top of page

Multimodal Information Extraction for Scalable User Research

This project explored how LLMs and emotional analysis can help structure video-based diary data to better understand user experiences and behaviors over time.

I developed a multimodal information extraction method that analyzes video diaries to capture both measurable progress and individuals’ perceived progress. By transforming unstructured video content into structured insights, the approach enables scalable data collection and analysis within specific video formats, providing researchers with a new way to study real-world user perceptions and behavioral patterns.​

Background and Goals

We have some key goals in order to guide our user-centered research.

Background

  1. In skin-related treatments, "noticeability" means visible changes to the naked eye. From a business perspective, noticeability with consumers means managing their expectations and helping them understand potential outcomes and the improvement timeline.​

  2. Individuals share their treatment journey online through social-technical systems, such as YouTube, to increase awareness, exchange information, and provide community support. This user-generated content provides a valuable source of real-world data for user research.

  3. User research with video-based diaries generates large volumes of unstructured data, making it challenging for researchers to systematically analyze user experiences and behavioral patterns over time.

Generative Research Goals:

Research Methods and Process

  1. How can LLM-assisted analysis help researchers identify patterns in user perceptions, emotions, and behavioral changes at scale?​​​

  2. How can insights from video diaries help businesses better understand consumer expectations around treatment progress and timelines?

  3. How does perceived progress shared in video diaries align with measurable treatment progress (i.e., visible changes)?

​

Data Collection

  • Collected and curated 350+ health-related video diaries from YouTube (vlogs) related to patients dealing with eczema treatment. ​​

      Methods: web scraping, YouTube API, keyword searching 

​

Multimodal Information Extraction

  • Developed a multimodal analysis method to extract structured information from video diaries, including speech content, visual cues, and contextual metadata. Leveraged LLMs, specifically OpenAI GPT API, to identify users' perceptions of treatment progress, while emotional analysis was used to detect patterns in users’ emotional responses throughout their treatment journeys.

     

  • Analyzed both measurable progress (visible skin changes) and perceived progress expressed by individuals in their narratives to understand how users interpret treatment outcomes over time.

       

       Tech: Transformer-based emotion models, LLMs (GPT), sentiment analysis, Whisper, Librosa, OpenCV

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Longitudinal Pattern Analysis

  • Conducted longitudinal analysis across video diaries to identify behavioral patterns and treatment retention trends over time.

       Methods: Statistical analysis (Pearson correlation), deep learning (Autoencoder), data visualization

Screenshot 2026-03-15 at 10.19.41 PM.png

Findings and Crucuial Insights

  • Identified socio-emotional patterns throughout treatment journeys

       Emotional analysis showed that users often experienced early uncertainty, mid-stage frustration, and later satisfaction,         suggesting that treatment experiences evolve emotionally over time.

 

  • Demonstrated that multimodal analysis can structure unstructured video diaries​

       Combining audio analysis, visual cues, and contextual metadata enabled the conversion of large volumes of video              diaries into structured insights for user research.

​

  • Enabled scalable analysis of real-world user experiences
    The method allowed researchers to analyze hundreds of video diaries and thousands of related entries, revealing behavioral and perception patterns that would be difficult to detect through manual analysis.

 

  • Highlighted the importance of expectation management in treatment communication

       Insights showed that misunderstandings about treatment timelines and noticeability can influence user satisfaction,             emphasizing the need for clearer communication about expected outcomes.

​

  • Discovered that users rely on visual proof to evaluate treatment success
    Video diaries frequently focused on before–and–after comparisons and close-up visual documentation, indicating that visible changes play a critical role in how users judge treatment effectiveness.

Business Impacts

  • Built a data pipeline that structured video-based diaries to support user research.

  • Informed product and communication strategies by revealing how users perceive treatment outcomes over time.

  • Supported data-driven decision-making for health and skincare research teams.

  • Reduced manual effort in analyzing unstructured video data through multimodal AI-assisted extraction.

  • R&D team at Procter & Gamble Company

Timeline

May 2023 - August 2023

bottom of page