Towards behaviour-aware multimodal video summarization : integrating visual, audio, and textual cues for human-centric content analysis

Video summarization models for human-centric content face critical challenges in capturing behavioural cues that convey essential interpersonal communication signals, as existing approaches either rely solely on visual features or employ basic multimodal combinations that neglect synchronized behavioural signals across modalities. This thesis proposes a behaviour-aware multimodal video summarization pipeline that integrates visual, audio, and textual modalities to identify moments of communicative significance through synchronized behavioural signals including gestures, facial expressions, and vocal prosody. The pipeline employs two complementary approaches: a transformer-based framework with cross-modal attention mechanisms and a heuristic approach utilizing multimodal bonus words, terms emphasized across multiple modalities that improve semantic relevance and expressive clarity of summaries. To address dataset scarcity, a scalable pseudo-ground truth generation pipeline using large language models (LLMs) enables automated evaluation without human annotations. Experimental validation on the ChaLearn First Impressions dataset demonstrates substantial performance improvements over existing state-of-the-art methods across multiple evaluation metrics. Comprehensive analysis confirms that multimodal integration significantly outperforms individual modalities, while temporal alignment emerges as crucial for maintaining narrative coherence and summary quality. This work contributes a novel methodology for preserving both semantic content and behavioural expressiveness in video summarization, enabling applications in educational technology, content analysis, and human-computer interaction.