Generative Multimodal Analysis (GMA) for Learning Process Data Analytics

Abstract

Abstract

This paper introduces Generative Multimodal Analysis (GMA), a novel method designed for utilizing Artificial Intelligence (GenAI) in the analysis of multimodal data derived from learning processes. The method is encapsulated in a systematic framework that integrates and optimizes GenAI technology with multimodal large language models (MLLMs) for application in multimodal learning analytics. The recent emergence and advancement of GenAI, particularly MLLMs, has opened new avenues for the automated interpretation and meaningful analysis of varied data sources. Current research in the field has sightseen diverse applications of GenAI in transforming learning and teaching practice. However, there is a noticeable gap in systematic methodologies for applying GenAI to scrutinize learning process data. This paper aims to bridge this gap by proposing the GMA method in the sphere of multimodal learning analytics with learning process data. In addition to the proposed methodological framework, this study also proposes an operational prototype for the practical implementation of GMA. This prototype serves as a tool for examining multimodal data in learning processes. To demonstrate the applicability and effectiveness of our proposed method, we conducted and presented a case study. Our approach offers essential guidance for learning scientists and educational technology application developers, reflecting the contemporary trends and needs in educational technologies. By providing a structured, innovative approach for employing GenAI in learning process data analysis, this study contributes significantly to the advancement of learning analytics methods.