Enhancing gen AI with multimodal RAG systems

Abstract

Organizations can achieve better decision-making and operational efficiency and regulatory compliance through artificial intelligence analysis of unstructured financial data. The current AI systems either concentrate on text-based processing alone or neglect to handle multimodal financial information, including tables and graphs and scanned documents which results in fragmented insights and limited usability. The research develops a Multimodal Retrieval-Augmented Generation (RAG) system using only open-source tools for its design and implementation and evaluation. The study addresses a literature gap by using Action Design Research (ADR) to develop and improve the system through iterative refinement within a realistic financial environment. The research evaluates open-source large language models (LLMs) through systematic benchmarking to demonstrate their strengths and weaknesses in financial analytics tasks. The evaluation results show that open-source models like LLMs Phi-3 3.8B, DeepSeek-R1 7B, LLaVA-7B and Gemma-3 4B demonstrate good performance in retrieving structured data but encounter substantial obstacles when analyzing quantitative trends from multimodal input sources. The final artefact shows that a Multimodal RAG system built with proper construction improves retrieval accuracy and operational productivity yet requires future development for domain-specific fine-tuning and agent orchestration and explainability to reach enterprise standards