Scaling Dynamics in Multimodal Video-LLMs: A Scalability Analysis of Vision-Language Adapters

Multimodal Large Language Models (MLLMs) have achieved remarkable results in interpreting and contextualizing static images, but extending these capabilities to videos still remains a computational challenge. The main bottleneck is the high density of video data, the integration of both spatial and temporal information produces a volume of data that exceeds the input limits of traditional architectures. To solve this, Video-LLMs require projection strategies to efficiently compress the input sequence while preserving as much information as possible. This thesis presents a comprehensive scalability analysis of temporal projection methods for mapping video features into the LLM space. The research was conducted using the LLaVA architecture which combines a CLIP-based vision encoder with a Vicuna-based language model. The study systematically evaluates three distinct projection strategies: a baseline Mean Pooling, a Standard Perceiver, and a novel Patch-Wise Perceiver. These architectures are trained and evaluated across progressively scaled subsets of the InternVid dataset, ranging up to one million videos. By comparing the performance of these projection strategies under varying data scales, this work provides empirical insights regarding the trade-offs between architectural complexity and computational efficiency in multimodal alignment.