Training-free multimodal embeddings: extracting retrieval representations from a frozen vision-language model

Multimodal embedding models map images and text into a shared representation space, enabling cross-modal retrieval through nearest-neighbor search. State-of-the-art approaches typically obtain embeddings from the final layer of a contrastively fine-tuned Multimodal Large Language Model (MLLM) using a fixed token-pooling strategy. However, this choice of where and how to extract the embedding, which we refer to as the readout, is largely conventional. Prior work showing that intermediate layers and individual attention heads can capture specialized representations motivates a systematic study of alternative readouts. In this thesis, we investigate whether specific layer-head combinations within a frozen generative vision-language model can outperform standard final-layer pooling for cross-modal retrieval, without additional training. We exhaustively evaluate every attention head across all layers on the image tasks of the Massive Multimodal Embedding Benchmark (MMEB), using a split-task protocol in which readouts are selected on one subset of tasks and evaluated on the remaining tasks. Our results show that final-layer pooling is consistently outperformed by alternative readouts, with the best performance obtained from individual attention heads rather than full-layer representations. We further ask whether high-performing readouts can be identified without using retrieval performance for selection. Motivated by the Platonic Representation Hypothesis, we use Mutual K-Nearest Neighbor cross-modal alignment as a task-agnostic selection criterion. We find that this criterion closely tracks supervised retrieval accuracy and reliably identifies strong attention heads without access to downstream retrieval metrics. Concatenating the selected heads and applying all-but-the-top (ABTT) whitening yields the best-performing representation in all six configurations, exceeding the best single head available on the held-out tasks. Finally, we repeat the analysis across three models and two pooling strategies. The results show that both the head-level structure and the proposed selection criterion generalize consistently across model scales and pooling strategies.