Exploring Fine-Tuning Information Transfer via Activation Space

Fine-tuning enables pre-trained models to acquire capabilities tailored to specific downstream tasks. However, the parameter updates produced by this process are closely tied to the model in which they were learned, making them difficult to transfer to a different checkpoint or architecture. This thesis investigates whether the task-specific effects of fine-tuning can instead be transferred through activation space. The proposed approach describes fine-tuning in terms of the changes it induces in the source model’s representations. Rather than transferring a parameter update directly, it identifies the task-relevant effect of these changes and translates it into a correction for the target model’s representations. A predictor is then trained to generate the correction from the target model’s own activations. Once trained, this predictor allows the target model to operate without access to the source model at inference time, while keeping the target backbone frozen. The experimental results suggest that this form of activation-space transfer can improve the target model’s performance across different model configurations. They also reveal a distinction between transferring a task-specific effect and learning to reproduce it using only target activations: the latter depends on the amount and quality of the available calibration data. Taken together, these findings suggest that activation-space corrections can reproduce part of the task-specific effect of source fine-tuning in a target model, without directly transferring the source parameter update.