Adapting Pre-trained Neural Networks for Dynamic Inference: from Residual Weight Generation to Slimmable Fine-tuning

The rigid computational requirements of traditional neural networks often conflict with the need for hardware-agnostic deployment across a broad range of diverse real-world systems. Frameworks like Slimmable Neural Networks offer a dynamic alternative within a single model but traditionally require training from scratch, failing to leverage the presence of existing pre-trained weights. This work investigates adapting pre-trained models for runtime flexibility. We start from Neural Metamorphosis, a generative approach that uses hypernetworks to produce weights for various sub-network configurations. To address scalability issues in generating weights for larger architectures, we incorporate ideas from classical pruning and reframed the task as a residual problem. Instead of generating weights from scratch, the hypernetwork produces an additive change to be applied to the pre-trained weights, making the optimization landscape more tractable. Building on the insights of this first part, we transition to a strategy of direct fine-tuning. Specifically, we explore the possibility of adapting static, pre-trained neural networks into dynamic ones by applying the training principles of Slimmable Networks. This progression highlights the challenges of scaling weight generation and evaluates the viability of converting standard pre-trained models into dynamic ones without the necessity of full retraining.