Composing and Removing Concepts in Diffusion Models through Task Arithmetic

Stable Diffusion models can be adapted to specific visual concepts from a handful of examples through fine-tuning. Task arithmetic represents each adaptation as a task vector, the difference between the fine-tuned and the pre-trained parameters. Adding a single vector to the pre-trained checkpoint recovers the corresponding fine-tuned model; combining or subtracting vectors aims to compose or remove learned capabilities without further training. Prior work has studied task arithmetic in image classification and language tasks. Its behaviour when composing similar visual concepts in a text-to-image diffusion model remains less clear: independently learned concepts may interfere when their task vectors are combined, producing visual hybrids or causing one concept to suppress another. This thesis aims to deepen our understanding of task arithmetic in text-to-image generation by addressing three questions about the disentanglement of generative task vectors. First, can vectors learned separately on similar classes of the same dataset be composed while preserving the generation quality of the individually fine-tuned models and keeping the classes distinct? Second, can subtracting a vector from a composed model selectively remove the corresponding generation capability while preserving those contributed by the remaining vectors? Third, how do the fine-tuning procedure and textual conditioning influence these properties? To address these questions, the thesis fine-tunes a Stable Diffusion model on a dataset of closely related visual classes. The research first examines how different fine-tuning approaches affect the mergeability of the resulting task vectors. It then investigates the influence of different types of training prompts on this property. Each class vector is learned independently and evaluated through composition and subtraction. Across the four selected class pairs, we find that requested-class recognition under averaged composition is higher with attention-only adaptation for the recipes using class-specific prompts, with and without identifiers. The same pattern is observed with generic wording and an identifier on all four evaluated class pairs.