Seminar: Adaptation and Fine-Tuning of Foundation Models

Foundation models are pretrained once on broad and heterogeneous data distributions; however, real-world deployment typically requires reliable performance in narrow, specialized domains. This seminar will discuss fundamental and recent approaches to address the following question: How do we specialize foundation models to a target domain under computational and data constraints?

To explore this question, we will read and discuss basic and state-of-the-art methods on how to systematically adapt and fine-tune large models through parameter-efficient fine-tuning, prompt engineering, test-time adaptation, context augmentation and many more.

For an overview, see this paper: Zhang et al. Parameter-Efficient Fine-Tuning for Foundation Models (arxiv 2025)

Requirements

Familiarity with foundations of deep learning, including transformer architectures and in-context learning.

Topics

We will read the following papers:

Parameter-Efficient Fine-Tuning

• Houlsby et al. Parameter-Efficient Transfer Learning for NLP (NeurIPS 2025)
• Guo at al. Parameter-Efficient Transfer Learning with Diff Pruning (ACL 2021)
• Zhao et al. Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning (ICLR 2024)
• Hu at al. LoRA: Low-Rank Adaptation of Large Language Model (ICLR 2022)
• Mitchell et al. Fast Model Editing at Scale (ICLR 2022)

Prompt Engineering

• Chen et al. Auto-prompt: Eliciting knowledge from language models with automatically generated prompts (EMNLP 2020)
• Lester et al. The Power of Scale for Parameter-Efficient Prompt Tuning (ACL 2020)
• Li et al. Prefix-Tuning: Optimizing Continuous Prompts for Generation (ACL 2021)
• Guo et al. Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers (ICLR 2024)

Test-Time Adaptation

• Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020)
• Wei et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (NeurIPS 2022)

How the seminar will look like?

We will regularly throughout the semester. In the first few weeks, we will start with introductory lectures on adapting foundation models and how to critically review and present research papers. After that, we will have several sessions with presentations, each followed by discussions. In the end we will have one more concluding session.

Other Important information

Grading/Presentations: Grades will be based on your presentation, slides and active participation. Further details will be discussed in the introductory sessions.