Seminar: Adaptation and Fine-Tuning of Foundation Models

Foundation models are pretrained once on broad and heterogeneous data distributions; however, real-world deployment typically requires reliable performance in narrow, specialized domains. This seminar will discuss fundamental and recent approaches to address the following question: How do we specialize foundation models to a target domain under computational and data constraints?

To explore this question, we will read and discuss basic and state-of-the-art methods on how to systematically adapt and fine-tune large models through parameter-efficient fine-tuning, prompt engineering, test-time adaptation, context augmentation and many more.

For an overview, see this paper: Zhang et al. Parameter-Efficient Fine-Tuning for Foundation Models (arxiv’25)

Requirements

Familiarity with foundations of deep learning, including transformer architectures and in-context learning.

Topics

| Date | Content | |—————|——————–| | 22.04.2026 | intro I | | 29.04.2026 | intro II | | tba | sessions |

A non-complete paper pool for the seminar.

Parameter-Efficient Fine-Tuning

• Houlsby et al. Parameter-Efficient Transfer Learning for NLP (NeurIPS’25)
• Guo at al. Parameter-Efficient Transfer Learning with Diff Pruning (ACL’21)
• Zhao et al. Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning (ICLR’24)
• Hu at al. LoRA: Low-Rank Adaptation of Large Language Model (ICLR’22)
• Mitchell et al. Fast Model Editing at Scale (ICLR’22)

Prompt Engineering

• Chen et al. Auto-prompt: Eliciting knowledge from language models with automatically generated prompts (EMNLP’20)
• Lester et al. The Power of Scale for Parameter-Efficient Prompt Tuning (ACL’20)
• Li et al. Prefix-Tuning: Optimizing Continuous Prompts for Generation (ACL’21)
• Guo et al. Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers (ICLR’24)

Reinforcement Learning with Human Feedback

• Ouyang et al. Training language models to follow instructions with human feedback (NeurIPS’22)

Test-Time Adaptation

• Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS’20)
• Wei et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (NeurIPS’22)

How the seminar will look like?

We will regularly throughout the semester. In the first few weeks, we will start with introductory lectures on adapting foundation models and how to critically review and present research papers. After that, we will have several sessions with presentations, each followed by discussions. In the end we will have one more concluding session.

Other Important information

Grading/Presentations: Grades will be based on your presentation, slides, active participation and a short report. Further details will be discussed in the introductory sessions.