Seminar: AutoML in the Age of Large Pre-trained Models

Have you heard about AutoML? Do you wonder what LLMs can do for AutoML and vice versa? Then this seminar is for you!

TL;DR AutoML aims to support ML users (and ML researchers) by automating parts of the ML workflow. In this seminar we will read recent research papers in the field of AutoML with a focus on methods for and with LLMs and pre-trained models. For a detailed overview, see this paper.

Why should you attend this seminar?

Besides practicing your scientific communication skills, you will also

• learn about key contributions in the field of AutoML
• be able to discuss recent research on AutoML with and for large pre-trained models
• gain experience in reading, understanding and presenting research papers

Requirements

We strongly recommend that you know the foundations of machine learning and deep learning, including modern neural architectures including transformer models. Ideally, you also have some experience in applying ML to get the most out of this seminar.

Topics

The seminar focuses on understanding the underlying concepts of modern AutoML methods. Since this field is ever-growing, in this semester, we will focus on only a few topics as stated above.

Here is a tentative meeting schedule and a tentative paper list:

LargeModels for AutoML

1. [OptFormer] Chen et al. Towards learning universal hyperparameter optimizers with transformers (NeurIPS’22)
2. [LLM4BO] Liu et al. Large language models to enhance Bayesian optimization (ICLR’24)
3. [TabPFN] Hollman et el. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second (ICLR’23)
4. [CAAFE] Hollman et al. Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering (NeurIPS’24)
5. [MLAgent] Huang et al. Benchmarking Large Language Models as AI Research Agents (arxiv’23)
6. [GPT4NAS] Zheng et al. Can GPT-4 Perform Neural Architecture Search? (arxiv’23)
7. [GPT-NAS] Yu et al. GPT-NAS: Evolutionary Neural Architecture Search with the Generative Pre-Trained (arxiv’22)

AutoML for Large Models

1. [QuickTune] Arango et al. Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How (ICLR’24)
2. [Bandits4LLMs] Xia et al. Which LLM to Play? Convergence-Aware Online Model Selection with Time-Increasing Bandits (WWW’24)
3. [EvoPrompt] Guo et al. Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers (ICLR’24)

How the seminar will look like?

We will meet each week (with a few exceptions). In the first few weeks, we will start with introductory lectures on automated machine learning, Bayesian optimization, neural architecture search and how to critically review and present research papers. After that, each week, we will have presentations, followed by discussions.

Other Important information

Registration: Please register on ILIAS. The number of participants is limited and the registration opens on March 29th, noon. There will be a waiting list (please unregister to let other people take your place). Please come to the first lecture even if you are still on the waiting list. If you’re enrolled and don’t show up, your spot will be freed for someone on the waiting list.

Grading/Presentations: Grades will be based on your presentation, slides, active participation and a short report. Further details will be discussed in the intro session.