Test-Time Computation in Sequence Models: Adaptation, Scaling, and Reasoning

Module Description

Participants

All advanced CL Bachelor students and all CL master students. Students from MSc Data and Computer Science or MSc Scientific Computing with Field of Application Computational Linguistics are welcome after getting permission from the lecturer. MSc Scientific Computing students can only take the course as HS for 8 LP.  If the seminar should be oversubscribed, CL students will have priority.  

Prerequisites for Participation

Module Prerequisites

Introduction to Neural Networks or equivalent

Assessment

• Presentation
• Critical Reviewing of 2 other Presentations
• Second Presentation (depending on member count) OR Project

Contents

Neural sequence models are typically trained once and then deployed with fixed parameters. At test time, what you trained is what you get. A growing body of work challenges this by investing additional computation at inference. This takes many forms: updating model weights on the fly to adapt to new data (test-time training), dynamically adjusting how many computational steps each input receives (as in Universal Transformers), or training models via reinforcement learning to produce longer, more deliberate reasoning chains. Recent architectures like Titans and TTT layers, together with reasoning models like DeepSeek-R1, have brought test-time computation into the spotlight. This course covers works along this spectrum, from architectural ideas that blur the line between training and inference, to strategies for making models think harder when it actually matters.

Literature

» More Materials