Disenchanting AI: A Pseudo-Workshop on Fallacious LLM Patterns

Module Description

Participants

All advanced CL Bachelor students and all CL master students. Students from MSc Data and Computer Science or MSc Scientific Computing with Field of Application Computational Linguistics are welcome after getting permission from the lecturer. MSc Scientific Computing students can only take the course as HS for 8 LP.  If the seminar should be oversubscribed, CL students will have priority.

Prerequisites for Participation

• Module Prerequisites
• Completion of Introduction to Computational Linguistics, Introduction to Programming
• Mathematical Foundations of Computational Linguistics or Programming II heavily suggested
• Solid understanding of machine learning (Statistical Methods for NLP, Introduction to Neural Networks or similar) and how to generally work of NLP projects (ExpML, SWP, Thesis, other projects...)

Assessment

For everyone:

• 1 short paper (5 pages)
• 2 reviews of other students' papers
• 1 poster for the final presentation

For HS only:

• 10min oral exam about the work

Content

Large language models (LLMs) are often presented and perceived as capable and highly intelligent systems. However, a growing body of research reveals a plethora of ways in which these models fall short of correctly solving simple tasks by being easily manipulated, committing logical fallacies or reflecting cognitive biases. These patterns can be both: akin to human flaws, possibly inherited through pre-training, or uniquely LLM-specific and unlike anything known from human behavior. In this seminar, we set out to further disenchant the myth of the super-humanly intelligent language model by designing small evaluations to trigger and investigate these flaws.

The seminar is structured as a small-scale simulation of a conference or workshop submission cycle: Students will begin working on a project and final report as early as the second week of the lecture period. At the halfway point, reports in their current state will be exchanged anonymously for peer review and rebuttals. During the final weeks, we will hold one or two sessions of (digital) poster presentations. HS students will additionally undergo a 10-minute oral exam in which their work is discussed. There are no further requirements during the lecture-free period.

The sessions themselves are of two types: The first is a series of presentations by the lecturer on practical research skills, including an introduction to the topic, a discussion of the research landscape, how to read and write research papers, and how to write reviews and rebuttals. The second category is co-working sessions, in which students work on their projects and papers while having the chance to ask for advice and feedback on their current progress. No general reading outside the scope of the individual project is required for any session.

» weitere Kursmaterialien