Bilder vom Neuenheimer Feld, Heidelberg und der Universität Heidelberg

Lehrveranstaltungen
heiCO
Ressourcen	Fachschaft
Studien-FAQ	Technik-FAQ

Evaluation of NLP Systems

Kursbeschreibung

Studiengang	Modulkürzel	Leistungs- bewertung
BA-2010	AS-CL, AS-FL	8 LP
BA-2010[100%\|75%]	CS-CL	6 LP
BA-2010[50%]	BS-CL	6 LP
BA-2010[25%]	BS-AC, BS-FL	4 LP
Master	SS-CL, SS-TAC, SS-FAL	8 LP

Dozenten/-innen	Julius Steen
Veranstaltungsart	Proseminar/Hauptseminar
Sprache	English
Erster Termin	~~19.10.2022~~ 26.10.2022
Zeit und Ort	Mittwochs, 15:15-16:45, INF 327 / SR 3
Commitment-Frist	tbd.

Teilnahmevoraussetzungen

Completion of Programming I and Introduction to Computational Linguistics or similar introductory courses
Mathematical Foundations of Computational Linguistics (or equivalent) is heavily suggested

Leistungsnachweis

Active Participation
Presentation
Second presentation, term paper or implementation project

Inhalt

The great strides in Natural Language Processing in recent times have, to a great degree, been driven by the ready availability of large scale evaluation tasks and accompanying metrics. These tasks ideally allow for an unbiased comparison of different approaches, help us to quantify progress, and guide future research. However, as benchmarks get beaten in ever shorter timeframes, it becomes increasingly clear that our evaluation practices lack behind system development. In this seminar, we are going to study the problem of evaluation in NLP from a broad perspective. Topics of this seminar will include:

Automatic Evaluation Metrics
Protocols for eliciting human judgements of output quality
Acquisition of suitable evaluation datasets
Pitfalls and best practices in the evaluation of evaluation results

In the end, this seminar aims both to give participants a good overview over current practices and challenges of evaluation in our field, as well as to help make them more informed decisions for their own evaluation setups.

Literatur