Ruprecht-Karls-Universität Heidelberg
Institut für Computerlinguistik

Bilder vom Neuenheimer Feld, Heidelberg und der Universität Heidelberg

Evaluation of NLP Systems


Studiengang Modulkürzel Leistungs-
BA-2010 AS-CL, AS-FL 8 LP
BA-2010[100%|75%] CS-CL 6 LP
BA-2010[50%] BS-CL 6 LP
BA-2010[25%] BS-AC, BS-FL 4 LP
Dozenten/-innen Julius Steen
Veranstaltungsart Proseminar/Hauptseminar
Sprache English
Erster Termin 19.10.2022
Zeit und Ort Mittwochs, 15:15-16:45, INF 327 / SR 3
Commitment-Frist tbd.


  • Completion of Programming I and Introduction to Computational Linguistics or similar introductory courses
  • Mathematical Foundations of Computational Linguistics (or equivalent) is heavily suggested


  • Active Participation
  • Presentation
  • Second presentation, term paper or implementation project


The great strides in Natural Language Processing in recent times have, to a great degree, been driven by the ready availability of large scale evaluation tasks and accompanying metrics. These tasks ideally allow for an unbiased comparison of different approaches, help us to quantify progress, and guide future research. However, as benchmarks get beaten in ever shorter timeframes, it becomes increasingly clear that our evaluation practices lack behind system development. In this seminar, we are going to study the problem of evaluation in NLP from a broad perspective. Topics of this seminar will include:

  • Automatic Evaluation Metrics
  • Protocols for eliciting human judgements of output quality
  • Acquisition of suitable evaluation datasets
  • Pitfalls and best practices in the evaluation of evaluation results

In the end, this seminar aims both to give participants a good overview over current practices and challenges of evaluation in our field, as well as to help make them more informed decisions for their own evaluation setups.


Will be announced at the beginning of the course.

» weitere Kursmaterialien

zum Seitenanfang