Ruprecht-Karls-Universität Heidelberg
Bilder vom Neuenheimer Feld, Heidelberg und der Universität Heidelberg
Siegel der Uni Heidelberg

Title: Coherence Measures do not Predict Summary Coherence

Speaker: Julius Steen (ICL)


Automatic evaluation measures for summary coherence hold the promise of enabling more comprehensive evaluation of summary linguistic quality as well as improving summarizer coherence. However, there is no clear consensus on which coherence measures are most appropriate for this task. We thus investigate the efficacy of both general linguistic quality measures and specialized coherence measures for evaluating summary coherence. Unlike prior work, we evaluate on a large set of recent summarizer outputs and introduce an evaluation protocol that focuses on fine-grained coherence judgements instead of coarse-grained system performance. We show that none of the available measures can satisfactorily predict coherence judgements.
zum Seitenanfang