September 30th, 2023

At the recent KONVENS 2023 meeting in Ingolstadt, Laura Zeidler presented her thesis and was selected for the 2023 edition of the
Bi-annual Best Thesis Award. award

Her Bachelor thesis
“An Interpreted Dynamic Testsuite for the Evaluation of Semantic Similarity Metrics used in Parsing and Generation”
addressed the problem of NLG evaluation in AMR-to-Text processing.

Due to the abstract nature of AMR, sentences generated from AMR graphs may yield distinct surface structures as long as they preserve the abstract meaning encoded by the input AMR. This variability presents a challenge for current NLG evaluation metrics.

To support the development of AMR-to-Text evaluation metrics, Laura adopted the concept of CHECKLISTs (Ribiero et al. 2020) and designed
a checkist of selected linguistic phenomena that an NLG metric must be able to judge for semantic equivalence to a given AMR graph.
The testsuite aims to determine strengths and weaknesses of existing and new AMR-to-text generation evaluation metrics.

Next to evaluating existing textual, graph-based and contextualized NLG evaluation metrics, Laura also developed a new, hybrid, metric called LCG inspired by Lexical Cohesion Graphs (Sporleder and Li 2009). It measures the similarity of sentence pairs via their AMR graphs, by building a lexical cohesion graph from the concept nodes in a sentence’s AMR, where the nodes are represented by the corresponding text tokens’ contextualized word embeddings.

Laura’s thesis work has been further extended and published jointly with her supervisors in the *SEM Conference at NAACL 2022 as
A Dynamic, Interpreted CheckList for Meaning-oriented NLG Metric Evaluation – through the Lens of Semantic Similarity Rating”.

Congratulations to Laura for her great thesis work and
winning the Best Student Thesis Award of the GSCL !