At the recent KONVENS 2023 meeting in Ingolstadt, Laura Zeidler
presented her thesis and was selected for the 2023 edition of the
Bi-annual Best Thesis Award. 
Her Bachelor thesis
“An Interpreted Dynamic Testsuite for the Evaluation of Semantic Similarity Metrics used
in Parsing and Generation”
addressed the problem of NLG evaluation in AMR-to-Text processing.
Due to the abstract nature of AMR, sentences generated from AMR graphs may yield distinct surface structures as
long as they preserve the abstract meaning encoded by the input AMR. This variability presents a challenge for
current NLG evaluation metrics.
To support the development of AMR-to-Text evaluation metrics, Laura adopted the concept of CHECKLISTs (Ribiero
et al. 2020) and designed
a checkist 
of selected linguistic phenomena that an NLG metric must be
able to judge for semantic equivalence to a given AMR graph.
The testsuite aims to determine strengths and weaknesses of existing and new AMR-to-text generation evaluation
metrics.
Next to evaluating existing textual, graph-based and contextualized NLG evaluation metrics, Laura also developed a new, hybrid, metric called LCG inspired by Lexical Cohesion Graphs (Sporleder and Li 2009). It measures the similarity of sentence pairs via their AMR graphs, by building a lexical cohesion graph from the concept nodes in a sentence’s AMR, where the nodes are represented by the corresponding text tokens’ contextualized word embeddings.
Laura’s thesis work has been further extended and published jointly with her supervisors in the *SEM
Conference at NAACL 2022 as
“A Dynamic,
Interpreted CheckList for Meaning-oriented NLG Metric Evaluation – through the Lens of Semantic Similarity
Rating”.
Congratulations to Laura for her great thesis work and
winning the Best Student Thesis Award of the GSCL
!
