Homer meets BERT

Abstract

Recent advances in NLP have led to the creation of powerful language models for a wide range of languages, including Ancient Greek and Latin. While prior work on Classical languages unanimously chooses BERT, in our work, we create four language models for Ancient Greek that vary along two dimensions to study their versatility for tasks of interest for Classical languages. We explore encoder-only and encoder-decoder architectures using RoBERTa and T5 as strong model types, and create, for each of them, a monolingual Ancient Greek and a multilingual instance including Latin and English. We evaluate all models on a range of morphosyntactic tasks, and introduce a probing task that investigates the knowledge acquired by models trained on Classical texts. Furthermore, we assess strengths and weaknesses of our models along both dimensions and exploit the decoding abilities of T5 for lemmatization. Our results show that our models provide significant improvements over existing models. The comparative model analyses can inform future research in designing language models for Classical languages, including the development of novel generative tasks. The pre-training corpora established for our Ancient Greek models will be made available to support the creation of a larger comparable model zoo for Ancient Greek.