Information Theoretic View of Entity Based Coherence

Abstract

What makes a sequence of sentences feel coherent rather than disjointed? At the local level, coherence depends on two interacting mechanisms. Relational coherence captures how adjacent sentences are logically connected: one may explain, contrast with, elaborate on, or result from another. Entity-based coherence captures how sentences are linked through recurring entities, such as people, objects, places, or ideas. I focus especially on entity-based coherence. Coherent texts do not mention entities randomly: entities recur, shift roles, become pronominalized, or drop out in structured ways. When this pattern changes sharply, the text can feel locally unexpected unless a discourse relation licenses the shift. I propose to model this using information value, an information-theoretic generalization of surprisal. Given a context, we sample alternative next sentences from a neural language model and compare them to the actual next sentence. If the actual sentence differs strongly from the alternatives, it has high information value. The main contribution is to define this difference using interpretable entity-based distances. These distances compare whether the target and alternatives mention the same entities, introduce new entities, assign similar grammatical roles, or use similar referring forms such as pronouns, definite descriptions, and proper names. This allows us to estimate how unexpected a sentence is specifically in terms of entity coherence. Most empirical work is carried out on the GUM corpus, using its entity and discourse relation annotations. As sanity checks, these information value estimates will be evaluated against human reading times and acceptability judgements in dialogue and text.