From Relative to Rotary: Evolving Positional Encodings in Graph Language Models

Abstract

Graph data often uses natural language as node and edge features, making it important to have models capable of language understanding and graph reasoning. Thus, we presented Graph Language Models (GLMs) in previous work. GLMs adapt non-invasive alterations to a Language Model's (LM) architecture to convert it to a Graph Transformer (GT). The new GT architecture endows graph inductive biases to the GLM, while the pretrained LM parameters provide a strong foundation for understanding language. Thus GLMs are capable of reasoning over graph-structured data with natural language features.

In our previous work, we build GLMs from the T5 encoder, which uses so-called relative positional encoding. Newer LMs, such as LLaMa, Mistral or Falcon, use Rotary Positional Embedding (RoPE) instead, which are more efficient for language modeling but pose challenges in the GLM context. In this work, we extend the GLM framework to support LLaMa 3 by adapting RoPE to work in graph settings. We also present preliminary results demonstrating the feasibility and potential of this approach.