New GTFormer architecture replaces Transformer attention to process 4 million tokens

Researcher Elisabeth Fink posted a preprint paper to arXiv on October 27 introducing GTFormer, a language model architecture designed to replace Transformer self-attention. Tech Times reported that the new model uses a hierarchical graph neural network to achieve linear time and memory scaling. [1]

According to Tech Times, GTFormer achieves zero-shot length extrapolation from a training sequence length of 512 tokens up to 4 million tokens on a commodity GPU. This context window is more than 8,000 times longer than the sequences used during training, and the model scales without requiring additional fine-tuning. [1]

Traditional Transformer architectures rely on a multi-head attention mechanism where tokens are contextualized within a context window via parallel attention computations. In theoretical computer science, time complexity measures the amount of computer time an algorithm requires by counting its elementary operations. [2][4]

By replacing standard quadratic self-attention with a hierarchical graph network structure, GTFormer reduces the computational burden to linear O(N) memory complexity. Tech Times noted that the approach represents a fundamental redesign of the core language model backbone. [1]