Transformer

The transformer is a neural network architecture introduced in the landmark 2017 paper "Attention Is All You Need" by Vaswani et al. at Google. It replaced recurrent and convolutional approaches to sequence modeling with a purely attention-based mechanism, enabling unprecedented parallelization during training and establishing the architectural foundation for virtually all modern large language models, including GPT, Claude, Llama, and Gemini.

The core innovation of the transformer is the self-attention mechanism, which allows each element in a sequence to attend to every other element simultaneously, computing relevance scores that determine how much influence each position has on the representation of every other position. This parallel computation eliminates the sequential bottleneck of recurrent networks, where information must flow step by step through the sequence, and enables transformers to capture long-range dependencies effectively.

A standard transformer consists of an encoder and decoder, each built from stacked layers of multi-head self-attention and feed-forward neural networks, with residual connections and layer normalization. In practice, many modern models use only one half: encoder-only architectures like BERT excel at understanding tasks such as classification and extraction, while decoder-only architectures like GPT and Llama are optimized for text generation. Encoder-decoder models like T5 handle sequence-to-sequence tasks such as translation and summarization.

Scaling transformers to billions and trillions of parameters has driven the emergence of increasingly capable foundation models. Key innovations enabling this scale include FlashAttention for memory-efficient attention computation, rotary position embeddings for handling long sequences, grouped query attention for inference efficiency, and mixture-of-experts architectures that increase model capacity without proportionally increasing computation. Understanding transformer architecture is essential for practitioners working on model selection, fine-tuning, optimization, and deployment of modern AI systems.