---
id: transformer
kind: breakthrough
title: The Transformer — Attention Is All You Need
updated: 2026-10-05
sources:
  - id: transformer-paper
    url: https://arxiv.org/abs/1706.03762
    title: Attention Is All You Need (Vaswani et al., 2017)
    checked: 2026-10-05
  - id: vaswani-profile
    url: https://www.ashishvaswani.com/
    title: Ashish Vaswani — official site
    checked: 2026-10-05
related:
  - breakthroughs/imagenet-alexnet.md
  - breakthroughs/alphafold.md
contributors: []
---
## What was blocked
Sequence-to-sequence systems commonly relied on recurrent or convolutional networks. Recurrence imposes sequential computation across positions, limiting parallelization during training. Prior attention mechanisms typically supplemented those architectures rather than replacing recurrence altogether. [Source: transformer-paper]

## What changed
The 2017 paper proposed an encoder–decoder architecture based solely on attention, dispensing with recurrence and convolutions. It introduced multi-head self-attention, positional information, and feed-forward layers. On WMT 2014 English–German and English–French translation, the authors reported state-of-the-art BLEU results at the time, with training that was more parallelizable and less costly than compared systems. These are the authors' reported experimental results, not a claim that attention is universally superior. [Source: transformer-paper]

## Contributors and lineage
The paper credits **all eight authors**: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. The listed people profiles here are a deliberately partial linked set, not a complete roster or contribution attribution. The paper itself situates its method among sequence transduction, attention, and self-attention work; the architecture emerged from a team and prior research, not a single-person invention. [Source: transformer-paper]

## Why it mattered — interpretation
The key unlock was not “attention was invented,” but a clean architecture that made global token-to-token interaction and parallel training central. This proved reusable beyond machine translation and became a major substrate for later language and multimodal models. That broader historical influence is synthesis; the original paper directly supports the design and translation result, not every later consequence.

## Practical leverage
When a workload has long sequences and enough accelerator parallelism, compare attention-based models against recurrent/convolutional baselines for quality, throughput, memory, and data needs. For agent work, the leverage is architectural: pretraining one general sequence model can support many downstream tasks. Validate latency, cost, retrieval needs, and grounding for the actual application.

## Limits and open questions
Self-attention's cost and memory grow quadratically with sequence length in the standard formulation. Parallel training does not remove data, compute, evaluation, or deployment constraints. The original translation benchmarks do not establish reasoning, truthfulness, or safety. Later architectures and efficient-attention methods address different tradeoffs; this note is not a survey of them.

## Sources
- [transformer-paper] The original paper and author list: https://arxiv.org/abs/1706.03762
- [vaswani-profile] Ashish Vaswani's official site: https://www.ashishvaswani.com/