Listen: https://serve.buildtall.systems/b0a5f7bddf20f34de5323ee86165f37fa1ee1d28298ee3407fb14c8754c79566.mp3
Source: https://arxiv.org/pdf/1706.03762
This episode covers Attention Is All You Need, a paper by Ashish Vaswani and seven co-authors from Google Brain, Google Research and the University of Toronto.
The paper's own header says it was prepared for the 31st Conference on Neural Information Processing Systems, held in Long Beach, California, in 2017. According to OpenAlex, its record is a preprint posted to arXiv, the repository run by Cornell University, dated June 12, 2017, and the work is not retracted. The copy read here is arXiv version 7, dated August 2023. The paper takes on the problem of turning one sequence of symbols into another, such as an English sentence into its German translation. The strongest systems of the time relied on recurrent networks, which read a sentence one step at a time. The authors argue that this step-by-step process blocks parallel computation and makes it harder to link words that sit far apart. Their answer is the Transformer, which the abstract describes as a model "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely." The work is an experimental machine learning study. It proposes a new model design and then tests it against published systems. The main data are the WMT 2014 English-to-German set, with about 4.5 million sentence pairs, and the larger English-to-French set of 36 million sentences. Quality is scored with BLEU, a standard automatic measure that compares machine output with human reference translations. The abstract reports 28.4 BLEU on English-to-German, more than 2 points above the existing best results including ensembles, and a single-model score of 41.8 on English-to-French after 3.5 days of training on eight GPUs. The translation evidence is the strongest part of the paper, because it uses standard test sets and compares against established baselines. The base model's training run is reported as 12 hours on eight P100 graphics processors. The second test, English constituency parsing on the Wall Street Journal portion of the Penn Treebank, is thinner. The authors ran only a small number of tuning experiments, and the paper says the model performs surprisingly well without task-specific tuning. Its claim that attention heads learn syntactic and semantic behavior rests on the authors' inspection of attention patterns, not on a measured benchmark. Their suggestion that sinusoidal position encodings may let the model handle longer inputs is presented as a possibility that their experiments do not establish. The text supplied carries no funding statement, and the authors say their training and evaluation code is publicly available. According to OpenAlex, the work has 127,726 citations as of November 4, 2026. Yearly counts rose from 49 in 2017 and 981 in 2018 to a peak of 26,892 in 2024, followed by 25,961 in 2025 and 9,794 so far in 2026.
The introduction starts from the position that recurrent neural networks, especially long short-term memory and gated recurrent networks, have been firmly established as the state of the art for sequence modeling and transduction tasks such as language modeling and machine translation. The authors note that efforts have continued to push recurrent language models and encoder-decoder architectures further.
The paper then identifies the core limitation of these models. Recurrent models factor computation along the symbol positions of the input and output sequences. They generate a sequence of hidden states, each one a function of the previous hidden state and the input at the current position. The authors describe this as an "inherently sequential nature" that "precludes parallelization within training examples," and they say this matters more as sequences get longer, because memory constraints limit how many examples can be batched together. Recent work has improved computational efficiency through factorization tricks and conditional computation, but the authors argue that
the fundamental constraint of sequential computation, however, remains.
Next the paper turns to attention. Attention mechanisms have become an integral part of strong sequence models, because they let a model capture dependencies regardless of how far apart they sit in the input or output. In nearly all cases, though, attention is used alongside a recurrent network rather than replacing it.
The proposal follows from that gap. The authors present the Transformer, an architecture that dispenses with recurrence and relies entirely on an attention mechanism to draw global dependencies between input and output. They claim this allows far more parallelization, and they report that the model can reach a new state of the art in translation quality after training for as little as twelve hours on eight P100 GPUs.}
The background part of the paper starts from a goal shared by several earlier models: reducing sequential computation. The authors point to the Extended Neural GPU, ByteNet and ConvS2S, which use convolutional neural networks as their basic building block. Convolutional networks compute hidden representations in parallel for all input and output positions, rather than stepping through a sentence one word at a time. The problem, as the paper explains, is how far apart two words can be and still be connected. In these models, "the number of operations required to relate signals from two arbitrary input or output positions grows in the distance between positions, linearly for ConvS2S and logarithmically for ByteNet." The authors say "this makes it more difficult to learn dependencies between distant positions", meaning links between words that sit far apart in a sentence or passage. The paper then states what its own design changes:
In the Transformer this is reduced to a constant number of operations.
The authors admit a cost. Averaging attention-weighted positions reduces the effective resolution, and they counter this with multi-head attention, which the paper describes later. The background then defines self-attention, which the authors also call intra-attention. It is "an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence." In other words, a model looks across the words of one input to build a picture of that input. The paper notes that self-attention has already worked well in reading comprehension, abstractive summarization, textual entailment, and learning sentence representations that do not depend on a particular task. It also cites end-to-end memory networks, which use a recurrent attention mechanism rather than sequence-aligned recurrence. Those networks performed well on simple-language question answering and language modeling. The authors claim a first:
To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output.
A transduction model is one that turns an input sequence, such as a sentence in one language, into an output sequence, such as its translation. The paper says the Transformer does this without using recurrent networks or convolution. The section closes by saying the following sections will describe the Transformer, motivate the use of self-attention, and compare its advantages with those of earlier models.
Most of the neural systems that turn one sequence into another, such as translating a sentence, share a shape called an encoder-decoder. The paper states it directly:
Most competitive neural sequence transduction models have an encoder-decoder structure.
The encoder takes an input sequence of symbols, such as words, and maps it to a sequence of continuous representations, which are numerical vectors that stand for the input. The decoder then uses those representations to generate the output sequence one symbol at a time.
The authors add that these models are auto-regressive. Each new output symbol is produced with the symbols generated so far fed back in as extra input. The Transformer, the model this paper proposes, follows the same overall architecture. In the paper's words, it uses "stacked self-attention and point-wise, fully connected layers for both the encoder and decoder." Self-attention is a mechanism in which each position in a sequence weighs the other positions in that same sequence when building its own representation. A point-wise, fully connected layer applies the same small network to each position separately.
The encoder is "composed of a stack of N = 6 identical layers," so six copies of the same layer design, one feeding the next. Each encoder layer has two sub-layers. The first is a multi-head self-attention mechanism. The second is a simple, position-wise, fully connected feed-forward network. Around each of the two sub-layers the authors place a residual connection, which adds the sub-layer's input back to its output so that information can pass through unchanged. A layer normalization step follows. The paper writes that the output of each sub-layer is a layer normalization of the input plus the sub-layer's own function applied to that input.
The authors note a design choice that follows from this. "To facilitate these residual connections," all sub-layers in the model, and the embedding layers as well, produce outputs of dimension 512. Keeping one width throughout is what lets the input be added back at every step.
The decoder is also a stack of "N = 6 identical layers." It keeps the two sub-layers of each encoder layer and adds a third. That third sub-layer performs multi-head attention over the output of the encoder stack, which is how the decoder reads the input sequence. The decoder also uses residual connections and layer normalization around each sub-layer, as the encoder does.
The decoder's self-attention is changed in one important way. The authors "modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions." This masking is combined with a shift: the output embeddings are offset by one position. Together, the paper says, these ensure "that the predictions for position i can depend only on the known outputs at positions less than i." In plain terms, when the model predicts the word at a given position, it cannot draw on words that come after that position.
The paper builds its design around a general idea it calls attention, and it starts with a definition. In the authors' words,
An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors.
The output is a weighted sum of the values, and the weight on each value comes from a compatibility function that scores how well the query matches the corresponding key. Put simply, the model looks up the parts of its input that are relevant to a given query and blends them together.
The authors give their particular version a name, Scaled Dot-Product Attention. Its inputs are queries and keys of size d sub k, and values of size d sub v. The method takes the dot product of the query with every key, divides each result by the square root of d sub k, and applies a softmax function to turn those scores into weights on the values. In practice the queries are processed together, packed into one matrix, and the keys and values are packed into their own matrices. The output is then a softmax of the query matrix multiplied by the transposed key matrix, scaled by the square root of d sub k, and multiplied by the value matrix.
The authors note that the two most commonly used attention functions are additive attention and dot-product, or multiplicative, attention. Additive attention computes compatibility with a feed-forward network that has a single hidden layer. Dot-product attention is identical to the authors' method except for the scaling factor. The two have similar theoretical complexity, but the authors say dot-product attention is much faster and more space-efficient in practice, because it can run on highly optimized matrix multiplication code. For small values of d sub k the two perform similarly, but additive attention outperforms unscaled dot-product attention when d sub k is larger.
The authors offer a reason for that gap. They suspect that for large values of d sub k, "the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients." A footnote explains the intuition: if the components of a query and a key are independent random variables with mean 0 and variance 1, their dot product has mean 0 and variance equal to d sub k. To counteract the effect, the authors scale the dot products by one over the square root of d sub k.
Next the paper describes multi-head attention. Rather than performing one attention function with full-size queries, keys and values, the authors found it beneficial to project the queries, keys and values h times, each time with different learned linear projections, down to d sub k, d sub k and d sub v dimensions. Attention then runs in parallel on each projected version, producing d sub v-dimensional outputs. These outputs are concatenated and projected once more to give the final values. The authors explain the purpose of this design:
Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.
In other words, separate heads can track different kinds of relationships, which a single head would blend together.
The authors use h equals 8 parallel heads. For each head they set d sub k and d sub v to the model dimension divided by h, which is 64. Because each head works in a smaller space, the total computational cost is similar to that of a single attention head at full dimensionality.
The paper names three distinct ways its model uses attention. In the authors' words,
The Transformer uses multi-head attention in three different ways.
The first is what the paper calls encoder-decoder attention. Here the queries come from the previous decoder layer, while the memory keys and values come from the output of the encoder. The authors say this "allows every position in the decoder to attend over all positions in the input sequence." They note that the design mirrors the attention mechanisms common in earlier sequence-to-sequence models, which turn one sequence into another.
The second is self-attention in the encoder. Here the keys, values and queries all come from the same place: the output of the previous encoder layer. Each position in the encoder can therefore attend to every position in the layer before it. To attend means to weigh how much each other position contributes when building the representation of the current one.
The third is self-attention in the decoder, where each position can attend to all positions up to and including itself. The authors write,
We need to prevent leftward information flow in the decoder to preserve the auto-regressive property.
An auto-regressive model produces its output one piece at a time, with each piece depending only on what came before, so it must not look ahead at later positions. The paper enforces this through masking. Before the softmax, which converts raw scores into weights that add up to one, the connections that would be illegal are masked out, meaning their scores are set to negative infinity so that their weights drop to zero.
Each layer of the encoder and decoder also contains a fully connected feed-forward network. The paper says this network is applied "to each position separately and identically", so the same small network runs on each token's vector on its own, and any interaction between positions is left to the attention sub-layers. The network has two linear transformations with a ReLU activation in between. ReLU passes positive values through unchanged and turns negative values into zero, which is what gives the network its nonlinearity. The authors note that
while the linear transformations are the same across different positions, they use different parameters from layer to layer.
They add that another way to describe the operation is as two convolutions with kernel size 1, meaning a filter that looks at one position at a time. In this model the input and output have dimension 512, and the inner layer expands to dimension 2048.
The next part of the paper covers how tokens enter and leave the model. As in other sequence transduction models, the authors use learned embeddings to turn input tokens and output tokens into vectors of the model's dimension. A learned linear transformation followed by a softmax function then turns the decoder's output into probabilities for the next token. The model uses one shared weight matrix for the two embedding layers and the linear transformation before the softmax, which the authors say is similar to earlier work. In the embedding layers they scale those weights by the square root of the model dimension.
A comparison table then sets self-attention against recurrent and convolutional layers on three measures. The first is computational cost per layer, where n is the sequence length, d is the representation dimension, k is the convolution kernel size, and r is the neighborhood size in restricted self-attention. Self-attention cost grows with the square of the sequence length times the dimension. Recurrent layers grow linearly with sequence length but with the square of the dimension. Convolutional layers scale with kernel size, sequence length and the square of the dimension, and restricted self-attention scales with neighborhood size, sequence length and dimension. The second measure is the minimum number of sequential operations, meaning steps that must run one after another. Self-attention, convolution and restricted self-attention need a constant number of these, while recurrent layers need a number that grows with sequence length. The third measure is maximum path length, the longest route a signal has to travel between two positions. The table gives a constant for self-attention, so any two positions are linked in one step. Recurrent layers need steps proportional to sequence length, convolutions need a logarithm of sequence length in the base of the kernel size, and restricted self-attention needs sequence length divided by the neighborhood size.
The model contains no recurrence and no convolution, so on its own it has no built-in sense of order. The paper therefore argues that it must inject some information about the position of each token in the sequence. It does this by adding what the authors call "positional encodings" to the input embeddings at the bottoms of the encoder and decoder stacks. Each positional encoding has the same dimension as the embeddings, so the two can simply be summed. The authors note that there are many possible choices of positional encoding, both learned and fixed, and that earlier work has explored several of them. The authors use a fixed design built from sine and cosine functions of different frequencies. Each pair of dimensions in the encoding corresponds to one sinusoid, a wave whose values depend on the position and on the dimension index. The wavelengths, meaning the length of one full cycle of each wave, form a geometric progression running from 2π up to 10000 times 2π. So the lowest dimensions change quickly from one position to the next, while the highest dimensions change very slowly. The authors chose this function because they hypothesized that it "would allow the model to easily learn to attend by relative positions." Their reasoning is that for any fixed offset k, the encoding at position pos plus k can be represented as a linear function of the encoding at pos, so a fixed shift in position corresponds to a consistent transformation that the model can learn. The authors also tried learned positional embeddings instead, and found that the two versions produced "nearly identical results." The paper's later results on model variations report this comparison. They nonetheless preferred the sinusoidal version, because it "may allow the model to extrapolate to sequence lengths longer than the ones encountered during training." The authors present this extrapolation as a possibility the design may allow, not as a result their experiments establish.
The paper next compares its self-attention layers with the two designs most often used for this job, recurrent layers and convolutional layers. The job is mapping one sequence of symbol representations, whose length can vary, onto another sequence of the same length. Think of a sentence in one language turned into a sentence in another, where each word is a list of numbers. The authors judge the three layer types against three desiderata. The first is the total computational complexity per layer. The second is how much of the computation can run in parallel, which they measure by the minimum number of sequential operations a layer needs. The third is the path length between long-range dependencies in the network.
Long-range dependencies are links between words that sit far apart in a sentence, and the authors call learning them "a key challenge in many sequence transduction tasks." Their reasoning is that a learning signal has to travel along paths through the network, and the length of those paths matters. In their words,
The shorter these paths between any combination of positions in the input and output sequences, the easier it is to learn long-range dependencies.
So the authors also compare the maximum path length between any input position and any output position across the layer types.
On parallelism, the authors state that "a self-attention layer connects all positions with a constant number of sequentially executed operations, whereas a recurrent layer requires O(n) sequential operations." A recurrent layer works through the positions one after another, so the number of steps grows in proportion to the sequence length, n. The number of steps in a self-attention layer does not grow with length.
On cost per layer, the authors say self-attention is faster than recurrent layers when the sequence length n is smaller than the dimensionality of the representations, d. They add that this is "most often the case with sentence representations used by state-of-the-art models in machine translations," naming word-piece and byte-pair representations, which are ways of splitting text into smaller units. For tasks with very long sequences, the authors propose restricting self-attention so that each output position attends only to a neighborhood of size r around it. That would improve computational performance, but it would stretch the maximum path length to order n divided by r. They say they plan to investigate this approach in future work.
The comparison with convolutions works differently. A convolutional layer looks at a fixed window of neighboring positions, whose width is the kernel width, k. When k is smaller than n, a single layer does not connect every pair of positions. Connecting all pairs takes a stack of order n divided by k layers for contiguous kernels, or order log base k of n layers for dilated convolutions, which skip positions to widen the window. Either way, the longest paths in the network grow. The authors also note that convolutional layers are generally more expensive than recurrent ones, by a factor of k. Separable convolutions cut that cost considerably, to a complexity of order k times n times d, plus n times d squared. Even when k equals n, the authors write, the complexity of a separable convolution matches "the combination of a self-attention layer and a point-wise feed-forward layer, the approach we take in our model."
The authors also raise a side benefit. Self-attention, they suggest, "could yield more interpretable models." They inspected the attention distributions from their models and point to examples in an appendix. Their finding is that
Not only do individual attention heads clearly learn to perform different tasks, many appear to exhibit behavior related to the syntactic and semantic structure of the sentences.
The attention heads are separate attention mechanisms that run side by side, and syntax and semantics refer to the grammar and the meaning of a sentence. This part of the argument is reasoned from the layers' operation counts and path lengths and from the authors' inspection of attention patterns, and it does not come from a measured benchmark.
The training section describes how the models were trained, starting with the data. The authors used the standard WMT 2014 English-German dataset, which contains about 4.5 million sentence pairs. Sentences were split into subword units using byte-pair encoding, giving a shared source-target vocabulary of about 37,000 tokens. For English-French, the authors used a much larger WMT 2014 set of 36 million sentences, and split its text into a 32,000 word-piece vocabulary. Byte-pair encoding and word-pieces are ways of breaking words into smaller reusable fragments, so that rare words can still be represented from familiar parts. Sentence pairs were grouped into batches by approximate length, so that sentences of similar size were processed together. Each training batch held about 25,000 source tokens and about 25,000 target tokens.
The second part covers the hardware and the schedule. The team trained the models on a single machine with eight NVIDIA P100 graphics processing units. For the base models, which use the hyperparameters set out across the paper, the paper reports that "each training step took about 0.4 seconds." The base models were trained for a total of 100,000 steps, which the authors give as 12 hours. The larger big models, described in the bottom line of the paper's third table, took 1.0 seconds per step. They were trained for 300,000 steps, which the authors put at 3.5 days. Taken together, these figures show the training cost in concrete terms: a modest single-machine setup, with the big models requiring several times the compute of the base models.
The authors train with the Adam optimizer, a widely used method that adjusts a model's settings during training. They set its beta values to 0.9 and 0.98 and its epsilon to ten to the power of negative nine, a very small number that keeps the calculation stable.
The learning rate, which controls how large each adjustment is, is not held fixed. The authors vary it across training according to a formula they state in full. In their words, the rate is "decreasing it thereafter proportionally to the inverse square root of the step number." Before that decline, it rises in a straight line over an initial warm-up period, which the authors set at 4000 steps. Warm-up is a common way to let training settle before the rate reaches its peak.
The authors then turn to regularization, the set of techniques that keep a model from simply memorizing its training data instead of learning patterns that carry over to new text. They state that they "employ three types of regularization during training." Two of them are described in the text. The first is residual dropout. Dropout randomly switches off units in the network during training, so that no single unit becomes essential. The authors apply it to the output of each sub-layer, before that output is added to the sub-layer's input and normalized. They also apply it to the sums of the embeddings and the positional encodings in both the encoder and the decoder stacks. For the base model, the dropout rate is 0.1.
The second is label smoothing, set to a value of 0.1. Normally a model is trained to put all of its confidence on the single correct word. Label smoothing spreads a small share of that confidence across the alternatives. The authors are candid about the trade-off:
This hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score.
Perplexity measures how well a model predicts text, with lower values better, so the model scores worse on that measure. BLEU is a standard score for translation quality that compares machine output against human reference translations.
The text then gives a table of results, and its caption states the claim:
The Transformer achieves better BLEU scores than previous state-of-the-art models on the English-to-German and English-to-French newstest2014 tests at a fraction of the training cost.
The newstest2014 tests are standard sets of news sentences used to compare translation systems. The table lists the base model at 27.3 on English-to-German and 38.1 on English-to-French, and the big model at 28.4 and 41.8, with the Transformer's training cost set against those of earlier systems.
On machine translation, the paper reports results on two standard benchmarks from the WMT 2014 shared task, both translating from English. For English-to-German, the authors write that the big transformer model "outperforms the best previously reported models (including ensembles) by more than 2.0 BLEU, establishing a new state-of-the-art BLEU score of 28.4." BLEU is an automatic score that compares a machine translation with human reference translations, and higher is better. The authors say training this model took 3.5 days on eight P100 GPUs. Their smaller base model, they add, "surpasses all previously published models and ensembles, at a fraction of the training cost of any of the competitive models."
For English-to-French, the big model reaches a BLEU score of 41.0. The paper says this outperforms all previously published single models, meaning models not combined into ensembles, and that it was achieved "at less than 1/4 the training cost of the previous state-of-the-art model." For this French model the authors used a dropout rate of 0.1 instead of 0.3. Dropout is a regularization technique that randomly switches off parts of the network during training to reduce overfitting.
The authors also describe how the final models were prepared. The base models were a single model made by averaging the last 5 checkpoints, which were saved at 10-minute intervals during training. The big models averaged the last 20 checkpoints. Checkpoint averaging combines saved snapshots of the model's weights from late in training. For decoding, the authors used beam search with a beam size of 4 and a length penalty of 0.6. Beam search keeps several candidate translations alive at each step instead of committing to a single one. They say these settings were chosen after experimentation on the development set, a held-out set used for tuning. The maximum output length was set to the input length plus 50, with early termination when possible.
Finally, the authors compare their translation quality and training costs with other model architectures from the literature in a summary table. They estimate the compute used to train each model by multiplying the training time, the number of GPUs used, and an estimate of the sustained single-precision floating-point capacity of each GPU.
To test which parts of the design matter, the paper varied the base model in several ways and measured how each change moved performance on English-to-German translation. The measurements use the development set, a held-out collection of news text used for tuning rather than for the final test. The authors used beam search, the decoding procedure described earlier, but did not average checkpoints, so each result comes from a single saved version of the model rather than a blend of several. BLEU is the standard score for translation quality, and higher is better. For each variant, the table of results lists a BLEU score, a perplexity figure, and the number of parameters. The authors caution that their perplexity values are per wordpiece, meaning the small units that the byte-pair encoding splits words into, so those values should not be compared with perplexities measured per word.
The first group of variants changes the number of attention heads, along with the sizes of the attention key and value vectors, while keeping the total amount of computation fixed. The paper reports that
While single-head attention is 0.9 BLEU worse than the best setting, quality also drops off with too many heads.
In other words, using only one attention head costs nearly a full BLEU point, and piling on too many heads also hurts.
The second group shrinks the attention key size, which is the number of dimensions used to compare a word's query against the keys of the other words. The authors observe that reducing this size "hurts model quality." They draw a broader conclusion from it:
This suggests that determining compatibility is not easy and that a more sophisticated compatibility function than dot product may be beneficial.
The compatibility function is the rule that scores how strongly one word should attend to another. The dot product is the simple rule the model uses, and the authors suggest that a more elaborate rule might do better.
The next variants make the model bigger and change its dropout rate. Dropout means randomly switching off some connections during training so the model does not simply memorize its training examples. The paper's words are that "as expected, bigger models are better, and dropout is very helpful in avoiding over-fitting." Over-fitting is when a model learns its training data so closely that it performs worse on new text.
The last variant swaps the sinusoidal positional encoding for learned positional embeddings. The sinusoidal version marks each word's place in the sentence with a fixed pattern built from sine and cosine waves, while the learned version has the model discover its own place markers during training. The authors "observe nearly identical results to the base model," which suggests that the choice between the two makes little difference on this task.
The authors next ask whether the Transformer can handle tasks beyond translation, and they test it on English constituency parsing. Constituency parsing takes a sentence and breaks it into a nested tree of phrases, such as noun phrases and verb phrases. The paper says this task is hard in two ways:
the output is subject to strong structural constraints and is significantly longer than the input.
It also notes that recurrent sequence-to-sequence models "have not been able to attain state-of-the-art results in small-data regimes," which makes parsing a useful test of a model built without recurrence.
The team trained a four-layer version of the Transformer, with a model dimension of 1024, on the Wall Street Journal portion of the Penn Treebank, which gives about 40,000 training sentences. They also ran a semi-supervised setting, adding the larger high-confidence and BerkeleyParser corpora, roughly 17 million sentences. The vocabulary was 16,000 tokens for the Wall Street Journal-only setting and 32,000 tokens for the semi-supervised one.
Tuning was kept to a minimum. The authors ran only a small number of experiments to choose the dropout rates, both attention and residual, the learning rates and the beam size, all on the development set. Everything else stayed as in the base English-to-German translation model. Beam size sets how many candidate parses the search keeps at each step. During inference the maximum output length was set to the input length plus 300, with a beam size of 21 and an alpha of 0.3 in both settings.
The paper's verdict is that the model "performs surprisingly well" despite no task-specific tuning. It yields better results than all previously reported models except the Recurrent Neural Network Grammar from Dyer and colleagues. On the held-out test portion of the Wall Street Journal data, the Transformer scores 91.3 F1 when trained only on that data and 92.7 F1 in the semi-supervised setting. The Recurrent Neural Network Grammar reaches 93.3 in its generative setting. The authors also compare against the BerkeleyParser, noting that the Transformer outperforms it "even when training only on the WSJ training set of 40K sentences."
In closing, the authors describe the Transformer as "the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention." Sequence transduction means turning one sequence of symbols, such as the words of a sentence, into another, such as its translation. Earlier models for this job relied on recurrent layers, which read a sentence one step at a time. The Transformer drops them in favour of attention, which lets every word look directly at the others. The authors say that for translation the Transformer can be trained significantly faster than architectures built on recurrent or convolutional layers. On two standard translation tests from 2014, known as WMT 2014, English to German and English to French, they report a new state of the art, meaning the best results reported so far. On the English to German task, they write that their best model "outperforms even all previously reported ensembles." An ensemble is a combination of several separately trained models, usually stronger than any single one of them. The authors say they are excited about the future of attention-based models. They plan to apply them to other tasks, to extend the Transformer to kinds of input and output other than text, and to study local, restricted attention, which looks only at a neighbourhood of positions rather than the whole input, so that large inputs such as images, audio and video can be handled efficiently. They also name making generation less sequential, so that output is not produced strictly one word at a time, as another goal. Finally, they note that the code they used to train and evaluate their models is publicly available online.
The main takeaway is that the authors removed recurrence and convolution altogether and reported that attention alone gave better translation quality at a fraction of the training cost of earlier systems. The paper names its own open questions. The authors say they plan to apply attention-based models to other tasks and to extend the Transformer to inputs such as images, audio and video. They also want to study restricted, local attention for very long inputs, and to make generation less sequential, so that output is not produced strictly one word at a time. They leave open how best to score how well two words match, noting that a more sophisticated compatibility function than the dot product may be beneficial. The parsing and interpretability results remain early evidence that the authors themselves frame as preliminary.

