4 min read · 2026-08-28
You do not understand a diagram until you have had to make one work.
You can read about transformers for a long time and come away feeling like you understand them. There is a diagram everyone uses. There are boxes labelled attention and feed-forward, arrows going up the page, and a caption explaining that the model attends to relevant parts of the input.
I could reproduce that diagram from memory. Then I tried to write one, and found out that I could not.
This is a specific and slightly humiliating experience I would recommend to anyone. There is a gap between being able to describe how something works and being able to make it work, and the only reliable way to find out which side of that gap you are on is to try to build the thing.
The temptation is to start in the middle, at the attention mechanism, because that is the interesting part. I started at the bottom instead, with tokenization, which is the unglamorous business of turning text into numbers.
The simplest version is character-level. Every character gets a number. It is easy to write and easy to reason about, and it teaches you the problem with itself very quickly: your sequences become enormous. A short paragraph becomes hundreds of steps, and since attention cost grows with the square of the sequence length, you feel that immediately.
That is what makes byte pair encoding, or BPE, make sense. BPE looks at your text and finds the pairs of symbols that occur most often, then merges them into single units, over and over. Common words end up as one token. Rare words break into pieces. Nobody chooses the vocabulary by hand; it falls out of the statistics of the text.
Writing that yourself is genuinely clarifying. You stop thinking of tokens as a technicality and start seeing them as a decision that shapes everything above it.
Here is the thing the diagram does not make obvious. Attention has no idea what order anything is in.
It compares every position to every other position, all at once. That is what makes it fast, because there is no left-to-right loop. It is also why, without help, "the dog bit the man" and "the man bit the dog" look identical to it. The mechanism sees a bag of words.
So position has to be added deliberately. The original approach adds a positional signal to each token before attention runs. It works, but it treats position as something bolted on at the start.
RoPE, rotary position embedding, does something cleverer. Instead of adding position, it rotates the vectors by an angle that depends on where the token sits. When two tokens are then compared, the maths works out so that what matters is the difference between their angles, which is to say the distance between them. Position stops being an absolute stamp and becomes a relative relationship, which is usually what you actually care about. Word five relating to word three matters; whether they are at five and three or at 105 and 103 usually does not.
I could not have explained that before writing it. I had read it. Reading it is not the same.
The failures are rarely conceptual. They are shapes.
An enormous amount of the work is keeping track of which dimension is which: batch, sequence position, attention head, feature. Matrices happily multiply in the wrong order and give you a number rather than an error, and the model trains, and the loss goes down a bit, and everything looks fine while being subtly wrong.
The mask is the other classic. When a model is learning to predict the next word, it must not be allowed to see the words after it. Get the mask wrong and the model performs beautifully, because it is reading the answer. It is the same failure I later hit in a completely different project on basketball data, where the model was reading the box score of a game it was meant to be predicting. Different field, identical mistake: the model was given information it would not have at the moment it has to guess.
That pattern is worth internalising. When results look too good, the first suspicion should be that something has leaked, not that you are a genius.
The thing I built is not useful. It is small, it is slow, and anything I might actually want to use is better served by a library written by people with far more time and hardware.
That was never the point. The point was to find out whether I understood this, and the answer turned out to be no, followed by yes.
What changed is not that I memorised more. It is that the words became load-bearing. "Attention" stopped being a box on a diagram and became a specific operation with specific costs and specific ways of going wrong. When I read a paper now, I read it as something someone had to implement.
I would recommend the method for anything you believe you already understand. It is uncomfortable and it is fast, and it tells you the truth.