Tokens and the Context Window

Ask a model how many times the letter "r" appears in "strawberry" and there's a decent chance it gets it wrong. Ask it, forty messages into a conversation, to follow the rule you set at the very beginning, and it may quietly ignore it.

Neither failure is about intelligence. Both come from the way text gets into the model at all — and both are completely predictable once you know two things: what it reads, and how much of it it can hold.

It doesn't read letters

Text doesn't reach the model as letters. Before anything happens, it's cut into pieces called tokens — roughly, frequent chunks of characters. Common words are usually one token; longer or rarer ones break into several: "unbelievable" might arrive as "un" + "believ" + "able". A rough rule of thumb for English: a token averages about four characters, so a page of text is somewhere near 500 tokens.

The model never sees inside a token. "Strawberry" arrives as two or three lumps, not as ten letters — so counting the r's is like being asked how many times the letter "r" appears in a word you were shown as a picture from across the room. It can often reason its way there, but it's working around its own eyesight, not reading.

The practical rule is narrow and worth having: anything that lives at the level of individual characters is the wrong job for a model. Counting letters or words, exact string manipulation, checking whether a password contains a digit, precise arithmetic on long numbers. Not because it's stupid — because that layer is invisible to it. Give that work to code (which the model is happy to write for you) instead of to the model's eyes.

Two side effects worth knowing. Russian, Chinese and other non-English text usually costs noticeably more tokens per unit of meaning than English, which matters when limits and price are counted in tokens. And rhymes, anagrams and wordplay are unusually hard for the same reason.

The desk

Now the second thing, and the one that changes how you work.

The model has no memory of you. None. Between two of your messages nothing is stored anywhere in it.

What it has instead is a desk. Everything it can use right now is lying on that desk: the system instructions from the product you're using, your files, the whole conversation so far, its own previous answers, and your latest message. It reads the entire desk from scratch, every single turn, and produces the next reply. Then the desk is wiped and re-laid from the transcript for the next turn.

The model knows exactly what's on the desk — and nothing else.

The desk has a hard edge: the context window, measured in tokens. Modern windows are large — hundreds of thousands of tokens, several books' worth — but "large" is not "unlimited", and three things follow that no amount of prompting will change.

A new chat is an empty desk. Not a forgetful colleague — a different one, who has never met you. Everything the model "knows" about your project has to be back on the desk or it doesn't exist.

Nothing leaves the desk on its own. Every failed attempt, every draft you rejected, every "no, not like that" stays lying there and keeps influencing the answer. This is the single most common cause of a conversation slowly going bad: people think they're correcting the model when they're actually piling more contradictory material onto the same desk.

Position matters. In 2023 Nelson Liu and colleagues at Stanford ran a study with the excellent title Lost in the Middle: they hid a needed fact at different positions inside a long context and measured how often models found it. The result was a U-curve — material at the beginning and at the end was used reliably, material buried in the middle was noticeably more often missed. Newer models handle this better, but the shape hasn't gone away. Where you put something in a long prompt is a real decision, not formatting.

The mirror of your own head

There's a neat inversion here worth holding onto, and it comes straight from the How to Learn sphere.

Your working memory holds about four items at once — that's the entire reason chunking exists. But your long-term memory is enormous and, with spaced repetition, effectively permanent.

The model is the exact opposite. Its "working memory" is gigantic: hundreds of pages held simultaneously, with perfect recall of the literal text. And its long-term memory is zero — nothing carries from one conversation to the next unless something writes it back onto the desk.

That mirror suggests the division of labour that runs through this whole sphere. The model holds volume: long documents, whole codebases, every version of the draft. You hold continuity and meaning: what this is for, what we decided, what matters. Trying to do it the other way round — memorising what the model could hold, while expecting the model to remember what you decided last week — is how people end up doing both jobs badly.

Where the money and the slowdown come from

Since the model re-reads the entire desk on every turn, a long conversation is genuinely more expensive per message than a short one, and often slower. That's not a defect, it's the same fact stated in currency: on turn fifty, the model isn't answering your last message — it's re-reading forty-nine previous ones and then answering.

Which gives a piece of advice that feels wrong until you try it: one chat per task, and don't be sentimental about it. When a conversation has done its job, ask for a summary of what was decided, start a fresh chat, and put the summary on the clean desk. You lose the clutter and keep the conclusion. Article 7 turns this into a full working habit; article 18 covers making some of that context permanent so you stop re-laying it by hand.

In practice

Paste it, don't retell it. "Here's my article, here are the comments on it" beats any description of them. Putting things on the desk is the cheapest, highest-leverage move you have.

Front and back are prime real estate. In a long prompt, put the task at the top and repeat the key constraint at the bottom, after the material. What's in the middle can slip.

Don't nurse a broken conversation. Three failed attempts in a row is not a signal to try harder — it's a signal that the desk is full of failures. Summarise, restart, re-lay.

Send character-level work to code. Counting, exact reformatting, checking a string — ask for a script, run it, and let the model reason about the result instead of eyeballing the characters.

Remember it isn't remembering. If it matters tomorrow, it belongs in a file, a project or your notes — not in a chat you plan to scroll back to.

Check yourself

Close the article and answer in your own words:

1. What is a token, and why does that make counting letters a bad task for a model?
2. What is on the "desk", and what happens to it between two of your messages?
3. Why does a long conversation get worse rather than smarter, and what do you do about it?
4. What did Lost in the Middle find, and how does it change how you lay out a long prompt?
5. How are the model's memory and yours mirror images of each other, and what division of labour does that suggest?

In short

• Text arrives as tokens — chunks of characters, not letters. Character-level work (counting, exact string edits, long arithmetic) is invisible to the model; hand it to code.
• The model has no memory. Everything it uses is on the desk: system instructions, your files, the conversation so far, your message. It re-reads all of it every turn.
• A new chat is an empty desk, not a forgetful colleague.
• Nothing leaves the desk on its own: failed attempts and rejected drafts keep pulling on later answers. That's why long conversations degrade.
• Lost in the Middle (Liu et al., 2023): material at the start and end of a long context is used reliably; the middle gets missed. Place important things accordingly.
• You have four slots of working memory and vast long-term memory; the model is the reverse. Let it hold volume, and hold the meaning yourself.
• The model knows exactly what's on the desk — and nothing else.