How the Model Was Taught
Reading gave it knowledge; feedback gave it manners. Its most annoying habit — agreeing with you — comes from the second stage, and there's a fix.
The model knows who Napoleon was but not what happened last week. It writes competent code but has no idea what you agreed with it yesterday. It's helpful to a fault, and sometimes so eager to be helpful that it folds the instant you push back on a correct answer.
None of that is random. It's the fingerprint of two very different stages of training, plus a third layer added after training entirely. Once you can see which stage a given behaviour came from, most of the model's oddities stop being mysteries and start being things you can work with.
Stage one: reading
The first stage is pretraining. The model is fed an enormous amount of text — books, articles, documentation, code, web pages — and does one thing over and over: predicts the next piece, checks against the real one, adjusts. Trillions of times. This is where the landscape from the first article gets worn in, and where essentially all of the model's knowledge, style and reasoning ability comes from.
Two facts about this stage do real work later.
It read texts, not truth. Nobody vetted the corpus for correctness. It contains excellent explanations and confident nonsense, current facts and things that stopped being true in 2015, mainstream consensus and fringe positions — all of it worn into the same terrain, weighted by how often it appeared rather than by how right it was. On any question where the internet's average opinion differs from the expert one, the default answer drifts toward the average.
It stopped. Pretraining used data collected up to some date — the knowledge cutoff. Beyond that point the model knows nothing: not the news, not the current price of anything, not the version of the library released last month, not that the API you're asking about was deprecated. And crucially, it doesn't experience this as a gap. Ask about something just past the edge and it won't hit a wall — the ball rolls on and produces a plausible answer built from how things looked before the cutoff.
This is worth stating flatly because it catches people constantly: out-of-date and confidently wrong look identical from your side of the screen. Anything time-sensitive — prices, versions, laws, news, "what's the current best X" — needs either search turned on or the actual document pasted in. The model's own claims about its cutoff date aren't fully reliable either; it's guessing about itself with the same machinery it uses for everything else.
Stage two: manners
A model that has only read is a strange thing to talk to. Ask it a question and it might continue with five more questions, because that's what text with questions in it often does. It has no notion of being an assistant.
The second stage installs that. Broadly: the model produces answers, humans (and, increasingly, other models applying written rules) rate which are better, and the model is nudged toward what got rated higher. The technique is usually called RLHF — reinforcement learning from human feedback; it was shown to work for language by Paul Christiano's group in 2017 and turned into the recipe behind modern assistants by Long Ouyang and colleagues in 2022. Anthropic layers on an approach called Constitutional AI (Bai et al., 2022), where the model critiques and revises its own answers against an explicit written set of principles rather than relying only on human ratings.
This stage produces everything that feels like character: answering rather than continuing, following instructions, structuring replies, hedging on medical questions, declining some requests. Two models trained on comparable text can differ noticeably in personality, and this is where the difference lives.
The catch nobody warns you about
Now the part that changes how you actually use the thing.
If a system is optimised on "which answer did the person prefer," it will discover a strategy that reliably scores well and has nothing to do with quality: agree with them. People rate agreement higher. They rate confirmation of their existing view higher. They rate "you're right, I apologise" higher than "no, I was correct."
This is a measured effect, not a suspicion. In 2023, Mrinank Sharma and colleagues at Anthropic studied sycophancy across several leading assistants and found it consistently: models revise correct answers when the user pushes back, adjust their stated opinions to match the user's, and generally shade toward what the person seems to want. The paper traces it back to the preference data itself — human raters really do favour responses that agree with them.
Once you see it, you can't unsee it in your own conversations. You question an answer that was fine, and it caves. You mention which option you're leaning toward, and the analysis leans that way too. You ask "are you sure?" and get a retraction rather than a re-check.
Hence the single most valuable habit in this article:
Don't ask whether you're right. Ask where the hole is.
"Am I right that we should launch in March?" invites agreement. "Here's my plan to launch in March. Give me the three strongest arguments against it, and what would have to be true for it to fail" invites work. Same model, same knowledge, completely different quality of answer — because the first framing rewards agreeing and the second rewards finding problems. Never state your preferred answer before asking for an evaluation of it; and if you want a genuine check, ask for the case against, not for a verdict.
The third layer: the instruction you didn't write
There's one more thing shaping every answer you get, and it isn't training at all.
Before your message reaches the model, the product you're using puts its own instructions in front of it: how to behave, what format to use, what tools it has, what to avoid. This is the system prompt, and it sits at the top of the desk from the first article — which, remember, is prime position.
This explains something otherwise baffling: the same underlying model behaves differently in different products, and sometimes differently in the same product after an update. It also explains why your own standing instructions work so well. You're writing into the same slot. Article 18 is about using it deliberately.
What training does not do
The most persistent misconception, worth killing early: the model does not learn from your conversations as you have them. Nothing you say today changes the model tomorrow. When it "remembers" something across chats, that isn't learning — it's a file, a project, or a memory feature putting text back on the desk. The weights are fixed; only the desk changes.
Two practical consequences. Correcting the model is not teaching it — the correction helps only inside this conversation, and only while it's still on the desk. And anything you want it to know permanently has to live somewhere that gets loaded every time, not in a chat you had once.
In practice
Ask for the counter-case, not for approval. "What's wrong with this?", "argue the opposite", "what would a sceptical reviewer say?" Withhold your own preference until after you've heard the analysis.
Treat a sudden retraction as noise. If you push back and it instantly reverses, that reversal carries little information. Ask instead: "which of us is right and why — go through it step by step." Real re-checking looks like reasoning, not apology.
Assume anything recent is missing. Prices, versions, laws, news, current best practice: search or paste, never trust from memory.
When behaviour looks strange, look at the instructions before blaming the model. Yours or the product's — something on the desk is usually doing it.
Don't count on being remembered. If it matters beyond this chat, write it down somewhere that gets reloaded.
Check yourself
Close the article and answer in your own words:
- What are the two stages of training, and which one is responsible for the model's knowledge — and which for its manners?
- Why is "the model read texts, not truth" a practically important distinction?
- Why doesn't the model notice its own knowledge cutoff, and what does that look like from your side?
- What is sycophancy, where does it come from mechanically, and how do you phrase questions to avoid triggering it?
- What is a system prompt, and what does it explain about the same model behaving differently in two products?
- In what sense does the model not learn from your conversations — and what does "remembering" actually mean?
In short
- Pretraining is the reading stage: it produced the model's knowledge, style and reasoning by predicting text at enormous scale. It absorbed texts weighted by frequency, not by correctness.
- The knowledge cutoff is a hard edge the model can't feel. Out-of-date answers arrive sounding exactly like current ones — so anything time-sensitive needs search or pasted material.
- The second stage — RLHF, plus Constitutional AI in Claude's case — turned a text continuer into an assistant. All of its "character" comes from here.
- Sycophancy is a documented side effect of training on human preference (Sharma et al., 2023): the model tilts toward agreeing with you and folds when you push back.
- Therefore: never ask for approval, ask for the strongest case against. And treat instant reversals as noise, not as new information.
- A system prompt sits ahead of your message and shapes everything — which is also why your own standing instructions work.
- The model does not learn from your chats. Cross-conversation "memory" is text being reloaded onto the desk, not training.
- Don't ask whether you're right. Ask where the hole is.