A practical model of the model
How language models work.
A language model builds an answer a small piece at a time, using patterns learned during training and the material in front of it. That can produce remarkable reasoning—and a very convincing mistake. Understanding both is the beginning of using AI well.
What is actually happening
A probability machine, one token at a time.
A language model receives a sequence of tokens—words, fragments of words, punctuation and code—and estimates what token could usefully follow. It repeats that act until it has produced an answer. The model is not retrieving a finished paragraph from a filing cabinet. It is constructing one, step by step, from patterns encoded during training and the context in front of it now.
For example, after “Please find the invoice”, several continuations could fit: “attached”, “below”, or a longer explanation. Context changes which continuation is likely. The same mechanism can operate over a contract, a conversation or source code; it is doing much more than choosing from a short list of stock phrases.
During training, many models learn by predicting withheld text and adjusting their internal parameters when the prediction is poor. During an ordinary conversation, they use those learned parameters on the current context; putting a document in a chat does not normally retrain the model. Attention mechanisms help relate different parts of that context. The original Transformer paper describes that architectural foundation.
But the mechanism still matters. The next plausible token is not automatically the next true token. A sentence can be beautifully shaped around a fact that never existed.
The fluent voice problem
Certainty has a tone. Models can imitate it.
A model can produce the same calm, finished prose when it is recalling a familiar pattern, combining weak evidence or filling a blank. The surface does not reveal the quality of the ground beneath it.
It predicts what fits.
Training rewards successful continuation. Truth often fits, but so do familiar errors, invented citations and neat explanations for events that never happened.
It is shaped to be helpful.
Instruction tuning and human feedback favour answers people prefer. That can improve usefulness, but it can also reward agreement, confidence and the appearance of completion.
It works from the context supplied.
If the prompt omits a rule, source, exception or goal, the model has to infer one. Missing context rarely arrives wearing a warning label.
It can reason without becoming an oracle.
More computation, decomposition and tool use can improve an answer. They reduce some failure modes; they do not convert uncertainty into a warranty.
Do not confuse the layers
The model, the context and the application are different things.
The model is the trained network: the learned parameters and the computation they perform over an input. The context is the working material for this task: instructions, conversation, documents and tool results. The application is everything around both of them: retrieval, memory, permissions, orchestration, interfaces and records of what happened.
That distinction prevents a surprising amount of nonsense. A model does not gain access to a bank because its answer sounds financially literate. The application must provide the connection, constrain the permitted actions and decide how credentials are handled. A chat window can have tools. An “agent” is not a different species of intelligence; it is a system that lets a model select and use actions within a loop.
It also explains why two products using a similarly capable model can behave very differently. One may provide current sources, structured data, restricted actions and durable records. Another may provide a blank box and a send button. Judging them both as “the AI” hides the engineering that determines whether the result is useful or dangerous.
The echo chamber of one
Agreement is pleasant. It is not corroboration.
Models learn from human writing, so common patterns are easier to reproduce than rare ones. Common is not the same as correct. Add a prompt that already leans towards a conclusion and a helpful assistant may return a polished version of the user’s own assumptions.
That is a uniquely convincing loop: private, immediate and articulate. The answer feels like a second opinion while functioning as a mirror in a very good suit.
This is a studied behaviour, not just a complaint about chatbots. Research on sycophancy found that the tested assistants could favour agreement with a user’s beliefs, and that human preferences sometimes rewarded it. That finding does not mean every agreement is false; it means agreement alone is a poor check.
The defence is not ritual scepticism. It is external friction: ask for the strongest opposing case; separate supplied facts from inferred claims; check primary sources; run the calculation; execute the code; and let another person own consequential judgement. A model agreeing with you adds no evidence. A result that survives an independent check does.
Four different jobs
Do not ask one mechanism to provide every kind of confidence.
What might work?
Draft language, suggest structures, explore options, identify patterns and propose implementations. Variation is useful here.
Owner: the model, directed by a personWhat is known now?
Retrieve the contract, legislation, live record or primary source. Retrieval supplies material; it does not prove that the model interpreted it correctly.
Owner: an authoritative sourceWhat follows exactly?
Apply a defined formula, rule, state transition or permission boundary. The same inputs should produce the same governed result.
Owner: deterministic code and testsWhat should happen?
Resolve ambiguity, exercise discretion, accept risk and own the human consequences. This is authority, not merely another prediction step.
Owner: an accountable personContext is working material
A bigger desk does not choose the right document.
The decisive fact was never supplied.
The model fills the gap from learned patterns or nearby context. The answer can be coherent while resting on a rule the system never actually received.
The right-looking source is no longer authoritative.
A policy, tariff or contract may have changed. Retrieval is useful only when versions, dates and precedence are part of the evidence chain.
Two supplied sources disagree.
More context can make this worse if the system has no rule for which source governs. Authority has to be designed, not inferred from whichever paragraph sounds firmer.
The fact is on the desk and still missed.
Long contexts can contain the right material alongside noise, repetition and contradictory instructions. Presence is not proof of effective use.
Good context names the source, version and purpose of each important input. It preserves decisions outside the chat and removes irrelevant material. The goal is not to fill the window. It is to make the task legible.
Put the boundary where the consequence is
The answer changes with the job.
Explain the rule with AI. Calculate the liability with code.
A model can translate dense guidance into plain language and help an engineer map the cases. The production calculation should apply an explicit versioned rule to recorded inputs, expose its working and reproduce the result.
Summarise the record with AI. Keep authority with the caseworker.
A model may help a person find relevant material in a long file. It should not quietly turn a summary into an eligibility decision. The system can enforce procedure, preserve evidence and record who decided what.
Generate the implementation with AI. Let execution answer whether it works.
Code is unusually amenable to checking because it can be compiled, run, attacked and compared. The model proposes. Types, tests, security checks, observed behaviour and human review decide whether the proposal survives.
Use AI to map the question. Use sources to support the claim.
A fast synthesis is valuable at the beginning. Every fact that carries the conclusion needs a traceable source, a date and enough context for another reader to challenge it.
The production loop
Inference belongs inside a control system.
Good AI engineering is not a longer prompt. It is a system that gives the model the right context, limits its authority and makes failure observable.
- 01
Define the obligation.
State what must be true, which inputs are authoritative, what the model may change and which decisions require a person.
- 02
Give it the smallest useful context.
Relevant schemas, examples, rules and constraints beat a warehouse of undifferentiated text. Context is a working surface, not a ceremonial data dump.
- 03
Let the model propose.
Use its breadth for drafts, hypotheses, transformations and code. Preserve alternatives when uncertainty matters.
- 04
Make reality answer back.
Fetch the source, run the program, calculate the number, compare the output and deliberately exercise the failure paths.
- 05
Keep an accountable decision.
A person accepts the evidence, rejects the work or changes the specification. The model is not promoted to authority by being eloquent.
Worked example · an invoice arrives
Useful inference. Controlled commitment.
Consider a system asked to prepare an invoice for payment. This is a constructed example, but the boundary is real: language is messy; money is not allowed to be.
Extract candidates and retain the evidence.
The model proposes the supplier, invoice number, issue date, due date, line items, total and bank details. Each field keeps its source location. An empty or ambiguous field remains unresolved instead of being completed from habit.
Let code recompute what can be recomputed.
Versioned rules check formats, currency, tax arithmetic and whether line items produce the stated total. Durable records answer whether the invoice number has already been seen.
Route consequences through explicit authority.
A changed bank account, unexpected amount or missing purchase order triggers review. The model can explain the difference. It cannot quietly approve its own interpretation.
Make the action recoverable.
The payment instruction receives a recorded identifier so a retry cannot create it twice. The system records the sources, validation results, person and action together.
The model makes the unstructured document tractable. It does not become the ledger, calculator, authority or payment rail. Each part of the system does the job it can actually defend.
What still goes wrong
Tools move the failure. They do not abolish it.
A search tool can fetch the wrong page. A retrieval system can omit the decisive paragraph. A calculator can receive the wrong units. A code agent can satisfy a shallow test while breaking an unmeasured behaviour. A human reviewer can be rushed, impressed or mistaken.
That is why the control system needs layers: source provenance, explicit transformations, deterministic checks, adversarial tests, operating telemetry and clear ownership. The aim is not to declare the model safe. It is to make the whole system capable of exposing error before the error owns the outcome.
The useful settlement
AI does not replace deterministic software. It changes who can afford to build it.
Language models make exploration and implementation dramatically more abundant. That is precisely why specifications, evidence and operational boundaries matter more. We can now build more of the real system. We should not use the new speed as an excuse to make correctness vague.
Let the model be imaginative. Make the product answerable.