AI: the reality

Generative large language models excel at creating new items based on statistical patterns.

But LLMs struggle when they have to interpret a pre-existing artefact and constrain their output so that it is correct in terms of both the artefact and the system in which that artefact will be used.

We see this across software engineering, finance, medicine, and the law.

The long-term value of LLMs appears to be in problem spaces where they do not have to interpret something that already exists. Most value can be derived when an LLM generates a new artefact based only on reference to its statistical patterns and an input prompt.

This article discusses the LLM interpretation problem across four disciplines. Leaders must understand that LLMs fail when asked to modify existing artefacts because the language model is not designed to reason about domain‑level constraints ("If I change this, what breaks?"). This defines where AI is safe to use and where it introduces operational risk.

Most AI risk comes not from generation, but from misapplied modification.

LLMs

As of September 2026, leading large language models (LLMs) such as GPT-4 and Qwen2.5 are based on statistical models after processing approximately 20 trillion tokens.

Vast amounts of tokens

A token is the smallest unit of text an LLM can process. It is typically a word, part of a word, a punctuation mark, or it is whitespace, such as a tab, a space or newline character.

A very rough estimate is that one token is 3 to 4 characters (or letters) in the English language.

If each token were a word, twenty trillion words would fill about 200 million books. This is enough to stock every library on Earth many times over.

Put another way, if you read one token per second on your phone, you would finish in the year 636,000 AD.

Twenty trillion seconds ago, modern humans did not exist as the time period is well into the Middle Pleistocene era, long before agriculture, writing, or Homo sapiens. It predates modern humans, Neanderthals, or any known human civilisation.

Statistical patterns from language

The twenty trillion tokens are used to fit the model to the data. This is achieved by adjusting weights (numbers) through gradient descent.

When processing tokens, a large language model will try to predict the next token. If the input is "The cat sat on the". The target may be "mat".

If the model predicts "mat" incorrectly for the given input, the error is used to adjust weights. This adjustment is the gradient descent being applied.

Imagine teaching a child to finish sentences.

You say:

"The cat sat on the ..." and the child guesses "table.". You correct them:

"No, mat."

That tiny correction alters how the child will respond next time. Next time, they are slightly more likely to say "mat."

Now imagine doing this billions of times, with every possible sentence pattern:

• "once upon a..." → "time"”
• "as a result..." → "of"
• "side effects may include..." → "nausea"
• "force majeure shall include..." → "acts of God"
• "net present value is..." → "the sum of discounted cash flows"
• "for i in..." → “range”

Each correction nudges the model's internal representation of how words fit together. After enough nudges, the model can complete almost any sentence. Not because the model has stored all of them, but because the model's statistical representation of language has been shaped by countless tiny adjustments.

Why this works

This process of predict-and-adjust works for English because the way people write English produces text with a highly familiar structure that can be accurately modeled with statistics.

When you optimise a model’s weights with trillions of English tokens, those tiny updates allow the model to build a mirror that accurately reflects the regularities (the common usage) of the language. That mirror is so accurate (as so many tokens have been processed) that an LLM's output looks like understanding. But it is not, the mechanism is purely statistical.

This approach is possible for three main reasons.

Firstly, English has a lot of predictable continuations (such as "in conclusion..." → ",") so such a prediction is likely to be correct for a reader.

Secondly, English composes small ideas into large ones: "claim → evidence → conclusion". Processing trillions of tokens results in a model that has a representation of how the smaller ideas compose into the parts of a larger idea.

This lets the model recombine patterns in new ways, which, to a human, looks like reasoning.

Lastly, in written English, authors repeat ideas through various mechanisms: paraphrasing, templates, boilerplate legal language, standard medical phrasing and recurring coding idiom. This repetition performed a vast number of times, reinforces the model's statistical representation of that repetition, improving the model's output accuracy.

What we get from an LLM

When you expose a model to twenty trillion English tokens, each weight update nudges its mirror of correlations. After trillions of nudges, the model has a statistical representation of how correct English is used. None of this is understanding.

But the model's output behaviour aligns so well with English language structure that it looks like understanding. It is the human reader that adds the I to AI.

Large language models are statistical machines. Nothing else.

Software engineering

LLMs do a good job when creating new items based on their statistical English-language model.

In software engineering, creating a new item is the generation of computer code based on nothing other than a prompt.

This works well as the LLM has statistically modeled the relationships between the items that compose the text of a programming language. When prompted to output a new file, the LLM generates accurate program text because the model has seen a vast number of examples. These valid examples are encoded in its correlations that mirror how that programming language is correctly used.

But if an LLM has to interpret something that already exists to get its output correct, that is when they struggle.

Interpreting code

In software engineering, updating code is vastly more common than writing new code.

A computer system that solves a business problem has code running within it. To execute properly, this code relies on certain aspects of the system remaining true. If such as aspect is ever not true, the code will fail.

To successfully update code, any change must result in code that keeps correctly solving the original business problem that the code was first written for, while also correctly solving the new problem that the change represents. The changed code must also ensure that all aspects of the system that must be true, remain so.

An LLM struggles to perform such a change.

Updating code requires first understanding how the various components of a system fit together, why they fit the way they do, and reasoning to decide on the change that will work, while preserving the safe operation of the code.

But an LLM only has statistical next‑token correlations. The LLM is not capable of processing the pre-existing code, together with a prompt, in a way that answers the question: "If the change is this, what breaks?". Without the model being able to calculate this what-if scenario, an LLM cannot safely change a piece of code to satisfy the old requirement and the new requirement, while also preserving all the aspects of the system that must remain true.

We see the same LLM interpretation problem in finance, medicine, and the law.

Finance

An LLM generating a new artefact in finance will excel as there is no pre-existing object for the model to both interpret and keep consistent with the item generated. The LLM can rely entirely on its internal statistical structure.

Examples are drafting an explanation of discounted cash flow, producing a risk-adjusted return formula, or an explanation of why a movement in interest rates affects inflation.

These tasks succeed because they are pattern‑completion problems. The model has seen millions of examples of these so can generate plausible text that fits the statistical patterns in its weights.

There is no external system to keep consistent, and no other artefact to process with respect to that system before the result can be produced.

Interpreting financial artefacts

Imagine a spreadsheet that already calculates monthly loan repayments.

To do this, it has a cell that contains the amount of money borrowed, a cell with the repayment interest rate, a cell containing the length of time repayments will be made, and based on all of this a formula that outputs the monthly payment.

The finance team has a number of summaries, charts and reports that all depend on that formula remaining unchanged.

If an LLM is prompted to update the spreadsheet to also calculate weekly payments, the system that the spreadsheet is a part of (all of the dependent summaries, charts and reports) requires that any change does not break anything.

Because the LLM only has next-token correlations from its training data that do not reference the spreadsheet under consideration, an LLM might:

• rewrite the monthly formula
• replace the original logic without extending it to account for weekly payments
• rename cells
• change layout
• break spreadsheet references

The LLM produces a plausible spreadsheet, not a safe modification.

The fundamental difference

In the first example, the LLM only needed to output text with reference to its own correlations, based on the prompt.

In the second example, the presence of a pre-existing artefact, the spreadsheet, constrains the problem, requiring the LLM to restrict its token choice so that generated text does not break any dependency.

The LLM struggles with this because the information that describes what is and what is not breaking change in this case is not part of the LLM's statistical weights.

The LLM only generates a plausible spreadsheet with respect to its weights. What is generated may be a valid spreadsheet in isolation but a broken solution when the spreadsheet is considered within the context of the larger system (the summaries, charts and reports that rely on the formula still working).

Medicine

An LLM can easily write a description of how a painkiller works, list the common side effects and explain a typical adult dosage.

But updating an existing label without introducing an error is where it will struggle.

Imagine a medicine label already contained dosage instructions, warnings and side effects and that the maximum daily dosage was to be reduced from 2,400mg to 2,000mg.

A safe update must:

• change only the maximum daily dose
• keep all other doses intact
• preserve warnings
• preserve side-effect lists
• avoid conflicting changes
• avoid removing required regulatory data

Because an LLM does not understand the domain, it only has next-token correlations from millions of medicine label examples, an LLM might:

• rewrite the entire label
• change other doses
• remove required regulatory text
• introduce inconsistencies: a max dose of 2,000mg in one place, 2400mg in another

The output is a plausible medical label, not a safe one with respect to local medical practice.

The LLM does not understand the need for this safety. The LLM does not understand medicine, its standards, norms, and regulatory requirements.

Law

An LLM can easily draft a confidentiality clause for a fresh contract.

But if an update is required such as "Change the notice period from 30 days to 14 days." the LLM will struggle. Once again, a safe update must:

• change only the notice period
• keep all definitions intact
• avoid introducing new legal meaning

Correct change is hard. A change is always with respect to some other artefact. In our case, this other artefact is a running production software system, a corporation's financial system, the safe public consumption of medicine, and the effect on clients of a law firm.

As the LLM has no awareness of these domains, it does not see the need for consistent change with respect to that system.

The generated artefact will be plausible with respect to the model's statistical correlations but that tells you nothing about correct and safe updates within software engineering, finance, medicine, the law, or any other field of endeavour.

Solutions adopted

The four fields attempt to solve the LLM's limitation in the same way:

• constrain the LLM so that rewrites are not possible, so the LLM only fills in the blanks
• validate the output using domain tools such as static code analysis, or clinical safety checks
• minimise the number of items that can be changed
• surround the LLM with systems that attempt to understand the domain

These are mitigations only that reduce the likelihood of LLM error but none of them eliminate LLMs errors when updating an artefact with respect to some system.

These four mitigation strategies reduce the risk of an LLM change being wrong, but they do not eliminate it. They work best when the domain has strong structure and strong external validators (such as software engineering) but they work least well when the domain has less structure, or the domain is contextual or interpretive, as it is in medicine and law.

Conclusion

LLMs initially held immense potential but now that they have been stress tested in real work scenarios, their correlated textual-surface outputs have been shown to be lacking when requiring domain knowledge.

Use of LLMs is best when there is no artefact to change so that there is no external domain system to keep consistent.

LLMs are most safely used when producing new text based artefacts that have no other input other than the prompt.

Use LLMs as generators, not maintainers. If you are modifying or maintaining something, a human is still best.

Read next: Agents Cannot Maintain Systems
For the system‑level consequences of this limitation, read Agents Cannot Maintain Systems.

Related Articles

• The Myth of Complete Specifications
• The Limits of Stateless LLMs
• Vibe Coding Is Not Engineering
• What Software Engineers Need to Know About LLMs
• Building Safe, Compliant and Sustainable LLM Systems