Same request, two results: your LLM does not calculate

Analogy: the LLM is like an engineer

Think of an engineer in the practice.

Their ability depends on professional preparation: training, experience, judgement. Some engineers are more capable, some less, for a given task. A large language model — LLM — is like that: models more or less capable depending on how they were trained and sized. Not every “digital engineer” performs equally on the same problem.

But even the best engineer works differently depending on the environment:

• With more tools (office software, an audited spreadsheet, GIS).
• With a better workplace (clear policy, versioned files, review).
• With better machinery (engines, scripts, checks).

That, around an AI model, is the harness: the frame / workshop the “engineer” works inside. The same LLM behaves differently depending on the harness. Same model, different workshop: different quality of work, different evidence, different risk.

This note is part of the series on the AI harness for civil engineering. Read it with the system prompt and prompt engineering: the workshop, the policy, and the work order.

What an LLM is in one office sentence

An LLM predicts the next fragment of text from a probability distribution conditioned on the context. At each step it chooses — with more or less randomness depending on the settings — among plausible alternatives.

That produces useful text, summaries, drafts, and proposed steps. On its own it does not produce the numerical solution of a structural model with the same semantics as OpenSees, CalculiX, or ETABS.

Probabilistic versus deterministic, in site language:

• Chat / LLM: plausible text. Two runs of the same request can differ. That is sampling, not a rare bug.
• Engine (OpenSees, ETABS, CalculiX, and so on): given the same model, the same engine version, and the same parameters, the result is deterministic. If it changes, you investigate the input or the engine — not the system’s “creativity.”

We are not inventing figures here. The lesson does not need a made-up number: it needs you not to treat a conversation as if it were a solve.

Same request, two results

Illustration — a desktop demo, not a client case.

1. The same request twice to the chat: two texts or two numbers. Note the difference.
2. The same physical model to an engine (OpenSees or CalculiX, for example) with the same inputs: the result is deterministic given the model, the engine version, and the parameters.

This piece does not repeat the whole harness article. The focus here is the role: why AEC does not treat the paragraph as if it were ETABS, and why the same LLM looks different when the harness gives it hands, policy, and a stop.

Lowering temperature reduces variety. It does not turn the chat into a finite-element engine. Reproducibility aids in some APIs (seeds and the like) can stabilize outputs; they are still sampling over a language model, not the solution of a system of equations with your mass, your stiffness, and your versioned solver.

Signs you are about to sign a conversation

• You cannot name file, version, engine, and parameters.
• The number appeared in prose, with no tool trace.
• The second run does not match and you still think “it is close.”
• You confuse rhetorical confidence with convergence.
• You use temperature 0 as if it were a structural quality stamp.
• You believe “the model is very good” substitutes for engine evidence.

If any of those is true, do not paste. Ask for engine evidence or leave the text as a drafting draft.

What the LLM can do (without lying to yourself)

• Help frame the problem and list assumptions.
• Review a deck or a script before you run it.
• Explain a result that already came out of the engine.
• Draft a note that separates facts, assumptions, and warnings.
• Prepare a clear work order (see prompt engineering) or respect persistent policy (see the system prompt).

What it cannot do: replace the solve or transfer your license.

Mission of this piece: do not trust the chat’s probability for engineering numbers. The serious number comes from the engine with a trace. The chat can help you think; it cannot help you sign.

The same LLM, a different harness

Back to the engineer analogy.

The same professional, without tools and without policy, will improvise. With tools, policy (system prompt), checks, and a signature, the work is different. The harness does not make the LLM “calculate” like ETABS; it makes the work around the LLM auditable: what was asked, what was run, what was checked, who approved.

That is why “change the model” is not the only lever. Sometimes the model is enough and the workshop is missing. Sometimes the workshop is empty and the model only writes.

For the full map of the harness, read what an AI harness is.

Sources

• OpenAI — documentation on advanced use and non-determinism of completions (see the official guide).
• OpenSees — official engine site

Evaluation and limitations

Treat the notes below as evaluation context, not as a verified calculation or code check.

If you need the same result twice, the calculation must live in a deterministic engine with evidence, not in the chat.