Replaying 55 historical setups: one comparison

What was replayed

Every cycle is recorded: the prompt that was sent, the model's answer, the parsed decision and the market context. The replay took 55 recorded setups — 45 cycles that historically produced an entry (BUY or SELL) and 10 that produced a CLOSE — and sent each one back through the pipeline with a fixed prompt.

Three variants, all on the same replay data:

• K — the prompt version in use before the rewrite, on the model generation of the time;
• W — the current prompt, on the same model generation;
• W36 — the current prompt, on the previous model generation.

Each variant produced 55 calls, 165 in total, run once, on 2026-09-18.

What was measured

Two things only: whether the variant entered on a historical entry setup, and whether it chose the same direction as the historical decision. No trade outcome, P&L, slippage or fee was scored — the replay does not know what would have happened, only what each variant said.

On the 10 CLOSE setups the same difference appears: the old prompt produced CLOSE 2, UPDATE 2 and HOLD 6; the current prompt produced HOLD 10.

Where the variants disagree

Comparing the two model generations under the same current prompt: 31 of 45 decisions matched. Of the disagreements, 10 are cases where the older model entered and the newer one held, and 4 where the newer one entered and the older one held; the remaining agreement includes 5 setups where both entered in the same direction.

That matrix is the most useful part of the result: changing the model moved more decisions than the prompt rewrite did, and neither change was evaluated against an outcome.

Run quality

• 165 calls, 0 failures, 0 retries required;
• every response finished normally (finishReason: STOP), no truncation;
• mean call latency 20.4 s.

Those numbers only say the replay itself behaved; they say nothing about the quality of the decisions.

What it established

One concrete finding: the old prompt did not enter on any of the 45 historical entry setups, while the current one entered on 9. Whatever else is true, the rewrite changed behaviour from "almost never act" to "act sometimes". That is a calibration change, and calibration is not skill.

It also made the model-generation difference visible and measurable, which is the reason to keep recorded cycles at all: without them, the only comparison available would be a memory of how the bot used to feel.

What it does not establish

• No outcomes were scored. Entering more often is not better; it is only different. This replay cannot say which variant would have made money.
• The sample is small and from one period. 55 setups, one instrument, one language model family.
• One run per variant. No repeats, so the run-to-run variance of the model is not measured.
• Historical setups are not a neutral test. They were selected because the bot acted on them at the time; the replay deliberately excludes the cycles where it did nothing.
• The replay scripts are not in the public repository. The numbers here are transcribed from the run's summary output; the method is described above so it can be repeated, but this page is currently the only published record of it.