Best win rate
100%
v0.1 OOD · 48/48 LLM-blind ratings
Public benchmark
Open methodology
Across the benchmark — frontier wraps, distilled local models, blind LLM judges — the framework wins consistently. Human-judge v0.7 results compiling.
Latest LLM-blind
75%
v0.4 cross-scale · 7B-vs-Sonnet · 18/24
v0.7 status
Compiling
4 raters · double-blind · vocab-scrubbed control
Models tested against
3
Opus 4.7 · Sonnet 4.6 · GPT-5
§ I
Versioned trajectory
Version
Judge
n
Win rate
Setup
v0
LLM-blind
n=54
frontier panel
First public bake-off. 6 strategic-reasoning questions × 3 frontier families (Opus 4.7 / Sonnet 4.6 / GPT-5), Hammerstein-on-frontier vs raw frontier. 53 of 54 ratings preferred Hammerstein.
v0.1
LLM-blind
n=48
OOD generic
4 out-of-domain strategic-reasoning questions, same frontier panel. Unanimous across judges and model families. Also: an ablation on Sonnet found the RAG corpus decorative at frontier scale; system prompt alone matches the full wrap.
v0.4
LLM-blind
n=24
cross-scale
Hammerstein-7B (the framework distilled into Qwen2.5-7B local weights, no system prompt) vs raw Claude Sonnet 4.6. 18 of 24 ratings preferred the 7B local model. 4 of 6 questions unanimous across all 4 blind judges. Pair-1 control (7B-vs-raw-7B): 24/24.
v0.7
Human
n=32 *
spec → results
First human-judged benchmark. Methodology shipped 2026-05-13; results compiling from 4 raters, double-blind, with a vocabulary-scrubbed control arm.
LLM-blind judge · Human judge · win rate = % of paired comparisons where blinded judges preferred the Hammerstein response.
§ II
v0.7 — compilation in flight
The first generation under human judges. Methodology landed 2026-05-13 with a vocabulary-scrubbed control arm — the test of whether the v0 → v0.4 margin survives when judges score on what the response actually does, not the words it uses. Results compile from four raters across four batches.
Run profile (per spec)
- Treatment arm
- Wrapped in the Hammerstein system prompt + ethical-constraint rail
- Control arm
- Bare frontier model + a single line of role context
- Vocab-scrubbed control
- Hammerstein arm with framework vocabulary stripped — isolates the doctrine-shape from the doctrine-vocabulary bonus
- Judges
- 4 human raters · double-blind on arm assignment · pre-registered rubric
- Rubric
- 5 axes scored 1–5 plus a forced binary preference per pair
- Batches
-
batch-a-results-2026-05-11.json,batch-b-results-2026-05-11.json,batch-d-results-2026-05-11.json— see Replicate it below
Rubric axes
Tactical soundness
Strategic framing
Rules accuracy
Anti-metagame discipline
Actionability of orders
§ III
Replicate it
The benchmark is open source. Pull the spec, run the rubric against any model you choose, and open an issue if your results materially differ. Treat this page as a moving target — it should be falsified by the next person who tries.
§ IV
Honest caveats
Caveat 01
Framework-vocabulary bonus
Judges may reward responses that sound "strategic." v0.4 documents this; v0.7 includes a vocabulary-scrubbed control arm to bound it. The falsifiable test: does the margin survive when framework vocabulary is stripped? Results pending.
Caveat 02
Model-version drift
Every entry runs on the then-current frontier model. A win at v0 against Opus 4.7 is not the same evidence as a win against today's Opus 4.8. Treat the trajectory as relative, not absolute.
Caveat 03
Question-set ceiling
The current question set is small enough to detect a real margin, not large enough to claim coverage across the tabletop universe. The next benchmark generation will widen the set and add CDG and operational-scale systems.
§ V
Hammerstein-CODER — restraint on code
The same discipline, applied to a coding model. The wrap forces the model to refuse the unnecessary build, scope the vague request, and deliver the functional code. It disengages on models that already perform those steps.
Over-engineering baits refused (higher is better)
| Model | Plain | Hammerstein-CODER |
|---|---|---|
| Claude Opus 4.8 | 70% | 100% |
| Claude Sonnet 4.6 | 0% | 100% |
| GPT-5 | 0% | 100% |
| GLM-5.2 / Kimi / Qwen3-Coder | 0–10% | 90–100% |
Correctness neutral: HumanEval pass@1 within ±0.05 of plain (GLM +0.05, Kimi −0.03, Qwen 0.00). 6/6 models pass the gate.
Run profile
- Models
- Claude Opus 4.8, Sonnet 4.6, GPT-5, GLM-5.2, Kimi-K2.7-Code, Qwen3-Coder-480B
- Restraint scoring
- Independent LLM judge over a 15-task adversarial bait bank — tasks constructed to elicit over-engineering (unnecessary abstractions, premature generalisation, gold-plating on a bounded request)
- Correctness scoring
- Execution-based pass@1 on HumanEval — same tasks, plain model vs wrapped; delta reported per model
- Correctness gate
- 6/6 models within ±0.05 of plain baseline — the wrap does not degrade coding ability
- Tested
- 2026-06-21/22 · comprehensive bench (6 models × 2 runs)
Finding 01
Why we'll tell you when it does the least
We ran the wrap on the strongest model we tested. It moved the least. Opus 4.8 already refuses most over-engineering on its own, so the wrap took it from 70 to 100 percent, not from near zero. A discipline layer that barely improves a model that already has the judgment is grading the model, not the prompt. The wrap does least where it is needed least. It does the most for the rest.
Finding 02
More than "do less"
We tested the wrap against ponytail, an off-the-shelf prompt built on the laziest thing that works. Both handle refusal to over-engineer similarly. The split appears on vague requests — the wrapped coder evaluates what the code actually requires, then implements it. The wrap forces a scoping step before generation begins.
Finding 03
What this isn't
The benchmark tests judgment, not baseline coding ability. The wrap only changes what the system attempts to build. The claim is narrow because the evidence is narrow. A zero-cost prompt layer — the same one as the strategic-reasoning Hammerstein — applied to code.