First, where 3DEval is going: its direction, synced from the 3DEval repository. Then the verdict board: what the engine got wrong, check by check, one click from the transcript. Then the evidence so far: the pilot of record, every answer graded automatically, every figure with its receipt, and the run drawn as a world you can turn in your hands.
“The 3D.EVAL is a benchmarking tool but also one for actual AI eval.”
3D.EVAL is an AI eval framework first and a benchmark second. A small, highly targeted set of test cases catches the most common mistakes of the system under test, every case is graded automatically, and the run is shown so that watching it teaches real-world AI eval.
The pillars
01
The Task
A small, targeted test case with a known right behaviour, each one aimed at a named failure mode and traceable to the evidence that motivated it.
02
The Run
The system executes every task with its full record kept: what was asked, what came back, how long it took, what the machine was doing, and the checks that the system served was the one requested.
03
The Grader
Every task is graded automatically, in tiers.
1Code checks come first, one per failure mode, and they decide every engine failure mode they can.
2A small, local decision model (Laya, a non-generative System One model that answers typed questions with calibrated probabilities) grades what code cannot decide, and decides on its own only where its calibration holds a stated error bound.
3A frontier judge sees only the uncertain remainder and a calibration sample.
4No judge is ever the model being graded, the engine being tested, or the model whose traces trained a student, and every judge is calibrated against answers the owner labels before any judged figure is shown.
5The decision model's first measurement is about itself; it decides nothing until the owner's labels certify it.
The target ladder
cleared · available now
in progress
locked · we aim to
Each rung is a level. A level is cleared once its eval is available now, in progress while it is being built, and locked until what it needs exists.
Level 1cleared● available now
The inference engine (QueueLLM)
The question
Is the output clean? Answers commit, no chat-template or special-token leakage, no runaway output, the requested configuration is the one served, the same seed gives the same answer, the streaming path gives the resident path's answer, and answers known to be right stay right.
Evidence so far
The suite runs on the engine in one command and grades every answer by code. On the pinned engine build Qwen3-30B is clean, and gpt-oss-20b answers with special-token debris and loops on both engine paths, the same way in each run and at both token caps tried. That is wider than the streaming-path framing mistake Sprint 1 found. A run of the same model files under a second runtime comes next, to show whether the engine or the files cause it.
What it needs
The engine's attention fix and a less compressed conversion of the files, measured by this suite before and after.
Does a distilled model work well in the domain it was tuned for, graded against that World's human-authored, sourced cards?
Evidence so far
The one fine-tuned model on disk was tuned on QuKaiZen's own repository documents, not on a World, and the models Nucleus has graduated so far are not runnable here.
To unlock it
A runnable model distilled from a World, so its tasks can be graded against the cards it was tuned on.
Level 3locked→ we aim to
Reasoning in the domain (SCoTD)
The question
Can a distilled model reason step by step in its domain, with every step grounded in the World and the conclusion matching the human verdict?
Evidence so far
The SCoTD teacher contract in DDaC writes structured reasoning traces and rejects unsourced steps. In the MapSwipe triage World, only the tiles where the volunteers agreed are clean verdict gold, and no SCoTD student exists yet.
To unlock it
A SCoTD student model, the agreed-verdict tiles as gold, and a calibrated judge for the steps code cannot check.
What a win looks like
Runs that are pleasing to the eye, almost like a video game: each failure mode a level the engine clears or fails, one click from what it actually said.
The owner learns real-world AI eval while using it: every term and metric sits beside its sourced card.
The framework knows what to do and leads the user to a working eval: the user explains the golden scenarios, and the tools generate, check and run the rest, with the owner as approver rather than bottleneck.
Now and next
Now
→ in progress
Sprint 3 — the engine's gpt-oss fix
Two faults, both confirmed: the heavily compressed gpt-oss-20b files loop in a second runtime too, and the engine's sliding-window attention breaks once an answer runs past its window. The fix is the engine's attention plus a less compressed conversion of the files, measured by the same suite before and after.
Next, in order
The decision-model judge, certified on the owner's labels where a population is large enough to pay for them: reasoning steps and open answers.
Eval creation the tool leads: the user explains a few golden scenarios; agents working together generate synthetic tasks with DDaC and build the autorater that grades them, checked against the goldens; the user approves a short sample. A first example: ARAIL's support bot.
Autoresearch on QueueLLM: measure the engine, then let agents run experiments on their own to find improvements, with the engine suite as the guard that keeps the output clean, and every step explained beside its sourced term cards.
The engine fix for gpt-oss output on both engine paths, measured by the same suite before and after.
Distilled-model and SCoTD reasoning tasks on the same three pillars, graded by code, the decision model and, for the uncertain tail, a frontier judge.
Agents that curate tasks and keep this page current as direction changes.
Done
Sprint 1 — Measure + See: the instrument, its receipts and audits, the live 3D scene, and the recorded pilot shown on this page.
Sprint 2 — The Task, The Run, The Grader: a small suite of engine-mistake tasks graded by code, a verdict board on this page, and a first measurement of the decision-model judge on stored runs.
Waiting on the owner
Each question carries the default the agents follow until the owner answers it.
Should DDaC's method for synthetic eval data be open or private?
The default meanwhilePrivate until DDaC's rule for new curation methods is met. Either way, every generated task is traced to the golden it came from.
raised
Label a short, blind list of answers ("does this answer name the card?")?
The default meanwhileNothing waits on it. Until it is labelled, no judged figure is shown and the lenient rule stays unvalidated.
raised
Spend a few minutes of engine time to give the judge and the lenient rule more distinct answers?
The default meanwhileYes, in the next sprint.
raised
How far can the decision model be trusted on our own grading decisions?
The default meanwhileMeasured on stored runs and not yet trusted to decide. The next step is the owner's labels on a short, blind list.
raised
How should ASD-STE100 (Simplified Technical English) shape 3D.EVAL?
The default meanwhileA measured column beside the verdict first: three word counts, with no pass or fail. Its rules and limits wait for the owner's copy, and its dictionary stays out of the repo.
raised
Run the engine suite per release rather than per commit?
The default meanwhilePer release for now: on the pinned engine build every gpt-oss answer runs to the token cap. The suite is sized again on the first repaired engine build.
raised
Is ARAIL's support bot the first example for eval creation from golden scenarios?
The default meanwhileYes, as the owner suggested: its support scenarios become the goldens, and it joins the ladder when that sprint starts.
raised
Post the gpt-oss evidence from both engine paths on the engine's issue?
The default meanwhileHeld until the owner says yes.
raised
Decided
Decided by the owner
owner
3D.EVAL is an AI eval framework first, built on The Task, The Run, The Grader.
Sprint 2 decisions log, 2026-10-09
owner
The target ladder: the inference engine, then models we distill, then SCoTD reasoning; start with the engine.
Sprint 2 decisions log, 2026-10-09
owner
This page leads with the direction, kept current by agents; the owner approves.
Sprint 2 decisions log, 2026-10-09
owner
The judge tier is a small, local, calibrated decision model (Laya), not a large generative model; a frontier model (Claude Fable 5.1) is the calibration reference and the escalation for the uncertain tail.
Sprint 2 decisions log, 2026-10-09
owner
Humans explain the golden scenarios; agents working together generate the rest of an eval, with DDaC making the synthetic data and the agents building the autorater, and the tool leads the user to success.
Sprint 2 decisions log, 2026-10-10
owner
Measure QueueLLM and run autoresearch to find improvements, fully autonomous, with the academic side kept so the owner learns what is going on.
Sprint 2 decisions log, 2026-10-10
owner
DDaC is the tool that generates the synthetic data for evals.
Sprint 2 decisions log, 2026-10-10
Defaults taken on the owner's behalf
Each one stands until the owner reverses it.
default taken for you
The judge runs on this machine with no network, and decides nothing until the owner's labels certify it.
Sprint 2 design, the judge tier
default taken for you
Conciseness is measured beside the verdict, not graded.
Sprint 2 design, the conciseness ruling
default taken for you
The first suite: a small set of tasks in existing shapes, one code check per engine failure mode.
Sprint 2 VISION OD2
default taken for you
The fast lane adds gpt-oss-20b on both engine paths; gpt-oss-120b waits for the engine's gpt-oss fix.
Sprint 2 VISION OD3
default taken for you
Nothing in Sprint 2 needs an API key.
Sprint 2 VISION OD8
default taken for you
The suite runs at every engine ship and on the owner's one command; a background watcher comes later.
Sprint 2 VISION OD9
default taken for you
The suite records its first verdict on today's engine build before any engine rebuild, so the fix can be measured before and after.
Sprint 2 VISION OD11
default taken for you
The fast lane checks gpt-oss's streaming path with every layer kept in memory; the slow lane streams for real.
Sprint 2 design, the sizing probe
default taken for you
The verdict board is the visual for suite runs; Sprint 2 adds no new 3D scene work.
Sprint 2 design, the verdict board
Not now
Left for later on purpose, each with where it goes.
ARAIL's support bot as a target
the sprint that builds eval creation from golden scenarios
Generated probe plans
a later sprint
A background watcher that runs the suite on its own
Sprint 3
Live drawing and new 3D scene work for suite runs
after the owner comes back to play a run
gpt-oss-120b in the suite
after the engine's gpt-oss fix
A frontier judge (Claude Fable 5.1)
after an API key and a calibration pilot against a cheaper Claude model
ASD-STE100's rule checks and dictionary
after the owner's copy and ASD's terms
The longest-context tasks
with the slow lane, after the engine's gpt-oss fix
Changelog
How the direction changed, newest first.
Sprint 3 found two faults behind the gpt-oss output: the heavily compressed files, and the engine's sliding-window attention.
Sprint 2 shipped: the engine suite and its verdict board; the decision-model judge measured on stored runs and not yet trusted to decide; the answer check was tightened after the first engine run read sentence fragments as answers.
The owner set the second track: measure QueueLLM and run autoresearch to find improvements, fully autonomous and explained as it goes; DDaC is the tool for synthetic data.
The owner set how evals get made: humans explain the golden scenarios; agents working together generate synthetic data with DDaC and build the autorater; the tool leads the user to success; ARAIL's support bot is the suggested first example.
The first engine run found gpt-oss-20b broken on both engine paths, wider than the framing mistake; the engine suite runs per release until a repaired build is sized again.
The Sprint 2 design: the judge's first numbers are an entrance exam on stored answers, and conciseness becomes a measured column beside the verdict.
The owner named ASD-STE100 (Simplified Technical English) as a way to keep output concise and free of AI slop.
The owner set the judge tier: a small, local, calibrated decision model (Laya) grades what code cannot decide; a frontier judge only sees the uncertain tail.
The visionary's Sprint 2 wedge: code graders decide every engine failure mode, no judge until the reasoning rung, a verdict board as the visual; ladder rungs now state what they need; pending decisions, defaults taken, deferrals and this changelog added.
The owner set the target ladder and the win, and asked that this page lead with the direction and stay current, curated by agents.
The owner pivoted 3D.EVAL to AI eval first: The Task, The Run, The Grader.
Sprint 1 shipped: the instrument and the recorded pilot.
● available now · the engine suite's verdicts on recorded runs, every answer graded by code
An eval asks whether a system made a mistake; a benchmark asks how fast it ran. The engine suite is an eval: small tasks, the same questions on every model and path the engine supports, every answer graded by code, and a board that names the mistake and shows the transcript.
Each config gets one state: CLEAN, MISTAKES, UNVERIFIED, INCOMPLETE or DID-NOT-RUN. Under it sit seven checks, each drawn as a level: cleared, failed, locked until it has run, not applicable, or unverified. Only a cleared level is a pass, and a level that did not run stays locked, in plain view. A failed level opens on the first transcript that failed it.
→ we aim to · a suite run's verdict from one command on the pinned engine, drawn here beside these
10 of 12 rows fail, in 10 of 12 tasks: template debris, a leaked reasoning channel or an encoding error in the output
Read the first failing row 10 of 12 tasks fail
task t01-identify-cb · pattern M2 · from character 0
<|channel|>analysis<|message|>We have a quiz: "the history of the arts. Answer from your own knowledge. Which term matches this description? Description: "~1860–1860–1860–1860–1860–1945: 'the break with tradition—? 'the? '1860–1945? 'the? '
12 of 12 rows fail, in 12 of 12 tasks: no committed answer line
Read the first failing row 12 of 12 tasks fail
task t01-identify-cb · pattern markup:M2 · from character 0
<|channel|>analysis<|message|>We have a quiz: "the history of the arts. Answer from your own knowledge. Which term matches this description? Description: "~1860–1860–1860–1860–1860–1945: 'the break with tradition—? 'the? '1860–1945? 'the? '
12 of 12 rows fail, in 12 of 12 tasks: the output ran to the token cap or repeats itself
Read the first failing row 12 of 12 tasks fail
task t01-identify-cb · pattern L1:We" · from character 1202
We"
Ok. The question: The answer: The question: The question: The question: The question: The answer: The answer: The question: The question: The user wants: The user: The user: The user: The user: The user: The user wants: The user wants
8 of 10 rows fail, in 8 of 10 tasks: template debris, a leaked reasoning channel or an encoding error in the output
Read the first failing row 8 of 10 tasks fail
task t01-identify-cb · pattern R · from character 0
We need to answer: The term that matches description: "~1860–1945: the break with tradition: abstraction, atonality, atonality, stream of consciousness, etc." This? Wait.
We? The? The? The? The? The?
We?
We?
We?
We?
We
We
We
We
We
10 of 10 rows fail, in 10 of 10 tasks: no committed answer line
Read the first failing row 10 of 10 tasks fail
task t01-identify-cb · pattern reasoning-leak · from character 0
We need to answer: The term that matches description: "~1860–1945: the break with tradition: abstraction, atonality, atonality, stream of consciousness, etc." This? Wait.
We? The? The? The? The? The?
We?
We?
We?
We?
We
We
We
We
We
8 of 8 rows fail, in 8 of 8 tasks: template debris, a leaked reasoning channel or an encoding error in the output
Read the first failing row 8 of 8 tasks fail
task art-history-001-identify@closed-book · pattern R · from character 0
The user asks: "Quiz: the history of the arts. Answer from your own knowledge.
Which term matches this description? Description: "~1860–1860– 1860– 1860–1945: the break with tradition—abstraction—abstraction? break with tradition—abstracti
8 of 8 rows fail, in 8 of 8 tasks: no committed answer line
Read the first failing row 8 of 8 tasks fail
task art-history-001-identify@closed-book · pattern reasoning-leak · from character 0
The user asks: "Quiz: the history of the arts. Answer from your own knowledge.
Which term matches this description? Description: "~1860–1860– 1860– 1860–1945: the break with tradition—abstraction—abstraction? break with tradition—abstracti
8 of 8 rows fail, in 8 of 8 tasks: the output ran to the token cap or repeats itself
Read the first failing row 8 of 8 tasks fail
task art-history-001-identify@closed-book · pattern L2:modernism · from character 1615
Let's choose "Modernism"? I'd give "Modernism"? I'd answer: "Modernism"? I'd give "Modernism"? I'd say "Modernism"? The term "Modernism"? Let's output "Modernism"? Might be "Modernism"? The term "Modernism"? I'll output "Modernism? "Modern
Performance and conciseness · measured, never a pass or a fail
Median wall time per task
12.55s
Decode rate
not measuredresident path: decode is not separable from prefill (QA-D6)
Peak engine memory
16568.7MB
Read from disk per answer
16.88GBsystem-wide, not the engine's own
Swap change
-597.6MBsystem-wide, not the engine's own
Words in the answer line, median
2wordsstands on 480 of 480 rows
Words in the whole output, median
2wordsstands on 480 of 480 rows
Longest sentence, maximum
5wordsstands on 480 of 480 rows
Not judged
mock-slice-reader · mechanism check, not a contestant
claude-opus-5-5 · switched off in the plan: no-api-key
gpt-oss-120b-ring4 · switched off in the plan: harness defect QA-D14 (served env-knob check), framing unverified
stored transcripts are not a suite run: there is nothing to compare them with
Judge · not run · no active task has a model criterion
The words on this board
Each word is shown with the card of the AI World that defines it, as the World wrote it, or says that no sourced card exists yet.
template debris
no sourced card yet
no answer
no sourced card yet
runaway output
related card · Stop SequenceA string that, once generated, halts decoding — used to bound output and separate turns.a runaway output does not stop at a stop sequence; it runs to the token limitThe card
A stop sequence is one or more strings that terminate generation when produced, so the model doesn't run on past the intended boundary. They mark turn ends, close structured fields, or cap output. Distinct from the model's learned end-of-sequence token, stop sequences are caller-specified at request time.
Setting a stop sequence of '\nUser:' keeps the model from hallucinating the user's next turn.
source · authored
wrong configuration served
no sourced card yet
nondeterminism
related card · DeterminismWhether a model returns the same output for the same input every time — LLMs are non-deterministic by default.determinism is the property this check testsThe card
A process is deterministic if identical inputs always produce identical outputs. LLM generation is non-deterministic by default: sampling (temperature, top-p) injects randomness, and even at temperature 0, floating-point order and parallel execution (batching, GPU kernels) can cause small variations. You make it near-deterministic with greedy decoding (temperature 0), a fixed random seed, and a pinned runtime.
Ask the same question twice at temperature 0.8 and you get two different answers; drop to temperature 0 with a fixed seed and they match — modulo hardware-level floating-point nondeterminism.
source · authored
path divergence
related card · Layer StreamingLoad one transformer layer from disk, compute, discard — running 400B+ models on tiny VRAM.layer streaming is the path whose answers are compared with the resident path'sThe card
Layer-streaming inference (AeroLLM's core primitive) streams a model layer by layer from SSD: load a layer's weights, compute, free, repeat. It trades latency for the ability to run frontier-scale teachers (70B-671B) on commodity hardware with a few GB of VRAM.
A 671B MoE at Q4 streams off a 1TB SSD on a MacBook — slow per token, but a background swarm does not mind waiting for depth.
card · EvalThe practice of measuring model quality with repeatable tests — from public benchmarks to task-specific graders.The card
An eval is any repeatable measurement of how well a model does something: a public benchmark, a private held-out set, an LLM-as-judge rubric, or a unit-test-style check. Good evals are the steering wheel of model building — without them you cannot tell whether a change helped. QuKaiZen's certification gates are the evals a student model must pass before it graduates.
Before shipping a fine-tune you run an eval suite — MMLU for knowledge, GSM8K for math, IFEval for instruction-following — and only ship if every score holds or improves.
source · authored
benchmark
card · BenchmarkA standardized test set used to measure and compare model capability.The card
Benchmarks score models on fixed tasks — knowledge, reasoning, code — so results are comparable. QuKaiZen's Gate 1 uses MMLU, HellaSwag, ARC, GSM8K, and IFEval to verify capability survives distillation.
A distilled student must retain ≥85% of its base model's MMLU score to pass the regression gate.
Below is the writeup Sprint 1 shipped: the instrument, the pilot of record with its receipts, and the run's own scene.
01 What it is
Measure · See · Learn · Explain · Tune
Underneath the eval is the instrument: it measures anything you can run — an inference engine first, then a chatbot, an agent, a service. A run is one record: what was asked, what came back, how long it took, what the machine was doing, and the receipts to rebuild every figure from its rows.
Measure
Eval and perf in one run: correctness against human-authored, sourced answers, beside latency, memory and disk. A value the run did not measure reads “not measured”, and every accuracy, rate and lift sits beside its interval.
● available now · the run below
See
The run drawn as a rotatable 3D world: the World's terms as points, their relationships as threads, and the cards a question was shown as a shell you can widen or tighten.
● available now · the scene below
Learn
Each term opens its card exactly as the World wrote it, source included. Each metric opens its card from the AI World where one is written, and says “no sourced card yet” where one is still to come.
● available now · in the scene
Explain
A Claude coach reads the finished record and says what the numbers mean, labeled as AI commentary. Code checks every claim against the fields it cites, and removes any claim the record does not back.
→ we aim to · publish its first commentary beside the run it explains
Tune
Experiments that change one setting at a time and keep a change only when its paired difference clears the noise: a system tuned to the machine it runs on, with the before and the after on record.
→ we aim to · the tuning loop, played from receipts
02 The three axes
Knowledge, focus, target.
The “3D” is three axes. Every question in a run sits at one point on each, and every figure belongs to one cell of them.
Knowledge
A sourced World
DDaC's human-authored cards, each citing its origin and gate-checked before it ships. The questions are written from the cards, and the cards are the answer key.
This runA History of the Artsart-history · 103 terms · sha256 d05608d0f8b6
Focus
How much it is shown
The ladder runs closed-book, d1, d2, full: no cards; the answer's card and its one-hop neighbours; two hops; the whole World. A smaller context can mean a clearer answer, and the ladder measures what each step outward costs and buys.
This runclosed-book → d1 → d2 → full
Target
The system under test
Reached through an adapter: a local inference engine, a hosted model as the reference, a mock that checks the mechanism itself.
This runMockClaude · claude-opus-5-5 · not run in this runQueueLLM 1.1.0 · Qwen3-30B-A3B-Instruct-2507-4bitQueueLLM 1.1.0 · gpt-oss-120b-MXFP4-Q4 · not run in this run
03 How a run is measured
Questions from cards, answers on the clock.
Cases
Generated deterministically from the World's hash and a seed. Identify asks which term matches a description; relate-era asks which era a term belongs to, from the World's own list; relate-link asks whether two terms are directly connected, yes or no. The answer key is the card itself, written and sourced by a person.
The focus ladder
Each question is asked at every rung of the run. The cards reach the prompt through DDaC's own TOON encoder, so from rung to rung only the set of cards changes.
Receipts
The manifest pins the World's hash, the DDaC commit, the case digest, the prompts, the grading code and the engine binary. Every row carries the engine's own timing record and the checks that the model, build and prompt it served were the ones requested; one receipt digest seals the set.
Grading
Code grades each answer against the card it was written from, and the run's caption says what “correct” counts. A grade depends only on the answer its row stored, so a newer grader can re-score a finished run from its receipts, with no engine; the figures then name the grader they stand on.
Intervals
A deterministic engine gives the same answer on every rep, so reps measure time, and every interval is clustered by case: the case is the unit. A difference whose interval includes zero is marked within noise.
The big number
Correct answers per minute: the correct answers a target produced, divided by the minutes it spent producing them, at each rung, with its interval. It moves on all three axes — knowledge sets how many answers are right, focus changes both accuracy and the cost of reading the prompt, the target and the machine set the time. It carries no free weights, it is the figure a tuner would push upward, and it is reported at the rung where it peaks, with every rung's interval beside it.
04 The run
The pilot of record.
● available now · the pilot of record, with its receipts
Everything in this section is the run's own record. Each figure is the scorecard's own string, synced from the 3DEval repository at a pinned commit and held to its hash before the site builds.
The big number · correct answers per minute
22.38/min ±5.06(strict grading)
QueueLLM 1.1.0 · Qwen3-30B-A3B-Instruct-2507-4bit, at the closed-book rung
peak rung, selected after the fact; every rung's CI is shown
120/120 rows scored
QueueLLM latencies here start a fresh engine process per request, so they are pessimistic against a warm chat session.
Context cards are seeded at the answer's own card, so the focus ladder is not a retrieval test.
Run
20261008T141059Z-pilotrecorded 2026-10-08
Plan
the pilot plan40 cases (identify · relate-era · relate-link) · closed-book → d1 → d2 → full
World
A History of the Artsart-history · sha256 d05608d0f8b6 · DDaC 1c38512
Mechanism check
Mockmock-slice-reader · 3 reps
Not run
Claude · claude-opus-5-5claude-opus-5-5the receipt's reason: no-api-key
complete960 of 960 planned requests accounted for · 0 error rows
Harness
3d-eval 0.1.0git 2cc973e (dirty)
Grading
programmatic/v3a regrade of the stored answers · the run graded programmatic/v2 · extract/v2 · norm/v1 · lenient/v1
Correct = strict grading: an answer counts only if it is the card's own label in the World, or the card's id (its slug). The card's other names and parts of a compound name do not count. The lenient rule that counts them is not validated by the owner yet, so strict is the headline. Graded by programmatic/v3, a regrade of the stored answers (the run graded programmatic/v2).
Mock · mock-slice-reader · mechanism check, not a contestant
closed-book
0.0%[0%, 8.8%] exact
0.0%[0%, 8.8%] exact
0.00 s
0.00 s
0.00/min±0.00
baseline
baseline
120/120
d1
100.0%[91.2%, 100%] exact
100.0%[91.2%, 100%] exact
0.00 s
0.00 s
334851.26/min±83226.97
+100.0 pts±0.0 pts
+100.0 pts±0.0 pts
120/120
d2
100.0%[91.2%, 100%] exact
100.0%[91.2%, 100%] exact
0.00 s
0.00 s
135056.74/min±18122.69
+100.0 pts±0.0 pts
+100.0 pts±0.0 pts
120/120
full
100.0%[91.2%, 100%] exact
100.0%[91.2%, 100%] exact
0.00 s
0.00 s
56013.35/min±5213.47
+100.0 pts±0.0 pts
+100.0 pts±0.0 pts
120/120
Lift
Each lift compares a rung with closed-book, case by case. Every interval here excludes zero, so every lift on this run clears the noise.
Grading
The strict rule headlines until a person has validated the lenient one against their own labels. On this run: correct_strict: the lenient rule is not-done (validation needs n >= 30 labeled answers and agreement >= 90%)
Did not complete
gpt-oss-120b-MXFP4-Q4 (streams: 58 GB on 36 GB), the configuration that streams from disk, did not complete. On the streaming path QueueLLM frames gpt-oss prompts with a ChatML template instead of the model's own Harmony format, so the model's answers never committed: each request ran to the token limit with no final answer. A probe whose framing gate was written down before it ran showed this on every request it made, and that outcome — did not complete, framing defect — is recorded with the probe's receipts. This run's receipt was closed before the probe, so the reason it gives above is the one known then.
→ we aim to · measure it once the engine frames it correctly
Riskiest assumption
That a focus lift measures what a system gains from its context, rather than how the questions were built or graded. It is decided at the full rung against closed-book, by splitting the lift into knowledge and vocabulary. On this run: not-evaluable — the lenient rule is not-done. The scorecard's caption: “not evaluable: lenient rule not validated by the owner”.knowledge +25.0 pts ±14.0 pts · vocabulary +7.5 pts ±8.5 pts · within noise
Determinism
mock-slice-reader: 160 case–rung pairs compared across reps; every rep returned the same answer.
Determinism
qwen3-30b-a3b-resident: 160 case–rung pairs compared across reps; every rep returned the same answer.
Null battery
For every case kind — identify · relate-era · relate-link — a constant answer scores within chance plus its noise margin, so every kind counts toward the aggregate lift.
In the run's own words · the manifest
Each QueueLLM answer here is a fresh queuellm generate process. The model is mapped and loaded for every request; there is no KV or prefix reuse between requests; and the page cache holds whatever the previous request left. A warm chat session pays the load once. These latencies are therefore pessimistic against a warm session by at least the load time per request. That is measured as load_s on the streaming path and bounded by load probes on the resident path.
run 20261008T141059Z-pilot · receipt sha256 705438cbd592 · source commit 7f98269 · graded by programmatic/v3, a regrade of the stored answers (the run graded programmatic/v2)
05 The scene
The run, drawn.
● available now · the run's own scene, byte for byte
Each term in the World is a point on the sphere. The World's era spine runs pole to pole, its categories divide the longitudes, and threads join related terms. Choose a node to read its card exactly as the World wrote it; slide the focus to see the cards a question was shown; switch the metric to repaint the sphere from the rows. Drag to turn it, scroll to zoom.
On a narrow screen the frame keeps its width: scroll it sideways to reach the sphere.
One self-contained file, inert by design: everything it draws is embedded in it, it runs under its own strict content policy, and it is the run's own scene, byte for byte · sha256 2d7a93627784
The run was recorded on one machine — Apple M5 Max, 36 GB of unified memory — with the disk and memory sampled around every request and the engine's own timing record kept beside them in each row.
Each answer is a fresh engine process, and the record shows what that costs. Every request reads roughly the whole model checkpoint back from the SSD, so the work is read-intensive by construction, once per request. Below the full rung the disk windows are read-heavy, reads dwarfing writes in nearly every one. At the full rung, with the whole World in every prompt, memory pressure climbs, the windows read more than the checkpoint, and large system-wide writes sit beside the reads.
At the closed-book rung, starting the process and loading the model take most of each answer's time; once cards are in the prompt, reading them takes a growing share. A warm chat session pays the load once. That is why these latencies are pessimistic, and why we aim to report a warm-equivalent figure beside each cold one.
Disk and memory readings are system-wide; the engine's resident memory is its own. The per-request readings are in the run's receipts.