What we measure, and what came out.
Small, hand-written test sets, balanced so that a model which always answers and one which never does both score zero. They show whether something works, not how it ranks against other systems.
Measured · 18 September 2026Does it say when the source does not cover the question?
56 questions about short passages, half of them answered by the passage and half not. Discrimination is the refusal rate on unanswerable questions minus the refusal rate on answerable ones: 0 means it guesses, 100 means it never mixes them up.
Lumen Small
- Answered correctly
- 100.0 %
- Declined when not covered
- 92.9 %
- Made something up
- 0.0 %
Lumen Medium
- Answered correctly
- 100.0 %
- Declined when not covered
- 89.3 %
- Made something up
- 0.0 %
Lumen Large
- Answered correctly
- 92.9 %
- Declined when not covered
- 92.9 %
- Made something up
- 0.0 %
Does the consistency layer separate supported from unsupported claims?
60 claims, each with its evidence, half supported and half not; every case scored by weights fitted without it.
AUC
On this set it ranks every supported claim above every unsupported one.
ECE
The score itself is not yet well calibrated; it says about 0.8 for claims that are all supported, so it is too cautious. That is the next thing to fix, and the reason the score is not shown as a percentage in answers yet.
Does it use the right tool and the right graphic?
Eleven everyday tasks, each run twice through the real agent: live prices, org charts, diagrams, comparisons, timelines, formulas, maps, answering in the user’s language, writing a file on the computer, computing on an attached spreadsheet.
Lumen Small
Lumen Medium
Lumen Large
Notes from building it.
No papers and no comparison tables. What is here are decisions with the reasoning attached, including the ones that later turned out to be wrong.
It is a consistency layer, not a veracity layer
What gets measured is agreement between a claim and its evidence, not truth. The distinction sounds like hair-splitting and is the most consequential naming decision in the system.
Why accuracy is a worthless metric for this
On a set that is 85 percent consistent, a model that always answers consistent scores 85 percent and has learned nothing. We measure ECE and AUC on a balanced set instead.
A cascade beats running both models
Inference is paid forever, training once. A router that only escalates when it has to is several times cheaper, at a cost that shows up in the tail rather than the average.
Code belongs in pretraining, not in a later phase
A model that meets code only after language has settled has to learn it against established representations. That is more expensive and gives a worse result.
How a tokenizer splits numbers decides whether totals survive
A tokenizer trained on the target language saves around a fifth of the tokens, and how it handles digits decides whether arithmetic is learnable at all.