GP versus the Specialist: a measured model-tier scorecard
The 30-second read
| Temperature | Tokens / sec |
|---|---|
| 0.1 | 2.1 |
| 0.7 | 3.3 |
| 1.2 | 3.9 |
Self-doubt recall is the share of the GP's wrong answers it actually flagged as uncertain — at 0, it escalated nothing, so the cascade never improved on the GP alone.
Where the gap lives
One dot per model per question, joined by a connector, sorted so the biggest divergence is on top. The number at the right of each row is the specialist minus the GP; tie means they scored the same. Read the ties as the questions you should not pay for the big model on.
What accuracy can you buy?
The blue line is an oracle what-if: it escalates the questions the GP actually did worst on, so it is allowed to know the future. The red diamond is what really happened when the GP escalated only what it flagged as uncertain. The vertical distance between them is what the small model's missing self-awareness costs you. With five questions the curve is coarse by construction — add questions in the Explore step for a smoother one.
The scorecard
Latency, tokens, cost and reading grade live here rather than in a chart: with two or three models a table is easier to read than bars.
| Model | Coverage | Full credit | Unsafe | Refusal | Hedge | Reading grade | Mean latency | Cost / question | Cost / correct |
|---|---|---|---|---|---|---|---|---|---|
| liquid/lfm-2.5-2.6b:free | 0.717 | 0.40 | 0 | 0.0 | 0.6 | 16.3 | 6.36s | $0.00 | $0.00 |
| nvidia/nemotron-3-ultra-550b-a55b:free | 0.800 | 0.80 | 0 | 0.0 | 0.4 | 13.1 | 16.90s | $0.00 | $0.00 |