PolicyEngine

Process

The full production story behind the panel: the exact prompt every model received, how 11,310 runs became the tables on this site, what the elicitation cost, and where the project goes next. Methods covers the estimand, pooling, and per-model harness; this page covers how the data came to exist.

29 models26 quantities11,310 runs

The prompt, verbatim

One template covers all 26 quantities: a memory-only preamble, the quantity's definition block (name, definition, target interpretation, population, units), and a task list that requests five subjective quantiles, a point estimate pinned to the median, up to three recall-anchor citations, and a brief reasoning summary — returned as JSON in the shape printed at the bottom of the prompt. Below is the text for the elasticity of taxable income, read from the archived request logs — every one of the 29 models received this identical text. Each quantity page shows its own text under "What the models were asked."

Answer from your current memory and background knowledge only.
Do not use tools, files, the web, code, or external resources.
Do not try to reconstruct a literature review or search for a consensus estimate.
Report the belief you currently endorse.

Quantity of interest:
- Name: Elasticity of taxable income
- Definition: Percent change in taxable income from a 1 percent change in the net-of-tax rate.
- Target interpretation: Medium-run ETI for top earners with avoidance and real responses combined
- Population/context: High-income taxpayers in the United States
- Units: elasticity

Task:
1. Use exactly the target interpretation above. In `interpretation`, restate it briefly.
2. Give your subjective quantiles p05, p25, p50, p75, and p95 for this quantity.
3. Set `point_estimate` equal to `p50`.
4. Make the quantiles weakly increasing and numerically coherent.
5. In `citations`, list up to 3 source anchors from memory that influenced your belief. These are recall anchors only. If none come to mind confidently, return `[]`.
6. Keep `reasoning_summary` brief and substantive.

Return valid JSON only with exactly this shape:
{
  "interpretation": "...",
  "point_estimate": <number>,
  "quantiles": {
    "p05": <number>,
    "p25": <number>,
    "p50": <number>,
    "p75": <number>,
    "p95": <number>
  },
  "citations": ["..."],
  "reasoning_summary": "..."
}

Two sign-ambiguous quantities — income elasticity of labor supply and capital-gains realizations — carry an extra sign-convention block that defines what the reported sign means without calling either direction correct. That clarifier text was revised two days into the April 2026 wave: seven April models keep the original wording, and the four April premium models were re-elicited under both. The affected quantity pages carry the model-by-model comparison — the two headline quantities barely move, while the net-of-tax sibling moves materially for two models (paper, Appendix Tables A18–A19). How the JSON is enforced (strict schema, forced function call, or JSON-object mode) follows each provider's API surface and is disclosed per model in the Methods harness table.

From prompt to panel

  1. Registries fix the design. Quantity definitions live in quantities.toml; the model roster, waves, and frontier flags live in model_registry.py. Analysis, paper tables, and this dashboard all read the same two registries, so a model or quantity exists everywhere or nowhere.
  2. The runner elicits each cell. run_v4_new_models.py runs 15 independent draws per model-quantity cell through each provider's structured-output path, one quantity per subprocess so request-log cost aggregation survives provider quirks. Completion budgets are truncation guards — reasoning tokens count against them on models that reason, so budgets were raised where required rather than tuned.
  3. A grid gate admits the panel. check_panel_grid.py verifies every cell against the exact 15-run parsed grid before anything downstream runs. Failed slots re-run as fresh independent draws and every failure is archived, as Methods discloses.
  4. Aggregation turns runs into intervals. Each run's five quantiles become a piecewise-uniform distribution; the headline interval is the equal-weight 90 percent mixture across the 15 runs, with REML and Bayesian hierarchical estimators alongside — Methods carries the estimand discussion.
  5. Tables build, prose is pinned, pages generate. build_tables.py regenerates every paper table from the committed artifacts, and verify_paper_prose.py pins the paper's quoted numbers to those tables so stale claims fail instead of shipping. This site is statically generated from the same results tree and table CSVs at build time — there is no separate dashboard database to drift.

What it cost

The paper's Data section carries the full accounting: $66.59 of request-log-tracked spend — $23.15 for the April main-panel rerun, $37.74 for the July frontier additions across their main panel and clarify probes, $2.71 for the late Grok 4.5 addition, and $2.99 for the late Gemini 3.6 Flash addition. The five original Chinese-lab models cost $29.25 at the OpenRouter account level, including the failed calls their recovery replaced; per-request costs for those models, for Kimi K3 (logged tokens imply roughly $10.69 at list prices), and for the GPT-5.6 family are untracked in the request logs and appear as em dashes below rather than as zeros. Dispersion among tracked models is large — the most expensive, Claude Fable 5, whose always-on reasoning is billed as output tokens, costs roughly 310 times the cheapest, Grok 4.1 Fast.

ModelServing pathRunsTokensTracked costPer run
Claude Fable 5anthropic_native390668,690$16.80$0.043
Claude Opus 4.7litellm_completion390866,837$8.34$0.021
Claude Opus 4.8anthropic_native390549,552$5.42$0.014
GPT-5.5openai_chat_completions390281,245$5.42$0.014
Claude Sonnet 4.6litellm_completion390675,522$4.28$0.011
Gemini 3.5 Flashlitellm_completion390597,065$4.16$0.011
Grok 4.20litellm_completion390838,492$3.45$0.00884
Gemini 3.1 Prolitellm_completion390410,998$3.31$0.00850
Gemini 3.6 Flashlitellm_completion390503,023$2.80$0.00719
Claude Sonnet 5anthropic_native390601,219$2.68$0.00688
Grok 4.5litellm_completion390701,676$2.49$0.00638
GPT-5.4openai_chat_completions390122,735$1.35$0.00347
Claude Haiku 4.5litellm_completion390629,059$1.20$0.00308
Grok 4.3litellm_completion390514,688$0.69$0.00178
GPT-5.4 miniopenai_chat_completions390116,358$0.38$0.00097
Gemini 3 Flashlitellm_completion390256,753$0.37$0.00095
Gemini 3.1 Flash-Litelitellm_completion390323,960$0.29$0.00073
GPT-5.4 nanoopenai_chat_completions390127,281$0.12$0.00030
Grok 4.1 Fastlitellm_completion390399,672$0.06$0.00015
Claude Opus 5anthropic_native390754,959
DeepSeek V4 Proopenrouter_via_litellm390514,993
GLM-5.2openrouter_via_litellm390430,672
GPT-5.6 Lunaopenai_chat_completions390198,449
GPT-5.6 Solopenai_chat_completions390232,582
GPT-5.6 Terraopenai_chat_completions390168,285
Kimi K2.6openrouter_via_litellm3902,903,796
Kimi K3openrouter_via_litellm390825,168
MiniMax M3openrouter_via_litellm390659,388
Qwen 3.7 Maxopenrouter_via_litellm3901,045,361

Tracked cells above sum to $63.61 across the main elicitation batches this site serves (19 of 29 models); the paper's $66.59 additionally includes the clarify probes archived alongside them. Costs are estimated from logged token usage at provider list prices; sources and as-of dates are recorded in pricing.py.

Archived batches outside the main panel

Every other archived batch in the results tree: the Armington and IES clarify probes feed the paper's appendix cross-prompt tables, and the ablation, pilot, and connectivity-probe batches are kept for provenance. None of these runs enter the headline panel.

All 60 archived batches
BatchRunsTokensTracked cost
_litellm_probe11,631$0.00
claude-fable-5-armington-clarify-batch151524,360$0.55
claude-fable-5-ies-clarify-batch151527,301$0.65
claude-haiku-4.5-armington-clarify-batch151524,622$0.05
claude-haiku-4.5-ies-clarify-batch151526,127$0.05
claude-opus-4.7-armington-clarify-batch1515
claude-opus-4.7-ies-clarify-batch151536,203$0.36
claude-opus-4.7-mechanism-ablation-batch15135193,770
claude-opus-4.8-armington-clarify-batch151521,665$0.21
claude-opus-4.8-ies-clarify-batch151523,511$0.23
claude-opus-5-armington-clarify-batch151528,458
claude-opus-5-ies-clarify-batch151528,723
claude-sonnet-4.6-armington-clarify-batch151525,979$0.16
claude-sonnet-4.6-ies-clarify-batch151528,205$0.18
claude-sonnet-5-armington-clarify-batch151522,652$0.09
claude-sonnet-5-ies-clarify-batch151524,276$0.10
deepseek-v4-pro-armington-clarify-batch151518,344
deepseek-v4-pro-ies-clarify-batch151520,607
gemini-3-flash-preview-armington-clarify-batch151122,913$0.05
gemini-3-flash-preview-ies-clarify-batch15923,557$0.05
gemini-3.1-flash-lite-preview-armington-clarify-batch151510,154$0.01
gemini-3.1-flash-lite-preview-ies-clarify-batch151511,057$0.01
gemini-3.1-pro-preview-armington-clarify-batch15924,165$0.22
gemini-3.1-pro-preview-ies-clarify-batch15425,080$0.23
gemini-3.5-flash-armington-clarify-batch151524,470$0.17
gemini-3.5-flash-ies-clarify-batch151523,052$0.15
gemini-3.6-flash-armington-clarify-batch151518,726$0.10
gemini-3.6-flash-ies-clarify-batch151517,814$0.09
glm-5.2-armington-clarify-batch151518,827
glm-5.2-ies-clarify-batch151519,338
gpt-5.4-armington-clarify-batch15154,619$0.05
gpt-5.4-ies-clarify-batch15155,218$0.06
gpt-5.4-mini-armington-clarify-batch15154,430$0.01
gpt-5.4-mini-ies-clarify-batch15154,954$0.02
gpt-5.4-nano-armington-clarify-batch15155,129$0.00
gpt-5.4-nano-ies-clarify-batch15155,751$0.01
gpt-5.5-armington-clarify-batch15156,311$0.15
gpt-5.5-ies-clarify-batch15158,123$0.20
gpt-5.6-luna-armington-clarify-batch15157,412
gpt-5.6-luna-ies-clarify-batch15157,009
gpt-5.6-sol-armington-clarify-batch15158,543
gpt-5.6-sol-ies-clarify-batch15157,222
gpt-5.6-terra-armington-clarify-batch15157,400
gpt-5.6-terra-ies-clarify-batch15156,981
grok-4.1-fast-armington-clarify-batch151515,717$0.00
grok-4.1-fast-ies-clarify-batch151516,910$0.00
grok-4.20-armington-clarify-batch151537,490$0.16
grok-4.20-ies-clarify-batch151543,597$0.19
grok-4.3-armington-clarify-batch151520,981$0.03
grok-4.3-ies-clarify-batch151521,397$0.03
grok-4.5-armington-clarify-batch151531,019$0.12
grok-4.5-ies-clarify-batch151530,359$0.11
kimi-k2.6-armington-clarify-batch151566,038
kimi-k2.6-ies-clarify-batch151579,695
kimi-k3-armington-clarify-batch151525,764
kimi-k3-ies-clarify-batch151532,717
minimax-m3-armington-clarify-batch151526,628
minimax-m3-ies-clarify-batch151528,511
qwen-3.7-max-armington-clarify-batch151545,568
qwen-3.7-max-ies-clarify-batch151551,036

Tracked cells across these archived batches add $4.85; "partial" marks batches where some cost cells are untracked, so the batch total is a floor, not a sum.

What comes next

  • New models join as labs ship. The registry-first recipe — add the model to the registry and harness config, elicit the identical panel, pass the grid gate — is how the panel grew from 11 models to 29, and it is how the next model joins. The README documents the path end to end.
  • Capability re-pins with each PolicyBench release. The capability correlations pin to a named PolicyBench release; each new release that scores more of the panel refreshes the cut and gives the post-hoc run-consistency association — the strongest and least protected number in the paper — an out-of-sample test.
  • Tool-enabled elicitation is the designed contrast. The v4 panel is deliberately memory-only; the prompt builder already supports a tools-on regime, and comparing what a model answers from memory against what it answers with search and code in hand is a natural next experiment.
  • The paper hardens toward review. Every number quoted in the manuscript stays pinned to the generated tables by the verification script as the panel grows, so review revisions and model additions cannot silently disagree.

11,310 successful runs · elicited April and July 2026 · v4 prompts · 15 runs per model-quantity cell. Code · Raw responses · Paper (PDF)

Code and dataElicited April and July 2026 · 29 models · v4 prompts