2026-08-10

How we treat benchmark scores

Two score rows may share a table only if their comparability keys match. Tier A sources may be republished; Tier B is cite-and-link; Tier C is link-out only. The build fails on redistribute:false.

The benchmark desk exists to correct a specific dishonesty: treating numbers from different harnesses, shots, or scoring methods as if they ranked the same thing. Every score row carries a comparability_key of the form benchmark_id / benchmark_version / split / n_shot / scoring_method / harness_family. Mixed keys render in separate panels with a one-line explanation of what differs. There is no normalize-across-harnesses feature. Licensing is enforced at build time. We republish only Tier A material. A redistribute:false row reaching a rendered artifact fails the build. Full policy lives on /methodology and /sources.