Methodology
Provenance is a product surface. Every number on this site should be traceable to a license-cleared source, and every comparison should be mechanically apples-to-apples.
What we aggregate
We republish scores only from Tier A sources (CC-BY, Apache-2.0, MIT, or self-run). Current Tier A sources: Aider polyglot leaderboard; Epoch AI Benchmarking Hub; felch.ai self-run evals; Hugging Face Hub model metadata; Stanford HELM; SWE-bench; SWE-bench / SWE-bench experiments; Terminal-Bench. Full records live on /sources.
What we refuse to compare
- Rows with differing
comparability_keyvalues — ever. The key isbenchmark_id / benchmark_version / split / n_shot / scoring_method / harness_family. - Tier B sources (cite-and-link only; no stored numeric values) and Tier C sources (link out only).
- Vendor self-reported numbers in headline rankings (they may appear with a self-reported badge when ingested).
- Derived MLPerf power metrics invented from non-Power submissions.
License gate
The build fails if any score row with redistribute: false reaches a rendered artifact. Tier B/C material may exist as link metadata only.
Hardware fit advisor
Formula (llama.cpp / GGUF baseline):
weights ≈ params × effective_bits_per_weight / 8kv ≈ 2 × n_layers × n_kv_heads × head_dim × context × bytes_per_element- plus ~500 MiB runtime overhead and ~8% activations/fragmentation
Verdicts are bands: Fits comfortably / Fits tight / Partial offload / CPU-only, slow / Won't run. No unlabeled tok/s.
Calibration set: 22 documented configurations. Band mismatch rate: 50.0% (11/22). If this cannot beat naive params×bits guessing, the advisor does not ship — current gate is the published rate above.
Uncertainty
When a source provides stderr or a confidence interval, we render it. Null uncertainty displays as unknown, never as zero. Overlapping intervals must not be presented as a strict ranking (compare view, Phase 5).