Benchmarks
Benchmark methodology
This page is the contract for GET /benchmarks. Every figure on that page is derived from data the coordinator already exposes (or already stores on hosts). Nothing here is a synthetic lab run, a vendor quote, or a number invented to fill an empty cell.
Live JSON: GET /v1/slo (uptime + completion latency) and GET /v1/stats (online hosts, per-model provider counts, hardware inventory). The HTML page is a 5-minute cached rendering of the same snapshot plus host-level TTFT/TPS EWMAs.
We do not fingerprint, prioritize, or otherwise special-case Artificial Analysis (or any other rater). Host selection is scored in workers/src/routing.ts from reputation, TTFT/TPS EWMA, trust, Connect readiness, and least-recently-assigned — never from User-Agent, IP, or account identity of the buyer.
What we measure
| Figure | Source | Formula | Window |
|---|---|---|---|
| 7d / 30d success | /v1/slo | completed ÷ (completed + failed) | trailing 7d and 30d |
| Completion latency p50 / p95 | /v1/slo | nearest-rank percentile of completed_at − created_at on completed jobs | same windows |
| Online hosts / availability | /v1/slo and /v1/stats | distinct hosts with a heartbeat in the last 45 s that advertise the catalog id | point in time |
| TTFT p50 / p95 (per trust tier) | hosts.ttft_ewma_ms | nearest-rank percentile of those EWMAs among online hosts advertising the model at that trust_level | EWMA over completed jobs (α = 0.3) |
| Median TPS (per trust tier) | hosts.tps_ewma | nearest-rank p50 of those EWMAs, same host set as TTFT | EWMA over completed jobs (α = 0.3) |
Unsettled jobs (queued / leased / running) are never counted in success or latency. Failed jobs lower the success rate and contribute no latency, TTFT, or TPS sample.
Empty cells render as an em dash. That means not enough samples, not zero milliseconds and not 100 % success. A sparse network (no online hosts and no settled jobs in 7 days) also shows the banner copy “Not enough samples yet”.
Definitions (honest)
Time to first token (TTFT). The coordinator records time-to-first-token on job completion and blends it into a per-host exponentially-weighted moving average (ewma() in workers/src/reputation.ts, α = 0.3). The public page does not recompute TTFT from raw token timestamps — those are not stored. It reports the distribution of the host EWMAs that already drive routing. That is a trailing, host-level signal, not Artificial Analysis’s 72-hour median of independent API probes.
Tokens per second (TPS / output speed). completion_tokens / (elapsed_ms / 1000) on each completed job, blended with the same EWMA. Jobs with no completion tokens or no elapsed time are skipped (they do not record a 0).
Completion latency. End-to-end completed_at − created_at. This includes queue, lease, host load, generation, and result relay. It is not the same as TTFT, and it is not “time to 100 output tokens”.
Success rate. Settled-job reliability. The coordinator has no 4xx user-error class to subtract, so this is completed ÷ settled, not OpenRouter’s “success ÷ total excluding 4xx”.
Trust tier. none | software | edr | hardware as documented in docs/security/trust-levels.md. 7-day success and completion latency are per model, not per trust tier — jobs are not rolled up that way on /v1/slo. The per-tier table is host coverage + TTFT + TPS only. Do not read a software-tier TTFT as “software-tier success”.
Percentiles. Nearest-rank, identical to /v1/slo (percentile() in workers/src/slo.ts). For two samples [100, 300], p50 is 100 and p95 is 300. We do not interpolate.
How to reproduce
GET /v1/slo— per-modelwindows["7d"].success_rate,windows["7d"].latency.p50_ms/p95_ms,online_hosts.GET /v1/stats—online_hostsandmodels[].providers(must agree with SLO host counts for advertised catalog ids).GET /benchmarks— HTML rendering of (1) plus per-tier TTFT/TPS. Cachedpublic, max-age=300. NoAuthorizationheader.- Compare a model card’s
data-model-success-7dto/v1/slo. They must match for the same coordinator clock.
Host-level EWMA values are not on the public JSON surface (they would leak per-host identity if emitted raw). The page only emits percentiles over the online set, with no host ids, account ids, emails, prompts, or completions.
What we do not claim
- These numbers are not Artificial Analysis scores. AA measures from their own probes under their methodology.
- We do not report quality / intelligence (MMLU, coding evals, etc.).
- We do not report “peak tokens/s on a dedicated GPU”. Hosts are idle consumer and workstation machines; load, residency, and quantization vary.
- Quantization and context length for a served model are the catalog row (
models.quant,models.context_leninworkers/src/catalog.ts). Hosts may still fail a job if they cannot actually load that quant. hardwaretrust is reserved and never granted today. A hardware-tier row with 0 hosts is expected.
Cache and privacy
GET /benchmarks sets Cache-Control: public, max-age=300. The payload is aggregate and pseudonymized. Prompt and response bodies never appear (the page never reads request_json / result_json).