Every 15 minutes the evaluator subscribes to all configured relays, observes a 10-minute window of #commit events, builds a cross-relay truth set of what verifiably happened, and scores each relay against it. Philosophy: truth = what verifiably happened (inclusive), scoring = fair aggregation (volume-capped), classification = why a miss happened (policy ≠ breakage).

1 · truth construction

Truth is the set of (did, rev) commit identities seen by at least K distinct quorum groups (default K=3) among the live references. A quorum group is the relay's backend if configured (relays that are fronts for the same physical backend count as one vote), else its operator, else the host itself — so one operator running many relays, or one backend behind many hostnames, can never vouch a commit into truth alone.

reference tightening

After a provisional truth set is built, each live reference's balanced coverage of it is computed. The survival threshold is relative, not absolute:

threshold = min(0.99, median(reference balanced coverage) − 0.015)

References at or above the threshold rebuild the final truth set. If fewer than K groups survive, the top K groups by balanced coverage are used instead — the evaluator never falls back to the loose (untightened) reference set, which historically let sub-1%-coverage relays vote on truth during spam bursts. Each run records truth_quality = tight or topk.

skew diagnostics

Each run stores truth_unique_dids and truth_top_did_share (the largest single DID's fraction of truth). The dashboard shows a red skew badge when the top DID exceeds 2% of truth — a single account dominating the window means raw coverage is measuring rate-limit policy, not relay quality.

2 · balanced coverage (headline)

Raw commit coverage is micro-averaged, so one account emitting 37k commits in 10 minutes can control 15 points of the score. The headline metric instead winsorizes each DID's contribution:

n_d      = number of truth commits for DID d
C        = max(20, p99 of n_d over all truth DIDs)   # winsorization cap
weight   = min(n_d, C) / n_d                          # per-commit weight for DID d
seen_d   = relay's truth-intersected commits for DID d

balanced = Σ_d ( seen_d · min(n_d, C) / n_d )  /  Σ_d min(n_d, C)

Balanced coverage equals raw coverage when no DID exceeds the cap. A spam DID at 37,000 commits per window contributes at most C commits to the denominator (≈0.01% of a typical window) instead of ~15%.

supporting metrics

• event completeness — events-in-window vs. the reference median, capped at 1.0.
• live ratio — events arriving in [WindowStart, WindowEnd) ÷ events arriving in [WindowStart, WindowEnd + grace). The warmup period is excluded from the denominator, so a steady stream reads 100%.
• freshness samples — recorded only for commits that are in the wall window and have an in-window TID and pass the liveness check, so old-rev backfill no longer pollutes p95. Samples are clamped to [0, 24h] and commits with implausible TIDs (before 2023-01-01 or >10min in the future) are excluded from freshness only.

3 · grace window

Collection runs until WindowEnd + arrival_grace (default 10s — the maximum delivery latency tolerated of a reference). Events arriving in the grace tail still count toward the relay's seen-commit set (if their TID is in-window and they pass liveness) but not toward wall-clock counters or freshness. Without it, a slower-but-complete relay would carry a structural penalty of max(0, latency − edge) / 600 per run purely for delivery latency. The classifier additionally forgives misses whose TID falls within edge_seconds (default 10s, was hardcoded 2s) of the window boundary; with the grace window this bucket should be near zero and is kept as a guardrail.

4 · miss taxonomy

Every truth commit a relay did not deliver is classified:

DID→PDS resolutions are cached persistently (7d TTL for active/deactivated, 6h for unresolvable); resolver 429/5xx responses are retried once with backoff, then land in unclassified rather than being miscounted as unresolvable. Network lookups are capped at 20,000 DIDs per run; misses beyond the cap are unclassified.

host-status probing

After each run, for every relay × its top-5 missed PDS hosts, the evaluator issues read-only GET /xrpc/com.atproto.sync.getHostStatus?hostname=<pds> requests against the relay — two samples ≈20s apart — yielding the relay's cursor (seq) into that PDS and its consumption rate. The cross-relay head seq for a PDS is the max seq any relay reports (seq spaces are comparable per host, never across hosts). Each (relay, pds) pair is classified:

Implementation quirk: rsky relays report every host active, so status-based classes (banned/throttled) are never taken from an rsky status field — only seq evidence counts there. Each missed PDS is probed against every connected relay (not just the ones that missed it) so the head baseline comes from a relay that actually keeps up.

The drawer shows the probe verdict as a chip next to each missed PDS, e.g. ⊘ limited · 0.2 ev/s · 13.1M behind.

5 · tiers

Single-window ranks inside the top cohort are statistical noise (~13% of adjacent ranks swap run-to-run on <0.1pp differences). The tier column groups relays whose trailing-24h performance is indistinguishable: per relay, the median balanced coverage over its non-invalidated ok rows in the last 24h, with a bootstrap 95% confidence interval (1000 resamples). The tier score is availability-weighted: median × ok-fraction, where ok-fraction is the share of scheduled windows the relay was measurable in (quality ok, not invalidated). Without this, a relay whose bad runs go thin/invalidated — e.g. a cursor-lag oscillator delivering stale backfill every other window — would tier on only its good runs (survivorship bias). Relays sorted by weighted score; a relay joins the current tier if its (availability-scaled) CI overlaps the tier leader's, otherwise it starts the next tier. Order within a tier is not meaningful. A relay needs at least 8 runs (~2h) of history before it holds a tier — until then it shows · and sorts after tiered relays, so one clean cold-start run can't out-rank relays with real history.

6 · known artifacts

• TID skew tolerance — commits are accepted with TIDs up to 60s outside the wall window to tolerate PDS clock skew. A PDS with a badly skewed clock can push its commits into the edge bucket (forgiven) or, beyond the tolerance, out of truth entirely.
• reference self-vouching — reference relays are scored against a truth set they helped build, a small upward bias for references. Leave-one-out truth is a known improvement, not yet implemented.
• edge handling — commits created in the last seconds of the window are judged by TID plus the grace window; the residual edge bucket is forgiven, so a relay systematically dropping end-of-window events within edge_seconds would not be charged for them.
• reference-set disclosure — every run's payload carries its reference_set; the status-bar tooltip on the leaderboard lists the relays that voted on that run's truth.

7 · notes

Runs recorded before 2026-06-12 predate some of these metrics: balanced coverage falls back to raw coverage, the policy/backlog/unclassified buckets are empty, live ratio carried a structural ~95.2% ceiling, and freshness percentiles may include backfill samples. Cross-era comparisons of those metrics are not apples-to-apples.

Not yet implemented: leave-one-out truth for references; #identity/#account/#sync completeness scoring; a per-PDS coverage matrix from full DID→PDS attribution of truth; freshness Apdex.