Real work.
Repeated.
Then checked.
How fast can your computer write with local AI? We time the same model and prompts on every machine, repeat the test three times, and check the evidence before ranking a result.
Count the output.
Time the work.
A token is a small piece of text, not always a whole word. We count the tokens the model actually writes.
From launch to rank
What happens when you press run?
Explore each stage. Every competitive run follows this order, and every unscored step is kept outside the throughput numerator.
Explore the process
Follow one score from machine to leaderboard.
Preflight
Find interference before it becomes a score.
The CLI inventories the real machine and checks visible GPU load, occupied VRAM, temperature, CPU contention, and available model memory before measuring.
- GPU load >= 25% blocks ranking
- VRAM occupancy >= 50% blocks ranking
- GPU temperature >= 82 C blocks ranking
Why it mattersA sensor the operating system does not expose is marked not evaluated; it is never invented.
Score construction
Three metrics. Three different questions.
The primary score is decode throughput. Prefill and first-token decode latency remain visible because a single number cannot describe the full interaction.
Writing speed (decode)
median( 256 / decode_seconds_i)Each of the three canonical prompts generates exactly 256 tokens. The median prompt rate becomes the pass score in tokens per second.
Prompt reading speed (prefill)
sum(prompt_tokens_i)/sum(median_prefill_s_i)Every prompt is expanded deterministically to 2,048 tokens. Three real probes are timed per prompt; median_prefill_s_i is the median of those three probe times.
Time to the first token after reading
median( first_token_ms_i)The historical API field is TTFT p50, but its clock starts after prefill. It measures the first decode token, not end-to-end request latency. Lower is better.
Keeping prefill and first-token decode separate avoids charging long prompts twice and lets readers diagnose whether prompt processing or generation is the bottleneck. It also means VRAM Check must not describe the current field as full request TTFT.
Strict repeatability
A result should be repeatable. Not just fast once.
The full pass runs three times. Relative range catches disagreement; a weighted distance to the three medians selects one internally coherent pass for the published score.
| Pass | Decode | Prefill | First token | Decision |
|---|---|---|---|---|
| 01 | 157.3 tok/s | 2,320 tok/s | 351 ms | Observed |
| 02 | 158.1 tok/s | 2,338 tok/s | 347 ms | Observed |
| 03 | 157.8 tok/s | 2,326 tok/s | 349 ms | Representative |
Relative range
(max - min) / mean x 100All three gates must pass. Coefficient of variation is reported for diagnosis, but relative range is the eligibility rule.
Representative distance
0.50 x delta decode + 0.30 x delta prefill + 0.20 x delta first-tokenEach delta is normalized against that metric's median. The lowest combined distance wins; decode breaks an exact tie.Fail-closed eligibility
A stable number is necessary. It is not sufficient.
Ranking is the intersection of measurement integrity, execution evidence, system health, and release trust. A useful run can be saved without being promoted to competitive truth.
Profile, methodology, prompts, model revision, file size, and model hash must match.
GPU backend, active offload, device identity, and execution profile must agree.
Visible preflight contention, degraded cohort health, or unstable passes keep the run provisional.
The executable hash must belong to a release artifact explicitly trusted for leaderboard submissions.
Three uniquely indexed passes and the selected pass must be present and mathematically consistent.
The API, not the client label, decides the final trust state and leaderboard eligibility.
Independent consistency checks
A reported speed is checked, not just accepted.
A signed session prevents profile substitution. On submit, the API validates the raw measurements before writing a leaderboard-eligible result.
- 01Bind
Issue a signed, expiring, one-use challenge tied to the canonical profile and prompt pack.
- 02Identify
Match the model repository, filename, revision, model SHA-256, and release binary SHA-256. The client separately validates the exact byte size.
- 03Recalculate
Derive every sample rate from token counts and monotonic-clock seconds, then rebuild pass aggregates.
- 04Cross-check
Recompute three-pass ranges and confirm the declared representative pass matches its submitted metrics.
- 05Classify
Apply execution, preflight, performance-health, repeatability, and binary-trust gates on the server.
- 06Preserve
Store useful noncompetitive measurements as provisional rather than erasing or disguising them.
Verify an official Windows artifact locally
vramcheck verify-release --file .\vramcheck-windows.exeClaim boundary
What the evidence proves, and what it does not.
Precision requires a narrow claim. These boundaries are part of the methodology, not fine print.
Supported by the evidence
- Observed throughput for this exact canonical llama.cpp workload.
- Within-session repeatability across three complete passes.
- Whether the submitted evidence satisfies the public ranking policy.
- Comparable ordering among runs using the same frozen contract.
Not claimed
- Exact speed for every model, quantization, context, or application.
- Model quality, image generation, energy efficiency, or total app latency.
- Cryptographic hardware or operating-system attestation.
- Identical results across drivers, board variants, thermals, and power limits.
- Full prompt-level raw traces for the two non-representative strict passes; those passes currently submit aggregate timing evidence.
Result vocabulary
Measured does not automatically mean ranked.
Four labels keep observed data, compatibility evidence, and catalog estimates from collapsing into one misleading category.
Canonical contract matched, GPU acceleration was active, strict evidence passed, the binary was trusted, and the server admitted the run to ranking.
The measurement is preserved, but one leaderboard gate failed. The result page names the blocker instead of silently converting it into a ranked score.
Real inference completed without the ranked GPU execution profile. It remains useful for the owner, but cannot compete with verified accelerated runs.
A catalog fit or planning estimate. It never enters the measured leaderboard and is always labeled separately from observed throughput.
Frozen public contract
Identity is part of the measurement.
Changing the workload without changing its version would invalidate comparisons. The client therefore refuses a canonical profile whose frozen fields or derived prompt hash drift.
- Profile
- canonical_v1_1_tinyllama_q4km
- Methodology
- v1.1
- Timing semantics
- v1.4.1
- Scored output across the session
- 2,304 tokens: 3 passes x 3 prompts x 256 output tokens. Warmup is excluded.
- Prompt processing across the session
- 55,296 tokens processed: 3 passes x 3 prompts x 3 probes x 2,048 tokens. Includes repeated readings, not unique text.
- Model artifact
- TinyLlama 1.1B (Q4_K_M)
- Exact size
- 668,788,096 bytes
- Model revision
52e7645ba7c309695bec7ac98f4f005b139cf465- Model SHA-256
9fecc3b3cd76bba89d504f29b616eedf7da85b96540e490ca5824d3f7d2776a0- Prompt-pack SHA-256
b7fd979c78c2db84cf32ea05456ca0e0e70691e860ce6ad97b03660809951b03
Questions worth asking
Skepticism is welcome. Ambiguity is not.
Why use TinyLlama instead of a larger or newer model?
A benchmark needs a stable ruler. The frozen 1.1B Q4_K_M artifact runs on a broad range of hardware and produces enough repeated work to compare execution consistently. It is not a claim that TinyLlama represents every model or model-quality workload.
Can two healthy runs on the same GPU differ?
Yes. Driver versions, power limits, thermals, clocks, operating-system scheduling, other applications, and board variants can move the result. Strict repetition filters noisy sessions; it does not erase legitimate system-to-system variation.
Why is a repeatable result sometimes still provisional?
Repeatability only says the passes agree. A machine can repeat a degraded result while a game, power cap, thermal limit, or other persistent constraint remains active. Cohort health, preflight state, binary trust, and canonical evidence are separate gates.
Does Verified mean the hardware was remotely attested?
No. Verified means the submitted evidence passed the published contract and server consistency checks. VRAM Check does not currently provide secure hardware attestation and does not claim to prevent a fully modified operating system or client from lying about device identity.
Does the canonical score predict every local AI workload?
No. It compares one frozen llama.cpp text-generation workload. Model architecture, quantization, context length, batching, multimodal work, image generation, and application overhead can produce different performance relationships.
Inspect the evidence. Then add your machine.
Every public result exposes its trust state, score components, run conditions, and eligibility decision.