WriteAmp Bench · v1 results
The v1 run against the three-tier local catalog: 31 cases (28 positive, 3 negative) across email, chat, document, browser, web-form, and mid-line surfaces. Generated 2026-06-29. Versioned and immutable — this directory never changes after publication.
Latency (the good news)
Every tier completes well under the old 250 ms thinking-dot threshold, and the first word streams in from the first decode batch. These are report-only numbers from a single warm machine:
| Tier | p50 total | p95 total | p95 generation |
|---|---|---|---|
| mini | 51 ms | 110 ms | 107 ms |
| midi | 92 ms | 109 ms | 108 ms |
| max | 123 ms | 147 ms | 145 ms |
Latency is end-to-end (generation + prompt build), measured on the same reference Apple Silicon hardware. Apple Intelligence latency is benchmarked separately as a report-only canary and is not represented here.
Quality
WriteAmp's scoring prefers suppression to a wrong suggestion, showing nothing is often the correct outcome. The quality score below penalises wrong shows heavily; negative scores mean the run was more conservative than precise. The v1 corpus is deliberately hard, and confidence-threshold tuning is still in progress.
| Tier | Quality score | Precision when shown | Positive coverage | Wrong-show rate |
|---|---|---|---|---|
| mini | -1.12 | 13.0% | 10.7% | 64.5% |
| midi | -1.02 | 17.4% | 14.3% | 61.3% |
| max | -1.02 | 17.4% | 14.3% | 61.3% |
These are current-state reference numbers, not a finished benchmark claim. We publish them for reproducibility and honesty while the confidence threshold continues to be tuned. When a tuned run finalises, it ships as /benchmarks/v2/, v1 here stays untouched.
Scoring taxonomy
- correctInsert, a suggestion was shown and was the right completion.
- correctSuppression, nothing was shown and that was right.
- acceptableSuppressionOnPositive, a positive case where the model conservatively suppressed.
- wrongShown, a suggestion was shown and it was wrong.
- incorrectSuppression, nothing was shown but something should have been.
Dataset (immutable)
- aggregate.json, per-tier summary: latency percentiles, quality, precision, coverage, wrong-show, per-tag breakdown.
- manifest.json, run configuration and exact model/quantisation identifiers.
- rows.jsonl, one row per case per tier: outcome, candidates, score contribution, tags.
Paths that referenced the reference machine's filesystem are removed from the public files. Full methodology: WriteAmp Bench overview.
“I just left it on mini. Works fine. Didn’t overthink it,” says Ken, who writes code comments all day in Osaka. He tried midi once. Went back.
Mini is the default for a reason.