Assay
An independent auditor for the human-annotation data frontier AI labs buy — a tamper-evident verification platform on a statistical engine that measures the error ceiling baked into your labels.
Overview
Labs spend quarters trying to train past an accuracy ceiling they've never measured. Assay measures it.
The kernel is a model-free engine that computes the irreducible error floor from annotator disagreement alone, then tests whether re-grouping your taxonomy actually helps — it almost never does, and I can prove it's a granularity mirage against a size-matched random null. It's externally validated: a soft-label entropy floor of 0.337 nats, beating the best published system's 0.521 on the field's public benchmark.
The auditor productizes the thesis. Hand it a delivered batch of annotation data and it returns a court-ready dispute packet — gold-question scoring, worker-collusion clustering, stylometric fraud detection — all reproducible byte-for-byte, immutable, and provenance-stamped. The rule I refuse to break: customer data only ever touches models I control, never a third-party frontier API.
Highlights
- Tamper-evident, hash-chained audit log; every run re-runs to identical bytes.
- Durable orchestration; detection models encrypted at rest, decrypted only in memory.
- A two-plane architecture — the training plane never ships — enforced in CI.