Programme · 19 Jun 2026
The verification programme, and what it pays
Four hundred hours at fifteen pounds, paid per verified hour rather than per item.
Specifics
The numbers, and where they come from
| Quantity | Value | Basis |
|---|---|---|
| Rate | £15 per verified hour | Stated |
| Programme size | ~400 hours | Estimated |
| Seeded gold | 5% of items | Stated |
| Triple-annotated | 20% of items | Stated |
The terms
Paid by the hour, on purpose
Annotators are paid fifteen pounds per verified hour rather than per item. Piece rates pay people to rush the cases they are unsure about, which are exactly the cases where a considered judgement is worth most. Paying by the hour costs more and buys a knowledge base we can defend.
IP assignment and confidentiality terms are shown before anyone starts work, not after a first batch. The full quality regime is published on this page rather than described in an onboarding call.
The regime
How quality is measured
- 01
Five per cent seeded gold
Items with known answers mixed into every batch and scored continuously, so drift is visible while it is happening.
- 02
Twenty per cent triple-annotated
Fleiss’ kappa reported per batch. A single aggregate figure would let a bad batch hide inside a good month.
- 03
Expert adjudication
Every disagreement and every unsure goes to an adjudicator. Majority voting was withdrawn in July because annotator errors are not independent on ambiguous items.
- 04
Governance
Lexicon and qualia are safe to crowd-verify. Grammar rules and frames are not, and stay with one author.
What the work is
Two tasks, one much harder
Qualia judgements are binary and learnable in an afternoon: does this relation hold between these two things. Sentence auditing needs real linguistic judgement and takes longer to train, and it is the task where adjudication earns its cost.
Honest about the scale
This is roughly four hundred hours of work, not a permanent role. It is contract work, remote, and it will end. Anyone applying should know that before they start.