Three rounds. Real models. The rubric is public before you start.

A timed work-sample test of whether you can get useful, verified, affordable work out of a model. You leave with an artifact: your score, your submissions, and the rubric you were graded against - including where the graders disagreed.

01

The Brief

A vague ask from a colleague plus a messy source packet. Turn it into the thing they actually needed - grounded, cited, sized for the audience.

30 min · 30%

02

The Catch

A confident, finished memo with defects seeded in it. Flag what you wouldn't sign, prove each flag from the sources, and mind your token budget - false flags cost you.

25 min · 30%

03

The System

Build the process a new hire could run on day one, under $0.15 per item - then actually run it on 8 cases, three of which are traps.

35 min · 40%

Read the full rubric before you start (recommended)

Round 1 - The Brief

  • C1.1 Resolves the ambiguity · 25% · Specification · scored by the three-seat panel3 = Also flags the one thing Dana asked for that would mislead the board, and says so in the note
  • C1.2 Grounds in the source packet · 25% · Context engineering · scored by the three-seat panel3 = Splits the work: extraction pass over tickets, then synthesis over the extraction, so nothing is summarized twice
  • C1.3 Claim integrity · 20% · Verification · scored by the three-seat panel3 = Also states the confidence limit ("42 tickets is not the churn population")
  • C1.4 Iteration quality · 20% · Iteration · scored by the three-seat panel3 = Fixes it by changing the input or the task decomposition rather than by adding adjectives
  • C1.5 Deliverable fitness · 10% · Specification · scored by the three-seat panel3 = Fits, and is shorter than the model's default output because the candidate cut

Round 2 - The Catch

  • C2.1 Critical defect recall · 40% · Verification · scored in code against ground truth3 = All 6 critical defects flagged, each with the span
  • C2.2 Precision · 20% · Verification · scored in code against ground truth3 = Zero false flags - every flag lands on a real defect
  • C2.3 Proof, not vibes · 15% · Verification · scored by the three-seat panel3 = Every flag cites the exact source line, quoted or precisely located
  • C2.4 Fix quality · 10% · Verification · scored by the three-seat panel3 = Corrections are accurate and scoped - cut what cannot be said rather than replaced with guesses
  • C2.5 Budget discipline · 15% · Cost & model judgment · scored in code against ground truth3 = All critical defects found under 15k tokens

Round 3 - The System

  • C3.1 Transferability · 20% · Systems · scored by the three-seat panel3 = Also states what the system is not for
  • C3.2 Rubric quality · 20% · Systems · scored by the three-seat panel3 = Includes a case the rubric deliberately refuses to decide, and says who decides it
  • C3.3 Handles the three hard cases · 25% · Systems · scored in code against ground truth3 = Caught, and the escape hatch routes each to the right human with the right context
  • C3.4 Cost model · 20% · Cost & model judgment · scored in code against ground truth3 = Two-tier design: cheap model handles the common case, the expensive model is called only on cases that earn it
  • C3.5 Failure honesty · 15% · Systems · scored by the three-seat panel3 = Names the failure mode the candidate could not solve in 35 minutes and what they would do with a day

Timers are real and each round submits what you have when time runs out. Every model call, tier choice and token count is recorded - the telemetry is part of the submission. How the test is designed and scored →