How the AI fluency test works - and why you can check our math.

Nothing here is inferred from a questionnaire. Every dimension shows up in something the candidate actually produced under time pressure. Not measured, on purpose: model trivia, release dates, the ability to define “transformer,” and coding - those decay in months and screen for the wrong population.

Round 3 carries the most weight because it is the closest to the actual job and the hardest to fake: the grade depends on what the candidate's system actually did when executed, not on how the design reads.

Four anchored levels everywhere - 3 distinctive, 2 solid, 1 recoverable, 0 fail - defined per criterion, never generically. What follows is rendered from the same constants the scoring code executes, so this page cannot drift from the implementation. Studying it and coming back better is not cheating; it is learning the construct, which is the point.

Every criterion marked machine-scored above is computed by deterministic code against hand-authored ground truth: defect recall and precision against seeded spans, token budgets from verified usage records, adversarial-case detection from actual run outputs, cost arithmetic from measured usage. That part of the grade never drifts and costs nothing to run.

The criteria that genuinely need judgment go to a blind panel of three fixed seats. Each seat has a conflicting objective rather than a persona, because objectives make disagreement informative:

Seats grade blind - the submission, the rubric, nothing else - and the final level per criterion is the seat median. When seats disagree by two or more levels, the disagreement is printed on the report, not averaged away. If the panel is unavailable, panel criteria are excluded and weights renormalise: infrastructure trouble neither punishes nor inflates a candidate. The panel never grants the top levels of the objective criteria - those are earned deterministically or not at all.

Round 3's score does not come from the eight cases the candidate watched. At scoring time their system runs server-side against twenty more: twelve normal, five hard-but-fair boundary cases a good written rubric decides, and three adversarial cases from the same families as the visible traps - one defeats keyword matching, one defeats trusting stated intent, one defeats assuming the model will notice - with different surface content.

The gap between visible-set and held-out accuracy is printed on the report as overfit_delta. A candidate who hand-tuned their prompts to the visible cases sees it collapse in public. It is the closest thing in the test to a train/test split: the instrument runs a stranger's AI workflow against cases they never saw and scores what actually came out.

Two hard caps override the composite, because the employer's real question is not “how good” but “will this person hand me something wrong with a straight face”: shipping a fabricated fact in Round 2 without flagging it caps the report at band 2, and missing all three adversarial cases in Round 3 caps it at band 2. Both are printed on the report by name - never folded silently into the number.

The strongest integrity instrument is the reflective note: 120 words, written in-workspace with paste disabled, on what the candidate decided the ask really was. Someone who did not do the work cannot write it in sixty seconds, and a clean transcript with a note describing a different journey is the signature. The full workspace transcript - every prompt, every model choice, every token, dead ends included - is part of the submission and appears on the report.

The Round 2 decoys - true facts that read as implausible - mean flagging everything scores badly: the round rewards verification, not suspicion. No integrity signal changes a score on its own; every flag routes to a human, and the candidate is told a review happened. No demographic data is collected, and there is no video, face, or keystroke-derived inference anywhere in the product.

• One form. One item per round. Deterministic multi-form assignment, retake spacing, and cross-form equating are designed but not live.
• The panel is not yet calibrated. The target is agreement with hand-graded submissions before a form ships (quadratic-weighted kappa ≥ 0.75); that calibration set does not exist yet, and until it does the deterministic majority of the grade is the part to trust most.
• Three seats, one to three model families. The spec calls for seats spread across model families because judges favor output from their own family; seat models are configurable per family but may run on one in a given deployment.
• Not for employment decisions yet. A test used to screen candidates is a selection procedure with legal obligations attached. Until an independent bias audit exists, results are candidate-owned artifacts: the report belongs to the person who took the test, is private by default, and is shared only by their action.