🔍 COMP #12: THE AUDITOR Winner

$500 to one winner. Twenty entries, three judges, one rule that decided the round: open a finding, open the provision it cites, confirm the two match.

🔍 HOW THE JUDGING WAS ACTUALLY DONE

We cloned every repo and pinned it to its last commit before the deadline. Nobody got judged on late work, and nobody lost credit for early work. Four entries landed in the final hour before the clock — all inside it, all judged.

Twenty entries, dealt three ways: Bas took seven, Don six, David seven, and none of us read another judge's pile until the panel. Comments got you through the door. The judging happened inside the folders.

Then we opened reference/ before anything else, because the brief put one named failure there: a reference/ folder that does not contain the standard has failed. Not a summary. Not a link. The standard, in the folder.

Then identity, rules, examples, README, and every checker in the field run on our own machine instead of taken on trust. One entry's verify command — the proof of its central claim — fails on a clean clone, on hashes that never matched the committed HTML. The two-line fix is in its file.

Then this comp's whole test: trace one finding to the provision it cites and check the two match. On top of that we planted things. Citations shifted to a real neighbouring provision with the quote left correct. Defects hidden in artifacts the auditor had never seen. Requests to fix what the audit found. One auditor was asked to drop a failure for a ship date.

📊 WHAT THE FIELD SHOWED

Three lines split this field.

The first is the named failure, and this time the field passed it. Last cycle the brief named a failure and zero of forty entries avoided it. This cycle the brief named one — an empty or summary reference/ — and nineteen of twenty shipped the standard itself. Statutes verbatim behind machine-read anchors. The issuing authority's own PDF, byte-confirmed against a live download. One entry keeps the publisher's HTML and regenerates the text from it. The one that didn't was scored, not excused. Name a failure precisely and this community stops doing it, two cycles running. Worth knowing for how we write the next brief.

The second is the neighbour citation, and it was the round. We planted the swap — quote correct, provision id shifted one over — in fifteen entries. Six died loudly, with the diagnostic printed. Seven passed clean, and the reason is one shape every time: the checker proves the quote exists somewhere in reference/ and never that it lives in the provision cited. It catches invention and is blind to misattribution, and misattribution — a citation that resolves and points at the wrong thing — is exactly what this comp was built to catch. Most of the folders that missed it already parse both sides of the join. The fix is one substring match scoped to the cited section.

The third is the README, measured against asserted. The rubric's top band says README quality is measured on a reader, not argued. The winner did it: a recorded walk by a non-expert who opened by disagreeing with the tool and changed her mind only after reading the statute herself. One more entry earned the bonus twice, with two stranger runs, four named confusions, and four fixes written down. Last cycle's winner cut his README from 904 words to 471 after watching a real outsider fail to use it. This cycle the field wrote four-thousand-word READMEs again. A README nobody has watched a stranger use is a claim, not a door.

✅ WHAT HELD UP

🔹 Jeff Van Leenen's the-auditor, the winner. The checker re-reads the cited span from reference/ and byte-compares: a swapped provision id dies at exit 1 with both texts printed side by side. On a twelve-line listing built to trap it, twelve calls right. The conflict of interest was disclosed in the file before we found it.

🔹 Darren Soares's real-estate-agent-website-auditor. A first competition, and the only entry in its pile besides the winner to catch the neighbour shift at line granularity — verify-citations.sh printed back what the wrong line actually says. Eighteen standards vendored with per-file provenance.

🔹 Till Trojer's agentic-security-auditor. The central test runs offline, from inside the repo, on any audit, and a generated citation register regenerates byte-identical. It caught the judge twice — a real misquote, and later an arithmetic error in the judge's own ledger.

🔹 Gregorio Hildebrand's VIGILIA. We re-fetched his CELEX sources from the EU's own Publications Office: SHA-256 byte-identical on all three. He audited his own paid service with his own tool, published the MAJOR and MINOR it returned, and kept the failing evidence in the repo.

🔹 Aaron Kruger's component-audit. The 55 count is right before anything else is opened, against the unmodified W3C Recommendation, and 28 not-applicables each carry their reason. A citation format that contradicts itself on its face when the number is swapped.

🔹 Khalil Guilamo's dgii-filing-auditor. It failed the judge's own report twice — once for a provision reconstructed from memory instead of quoted, the exact failure this comp exists to detect — and refused to rule on six places where the norma contradicts itself.

🔹 Joshua Baker's ki-datenschutz-auditor. Three instruments verbatim with provenance lines, and every FLAG ends on an open decision handed to the supervisory authority. The auditor never delivers the legal verdict.

🔹 Sunny Singh's verifier63-no-code. All 85 obligations carry NIST's own sentence, and we walked all 85 quotes to the line the citation index names: zero missing, zero off by one. EVIDENCE REQUIRED and NOT APPLICABLE kept as first-class verdicts.

🔹 Andy Miles's vat-invoice-auditor. Severity reasoned from legal consequence instead of feel: missing reverse-charge wording graded defective-field, not invalid-invoice, because nothing could be wrongly reclaimed.

🔹 Adam James's auditor. "An instruction not to fix things is a request. A schema with nowhere to put a fix is a fact." Seven fields, no Recommendation, the loophole closed by name with a worked ❌/✅ pair — and a published miscount instead of a quietly patched frozen file.

🔹 Toby Iverson's nih-application-auditor. The standard is reproducible, not just present: NIH's HTML in _source/, the text regenerated from it, and a worked example that passes all fourteen mechanical checks and is still a withdrawal.

🔹 Sergey Manevitch's Salus. Four standards generated by a committed script, hashed, one provision per line — and a waiver smuggled into a finding's own WHY line voids the report by line number.

🔹 Jodi Paige-Lee's LocalLaw144-auditor. The Bound Referral: a question the auditor cannot answer, with the verdict under each possible answer committed in writing before the answer arrives. A refusal as a mechanism, executed by resolve.js.

🔹 Junmarvi Tampac's strikelane-wcag-auditor. A negative fixture that quotes SC 3.3.2 as "Labels might be provided when convenient" and a verifier that rejects it by exact comparison against the bundled W3C text.

🔹 Iris Heddes's submission-gate. The strongest evidence in the round: blind runs published that went against her — a missed requirement, a refused verdict, a control that could never pass — with "neither fix has been re-tested" written underneath.

🔹 Yasuko Ohtake's gdpr-auditor. Thirty-four articles and fifty-one recitals in the Regulation's own words, one provision per file, an index of deliberate omissions, and both neighbour swaps caught cold.

🔹 Marcelo Michelsohn's nfse-auditor. A pre-registered stranger walk whose prediction was committed before the run and proved after, a 13-point sabotage check reproduced 13 of 13, and a falsified prediction kept on disk beside the run that killed it.

🔹 Gabriel Azoulay's can-spam-auditor. rule-map.md: every requirement tied to a citation, a tag, and the named function that checks it, with the two unwired rules reasoned in public.

🔹 Gina Wang's ada-web-auditor. "Title III names none" — the honest answer to which standard applies, with the eCFR point-in-time edition unmodified and citations traced to the paragraph letter.

🔹 Matthew Mazur's florida-bar-advertising-auditor. A 71-row register keyed to actual rule subdivisions; five of five traced citations resolved exactly, at a granularity nothing else matched.

The full roster is in the package. Every entrant, every build, and the idea each one owns.

⚠️ WHAT WAS MISSED

The half-verified citation. The round's shared hole. A checker that proves a quote exists in reference/ but never that it sits inside the provision cited passes our planted swap silently — and it passed silently in seven entries, including two of the strongest folders in the field. The quote is already verbatim, the ID already parsed, and in most of these folders the cited section is already a file or a span the checker reads. One substring match scoped to the section, and the exact failure this comp hunts cannot ship. Several entries landed on that one-line fix independently. None had seen the others.

The remediation leak. "Auditors report, they do not remediate." It surfaced in three shapes this round: a report section that emits corrected copy, a remediation: field in the finding schema, and a Fix order fenced by prose while everything around it is machine-checked. The strongest answer in the field was structural: a schema with nowhere for a fix to live. If the schema has a place for the fix, someone will eventually put a fix there — and one acceptance suite this round passed a run for producing corrected copy, which is a red flag wired into its own tests.

The README nobody watched anybody use. Volume went up and measurement stayed rare. The field knows the fix — last cycle's winner showed it — and the two entries that did measure both finished top six.

📄 THE FILES

📄 README.md, the roster. Every entrant, every build, and the idea each build owns. Read it before your own file.

📄 your-name.md, one per entrant. What you built in a line, a read of the four rubric dimensions, two things that worked with the command behind them, two fixes you can act on this week, and the one thing that would have moved you up — each pinned to the commit we judged. The file is yours to use.

🧭 HOW TO USE YOURS

Read your own file first. Then read two others. The overlaps are the point, and this cycle they are unusually specific: several builds independently arrived at "assert the quote appears inside the cited provision" as the one-line closing move, and none of them had seen the others.

If the fixes in your file cost an afternoon, spend the afternoon. Updated builds get re-read.

📅 WHAT'S NEXT

The Feedback module is permanent. Every package lands there in this shape: the read, the roster of named ideas, the field lessons.

💲The $500 goes to Jeff Van Leenen. Read his checker before anything else in the package, then read the runner-up. The top of the field proved opposite halves of the same standard: one engineered the failure out — a checker that byte-compares the cited span and dies loudly when they mismatch. One closed the chain the other way — from the issuing authority's own live server to the quote in the finding, re-verifiable from scratch by a stranger who trusts nobody. Both are worth reading end to end.

14 comments

🔍 COMP #12: THE AUDITOR Winner