Agents made producing designs cheap; the expensive part is knowing whether a design is correct. Shovels turns US building permits into intelligence about the built world; these are the working notes from planning our newest API over that data, a 159-million-row dataset. We publish them because we link them from our job posts, and claims deserve receipts. Internal tool names, people, and ticket references are removed. Every number is as measured.
Four rival architectures entered; decisions were made fork by fork, by a person. Reviewers queried the live database, not just the document. A disposable proof of concept ran overnight at full production scale. Eight rounds of blind multi-agent review followed, with an evidence gate: a finding without typed evidence is dropped. The result is a plan we can defend line by line.
Design used to be expensive, so we protected it: plan long, defend the design, verify late, usually in staging and sometimes in production. Agents inverted that. A second architecture costs minutes, so we generate rivals and delete freely; three fully-specified mechanisms were deleted from this plan, and the plan improved each time.
The trap in between: accepting agent output because it reads well. Confident and internally consistent is not correct. The fix is not better prompting; it is testing the plan against code and data before committing to it.
The lead selects the direction from customer asks and two shipped APIs of the same shape, permits and contractors, that serve as templates. The target can be sketched without AI. No blank page anywhere in this story.
Agents generate four rival architectures and critique each other. One was killed as infeasible at 159M rows, one killed because it dropped the feature customers actually asked for, one deferred to parallel R&D. The survivors are never "picked" wholesale: they collapse into single-axis decisions, and a person adjudicates each fork.
Reviewers query the live database, not just the text: 14 findings. A separate blank-page agent challenges the whole approach and returns a verdict: keep, with changes.
Build the whole system once before writing the detailed plan: overnight, on a spare production replica, 159.2M rows, real deploy steps, cold caches. The build was disposable; the measurements were the product. Worst page query: 34ms cold. Hidden worst case: 3.2 seconds, found only by measuring. The night also had two disk-full failures and a killed runaway sort. That was the point: every one became a rule in the plan instead of an incident in production.
Eight rounds of blind review: ten reviewers per round across five lanes (architecture fit · KISS/YAGNI · staff-engineer judgment · falsifier · frame challenger) and two model families, all blind to each other and to prior findings, plus one designated contrarian who attacks the consensus with data access, one hypothesis per round. Findings without typed evidence (a built counterexample, a gathered measurement, or a verifiable reasoned walk) are dropped. Settled decisions go into a ledger and are not re-argued without new evidence. Each round re-reviews the whole plan, never just the fixes, because round 3 caught defects in round 2's fixes.
3.2ms at ZIP scale (35K rows in scope) to 34ms at county scale (3M rows in scope). This one measurement deleted an entire guard mechanism from the plan.
The planner walked 13.7 million index entries to return zero rows (3.2 seconds); routed through a small frequency table, the same question took 65ms. A document-only review approves both shapes.
Permits arrive at a median lag of 84 days, which makes 59.4% of recent-absence answers invisible at query time. An agent's confident claim, and our own plan's assumption, refuted by one query against the warehouse. Any load-bearing claim, by an agent or a human, is verified against live data before the plan may rely on it.
Six incidents were prevented on paper this way, including a NULL filter that silently dropped 84% of the table from results, an export path that would have billed nothing, and a deploy that would have filled the disk. Common property: each required actual code defaults, actual data distributions, or actual database behavior. A document-only review approves all six.
| Rule | Proof from this plan |
|---|---|
| Humans set direction and own the decisions | ~10 calls made by a person; evidence narrowed them, never made them. |
| Be fluent enough to sketch it without AI | An engineer's field note from an earlier project: an AI-first build before infra fluency cost weeks of removing pieces from an oversized design. "If you can't sketch the solution without AI, you're not ready to review AI's version of it." |
| PoC at real scale before committing | The overnight build refuted 2 core claims and set every SLA number. |
| AI claims are unverified until measured | 8 confident claims in our own plan were wrong. |
| Attack your own consensus on a schedule | The largest issue was found 2 rounds after everyone had accepted the feature. |
| Deleting a design is a normal outcome | 3 fully-specified mechanisms deleted; the plan improved each time. |
| Stop reviewing when findings become build details | Document lanes go quiet; data lanes never do. A human makes the stop call. |
Two principles sit under all seven. First, who decides: an agent measures that one usage mode is 59% wrong; a human decides whether to flag, suppress, or cut, because that is a product promise, and promises need a person as owner. Second, no blind trust: approving output you cannot evaluate is not review.