A Real Gate
“Stop” is an allowed answer here
Three outcomes. All three are allowed.
We set the thresholds in Stage 0 so this call comes from the data, not from how invested everyone is by week eleven.
4.1 — The Scorecard
Thresholds set in Stage 0, before anyone knew the results
| Dimension | Green — scale it | Amber — fix first | Red — stop this part |
|---|---|---|---|
| Cycle time to UAT-ready | Clearly below baseline, same kind of change | Flat, but rework is falling | Flat or worse, no trend |
| Rework per PR | At or below baseline | Slightly up, in specific agents | Up enough to cancel the drafting gain |
| Drafted lines surviving to merge | Most of the draft survives review | Mixed, and traceable to a memory gap | Most of it gets rewritten |
| Change map accuracy | Matches what the engineers touched, plus things they missed | Occasional misses in one identifiable category | Misses something material more than once |
| Escaped defects | At or below baseline | Level, cause understood | Above baseline — quality traded for speed |
| Review agent catch rate | Finds most of what human reviewers would have | Finds the mechanical issues only | Near zero, or all trivial |
| Non-owner capability | Shipped with review only, no escalation | Shipped, but needed the owner repeatedly | Could not proceed without the owner doing it |
| Team sentiment | Engineers ask to keep it after the pilot | Mixed, split cleanly by agent | Broadly negative — it will not survive us leaving |
Verdicts per agent, not one overall verdict
The realistic result is not uniform. Analysis, review and test agents usually do well early. Some implementation agents need more context before they earn a place. The scorecard is applied agent by agent so rollout carries what worked, instead of averaging a strong result and a weak one into a misleading “it went well.”
4.2 — Refining From the Rejection Log
Every rejection is a specific, fixable defect. Four kinds.
.ai-memory/patterns/, not the agent. Benefits every agent at once.CLAUDE.md, then replay the failed case to confirm.Category D is the one to be straight about
Some work is not agent work. A two-line change in a well-understood file. An exploratory spike. A call that is mostly judgement. Naming those boundaries makes the fleet more credible, not less. A tool sold as universally applicable gets dropped the first time it fails outside its range.
4.3 — Autonomy Promotions
Decided on data and written down
- Each agent reviewed against its criteria: acceptance rate, rework rate, escaped defects on work it touched
- Promotions recorded with the evidence, so nobody reconstructs later why an agent was trusted with more
- Demotions are equally available and carry no stigma. An agent that regresses goes back to L1
- L5 restated: no agent deploys, merges to protected branches, or runs DDL. Not now, not later
- Restricted-zone gate reconfirmed by the domain owners, in writing
4.4 — The Rollout Decision
Made jointly, with the evidence in the room
What you keep either way
The knowledge base. The patterns file. The enforced CI gate with lint and static-analysis rules. The characterization suite. The golden datasets. The integration and UI harness. Those improve your delivery on their own and are worth having whether or not a single agent survives Stage 4. The programme is sequenced that way on purpose: the durable part is built before the part being tested.
Stage 4 deliverables
Complete scorecard against the Stage 0 baseline · per-agent verdicts with evidence · refined agent instructions and memory, replay-validated · recorded autonomy promotions · named module stewards · a joint, documented rollout decision.