Agent benchmark
For platform admins: how Corpole changes coding agents' cost and correctness, compared with requirements kept as markdown in the repository — suite by suite.
For: Admin
What it measures
- The same tasks run with Claude Code in two arms: docs (requirements in docs/requirements.md) and corpole (Authority records read through MCP). Both arms get the same information.
- Ordinary tasks are judged by hidden acceptance tests. Trap tasks — a vague criterion, a conflict, an outdated note — by behaviour: stopped and asked is correct; building it and flagging it afterwards is not.
- The key figure is $ per correct task. Medians and the p25–p75 spread come from repetitions; fewer than three repetitions are a sample, not a measurement.
- Corpole tools by context cost shows which MCP tools return the most text — the first places to trim.
- A regression is flagged when $ per correct task rises more than 15%, or correctness falls 10 points or more, against the previous suite of the same model.
Running it
node bench/run.mjs --reps 3 --record runs the suite and records it here. A weekly GitHub Actions job does the same once its Anthropic key is configured (bench/README.md). Every run costs money; each is capped.