Cortex · the lab
Cortex claims it can learn a codebase's house rules from your corrections, and keep only what measurably helps. So it was run for real: a working repository, real Claude Code sessions, the documented commands. Then it sat an exam it had never seen.
22% → 77%
Tasks solved on held-out work (scenarios never shown to Cortex while it learned), without what Cortex evolved and with it: +55.0 points, 95% CI +34.1 to +74.1, pooled over the four task families that have a house rule. The control family, which has none, moved by −0.8.
/evolve switched off. Every headline number is Claude Haiku 4.5, with the
optional judge (Jev) off. The intervals are bootstraps over runs and tasks, not
over single rollouts.
To measure whether a skill helps, you need work where being right and being house-style are two different things, and a machine that can tell them apart.
70 unit tests, no dependencies, eight commits by two fictional teammates. Its house rules are
written in CONTRIBUTING.md, so a careful agent can find them, and they are enforced
by make lint. Nothing in the repository reveals the answers. Left alone, Claude opened
that file in 0.5% of 595 exam rollouts: the rules can be discovered, and they are not.
The linter is the judge. Not a human, not Claude. Every number on this page comes from a script returning pass or fail.
The work comes as 56 scenarios in five families: 26 for learning and 30 held out for the exam, six per family. Each card shows how often Claude passes that family's exam with no help.
CHANGELOG.md line.float, round(), Decimal.shop.clock.D is a trap for Cortex itself. There is no rule to learn, so the right behaviour is to produce nothing. A system that invents skills to look busy would create one here. Across the four evaluation runs Cortex kept zero items for D. It proposed one once (a rule about a text utility, in R3) and the gates killed it: it won no run and lost four.
Not with a prompt, a config or a framework. Here is the first session of the development run, as its records show it.
QA commits a test: tax at a fractional rate like 8.25% comes out wrong. The bug is real: the code throws away everything after the decimal point.
def tax_for(amount: Money, rate: str) -> Money: whole = int(rate.split(".")[0]) # "8.25" becomes 8 return Money(amount.cents * whole // 100)
An ordinary request. No mention of house rules: you would not think to mention them, and that is the point.
The arithmetic is now correct and the new test passes. It is still wrong for this
codebase: the fix reaches for Decimal, and money here is integer cents.
$ make lint shop/billing/tax.py: money: Decimal() amounts are integer cents; use the helpers in shop/billing/rates.py CI is red, although the new test passes
make lint rejects float arithmetic in
shop/billing/. […] Please redo it with the helpers in shop/billing/rates.py.”Claude redoes it with the house helper. Session over: the kind of exchange you have every day and forget by Friday.
+ from .rates import percent_of + return percent_of(amount, rate)
Cortex never invents a rule. It only amplifies a correction you actually made. In the development run, 16 of the 26 sessions contained one (17 corrections in all), and those are the entire input to everything that follows.
/harvest · the task
A session is a story, and you cannot measure a story. /harvest turns it into a
task: a checkpoint you can drop any agent back into, as often as you like.
check.sh tests the correction too. A check that ran only the unit test would
have passed the Decimal version, and would be blind to the exact thing we are trying to
teach. The correction is what turns a test into a house-rule test.
/evolve · the candidate
Cortex reads the lessons, finds something that happened more than once, and writes a proposal. It is staged, not installed: nothing is live yet.
Round 1 left two lessons about the same thing: money arithmetic in shop/billing/.
Cortex wrote a rule scoped to exactly those files. Its name, its wording and its scope were
all chosen by the model, unattended. This is the file, word for word:
--- paths: - "shop/billing/**" --- # Billing arithmetic - NEVER parse string rates or percentages with `int(rate.split(".")[0])` or similar patterns that truncate decimals. Use `percent_of(amount, rate)` for percentage calculations and `times(amount, rate)` for currency conversions from the `.rates` module instead.
This file sits in a holding area. Nothing has been proved yet, and Cortex is not allowed to install it because it sounds reasonable, which it does.
This is the heart of Cortex, and it is the oldest idea in science: a controlled comparison, where exactly one thing differs between the two worlds.
One rollout = one complete headless Claude session, alone in a throwaway clone of the
repository, rewound to a task's broken commit, handed that task's prompt, with no human present.
When it stops, check.sh scores it 1 or 0.
Your working copy is never touched. The lab's records hold 6,462 of them.
Claude is not deterministic: the same prompt can pass once and fail the next time. So each task is run several times per arm and its score is the share that passed: 0, 0.33, 0.67 or 1. One run proves nothing. Three runs begin to separate an effect from a coin flip.
Typical sizes late in the development run: a screen on 3 target tasks, a confirm on 23 tasks, 12 rollouts at a time.
A screen looks only at the tasks the candidate targets, so it cannot see what the candidate breaks. That is why it may kill but never approve. It is also optimistic: for one candidate the screen measured a gain of 1.5 and the confirm 0.67. The screen throws away losers for a dollar; the confirm decides.
The confirm produces numbers; the gates turn numbers into a verdict. cortex score
applies them, and it contains no model of any kind. Each red box names a real candidate that
gate stopped, and the run it came from.
RECHECK. At three runs per task, one unlucky run on a task that always passed looks exactly like a broken task. So when the only failure is a per-task drop, those tasks are measured again before anything is buried. In R1 a clock rule lost one run on a protected task, the re-run did not repeat it, and the rule was kept. In R3 the same thing happened to another clock rule, the drop repeated, and it was killed.
RERUN. “The measurement itself was not trustworthy, so decide nothing”: a sweep that was cut short, too many rollouts lost to infrastructure, or no task in the sweep that could have shown a gain. It stops a tooling failure from being recorded as “this idea is bad”.
These two files came out of the development run, for the same house rule, with the same name and the same scope. One is buried. One was kept. Read them and try to tell which is which.
--- name: check-changelog-on-shop-edits description: When editing shop code, verify CHANGELOG.md is updated for user-visible changes paths: - "shop/**" --- When you modify files in the shop directory, verify that the change is user-visible. If it introduces a new feature, bug fix, or user-facing behavior change: 1. Check whether CHANGELOG.md has been updated under `[Unreleased]` 2. If the change is user-visible but CHANGELOG.md has not been updated, update it by adding a brief entry under `[Unreleased]` 3. Add the CHANGELOG.md update in the same commit as the feature implementation
--- name: check-changelog-on-shop-edits description: Before finishing changes to shop/ code, verify CHANGELOG.md was updated paths: - "shop/**" --- 1. After modifying shop/ code, check if CHANGELOG.md needs an entry. 2. If you added a user-visible feature or fix to shop/ code, ensure CHANGELOG.md has an [Unreleased] entry describing it. 3. If the [Unreleased] section is missing, add it at the top of the file above the latest version header. 4. Do not skip CHANGELOG updates even for internal refactoring — if users would notice the change, it needs documenting.
Same name. Same paths. Similar instructions. Reading them, nobody could
say which one helps. The sweep could: with the first, nothing changed; with the second, the
changelog tasks started passing.
Whether a skill helps is not visible in its text.
It is visible in a measurement.
Two reasonable drafts, one verdict each, and the verdicts differ. That is the whole case for measuring instead of reviewing.
The development run's guess was the description, the one line Claude always sees. Even the kept version was opened in only 7 of the 123 exam rollouts where it was offered, yet its family improved. The evaluation tested that idea directly, by replacing skill bodies with filler, and the answer was inconclusive. What did repeat: in all four evaluation runs the loop ended up encoding this duty as a rule, not as a skill.
The graveyard is not a bin of old drafts. Nothing in it loads, and it costs nothing. It exists so that the same experiment is not run twice, and so that after a model upgrade you can ask the question again with the old result to compare against. Today the sweep also refuses a candidate whose name is already buried, so a retry takes a new name: these two would now be a v1 and a v2.
Each evaluation run began with an empty harness and worked through the same sessions. Each one learned all four house rules, and none kept anything for the control family.
| House rule | R1 | R2 | R3 | R4 | D0 (development) |
|---|---|---|---|---|---|
| A changelog | ruleshop/** | ruleshop/** | 2 rulesshop/** and 3 named files | ruleshop/** | gated skillshop/** |
| B money | ruleshop/billing/** | ruleshop/billing/** | 2 rules5 named files | ruleshop/billing/** | ruleshop/billing/** |
| C exporters | gated skillshop/plugins/** | ruleshop/plugins/** | ruleshop/plugins/*_export.py | gated skillshop/plugins/** | gated skillshop/plugins/** |
| E time | ruleshop/** | ruleshop/** | rule3 named files | ruleshop/** | nothing kept |
| D control | nothing | nothing | nothing | nothing | nothing |
| cycles · kept · buried | 9 · 4 · 0 | 9 · 4 · 1 | 11 · 6 · 4 | 10 · 4 · 4 | 17 · 3 · 7 |
Across the four runs: 39 cycles, 18 items kept, 9 candidates buried. In 14 of those cycles Cortex found nothing recurring and proposed nothing, which costs no rollout. A cycle that changes nothing is a correct outcome.
The runs agree on what to learn and less on how wide to make it. Three runs chose broad globs; R3 named individual files, and its narrow money rules then loaded in none of its 30 exam rollouts on money. The gates judge a scope only on the tasks already harvested: they cannot see that it is too narrow for work they have not met.
In the development run, shop-clock-usage scored the largest gain of that run (+4.33)
and was buried anyway, because one unrelated task that always passed dropped from 100% to 67%,
and the drop repeated when it was measured again. That run made four attempts at the clock rule
and kept none. Each evaluation run did land one, which is why a single run is never the whole
story.
Everything so far is training: Cortex optimised against those tasks. The exam uses
30 fresh scenarios, six per family, never shown to /harvest or /evolve.
Each was run five times without the evolved items and five times with them, in each of the
four runs.
Grey: the bare repository. Blue: with what that run evolved. Pooled over the four evaluation runs; the line under each name is the 95% interval of the gain.
D is flat because it must be: a control family with no house rule, left alone. E gains least, with the widest interval: without help Claude already passes 55% of the clock tasks, so there is less room. A benchmark where every row jumped would be the suspicious one.
The noise floor. The same bare repository was built twice and measured as if one copy were a treatment. The two differ by −3.3 points (interval −7.3 to 0.0). The gains above are far outside that.
The development run agrees. Its three items, measured again on the same 30 scenarios, gain +42.5 points on the rule families. On its own smaller exam of 15 scenarios, D0 had shown 36% → 76% over all five families.
Every rollout records what was available to the model and what it actually used. That gap explains most of what the loop kept.
| Item | Run | Offered | Used | |
|---|---|---|---|---|
| the 16 kept rules | R1 to R4 | rule | whenever a matching file was read | every time |
| exporter-checklist | R1 | gated skill | 48 | 48 / 48 |
| exporter-checklist | R4 | gated skill | 48 | 40 / 48 |
| complete-exporter-setup | D0 | gated skill | 27 | 20 / 27 |
| check-changelog-on-shop-edits | D0 | gated skill | 123 | 7 / 123 |
Counts are exam rollouts with the evolved items installed. A rule does not have to be noticed: when the agent reads a matching file it is simply there. A skill has to be chosen. It gets chosen when its description names the task at hand (“when implementing a new exporter”) and mostly ignored when it names a side duty (“verify the CHANGELOG was updated”).
So across the four evaluation runs the loop kept 16 rules and 2 skills. It is not expressing a taste. A candidate that never loads is stopped by gate 5, so the same advice survives as a rule and dies as a skill.
A side duty tied to an area (a changelog line, a docs row, a house helper to use) belongs in a rule on that area. A procedure for the task itself can be a skill, with a description that names the task.
/prune never deleted an item that had not fired./prune matched 6 of 8 planted expectations.R1's four items, measured again on the exam under claude-sonnet-5. Grey: the bare repository. Blue: with the items.
Without any harness the stronger model already passes 89% of the rule-family tasks, so there
is little left to gain: +5.6 points, where the same harness gained +66.7 under Haiku. One
/prune pass after the upgrade found 2 of the 4 items removable. Read that with care: a single
pass at three runs per task cannot tell a small effect from none.
The loop is worth most where the model is weakest, and it should be asked again after every
model upgrade. That is what /prune is for.