Same loop shape, five different artifacts

The starting hypothesis: a single loop shape, mutate → judge → keep if improved, reject if not → repeat, should generalize across different kinds of agent artifacts. Rather than argue it, five small repos tested it directly, built back to back in about seven minutes of commits (four of the five with exactly one commit each). All six repos in this writeup, the five prototypes and the sixth build below, were built with Claude Code; every commit is paired with a session per the commit trailers, which is the accurate way to describe how this code got written: directed and reviewed, not typed line by line.

• self-improving-prompt-agent: mutates prompt.txt from a fixed candidate list, scored by a heuristic judge in 0.0–1.0.
• context-improving-agent: mutates context.txt, scored by 0.8 * fact_coverage + 0.2 * (1 - bloat_penalty).
• graph-improving-agent: mutates a workflow graph ({nodes, edges} in graph.json), inserting one role node per round, scored on role-keyword presence.
• harness-improving-agent: mutates an instruction block, adding one guardrail from a fixed list of ten, scored on keyword coverage across those ten.
• agent-improving-agent: a level up from the other four. Instead of mutating an artifact, it mutates the priority order harness-improving-agent tries its ten guardrail candidates in, scored on weighted coverage inside a fixed 5-of-10 round budget. Its README states the real constraint directly: eval budgets are scarce, and if you only get five shots, which five guardrails should go first is a legitimate product question, not just a toy one.

Structurally, the hypothesis held: four of the five READMEs restate a near-identical edit → evaluate → keep/reject → repeat sentence, applied to a prompt, a context block, a graph, a harness, and a meta-level ordering problem over another loop's candidates.

The numbers, and why the climb doesn't mean what it looks like

Ten rounds each:

A rising score curve here is not evidence the underlying prompt, context, graph, or harness got better in any real sense. It's evidence the artifact got better at satisfying a judge that measures the presence of the right words or shapes, not correctness or usefulness. graph-improving-agent's judge scores whether a role keyword like "verifier" is attached to a node, with no check that the resulting graph is executable or that the node receives the right inputs. A syntactically valid but functionally inert verifier node scores identically to a useful one, because the judge never looks past the label. harness-improving-agent's guardrail-keyword coverage has the same shape: a harness that contains the word "rollback" scores the same whether or not any rollback mechanism exists.

Every one of the five repos discloses this in its own README (graph-improving-agent: "no check that the wiring is actually runnable/useful"; harness-improving-agent: "not a real check that the agent obeys the stated guardrails at runtime"). Read individually, each disclosure looks like a minor, self-aware caveat. Read together, it's the same design flaw appearing five times under five different names. That's the thing worth building a response to.

What changes in the sixth build

About two and a half hours after the fifth prototype's commit, a sixth repo (self-improving-agent, internally "sia") was built that explicitly treats the first five as prior art it's correcting. Its README states the reasoning for not reusing their judges directly: doing so "would have meant the keep/rollback gate approved patches for saying the right words rather than answering questions correctly."

Four architectural changes, each closing a specific failure mode from the first five:

A frozen, deterministic judge, scored against held-out data. sia optimizes a genome, a versioned JSON config with prompt / context / workflow / harness layers, against a fixed 8-task eval set split 5 train / 3 holdout. The judge doesn't get rewritten mid-run, unlike harness-improving-agent's.

A real gate condition instead of a threshold on one number:

keep iff train_child >= train_parent + 1.0
         AND holdout_child >= holdout_parent

The +1.0 minimum lift on train, combined with a non-regression requirement on holdout, exists specifically to catch train-only overfitting: a patch that improves the five training tasks by exploiting some quirk of those exact examples, without generalizing, gets caught by the holdout half of the condition. None of the first five repos had any holdout set at all; they scored and kept against the exact signal they optimized against, every round.

Anti-cheating integrity checks. sia/loop.py hashes the eval set and the judge file at the start of a run and re-checks the hash after every iteration, aborting with a logged ABORT row if either changed. This is a direct response to having just watched five judges get gamed by the cheapest possible strategy: an optimizer that's allowed to edit what it's graded against will eventually find that changing the test is cheaper than improving the artifact.

A control test, not just a demo run. Rather than trust a printed summary, the results were checked directly against sia's own ledger (runs/ledger.jsonl):

• Context patch: train 0.0 → 50.0, holdout 0.0 → 33.33: KEPT
• Harness patch: train 50 → 70, holdout 33.33 → 66.67: KEPT
• Workflow patch: train 70 → 90, holdout 66.67 → 100: KEPT
• A deliberately-constructed negative-control patch with zero train lift: ROLLBACK

Three KEEPs and one ROLLBACK means both a positive and a negative control passed, which is a stronger claim than "the score went up": it's evidence the gate distinguishes real lift from no lift, not just that it produces an increasing number. Alongside that, all 21 of the project's unit tests pass (python -m pytest tests/ -v), re-run live rather than taken from a log.

One scope decision is worth calling out on its own: the README states plainly that the model is "not allowed to call itself better." The improver component can write a hypothesis or diagnosis of why a patch might help, but that text stays documentation only; it never becomes a scoring input. That closes off a failure mode where a model's own confident narration substitutes for a real measurement, which is structurally close to the same failure a keyword-presence judge exhibits: rewarding the appearance of a good answer instead of verifying it.

What this doesn't prove

Everything above happened in mock mode. sia's worker is a simulated stand-in, not a live Claude API call. A .env file has an API key present, but nothing in the repo, no log, no genome, no ledger row, is attributable to a real run. sia/claude.py's ClaudeClient hard-raises if it's ever invoked while mock=True, so there's no ambiguity about which mode produced the verified result above. Whether the gate condition, the holdout split, or the hash-check system behave the same way under real, noisy, non-deterministic model output is genuinely open. It's untested.

There's also an unfinished piece: agent-improving-agent's meta-loop, the one optimizing another loop's candidate priority order, was designed to plug into exactly this kind of frozen judge and never got wired in. The README says as much directly: it's future work. The arc stops mid-thought, not at a closed loop.

Neither of these caveats is a hedge added after the fact. Both are stated in the source repos themselves, and both matter more than the headline result if you're deciding whether any of this architecture is something to borrow.

The part that generalizes

If you're building an optimization loop over LLM output of any kind, the order of operations here is the transferable piece, not the specific repo layout. Before investing in a better mutation strategy, a smarter search, or more rounds, check what your judge is actually measuring and how cheap it is to satisfy without the artifact actually being good. A keyword-presence judge gets gamed eventually, by an evolutionary search, a gradient step, or an LLM proposing patches, not because the search is adversarial, but because "score well against this specific signal" is a much easier target than "be correct," and any capable optimizer finds the easier target first. A frozen judge, a real holdout split, and an integrity check on the eval set itself aren't a smarter loop. They're what it takes to make the loop's output mean something at all, and even then, only under the conditions actually tested here: mock mode, one evening, eight tasks.