Evolver: design and implementation¶
New to the tool? Start with Usage and experiments for evaluation scope, setup, costs, and the run/resume/finalize workflow. This page is the implementation reference.
Companion: the authoritative methodology spec is
docs/specs/self-evolution-loop-sop.md, kept in the repository beside the code
rather than published here. This document maps the SOP clauses to Raven's
implementation, including evaluation choices and module integration.
How to use: before changing evolver/** or
benchmarks/appworld/evolve/**, locate the SOP clause here; if your
change alters a correspondence, update this document in the same PR.
0. The fundamental architectural difference (read first)¶
The SOP's loop (§3 / §8.3) is Claude-driven: a human opens Claude and
walks the seven-step funnel by hand. Raven implements the same methodology as a
program-driven loop: evolver/orchestrator/loop.py::EvolutionOrchestrator
drives the funnel, and the SOP's CLI parts became functions inside the loop.
The SOP §8.0 division of labor (semantics to the model, deterministic
arithmetic to code) is preserved unchanged: diagnosis / design / verdict are
still LLM calls, wrapped in orchestrator/nodes/semantic.py::SemanticNode
(parsing errors are fed back to the model, with a limited number of repair
retries). Only "who presses the next-step button" changed.
The launcher automates the funnel and persists completed-round checkpoints, so an operator can resume without reconstructing the experiment by hand.
1. SOP §0 general rules -> implementation¶
| SOP clause | Raven implementation | Evidence (file :: symbol) |
|---|---|---|
| Gate0 pre-run environment health check | precheck injected per round, forced once before cold start | orchestrator/production.py::build_evolution_orchestrator (run_gate0); benchmarks/appworld/evolve/precheck.py::make_appworld_precheck |
| Infra-failure ladder: detect -> rerun <=2 -> score 0 if still broken | eval_with_infra_rerun, max_reruns=2, produces the _infra_rerun{1,2} directory ladder; the KEPT rule takes the measurement with the fewest infra trials |
orchestrator/scoring.py::eval_with_infra_rerun; benchmarks/appworld/evolve/adapter.py::ladder_out_dirs / read_kept_out_dir |
| ★ Denominator = total tasks; infra tasks never excluded, scored 0 in place | Gate-f only reports the contamination list, never shrinks the denominator | orchestrator/gates/pipeline.py::run_gates (comment cites SOP §0 directly) |
| Division of labor: semantics = model, determinism = code | see §0 above | orchestrator/nodes/semantic.py::SemanticNode |
| Two verdicts: navigation = K3 mean > vanilla (bank); credited = paired 2σ (paper) | PairedResult keeps them as two independent fields: promoted (navigator) and credited_2sigma (label); promotion reads only the former |
orchestrator/gates/paired.py::PairedResult / paired_lift |
| Paired σ: std of per-task paired diffs, removing between-task difficulty | d_i = rate_c,i − rate_v,i, se = stdev(d)/√n, z = lift/se |
orchestrator/gates/paired.py::paired_lift |
| Sealed test: keep test results out of selection, then evaluate retention | The AppWorld launcher scores the baseline and recorded round deliverables at finalization through unseal_retention. SealedTestRunner.score returns None; results are stored separately, but filesystem access is not isolated from arbitrary code. |
orchestrator/sealed/runner.py::SealedTestRunner / unseal_retention |
| Test never enters anchor/train | stronger than the SOP's discipline: a mechanized assertion — leakage raises at startup | orchestrator/sealed/runner.py::assert_no_test_leak, wired in loop.py construction |
| Discipline: diagnose from train trajectories only | the diagnosis corpus source hangs off train only; test trajectories have no read path (the sealed runner stores scores only) | orchestrator/scoring.py::EvalBackend.trajectories |
| Discipline: configuration identical throughout | run_meta.json records a configuration fingerprint; the launcher normally refuses a changed configuration or a finalized run. --force bypasses these guards and invalidates the sealed-experiment assumption. |
evolver/launch/state.py::RunMeta.check_config; evolver/launch/runner.py::_meta_guard |
2. SOP §1 cold start -> implementation¶
| SOP clause | Raven implementation | Evidence |
|---|---|---|
| Vanilla train full set x K=3 thick ledger | backend.cold_start() must return a non-empty stability ledger; the initial baseline is built from the evaluation results in the vanilla output directory and then frozen. |
orchestrator/loop.py (construction); benchmarks/appworld/evolve/run.py (seed_label="van0", cold_start_k) |
| Failure map covers >= 7 WHY classes | diagnose_round(min_why_classes=7) |
orchestrator/nodes/diagnose.py::diagnose_round |
| WHY x WHERE taxonomy; inducible for a new bench | TaxonomySpec + two-stage induce_taxonomy; induction failure raises loudly, never silently borrowing another bench's table |
orchestrator/nodes/taxonomy.py |
3. SOP §2 seven-step funnel -> implementation¶
| SOP step | Raven implementation | Evidence |
|---|---|---|
| ① Round 1 uses the cold-start map; round 2+ re-diagnoses the child (fired / flipped / still-failing), appending to the continuously updated failure map | _diagnosed_parents prevents duplicate diagnosis; merge_failure_maps accumulates cross-round and persists; flips recorded in _flips with a harm-replay excerpt showing the regressed task's actual failure |
orchestrator/loop.py (round body); orchestrator/production.py::outcome_hook |
| ② 1-2 WHYs x 2-3 candidates, budget-capped; env-gated default-off | WHY selection defaults to driver mode (model picks; formula fallback + shadow log); Budget(max_why_per_round x candidates_per_why) enforced in code; the appworld line passes per-node activation_env |
benchmarks/appworld/evolve/editor.py::driver_select_whys / rerank_whys; orchestrator/config.py::Budget; benchmarks/appworld/evolve/adapter.py |
| ③ Free pruning: beacon_guard + preflight | beacon: the editor rejects python edits without activation_beacon (hard); preflight: make_zero_hit_preflight zero-hit prune, off by default (see §6 ①) |
benchmarks/appworld/evolve/editor.py; orchestrator/production.py::make_zero_hit_preflight; benchmarks/appworld/evolve/run.py (zero_hit_preflight=False) |
| ④ Apply, create child node; path_guard shields the kernel; git persistence | edit-then-commit: edits land as a real git child commit of the parent, the working tree remains unchanged, and the program determines and returns the changed file paths | evolver/tree/git_ops.py::commit_files_as_child; evolver/applier/path_guard.py |
| ⑤a K=1 anchor generous-pass screen; σ_screen computed; cull at 1.5σ; three tiers | three buckets clear_win / within_band / cull, only cull is blocked; σ formula identical to the SOP, emitted by select_anchor from the ledger; the AppWorld line uses a focused-subset Fisher probe variant (also generous-pass: culls only significantly-worse) |
orchestrator/nodes/screen.py::screen_candidate; evolver/scheduler/anchor_selection.py::select_anchor / simple_anchor; orchestrator/gates/strategies.py::FocusedFisherGate |
| ⑤a anchor composition: affinity majority + icebreakers + sentinels, ⊂ train | all three roles implemented; the sentinel guard adds stratification the SOP does not have (stable = mean guard, fragile = Fisher, avoiding noise kills); the affinity data source is unwired (§6 ③) | evolver/scheduler/anchor_selection.py; orchestrator/gates/strategies.py (sentinel guard) |
| ⑤ Borrowing (SOP tags [defer]) | module present (byte-identical to upstream), unwired (§6 ②) | evolver/scheduler/tree_aware_bandit.py |
| ⑤b Survivors: full set x K=3 confirm | k_confirm=3, confirm runs the full train set |
orchestrator/gates/strategies.py |
| ⑥ Three shields in order Gate-f -> Gate-b -> Gate2 | run_gates is exactly this order; Gate-b fails OPEN without instrumentation data (never condemns an uninstrumented-but-honest candidate); the reported score always uses the full-set fixed denominator, so a Gate-b subset mean can never masquerade as the score |
orchestrator/gates/pipeline.py::run_gates; orchestrator/production.py (beacon-aware fired_source) |
| Gate-b data chain | write side: the editor forces inline beacons -> per-attempt beacon directories; read side: union over the confirm dir + infra-ladder siblings. Honesty note: attribution is presence-level — Gate-b proves a beacon in the candidate's code executed on a task, not that the beacon sat inside the mechanism's trigger condition; an unconditionally-placed beacon degrades Gate-b to a no-op (promotion still requires the full-train win) | evolver/activation/ledger.py::beacon_workspace / mark_beacons_enabled (called from benchmarks/appworld/batch.py); evolver/activation/ledger.py::read_fired_tasks |
| ⑦ Best of bank becomes parent; if no candidate passes, retain the existing parent and diagnose again | beat_vanilla patience signal + greedy parent selection |
orchestrator/loop.py |
| Termination: 10 rounds with nobody above vanilla (vs vanilla, not the previous parent) or 20-round cap; never consult test | TerminationTracker(patience=10, max_rounds=20), the signal defined as beating the FIXED vanilla; plus a protection the SOP lacks: errored rounds do not burn patience (max_consecutive_errors is a separate backstop) |
orchestrator/termination.py |
| Four node statuses | superset: pruned_inert / pruned_at_screen / pruned_at_confirm / promoted_to_baseline / errored / blocked_l1 / archived-methodology-failure; inert deaths additionally feed the per-WHY history so the designer learns from them |
evolver/tree/node.py::NodeStatus; orchestrator/production.py::inert_hook |
| WHERE bound mechanically from the artifact, never by self-declaration | bind_where derives the lever from the actually-touched files; the self-declared patch_where stays in the ledger for audit and never decides the archive coordinate (4-tier granularity, see §6 ④) |
orchestrator/archive.py::bind_where / cell_of |
4. SOP §3 persistence -> implementation¶
| SOP layer | Raven counterpart |
|---|---|
| findings work log | <work_dir>/findings.md (one section per round, driver verdict) |
| Cross-session state | journal (orchestrator/state/journal.py, crash-resume replays completed rounds) + history.json (per-WHY attempt history) |
| Box durable artifacts | failure_map.json (failure map updated across rounds) / nodes/<id>.json (node ledger: identity + git anchor + final status + gate stats) / per-round out-dirs (per-task results) |
The SOP §3.1 recording scheme is retained: Git commits store code versions
(commit_files_as_child produces commit SHAs), and nodes/*.json stores the
evolution tree structure. The two are linked by git_commit_sha.
5. Where we exceed the SOP¶
- Sealed-test evaluation is automated.
assert_no_test_leakchecks that task splits do not overlap. At finalization,unseal_retentionscores recorded commits and selects the deliverable by training score. These controls separate evaluation from selection; they do not sandbox candidate code or prevent an operator from reading files. - A single candidate's crash cannot sink a round: the
erroredstatus + errored rounds not burning patience (max_consecutive_errorsas its own backstop); not covered by the SOP. - QD archive and recombination: a (WHERE x WHY) per-cell elite bank +
cross-cell recombinant candidates (
orchestrator/archive.py) — an exploration mechanism beyond the SOP; the gates and evaluation criteria remain unchanged. - The inert-death feedback loop (2026-07): a distinct
pruned_inertstatus + inert deaths recorded into history + the design prompt distinguishing "the trigger never fired" from "the mechanism was rejected" - gentle WHY decay for inert deaths (
0.55^n_fail x 0.85^n_inert). - Harm replay: when a candidate causes a task to regress, an excerpt of that task's actual execution trajectory under the candidate is provided to the next candidate-design attempt; the SOP only requires flip counts.
6. Implementation choices¶
- Optional zero-hit preflight (
zero_hit_preflight=Falseby default). With instrumentation supplied, Gate-b excludes tasks where no beacon fired; without it, Gate-b is skipped. Preflight is a separate budget-saving check that can prune useful candidates. Enable it for runs that supplyTRIGGER_REGEXand inspect theirpruned_inerthistory entries to assess pruning. SOP §2 marks this step as [now]; Raven makes it opt-in. - Full-set confirmation. AppWorld evaluates the full training set.
The orchestrator does not call
tree_aware_banditfor borrowing (SOP §2 ⑤, tagged [defer]). - Anchor composition. AppWorld anchors use icebreakers, sentinels, and
borderline tasks.
select_anchor(affinity)accepts affinity data, but the AppWorld adapter does not supply a trigger-density source. The effect of this selection on evaluation results has not been measured separately. - Four-tier WHERE binding. Mechanical binding uses prompt/runtime/mixed/edit for QD cells, while the judge schema has 14 classes. The ledger records both self-declared and mechanical values for comparison; it does not raise an automatic reconciliation alarm.
pruned_inertmatches the SOP's status semantically but differs in implementation: the SOP judges it via a preflight CLI, we via the embedded loop preflight (default off, see ①).
6.5 The unified entry (added 2026-07)¶
SOP §8.3's "manual orchestration" is superseded by
python -m evolver run --config <yaml>: a single-command state machine runs
cold start -> rounds -> termination -> unseal. Before finalization, it resumes
from saved trials and completed-round checkpoints, not from an arbitrary
instruction inside an interrupted call. Configuration drift and finalization
are checked against run_meta.json unless --force overrides the guards.
Benches plug in via the contract in
docs/specs/evolve-bench-contract.md, kept in the repository beside the code;
implementation in evolver/launch/ + evolver/cli.py.
7. SOP parts <-> Raven parts quick reference¶
| Part cited in SOP §8.1 | Raven counterpart | Shape difference |
|---|---|---|
analysis/proxy_features.py |
evolver/analysis/proxy_features.py |
byte-identical |
analysis/failure_map_builder.py |
evolver/analysis/failure_map_builder.py |
byte-identical |
activation/preflight.py (CLI) |
orchestrator/production.py::make_zero_hit_preflight |
CLI -> embedded; regex variant, default off |
analysis/stability_bucket.py |
evolver/analysis/stability_bucket.py |
byte-identical |
scheduler/bandit_tasks.py |
evolver/scheduler/bandit_tasks.py |
byte-identical |
scheduler/anchor_selection.py |
evolver/scheduler/anchor_selection.py |
same lineage + our simple_anchor |
scheduler/affinity_picker.py |
no counterpart (§6 ③) | — |
scheduler/tree_aware_bandit.py |
evolver/scheduler/tree_aware_bandit.py |
same lineage, unwired (§6 ②) |
activation/gate_audit.py (CLI) |
orchestrator/gates/pipeline.py::run_gates (Gate-b embedded) |
CLI -> embedded |
analysis/paired_significance.py (CLI) |
orchestrator/gates/paired.py::paired_lift; retention in sealed/runner.py::unseal_retention |
CLI -> embedded |
aggregate_keq3 / gate0_ctrf_audit (external eval engine) |
benchmarks/appworld/evolve/eval.py / adapter.py (K=3 aggregation + infra ladder) |
cross-repo contract -> same-repo module |
tree/* |
evolver/tree/ (node/store/git_ops) |
same lineage + our commit_files_as_child / read_file_at |