01 — OVERVIEW
EvoAgentBench
Benchmarking Agent Self-Evolution via Ability Transfer
EvoAgentBench evaluates whether agents can transfer reusable procedures from past experience to new tasks. Its Ability-guided split covers web research, algorithmic reasoning, software engineering, and knowledge work, with verified training-side support for every test task.
Multi-Domain Evaluation
4 agentic domains — web research, algorithmic reasoning, software engineering, and knowledge work — with 528 training tasks and 267 supported test tasks.
Multi-Agent Support
Plug in any CLI-based agent — Nanobot, OpenClaw, or your own. Each task runs in isolated config with independent workspace, supporting concurrent execution and automatic retry.
Self-Evolution Comparison
A standardized train → evolve → evaluate protocol comparing Memento, ReasoningBank, GEPA, and the diagnostic Anchor Skill† reference under matched conditions.
02 — LEADERBOARD
Method ranking per configuration
For each (agent, model, domain) cell, methods are sorted by Δ gain (method score − Vanilla). The Vanilla score is shown next to each method.
Web Research
Algorithmic Reasoning
Software Engineering
Knowledge Work
Web Research
Algorithmic Reasoning
Software Engineering
Knowledge Work
Web Research
Algorithmic Reasoning
Software Engineering
Knowledge Work
Web Research
Algorithmic Reasoning
Software Engineering
Knowledge Work
Web Research
Algorithmic Reasoning
Software Engineering
Knowledge Work
Web Research
Algorithmic Reasoning
Software Engineering
Knowledge Work
Bar length encodes Δ magnitude. See the full leaderboard for per-cell scores and the paper’s turn-cost analysis.
03 — WHY THIS MATTERS
Why Self-Evolution Matters, and What We Learned Running This
Why self-evolution matters
If an agent solves a problem today, it shouldn’t have to start from zero on a similar one tomorrow. Useful experience isn’t just a log of what happened — it’s a way of working: a search habit, a debugging move, a verification step, a recipe for producing something useful. Self-evolution is the question of whether an agent can pick up these habits on its own, from its own past attempts, without retraining the model underneath.
Most benchmarks today don’t really test this. They either ask “can the agent solve a fresh task?” or “can the agent remember what it saw?” Neither tells you whether yesterday’s way of working actually shows up when the agent tries something new today. EvoAgentBench is built around that specific question.
What the experiments show
Across two agent scaffolds, three backbones, and four domains, the results expose where current self-evolution systems still break.
Ability content transfers across model families. Anchor Skill† improves every scaffold–backbone–domain cell even though its construction backbones are disjoint from the evaluation backbones. The procedural content is transferable when it is extracted and routed correctly.
Every automatic method still shows negative transfer. Memento, ReasoningBank, and GEPA each improve some settings, but none remains positive in every cell. One striking mismatch is Memento on Nanobot / Qwen3.5-27B / software engineering, where performance drops by 36.3 points.
Extraction and routing remain the bottleneck. Automatic methods see the same training prompts, trajectories, and verifier outcomes. Their uneven gains therefore point to how reusable content is extracted, indexed, and delivered at test time, not to missing training-side support.
What we did to make the comparisons fair
For numbers to be worth comparing, the setup has to be tight. Three things we cared about most:
Nobody gets to peek at the answers. Methods can only look at training tasks: the question, what the agent tried, and whether it worked. The actual test answers, and the test trajectories that succeeded, are off-limits during evolution. We didn’t make this a rule for people to follow; we wired it in so the two paths simply don’t meet.
Training and test tasks are related, not random. The Ability Graph links tasks through trace-grounded procedural overlap. Every test task has verified support from training-side Abilities, so a failed transfer cannot be dismissed as an unsupported test case.
Every cell on the leaderboard is real. Two agent frameworks, three backbones, four domains, and four evolution conditions produce 96 method comparisons, all measured against matched Vanilla baselines. Tasks, tools, timeouts, scoring, and base agent configuration are fixed across methods; results are averaged over three independent runs per instance.
04 — DOMAINS
Evaluation Domains
EvoAgentBench builds an Ability Graph over four agentic domains and selects a 528/267 train/test split with verified training-side support for every test task.
| Domain | Base Benchmark | Description | Abilities | Communities | Abilities / Task | Train | Test |
|---|---|---|---|---|---|---|---|
| Web Research | BrowseComp-Plus | Search a controlled corpus and verify evidence for multi-constraint questions. | 38 | 13 | 2.42 | 154 | 65 |
| Algorithmic Reasoning | LiveCodeBench | Solve programming problems against hidden test cases with pass@1 scoring. | 58 | 22 | 2.12 | 182 | 86 |
| Software Engineering | SWE-Bench Verified | Repair real-world repositories and pass the designated hidden test suites. | 38 | 15 | 2.21 | 87 | 56 |
| Knowledge Work | GDPVal | Produce professional deliverables evaluated against expert references. | 36 | 6 | 2.63 | 105 | 60 |
05 — SELF-EVOLUTION
Evolution Conditions
EvoAgentBench provides a standardized protocol for evaluating agent self-evolution methods — techniques that let agents learn from past experience and improve future performance. The paper evaluates three automatic methods plus Anchor Skill†, a diagnostic reference.
Memento
Retrieval-based (CBR)Retrieves the closest retained training case, using surface similarity to route prior task experience.
GitHubReasoningBank
Memory + reasoningDistills natural-language strategies from training trajectories and retrieves them by strategy similarity.
GitHubGEPA
Prompt evolutionEvolves one prompt on the training split and broadcasts it to every test task without per-task retrieval.
GitHubAnchor Skill†
Diagnostic referenceDiagnostic reference using train-side Ability-grounded skill content and curator-side Ability-label routing. It is not a deployable automatic method.