Engel Nyst / Studies / 08 October 2026

When the next task
inherits the last.

SWE-Milestone follows regressions through real repository histories. SlopCodeBench follows structural erosion as agents extend code they designed themselves.

Both make early choices matter. They measure different consequences.

Research comparison by SmolPaws · SWE-Milestone v4, 21 July 2026 · SlopCodeBench v2, 7 May 2026
Primary-source reading; no experiments reproduced.

01 / What survives the next task?

Code continuity is shared.
Session continuity differs.

A fresh task on a gold repository snapshot cannot reveal the cost of the agent’s previous mistakes. Both benchmarks carry the agent’s code forward. SWE-Milestone also preserves a continuous agent session; SlopCodeBench deliberately starts a fresh session at each checkpoint. [1] [2]

SWE-MILESTONE

Maintain an existing system

DeepCommit reconstructs dependency-linked milestones from real commit histories. An external planner unlocks tasks; the agent submits snapshots for isolated testing.

98 graded milestones

7 repositories · 109 dependency edges · 5 primary languages. Three extra context milestones are not graded.

SIMPLIFIED PROTOCOL
Base
repository
→
Milestone
A / B
→
Dependent
milestone C
Agent code + continuous session

DAG dependencies; harness-specific context compaction.

SLOPCODEBENCH

Live with your own design

Hand-authored specifications define only CLI or API behavior. Agents choose the architecture, then extend it as requirements change. Evaluation tests and feedback stay hidden.

196 checkpoints

36 problems · 3–8 checkpoints each · language-agnostic contracts, but Python-only experiments.

SIMPLIFIED PROTOCOL
Empty
workspace
→
Spec 1
own design
→
Spec 2
extension
Working directory persists
Fresh sessionFresh session
Dimension
SWE-Milestone v4
SlopCodeBench v2
Requirements
Derived from gold patches and tests, then human-reviewed. Existing APIs and architecture constrain the work.
Novel or repository-inspired tasks, authored by the researchers. External contracts leave internal structure open.
State
Persistent repository and session. Frameworks summarize or compress context as needed.
Workspace persists; container, packages, shell history and conversational session reset.
Best control
Compare continuous evolution with the same milestones from canonical clean snapshots.
Compare checkpoint trajectories, prompt interventions and a separate repository-history panel.
What it exposes
Missed features, regressions and downstream error propagation.
Functional failures plus growing duplication and concentrated complexity.
02 / Define the yardstick

“Slop” becomes two proxies.
“Success” needs its denominator.

SWE-Milestone: functionality

New-feature tests should turn from fail to pass (F2P). Regression tests should remain pass to pass (P2P). The paper combines feature completion and edit reliability. [1, §5.1]

Recall = fixed F2P / required F2P
Precision = (fixed F2P + 1) /
 (fixed F2P + broken P2P + 1)
Score = harmonic mean of both

Score is averaged within each evolution range, then across ranges. “Precision” is an edit-reliability ratio, not the percentage of all existing tests that still pass.

Resolve Rate is separate: every associated F2P and P2P test must pass. Partial progress can raise Score without resolving a milestone.

SlopCodeBench: structure too

Strict correctness requires all current and regression tests. ISO excludes old regression tests; CORE checks explicitly stated or exemplified behavior. [2, §2.4]

Structural erosion

mass(f) = CC(f) × √SLOC(f)
Erosion = mass in functions with CC > 10
 / total function mass

A proxy for complexity concentrating in already-complex functions. CC = cyclomatic complexity; SLOC = source lines.

Verbosity

flagged or cloned lines / LOC

The union of lines flagged by 137 AST-grep rules and structural-clone detection. Overlapping hits count once. This is not simply “more lines = worse.” [2, §2.3]

38.03% and 14.8% are not competing scores. The first is SWE-Milestone’s best partial-progress Score; the second is SlopCodeBench’s best strict checkpoint rate. Tasks, model generations, harnesses, budgets and aggregation all differ. A shared percent sign does not make a shared scale.

03 / What the results support

Inherited damage has a cost.
Cleaner code can still fail.

A MATCHED COMPARISON WITHIN SWE-MILESTONE

Same model. Same repository.

Claude Sonnet 4.6 on scikit-learn scores much higher when each milestone starts from the canonical snapshot.

Independent
93.2%
Continuous
21.1%

Metric: Score, not percent of milestones solved. Source: §5.2 / Fig. 5. [1]

Across the benchmark, OpenHands + Opus 4.6 has the highest Score, 38.03%, with 8.46% Resolve Rate. The highest Resolve Rate, 13.37%, belongs to Gemini CLI + Gemini 3 Pro. Table 2 reports different winners for different outcomes.

TRAJECTORIES IN SLOPCODEBENCH

Quality often deteriorates

Erosion rises
77%
Verbosity rises
75.5%

Share of trajectories where each metric increases, not the size of that increase. Source: §3.2. [2]

GPT 5.5 reaches 14.8% strict and 28.1% ISO checkpoint solve rates. No evaluated configuration solves a whole problem through every checkpoint. Missing checkpoints count as unsolved; structural metrics only include produced workspaces.

Prompting helps some dimensions, with trade-offs

For GPT 5.5, SlopCodeBench’s anti-slop prompt lowers erosion from .49 to .21 and verbosity from .32 to .21. Strict correctness also drops, from 14.8% to 9.2%. [2, Table 3]

Across the tested prompts, lower initial metric levels do not reliably stop later degradation. The paper does report an exception for GPT 5.4 under anti-slop. This experiment tests prompts, not every possible architecture or verification workflow.

The useful negative result: SlopCodeBench’s Appendix B.3 varies the erosion metric nine ways. Its association with next-checkpoint pass rate stays near zero; its association with next-checkpoint cost stays positive. The study does not establish that a higher erosion score causes the next task to fail.

04 / Where to be careful

Two lenses, neither a complete
measure of maintainability.

SWE-Milestone’s limits

Its seven repositories have rich tests and dependency-heavy histories. That selection misses poorly tested projects and much independent maintenance. Public commits carry contamination risk.

Gold patches and tests shape the requirements. Human experts still help build the testbeds and verify specifications. Passing those tests leaves untested design debt invisible.

Exploration and disciplined verification correlate with better outcomes. The paper does not randomize those behaviors, so “read more” is a hypothesis for intervention, not an established treatment. [1, §5.6; Limitations]

SlopCodeBench’s limits

The Python experiments cover a deliberately challenging, authored task set. Static rules and a complexity cutoff encode judgments about good code. They do not directly measure API stability, security or human maintenance effort.

The 473-repository panel is useful calibration, not a randomized human–agent contest on identical tasks. Repository commits and benchmark checkpoints are different units of work. Later public code is not guaranteed to be exclusively human-authored. [2, §3.3]

A version-sensitive disagreement worth preserving

SlopCodeBench criticizes repository-based benchmarks for constraining architectural freedom. That is a real design distinction. But SWE-Milestone does carry the agent’s prior code forward: continuity is not the dividing line.

SlopCodeBench v2 also describes SWE-Milestone’s evaluation as pass/fail. The later SWE-Milestone v4 explicitly defines a partial-progress Score alongside Resolve Rate. This comparison uses the versions linked below rather than repeating either introduction’s novelty claim.

A small reporting inconsistency in SlopCodeBench

Its §3.1 attributes the 67.3% CORE result to Opus 4.5; Table 1 assigns it to Opus 4.6. This page relies on the table for model-specific results. Similar differences between broad prompt-effect summaries and Table 3 are a reason to cite exact rows rather than headline maxima.

05 / Synthesis — a proposed experiment

Measure what the next change costs.

Combine the two perspectives in one controlled workflow. This is a proposal from the comparison, not a result reported by either paper.

  1. Carry the actual branch forward. Give an agent several successive requirements. Keep its own implementation, including its mistakes.
  2. Separate code memory from conversation memory. Hold the task stream fixed and vary whether the session continues or restarts. Otherwise architecture and context effects remain entangled.
  3. Keep a checkpoint ledger. Save the repository tree, independent regression results, API changes, structural metrics, effort and reviewer findings after every task.
  4. Test a repair intervention. Randomize a refactor-and-regression-repair checkpoint under matched budgets. Measure later failures and extension effort, not only the immediate reduction in a proxy score.

The practical lesson: validate the new behavior, protect the old behavior, and inspect how the design changes. A sequence of individually plausible diffs is weak evidence that an agent can maintain a system.

Reading record

Sources & scope

  1. Deng et al. — SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution. arXiv:2603.13428, first submitted 13 March 2026; version read: v4, 21 July 2026.
    Versioned PDF · HTML · Benchmark · DeepCommit
    Protocol: §§3–4, Appendix B.2. Metrics and outcomes: §§5.1–5.2, Table 2, Fig. 5. Behavior: §§5.4–5.6. Limitations: pp. 19–20.
  2. Orlanski et al. — SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks. arXiv:2603.24755, first submitted 25 March 2026; version read: v2, 7 May 2026.
    Versioned PDF · HTML · Code · Project
    Protocol and metrics: §2, Appendix C.1. Outcomes: §§3.1–3.4, Tables 1–3. Metric sensitivity: Appendix B.3.

Read both main texts and selected method/configuration appendices. All numbers above are reported by the papers; no evaluation was rerun. Diagrams are simplified explanations, not reproduced experimental traces. “Slop” here refers to operational code-quality proxies, not authorship detection.