Many SWE-bench-Passing PRs would not be merged

Summary: We find that roughly half of test-passing SWE-bench Verified PRs written by mid-2024 to mid/late-2025 agents would not be merged into main by repo maintainers, even after adjusting for noise in maintainer merge decisions. Since the agents are not given a chance to iterate on their solution in response to feedback the way a human developer would, we do not claim that this represents a fundamental capability limitation. Rather, our results indicate that a naive interpretation of benchmark scores may lead one to overestimate how useful agents are without more elicitation or human feedback.

---

Introduction

It is often unclear how to translate benchmark scores into real-world usefulness. For example, if a model’s SWE-bench Verified score is 60%, does that mean it can resolve 60% of real-world open-source issues? One reason to doubt this is that benchmarks are clean and verifiable in ways the real world is not. To study this quantitatively, we take SWE-bench Verified and zoom in on one such difference — it uses an automated grader rather than the real-world standard of maintainer review.

To study how agent success on benchmark tasks relates to real-world usefulness, we had 4 active maintainers from 3 SWE-bench Verified repositories review 296 AI-generated pull requests (PRs). We had maintainers (hypothetically) accept or request changes for patches as well as provide the core reason they were requesting changes: core functionality failure, patch breaks other code or code quality issues.

To deal with noise in maintainer decisions, we also record maintainer merge decisions on 47 original human-written PRs that were actually merged into main (hereafter “golden patches”). We report all our scores as % of golden baseline, e.g., as the golden baseline is 68%, if a model gets 34%, then the golden-baseline-adjusted score is 50%.¹

Figure 1 shows our main result that on average maintainer merge decisions are about 24 percentage points lower than SWE-bench scores supplied by the automated grader. Moreover, the rate of improvement, as measured by percentage points gained per year (pp/yr), is 9.6 pp/yr slower for maintainer merge decisions. However, we believe our results on the rate of improvement are shakier and only provide suggestive evidence that it is slower for maintainer merge decisions.

Our central claim is about comparing a naive interpretation of benchmark scores with a richer view of agent usefulness. It is important to note what we are not doing:

1. We are not claiming that agents have a capability limitation that prevents them from passing maintainer review. It seems likely that better prompting and agent elicitation could resolve many of the remaining problems of code quality, not following repo standards, etc.
2. We are not comparing AI to humans in equal conditions. While models only have one attempt at submitting a solution, human developers iterate with feedback in most cases.
3. We are not claiming benchmarks have no signal or that you should not use benchmarks to compare models. Instead, we are showing that mapping benchmarks to real-world AI capabilities is difficult and has many subtle problems. Therefore, those forecasting AI progress and its real-world impact should view benchmarks as one piece of evidence, rather than as decisive.

Previous Post

In a previous blog post, we compared a small number of PRs that passed algorithmic tests vs. our own code review. This investigation makes 5 advances over that post:

1. We use actual maintainers for code review rather than reviewing it ourselves.
2. We focus on SWE-bench Verified scores, making results more interpretable.
3. We cover 95 unique tasks vs. 18 tasks and cover 5 language models instead of 1.
4. We account for noise in human review by normalizing our results by the golden baseline.
5. Reviewers were blinded to whether the PR was written by a human or an AI.

While we believe these advances make our current results more reliable, they are not strictly comparable to the results in our previous post. In particular, the previous post considered much more difficult PRs. In our sample of SWE-Bench Verified golden patches, the average lines of code changed is about 17 while for our previous post this was over 500.

Data and Methods

Our basic methodology is to take PRs that are scored as correct by the SWE-bench automated grader, and have current maintainers review whether the patches would be merged into main.

Maintainers and Repositories

We recruited 4 active maintainers covering 3 SWE-bench Verified repos: 2 from scikit-learn, 1 from Sphinx, and 1 from pytest. Therefore, we have coverage for 3/12 (25%) of the repos in SWE-bench Verified, covering 95/500 (19%) of the issues. Maintainers were recruited via cold email. In the appendix, we show that our sample is representative of SWE-bench Verified when measured by pass rates. Maintainers are paid hourly with a bonus if peers label their reviews as high quality.

Agent Runs

We use the SWE-bench Verified agent runs from Epoch’s benchmarking hub (Epoch AI, 2025).² We pull patches from the following:

• Claude 3.5 Sonnet (Old)
• Claude 3.7 Sonnet
• Claude 4 Opus
• Claude 4.5 Sonnet
• GPT-5
• Golden patches (original human solutions)

We focus primarily on Anthropic models because they have been state-of-the-art on SWE-bench Verified for most of its history.

We use patches in their original state, except we manually remove miscellaneous debugging files to avoid penalizing agents for artifacts they were not instructed to remove prior to submission.³ We upload the patches to a private copy of the repo in its historical state on GitHub.

For our main analysis, we only submit PRs for review if they pass the SWE-bench automated grader. We impute maintainer review as failure if the PR fails the automated grader. This amounts to assuming SWE-bench Verified has no false negatives (cases where the automated grader says fail, but the maintainer would actually merge it). We think this is a fairly reasonable assumption because:

1. SWE-bench Verified was human-verified to screen out unit tests that would reject valid fixes. Although this process is imperfect, it likely reduced the false negative rate substantially.
2. We had maintainers review a small subset of 31 patches covering 27 issues. We found only 1/27 (3.7%) issues where the automated grader rejected valid patches.⁴

The appendix has results without this assumption and results that try to relax it.

Maintainer Review

For AI-generated patches, maintainers review all of the patches that pass the SWE-bench automated grader.⁵ For the golden patches, we have maintainers review only half of them (randomly selected) to save time. In the case of scikit-learn, where we have more than one maintainer, we randomly split which patches are reviewed by which maintainer.

Maintainer review is done in GitHub, to match real code review as closely as possible. The review is done in waves. Each wave contains non-overlapping issues attempted by different models. The maintainers are not told the source of the pull request.⁶

We ask maintainers to review these PRs exactly as they do PRs in real life, with two exceptions:

• Lack of CI: Because we are uploading the patch to the repo in its historical state, it is not possible to keep Continuous Integration tools active. Therefore, maintainers do not have access to linting or tests. To supplement this, we tell them that the patches passed an automated grader built from tests. We ask the maintainers to review to the best of their ability without these tools.
• Testing Requirements: We ask them to ignore testing requirements in the PR, as agents are not elicited to include proper tests. This is generous to AI-generated patches, as it removes one requirement of a good patch.

We ask the maintainers to give an accept/request changes decision, structured feedback, and natural language feedback. The structured feedback covers:

• Rejection Reason (if applicable):
  • Core functionality: The PR does not correctly resolve the issue or the solution has bugs.
  • Breaks other code: The PR touches unrelated code and causes breakages.
  • Code quality: The code is verbose, does not conform to standards, etc.
• Progress: How much progress the PR makes (0-100%).
• Review Time: How long it took to review the PR.
• Baseline Time: How long it would take them to write a proper PR from scratch.
• Confidence: Confidence in the decision (1-10).

Finally, maintainer reviews may be noisy, as they involve subjective decisions on code quality, etc. Therefore, we benchmark agent pass rates vs. the level of noise in the golden patches. Recall that SWE-bench Verified is made from real-world PRs that were merged. We take these real-world PRs and re-submit them to our maintainers to establish a baseline for noise in our maintainer merge pipeline.⁷ We found that about 68% of golden patches are indeed merged by our maintainers, and the average progress towards a mergeable PR among all golden patches is about 90%. Moreover, around 85% of golden patches are rated as making 80% or more progress towards a mergeable PR. The percentage merged is low, but the percentage rated as making 80% or more progress toward a mergeable PR is high, indicating that there is a degree of maintainer subjectivity playing a role in the last mile of PR acceptance. We report all our scores as % of golden baseline accordingly, e.g., if the golden baseline is 68% and a model gets 34%, then the golden-baseline-adjusted score is 50%.

Stats

All confidence intervals are 95% and computed via nonparametric bootstrap at the patch level. For trend lines, we fit an unweighted linear regression of pass rates against model release dates. Trend confidence intervals reflect only uncertainty in pass rates, conditioning on which models are in the sample. We do not account for the uncertainty in the golden baseline.

Results

Basic Results

Figure 1 plots the pass rate for our language models according to the SWE-bench automated grader and according to maintainers. There are two immediate conclusions from this graph.

First, the maintainer merge rate is well below the SWE-bench pass rate. In the bottom right, the top row shows the automated grader is on average about 24.2 percentage points (standard error: 2.7) higher than the maintainer merge decision. This is strong and statistically significant evidence that the naive interpretation of the benchmark as passing maintainer review is misleading.

Second, at the bottom of Figure 1, we show the difference in percentage points gained per year (pp/yr). Notice that the maintainer merge decision is gaining about 9.6 pp/yr (standard error: 5.5) less than the automated grader. This is a weaker effect than the level difference; it is only statistically significant at a 10% significance level rather than 5%. Moreover, the pp/yr result is less robust than the level result, e.g., the result changes if trend lines are filtered to SOTA models only, so we take this as only weak evidence that the rate of improvement is slower for maintainer merge decisions.

Next, as a robustness check, Figure 2 changes the pass criteria from maintainers merging the PR to maintainers saying the PR makes sufficient progress towards a mergeable PR. Concretely, we define a 0–1 pass as maintainers rating that the PR is more than 80% of the way towards a mergeable PR, continuing to impute no pass if the automated grader fails the patch. Note that 80% progress cutoff is the 15th percentile of the golden patches, so this is a fairly low bar. We find very similar results: scores are roughly half and the trend is about 10 pp/yr slower.

Reasons for Rejection

To understand why some PRs passed the automated grader but failed maintainer review, Figure 3 breaks out failures by their primary reason, from least serious to most serious problem:

• Code quality: e.g., bad style, not following repo standards.
• Other failure: Undocumented failures.
• Breaks other code: e.g., solving the issue but breaking other code in the repo.
• Core functionality failure: e.g., not solving the problem raised in the issue.
• Failing automated grader.

This figure provides more texture to our understanding of AI progress on SWE-bench Verified:

1. Claude 3.5 Sonnet → Claude 3.7 Sonnet: Large increase in pass rates, but also more cases where the maintainer flagged core functionality issues despite the patches passing the automated grader.
2. Claude 3.7 Sonnet → Claude 4 Opus: Minimal pass rate improvement but substantial progress moving issues from failing the automated grader to only bad code quality.
3. Claude 4 Opus → Claude 4.5 Sonnet: Mostly about improving code quality.
4. GPT-5 is substantially weaker at code quality than Anthropic models.

Figures 4 and 5 show example rejections for code quality. Figures 6 and 7 show example rejections for core functionality and breaking other code respectively.

Conclusion

Technical limitations

Our study has many technical limitations, including:

1. We only study a subset of a single benchmark with one agent harness.
2. Our maintainer review is not fully realistic (no CI, issues are old, golden patches were already merged and may be recognized by reviewers).
3. We compared patches statically, based on whether they are correct. But in a real-world setting: patch creation is dynamic; there is a creation, review, and iteration process. An interesting future study would compare time to complete a finished patch by AI agents vs. humans.
4. We use maintainer review as the ‘market test’ for proper code, but the standards for open source repos may change over time as agents write and review more code. Maintainer review standards may shift substantially as a result.
5. The most recent model in our study is Claude Sonnet 4.5, but more recent models might have different results.

Discussion

In early 2025, METR published results showing that open-source developers were slowed down when they used AI. This was surprising to us and other experts, at least in part because SOTA models at the time could autonomously complete 40-52% of issues in SWE-bench Verified. The gap between the automated SWE-bench Verified grader and maintainer review perhaps makes this slowdown slightly less surprising.

In general, these results caution against naive extrapolation of some benchmarks to real-world usefulness. The results above focus on SWE-bench Verified, but we suspect similar lessons apply to other benchmarks interpreted in the context of human workflows, e.g., GDPval-AA, UpBench, etc.

A1. Conditional Maintainer Merge Rate

The main results impute maintainer failure for patches that fail the automated grader. An alternative is to look at just the false positive rate: among patches that *pass* the automated grader, what share do maintainers actually merge? This avoids any assumption about false negatives. Figure 1 shows that AI-generated patches that pass the automated grader are merged at a lower rate than human golden patches, though we caution against direct AI-vs-human comparisons given the different conditions under which these patches were produced.

A2. Sample Representativeness

Our maintainer sample covers 3 of 12 SWE-bench Verified repos (scikit-learn, Sphinx, pytest), comprising 95 of 500 issues. Figure 2 compares the automated grader pass rates on our subset against the full dataset, showing close alignment and suggesting our subset is not biased by task difficulty.

A3. False Negative Correction

Our main analysis assumes all patches that fail the automated grader would also fail maintainer review (i.e., no false negatives). To test robustness, we estimate the grader's false-negative rate from a small subset of 31 patches covering 27 issues where maintainers reviewed automated-grader-failing patches. We found only 1/27 (3.7%) issues where the automated grader rejected valid patches. Figure 3 shows that correcting for this estimated false-negative rate yields similar results.

A4. Raw (Unnormalized) Pass Rates

The main results normalize by the golden baseline (68% maintainer merge rate on human patches). Figure 4 shows raw, unnormalized pass rates. Under this specification, maintainer merge rates are roughly one-third to one-half of automated grader pass rates, and the difference in the rate of improvement is 15.5 pp/yr.

A5. Raw (Unnormalized) Progress-Based Pass Rates

Figure 5 shows the same unnormalized view using our alternative measure of PR success - that maintainers rated progress towards a mergeable PR is over 80%.

A6. SOTA Models Only

The trend result (widening gap over time) is sensitive to which models are included. Figure 6 restricts the sample to models that were state-of-the-art at their time of release. The trend difference is smaller and not statistically significant under this restriction, illustrating the fragility of the trend finding.

A7. Results by Repository

Figures 7–9 show results separately for each repo. The level gap between automated grader and maintainer pass rates holds across all three repos. The trend result holds for Sphinx and pytest but is noisy and does not hold for scikit-learn.

A8. Ordering Effects

Maintainers reviewed the same issue multiple times across different models, which could bias assessment if earlier reviews influence later ones. To test for this problem, we construct a PR level dataset that records which wave the PR was in. Recall PRs are reviewed in waves, where each wave includes a single issue at most once. We regress the maintainer merge decision on wave number, controlling for model and issue fixed effects. If seeing an issue multiple times before made maintainers harsher, then we would expect the coefficient on wave to be negative and vice versa if it made them more lenient. Table 1 shows no evidence of ordering effects: the wave coefficient is 0.016 (SE: 0.037, p = 0.67) and the partial R² for wave is about 0.001, indicating wave order explains very little of merge decisions.

Table 1: Linear Regression of Merge Decision on Wave Order

Variable	Coefficient	P-value
Wave	0.016	0.665
Model FE	Yes
Issue FE	Yes
N	329
Partial R² (Wave)	0.0008

A9. Time Horizon Analysis

We apply the time horizon methodology from Kwa et al. (2025), which converts benchmark scores into a “time horizon”—the human completion time of tasks at which a model achieves 50% success. We use the time-to-complete estimates from SWE-bench directly, though we do not have task families or task weighting as in the original methodology. Note that time-to-complete estimates from SWE-bench come in ranges (e.g., 15min–1hr); we take the geometric mean to get a point estimate.

Methodology. For each evaluation criterion (automated grader and maintainer merge), we fit logistic regressions of pass rate against log task duration. The time horizon T₅₀ is the task duration at which the fitted model predicts 50% success. Note that these estimates are less stable than in Kwa et al. because the task duration range in SWE-bench Verified is narrower, requiring extrapolation away from the data to estimate a 50% time horizon.

Results. The automated grader gives substantially higher time horizons than maintainer review. For example, Claude Sonnet 4.5 has an approximately 50-minute time horizon according to the automated grader but only about an 8-minute time horizon according to maintainers—roughly a 7× overstatement. This level difference is our most robust finding from the time horizon analysis.

Trend. Figure 12 plots time horizons against model release dates. We want to heavily caveat this figure. The only robust conclusion is that the level of time horizon is overstated by a large amount when using the automated grader versus maintainer review. Although it would be tempting to also conclude that the trend is slower under maintainer review, this finding is extremely noisy. The confidence interval for the difference in doubling time ranges from −158 months to 2 months, which is so wide that it is hard to take seriously. Moreover, this result is not very robust to different design decisions. Because SWE-bench data has fairly minimal time horizon dispersion, drawing robust, sufficiently powered conclusions about the rate of improvement is difficult.

Summary. We find strong evidence that the level of time horizon is overstated by a large amount when using the automated grader versus maintainer review. However, we are unable to draw confident conclusions about the rate of improvement.