🔍 Read the full analysis: FrontierCode: What Does It Take For A Maintainer To Merge AI’s Pull Request? on ThorstenMeyerAI.com
Get privacy and security gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, testing 23 AI models on whether repository changes meet maintainer-style review criteria. Claude Opus 5.5 ranked first, but passed every blocking check in 57.7% of attempts; results also show substantial cost differences between models.
Artificial Analysis tested 23 AI models on FrontierCode, a benchmark designed to judge whether a coding agent’s changes would meet maintainers’ criteria for merging, rather than simply pass visible tests. Claude Opus 5.5 ranked first at 52.2%, but cleared every blocking check in 57.7% of attempts, leaving substantial room for human review.
FrontierCode’s score combines checks covering hidden-test correctness, regressions, test quality, scope limits and code review. A failure on any blocking check gives an attempt a score of zero. An attempt that clears those checks receives a score between 70% and 100%, depending on its quality-check results. The benchmark also penalizes oversized changes and edits outside relevant files.
In Artificial Analysis’s results, GPT-6.1 Sol scored 49.1% and Claude Fable 5.1 scored 49.0%, placing second and third. The top nine models ranged from 43.2% to 52.2%. Below that group, Grok 4.7 scored 33.8%, MiniMax-M3 23.0%, Inkling 15.8%, Muse Glimmer 13.9% and Nemotron 3 Ultra 10.5%. The source says nine of the benchmark’s 100 tasks were unsolved by every model on every attempt.
Artificial Analysis ran the models using the same open-source agent harness, Stirrup, with up to 250 turns per attempt and three attempts per task. Its review step used GPT-6 Luna (medium) as the judge. Cost per task varied from $0.09 to $12 across the reported models. GPT-6.1 Sol scored 49.1% at $0.81 per task, while Opus scored 52.2% at $10.15.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Passing Tests Is Not Enough
FrontierCode focuses on a practical problem for software teams: a patch can pass tests and still be difficult or unsuitable to merge. By checking scope, test usefulness and review criteria as well as correctness, the benchmark attempts to measure parts of the work a maintainer considers before accepting a pull request.
The leading result also gives a qualified measure of how much oversight remains necessary. Opus cleared all blocking checks in 57.7% of attempts, according to Artificial Analysis. That is a benchmark result, not evidence that the model can safely take over review or merge work in every repository. The tasks, evaluation setup and review judge shape what the score means.
The cost figures show a second trade-off for teams considering these tools. Moving from GPT-6.1 Sol’s 49.1% score to Opus’s 52.2% costs substantially more per task in this evaluation. These figures reflect the benchmark runs and should not be treated as a universal estimate for other workloads or deployment arrangements.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Scores Patches
Cognition developed FrontierCode with open-source maintainers and announced it in June 2026; version 1.1 followed in July. According to the source, more than 20 maintainers created tasks drawn from 36 repositories, including Celery, Budibase, uppy and Mattermost, spending more than 40 hours on each task. The full benchmark contains 150 tasks, while Artificial Analysis ran its 100-task Main set.
The tasks are private to reduce the chance that models have encountered their solutions during training. Version 1.1 also introduced rules against looking up solutions online, and Cognition’s trajectory scanner can assign a zero to an attempt flagged for doing so. These design choices are intended to test agents on repository work without allowing a result to rest only on familiar examples or a patch that passes ordinary checks.
The evaluation uses several kinds of tests. Reverse tests check that an agent’s own tests fail on the original code, while adaptive tests respond to the agent’s implementation for more complex behaviour. A large language model also reviews changes against maintainers’ criteria. Those features make the result different from a conventional test-pass rate, but they do not establish how every human maintainer would judge a patch.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, as quoted in Cognition’s announcement
software developer testing laptops
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Current Results
The results cover 23 models and Artificial Analysis’s 100-task Main set, not every model or all 150 FrontierCode tasks. The source says Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no result as of October 13, 2026. Their absence means the reported ranking cannot show how they would perform.
Scores also depend on the benchmark’s harness, attempt limits and review process, including GPT-6 Luna as the code-review judge. The available material does not establish how closely these scores predict acceptance rates in live projects, where maintainers, codebases and review standards differ. Nor does a score reveal how much human correction an individual patch may still need.
The source describes cost per task for tested configurations, but does not establish that those prices will hold for other usage patterns. It also reports that nine tasks defeated every model on every attempt without identifying those tasks here. What causes the failures, and how results vary across task types, remains unclear from the supplied results.
As an affiliate, we earn on qualifying purchases.
More Models and Tasks to Watch
The immediate next comparison will come if models without reported FrontierCode results are added, particularly the four named as missing in the October 13 account. Their results could change the ranking or expand the comparison between coding benchmarks and broader model evaluations. No release dates for those results are provided.
For now, readers can compare the published scores with the benchmark’s stated setup, rather than treating the leaderboard as a guarantee of merge-ready code. Further reporting would need to establish how well FrontierCode performance corresponds to decisions by maintainers on real pull requests, and whether results remain similar across the full task set and different agent configurations.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates whether AI-generated changes satisfy a set of repository-level checks, including hidden tests, regressions, test quality, scope limits and an LLM code review against maintainers’ criteria. It is designed to assess more than whether visible tests pass.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s results with a 52.2% FrontierCode score. It cleared all blocking checks in 57.7% of attempts.
Does a high score mean an AI can merge pull requests without review?
No. The score is a result on a defined benchmark, not a guarantee about live repositories. The leading model failed to clear every blocking check in 42.3% of attempts, based on the reported 57.7% pass rate.
How much did the tested models cost per task?
Artificial Analysis reported costs ranging from $0.09 for GPT-6 Luna to $12 for Claude Fable 5.1. Among the listed cost-frontier models, GPT-6.1 Sol scored 49.1% at $0.81 per task and Claude Opus 5.5 scored 52.2% at $10.15.
Which models were missing from the results?
As of October 13, 2026, the source listed Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as having no FrontierCode result.
Source: ThorstenMeyerAI.com
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
