SWE Atlas - Codebase QnA Extended

Name: SWE Atlas - Codebase QnA Extended
Creator: Mercor

Codebase comprehension, testing, and refactoring.

Get in touch Read research

50Mercor tasks

124Public tasks

40.0%Highest score

The SWE Atlas - Codebase QnA Extended leaderboard

Benchmarks coding agents beyond issue resolution across Codebase Q&A (124 tasks), test writing (90), and refactoring (70), combining programmatic validation with rubric-based software-quality scoring (maintainability, abstractions, hygiene). Uses under-specified, agentic task formulations.

Paper Data Code Sample task

Model

Score

Opus 5Max

Score on the original open-source benchmark: 50.3% / Score on this Mercor-extended benchmark: 40.0%

Sonnet 5Max

Score on the original open-source benchmark: 39.5% / Score on this Mercor-extended benchmark: 32.7%

Kimi K3Max

Score on the original open-source benchmark: 52.7% / Score on this Mercor-extended benchmark: 24.0%

Opus 4.8Max

Score on the original open-source benchmark: 44.4% / Score on this Mercor-extended benchmark: 24.0%

GPT-5.6 LunaMax

Score on the original open-source benchmark: 46.0% / Score on this Mercor-extended benchmark: 19.3%

10%

20%

30%

40%

50%

60%

70%

SWE Atlas - Codebase QnA Extended

The SWE Atlas - Codebase QnA Extended leaderboard

The latest on frontier AI performance, straight to your inbox.