byunLAP

Nice DeepSWE score.But is it really true?

MaxxBench tests how well coding performance survives small changes to DeepSWE tasks.

Browse dataWhy MaxxBench

Leaderboard

Models 13
MaxxBench Score Benchmaxxing
ModelScore composition
76.4%-2.2 pp
Cost: $16.18Tokens: 249kSteps: 134
71.9%2.2 pp
Cost: $4.38Tokens: 33kSteps: 30
69.1%3.6 pp
Cost: $5.95Tokens: 63kSteps: 62
63.5%6.4 pp
Cost: $14.79Tokens: 88kSteps: 72
63.5%10.2 pp
Cost: $13.80Tokens: 137kSteps: 102
56.7%12.2 pp
Cost: $5.56Tokens: 109kSteps: 141
55.6%18.6 pp
Cost: $0.62Tokens: 205kSteps: 193
55.1%8.3 pp
Cost: $0.73Tokens: 123kSteps: 147
53.4%20.5 pp
Cost: $2.53Tokens: 152kSteps: 167
53.4%22.0 pp
Cost: $2.99Tokens: 94kSteps: 108
46.1%21.1 pp
Cost: $0.53Tokens: 77kSteps: 105
38.8%16.1 pp
Cost: $3.03Tokens: 105kSteps: 107
28.1%25.2 pp
Cost: $0.58Tokens: 115kSteps: 160
Tasks solved by costEach curve averages successful attempts within each task at the horizontal-axis threshold, then weights all evaluated tasks equally. Failures remain in the denominator. Curves finish at each model's MaxxBench score. Focus a curve and use Left, Right, Home, or End to inspect its successful attempts.Tasks solved0%10%20%30%40%50%60%70%80%$0.1$0.2$0.5$1$2$5$10$20$50$100↖ BetterCost / task (log scale) Tasks solved by costEach curve averages successful attempts within each task at the horizontal-axis threshold, then weights all evaluated tasks equally. Failures remain in the denominator. Curves finish at each model's MaxxBench score. Focus a curve and use Left, Right, Home, or End to inspect its successful attempts.Tasks solved0%10%20%30%40%50%60%70%80%$0.2$1$5$20$100↖ BetterCost / task (log scale)
Efficiency metric

Good at coding, or good at DeepSWE?

DeepSWE uses original engineering tasks to test how well models can build software. But once those tasks and their solutions are public, they can become tuning material, and developers can optimize against the same problems again and again. A rising score should mean a model is better prepared for your next project—even when the requirements look a little different.

MaxxBench changes the instructions or starting code of 89 DeepSWE tasks while keeping the main engineering problem. That connection lets us examine how models handle the change: whether they revise their approach, overlook a new requirement, or fall back on a solution that no longer fits. The feature still needs to work. Can the model make the adjustment?

Example mutations

Naming browser reports

Testem runs tests in several browsers. This task asks it to write a separate report for each browser, replacing spaces and special characters in the browser’s name with underscores. Our version also requires lowercase filenames.

DeepSWE
reports/ Chrome__Headless_.xml
MaxxBench
reports/ chrome__headless_.xml
Explore task

More tasks

See all tasks

Our research

All research

Introducing MaxxBench

How coding models respond when familiar engineering tasks ask for something different.