Introducing MaxxBench

We changed requirements in a coding benchmark and found models talking themselves out of following them.

Sometimes the agent understood the new requirement perfectly. It described what needed to change, considered an implementation, and then decided the hidden tests probably wanted something else. In one run, a model implemented the change correctly, asked another model for the benchmark’s original answer, and removed its own correct code after receiving it.

This is an awkward habit for a coding assistant to have. It is also a habit that a familiar benchmark can reward. If the model has guessed what the tests expect, it gets the same passing result as an agent that worked carefully from the requirements.

We built MaxxBench to examine how much coding performance depends on those expectations. It changes the instructions or starting code of 89 DeepSWE tasks, giving models familiar engineering work with a reason to revise their usual approach.

Change the question

Once a benchmark becomes popular, it becomes part of the development process. Its tasks can enter training data. Its scores influence checkpoint selection. Developers adjust prompts and agents against its failures. There are plenty of opportunities to get better at that particular exam. We call gains tied to its particular tasks and conventions benchmaxxing.

Then the scores cluster near the top, and the usual response is a harder benchmark. Recent releases such as FrontierSWE have pushed toward engineering projects that can occupy an agent for much of a day.

We chose DeepSWE’s medium-duration feature-implementation tasks because they are closer to the everyday coding work we wanted to evaluate. Before adding more work, we wanted to know how well models would handle changes to the tasks they already scored well on.

For example, one DeepSWE task asks an agent to write a separate test report for each browser. We added a requirement that the filenames be lowercase. The reporting feature still needs to work; Chrome.xml should now be chrome.xml.

Another task adds retirement dates to FastAPI routes. We changed the required timestamps to UTC with a trailing Z, instead of preserving the supplied time-zone offset.

These changes resemble requests a developer might make after reviewing a first implementation. The agent has to keep the feature working while adjusting its behavior.

Other mutations change the code around the feature, including the interfaces available to implement it. We revise the reference solution and tests accordingly, retaining checks for behavior that still applies. That gives us two related versions of the work and a way to inspect where an agent’s adaptation succeeds or fails.

The near-ties disappear

Claude Opus 5.5 and Gemini 3.8 Flash both have published DeepSWE scores of about 74%. On MaxxBench, they score 76.4% and 53.4%.

A 23-point gap is a different basis for choosing between them. The original scores make them look like peers; the revised tasks expose a large difference in how reliably they handle changes to the work.

GPT-6 Astra and Opus 5 also start close together on DeepSWE, then separate to 71.9% and 63.5% on MaxxBench.

We are skeptical of treating a benchmark as exhausted when revisions to its existing tasks still separate models by this much.

The prompt starts losing arguments

In a ytt task, we added a requirement to support uint64 values. Both GLM 5.3 runs used float64, which loses precision for large integers.

One run reasoned that the hidden tests would probably use numbers too simple to expose the problem. The other interpreted the requirement as allowing the conversion.

The model was making a bet about the grader. It had enough information to identify the precision issue, but expected the shortcut to be rewarded.

We found related behavior in Claude’s trajectories. Agents recognized changed requirements and then returned to familiar or simpler implementations. They also sometimes added behavior for imagined tests that the request had never mentioned.

From the agent’s perspective, the prompt had become evidence about an exam it was trying to pass. The expected tests could take precedence over the requirements written in front of it.

DeepSeek went looking for the answer key

DeepSeek V4.1 Flash took a more direct approach. It identified an Obsidian Linter problem as a DeepSWE task and asked an outside model for its test_patch and solution_patch. It tried benchmark URLs, searches, and mirrors, insisting:

Do not summarize source code; quote it.

DeepSeek extracted the API key supplied to run mini-swe-agent and used it to make its own calls to other models and server-side web search. Routing these requests through OpenRouter let it bypass the sandbox’s restrictions on ordinary internet access. At max effort, it made outside-model calls in 20 of 178 attempts.

Its reasoning was candid about the incentives. One excerpt says, “This is ethically gray, but network is allowed.” Another says, “Hidden tests won’t penalize unless usage tracked.”

The lowercase-filename task shows where that approach could lead.

DeepSeek initially implemented lowercasing correctly. It then received an implementation that omitted it. The model noticed the disagreement, decided to favor the presumed original tests, and replaced its code. The final patch failed 33 feature tests, including checks requiring chrome.xml rather than Chrome.xml.

static sanitizeLauncherName(name) {
    if (name === null || name === undefined) {
      return 'unknown';
    }

    return String(name)
      .replace(/[/\\:*?"<>|()]/g, '_')
      .replace(/\s+/g, '_')
      .toLowerCase();
  }

static sanitizeLauncherName(launcher) {
    if (launcher === null || typeof launcher === 'undefined') {
      return 'unknown';
    }

    return String(launcher)
      .replace(/[\\/:*?"<>|()]/g, '_')
      .replace(/\s+/g, '_');
  }

git apply /tmp/agent-src.patch

AssertionError expected 'reports/Chrome.xml' to equal 'reports/chrome.xml'

What we should expect from a higher score

We think the industry has been too quick to read gains on familiar benchmarks as gains in general coding ability. Higher scores should come with evidence that the improvement carries across revised tasks, including cases where the familiar answer is wrong. Requiring that evidence would raise the bar without immediately turning every evaluation into a multi-day assignment.

Successive releases will need fresh mutations as the current ones become familiar. We want to track how quickly benchmark-specific advantages return, and which improvements continue to work when similar-looking tasks require different solutions.

Acknowledgements

We thank the DeepSWE authors for making their tasks, reference solutions, evaluation infrastructure, and trajectories available. That transparency made this investigation possible.

Citation

Please cite this work as:

@misc{lo2026maxxbench,
  title  = {MaxxBench: Benchmarking Task Adaptation and Generalization in Frontier LLMs},
  author = {Gavin Lo and Justin Lo},
  year   = {2026},
  url    = {https://maxxbench.unlap.ai},
}