Data
Pass rate100%No evaluation
Task
Claude Opus 5.5 max
GPT-6 Astra xhigh
GPT-5.6 Sol max
Claude Fable 5 xhigh
Claude Opus 5 max
GLM-5.3 max
DeepSeek V4.1 Flash max
GLM-5.3 Flash max
Gemini 3.8 Flash high
Muse Spark 1.3 xhigh
GPT-5.6 Luna max
Muse Spark 1.2 xhigh
DeepSeek V4 Flash max
Mean
Mean pass rate for Add conditional required attributes to schemas: 100%
Mean pass rate for Add deterministic multi-key sorting to fd: 100%
Mean pass rate for Add entity snapshot and rollback APIs to Koota: 100%
Mean pass rate for Expose accumulated streamed function-call args in SDK surfaces: 100%
Mean pass rate for Add default arguments to Anko function parameters: 96%
Mean pass rate for Add partial structuring with error recovery to cattrs: 96%
Mean pass rate for Add boundary modes to @stencil: 92%
Mean pass rate for Add unified manifest stream output across Helm commands: 92%
Mean pass rate for Add deterministic map conflict detection to Y.Map writes: 88%
Mean pass rate for Add flattened dataclass fields to Mashumaro field options: 88%
Mean pass rate for Add grouped test phases with synchronized barriers: 88%
Mean pass rate for Add ShapeIndex encoding and decoding: 88%
Mean pass rate for Add task graph export with JSON, DOT, and text output: 88%
Mean pass rate for Add trap coredump generation to wasmi: 88%
Mean pass rate for Add worktree merge conflict handling: 88%
Mean pass rate for Add a persistent analysis cache to Vulture: 85%
Mean pass rate for Add drift detection and compliance baselines: 85%
Mean pass rate for Add implicit HEAD and automatic OPTIONS responses to FastAPI routes: 85%
Mean pass rate for Add scoped per-rule ignore markers to Obsidian Linter: 85%
Mean pass rate for Add task snapshots, inspection, and diffing to aiomonitor: 85%
Mean pass rate for Add XML diff, patch, and merge operations to etree: 85%
Mean pass rate for Preserve restored query state in persisted snapshots: 85%
Mean pass rate for Restore RichLog follow-state parity and expand reflow behavior: 85%
Mean pass rate for Fix isolated Go-side calls for Tengo callables and closures: 81%
Mean pass rate for Partition report files by launcher and expand report templates: 81%
Mean pass rate for Add \multicolumn column spans to array-like environments: 77%
Mean pass rate for Add JSONPath query APIs to orderedmap and Starlark modules: 77%
Mean pass rate for Harden module loading, cache introspection, and script flags: 77%
Mean pass rate for Add interprocedural taint checks for Bandit injection sinks: 73%
Mean pass rate for Add HTML5 document format handling to Dasel: 69%
Mean pass rate for Add recursive schema composition to Valibot: 69%
Mean pass rate for Add tube multiplexing to pwntools: 69%
Mean pass rate for Add a deferred mutation buffer to batch entity changes: 65%
Mean pass rate for Add atomic signal selectors to Kea: 65%
Mean pass rate for Add deprecation, sunset, and successor headers to FastAPI routes: 65%
Mean pass rate for Add durability callbacks and wait APIs for sync writes: 65%
Mean pass rate for Validate daemon watch, status, and log lifecycle: 65%
Mean pass rate for Add bidirectional TOML table converters: 62%
Mean pass rate for Add config file parsing to Cliffy commands: 62%
Mean pass rate for Add go:embed directive support for interpreted packages: 62%
Mean pass rate for Add hierarchical evaluation cancellation to Boa: 62%
Mean pass rate for Fix PromQL label sorting across typed and untyped values: 62%
Mean pass rate for Add composite trait aspects to Koota: 58%
Mean pass rate for Add matchEach to ts-pattern: 58%
Mean pass rate for Add destructuring bindings to Tengo: 54%
Mean pass rate for Add input key aliases to name mapping: 54%
Mean pass rate for Add RFC 5545 timezone interoperability to dateutil recurrence parsing: 54%
Mean pass rate for Add scoped state data to state machine callbacks and history: 50%
Mean pass rate for Add session bundle recording and replay to IPython: 50%
Mean pass rate for Add stepped slices for arrays and strings: 50%
Mean pass rate for Complete Kitty keyboard phases and stable key metadata: 50%
Mean pass rate for Implement a deterministic IntersectionObserver in Happy DOM: 50%
Mean pass rate for Add conditional option dependencies to Optique: 46%
Mean pass rate for Add duration encoding to TableVectorizer: 46%
Mean pass rate for Add JSON Schema refs and dependency keywords: 46%
Mean pass rate for Add lazy recursive schemas with DTO and JSON Schema export: 46%
Mean pass rate for Add multiplexed ordered streams over KCP: 46%
Mean pass rate for Add rule evaluation profiling to Rego: 46%
Mean pass rate for Add single-active-consumer priority and cancel tracking to virtual transports: 46%
Mean pass rate for Add value-based query predicates to Koota: 46%
Mean pass rate for Add a checker for broken doc comment links: 42%
Mean pass rate for Add pair-level relation tracking modifiers: 42%
Mean pass rate for Add request coalescing to Runnable: 42%
Mean pass rate for Add transparent encryption to dump uploads: 42%
Mean pass rate for Persist the fitted feature schema across evaluate, predict, serve, and export: 42%
Mean pass rate for Reconstruct template strings in partial evaluation output: 42%
Mean pass rate for Add dead-lettering, TTL, and overflow handling to virtual queues: 38%
Mean pass rate for Add safe import checkpoints and invariant validation: 38%
Mean pass rate for Add streaming JSON iteration to HTTPX responses: 38%
Mean pass rate for Add keyset cursor pagination to $find: 35%
Mean pass rate for Add build-time grammar conflict analysis to participle: 27%
Mean pass rate for Reuse one toolbar across multiple Quill editors: 27%
Mean pass rate for Add bounded-memory spilling to SCC aggregation: 23%
Mean pass rate for Add dependency-aware async initialization to the container: 23%
Mean pass rate for Add grouping-set and window-frame SQL helpers: 23%
Mean pass rate for Add multipart response parsing to HTTPX: 23%
Mean pass rate for Add shorthand expansion and compression to the lexer: 23%
Mean pass rate for Add async autocomplete options and fetch lifecycle handling: 19%
Mean pass rate for Add duration-aware sharding to Vitest: 19%
Mean pass rate for Add link format conversion between wiki and markdown syntax: 19%
Mean pass rate for Add try/catch error recovery to expr: 19%
Mean pass rate for Add consistent hash policy support to TrafficPolicy: 15%
Mean pass rate for Add CSS Grid layout to the Box component: 15%
Mean pass rate for Add error stack serialization to SuperJSON: 15%
Mean pass rate for Add typed variable bindings to Anko: 15%
Mean pass rate for Add incremental cache controls to Bandit: 8%
Mean pass rate for Implement recursive agent delegation through delegate_task tool calls: 8%
Mean pass rate for Add method declarations and interface dispatch to Scriggo: 4%
Mean pass rate for Add bail-on-test-failure handling to Testem: 0%