angelX
136 repository-repair tasks · 48 JavaScript, 34 Python, 30 Rust, 24 C++ · independently graded task tests
One attempt per task, 600-second agent limit. Medians are per-attempt agent wall time; failures remain in the cohort.
| Date | Version | Model / setting | Solved | Median wall | Input tokens |
|---|---|---|---|---|---|
| 2026-09-30 | 0.1.8 | DeepSeek V4.1 Flash · thinking off · run 1 | 135 / 136 | 9.9 s | 8.7 M |
| 2026-09-30 | 0.1.8 | DeepSeek V4.1 Flash · thinking off · run 2 | 135 / 136 | 9.1 s | 6.9 M |
| 2026-09-23 | 0.1.6 | GLM-5.3-Flash · thinking low | 134 / 136 | 46.1 s | 9.4 M |
| 2026-09-23 | 0.1.6 | Grok 4.7 · thinking low | 136 / 136 | 17.5 s | 6.7 M |
| 2026-09-23 | 0.1.6 | Muse Spark · thinking low | 136 / 136 | 25.5 s | 5.9 M |
Configuration: Prime Intellect Verifiers v0.3.1, task-native tests, temperature 0, fresh environment per attempt. The September 23 Muse cell used its recommended temperature 1.0. These are historical binaries; the current 0.2.3 release has no published full-suite result. Host CPU/GPU and memory were not recorded in the public summaries, so these figures are not hardware throughput claims.
Source: public benchmark tables and methodology · full-cohort timing audit and comparison limits.
The interactive plots below use the original September 21 cells. They do not describe the current release. Agent wall time includes failed attempts; it is not a measure of overall campaign elapsed time.
| Harness | DeepSeek V4.1 Flash thinking off |
GLM-5.3-Flash thinking low |
Total both models |
|---|---|---|---|
| angelX | 133 / 136 | 135 / 136 | 268 / 272 98.5% |
| OpenCode 1.18.31 | 132 / 136 | 133 / 136 | 265 / 272 97.4% |
| oh-my-pi 18.2.4 | 86 / 93 200M-token budget cap | 133 / 136 | — partial DeepSeek cell |
Tokens generated and model calls made per task (cap at 2.5k)
Wall time per attempt in run order (medians dotted)
Cumulative, cached and uncached
Share of input tokens served from provider cache
Graded by task-native test suites on Prime Intellect evaluators (Verifiers v0.3.1) · 600 s wall cap · 2026-09-21. DeepSeek used thinking off; GLM used thinking low. Machine-readable September 21 source.