Here's a benchmark table which compares various AI models on the SWE-bench Verified software engineering benchmark:
| Benchmarks | Qwen3-Coder 480B-A35B-Instruct | Kimi-K2 Instruct | DeepSeek-V3 0324 | Claude Sonnet-4 | GPT-4.1 |
|---|---|---|---|---|---|
| Agentic Coding | |||||
| Terminal-Bench | 37.5 | 30.0 | 2.5 | 35.5 | 25.3 |
| SWE-bench Verified | 69.6 | – | – | 70.4 | – |
| └─ w/ OpenHands, 500 turns | 67.0 | 65.4 | 38.8 | 68.0 | 48.6 |
| └─ w/ OpenHands, 100 turns | – | 65.8 | – | 72.7 | 63.8 |
| └─ w/ Private Scaffolding | 26.3 | 22.3 | 13.0 | 27.7 | – |
| SWE-bench Live | 54.7 | 47.3 | 13.0 | 53.3 | 31.5 |
| SWE-bench Multilingual | 25.8 | 19.3 | 7.5 | 24.8 | – |
| Multi-SWE-bench mini | 27.0 | 20.7 | 7.0 | 25.0 | – |
| Multi-SWE-bench flash | 60.8 | 56.0 | 56.9 | 56.4 | 52.4 |
| Aider-Polyglot | 31.1 | 25.2 | 12.8 | 31.1 | 16.5 |
| Spider2 | – | – | – | – | – |
| Agentic Browser Use | |||||
| WebArena | 49.9 | 47.4 | 40.0 | 51.1 | 44.3 |
| Mind2Web | 55.8 | 42.7 | 36.0 | 47.4 | 49.6 |
| Agentic Tool Use | |||||
| BFCL-v3 | 68.7 | 65.2 | 64.7 | 73.3 | 62.9 |
| TAU-Bench Retail | 77.5 | 70.7 | 59.1 | 80.5 | – |
| TAU-Bench Airline | 60.0 | 53.5 | 40.0 | 60.0 | – |
Benchmark Glossary
- Terminal-Bench – Evaluates model's ability to perform multi-step coding tasks in a terminal.
- SWE-bench Verified – Measures end-to-end software engineering problem-solving with test verification.
- SWE-bench Live – Real-time coding and debugging benchmark for software engineering tasks.
- SWE-bench Multilingual – Tests coding and debugging tasks across multiple programming languages.
- Multi-SWE-bench (mini/flash) – Aggregated SWE-bench tasks in smaller or faster evaluation modes.
- Aider-Polyglot – Assesses multi-language code editing and refactoring using Aider framework.
- Spider2 – Evaluates complex SQL generation and query understanding.
- WebArena – Benchmarks model’s ability to perform real-world web browsing tasks agentically.
- Mind2Web – Measures structured information extraction from web pages.
- BFCL-v3 – Assesses agentic tool use for file and command-line tasks.
- TAU-Bench (Retail/Airline) – Measures task automation and reasoning using simulated business scenarios.