Mistral Codestral has been tested across a wide range of benchmarks, demonstrating state-of-the-art performance in multiple domains. Here are the results: (for more details read here)
Base Benchmarks
| Model | Context Length | HumanEval | MBPP | CruxEval-O | RepoBench | Spider | HumanEvalFIM Avg. | HumanEval Avg. |
|---|---|---|---|---|---|---|---|---|
| Codestral 22B | 32k | 81.10% | 78.20% | 51.30% | 34.00% | 63.50% | 91.60% | 61.50% |
| CodeLlama 70B | 4k | 67.10% | 70.80% | 47.30% | 11.40% | 37.00% | - | 51.90% |
| DeepSeek Coder 33B | 16k | 77.40% | 80.20% | 49.50% | 28.40% | 60.00% | 78.20% | 57.60% |
| Llama 3 70B | 8k | 76.20% | 76.70% | 26.00% | 18.40% | 67.10% | - | 61.20% |
Language-Specific Benchmarks
| Model | HumanEval Python | HumanEval C++ | HumanEval Bash | HumanEval Java | HumanEval PHP | HumanEval TypeScript | HumanEval C# | HumanEval Average |
|---|---|---|---|---|---|---|---|---|
| Codestral 22B | 81% | 65.20% | 42.40% | 63.30% | 68.90% | 68.60% | 43.70% | 61.50% |
| CodeLlama 70B | 67% | 54.00% | 28.50% | 57.60% | 62.10% | 60.40% | 36.10% | 51.90% |
| DeepSeek Coder 33B | 77.40% | 67.10% | 39.90% | 60.10% | 54.70% | 62.90% | 43.70% | 57.60% |
| Llama 3 70B | 76.20% | 67.70% | 39.20% | 60.80% | 66.50% | 74.20% | 46.20% | 61.20% |
Fill-in-the-Middle (FIM) Benchmarks
| Model | HumanEvalFIM Python | HumanEvalFIM JavaScript | HumanEvalFIM Java | HumanEvalFIM Average |
|---|---|---|---|---|
| Codestral 22B | 89.40% | 95.10% | 90.30% | 91.60% |
| CodeLlama 70B | - | - | - | - |
| DeepSeek Coder 33B | 65.40% | 82.50% | 86.60% | 78.20% |
| Llama 3 70B | - | - | - | - |
Benchmarks Glossary
- HumanEval: Assesses general code generation performance.
- MBPP: Evaluates Python programming tasks.
- CruxEval-O: Measures output prediction accuracy for Python.
- RepoBench: Tests long-range repository-level code completion.
- Spider: SQL benchmark evaluation.
- HumanEvalFIM: Evaluates fill-in-the-middle performance.
- HumanEval Python: Performance on Python-specific tasks.
- HumanEval C++: Performance on C++ programming tasks.
- HumanEval Bash: Performance on Bash scripting tasks.
- HumanEval Java: Performance on Java-specific tasks.
- HumanEval PHP: Performance on PHP programming tasks.
- HumanEval TypeScript: Performance on TypeScript programming tasks.
- HumanEval C#: Performance on C# programming tasks.
- HumanEval Average: Average performance across multiple languages.