Benchmarks

AI Model Performance Benchmarks

• MMLU/MMLU-Pro: Measures performance on massive multitask language understanding across subjects (math, medicine, law, etc.).
• GPQA (Graduate-Level Physics QA): Evaluates advanced scientific reasoning.
• AIME: Assesses mathematical problem-solving, often used for math Olympiad-level questions.
• SWE-Bench: Quantifies proficiency in real-world software engineering tasks.
• LiveBench: Focuses on robust, contamination-resistant evaluation for coding, reasoning, and multi-step tasks. ^{[nxzyo6] [s3j99t] [8kt4xe] [gj402g]}
• Chatbot Arena (Elo Scores): Dynamically ranks models based on head-to-head user engagement and feedback.
• Other notable benchmarks: SuperGLUE (natural language understanding), LAMBADA (word prediction), HumanEval (coding), and others for multimodal capabilities. ^{[8kt4xe] [i48jo2] [dtfaf1]}

• Since the debut of Chat GPT (based on GPT-3.5), benchmark scores have surged:
  • On benchmarks like MMMU, GPQA, and SWE-Bench, scores rose by 18–67 percentage points within a year. ^[s3j99t]
  • Inference costs for GPT-3.5–level performance dropped more than 280-fold; hardware costs and energy use decreased by 30–40% per year. ^[s3j99t]
  • Model scale (number of parameters and training compute) continues to double every few months, yet the performance gap between top models has shrunk from 12% to just 0.7%, making the frontier highly competitive. ^[s3j99t]
  • Models now outperform humans in some programming and reasoning tasks with time constraints, and open models have nearly rivaled closed models on certain benchmarks. ^[s3j99t]

• Current state-of-the-art models such as DeepSeek R1, Grok 4, GPT-4.5, Gemini 2.5 Pro, and Claude 4 Opus consistently achieve scores above 85%—and in some cases above 90%—on historically challenging tasks, reflecting dramatic advances in reasoning, coding, multilinguality, and robustness even compared to GPT-4. ^{[nxzyo6] [sab6tu] [i48jo2] [0ueat7]}

• Open-source models (e.g., DeepSeek, LLaMA, Qwen) have not only become more efficient (requiring less compute per output), but have closed most of the quality gap with proprietary models. ^{[s3j99t] [sab6tu]}
• Benchmarks themselves are evolving rapidly (e.g., LiveBench, MMLU-Pro), specifically to detect and discourage training data leakage and saturation, further driving genuine innovation. ^{[s3j99t] [gj402g]}

Citations