HumanEval

HumanEval was created by Mark Chen and colleagues at OpenAI (2021) as a benchmark for code generation. The dataset contains 164 Python programming problems, each consisting of a function signature and docstring. The model must complete the function body. Each problem has an average of 7.7 unit tests for evaluation.

The primary metric is pass@1: the fraction of problems where the model's single generated solution passes all unit tests. The 2021 baseline was approximately 29% (Codex, the first dedicated code model). GPT-4 reached 67% in 2023. By 2024, frontier models were at 90%+ and the benchmark had effectively saturated; later generations cluster in the high 90s, where a one or two point difference is a handful of problems out of 164 rather than a capability signal.

At 96%+ pass@1, HumanEval has no discriminating power between frontier models. A 97% score and a 98% score differ by fewer than 2 problems out of 164. The problems are also simple by 2026 standards: they are single-function implementations of well-defined algorithms, mostly solvable with standard library functions. A capable model cannot be meaningfully ranked on this benchmark.

The table below traces the launch-era saturation curve using published, verifiable figures, and deliberately stops at the 2024 frontier. Quoting current-generation models on a saturated benchmark adds no information, and a cross-vendor table of current scores could not be sourced to a leaderboard we trust, so we do not print one.

MBPP

MBPP (Mostly Basic Python Problems) was created by Jacob Austin and colleagues at Google Research (2021). The benchmark contains 974 Python programming problems ranging from simple mathematical calculations to basic data structure operations. Problems are intended to be solvable by beginning programmers.

MBPP has the same saturation problem as HumanEval. Frontier models score 97%+ pass@1. The benchmark was valuable in 2021-2022 for demonstrating that language models could handle basic programming tasks. By 2026 it is only useful for testing smaller models (7B-30B range) where there is still meaningful separation, or for quick regression testing.

LiveCodeBench - The Anti-Contamination Solution

LiveCodeBench was created by Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica (multiple institutions), published in 2024. Its key innovation: it draws problems from LeetCode and AtCoder that were released after a specified date. When evaluating a model, only problems released after the model's training cutoff are used.

This is structural contamination prevention. A model trained with a cutoff of January 2025 is evaluated only on problems published after January 2025. By construction, the model cannot have seen the test problems during training. HumanEval's problems are from 2021 and appear in web crawls; LiveCodeBench's problems are continuously refreshed.

Because LiveCodeBench refreshes against post-cutoff problems, it still has real headroom at the frontier where HumanEval has none, and a meaningful gap separates frontier models from mid-tier ones. We do not reprint a per-model LiveCodeBench score table here: the cells cannot be sourced to a leaderboard we trust for the 2026 frontier. See the LiveCodeBench deep dive for the methodology and the official leaderboard. LiveCodeBench is the recommended benchmark for code generation comparisons in 2026.

Where to Look Instead

Frequently Asked Questions