Abstract

WhatCross-file code completion benchmark, roughly 27,000 examples from real GitHub Python and Java repositories.

WhoLiu, Xu, McAuley (UC San Diego, ICLR 2024).

Launch baselineGPT-4 at 36.4% Edit Similarity on RepoBench-P Python (original paper).

Section I.ix Industry Domain|Reviewed 2026

RepoBench: 27,000 Multi-File Code Completion Tasks, GPT-4 at 36.4%

Single-file benchmarks like HumanEval do not measure what your IDE assistant actually does.

Construction

RepoBench is built from real public GitHub repositories in Python and Java. The dataset was filtered for compilable, well-tested projects. Each example places the model cursor at a specific line in a file, with the rest of that file and the rest of the repo available as candidate context. The task is to emit the next line or block at the cursor.

Three splits: RepoBench-R (retrieval only: pick the right cross-file snippets), RepoBench-C (completion only: assume the right context is given), and RepoBench-P (pipeline: do both end-to-end). The pipeline split is the headline because it matches the real IDE-autocomplete deployment.

Scores and the Launch Baseline

The original Liu et al. paper (ICLR 2024) reported GPT-4 at 36.4% Edit Similarity on the RepoBench-P (pipeline) Python split, with UniXcoder and CodeBERT baselines trailing in the teens. Beyond that published launch baseline we do not reprint a per-model RepoBench score table: there is no maintained official leaderboard for the current frontier, and vendor-reported completion numbers come from harnesses we cannot inspect, so any table we copied would be unverifiable. For methodology and the original numbers, read the paper; start from the homepage task picker to find the benchmark that matches your use case.

III

When to Pick RepoBench

Pick RepoBench when the question is IDE-autocomplete quality at the cursor. Pick HumanEval or MBPP for single-function generation. Pick SWE-bench Verified for end-to-end patch generation against failing tests. Pick LiveCodeBench for problem-solving over fresh competitive problems. The four benchmarks measure four distinct coding-agent capabilities and a strong score on one does not imply a strong score on the others.

SWE-bench Verified →HumanEval, MBPP →LiveCodeBench →

Reader Questions

Q.01What does RepoBench test?+

RepoBench tests three subtasks: Retrieval (given the cursor position, retrieve the right cross-file context), Code Completion (predict the next line or block given context), and Pipeline (chain retrieval and completion end-to-end). It draws from real GitHub repositories in Python and Java and explicitly requires cross-file reasoning, unlike HumanEval which is single-file.

Q.02What was the headline launch number?+

Liu et al. (ICLR 2024) reported GPT-4 at 36.4% Edit Similarity score on the RepoBench-P (pipeline) Python split. UniXcoder and CodeBERT baselines trailed in the teens. The benchmark stresses cross-file context, which made it harder than HumanEval at the time and remains harder in 2026 even as models improve.

Q.03How is RepoBench different from SWE-bench?+

SWE-bench Verified tests patch generation for real GitHub issues with a failing test as ground truth. RepoBench tests autocomplete-style code completion at the cursor position, like a Copilot inline suggestion. Both touch multi-file repositories, but the task is fundamentally different: patch generation (SWE-bench) vs context-aware completion (RepoBench).

Q.04Why does retrieval matter so much for RepoBench?+

The completion task often requires referring to a class, function, or import defined in another file. A naive context window can fit only the current file. Strong RepoBench performance requires either long-context models that can fit the whole repo, or a retrieval step that pulls the right cross-file snippets. The original paper showed that retrieval quality bounds completion quality.

Q.05Is RepoBench gameable through training-set overlap?+

Partially. The dataset is built from public GitHub repos, which are also pre-training data for most code models. The authors address this with a contamination-filtered subset that excludes repositories with high overlap to the model's training cutoff. Numbers on the filtered subset are roughly 5 to 10 points lower than the full-set numbers and are the more meaningful comparison.

Sources

[1] Liu et al. (2023): arxiv.org/abs/2306.03091
[2] RepoBench dataset on Hugging Face: huggingface.co/datasets/tianyang/repobench-p
[3] RepoBench repository: github.com/Leolty/repobench