BrowseComp: The Open-Web Search Benchmark

Questions that are trivial to check and brutal to find. BrowseComp puts a number on deep-research capability: can an agent navigate the live internet persistently enough to pin down a single hard-to-locate fact.

BrowseComp (Browsing Competition), released by Wei et al. at OpenAI in April 2025, is built on a single design inversion. Most benchmarks ask questions that are hard to answer once you have the facts. BrowseComp asks questions that are trivial to answer once you have the facts, but where obtaining the facts requires navigating the open web across many pages. Each of the 1,266 questions specifies a target through a conjunction of constraints obscure enough that exactly one entity satisfies them, and the answer is a short string: a name, a date, a title, a number.

This makes BrowseComp a clean test of one narrow but valuable capability: persistent, creative information-seeking on the live internet. It deliberately does not test reasoning depth (the answers require little computation), UI manipulation (there is no sandbox site to click through), or long-form synthesis. It tests whether an agent can keep searching, reformulate queries, cross-reference partial matches, and refuse to give up. That capability is what the "deep research" product category is built on, which is why BrowseComp became the reference number for that category almost immediately.

The difficulty is calibrated against people, not models. The questions were written so that a different person could not find the answer within ten minutes. When the authors gave the finished benchmark to human trainers with a two-hour limit, the trainers gave up on 70.8% of the questions (888 of the 1,255 they attempted). On the 29.2% they did solve, their answer matched the reference 86.4% of the time, which confirms that the questions have a single findable answer and are not merely ambiguous. A benchmark that strong humans fail most of the time, but where the answer is objectively checkable, is an unusually clean measuring stick.

The two-hour human give-up rate is the number to keep in mind when reading model scores. It sets the scale: this is not a benchmark where 90% is the eventual ceiling and everything above it is noise, the way HumanEval saturated. There is genuine, human-hard headroom, and the spread between models at launch was correspondingly wide.

The April 2025 paper reported the following accuracies. These are the fixed, published launch figures; for current-generation results, read the numbers each vendor and the open-source community report in their own harnesses, because a BrowseComp score is inseparable from the tooling that produced it.

The headline is the gap between rows two and five. Giving GPT-4o a browsing tool lifted it from 0.6% to 1.9%, a rounding error against the task. Deep Research, a model trained end-to-end for agentic browsing and run in OpenAI's own harness, reached 51.5%: an 85x jump over base GPT-4o. The lesson the authors draw is that browsing is a trained skill. A capable base model plus a search API is not a browsing agent; the persistence, query reformulation, and stopping behaviour have to be learned. Note also that o1 (9.9%, no browsing tool at all) beat GPT-4o with browsing, which shows reasoning helps even before tools enter the picture.

Each question ships with a short reference answer. The model is prompted to end with a structured "Exact Answer" line, and an AI grader checks semantic equivalence between the prediction and the reference using the same grading prompt as Humanity's Last Exam. Because the answers are short and unambiguous, this is far more reproducible than the LLM-as-judge scoring used for open-ended generation; it sits closer to the exact-match hygiene of GAIA than to the judge-noise of end-to-end RAG evaluation.

The paper also reports calibration: whether a model's stated confidence tracks its actual accuracy. On BrowseComp, models tend to be over-confident, asserting answers they have not actually verified. This matters for deployment: a deep-research product that returns a wrong answer with high confidence is more dangerous than one that says it could not find the answer. Calibration is a second axis worth reading alongside raw accuracy.

BrowseComp is the open-web-search corner of the agent-benchmark map. It complements rather than competes with the others. WebArena, Visual WebArena, and OSWorld test UI manipulation inside sandboxed environments; GAIA tests everyday assistant tasks against a private answer set; Tau-Bench tests tool-use dialogue. BrowseComp tests none of those directly. It asks only: can the agent find the needle. For a product whose job is to research a question by reading the live web, it is the most on-target public benchmark available.

The limitations follow from the narrow design. BrowseComp says nothing about answer synthesis quality (answers are one line), nothing about UI or form-filling, and nothing about multilingual retrieval (the questions are English). Its answers can go stale as the web changes underneath a fixed reference, and because Deep Research and similar agents are products rather than open harnesses, the top scores are not straightforwardly reproducible. A follow-up benchmark, BrowseComp-Plus, was introduced in 2025 to make deep-research evaluation more transparent by fixing the retrieval corpus; treat it as the fairer cross-agent comparison and BrowseComp as the original open-web reference.

Reach for BrowseComp when the agent's job is open-ended web research: find a fact, a source, a person, a filing, anywhere on the internet, and return a checkable answer. It is the right headline for deep-research assistants and open-web retrieval agents. Pair it with a calibration read so you know whether the agent knows when it has failed. For agents whose work is UI-driven or assistant-shaped rather than search-driven, prefer the browser-agent benchmarks or GAIA instead; a strong BrowseComp score does not imply competence at clicking through a checkout flow, and vice versa.