Abstract

What162 legal reasoning tasks tagged against the IRAC framework, contributed by practising lawyers and law-school researchers.

WhoGuha, Nyarko, Ho, Re, Chilton, et al. (Stanford CRFM, NeurIPS 2023 Datasets and Benchmarks track).

Launch baselineGPT-4 at 77.0 macro-F1 across the 162-task suite (original paper).

Projecthazyresearch.stanford.edu/legalbench

Section I.vii Industry Domain|Reviewed 2026

LegalBench: 162 Legal Reasoning Tasks, GPT-4 Macro-F1 at 77.0

The benchmark that maps legal reasoning to LLM evaluation under the IRAC frame.

Construction

LegalBench was assembled through a 2022 to 2023 call for task contributions from practising lawyers, law professors, and researchers. The result is 162 tasks organised under six categories: Issue-Spotting, Rule-Recall, Rule-Application, Rule-Conclusion, Interpretation, and Rhetorical Analysis. Each category maps to a step in IRAC analysis.

Task formats vary by reasoning type. Some tasks are binary classification (is this clause an arbitration clause?). Others are multi-class (which area of law does this case touch?). A handful are free-text generation evaluated by exact match or by judges. The headline metric is unweighted macro-F1 across all 162 tasks.

Scores and the Launch Baseline

The original Guha et al. paper reported GPT-4 at 77.0 unweighted macro-F1 across the 162-task suite. Beyond that published launch baseline we do not reprint a per-model LegalBench score table: there is no maintained official leaderboard tracking the current frontier, and vendor-reported legal-reasoning numbers come from harnesses we cannot inspect, so any table we copied would be unverifiable. For methodology and the original results, read the paper; start from the homepage task picker to find the benchmark that matches your use case.

III

Where Models Underperform

The original paper found Rule-Application (mapping a given set of facts to a stated legal rule) was the hardest category, while Issue-Spotting and Rule-Recall were easier, and Conclusion lagged further. This category ordering is one of the more durable findings in the paper and matches the intuition that pattern recognition is easier than legal judgement; expect the same relative shape to hold even as absolute scores rise.

MedQA for medical reasoning →RepoBench for multi-file code →Contamination in domain benchmarks →

Reader Questions

Q.01What is LegalBench?+

LegalBench is a collaboratively-built benchmark for evaluating LLM legal reasoning. It collects 162 tasks contributed by legal practitioners and researchers, organised under the IRAC (Issue, Rule, Application, Conclusion) framework that mirrors how lawyers actually reason. Tasks span case classification, doctrine application, statute interpretation, and contract clause detection.

Q.02What was the headline number?+

The original Guha et al. paper (Stanford CRFM, 2023) reported GPT-4 at 77.0 macro-F1 across the 162-task suite, with mid-tier models of the era trailing further back. There is no maintained official leaderboard tracking the current frontier on LegalBench, and task-level variance across the 162 tasks is significant, so we report only the published launch baseline rather than a current per-model figure we cannot source.

Q.03Why use IRAC as the structuring framework?+

IRAC (Issue, Rule, Application, Conclusion) is the standard legal-analysis pattern taught in U.S. law schools and recognised in most common-law jurisdictions. By tagging tasks against the IRAC stage they require, the benchmark surfaces where model failure clusters. The original paper found models do well on Issue spotting and Rule recall but worse on Application (mapping facts to a rule).

Q.04Is LegalBench US-centric?+

Mostly. The bulk of tasks derive from U.S. statutes, U.S. case law, and U.S. contract conventions. Some tasks (CUAD-derived contract analysis) are language-of-the-document rather than jurisdiction-of-law, so they generalise better. For UK, EU, or commonwealth legal reasoning, LegalBench results overestimate capability and should be supplemented with jurisdiction-specific evaluation.

Q.05Should I rely on LegalBench scores for production legal tools?+

No, not alone. LegalBench measures benchmark-task performance; production legal work requires confidentiality controls, citation accuracy, hallucination detection, and human-in-the-loop review. A high LegalBench score is a necessary but not sufficient signal. The Stanford CRFM authors are explicit about this in the paper limitations section.

Sources

[1] Guha et al. (2023): arxiv.org/abs/2308.11462
[2] LegalBench project: hazyresearch.stanford.edu/legalbench
[3] LegalBench repository: github.com/HazyResearch/legalbench