LLM Eval Tools Compared 2026: Braintrust, Langfuse, Promptfoo, DeepEval, Galileo & Alternatives
Almost every existing comparison of these tools is published by one of the vendors themselves. This is a neutral side-by-side comparison with honest “avoid if” notes, head-to-head matchups, and an alternatives guide. No tool is the best for every team - the right choice depends on your stack, team size, and budget.
Tool links on this page go directly to each vendor and are not monetised. The site takes no payment for placement; any affiliate link added later would be labelled and would not influence rankings. Ranking methodology: OSS availability, self-host option, eval feature depth, CI integration quality, pricing transparency, and honest weakness assessment.
At-a-Glance Matrix
| Tool | OSS / Cloud | Free tier | Self-host | Best for | Avoid if |
|---|---|---|---|---|---|
| Braintrust | Cloud + hybrid self-host | Yes (generous) | Enterprise (hybrid) | Developer experience, CI integration, prompt management | You are a small team and want self-host without an Enterprise contract - the hybrid data plane is Enterprise-plan only |
| Langfuse | OSS + Cloud | Generous (cloud) + Free (OSS) | Yes - full | Self-hosting, cost control, teams that want to own their data | You need the best-in-class CI integration or prompt management UX (Braintrust is stronger here) |
| LangSmith | Cloud | Limited | No | Teams already using LangChain or LangGraph | Your stack does not use LangChain - instrumentation is much more work |
| Arize Phoenix | OSS + Arize AI (cloud) | Yes - OSS is fully free | Yes - full (OSS) | Production monitoring, tracing, multimodal evaluation, embedding analysis | You just need offline eval - Langfuse or Braintrust are simpler for that use case |
| Promptfoo | OSS + Enterprise cloud | Yes - OSS (MIT) | Yes - full (OSS) | Config-as-code eval, CI gating, red-teaming and security testing | You want a hosted dashboard with long-term trace storage as the primary interface - Promptfoo is CLI-first |
| Galileo | Cloud / Enterprise (SaaS, VPC, on-prem) | Yes - free tier | Enterprise (VPC / on-prem) | Agent evaluation and observability, guardrails, low-cost eval at scale | You want a fully open-source, self-contained framework - Galileo is a commercial platform, not OSS |
| HoneyHive | Cloud | Trial | No | Agent observability, multi-step trace debugging, agentic workflow evaluation | You are evaluating a simple single-turn LLM call - HoneyHive is overkill |
| Weights & Biases (Weave) | Cloud | Yes (personal use) | No | Teams already using W&B for ML training, want unified ML + LLM observability | You are not using W&B for ML training - the LLM eval features alone do not justify the platform |
| Confident AI / DeepEval | OSS + Cloud | Yes - OSS fully free | Yes - eval runs locally | Engineering teams, CI integration, pytest-style eval, OSS-first | You need a rich dashboard for non-technical stakeholders - the UI is functional but not polished |
| Vendor pricing changes frequently - check each vendor's pricing page for current prices. Links go directly to each vendor and are not monetised. | |||||
Tool Deep Dives
Braintrust
Braintrust was founded in 2023 by Ankur Goyal (previously VP Engineering at SingleStore/MemSQL and founder of Impira, acquired by Figma) and has a strong developer-experience focus. The SDK (Python + TypeScript) is clean, the CI integration is first-class, and the prompt playground is excellent. Dataset versioning and experiment tracking are the strongest in class for offline eval. Self-host is available on the Enterprise plan via a hybrid deployment: the data plane (traces, datasets, results) runs inside your own AWS/GCP/Azure VPC through Terraform or Docker Compose while Braintrust hosts the control-plane UI; Starter and Pro run on shared cloud. It carries SOC 2 Type II and HIPAA compliance. Production tracing exists but is less mature than Arize or Langfuse for online monitoring.
Langfuse
Langfuse is a mature open-source LLM observability and eval platform. Self-host via Docker Compose or Helm chart in under 30 minutes. Strong tracing (OpenTelemetry-compatible), decent eval functionality, and an active community. The cloud version is hosted on EU/US infrastructure with GDPR compliance. The main limitation: the eval UX is less polished than Braintrust, and the CI integration requires more manual setup.
LangSmith
LangSmith is the observability and eval platform built by LangChain, Inc. If you are using LangChain, adding LangSmith is a one-line change that gives you automatic trace capture, latency tracking, and token counts. The eval features work with any model API. Outside the LangChain ecosystem, the instrumentation is more manual and the developer experience is not as smooth as Braintrust or Langfuse. Cloud-only, US/EU options.
Arize Phoenix
Arize Phoenix is the OSS version of Arize AI's production monitoring platform. It installs as a Python package and runs a local UI server. Extremely strong for tracing, embedding drift analysis, and hallucination detection in production. The eval API supports Ragas metrics natively and integrates with LlamaIndex, LangChain, and direct API calls. The Arize AI cloud version adds real-time monitoring, alerting, and team features. Best choice for teams that need both offline eval and production monitoring with a single tool.
Promptfoo
Promptfoo is an open-source (MIT) eval and red-teaming tool that is CLI-first: you declare test cases, providers, and assertions in a YAML config, run them locally or in CI, and view results in a local web UI. It evaluates prompts, models, and RAG pipelines side by side, and its second core capability is adversarial red-teaming - simulating prompt injection, jailbreaks, data-leak and insecure-tool-use attacks across a large library of vulnerability types. Self-host is the default (it runs on your machine); an Enterprise cloud adds a shared dashboard and managed scans. Best when you want eval and security checks to live in version control and block a bad deploy in CI.
Galileo
Galileo is a commercial evaluation and observability platform aimed at teams building agents and RAG systems at enterprise scale. It ships 20+ out-of-the-box evaluators for RAG, agents, safety and security, plus custom-evaluator building. Its signature feature is the Luna evaluation models, which distil expensive LLM-as-judge evaluators into compact purpose-trained models so monitoring stays affordable at scale. It also turns offline evals into production Guardrails that can gate agent actions and tool access, and an insights engine that surfaces agent failure modes. Deployment is SaaS, VPC, or on-prem (not open source).
HoneyHive
HoneyHive is purpose-built for agentic workflows - multi-step chains, tool calls, and agent loops where debugging requires understanding the full trace of actions, not just input and output. Strong agent-specific evaluation features: step-by-step trace analysis, tool-use attribution, and cost-per-step breakdown. Newer than Langfuse or Braintrust; feature set is growing. Best for teams building complex agent pipelines who need to understand why an agent took a particular path.
Weights & Biases (Weave)
Weights & Biases added LLM evaluation and tracing as 'Weave' in 2024. For teams already using W&B for training run tracking, Weave is a natural extension - unified dashboards for model training metrics and LLM eval results. The eval features are solid but not as mature as Langfuse or Braintrust. The main value is the W&B ecosystem integration, not the LLM eval features in isolation.
Confident AI / DeepEval
DeepEval is an open-source Python eval framework with a pytest-style API. Evals are written as Python functions, run locally or in CI, and optionally push results to the Confident AI cloud dashboard. Supports a wide range of metrics: G-Eval, faithfulness, relevancy, hallucination, contextual precision/recall, RAGAS metrics, and custom LLM-as-judge metrics. The strongest choice for engineering teams that want maximum control, no cloud dependency, and integration with existing Python test suites.
Decision Trees
I want OSS, self-hosted, no vendor lock-in
- 1.Langfuse - most feature-complete OSS
- 2.Arize Phoenix - if production monitoring is primary
- 3.DeepEval - if you want pytest-style eval in CI with no cloud
I want the best developer experience and CI integration
- 1.Braintrust - first choice
- 2.DeepEval - if you need it fully local
- 3.LangSmith - if you are on LangChain
I am on LangChain and want zero-friction setup
- 1.LangSmith - auto-traces LangChain with one env var
- 2.Langfuse - good LangChain integration, plus self-host option
Head-to-Head Matchups
Langfuse vs Promptfoo
Different shapes, not direct substitutes. Langfuse is a hosted-or-self-hosted observability platform: a dashboard, long-term trace storage, and datasets you manage over time. Promptfoo is CLI-first and config-as-code: you declare tests in YAML and run them locally or in CI, and it adds red-teaming that Langfuse does not. Pick Langfuse when the dashboard and production tracing are the point; pick Promptfoo when you want eval and security checks living in version control and gating a deploy. Some teams run both - Promptfoo in CI, Langfuse for production traces.
DeepEval vs Braintrust
Both are strong for CI-integrated offline eval; the split is OSS-local versus hosted-hybrid. DeepEval is an open-source pytest-style framework that runs entirely locally with no cloud dependency, ideal when you want maximum control and to keep everything in your own test suite. Braintrust adds a polished cloud dashboard, prompt playground, and best-in-class experiment tracking, with self-host available only on its Enterprise hybrid plan. Choose DeepEval for OSS-first, fully-local eval; choose Braintrust when the dashboard and prompt-management UX justify a paid platform.
Galileo vs Braintrust
Two commercial platforms with different centres of gravity. Braintrust leads on developer experience, prompt management, and offline experiment tracking. Galileo leads on agent evaluation and observability, its Luna distilled-judge models for low-cost eval at scale, and production Guardrails that can gate agent actions. Choose Braintrust if your work is prompt-and-model iteration with strong CI; choose Galileo if you are running agents in production and want cheap continuous eval plus runtime guardrails.
Langfuse vs Humanloop
This matchup no longer has two live options. Humanloop was acqui-hired by Anthropic and sunset its platform on 8 September 2025 (billing stopped 30 July 2025); it is not a tool you can adopt today. For the prompt-management-plus-eval workflow Humanloop offered, Langfuse is the closest self-hostable replacement, and Weights & Biases (Weave) is the migration path Humanloop itself pointed departing customers to.
Alternatives, by Tool
Braintrust alternatives
If Braintrust's cloud-first model, per-seat pricing, or Enterprise-only self-host does not fit: Langfuse is the leading open-source alternative with full self-host; DeepEval gives you a pytest-style framework that runs fully locally in CI; Promptfoo covers config-as-code eval plus red-teaming; and Galileo is the closest commercial alternative if agent observability and guardrails matter more than prompt-playground UX.
Arize Phoenix alternatives
For open-source observability and tracing without Arize Phoenix: Langfuse is the most feature-complete OSS alternative and self-hosts in under 30 minutes. If your primary need is offline eval rather than production tracing, DeepEval (local, pytest-style) or Promptfoo (config-as-code plus red-teaming) are simpler. Galileo is the commercial alternative when you need agent-scale observability with managed guardrails.
Recently sunset: Humanloop
Humanloop, a prompt-management and eval platform, was acqui-hired by Anthropic in August 2025 - the founders and much of the team joined Anthropic, though Anthropic did not acquire Humanloop's assets or IP. The Humanloop platform shut down on 8 September 2025, with billing stopped from 30 July 2025. We have removed it from the live comparison above. If you are migrating off Humanloop, its own guidance pointed customers to Weights & Biases (Weave); Langfuse is the closest self-hostable equivalent for the prompt-plus-eval workflow.
When You Don't Need a Tool
If you have fewer than 100 test examples and no production traffic, a Python script and a CSV is fine. Write your golden dataset in a spreadsheet, run inference in a loop, call the LLM judge API, and append results to a CSV. Setup is typically a day or two, and the per-run API cost is small at this scale.
A dedicated tool earns its cost when you have: (a) regression tracking across model versions that requires a database to manage, (b) team collaboration on prompt versions where multiple people need visibility, or (c) production traces at scale where you need sampling, alerting, and aggregation. For most teams before product-market fit, the CSV approach is the right starting point.