RAGAS: Automated RAG Evaluation

RAGAS is not a benchmark you look up a model on; it is the framework you run over your own RAG pipeline to score it. Its defining move is reference-free evaluation: it can tell you whether an answer is grounded in the retrieved context and whether it answers the question without a human-written gold answer. That makes it the fastest way to put numbers on a pipeline, and its reliance on an LLM judge is exactly why those numbers need reading with care.

RAGAS (Retrieval Augmented Generation Assessment), introduced by Es, James, Espinosa-Anke and Schockaert in 2023, is an open-source Python framework for scoring RAG systems. The paper's central contribution is reference-free evaluation: rather than comparing a pipeline's output to a human-written gold answer, RAGAS uses an LLM to judge properties of the answer directly, such as whether every claim it makes is supported by the retrieved context. That removes the biggest bottleneck in RAG evaluation, which is producing labelled ground truth, and lets a team score a pipeline on hundreds of queries the same day it is built.

The distinction that matters most is that RAGAS is a framework, not a benchmark. A benchmark like BEIR or MTEB fixes a dataset and ranks models against each other; there is no equivalent RAGAS leaderboard. You point RAGAS at your own test set and your own pipeline, and it returns per-metric scores for that pipeline. The two are complementary: BEIR and MTEB help you pick an embedding or retrieval model, and RAGAS tells you how the assembled system behaves once that model is wired into prompts, chunking and generation.

By 2026, RAGAS (package version 0.4.3, released January 2026) has grown beyond RAG. The framework now describes itself as an evaluation framework for RAG and LLM applications, with metrics and dataset tooling extending to agent and tool-use workflows and multimodal inputs. The RAG metrics below remain the reason most teams reach for it, but it is no longer a RAG-only library.

The documentation lists eight RAG metrics. The load-bearing four are faithfulness and response relevancy on the generation side, and context precision and context recall on the retrieval side; the remaining metrics probe entity-level recall and robustness to retrieval noise. A key detail is which metrics need a reference answer: faithfulness and response relevancy are fully reference-free, while context recall and context entities recall are computed against ground truth.

All the LLM-scored metrics report on a 0 to 1 scale where higher is better (noise sensitivity is the exception: lower is better). Alongside these, RAGAS also exposes traditional reference metrics such as answer similarity and BLEU/ROUGE for teams that do have gold answers and want a lexical or embedding-based comparison.

Faithfulness is the clearest to reason about: RAGAS uses an LLM to break the answer into individual claims, checks whether each claim can be inferred from the retrieved context, and scores the fraction supported. Two claims, one supported, gives 0.5. A score near 1.0 means the answer is fully grounded in what was retrieved; scores below roughly 0.70 indicate meaningful hallucination and are the first thing to investigate, because ungrounded output is the most damaging RAG failure.

Response relevancy (renamed from answer relevancy) works in reverse. RAGAS generates a set of artificial questions from the answer, defaulting to three, then computes the mean cosine similarity between the embeddings of those generated questions and the original user question. The premise is that a good answer lets you reconstruct the question it was answering. It deliberately says nothing about factual accuracy, which is why it is always read next to faithfulness: an answer can be perfectly on-topic and still wrong, or perfectly grounded and still evasive.

Context precision scores retrieval ranking as the mean of precision@k over the retrieved chunks, rewarding pipelines that put the relevant chunks above the irrelevant ones rather than merely including them somewhere. It ships in several variants (LLM-judged against a reference answer, LLM-judged against the generated response, a non-LLM similarity version, and an ID-based version), so it can run with or without ground truth. Context recall asks the complementary question, how much of the information needed to answer was actually retrieved, and is computed against a reference answer, so it does need ground truth. Precision and recall trade off: a retriever tuned to fetch more chunks lifts recall but usually costs precision.

RAGAS is more than a bag of metrics. It also generates synthetic test sets: given your document corpus, it can build question-answer pairs of varying difficulty (including multi-hop questions that require combining passages) so you are not blocked on hand-writing an evaluation set before you can measure anything. Those generated sets are a starting point rather than a substitute for real user queries, but they let a team put a first number on a pipeline within an afternoon.

The 2026 framework extends past RAG into agent and tool-use evaluation and multimodal inputs, reflecting how many production systems have moved from single-shot retrieval to multi-step agents that call tools. This is the same direction the wider field has taken, and it puts RAGAS alongside the general-purpose eval platforms covered in our eval tools comparison. For the specific question of scoring an agent's trajectory rather than a single answer, read that comparison and our LLM-as-judge methodology together, because the judge reliability caveats compound as the thing being judged gets longer.

Almost every RAGAS metric that matters is scored by an LLM, and that is both its strength and its weak point. Three caveats travel with any RAGAS number. First, non-determinism: the same pipeline can score slightly differently across runs because the judge model samples, so a small metric movement between two runs can be noise rather than a real regression. Second, judge bias and capability: an LLM judge shares the blind spots of its training, and a judge that is weaker than or the same as the model being evaluated cannot reliably catch errors it would make itself; the judge should be at least as strong as the system under test. Third, cost and latency: because scoring calls an LLM per claim and per generated question, a large evaluation set is a real API bill, which pushes teams toward smaller judge models exactly where accuracy matters most.

None of this makes RAGAS unreliable; it makes it a fast, directional instrument that has to be operated carefully. Hold the judge model and its version fixed across every run you intend to compare, or a metric change will confound a pipeline change with a judge change. Treat RAGAS scores as a continuous signal for iteration and regression detection, and gate releases on human review of a sample rather than on a RAGAS threshold alone. This is the same lesson the rest of the field keeps relearning: automated scores are for speed, and the things benchmarks miss are found by looking at outputs.

Reach for RAGAS when you have a RAG pipeline assembled and need to put numbers on it quickly, especially before you have a hand-labelled evaluation set. Start with faithfulness and response relevancy, which need no ground truth, to catch hallucination and off-topic answers; add context precision and context recall when you are specifically tuning the retriever, accepting that recall needs reference answers. Keep the judge model strong and fixed, use RAGAS scores for iteration and continuous regression tracking, and confirm with human review at release. For choosing the embedding or retrieval model that feeds the pipeline in the first place, use BEIR and MTEB; see our RAG benchmark comparison for how the model-selection benchmarks and the end-to-end framework fit together, and our RAG evaluation deep dive for the metric methodology in full.