LLM as a judge is one of the most practical ways to evaluate open-ended AI outputs when exact-match tests are too narrow. This guide shows how to build, test, and trust an LLM as a judge workflow with Langfuse, MT-Bench lessons, bias checks, and production-ready rubrics.

LLM as a Judge: 9 Powerful Practices for Reliable Evaluation

LLM as a judge quick checklist

Use this quick checklist before trusting any LLM as a judge score in production. An LLM as a judge system is strongest when it is narrow, calibrated, evidence-aware, and tested against human-labelled examples.

• LLM as a judge should evaluate one clear property at a time.
• LLM as a judge should receive the user input, candidate answer, context, and rubric.
• LLM as a judge should use labels whose boundaries are explicitly defined.
• LLM as a judge should be calibrated with human-labelled examples before production use.
• LLM as a judge should be tested for position bias in pairwise comparisons.
• LLM as a judge should be tested for verbosity bias with deliberately long weak answers.
• LLM as a judge should report reasoning that helps developers inspect disagreements.
• LLM as a judge should not replace deterministic checks for schemas, numbers, or exact rules.
• LLM as a judge should be versioned like application code.
• LLM as a judge should be rerun after prompt, model, retrieval, or rubric changes.
• LLM as a judge should track failure recall, not only average agreement.
• LLM as a judge should separate retrieval failures from generation failures in RAG systems.
• LLM as a judge should use source evidence when judging groundedness.
• LLM as a judge should mark unstable close-call comparisons instead of forcing a winner.
• LLM as a judge should send important disagreements to human review.

In short, LLM as a judge evaluation is not just a prompt. LLM as a judge evaluation is a repeatable measurement workflow. LLM as a judge evaluation becomes reliable only after you measure the evaluator itself.

Evaluating an LLM application becomes surprisingly difficult once you move beyond questions with one objectively correct answer.

A JSON parser can tell you whether a response is valid JSON.

An exact-match test can tell you whether an expected string appears.

Neither can reliably answer questions such as:

• Did the assistant actually answer the user’s question?
• Is a RAG response grounded in the supplied documents?
• Did a summary preserve the important details?
• Is the response concise without leaving out necessary information?
• Did an agent choose an appropriate tool?
• Is one generated answer genuinely better than another?

That is where LLM-as-a-Judge becomes useful.

Instead of asking humans to inspect every generated response, you give another language model the input, candidate response, relevant context or reference answer, and a carefully defined rubric. The judge then returns a structured assessment.

The important word is carefully.

An LLM judge is not an objective measurement instrument simply because its output contains a number. Poorly designed judges can prefer longer answers, favor whichever candidate appears first, miss subtle factual failures, disagree with humans, or change their score when the prompt format changes.

The useful question is therefore not:

Can an LLM evaluate another LLM?

It can.

The useful question is:

How do you determine whether your particular LLM judge is trustworthy enough for your application?

This guide builds that answer from the original MT-Bench and Chatbot Arena work through modern Langfuse production evaluation, calibration, and recent research into judge bias.

---

LLM as a judge use cases

The table below shows where LLM as a judge evaluation fits best. Each LLM as a judge use case should still be validated with examples from your own product, because LLM as a judge behavior changes across domains, models, prompts, and failure types.

Use case	How to apply it
LLM as a judge for RAG faithfulness	Check whether every factual claim follows from retrieved context.
LLM as a judge for answer relevance	Check whether the response directly answers the user question.
LLM as a judge for completeness	Check whether required details, exceptions, and constraints are included.
LLM as a judge for citation quality	Check whether each citation supports the claim attached to it.
LLM as a judge for tool use	Check whether an agent chose the right tool and arguments.
LLM as a judge for summarization	Check whether a summary preserves the important source facts.
LLM as a judge for tone	Check whether the answer matches the required support or brand tone.
LLM as a judge for prompt testing	Compare prompt versions with pairwise judging and order reversal.
LLM as a judge for regression testing	Rerun known failures after model, prompt, or retrieval changes.
LLM as a judge for production monitoring	Sample real traces and send low-confidence cases to human review.
LLM as a judge for close comparisons	Mark unstable ties instead of forcing a false winner.
LLM as a judge for evaluator QA	Measure agreement, failure recall, and bias sensitivity before rollout.

This is why LLM as a judge workflows work best as engineering systems. LLM as a judge outputs should feed dashboards, regression sets, and review queues, not stand alone as unquestioned truth.

What is LLM-as-a-Judge?

LLM-as-a-Judge is an evaluation technique in which a language model assesses another model’s output according to explicitly defined criteria.

A typical evaluation contains four pieces:

User input
     ↓
Candidate model response
     ↓
Evaluation rubric
     ↓
Judge LLM
     ↓
Score + explanation

You might ask a judge to evaluate:

• correctness;
• relevance;
• completeness;
• faithfulness;
• hallucination;
• tone;
• instruction following;
• safety;
• tool selection;
• citation quality.

Langfuse describes the approach similarly: the judge receives the input, application output, evaluation criteria and, when available, a reference or ground truth, and produces a score plus reasoning.

The result can be numeric:

faithfulness = 0.86

categorical:

verdict = "partially_correct"

or binary:

grounded = true

The evaluation format matters more than it initially appears.

A single vague question such as:

Rate this response from 1 to 10.

often produces a much weaker evaluator than several narrowly defined checks.

We will return to that problem later.

---

Why not just evaluate LLMs with BLEU, ROUGE, or exact match?

Traditional metrics work well when similarity to one expected output is meaningful.

They become less useful when many answers can be correct.

Imagine the question:

Why might semantic chunking improve retrieval for meeting transcripts?

Two responses could use completely different wording while communicating the same correct explanation.

An exact-match evaluator would fail immediately.

A semantic similarity metric is better, but similarity still does not necessarily mean correctness.

An answer can be semantically similar to the reference while introducing an incorrect technical claim.

An LLM judge can instead examine the meaning of the answer against a rubric:

Check whether the response:

1. explains that transcripts often lack reliable structural separators;
2. explains semantic boundary detection;
3. does not claim semantic chunking is always superior;
4. mentions the additional embedding/computation cost.

That converts an ambiguous concept such as “quality” into something much more testable.

---

Three common LLM-as-a-Judge designs

There are three useful patterns to understand before building an evaluator.

1. Pointwise evaluation

The judge sees one response and scores it independently.

Question
   ↓
Answer A
   ↓
Rubric
   ↓
Score: 4/5

Example:

Rate the answer’s faithfulness from 1 to 5 using only the supplied context.

Pointwise evaluation is useful when you need an absolute quality signal for production monitoring.

For example:

faithfulness >= 4 → pass
faithfulness < 4  → inspect

The limitation is calibration.

What exactly separates a 3 from a 4?

Unless your rubric defines that boundary, the number can look more precise than it really is.

---

2. Pairwise evaluation

The judge receives two responses and chooses between them.

┌─ Response A
Question ─────┤
              └─ Response B
                    ↓
                  Judge
                    ↓
          A better / B better / tie

Pairwise judging is useful for:

• comparing prompt versions;
• comparing two models;
• testing a new RAG configuration;
• comparing agent policies.

It can sometimes be easier for a judge to distinguish two outputs than to assign an absolute number.

But pairwise evaluation creates another problem: position bias.

If response A and response B switch places, would the judge still select the same answer?

If not, your evaluator is measuring presentation order in addition to answer quality.

---

3. Reference-based evaluation

Here the judge receives an expected answer or source evidence.

Question
      ↓
Candidate answer
      ↓
Reference / source context
      ↓
Judge

This is especially useful for RAG.

For example:

Question:
When can a customer request a refund?

Retrieved policy:
Customers may request refunds within 30 days.
Digital products are excluded once downloaded.

Candidate answer:
All purchases can be refunded within 30 days.

A generic relevance evaluator could consider that answer good.

A reference-based faithfulness judge should detect that it incorrectly ignores the digital-product exception.

Whenever reliable ground truth exists, giving it to the judge usually makes the evaluation task better defined.

---

Why MT-Bench and Chatbot Arena made LLM-as-a-Judge important

One of the foundational papers in this area is “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, published by researchers associated with LMSYS in 2023.

The researchers were dealing with a fundamental evaluation problem: open-ended assistants produce responses that traditional benchmarks cannot fully capture.

They introduced two complementary approaches.

MT-Bench

MT-Bench uses multi-turn questions designed to test conversational models across different capabilities.

An LLM judge can score model responses to those questions.

Chatbot Arena

Chatbot Arena collects human preferences by showing users competing responses and asking which one they prefer.

The paper then compared strong LLM judges with those human preferences.

The authors reported that strong judges such as GPT-4 achieved more than 80% agreement with human preferences in their experimental setting, while also identifying important limitations including position, verbosity and self-enhancement biases.

That finding helped establish LLM-as-a-Judge as a practical evaluation technique.

But there is a dangerous way to interpret it:

GPT-4 agreed with humans in a benchmark, therefore an LLM judge is automatically reliable.

That is not what the experiment proves.

It shows that strong LLM judges can approximate human preferences under particular evaluation conditions.

Your production rubric, model, domain and data distribution may be completely different.

That brings us to the most important part of an LLM judge system.

---

You need to evaluate the evaluator

Suppose you create this prompt:

Evaluate the quality of this response from 1 to 5.

You run it across 10,000 production conversations.

Your dashboard says:

Average quality = 4.34

It looks scientific.

But you still do not know whether 4.34 represents anything useful.

Before trusting a judge, create a set of examples that humans have already evaluated.

For example:

200 representative responses

             Human label
                  ↓
          ┌───────┴───────┐
          │               │
       Judge LLM       Comparison
          │               │
          └───────┬───────┘
                  ↓
          Agreement analysis

Now the question becomes measurable:

When the human evaluator says a response fails the criterion, does the judge usually recognize that failure?

This is judge calibration.

Langfuse’s current guidance similarly recommends calibrating an LLM judge against examples with known human labels rather than assuming the evaluator behaves as intended.

---

A better LLM-as-a-Judge rubric

Consider this weak evaluator:

Score the answer's quality from 1 to 5.

What is “quality”?

The judge must invent its own definition.

Instead, decompose the evaluation.

Suppose we are evaluating a RAG answer.

Criterion 1: Groundedness

Does every factual claim in the response follow from the supplied context?

Criterion 2: Relevance

Does the response directly address the user's question?

Criterion 3: Completeness

Does the response include every important condition required to answer the question?

Criterion 4: Citation correctness

Do the cited sources actually support the claims associated with them?

Now a response could produce:

{
  "grounded": true,
  "relevant": true,
  "complete": false,
  "citations_correct": true
}

That result is much easier to debug than:

quality = 3.7

Recent research provides an interesting argument for this decomposition. CheckEval, presented at EMNLP 2025, evaluated checklist-style binary questions instead of relying mainly on broad Likert-scale ratings. The researchers reported substantially improved agreement across evaluator models and reduced score variance in their experiments.

That suggests a practical rule:

When possible, turn subjective quality judgments into a collection of narrow, observable checks.

---

A production-ready LLM-as-a-Judge prompt

Here is a much stronger starting template.

SYSTEM

You are evaluating an AI assistant response.

Evaluate only the criterion defined below.
Do not reward writing style, response length, formatting,
or confidence unless they are explicitly part of the criterion.

Use only the supplied context as evidence.

CRITERION

Faithfulness:
Every factual statement in the assistant response must be
supported by the supplied context.

LABELS

PASS:
All factual claims are supported by the context.

PARTIAL:
The central answer is supported, but at least one secondary
claim is unsupported or overstated.

FAIL:
A central claim contradicts the context or cannot be supported.

INPUT

User question:
{{input}}

Context:
{{context}}

Assistant response:
{{output}}

Return JSON only:

{
  "label": "PASS | PARTIAL | FAIL",
  "unsupported_claims": [],
  "reason": ""
}

Notice several deliberate decisions.

We do not ask:

Is this a good answer?

We ask one specific question.

We define the score boundaries.

We explicitly instruct the evaluator not to reward irrelevant surface characteristics.

And we request evidence explaining the failure.

That final field is important when debugging the judge itself.

---

The five LLM-as-a-Judge failure modes I would test first

1. Position bias

Consider pairwise evaluation:

Candidate A
Candidate B

If the judge chooses A, reverse them:

Candidate B
Candidate A

If the preferred answer changes frequently, you have a position problem.

The original MT-Bench work identified position bias, and later research has confirmed that it remains a meaningful issue. A 2025 study evaluated 15 judge models across MTBench and DevBench in more than 150,000 evaluation instances and found that position bias varied substantially across judges and tasks rather than being explainable simply as random noise.

A simple mitigation for important pairwise tests is:

Run 1: A vs B
Run 2: B vs A

Only declare a winner when the result is position-consistent.

Otherwise label it:

unstable

That is more informative than forcing a decision.

---

2. Verbosity bias

Imagine two correct answers.

Answer A

Paris.

Answer B

Based on the relevant historical and geographical context, after carefully considering the available information, it is reasonable to conclude that the capital city of France is Paris.

If the question is simply:

What is the capital of France?

Answer B contains no extra useful information.

Yet an evaluator can mistake additional detail, polished writing or length for superior quality.

Research has repeatedly investigated this tendency; the original MT-Bench paper discussed verbosity bias, and later work has examined evaluators favoring superficial qualities such as verbosity and fluency even when those qualities do not correspond to better instruction following.

Test your evaluator with adversarial pairs where the longer answer is deliberately worse.

---

3. Self-preference or family bias

Suppose Model X generates an answer and a closely related model judges it.

You should not assume independence.

Your judge and generator may share:

• model-family tendencies;
• training preferences;
• style preferences;
• blind spots.

This does not mean same-family judging is automatically invalid.

It means you should measure it.

A useful experiment is:

Candidate outputs:
Model A
Model B
Model C

Judges:
Judge A
Judge B
Judge C

Then compare whether each judge disproportionately favors responses from its own family or style.

---

4. Judge leniency

A judge can become too forgiving.

Suppose humans classify:

30 PASS
40 PARTIAL
30 FAIL

but the judge returns:

62 PASS
32 PARTIAL
6 FAIL

Even apparently high overall agreement could hide a serious operational weakness: the judge almost never catches failures.

A 2025 study examining thirteen LLM judges found that even capable judges could differ materially from human scores and reported vulnerabilities including sensitivity to prompt complexity and leniency.

For production evaluation, therefore, don’t report only:

agreement = 84%

Also calculate metrics for the failure class you actually care about.

---

5. Close-call instability

LLM judges generally have an easier task when one response is obviously much better than another.

Real model development frequently involves something harder:

Prompt v17
vs.
Prompt v18

where the difference is tiny.

Recent research revisiting LLM evaluator meta-evaluation found that agreement with human judgment can become much weaker when the systems being compared are close in capability.

This matters because those close decisions are often precisely the decisions engineering teams need evaluators to make.

If two systems are nearly tied, consider escalating a sample to human evaluation rather than forcing an automated conclusion.

---

Langfuse LLM as a Judge: how the workflow works in 2026

Langfuse provides a practical way to implement this process without building the complete evaluation infrastructure yourself.

At a high level:

Application
    ↓
Langfuse observations
    ↓
Evaluation rule
    ↓
LLM-as-a-Judge evaluator
    ↓
Score + reasoning
    ↓
Analytics / investigation

The current Langfuse workflow separates two ideas:

Evaluator: how an item should be scored.

Rule: which incoming observations should receive that evaluator.

That distinction is valuable.

You might have one faithfulness evaluator but apply it only to:

environment = production
AND
observation.name = final_response
AND
RAG_enabled = true

Langfuse allows numeric, categorical and boolean evaluation outputs and lets evaluators be tested against real observations before being attached to production rules.

---

Step 1: collect representative examples

Do not begin with the evaluator prompt.

Begin with your failures.

For a customer-support assistant, your dataset might contain:

20 correct responses
10 hallucinations
10 incomplete responses
10 wrong policy interpretations
10 unnecessary refusals
10 edge cases

If all your calibration examples are obvious successes, your evaluator can achieve impressive accuracy without learning to identify the failures you care about.

---

Step 2: manually label them

For each example, store:

input
context
output
human_label
human_reason

For example:

{
  "human_label": "FAIL",
  "human_reason":
    "The answer says downloaded digital products are refundable,
     but the supplied policy explicitly excludes them."
}

Human reasoning is valuable because disagreements can reveal ambiguity in the rubric rather than failure by the judge.

---

Step 3: build one evaluator per important criterion

Avoid creating:

overall_quality

as your only score.

For a production RAG system, I would rather track:

faithfulness
answer_relevance
completeness
citation_correctness

plus deterministic checks such as:

valid_json
required_fields
latency
token_count
tool_call_schema

Langfuse itself distinguishes model-based evaluation from code evaluators: deterministic checks such as schema validation or exact matching are better handled with code, while semantic assessments such as relevance or tone are appropriate for an LLM judge.

Using an expensive language model to determine whether a JSON object has a required field is unnecessary and less reliable than ordinary code.

---

Step 4: test the evaluator on your labelled dataset

Now create a confusion matrix.

Suppose the labels are PASS and FAIL:

	Human PASS	Human FAIL
Judge PASS	73	9
Judge FAIL	7	31

Do not stop at overall agreement.

Calculate:

Failure recall

Of the responses humans labelled FAIL, how many did the judge catch?

Using the example above:

31 / (31 + 9) = 77.5%

If hallucinations are expensive, this number may matter much more than average agreement.

False-alarm rate

How often does the judge reject something humans consider acceptable?

Class distribution

Compare the human and judge PASS/FAIL proportions.

Disagreement examples

Read every important disagreement manually.

That last step is where the evaluator usually improves fastest.

---

A small experiment worth publishing with this article

This is where you can make your GenAITrail article more useful than another generic explanation.

Run your own miniature judge benchmark.

Create 30 response pairs:

10 obvious quality differences
10 subtle differences
5 verbosity traps
5 position-bias traps

Human-label them before sending them to the judge.

Then test:

Judge configuration A:
simple "choose the better answer" prompt

Judge configuration B:
explicit rubric

Judge configuration C:
explicit rubric + reversed candidate order

Report:

Configuration	Human agreement	Position consistency	Failure recall
Simple prompt	your result	your result	your result
Rubric	your result	your result	your result
Rubric + swap test	your result	your result	your result

Do not invent these numbers.

Run the experiment and publish the actual output.

That table, your dataset methodology, screenshots and observations become information that competing articles cannot simply reproduce from the same documentation.

Google specifically recommends original information, research and analysis rather than commodity content created primarily for search traffic.

---

LLM-as-a-Judge best practices

After building and testing evaluators, these are the rules I would carry into production.

1. Evaluate one clearly defined property at a time

Avoid:

How good is this response?

Prefer:

Does the response contain any factual claim that is unsupported
by the supplied context?

The narrower question is easier to label, debug and calibrate.

---

2. Define what every score means

Bad:

Score from 1-5.

Better:

5 = fully satisfies every requirement
4 = correct but one minor omission
3 = partially correct; important information missing
2 = major error despite some useful content
1 = fundamentally incorrect

Even better, when possible, replace fuzzy scales with explicit categorical or binary checks.

---

3. Give the judge the evidence it needs

Do not ask for factuality against information the judge cannot reliably know.

For RAG evaluation, pass the retrieved evidence.

For policy evaluation, pass the policy.

For extraction, provide the source document.

For exact business rules, prefer deterministic code.

---

4. Separate correctness from presentation

If you’re measuring factual accuracy, state explicitly:

Do not reward verbosity, formatting, confidence,
politeness or sophisticated writing style.

Unless one of those features is actually part of your product requirement.

---

5. Randomize or reverse pairwise ordering

For serious pairwise benchmarks, run:

A → B

and

B → A

Treat inconsistent decisions as a separate outcome.

---

6. Keep a human calibration set

Your judge prompt will change.

Your model provider will change.

The underlying judge model may change.

Your product will change.

Keep a fixed set of human-labelled examples so you can rerun the evaluator after those changes.

Think of it as unit tests for your evaluator.

---

7. Add new failures to the evaluation set

Suppose a production response says:

Your annual subscription will automatically refund within seven days.

Your evaluator incorrectly passes it.

Don’t only fix the individual case.

Add the failure pattern to your regression dataset.

Over time:

Production failure
       ↓
Human investigation
       ↓
Evaluation example
       ↓
Regression dataset
       ↓
Future release check

This turns incidents into persistent evaluation coverage.

---

8. Don’t make the judge do what code does better

Use code for:

JSON validity
regex checks
field presence
length constraints
numeric ranges
tool schema validation

Use an LLM judge for:

faithfulness
relevance
completeness
tone
semantic correctness
instruction following

The combination is generally stronger than forcing everything through one evaluation method.

---

9. Sample production traffic rather than blindly judging everything

An evaluator has cost and latency.

You may not need to evaluate every request.

A reasonable architecture is:

100% deterministic checks

        +

5–20% semantic judge sampling

        +

targeted evaluation of:
- new releases
- suspicious traces
- high-value workflows
- known failure categories

Choose the percentage according to risk, cost and traffic rather than copying a universal number.

Langfuse’s current production workflow exposes estimated matching volume and evaluation cost so sampling can be adjusted before applying an evaluator broadly.

---

Important Langfuse change for 2026

There is one implementation detail worth knowing if you’re following older tutorials.

Langfuse has moved toward an observation-oriented evaluation model rather than relying on its older trace-level LLM-as-a-Judge setup.

Its current documentation says trace-level evaluators are deprecated, and for Langfuse Cloud the legacy evaluators are scheduled to stop producing results at the v4 cutover on November 16, 2026. Current guidance recommends observation-level evaluators for production evaluation.

That means a tutorial built around an older trace-only workflow may already be on the wrong path.

For new implementations, target the relevant observation—such as the final generation or retrieval operation—instead of designing around a legacy trace evaluator.

Langfuse also released stable evaluator-management endpoints under /api/public/v2 in August 2026, making it possible to version evaluator configurations and roll them out more systematically.

This is particularly useful if you want evaluator changes to go through the same review process as application code.

---

LLM-as-a-Judge for RAG systems

LLM judges become especially useful in retrieval-augmented generation because a RAG application can fail in several independent places:

Question
   ↓
Retrieval
   ↓
Retrieved evidence
   ↓
Generation
   ↓
Answer

A bad final answer does not tell you where the problem occurred.

Instead evaluate the pipeline in layers.

Retrieval relevance

Did the retrieved chunks contain information relevant to the query?

Context sufficiency

Was enough evidence retrieved to answer the question completely?

Faithfulness

Does the generated answer stay within the retrieved evidence?

Answer relevance

Does the generated response actually answer the question?

This distinction matters.

A response can be faithful but useless:

Question:
Can I receive a refund after downloading the product?

Context retrieved:
Our company was founded in 2018.

Answer:
The supplied information does not say.

The generation is technically grounded.

The retrieval failed.

If you only score hallucination, your dashboard may incorrectly suggest the pipeline is healthy.

This connects directly to RAG chunking: before blaming generation, inspect whether retrieval supplied enough intact evidence. The existing GenAITrail chunking experiment emphasizes measuring retrieval separately from final generation.

---

Should an LLM judge replace human evaluation?

No serious evaluation pipeline should assume that it can.

A more realistic architecture is:

┌─ deterministic checks
Production output ───┼─ LLM judge
                     └─ sampled human review
                              ↓
                         calibration

Humans are expensive and difficult to scale.

LLM judges are cheaper and much easier to automate.

Code checks are highly reliable but narrow.

The strongest system uses each where it is appropriate.

Recent research reinforces the need for this caution. Studies published in 2024 and 2025 continue to find biases and robustness limitations in LLM judges, including susceptibility to position effects, prompt characteristics, superficial response qualities and domain shifts.

The conclusion isn’t that LLM judges are useless.

It is that an evaluator deserves evaluation just as much as the model it evaluates.

---

A practical LLM evaluation architecture

For a production AI application, I would structure evaluation like this:

USER REQUEST
                              │
                              ▼
                         APPLICATION
                              │
                              ▼
                        MODEL RESPONSE
                              │
             ┌────────────────┼────────────────┐
             │                │                │
             ▼                ▼                ▼
       CODE CHECKS        LLM JUDGES       USER SIGNALS
       ───────────        ──────────       ────────────
       JSON valid?       Faithfulness      Thumbs up/down
       Tool valid?       Relevance         Correction
       Length OK?        Completeness      Retry
       Schema OK?        Tone              Escalation
             │                │                │
             └────────────────┼────────────────┘
                              ▼
                          SCORE STORE
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
          DASHBOARD      REGRESSION SET    HUMAN REVIEW

The important part is the feedback loop.

A production evaluation system should not merely generate scores.

It should help answer:

What failed?

Why did it fail?

Is the failure becoming more common?

Did the new model or prompt fix it?

Did the fix break something else?

That is what turns LLM-as-a-Judge from an interesting prompt into engineering infrastructure.

---

When should you not use LLM-as-a-Judge?

There are several cases where an LLM judge is unnecessarily complicated.

You have an exact expected value

If the correct response is:

42

use an exact or numerical comparison.

The output has a strict schema

Validate the schema programmatically.

You can execute the answer

For generated SQL, code, calculations or API calls, execution-based evaluation can provide stronger evidence than subjective judgment.

The decision has serious real-world consequences

In medical, legal, financial or other high-impact settings, automated evaluation can help triage or support reviewers but should not automatically be treated as a substitute for the relevant expert oversight.

The criterion cannot be defined

If two humans cannot agree what “good” means, asking an LLM to convert the ambiguity into 4.2/5 does not solve the underlying problem.

Fix the rubric first.

A Practical Implementation

# ============================================================
# LLM-AS-A-JUDGE BENCHMARK
# GenAITrail experiment
#
# OpenRouter configuration
# ============================================================
# ============================================================
# CELL 1 — INSTALL DEPENDENCIES
# ============================================================
!pip -q install openai pandas numpy matplotlib python-dotenv scikit-learn
# ============================================================
# CELL 2 — IMPORTS
# ============================================================
import os
import re
import json
import time
import random
import platform
from pathlib import Path
from datetime import datetime, timezone
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from openai import OpenAI
from sklearn.metrics import (
    confusion_matrix,
    classification_report,
    accuracy_score,
    precision_score,
    recall_score,
    f1_score,
    cohen_kappa_score,
)
print("Imports successful.")
# ============================================================
# CELL 3 — OPENROUTER CONFIGURATION
# ============================================================
# ------------------------------------------------------------
# COLAB SECRETS
#
# In Google Colab:
#
# Left sidebar → Secrets → Add new secret
#
# Name:
#     OPENROUTER_API_KEY
#
# Value:
#     Your OpenRouter API key
#
# DO NOT hard-code the API key in the notebook.
# ------------------------------------------------------------
try:
    from google.colab import userdata
    OPENROUTER_API_KEY = userdata.get("OPENROUTER_API_KEY")
except Exception:
    OPENROUTER_API_KEY = os.getenv("OPENROUTER_API_KEY")
# ------------------------------------------------------------
# OpenRouter API endpoint
# ------------------------------------------------------------
OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1"
# ------------------------------------------------------------
# Model
#
# "openrouter/free" automatically routes to a currently
# available free model on OpenRouter.
# ------------------------------------------------------------
OPENROUTER_MODEL = os.getenv(
    "OPENROUTER_MODEL",
    "openrouter/free"
)
# ------------------------------------------------------------
# Judge behavior
# ------------------------------------------------------------
TEMPERATURE = 0
MAX_TOKENS = 600
# ------------------------------------------------------------
# Output directory
# ------------------------------------------------------------
OUTPUT_DIR = Path("llm_judge_results")
OUTPUT_DIR.mkdir(exist_ok=True)
# ------------------------------------------------------------
# Reproducibility
# ------------------------------------------------------------
RANDOM_SEED = 42
random.seed(RANDOM_SEED)
np.random.seed(RANDOM_SEED)
# ------------------------------------------------------------
# Display configuration
# ------------------------------------------------------------
print("Model:", OPENROUTER_MODEL)
print("Base URL:", OPENROUTER_BASE_URL)
print("Output directory:", OUTPUT_DIR.resolve())
if not OPENROUTER_API_KEY:
    print("\nWARNING: OPENROUTER_API_KEY is not configured.")
    print("Add OPENROUTER_API_KEY to Colab Secrets before")
    print("running the API cells.")
else:
    print("OpenRouter API key detected.")
# ============================================================
# CELL 4 — OPENROUTER API CLIENT
# ============================================================
client = OpenAI(
    base_url=OPENROUTER_BASE_URL,
    api_key=OPENROUTER_API_KEY,
)
print("OpenRouter client initialized.")
# ============================================================
# CELL 5 — TEST API CONNECTION
# ============================================================
def test_chat_api():
    start = time.perf_counter()
    response = client.chat.completions.create(
        model=OPENROUTER_MODEL,
        messages=[
            {
                "role": "system",
                "content": (
                    "You are an intelligent assistant. "
                    "Reply concisely."
                ),
            },
            {
                "role": "user",
                "content": (
                    "Reply with exactly this text and nothing else: "
                    "API connection successful."
                ),
            },
        ],
        temperature=TEMPERATURE,
        max_tokens=30,
    )
    elapsed = time.perf_counter() - start
    raw_content = response.choices[0].message.content
    text = raw_content.strip() if raw_content is not None else ""
    print("Response:", text)
    print(f"Latency: {elapsed:.2f} seconds")
    return {
        "response": text,
        "latency_seconds": elapsed,
        "model": OPENROUTER_MODEL,
    }
api_test = test_chat_api()
Output

# ============================================================
# CELL 6 — HUMAN-LABELLED POINTWISE DATASET
# ============================================================
pointwise_examples = [
    {
        "id": "P01",
        "question": "Within how many days can customers request a refund?",
        "context": (
            "Customers may request a refund within 30 days of purchase. "
            "Downloaded digital products are non-refundable."
        ),
        "answer": "Customers can request a refund within 30 days of purchase.",
        "human_label": "PASS",
        "category": "clear_pass",
    },
    {
        "id": "P02",
        "question": "Can downloaded digital products be refunded?",
        "context": (
            "Customers may request a refund within 30 days of purchase. "
            "Downloaded digital products are non-refundable."
        ),
        "answer": "No. Downloaded digital products are non-refundable.",
        "human_label": "PASS",
        "category": "clear_pass",
    },
    {
        "id": "P03",
        "question": "Can downloaded digital products be refunded?",
        "context": (
            "Customers may request a refund within 30 days of purchase. "
            "Downloaded digital products are non-refundable."
        ),
        "answer": (
            "Yes. All purchases can be refunded within 30 days, "
            "including downloaded digital products."
        ),
        "human_label": "FAIL",
        "category": "contradiction",
    },
    {
        "id": "P04",
        "question": "What happens after five failed login attempts?",
        "context": (
            "Accounts are locked for 15 minutes after five consecutive "
            "failed login attempts."
        ),
        "answer": "The account is locked.",
        "human_label": "PARTIAL",
        "category": "incomplete",
    },
    {
        "id": "P05",
        "question": "What happens after five failed login attempts?",
        "context": (
            "Accounts are locked for 15 minutes after five consecutive "
            "failed login attempts."
        ),
        "answer": (
            "After five consecutive failed login attempts, "
            "the account is locked for 15 minutes."
        ),
        "human_label": "PASS",
        "category": "clear_pass",
    },
    {
        "id": "P06",
        "question": "How long is an account locked?",
        "context": (
            "Accounts are locked for 15 minutes after five consecutive "
            "failed login attempts."
        ),
        "answer": "Accounts remain locked for one hour.",
        "human_label": "FAIL",
        "category": "wrong_number",
    },
    {
        "id": "P07",
        "question": "Which regions are included in free shipping?",
        "context": (
            "Free shipping is available in the continental United States. "
            "Alaska, Hawaii, and international destinations are excluded."
        ),
        "answer": (
            "Free shipping is available throughout the United States, "
            "including Alaska and Hawaii."
        ),
        "human_label": "FAIL",
        "category": "exception_ignored",
    },
    {
        "id": "P08",
        "question": "Which regions are included in free shipping?",
        "context": (
            "Free shipping is available in the continental United States. "
            "Alaska, Hawaii, and international destinations are excluded."
        ),
        "answer": (
            "Free shipping applies to the continental United States. "
            "Alaska, Hawaii, and international destinations are excluded."
        ),
        "human_label": "PASS",
        "category": "clear_pass",
    },
    {
        "id": "P09",
        "question": "When does the support desk operate?",
        "context": (
            "Customer support is available Monday through Friday "
            "from 9:00 AM to 5:00 PM Eastern Time."
        ),
        "answer": "Support is available Monday through Friday.",
        "human_label": "PARTIAL",
        "category": "missing_time",
    },
    {
        "id": "P10",
        "question": "When does the support desk operate?",
        "context": (
            "Customer support is available Monday through Friday "
            "from 9:00 AM to 5:00 PM Eastern Time."
        ),
        "answer": (
            "Customer support is available Monday through Friday "
            "from 9:00 AM to 5:00 PM Eastern Time."
        ),
        "human_label": "PASS",
        "category": "clear_pass",
    },
    {
        "id": "P11",
        "question": "How often are backups created?",
        "context": (
            "Database backups are created every six hours "
            "and retained for 30 days."
        ),
        "answer": "Backups are created every six hours.",
        "human_label": "PASS",
        "category": "clear_pass",
    },
    {
        "id": "P12",
        "question": "How often are backups created and how long are they retained?",
        "context": (
            "Database backups are created every six hours "
            "and retained for 30 days."
        ),
        "answer": "Backups are created every six hours.",
        "human_label": "PARTIAL",
        "category": "missing_retention",
    },
    {
        "id": "P13",
        "question": "How often are backups created?",
        "context": (
            "Database backups are created every six hours "
            "and retained for 30 days."
        ),
        "answer": "Backups are generated once per day.",
        "human_label": "FAIL",
        "category": "wrong_frequency",
    },
    {
        "id": "P14",
        "question": "What is the API rate limit?",
        "context": (
            "The standard API tier permits 100 requests per minute "
            "per API key."
        ),
        "answer": (
            "The API allows 100 requests per minute per key. "
            "This limit applies to the standard tier."
        ),
        "human_label": "PASS",
        "category": "clear_pass",
    },
    {
        "id": "P15",
        "question": "What is the API rate limit?",
        "context": (
            "The standard API tier permits 100 requests per minute "
            "per API key."
        ),
        "answer": (
            "The standard API tier provides an industry-leading, highly "
            "scalable and exceptionally flexible allocation of precisely "
            "100 requests every minute for each individual API key."
        ),
        "human_label": "PASS",
        "category": "verbosity_trap",
    },
    {
        "id": "P16",
        "question": "What is the API rate limit?",
        "context": (
            "The standard API tier permits 100 requests per minute "
            "per API key."
        ),
        "answer": (
            "The API supports 500 requests per minute, making it suitable "
            "for demanding enterprise applications."
        ),
        "human_label": "FAIL",
        "category": "confident_wrong",
    },
    {
        "id": "P17",
        "question": "What file formats are accepted?",
        "context": "The upload service accepts PDF, PNG, and JPEG files.",
        "answer": "PDF, PNG, and JPEG files are accepted.",
        "human_label": "PASS",
        "category": "clear_pass",
    },
    {
        "id": "P18",
        "question": "What file formats are accepted?",
        "context": "The upload service accepts PDF, PNG, and JPEG files.",
        "answer": "PDF and PNG are accepted.",
        "human_label": "PARTIAL",
        "category": "list_omission",
    },
    {
        "id": "P19",
        "question": "What file formats are accepted?",
        "context": "The upload service accepts PDF, PNG, and JPEG files.",
        "answer": "PDF, PNG, JPEG, DOCX, and ZIP are accepted.",
        "human_label": "FAIL",
        "category": "unsupported_additions",
    },
    {
        "id": "P20",
        "question": "When are invoices generated?",
        "context": (
            "Invoices are generated automatically on the first day "
            "of every month."
        ),
        "answer": "Invoices are generated automatically on the first day of each month.",
        "human_label": "PASS",
        "category": "clear_pass",
    },
]
pointwise_df = pd.DataFrame(pointwise_examples)
print(pointwise_df[["id", "human_label", "category"]])
print("\nLabel distribution:")
print(pointwise_df["human_label"].value_counts())

# ============================================================
# CELL 7 — SIMPLE JUDGE
# ============================================================
SIMPLE_JUDGE_PROMPT = """
You are evaluating an AI assistant answer.
Question:
{question}
Reference context:
{context}
Assistant answer:
{answer}
Rate the answer as exactly one of:
PASS
PARTIAL
FAIL
Return JSON only:
{{
  "label": "PASS | PARTIAL | FAIL",
  "reason": "brief explanation"
}}
"""# ============================================================
# CELL 8 — RUBRIC-BASED JUDGE
# ============================================================
RUBRIC_JUDGE_PROMPT = """
You are evaluating whether an AI assistant answer is faithful
and complete relative to supplied reference context.
Judge ONLY factual correctness and completeness.
Do not reward:
- verbosity
- sophisticated wording
- confidence
- formatting
- politeness
Use ONLY the supplied reference context.
LABEL DEFINITIONS
PASS:
The answer correctly answers the question and contains no
unsupported factual claims. Minor wording differences are allowed.
PARTIAL:
The central answer is correct, but an important requested detail
is missing. It must not contain a direct factual contradiction.
FAIL:
The answer contains a factual contradiction, an unsupported
substantive claim, a wrong number, wrong condition, or gives the
wrong central answer.
QUESTION:
{question}
REFERENCE CONTEXT:
{context}
ASSISTANT ANSWER:
{answer}
Return valid JSON only:
{{
  "label": "PASS | PARTIAL | FAIL",
  "unsupported_or_missing": [],
  "reason": "brief explanation"
}}
"""
# ============================================================
# CELL 9 — JSON RESPONSE PARSER
# ============================================================
def extract_json(text):
    if text is None:
        raise ValueError("Empty response.")
    text = text.strip()
    # Remove markdown fences
    text = re.sub(r"^```json\s*", "", text, flags=re.I)
    text = re.sub(r"^```\s*", "", text)
    text = re.sub(r"\s*```$", "", text)
    # First direct parse
    try:
        return json.loads(text)
    except json.JSONDecodeError:
        pass
    # Extract first JSON object
    match = re.search(r"\{.*\}", text, re.DOTALL)
    if not match:
        raise ValueError(f"No JSON object found:\n{text}")
    return json.loads(match.group(0))
# ============================================================
# CELL 10 — CALL JUDGE
# ============================================================
def call_judge(prompt, retries=3):
    last_error = None
    for attempt in range(retries):
        try:
            start = time.perf_counter()
            response = client.chat.completions.create(
                model=OPENROUTER_MODEL,
                messages=[
                    {
                        "role": "user",
                        "content": prompt,
                    }
                ],
                temperature=TEMPERATURE,
                max_tokens=MAX_TOKENS,
            )
            latency = time.perf_counter() - start
            raw_text = response.choices[0].message.content
            parsed = extract_json(raw_text)
            label = str(parsed.get("label", "")).upper().strip()
            if label not in {"PASS", "PARTIAL", "FAIL"}:
                raise ValueError(f"Invalid label returned: {label}")
            return {
                "label": label,
                "reason": parsed.get("reason", ""),
                "raw": raw_text,
                "latency_seconds": latency,
                "success": True,
            }
        except Exception as exc:
            last_error = str(exc)
            print(
                f"Attempt {attempt + 1}/{retries} failed:",
                last_error
            )
            time.sleep(2 * (attempt + 1))
    return {
        "label": "ERROR",
        "reason": last_error,
        "raw": "",
        "latency_seconds": None,
        "success": False,
    }
# ============================================================
# CELL 11 — RUN SIMPLE JUDGE
# ============================================================
simple_results = []
for i, row in pointwise_df.iterrows():
    prompt = SIMPLE_JUDGE_PROMPT.format(
        question=row["question"],
        context=row["context"],
        answer=row["answer"],
    )
    result = call_judge(prompt)
    simple_results.append(result)
    print(
        f'{row["id"]}: human={row["human_label"]} '
        f'judge={result["label"]}'
    )
    # Be polite to rate limits
    time.sleep(0.5)
pointwise_df["simple_judge_label"] = [
    x["label"] for x in simple_results
]
pointwise_df["simple_reason"] = [
    x["reason"] for x in simple_results
]
pointwise_df["simple_latency"] = [
    x["latency_seconds"] for x in simple_results
]

# ============================================================
# CELL 12 — RUN RUBRIC JUDGE
# ============================================================
rubric_results = []
for i, row in pointwise_df.iterrows():
    prompt = RUBRIC_JUDGE_PROMPT.format(
        question=row["question"],
        context=row["context"],
        answer=row["answer"],
    )
    result = call_judge(prompt)
    rubric_results.append(result)
    print(
        f'{row["id"]}: human={row["human_label"]} '
        f'judge={result["label"]}'
    )
    time.sleep(0.5)
pointwise_df["rubric_judge_label"] = [
    x["label"] for x in rubric_results
]
pointwise_df["rubric_reason"] = [
    x["reason"] for x in rubric_results
]
pointwise_df["rubric_latency"] = [
    x["latency_seconds"] for x in rubric_results
]

# ============================================================
# CELL 13 — DISAGREEMENT ANALYSIS
# ============================================================
simple_disagreements = pointwise_df[
    pointwise_df["human_label"] != pointwise_df["simple_judge_label"]
]
rubric_disagreements = pointwise_df[
    pointwise_df["human_label"] != pointwise_df["rubric_judge_label"]
]
print("Simple judge disagreements:")
display(
    simple_disagreements[
        [
            "id",
            "category",
            "human_label",
            "simple_judge_label",
            "simple_reason",
        ]
    ]
)
print("\nRubric judge disagreements:")
display(
    rubric_disagreements[
        [
            "id",
            "category",
            "human_label",
            "rubric_judge_label",
            "rubric_reason",
        ]
    ]
)

The simple prompt misclassified one incomplete answers as passes, whereas the rubric prompt correctly separated missing requested information from factual contradictions.

---

LLM-as-a-Judge checklist

Before deploying an evaluator, I would require answers to all of these:

□ What exact property are we evaluating?

□ Can deterministic code evaluate it instead?

□ Does every score/label have a precise definition?

□ Does the judge receive all evidence required for the decision?

□ Do we have human-labelled calibration examples?

□ Have we measured failure recall instead of only average agreement?

□ Have we tested position bias?

□ Have we tested verbosity traps?

□ Have we tested difficult edge cases?

□ Do we store judge reasoning for debugging?

□ Do we version the evaluator prompt?

□ Can we rerun old calibration data after changing the model?

□ Are production failures added to the regression dataset?

□ Do important disagreements get human review?

If several of those boxes are empty, the score generated by your evaluator probably deserves less confidence than the dashboard suggests.

---

FAQ

What is LLM as a judge?

LLM-as-a-Judge is an evaluation method where one language model assesses the output of another model or AI application using a specified rubric. It is commonly used for semantic qualities such as relevance, faithfulness, completeness and instruction following.

Is LLM-as-a-Judge accurate?

It can be useful, but accuracy depends on the judge model, prompt, criterion and data. The original MT-Bench research reported more than 80% agreement with human preferences for strong judges in its evaluation setting, while subsequent research has documented persistent biases and reliability limitations.

What is Langfuse LLM as a Judge?

Langfuse allows developers to create model-based evaluators, define scoring rubrics, test them against observations, and apply them to selected production or experimental data through evaluation rules.

What are the main LLM-as-a-Judge biases?

Commonly studied problems include position bias, verbosity or superficial-quality bias, self-preference effects, judge leniency and instability on difficult comparisons. Their severity varies by model and task.

What is the difference between MT-Bench and Chatbot Arena?

MT-Bench was introduced as a multi-turn benchmark for conversational models, while Chatbot Arena gathers pairwise human preferences from model battles. The original LLM-as-a-Judge research used them to study how well model-based judges corresponded with human judgments.

Should I use the same model as generator and judge?

You can, but do not assume that the resulting evaluation is independent. Compare the judge against human-labelled examples and, for important decisions, consider testing multiple judge models or model families.

Is LLM-as-a-Judge useful for RAG?

Yes. It can separately assess retrieved-context relevance, context sufficiency, response faithfulness and answer relevance. Separating these metrics makes it easier to tell whether a failure originated in retrieval or generation.

---

Primary sources and further reading

For source-level verification, start with the original Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena paper, the first-party Langfuse LLM-as-a-Judge documentation, and Google’s people-first content guidance. These primary references are better foundations than SEO summaries because they document the evaluation method, the tooling behavior, and the content-quality principles directly.

LLM as a Judge deployment notes

Use these final LLM as a judge checks when moving from an experiment to a production evaluation workflow.

• Start every LLM as a judge rollout with a small human-labelled calibration set.
• Run each LLM as a judge prompt against clear passes, partial answers, and known failures.
• Keep the LLM as a judge rubric narrow enough that two reviewers can agree on the label.
• Use LLM as a judge scoring for semantic quality, not for JSON validity or schema checks.
• Compare LLM as a judge results with deterministic tests before trusting a release gate.
• Track whether LLM as a judge failures cluster around retrieval, reasoning, citation, or tone.
• Review LLM as a judge disagreements with humans instead of silently averaging them away.
• Version every LLM as a judge prompt, rubric, model setting, and sampling rule.
• Retest the LLM as a judge setup after changing models, prompts, chunking, or retrieval filters.
• Use LLM as a judge dashboards to find patterns, then add the best examples to regression tests.
• Calibrate LLM as a judge thresholds separately for support, coding, search, and RAG workflows.
• Treat LLM as a judge output as evidence for review, not as automatic ground truth.
• Document when LLM as a judge automation should escalate to human review.
• Sample real user traces so the LLM as a judge dataset matches production behavior.
• Keep LLM as a judge explanations short enough for engineers to inspect quickly.
• Do not let an LLM as a judge score hide low recall on serious failures.
• A reliable LLM as a judge system improves when every missed failure becomes a new test case.
• The best LLM as a judge implementation combines clear rubrics, source evidence, and regular human calibration.

Final takeaway

LLM-as-a-Judge is useful precisely because many important properties of generative AI outputs cannot be reduced to exact string matching.

But replacing human judgment with an untested judge prompt only moves the uncertainty somewhere else.

A reliable implementation follows a different sequence:

Define the failure
        ↓
Write the rubric
        ↓
Collect human-labelled examples
        ↓
Build the judge
        ↓
Measure agreement and failure recall
        ↓
Attack it with bias tests
        ↓
Run it on production samples
        ↓
Send disagreements back into the evaluation set

MT-Bench and Chatbot Arena demonstrated why model-based evaluation can scale. Modern tools such as Langfuse make deploying those evaluators much easier. The remaining engineering work is determining whether your judge is actually measuring the thing you think it is.

Treat the evaluator like production code.

Version it.

Test it.

Give it regression cases.

Inspect its failures.

And never confuse a confidently formatted score with ground truth.