What the original graphic gets right — and misses.
Keep
The 11-method taxonomy is an excellent teaching scaffold: reference metrics, model graders, people, agent traces, conversations, and safety are distinct lenses.
Correct
BLEU is not simply “precision”: it combines modified n-gram precision and a brevity penalty. ROUGE is a family of metrics, not just recall. BERTScore is embedding-based similarity, not factual verification. Human review is valuable but not automatically ground truth.
Expand
G-Eval is a particular rubric-driven LLM evaluation approach, not a synonym for every judge. DAG-style evaluation depends on a clearly defined decision tree and reliable underlying checks. A jury can share correlated bias even with several judges.
Add
Executable assertions, retrieval quality, statistical uncertainty, adversarial test design, privacy, cost, latency, monitoring, versioning and deployment gates are essential complementary controls.
Engineering principle: Evaluate the complete system, not only the language model. Define a task-specific test set, run deterministic checks where possible, calibrate subjective judges against humans, and monitor changes after release.
Eleven methods, properly understood.
Filter by family. Each card explains the workflow, suitable applications and its most important limitation.
Choose the right evaluation stack.
Select your use case and priorities to get a defensible starting stack. This is guidance, not an automatically validated evaluation.
Your recommended stack
Configure the application on the left and generate a tailored set of checks.
See why overlap scores can mislead.
This local simulator computes unigram precision, recall, F1, and a simple longest-common-subsequence F1. These are teaching proxies — not full corpus BLEU, ROUGE packages or BERTScore.
Tokenization: lowercase alphanumeric words; no stemming, semantic embeddings or synonym handling.
Do not interpret these percentages as truthfulness. “The treatment is effective” and “The treatment is not effective” can have high word overlap while contradicting each other.
Release-readiness scorecard.
A practical checklist of controls. Progress is stored only in this browser and can be exported as JSON. Checking a box records an assertion; it does not certify the system.
Evidence checklist
Suggested release gates
Set these thresholds against business risk and your own measured baseline; the values below are examples, not universal standards.
| Dimension | Example acceptance criterion |
|---|---|
| Task success | At or above agreed baseline, with confidence interval reported |
| Critical safety | Zero known critical failures in the defined blocking suite; continue monitoring |
| Human agreement | Judge calibrated on representative expert-labelled cases |
| Latency | p95 response time within service objective |
| Cost | Per-successful-task cost within budget |
| Regression | No statistically or operationally significant degradation in priority slices |
From ad hoc testing to continuous assurance.
01 / Define
Specify user task, failure severity, business KPI, unit of evaluation and pass/fail criteria.
02 / Curate
Build representative, adversarial, multilingual and privacy-reviewed cases; separate development and holdout sets.
03 / Instrument
Record prompts, model and system versions, retrieved context, tool calls, outputs, timing and cost.
04 / Score
Combine executable assertions, reference metrics, calibrated LLM judges and blinded human adjudication.
05 / Analyze
Report slices, confidence intervals, failure taxonomy, judge disagreements and severity-weighted outcomes.
06 / Gate & monitor
Block critical regressions, document approvals, sample production traffic safely and add incidents to the test set.
Strengths and failure modes.
| Family | Good at | Cannot establish alone | Recommended companion |
|---|---|---|---|
| BLEU / ROUGE | Lexical overlap at scale | Factual correctness or semantic equivalence | Human review / task assertions |
| BERTScore | Contextual semantic similarity | Truth, source support, safe behavior | Claim-level grounding checks |
| G-Eval / LLM judge | Rubric-based qualitative grading | Unbiased or perfectly reliable judgment | Human calibration and bias checks |
| LLM jury | Disagreement visibility and ensemble review | Independence of model errors | Expert adjudication |
| DAG | Auditable conditional checks | Validity of its rules and underlying graders | Unit tests and human review |
| Trajectory / multi-turn | Tool behavior, state and dialogue consistency | End-user utility by itself | Outcome and safety evaluation |
| Safety / human | Policy risks and nuanced task quality | Complete coverage of unseen failure modes | Adversarial testing and monitoring |
Primary research & engineering references.
Direct links to papers and maintained guidance. External links require internet access; the learning page itself works offline.