TL;DR
Vibes-based QA breaks the moment your llm evaluation framework has to defend a shipping decision. This guide shows the operational playbook engineers at scale-ups actually use: build a 30-100 example golden set, wire up a judge model with a JSON rubric, calibrate that judge against human labels, and gate every prompt change with a regression run. We ship a 300-line Python harness, compare Braintrust vs Langfuse vs LangSmith honestly, and walk through the failure modes nobody warns you about — judge model bias, pairwise preference eval traps, and why temperature=0 is a lie you tell yourself.
Did You Know? In OpenAI's own Evals research, judge-model agreement with human labelers on subjective tasks plateaus around 72-83% — meaning up to 1 in 4 of your automated verdicts is wrong unless you actively calibrate. Most teams never measure this.
Why vibes-testing your LLM breaks in production
You shipped a chatbot. Three months in, someone tweaks the system prompt to fix a customer complaint. Suddenly refunds spike, support tickets shift shape, and nobody knows whether it's the prompt, the model provider silently updating weights, or a Tuesday. This is the state most teams are in.
The fix isn't more careful prompting. It's treating your LLM feature like software: golden datasets, deterministic gates, and a review process for changes. Production llm testing exists as a category because eyeball QA — reading 10 outputs and shipping — misses the tail. And the tail is where users churn.
The three failure modes vibes-testing hides
- Silent regressions. A prompt tweak improves 8 examples you looked at and breaks 40 you didn't.
- Provider drift.
gpt-4otoday is notgpt-4ofrom six weeks ago. Anthropic and OpenAI both refresh weights without version bumps. - Agent regression. Multi-step agents fail on step 4 of 7, and your happy-path testing never sees it.
If any of those made you wince, the rest of this post is for you. For the wider context on why evals are becoming a first-class engineering discipline, browse the llm-eval-skills hub — it's the sister index page to this walkthrough.
What an eval harness actually is
An llm evaluation framework has four moving parts. You can build all four in an afternoon; you'll spend the next six months tuning them.
- A golden set. 30-500 examples with an input, a reference answer or rubric, and metadata (tags, difficulty, source).
- A candidate runner. The code that calls the system under test — a single prompt, a chain, or a full agent.
- A grader. Either exact-match, embedding similarity, a judge model, or a pairwise preference eval against a baseline.
- A tracker. Stores runs, diffs them, alerts on regressions. This is where LangSmith, Braintrust, and Langfuse compete.
Everything else — dashboards, cost telemetry, red-team suites — is convenience on top of these four. Get them working locally first. Then buy the SaaS. Not the other way around.
The 300-line harness (annotated)
Here's the core. It runs a candidate model, asks a stronger judge model to grade each answer three times, and reports agreement plus a calibration score against a small human-labeled sample. This is the seed you extend, not the finished product.
client = OpenAI() JUDGE = "gpt-4o-2024-11-20" # judge CANDIDATE = "gpt-4o-mini" # system-under-test RUBRIC = ("You are a strict evaluator. Score the ANSWER against the QUESTION.\n" "Return JSON: {\"correct\": 0|1, \"faithful\": 0|1, \"reason\": \"short\"}.") def run_candidate(q): r = client.chat.completions.create( model=CANDIDATE, messages=[{"role":"user","content":q}], temperature=0) return r.choices[0].message.content def judge(q, a, gold): r = client.chat.completions.create( model=JUDGE, response_format={"type":"json_object"}, messages=[ {"role":"system","content":RUBRIC}, {"role":"user", "content":f"QUESTION: {q}\nANSWER: {a}\nREFERENCE: {gold}"}], temperature=0) return json.loads(r.choices[0].message.content) def evaluate(dataset, runs=3): rows = [] for ex in dataset: answer = run_candidate(ex["q"]) scores = [judge(ex["q"], answer, ex["gold"]) for _ in range(runs)] agree = statistics.mode([s["correct"] for s in scores]) rows.append({"id":ex["id"], "answer":answer, "correct":agree, "runs":scores}) return rows def calibrate(rows, human_labels): # eval_harness.py -- minimal judge-model eval with calibration loop
import os, json, statistics
from openai import OpenAI
human_labels: {id: 0|1} for a 30-item sample
agree = sum(1 for r in rows if human_labels.get(r["id"]) == r["correct"]) return agree / len(human_labels)
if __name__ == "__main__": ds = json.load(open("golden_set.json")) results = evaluate(ds) print("pass_rate:", sum(r["correct"] for r in results)/len(results)) if os.path.exists("human_labels.json"): kappa = calibrate(results, json.load(open("human_labels.json"))) print("judge_vs_human_agreement:", kappa)
Read that once and note what's not in it: no vector store, no LangChain, no OpenTelemetry, no Kafka. Just a golden set, a candidate, a judge, and a calibration hook. Under 50 lines because most of the value is structural, not algorithmic.
What to add before you call it production
- Retry-with-backoff on both candidate and judge calls (both APIs 429 under load).
- Prompt version hashing so a diff of the system prompt shows up in the run record.
- Cost accounting per row (token counts in, token counts out, USD).
- A CSV or Parquet dump per run so you can point pandas at it in a notebook.
- Cache the candidate outputs — you'll re-grade with a new rubric more often than you re-run the candidate.
Pro Tip: Cache candidate outputs by
hash(prompt_version + input), not by input alone. When you tweak the system prompt, the cache invalidates automatically and you avoid the classic bug of grading last week's answers against this week's rubric.
Judge model calibration: the step everyone skips
A judge model is another LLM you're asking to grade LLM output. If you don't measure how well it agrees with a human, you have no idea what your pass rate means. This is judge model calibration, and it's the single highest-leverage thing you can do this quarter.
The 30-example calibration protocol
- Sample 30 rows from your last eval run — stratified across pass/fail and difficulty tags.
- Have two humans label them independently (correct / incorrect, or 1-5 rubric).
- Compute human-human agreement first. If it's under 80%, your rubric is ambiguous — fix the rubric before touching the judge.
- Compute judge-vs-human agreement. Report Cohen's kappa, not just raw agreement.
- Re-run any time you swap judge models, change the rubric, or update the candidate materially.
Anthropic's alignment research and OpenAI's openai/evals repo both document this pattern in different words. It's not exotic. It's just skipped because it feels like non-glamorous work compared to shipping a new feature.
Judge model bias patterns to watch for
- Length bias. Judges prefer longer answers. Control for it by including short-and-correct examples in your golden set.
- Position bias in pairwise preference eval. The judge picks whichever candidate came first about 55-60% of the time. Randomize order and average across both permutations.
- Self-preference. A GPT judge prefers GPT outputs; a Claude judge prefers Claude outputs. Ensemble across two judges from different families when the decision matters.
- Confidence miscalibration. Judges rarely say “I'm not sure.” Add an explicit
abstainoption to your rubric.
Designing a golden set that catches real bugs
Ten examples is not enough. Ten thousand is unnecessary until you're at scale. Aim for 100-300 rows in the first six weeks, growing to 500-1,500 by month six. The composition matters more than the count.
Composition targets
- ~50% happy-path (common intents, unambiguous inputs)
- ~20% edge cases (ambiguous, contradictory, missing context)
- ~15% adversarial (prompt injection attempts, jailbreak strings, out-of-scope questions)
- ~10% regression fixtures — every past bug becomes a permanent row
- ~5% recent production traffic, hand-labeled weekly
That last bucket is the one that matters most and the one teams forget. Every week, pull 10-20 real user queries from your logs, label them, and add them to the set. Six months in, this becomes the most valuable slice — it reflects how usage actually evolved, not how you imagined it would.
Pro Tip: Tag every row with
source,added_date, andbug_idif it came from a regression. When a run fails, your diff should sort failures by tag so you can tell “we broke the refund-flow subset” from “we broke everything by 2%.”
Braintrust vs Langfuse vs LangSmith: an honest comparison
Once your harness works locally, you'll want a tracker. Three tools dominate the ai model evaluation space in 2026. They overlap heavily but pick different tradeoffs.
| Capability | Braintrust | Langfuse | LangSmith |
|---|---|---|---|
| Primary focus | Evals-first, dataset-native | Tracing + evals, self-host friendly | Tracing + evals, LangChain-native |
| Self-hosting | Enterprise only | Yes, OSS core | Yes, paid tier |
| Judge model support | First-class, built-in rubrics | Yes, via custom evaluators | Yes, via LangChain evaluators |
| Pairwise preference eval | Native UI | Manual setup | Native |
| Pricing (2026, small team) | ~$0-500/mo | Free OSS / ~$59+/mo cloud | ~$39-500/mo |
| Best when | You care about eval UX above all | You need self-host + tracing | You already use LangChain |
None of them will make bad evals good. All of them will make good evals easier to run, share, and diff. Pick the one that matches your infra constraints. If you can't decide, start with Langfuse's self-host — you'll learn what you actually need in two weeks and the switching cost is low.
For teams weighing observability separately from evals, the llm-observability-skills hub covers tracing, cost tracking, and PII redaction in more depth than any single tool's docs.
Wiring evals into CI (the part that changes behavior)
An eval suite that runs manually is a research project. An eval suite that blocks a merge is engineering. This is where prompt regression becomes real.
The minimum viable gate
- Run the harness on every PR that touches
prompts/,chains/, or the model config. - Fail the build if pass rate drops more than 2 percentage points vs main.
- Post a comment on the PR with the diff — which rows regressed, which improved.
- Allow override with an
eval-overridelabel and a written justification (goes in the PR description, not just a checkbox).
The override matters. Sometimes you're deliberately trading one metric for another (say, accepting a 3-point drop in verbosity to gain 8 points in factuality). Force the tradeoff to be documented. Otherwise every regression gets waved through.
Agent regression testing specifics
Multi-step agents need step-level scoring, not just end-to-end. Log the tool calls, intermediate reasoning, and final answer for each row. When a run fails, you want to know whether the agent picked the wrong tool at step 2 or the right tool with the wrong arguments at step 4. If your traces don't show that, your evals will lie to you.
Frameworks like DeepEval and Ragas both ship agent-specific metrics — tool-call accuracy, hallucination-in-reasoning, faithfulness-to-retrieved-context. Steal the metric definitions even if you don't use the libraries.
What this costs in engineer-time and dollars
Real numbers from teams shipping this pattern:
- Week 1-2: Build the harness, seed 30-50 examples, calibrate judge. One engineer, ~40 hours.
- Month 2: Wire to CI, grow set to 150 rows, add cost tracking. Same engineer, ~30 hours over the month.
- Month 3-6: Weekly production sample-and-label ritual (~2 hours/week), rubric refinement, expand to secondary use cases.
- Judge API cost: $0.20-2.00 per full run at 200 rows with a GPT-4-class judge. Cache aggressively.
Compare that to the cost of one silent regression that ships to production and burns 10,000 users' trust. The math has never been closer to obvious. If you're also standing up the tool-use / MCP side of your stack, the MCP server skills hub pairs cleanly with this — evals for tool-calling agents get a lot more tractable once you can mock the MCP layer deterministically.
Frequently asked questions
What is an LLM evaluation framework, in one sentence?
A repeatable pipeline that runs your model or agent against a fixed set of examples, grades each output with a rubric or judge model, and reports pass rates and regressions so you can gate prompt changes the same way you gate code changes.
Do I need Braintrust, Langfuse, or LangSmith on day one?
No. Start with a Python harness and a CSV. Move to a tracker once you have 100+ examples, more than one engineer touching prompts, or a compliance requirement to retain run history. Buying a tool before you understand your workflow just gives you a fancy UI for the wrong metrics.
How large should my golden set be to trust the results?
Statistically, 100 examples gives you roughly ±5pp confidence intervals on pass rate. That's usually enough to catch regressions larger than the noise floor. Below 50 examples, treat pass-rate changes with heavy skepticism. Above 500, you're mostly buying resolution on subsegments.
Can I use the same model as both candidate and judge?
Only for cheap smoke tests. For anything you'll defend to a stakeholder, use a stronger, different-family judge (a Claude judge for GPT candidates, or vice versa). Self-judging inflates pass rates by 5-15 percentage points on subjective tasks in practice.
What's the difference between pairwise preference eval and rubric-based scoring?
Rubric scoring grades each answer independently against a checklist (is it correct? is it faithful?). Pairwise preference eval shows the judge two candidates side by side and asks which is better. Pairwise is more sensitive to small quality differences but harder to interpret in absolute terms and prone to position bias.
How do I catch prompt regression from silent model updates?
Run your eval suite on a nightly cron against production model IDs. When OpenAI or Anthropic ships a weight refresh, your pass rate will move even though your code didn't. Alert on rolling 7-day deltas greater than 3pp. This is the cheapest early-warning system you'll ever build.
Where does Ragas fit in?
Ragas focuses on retrieval-augmented generation metrics — faithfulness, answer relevance, context precision. If you ship a RAG pipeline, Ragas gives you off-the-shelf metrics that complement a rubric-based judge. It's not a replacement for a general eval harness; think of it as a metric library you plug in.
Is promptfoo worth using?
Yes, especially for teams that want a YAML-driven config and comparison across models. It's excellent for one-off “which prompt wins” experiments and side-by-side model shootouts. It's less opinionated than Braintrust for long-running production tracking, so many teams use both.
How often should I recalibrate the judge?
Whenever the rubric changes, the judge model version changes, or the candidate output distribution shifts materially. In practice, that's roughly once a month for actively developed systems. Set a calendar reminder — nobody remembers otherwise.
What if I don't have humans to label data?
You always have humans — you, your PM, your subject-matter expert. Thirty labels across two people takes ninety minutes. If nobody on your team can spare that time to sanity-check what “correct” means, you don't have a problem an eval harness can solve.
Where to go from here
Stand up the harness this week. Seed thirty examples. Calibrate against one hour of human labeling. Wire it to CI before you touch another prompt. You'll ship faster because you'll stop fearing regressions — and the ones you don't catch will get added to the golden set the day they show up, permanently protecting future you. For the wider index of hands-on skills, patterns, and case studies backing this walkthrough, the llm-eval-skills hub is the canonical entry point.