Beyond 'Looks Good': Evals for Production AI

Let's be honest. For the past few years, a lot of AI feature development has been vibes-based. You tweak a prompt, the output 'feels better' in the playground, and you ship it. This was fine when a chat interface was a novelty. It is not fine in 2026, where AI is deeply embedded into critical workflows.

The 'good enough' bar for AI is rising fast. Users now expect reliability, consistency, and correctness. They don't care about your clever prompt chain. They care if the AI-generated summary is accurate or if the AI-suggested code actually runs. Relying on gut feelings to ship products is now a form of engineering malpractice. The wild west phase is over. What got us here-rapid prototyping and a high tolerance for weirdness-won't get us there. We need to move from 'it feels better' to 'it is better, and here is the data to prove it'.

The-Old-Way-Is-Broken

Traditional software testing methods are necessary but wildly insufficient for evaluating LLM-based features. Your classic testing pyramid-unit tests, integration tests, E2E tests-was built for a deterministic world. It assumes that for a given input, there is a single, correct output. This assumption shatters the moment you introduce a non-deterministic, generative model into your stack. You can't write an `assert output == 'expected_string'` when the model might produce thousands of valid, stylistically different variations.

Sure, you still need unit tests for the deterministic parts of your code. Your API endpoints must work. Your data parsing must be correct. But these tests tell you nothing about the quality of the AI's output. They can confirm the model produced a JSON object, but not whether the contents of that JSON are coherent, factually correct, or helpful to the user. The old QA paradigm of discrete pass-fail states simply doesn't map to a world of probabilistic, creative, and sometimes subtly incorrect output. We need a new stack.

Introducing-the-Modern-Eval-Stack

The modern evaluation stack is a multi-layered system for assessing AI quality from the lab to the real world. Think of it not as a single tool, but as a practice that combines different methods to give you a holistic view of your feature's performance. It is your defense against regressions, your guide for iteration, and your proof of progress.

Here's what that stack looks like:

1. Unit Evals (The Foundation): These are fast, automated tests that check for specific, verifiable behaviors at the model or prompt level. They are your new unit tests. They run on every code change and provide an immediate feedback loop. Examples include checking for JSON format adherence, flagging safety-policy violations, detecting prompt injections, or verifying simple factual recall on a known set of questions.

1. System Evals (The Integration Test): This is where you assess the quality of the entire AI feature on a curated 'golden dataset' of realistic inputs. Here, the definition of 'correct' is fuzzy. Instead of pass-fail, you look at preference. Is version B better than version A? This is where you use more sophisticated techniques like LLM-as-a-judge for scaled-out comparisons or human annotators for the highest-fidelity signal.

1. Online Evals (The Source of Truth): This layer measures how the feature performs in production with real users. Metrics here are a mix of explicit and implicit signals. Explicit signals are things like thumbs up/down buttons or user correction forms. Implicit signals are often more powerful. Did the user copy the AI's answer? Did they retry their prompt? Did they abandon the workflow? Ultimately, this layer connects AI performance to business outcomes like conversion rate, task completion time, or support ticket deflection.

Building-Your-Golden-Dataset

Your golden dataset is the most valuable, non-model asset you will build for your AI product. It is a curated collection of high-quality, representative examples that you use for all offline system evaluations. It is the benchmark against which you measure every change to your prompts, models, and retrieval strategies. Without a good dataset, your evals are meaningless.

The thing is, this dataset is a product, not a data dump. It requires deliberate effort to create and maintain.

• Sourcing: Start with what you have. Logged production data is the best source, but it's messy. You need to sample it, clean it, and de-identify it. You can also have internal users 'dogfood' the product and submit interesting examples. For new features with no data, you can even use a powerful LLM like GPT-4 or Claude 3 Opus to synthetically generate a diverse set of test cases based on your product specs. Mix these sources for robustness.

• Curation: Your dataset must cover the good, the bad, and the ugly. Include your 'happy path' examples, but also load it with edge cases, adversarial attacks, and examples that have historically caused problems. Each entry should have a clear input, an 'ideal' output if one exists, and metadata like user persona or difficulty level. Version your dataset in Git or another system just like you version your code.

• Maintenance: This is not a one-off task. Your golden dataset is a living artifact. You must have a process for continuously adding new, interesting cases found in production logs. Review and prune the dataset periodically to ensure it still reflects current user behavior. Stop arguing about prompts in pull requests. Instead, argue about which examples should be added to the golden dataset. That is a much higher-leverage activity.

LLM-as-a-Judge-Strengths-and-Pitfalls

Using a powerful frontier model to evaluate the output of your application's model is one of the most important techniques for system-level evals. LLM-as-a-judge is fast, cheap, and scalable. It allows you to run a comparison between two model versions across your entire golden dataset in minutes, not days. This is a superpower for rapid iteration.

However, you must be aware of its limitations. A judge LLM is not a source of ground truth-it's a source of scaled-out preference. These models are notoriously susceptible to biases. They tend to prefer longer, more verbose answers (verbosity bias). They can prefer the answer in position A over position B (positional bias). They often favor answers that sound like themselves (self-consistency bias). You cannot simply ask an LLM 'Is this a good answer?'.

To use a judge LLM effectively, you need a detailed, specific rubric. Instead of a single score, have it evaluate multiple dimensions: Is the answer factually correct according to the provided context? Does it adhere to the requested format? Is the tone appropriate? By breaking down the problem, you get a much more reliable and debuggable signal. Always treat the judge's output with skepticism and calibrate it with periodic human evaluation.

| Criteria | LLM-as-a-Judge | Human Evaluation | | :--- | :--- | :--- | | Speed | Seconds | Hours to Days | | Cost | Pennies per eval | Dollars per eval | | Scale | Virtually unlimited | Limited by human resources | | Nuance | Low to Medium | High | | Consistency | High (with fixed temp) | Medium (inter-annotator agreement) | | Best For | Iterative development, regression testing, pairwise comparisons | Golden set validation, subjective quality, user preference |

Beyond-Accuracy-Evaluating-User-Experience

Measuring what actually matters to users is the final and most important piece of the puzzle. Academic benchmarks like BLEU or ROUGE are mostly irrelevant for product work. A high score on a text-similarity metric doesn't mean the user found the feature helpful. In fact, it might just mean the model produced a generic, uninspired answer.

To evaluate user experience, you must connect model performance to user behavior and business goals. This is where your online evaluation system comes in. Instrument your product to track key proxy metrics that correlate with user satisfaction.

Here are some examples:

• Interaction Metrics: Do users accept the AI's suggestion, or do they edit it heavily? Do they have to re-run their prompt multiple times to get a good result? High retry rates are a strong negative signal.
• Latency Metrics: How long does the user have to wait? Time-to-first-token is critical for perceived responsiveness. Total generation time impacts the user's flow. Slow AI is often worse than no AI.
• Workflow Metrics: Does the AI feature actually help the user accomplish their goal faster or more effectively? For a code assistant, this could be 'accepted completions per day'. For a customer support bot, it's 'automated resolution rate' or 'reduction in ticket escalation'.

These business-level metrics are the ultimate arbiter of quality. An improvement in an offline eval score is only meaningful if it eventually leads to an improvement in these online KPIs. Your goal is to build a causal link between the changes you make and the value your users receive.

How-do-I-start-if-I-have-no-data?

The solution to the cold start problem is to create your initial data yourself. Do not wait for production traffic to start thinking about evaluation. Your initial 'golden dataset' can be built from a few dozen examples that you and your team write by hand. Pretend you are different types of users and create inputs that test the core functionality, potential failure modes, and edge cases. This founder-led annotation is invaluable for encoding your product intuition into a testable asset from day one. You can then augment this with synthetically generated data from a powerful model like Claude 3 Opus or GPT-4o to increase coverage. It won't be perfect, but it's a thousand times better than having no benchmark at all.

How-often-should-we-run-our-evals?

You should run evals at a cadence that matches your development velocity and the risk of the change. A good starting point is a tiered approach. Run your fast, automated 'unit evals' on every single pull request in CI/CD. This catches simple regressions before they are ever merged. Run your larger, more expensive 'system evals' on a nightly basis against your main branch to track overall quality trends. Finally, your 'online evals' are not 'run' at all-they are a real-time monitoring system that is always on, constantly feeding you data about how your feature is performing in the wild. This combination gives you immediate feedback, a daily health check, and a continuous pulse on user experience.