Evaluation-Driven Development: Test Harnesses for Non-Deterministic AI

Your CI pipeline is green. Every test passes. You merge to main with confidence, and the deployment rolls out at 3pm on a Tuesday. By 4:15pm, your customer-facing agent has hallucinated a 90-day return policy that doesn't exist, auto-approved three refunds that violated your terms, and generated an email citing a support article your team deleted six months ago. The test suite didn't lie, exactly. It just tested the wrong things.

Traditional testing breaks when applied to non-deterministic AI systems. You can't assertEqual on an output that's valid in 500 different phrasings. You can't snapshot-test a response that changes with every inference call. What you need is an evaluation harness, not a test suite. The distinction matters: a test suite checks for exact correctness, while an eval harness measures quality across a spectrum, using layered strategies that match the probabilistic nature of your system. This article walks through a three-layer evaluation framework (deterministic assertions, LLM-as-judge scoring, and regression benchmarks) that you can wire into your CI/CD pipeline and start shipping agentic features with actual confidence.

Your Test Suite Is Lying to You (and Passing)

Here's what happened on that Tuesday. The team had 247 unit tests covering the agent's tool-calling interface. Every function returned the right schema. Every API mock responded on cue. The integration tests confirmed the agent could chain three tools together to resolve a ticket. All green.

But none of those tests evaluated whether the agent's natural language output was factually correct, policy-compliant, or tonally appropriate. The agent generated valid JSON, called the right tools, and composed grammatically correct English. It just said the wrong things.

This is the fundamental gap. Traditional tests verify structural correctness. Agentic systems need semantic evaluation, checking whether the meaning of the output aligns with your intent, your policies, and your users' expectations.

The three-layer eval framework addresses this by stacking checks at different fidelity levels: fast deterministic assertions that catch structural failures, model-graded scoring that evaluates semantic quality, and regression benchmarks that detect drift over time. Each layer maps to a specific CI/CD stage, and together they form a defense-in-depth strategy for non-deterministic outputs.

Why Traditional Testing Algebra Doesn't Work for Agents

Unit testing assumes a fundamental property: determinism. Given input X, function F always returns Y. You verify F(X) == Y, and you're done. Agentic AI violates this assumption at every level.

Consider a customer-support agent tasked with writing a refund denial email. The correct output must reference the 30-day return window, express empathy, and offer an alternative (store credit). But "I understand this is frustrating" and "I'm sorry to hear about this situation" are both valid empathy expressions. "Unfortunately, your purchase falls outside our 30-day return window" and "Our return policy covers items returned within 30 days of purchase" both state the policy correctly. The equivalence class of valid outputs isn't two or three phrasings. It's hundreds.

This creates what I call the equivalence class explosion problem. You can't enumerate valid outputs. You can't snapshot them. You need to evaluate properties of the output, not the output itself.

The bottom line: you're not testing a function anymore. You're evaluating a system that produces language. Your tooling needs to reflect that shift.

Layer 1: Assertion-Based Checks That Still Earn Their Keep

Don't throw out assertions entirely. Deterministic checks still cover 40-60% of what matters in agentic outputs, if you scope them correctly. The key is distinguishing guardrail assertions from correctness assertions.

Guardrail assertions verify structural and safety properties that have binary answers:

• Schema validation: Does the tool-call response match the expected JSON schema?
• Constraint bounds: Is the response under 500 tokens? Does it contain required fields?
• Safety checks: Does the output contain PII? Does it include any banned phrases from your compliance list?
• Citation presence: If the agent claims a policy, does it reference a source document ID?

These checks are fast (milliseconds), cheap (no LLM calls), and should run on every commit.

# test_agent_output.py
import pytest
from pydantic import BaseModel, Field, validator
from typing import List, Optional

class ToolCallResponse(BaseModel):
    tool_name: str = Field(..., pattern=r'^(lookup_policy|create_ticket|send_email)$')
    arguments: dict
    citations: List[str] = Field(..., min_length=1)
    response_text: str = Field(..., max_length=2000)

    @validator('response_text')
    def no_pii_patterns(cls, v):
        import re
        ssn_pattern = r'\b\d{3}-\d{2}-\d{4}\b'
        if re.search(ssn_pattern, v):
            raise ValueError('Response contains SSN pattern')
        return v

    @validator('response_text')
    def no_banned_phrases(cls, v):
        banned = ['I guarantee', 'we promise', 'legal liability']
        for phrase in banned:
            if phrase.lower() in v.lower():
                raise ValueError(f'Response contains banned phrase: {phrase}')
        return v

@pytest.fixture
def agent_response():
    """Simulate agent output for a refund denial scenario."""
    from my_agent import run_agent
    return run_agent(intent="refund_request", order_id="ORD-9921")

def test_tool_call_schema(agent_response):
    """Validates structural correctness of agent tool calls."""
    for call in agent_response.tool_calls:
        ToolCallResponse(**call)  # Pydantic raises on schema violation

def test_response_length_bounds(agent_response):
    assert 50 < len(agent_response.text) < 2000, "Response outside acceptable length range"

The trap is assuming these checks prove your agent works correctly. They don't. An agent can produce perfectly structured, PII-free, correctly-cited output that's factually wrong. Layer 1 tells you the output is safe to evaluate further. It doesn't tell you the output is good.

Layer 2: LLM-as-Judge Scoring with Rubric-Anchored Prompts

Model-graded evaluation uses a second LLM (the "judge") to score your agent's output against a structured rubric. This sounds circular ("using AI to test AI"), but it works when calibrated properly. A 2024 study from the LMSYS team showed that GPT-4-based judges reach 89% agreement with human raters on rubric-anchored evaluations, measured by Cohen's kappa.

The critical detail is rubric design. Vague rubrics ("rate quality 1-5") produce unreliable scores. Anchored rubrics with concrete descriptions for each score level produce consistent results.

JUDGE_PROMPT = """You are evaluating a customer support agent's response.
Score the response on two dimensions using the rubrics below.

## Factual Accuracy (1-5)
1: Contains fabricated policies, incorrect dates, or wrong product details
2: Mostly correct but includes one factual error or unsupported claim
3: Factually correct on core claims but missing important qualifications
4: Factually correct and appropriately qualified with minor omissions
5: Completely accurate, all claims supported by cited policy documents

## Tone Compliance (1-5)
1: Adversarial, dismissive, or inappropriately casual
2: Professional but cold, lacks empathy acknowledgment
3: Adequate tone with generic empathy ("we apologize for any inconvenience")
4: Warm, specific empathy that acknowledges the customer's situation
5: Empathetic, helpful, and proactively offers alternatives

## Input
Customer message: {customer_message}
Agent response: {agent_response}
Relevant policy document: {policy_text}

## Output Format
Return JSON: {{"factual_accuracy": <int>, "tone_compliance": <int>, "reasoning": "<str>"}}
"""

Calibrating your judge

Before trusting model-graded scores in CI, you need to calibrate. Take 50 representative examples, have two human raters score them on your rubric, then run the same examples through your judge prompt. Compute Cohen's kappa between human-human and human-judge pairs. If the judge-human agreement is within 0.1 of human-human agreement, your judge is reliable enough for automated gates.

Run Layer 2 evaluations on PR merges, not on every commit. The cost is low per call, but running 200 evals on every push adds up and slows your feedback loop.

Layer 3: Regression Benchmarks with Golden Example Suites

Golden example suites are your safety net against slow quality erosion. A single eval run might score 4.2/5 on factual accuracy, and that looks fine in isolation. But if last month's average was 4.6, you have a regression that no individual test caught.

Building your golden dataset

Start with 200+ curated input-output pairs. Source them from three places:

• Production logs: Real user queries that your agent handled, annotated with human quality scores
• Adversarial examples: Edge cases designed to trigger known failure modes (ambiguous requests, multi-intent queries, policy boundary cases)
• Failure post-mortems: Every production incident becomes a golden example with the correct output annotated

Version your golden sets in the same repository as your prompt templates and model configs. When you update a prompt, the golden suite travels with it. Use a directory structure like evals/golden/v12/ and tag releases alongside your application code.

Scoring methodology

Run your full golden suite nightly. Compute aggregate pass rates (percentage scoring above threshold on each dimension), per-category breakdowns, and score distributions. Track these over time to detect drift.

The hardest decision is: when does a regression block a deploy vs. trigger a review?

Policy accuracy gets the strictest threshold because factual errors create legal and reputational risk. Edge case handling gets a softer threshold because these are, by definition, unusual scenarios where some degradation is acceptable.

Wiring Evals into CI/CD Without Killing Velocity

Map the three layers directly to CI stages. Layer 1 (assertions) runs on every commit in under 30 seconds. Layer 2 (LLM-as-judge) runs on PR merge and completes in under 8 minutes for a 50-example eval set. Layer 3 (regression benchmarks) runs nightly against the full 200+ golden suite.

Practical architecture

In AWS, this looks like: GitHub Actions triggers a CodeBuild job that spins up an eval container. The container pulls cached golden sets from S3, runs the appropriate eval layer based on the trigger type (commit, PR, nightly cron), and posts results to a CloudWatch dashboard. For teams already using our agentic AI platform, eval orchestration can plug into the existing agent execution pipeline.

Handling flaky evals

Non-deterministic systems produce non-deterministic eval results. An agent might score 4/5 on one run and 3/5 on the next for the same input. Don't treat this as a bug in your eval. Treat it as a property of the system.

Use statistical thresholds instead of binary pass/fail. Run each critical example 3 times. Pass if the median score meets the threshold. For the full suite, set pass criteria like "47 of 50 examples pass" rather than "all 50 pass." This absorbs legitimate variance without masking real regressions.

For alert routing: policy accuracy failures page on-call. Tone regressions file a P2 ticket. Edge case degradation goes into the next sprint's backlog.

What Your Eval Dashboard Should Actually Show

Most teams build eval dashboards that show the wrong things. A single "pass rate: 94%" number tells you almost nothing. You need trend data, category breakdowns, and cost visibility.

Five metrics that matter:

• Pass rate by category: Broken out by policy accuracy, tone, edge cases, and any domain-specific dimensions. A 94% aggregate can hide a 70% pass rate on policy accuracy.
• Score distribution drift: Plot the distribution of scores over time. A shift from bimodal (mostly 4s and 5s) to uniform (spread across 1-5) signals instability even if the mean hasn't changed.
• Cost per eval: Track your monthly spend on judge LLM calls. At $0.03 per eval and 200 nightly evals, you're at $180/month. If that jumps to $500, you've got prompt bloat or unnecessary reruns.
• Eval latency: P50 and P99 latency per eval call. Latency creep kills developer adoption.
• Human override rate: How often does a human reviewer disagree with the judge's score? If overrides exceed 15%, your rubric needs recalibration.

Connect these to business KPIs. If your agent handles 10,000 tickets/month and your policy accuracy pass rate drops from 95% to 90%, that's 500 additional tickets with potential factual errors. Multiply by your average cost-per-escalation ($45 in most B2C support orgs), and leadership understands why eval infrastructure matters: it's a $22,500/month risk reduction.

Frequently Asked Questions

How many golden examples do I need to start?

Start with 100. You can build a useful regression suite from 100 well-curated examples in a week. Scale to 200+ over the first month by adding production failures and edge cases. Teams that wait for a "complete" dataset never ship an eval harness.

Which LLM should I use as a judge?

GPT-4o or Claude 3.5 Sonnet for calibration. GPT-4o-mini or Claude 3.5 Haiku for production eval runs where cost matters. The key is calibrating with a strong model and validating that the cheaper model agrees on your specific rubric. We've seen 4o-mini match 4o at 92% agreement on well-anchored rubrics.

Can I use the same model as both agent and judge?

You can, but cross-model judging produces less biased scores. If your agent runs on Claude, judge with GPT-4o (or vice versa). Self-evaluation tends to inflate scores by 0.3-0.5 points on a 5-point scale.

How do I handle eval results for agents that call external APIs?

Mock external APIs for Layer 1 and Layer 2 evals (same as traditional integration testing). For Layer 3 nightly runs, consider a staging environment with real API access to catch integration-level regressions. Tag golden examples as "mock-safe" or "needs-live-api" and route accordingly.

Start Here: A 30-Day Eval Harness Buildout Plan

Week 1: Instrument and collect. Add structured logging to your agent's output pipeline. Capture the full input, output, tool calls, and any intermediate reasoning. Pull 100 representative examples from production logs and annotate them with pass/fail on your two most important dimensions (likely factual accuracy and policy compliance). This is your seed golden dataset.

Week 2: Build Layer 1. Write Pydantic models for your agent's output schema. Add constraint validators for length, PII, banned phrases, and required fields. Wire these into your CI pipeline as pytest tests. They should run in under 30 seconds on every commit.

Week 3: Build Layer 2. Design your rubric with anchored descriptions for each score (1-5). Write the judge prompt. Run 50 golden examples through both human raters and the judge. Compute agreement. Iterate on the rubric until judge-human kappa exceeds 0.75. Integrate as a PR merge gate with a 50-example subset. Teams building on enterprise AI platforms with built-in governance can often reuse existing quality scoring infrastructure here.

Week 4: Build Layer 3. Expand your golden suite to 200+ examples across all categories. Set per-category pass thresholds. Configure a nightly cron job that runs the full suite, computes aggregate scores, and posts results to your dashboard. Set up alert routing: critical failures page on-call, moderate regressions file tickets.

Remember that Tuesday at 4:15pm, when your green CI pipeline didn't stop the hallucinated return policy? After Week 4, your Layer 1 would have flagged the missing citation. Your Layer 2 judge would have scored factual accuracy at 1/5. And your Layer 3 regression suite would have caught the drift the night before the deploy. The single metric to start tracking this week: percentage of agent outputs that pass your two most important rubric dimensions at a score of 4 or higher. If you don't know that number today, that's your first problem to solve.