How I built an LLM eval harness that took AI output from 24% to 97% valid

At Krofile, every digital business card is written by a model. A customer describes what they do, and the model returns a complete React component that renders live. That makes "did this prompt change help?" a question with real money attached: a worse prompt means broken cards, refunds and churn.

For a while we answered it the way most teams do. We'd change a prompt, generate a handful of cards, look at them and decide they felt better. This post is about replacing that with an evaluation harness, and about the bug in our own harness that taught me more than any of the wins.

What an LLM eval harness needs to answer

An eval harness for generated code has to answer two different questions, and mixing them up is the most common mistake I see.

1. Is the output valid? Does it parse, stay inside the allowed scope and pass the hard checks? This is deterministic, so no model should be judging it.
2. Is valid output good? Given two cards that both work, which one would a customer rather share? This is a judgement call, which is where an LLM judge earns its place.

So the harness runs in that order. Every candidate card first goes through six deterministic validator groups: schema, compile and AST allowlist, OKLCH contrast, semantic checks, content checks and size. Only cards that pass reach the judge.

Step 1: fix the inputs

The brief set is 34 real-world descriptions covering photographers, clinics, restaurants, agencies and solo consultants, plus 10 deliberately thin briefs ("I sell candles") to see how the pipeline copes with almost no information. The set never changes between runs. If you swap briefs between runs, you're measuring the briefs.

Prompts come from a versioned registry: 14 prompt families, each version immutable once published. A run records the exact prompt version, model ID and settings it used. Without that, a result six weeks later is just a number with no provenance.

Step 2: judge blind, and judge twice

For each brief, we generate one card with the candidate configuration and one with the production baseline. A judge model sees both, unlabelled, and picks the one a real customer would rather share. Then it judges the same pair again with the order swapped.

const first = await judge({ brief, a: candidate, b: baseline });
const second = await judge({ brief, a: baseline, b: candidate });

// Only a verdict that survives the swap counts
const verdict =
  first === "a" && second === "b" ? "win" :
  first === "b" && second === "a" ? "loss" :
  "split"; // position bias, or a genuine coin toss

Judges have a measurable bias toward whichever option they see first. Swapping the order and only counting agreements turns that bias into a split, which is honest, instead of a fake win.

The bug that inverted our rankings

Our first full bake-off ranked the models in an order that didn't match anything we saw by hand. The model that produced the nicest cards in manual testing came out near the bottom.

The cause was mundane: the harness never sent the effort setting. Production called the larger models at a higher effort level; the harness called every model at the default. We had carefully measured a configuration that no customer would ever see.

An eval only means something when it matches the configuration you actually ship.

After the fix, the ranking flipped. The strongest model won 23 of its decisive comparisons and lost none, and four models were signed off for customers. The lesson I now apply everywhere: before trusting any eval result, diff the harness's request against a real production request, field by field.

From 24% to 97% valid structured output

The second big result came from the deterministic half. We tried moving generation onto structured outputs (JSON schema in, JSON out). The first run was only 24% valid. Most failures were truncated responses and schema keywords the API rejected, not bad model behaviour.

Fixing the schema at the source rather than adding retries took the same path to 97% valid. Retries would have hidden the problem and multiplied cost; the eval made the real failure visible.

Telemetry: evals' other half

Evals tell you what should happen. Production telemetry tells you what does. Every model call logs the attempt, tokens, cache hits, cost and whether it failed over to the backup provider.

That logging caught something no eval would have: production had been calling a retired model ID that returned 404, and every request was quietly failing over to the backup provider. Cards still rendered, so nothing looked broken. Only the failover rate gave it away.

What I'd tell a team starting LLM evals

• Separate validity (deterministic checks) from quality (an LLM judge). Don't ask a model whether code compiles.
• Freeze the brief set and version every prompt. Record model, prompt version and settings with every run.
• Judge blind and swap the order. Count only verdicts that survive the swap.
• Diff your harness request against production before you believe a ranking.
• Instrument every production call. Failover rates and cost per attempt find problems evals can't.

You can see what this pipeline produces on the Krofile case study, including a live card generated by the system.