How I built an LLM eval harness that took AI output from 24% to 97% valid
A blind pairwise judge, a pinned prompt registry and one harness bug that silently inverted our model rankings. A practical guide to LLM evals for code generation.
Krofile - AI Digital Business Card
Describe what you do and Krofile AI writes a unique, on-brand business card as live React in seconds, validated, sandboxed and editable section by section.
Generative AI
Claude API
LLM Pipeline
AI Code Generation
LLM Evals
MCP Server
Prompt Caching
Sandboxed Rendering
Stripe Credits
Lead Capture
Analytics
Project Overview
Krofile is an AI digital business card platform. You describe what you do, and Krofile AI writes a unique, on-brand card as live React that you share by link, QR code or NFC, with lead capture and analytics built in. I led V2, the AI-first rebuild that went to production in October 2026, end to end: the web apps, backend, DevOps and AI.
I built the AI engine end to end: a Haiku pre-flight with 20 industry playbooks, Sonnet 5 generation behind a multi-model gateway with GPT-5 failover, prompt caching and per-call cost telemetry, a validate-and-repair quality gate, section-level AI editing, and a shared contract package plus sandboxed renderer that make it safe to run model-written code. I also shipped Stripe credit billing, Teams and an OAuth 2.1 MCP server, so customers' AI assistants can work with their leads directly.
Everything was measured with a 34-brief eval harness and a blind LLM judge. That is how four Claude models were approved for users and how the structured path went from 24% to 97% valid. I worked with Claude Code inside a multi-agent workflow I set up across the six repositories.
Before V2, I architected the original platform (an Express.js and Nest.js microservice backend behind the web apps) and led a team of five developers building subscriptions, promotions, QR analytics and automation workers. That version has its own V1 case study.
97%
Valid AI Cards in Evals
4
Claude Models Eval-Approved
12
MCP Tools Shipped
16
Validators Guarding AI Code
Try a Live Card
This is my own Krofile card, served live by the production renderer. Tap through it, save the contact or open the full card to see it the way recipients do.
Open the full cardKey Features
Core functionality and capabilities that make this project stand out
Prompt-to-Card Generation
Krofile AI runs a fast pre-flight on the brief to pick one of 20 industry playbooks, then writes the whole card as a React (TSX) component that renders live.
Self-Checking Quality Gate
Every card passes six validator groups (schema, AST allowlist, OKLCH contrast with auto-fix, content and size) and gets one targeted AI repair before a user ever sees it.
Element-Level AI Editing
Click any section and describe the change. Copy edits route to Haiku and structural edits to Sonnet, so a card is never regenerated from scratch.
Multi-Model Gateway & Credits
Sonnet 5, Opus 5, Fable 5 and Fable 5.1 behind one gateway with GPT-5 failover, prompt caching, per-attempt cost telemetry and a Stripe credit ledger that never charges for a failed run.
Safe Execution of AI-Written Code
A shared card-contract package (80 scope keys, 16 validators) and a cross-origin sandboxed iframe make it safe to run model-written React across all three renderers.
Evals & MCP Server
A versioned prompt registry and an eval harness with a blind LLM judge, plus an OAuth 2.1 MCP server with 12 tools so customers' AI assistants can work with their leads and analytics.
Technology Stack
Key Technologies I Used on This Project
AI & LLM
Claude Sonnet 5 · Haiku 4.5Claude Opus 5 · Fable 5.1OpenAI GPT-5 (failover)Anthropic SDKPrompt CachingStructured OutputsMCP (OAuth 2.1)
Backend
TypeScriptExpress 5MongoDBMongoose 8RedisZod
Rendering & Frontend
React 19 + ViteNext.js 16react-liveBabel ASTTailwind CSS
Billing & Analytics
StripePostHog
Infrastructure
AWS S3 + CloudFrontDigitalOceanVercelGitHub Actions
Testing
JestVitestPlaywrightfast-check
Challenges Overcome
• Running model-written React safely: closing validator bypasses (optional chaining, destructuring, reflection) and a sandbox handshake race across three renderers.
• Taking structured output from 24% to 97% valid by fixing truncation and the JSON-schema keywords the API rejected, at the source.
• Catching, through cost telemetry, that production was silently calling a retired model ID and failing over to GPT-5.
• Keeping evals honest: a harness bug that never sent the effort setting had inverted the model rankings until it was found.
• Shipping V2 to production with a rehearsed V1-to-V2 migration while existing customers kept their cards live.
Key Learnings
• Instrument every model call (tokens, cache hits, cost, failovers) before optimizing anything.
• An eval only means something when it matches the configuration you actually ship.
• Treat generated code as hostile: allowlists, validators and sandboxing, in depth.
• Ship behind flags whose off path behaves exactly like before.
• Agentic development with Claude Code plus adversarial review agents catches money and security bugs that green tests miss.
Writing about this project
Deeper dives into how parts of it were built.
Sandboxing AI-generated React code: four layers that make it safe to run
When a model writes the React your users see, treat its output as hostile. Here's the defence in depth we use: a shared contract, AST validators, a quality gate and a cross-origin sandbox.
Building a production MCP server with OAuth 2.1: lessons from shipping one
What it took to let customers connect Claude and ChatGPT to their own data safely: OAuth 2.1 with PKCE, dynamic client registration, scoped tools, live permissions and rate limits per connection.
Interested in This Project?
This project showcases practical development skills and problem-solving experience. Explore the features, review the implementation, or get in touch to discuss similar work.