Cheap verification is your biggest unlock when building ML products
You’re three weeks into a client POC. The demo went well - the outputs look plausible, the stakeholders are nodding. Your pipeline takes a document, extracts structured data, runs it through a few model calls, and produces a result. It mostly looks right.
But “mostly looks right” is doing a lot of work in that sentence.
Every morning, someone on the team pulls a sample of outputs and reads through them. They flag the ones that seem off. You fix something, re-run, read through again. The cycle is slow - not because the models are slow, but because you are the verification step. Your eyes, your judgment, your time.
You can’t run 500 experiments when each one requires a human to sign off. So you run five, call it a sprint, and hope the trend is pointing in the right direction.
This is the default state of most ML product development. It’s not a tooling problem, or a data problem, or a model problem. It’s a verification problem.
The teams that move fast have figured out how to get a cheap, reliable signal that tells them whether their pipeline is working. Not perfect. Not human-level accurate. Just cheap enough to run at scale, honest enough to catch regressions, and fast enough to give feedback in minutes rather than days.
That signal is the thing worth building before anything else.
The hidden variable
When you look at where AI has genuinely succeeded - not “impressive demo” succeeded but “reliably deployed at scale” succeeded - a pattern emerges.
Chess engines. Protein folding. Code generation. Math reasoning. Recommendation systems. These are the domains where AI has compounded over decades, where each generation of models meaningfully outperformed the last.
Compare that to: open-ended research agents, general-purpose assistants for complex decisions, systems that synthesize across large ambiguous corpora and produce actionable judgment. These are the domains where progress is slower, deployment is shakier, and the gap between demo and production is widest.
Did the chess engine win? Yes or no. Does the code compile and pass the tests? Yes or no. Is the predicted protein structure consistent with experimental data? Checkable. The feedback loop is tight, unambiguous, and scalable. You can run a million games, a million test cases, a million structure predictions - and every single one tells you something true.
In open-ended research, creative synthesis, strategic judgment - there’s no equivalent check. The output either seems right or it doesn’t, and “seems right” is a function of whoever’s reading it, under whatever time pressure, with whatever domain knowledge they happen to have that day.
This isn’t a new insight. Andrej Karpathy put it plainly: the eval IS the product. Not the model. Not the prompt. The evaluation. Without a reliable signal, you’re not doing engineering - you’re doing intuition. You change something, it feels better, you ship it. Maybe it is better. You won’t know until production tells you, which is a brutal way to iterate.
You can’t improve what you can’t measure. And you can’t measure what you can’t verify.
This goes deeper than engineering
The AI safety community has a name for this: scalable oversight. Paul Christiano framed it precisely - the problem of correctly rewarding intended behaviours even when those behaviours are beyond humans’ ability to efficiently judge. That’s not a superintelligence problem. It’s the problem you hit at week three of a POC.
The consequence is reward hacking: models optimize for whatever signal you give them. If the signal is a proxy for quality rather than quality itself, the model finds the gap. A weak verifier doesn’t just slow you down - it actively shapes the model toward satisfying the metric, not the underlying task. Goodhart’s Law, applied.
Verifier quality sets the ceiling on what you can build. Which brings us back to the question every ML team is quietly asking: where does that signal come from?
Some AI research approaches that didn’t work
Between 2022 and 2024, there was a wave of serious attempts to build AI systems that could do deep, open-ended work. Smart people, well-resourced teams, compelling demos. Some of them failed in production - not because the models were bad, but because the systems had no way to know when they were wrong.
AutoGPT and BabyAGI
The idea: give the model a goal, let it spawn its own subtasks, prioritize them, and loop until done. True autonomy.
In practice, they’d loop indefinitely on trivial things, confidently pursue wrong directions, and had no mechanism to know when they were finished. The “agent decides what to research next” part required the model to be its own supervisor - to judge whether its own output was good enough to stop. It wasn’t. GPT-4 at the time was not reliable enough to evaluate GPT-4.
Without an external check - a test that passes, a score that converges, a human who says “yes, that’s right” - the loop had no ground truth to anchor it. It optimized for activity rather than correctness.
Multi-agent debate
The proposal: use several models. One researches, one critiques, one fact-checks. Let them argue, surface truth through disagreement.
The failure mode was subtler. Models turned out to be sycophantic under pressure from other models. The critic would raise a valid objection. The researcher would agree and revise - even if the original answer was correct. They’d converge on confident-sounding consensus that wasn’t necessarily accurate.
Each model was optimizing to appear agreeable, because that’s what training had rewarded. The debate had no external referee - no ground truth to appeal to when the models disagreed. So they stopped disagreeing.
RAG at scale
Early retrieval systems tried to ingest massive corpora into vector stores, then retrieve relevant chunks for any query.
Semantic similarity search pulled chunks that were topically adjacent but not actually relevant to the specific question. You’d ask about a niche aspect of a topic and get back the most popular chunks about the general topic. The chunks looked right. There was no automated check to verify they were actually useful.
Teams spent months on chunking strategies, embedding models, reranking pipelines. Results were still inconsistent enough that it never became reliable for deep research - because the fundamental problem wasn’t retrieval quality. There was no cheap signal to tell you whether the retrieved context was actually answering the question.
Recursive summarization
For large document collections: chunk, summarize each chunk, summarize the summaries, repeat.
Works for compression. Fails for research. Early summarization steps lose details that turn out to be important later - but there’s no signal that tells you what was lost. If the answer to your question lives in a paragraph that got summarized away in round one, it’s gone. The final output still reads as coherent and complete. You won’t know anything is missing.
The common root
Every one of these failed for the same reason: errors compounded silently. Unlike code - where a test either passes or doesn’t - there was no mid-pipeline check that could catch a wrong turn early. The system would proceed through a chain of plausible-looking steps and produce a plausible-looking output that was wrong in ways no one had instrumented to detect.
The scaffolding was trying to reason for the model. As models got stronger, simpler approaches started outperforming elaborate ones. Perplexity - essentially search plus one-shot summarization - works better in practice than AutoGPT ever did. Not because it’s smarter. Because it’s shorter, and a shorter chain has fewer places to silently go wrong.
The map is almost too clean
Once you see the verification pattern, it’s hard to unsee it.
Where AI has compounded fastest - code generation, math, games, structured data extraction - you can verify the output automatically. Tests pass or fail. The proof checker doesn’t care how confidently the model stated an incorrect step. AlphaGo didn’t need a human to read its moves and say “that felt like a good game.” The board told it.
Where progress is slower and deployment is shakier - open-ended research, strategic advice, creative judgment - verification is expensive or impossible. Is this synthesis accurate? Experts disagree. Was this the right recommendation? You won’t know for months. Is this good writing? Depends who’s asking.
The map overlaps with model capability, yes. But it overlaps more cleanly with verification cost.
The industry’s answer: manufacture the signal
Tools like empirical.run, Braintrust, and promptfoo exist because the industry hit this wall and decided to build around it: if your domain doesn’t naturally produce a verification signal, construct one. In practice, you build a dataset of representative inputs, score outputs automatically with rule-based checks or a judge model, and gate deploys on the result - a prompt change that degrades quality fails the build, not the demo. This turns vibes-based iteration into something that compounds: instead of hoping a change is better, you run it against 500 cases and read a number.
The twist
Most of these tools, when you need to evaluate something subjective - does this summary capture the right information, did the agent take the right steps - reach for the same solution: LLM-as-judge. Use a stronger model to score the outputs of your pipeline.
It works. Though depending on the domain, you may have to look at it closely.
- Research at NeurIPS 2024 found that models have a measurable self-preference bias - they rate outputs that resemble their own style more favorably, and the correlation between self-recognition capability and self-preference strength is linear. Studies have catalogued at least 12 distinct bias types: verbosity bias (longer answers score higher regardless of quality), authority bias (confident-sounding outputs score higher), position bias (the first option in a comparison tends to win). A nonsense response, crafted to sound authoritative, can consistently fool a GPT-4 evaluator.
- There’s also preference leakage: when the model generating your outputs and the model judging them are trained on related data - which they almost always are - the judge’s preferences are contaminated by familiarity with the generator’s patterns. The eval is no longer independent.
LLM-as-judge moves the problem up one level rather than solving it - but for most domains, it’s still the cheapest reliable verifier you’ve got. Know the biases, calibrate regularly, and it holds up.
What we actually did
At Setu, we were building an ML pipeline for a client POC - document ingestion, extraction, a few model calls in sequence, structured output at the end. The kind of pipeline where each step looks reasonable in isolation and errors accumulate quietly across them.
Early on, we were doing what everyone does: sampling outputs manually, reading through them, flagging the ones that seemed off. It was slow. More importantly, it was a bottleneck on experimentation. Every time we changed a prompt, adjusted a model, or tweaked the extraction logic, someone had to sit down and re-evaluate a sample by hand. We were running maybe three or four real experiments a week.
We knew LLM-as-judge had its limitations and weren’t under any illusion that a model grading model outputs was the same as ground truth - but we used it anyway, and it worked. Not perfectly, but enough.
We set up a larger model to score outputs against criteria we’d defined - extraction completeness, field accuracy, schema compliance - against a fixed dataset. Every experiment produced a number. Manual review didn’t disappear, but shifted: instead of reading outputs to judge quality, we read them to calibrate the judge.
That shift changed the economics. Four experiments a week became dozens. The POC landed in two weeks.
You don’t need a perfect verifier. Perfect is not the point. The point is replacing five slow, expensive, inconsistent human evaluations with five hundred fast, cheap, consistent automated ones - knowing they’re imperfect, calibrating them regularly, and using the headroom to run experiments you otherwise couldn’t afford.
The teams shipping reliable AI products aren’t the ones with the best models. They’re the ones who figured out how to get a signal - any signal, cheap enough to scale - and built their iteration loop around it.
Find that signal first. Everything else compounds from there.