A cheap artificial intelligence (AI) model can answer quickly and still be the wrong first judge for a two-model system. The question is not just how often it is right. It is whether its confidence helps identify the cases that need another opinion. In a November 4 engineering report, Hightouch tested that distinction by combining the low-cost Jev model with Claude Opus. Its strongest lesson is narrower than the headline's fourfold cost saving: the difficulty of the data changed both the savings and the value of the confidence scores.
What the models were checking
Hightouch studied identity resolution, which means deciding which data records belong to the same person. Its models checked whether a profile had incorrectly combined multiple people. That mistake is called over-merging. Similar names, shared emails and typing errors can make it hard to distinguish a genuine match from a mistaken combination.
The team used pseudopeople, a tool that generates fictional population records, and added realistic errors. Its first test contained 1,000 identities: half represented one person and half were over-merges. The later test deliberately emphasized difficult cases, including similar people and conflicting details. These were experiments on generated data, not proof of performance on every customer database. Hightouch explicitly says that its examples contain no real people.
Separate the decision from the referral rule
The report's tables describe two different thresholds. One decides whether a profile should be classified as over-merged. Another decides which cases should be sent to Opus for a second opinion. Changing a classification threshold trades one kind of mistake for another; choosing a referral range changes how much work the stronger model receives. They are not the same control.
For the easier test, sending Jev cases in the reported 30–90% confidence range to Opus referred 22% of identities. The combined system achieved 98.8% accuracy at an observed cost of $0.88 per 1,000 identities. Opus alone achieved 99.0% at $3.86. The cheaper result was close, but it was not identical accuracy.
On the harder test, the same Jev referral range produced 95.3% accuracy at $2.94 per 1,000 identities, compared with 95.1% at $5.39 for Opus alone. The cost advantage was smaller. Hightouch also warns that its measured costs and response times depend on its setup. Neither table is a current price quote or a promise about another workload.
Why confidence needs its own check
Calibration means comparing a model's confidence with what actually happens. A score is useful for referral only if it helps separate more reliable decisions from less reliable ones. Hightouch observed that Jev's errors were more concentrated in uncertain cases, while GPT-6 Luna sometimes assigned high confidence to mistakes. On its harder test, the combined Luna-and-Opus system achieved 91.6% accuracy, below the Jev-and-Opus result. A strong second model did not make the first model's confidence irrelevant.
These are the authors' findings for this task, not a ranking of the models for all uses. Their conclusion that the pattern may help elsewhere remains a hypothesis to test. A content checker, fraud detector or support router would need evidence from its own task before inheriting the same referral rule.
A certification exercise that goes beyond the headline
The official AWS Certified AI Practitioner objectives include accuracy, precision, recall and business measures such as cost per user. This experiment is a useful reading exercise for those concepts, not an announcement of a new AWS feature or a claim about actual exam questions.
- Define the positive result. Here it is a profile judged to contain multiple people. A false positive wrongly flags a valid one-person profile; a false negative misses a mistaken merge. Write down the operational consequence of each before choosing a threshold.
- Keep confidence and correctness separate. Ask whether confident answers were checked against known answers, not merely whether the model supplied a score.
- Read quality and cost together. Compare the easier and harder test tables. State which system, dataset and referral rule a quoted saving belongs to.
- Ask what has not been demonstrated. The report does not establish that these results transfer to your records or that a small accuracy difference will persist on another sample. Treat transfer as a new evaluation question.
For further practice, use the Nex Arc AIF-C01 practice questions alongside the official objectives. Keep the takeaway precise: a cheap first check can reduce spending, but routing should be judged by the mistakes it catches and misses, not by how confidently it talks.