We tested Jev on 400 feature decisions

Jev cut the cost and time of one scoring step. Its first version also credited features to the wrong product, so we held it and revised the decision rule.

Kevin Kho

Co-founder, KnitKnot

A frozen baseline and later observation arranged as a measured evaluation window

Evidence plate / researchObserved · sourced · reviewed

In one AI answer we tested, the model discussed a company’s forecasting product. Jev credited features defined for a different product from the same company. The answer never established that the named product had those features.

That decision sits inside KnitKnot’s scoring pipeline. For each feature in an AI vendor comparison, we need to decide whether the answer attributes it to the company we’re measuring. A competitor’s feature doesn’t count. Neither does a general statement about what vendors should offer.

We had been using a three-pass language model vote. TypeSafe’s Jev offered a cheaper way to return predefined choices: credited or not_credited for each feature. Our first trial confirmed the speed gain, then failed our consistency check.

Eight answers, 400 feature decisions

Our trial used eight saved AI answers from two technology companies: 400 feature decisions per repeat. We ran three paired repeats of the existing three-pass GPT route and a single Jev call, plus 12 written control cases. The Jev model was pinned to jev-1.13.0. Both routes saw the full feature definitions and answer text.

Measure on this trial	Three-pass GPT route	One-pass Jev route
Estimated cost per 1,000 executions of this step	$18.66	$0.44
Median time for this step	1.89 seconds	0.35 seconds
Same complete feature set across all three repeats	3 of 8 answers	5 of 8 answers
Written controls correct across three repeats	36 of 36	36 of 36

We measured this scoring step, not the time or cost of a complete KnitKnot report. The controls were written by the team that built the trial. Agreement with GPT checked compatibility; GPT’s vote was not a ground-truth accuracy score.

Our advance rule required the same complete feature set in at least 95% of repeated saved answers. Jev met that bar on 5 of 8. We held the initial version.

What the output got wrong

The product mix-up was one error. In another answer, a mention of longer forecasting horizons became credit for a dedicated long-horizon model. The answer hadn’t established that model existed.

Jev returned a valid credited value in both cases. A fixed set of outputs controls the format; it cannot verify the underlying claim.

We revised the decision rule to require evidence for the named product, model variant, or implementation. The model and full inputs stayed the same. After follow-up trials, we reviewed all 400 feature decisions against their source answers and definitions. Nineteen were clear positives, 377 clear negatives, and four remained ambiguous. The reviewer had seen earlier outputs, so this was not an independent blind audit.

Across three repeats on the 396 clear pairs, the selected Jev version made 12 false-credit predictions and missed none. The existing GPT route made 24 false-credit predictions and missed two. That’s 1,188 predictions over 396 pairs, not 1,188 independent examples. Jev still over-credited features. Eight answers from two companies cannot establish how it will perform elsewhere.

The job Jev gets

We use Jev for this narrow feature-credit decision, with the original answer and feature definition kept alongside the result. When a claim is disputed, we check its source and review the evidence.

Our scoring work uses several inspectable decisions instead of asking one model who won the comparison. A marketing or competitive intelligence team can trace a feature credit to the answer that produced it and see which product the answer discussed.

We selected the revised Jev route knowing that false credits remain possible. The measured savings apply to this step in a local trial. We have not measured a change in live report costs or confirmed its accuracy across industries.