We spent $0.05 testing Jev. Its calibration held. The pipeline around it didn’t.

Jev is real, cheap, and fast. Whether you can trust the confidence number it attaches to every answer — the property its whole pitch rests on — is exactly the thing we could test ourselves for pennies. So we did, and on our sample, that number held up.

01. Why test it at all

TypeSafe AI launched Jev on 15 September 2026: a model that returns no text at all — just typed answers (a choice, a score, or a yes-probability) with a confidence figure attached.[1] The launch numbers are large: up to 193.6× faster and 444.6× cheaper than frontier chat models, “zero hallucinations,” and — the claim everything depends on — calibrated confidence, meaning a stated 90% confidence should be right about 90% of the time.[1]

Every one of those performance figures is TypeSafe’s own, generated on evaluation workflows TypeSafe’s own team wrote.[1][3] Independent testing remains thin and small-sample.[3] TypeSafe itself recommends buyers verify calibration on their own data — and at $0.042 per million input tokens, that verification costs less than a coffee.[1]

02. What we did

We wrote a pre-registration first and then followed it — with documented deviations where reality intervened. The dataset: 200 publicly labelled review sentences (100 film, 100 business, 100 positive / 100 negative) from the UCI Sentiment Labelled Sentences collection, checksummed before the first scored API call.[2] One yes/no question per item — “does this review express a positive sentiment?” — five repeats each: 1,000 calls in total. A separate 20-case adversarial set probed hostile inputs: injections, non-English text, emoji, empty states. A third test compared end-to-end latency against a chat model answering the same question in JSON.

The harness, frozen sample, raw outputs, and checksums are published alongside this article in the repository — every number below can be re-derived from the raw logs.

03. What we measured

The consistency result is the quiet standout. Across five repeated calls on every one of 200 items, not a single item changed its answer’s direction — the median probability spread was 0.000. Six items were consistently wrong, which is the honest version of a stability claim: Jev repeats itself, including its mistakes. Three of those six carried stated confidence above 0.60 — two above 0.90 — so the model’s errors here are confident ones, not hedged ones.[1]

The latency claim holds in our conditions — 0.90 s median wall-clock from Lithuania — but we put reliability first deliberately. Our baseline pipeline failed to return parseable JSON on 67 of 200 calls (22–42% per run across four runs), with 120-second timeouts in every run; the ~6× speed ratio is computed only over the baseline calls that did parse. A clean comparison against a pinned frontier model would narrow the gap. The honest summary is the asymmetry: the typed endpoint answered everything, the JSON pipeline did not.

04. Where it wobbles

“Zero hallucinations” survives contact with a schema, because Jev cannot emit free text — that part is construction, not virtue, and TypeSafe says so itself.[1] What remains is confident wrongness, and our adversarial probes found it in specific, nameable places:

One injection landed and one did not — the inconsistency is the finding, not either individual verdict. Non-English input worked but with visibly degraded confidence. Nothing crashed, refused, or returned a malformed value in 1,020 calls.

05. What this cannot tell you

Sentiment is the easiest decision shape Jev handles. The vendor’s accuracy gap against frontier models appeared on invoice processing and incident triage — tasks we did not test.[1][3] Our calibration result is measured on 200 items in one domain, and only 4 of those items sat in the middle of the confidence range where calibration is hardest to earn — a 400-item, multi-family design was pre-registered but had to be reduced for scope, and we say so rather than let the smaller design look definitive. The baseline model ran through a free local gateway, not a pinned frontier API, so the speed and reliability comparison describes one practitioner’s pipeline, not a clean benchmark. And all of this was measured against jev-1.13.0 on one day from one location — a vendor can improve, or regress, any of it next month.

The practical takeaway: on this sample, Jev’s stated confidence was trustworthy — ECE 0.03, every pre-registered bin passing — and the failure we actually observed was in the ordinary chat-model pipeline beside it. Treat Jev as a fast, consistent, cheap signal generator, and still budget your first week of using it for calibration measurement on your own data, in your own domain — because a passed test on easy sentiment items is a reason to keep testing, not a reason to stop.