Say you’re paying an LLM to route every support ticket to the right team. You could put a cheaper model in front of it, let that model handle the tickets it’s confident about, and send only the rest to the LLM. That only works if “confident” means “correct”.
Post from 4 November 2026
I’ve been reading about one such model, Jev, which TypeSafe AI released in early access in September. It doesn’t write text. You give it a ticket and a list of teams, and it returns a probability for each team. TypeSafe pitches it as much cheaper than an LLM, though that’s its own claim.
A customer writes “I was charged twice and need this fixed.” The model comes back with something like billing: 0.94, its probability that this ticket belongs to billing. That’s confident enough to clear the threshold you set, so plain code sends the ticket to billing. A ticket where the top probability is lower goes on to the LLM or a person. Say 80 of every 100 tickets clear the threshold (my number, just to show the shape). The cheaper model still runs on all 100, but your LLM only sees 20.
So the cheaper model doesn’t have to be as good as the LLM on everything, because the hard tickets still reach the LLM. What I’d look at first is what happens when it’s wrong. A low probability costs you an extra LLM call. A wrong answer at 0.94 goes straight to code, and nothing in the pipeline checks it until someone downstream notices.
I’d want to know whether those probabilities are calibrated: when the model says 0.9, is it right about 90% of the time? I don’t know yet whether Jev’s probabilities are calibrated, so I’d test that on real tickets before trusting the threshold.
That’s a check before you launch. Once it’s live, I’d have someone review a random sample of the tickets it routed on its own. That won’t catch every mistake, but it tells you how many are getting through.
You’ll hear that two models means two things to monitor. True, and that’s why I’d measure the whole pipeline instead of the cheaper model alone:
- Mistakes that get through: wrong answers the cheaper model acted on without passing them to the LLM or a person, counted from that sample.
- Escalation rate: if most tickets get passed up anyway, you’re paying for both models on most of them and the savings shrink.
- End-to-end cost and delay: including the hard tickets, which now wait for both models.
So before you put a cheaper model in front of your LLM, decide how you’ll catch its confident mistakes.
Loading comments…
Comments aren’t loading right now.
The post is unaffected; the comment service didn’t respond. Try again in a moment, or come back later.