“The PoC for an agentic AI implementation isn’t delivering value. Let’s try a better model.”
That recommendation already assumes a diagnosis we haven’t established.
Reaching for a stronger model assumes model capability is what’s preventing the current implementation from delivering the expected results. It might be. But that’s still a hypothesis, which should be tested before it gets funded.
Here’s a concrete case:
A team runs a PoC where AI drafts customer-support replies. Drafting drops from 5 minutes to 5 seconds. But human support agents then spend 2 minutes checking the draft reply and 3 minutes rewriting it. End to end, a ticket takes about as long as it did before.
We know the time savings disappeared into review and rewriting. We don’t yet know why the human support agents have to do that work.
The answer changes what you try next.
If agents are rewriting drafts to add customer account history the AI never received, a better model probably won’t help much. It still won’t have the account history. What might help is giving it the context that it needs in the first place.
If the model already has the context and keeps using it incorrectly, testing a more capable model may be justified.
The symptom is the same in both cases, but the fix is different. Knowing the cause tells you whether to spend the next iteration on context or on the model. Without it, a model upgrade is just the most expensive guess.
So before choosing the next model, name the problem you expect it to solve, and the evidence that points there.
Loading comments…
Comments aren’t loading right now.
The post is unaffected; the comment service didn’t respond. Try again in a moment, or come back later.