Someone at work asks whether AI could handle this for you, and the demo makes it look easy. But a draft that appears in 5 seconds doesn’t mean the work got done any faster. Here’s how I’d judge AI on your own work: by the kind of task, the context the model gets, and what happens to its output afterwards.
This video uses a customer support pilot and a coding-agent ticket to show why AI often moves work rather than removes it, what the DORA and METR research says about that, and five checks to run before you reach for a better model.
Transcript
00:00 Could AI handle this for us?
So someone at work asks you a simple question. Could AI handle this for us? Maybe it’s customer support replies, maybe it’s code reviews, or test generation. You’ve seen the demo where it writes a perfect answer in a few seconds. You’ve also read the threads where engineers say they were let down. And you realize you can’t actually answer this question. You can’t tell what AI can do reliably, or how much of what you’re hearing is just hype.
Does that sound familiar? You’re not the only one. On Hacker News, someone asked engineers how AI-assisted coding was going for them. The thread drew more than 600 comments. That’s a lot of engineers trying to sort out what’s actually working and what isn’t. I think the question “how good is AI” doesn’t have one answer. It hides three smaller questions: what kind of task it is, what the model was given to work with, and what happens to its output afterwards. Now, let’s look at all three through one example.
00:59 The support pilot: 5-second drafts, same 8-minute ticket
Imagine a team runs a pilot where AI drafts customer support replies. Before the pilot, a customer support representative reads the tickets, writes the reply, and sends it. Writing the reply takes about 5 minutes. With AI, the draft appears in about 5 seconds. That looks like a huge win, and if you stop measuring it there, you would report it as a win.
But the customer support representatives don’t just send the draft as it is. They spend about 2 minutes checking it, and about 3 minutes rewriting it. So end-to-end, a ticket takes about as long as it did before. So, did the AI really improve the workflow here? The model did exactly what it was asked: it produced a draft, fast. The workflow didn’t get faster at all. So it looks like AI often moves work rather than removes it.
01:50 Question 1: what kind of task is it?
But the first question we need to ask here is: why are human agents rewriting it? One possibility is because of the kind of task. Language models generate answers token by token, based on patterns in their training data. They don’t actually compute. For example, ask a model to multiply two large numbers, and what you find is that it isn’t doing the arithmetic, it’s predicting what the answer looks like.
Surprisingly, that works most of the time. And then it fails unpredictably on some edge cases. That’s why a model can write a fluent reply and still get a precise calculation wrong. Agent builders often handle this with a tool for the parts that require an exact answer. It’s usually called a code interpreter, a small, isolated place where the model can run code. So instead of guessing the number, the model writes a few lines of code that calculate it.
The code runs, a real computer does the arithmetic, and the model puts that exact result into its answer. The model does the language part, understanding the request and writing the code. The computer does the calculation parts that require an exact answer. So as you can see, the same model can be very reliable on one part of a task and unreliable on the next part. So how good is it? Depends on which part you’re asking about.
03:17 Question 2: what did the model have to work with?
Another possibility is a context. Suppose the human agents are rewriting drafts to add the customer’s account history. The model never received that history. It wrote a reasonable reply to a ticket it only half understood. A better model probably won’t help much there, it still won’t have the account history context. What might help here is giving the model the context it needs in the first place.
Now suppose the model does have the context about the account history and it keeps using it incorrectly, then testing a more capable model may be justified here. So the symptom is the same in both cases, as you can see here, but the fix is different. Which is why saying that AI isn’t good enough is a guess until you know which one you’re looking at.
04:10 Question 3: what happens after the output?
The third question is the one demos usually skip: what happens after the output is generated? In the pilot, the AI step got faster, while checking and rewriting took up the time it saved. The same thing often shows up in software. Generation gets faster, while review, verification, and rework grow.
Say, a ticket normally takes an engineer 230 minutes from start to finish. Implementation is 120 of that. With a coding agent, generating the implementation takes 25 minutes. Now, commonly that’s the number that ends up on a PowerPoint slide. Something like “Coding got 79% faster”. But now someone spends 35 minutes inspecting the generated code. Review and rework both take longer too. So end-to-end, the ticket takes 205 minutes instead of 230. That’s an 11% improvement, not 79%. 11% might still be worth it, but it’s definitely a different conversation from 79% gain.
05:18 AI as an amplifier (DORA)
DORA stands for DevOps Research and Assessment. It’s a long-running research program at Google Cloud that studies how software teams deliver and run software. Its 2025 report describes AI as an amplifier. It magnifies the strengths of high-performing organizations and the dysfunctions of struggling ones. In other words, the same AI can make a strong team stronger, while also make a struggling team’s problem bigger.
For example, a team with good practices, such as clear requirements, solid tests, and quick reviews, get more out of AI, while a team with weak practices get its problems magnified, more code piling up in review, more rework. What you get out of it depends a lot on the work around the model, not only on the model itself. That’s why the speed of the AI steps, like that 79%, isn’t a reliable guide on its own. And your own sense of how fast you’re going isn’t a reliable guide either.
06:27 Felt faster, measured slower (METR)
There’s a study that showed this very well. It’s from METR, Model Evaluation and Threat Research, a nonprofit that tests what advanced AI model can do. In early 2025, they worked with 16 experienced open-source developers. Each of them had worked on large, mature projects for years. The developers brought real work from their own projects, like bug fixes, features, and refactors, 246 tasks overall. Each task was randomly assigned. Either AI was allowed, or it wasn’t. Then they measured how long each task took.
Before starting, the developers expected AI to make them about 24% faster. When the researchers measured, tasks with AI took about 19% longer, and afterwards, the developers still believed AI had sped them up by about 20%. That gap is the point I care about here. These were experienced engineers, working on code they know well, and their sense of how much AI helped them pointed the wrong way.
Now, there are some caveats I should call out here. It was a small study, with the tools of early 2025. METR itself doesn’t claim these developers represent most software work. Its later works actually suggest today’s tools probably help developers like these more. But METR calls that evidence weak. And a much larger study found gains. In field experiments at three companies, with nearly 5,000 developers, those given an AI coding assistant completed about 26% more tasks. So less experienced developers gained the most.
So, I wouldn’t take this as AI makes you slower. I would take this as: how fast it felt isn’t evidence. You have to measure it.
08:24 How to measure it on your own task
Here’s what I would try, on one task your team actually does. First, write down the whole workflow before AI, with rough times for each step. That’s your baseline. Second, measure where useful work leaves the workflow: the reply sent, or the change in production, not the step where the model produces output.
Third, watch the handoff. How long do people spend checking? How often do they rewrite stuff? And be careful with acceptance rate. If people accept almost everything the AI suggests, that could mean the AI is very good, but it could also mean people stop checking altogether.
Fourth, if it underperforms, find the first place it breaks before reaching for the better model. Go through the following five checks, in order, and stop at the first one that fails. Is the output good enough? Do people actually use it? Does the workflow get better? Does the outcome you care about get better? Is it worth what it costs? Each of these questions is necessary. And none of the earlier ones proves the later ones.
Fifth, decide where a person actually needs to check. Putting a human on every output can wipe out the gain. Two things can help here. How ambiguous the task is, and how costly a mistake would be. For a coding agent, AI can do more on its own with formatting, boilerplate, and test scaffolding. Human judgment matters more for things like authentications, schema migrations, and security boundaries.
10:02 Did it save the work, or just move it?
Now measuring the whole workflow takes more effort than timing the AI step. I know that, but think about what that one step’s number tells you. The draft that appears in 5 seconds, a coding that got 79% faster. Each of those numbers measure one step, the moment the AI produces output. It doesn’t tell you what kind of task that step was. It doesn’t tell you what the model had to work with. And it doesn’t tell you what happened afterward, like the checking, the rewriting and the review. In the pilot, that’s where the time went.
That’s the whole idea of this video. AI isn’t one skill level. It depends on the task, the context it gets, and the work around it. The speed of one step can’t answer any of those for you. So next time someone asks you, “Could AI handle this for us?”, or show you a number like that, don’t stop at how fast the AI step was. Remember why the human agents in the pilots was rewriting. The drafts weren’t good enough to send as they were. Whatever the cause, that’s what “unreliable” looks like from the inside. And ask the question the support pilot left us with. Did it save the work? Or did it just move it?
I hope you find this video useful. If you like more on judging AI in your own work, like this video or drop a comment below, so I know what content serves you well. Thank you, and see you next time.
Loading comments…
Comments aren’t loading right now.
The post is unaffected; the comment service didn’t respond. Try again in a moment, or come back later.