Human-on-the-Loop: How Service Teams Can Keep Making AI Better

AI in customer service is moving from answering questions to getting things done. An AI agent can increasingly understand what a customer wants, figure out what needs to happen next and take action across different systems. That is a much bigger shift than simply making chatbots better, because it changes not only what AI can do but also how service teams need to work around it.

‍

Until recently, the big question around AI in customer service has been how much of the workload it can automate. That still matters, of course. But as AI starts handling more of the actual work, we think another question becomes just as important: how do you keep making it better?

‍

No matter how good an AI agent is when you launch it, customer service never stands still. Products change, policies change, new questions appear and customers find situations nobody thought about when the system was first set up. This is where the idea of Human-on-the-Loop gets interesting.

‍

So, what is Human-on-the-Loop?

‍

Human-on-the-Loop is a way of describing systems that can work independently while people supervise how they perform, set the boundaries and step in when needed. That is slightly different from Human-in-the-Loop, where a person is still directly involved in completing or approving the task.

‍

For customer service, though, we think there is another important part to it. The human should not only be there to supervise the AI, but also help make it better over time.

‍

Think about a fairly normal situation. An AI agent gets a customer request it cannot handle and passes it over to a person. The service agent figures out what is going on, helps the customer and closes the case. The immediate problem is solved, but the handover also tells us something useful about the AI.

‍

Maybe the information was missing from its knowledge. Maybe the information was there but unclear. Perhaps the AI understood exactly what needed to happen, but did not have access to the system needed to do it. Or maybe this was genuinely a situation where someone needed to use human judgement.

‍

Those are very different problems. If the organization can understand which one it was, the handover becomes more than a handover. It becomes a clue about what the AI needs to get better at next.

‍

Building a continuous improvement loop

‍

Continuous improvement is hardly a new idea in customer service. Good teams have always reviewed conversations, updated knowledge, fixed processes and trained people based on what they learn from customers. AI makes that loop much more powerful because improvements to the system can scale immediately across future interactions.

‍

If a person learns how to handle a new type of problem, that learning normally has to spread through documentation, training or experience. If the AI itself is improved, one change can potentially affect every similar interaction that comes afterwards.

‍

A missing answer can lead to better knowledge. A recurring misunderstanding can lead to clearer instructions. A process the AI understands but cannot complete can point to an integration or action that is missing. And a situation that really should be handled by a person can help define a better escalation rule.

‍

The basic loop is simple: serve customers, see where the AI struggles, understand why, improve the system and check whether it actually got better. Then repeat.

‍

‍

This also changes how useful customer conversations become. They are not just a history of tickets that have been opened and closed. Together, they show you what customers are trying to achieve, where the AI is getting stuck and where the service operation itself might need to improve.

‍

That matters even more as AI moves from answering to acting. Gartner recently found that 58% of AI users had already used AI to complete a task on their behalf, rising to 74% in B2B. In other words, people are getting used to AI not only telling them what to do, but actually doing things for them.

‍

Once an AI starts making changes, submitting requests or taking action in other systems, improving how it behaves becomes just as important as automating the work in the first place.

‍

When AI starts helping improve itself

‍

There is an interesting parallel happening in AI research right now. The leading AI labs are increasingly using AI to help build better AI. Anthropic has described its work toward what is known as recursive self-improvement, the idea that AI can take on more of the research, coding and experimentation involved in developing more capable AI systems. OpenAI is also evaluating how useful its models are at tasks involved in improving AI development itself.

‍

We are obviously not suggesting that a customer service AI should start rewriting itself without oversight. But the underlying idea is relevant: AI can increasingly take part in identifying how it could become better.

‍

Imagine an AI agent notices that the same type of refund request keeps ending up with a person. It can help analyse those conversations and work out why. Maybe a policy is missing, the instructions are confusing or it knows what needs to happen but cannot access the billing system.

‍

The AI can identify the pattern and suggest what might need to change, while a person still decides whether that change should actually be made. Should new knowledge be added? Should the instructions change? Does the AI need access to another system? Should it be allowed to perform a new action, or is this a type of case where human involvement should remain mandatory?

‍

This creates a human-governed improvement loop where AI can help identify problems, suggest improvements and evaluate whether a change produced a better result, while people remain responsible for how the system is allowed to evolve.

‍

That matters for both quality and accountability. As AI takes responsibility for more work, it becomes increasingly important to know who decides what the system can do, what information it can rely on and how its capabilities are allowed to change.

‍

The human role changes as the loop changes

‍

As AI handles more customer interactions, having a person approve every single answer or action quickly stops making sense. If an AI is dealing with tens of thousands of conversations, adding a manual approval step to every one of them would remove much of the point of using AI in the first place. That does not mean human accountability disappears, but it does change where that accountability sits.

‍

Instead of being directly involved in every interaction, people can define which knowledge the AI should trust, which systems it is allowed to use, which actions it can take independently and where human involvement should still be required. They can also decide which improvements the AI is actually allowed to make.

‍

This becomes especially important as AI moves from answering questions to taking action in the systems where the actual work happens. An incorrect answer is one thing. An incorrect action can have very different consequences, which makes clear ownership over how the AI operates and evolves increasingly important.

‍

Human-on-the-Loop is therefore not about removing people from customer service. It is about applying their expertise at a different level. Instead of solving the same problem over and over again, a service team can spend more time making sure the AI handles it properly next time. Failed conversations and edge cases become useful signals for understanding what the system needs to improve next.

‍

The people closest to customers are particularly important in that process. They understand why a technically correct answer can still be unhelpful, which exceptions actually matter and where a process that looks simple on paper breaks down in reality. In a Human-on-the-Loop setup, that knowledge should continuously feed back into the system.

‍

Maybe automation rate is only half the story

‍

Automation rate will continue to matter. If an AI agent can resolve more customer requests, that creates real value. But it only tells you what the AI can handle at a particular point in time.

‍

Imagine two companies using roughly the same AI. One deploys it, tracks its automation rate and sends everything it cannot handle to a person. The other looks closely at those same handovers, works out why they happened and keeps improving its knowledge, instructions, integrations and processes.

‍

After six months, those two companies may be getting very different results, even if they started with the same underlying models. The difference is not only how good the AI was on day one, but how quickly the organization was able to learn from what happened afterwards.

‍

That is why we think the next big question in AI customer service may not simply be how much can we automate? It may increasingly be how fast can we make the AI better?

‍

Human-on-the-Loop gives us a useful way to think about that shift. AI handles more of the work and increasingly helps identify where the system can improve. People decide what should change, set the boundaries and remain accountable for those decisions.

‍

At Ebbot, we think that continuous improvement loop will become an increasingly important part of how modern customer service is built.