Lessons from running a multi-agent assistant in production
We shipped a multi-agent chat assistant to production nine months ago. It routes customer questions across a handful of specialist agents, and most days it works quietly and well. This is what I’d tell myself before doing it again.
The setup
The system routes an incoming message to one of several specialist agents based on intent, lets that agent call tools and hold context, then hands the reply back through a shared formatting layer. On paper it’s a clean separation of concerns. In practice, the routing layer is where almost everything interesting happens.
@Service
class AgentRouter {
List<Agent> agents;
Response route(Intent intent) {
return agents.stream()
.filter(a -> a.supports(intent))
.findFirst()
.map(a -> a.handle(intent))
.orElseThrow();
}
}This is the version we shipped with. It’s honest about what it does, and that honesty is exactly why it held up better than a fancier LLM-based router once things started going sideways.
What broke
The failures that mattered weren’t wrong answers from a single agent — those are easy to catch and score. They were disagreements about which agent should have answered at all: two agents both confident, both partially right, both talking past the actual question.
We started logging routing confidence alongside every response and alerting on the gap between the top two candidates, not just on low confidence outright. That single change caught most of the incidents that used to surface as confused users instead.
Going simpler
We tried an LLM-based orchestrator that reasoned about which agent to call. It was more flexible and noticeably harder to debug at 2am. We rolled most of that logic back into explicit rules plus a narrow fallback, and reliability went up while flexibility barely dropped — most real traffic clusters into a small number of clear intents anyway.
The failures that mattered weren’t the agents giving wrong answers — they were agents disagreeing about who should answer at all.
Takeaways
- Log routing confidence, not just final answers — the gap between candidates is often more useful than the top score.
- Prefer boring, explicit routing until traffic proves you need something smarter.
- Treat agent disagreement as its own alert class, separate from low confidence.
- Every agent you add multiplies the routing surface, not the capability surface. Adding the fourth agent cost us more than adding the second and third combined.