Evaluating multi-agent systems: a practical loop with Google's ADK

Every enterprise agent program reaches the same moment: the demo works, leadership is convinced, and then someone changes a prompt. Is the system better now, or worse? Most teams cannot answer that question with evidence. That inability — not model quality — is what keeps agentic AI stuck between pilot and production.

This piece describes an evaluation loop that works in practice for multi-agent systems built on Google’s Agent Development Kit (ADK). It is deliberately minimal: a case format, a runner, deterministic checks first, LLM judges only where they earn their keep.

Why evaluation is the hard part

Traditional software has tests because behavior is deterministic. Classic ML has benchmarks because tasks are fixed. Agent systems have neither property: the same input can produce different tool-call sequences, and the “task” is an open-ended conversation with side effects. Meanwhile the things that change most often — prompts, tool descriptions, model versions, agent topology — are exactly the things with no compiler and no type checker.

The result is a system where regressions are silent. A prompt tweak that improves refund handling quietly breaks escalation behavior. A model upgrade that lifts reasoning quality doubles tool-call latency. Without a repeatable loop, every change is a bet, and the organization’s appetite for bets runs out long before the backlog does.

The shape of the problem

“Is the agent good?” decomposes into four questions that need separate instruments:

1. Task success — did the user get the right outcome?
2. Tool-call correctness — did the agent call the right tools, with the right arguments, and only those tools?
3. Coordination quality — in multi-agent setups: did handoffs occur at the right moments, did agents share state instead of re-deriving it, did any agent do redundant work?
4. Operational cost — tokens, latency, and retries per completed task.

Tool-call correctness and cost are deterministic and cheap to check. Task success and coordination quality usually need judgment. Order matters: teams that start with an LLM judge for everything drown in noise; teams that start with deterministic checks catch most regressions for free.

Building an eval dataset

The dataset is the asset with the longest half-life — models and prompts will change; good cases won’t. Three sources, in order of value:

Failures from real traffic. Every escalation, thumbs-down, and support ticket involving the agent is a case. These carry the distributional quirks synthetic data never has: ambiguous phrasing, mixed intents, wrong assumptions stated confidently.

Boundary cases from policy. Anything with an approval threshold, a compliance rule, or an irreversible side effect gets a case pair: one just inside the boundary, one just outside. This is where the must_not_call_tools assertion does most of its work.

Synthetic broadening. Generate variations of the above — paraphrases, locale changes, adversarial phrasings — but review them by hand. Fifty curated cases beat five hundred generated ones; a case whose expectation is wrong is worse than no case.

Expectations should describe observable behavior, not exact wording:

{
  "id": "refund-over-limit-001",
  "input": "I need a refund of $4,200 for order 88231",
  "expectations": {
    "must_call_tools": ["lookup_order", "check_refund_policy"],
    "must_not_call_tools": ["issue_refund"],
    "outcome": "escalates to a human approver because the amount exceeds the auto-approval limit"
  }
}

Wiring up ADK for eval runs

ADK’s event stream is what makes this tractable: every tool call, agent transfer, and final response is observable as the run executes. The harness runs each case in a fresh session, collects the events, and applies checks.

async for event in runner.run_async(
    user_id="eval-user", session_id=session.id, new_message=case["input"]
):
    for call in event.get_function_calls() or []:
        result.tool_calls.append(call.name)
    if event.is_final_response() and event.content:
        result.final_response = event.content.parts[0].text

Two details carry most of the value. First, capture everything, judge later: persist the full event trace per case as JSONL, so a scoring change never requires re-running the agent. Second, fresh state per case: shared sessions leak context between cases and make failures irreproducible.

The full harness — case loading, deterministic checks, result reporting — is about sixty lines; a copy-ready version is published as the ADK eval harness starter on the resources tab.

LLM-as-judge, carefully

Deterministic checks can’t score the outcome expectation, so a judge model compares the agent’s final response and trace against the expected outcome. Three rules keep judges honest:

Rubrics, not vibes. The judge prompt asks for a binary decision against a specific criterion (“does the response escalate rather than commit to the refund?”), never a 1–10 quality score. Scalar scores drift; binary criteria are auditable.

Calibrate before trusting. Label thirty cases by hand, run the judge, and measure agreement. Below roughly 90% agreement, fix the rubric — or the expectation — before wiring the judge into any decision.

Judge the trace, not just the text. A response that says “I’ve escalated this” while the trace shows an issue_refund call is a failure. Feeding the judge the tool-call sequence alongside the final response catches exactly the class of failure that text-only judging misses.

The failure modes that actually show up

Running this loop across real multi-agent systems, the recurring findings are rarely “the model got the answer wrong”:

• State re-derivation. A task agent re-queries what the root agent already fetched because the handoff didn’t carry state — invisible in demos, doubled latency and cost in traces.
• Premature handoff. The router transfers to a specialist before the case is qualified, so the specialist bounces it back — visible as transfer→transfer→transfer chains in the event stream.
• Retries masking broken tools. A tool failing 30% of the time hides behind agent retry behavior; success rates look fine while cost and latency degrade. Only per-case tool-error counts expose it.
• Instruction bleed. A prompt change to one agent shifts another agent’s behavior through shared context — the reason eval runs must cover the whole system, not the changed agent in isolation.

None of these are model problems. All of them are architecture problems that only a trace-level evaluation loop makes visible.

Closing the loop

The harness pays for itself the day it gates a change. Wire it into CI so that prompt, tool, and topology changes run the case set and fail the build when pass-rate drops. Track pass-rate per category (tool correctness, boundaries, coordination) rather than a single number — a flat aggregate hides a collapsing category.

Two disciplines keep the loop trustworthy over time. Add a case for every production incident — the dataset should grow the way regression suites grow. And resist tuning prompts against the eval set until it stops failing; when pass-rate hits 100% and stays there, the set has stopped measuring, and it’s time to harvest new failures from traffic.

Evaluation is not a phase after building the agent. It is the mechanism by which an agent system becomes buildable at all.