Designing an agent people can rely on

That is where I would start designing the experience: with what the person needs to accomplish, including what they need to understand before they can comfortably leave it alone. The conversation is part of that experience, along with the actions taken elsewhere and the work left for the person afterwards.

Consider a hypothetical support agent handling a return. It finds the order, checks the policy, and submits a refund request. Everything it attempted succeeds. Yet if its final message says the money has been refunded when the request is only awaiting processing, it has given the customer the wrong expectation. A successful action and an accurate explanation of that action need separate checks.

Anthropic makes a related distinction in its guidance on agent evaluation: the conversation records what happened during an attempt, while the outcome concerns the resulting state in the connected system. Its guidance for conversational agents includes evaluating both task completion and the quality of the interaction. That gives product teams a useful starting point, without establishing whether any particular experience is good. [1]

For the return, success needs a more careful definition than the conversation ending. Was the correct order selected? Did the refund request reach the payment system? Was the customer told what remained pending and when to expect an update? Payment may take several days, so the agent cannot always finish the entire process in one sitting. It can still make the current state clear and provide a way to follow it.

Once that expected experience is explicit, the data has something useful to explain.

Suppose customers keep returning to ask where their money is. Those contacts are a reason to investigate, rather than immediate proof that the agent needs to sound more reassuring. The payment might be late, the stated timeline might be wrong, or the confirmation might never have reached the customer. Reviewing the conversation alongside the refund record and the notification history helps separate problems that would otherwise look much the same.

Each explanation calls for a different change. A delayed payment needs a truthful update and a route to help. A missing confirmation needs a reliable notification. An unclear completion message needs to distinguish a submitted request from a completed payment. Rewriting the response before understanding the cause could make the conversation more pleasant while leaving the problem in place.

The same care applies to effort during the interaction. Repeated questions might reveal that an agent has lost information already supplied. They might also be necessary because there are two similar orders and choosing incorrectly would be costly. Counting questions cannot distinguish those situations; a review needs to establish what each question accomplished and whether the answer was already available.

I would want the product to preserve what someone has already explained, while making it easy to correct anything misunderstood. If the wrong item has been selected, changing it should not require starting the return again. Microsoft's human–AI interaction guidelines include efficient correction as a design concern; Google's People + AI Guidebook similarly treats a path forward after failure as part of the experience. These are established principles worth carrying into agent products. [2][3]

Recovery also changes what needs measuring. For a handover to a person, recording that the transfer succeeded says little about whether the customer had to explain the whole problem again. A useful review would examine whether the next person received the relevant context, what the customer had to repeat, and whether the issue was eventually resolved. The agent's run can end before the customer's work does.

There is a temptation to compress all this into a score. Shorter conversations, fewer transfers, and faster responses are convenient to compare, but each can improve for an unhelpful reason. An agent can shorten a conversation by skipping a necessary question. It can avoid a transfer by continuing with a problem it cannot solve. Those measures become useful when read alongside verified outcomes and a review of the interactions behind them.

Even then, the records leave gaps. Silence after a response might mean the person is satisfied, distracted, or looking for help somewhere else. A brief follow-up or a usability session can help explain behaviour the event log cannot. Being guided by data includes recognising when the available measurement does not answer the question.

For a proposed change, I would write down the improvement expected in the person's experience before choosing the measure. If the change is a clearer refund confirmation, the expectation might be fewer contacts caused by uncertainty about status. Testing that requires distinguishing those contacts from complaints about genuinely late payments, and checking that customers still know how to reach help. Where practical, a small randomised rollout would help separate the effect of the new design from differences in the customers or problems arriving that week. A lower contact rate would be encouraging only if the underlying issue was resolved and help remained accessible.

The difficult cases belong in that testing too: a payment service that does not respond, an order the agent cannot identify, or a customer who changes their mind halfway through. Google's guidebook points out that some errors remain invisible to users and therefore will not appear in their feedback. Deliberately testing such cases helps cover what production feedback alone can miss. [3]

An agent experience improves when evidence leads to a specific change that makes the person's task easier to complete. For the customer waiting on a refund, that might be a clear pending state, a dependable update, or a handover that carries their explanation forward. The test is whether they can get on with their day without having to keep managing the agent.

Sources

1. Anthropic, Demystifying evals for AI agents, 9 January 2026. Engineering guidance; not a controlled study of this article's illustrative workflow.

2. Amershi et al., Guidelines for Human-AI Interaction, CHI 2019. See the guideline on efficient correction.

3. Google PAIR, Errors + Graceful Failure, People + AI Guidebook. Design guidance on error diagnosis, recovery, and errors that user feedback may not reveal.

Editorial note: The return and refund scenario is illustrative. This draft reports no original deployment results or measured improvements.