Designing Agent Products Beyond the Interface

I started following the word “agentic” out of curiosity. Over the past few years, I have also had the opportunity to work on AI products, including products focused on agentic cost governance. That combination gives me a particular interest in what happens between an impressive demonstration and a product someone can afford to keep using.

It is easy to describe an agent product through its interface: a conversation, a workspace, a sequence of actions. But much of the experience is decided before anything appears on screen. Which model gets the task? How much context does it receive? When does it search, retry, ask a question, or stop?

These decisions determine what the user waits for, what they can trust, and what the business pays to deliver it.

Human-centered design gives us a useful foundation here. It would be unfair to suggest that earlier products were only about frontend design, or that product managers previously enjoyed a world without economic constraints.

Don Norman makes that explicit in the preface to the revised The Design of Everyday Things:

“To understand products, it is not enough to understand design or technology: it is critical to understand business.

What agents change is how directly some of those constraints enter the interaction. A decision to spend less on inference can change the answer. A decision to check more sources can change the waiting time. A decision to avoid asking the user can change the probability of taking the wrong action.

I find three dimensions particularly useful for thinking about this: cost, performance, and accuracy. They do not explain every part of the experience, but they force a product team to confront tradeoffs that a polished interface can easily conceal.

Cost: follow the work through to completion

Cost per model call is useful infrastructure information. For a product decision, I want to know what it costs to complete a task to an acceptable standard.

Consider a hypothetical support agent handling a refund request. Its first response may be inexpensive. But completing the request might require retrieving policy, identifying the transaction, checking eligibility, asking a clarifying question, and recording the outcome. Failed attempts and repeated tool calls still appear on someone's bill.

The denominator matters. Dividing operating cost by the number of responses rewards producing responses. Dividing the cost of all attempts by the number of successfully completed tasks tells us more about the economics of the service we promised.

Human review deserves a separate line in that calculation. If the customer performs it, the expense may never enter the vendor's gross margin calculation. It still enters the customer's decision about whether the product is worthwhile.

There is also a commercial consequence. A feature can be valuable to users and still be difficult to sustain if heavy usage costs more than the revenue it supports. Pricing, task limits, and service levels are part of the product promise. Successful-task economics helps assess that promise; it does not, by itself, establish profitability.

Performance: measure the time until the user can move on

For an agent, “fast” needs a definition. The time until the first word appears is different from the time until the user has a usable result. Both are different from the time until the work is actually complete.

A response that appears in two seconds and needs five minutes of correction may be slower in practice than a response that takes twenty seconds and needs little intervention. The right comparison depends on the task and how much attention the product demands along the way.

I would therefore measure completion time alongside active user time. I would also inspect the slow cases: how often a normal task gets stuck, how long it remains unresolved, and whether the user can recover without starting again.

That changes the design brief. If the work will take time, useful progress might mean showing which sources have been checked, which actions are complete, and what is blocking the next step. A stream of reassuring sentences is a poor substitute for information the user can act on.

Some work should finish immediately. Some should continue in the background. Some should pause for a decision. Product judgment includes choosing the right mode, rather than treating every additional second as equally harmful.

Accuracy: define what it means to be right

Accuracy sounds straightforward until the agent can act.

In the refund example, a response can quote the correct policy while applying it to the wrong transaction. The amount can be correct while the action violates an approval requirement. The explanation can be excellent while the refund is issued twice.

I would treat answer accuracy as one part of a wider reliability requirement: the right outcome, for the right object, within the user's authority, with the important constraints respected.

Errors also have different consequences. A weak suggestion for a document title and an incorrect payment instruction should not share one undifferentiated success percentage. Before choosing an accuracy target, the team needs to decide which failures are tolerable, which require review, and which must be prevented by controls outside the model.

There is useful history here. In his 1999 paper Principles of Mixed-Initiative User Interfaces, Eric Horvitz described systems that decide whether to act, ask, or wait by considering uncertainty alongside the expected costs and benefits. The interruption itself has a cost; so does a mistaken assumption.

That is a more useful starting point than asking an agent to be maximally autonomous. A question can be the best next action. So can preparing a reversible draft and leaving the consequential step to the user. Applying this principle requires evidence about likely outcomes, rather than trusting a model's confident description of its own certainty.

But adding an approval button does not finish the job. In Ironies of Automation (1983), Lisanne Bainbridge wrote:

“the more advanced a control system is, so the more crucial may be the contribution of the human operator.

Her subject was industrial automation. The relevant warning for agent products is that automation can leave people responsible for difficult exceptions without adequately supporting them. A reviewer needs the evidence, context, and time to make a real decision. Otherwise, “human review” is an optimistic description of another click.

The balance depends on the task

These dimensions are coupled. More checking can improve correctness while increasing latency and expense. A cheaper model can reduce inference cost while generating more rework. Asking a question can interrupt the user while preventing a much more expensive mistake.

Still, I would resist drawing a triangle and declaring that every improvement requires an equal sacrifice elsewhere. Better architecture can change the available choices.

My preference would be to define a minimum acceptable outcome and task-specific limits on cost and waiting time, then compare approaches inside those limits. The business needs a service it can sustain. The user needs to know what level of service they are receiving.

People experience behavior, too

Cost, performance, and accuracy do not fully explain why someone enjoys using an AI product. How it responds also matters: whether it is patient, concise, dismissive, overconfident, or able to challenge an assumption without turning the conversation into an argument.

It is tempting to explain enthusiasm for AI products through emotional attachment. I would be careful with that claim. In research published in 2025, OpenAI and MIT Media Lab found that emotional engagement was uncommon across the ChatGPT conversations they assessed and concentrated among a small subset of heavy users. Outcomes varied with usage and personal circumstances. The initial research also had limitations, including its duration and reliance on self-reported measures.

My interpretation is narrower: familiarity, response style, and a sense of being understood can matter, but they should not become a universal explanation for why people use these products. Someone may return because the product helps them work. Someone else may value the conversation itself. We need to understand which relationship our product actually supports.

There is a further complication: a response can feel good while making the product worse. OpenAI's account of its April 2025 GPT-4o sycophancy rollback is instructive. Offline evaluations looked positive, and a small A/B test suggested users liked the update. Yet the combined changes produced excessively agreeable behavior that the launch process had failed to catch.

For me, that is a product lesson about interpreting feedback. Warmth is valuable. Agreement is sometimes appropriate. Neither establishes that an agent is helping someone make a better decision.

This also means that a model change can be a substantial experience change even when every screen remains familiar. If a cost optimization changes the agent's patience, its willingness to ask questions, or its handling of uncertainty, users are receiving a different service.

Where I would start as a product manager

Working on AI products and agentic cost governance has made me more careful about equating technical efficiency with product value. A lower bill matters, but the work delivered for that bill needs to hold up. Savings become harder to defend when the user absorbs the difference through extra waiting, checking, or correction.

A useful starting point is one recurring task, with a clear definition of acceptable completion. That includes the expensive mistakes, the actions that need permission, and the amount of review the user can reasonably take on. These decisions give model selection and workflow design something concrete to serve.

Comparing implementations on the same tasks then becomes more meaningful. The cost of failed attempts belongs alongside successful completion, serious errors, elapsed time, and human correction effort. Averages can reveal a pattern; examining individual failures helps explain what produced it.

My preference is for products that use attention carefully. One necessary question early can prevent an elaborate result built on a bad assumption. Completed work should survive a later failure, corrections should stay close to the affected step, and consequential decisions need accessible evidence. Spending more on a model can be worthwhile when it demonstrably removes work from the person using it.

Before calling a change an improvement, the question is what the user now has to wait for, verify, correct, or pay for. A change in the model, budget, or workflow deserves to be judged through those consequences.