Panel: AI agents in production, what actually broke

Meera Iyer, who moderated, started by asking each panellist what their agent actually does in production today. The answers were all narrow: support ticket triage at Umbrella Retail, invoice matching at Initech, and onboarding calls at Hooli Labs. All three are live with paying customers.

Every team had started broad. Tomás Reyes described Umbrella's first attempt as an assistant that could answer anything about an order, and said it was impossible to test because there was no clear definition of done. The version that stuck triages tickets into five buckets and hands everything else to a person.

Sana Qureshi put it in terms of time: the demo works on day one, and week three is when you find out what you actually built. Narrow jobs survive week three because you can tell when they're wrong.

Arjun Bhatt explained Initech's release gate. They keep about four hundred real cases pulled from production and labelled by the ops team. Every prompt or model change runs against all of them, and if the pass rate drops, the release is blocked, the same as a failing unit test. Nobody on his team changes a prompt without running the evals.

Sana pushed back. Four hundred cases is great with Initech's traffic, but Hooli had eleven customers when they started, and their eval set was her, a spreadsheet and a lot of coffee. Arjun agreed that counts: fifty good cases beat a fancy tool with no cases.

The surprise of the panel came from Tomás. Umbrella pushed the agent's accuracy from 81% to 89% and nobody noticed. What users did notice was when the agent started showing its steps: which order it looked up, which policy it applied, and an undo button on every action.

Making the agent slower and more visible did more for trust than any accuracy gain users couldn't see. People want to see it think.

Pricing was the least settled topic. Two panellists had moved off per-seat pricing, toward per task or per outcome, and one had moved back. Tomás said they tried charging per task: finance hated it, customers loved it, and they're still arguing about it internally.

The shared worry was that outcome pricing makes revenue depend on how you define a successful outcome, and that definition becomes a negotiation with every customer.

On hiring, the panel agreed the most valuable PM skill right now is reading a trace. Arjun said he doesn't need PMs who write clever prompts; he needs one who can look at a failed run and explain why it went wrong. They also want engineers who enjoy writing evals, and designers who can show uncertainty in an interface without making it scary.

The first audience question was who's liable when an agent gets it wrong. Meera's short answer: you are. Every company on stage keeps a person who can say no before money moves or a customer is told something irreversible.

The second was whether small teams should build or buy. The panel leaned towards buying the plumbing (models, eval tooling, tracing) and building only the narrow job that is specific to your business.