Panel: AI agents in production, what actually broke · Notes

The short version

Three teams running agents with real customers agreed the hard part isn't the model, it's knowing when the agent is wrong. Narrow jobs, evals as a release gate and showing every step came up again and again.

5 takeaways

Key takeaways

01Start with one narrow job the agent can finish end to end. Broad assistants stalled at every company on stage.
02Write the evals before the prompt. Initech runs 400 real cases on every change and blocks the release if the pass rate drops.
03Users trusted agents more when they could see each step, even when that made them slower.
04Outcome pricing is tempting and messy. Two panellists moved off per-seat pricing, one moved back.
05On hiring, they want PMs who can read a trace and explain a failure, not PMs who write clever prompts.

7 chapters

Chapters

Timed to the recording. Tap a time to link to it.Read each chapter in full

0:00
What's actually in production
Support triage at Umbrella, invoice matching at Initech, onboarding at Hooli. All narrow, all live.
5:40
Narrow jobs beat broad assistants
Every team started broad and cut back. The agents that stuck do one job and hand off the rest.
12:15
Evals as a release gate
Initech's 400-case eval set, who maintains it, and why it blocks releases like a failing test.
20:30
Trust means showing the steps
Visible steps and an easy undo did more for trust than accuracy gains users couldn't see.
28:45
Pricing agents
Per seat, per task, per outcome. What each panellist tried and what finance said.
34:20
Who to hire now
Trace-reading PMs, engineers who like writing evals, and designers who can show uncertainty.
40:10
Audience questions
Liability when an agent gets it wrong, and whether small teams should build or buy.

5 quotes

Worth quoting

The demo works on day one. Week three is when you find out what you actually built.
Sana QureshiFounder, Hooli Labs7:48
Nobody on my team gets to change a prompt without running the evals. It's the same rule as tests.
Arjun BhattStaff Engineer, Initech AI13:22 in the transcript
Our agent got more trusted when we made it slower and showed every step. People want to see it think.
Tomás ReyesDirector, AI Platform, Umbrella Retail21:10 in the transcript
We tried charging per task. Finance hated it, customers loved it. We're still arguing about it.
Sana QureshiFounder, Hooli Labs29:40
I don't need a PM who writes great prompts. I need one who can read a trace and tell me why it went wrong.
Meera IyerHead of Product, Globex Payments35:30

Transcript excerpt

From the transcript

An excerpt, lightly cleaned up. Speakers were matched by voice, then checked by a person.

12:15Meera Iyer
Arjun, you said something backstage I want you to repeat. About prompts.
12:24Arjun Bhatt
Sure. So we have about four hundred real cases, pulled from production, labelled by the ops team. Every prompt change runs against all of them.
13:22Arjun Bhatt
Nobody on my team gets to change a prompt without running the evals. It's the same rule as tests.
13:50Sana Qureshi
Can I push back a little? Four hundred is great when you have the traffic. We had eleven customers. Our eval set was me, a spreadsheet and a lot of coffee.
14:10Arjun Bhatt
Honestly, that counts. Fifty good cases beat a fancy tool with no cases.
20:30Tomás Reyes
For us the surprise wasn't accuracy. We got the agent from eighty-one to eighty-nine percent and nobody noticed. What they noticed was when we started showing the steps.
21:10Tomás Reyes
Our agent got more trusted when we made it slower and showed every step. People want to see it think.
40:15Audience
Who's liable when the agent gets it wrong?
40:22Meera Iyer
Short answer, you are. Which is why every one of us has a human who can say no before money moves.

The short version

5 takeaways

Key takeaways

01Start with one narrow job the agent can finish end to end. Broad assistants stalled at every company on stage.
02Write the evals before the prompt. Initech runs 400 real cases on every change and blocks the release if the pass rate drops.
03Users trusted agents more when they could see each step, even when that made them slower.
04Outcome pricing is tempting and messy. Two panellists moved off per-seat pricing, one moved back.
05On hiring, they want PMs who can read a trace and explain a failure, not PMs who write clever prompts.

7 chapters

Chapters

Timed to the recording. Tap a time to link to it.Read each chapter in full

0:00
What's actually in production
Support triage at Umbrella, invoice matching at Initech, onboarding at Hooli. All narrow, all live.
5:40
Narrow jobs beat broad assistants
Every team started broad and cut back. The agents that stuck do one job and hand off the rest.
12:15
Evals as a release gate
Initech's 400-case eval set, who maintains it, and why it blocks releases like a failing test.
20:30
Trust means showing the steps
Visible steps and an easy undo did more for trust than accuracy gains users couldn't see.
28:45
Pricing agents
Per seat, per task, per outcome. What each panellist tried and what finance said.
34:20
Who to hire now
Trace-reading PMs, engineers who like writing evals, and designers who can show uncertainty.
40:10
Audience questions
Liability when an agent gets it wrong, and whether small teams should build or buy.

5 quotes

Worth quoting

The demo works on day one. Week three is when you find out what you actually built.
Sana QureshiFounder, Hooli Labs7:48
Nobody on my team gets to change a prompt without running the evals. It's the same rule as tests.
Arjun BhattStaff Engineer, Initech AI13:22 in the transcript
Our agent got more trusted when we made it slower and showed every step. People want to see it think.
Tomás ReyesDirector, AI Platform, Umbrella Retail21:10 in the transcript
We tried charging per task. Finance hated it, customers loved it. We're still arguing about it.
Sana QureshiFounder, Hooli Labs29:40
I don't need a PM who writes great prompts. I need one who can read a trace and tell me why it went wrong.
Meera IyerHead of Product, Globex Payments35:30

Transcript excerpt

From the transcript

An excerpt, lightly cleaned up. Speakers were matched by voice, then checked by a person.

12:15Meera Iyer
Arjun, you said something backstage I want you to repeat. About prompts.
12:24Arjun Bhatt
Sure. So we have about four hundred real cases, pulled from production, labelled by the ops team. Every prompt change runs against all of them.
13:22Arjun Bhatt
Nobody on my team gets to change a prompt without running the evals. It's the same rule as tests.
13:50Sana Qureshi
Can I push back a little? Four hundred is great when you have the traffic. We had eleven customers. Our eval set was me, a spreadsheet and a lot of coffee.
14:10Arjun Bhatt
Honestly, that counts. Fifty good cases beat a fancy tool with no cases.
20:30Tomás Reyes
For us the surprise wasn't accuracy. We got the agent from eighty-one to eighty-nine percent and nobody noticed. What they noticed was when we started showing the steps.
21:10Tomás Reyes
Our agent got more trusted when we made it slower and showed every step. People want to see it think.
40:15Audience
Who's liable when the agent gets it wrong?
40:22Meera Iyer
Short answer, you are. Which is why every one of us has a human who can say no before money moves.

Key takeaways

Chapters

What's actually in production

Narrow jobs beat broad assistants

Evals as a release gate

Trust means showing the steps

Pricing agents

Who to hire now

Audience questions

Worth quoting

From the transcript

Key takeaways

Chapters

What's actually in production

Narrow jobs beat broad assistants

Evals as a release gate

Trust means showing the steps

Pricing agents

Who to hire now

Audience questions

Worth quoting

From the transcript