From 34 minutes to 40 seconds: what an AI gatekeeper actually cost me

Our first reply to an inbound message used to take 34 minutes during business hours, and until the next morning outside them. It now takes 40 seconds, around the clock. Conversion from inbound message to an actual conversation with a salesperson went from 52% to 71%. The whole thing runs on $45 a month.

This is a case study, not an architecture write-up. I run marketing, I built this for my own team, and the only question I cared about was whether the leak cost more than the fix.

The part nobody warns you about

Start here, because it decides whether any of this works.

Three of the five weeks this took went into getting written answers to questions everyone in the company already knew. It turned out we had no single answer to "how long will it take." Every salesperson quoted a different lead time, and every one of them was defensible. Until that answer exists in writing, the model has nowhere to get it, so it invents one.

Which it did, in week one. The agent quoted a customer a price we do not charge, with a delivery date, confidently, complete with reasoning. I had not connected it to our live pricing, assuming general phrasing would be enough. It wasn't. A model never says "I don't know." It fills the gap with something that reads true.

Then there was my definition of a qualified lead, which I wrote in about half an hour and which was wrong. It caught anyone who asked about price, and 34% of our inbound is not customers at all: agencies pitching services, suppliers, people who got the wrong address. I rewrote it three times. What finally worked came out of reading roughly two hundred agent decisions over two weeks, one after another. Tedious, and not delegable. Whether a lead is worth your team's attention is a business call, not an engineering one.

The last thing that broke was human. For the first few days my team didn't trust the agent and re-read every original thread anyway, which cancelled out the entire saving. What fixed it wasn't persuasion. It was putting a link to the raw conversation in the summary card, so checking took one click. They stopped checking within a week.

Why I got here in the first place

Not out of any interest in AI. I was working out why our deal count sat flat while the budget grew, and landed somewhere unglamorous. We had enough inbound, 380 messages a month across all channels, roughly a dozen a day. The problem was the gap between someone writing to us and someone from my team actually replying. Some of them filled that gap by going elsewhere.

On a report it looked like weak lead-to-conversation conversion. We spent the best part of a year treating it with sales scripts. Wrong diagnosis.

Why I skipped the two obvious fixes

The standard advice is to put a junior on first-line response, or install a chatbot with a button tree.

A junior only covers business hours, and 41% of our messages arrive in the evening or at the weekend. That is not a rounding error, that is nearly half the problem left untouched. Staffing a night shift for 380 messages a month doesn't add up on any spreadsheet I could build.

The chatbot is worse. It doesn't filter so much as annoy: someone has already typed their question in their own words, and the thing responds by asking them to pick a menu category. We had tried it. Engagement dropped hard enough that we pulled it after three weeks.

What I wanted was a third thing. Something that reads free-form text, works out what is being asked, asks one or two clarifying questions, and hands the salesperson a situation rather than a transcript.

One scenario, not ten

I automated exactly one thing: triage of an inbound message and handoff.

That mattered more than any technical decision I made. There was an obvious temptation to do everything at once, FAQs and booking and reminders and follow-ups. I parked it, because a bundle like that has no moment where you can say whether it worked. One scenario does, and it is measured in conversations that happened.

Two weeks in, the numbers moved. Six weeks in, I expanded.

The one thing I rolled back

I tried handing follow-ups to the agent. A polite nudge after a day, another one three days later.

It worked, technically. I killed it anyway. When you write to someone who showed interest and then went quiet, any mechanical note in the message is obvious immediately, and instead of restarting the conversation you get quiet irritation. Two customers told me so in plain language. Follow-ups went back to people.

The line I settled on: the agent works up until the first real contact with a salesperson, and not one step past it.

What I actually bought

Triage was eating about nine hours of my team's week. It doesn't now.

That is the purchase. Not a technology, nine hours and a response time that no longer depends on what time it is.

If you run marketing and you're thinking about this

Don't start by asking what AI could do for you. There is no answer to that question and you will drown in demos.

Start from the other end. Find the place where you lose money on something that repeats daily and requires no judgment. It will almost certainly be embarrassingly mundane: unanswered evening messages, sorting inbound, rebuilding the same report. Automate that one thing and look at the number two weeks later.

And keep in mind what you are buying. If neither your team's time nor your response speed changed after you shipped it, you didn't buy a system. You bought a demo.

A version of this piece was first published on Medium.