This is part two of a three-part series on how much authority to hand an AI agent. Part one is the agent that only watches. Part three is the agent that earns the right to act on its own.
Every other Wednesday, an agent hands our team a list of products that are live on our site and impossible to buy. The last run started with 469 of them.
Nobody can do anything useful with 469.
That gap is the entire reason this agent exists, and it is also why the agent does not fix a single one of those listings on its own. It narrows 469 down to about 49, writes them into a spreadsheet with checkboxes, posts a link, and stops. Humans decide. A separate task does the actual work later, and only on the rows that got checked.
This is the human-in-the-loop AI agent pattern, and I reach for it any time an automation's mistake would be expensive to undo.
A quick word on the names before we go further. The first agent we built got called Leo, after Leonardo DiCaprio, for no reason beyond needing a name and that one sticking. Then somebody asked for a second agent. Once you have a Leo on the team, there is exactly one name available. Kate runs the inventory audits now. I regret nothing.
Key takeaways
-
The audit finds products that are live on the storefront, sold out, and have nothing on order. Customers can find them and cannot buy them.
-
A recent run filtered 469 sold-out live listings down to 49 that actually needed a decision.
-
The agent writes those 49 into a permanent Google Sheet with real checkboxes, then stops. It has no ability to change anything in the store.
-
A separate task re-checks each live product and then executes, acting only on rows a human checked.
-
The agent does homework first, pulling sales velocity and pre-ticking items that already have a reorder point, so approving is fast and disagreeing is faster.
-
Certain products cannot be archived at all, because the checkbox for it is locked rather than merely discouraged.
-
This one costs nothing recurring to run, same as every other agent we have built.
What problem is the audit actually solving?
Products that are sold out, still listed, and have nothing on the way.
Every one of those is a small dead end. A customer searches, finds the thing, decides they want it, and cannot buy it. A few hundred of those across a catalog adds up to a storefront wasting a real slice of its own traffic.
The right answer is different for every row, which is exactly what makes it hard. Some should be reordered because they sell well and the gap is a purchasing oversight. Some should be hidden temporarily. Some should be archived because they are finished. A few belong to a monthly themed mission and get handled a completely different way.
We run the inverse audit too, the one that finds products that are hidden or excluded but do have stock sitting in the warehouse. That first pass surfaced roughly $50,000 of product, at cost, that was in the building and not offered for sale anywhere. Already bought, already paid for, invisible.
Why doesn't the agent just fix them?
Because every action available to it is one you would hate to get wrong 49 times in a row.
Archiving a product that should have been reordered kills a listing with real history behind it: reviews, search ranking, inbound links. Hiding an item the buyer placed an order for last week is a self-inflicted outage. Reordering something genuinely discontinued is money spent on a mistake.
The information needed to make those calls does not live in Shopify. It lives in people's heads. A vendor promised a date over the phone. A product is getting replaced next quarter. An item is part of a mission, and there is a reason it looks dead. No agent is reading any of that out of an API.
The agent is very good at finding the 49 rows. It has no idea which of them matter.
How does the agent hand the decision to a human?
It writes the list into a Google Sheet with real checkboxes, posts a link in Slack, and stops there.
What happens next is a two-phase triage, and the two phases are deliberate. In the first phase, the people who own each product channel go into the notes column and make their case: keep it, kill it, here is what I know that the sheet does not. About a day later, in the second phase, the people who actually make the call read those notes and check the boxes.
That gap of a day is doing real work. It separates the person with context from the person with authority, and it means nobody is checking boxes in the first ten minutes based on a list they just met.
A separate task, started deliberately by a person, then reads the checked boxes and executes. The agent proposes, humans approve, and a second run acts.
Why one permanent sheet instead of a new one every run?
A link nobody bookmarks is a link nobody opens.
The audit writes into one permanent Google Sheet with a stable URL. Every run wipes it, rewrites it with the current rows, reapplies the checkboxes, and renames the tab to the run date. The dated archive of each cycle gets saved elsewhere for our own records.
Generating a fresh sheet every cycle feels tidier and quietly kills adoption. Two weeks later, half the team is looking at the old one, somebody is asking which link is current, and the thing turns into a scavenger hunt nobody signed up for. One permanent bookmark beats a clean filing system every time.
What does the agent do to earn the team's time?
Homework. An approval queue is only worth building if a human can clear it quickly.
Before it posts anything, the agent pulls units sold over the last 30, 90, and 365 days for every flagged product and puts those numbers right on the row. The reorder question stops being a guess and becomes a comparison.
It also checks whether each item already has a reorder point configured in our inventory system, and pre-ticks that column with a short note explaining why. On the last run that covered 21 of the 49 rows, which is 21 lookups nobody had to do.
Hand somebody a bare list of 49 product names, and you have not automated anything. You have created homework and put your name on it.
What can the checkbox not do?
Some rows cannot be checked for archive at all, because that cell is locked.
Certain products belong to a monthly themed mission, and mission items are never archived. That rule exists for reasons that have nothing to do with sales data. Rather than writing "please do not archive mission items" into the instructions and hoping, the sheet locks the archive cell on those rows to "n/a." The option is not there. Nobody can tick it, not even by accident at 4:55 on a Friday.
The execution task carries the same rule independently. It re-reads the live product before acting and skips any mission item that somehow made it through.
A guardrail the agent has to remember is the weakest kind. Removing the option entirely, in two places that do not depend on each other, is the strong version.
Why is the execution a completely separate task?
Because the agent that finds problems has no ability to change anything, by design rather than by configuration.
The audit can read the store, read the inventory system, and write to exactly one spreadsheet. It cannot edit a product, change a tag, or archive anything. If it went sideways tomorrow, the worst case is a spreadsheet full of nonsense that a human declines to check.
The task that applies decisions is a different job with different permissions, started on purpose by a person. Before it changes anything, it re-reads the live product, because the checkboxes might be a day old and something might have shipped in the meantime. It is also written so that running it twice does nothing the second time, which means a nervous re-run is harmless.
Splitting the two means a bug in the finder can never become a change to the store. If you build only one thing from this article, build the split.
Does a checkbox actually count as human oversight?
Only if the person checking it has enough information to disagree.
Article 14 of the EU AI Act covers human oversight for high-risk AI systems, and it names the failure mode directly: automation bias, the tendency of people to over-rely on whatever the system puts in front of them. Our inventory sheet is nowhere near a high-risk system in the regulatory sense. The failure mode is identical anyway.
Give somebody 49 rows with no evidence attached, and they will approve all 49 in four minutes, and what you built is approval theater with extra steps. The velocity numbers and the pre-ticked reorder points exist so that saying "no, not that one" takes about three seconds. Make disagreement cheap or the approval step is decoration.
What does this cost to run?
There is no recurring bill on it, which is the same answer as everything else we build.
No per-agent license, no monthly fee. It runs on the AI subscription we already pay for. The cost was the time spent writing down what to check, what to filter out, and what the agent is never allowed to do.
Against that, the audits found roughly $50,000 of product sitting in the building and not for sale. No software subscription was going to hand us that, and if one had, it would have billed us every month afterward for the privilege.
Why is the funnel the actual product?
The raw number is useless. The filtered number is a meeting.
Here is what one recent run looked like, step by step. It started with 469 active products showing zero inventory. Dropping the ones already tagged hidden or excluded, because somebody has handled those, took it to 171. Dropping everything with a purchase order already on the way took it to 59. A final pass landed on 49 rows, ten of which are mission items that get handled their own way.
The two cycles before that ran 474 down to 51, and 462 down to 57. The shape holds.
Roughly 90 percent of what the agent finds is noise somebody has already dealt with. Anyone can write the query that returns 469. The work is in the filters that make it 49, and every one of those filters is a piece of business logic somebody had to explain out loud once.
Where else does this pattern fit?
Anywhere the action is expensive or annoying to reverse.
Refunds and cancellations. Price changes. Deleting or archiving anything. Sending anything to a customer. Publishing to the storefront. Deactivating an account. If your first reaction to "the agent will just handle it automatically" is a small flinch, that flinch is the signal to put a checkbox between the finding and the doing.
Genuinely reversible, low-stakes work does not need any of this. Adding a tag, updating an internal note, refreshing a report. Making those wait on a human is ceremony, and ceremony is how a good automation gets abandoned.
Frequently asked questions
What is a human in the loop AI agent?
It is an AI agent that identifies work and proposes actions, then waits for a person to approve before anything is executed. In our version, the agent writes findings into a spreadsheet with checkboxes, and a separate task acts only on the rows a human checked.
Why not let the AI agent make the changes automatically?
Because the actions involved are hard to undo and the context needed to judge them lives with people, not in the data. A vendor conversation or a discontinuation plan never shows up in an API response, and getting it wrong across dozens of products at once is expensive.
How do you stop approval from becoming a rubber stamp?
Attach evidence to every row and pre-answer the lookups. Ours puts 30, 90, and 365 day sales figures on each line and pre-ticks items that already have a reorder point. If disagreeing with the agent is fast, people actually disagree. If it is slow, they approve everything.
Should the agent that finds problems also be the one that fixes them?
No. Keep them as separate tasks with separate permissions. The finder should have no ability to change anything, so a bug in the finder can never become a change to your store.
What is the best way to enforce a rule an agent must never break?
Remove the option instead of writing a rule. We lock the archive checkbox on products that must never be archived, and the execution task independently skips them. A guardrail that depends on the agent choosing to follow it is the weakest kind.
Should the agent create a new spreadsheet for each run?
No. Use one permanent sheet with a stable URL that the team bookmarks, and wipe and rewrite it each run. A new link every cycle means half your team is looking at a stale one.
What does it cost to run an AI agent like this?
Nothing beyond the AI subscription you are already paying for. There is no per-agent license and no recurring software bill. The cost is the time spent writing the instructions, paid once.
Final thoughts
The temptation with any agent is to let it finish the job. It found the 49 rows; it clearly knows what it is looking at, why make a person click 49 times?
Because the clicking is where somebody says "hold that one, the vendor called me Tuesday," and there is no version of this where the agent knew that. What the agent removed was the list-pulling and the purchase-order cross-referencing that used to sit between us and that conversation.
Hand the agent the searching, keep the deciding for yourself, then spend your effort making the deciding fast. An approval step nobody has time for is the same as no approval step at all.
Interested in learning more?
This is part two of a three-part series on agent authority. Part one is How We Built an AI Agent to Catch Discount Code Fraud Every Morning, the agent that only watches. Part three is How We Let an AI Agent Earn Autonomy One Action at a Time.
For more of the operator side of running a store, take a look at Always Delete Your Test Coupon Codes Before a Bot Finds Them and our series on how we built a seven-figure live-selling channel.
