Our Build Log: What Breaks in Production Automations

The reasoning, the prompt, the branching logic, the bit that took the longest to get right: that part survives. What goes down is the plumbing around it. A token expires. A source system renames a field. A per-second cap gets crossed on the one morning somebody runs a backfill. None of it is interesting and all of it is load-bearing.

The field is very quiet about this, which is the first thing worth noticing. Search for why automations fail and you get a dozen pages listing six or eight failure modes with no citation on any of them, written by shops that sell reliability. The closest thing to a public number is old: in Forrester Consulting research commissioned by Tricentis back in 2020, 45% of firms said they dealt with bot breakage weekly or more often, and 99% said their bot logic required some scripting. Six years on, that is still the number people cite, which tells you how little anybody has published since.

We are not going to pretend we have a tidy internal dataset to put next to it. What we have is a list, an ordering, and the vendor documentation that explains the mechanism behind each item. The ordering is ours and it is a judgment, not a measurement. The mechanisms are not judgments: every one of them is written down by the vendor, in public, usually in a help article nobody reads until the second time it happens.

This is the single most common way a working automation stops working, and the mechanism is published. Google's OAuth documentation says a refresh token stops working if the user revokes access, if it has not been used for six months, if the user changed their password and the token carries Gmail scopes, if an admin restricted one of the scopes, or if the account has hit its ceiling on live refresh tokens. Then there is the one that catches almost everybody: a project whose OAuth consent screen is still in Testing gets tokens that expire in seven days. Not seven days from a warning. Seven days from issue, and then invalid_grant.

Seven days is exactly long enough for a build to be signed off, demoed, praised, and forgotten about. Nobody moves the consent screen to production because nobody knew that was a step, and the thing dies the following Tuesday.

Microsoft publishes its own clocks. Entra refresh tokens last 24 hours for a single-page app and 90 days for everything else, self-renewing on each use, and the same table lists which kinds of password change and admin action revoke them outright. That table is worth reading before you build anything against a Microsoft tenant, because half the rows say the token stays alive and half say it dies, and the difference is whether the client is confidential.

A workflow that runs twelve times a day will never see a rate limit. The day you migrate three years of history into it, you will see nothing else.

HubSpot publishes its numbers: 100 requests per 10 seconds on Free and Starter, 190 on Professional and Enterprise, and a daily ceiling of 250,000, 625,000 or 1,000,000 depending on the tier. Cross either and you get a 429 with errorType: RATE_LIMIT and a policyName telling you whether you hit the per-second cap or the daily one. Read the second half of that page carefully, because it contains the part that actually bites: the burst limit is per app, but the daily limit is shared across every app on the account. Somebody else's integration can spend your budget, and the error will arrive in your logs.

The fix is a queue and a concurrency ceiling, and it is not hard. The expensive part is that you find out on migration day, with 40,000 records half moved and no way to tell which half.

Every platform retries, and retrying is correct. The trouble is that retries multiply through a stack instead of adding up.

The Google SRE book puts it plainly: if a database is struggling and the backend, the frontend and the browser script each retry three times, one user action lands as 64 attempts on the database. The chapter's recommendation is a cap rather than a longer wait. Limit retries per request, and give the whole process a budget, their example being 60 retries a minute, after which you stop retrying and fail honestly.

On an automation with a model in it, the multiplier is also a price. A retried step is a repriced step, and a workflow that quietly went from one attempt to five overnight has quintupled its own token bill without anybody changing a line of it. We put a spend ceiling on every workflow that calls a paid API, for exactly this reason. It is a cheap control, and it is the one that tells you about a retry loop before the invoice does.

This is the one that costs a client relationship rather than an afternoon, so it is worth knowing the exact numbers.

Zapier's own help article on replay says Autoreplay retries a failed step up to five times, on a backoff of 5 minutes, then 30 minutes, then 1 hour, then 3 hours, then 6 hours. Their worked example: a Zap that errors at 1:00pm gets its final attempt at 11:35pm. Ten hours and thirty five minutes. And then the sentence that matters more than any of it, quoted from that same page: "Zapier will not send any error notification emails until the final autoreplay attempt fails."

So a step that dies at nine on Monday morning generates its first email at half past seven that evening, into an inbox nobody is reading, and gets looked at on Tuesday. The workflow was down for a full working day and the run history said pending the whole time.

The other threshold on Zapier's error troubleshooting page is worth holding next to it. A Zap turns itself off when 95% of its runs error over the last seven days. That is a sensible circuit breaker and it is also a floor: a Zap failing on nine runs in ten stays on indefinitely, quietly dropping 90% of the work, because 90 is not 95.

Two flavors, and the loud one is the easy one.

The loud flavor is a version being retired. Salesforce retired platform API versions 21.0 through 30.0 in the Summer '25 release, and documents exactly what a retired call gets back: REST returns 410 GONE, SOAP returns 500 UNSUPPORTED_API_VERSION, and Bulk returns 400 InvalidVersion. That is a good failure. It is unmissable, it is dated years in advance, and you can grep for it.

The quiet flavor is the one that costs money. A client's ops lead renames a picklist value, or adds a required field, or changes a currency column from a number to a string with a symbol in it. Every call still returns 200. Your mapping now writes an empty string into a field that used to hold a value, and it does that a few hundred times before anybody looks at the output closely enough to notice.

Why this stays invisible is not a mystery either. Postman's 2025 State of the API report surveyed developers on exactly this discipline, and the answers are the reason drift is a category rather than an incident.

Any workflow that reads a page rather than an API inherits somebody else's release schedule. A container gets renamed, a table becomes a set of divs, a login flow adds a step, and your selector matches nothing.

We went looking for a citable number on how often this happens and there is not one. Plenty of pages assert that websites change constantly. None of them measured it, and the ones that came closest are scraping vendors with an obvious interest in the answer. So the honest version is: nobody publishes this, and the absence is itself worth knowing when somebody quotes you a confident uptime figure on a scraped source.

What we can say precisely is what the failure looks like, because it is always the same shape. The step succeeds. It returns zero rows. Downstream, a report renders with no data in it and a green tick next to it. If your monitoring only watches for errors, this is invisible forever.

Scheduled runs are where geography turns into an outage.

The IANA time zone database shipped three releases in the first seven months of 2026. 2026a in March, 2026b in April, 2026c in July. Between them: British Columbia went to permanent -07 on March 9, Alberta went to permanent -06 in June with the Northwest Territories expected to follow, and Morocco moves to permanent UTC on September 20. Every one of those is a jurisdiction where a nightly job that fires at 2am local either runs twice, skips a day, or lands an hour into the next reporting period.

The month-end version is quieter and just as common. A monthly report scheduled for the last day of the month, run against a system that closes the period at midnight in a different zone, produces a report of a period that has not closed yet. It looks fine. Every number in it is one day short.

Every document workflow is built against a sample the client picked, and clients pick the good one.

Then production arrives: a scan rotated ninety degrees, a merged cell spanning three columns, a second page nobody mentioned, a supplier who puts the total in a footer, a PDF that is an image of a spreadsheet. The extraction step does not error. It returns a number, and the number is wrong, and it is wrong in a field somebody downstream trusts.

This is the category where a model genuinely helps, because a rules engine has no way to handle the case it was not shown. It is also the category with the highest cost of being confidently wrong, which is why the judgment steps in our builds get an eval suite rather than a spot check. A step that reads a document and cannot tell you how sure it is has no business writing to a ledger.

The pattern in that table is the whole lesson. Detection is the expensive half, so it gets the budget. It is also the half nobody adds to an agent an employee built themselves, which is worth remembering as the DIY workbenches spread.

Five things now go into every build before the interesting logic does. A heartbeat that alerts on silence rather than on error, because five of the eight categories above never raise an error at all. A contract check that reads the source schema and fails loudly the moment a field it depends on changes shape. A cap on retries and a hard spend ceiling on anything that calls a paid API. Idempotency keys, so that a replay cannot double-post an invoice. And a dead letter queue with a named human owner, because a queue nobody owns is a folder.

None of that is clever. All of it is the difference between an automation that has been running since March and one that stopped in April and got noticed in June. It is also why our builds take the shape they do: what we actually build is a workflow plus the instrumentation that tells you when it stopped, and the second half is not an upsell.

Three questions, and they take about four minutes on a first call.

Ask what happens on day eight of a Google token. If the answer is a blank look, the person has not run an integration long enough to have been bitten, and you are paying for their education. Ask what the retry cap is and what it costs when it trips, because an uncapped retry on a metered API is an open tab. And ask to see a run that failed, in the tool, with the timestamps visible. That last one is the fastest read on somebody's real operating history, and it is the centerpiece of how to vet an automation consultant when you cannot read code.

The wider decision, whether you need an agency automation consultant at all or whether your ops person plus a no-code tool covers it, turns on the same thing this post has been circling. Everything above is error handling, and error handling is the part nobody scopes.

A no-code build is fine right up until it needs that, and the boundary is the real reason to move off Zapier rather than the monthly price. So whoever ends up owning the workflow, put the reliability work in the scope on day one. Bolted on after the first outage it costs more, and it costs it while the client is watching.