How to Vet an Automation Consultant When You Cannot Read Code

Three questions, and only three. Can this person build the thing. Will it still be running in March. Can you leave without losing it.

Everything else on a sales call is texture. The demo, the logo wall, the case study with the percentage in it: all of that is aimed at question one, and question one is the easy one. Plenty of people can build something that works on a Tuesday afternoon in front of a prospect. The engagements that go wrong almost never go wrong on question one. They go wrong four months later, when a vendor changes an API and the thing that used to sort your inbound stops sorting your inbound, and nobody notices for nine days because nobody was told to.

So the vetting has to aim at questions two and three, and the person selling to you has no natural reason to bring either of them up. You have to. The good news is that every one of the checks below produces a thing rather than an answer. A question can be handled by anyone with a decent memory for words. An artifact either arrives or it does not.

Because a demo is a rehearsed path through a system, on data the builder chose, on a day they picked. That is not an insult to demos. It is what a demo is for.

The more useful reason to distrust it is that the people building this stuff cannot reliably tell you how well it is going either. METR ran a randomized trial in early 2025 with sixteen experienced open-source developers working 246 real issues in repositories they already maintained. Before starting, they expected AI tooling to speed them up by 24%. They came out 19% slower. Afterwards, having lived through it, they still believed they had been sped up by about 20%. That is expert practitioners, measuring their own work, on their own code, getting the direction of the effect wrong.

If that is what self-assessment is worth from the inside, a forty-minute presentation from the outside is worth less. And the failure mode is specific rather than dramatic. In Stack Overflow's 2025 survey of more than 33,000 developers, the single most-cited frustration, at 66%, was AI output that is almost right but not quite. Only a third said they trust the accuracy of what comes back. Almost right is exactly the thing that survives a demo and dies in production, because almost right looks identical to right until the one record it mangles turns out to be the client invoice.

Ask anyway. You are not reading the code in it. You are reading the history around the code, and the history is written in plain dates and names that anybody can read.

Say this: "Send me read access to a repository for something you built and still run. I am not going to open the files." Watch what happens next, because the reaction is half the test. Somebody who builds things says yes and asks which client they can get permission from. Somebody who does not sends you a PDF.

When it arrives, six things are visible on the landing page before you click anything.

Plenty of good automation work has no code in it, and a consultant who builds in n8n, Make or Zapier is not hiding anything by not having a repository. The artifact just changes shape.

Ask for the exported workflow file and the version history inside the platform. Both exist. n8n and Make export a workflow as JSON you can save, and every one of these tools keeps a record of who edited what and when. You are looking for the same three things: whether the work happened over time, whether more than one person touched it, and whether anybody has been in there recently. A consultant who cannot export the thing they built for you is telling you something important about what happens when you stop paying them, and it is worth knowing before you start. We have written the longer version of where no-code stops holding up, which is a different question from this one but tends to arrive at the same table.

If the answer to every version of this request is no, you have your answer. There is no confidentiality rule that prevents somebody from showing you a workflow they built for themselves.

This is the check that separates people who run systems from people who ship them. Ask them to open the run history on something they operate today and find you a failure.

Every real system has them. Zapier's own documentation says a failed step gets replayed up to five times, backing off at five minutes, thirty minutes, one hour, three hours and six hours, so the last attempt lands about ten and a half hours after the first error. Then it stops. That is the vendor telling you, in writing, that a system will eventually stop trying and somebody has to be there when it does. Autoreplay is also only on the Professional plan and up, which means "we will be on Zapier" is not an answer to "what happens when it fails", and the two get conflated constantly.

So ask a specific question and listen for a specific answer. A good one sounds like a story with a timestamp in it: this one failed at 3am on the fourteenth because the CRM returned a 502, it retried twice, it landed in an error queue, and Priya cleared it at nine the next morning. A bad one sounds like reassurance. If somebody tells you their automations do not fail, they are either not running any or not watching the ones they are running, and you cannot tell which from where you sit.

The research points the same way. Google's 2025 DORA report surveyed nearly 5,000 technology professionals and found 90% now using AI at work, with a positive relationship between AI adoption and delivery throughput and a negative one with delivery stability. More gets shipped and more breaks. The teams where that trade still comes out well are the ones with a control system around the work: version control, automated testing, fast feedback when something goes wrong. Tooling does not supply any of that on its own. Which is the thing you are checking for, and the reason the question is about failures rather than features.

Two names, in writing, in the scope document. The person who does the building and the person who checks it. Very often the answer to both is the same person, and that is fine at this size, as long as it is said out loud rather than implied by the seniority of whoever ran the call.

The reason to insist on the second name has changed in the last two years. Almost everybody building automation now writes some of it with a model, including us, and that is not a scandal. It is a reason to care about review. Veracode's 2025 GenAI code security testing ran more than 100 models across four languages and found 45% of the resulting code introduced an OWASP Top 10 vulnerability. It compiled. It ran. It did the task. It was also insecure, and the failure rate did not improve with newer or larger models, which means this is a property of the method rather than a phase it will grow out of.

Ask what reviews the machine's work before it touches your accounts, and listen for whether the answer contains a person or a process. "We test it" is a start. "The model checks its own output" is not an answer, it is the problem restated confidently. Somewhere in a credible reply there is a human reading a diff, or a test suite that runs on every change, or a staging environment that is not your live CRM. Checking that machine output is right before anybody depends on it is close to the whole job now, and it is why we build evals around agents rather than treating a passing demo as the finish line.

There is a second reason to care, which is what happens to the thing you paid for over the following year.

Four things need to be in your name before any money moves, and all four are checkable by you in an afternoon.

The accounts are yours. Not a seat on their Zapier org, not their OpenAI key with your usage on it. Your workspace, your billing, your API keys issued from your own accounts, with them added as a user you can remove. The single most common way an automation build turns into a hostage situation is that the whole thing runs on infrastructure billed to somebody else, and the day you want to leave you discover that leaving means rebuilding.

The repository or the export is yours, in your organization, from the first commit rather than at the end. "We will hand it over on completion" is a promise about a moment that may never arrive.

The documentation exists and a stranger can follow it. Ask for the handover doc from a previous engagement, redacted as much as they like. One page that says what runs, when it runs, what breaks it, and who to call is worth more than forty pages of architecture diagrams.

And the copyright is assigned to you in writing, in those words. This is the one people get wrong, because it sounds like it is already handled. Under US law, a commissioned work only counts as a work made for hire if it falls into one of nine specific categories, and the Copyright Office lists them: contributions to collective works, parts of audiovisual works, translations, supplementary works, compilations, instructional texts, tests, answer material, and atlases. Software is not on that list. So a contract that says the work is a work made for hire, and stops there, can leave the copyright sitting with the person who wrote it. What you want next to that clause is an express assignment of copyright to you. We are not your lawyers and this is not legal advice, but it is a specific question worth putting to somebody who is, and it costs one email.

Everything above is free to run and takes about a week. The last check costs money, and it is the only one that tells you what somebody is actually like to work with.

Buy a small piece first. One workflow you currently do by hand, scoped in a paragraph you could read aloud. Two weeks. A fixed price agreed before it starts. Written acceptance criteria, which means a sentence that can be true or false on day fourteen rather than a feeling about whether things went well. If the document that comes back has no such sentence in it, that is one of the six things in a proposal worth stopping the meeting over. And three deliverables: the thing running in your accounts, the repository or the export in your organization, and one page of documentation.

Make it paid. A free pilot gets staffed with whoever is idle, and it gives you a version of the engagement that does not resemble the engagement. Paying also buys you the right to be a demanding client for two weeks, which is exactly the condition you want to observe them under.

On what that should cost, our own numbers, since almost nobody in this category publishes theirs. A one-off starter build with us is $1,500 to $2,500 fixed. A two-week production sprint is $5,000 fixed. Retainers start at $5,000 a month, and that last number is the one to keep out of a first engagement: a trial priced like a retainer is not a trial, it is a contract with a friendlier name. The full ladder is on our pricing page.

The case for buying a first build rather than working it out internally is reasonably well evidenced. The MIT NANDA study behind the widely quoted finding that 95% of generative AI pilots showed no measurable return also found that buying from specialist vendors and building through partnerships succeeded about 67% of the time, against internal builds succeeding roughly a third as often. That gap is the argument for hiring an agency automation consultant at all. It is not an argument for hiring this one, which is what the two weeks are for.

Ask for a reference whose engagement has ended. Not the happiest current client, who has every reason to be generous about work they are still paying for, and who was chosen for that reason.

Then ask three questions and stop talking. What broke, and how did you find out. Who fixed it and how long did it take. What happened the first time you wanted to change something after they had gone.

That third question is the whole call. It is the only one that tests handover rather than delivery, and handover is where an automation engagement turns into either an asset or a subscription you cannot cancel. A reference who says they changed it themselves has told you the documentation was real. A reference who says they had to call them back, and were glad to, has told you something honest and also something specific about what year two costs. A reference who says they stopped using it is the most useful call you will make, and it is why the request has to be for an ended engagement rather than a current one.

While you are on the phone, ask who actually did the work. Being sold by a senior and built by somebody you never met is common enough to be worth ten seconds of checking, and it is the sort of thing a former client will tell you plainly.

Some of the standard advice in this category is theater, and skipping it buys you time for the checks that are not.

Certifications and platform partner badges tell you somebody paid a fee and passed a test on a product. They say nothing about whether a system they built is still running. A logo wall tells you a company existed in the same room as them once, and logo walls are not audited. Team size is a vanity number, and a shop of four that ships beats a shop of forty where your work sits behind nine other projects. Years in business is worth something in a stable trade and much less in one where the tooling changed completely inside three years.

The one piece of theater worth naming directly is the guaranteed outcome. "Guaranteed ROI in ninety days" is a claim about your business, made by somebody who has not yet seen your data. Confidence would sound different, and usually quieter. It is also the kind of claim that has become an enforcement matter: the FTC's Operation AI Comply opened in September 2024 against companies making exactly these promises, and one of them settled for $193,000. You are not going to sue anybody over a bad automation build. But when a regulator has to open a sweep against a marketing pattern, that pattern is common, and you should price it as common when you hear it on a call.

Being straight about the limits, because a checklist that claims completeness is doing the same trick as the testimonials.

None of this predicts whether somebody will still care in month six. Attention is the real failure mode in small engagements, and there is no artifact for it. None of it catches a genuinely good builder who is bad at process, and those exist: someone whose repository is a mess and whose systems have run for three years without an incident. Judge the ratio rather than any single miss, and weight the failure story and the ended-client reference highest, because those two are the hardest to stage.

And none of it removes the need to be a decent client. The engagements we have seen go wrong from the inside usually went wrong on both sides: a scope that kept moving, a decision nobody would make, an internal person who was supposed to provide access and did not. Vetting is the half you control. Do it in a week, spend the $1,500 to $2,500, and find out.