Six ways an automation project quietly fails after go-live

Most automation post-mortems are written about projects that never shipped. Those are the easy ones: the budget went, the scope drifted, somebody left. The harder case is the project that shipped, worked, was celebrated — and eighteen months later nobody can say what it does.

That decay has a shape. Here are the six modes we design against, and the specific check that catches each one before it becomes permanent.

1. The exception queue nobody owns

Every automation has a confidence threshold. Above it, the machine acts; below it, a human decides. The pilot always staffs that queue — usually with the person who championed the project, who is motivated and fast.

Then the champion moves on. The queue keeps filling. Nobody is named as its owner because at pilot volume it never needed one.

Six months later the queue is the process. People work around the automation to avoid feeding it.

The check: before go-live, name the queue’s owner in writing, with a volume ceiling and an escalation rule. If the answer to “who works this queue in month nine?” is a role that does not exist yet, the automation is not ready.

2. Confidence thresholds set once and never revisited

A threshold is fitted on the data you had. Your data moves: a new supplier’s invoice layout, a new product category, a rebrand that changes every document header.

The model does not know the distribution shifted. It keeps returning confident answers on inputs it has never seen.

The check: track the rate of below-threshold routing weekly, not just accuracy. A queue that suddenly halves is as alarming as one that doubles — it usually means something is being confidently mishandled rather than correctly caught.

3. The integration that succeeds at the wrong thing

The hardest integration bugs are not failures. A failure raises an alert. The dangerous case is a write that succeeds against a stale interface: the field moved, the value lands in a column nobody reads, and every dashboard stays green.

We have watched this pattern for a year in a system where a status field was written to a deprecated table. Everything reported success. Nothing downstream changed.

The check: assert on the read-back, not the write. After writing, read the record through the same interface the business uses, and compare. If your integration test does not do a round trip, it is testing your own code rather than the system.

4. The measurement that stopped matching the baseline

This is the one that kills renewals, and it is entirely self-inflicted.

You measure a baseline by one method — a timed sample, a manual audit — then measure the result by another, because by then you have telemetry and telemetry is easier. The improvement you report is partly real and partly an artefact of the change in instrument.

A number produced by a different method than the baseline is not a result. It is a coincidence.

The check: freeze the measurement method with the baseline, in writing, before any build starts. If you must change instruments, run both for one full cycle and publish the delta between them.

5. Documentation written last, and never tested

Handover documents are written at the end, by the person who no longer needs them, for a reader who does not exist yet. They describe the happy path in the present tense and stop.

The test is not whether the document exists. It is whether someone who was not in the project can use it to recover the system.

The check: a recovery drill before final invoice. Someone outside the build takes the runbook and restores service to a broken instance. Whatever they ask out loud is the documentation gap. We failed this test ourselves the first time we ran it.

6. The process that was broken before you automated it

The most expensive failure, because it looks like success for the longest time.

If a process needs a workaround at step four, automating it means the workaround now happens 4,000 times a month at machine speed and nobody can see it. Throughput rises. Error volume rises with it, but distributed thinly enough that no single person notices.

The check: read the event log before the workshop diagram. The diagram describes the process people believe they run; timestamps describe the one they actually run. Where the two disagree, fix the process before automating it — or automate the corrected version and say so explicitly.

What this adds up to

None of these are model problems. Five of the six are organisational, and the sixth is a testing discipline. That matches what the research says about pilot attrition: the constraint is rarely the technology.

It is also why our engagements end with a stopping condition rather than a launch party. Something that cannot fail visibly cannot be trusted, and something nobody owns in month nine was never really delivered.