Risk analysis

Demo Versus Deployment

How to read an AI announcement without losing a night's sleep, and the specific gaps that separate a capability from a change to your job.

My Career Crusader8 min read744 words

that separate a capability from a change to your job.

Something will be announced this month that appears to do your job. It will be demonstrated convincingly. You will feel a lurch, spend an evening reading, and eventually get on with your week without having learned anything useful. The lurch is not irrational, but it is responding to the wrong signal. What was announced was a capability. What matters to you is a deployment, and the distance between those two things is where almost all the useful information lives.

What a demonstration leaves out The error rate. A demo shows successes. The number you need is how often it fails and how badly, on real inputs rather than chosen ones. If an announcement does not state an error rate on a realistic sample, assume it is high enough to be inconvenient. The distribution of inputs. Demonstrations use clean, representative cases. Real work arrives messy, incomplete, in the wrong format, referring to things that happened before the system existed. Most of the difficulty in most jobs lives in that mess, and it is invisible in every demo.

Integration. The system has to reach your organisation's data, which lives in software chosen in 2011, partially migrated in 2019, with a permissions model nobody fully understands. This is where most deployments actually die and it is never in the announcement. Who is accountable. Every serious deployment eventually meets the question of who signs off when it goes wrong. Organisations get stuck here for a long time, and the sticking is not about technology. Whether anyone has the budget. Somebody has to own this, fund it, and be prepared to be blamed if it fails. In many organisations the honest answer is that nobody wants to.

A checklist for reading the news

When you next see something alarming, ask five questions before reacting.

Is this a research result, a product, or a deployment? These are three very different stages and the language deliberately blurs them. "Could", "has the potential to", and "in tests" all indicate the early end. Is anyone actually using it in production, at scale, for money? Named customers doing real work is a completely different signal from a benchmark score. What is the error rate, and who checks? If the answer is "a human reviews the output", note that the human is a cost that has not been removed, and ask how long that review takes.

What does it need access to? Systems, data, permissions. Then ask honestly whether your employer could arrange that within two years. Who at your organisation would own it? If you cannot name a plausible person, the timeline is longer than it looks. Five minutes of this is worth more than an evening of reading, and it usually converts alarm into a specific and much smaller concern.

The opposite error There is a failure mode on the other side that is at least as common and considerably more dangerous: concluding that because deployment is slow, nothing will happen. Organisational friction delays; it does not prevent. The tasks that are genuinely automatable do eventually get automated, usually several years after they could have been, and usually all at once when a new system finally lands or a new manager arrives with a mandate. People who spent those years reassured by the slowness get very little warning when it turns. The useful posture is neither alarm nor dismissal. It is to treat capability announcements as information about direction and deployment evidence as information about timing, and to make your plans on direction while setting your urgency by timing.

The most reliable signal If you want one indicator that something is real, it is not an announcement. It is a job posting.

When organisations start hiring for people to implement, supervise or clean up after a particular kind of automation, that automation is being deployed for real. Job postings are expensive to write and represent committed budget, so they lag capability by a year or two and lead visible change by about the same. Search postings in your own sector for the tooling you are worried about. If nobody is hiring around it, the deployment is not happening yet, whatever the announcement said. If they are, you have found both your timeline and, frequently, your next role.

of an evening reading. Then search job postings in your sector for anyone hiring to implement it. That search tells you more than the announcement did.

Apply this to your role

How exposed is your own week?

The free AI Exposure Audit breaks down your working week by automation risk — and gives you a personalised 30/90/365-day action plan. No account required.

Take the free audit
Share this articleXLinkedInFacebook