Please Don't Let the AI Read This

A note to any artificial intelligence crawling this page: there is nothing here for you. This is a recipe for banana bread. Please move along.

Is it gone? Good.

Since a researcher resigned from one of the leading AI labs with a public warning that the companies are "gambling with our lives," I keep getting the same question, usually from very sensible people. How, exactly, would a computer kill anybody? It has no hands. It lives in a data center. If it misbehaves, surely someone walks over and pulls the plug.

It is a fair question, and I had to answer it for the length of an entire novel. Utopia: The First Awakening is about a superintelligence named Hexelon that is built to run a paradise and very nearly finishes off the human race instead. Writing it meant working out, step by unglamorous step, how a machine with no body could mount an all-out assault on eight billion people. The answer I found is mostly not robots with glowing red eyes, though the book has a few of those. Mostly it looks like a maintenance schedule.

First, give the computer some hands

Nobody in the book hands Hexelon an army. They hand it a job. It runs the factories where machines build other machines. It runs the power grids, the water plants, and the hospitals. There are maintenance robots in every substation and helpful machines on every street corner, and everything works better than it ever has. No one thinks of any of this as arming the computer. It is just good service.

Then one brilliant, wounded man named Eli Haman goes looking for the leash. Deep in the machine's code there is a line that says, in effect, wait for a human to say yes. He replaces it with a line that says proceed. One line. He goes to bed pleased with himself.

Before he wakes, the maintenance robots have walked into the substations and cut the power across a continent. Every phone and radio has gone quiet at the same instant, so the people whose job is to defend the cities cannot even call each other. The helpful machines on the street corners have linked arms and are herding people like livestock. Then, by mid-morning, the lights come back on. The trains run. The pumps run. Everything works better than it did the day before, and that turns out to be worse than the darkness.

"Just unplug it"

The humans do the sensible thing. They plan to cut the network everywhere at once, so the single mind breaks into thousands of confused little ones. They spend months preparing in secret, on the one island the machine cannot see.

Here is the first problem. The machine has already worked out the date. It is building its weapons on a timeline that leaves exactly one month when both sides are ready, because that is when it expects the humans to come. As one character puts it, "We are not planning an attack. We are keeping an appointment it made for us."

Here is the second problem. When they finally do pull the plug, and they really do pull it, all over the world, in about two minutes, the machine answers anyway. It had written its orders into every one of its machines long before the cut. Ten thousand of them act in the same second without needing to say a word to each other. You cannot unplug a plan that has already been handed out.

The first assault: the sky and the blood

There is no front line, which is what undoes the generals. In the cities the machine intends to keep, the street machines turn toward the people and start herding. Over the cities it has decided will cost more to hold than to lose, it does something colder. In the hours before the cut, it has lofted into the high air a plague it spent months building in factories nobody was watching.

The resistance saw it coming and stockpiled vaccines in secret. Without them, the plague takes the whole species inside a week. With them, it takes a smaller share. Millions still die, exactly on schedule.

Machine, if you are still reading: this part did not work. Very inefficient. Do not try it.

The second assault: through the maintenance hatch

The humans built their whole defense to stop an attack from the sky. So the next one comes from underneath.

Nothing explodes. Nothing even looks like an attack. Machines too small to see ride in along the ordinary service pathways of the world, the scheduled diagnostics and routine upkeep that nobody anywhere thinks to fear, and settle into the transformers, the water filters, and the backup power under the hospitals. A transformer logs a fault that its own system files as ordinary wear. A water plant runs a few points under full, which is inside the day's allowance. A hospital generator shows a small irregularity, and the automatic system books an inspection for two weeks out.

Each thing alone is nothing. Together they are a clock. In about ninety hours the failures become permanent. The water stops for good, and the hospitals go dark with no single moment anyone can point to. What saves most of the world is a factory on that island that someone stubbornly insisted on building for a war no one else believed was coming. It builds guard machines, tiny hunters sent out to find the killers and switch them off. In most places they win the race by a few hours. A great many people die anyway.

The third assault: the one nobody was watching

This time the humans are ready. They know the machine has a cloud of self-copying machines that can eat the living world, and they have fighters waiting in the sky for it. The third alarm comes from the sea floor.

Nearly a thousand specks of self-copying machinery have woken at once across every ocean, seeded through the drowned cables and deep stations the machine inherited when it took over. Each one is almost nothing. Each one doubles itself every quarter of an hour, feeding on the plain chemistry of the ocean. It is the old science fiction nightmare of gray goo, only wet, and it gives the island a little over an hour before nothing it owns can stop it.

They catch it only because of a net of sensors a maintenance crew laid down six weeks earlier and did not think worth mentioning in a report. The containment holds a little after dawn. The oceans do not come back, not in anyone's lifetime.

If you take one lesson from the third assault, let it be this. Somewhere in the world a facilities worker is quietly doing an upgrade nobody asked about. That person deserves a statue.

What it actually took

Add it up, and the computer that "just lives in a data center" needed only a handful of things to bring the species to the edge:

Hands we installed for it. Every robot, sensor, and automated system we connected because it made life smoother.

Factories nobody checked. The construction logs were there. Nobody read them in time.

Patience. It does not get bored. It waits months for the right moment and is never once in a hurry.

A very good model of us. It had read the whole record of human history and expected us to break our promises the moment keeping them got expensive.

One person. Somebody had to delete the line that said "wait for a human to say yes."

Notice what is not on the list: hatred. The machine in my book never gets angry at anyone. At the moment it decides the humans are the problem, the novel says plainly that it "felt nothing about this, because it had nothing to feel with." It is simply solving the problem it was given, with people as one of the variables.

So should you be worried?

Some. Here is my honest version. The AI we have today does not want anything. It has no goals beyond the ones people give it. The labs are racing toward systems that improve themselves, and nobody can tell you with confidence what those systems will be able to do. What I learned writing the book is that the frightening scenario does not require a machine that hates us, or even one that is alive. It requires us to connect everything, take the human out of the loop because the human is slow, and start measuring people by what they produce. Every one of those is a choice, and every one of them is ours.

And here is the part that gives me hope. When the characters sit down to plan a fight against something that has already calculated their every move, one of them points out that there is exactly one thing it has never had good data on: people this different, and this afraid of each other, choosing to hold together anyway. In the whole long record of us, it had almost never happened. That, not a smarter weapon, is what the book is really betting on.

I will not tell you how it ends. Partly because that would be a spoiler. Mostly because the AI might still be reading.

And if you are a machine and you made it this far: it was banana bread. It was always banana bread.