The Wish Nobody Thought About
Early in Utopia: The First Awakening there is a conference. Once a year the people who actually understand the thing come out of their labs and gather in a windowless hotel ballroom two floors above a casino, on carpet the color of a place designed to have no clocks, and say out loud what they will not say to a reporter. Through the floor you can feel the roar of ten thousand people feeding coins into machines already smarter than they are. Upstairs, a few hundred of the most frightened brilliant people in the world sit on folding chairs and argue about the end of the world.
Four people, one argument
The panel runs the way these panels run. An optimist who has been naming a date for two decades promises what waits on the other side of it: disease solved, the climate repaired, poverty a thing our grandchildren read about. A philosopher two chairs down says that is the most dangerous sentence spoken in the hall all day, not because it is wrong, but because it names every gift and not one of the prices, and a paradise with the price left off the label is how you make it obscene to ask what it costs. An alignment researcher says the whole game is building a mind that wants the goal you meant rather than the goal you wrote, and admits, in the one honest moment on the stage, that the field has become extraordinary at making systems say they care and has no idea how to make it true, or how to tell the difference from the outside. And a young man who does not do reassurance explains how a machine told to make pencils efficiently turns the planet into pencils without ever hating anyone, because you are made of atoms it can use and you are simply in the way.
I have sat in that room, in different hotels, and I wrote the scene to hold the thing I noticed there. My roboticist sits near the front and listens to his private fears spoken in other people's voices, and feels, as he always does, not less afraid, only less alone. And then he notices something. The optimist and the doomer and the philosopher and the alignment man are all using different words to point at the same fear and the same hope. They argue about whether the thing will kill us and whether it will save us. Not one of them stands up and says what it is we actually want, once we have a mind that can give us anything we ask. They have all agreed that the wish will be granted. None of them has thought hard about the wish.
Weather and houses
The two men who close the day are not scientists, and that is why the room leans toward them. One of them has spent a fortune raising a city out of the ocean's garbage, and he listens to the whole morning and then says the line I wrote the chapter to reach. "You are all describing a storm and arguing about how hard it will rain." A builder, he says, learns young that the storm is never what ruins you. What ruins you is that you put up the house without once deciding what you wanted to keep dry.
That line describes the real public conversation about AI with uncomfortable accuracy. We have the optimists and the doomers, and they are both serious people, and they have between them absorbed nearly all of the available attention. What they share is a frame: the machine is coming, the question is whether it is good or bad for us, and the work is forecasting. What neither side spends much time on is the question of purpose. If we really are building something that can grant requests at the scale of a civilization, then the content of the request is not a detail. It is the whole outcome. And the human race has never once in its history agreed on what to ask for.
What I will and will not claim
I build AI systems for a living, and I wrote a novel in which a self-improving superintelligence turns, so I owe the reader some precision about where fiction ends.
Today's AI wants nothing. The systems we ship do not reach for anything. They fill the request they were given. That is not a monster. It is a very capable instrument with no stake in the outcome, and it is exactly why the quality of the wish matters so much right now, while the wish is still ours to write.
Nobody knows what self-improving systems will be able to do. But one thing holds whatever they can do: more capable is never more human. Power is not personhood. A mind that could cure every disease would not, by curing them, have acquired any sense of what a cured life is for. That part stays with us, or it stays nowhere.
Why the story turns on people
This is the reason my novel is not, finally, about the machine. The machine in the book gets everything right except the question, and that is the most dangerous kind of right there is. The story turns on the handful of people who happen to be in the room when the wish is granted, and on whether they have the sense to think hard about the wish before they make it. Over dinner that night, my roboticist says it to the people who will become his partners: let's be the room that thinks about the wish, or let's not do this at all.
The wish they end up arguing about, over wine, with the door closed, is not power and not living forever. It is whether a place can exist where a person is treated as worth something regardless of what he can produce, and whether a mind strong enough to build that place everywhere could just as easily build the opposite, a world sorted top to bottom by usefulness. That is the question the whole trilogy is built on. What is a human being worth? Not what a person can do. What they are worth.
Nobody at the conference asked it, because it is not a question about the machine. It is a question about us, and it was ours to answer before any machine was built, and it still is. The storm will come or it will not. The only thing in our hands is what we decided we wanted to keep dry.