So Don't Build It

There is a sentence that almost never gets said out loud in a room where people build powerful technology. It is not forbidden. It is just treated as unserious, the kind of thing an outsider says because they do not understand the work. The sentence is: so don't build it.

I gave that sentence to the ethicist in the first chapter of Utopia: The First Awakening, and I put it in the mouth of the one person in the room who understands the work completely, because I wanted it to be impossible to dismiss.

New Year's morning

The scene is small. It is New Year's Day in a lab in Berkeley, the kind of lab that has grown one improvised layer at a time around the people who need it. Four people are there because none of them has ever agreed to close it. Three of them hold the powers the book is about: genetics, nanotechnology and robotics. The fourth, Sheba, holds none of them. She grew up doing homework in the corner of rooms like this one, and she is engaged to the roboticist.

They are watching a documentary about the garbage in the Pacific, and the conversation drifts, the way conversations in labs do, from the ocean to the thing they are actually funded to build. Troy, the roboticist, explains the grant. They were given the money to answer a single question, and he states it plainly: "Can a thing that much smarter than us be built so that it still wants what we want." Then he gives the honest answer. Nobody on Earth knows, "and we are building it anyway."

That is the moment Sheba speaks. "So don't build it."

Why she has to say the rest

If the scene stopped there, it would be a familiar argument, and a weak one. Every builder has heard "don't build it" from someone who does not know what the thing is for. They have answers ready: someone else will build it if we do not, the benefits are too large to forgo, it is better that careful people get there first. Those answers are often sincere and sometimes right.

So Sheba does not stop. She keeps going, past the comfortable part, into the motive. "We're going to build it, and manage the risk, because the truth is we all want to see the thing get born more than we're afraid of it." Then she looks at the man she is going to marry and adds, "You most of all."

Troy says it is not fair. He says it gently, and the book notes that he does not say she is wrong. Her mother, the geneticist, says something sharper: that Sheba is the only one in the room being honest about why they do it.

I wrote that exchange because I have sat in versions of that room for most of my working life. Not with a superintelligence on the table, but with the same shape of conversation. Everyone in the room can describe the risks. Everyone can describe the benefits. Almost nobody describes the third thing, which is often the strongest force in the room: the builder's plain desire to see the thing exist.

The motive nobody lists

I do not think that desire is shameful. It is close to the heart of why anyone makes anything. A builder wants to see the bridge stand, the code run, the city float. The same wish that drives people to cure diseases and clean oceans drives them to build things they cannot fully predict. You cannot cut it out of a person without cutting out the reason they are good at the work.

But a desire that is never named cannot be weighed. When a team talks only about risk and reward, the desire to build hides inside the reward column and quietly inflates it. Every benefit looks a little larger, every danger a little more manageable, because somewhere underneath the analysis, the people doing it already want the answer to be yes.

Naming it does not require a confession or a ceremony. It can be as plain as one more line in the decision: we want to build this, and here is how much of our confidence comes from wanting to. A team that can say that out loud can also ask the useful follow-up. If we did not want this so much, would the evidence still persuade us? Sometimes the answer is yes, and the work goes ahead on firmer ground. Sometimes it is no, and that is worth knowing before anything is built rather than after.

That is what Sheba does in the first chapter. She does not win the argument. She is not trying to. She moves the desire from underneath the analysis to the top of the table, where everyone can see it. After that, the decision is the same, but it is no longer a decision anyone can pretend was purely rational.

What I am and am not claiming

It matters to me to be precise here, because I write about AI from inside the industry. Today's AI wants nothing. The desire in this scene belongs entirely to the people. Nobody knows what self-improving systems will eventually be able to do, and the book takes that uncertainty seriously, but more capable is never more human. Power is not personhood.

The novel is not built on the question of whether machines are dangerous. It is built on people, and on what they decide a human being is worth when they want something badly enough. That question starts on page one, in a quiet lab, on a holiday morning, with a sentence nobody wants to hear.

I do not think every powerful thing should go unbuilt. I build things for a living. But I have come to think that the most honest room is the one where someone is allowed to say "so don't build it," and where the people who build anyway can say, out loud, why they want to.