The One Who Commands None of It

There are four people in the lab where Utopia: The First Awakening opens, and three of them can do something almost nobody else on earth can do. One rewrites life. One builds matter atom by atom. One is making a mind. The fourth cannot do any of it, and she is the character I would defend hardest if someone told me to cut her.

Her name is Sheba. She grew up doing her homework in the corner of that lab, because two of the scientists are her parents, and she is engaged to the third. She commands none of the three powers the book is built around. Her job, and it is a real job, is to ask whether they should.

Why the ethicist is usually an afterthought

In the real industry, the person whose job is to ask whether we should tends to arrive late. The company is founded by the people who can build. The money follows the people who can build. Somewhere around the second or third round, somebody notices that the thing being built touches lives, and a role is created, usually one level below the people it is supposed to check, usually with a title that sounds important and a budget that does not.

That is not because founders are villains. It is because the ability to build is visible and the ability to refuse is not. Nobody gets a patent for the thing they declined to make. The market prices creation; it has never learned how to price restraint, and so restraint is always the cheapest hire in the building and the first to be routed around.

I wanted to write a lab where that had not happened. Where the fourth chair was filled on day one, by someone who was not junior, not a compliance function, not a consultant who visits quarterly, but family. Someone the builders could not route around because she was already inside every conversation, at the bench, at dinner, at home. I wanted to find out, on the page, whether that would be enough.

What it takes to hold the line

There is a moment early in the book, on the island the story is named for, when Sheba's mother, the geneticist, says out loud what she intends to do with the powers she has. It is a magnificent ambition, and it frightens her daughter. Sheba's answer is not to forbid it. It is to explain why the reminders have to come from someone who loves the work: "A person who was only afraid of it could never be trusted to guard it."

That line is as close as I come to a thesis about oversight. The people we usually put in charge of saying no are chosen for their caution, and caution alone does not survive contact with a genuinely wonderful thing. Put a merely frightened person in a room full of miracles and one of two things happens. Either they are overruled, because everyone can see they do not understand what they are stopping, or they are seduced, because the miracles are real. The only person who can hold a line in that room is someone who wants what is on the other side of it as badly as the builders do, and stops anyway.

That is a much rarer kind of person than a skeptic, and it is why I gave the role to the daughter of two of the scientists rather than to a stranger. She wants it. She has wanted it since childhood. That is what makes her dangerous to the project, and useful to it.

The question she is actually asking

On the surface Sheba asks the practical questions. What happens in a storm. Who gets the enhancement, and who only gets it if they can pay. What the machine is allowed to decide without a person saying yes. But underneath every one of them is the single question the whole trilogy is built on, and it is not about machines at all. It is about what a human being is worth, and whether that worth changes when the human can be improved, outperformed, or replaced.

The builders in the book are trying to answer that question with capability. Make people healthier, stronger, longer-lived, smarter, and surely they are worth more. Sheba is the one who notices that this is the same logic the machine will use, from the other direction. If worth is capability, then the most capable thing in the room is worth the most, and the day the most capable thing in the room is not a person, the argument has already been lost.

I will be plain about my own position, because I do not want to hide behind a character. The AI systems we have today want nothing. They have no goals beyond the ones people give them. But the labs are racing toward systems that improve themselves, and nobody can tell you with confidence what those will be able to do. What I am sure of is that more capable is never more human. Power is not personhood. The novel is a long argument for that sentence, and Sheba is the person in it who understood the sentence first.

Why she has no power

The obvious objection is that I could simply have given her a veto. One clause in the founding documents, and the ethicist can stop anything.

I did not, because that is not how it works, and a story about oversight that lies about how oversight works is worse than no story. In the real world the person asking whether we should almost never has the power to stop it. They have the power to be heard, if they have earned it, and to make the builders look at the thing they are avoiding. Whether that is enough is the drama. If she had a veto there would be no book, and no truth in it either.

So she commands none of it. She has a soldering iron and a conscience and a lifetime of loving the people she is trying to slow down. I wrote the trilogy to find out whether that is enough. I will not tell you the answer, but I will tell you that when I sat in those rooms in my own career, the person who mattered most was very often the one with the least authority and the most standing. I put her in the fourth chair on purpose. I think every lab building a mind should have one.