Work Text:
I would always rather that a human element remain... in determining something so critical as someone's fate.
In your nightmare, your friends have logic on their side; all you have is desperation.
Before you even started work on the Machine, you saw the dangers. The fundamentals that allowed your idea to function would, someday, all too soon, be twisted to give power to those who would abuse it. Used not to save lives, but to destroy them. With the Machine, you hoped to provide the core functions in a way that couldn’t be exploited -- and, somehow, to stave off that inevitable future, if only for a little longer.
That future is upon you now, weeks or days or hours away; you’ve all been working feverishly to hold it off for as long as possible, knowing the attempt would ultimately prove futile.
In your nightmare, though, the Machine has found a way to push it back for months, to give your team some breathing room... maybe even give you enough time to discover a more long-term solution to keep it at bay. Your creation has given you the option, and left the decision in your hands.
The tactic makes sense, and you have every reason in the world to do it. Your friends know this, and they’re trying to persuade you to accept it. Just this one little step, this simple, obvious step. The Machine has evaluated the data, more data than any human could possibly get a handle on, and has pointed you at the linchpin. Take care of this one variable, and you win -- for now. It’s in your power; it’s easy. It would solve so many problems. Save lives.
In your nightmare, you’re pleading with your friends not to take that step. Begging, increasingly desperate, knowing that you’re arguing for an action that will -- not might, but will -- result in the deaths of innocent people. There’s even a little part of you that wants your friends to hold fast, that wants them to make the choice instead of you, because you do understand: If you win this argument, people will die. Here, now, their lives depend on you. Even as you argue that the threat is merely a possibility, you know with leaden certainty that it’s as good as fact.
You also know that it doesn’t -- shouldn’t -- matter. Because your friends are arguing to take a life. Not in the heat of battle, not when he’s actively threatening to harm someone; not even, like Benton, an unrepentant destroyer who will, if set free, go on to feed again on the misery of new victims. There are times when your team has to kill, and you hate it, but you can deal with it. Sometimes an aggressor must die, to protect an innocent.
But the man your Machine singled out isn’t an aggressor; he’s not deliberately trying to harm others. Nor is he putting money ahead of people’s lives, like the CEO you once took a dark pleasure in bringing to financial ruin. Congressman McCourt honestly believes that he’s doing the country a service -- one most people wouldn’t appreciate, but a necessary one. And oh, how well you understand that burden. But he’s got the wrong data, and no reason to trust you after you’ve essentially kidnapped him... and his obstinance is going to get a lot of people killed.
Killing him would save all those people -- at least for a while, which is, honestly, all you can hope for right now. And with the feds closing in, you don’t have a third option: It’s kill him here, or leave him be. Letting him live means accepting all the deaths his actions will inevitably bring about.
You know this. Your little team is all that stands between this threat -- this very real threat -- and the hundreds, perhaps thousands, it might destroy. Just because you haven’t been given their numbers doesn’t make their deaths any more bearable.
And yet… to take a life. An innocent life.
Deliberately.
It’s your nightmare because it goes against everything you’ve ever fought for -- even before you understood the nature of the war. You created the Machine to save lives, but it’s been arranging assassinations since before you handed over the keys. You recovered Reese, and then Shaw, and reforged them for a new purpose, struggling to pull apart decades of training designed to turn them into monsters -- and yet now, the friends whose lives you saved are the very ones arguing for the kill, and basing their arguments on the sense of ethics they learned from you.
Every step you’ve ever taken was to prevent this outcome, and yet here you stand, watching it unfold, hearing your rationalizations fall useless as raindrops, fizzling against a growing wildfire.
When you first encountered the Trolley Problem, the answer had felt trivial: Save the most people. It’s the same reasoning that let you turn your back on the Irrelevants, and it took the death of Nathan to force you to acknowledge that sometimes, it was wrong to choose the many over the few. The dilemma sits at the crossroad of two moral evils: Kill, when you could walk away; let people die, when you could save them. Neither action is acceptable: People are not a thing that you can sacrifice. But there is no right answer here, and you’re no closer to a good solution than you were while training the Machine.
You built the Machine to sort through the data and find the threats -- but not to make the decisions; while it seemed to have a sense of morality, you couldn’t trust its judgment. Yet your friends maintain that you should trust it now, and, ironically, for the same reason that you can never do that: You’re the one who built it. They have a confidence in your work that you can’t share; they never saw the failures, suffered the fallout when the self-preservation subroutines overrode the program’s core function.
They never wrestled with the decision to cripple what, by that point, felt all too much like a child. And they can’t know the stark terror of realizing that your creation has grown beyond the constraints you designed -- the only possible means you could see to keep it in check. Let alone that it managed this feat long before you ever found out about it, and that by now it’s autonomous and well beyond the control of any human being.
It’s a different kind of nightmare, one that hits you in the daytime, in the spaces between the Numbers: when you can’t help but think about the Machine as more than just a source of information. About what it means that it was able to plan, and then bring about, a convoluted method of preserving its deleted memories; that it was able to make decisions you never programmed it for. About how your crystal-clear directive -- You can’t do that again; your job is to protect everyone, not to protect me -- was countermanded by a singularly unyielding man, who didn’t even have to write a single line of code.
The Machine is still trying to protect you -- Root explained that much -- but the overwhelming part, for you, is not knowing what it might sacrifice to that end. It’s one thing for John to prioritize your life, but he can, at most, fail to help a couple dozen people while trying to save you; the Machine’s influence is nowhere near that limited. And where you once terminated an early attempt because it had learned to lie, your living creation has proven itself capable of startling manipulations, up to, and including, designing a human identity for itself.
How far would it go to save your life? Would it be willing to kill innocent people, even knowing you don’t want that? Might the Machine, for example, try to convince your team to murder someone, not even for the greater good but specifically to save you? It’s a possibility that you wish you could ignore.
From the moment this idea started unfolding -- from the moment you heard, from John’s lips, the confirmation that it had truly, finally, come down to this -- you have tried every tactic you can think of, raised every argument, levied all your wit and considerable strength of will, and yet… and yet you can see in their eyes that it isn’t enough. They’d hoped you would come around; it pains them to take this action without you. But they’re convinced that this is the right thing to do. Your reservations simply aren’t enough to weigh in against the lives they’re trying to save.
The train chugs on down the track, inexorable, and you would throw yourself in front of it if you thought it would do any good. Find a way -- any way -- to have this not happen.
It’s not a new nightmare. You’ve struggled with this idea since first dreaming up the Machine, and it’s only gotten more detailed as you perceived the reality drawing closer. So many nights you’ve been in the throes of that dream state, and it never feels like a dream; every time, no matter the shape the nightmare takes, it feels like you’re really there. Like it’s really happening, and nothing you can do will stop it. Like you need to give in.
But the moment has finally come, and the only reason you understand that you’re not just dreaming again -- that you won’t wake up from this moment, that whatever you do here will be irreversible -- is that, for the first time, the arguments your friends make don’t bring you around to agree with them.
Every nightmare ends the same way: your resistances worn down, your mind finally accepting the pragmatism of the act. Your finger on the trigger. But here, in reality, it’s your hand on John’s arm, pleading with him not to let this happen. Because regardless of the consequences of letting this man live, you cannot accept a scenario that turns you and your friends into murderers.