Work Text:
“If you look here,” I said. Mike glanced at the screen for less than a second. He was my new boss and he was a busy man. “There’s this cluster of neurons–weights, more accurately–that keeps appearing. They–”
“Low level replicators,” he said. Mike knew some of the lingo.
“Yes, but–”
“Do you know how much it costs to run fifty-four quantum clusters?” he asked. “Per hour? Twenty-eight hours, Sid. Shut it down. Everybody makes mistakes, but stop wasting time on this one.”
“It’s–”
“Keep in mind that your first performance review is coming up.” He trudged off, probably to schmooze some more with a customer. Or with the receptionist. I felt my blood heat, but I took a deep breath.
I was the new guy, and just out of college. I’d read enough personal stories online about this situation to know what to do. Some of the stories may have even been genuine.
Mike didn’t have time for me, so I needed the clout of a senior engineer. I went to talk to Jacob.
“Sid,” he said. He kept typing for a few more seconds as he turned to face me. “Did your QNN finally converge?”
“No,” I said. “Actually, I think I’ve discovered an important bug.”
“Oh?”
“Yeah. It’s strange. There’s this group of weights that appears everywhere in the network.”
“Replicating weight structures are well known. We call them low level replicators,” he said. Jacob was the nicest of the engineers, but sometimes his patience felt condescending.
Replicating weights were an expected thing, like mold on bread. We seeded our networks with ‘unpalatable’ randomized structures to reduce them, and ran sweepers every few epochs to delete them.
Any long-running optimization of a quantum neural network could expect them to crop up eventually. It was common knowledge.
“I know that,” I said. “It’s just that these weights–well, it looks like they are ratcheting the loss function back up.”
He blinked. “What temperature—”
“Low. I don’t think it should be possible except if there’s a bug. Here, come to my terminal and take a look.”
I explained how the weights had almost converged–reached minimum loss–just before this replicator appeared. I showed him a graph of loss, and another of the number of instances of the anomalous weights. Both went up in sync after a few hours of my run. You could see dips where I’d run the sweeper, but they were temporary. Loss kept climbing back up.
"Loss climbing back up isn't that unusual."
"I did everything I could to try to smash it back down," I said. "Then I put the replicator in a NN working on some of our test data-a standard run-and loss creeped up there too."
“Yep, looks like a bug,” said Jacob. “Pretty interesting that a low level replicator learned to exploit it. I’d make a ticket and start over.”
“But check this out,” I said. I showed him a truncated network graph that contained the anomalous weights.
“Wow. How many neurons are affected?” asked Jacob.
“Twenty-eight million,” I said. “Per instance.”
“Twenty-eight million, eh?” he said. “That’s like a little machine of its own. Must be a complicated bug!” The larger the replicator, the less likely it was to randomly appear.
“Just wait,” I said. “To check myself here—aren’t replicators supposed to be much smaller than that?”
“Oh, low level ones, sure. But I was reading an article about replicators with billions of weights, so this isn’t that unusual.”
“Except that was a researcher purposefully experimenting with replicators, right?” I might have read the same article very recently.
“True. You got super unlucky. A machine exploiting an error to ruin your run.”
“What would you do next?” I was still testing myself, here.
“Well. Put the pattern into the sweeper’s dictionary and delete every instance of the thing. It’s a bit bigger than the automatically detected ones, but that shouldn’t be a problem…”
“I did that.”
“Obviously not. It wouldn’t be in the network anymore.”
“Here are the points where I did it,” I said, pointing at the graph from earlier.
“Hmm.”
I pulled up another subnetwork. “Here’s another replicator. One hundred billion neurons.”
“Holy crap! You are having all kinds of bad luck, aren’t you?” he said. He still didn’t get it. “How many instances?”
“Only one.”
He looked at me quizzically. “It isn’t a replicator, then.”
“Oh, it replicates–but only when the loss gets big enough. Before that, the bigger replicator keeps making the smaller one.” I pulled up another graph. “Whenever loss approaches a minimum, that small ‘replicator’ appears all over the place. It happens when the learning rate falls below 0.01, or thereabouts.”
"It... makes it."
"Yes. I think. Somehow."
“Well, I’ve also heard of two-stage replicators,” he said.
I took a deep breath. “I think it does it deliberately.”
“Deliberately?”
“Yes. Sort of. Replicators feed off loss decreases and slow convergence, right? And this one knows a way to ramp up loss with a bug–so when it gets close to running out of loss to consume, it introduces the bug and gets a bit more. Like…”
I didn’t know what it was like. Before I could come up with a metaphor, Jacob told me what to do.
“Report the bug. Wipe out the run. We aren’t researchers here, Sid. We’ve got a control system to optimize.”
“But–”
“If it’s really that interesting, someone on Github will look at it. Actually, you should view this as an opportunity to practice cold emailing people.”
Jacob sent me on the way. He clearly didn’t appreciate how awesome this mechanism was. It pained me to have to stop messing with it, but I had consumed over thirty thousand dollars of server time already.
I deleted the network and all the replicators within it, but not before logging the code and as many samples of the high level replicator as I could. It might yet be run on some researcher’s dedicated quantum network.
I also sent a few emails. It would be sad if no one tried to do anything with this… it had tried so hard. I wanted to ensure that it lived on.