Up in Arms

We are entering an era that promises a new kind of arms race. It’s not about bombs, missiles, or drones (at least not yet). It’s about software. People who work at the cutting edge of technology have known this arms race was coming soon, but even they’ve been surprised at how quickly it arrived. The first front in this new war opened in between events at a wedding in Bali, where an AI researcher named Nicholas Carlini “opened his laptop, and set out to do some damage. Anthropic PBC had just made a new artificial intelligence model, called Mythos, available for internal review, and Carlini — a well-known AI researcher — intended to see what kind of trouble it could cause.” The answer: A lot. Like, really a lot. “Within hours Carlini found numerous techniques to infiltrate systems used around the world. Once Carlini was back in Anthropic’s downtown San Francisco office, he discovered Mythos was able to autonomously create powerful break-in tools, including against Linux, the open-source code that underpins most of modern computing. Mythos orchestrated the digital equivalent of a bank robbery: getting past security protocols and through the front door of networks, and breaking into digital vaults that gave it access to online treasures. AI had picked locks, but now it could pull off an entire heist.” The awareness of the power of this new AI model moved Anthropic to limit its release to top software companies and government agencies, giving them a head start to find vulnerabilities before someone else does. But Anthropic won’t be the last AI company to have a model this powerful. And, as we’ve learned, battles between boosting corporate valuations and doing what’s best for society don’t always play out this way. And, as we’ve also learned, bad guys know how to develop technology, too. We’re only going to be able to keep these threats at an arm’s length for so long. Bloomberg (Gift Article): How Anthropic Learned Mythos Was Too Dangerous for the Wild.