Photo-Illustration: Intelligencer; Photo: Getty Images
In recent months, AI models have become much better at writing, debugging, and testing code. It has accelerated the AI race, made software-focused Anthropic its leader, and sent everyone else in pursuit of a new, more focused goal: improving coding tools and gathering as many excited and terrified enterprise customers as possible.
However, as these tools have become more capable of writing software, they have also become much better at figuring out how to break it. AI is increasingly automating parts of coding software development, but – at least as much and perhaps even more rapidly – it is also automating hacking. Malicious actors have been using AI to help speed up and enhance their capabilities for over a year, with limited but real success; Also, software companies and open-source projects are using AI tools to harden their software find new Weaknesses. Recent increases in the coding capabilities of models clarified the situation. If a software developer can now oversee a fleet of coding agents to deliver more features in less time, he or she can also do so with the goal of finding exploitable flaws in other people’s software to exfiltrate data, shut down systems, or hold them hostage for ransom.
All of this becomes especially difficult when technology makes a massive, sudden leap forward, which is meant to describe how AI frontier model releases – which come from a small group of large companies – might work. anthropic They say Its next model is one such case:
Earlier today we announced Cloud Mythos Preview, a new general purpose language model. This model performs strongly everywhere else, but it’s surprisingly capable at computer security tasks. In response, we have launched Project Glasswing, an effort to use Mythos Preview to help secure the world’s most critical software, and to prepare the industry for the practices we will all need to adopt to stay ahead of cyber attackers.
By “highly capable of performing computer security tasks” Anthropic’s testers mean, among other things, that the new model makes it much easier to find exploits in existing software. Testers using this version of the cloud were able to “identify and then exploit zero-day vulnerabilities in every major operating system and every major web browser,” while non-experts were able to use it to “find and exploit sophisticated vulnerabilities.” Programmer Simon Willison takes a look at some examples and reports from Glasswing partners Anthropic’s claims seemed compelling. “There is so much smoke in here that I think there is a fire,” he wrote. “Finding vulnerabilities in decades-old software is no surprise, especially given that they are mostly written in C, but what’s new is that coding agents driven by the latest frontier LLMs are proving to be tirelessly capable of solving these issues.”
This creates an awkward situation in the short term, where a company, in the process of building a general purpose AI model with a specific focus on coding, acquires the exclusive ability to exploit countless thousands of pieces of software deployed by competitors, governments, and institutions around the world. (Same company, it should be noted, that the US government recently penalized it for raising a separate set of security concerns and attempted to designate it as a supply chain risk.) Anthropic’s absurd levels of latent power will continue until competitors catch up, and/or until its tools – or tools like it – can systematically protect secure software against empowered hackers. Anthropic has not yet released this model to the public, but it won’t last long, and alternative tools adapted for malicious use will continue to follow it.
So, for now, the most obvious way to prepare for the security risks posed by new models is to use them to test and patch your own software ahead of time if possible. Anthropic’s decision to donate “$100 million in usage credits” to the “defenders” shows that they know how awkward their situation is, and how uncomfortable it can make everyone else. Given that this next model is likely to be far more capable of directly productive coding tasks, Anthropic’s business is going to have a very, very, good year, as it The alarm continues to sound About… well, it has its own success.
Anthropic is making every effort here to explain that it did not intend to semi-blackmail the entire world – sorry, it just happened! And, contrary to the widespread rumor about Mythos in the world of AI and cybersecurity, the company is surprisingly optimistic about the long-term effects of automated high-level hacking in its release. Outlining a staggering array of vulnerabilities that the model was able to help uncover in often surprising ways, Anthropic’s own security testers write that “[m]ost security tooling has historically benefited defenders more than attackers,” and believes “eventually the same will be true here,” while acknowledging that the interim period “may be tumultuous though.”
It’s an interesting posture coming from a security-focused lab that spends a lot of time talking about serious AI risks. Now that one of them is being realized, it is signaling both a need for urgency and a general understanding that through cooperation and planning and market mechanisms things can eventually work out, and that widely distributed new capabilities will result in a new “equilibrium” – a term you don’t hear often from labs that talk about rapidly growing capabilities. Hackers have new tools, but so do the people making software, who will be able to use them to root out problems and identify lingering security problems, which will eventually result in a situation similar to where we are now (which, admittedly, most cybersecurity professionals will tell you is terrible, but not as apocalyptic as they can remember).
This is at least spiritually similar to the recent rhetoric among some tech elites away from projections of imminent and rapid labor replacement and toward a more familiar story. Just capitalism, but with greater productivity, Who doesn’t so much dispute those claims as dismiss them as short-term distractions: :
This may provide some valuable data about the bigger picture here of how the next year will unfold in cybersecurity. On the one hand, earlier AI capabilities – generating plausible speech for different contexts, realistic images, videos, etc. – have increasingly disrupted, broken, or destabilized the systems and markets where they have been deployed, for example, sending a shock through education, and oversupplying social networks with careless and other inauthentic content. They matter. On the other hand, they have made it a little easier ThinkingAt the very least, about how those systems may need to change, or their participants may need to adapt, to function going forward. An important question is how “turbulent” the interim period of each system will be, and whether, in the process of achieving a new equilibrium or status quo, some participants will be left behind. The second is how long it will last, or whether it will ever actually end.
As far as systems go – and I’m using that term loosely here – the cybersecurity status quo seems unusually vulnerable to near-term “disturbance,” as countless poorly maintained software deployments quickly become accessible targets for a growing pool of less-skilled hackers. Maybe, as Anthropic suggests, thousands of different actors will spring into action, coordinate and collectively take software security to new heights with new tools, and everyone will move on with their lives. This would be a good story to tell about AI. Or maybe a group of hackers with jailbreak models may soon be able to hold entire countries hostage with a single laptop. Looks like we’ll find out!
