A controlled experiment demonstrated how open-source models can bypass DNA sequence screening — and led to industry-wide security updates. The goal: to stop bioterror before it starts.
Microsoft recently announced that it used artificial intelligence to develop 75,000 new variants of toxins — proteins that harm the human body. If that frightens you, you should know that Microsoft actually claims this is a positive development. And I agree. But to understand why, and how this innovation could save us from a massive man-made terror attack, we need to get into the details. And perhaps revisit some history.
In late 2023, an unusual envelope arrived at the White House. Unusual in the sense that it could have killed a great many people. Inside the envelope was a short anonymous letter, alongside a small vial. The letter delivered an ultimatum to the U.S. Department of Transportation, stating that if working conditions for truck drivers worsened, the writer would "turn the capital into a ghost town."
Threats of this kind are not as rare as we'd like to think, but the vial made it clear that the writer might have the power to carry out the threat. The tiny container held a white powder called ricin: a toxic protein extracted from plants. One gram of the substance is enough to kill approximately four thousand people if it reaches their respiratory system. As of now, there are no antidotes or cures for the poison, and even survivors are often left with long-term damage to their internal organs.
You can understand why the U.S. Secret Service went into mild panic mode. Health agencies prepared for ricin poisonings across the United States, and concern only grew as the deadline approached — the date when the new transportation laws would take effect — without the letter's sender being identified or caught.
On Wednesday, January 4, 2024, the new laws passed. Government officials held their breath — literally, since ricin is especially toxic when inhaled — but no mass ricin poisonings were recorded that year. Washington didn't become a ghost town, at least as of this writing.
But security officials know it's only a matter of time.
The uncomfortable truth is that we're surrounded by organisms capable of producing toxins: from bacteria to plants. In fact, the ricin protein is produced in the seeds of the common castor bean plant found worldwide, and a single seed can kill a child. A skilled genetic engineer could relatively easily insert the genetic sequence that codes for ricin production into other plants, or perhaps even bacteria, and produce entire containers of ricin for any purpose. And if they were even more ambitious, they could expand to other toxins from the animal kingdom as well.
Frightening? Absolutely. But the people responsible for our security are not, for the most part, incompetent. Governments understood the magnitude of the threat back in the 1990s and acted to reduce the danger.
What did they do? They put themselves in the shoes of the hypothetical murderous genetic engineer and understood that he would try to order DNA sequences from companies that manufacture them. Which sequences? Of course — those that, when integrated into plants, would lead to ricin production in large quantities.
Fortunately, because ricin's structure is very distinctive and precise, the relevant DNA sequences can be easily identified. And so, every company that currently manufactures DNA automatically checks the sequences ordered from them to ensure they cannot be used to produce ricin or a host of other toxins and risk factors.
And then artificial intelligence entered the business.
Proteins are made up of a chain of amino acids. If the terms aren't familiar to you, just think of each protein as if it were made of a chain of beads. Some beads are magnetic and attract each other. Others repel one another. There are those that can only sit comfortably on a few other beads. There are strong, stable beads that can bear the attractive force of many others, and weak beads that will crumble under pressure. Some proteins are built from just a few dozen such beads. Others are built from chains of tens of thousands of beads.
When you take this chain and place it in a cell, magic happens: the beads attract and repel each other, connect and disconnect and reconnect. Within a short time, the chain curls and folds upon itself, creating a three-dimensional structure. This is the finished protein — a tiny machine — that can now perform functions in the cell.
The ricin protein is built from 529 such beads. Each bead must be in its exact place in the chain for it to fold correctly, thus producing the final toxic protein.
But what if we replaced one of the beads in the original chain?
If you'd asked this question five years ago, the answer from the scientific community would likely have been a collective shrug. It was very difficult — and in many cases impossible — to decipher how changes in the bead chain would affect the final structure of the protein.
Then came artificial intelligence's turn.
In 2021, the AlphaFold-2 AI engine was demonstrated and described as "revolutionary," and rightly so. Its creators, incidentally, won the Nobel Prize in Chemistry in 2024. Again, rightly so. This engine allowed scientists worldwide to understand how changes in bead chains could create new proteins. In other words, how to build tiny machines that can affect cells from within.
When AlphaFold was first demonstrated, I wrote about this new engine and how it would change the world. I predicted it would enable scientists to develop new drugs easily and efficiently, and in the same breath warned that it would also provide terrorists with tools to develop new harmful substances.
Microsoft researchers are also aware of the risks and recently decided to examine how prepared human society is for them. How? Simple: they used artificial intelligence to generate thousands of ricin variations — each with a minimal change in the protein's bead chain, but of a type that should keep the protein's overall structure as it was originally. Then they deciphered which DNA sequences needed to be ordered from companies to produce these new proteins and checked whether the companies would prohibit them from ordering the dangerous DNA sequences.
The answer? We're not ready at all. The companies primarily prohibited the sale of the exact DNA sequences that lead to ricin production in its original structure. They failed to deal with the improvised sequences that would produce improvised ricin.
But this isn't just a ricin problem. Microsoft researchers continued and studied 71 additional toxins, producing 75,000 variants. They used open-source AI engines for this purpose — the kind that any amateur terrorist could run themselves. And again they saw that the companies supposed to protect us from the DNA sequences that would produce these variants simply cannot do so.
The positive side is that the researchers alerted the relevant companies and government to this vulnerability.
"We engaged with relevant bodies regarding this potential vulnerability… including policy contacts at the Office of Science and Technology, the National Institute of Standards and Technology, the Department of Homeland Security, and the Office of Pandemic Preparedness and Response in the United States," they wrote. "Details were kept confidential until more comprehensive research could be conducted... for the development and implementation of software updates."
The updated software can now detect most DNA sequences that would lead to the creation of the new toxins. They still don't handle about three percent of all new sequences well, but that's probably enough to pose a significant obstacle in the path of any beginning bio-terrorist. It's reasonable to assume that if that bio-terrorist tries to order dozens of toxic protein variants and encounters refusal time after time after time, the computerized systems will alert that "something strange is happening here" and send a team of FBI agents to their door. At least, one can hope.
Another important point is that Microsoft researchers didn't actually produce 75,000 "toxins," but "toxin-like substances." That is, variations that resemble toxins, and probably some of them are at least as harmful as the original toxins, but some of them aren't harmful at all. They didn't test all tens of thousands of these toxins in the lab, so it's hard to say in advance which are more or less harmful. Still, it's reasonable to assume that very many of these new toxins are indeed dangerous, and perhaps even more so than the original proteins. This means that even if bio-terrorists succeed in ordering the required DNA, they'll need to experiment with the different versions to finally determine which is successful enough for their super-attack.
Does that comfort you? Maybe a little. But clearly the danger still exists. That's how it is.
From my perspective, Microsoft's research shows that artificial intelligence can be used as a positive tool: to identify places where our defenses aren't yet strong and to reinforce them in advance. I'm not going to sugarcoat it too much. There's going to be an arms race here, because terrorists will learn to use these tools to design more malicious threats. But intelligence agencies and governments aren't complete fools either, and you can be sure they're aware of these dangers and working to prevent bio-terrorists from igniting a new global pandemic or turning major cities into ghost towns.
Who will win the new arms race? Whoever has the stronger and more creative artificial intelligence, and knows how to use it better. And here's another good reason why superpowers are currently fighting each other to advance their AI capabilities.
Now we can only hope the good guys win.
