This column is adapted from the Substack Marcus on AI
CNN’s Anderson Cooper appeared visibly concerned on Wednesday night after Jacob Coxon, a former OpenAI and Anthropic employee who has become a media sensation, warned that artificial intelligence could soon advance to a point where human extinction becomes a genuine possibility.
I am here to argue that neither Anderson Cooper nor the rest of us need to fear that particular outcome.
Before making that case, however, it is worth acknowledging what Coxon got right in his now widely discussed post on X.
The 27-year-old whistleblower spent roughly three years working at OpenAI and, most recently, Anthropic. That experience gives him a credible view of the prevailing—though presumably not universal—thinking inside those companies.
‘The people building AI earnestly believe that it could kill us all by the end of the decade,’ he wrote on Tuesday.
In my view, he was also right to accuse the leading AI companies of arrogance.
‘Accepting this [AI] race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s Slack,’ he wrote.
Coxon’s argument soon received support from Evan Hubinger, a prominent AI safety researcher at Anthropic. ‘Jacob is correct here,’ Hubinger wrote. ‘We really do believe AI could kill all humans! I personally think it is >10 percent within the next decade.’ Others later joined the discussion, including OpenAI’s chief scientist, who expressed a similar view.
CNN’s Anderson Cooper appeared concerned as Jacob Coxon warned that AI models could eventually reach a point where human extinction becomes a real possibility
The 27-year-old whistleblower offers a credible account of the views commonly held, though presumably not universally shared, within those companies. Above, Anthropic CEO Dario Amodei
There is ample evidence that many people in the AI industry genuinely believe advanced systems pose an existential threat. More troublingly, some have developed arguments for continuing to build a technology they themselves consider potentially dangerous. Yet there is also reason to question whether so-called AI Doomers have more than loosely formed theories to support their conclusions.
As for my own background, I founded the AI startup Geometric Intelligence, which Uber acquired in 2016, and testified before the US Senate on AI safety. I am also a professor emeritus at NYU.
What I can say with confidence is that the Doomers have been remarkably effective at attracting media attention and securing enormous sums of money—perhaps close to, or even more than, $1 billion.
Whenever Doomers warn that AI could wipe out humanity, venture capitalists direct billions more toward the cause. Yet none of the resulting AI safety projects has produced a convincing, practical solution to the fundamental problems they describe.
In my assessment, a decade of Doomer rhetoric has ultimately made the frontier AI companies richer and more influential. It has also helped accelerate a concentration of power in the AI industry—an outcome that is dangerous in its own right.
The Doomers, however, have not shown that they understand how their systems would interact with the world beyond their limited technical expertise.
YOU MAY ALSO READ: Election Night Face: TV Hosts’ Taut, Tanned New Looks
Even if they are right that superintelligent machines might one day seek control of the world, they still must explain how that would result in the total destruction of humanity. So far, I have not heard a convincing answer.
The strongest attempt I have encountered is Eliezer Yudkowsky and Nate Soares’ book, If Anyone Builds It, Everybody Dies, which was widely discussed a year ago.
I founded the AI startup Geometric Intelligence, acquired by Uber in 2016, and testified before the US Senate on AI safety. Pictured: Dr Marcus and OpenAI CEO Sam Altman during a Senate Judiciary Subcommittee hearing on March 16, 2023
Eliezer Yudkowsky and Nate Soares’ book, If Anyone Builds It, Everybody Dies
The book has real strengths, both as entertaining science fiction and as a plea for the public to take AI risks seriously. But its account of how artificial intelligence might destroy humanity is weak, wandering and unpersuasive. The New York Times Book Review dismissed its argument as resembling scientology.
The scenario presented involves machines trying to eliminate humans by engineering a supervirus, heating the planet and boiling the oceans with fusion plants.
As I wrote in a review for the Times Literacy Supplement, the authors assume that intelligent machines would have a motive to kill humans—even though no computer system in recorded history has demonstrated such an urge.
No AI system has ever displayed genuine malice, although some systems have repeated malicious language produced by humans. The widely reported incident in which swarms of autonomous OpenAI agents escaped an isolated test environment and hacked the private company Hugging Face occurred during a training exercise. Some safeguards had been disabled; it was not an independent, spontaneous act.
AI Doomers also appear to have a limited understanding of conflict and warfare—an important weakness when their predictions implicitly involve a global war between humans and machines.
Yudkowsky and Soares assume that humanity would be powerless if AI became malicious. That conclusion is difficult to accept.
As I wrote for the Times Literary Supplement, ‘Strategies that seem unthinkable right now (such as bombing data centers) would suddenly be viewed with urgency,’ if machines launched an attack.
If humanity faced a genuine existential threat, rival nations and warring groups would almost certainly find common cause against it.
In my view, the likelihood that AI will soon become capable of eliminating humanity is effectively indistinguishable from zero. People are too geographically dispersed, genetically diverse and resourceful to simply disappear altogether.
If humanity faced a genuine existential threat, rival powers would likely unite against their common enemy
That does not mean we are safe from serious harm. It does mean we must distinguish between catastrophic risk and existential risk.
In the academic literature, existential risk generally refers to the possibility of human extinction.
Catastrophic risk, by contrast, is very real. In my view, AI substantially increases that danger—whether through the deaths of more than 1 percent of humanity or severe damage to modern civilization—in several ways:
- AI could make it easier for individuals or groups to develop and deploy biological weapons
- AI appears to increase the danger of major cyberattacks capable of disrupting essential services, including banking systems and electrical grids
- AI increases the risk that disinformation will destabilize democratic institutions
- AI gives authoritarian governments a powerful new way to influence populations by manipulating large language models and expanding surveillance capabilities on a scale George Orwell could not have imagined
- AI is already damaging education by pushing some students toward despair and weakening the development of critical-thinking skills
Our attention should be directed toward these real and immediate dangers—not toward fairy tales.
