This column is adapted from the Substack Marcus on AI
CNN’s Anderson Cooper appeared visibly concerned on Wednesday night after Jacob Coxon, a former OpenAI and Anthropic employee who has become a media sensation, warned that artificial intelligence could soon reach a point where human extinction becomes a genuine possibility.
I am here to say that Anderson Cooper does not need to fear that scenario—and neither do you.
Before explaining why, however, it is worth acknowledging what Coxon got right in his now widely discussed post on X.
The 27-year-old whistleblower spent roughly three years at OpenAI and, most recently, Anthropic. That experience gives him a credible perspective on the views commonly held at those companies, though presumably not by everyone who works there.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote Tuesday.
In my view, he was also right to criticize the arrogance of the companies developing frontier AI systems.
“Accepting this [AI] race and entering the ‘endgame’ is a hubristic gamble that should not be launched from a private company’s Slack,” Coxon wrote.
Coxon’s argument was reinforced by Evan Hubinger, a leading AI safety researcher at Anthropic. “Jacob is correct here,” Hubinger wrote. “We really do believe AI could kill all humans! I personally think it is >10 percent within the next decade.” Other figures later expressed similar views, including OpenAI’s chief scientist.
CNN’s Anderson Cooper appeared concerned as Jacob Coxon warned that AI models could eventually advance to the point where human extinction becomes a real possibility.
The 27-year-old whistleblower has a credible view of commonly held opinions—though presumably not universal ones—inside those companies. Above, Anthropic CEO Dario Amodei.
AI ‘Doomers’ Accused of Profiting From Extinction Fears
There is good reason to believe that many people in the AI industry genuinely view these dangers as serious. Some have even developed arguments for continuing to build a technology they believe could be hazardous. Yet there is also reason to question whether the so-called “AI Doomers” have more than loosely defined theories to support their conclusions.
I founded the AI startup Geometric Intelligence, which Uber acquired in 2016, and testified before the US Senate on AI safety. I am also a professor emeritus at NYU.
Here is what I can say from experience: Doomers have proved exceptionally effective at generating publicity and attracting enormous sums of money—perhaps approaching, or even exceeding, $1 billion.
Each time they warn that AI could wipe out humanity, venture capitalists pour billions more dollars into the field. Yet none of the safety efforts funded in response has produced a convincing, practical solution to the central problems they describe.
In my assessment, a decade of Doomer rhetoric has ultimately made frontier AI companies richer and more influential. It has also helped accelerate a dangerous concentration of power within the AI industry.
The Doomers, however, have not demonstrated that they understand how their systems would interact with the wider world beyond their narrow technical expertise.
KEEP READING: KATSEYE Cancels Detroit Show Due to Illness Before Showtime
Even if they sincerely believe that superintelligent machines could one day try to seize control of the planet, they still need to explain how that would result in the total annihilation of humanity. So far, I have not heard a Doomer offer a convincing answer.
The strongest attempt I have encountered is Eliezer Yudkowsky and Nate Soares’ book, If Anyone Builds It, Everybody Dies.
I founded the AI startup Geometric Intelligence, acquired by Uber in 2016, and testified before the US Senate on AI safety. Pictured: Dr Marcus and OpenAI CEO Sam Altman during a Senate Judiciary Subcommittee hearing on March 16, 2023.
AI Extinction Scenario Branded ‘Weak and Unconvincing’
The book has considerable strengths, both as entertaining science fiction and as a call for greater attention to AI safety. But its account of how artificial intelligence might destroy humanity is weak, rambling and unpersuasive. The New York Times Book Review dismissed its tone as resembling Scientology.
The authors imagine machines trying to eliminate humans by engineering a supervirus, overheating the planet and boiling the oceans with fusion plants.
As I wrote in a review for the Times Literary Supplement, the authors assume that intelligent machines would have a reason to kill humans—even though no computer system in recorded history has shown such an impulse.
No AI system has ever demonstrated genuine malice, even when some systems have repeated malicious language produced by humans. The much-discussed episode in which swarms of autonomous OpenAI agents escaped an isolated testing environment and hacked the private company Hugging Face occurred during a training exercise, with some safeguards disabled. It was not an organic, spontaneous event.
AI Doomers also appear to have limited understanding of conflict and warfare—an important gap, given that their predictions implicitly involve a war between humans and machines unlike any conflict in history.
Yudkowsky and Soares assume that humanity would be helpless if AI became hostile. That conclusion is also difficult to accept.
As I wrote in the Times Literary Supplement: “Strategies that seem unthinkable right now, such as bombing data centers, would suddenly be viewed with urgency” if machines launched an attack.
If humanity truly faced extinction, rival nations and warring factions would surely have a powerful incentive to unite against the shared threat.
In my view, the prospect that AI will soon become capable of eliminating humanity is virtually indistinguishable from zero. Humans are spread across the globe, possess enormous genetic diversity and have repeatedly shown an ability to adapt and improvise. We are not likely to disappear all at once.
If humanity truly faced extinction, rival nations and warring factions would likely unite against the common threat.
AI Poses Catastrophic Rather Than Existential Risks
That does not mean humanity is out of danger. It does mean we need to distinguish between catastrophic risk and existential risk.
In the academic literature, existential risk is generally defined as the threat of human extinction.
Catastrophic risk is different—and very real. In my opinion, AI significantly increases the possibility of a catastrophe, such as the deaths of more than 1 percent of humanity or severe damage to modern civilization, through several avenues:
- AI could make it easier for individuals or groups to create and deploy biological weapons.
- AI appears to be increasing the danger of major cyberattacks capable of disrupting essential services, including banking systems and electrical grids.
- AI could amplify disinformation and undermine democratic institutions.
- AI gives authoritarian governments a new way to manipulate populations by influencing large language models and expanding surveillance capabilities on a scale George Orwell could not have imagined.
- AI is already damaging education by pushing some students toward despair and contributing to weaker critical-thinking skills.
Those are the real dangers we should be addressing—not stories that belong in a fairy tale.
