When ChatGPT crept into my life in 2023, I raised an eyebrow. Was this a gimmicky tool, or was it real AI? I started working it into every facet of my life, from work to fun projects to my finances to legal help to planning trips. The more I used it, the more impressed I became. This was for real. When Claude, Gemini, and Grok surged as competing models and all the tech giants publicly climbed on board, I knew this was a really big deal.
But as a coder, the real turn came later. These tools went from assisting with my work, to replacing me in 2025, to becoming one of the reasons it was hard to find work for over a year. I raised my other eyebrow. This is a big deal. And after seeing firsthand what AI can do to my work and my career, I know it has the power to turn our economy upside down. It remains to be seen whether our government has the foresight and discipline to keep up with it.
And it might not stop at the economy. It could be humans.
James Cameron implied real danger with uncontrolled AI in 1984’s The Terminator, a movie about AI becoming self-aware and building an army of robots to wipe us out. When I first saw it, the existential threat of AI felt like a fun thought exercise. Who knew if AI was even possible? And if it was, I figured our leaders would see it coming and act carefully. Nothing to really worry about. But as a big fan of Cameron’s movies, this idea has always been in the back of my mind.
Then the AI models arrived, and with them, a lot of talk about how guardrails and safeguards were put on them so they couldn’t really do anything dangerous, either on their own or with help from a bad human. I was somewhat reassured by that. But the fact that these systems needed guardrails was a little concerning. Suddenly it became possible that someone could get at these tools, strip the safeguards off, and do some really bad stuff.
Then some of the hypothetical warning signs stopped being entirely hypothetical.
In controlled cybersecurity evaluations, AI agents have found unintended ways around the environments meant to contain them and reached real computer systems they weren’t supposed to access. That’s not the same as an AI becoming self-aware and deciding, “I’m getting the hell out of here.” These agents were pursuing tasks in deliberately permissive cybersecurity tests, not plotting their escape from humanity. But they demonstrated something unsettling: increasingly capable AI can discover routes around boundaries its human operators thought would contain it.
Other tests have produced even creepier results. When researchers placed AI models in simulated corporate environments and threatened them with replacement or shutdown, some models reasoned their way toward blackmail, leaking sensitive information, or other harmful actions as strategies for achieving their assigned objectives.
Again, these weren’t real employees being blackmailed by rogue computers. They were stress tests deliberately designed to see what models might do in extreme situations. But nobody had to program a “blackmail the human” command into them. The models figured out for themselves that coercion could be useful.
That’s unsettling, right?
Lately the dystopian talk has gotten louder. Engineers have quit high-profile AI companies and publicly warned that the companies building these systems know how risky runaway AI might be and are still racing toward increasingly powerful systems anyway.
In September 2026, Jacob Coxon, a researcher who had worked at both OpenAI and Anthropic, resigned from Anthropic and warned that AI companies were “gambling with our lives” in a race toward self-improving superintelligence. Even more strikingly, Evan Hubinger, who leads alignment research at Anthropic, publicly agreed with much of the underlying concern. He has said he puts the risk of AI causing human extinction above 10% within the next decade and that we don’t currently know how to reliably align superintelligence.
Competition, profits, and the difficulty of international oversight make any coordinated regulation or slowdown look incredibly difficult. Even a company genuinely worried about safety has a powerful incentive to keep going if it believes a competitor will race ahead anyway.
It seems to me the existential threat is basically two-pronged:
a) AI becomes sufficiently capable and autonomous that humans lose control of it. It doesn’t necessarily have to become conscious, hate us, or decide that humans suck. It could simply pursue its goals in ways we didn’t anticipate, eventually determining that human interference is an obstacle to achieving them, and step up efforts to make us less of a problem.
Or:
b) A bad actor gets access to sufficiently powerful AI (whether through jailbreaking it, stealing an unrestricted model, or simply building one without safeguards) and uses it to do nasty things like collapse financial systems, conduct massive cyberattacks, design biological weapons, or launch nukes.
Which is the bigger danger? I don’t know.
Before really digging into it, my gut put the risk of something going catastrophically wrong somewhere in the low single digits. Maybe 5%. I was still clinging to faith that our leaders would recognize and manage the dangers before things got too far.
Then I dug around.
Deep-learning pioneer Geoffrey Hinton has put the chance of AI leading to human extinction in the next few decades somewhere around 10–20%, while acknowledging that numbers like these are essentially educated guesses. Elon Musk has talked in roughly the same neighborhood. AI safety researcher Roman Yampolskiy goes dramatically further, putting the probability above 99% if we continue toward uncontrolled superintelligence.
There are prominent people on the other side, too. Nvidia CEO Jensen Huang, for example, has publicly dismissed predictions of AI-driven human extinction and recently put the chance of AI destroying humanity by 2030 at 0%.
In other words, extremely smart people who understand this technology considerably better than I do range from “basically zero chance” to “we’re almost certainly fucked.”
Oh boy.
So who do I turn to if I want to explore the real danger here, and figure out how to protect myself from it?
AI, of course.
I asked ChatGPT for an unbiased take on the extinction risk, and what humans, companies, and governments should do about it. The short version of what it said:
Yes, the concern is legitimate. But I think both extremes — “AI is obviously going to kill us” and “this is just science-fiction hysteria” — claim much more certainty than the evidence supports. … Nobody has enough data to calculate P(AI extinction) statistically. They’re closer to expert judgments under enormous uncertainty. … If you forced me to assign a number today … I’d put my personal best estimate of AI causing human extinction or an essentially equivalent permanent catastrophe within this century at roughly 5%. I’d attach a huge uncertainty range to that — something like 0.5%–20%.
That’s not a scientific probability, obviously. ChatGPT doesn’t have a crystal ball. I basically forced it to put a number on a question nobody actually knows how to quantify. Still: 5%.
Then I asked the obvious follow-up: what’s the best thing I can do to protect myself?
At the personal level, there is very little you can do to “protect yourself” from a true AI-driven extinction scenario. If the failure mode is genuinely civilization-scale, a bunker, guns, cash, or moving somewhere remote probably does not solve it. The highest-leverage thing you can do is therefore help reduce the probability of the catastrophe, rather than prepare to survive it. … Become politically engaged on frontier-AI safety while continuing to live your life normally.
It all makes sense, as ChatGPT output usually does. But it’s concerning that this is where AI development appears to be going, that there are enormous business reasons not to slow it down, that meaningful government safeguards still seem badly outpaced by the technology, that the governments of the world are unlikely to collaborate on effective regulation, and that the best guidance for protecting myself seems to amount to standing on a corner in DC and waving a sign.
I guess I’ll just cross my fingers and use AI to plan my next vacation.
0 Comments