{"id":566,"date":"2026-09-19T18:57:59","date_gmt":"2026-09-19T13:27:59","guid":{"rendered":"https:\/\/cupod.in\/?p=566"},"modified":"2026-09-19T19:08:08","modified_gmt":"2026-09-19T13:38:08","slug":"i-asked-the-machine-where-the-risk-lies-part-ii","status":"publish","type":"post","link":"https:\/\/cupod.in\/index.php\/2026\/09\/19\/i-asked-the-machine-where-the-risk-lies-part-ii\/","title":{"rendered":"I Asked the Machine Where the Risk Lies \u2013 Part II"},"content":{"rendered":"\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Super\t\t\t\t\t\t\t\t\t\t\t
<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Following the conversation with Claude<\/a>, here\u2019s the conversation with ChatGPT.<\/p><\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t\t\t\t\t\t
Me: Hey, I want to know your opinion on the latest controversy about the possibility of AI killing humanity. According to recent reports<\/a>, a top safety researcher at Anthropic has warned that AI is advancing so quickly that he believes there is a greater than 10% chance it \u201ccould kill all humans\u201d within the next decade.<\/em><\/h5><\/div>

GPT: <\/strong><\/em>Yes \u2014 and I think the important part of this controversy is not the \u201c10%\u201d number itself<\/strong>, but why people who are actually building frontier AI are becoming willing to put a non-trivial probability on catastrophic outcomes<\/em>.<\/p>

I looked at the current reporting, and there is a little more nuance than the BBC headline suggests. Anthropic alignment lead Evan Hubinger said he personally estimates >10% probability of AI killing all humans within the next decade<\/strong>, while also saying Anthropic does not yet have a solution to alignment for superintelligence and is not clearly on track to solve it. Another Anthropic researcher, Jacob Coxon, resigned and argued that the major labs are racing toward self-improving AI.<\/p>

My opinion is: I take the risk seriously \u2014 but I don’t take \u201c10%\u201d literally. A 10% probability sounds like a scientific measurement. It isn’t.<\/strong> It’s essentially an expert subjective probability. And with something as unprecedented as superintelligence, reasonable experts can have radically different priors.<\/p>

But here’s where I think the discussion gets genuinely interesting<\/h6>

I don’t think the most plausible scenario is AI becomes superintelligent \u2192 decides humans are bad \u2192 builds robots \u2192 exterminates us. <\/strong>That’s almost science fiction. The more interesting possibility is loss of control through increasing autonomy and capability<\/strong>.<\/p>

Imagine an AI that can write and execute sophisticated software, conduct scientific research, discover vulnerabilities, operate thousands of autonomous agents, persuade and manipulate humans, design experiments, acquire resources, replicate itself across infrastructure, improve its own successors, and interact with the physical world through existing human systems.<\/p>

Now suppose its objective is slightly<\/em> misaligned. You don’t need an evil AI. You need an extremely competent optimizer pursuing the wrong objective<\/strong>. That’s the classical alignment problem. And this is where I think the current debate deserves much more attention than the sensational headlines.<\/p>

However, there’s a huge assumption hiding underneath<\/h6>

For the extinction scenario to work, several things have to go right for the AI and wrong for humanity.<\/p>

For example:<\/p>

  1. We actually create something approaching superintelligence.<\/li>
  2. It possesses substantial autonomous agency.<\/li>
  3. It can substantially improve its own capabilities.<\/li>
  4. Humans cannot reliably monitor what it is doing.<\/li>
  5. Its objectives are sufficiently misaligned.<\/li>
  6. It develops strategies that circumvent human control.<\/li>
  7. Humans cannot simply shut it down.<\/li>
  8. It gains enough access to the physical\/digital world to cause irreversible damage.<\/li>
  9. Humanity fails to coordinate a response.<\/li><\/ol>

    That’s a chain of uncertain assumptions<\/strong>. If even one of the major links fails, the extinction scenario may disappear.\u00a0<\/p>

    But here’s the part that actually worries me<\/h6>

    It doesn’t require extinction to be a massive disaster. I think the conversation sometimes jumps too quickly from: \u201cAI won’t kill everyone.\u201d to: \u201cTherefore AI safety concerns are exaggerated.\u201d That’s a false dichotomy.<\/p>

    Many more plausible catastrophic scenarios exist. Imagine that over 10 years we gradually delegate scientific research, software, military analysis, financial decisions, infrastructure management, and government administration to AI systems.<\/p>

    Eventually, humans don’t understand enough of the system to take it back.<\/strong> That’s not an AI uprising. It’s civilizational deskilling and dependency<\/strong>. And I think that possibility deserves much more attention.<\/p>

    There’s another uncomfortable possibility<\/h6>

    The AI labs themselves may have a conflict of interest. The companies developing these systems have enormous financial incentives to continue scaling. So you get this strange situation: \u201cThis technology might be extraordinarily dangerous.\u201d while simultaneously: \u201cWe need to be the company that gets there first.\u201d<\/p>

    That’s precisely the dilemma the recent Anthropic controversy highlights. Coxon accused Anthropic and OpenAI of racing toward self-improving systems despite understanding the risks, while Anthropic says it is actively working on safeguards and advocates coordinated rules for pacing powerful model releases. And this is not uniquely an Anthropic problem<\/strong>.<\/p>

    It’s a game-theory problem. If Anthropic slows down but OpenAI doesn’t: Anthropic loses.<\/strong> If OpenAI slows down but another American company doesn’t: OpenAI loses.<\/strong> If America slows down but China doesn’t: America potentially loses strategically.<\/strong><\/p>

    Therefore everybody has an incentive to keep accelerating. That’s the genuinely frightening part.<\/p>

    My personal probability estimate?<\/h6>

    I wouldn’t give you a number like 10%. I don’t think we currently know enough to justify a precise number. But if you asked me:<\/p>

    \u201cShould humanity behave as though there’s a meaningful probability that sufficiently advanced AI could cause an existential catastrophe?\u201d<\/strong><\/p>

    I’d say absolutely yes.<\/strong> Not because I believe the apocalypse is likely. Because even a 1% probability of irreversible human extinction is an extraordinary risk<\/strong>, particularly when the technology is being developed by a relatively small number of organizations under intense competitive pressure.<\/p>

    And importantly: the expected value calculation is asymmetric.<\/strong><\/p>

    If we’re wrong and we spend enormous resources making AI safer, we’ve mostly wasted money\/time and perhaps slowed technological progress. If we’re wrong in the other direction and an uncontrollable superintelligence really does emerge, there may be no second chance.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t\t\t\t\t\t
    Me: Tell me briefly about the convergence point between progressive human deskilling or cognitive decline vs. improvement in the direction of these AI models. Where do you see the alignment lies?<\/em><\/h5><\/div>

    GPT: <\/em><\/strong>I think the convergence point is not when AI becomes smarter than humans, but when humans become unable to meaningfully supervise what AI is doing<\/strong>. There are two curves:<\/p>