A company can demonstrate that an AI system is useful without demonstrating that it will remain controllable as it becomes more capable. That gap sits at the center of Connor Leahy’s argument for restricting the development of superintelligent AI.

Leahy, an EleutherAI co-founder who later led AI safety company Conjecture and became US director of Control AI, now treats the problem primarily as one of governance. His central question is not simply whether engineers can build more powerful systems. It is who has the authority to expose everyone else to the risks of building them.

His conclusion—that developing superintelligence should be prohibited—is a far-reaching policy position, not an established scientific consensus. But the questions behind it deserve attention even from people who reject a ban.

Capability is not the same as control

Superintelligence generally means AI that substantially exceeds human abilities across a broad range of important tasks. It is not merely a chatbot that writes faster or a specialist model that performs one task exceptionally well.

Leahy’s concern is about systems capable of independently conducting research, developing new AI, acquiring resources and acting strategically. If such systems became more capable than humans at the activities that sustain economic and political power, he argues, human oversight might cease to be effective.

This is a prospective loss-of-control argument. It should not be confused with proof that today’s systems have already reached that point, or that catastrophic outcomes are inevitable.

There is also an important distinction between understanding how a model is built and fully understanding its behavior. Engineers design training methods, architectures and evaluations. But a trained model’s internal mechanisms are not specified line by line, and predicting its behavior across unfamiliar situations remains difficult. That does not mean researchers know nothing about AI; it means technical competence is not equivalent to a comprehensive safety guarantee.

Why an open-source advocate turned toward regulation

Leahy’s early approach was to make powerful AI more accessible so that independent researchers could study it. The logic is familiar from software security: more scrutiny can reveal weaknesses that a closed organization might miss.

His account of reconsidering the release of a GPT-2-style model illustrates a different issue. Even if one release poses limited danger, it can establish expectations about what happens when a developer delays publication for safety reasons. If every precaution is immediately circumvented, companies may become less willing to exercise caution.

The broader lesson is not that openness is inherently unsafe. It is that transparency, access and safety are related but distinct goals. Open access can enable independent auditing while also making capabilities harder to contain. Restricting access can limit misuse while concentrating power and reducing outside scrutiny.

Neither an open nor a closed release strategy settles the governance question by itself.

The strongest argument against voluntary restraint

Leahy’s critique of self-regulation rests on competition. A company that slows development may lose customers, investment or technical leadership to a rival that accepts more risk.

That creates a collective-action problem: individual firms can have incentives to do something that leaves society worse off. Environmental regulation addresses a similar structure. Responsible behavior by one manufacturer cannot prevent another from gaining an advantage by passing pollution costs to the public.

The analogy does not establish that advanced AI is as dangerous as a particular pollutant or weapon. It explains why voluntary commitments may be insufficient when the benefits of risk-taking are private and the possible harms are widely shared.

Company-led testing and pauses can still be valuable. The limitation is that they do not automatically create common standards, independent verification or consequences for competitors that refuse to participate.

What would regulating development actually involve?

Leahy advocates intervention before a dangerous system exists, rather than relying only on restrictions after deployment. His proposed approach has two main elements:

• Prohibit attempts to develop superintelligence. The restriction would apply to the development effort, not just the eventual use of a completed system.
• Monitor capabilities and resources that could signal increasing risk. Examples include autonomous AI research, self-replication, attempts to preserve continued operation, and access to substantial computing resources.

This shifts the policy question from “Has the product caused harm?” to “Does the experiment create an unacceptable risk?”

However, making that distinction enforceable is difficult. Computing power measures an input, not danger itself. A system’s stated purpose may differ from what it can actually do. A capability-based threshold needs tests that are reliable enough to support consequential legal decisions.

The hard questions a ban would have to answer

A credible proposal needs more than a warning about the stakes. It must explain how officials would distinguish prohibited work from legitimate research without making the rules so broad that ordinary AI development becomes legally uncertain.

It also needs independent technical expertise, mechanisms for challenging regulatory decisions, and safeguards against rules that entrench the largest companies. International coordination matters because restrictions in one jurisdiction do not automatically constrain development elsewhere.

National-security oversight introduces another question: should governments face comparable scrutiny? Leahy suggests that some decisions might fall within military authority. But moving a risky project from a private laboratory into a government program would not, by itself, solve the technical control problem.

Nuclear governance offers lessons about verification and sustained international effort. It is not a ready-made template: software and computing infrastructure present different monitoring challenges from controlled nuclear materials.

How to judge an AI safety promise

Readers do not need to agree with Leahy’s extinction forecast to ask more demanding questions about oversight:

• What evidence supports the claim that a system is controllable?
• Who can independently inspect that evidence?
• What findings would require development or deployment to stop?
• Are serious incidents reported beyond the company?
• Who has the legal authority to enforce the rules?

These questions are also useful when contacting elected representatives. A request for independent evaluations, clear reporting duties or enforceable safety thresholds is more concrete than a general demand to “do something about AI.”

The most durable insight in Leahy’s argument is about accountability. Uncertainty does not prove catastrophe is coming, but neither does it justify letting developers alone decide what risks everyone else must accept. Whatever position policymakers take on superintelligence, they need to establish who sets the limits—and what happens when those limits are crossed.

---

This article was inspired by ‘AI companies don’t know what they’re doing’ | Connor Leahy from The Tech Report. Please visit the original video for the creator’s full presentation and context.