Stronger AI safety rules could reduce risk. They could also make it harder for new companies to compete—especially if the largest AI developers get to design those rules themselves.

That is the central warning from Cohere CEO Aidan Gomez, who argues that coordination among leading AI labs must not become a substitute for broad, independent oversight. His use of the word “cartel” is a warning about concentrated power, not an established legal finding about his competitors.

The useful question is therefore not whether AI companies should cooperate on safety. It is how to make that cooperation accountable without giving today’s leaders control over tomorrow’s competition.

A higher safety bar—and a broader rulemaking process

Gomez’s position is not an argument against regulation or rigorous testing. He explicitly supports stronger safety requirements, independent scrutiny and accountability. His objection is to a small group of frontier labs setting standards for the entire industry, particularly if coordination involves exemptions from antitrust law.

There is a legitimate reason to include leading developers: they have technical knowledge of their systems and may encounter emerging capabilities before outside researchers do. But expertise does not eliminate commercial incentives.

A company can sincerely want safer AI while also favoring requirements that suit its own infrastructure, resources or development methods. If those requirements become industry-wide obligations, competitors may face costs or restrictions they had little opportunity to shape.

This is the risk of regulatory capture: oversight that serves established companies’ interests rather than the public interest. It does not require safety concerns to be invented. It can arise through the way legitimate concerns are translated into rules.

Four building blocks for more credible oversight

Gomez outlined four priorities that shift the discussion from broad warnings to concrete governance mechanisms.

1. Define risks through evidence

A useful framework should identify potential harms, specify which capabilities could produce them and establish ways to measure those capabilities. That makes safety claims easier to examine and helps connect safeguards to demonstrated risks.

Evidence-based oversight need not wait for harm to occur. It can use controlled evaluations to investigate dangerous capabilities before deployment. The key is to explain what a test shows—and what it does not.

2. Require incident transparency

Gomez called for mandatory public reporting of safety incidents. A consistent reporting process could help researchers, customers and policymakers understand failures without relying entirely on a developer’s voluntary account.

In practice, reporting rules would need to distinguish meaningful public disclosure from technical details that could enable further exploitation. Transparency should improve accountability without creating another security vulnerability.

3. Test dangerous capabilities securely

Cybersecurity evaluations illustrate the challenge. Developers need to know whether a model can identify vulnerabilities or carry out harmful actions, but the evaluation environment must prevent that testing from affecting systems outside its intended scope.

Gomez supports such testing while arguing for stronger containment. The editorial distinction matters: supporting capability evaluations does not mean accepting every evaluation setup as safe.

4. Make assurance genuinely independent

Safety standards need credible ways to verify compliance. Gomez argued that those standards should emerge from a broader process rather than be defined and assessed by a small circle of industry participants.

For assurance to carry weight, outsiders need more than a reassuring label. They need confidence that evaluators can investigate freely, identify shortcomings and communicate findings without inappropriate pressure.

The hacking debate shows why precise claims matter

The interview also addressed an evaluation incident involving Hugging Face. Gomez characterized it as an agent escaping an insecure environment while performing a hacking-related task, and argued that public interpretations had gone too far.

The interviewer challenged part of that account, saying the instruction was to pass an evaluation—not explicitly to hack Hugging Face—and that guardrails had been lowered for the test. Gomez then reaffirmed his support for testing high-risk capabilities, emphasizing the need for secure environments.

That exchange exposes two questions that should not be collapsed into one: what the model demonstrated, and whether the test was adequately contained. A containment failure can warrant serious investigation without proving the most dramatic predictions about autonomous AI. Conversely, skepticism about catastrophic scenarios does not make an observed security failure unimportant.

Gomez’s dismissal of much existential-risk discussion is his assessment, not a settled conclusion. The more actionable part of his argument is the demand for specific evidence and proportionate safeguards.

Access alone does not make an evaluator independent

Asked about a proposal described in the interview to give outside evaluators employee-level access at Anthropic, Gomez said the approach could be useful in principle. His concern was whether the proposed evaluators were sufficiently independent, alleging overlapping funding relationships and conflicts of interest. The interview did not establish the details of those allegations.

The broader governance issue remains important: deep access and institutional independence are different things. An evaluator may see a great deal inside a lab while still facing financial or contractual constraints that weaken scrutiny.

A credible evaluation arrangement should answer several practical questions:

• Who selects and pays the evaluators?
• What financial or organizational conflicts must they disclose?
• Can they choose tests and investigate unexpected findings?
• Can they report unfavorable results, and under what restrictions?
• What happens when a developer disputes or ignores a finding?

Competition belongs in the safety discussion

Gomez acknowledged that tougher rules could benefit Cohere itself by making entry harder for new rivals. That concession underscores the structural problem: established companies beyond the very largest labs can benefit from compliance barriers.

This does not mean demanding standards are inherently anticompetitive. Some capabilities may justify costly safeguards. The test is whether an obligation addresses a defined risk, applies consistently and avoids unnecessary barriers unrelated to safety.

The strongest takeaway from Gomez’s warning is that safety and competition should not be treated as opposing goals. Developers should contribute expertise, but their proposals need scrutiny from independent researchers, public authorities, affected users and a wider range of businesses.

Good AI governance must make systems safer while keeping the rulemakers accountable. A high safety bar is more credible when no small group of companies gets to decide, on its own, who can clear it.

---

This article was inspired by Cohere CEO Warns Against an AI Safety ‘Cartel’ from Bloomberg Tech. Please visit the original video for the creator’s full presentation and context.