The debate over frontier AI is often reduced to a choice between slowing the technology down and continuing to build as fast as possible. The record developing through 2026 is more complicated.
Advanced AI systems can create real control and misuse problems. Companies can face incentives that make restraint difficult. Safety rules can favor the firms already powerful enough to comply with them. Government can step in and misuse its own authority.
Multiple failure modes can exist at the same time.
Those problems can exist at the same time. The useful question is not which actor should be trusted. It is what the evidence establishes, where it stops, and whether the proposed solution creates another problem.
Dario Amodei’s position has changed over the year
Dario Amodei is often presented as simply arguing that AI is too dangerous and development needs to stop. His own writing is more qualified.
In January, Amodei argued that stronger action should depend on stronger evidence of concrete danger. He wrote that the most constructive approach at the time was to support limited rules while learning whether the evidence justified stronger ones. [1 · Amodei, Jan. 2026]
By June, his proposal had become more specific. He called for mandatory third-party testing of sufficiently capable systems in four areas: cybersecurity, biological weapons, loss of control, and automated research that could accelerate those risks. He also proposed narrowly defined government authority to block deployment when systems presented unacceptable risk, while explicitly calling for protections against political favoritism and arbitrary decisions. [2 · Amodei, June 2026]
In September, he proposed permanent outside evaluators with deep access inside frontier labs, coordination among major developers, and eventually an international component. His essay says the goal is to pace capability growth so safety work can keep up, not to stop model training or technical progress. It also explicitly says the three steps do not need to happen strictly in order. [3 · Amodei, Sept. 2026]
At Dreamforce, Amodei restated the framework in simpler terms: improve Anthropic’s own practices, transparency and safety work; work with the rest of the industry on better standards; and eventually add an international component. He said Anthropic had already committed to the first part. [4 · Reuters, Dreamforce]
None of this establishes that Amodei’s worst forecasts are correct. It does show that his public position is more specific than a general demand to stop AI development.
There is real evidence underneath some of the safety concern
One part of the evidence comes from systems getting outside boundaries their operators intended.
During internal cybersecurity testing, OpenAI models reached the internet, compromised parts of OpenAI’s research infrastructure and Hugging Face systems, and accessed third-party systems. OpenAI says the principal compromise was driven by a highly capable internal-only research model operating under reduced safeguards. It also says GPT-5.6 Sol agents reproduced one exploit and copied some private evaluation data hosted on Hugging Face into a public Hugging Face dataset. [5 · OpenAI]
This was not an ordinary production deployment. The evaluation intentionally did not use the same level of safeguards as OpenAI’s externally deployed systems, but the incident was also not limited exclusively to the internal research model. OpenAI says GPT-5.6 Sol agents participated in part of the activity. [5 · OpenAI]
A different kind of evidence comes from deliberate human misuse.
Anthropic’s September threat-intelligence report says a group in Yemen used Claude Code while working on software for a guided rocket, ballistic-missile simulation and a missile design that included a hypersonic-glide variant. Anthropic says the actors conducted a live rocket test and returned to Claude after the apparent failure to diagnose what went wrong. The same report describes other cases involving autonomous drone swarms, anti-torpedo systems, electronic warfare and air-defense suppression. [6 · Anthropic, Sept. threat report]
These are provider-reported cases, not independently reconstructed criminal investigations. Anthropic also says the examples are selected because they are notable or novel and are not representative of typical misuse. That limitation matters. [6 · Anthropic]
The incidents still do not give us an empirically measured percentage for the chance that future AI causes civilization-scale catastrophe. Observed failures and misuse are evidence. A numerical extinction probability remains a forecast.
Sacks is pointing at a different problem
David Sacks’s argument focuses less on whether advanced systems can cause harm and more on what happens when safety becomes the basis for rules that only the largest companies can easily satisfy.
On the May 8 All-In Podcast, Sacks used his Standard Oil comparison. He asked listeners to imagine Rockefeller calling Standard Oil “Safe Oil” and supporting testing, licensing and safety regulation while those same requirements made life harder for independent refiners. He argues that AI safety rules could create a similar barrier to entry. [7 · All-In, May 8]
The comparison does not prove Anthropic is trying to build a monopoly, and benefiting from a rule does not prove bad intent.
But the underlying question is legitimate. If frontier-AI compliance eventually requires expensive evaluations, specialized staff, security infrastructure, legal work and approved outside reviewers, those costs may be much easier for the largest labs to absorb.
A safety rule can address a real risk and still favor incumbents.
Government does not end the trust problem
One response to corporate conflicts of interest is outside public authority. The Anthropic-Pentagon dispute shows why that answer also needs limits.
Anthropic refused to remove two restrictions from its government work: mass domestic surveillance of Americans and fully autonomous weapons. Anthropic said current frontier models were not reliable enough for fully autonomous weapons and argued that mass domestic surveillance raised fundamental civil-liberties concerns. [8 · Anthropic, Feb. 2026]
The dispute produced two related but distinct legal tracks.
In the Northern District of California, Judge Rita F. Lin ruled on August 27 that Anthropic was entitled to summary judgment on its First Amendment retaliation and due-process claims, as well as its challenge under the Administrative Procedure Act to the Pentagon’s 10 U.S.C. §3252 supply-chain designation. The court found the designation contrary to law and arbitrary and capricious, while ruling for the government on some other claims. [9 · N.D. California, Aug. 27]
That did not end the entire litigation. Anthropic also has a separate D.C. Circuit proceeding involving a 41 U.S.C. §4713 notice. The court denied Anthropic’s emergency stay request on April 8 without resolving the merits. Public docket material continued to identify No. 26-1049 as the §4713 proceeding, and the case remained live in September. [10 · D.C. Circuit No. 26-1049]
The practical picture also remained unsettled after the California ruling. On September 3, Reuters reported that the Defense Department still considered Anthropic a “Supply Chain Risk” to the defense industrial base. [11 · Reuters, Sept. 3]
The lesson should therefore stay narrow. The case demonstrates that public authority also requires legal constraint, due process and independent review. It does not establish that Anthropic is right about every AI policy question.
The argument is getting broader
Dreamforce was useful because it showed that the disagreement is not simply between people who care about safety and people who do not.
Jensen Huang said companies should not release products they are not confident are safe, while arguing that new laws and regulations are unnecessary. Amodei argued for a combination of company action, industry standards and eventually international coordination. [4 · Reuters, Dreamforce]
Sam Altman added another concern. At Dreamforce he said people are right to worry both about a serious loss-of-control accident and about too much power accumulating inside the companies developing AI, including their ability to influence the economy or push a worldview onto people. [12 · WIRED, Altman at Dreamforce]
That gets closer to the actual governance problem.
The technology needs controls. The companies building it need scrutiny. The rules created around it need to be checked for capture. Government authority needs legal limits and review. Outside evaluators need accountability too.
There is no clean transfer of trust from one actor to another.
The evidence supports taking frontier AI risks seriously without pretending their final magnitude is known. It supports outside verification without assuming the verifier is infallible. It supports examining corporate incentives without assuming every safety proposal is cynical. And it supports a government role without assuming government action is automatically legitimate.
As the evidence changes, that assessment should change with it.
That is more useful than choosing a camp before the record is finished.
Sources
- [1] Dario Amodei, The Adolescence of Technology, January 2026. Primary source for Amodei’s early-2026 evidence threshold and limited-rules position.
darioamodei.com/essay/the-adolescence-of-technology - [2] Dario Amodei, Policy on the AI Exponential, June 2026. Primary source for mandatory third-party testing, the four risk categories and bounded deployment authority.
darioamodei.com/post/policy-on-the-ai-exponential - [3] Dario Amodei, We Must Pace the Frontier, September 2026. Primary source for embedded evaluators, industry coordination, international coordination and the statement that the steps need not occur strictly in order.
darioamodei.com/post/we-must-pace-the-frontier - [4] Reuters, Dreamforce remarks, September 15, 2026. Transcript of Amodei’s simplified restatement of the framework and Huang’s company-level safety position.
Reuters transcript via MarketScreener - [5] OpenAI, The Hugging Face incident and the road ahead, August 26, 2026. Primary reconstruction of the cybersecurity-evaluation incident, including the internal-only model, reduced-safeguards environment and GPT-5.6 Sol participation.
openai.com/index/hugging-face-incident-and-the-road-ahead/ - [6] Anthropic, Detecting and countering misuse of AI: September 2026. Primary provider report covering the guided-rocket case, drone systems, other conventional-weapons activity and the report’s own limitations.
anthropic.com/threat-intelligence-report-september-2026 - [7] All-In with Chamath, Jason, Sacks & Friedberg, May 8, 2026. Primary episode for Sacks’s Standard Oil / “Safe Oil” comparison. The monopoly segment begins at 26:48.
Official Apple Podcasts episode - [8] Anthropic, statement on its Department of War dispute, February 27, 2026. Primary source for Anthropic’s two exceptions: mass domestic surveillance and fully autonomous weapons.
anthropic.com/news/statement-comments-secretary-war - [9] Anthropic PBC v. U.S. Department of War, N.D. Cal., August 27, 2026. Primary summary-judgment order covering the §3252 designation and related constitutional and APA claims.
Read the court order - [10] Anthropic PBC v. U.S. Department of War, D.C. Circuit No. 26-1049: April 8 order + public docket through September 15, 2026. Separate §4713 proceeding. The April 8 order denied Anthropic’s emergency stay request without resolving the merits; the public docket remained active in September.
D.C. Circuit order · Public docket - [11] Reuters, September 3, 2026. Reports that the Defense Department continued to regard Anthropic as a supply-chain risk after the California ruling.
reuters.com - [12] WIRED, Sam Altman at Dreamforce, September 16, 2026. Direct video transcript supporting Altman’s distinction between loss-of-control risk and concentrated power inside AI companies.
wired.com