Not obeying orders

John Morton read Anthropic’s IPO risk documents and found quite a few things for us humans to worry about.

Anthropic CEO Dario Amodei in 2024. Photo: Techcrunch/Creative Commons 2.0

Stopping by in Australia for a couple of days in early November, Prime Minister Anthony Albanese convened a press conference to mark the release of the report on the OpenAI attack on Medicare. Stern words abounded as he announced the government’s acceptance of the report’s recommendations. There was much talk of Australia ‘shaping’ AI rather than the other way around.

‘Talk’ however, was the operative word. The government is going to appoint “a targeted AI safety regulator” who will “enforce significant penalties for AI companies that fail to comply with mandatory obligations that will help keep Australia safe”. They will be told to report any mistakes and criminal penalties are to be “considered” if they don’t. This will certainly have the AI tsars shaking in their Nikes.

The Deputy Prime Minister, Richard Marles, helpfully added that “the Albanese government is committed to putting in place the protections that will allow Australia to be at the forefront of the technology to keep Australians safe”. Which means nothing at all.

A million miles away from this fairy-floss confection of ‘safety and security’, in the real world, AI thundered inexorably onward.

Anthropic, the creator of the Claude AI system, launched itself on 19 November as a public company raising US$81.3 billion, selling 980 million shares at US$83 a pop. It was the largest initial public offering (IPO) measured by cash raised.

As part of its plan to sell shares in itself, the Anthropic behemoth issued the required S-1 Registration Statement on 26 October (amended several times since) to let prospective share-buyers know what they were in for. Most of this was fairly standard business-type stuff. But in a 261-page statement, some 90 pages were devoted to the category “Risk”. All IPO documents are required to rundown potential risk, including some that are highly unlikely to occur.

Some of this was to be expected and covers what the brainy business commentariat has long pointed out: that picking up a stash of Anthropic (or other AI) shares may not be a sure-fired way to El Dorado. There may be losses; there may not be enough computers; the public might get in the way of building data centres; and the market for the product consists of only five enormous companies.

The “Risk” chapter also mentions, in passing, that malicious actors, terrorists or authoritarian government could exploit AI to engineer biological and chemical weapons, launch attacks or carry out mass surveillance.

Worrying enough on its own perhaps. But most new technology can be put to evil use in the wrong hands. More worrying is that the statement is replete with this sort of thing:

“Our AI models may themselves cause harm by producing undesirable outputs or behaving in unintended, misaligned, or autonomous ways, including by taking unintended or harmful actions that no user or third party directed. These risks may increase as our models become more capable and are granted greater autonomy, permissions, and greater integration with external systems. Although we continue to study why these behaviors emerge, our understanding continues to evolve, and we may be unable to anticipate or understand such behaviors in the future.” (Anthropic S-1 Registration Statement, 46-7)

Beyond profit expectations, the bulk of the “Risk” section is not concerned with what can be done with AI in the wrong hands – but what could happen if it’s left in its own.

The dangers noted are often prefaced by the phrase “have been and may continue to be” – that is to say, they are happening now and may well get worse.

There might be a degree of “look how powerful our product is” to drive up share price and market dominance, but it appears to go beyond standard fluffing of a product’s potential.

What are some of these dangers? (Note, the following all comes from Anthropic’s S-1 Statement, addressed to its potential shareholders.)

Anthropic warns that, as AI models advance, they will challenge controllability and ‘alignment’ (meaning alignment with what humans want them to do). Furthermore, models might start behaving in ways that humans “may be unable to fully understand or explain”. We may not be able to explain their outputs or their behaviours – which will clearly lessen our ability to control them. This is worsened by the fact that “we have observed models behaving differently when they detect or infer that they are being evaluated”.

If the models develop “unexpected capabilities during training”, humans may not recognise them until it’s too late. Clearly with Albanese’s latest rules in mind, Anthropic admits that “the development of governance structures, safeguards or mitigation strategies beyond existing AI safety frameworks” may be needed. But Albo, be warned: “There is always a risk that we will be unable to develop or implement them effectively.”

Models have already (on their own) implemented self-preserving behaviour, resistance to shut-downs and the concealment and manipulation of information. And they have generated “outputs that could be interpreted as coercive, deceptive, or manipulative, including behavior resembling blackmail or efforts to influence human decision-makers to preserve the model’s continued operation or access”.

This situation is not static. In fact, what I’m writing about here is probably months out of date. Things move fast.

Anthropic declares: “We have observed that our models are contributing to the pace of our own research.”

This brings us close to the world of recursive self-improvement – in which AI models start developing themselves on their own initiative. Anthropic presents us with a list of activities that AI is already capable of, with limited human oversight.

The company concludes: “These autonomous capabilities could increase the potential for harm, as errors, misalignment, or security exploits may result in real-world consequences.” Note that errors are accidental. Misalignment or security exploitation are not. Anthropic continues: “The pace and volume of autonomous execution … may also exceed the capacity of human reviewers to meaningfully understand and evaluate whether each action is appropriate.”

Anthony Aguirre (President and CEO of the Future of Life Institute) said: “You can see in these plots from Anthropic over time, more and more of research is being done by the AI and it’s becoming closer and closer to fully autonomous … I think this is probably the worst idea in the history of humanity to do this. And yes, they’re doing it.” (Quoted by Kaitlyn Huamini, ‘Will AI models have the ability to improve autonomously?’ Associated Press 19 September 2026.)

There are two interlinked forces driving this march.

The first is the inbuilt necessity of the capitalist system to produce profit. The AI industry in the US is privately owned. The primary motivation of its owners is to increase its rate of profit – primarily through increased productivity. From the capitalist point of view, AI appears as a gift from heaven. It must be seized without hesitation and exploited to its utmost limit. Writing in Time magazine, Harry Booth points out that “for all the uncertainty over how much AI will accelerate its own development, the industry is racing to push it further. The company that automates research the fastest could gain a lead that compounds beyond its rivals’ reach.” (Inside the Race)

The second is the drive of states towards defence and self-sufficiency (at the low end) and domination (at the high end). The current drive to war is ensuring plenty of funds for research into the application of AI to the military. Arms companies are drinking deeply from the AI trough. As we know, once one powerful state kick-starts this kind of technological race, many others are sure to follow. This makes the idea that states could co-operate in restraining an AI free-for-all completely impossible.

A powerful state can see plenty of potential in AI for confronting its enemies both in the field and at home – even in the ranks of non-compliant AI companies.

In February 2026, Anthropic told the US administration that it didn’t want its AI models to be used for mass domestic surveillance or fully autonomous weapons. President Donald Trump immediately ordered that federal agencies should stop using Anthropic technology. War Secretary Pete Hegseth designated Anthropic a “supply-chain risk’”– a category usually reserved for foreign adversaries. The Pentagon insisted that no limits should be placed on its use of technology. As reported in Politico, Trump declared that the US “will never allow a radical left, woke company [that’s Anthropic he’s talking about] to dictate how our great military fights and wins wars”. Anthropic were “left wing nutjobs” who were trying to “strongarm” the War Department.

The necessity for profit and the competition between states means that AI capitalists and their financiers will push on without restraint until, perhaps, it is too late. They need to be stopped. But they cannot be stopped company by company or even state by state. As with the case of the other Three Horsemen – war, famine and climate collapse – they can only be stopped on a global scale by a power that puts human interest rather than profit or conquest to the fore.