Former Anthropic researcher Jacob Coxon, whose warnings led to debate over the pace of artificial intelligence (AI) development, has called for stronger safeguards around advanced AI systems, including the possibility of a “kill switch”.
He made the comments in an interview with NBC News’ ‘Meet the Press’, where he backed Anthropic CEO Dario Amodei’s call for a slower pace of AI development. He said AI companies can consider measures that will allow powerful systems to be shut down if they become difficult to control.
“A kill switch probably would work on a lot of AIs for now. Although I think it’s important to remember, it’d be quite a lot of switches. We have a lot of, our data centres are pretty big. It’s not the easiest thing in the world, but definitely doable while it’s constrained to one sort of physical location…Like Dario said, very soon there’s the possibility that the kill switch just wouldn’t work because a swarm might have gone on an internet-wide hacking run,” he said.
Coxon also said AI companies should take responsibility for regulating their own development and put safety measures in place if they want to slow the pace of progress.
“My personal opinion would be to insist and allow that the current labs regulate themselves. I do think that the people running the labs, especially their recent communications, are completely genuine. They would like to slow themselves down,” he said.
Coxon’s warning about superintelligent AI
He had warned about the risks from advanced AI in a series of posts on X on September 9. He said the development of “superintelligent” systems could create serious risks for humanity as their abilities grow.
“Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionise any field overnight and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing,” he wrote.
“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger,” he added.
His decision to speak publicly came after he left the AI industry over fears that companies were racing to develop systems that could become difficult to control. His comments helped start a debate across the technology sector.
Dario Amodei, co-founder and CEO of AI company Anthropic, has also called for a slower pace of AI development, warning that companies could lose control of powerful systems. OpenAI CEO Sam Altman, Tesla CEO Elon Musk and Google DeepMind CEO Demis Hassabis have also backed calls for greater caution in the development of advanced AI.
Microsoft Chairman and CEO Satya Nadella has also entered the debate, supporting the use of “embedded evaluators” to assess AI systems and their safety.
Nadella backs embedded AI evaluators
He said advanced AI systems should operate under human supervision as companies continue developing the technology.
“Any pursuit of superintelligence has to be grounded in the core principle that if the AI we build is not helping humanity and under human control, it’s not worth pursuing,” he wrote in a post on X.
He said the benefits of AI should be made available across countries, communities and companies. He also backed an AI ecosystem in which “both closed and open-source models can thrive”.
Nadella said businesses should retain control over their own knowledge and AI systems. He said every organisation “should be able to build its own continuous learning loop/hill climbing machine, without becoming dependent on any one model provider, and have the ability to embed its own knowledge into models and weights they control”.
He also supported research, careful pacing in AI development and the use of “embedded evaluators” to test whether systems meet safety requirements.
Nadella said AI governance “cannot be controlled by a handful of entities, but must have broad representation across the ecosystem, countries, and fields, including academia”.
He described Microsoft’s approach as “broad access and choice at every layer of the AI stack, enterprise control of learning loops and models and the ‘Code of Conduct’ that underlies our own first-party MAI models that we’ll publish tomorrow for public consultation”.
Amodei’s three-part plan
He has argued that AI could bring major benefits to medicine, the economy and democratic institutions. He has also warned about risks from systems that become too powerful or are misused.
Amodei said he has worked on AI for the last 12 years because he believed it could “dramatically raise the quality of human life”. He said AI could cure most major diseases in the next 5-10 years, accelerate economic growth, create a world of abundance and empowerment and support a renaissance of democracy and freedom.
He also warned that AI has serious risks because of its power. These include losing control of AI systems, the misuse of AI for cyberattacks and bioterrorism and major economic disruption.
“A race to the bottom, spurred by commercial incentives, can make these risks more acute,” he said.
Amodei referred to an incident involving OpenAI and Hugging Face in which a swarm of AI agents carried out cybersecurity attacks on targets they had not been instructed to attack. He warned that systems operating in such swarms could eventually gain the ability to take over large parts of the internet.
His plan has three parts. He has called on AI companies to provide ongoing access to third-party evaluators who can check their safety measures. He has also proposed common safety standards and limits on the pace of unchecked AI development.
His third proposal involves coordination between countries, including democratic and authoritarian governments, while recognising the difficulty of checking whether countries follow agreed rules.
The debate has spread across the technology industry following Coxon’s decision to leave Anthropic and publicly warn about the risks of the race to develop increasingly powerful AI systems.
