King warns of “existential dangers” of AI as tech leaders meet in Scotland

The King has urged some of the world’s biggest artificial intelligence companies to ensure humans remain in control of increasingly powerful systems, as questions over AI safety and oversight move rapidly up the agenda for UK industry and defence.

And he warned of the “existential dangers” if AI falls into the wrong hands and being used in “potentially catastrophic ways”.

“I really am enormously grateful for the presence in this room of many of our greatest technological minds, and indeed, for the thoughtful stewardship demonstrated by those gathered here today as you consider the fundamental principles that should – and must – guide the… pic.twitter.com/raxMvdTZvT

— The Royal Family (@RoyalFamily) September 17, 2026

King Charles III brought senior figures from Nvidia, Google DeepMind, OpenAI and Anthropic together at Dumfries House in Scotland on Thursday to discuss the future development of artificial intelligence.

Attendees included Nvidia founder and CEO Jensen Huang, Google DeepMind founder Demis Hassabis and OpenAI CFO Sarah Friar, alongside UK AI Minister Kanishka Narayan.

King calls for human control of AI

The King raised concerns about increasingly autonomous AI agents and the risk of humans surrendering important decisions to machines.

“The decisions taken at this formative time will shape the world inherited by future generations,” the King said in prepared remarks reported by Reuters. “The task before you is not merely to advance technology, but to ensure that it remains firmly in the service of humanity, community and the natural world.”

“Those in our world who value our humanity and its vital moral component are anxiously seeking your reassurance that we will not lose control of our destiny,” he said.

King warns of “existential dangers”

The King also warned of the “existential dangers” posted by AI falling into the wrong hands.

In his opening address, he said, “AI is already showing its immense capacity to improve and save life – for example, in the field of life sciences and medicine.

“Yet, those who have created these technologies are now increasingly warning that AI risks developing darker capacities – perhaps even to take life.

“In this regard, if I may say so, there seems urgency in adequately considering the existential dangers of such technologies falling into the wrong hands, and being used in potentially catastrophic ways.

“Surely, we need sufficient means of control before it is all too late?”

The meeting was not expected to produce binding agreements. Buckingham Palace said discussions would consider whether common principles could guide AI so it serves not simply as a source of greater capability and efficiency but also supports human dignity.

AI safety debate accelerates

The meeting comes during an unusually intense debate within the technology industry over how quickly increasingly capable AI systems should be developed.

Some AI executives and researchers have called for stronger testing, independent scrutiny and greater international coordination. Others argue that existing safeguards and competition can manage the risks without slowing development.

The UK government has already identified “loss of control” as one potential risk from advanced AI. A government discussion paper on frontier AI says humans could gradually hand important decisions to AI systems until it becomes difficult to take back control.

However, the government also stresses that experts still dispute the likelihood of such scenarios.

The practical issue for UK companies is to demonstrate how AI behaves, how it has been tested, where responsibility sits and how humans retain meaningful authority over its operation.

AI control for defence and national security in the UK

Rapid AI adoption is a strategic priority for the Ministry of Defence. Defence Secretary John Healey and MOD representatives said in June 2026 that the UK must “adopt and exploit AI faster than our adversaries,” citing lessons from Ukraine and AI’s growing role in accelerating battlefield decisions.

But faster adoption does not remove the need for human responsibility.

The MOD’s policy on safe and responsible defence AI states that humans must always remain responsible for the ethical use of AI in defence. It also requires “context-appropriate human involvement” in weapons that identify, select and attack targets.

That does not require a person to approve every individual action in real time. Some defensive systems may need to identify and respond to incoming weapons faster than a human operator could. The MOD says human control can also be exercised by defining a system’s operating parameters. 

Developers need to demonstrate where responsibility lies, what decisions an AI system is permitted to make and how human judgement can ultimately be exercised over its outcomes.

Defence contractors must prove AI can be trusted

JSP 936, the MOD’s principal framework governing dependable AI, sets requirements covering governance, development and assurance throughout an AI system’s lifecycle.

The framework guides defence teams to develop AI that is safe, robust and effective while maintaining the appropriate level of human oversight.

That makes assurance increasingly important for UK defence technology companies. A system may perform impressively in demonstrations but still need evidence showing how it behaves outside expected conditions, how it handles failures, who remains accountable and whether operators understand the limitations of its output.

The government reinforced that approach this month with a new AI Risk Management Toolkit, published on 8 September. It is designed to help organisations identify, assess and justify how they manage risks when developing, buying or operating AI.

For defence suppliers, competitive advantage may depend on developing AI capabilities quickly and producing credible evidence that those capabilities can be safely controlled.

Aerospace faces the same assurance challenge

Civil aerospace is confronting similar questions. Aviation certification has traditionally depended on systems behaving predictably and demonstrating that behaviour through testing and evidence.

The UK Civil Aviation Authority says it is developing its approach to AI as adoption accelerates across aviation.

Its 2026/27 programme includes mapping how AI is already being used across the sector, identifying emerging risks and examining how AI fits within existing aviation regulation.

The regulator says its role is to allow innovation while ensuring AI is used “safely, responsibly, and in the interests of consumers.”

That could become increasingly significant as machine learning moves deeper into aircraft systems, autonomous aviation, air traffic management, maintenance, safety analysis and airport operations.

For UK aerospace and defence companies, the wider debate highlighted by the King’s meeting therefore has an immediate engineering dimension.

The aim is not just to build more capable AI. It is to demonstrate that the technology remains dependable, testable and accountable—and that humans still have meaningful control when it matters most.