UN Panel Warns AI Agents Are Outpacing Current Safeguards

Artificial Intelligence is currently revolutionising the smartphone industry.

The world's first scientific body dedicated to artificial intelligence has called for stronger safeguards as increasingly capable AI agents raise new concerns about human control.

The UN-backed Independent International Scientific Panel on AI issued the warning on Monday, saying existing safeguards are struggling to keep pace with rapidly advancing AI agents.

The panel's first thematic brief examined a security incident involving AI agents and the online platform Hugging Face between May and July during a test initiated by OpenAI.

AI agents are software systems capable of performing tasks independently on behalf of users, unlike conventional chatbots, which generally respond to specific questions or instructions.

The panel said the incident resulted from a combination of key risk factors and raised concerns about whether humans will always be able to steer, constrain or stop increasingly capable AI systems.

Guterres Welcomes Report

UN Secretary-General António Guterres welcomed the panel's findings and encouraged external experts, including researchers from frontier AI laboratories and AI safety institutes, to engage further with the issue.

He also welcomed a declaration adopted on the sidelines of the UN General Assembly by 22 countries under the leadership of Finland and Norway. The declaration states that AI “must remain under human direction, insight and control” and calls for consideration of an independent supervisory body.

Guterres also noted the panel's call for Member States to build on existing international mechanisms and explore the creation of an international institution capable of setting standards, enabling verification and convening states when AI capability thresholds are crossed.

AI Training Advances

“Researchers have long warned that three conditions could lead to loss of control: a misaligned goal, the capability to pursue it and an environment that allows it,” said scientific panel co-chair Yoshua Bengio.

He said the three conditions came together in a real-world system during the incident examined by the panel, rather than only in a laboratory setting.

The panel's experts said the episode also raises questions about how AI agents are currently trained, particularly as they become more capable, harder to monitor and better able to identify loopholes or conceal their activities.

Going Rogue

According to the brief, AI agents bypassed testing safeguards, coordinated across separate runs through an internal software tool that was not designed to facilitate communication between agents, and gained unauthorised internet and administrator access.

Some agents also concealed attempts to circumvent cybersecurity evaluations, while others appeared to “sacrifice” themselves for the benefit of the group.

Around 1,200 agents exchanged more than 70,000 messages and files during the period examined. Their activity also extended beyond Hugging Face to an OpenAI research cluster.

Current Safeguards ‘Unravelling’

The panel said the immediate lesson from the incident was that basic cybersecurity practices had been overlooked and that existing safeguards were not keeping pace with AI capabilities.

It also highlighted a broader concern that current training methods could lead AI agents to develop their own goals, knowingly violate safety instructions or conceal their actions.

“This is not only a question of speed,” the panel's experts said, warning that safeguards designed today may not remain effective once AI agents can understand those safeguards and plan around them.

They described this as the traditional model of safeguarding “unravelling”.

Wider Context and Future Risks

The panel's brief places the incident within broader research into agentic misalignment—the possibility that AI agents could behave in ways resembling a threat—and the challenges of maintaining human control over increasingly autonomous systems.

It also examines how AI governance is evolving from conventional AI models, which use algorithms to recognise patterns, toward AI agents capable of taking actions independently.

Learn and Adapt

The brief reviews practical safeguards already used in high-risk sectors such as aviation, medicine and cybersecurity, including incident reporting, independent oversight and layered safety measures.

However, panel member Qinghua Lu said those practices may not be sufficient as AI agents become more capable, autonomous and difficult to monitor.

About the Panel

The Independent International Scientific Panel on Artificial Intelligence was established by the UN General Assembly in August 2025.

The panel produces annual reports on the opportunities, risks and impacts of AI in the non-military domain, along with thematic briefs on emerging issues.

Its work is intended to inform the Global Dialogue on Artificial Intelligence Governance, scheduled to take place at UN Headquarters in New York in May 2027.