Our goal
\nTo usher in a new era for AI safety, allowing us to unlock the full economic and social benefits of advanced AI systems while minimising risks.
\n\n
Why this programme
\nAs AI becomes more capable, it has the potential to power scientific breakthroughs, enhance global prosperity, and safeguard us from disasters. But only if it’s deployed wisely. Current techniques working to mitigate the risk of advanced AI systems have serious limitations, and can’t be relied upon empirically to ensure safety. To date, very little R&D effort has gone into approaches that provide quantitative safety guarantees for AI systems, because they’re considered impossible or impractical.
\n\n
What we’re shooting for
\nBy combining scientific world models and mathematical proofs we will aim to construct a ‘gatekeeper’, an AI system tasked with understanding and reducing the risks of other AI agents. In doing so we’ll develop quantitative safety guarantees for AI in the way we have come to expect for nuclear power and passenger aviation.
\n\n\n
Read the accessible version of the programme thesis
","mediumLayout":false,"fullWidth":false},{"type":"spacer-comp","cssSizeClass":"small","height":0},{"type":"quote","quote":"\"When stakes are high, promising test results aren’t enough; we need quantitative safety guarantees. We’re building a workflow to harness general-purpose AI to construct guaranteeable domain-specific AI systems for high-risk contexts.\"
","author":"David 'davidad' Dalrymple","role":"Programme Director","image":{"src":"/media/4xuaytke/davidad-pd-1.jpg?width=600&height=750&format=webp&v=1db3acb9411d3d0","alt":"Davidad PD 1"}},{"cards":[{"title":"Scaffolding","text":"We can build an extendable, interoperable language and platform to maintain formal world models and specifications, and check proof certificates.","step":"TA1"},{"title":"Machine Learning","text":"We can use frontier AI to help domain experts build best-in-class mathematical models of real-world complex dynamics + train verifiable autonomous systems.","step":"TA2"},{"title":"Real-World Applications","text":"A safeguarded autonomous AI system with quantitative safety guarantees can unlock significant economic value when deployed in a critical cyber-physical operating context.","step":"TA3"}],"type":"numbered-cards","heading":"Technical areas","subHeading":"This programme is split into three technical areas (TAs), each with its own distinct objectives."},{"type":"standout-copy","title":"Live funding opportunities","text":"We're currently accepting applications for funding in this programme.
\nSubmit your 'Late' TA2 Phase 1 application until 17 August.
","link":{"isActive":false,"text":"Learn more ","href":"/opportunity-spaces/mathematics-for-safe-ai/safeguarded-ai/funding","title":"about the funding call"},"color":"99e0d9","isWhiteText":false,"fullWidth":false},{"type":"spacer-comp","cssSizeClass":"small","height":0},{"type":"team-cards","items":[{"image":{"src":"/media/4xuaytke/davidad-pd-1.jpg?width=1000&height=1000&format=webp&v=1db3acb9411d3d0","alt":"Davidad PD 1"},"name":"David 'davidad' Dalrymple","text":"davidad is a software engineer with a multidisciplinary scientific background. He’s spent five years formulating a vision for how mathematical approaches could guarantee reliable and trustworthy AI. Before joining ARIA, davidad co-invented the top-40 cryptocurrency Filecoin and worked as a Senior Software Engineer at Twitter.","position":"Programme Director"},{"image":{"src":"/media/dzcho2ye/yasir-bakki.jpg?rxy=0.4821575042765275,0.40508862206975416&width=1000&height=1000&format=webp&v=1db4d76e69b2ad0","alt":"A photo of Yasir Bakki smiling against a grey background."},"name":"Yasir Bakki","text":"Yasir is an experienced programme manager whose background spans the aviation, tech, emergency services, and defence sectors. Before joining ARIA, he led transformation efforts at Babcock for the London Fire Brigade’s fleet and a global implementation programme at a tech start-up. He supports ARIA as an Operating Partner from Pace. ","position":"Programme Specialist"},{"image":{"src":"/media/zqzhenhm/nora-ammann.jpg?rxy=0.4681235005520459,0.4680113289268752&width=1000&height=1000&format=webp&v=1db4d76de7857b0","alt":"A photo of Nora Ammann smiling at the camera against a white background."},"name":"Nora Ammann ","text":"Nora is an interdisciplinary researcher with expertise in complex systems, philosophy of science, political theory and AI. She focuses on the development of transformative AI and understanding intelligent behavior in natural, social, or artificial systems. Before ARIA, she co-founded and led PIBBSS, a research initiative exploring interdisciplinary approaches to AI risk, governance and safety.","position":"Technical Specialist"}],"title":"Meet the programme team","text":"Our Programme Directors are supported by a Programme Specialist (P-Spec) and Technical Specialist (T-Spec); this is the nucleus of each programme team. P-Specs co-ordinate and oversee the project management of their respective programmes, whilst T-Specs provide highly specialised and targeted technical expertise to support programmatic rigour.
"},{"type":"spacer-comp","cssSizeClass":"medium","height":0},{"type":"news-insights","title":"Featured insights","sliderTitle":"","sliderLink":null,"featured":{"title":"Can AI be used to control safety critical systems? ","text":"Fortune
\nThe U.K.'s Advanced Research and Invention Agency (ARIA) is funding a project to use frontier AI models to design and test new control algorithms for safety critical systems.
","image":{"src":{"mobile":"/media/svilqygq/aicontrolsafetycriticalsystems_fortune.webp?width=300&height=375&format=webp&v=1dbdad9ebded7d0","tablet":"/media/svilqygq/aicontrolsafetycriticalsystems_fortune.webp?width=440&height=550&format=webp&v=1dbdad9ebded7d0","desktop":"/media/svilqygq/aicontrolsafetycriticalsystems_fortune.webp?width=600&height=750&format=webp&v=1dbdad9ebded7d0"},"alt":"Nuclear power plant with cooling towers and transmission lines","title":""},"link":{"target":"_blank","isActive":true,"text":"Read more ","href":"https://fortune.com/2025/06/10/aria-safeguarded-ai-safety-critical-systems-davidad-eye-on-ai/","title":"on fortune.com"}},"items":[{"title":"Towards Provable Safe AI with Davidad","tag":"Insights","date":"05 September 2024","outlet":"Gradient Podcast","image":{"src":"/media/h4wb400w/the-gradient-towards-provable-safe-ai.webp?rxy=0.21264866933424656,0.4556371535966328&width=400&height=400&format=webp&v=1db9d6306658040","alt":"A graphic image of davidad, on the Gradient podcast. The writing says 'Davidad Dalrymple: Towards provably safe AI'."},"link":{"target":"_blank","isActive":true,"text":"Listen now ","href":"https://thegradientpub.substack.com/p/davidad-dalrymple-towards-provably","title":"on The Gradient Substack"}},{"title":"Yoshua Bengio joins Safeguarded AI as Scientific Director","tag":"News","date":"07 August 2024","outlet":"MIT Tech Review","image":{"src":"/media/csmlpui4/aria_june24-10.jpg?rxy=0.49053415203117023,0.38350769869251755&width=400&height=400&format=webp&v=1db9d61d25a1870","alt":"A photo of Yoshua Bengio and David 'davidad' Dalyrmple smiling for the camera."},"link":{"target":"_blank","isActive":true,"text":"Read more ","href":"https://www.technologyreview.com/2024/08/07/1095879/ai-godfather-yoshua-bengio-joins-uk-project-to-prevent-ai-catastrophes/","title":"on MIT technologyreview.com"}},{"title":"'Towards Guaranteed Safe AI: A framework for ensuring robust and reliable AI systems' ","tag":"Insights","date":"10 May 2024","outlet":"arXiv","image":{"src":"/media/krsbyppo/davidad-workshop.jpeg?width=400&height=400&format=webp&v=1db4d770da20310","alt":"A photo of a group of people sitting around the table and looking at davidad writing on a whiteboard. "},"link":{"target":"_blank","isActive":true,"text":"Read more ","href":"https://arxiv.org/abs/2405.06624","title":"on arxiv.org"}}]},{"type":"spacer-comp","cssSizeClass":"small","height":0},{"type":"standout-copy","title":"Sign up for updates","text":"Stay up-to-date on our programmes and opportunity spaces.
","link":{"isActive":false,"text":"Sign up ","href":"/sign-up-for-updates","title":"for updates on our programmes and opportunity spaces"},"color":"99e0d9","isWhiteText":false,"fullWidth":false},{"type":"double-card","items":[{"title":"Programme Directors","text":"Our Programme Directors are exceptional scientists and engineers with the curiosity to explore uncharted territory","link":{"isActive":false,"text":"Find out more ","href":"/about-aria/our-team/programme-directors","title":"about ARIA's Programme Directors"},"colour":null},{"title":"How we work","text":"We seek out exceptional scientists and engineers and empower them with the resources and autonomy to turn their ideas into reality","link":{"isActive":false,"text":"Learn more ","href":"/how-we-work","title":"about how ARIA work"},"colour":null}],"fullWidth":false},{"type":"spacer-comp","cssSizeClass":"medium","height":0}],"scriptsAtTop":" \n\n\n\n\n\n\n\n\n\n\n","scriptsAtBottom":"","id":2291,"languageCode":"en","pageName":"Safeguarded AI","scrollSmoothness":0.0,"gtmCode":"GTM-T8Z5F55","topNav":{"logo":{"text":"ARIA - Home","href":"/media/hsxblxrf/logo-aria-black.svg"},"nodes":[{"text":"About us","href":null,"target":null,"id":null,"nodes":[{"text":"About us","href":"/about-aria","target":null,"id":"2207","nodes":[],"noOpener":false,"noReferrer":false},{"text":"Our team","href":"/about-aria/our-team","target":null,"id":"2159","nodes":[],"noOpener":false,"noReferrer":false},{"text":"Programme Directors","href":"/about-aria/our-team/programme-directors","target":null,"id":"2831","nodes":[],"noOpener":false,"noReferrer":false},{"text":"Reporting and policies","href":"/about-aria/reporting-and-policies","target":null,"id":"2160","nodes":[],"noOpener":false,"noReferrer":false}],"noOpener":true,"noReferrer":false},{"text":"What we do","href":null,"target":null,"id":null,"nodes":[{"text":"How we work","href":"/how-we-work","target":null,"id":"2817","nodes":[],"noOpener":false,"noReferrer":false},{"text":"Live opportunity spaces","href":"/opportunity-spaces","target":null,"id":"2215","nodes":[],"noOpener":false,"noReferrer":false},{"text":"Emerging areas","href":"/emerging-areas","target":null,"id":"2963","nodes":[],"noOpener":false,"noReferrer":false},{"text":"Activation Partners","href":"/activation-partners","target":null,"id":"2174","nodes":[],"noOpener":false,"noReferrer":false}],"noOpener":true,"noReferrer":false},{"text":"Funding","href":null,"target":null,"id":null,"nodes":[{"text":"Funding opportunities","href":"/funding-opportunities","target":null,"id":"2164","nodes":[],"noOpener":false,"noReferrer":false},{"text":"Applicant guidance","href":"/funding-opportunities/applicant-guidance","target":null,"id":"2162","nodes":[],"noOpener":false,"noReferrer":false},{"text":"Being funded by ARIA","href":"/funding-opportunities/the-creator-experience","target":null,"id":"2433","nodes":[],"noOpener":false,"noReferrer":false},{"text":"FAQs","href":"/funding-opportunities/faqs","target":null,"id":"2649","nodes":[],"noOpener":false,"noReferrer":false}],"noOpener":true,"noReferrer":false},{"text":"Careers","href":"https://aria-jobs.teamtailor.com/","target":"_blank","id":null,"nodes":[],"noOpener":false,"noReferrer":false}],"headerLinks":[{"target":"","isActive":false,"text":"Become a Programme Director","href":"/about-aria/our-team/programme-directors/cohort-3","title":""},{"isActive":false,"text":"Opportunity spaces","href":"/opportunity-spaces","title":""}],"menuLinks":[{"isActive":false,"text":"Contact us","href":"/contact","title":""},{"isActive":false,"text":"Terms of use","href":"/terms-and-conditions","title":""}],"activeItem":"2291","menuText":"Menu","menuTextClose":"Close menu","search":{"isActive":false,"text":"Search","href":"/search","title":""},"animationOffText":"Motion: Off","animationOnText":"Motion: On"},"footer":{"header":"Help us drive momentum ","text":"ARIA is a non-departmental public body, sponsored by the Department for Science, Innovation + Technology.","copyright":"Copyright 2026","headerLinks":[{"isActive":false,"text":"Contact","href":"/contact","title":""},{"target":"_blank","isActive":true,"text":"Careers","href":"https://aria-jobs.teamtailor.com/","title":""}],"socialHeader":"Follow our progress","footerLinks":[{"isActive":false,"text":"Privacy Policy","href":"/privacy-policy","title":""},{"isActive":false,"text":"Terms and Conditions","href":"/terms-and-conditions","title":""},{"isActive":false,"text":"Cookie Policy","href":"/cookie-policy","title":""},{"isActive":false,"text":"Accessibility Statement","href":"/accessibility-statement","title":""},{"isActive":false,"text":"Accessibility Support","href":"/funding-opportunities/accessibility-support","title":""},{"target":"_blank","isActive":true,"text":"Careers","href":"https://aria-jobs.teamtailor.com/","title":""},{"isActive":false,"text":"Supply to us","href":"/supply-to-us","title":""}],"socialItems":[{"link":{"target":"_blank","isActive":true,"text":"ARIA Research Twitter","href":"https://x.com/aria_research","title":""},"type":"x"},{"link":{"target":"_blank","isActive":true,"text":"LinkedIn","href":"https://www.linkedin.com/company/advanced-research-and-invention-agency","title":""},"type":"Linkedin"},{"link":{"target":"_blank","isActive":true,"text":"ARIA Research Substack","href":"https://ariaresearch.substack.com/","title":""},"type":"SubStack"}],"formText":"","formSubmit":"Sign up","formId":"","recaptchaSiteKey":"6LcuyY4bAAAAAE2OzRFfA7vDWyohfZXyl2uTuHLa","verificationToken":"","redirectUrl":null,"submitMessage":"","formTheme":""},"seo":{"title":"Safeguarded AI","description":"Backed by £59m, this programme aims to develop the safety standards we need for transformational AI.","keywords":"safeguarded ai, ai safety, aria programme research funding, aria opportunity space, research programmes, transformational ai","includeInSitemapXml":true,"canonicalTag":""},"sharing":{"og":[{"property":"og:title","content":"Safeguarded AI"},{"property":"og:description","content":"Backed by £59m, this programme aims to develop the safety standards we need for transformational AI."},{"property":"og:image","content":"https://aria.org.uk/media/ipab0gpu/safeguarded-ai-programme.jpg?width=600&format=webp&quality=100&v=1db669bb49aced0"},{"property":"og:url","content":"https://aria.org.uk/opportunity-spaces/mathematics-for-safe-ai/safeguarded-ai"}],"twitter":[{"name":"twitter:card","content":"summary_large_image"},{"name":"twitter:site","content":"@ARIA_research"},{"name":"twitter:title","content":"Safeguarded AI"},{"name":"twitter:description","content":"Backed by £59m, this programme aims to develop the safety standards we need for transformational AI."},{"name":"twitter:image","content":"https://aria.org.uk/media/ipab0gpu/safeguarded-ai-programme.jpg?width=600&format=webp&quality=100&v=1db669bb49aced0"},{"name":"twitter:url","content":"https://aria.org.uk/opportunity-spaces/mathematics-for-safe-ai/safeguarded-ai"}]},"pageType":"opArticlePage"}