Research institute · Research orgs · Oxford, UK · Est. 2024
Forethought Research (operating as “Forethought”) is a UK-based, non-profit research organization focused on the transition to a world with superintelligent AI systems. Its stated mission is to research how to navigate this transition, with attention to both risks (including scenarios like rapid technological development, AI-enabled autocracies, and conflicts) and the pursuit of “really good futures.”
Forethought describes an approach built around a small, focused team that works on what it considers some of the most important questions outside the current “Overton window,” and that shares findings with AI research communities, policy actors, and the general public. The organization’s research portfolio emphasizes macrostrategy and technological forecasting—e.g., modeling or reasoning about the dynamics of “intelligence explosion” style progress, the sequencing of transformative AI impacts, and practical implications for governance. It also works on concrete technical/policy levers that could matter during a fast transition, such as model-spec design considerations, “AI character” (behavioral traits like honesty and cooperation), and work on credible communication between humans and advanced AI systems. Forethought also highlights related work themes such as AI-enabled changes to epistemics (how societies form beliefs) and preparedness for governance challenges that arise under rapid AI acceleration. Organizationally, Forethought is registered as a non-profit company in England and Wales (company number 16156175), incorporated on December 30, 2024, with leadership including Max Dalton (Director) and multiple Senior Research Fellows and Research Fellows listed on its website.
No curated programs yet. Check the careers page ↗.
One of the most important questions facing the world is whether AI will undergo a [fast intelligence explosion](https://www.forethought.org/research/will-ai-r-and-d-automation-cause-a-software-intelligence-explosion) and quickly automate most real-world work. A common argument ag
A nightwatchman on every probe: Superintelligent surveillance to prevent galactic anarchyOne vision of space settlement is "galactic anarchy," where probes sent to distant stars are each free to govern the societies they found however they like. We argue this vision is dangerous: across millions of light-years, no centralized authority can punish violations after the
The Dynamics of Intelligence ExplosionsAI is increasingly being used to help with AI&D. Under certain conditions this feedback loop might be able to produce an intelligence explosion, with rapidly escalating AI capabilities. I explore the mathematics of the most explosive possibilities, with an eye to understanding wh
Policy ideas to ensure responsible government deployment of AIGovernments will deploy frontier AI in areas where their powers are already least checked: military, intelligence, surveillance, and policing. Companies can't police those deployments, and legislatures largely can't see them. This could undermine democratic checks and balances, a
Risk-Averse AIsWe make the case for training AIs to be risk-averse in resources — specifically, to treat resources as having diminishing marginal utility. We argue that risk aversion can preserve AIs’ usefulness in the event that they turn out aligned, and that it provides an extra line of defe
What should go in a model spec?AI companies face a tangle of competing considerations when deciding what goes into a model spec. Should the AI be completely honest in every circumstance? Take proactive prosocial actions? Engage in whistleblowing? And so on. We lay out a checklist of plausible criteria for good
Will We Really Put Data Centers in Space?Google, SpaceX, Blue Origin, and others are betting that the next wave of AI data centers will be built in orbit. If they're right, it could reshape the economics of compute and create new headaches for AI governance. We dig into the technical and economic feasibility of orbital
Stickiness in AI Behavioral DesignToday's model specs are written for current and near-future versions of LLMs, and AI labs typically treat them as provisional. But what if the AI behaviors we set now stick around and end up governing far more capable future models by default? This piece traces four sources of "i
A draft honesty policy for credible communication with AI systemsIf humans and advanced AI systems are going to cooperate—to make honest deals and avoid negative-sum conflict—AIs will need reasons to trust us. By default, they won't have many: humans routinely lie to AIs in evaluations, and developers control much of what models see and believ
The Saturation ViewWill MacAskill presents a new theory of population ethics, the Saturation View. It is motivated mainly by the observation that virtually all existing population axiologies prefer a *homogeneous* universe, full of a vast amount of whatever sort of thing that axiology rates as best
Desired behaviour for AI advising humans on important decisionsAs AI gets smarter, people will increasingly turn to it for advice on important decisions — and those who do may gain outsized influence. So the quality of that advice really matters. Tom Davidson discusses key scenarios where people might seek AI guidance, drafts a model spec fo
AI impacts on epistemics: the good, the bad and the uglyFor better or worse, AI could reshape the way that people work out what to believe and what to do, with potentially enormous consequences for the decisions they go on to make. We explore how AI could enhance society's truth-tracking, or degrade it, whether unintentionally or deli
A new Forethought newsletter article extending its space-governance-related research theme, using a “nightwatchman”/minimal governance framing to discuss enforcement and governance challenges in a space-expansion scenario.
Notes on Implications of Scale-Dependent AlgorithmsA Forethought newsletter post analyzing how scale-dependent algorithmic improvements could affect inferences about future AI progress and arguments for or against intelligence-explosion-style dynamics.
Policy ideas to ensure responsible government deployment of AIA Forethought newsletter post outlining concerns about government deployment of frontier AI (especially in areas with existing exclusive governmental powers) and a set of policy ideas aimed at strengthening checks and balances.
Speed-up calculator: How much will automating AI R&D speed up AI software progress, absent a software intelligence explosion?Forethought describes a “speed-up calculator” for estimating how much faster AI software progress might become when AI R&D is automated, including cases even without a software-intelligence-explosion dynamic.
What should go in a model spec?A Forethought newsletter post presenting a framework for reasoning about what qualities/rules to include in an AI model spec and how criteria for model-spec design might clash.
A draft honesty policy for credible communication with AI systemsA Forethought newsletter post sharing a “draft honesty policy” concept intended to make human-to-AI communication more credible, including the idea of creating a paper trail that future models could use as evidence of genuine commitments.