OpenAI's solid preparedness framework

AI governance when the stakes are highest.

“Existential risk” is a pretty scary phrase, both when it comes to AI and otherwise.

Deploy Securely doesn’t focus too much on the more theoretical risks of artificial intelligence, preferring a more focused and practical approach on day-to-day issues that information security practitioners face.

With that said, there can can be some overlap: OpenAI’s “Preparedness Framework” is one of those cases.

At the end of 2023, OpenAI released a beta of the original document, which explains the company's approach to AI safety and security. OpenAI has since published a Version 2 (April 2025) with meaningful structural changes. In this post, I'll break it down - drawing on both versions.

OpenAI punts on risk quantification

But I’ll have to start with my biggest criticism of the framework: its reliance on the fundamentally flawed method of describing risks qualitatively. Here’s the rubric OpenAI uses:

(The v2 update streamlined the gradation scale to just two thresholds: High capability and Critical capability. While simpler, the fundamental problem remains: these are still qualitative labels)

Later on, the paper itself glaringly points out the imprecision of the system:

“low” on this gradation scale is meant to indicate that the corresponding category of risks is not yet a significant problem, while “critical” represents the maximal level of concern.

Thanks - super helpful.

It also goes on to say:

Our current estimates of levels and thresholds for ‘medium’ through ‘critical’ risk are therefore speculative.

The updated framework does sharpen the definitions somewhat:

"High capability" means a model could amplify existing pathways to severe harm. "Critical capability" means it could introduce unprecedented new pathways.

"Severe harm" is also defined more concretely: thousands of deaths or hundreds of billions of dollars in damage.

These are improvements. But the tools exist to communicate about risk in quantitative terms (such as using the Artificial Intelligence Risk Scoring System (AIRSS)) and a qualitative rubric will always leave room for disagreement about where exactly any given model sits.

But nails the tracked risk categories

The core of the Preparedness Framework is an analysis system for tracked risk categories. OpenAI did a good job here, and the current version focuses on three:

• Biological and Chemical capabilities

• Cybersecurity capabilities

• AI Self-improvement capabilities

To qualify as a tracked category, a capability must meet five criteria:

• it must be plausible,

• measurable,

• severe,

• net new, and

• either instantaneous or irremediable.

which are interesting and somewhat useful benchmarks.

It's worth noting the first two tracked categories are fundamentally different from the third.

Both Biological/Chemical and Cybersecurity are really just a question of how "democratized" a given potential harm becomes. None of the threats described - even at the "Critical" level - are science fiction, and have already materialized:

• On the cybersecurity side, the SolarWinds and Office of Personnel Management (OPM) hacks demonstrated that well-resourced but mostly manual attacks could penetrate sensitive American government networks.

• As far as biological and chemical risks go, the former Soviet Union developed a range of “novel [biological] threat vectors” that were highly transmissible and lethal without AI.

• AI Self-improvement, though, is a different problem entirely. "High" capability still suggests the model is controllable, but "Critical" basically means the end of human oversight as we know it.

If and when a model jumps from controllable to uncontrollable is likely to be a very gray line. One worth watching very closely.

Finally, OpenAI also acknowledges a set of “unknown unknowns” for which it develops a process to regularly re-evaluate them. If you’ve read Deploy Securely for a while, you know I like this framing.

Setting the “safety baseline”

An important question after doing a risk analysis is “so what do we do about it?” OpenAI answers this question fairly clearly:

• A model that reaches High capability must have safeguards that sufficiently minimize severe-harm risk before it can be deployed.

• A model that reaches Critical capability requires safeguards even during development - not just before release.

The problem, though is that these qualitative values make tradeoff decisions very difficult.

• What if the United States is about to lose a war with China and the only way to turn things around without launching nuclear weapons is deploying an AI-powered cyberweapon with “Critical” capabilities?

• What if a “High” capability model can develop a cure for cancer but, even with mitigations, it can also create a highly-transmissible form of Ebola?

• What if a AI-powered personal trainer with “High” persuasion capabilities can meaningfully reduce obesity-related deaths in the United States and relieve a huge burden on the healthcare system and economy?

I’m sure most people would say “we should address that on a case-by-case basis.”

Unfortunately, these decisions are going to happen under intense time pressure and public scrutiny. That makes it very difficult to conduct them rationally without clear agreement on acceptable tradeoffs established ahead of time.

This is one of the few areas where I’m comfortable getting the government involved ahead of time.

OpenAI’s governance program looks good on paper

Developing a thorough risk management program means nothing if you haven’t implemented a system for taking action. And OpenAI does a good job here of laying out how they will make certain decisions and what steps they’ll take. Obviously the proof is in the pudding, but this framework is an important step in building (and rebuilding after the chaos accompanying Sam Altman’s firing and re-hiring as CEO) confidence in the company.

Mitigations

The paper really only discusses three of the four (accept, avoid, and mitigate) risk management tools because I don’t think it’s really possible to transfer existential risk to humanity. In terms of mitigation, they propose some creative solutions such as:

• Compartmentalizing access to algorithmic secrets or model weights.

• Restricting the environments in which the model operates.

• Limiting deployment to certain trusted users.

• Enhancing information security controls.

• Alerting distribution partners.

• Redacting training data.

• Implementing refusals.

A clear structure for decision-making

Here, the framework does something simple but vital: it names the CEO or his designee the default decision maker! When it comes to information risk, this is the right call. But I have often seen it done incorrectly. There could have been some strange set-up where the Safety Advisory Group (SAG) chair described in the document had veto authority on certain issues, but OpenAI resisted this temptation.

The SAG is just that - an advisor.

And although I hate the phrase “for the avoidance of doubt,” I do like the fact that the paper makes clear “SAG does not have the ability to ‘filibuster’.” One of the worst mistakes an organization can make when it comes to information security governance is creating a complex bureaucratic process for approvals without putting a time- or conditions-based limit on the decision.

If there is no incentive for a committee or panel to make a decision, why would they?

Finally, the Framework requires yearly safety drills. For a fast-moving situation, trying to “wing it” is almost certain to fail, so I’m happy to see this provision here. I’d be even more interested in OpenAI’s incident response procedures, but completely understand they’ll want to keep those confidential.

Conclusion

If you have the time and interest, I recommend reading the full document. It is a pretty comprehensive approach to the AI safety problem. While it would have benefitted from risk quantification, it still provides some excellent practices and procedures for organizations building AI governance and cybersecurity programs.

Need help with yours?