Frame AI risk with the NIST RMF
Part 1 of a series on the National Institute of Standards and Technology's Artificial Intelligence Risk Management Framework.
What does GRC look like for AI?
Although it’s a rapidly-changing field, there is in fact a governance, risk, and compliance (GRC) framework for artificial intelligence (AI).
The National Institute of Standards and Technology (NIST) AI Risk Management Framework (RMF), less than 6 months old, fills this role.
Because of its “first mover advantage,” I think there is a strong possibility the AI RMF becomes the industry standard. This, as others have suggested, is what happened with the NIST Cybersecurity Framework (CSF, which arrived relatively early, in 2014) but did not happen with the NIST Privacy Framework (which arrived relatively late, in 2020).
So it’s time for a detailed teardown.
Over the next several issues, I am going to dive deep into the various sections of the AI RMF. In addition to summarizing the government-speak endemic to these sorts of documents, I’ll also make some actionable recommendations along the way.
In addition to the framework itself, NIST also provides an entire Trustworthy & Responsible AI Resource Center, which includes a detailed implementation playbook. During this series, I’ll examine the playbook as it pertains to the key “functions” of the AI RMF.
To help with those reading along with the full document, I’ll mirror its headings to the extent practical. For this issue, we’ll begin with Part 1 of the RMF, “Foundational Information.” In subsequent ones, we’ll go through the four functions:
Govern
Map
Measure
Manage
1. Framing Risk
The document begins by defining risk as:
the composite measure of an event’s probability of occurring and the magnitude or degree of the consequences of the corresponding event.
So far, so good. This sets you up for quantitative measurement, which I have frequently pointed out is the best approach.
1.1 Understanding and Addressing Risks, Impacts, and Harms
In terms of who might suffer the consequences mentioned above, NIST suggests there are three types of harms, those to:
People
An organization
An ecosystem
These categories are not MECE, and in the end, what anyone really cares about is the first category. So I don’t love this framing.
With that said, any compliance program ultimately built around the RMF will no doubt need to evaluate risk through these lenses.
Describing harms in terms of dollars and cents is applicable to all through categories, so aim for that.
1.2 Challenges for AI Risk Management
This is a relatively thoughtful section exploring risk measurement, tolerance, and prioritization. Although it gives lots of food for thought, unfortunately it doesn’t give many hard answers to practitioners.
1.2.1 Risk Measurement
Here NIST highlights difficulties associated with identifying and cataloging risk due to:
Third-party software, hardware, and data
This is pretty straightforward and a challenge for any technology system. Not using any of these things is basically impossible in the modern economy, though, so third-party risk is going to be omnipresent challenge. You’ll need to have an effective software supply chain risk management program, with AI-specific considerations, to stay on top of it.
Emergent risks
This is what I would call “unknown unknowns.” These are the things that aren’t even on your radar and thus have no way of trying to identify or mitigate. Brainstorming, red-teaming, and other thought experiments can help to uncover these. And you’ll want to make sure you regularly update your risk register as you identify them.
Availability of reliable metrics
The key quotation here is:
measurement approaches can be oversimplified, gamed, lack critical nuance, become relied upon in unexpected ways, or fail to account for differences in affected groups and contexts.
As we have seen from older cybersecurity frameworks, organizations will index heavily on certain metrics without fully thinking through the implications of doing so. This promotes (and is promoted by) a “box-checking” mentality which can lead organizations astray in a variety of ways.
Constantly asking “why is this metric important?” is a good way to keep yourself honest, although it is difficult to do in the face of bureaucratic inertia.
Characteristics of different AI lifecycle stages
“Shift left” for AI, anyone?
This section discusses how risk assessments for a developer training a model might different from the end user operating it in the real world. Especially when considering autonomous systems, making sure the end user understands for what purposes the model was developed - and uses it accordingly - is crucial.
Similarly, model developers should anticipate how their products will be used in the real world, and make clear which purposes and deployment models are appropriate and which ones are not.
Contrasting real-world and controlled settings
This goes hand-in-hand with the “emergent risks” category. As OpenAI quickly saw after the release of ChatGPT, clever people all over the world quickly found jailbreaks for their app, using it in ways they had never anticipated.
Similarly, even without human initiative, managing risk requires understanding how an AI would behave differently “outside the lab.”
Inscrutability
A better word here might be “opacity,” but this basically says that if one can’t understand an AI model well, it is difficult to assess risks from it.
You don’t say…
Human baseline
This section didn’t make too much sense to me, especially because NIST omits a critical point:
The human decisions on which an AI is trained could have been either objectively wrong or impossible to reconstruct after the fact.
For example, if doctors’ analysis of X-rays trains an AI, and the humans made mistakes, it might impossible to determine that they were wrong or why. Without this information, it would impossible to measure the accuracy risk in the resulting model.
So the key thing to keep in mind that your human baseline might be wrong.
1.2.2 Risk tolerance
Understanding organizational or personal risk tolerance1 is probably the most important step of any decision-making process. Many individuals and even large corporations only have an implicit sense of what their risk tolerance is. Conversely, they will often claim that they have none whatsoever.
Unfortunately, such unclear risk tolerance frequently leads to unexpected or sub-optimal outcomes. That’s because it makes systematic weighing of risks against rewards impossible and puts these critical decisions at the full mercy of human cognitive bias.
Especially when it comes to AI systems, where there are so many different factors to consider, establishing a risk tolerance - preferably in terms of annual loss expectancy, in dollars or other currency - is even more crucial.
Despite the importance of this step, the RMF does not prescribe or even suggest how to arrive at an appropriate risk tolerance, aside from the vaguest of statements.
While doing this for a specific situation would require substantial analysis (let me know if you need help here), some things to consider when determining your risk tolerance are:
Statutes and regulations
This includes asking questions like:
Could this AI put me or my organization in legal jeopardy?
Does operation of this AI trigger any reporting requirements?
What are the historical government fines associated with mishaps or incidents that my AI could conceivably cause?
Do changes in my user base impact (which might change without me knowing) impact which requirements I am subject to (e.g. the California Consumer Privacy Act or European Union General Data Protection Regulation)?
Contractual obligations
While breaching contracts won’t land you in jail, it could certainly land you in court on the receiving end of a lawsuit. Some things you should consider are:
What do my data processing and protection addenda say?
Are there any blanket bans on using AI in my agreements?
Have I made any forward-looking commitments to undergo audits by my customers? How will I comply with these? Can I even comply?
Compliance frameworks
While generally not required by law, maintaining compliance with frameworks like SOC 2, ISO 27001, or FedRAMP will often be a precondition for a sales process (or honoring an existing contract). Make sure you understand:
How do auditors view the use of AI tools?
Are there any specific compliance framework requirements related to the deployment or use of AI?
How can I use existing language of a given framework - which may or may not have anticipated the use of AI - to explain how my new technology complies with it?
Reputation
This is a big one from a business perspective. Since virtue-signaling a is a key component of any corporate communications strategy (I say this without sarcasm and understand that every company owes a fiduciary duty to its shareholders to maximize value through any lawful means available), you might consider:
Have customers or activist groups expressed concerns about algorithmic bias?
Should we leverage a tangentially-related event to cynically declare we will no longer offer a certain technology because it’s a small portion of our revenue and we are falling behind our competitors anyway, reaping the public relations (PR) rewards?
Or do we leverage the same event to declare a one-year pause in developing this technology, where we do have a competitive advantage, and then restart development once the heat has died down?
Has our AI system caused damage, whether physically, financially, or emotionally, to our customers or random bystanders? How will this impact the value the system - or our organization as a whole - can deliver in the future?
Cost
While AI can certainly help improve business processes and operations, it also has its own costs to deploy and manage. So you should ask:
Compared to the benefits delivered, how expensive is this AI?
Are there more cost-effective ways of doing this without AI?
Ethics
This is where it gets very challenging to put numbers on things, but it’s even more critical to do so here. For example:
How many numbers or types of jobs are we willing to destroy by using AI to automate this process?
On top of the legal, regulatory, and reputation penalties, what are the potential damages caused by the AI from a moral perspective?
1.2.3 Risk prioritization
This section talks a lot about prioritization at a high level, and isn’t especially helpful.
But an important acknowledgement is that:
Unrealistic expectations about risk may lead organizations to allocate resources in a manner that makes risk triage inefficient or impractical or wastes scarce resources.
When you are designing, developing, deploying, and operating an AI system, keep the concept of “risk mitigation return on investment” top of mind. This will help you focus on the biggest risks that can be mitigated most rapidly, without chasing long tail potential outcomes.
1.2.4 Organizational Integration and Management of Risk
Managing risk holistically is of course a central task of any leader. Financial, legal, technological, competitive, cybersecurity, privacy, and AI-specific risks all compete for attention and resources.
Thus, figuring out a comprehensive method for addressing all of them in the most economical manner will require a lot of bandwidth. Toward this end, NIST advises treating AI risk as similar to cybersecurity and privacy risk and also incorporating additional enterprise risk frameworks alongside the RMF.
Unfortunately NIST doesn’t provide any detail on how to do so.
Dealing with overlapping frameworks is major challenge for any organization, and this is an area that could use substantial work. So in the longer term, I’ll considering documenting a “trifecta” approach that overlays the NIST cybersecurity, privacy, and AI risk management frameworks.
2. Audience
I will skip this section entirely because it seemed quite out of place and mainly refers to an already-existing Organisation for Economic Co-operation and Development (OECD) Framework for the Classification of AI systems.
3. AI Risks and Trustworthiness
This section spells out the criteria through which someone using the RMF should evaluate an AI system. Through testing, evaluation, verification, and validation (TEVV) processes, those reviewing a system can use some or all of these characteristics to determine what to do with an AI.
When developing these in the real-world, I would advocate strongly for having quantitative criteria to drive TEVV. Due to organizational maturity or operational constraints, however, it may be necessary to provide qualitative ones.
In this case I strongly recommend avoiding terms such as “low,” “medium,” and “high.”
Rather, focus on categorical or binary outcomes. This will help to avoid bias and inconsistency.
I’ll give some examples of what these might look like below for each criteria. The quantitative ones will express probabilities of adverse events. Applying these across a 365 day period and multiplying by the expected impact of such an event (in dollar terms), will provide you with an annual loss expectancy (ALE).
Valid and reliable
The bottom line here is that the model needs to work and it needs to do so consistently. You’ll need quantitative methods to evaluate its effectiveness over time to help trade off accuracy against other desirable characteristics.
The traditional data science rubric for evaluating this is precision vs. recall.
Precision in a model refers to how many of the identified positives are actually true positives. High precision means that the model makes fewer false positive errors.
Recall measures how many actual positives a model captures by labeling them as positive (i.e. true positive). High recall means the model makes fewer false negative errors.
In some instances, improving precision can decrease recall, and vice versa, hence the tradeoff. This is because, to increase precision (and reduce false positives), the model might become more conservative in its positive predictions, potentially missing some true positives, which lowers recall.
Conversely, to increase recall (reduce false negatives), the model might predict positives more liberally, which can lead to more false positives and thus lower precision.
Quantitative example: a voice-to-text model must have:
Word Level Precision of 90%. This means the model transcribes 900 words correctly out of 1,000 words the model identified and decided to transcribe.
Word Level Recall of 90%. If the model correctly transcribes 900 out of 1,000 words spoken by the source.
Quantitative example: a facial recognition algorithm must have:
Image Level Precision of 80%. This would be achieved if the algorithm identifies 800 images correctly out of a total number of 1,000 images with identified faces.
Image Level Recall of 80%. If the system correctly identifies an individual’s face in 800 out of 1,000 images of the individual, the recall would be 80%.
Safe
This is pretty weak.
According to the RMF:
AI systems should “not under defined conditions, lead to a state in which human life, health, property, or the environment is endangered.”
Easy to say.
Basically impossible to implement with just this information.
The key here, in my view, is that the AI needs to do less harm than the alternative, incorporating all of the other costs involved in deploying it.
That’s why people freaking out about autonomous vehicle safety generally have it wrong in my view.
Depending on whom you ask, self-driving vehicles cause dozens of fatalities a year.
Compare this to 42,915 total traffic deaths in 2021, the vast majority of which resulted from human error!
Obviously autonomous vehicles have not been deployed nearly as fully as human-driven ones have. But assuming they replaced 90% of human-driven cars and caused 10,000 fatalities a year (which seems insanely high), that would be a great tradeoff in my mind!
14,292 (10,000 autonomous-caused + 4,292 human-caused) deaths is better than 42,915!
The key point here is that the mere fact an AI endangered human safety a single time should not cause you to shut it down.
You’ll need a holistic analysis to determine how relatively “safe” the system is.
Quantitative example: an robotic surgeon causes unnecessary tissue damage (i.e. not required by the procedure) in no more than 0.001% of operations.
Categorical example: an AI-driven forklift never decides to harm any human it encounters.
Obviously you’ll need sufficient data to determine if the AI decided to do something (see the “Accountable and Transparent” section below).
But I would say that if you uncover evidence of it happening, it would represent appropriate criteria to shut down and completely rebuild an AI.
Secure and resilient
This is another section of the RMF that could be an entire book.
NIST defines security and resilience in terms of responses to malicious activity, which I think is appropriate. Avoiding, protecting against, responding to, and recovering from attacks is critical to deploying AI securely and also separate from other concerns described in the RMF.
Since AI security and resilience is such a broad topic, and this is a cybersecurity-focused newsletter, I am going to weave analysis of it throughout this series.
But if you haven’t already seen these things, here are some actionable resources to get started with your AI security program:
Example security metrics include:
Quantitative: confidentiality loss of data from the AI system does not exceed $10,000 per year (using the Factor Analysis of Information Risk [FAIR] method for quantifying damages).
Categorical: the model requires multi-factor authentication (MFA) to access it, and none of the MFA options include an SMS code or email sent to the user.
Accountable and transparent
Another weak section here.
It begins saying:
Trustworthy AI depends upon accountability. Accountability presupposes transparency.
But then proceeds to not define or even discuss accountability any further.
How can a computer system be accountable for anything? The humans designing and deploying certainly may be, but to describe an AI as “accountable” makes zero sense to me. What are you going to do if it misbehaves? Sue it? Throw it in jail?
Transparency is discussed more in depth but still at a high level.
According to NIST, it
reflects the extent to which information about an AI system and its outputs is available to individuals interacting with such a system – regardless of whether they are even aware that they are doing so. Meaningful transparency provides access to appropriate levels of information based on the stage of the AI lifecycle and tailored to the role or knowledge of AI actors or individuals interacting with or using the AI system.
Super not helpful.
Breaking down exactly what “transparent” means for a specific system will be absolutely critical if organizations are ever going to attempt to actually implement the RMF.
Quantitative example: the model correctly returns the output of a given decision (randomly selected by a human auditor or cumulatively across all decisions) 99.9% of the time.
This is different than accuracy, in that we aren’t looking at how good the decision was. We are looking at whether or not the AI returned the correct output for a historical decision it has made. As we have seen with large language model (LLM) hallucinations, AI systems might not even correctly recall which information they have provided in the past.
How do you trust the AI on whether or not it is returning the correct output? You’ll probably want to track all of its outputs in a logically separated database for both recovery and auditing purposes. Comparing the two will allow you to figure out this value.
Explainable and interpretable
The TL;DR of this section is that for an AI system, the characteristic of:
Transparency answers “what happened?”
Explainability answers “how was the decision was made?”
Interpretability answers “why was the decision was made?”
In a separate callout, NIST makes a point with which I don’t agree saying “accurate but opaque and uninterpretable systems…are undesirable.”
This all comes back to the risk/reward tradeoff.
If an AI system cures cancer, writes a bestselling novel, and kisses babies on the cheek, then frankly, I am okay with it being a little opaque.
Now of course if the AI is just putting up a front while it plots world domination, then this might tip my analysis in the other direction.
But some opacity is probably a risk organizations will need to accept.
Quantitative example: when questioned, an AI x-ray analyzer provides an explanation for its diagnosis that matches a human doctor’s 95% of the time.
Categorical example: the model never refuses to justify or explain its actions.
Privacy-enhanced
NIST approaches the question of privacy at a high-level, recommending:
Privacy values such as anonymity, confidentiality, and control generally should guide choices for AI system design, development, and deployment.
The RMF talks about tradeoffs, e.g. using privacy preserving techniques like minimization might reduce accuracy, which all makes sense.
At a more concrete level, I would say:
You should redact personally identifiable (PII) and protected health information (PHI) before giving data to an AI system as long as doing so doesn’t reduce its effectiveness.
If there is a tradeoff to be had, get affirmative consent from the people whose data is being processed. Explicitly outline the risks and potential rewards. I talk about a related example here.
To get you started here, I would recommend the below resources:
An additional risk you’ll need to consider, from both a privacy and security perspective, is that of aggregation. Even if you don’t directly feed an AI someone’s home address, it’s certainly conceivable that it could intuit it if it has sufficient additional data.
This risk is unique to AI systems and will require well thought-out controls to mitigate.
As for examples, see below.
Quantitative: a pre-processing module correctly identifies and removes PHI from data before it is provided to the main AI model 99% of the time.
Categorical: the model never returns an unredacted social security number, even if prompted by a human user.
Fair - with harmful bias managed
No government document published in the past decade would be complete without some vague reference to harmful bias that doesn’t define the term. And NIST acknowledges that it is punting on this topic in general by saying:
Standards of fairness can be complex and difficult to define because perceptions of fairness differ among cultures and may shift depending on application.
Vagueness notwithstanding, NIST identifies three types of AI bias:
Systemic
Computational and statistical.
Human-cognitive.
While outside the cybersecurity-specific scope of this newsletter, you likely are going to need to be able to answer for all of these things if you are ever hoping to claim “compliance” under the AI RMF.
I’ll skip the examples for now, as does NIST. Wading into this subject seem like a no-win situation.
Conclusion
Because those at the “sharp edge” need to make decisions, my goal is to equip them with some tools to actually implement the AI RMF. It’s still early days, but I think we will begin to see best practices emerge.
Want to learn more about how to prepare your company for AI-powered transformation?
What the AI RMF refers to as “risk tolerance” is what I would define as “risk appetite,” but I will use NIST’s nomenclature in this series for consistency. For a detailed breakdown of the distinction between the two, please see Managing your risk surface.