AI security at enterprise scale: preventing data leakage from unintended training

Harnessing the AI revolution without exposing sensitive information.

Today’s post is a collaboration with Aishwarya Srinivasan, a senior AI Advisor at Microsoft.

The blistering speed of AI progress is rapidly reshaping the way companies do business. Especially impactful has been an explosion in generative AI capabilities, which allow enterprises to process huge volumes of unstructured data and create entirely new content easily.

Emerging technologies almost always introduce risks, though, and the AI revolution is no different. Companies leveraging generative AI need to contend with a host of challenges stemming from:

• Incorrect outputs that result from model “hallucinations”

• Ethical and intellectual property concerns

• Cybersecurity vulnerabilities

But what has been top of mind for security teams are concerns about data leakage.

Unintended training and the risk to data confidentiality

Many organizations have been hesitant to embrace generative AI tools because of the potential for accidentally exposing sensitive or proprietary data to them. This phenomenon occurs due to unintended training of AI models on things like:

• Personally identifiable information (PII)

• Technical documentation and code

• Confidential business strategies

After training on this information, these models can regurgitate it partially or entirely to those not authorized to see it. Especially for tools that continuously train on inputs by default, like ChatGPT, this can be a major problem. Companies like Amazon and Samsung have both reportedly fallen victim in this way.

While data leakage from unintended training is certainly a major challenge to enterprise adoption of AI tools, there are concrete steps companies can take to reduce its risk.

Administrative controls: employee awareness and training

The first line of any organization’s cybersecurity defense is its employees. If sufficiently trained and aware, they can avoid many threats to data confidentiality, integrity, and availability. To make sure they are properly equipped, enterprises should consider:

Operationalizing an AI policy

The bedrock of any security program - for AI or otherwise - is a policy that lays out, among other things:

• The organization’s risk appetite and tolerance

• Accountable leaders and responsibilities

• (Un)acceptable behaviors

Clearly specifying these can help employees to understand:

• Which types of data can be processed using which systems

• Whom to ask for clarification or exceptions

• Consequences of non-compliance

An accessible and concise policy can reduce the likelihood of well-intentioned team members exposing data to AI training which they should not.

Creating a streamlined procedure of AI tool assessment

Along with an AI policy providing high-level guidance, companies should also develop an actionable and easily-navigated procedure for onboarding new AI tools.

Data leakage from unintended training often occurs from employees simply trying to “get things done” under pressure. A well-designed, efficient, and risk-based approach to approving and reviewing AI tool use can help motivate employees to stay within the company’s risk appetite.

Regularly training employees on the risks of unintended training

Policies and procedures no one knows about don’t mitigate any risk. So regular training - preferably as part of an existing security awareness program - is key.

Boilerplate, slide-based training doesn’t generally engage employees effectively, so alternative approaches can be more effective, such as:

• Scenario-based role-play and interactive discussions

• Short quizzes to evaluate understanding

• Real-time, context-driven notifications

The last option - gently “nudging” employees into more secure behavior when they are about to or are already using AI tools - begins to overlap with the next category of controls: technical measures.

Stopping unintended training through automated enforcement

A layered defense is generally most effective when it comes to organizational cybersecurity. Protecting against data leakage from unintended AI training is no different. That’s why the most secure organizations enforce their policies and procedures through technical means like:

Pre-processing data

Even the best-trained employees can make mistakes. And automated solutions to redact sensitive data prior to sending it to an AI model can help. Both open source and commercial tools exist to identify and sanitize things like:

• Social security numbers

• Dates of birth

• Names

Removing these things from model inputs can help to reduce the likelihood of unintended training and subsequent leakage.

But this is not a fail-safe solution, unfortunately. Especially for less predictable or identifiable data structures, such as business plans or source code, rule-based pre-processing won’t catch everything.

Self-hosting AI models

While proprietary Software-as-a-Service (SaaS) models like GPT-4 still seem to have the edge when it comes to performance, open source peers like Mixtral are rapidly catching up. Running these models yourself has the added advantage of facilitating more observability over organizational data flows.

Whether hosting them using Infrastructure-as-a-Service (IaaS) providers or truly on-premises in an owned or leased data center, companies deploying open source models (or even developing their own) can apply more granular controls when it comes to access control and AI training policies.

With that said, this architecture requires substantial know-how from both a data science and security perspective. And due to the pervasive risk of cloud misconfigurations, it’s possible self-hosting can be even less secure than using SaaS tools from an overall perspective.

Blocking access to especially risky AI tools

As a final measure, enterprises can create denylists of AI tools they deem excessively risky due to their training or data retention practices. Many organizations have taken this path with tools like ChatGPT.

Unfortunately, constantly attempting to block newly-emerging tools will be a never ending game of “whack-a-mole.” Validating and implementing such rules are bound to absorb a substantial amount of engineering and information security resources. And according to one survey, 8% percent of employees at companies banning ChatGPT admitted using it nonetheless. If there is a will to use generative AI tools, people can generally find a way.

Balancing risk and reward with the AI revolution

While security and other challenges abound when deploying AI systems, companies also need to consider the risks of moving too slowly:

• Competitive pressures demand rapid advancements in productivity

• Customers may lose faith in vendors appearing to fall behind

• Stifling innovation will drive talented employees elsewhere

That means balancing all AI risks - security and otherwise - is the key task for the modern enterprise. Data leakage from unintended training is a serious but manageable one. With the right approach, companies can maximize productivity gains while minimizing cyber risk.