3 ways to evaluate open source AI model security

SCA, AI-powered code review, and pre-deployment testing.

1. Software composition analysis [SCA]

I recommend including these tools as part of any development pipeline, but remember they only catch KNOWN:

• Vulnerabilities

• Malicious code

• Data poisoning

SCA would catch findings logged against models in repositories like Hugging Face.

But only if:

• an ethical party has found it first, AND

• reported it to a vulnerability DB, AND

• you have access (if it's not free).

So this is not foolproof.

2. Manual or AI-enabled code review

An expert review the code for:

• Backdoors

• Unexpected callbacks

• Accidentally-introduced vulnerabilities

Of course, this can be very time-consuming.

So you can use another AI tool to do the heavy lifting, like I did with Llama 2.

If an attacker were really trying to be sneaky, they could hide malicious instructions as comments in the source code to prompt inject an AI tool reviewing it.

But this is risky for them: a human review would spot it easily.

3. Pre-deployment testing

There is no sure fire way to determine how a model will behave in the real world without testing. Things to look for:

• Adherence to business requirements

• Undesired but accidental bias

• Intentional data poisoning

Establish target scores (e.g. F1) ahead of time, and confirm your model meets your requirements.

Deploying open source models? Not sure how to vet them?

For those looking for more control and observability of their AI deployments, open source may be the way to go. While the risk of unintended training is lower, though, remember that misconfigurations are more common when using Infrastructure-as-a-Service (IaaS) vs. Software-as-a-Service (SaaS).

The key to making the right call: a thorough risk assessment.

Need help doing one? StackAware can help.