5 ways to evaluate open source AI model security

SCA, AI-powered code review, pre-deployment testing, model and data provenance analysis, and change control.

1. Software composition analysis [SCA]

I recommend including these tools as part of any development pipeline, but remember they only catch KNOWN:

• Vulnerabilities

• Malicious code

• Data poisoning

SCA would identify findings logged against models in repositories like Hugging Face.

But only if:

• an ethical party has found it first, AND

• reported it to a vulnerability DB, AND

• you have access (if it's not free).

So this is not foolproof.

2. Manual or AI-enabled code review

An expert can review the code for:

• Backdoors

• Unexpected callbacks

• Accidentally-introduced vulnerabilities

Of course, this can be very time-consuming.

So you can use another AI tool to do the heavy lifting, like I did with Llama 2.

If an attacker were really trying to be sneaky, they could hide malicious instructions as comments in the source code to prompt inject an AI tool reviewing it.

But this is risky for them: a human review would spot it easily.

3. Model and data provenance analysis

Even when downloading bona fide versions (i.e. not malicious ones pretending to be the original), there is still a risk with respect to data provenance and training.

I have encountered engineers using models trained on or by:

• Russian

• Chinese

• Unknown

sources, without security teams being aware.

These weren’t maliciously seeded into production environment, but rather intentionally put there by developers looking for the “latest and greatest.” This analysis, for example, lays out some of the risks of using even a locally-deployed version of DeepSeek (see point 5 in the article).

4. Pre-deployment testing

There is no sure fire way to determine how a model will behave in the real world without testing. Things to look for:

• Adherence to business requirements

• Undesired but accidental bias

• Intentional data poisoning

Establish target scores (e.g. F1) ahead of time, and confirm your model meets your requirements.

5. Actionable change control procedures

If your developers are going to be rapidly shifting models, you will need a way to evaluate them for:

• If following the standard, ISO 42001 Annex A adherence

• Security and compliance concerns described above

• Intellectual property issues

If you don’t have a comprehensive and continuously-updated asset inventory, along with clear release and approval gates, tracking and treating risks will be very difficult.

Deploying open source models? Not sure how to vet them?

For those looking for more control and observability of their AI deployments, open source may be the way to go. While the risk of unintended training is lower, though, remember that misconfigurations are more common when using Infrastructure-as-a-Service (IaaS) vs. Software-as-a-Service (SaaS).

The key to making the right call: a thorough risk assessment.

Need assistance doing one? StackAware can help.