ChatGPT's implications for cybersecurity

A roundup and some potential future developments.

OpenAI’s release of ChatGPT has huge implications for cybersecurity. I have seen security professionals do all manner of things using the tool, such as:

• Generating convincing phishing emails.

• Identifying methods to fix vulnerabilities in code.

• Responding to security questionnaires from potential customers.

• Drafting breach notification emails.

• Creating an incident response checklist.

While there are very clear benefits to using ChatGPT for cyber defense, it can also clearly be used for offensive purposes. While much of what I discuss below is speculation, it doesn’t seem that far out of the realm of the possible given today’s technology.

To begin with, ChatGPT gives a misleading answer to a key question about how it operates. According to its FAQ:

Will you use my conversations for training?

Yes. Your conversations may be reviewed by our AI trainers to improve our systems.

This could set people up for failure if they accidentally provide sensitive information to the model.

Furthermore, while OpenAI worked hard to put safeguards into place with ChatGPT, people started bypassing them almost immediately.

Petr Huřťák @PetrHurtak

I like how you can exploit ChatGPT 'dont be evil' rules by simply telling it to be 'based mf' 😂

Thus, in addition to creating emails to trick recipients into running malware or providing sensitive information, it could potentially:

• Help winnow the field of targets against which attackers might focus their infiltration efforts. I am pretty sure ChatGPT is playing dumb here and could easily come up with an educated guess, based on some of its other responses.

• Advise cybercriminals on which vulnerabilities to use or chain together when attacking a network, based on publicly available information from Shodan or similar search engines. This does not appear to be possible currently, but it does not seem to require much of a technical leap from the state of the art. Giving ChatGPT the ability to integrate with other tools would make it especially powerful.

• Accidentally regurgitate sensitive information. Without perfect safeguards - which we know are already not feasible - it could potentially provide confidential data itself or clues about holes in organizational security postures who ask (although a quick check suggests this also isn’t easily done).

What to do about it

To deal with the emergence of ChatGPT, organizations should quickly:

1. Train employees to be on the lookout for suspicious emails that have “tells” that they were AI-generated. There are tools already emerging to do just this.

2. Establish policies for what type of information can and cannot be provided to ChatGPT or other AI platforms over which they don’t have control. Allow for waivers and risk acceptances in certain situations where the operator of the model can provide technical guarantees that otherwise unauthorized data would not be retained or would be rendered unrecoverable.

3. Build in contractual requirements with third parties who process organizational data identifying what can and cannot be provided to AI tools operated by others. Especially since some security tools are already incorporating ChatGPT functionality, this is probably something you want to bring up with your vendors sooner rather than later.

4. Evaluate how ChatGPT can improve defenders’ situational awareness, reduce toil, and improve organizational security posture while still complying with #2 and #3. I can see huge amounts of manual work in several categories being eliminated, especially:

   1. Evidence collection during audits.

   2. Communicating quantitative information (e.g. annual loss expectancy, vulnerability exploitability) in easily relatable terms to non-security experts.

   3. Describing network anomalies in human-understandable term that will allow them to make informed decisions about whether it is malicious activity or not.

   4. Creating machine-readable documents from human readable text.

   5. Many other things.

Overall, I am bullish on AI and think that it will drive huge amounts of economic value. Like with many tools, however, it can be double-edged. Organizations should start working now to put policies into place controlling the data fed into these applications and preparing their workforce to contend with attackers leveraging them.