External audits: a better solution for 3rd (and greater) party risk management?
tl;dr - audit reports are better tools than security questionnaires for identifying external cybersecurity risk, but leave much to be desired. Don't rely solely on them.
External audits involve an (ostensibly) independent organization reviewing another’s cybersecurity posture and providing some sort of written assurance or certification that customers and other interested stakeholders can review.
Here’s the good news:
Knowing about the target’s policies, procedures, and past actions can provide some information on the likelihood of it introducing, identifying, and remediating vulnerabilities in systems upon which your data relies.
Having an outside party attest to the cybersecurity of a given company, product, or service is also certainly a step up from asking the organization in question to do so itself (e.g. via a security questionnaire).
The fact that many of these standards require controls on the target organization’s vendor management processes also helps to address the problem of 4th party cyber risk.
Finally, the simple fact that the audited organization was able to get its affairs in order sufficiently to pass the audit suggests that it approaches security with at least a minimum baseline of seriousness.
With that said, outside audits and attestations are not perfect tools for identifying information security risk in >3rd parties:
In general, there is an inherent conflict of interest in any situation where an organization pays another to vouch for it. In addition to historical occurrences of clear auditor misconduct, there is a more insidious problem: auditors are less likely to be tough because they don't want to anger their clients.
Furthermore, open-source organizations are almost certain to be unwilling (no financial incentive) or unable (insufficient governance mechanisms) to undergo such an audit, meaning that these tools are only useful for evaluating vendors you pay.
Even for those that you do, only companies that provide cloud hosting or Software-as-a-Service (SaaS) are likely to have such reports available. Traditional makers of software meant for on-premises deployment in a data center or your own cloud environment are unlikely to pursue the requisite audits because so little of the relevant criteria covered by many of them (e.g. board of directors independence, internal ethics policies, financial accounting standards, etc.) are likely to apply.
Finally, the vast majority of these external reviews are focused on whether or not certain types of controls are merely in place, with much less focus on their technical effectiveness in preventing damage to businesses that rely on the target organization. Thus, I would say that receiving a favorable opinion from an outside auditor is often correlated with strong security practices, as organizations that have sound processes and management practices are generally more likely to protect themselves against cyber threats. Unfortunately, having such controls in place does not necessarily cause the organization to be more resilient against malicious actors.
Turning to a specific framework, the System and Organization Controls (SOC) 2 attestation, specifically the Type II variant, is often seen as a cybersecurity “gold standard” for businesses in the United States.
I am not entirely sure why this is.
In addition to – what in my mind is – the hilarity of something promulgated by the professional accountancy organization (the American Institute of Certified Public Accountants) serving as the de facto industry standard for commercial cybersecurity, there are other problems.
Most glaringly, it is very challenging to make an “apples-to-apples” comparison between two organizations having received SOC 2 attestations by external auditors. The organization undergoing the audit is responsible for determining its scope as well as the nature of the controls in place. The external reviewer is merely responsible for ensuring that the audited company properly designs (Type I) and enforces (Type II) these controls. Similarly, the organization undergoing the audit can choose from a list of Trust Services Criteria (except for security, which is mandatory) for which it is evaluated.
Especially since your organization may rely upon 3rd parties handling vastly different criticality and sensitivity of data, the lack of a ready comparison between organizations with SOC 2 attestations makes determining the latent risk a difficult endeavor.
Furthermore, the information about organizational controls provided in SOC 2 attestation reports is usually qualitative and high level. For example, one of the criteria evaluated is whether the target organization conducts vulnerability scans. For a modern SaaS provider operating containerized applications, just knowing that it scans for vulnerabilities on a regular basis is not enough. If such an organization simply had a policy of running an open-source vulnerability scanning tool every month against its network - and complied with said policy - an auditor could quite reasonably confirm that the company has this control in place. That would be a far cry from industry standard practices such as integrating static and software composition analysis into the continuous integration/continuous deployment (CI/CD) pipeline. It is unlikely from reading a SOC 2 report, however, that you would be able to tell precisely which measures (names of tools, code coverage, CI/CD integration) the organization has in place.
In terms of other standards, there are others that are specific to certain industries and geographies. The ISO 27001 certification - popular for European businesses - is slightly more prescriptive in terms of its requirements but remains quite similar overall to the SOC 2 standard. Thus, knowing a vendor has one should give you the same (low-to-moderate) level of confidence in its cybersecurity.
The U.S. Department of Defense's Cybersecurity Maturity Model Certification (CMMC) is predictably applicable only to military contractors. Due to the endless controversy regarding the standard and the accompanying certification process as well as a subsequent Biden Administration Executive Order, much remains unclear about its future. Unless you are a government acquisitions officer and absolutely must check for CMMC compliance, at this point I would not take much solace in knowing that a vendor has the certification.
With all of the above said, an external audit is better than nothing for vetting your software supply chain. If another organization upon which you depend provides you with a SOC 2 or similar report, it is certainly worth your time to at least skim it. At a minimum, you should confirm that the report actually covers the functions and services upon which you will rely. A review of the type of controls in place – especially as they relate to vulnerability identification and remediation, threat identification, and incident response procedures – can shed additional light on the risk picture. Although these sections will likely lack granular detail, they can serve as a starting point for a conversation with the entity in question on how comprehensive their controls actually are.
Although they represent a step up from security questionnaires, external auditor reports are not a panacea and those managing technology supply chains should not rely upon them entirely. There is likely to be some correlation between an organization's ability to achieve a favorable auditor opinion and the strength of its cyber defenses, but remember that the former is not necessarily dispositive as to the latter. Additionally, there remains an inherent conflict of interest between any auditor and the organization it is reviewing. Finally, the relatively high-level information found in attestation reports and the extreme difficulty of comparing them to each other limits the effectiveness of these tools in managing 3rd (and greater) party cybersecurity risk.