The Evolution of Penetration Test Reports

04 Dec 2020

Is it Time to Treat Cyber Risk like Financial Risk?

Cybersecurity risk has long been seen as an internal information technology issue, with penetration test reports being used for IT risk management and to demonstrate regulatory compliance. But the relentless rise in high-profile security breaches, increases in fines, incident response costs, and lost business means cyber risk now needs to be communicated to boards and organisation stakeholders.

While penetration test reports have historically focussed on technical risks, and were therefore written for a technical audience, the shift in target audience from chief information security officer (CISO) and IT practitioners to board members and stakeholders means reports must be accessible to this wider audience.

Some examples of reporting deficiencies that cause problems for a non-technical audience are:

1. Unsatisfactory executive summaries.

   The report's executive summary is the most important section for a high-level audience. It should be directed at a non-technical audience and include an overview of the findings and their potential impact within the context of the business and sector in an accessible, unambiguous, and concise manner. Many reports contain unsatisfactory executive summaries with poor grammar, technical jargon, and sometimes a lack of understanding of the client's business sector. Drafting the executive summary is an important task that requires input from a senior practitioner with a good understanding of the client's business and sector.

2. Insensitive wording

   Most pentesters have not sat on the customer's side of the table during a stakeholder meeting, so do not appreciate the effect poorly formulated wording can have on clients or stakeholders. Pentesters should be aware that even a phrase that ostensibly sounds harmless can cause embarrassment to the client.

   Some phrases used within the tech industry are seen as exclusionary or even offensive to some people. The tech industry is starting to move away from these terms towards more inclusive terms, and cybersecurity providers should embrace this change to ensure reports are acceptable for a wide and diverse audience.

3. Imprecise or ambiguous wording

   Executive summaries sometimes contain imprecise wording, generalisations and opinions, whereas they should be factual, precise and neutral. For example, a statement such as "all your servers were well patched" should be avoided because "all servers" is imprecise and "well patched" is an opinion. A more precise statement would be: "The 30 servers listed in appendix A were found to be up to date with all important and critical security updates at the time of testing".

   Imprecise wording could cause different parties to have differing opinions on the exact meaning and thus degree of risk. In extreme cases this might result in legal liability if a security breach exploited a vulnerability that was not accurately communicated.

4. Inappropriate opinions

   Some reports contain opinions that the tester is not qualified to make. These opinions are often made with good intentions but can result in a misleading report. Pentest frameworks are generally designed to discover, verify, and report vulnerabilities rather than to prove the absence of vulnerabilities or give an opinion on the client's security posture.

   A common example is stating imprecise and unverifiable opinions such as "the system was well secured" with no context or supporting information. Another is stating that a vulnerability cannot be exploited because the tester was unable to exploit it.

As cyber risk management moves from an operational IT issue to a business issue with board and stakeholder visibility, cyber security firms need to ensure that their reports are accessible to this wider audience and provide the degree of confidence necessary for these groups to understand the level of risk and make informed decisions.

It may be time for cyber risk to be treated in the same way as financial risk, with pentest reports being written in a similar way to financial audit reports to ensure they are unambiguous and are understandable to wide audience.