Marc Krisjanous is one of the first CCSS Auditors and assisted C4 in the development of their auditors program.
In this article we will consider the use of sampling in a CCSS audit. We will review the sampling guidance provided by the CCSS committee and provide additional commentary from our own experiences in auditing information management systems using sampling.
But first we address the need for sampling during audit.
Why Sample?
Sampling is a technique auditors can utilize when the in-scope environment (people, process and technology) is too large or complex to realistically audit all components – factoring in time allowed to audit and audit budget. The CCSS Auditor Guide defines sampling for audit as:
“Audit sampling enables the auditor to obtain and evaluate audit evidence about some characteristic of the items selected in order to form or assist in forming a conclusion concerning the population from which the sample is drawn.“
The International Standard on Auditing (UK) 530 defines sampling for audit as:
“The application of audit procedures to less than 100% of items within a population of audit relevance such that all sampling units have a chance of selection in order to provide the auditor with a reasonable basis on which to draw conclusions about the entire population.“.
ISA 530, paragraph 5 (a)
For example, a CCSSA may find while auditing an assessed entity that there are over 100 servers within the in-scope environment. Some servers host the cryptocurrency wallets while others host security controls such as authentication and authorization controls, anti-virus management controls, log management controls. The allowed time and budget of the audit is not provisioned to accommodate the CCSSA reviewing each server therefore the CCSSA can select a sample of the servers for audit.
There is more to selecting components for a sample-set than just randomly selecting 10% of the total components population and we cover this in detail below.
Sampling Guidance from the CCSS Auditor Guide
We will now review the sampling guidance provided in the CCSS Auditor Guide.
The first paragraph for section 1.2.3. Sampling within the Auditor Guide is below.
“Where a CCSSA is testing the operating effectiveness of a control over a period of time a sampling approach should be followed. For a control with a high frequency of occurrence it is not practical to test all occurrences.”
This paragraph may lead to confusion for some CCSSAs due to the wording. For example, a CCSSA may interpret the statement using the term “frequency” as capturing network packets flowing into the in-scope environment every hour (the frequency) to ensure the data is protected by encryption while in transit (e.g. cryptocurrency protocol transaction data) and not recognize that the statement can also be applied to, for example, change tickets where a sample of change tickets is selected in such a way as to have a selection of change tickets that cover the previous 12 months and cover different types of changes such as user account role privilege changes or patching devices.
Next, the guidance addresses how the size of the sample can be determined.
“The sample size can be determined by the application of a statistically based formula or through the exercise of professional judgement.
The auditor must document their rationale behind the sample size selected and consider the following factors:
- What a tolerable rate of deviation will be given the size of the population;
- What the likelihood and impact is of errors occurring given the procedure being tested;
- How critical the procedure is and the level of certainty required given its important;”
So the CCSSA will need to ensure that however the sample is selected, it can be defended based on the bullet points above.
Defining Scope is Important for an Audit
To determine a sample-set size the CCSSA will first need to identify what is in-scope. The audit would not be effective if the CCSSA selected all servers which host cryptocurrency wallets and he/she was not aware of the servers which protect the wallet’s private keys while at rest. Clearly the “key management servers” would be in-scope for CCSS as there are several CCSS Aspects defining requirements for key management.
We will address defining the in-scope environment in an upcoming article but for now we will assume the CCSSA has identified all the personnel, processes, and technology in-scope for the CCSS audit.
Sampling Techniques
The Auditor Guide defines two techniques for gathering samples: (1) statistical and (2) professional judgement.
Statistical sampling is a technique where samples are selected at random using either a tool that generates a random output of items from a list of input items or manually such as assigning a number to each item in a population and selecting every item that is xN apart from the last selected item. Sometimes even rolling dice has been used as a random number generator for an audit.
There are other approaches to selecting samples such as: systematic selection, haphazard sampling, block sampling, which we will leave to the reader to research.
Professional judgement or “non-statistical” sampling technique utilizes the auditor’s skill and experience to select samples. For example, the CCSSA may select a higher percentage of servers for the sample-set of servers hosting a cryptocurrency wallet. The CCSSA is aware that these types of servers are critical to the in-scope systems and will be a high target for attack.
The CCSSA may find that the two techniques (statistical and non-statistical) can both be utilized during an audit. A statistical random-based approach can be used to select a sample of in-scope servers to ensure patch management is in-place for all servers. For a non-statistical sample selection, the CCSSA may decide to select a larger percentage sample of change tickets related to key management processes from the total change ticket population to ensure key management is in-place. Since the CCSS is an information security management standard for cryptocurrency wallets, key management is a critical component, and more attention needs to be focused on the management of keys than other changes to the in-scope environment.
The CCSSA should identify the critical personnel, processes, and technology in-scope for the CCSS audit and select a higher sample-set for the more critical components.
Standardized Processes can Guide Sample Size
Another consideration for the CCSSA is if the assessed entity uses standardized processes to ensure a consistent output from undertaking a process.
For example, if the assessed entity utilizes build scripts such as Terraform in the provisioning of servers hosting cryptocurrency wallets to ensure consistency of the build process output, then the CCSSA can review the Terraform scripts to ensure they meet the applicable CCSS requirements and only select a small sample-set of servers to review to ensure that the servers were built using the Terraform scripts. The Terraform scripts provide a consistent and measurable output that allows for the sample-set to be small in relation to the total population.
If, however, the assessed entity allows the IT department to build servers as they see fit then the CCSSA will need to ensure the sample-set for the servers is of a high percentage of the total population due to the inconsistency of the build process.
The CCSSA will also need to consider the risk of compromise or failure, and criticality of the components when defining the sample-set size. For our example, if the assessed entity allows the IT department to build servers hosting cryptocurrency wallets as they see fit, then this increases the risk of the build process introducing vulnerabilities. Factoring in the criticality of servers hosting cryptocurrency wallets, the CCSSA may decide to audit all servers hosting cryptocurrency wallets.
Finally, the Auditor Guide section on sampling provides the following guidance below. Which provides some helpful considerations when selecting samples.
Items selected for testing
When identifying the items to be tested, the auditor can use professional judgement, random selection, or a combination of the two techniques.
When identifying items to test using professional judgement the audit should consider factors such as the following:
- Items that are likely to be subject to manipulation;
- When the items occurred;
- Who performed procedure;
- Items that are outliers in the general population, etc.
When identifying items using random selection the auditor should make use of a randomized sampling technique such as the following:
- Simple random selection (i.e. using a random number generator within the range of the population)
- Systematic random (Identifying a starting point and then selecting items at a specific interval from this point)“
Summary
The sampling guidance provided within the Auditor Guide provides helpful guidance to the CCSSA. However, the CCSSA must be aware that sampling can apply not only to technology components of the in-scope environment but also the personnel and processes that are in-scope for CCSS compliance.
When selecting a sample-set the CCSSA must consider the following:
- The importance of identifying all personnel, processes, and technology that are in-scope. The CCSSA cannot consider sampling unless they know the different types of in-scope components and the total population size of each type.
- The different techniques to sampling (statistical, non-statistical etc..) and which is best suited for a component.
- If the assessed entity utilizes standardization in their processes to ensure consistency of output, then the sample-set size can be smaller than if the assessed entity provides no ability for consistently of process output.
- The criticality of a component and the risk to the security posture of the information systems if the component is compromised.
- Point 3 and 4 above assist in defining the sample-set percentage size of the total population.
