Audit evidence in an IT audit: what the auditor accepts and how to organise it

Every audit opinion rests on evidence. The auditor does not conclude that a control works because someone says so, but because he has seen something that demonstrates it: a ticket, a log, an export, a recorded decision. In practice an IT audit mostly stalls on evidence: evidence that does not exist, that arrives late, or that shows something other than what the organisation thought.

What an auditor means by evidence

The assurance standards ISAE 3000 and ISAE 3402 require evidence that is sufficient and appropriate, and the way a certification body works under ISO 27001 does not differ from that in substance. Sufficient is about quantity: enough observations to support a conclusion about the whole period. Appropriate is about quality, which breaks down into relevance and reliability. Relevant means the evidence is about precisely the control being tested. A screenshot of a firewall rule says something about network segmentation and nothing about the access review.

For reliability the auditor applies a ranking. What he sees in the system himself outweighs what is sent to him. An export straight from the system outweighs a spreadsheet someone has maintained. Evidence from an independent party outweighs evidence from the person who performs the control. An original outweighs a copy or an edited document. From this follows the rule that is underestimated most often: a conversation is not evidence. Inquiry is a valid audit procedure, but for a conclusion on the operating effectiveness of a control, inquiry alone is never enough. Something tangible always has to sit next to it.

Four ways the auditor tests

An auditor uses four procedures, and which one he chooses determines what you have to provide. Inquiry: he asks the owner how the control works. Observation: he watches while someone performs the control, such as a deployment to production or the revocation of an account. Inspection: he examines documents, records and configurations. Reperformance: he carries out the control again himself, for instance by recalculating an access review against the current account list.

Inspection and reperformance produce the strongest evidence and therefore make up most of the request list. Observation is of limited use, because the auditor only sees what happens at the moment he is watching. For a twelve-month period that says little. Once you understand that, you also understand why an auditor nods along during a walkthrough and then asks for the tickets anyway.

Design, existence and operation need different evidence

The question of which evidence is needed has three answers, depending on what is being tested. For design, the auditor wants to see that the control is well constructed: the policy, the procedure, the configuration or the workflow. For existence, he wants to see that the control is actually in place at a point in time: one instance, on one date. That is what a Type I report tests. For operating effectiveness, he wants to see that the control was performed consistently throughout the period, and for that he needs a population and a sample. That is the difference between Type I and Type II, and it is why the observation period determines so much; see the observation period for an ISAE 3402 Type 2.

Automated controls work differently. A system that blocks every change without approval does so the same way every time. The auditor then tests one instance and relies for the rest on the general IT controls around the system: who can change the configuration and whether changes to it are controlled. That is why access management and change management weigh heavily in every IT audit, whichever standard is being tested.

The population matters more than the sample

This is where things go wrong most often. An organisation submits twenty approved change tickets as evidence for change management. The auditor can do nothing with them, because he does not know how many tickets the twenty were chosen from or who chose them. He first wants the complete list of all changes in the period, and draws his own sample from it.

The completeness of that list is the real subject. A list from the ticketing system is only complete if every change goes through that system. Manual changes, emergency changes and changes made by a supplier have to be in it. The auditor therefore reconciles the deployment list from the pipeline against the tickets, and the HR list of leavers against the accounts that were revoked. If that does not add up, the control cannot be demonstrated, however tidy the submitted tickets are. How that reconciliation works for changes is covered in change management in the IT audit, and the access side in access control and IAM in the audit.

The size of the sample depends on the frequency of the control and on the risk. An annual control is tested for every instance; for a daily control the auditor draws a sample spread across the whole period. Ask the auditor in advance which sample sizes he uses, so you know how much to have ready.

What a usable piece of evidence contains

A piece of evidence has to answer five questions without explanation from the organisation: which system does it come from, when was it recorded, who performed or approved the action, which scope or filter is it based on, and is it unedited.

For a screenshot that means: the system name or URL in view, the date and time, the logged-in user, and the full screen rather than a cropped fragment. For an export: the query or filter used to produce it and the number of rows, so the auditor can establish that the whole population is in it. For an approval: the name of the approver, the timestamp and the object the approval relates to, from the system itself and not as a loose email. For minutes: date, attendees, decisions and actions. A management review without recorded decisions cannot be counted by the auditor as a review.

A screenshot is not prohibited, but it is the weakest evidence on the list. If the system can produce an export, an audit log or a report, provide that instead. An auditor testing something important will also frequently ask whether he can look in the system himself. Expect that and have a read-only account ready.

Evidence by type of control

Governance controls (policy, risk assessment, management review, internal audit) produce documentary evidence: the approved document with version history, the risk assessment with date and owner, the minutes with decisions. Under ISO 27001 these are the records required by clause 7.5, and the auditor walks through them under clauses 9.1 to 10.2.

Operational controls (change management, incident management, access management, supplier review) produce transactional evidence: tickets with approval, incident records with timeline and closure, joiners and leavers with the corresponding account actions, periodic access reviews with the decisions taken. This is where the population work sits.

Technical controls (encryption, network segmentation, vulnerability scanning, backup, monitoring) produce system evidence: configuration exports, scan reports with follow-up, backup logs with a successful restore test, alert rules and the alerts that actually fired together with what was done about them. For monitoring the auditor wants to see not only that it is configured but also an alert that was followed up; see logging and monitoring in the audit.

Evidence that cannot be created after the fact

Evidence of operation has to arise at the moment the control is performed. An access review that should have taken place in April and is done in August because the auditor asks for it is evidence of a review in August. For April the control remains unperformed, and that becomes an exception in the report. Reconstruction also shows: timestamps in systems, document metadata and the sequence of events in a ticket reveal when something was really created. An auditor who discovers that looks at the rest of the file with different eyes.

The second gap that cannot be closed: retention periods shorter than the period. If your monitoring logs are purged after ninety days and the observation period covers twelve months, no evidence exists for nine of them. The auditor does not report that as an exception in the control; he reports that he was unable to test operating effectiveness for those months, and a reader of the report weighs that just as heavily. So check before the period starts that every source of evidence retains at least the period plus the time the audit takes.

Organising evidence without turning it into a project

The approach that works attaches the evidence to the control rather than to the audit. For each control you record in advance which evidence demonstrates operation, which system it comes from, how often it arises, who owns it and where it is kept. Once that is in place, collection no longer happens in the weeks before the audit but at the moment the control runs. A quarterly review then produces its own evidence on the day it is done.

Keep it in one place, under the control number, with a file name that includes the period and the source. Evidence scattered across mailboxes, shared drives and local disks costs search time at every audit, and it is the reason a thirty-line request list sometimes takes weeks. Measure how long it takes to answer a request. If the evidence for a routine control takes days of searching, that says something about the control itself, and it is a good reason not to skip the audit readiness assessment.

Treat the auditor's request list as the leading document. It states per control what he expects, with a deadline. Deliver per line, say so when something does not exist instead of sending something similar, and keep track of what is open. An audit platform that keeps requests, evidence and review together per control replaces the email thread and the Excel tracker; what the Secure Audit platform does there is on the services page. Having the evidence flow continuously out of the systems is one step further; see continuous auditing.

Where it goes wrong in practice

Four patterns keep recurring. The organisation submits its own selection instead of the population, and then has to produce the full list after all. Screenshots without date, system or user come back with a request for an export. Periodic controls whose first quarters are undocumented, because collection only started when the audit was scheduled. And evidence that sits in the owner's mailbox, so that when that person leaves or the next audit comes round, it starts again from zero.

All four can be prevented by determining per control, before the period starts, what the evidence is and where it arises. That only has to be done properly once, and it pays for itself at every subsequent audit in the time the request list takes. How this fits into the wider preparation is covered in preparing for a SOC 2 audit.