What an AI code review audit trail must prove — a compliance checklist

Once an AI reviewer starts posting findings and controlling a merge gate, it becomes part of your change-management control — and an auditor will eventually ask about it. For a SOC 2 Type II or ISO 27001 engagement covering AI-assisted development, the question is not "do you use AI to review code?" It is "show me the evidence that every change was reviewed, that the gate decisions were governed, and that nobody quietly bypassed the control." The answer to that question is an audit trail.

This post is a concrete checklist of what that trail has to contain and what properties it needs to survive scrutiny — the who, what, when, and why; immutability; completeness; exportability; and override accountability. It is written to be useful no matter which reviewer you run. Gatekeep is the worked example at the end, honestly labeled, but the requirements are the point.

The checklist: what the trail must prove

Each item below is something a change-management auditor can reasonably ask you to demonstrate for AI-assisted development. Treat them as pass/fail questions about whatever tool guards your merges.

1. Every review is recorded — including the findings it suppressed

The trail must show that a review happened for each change, what the reviewer flagged, and at what severity. Crucially, it must also record findings that were suppressed or filtered below a comment threshold. A record that only logs what got surfaced lets a low threshold silently hide problems — and an auditor testing for completeness will ask exactly that: "what did the tool see but not tell anyone?"

2. Every gate decision is recorded with its reason and threshold

For a blocking merge gate, the pass-or-fail decision is the control action. The trail needs the decision, the policy that produced it, and the specific input that tripped it — not just "blocked," but "blocked because max severity high met the high threshold, tripping finding X." Reasoned decisions are what let an auditor confirm the control ran as configured rather than being waved through.

3. Every override is an actor plus a written justification plus a timestamp

Overrides are where audits are won or lost. When someone forces a blocked change through, the record must name who did it, capture a written justification (not an empty click), and stamp when. An override without a named human and a reason is an unaccountable bypass, and it is the first thing a skeptical auditor probes. Bonus points if the tool refuses empty or trivially short justifications outright, so the accountability is enforced rather than requested.

4. The record is tamper-evident

Evidence you can silently edit is not evidence. The trail needs a mechanism that makes deletion or after-the-fact editing detectable — a hash chain, a signature, write-once storage, or an external SIEM copy. You need not prove the log is physically unalterable; you have to prove tampering would show. "Append-only by policy" is a start; "append-only and verifiable" is the real answer.

5. It exports in a machine-readable format

Auditors sample. They want to pull a date range, filter to overrides, and reconcile counts against your ticketing system — which means the trail has to leave the tool as structured data (JSONL, CSV, a queryable table), not a screenshot or a scrollable UI. If getting evidence out requires a support ticket or a manual copy-paste, it will not survive a Type II observation window that spans months.

6. Retention and backup are answerable

A SOC 2 Type II covers a period, often six to twelve months, and ISO 27001 expects defined retention. You must be able to say how long records are kept, where they live, and how they are backed up. "The log rotated and we lost Q1" is an audit finding. This is usually your responsibility — but the tool has to store the data somewhere durable and let you own that store.

7. Coverage is complete — no PR bypasses the record

The single most important property, and the easiest to fail: the control must cover every merge path. If a repo isn't gated, if admins can merge without the check, or if there's a side door that skips review, then your trail is a record of the changes that happened to go through the front. Completeness is proven by branch protection that requires the check on every protected branch, plus a log that shows a review for every merged change. An auditor will look for the gap.

8. Human accountability is mappable

An AI approving its own findings is not accountability. The trail must let you map decisions back to people: which humans can override, who is on the allowlist, and — because the reviewer acts under an identity — that the bot's actions are attributable to a distinct, non-human account rather than blended into a developer's activity. Auditors think in terms of "who is accountable for this control operating," and the log has to answer in names.

9. Timestamps and identifiers are trustworthy and precise

Every record needs a reliable timestamp and enough identity to reconstruct the event: repository, pull request number, the exact commit SHA under review, and the actor. Records that can't be tied to a specific commit undermine the reconstruction an auditor is doing. Monotonic sequencing helps too — it proves you're not missing rows in the middle.

10. The trail is generated automatically, as a byproduct

Evidence assembled by hand the week before an audit is both expensive and suspect. The trail should be produced by the control itself, continuously, so the record is a natural consequence of shipping. This is a test of the other nine: if any require manual bookkeeping, they will decay.

Gatekeep as the worked example

Here is how those requirements map onto one reviewer, so you can see what "passing the checklist" looks like concretely. Gatekeep is a self-hostable, governance-first reviewer whose runtime is named mergegate; the audit behavior below is documented in full at /docs#audit.

An append-only, hash-chained log

Every governance-relevant event is written to an append-only store: PR reviewed, findings and severities, gate decision with the threshold used, override requested, override granted or denied. Each record carries a monotonic sequence id, timestamp, actor, repo, PR number, head SHA, event type, and payload — plus a hash chained to the prior record's prev_hash. That chain is the tamper-evidence mechanism from item 4: deleting or editing any row breaks the chain, and the break is detectable on export. The MVP store is a SQLite table with an application-enforced append-only contract — no UPDATE or DELETE paths in code — mirrored to a JSONL file for streaming to a SIEM.

This is what the chain looks like in the offline demo — five events, each hash linked to the one before it, ending in a verification pass:

seq= 1 pr_reviewed      actor=mergegate    hash=c97cc2ffe526 prev=000000000000
seq= 2 gate_decision    actor=mergegate    hash=03b379d6ab5c prev=c97cc2ffe526
seq= 3 override_denied  actor=random-dev   hash=d4e39b813765 prev=03b379d6ab5c
seq= 4 override_denied  actor=demo-lead    hash=d0fa5d77ee1a prev=d4e39b813765
seq= 5 override_granted actor=demo-lead    hash=d1dc99b5782b prev=d0fa5d77ee1a

chain verify: OK — unbroken

Read the sequence against the checklist. The review is recorded (seq 1), including findings suppressed below the comment threshold, which are still audited even though they aren't posted (item 1). The gate decision carries its threshold and the tripping finding (item 2). Overrides are governed: two attempts are denied — one from an unauthorized user, one with a justification too short — before an authorized lead overrides with a written reason recorded verbatim (item 3). None of it is reconstructed after the fact; it is the log the tool wrote as it happened (item 10).

Export as machine-readable JSONL, with a verification header

You fetch the full log as hash-chained JSONL from an authenticated export endpoint — the bearer token is the value of the env var named by audit.export_token_env (default AUDIT_EXPORT_TOKEN):

curl -H "Authorization: Bearer $AUDIT_EXPORT_TOKEN" localhost:8080/audit/export

The response body is JSONL (item 5), and the X-Audit-Chain-Valid header confirms the chain verifies end to end — so the auditor doesn't have to take the export on faith. Because Gatekeep is self-hosted, the durable store (/data, SQLite plus JSONL, optionally Postgres) lives on infrastructure you own, which is what makes retention and backup your answerable responsibility rather than a vendor's black box (item 6).

Where the checklist meets reality — and the roadmap

Two honest caveats, because the point of a checklist is to expose gaps. Coverage completeness (item 7) is not something the reviewer can grant on its own: it depends on you requiring the mergegate/review check in branch protection on every protected branch and not leaving admin bypass open. The tool provides the check; the completeness of the control is a configuration you own and an auditor will test.

And on tamper-evidence, the hash chain proves internal consistency — it detects edits and deletions within the log. Stronger guarantees are on the roadmap, not shipped today: full cryptographic signing of records and WORM (write-once-read-many) storage are a Phase 2 compliance-pack upgrade. Roadmap If your control framework requires signed or immutable-media evidence specifically, treat that as future work rather than a claim we're making now.

Frequently asked questions

Does an audit trail make us SOC 2 compliant?

No. A certification is an assessment of your whole control environment by an accredited auditor, and your process, access controls and people are most of it. What a good trail does is make the code review control auditable, turning "trust us, we review everything" into a record you can hand over. No tool grants compliance, and any vendor implying otherwise is overselling.

Are pull request comments enough?

Rarely. Comments can be edited or deleted, they live in a git host's activity data, and they were never designed as a retained record. If your answer to "show me this change was reviewed" is a link to a comment thread, you are relying on a third party's retention policy and an artifact with no integrity guarantee.

How long should we retain audit records?

Long enough for the framework you are assessed against, which is usually longer than your tooling defaults. HIPAA requires six years of documentation retention under §164.316, while default log retention in most development tools is 30 to 365 days. See HIPAA and AI code review for that gap in detail.

What should an override record contain?

Three things at minimum: a named actor, a timestamp, and a written justification. An override with an empty reason is an unaccountable bypass and is the first thing a sceptical assessor will pull on. Tooling that refuses trivially short justifications enforces the accountability rather than merely requesting it.

What actually makes a log tamper-evident?

A mechanism that makes deletion or after-the-fact editing detectable: a hash chain, cryptographic signing, write-once storage, or an external copy in a system the same people cannot edit. You do not have to prove the log is physically unalterable. You have to prove that tampering would show.

The short version

When an AI reviews or approves your code, the auditor's real question is whether the control operated on every change and whether its decisions were governed and accountable. A trail that passes proves ten things: every review recorded including suppressions; every gate decision reasoned; every override tied to a named actor, a written justification, and a timestamp; tamper-evidence; machine-readable export; answerable retention; complete coverage with no bypass; mappable human accountability; trustworthy timestamps; and automatic generation. Gatekeep's hash-chained JSONL, its /audit/export endpoint, its X-Audit-Chain-Valid header, and its override governance hit most of those — with signing and WORM labeled roadmap and coverage completeness left, correctly, in your hands. The tool doesn't make you compliant. It makes the review control something you can prove.