Publication as Evidence: What Auditors Actually Accept

Screenshots are not evidence

Ask most architecture teams for evidence that a control is modelled, traced to a requirement and assigned an owner, and you will get a screenshot of a diagram. It demonstrates that the information existed on the day the screenshot was taken, on the machine of whoever took it. It does not demonstrate that it was the approved position, that anyone else could see it, or that it has not changed since.

This is not auditor pedantry. The whole point of evidence is that a third party can verify it without trusting the person who produced it. A screenshot fails that test by construction.

Four things a publication has to record

1. What baseline it came from

Not "the repository" — a specific, named state. If your modelling tool supports baselines, the publication should name the one it rendered. If it does not, the generation timestamp plus a content hash is an acceptable substitute, provided the repository is not being edited during generation.

2. When it was generated and deployed

Generation and deployment are different events and can diverge — a publication that generated successfully but failed to deploy leaves the old portal in place. Record both.

3. Where it went

Which target, and therefore who could see it. "Published" without a destination does not establish that the intended audience had access.

4. What was skipped, and why

This is the one teams leave out, and it is the one that matters most.

Why the skip list is the important part

Every extraction from a real repository skips something. An element with an unsupported type. A diagram that failed to render. A relationship whose endpoint was outside the published scope. A package excluded deliberately because it contains work in progress.

If the publication does not record these, the portal makes an implicit claim it cannot support: this is everything. An auditor who finds one control missing from the portal now has to establish whether it was never modelled, was modelled and excluded, or was modelled and silently dropped by a rendering failure. Those have very different consequences, and without a record you cannot tell them apart either.

Warnings are not failures

A common overreaction is to treat any warning as a reason to block publication. In practice a mature repository always carries some: notes and boundary shapes have no names, some elements are intentionally undocumented, some relationships cross a scope boundary by design.

The useful distinction is between findings that make the output wrong and findings that make it incomplete in a known way. Unresolved references and failed renders are the first kind and should block. Unnamed visual-only shapes are the second and should be counted, reported and published.

Retention

Keep publications, not just the latest one. The cost is trivial — a portal for a repository of a few thousand elements is tens of megabytes — and the value appears the first time someone asks what the architecture looked like at the point a decision was taken.

A dated folder per publication, retained for as long as your decision records are retained, turns the portal from a current-state view into an architecture history that can be cited by date. That is a materially different asset, and it costs a naming convention.

What an auditor does with the evidence

It helps to understand the other side of the conversation. An auditor is not trying to catch you out; they are trying to reach a defensible conclusion with a limited sample and a deadline.

What makes that easy is evidence that is self-describing: the artefact says what it is, when it was produced, from what, and by which process. What makes it hard is having to reconstruct provenance through interviews.

Evidence has to outlive the tooling

A consideration that rarely comes up until it matters: retention periods for regulated evidence are often longer than the life of the tool that produced it.

This is an argument for the published artefact being plain files. A folder of HTML opens in eight years. A proprietary repository format, or a portal that needs a licensed server to render, may not — and the obligation to produce the evidence survives the licence.

The publication report as an artefact

The portal is what people look at. The publication report — the record of what the pipeline did on a given run — is what makes the portal evidential, and it is usually treated as build output to be discarded.

Kept, it answers questions the portal cannot. Which source revision this came from. What was excluded and by which rule. What failed to render and was therefore absent rather than nonexistent. How long it took, which matters only when someone asks whether a run completed.

The property that makes it useful is that it is machine-generated and unedited. A hand-written statement that the portal reflects revision 88 is an assertion; a pipeline-generated report saying so, produced at the time, is a record. Auditors distinguish between these more sharply than architects expect.

Reproducibility, and how far it needs to go

A strong form of evidence is being able to regenerate the exact portal that was published on a given date. It sounds demanding and it is mostly a matter of recording inputs rather than outputs.

What has to be pinned: the source revision, the configuration that controlled scoping and branding, and the version of the publishing tool. Pin all three and a regeneration produces the same output. Pin two and it produces something similar, which is worse than not claiming reproducibility at all.

The tool version is the one that gets forgotten, and it is the one that changes the output most visibly. A rendering improvement shipped in March means the portal regenerated today does not match the one published in February, and explaining that to someone comparing them is an unwelcome conversation.

For most estates the pragmatic position is to keep the published output rather than promise to reproduce it. Static files are cheap, and an archived portal is stronger evidence than a claim that one could be recreated.

What auditors ask about published architecture

The questions are more consistent than the variety of audits would suggest, and knowing them shapes what the publication should record.

• Who could see this? Not who did — who was able to. That is a question about where the portal was deployed and what protected it, and the answer belongs in the publication record.
• Was this the current architecture at the time? Answered by the source revision and its date, which is why both belong on the page rather than in a build log.
• What is missing from it? The exclusion rules. An auditor who discovers an omission you did not declare treats everything else as suspect.
• How would we know if it were wrong? The validation that ran before publication, and what it checked. This is the question that separates a generated portal from a curated one, and generation wins it comfortably.

Evidence for the negative case

Sometimes the claim under examination is that nothing changed — that the architecture in force during a period was stable, or that no unapproved modification reached a published state.

A sequence of publications each carrying its source revision supports this directly: identical revision identifiers across a period is a stronger statement than any narrative. Where the revision did change, the baseline chain shows whether each change passed through an approval.

What undermines it is a gap. A period with no publications at all is ambiguous — either nothing changed or the pipeline was broken — and the two are indistinguishable after the fact. Publishing on a fixed cadence even when nothing changed removes the ambiguity, and the run costs minutes.

Where the portal is deployed is part of the evidence

A publication that lands on an open network share is evidence of communication and also evidence of an access control failure, and which of the two an auditor writes down depends on what was intended.

The deployment target therefore belongs in the publication record alongside the source revision: where this went, and what controlled access to it. A portal published to an authenticated internal site is a different artefact from the same content on a file share, and six months later nobody remembers which was which.

This is also the check that catches the most common real mistake. A pipeline configured once and running unattended will happily keep publishing to a target that was reorganised, made world-readable, or moved. Recording the target on every run turns that into something a person could notice.

Signing the output

For estates where the publication genuinely is the evidence, signing it is worth considering, and it is simpler than it sounds because the output is a directory of files.

A manifest listing every file with its hash, signed once, gives anyone holding a copy the ability to verify that it is the publication that was produced and not a modified version. That is the property that makes an archived portal hold up when it is produced years later by someone who was not there.

The practical caveat is key management, which is where these schemes usually die. A signature verifiable only with a key held by the person who left in 2027 is not verifiable. If the organisation has a code-signing arrangement, use it; if it does not, a hashed manifest without a signature is still substantially better than nothing and has no operational tail.

Keeping the archive navigable

Retaining every publication produces, after three years, thirty-six directories with similar names and no index. The retention was the easy part; finding the March 2027 publication when it is asked for is the part that fails.

A single index file listing each publication with its date, source revision, scope and deployment target solves this, and it should be generated by the pipeline rather than maintained. It is the same information already in each publication report, collected into one place that a person can read.

The test is whether someone who has never seen the archive can find the publication that was current on a given date within a minute. If they cannot, the archive is storage rather than evidence.

Who signs off that the publication happened

A pipeline-generated record is strong evidence that a process ran. It is not evidence that anyone accountable looked at the result, and that is a distinction auditors draw.

The lightweight closure is a named person confirming, on a cadence, that the publication for that period was produced and reviewed. Not a signature on every run — that becomes a rubber stamp within two months — but a quarterly confirmation that the schedule operated, the failures were investigated, and the scope is still correct.

Where this earns its keep is the failure case. A quarter with two failed runs and a confirmation explaining what happened reads as a control operating. The same two failures with no record reads as a control that was not being watched, which is a finding about governance rather than about the pipeline.

Rehearse it with a friendly auditor

The final habit that separates practices whose evidence works from practices whose evidence exists: rehearsal. Once a year, before anyone official is due, ask a colleague from internal audit or the second line to spend an hour playing the part — pick a date from last year, pick a model, and ask the standard questions cold. What was published then? Who approved it? Show me the skip list. Prove this file is the one that was served. The exercise costs an afternoon including the preparation it inevitably shames someone into doing, and it finds the gaps while they are cheap: the retention job that silently stopped in February, the manifest field nobody populates, the archive folder whose naming convention changed mid-year and broke the lookup.

Two refinements make the rehearsal earn more. Have the person who would actually answer in a real audit do the answering — not the pipeline's author, who knows where everything is, but the practice lead who will be in the room. And time it: the difference between finding March in four minutes and finding it in four hours is the entire value proposition of everything this article describes, and the stopwatch makes that value legible to whoever funds the archive storage. Practices that run this drill report the same arc — awkward the first year, boring the second — and boring, in evidence production as in everything else operational, is the sound of the system working.

Taken together, these practices amount to a modest claim with an immodest payoff: that an architecture practice can treat evidence as a by-product of working normally, rather than as a genre of document produced under duress. The pipeline publishes anyway; making its publications citable costs a manifest, a retention job and an annual rehearsal. Auditors notice the difference immediately — not because the evidence is beautiful, but because it is boring, complete and fast to produce, which is what evidence from a functioning control looks like. The practices that struggle in audits are rarely the ones with imperfect architecture; they are the ones reconstructing their own past from mailboxes, and that is the fate this entire discipline exists to avoid.

Evidence, in the end, is a courtesy to the people who come after — including the version of you who no longer remembers. Extend it while remembering is cheap.