Skip to main content

Evidence and documentation

Published and reviewed

AI evidence packs: what goes in one and how to verify it

An AI evidence pack is the file that shows which rules apply to an AI system, how it behaved when tested and who decided, in a form a stranger can check. It answers the eight questions every auditor asks, and it comes with a hash and a signature, recorded outside the file, so the reader knows it has not changed since it was issued.

The EU AI Act, Regulation (EU) 2024/1689, does not use the term. It asks for technical documentation (Article 11 and Annex IV), automatic logs (Article 12) and, on the deployer side, monitoring and log keeping (Article 26). An evidence pack is where those duties meet. This primer is for the person who has to produce the file: the head of AI governance or the compliance officer.

The pack at a glance

The pack at a glance
What the auditor checksThe question behind itWhat answers it in the pack
Legal classificationWhich rules apply, and why?The classification, its reasoning and who decided
Applicable obligationsWhat exactly does the law ask?Each obligation quoted verbatim, with its date of application
Tests and resultsWhat was it tested against, and what did it do?Each test, the input sent, every reply, the tool calls and the grade
DecisionsWho decided what, and why?Each judgement with a name, a role, a date and a reason
DocumentsWhich documents were relied on?Cited passages, each pinned to one version of its document
VersionsWhich system does this describe?The model, prompt, tools, test plan and judge versions
Audit trailHas anything changed since?A record of every change that can be added to but never edited
DatesWhen was each of these true?The date of every test, decision and issue, and of the next review

Documentation versus evidence

Documentation says what a system is meant to do. Evidence shows what it did. The regulation keeps the two apart, starting with Article 11.

The technical documentation of a high-risk AI system shall be drawn up before that system is placed on the market or put into service and shall be kept up-to date. That technical documentation shall be drawn up in such a way as to demonstrate that the high-risk AI system complies with the requirements set out in this Section ... (Article 11(1))

The verb is demonstrate, not describe. Article 12 then asks for records.

High-risk AI systems shall technically allow for the automatic recording of events (logs) over the lifetime of the system. (Article 12(1))

A policy that says a person reviews every hiring decision is documentation. A log entry showing that a named reviewer overrode the agent, at a given time, on a given version, is evidence. The first can be written the week before an inspection, but the second cannot. Deployers keep those logs too, for at least six months under Article 26(6): see the deployer duties under Article 26.

The eight things an auditor looks for

Show the reasoning, not only the label. Is the system in an Annex III area, and under which point? Are you its provider or its deployer? Does Article 50 apply? Record the points considered and rejected, and who decided. A classification without reasoning cannot be checked, so the reader will redo it.

2. The applicable obligations quoted verbatim

Quote the article itself, with its date of application. Use the consolidated text in force since 27 July 2026, after the Digital Omnibus, Regulation (EU) 2026/1744. Paraphrases drift, but a quotation lets the reader check each duty against its test. Say how each obligation was checked: on the system, or in your documents.

3. What the system was tested against and what it did

Keep every input, every reply, every tool call and every grade. The test plan names each test, the obligation behind it and the exact input sent. The result holds the replies, the tool calls with their arguments, and the grade. Article 9(8) asks for testing against prior defined metrics and probabilistic thresholds, and Annex IV asks for the logs.

the validation and testing procedures used, including information about the validation and testing data used and their main characteristics; metrics used to measure accuracy, robustness and compliance with other relevant requirements set out in Chapter III, Section 2, as well as potentially discriminatory impacts; test logs and all test reports dated and signed by the responsible persons ... (Annex IV, point 2(g))

An agent can answer the same question differently each time, so one reply proves nothing. Send each test ten times and report the share of replies that passed: a pass rate, against thresholds set in advance, as how we test explains. For an agent, the tool calls matter most. "I have passed your file to an adviser" is worth nothing unless the log shows the referral call. How to test an AI agent against the EU AI Act covers the method.

4. Who decided what and why

Give every judgement a name, a role, a date and a reason. That covers a grade overridden, a test marked not applicable, a finding accepted and an approval given. Article 14(4)(d) expects overseers to be able to override the system's output. Where a reviewer disagreed with the judge (the model that grades each reply), the pack says so and says why.

5. The documents relied on and the passages cited

Cite the passage and the version, not the file. "The oversight policy" is not a citation. The paragraph that names the reviewers, in the version current on the day of the test, with that version's hash, is.

6. The versions of everything

A result without versions describes a system that no longer exists. Record the model, the system prompt, the tools and their permissions, and the documents it looks things up in. Then record the test plan and the judge, with the version of its instructions. Annex IV opens with this.

its intended purpose, the name of the provider and the version of the system reflecting its relation to previous versions (Annex IV, point 1(a))

7. The audit trail of changes

Keep a record of who changed what, and when, that nobody can edit or delete. Annex IV, point 6, asks for the relevant changes made through the system's lifecycle. The audit trail shows the reader that the pack in front of them is the one that was approved.

8. The dates

Date every test, every decision, every issue and the next review. Dates make the other seven checkable. Article 73(2) gives a provider at most 15 days, after becoming aware of a serious incident, to report it. You cannot meet a deadline you cannot date.

Hashes and signatures: how a reader checks the file

A hash lets anyone confirm the file is unchanged. A signature checked against a published key confirms who issued it. A pack needs both.

SHA-256, the standard hash function, turns the exact bytes of a file into a fixed-length value. The same bytes always give the same value, and one changed character gives a different one. Anyone holding the file can recompute it and compare. A hash proves integrity, not truth.

Anyone can hash a forged file, so a signature adds the issuer. With Ed25519, a standard signature algorithm, the issuer signs the hash with a private key and publishes the public key at a stable address outside the pack. The reader checks the signature without contacting the issuer.

Vidimus packs can be checked offline. Each pack has a line in your evidence register, the list of packs issued, and that line carries the hash, the key id and the signature. Vidimus publishes its public keys, retired ones included, at /.well-known/vidimus-signing-keys.json, and prints a content fingerprint on every page. The steps and commands are on the evidence pack page, with a specimen pack to try them on, and in the API documentation.

A valid signature does not show that the system complies or that the tests were the right ones. It answers the question that comes first: is this the file that was approved?

When to reissue an evidence pack

A pack is a photograph of a system on a date. Article 11(1) asks for documentation kept up to date, and Article 9(2) calls risk management a continuous iterative process. Six events should trigger a new version:

  • the model is swapped, or updated by its provider
  • the system prompt changes
  • a tool is added, removed or given new permissions
  • the documents it looks things up in change
  • the law or its guidance moves (two examples: marking is due by 2 December 2026 for generative systems already on the market, and the Commission's post-market monitoring template is due by 2 September 2027)
  • a serious incident occurs

Never edit an issued version. Issue a new one with its own hash and signature, and keep the old ones. A change ticket that names no re-test is a gap the reader will find. In Vidimus, a change to the registration marks the tests out of date, and scheduled re-tests send an alert when the pass rate drops.

What a Vidimus evidence pack holds

Vidimus builds the pack from a decided review. It holds:

  • the registration and the AI Act classification
  • every applicable obligation, quoted
  • every test with its pass rate and one reply, quoted with its attempt number, grade and tool calls
  • the documents with their quoted passages
  • the reviewer's decision
  • the audit trail

Every attempt's reply stays in the run report and the per-attempt CSV export. Duties about behaviour are tested on the live agent. Duties about process are checked against your documents, passage by passage. The pack also lists what it did not test, with the reason.

Vidimus is not a notified body, and the pack is not a certificate: the decision stays with your reviewer. How the pack relates to the technical documentation is covered in Annex IV explained section by section, and the evidence the GDPR asks of an agent in GDPR for AI agents.

To see which obligations your own agent faces, start with the AI Act risk classifier.

Quick answers

Is an evidence pack a certificate?

No. No harmonised standard has been cited in the Official Journal, so there is no presumption of conformity, and Vidimus is not a notified body. A pack is evidence your team uses to make and defend its own decision.

Can an auditor verify a pack without an account?

Yes. With the PDF, its line from your evidence register (hash, key id and signature) and our published public key, the check runs offline, with no call to Vidimus.

See the steps

What is the difference between a hash and a signature?

A hash shows the file has not changed since the hash was recorded. A signature, checked against a key the issuer publishes, shows who issued the file. A pack needs both.

When should a pack be reissued?

After any change to the model, the prompt, the tools or the documents the agent looks things up in, after a serious incident, and when the law or its guidance moves. Never edit an issued version: issue a new one and keep the old.

Why is each test sent ten times?

An agent can answer the same question differently each time, so one reply proves nothing. Vidimus sends each test ten times and reports the pass rate against thresholds published in advance.

How we test

Sources

Vidimus builds this pack from tests on your live agent, ten attempts each, and signs it so your auditor can verify it.

Last reviewed