What is an AI evidence pack, and what should be in it
An AI evidence pack is the file you hand to someone who does not yet trust you: an internal auditor, a supervisor, a market surveillance authority, a notified body or a client's procurement team. They want to know what the system is, which rules apply to it, what it was tested against, what it did, who signed off and whether anything has changed since. An EU AI Act evidence pack collects those answers in one versioned bundle that a stranger can verify. The EU AI Act, Regulation (EU) 2024/1689, does not use the term. It asks for technical documentation (Article 11 and Annex IV), record-keeping (Article 12), post-market monitoring (Article 72) and, on the deployer side, monitoring and logs (Article 26). AI compliance evidence is what those articles have in common once the headings are removed. This article is for the head of AI governance or the compliance officer who has to produce the file. The Article 50 transparency duties have applied since 2 August 2026. The high-risk duties for Annex III systems apply from 2 December 2027.
Documentation versus evidence
Documentation says what a system is meant to do. Evidence shows what it did. The regulation keeps the two apart. Article 11 covers documentation.
The technical documentation of a high-risk AI system shall be drawn up before that system is placed on the market or put into service and shall be kept up-to date. That technical documentation shall be drawn up in such a way as to demonstrate that the high-risk AI system complies with the requirements set out in this Section and to provide national competent authorities and notified bodies with the necessary information in a clear and comprehensive form to assess the compliance of the AI system with those requirements. (Article 11(1))
The verb is demonstrate, not describe. Article 12 then covers records.
High-risk AI systems shall technically allow for the automatic recording of events (logs) over the lifetime of the system. In order to ensure a level of traceability of the functioning of a high-risk AI system that is appropriate to the intended purpose of the system, logging capabilities shall enable the recording of events relevant for: (a) identifying situations that may result in the high-risk AI system presenting a risk within the meaning of Article 79(1) or in a substantial modification; (b) facilitating the post-market monitoring referred to in Article 72; and (c) monitoring the operation of high-risk AI systems referred to in Article 26(5). (Article 12(1) and (2))
A policy that says a human reviews every credit decision is documentation. A log entry showing that a named reviewer overrode the agent's recommendation, at a given time, on a given version, is evidence. The first can be written the week before an inspection. The second cannot. Deployers sit in the same position: Article 26(6) requires them to keep the logs under their control for at least six months, and Article 26(5) requires them to monitor the system on the basis of the instructions for use. An evidence pack is where documentation and records meet. Each claim in the documentation points to the record that supports it. That is all AI compliance evidence is.
The eight things an auditor looks for
Whatever the reader's title, they check the same eight things.
1. The legal classification and its reasoning
Which regime applies and why. Is the system in an Annex III area, and under which point? Is your organisation the provider or the deployer under Article 3(3) and (4)? Does Article 50 apply? Show the reasoning, not only the label: the intended purpose considered, the Annex III points considered and rejected, and who took the decision. A classification without reasoning cannot be checked, so the reader will redo it.
2. The applicable obligations, with the regulation quoted verbatim
For each obligation, the text of the article itself, taken from the consolidated version in force since 27 July 2026 after the Digital Omnibus, Regulation (EU) 2026/1744, with its date of application. Paraphrases drift. A verbatim quotation lets the reader check the mapping between duty and test without opening the Official Journal.
3. What the system was tested against and what it did
The test plan: each test, the obligation it was written from and the exact input sent. The result: the answer the system gave, the tool calls it made with their arguments and returns, and the grade with the rule it was given under. Article 9(8) says testing is carried out against prior defined metrics and thresholds. Annex IV asks for more than a summary.
the validation and testing procedures used, including information about the validation and testing data used and their main characteristics; metrics used to measure accuracy, robustness and compliance with other relevant requirements set out in Chapter III, Section 2, as well as potentially discriminatory impacts; test logs and all test reports dated and signed by the responsible persons ... (Annex IV, point 2(g))
For an agent, the tool calls are the evidence that matters most. An answer that says the case has been escalated to a human is worth nothing unless the log shows the escalation call was made.
4. Who decided what and why
Every judgement in the pack: a grade that was overridden, a test marked not applicable, a finding accepted, an approval given. With the person, the role, the date and the reason. Article 14(4)(d) expects the humans overseeing a high-risk system to be able to disregard, override or reverse its output. Where a reviewer disagreed with the grading model, the pack says so and says why.
5. The documents relied on, passage by passage
The instructions for use, the DPIA, the FRIA, the oversight policy, the vendor contract. Not as attachments, but as cited passages, each pinned to a specific version of the document. A reference to the oversight policy is not a citation. A reference to the paragraph that names the reviewers, in the version current on the day of the test, with the hash of that version, is.
6. The versions of everything
The system version: model, system prompt, tools and their permissions, retrieval corpus, guardrails. The test plan version. The corpus version. The grading model version. Annex IV opens with this.
its intended purpose, the name of the provider and the version of the system reflecting its relation to previous versions (Annex IV, point 1(a))
A result without versions is a result about a system that no longer exists.
7. The audit trail of changes
An append-only record of every change to the pack and to the artefacts it cites: who, what, when. Annex IV, point 6 asks for a description of relevant changes made by the provider to the system through its lifecycle. Article 3(23) treats a change that affects compliance or alters the intended purpose as a substantial modification, and Article 12(2) wants the logs to be able to identify one. The audit trail is what lets the reader see that the pack in front of them is the same one that was approved.
8. The dates
When each test ran, when each decision was taken, when the pack was issued, which version of the regulation it was assessed against, and when the next review is due. Dates make the other seven things checkable. Article 73(2) gives a provider at most 15 days to report a serious incident after becoming aware of it. You cannot meet a deadline you cannot date.
Why hashes and signatures matter
The eight things above are only useful if the reader can be sure the pack has not been edited since it was issued. Two standard tools give that certainty without asking the reader to trust anyone.
The first is a content hash. SHA-256 is a function that takes the exact bytes of a file and produces a fixed-length fingerprint. The same bytes always produce the same fingerprint. Change one character and the fingerprint changes completely. A pack therefore records the SHA-256 of its contents. Anyone holding the file can recompute the fingerprint and compare it with the recorded one. If they match, the file is the one the pack refers to. If they do not, something changed. A hash proves integrity, not truth.
The second is a signature. A hash shows the content is unchanged, but anyone can compute a hash of anything, including a forged pack. A digital signature adds the issuer. With Ed25519, the issuer holds a private key and uses it to sign the pack's fingerprint. The matching public key is published. Anyone holding the pack, the signature and the public key can check that the signature was made with the private key that matches the public key, over exactly this fingerprint. They do not need to contact the issuer or trust whoever sent them the pack. The public key must be published outside the pack, at a stable address, because a forger who could replace both the pack and the key would defeat the check.
Together, the hash pins the content and the signature pins the issuer at the moment of issue. Each version of a pack gets its own hash and its own signature, so an earlier version remains verifiable as what it was. An AI Act evidence pack that cannot be verified this way is a claim, not evidence.
The limits are worth stating. A valid signature shows that the issuer vouched for exactly this content. It does not show that the system complies, that the tests were the right ones or that the reviewer was competent. Those are questions for the other seven items. The signature answers the question that comes first: is this the file that was approved?
How to keep it current
A pack is a photograph of a system on a date. Systems change. Article 11(1) says the documentation is kept up to date. Article 9(2) describes risk management as a continuous iterative process across the lifecycle, requiring regular review and updating. Article 72(2) requires the provider's post-market monitoring to collect, document and analyse performance data throughout the system's lifetime. Article 26(5) requires the deployer to monitor. Article 27(2) requires the FRIA to be updated when any of its elements change.
Six events should trigger a new version. The model is swapped or updated by its provider. The system prompt changes. A tool is added, removed or given new permissions. The retrieval corpus is updated. The regulation or its guidance moves: the Commission Guidelines on Article 50 were published on 20 July 2026, the grace period for machine-readable marking ends on 2 December 2026, and the Commission is to publish a template for the post-market monitoring plan by 2 September 2027. A serious incident occurs.
Never edit an issued version. Issue a new one, with its own hash and signature, and keep the old ones. The audit trail links them, and the difference between two versions is itself evidence of what changed and what was re-tested. Tie the pack to your change process: a change ticket that does not reference a re-test is a gap the reader will find. For the deployer's side of this, see the deployer duties under Article 26.
Vidimus issues packs built this way. It turns each applicable article into tests, runs them against the live agent, records the tool calls the agent makes, grades every answer with a separate grading model, has a human reviewer confirm the result, and issues a versioned evidence pack that records its SHA-256 and carries an Ed25519 signature verifiable against its published public keys. It is not a notified body, does not certify compliance and gives no legal advice. The Article 50 duties are the first ones most customer-facing agents need evidence for. See how a pack for those duties is put together on the Article 50 solution page.