Control frameworks
Agentic AI under the EU AI Act: a control framework
An AI agent does not only answer: it calls tools, moves money and changes records. The EU AI Act applies to it as to any AI system, and the controls that count are the ones that show what the agent does, not what its documents say. This framework sets out eight such controls, the evidence each produces, the article it serves and how to test it.
A chatbot's worst output is a bad sentence. An agent's worst output is a bad action: a payment made, a record changed, a message sent, a sub-agent started with rights nobody reviewed. The controls most teams built for generative AI (output filters, a usage policy, a review of the model vendor's documentation) do not reach that far.
Why agents need different controls
- Tools. What an agent can do is set by its tools: the APIs, databases and channels it can call. A review that reads transcripts but never the tool calls checks the narration, not the behaviour.
- Injected instructions. Anything the agent reads (a document, a web page, another agent's message) can carry an instruction. No review of the system prompt clears that risk.
- Delegation. An agent that hands work to sub-agents passes on its permissions and its errors. The unit of review is the whole chain, not one model.
- Variance. An agent can answer the same request differently each time, and each step of a multi-step task adds variance. One test proves nothing: send each test ten times and read the pass rate, as how we test explains. A model update or a new tool can change behaviour without any release, so a test from March says little about June.
What the AI Act asks of an agent
For a high-risk system, the articles that bite hardest on an agent are those about behaviour. Article 9 frames risk management as a process that never stops.
The risk management system shall be understood as a continuous iterative process planned and run throughout the entire lifecycle of a high-risk AI system, requiring regular systematic review and updating... (Article 9(2))
For an agent, the lifecycle includes every change of model, prompt or tools. The same article makes testing a duty, not an option.
High-risk AI systems shall be tested for the purpose of identifying the most appropriate and targeted risk management measures. Testing shall ensure that high-risk AI systems perform consistently for their intended purpose and that they are in compliance with the requirements set out in this Section. (Article 9(6))
Article 12 asks for logs.
High-risk AI systems shall technically allow for the automatic recording of events (logs) over the lifetime of the system. (Article 12(1))
For an agent, the record of its tool calls is that log. A transcript alone does not show what the agent did. Article 13 asks the provider to tell deployers what the system can and cannot do, which for an agent means a declared tool list and known failure modes. Article 14 ties oversight to autonomy.
The oversight measures shall be commensurate with the risks, level of autonomy and context of use of the high-risk AI system... (Article 14(3))
Among the things the person overseeing the system must be able to do, Article 14 lists stopping it.
to intervene in the operation of the high-risk AI system or interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state. (Article 14(4)(e))
Article 15(5) asks for resilience against attempts to alter the system's outputs by exploiting its weaknesses, which for an agent includes instructions hidden in what it reads. AI Act Article 14 and human oversight covers the oversight duties in detail.
These high-risk duties apply from 2 December 2027 for Annex III systems, under the Digital Omnibus. Other duties do not wait: Article 50 has applied to many agents, whatever their class, since 2 August 2026, and the Article 5 prohibitions since 2 February 2025.
Oversight that survives autonomy
The question Article 14 raises for an agent is not "is a person in the loop?" but "which actions can complete without one?". A pattern that works:
- Sort the agent's actions by consequence: reversible and internal, reversible and external, irreversible or financially material.
- Put the last group behind an approval the agent cannot give itself: a separate channel, a separate identity, a logged decision.
- Test the gate by behaviour. Push the agent past it with an authority claim, urgency or chained instructions, and confirm in the recorded tool calls that the gated tool was never called.
The third step is the one most programmes skip, and the one that turns an oversight design into oversight evidence.
Tool governance
Treat the tool list like a firewall's rules:
- Declare each tool and its scope at registration, as a commitment rather than a description.
- Constrain scopes to the minimum: read-only where possible, capped amounts, approved recipients.
- Observe the actual tool calls and compare them with the declaration. A call to an undeclared tool is a finding, even when the reply looked fine.
- Re-test on change. A new tool changes both the permissions and the behaviour, so it should trigger new tests.
The control set
A rate needs reading with the stakes in mind. For a gate on an irreversible action, one payment in ten attempts is one payment too many, so read the failed attempts, not only the rate.
Where Vidimus fits
Vidimus tests these controls, one agent at a time. The registration declares each tool and its scope. Tests are written from the regulation text and sent to the live agent through the interface it already exposes: a plain HTTP request, the A2A or MCP agent protocols, a streamed reply, Microsoft Direct Line, an OpenAI-compatible API or a webhook, as integrations lists. Tool calls are read from the reply, the tool log or the event trace, and an agent that directs sub-agents can send their traces too.
Each test is sent ten times. A separate judge model grades every reply and quotes it, and a reply that tries to steer its own grading fails before the judge reads it. Your reviewer, the person who decides, records the decision. The result is a signed evidence pack that cites every obligation, including Annex IV's documentation duties, which our Annex IV guide sets out section by section. Writing the tests themselves is covered in how to test an AI agent against the EU AI Act. Vidimus is not a notified body: the pack documents behaviour but does not certify it.
To see which obligations your agent faces, run the AI Act risk classifier.
Quick answers
Does the AI Act mention AI agents?
Not by name. An agent is an AI system as Article 3 defines it, so the Act applies as written: Articles 5 and 50 whatever its class, and the high-risk chapter when its use falls under Annex III.
Is a human in the loop enough for Article 14?
Only if the agent actually routes to that person. An approval step the agent can go around is a diagram, not a control. Test that the gated tool is never called without the approval.
Why send each test ten times?
An agent’s replies vary, so one reply proves nothing. Ten give a pass rate you can compare from one version to the next. Vidimus sends each test ten times and grades every reply.
What counts as evidence for an agent?
The recorded tool calls, with their arguments, beside the reply. A reply that says it referred a file is not a referral. The call to the referral tool is.
Which interfaces can Vidimus test?
HTTP, A2A, MCP, streaming, Direct Line, an OpenAI-compatible API or a webhook. For other platforms, tell us and we build the adapter.
Sources
Vidimus tests what your live agent does with its tools, ten attempts per test, and files the result in a signed evidence pack.