Skip to content

About this practice

Status
draft
Endorsement
none yet
Last reviewed
2026-07-26

Best practice 4

Govern agent autonomy and accountability

Practice

When you run an agent in your own work, set its limits where they take effect: what it can access, how far it can act on its own, and when it must stop and ask. Stay the decision-maker for anything that affects results or other people. A personal agent connected to your mail and files is part of the institution's risk, even if no one else set it up.

Put the limits you promise in the system, not only in the docs, so an agent cannot exceed its scope by being told to. Give operators the stop, escalation, and shutdown controls they need, log agent actions under attributable identities, and record which role holds each responsibility. Roll out new agent features in stages, with clear success and go-live criteria.

Written policy alone does not bind a system that acts. Move the rules you care about into guardrails and enforced permissions. Give each responsibility a named role with clear escalation and shutdown paths, keep a list of the agents in use, require that their actions are logged and attributable, and require independent evaluation where the risk is high. Decide what autonomy the organisation is willing to grant, and treat governance as ongoing, not a one-time policy.

Reasons

An agent acts. A policy document tells a person what to do. It does nothing to a system that only follows the actions it is allowed to take. As agents get more autonomous, the gap between written rules and enforced limits becomes the main risk. To close it, put the limits where they take effect: what the agent can access, how far it can act without a human, and when it must stop or escalate. Accountability has to be just as concrete. If responsibility sits with "the institution" in general, no one is answerable; it has to sit with named roles, with clear paths to escalate and to shut an agent down. Accountability also needs a record: if an agent's actions are not logged under an attributable identity, no one can reconstruct what it did or hold the right role answerable. Personal agents with broad access to mail, files, and calendars sit outside all of this, so governance has to cover individual use, not only institutional deployments. This is the consensus position across the main governance frameworks (NIST, the EU AI Act, OECD, ISO/IEC 42001), which agree on enforced limits, named accountability, an agent inventory, logging, and independent review.

Examples

  • A policy document says the agent must not delete production records, but nothing in the system stops it; a prompt-injectedPrompt injectionAn attack that hides instructions inside content an agent reads (a web page, a document, a tool description), so the model follows them as if they came from the user. The user never sees the injected text, which is what makes it dangerous. Tool poisoning is one form, where the hidden instructions sit in an interface's tool description. instruction gets it to do exactly that. The rule was written, not enforced.
  • The same agent runs with a scoped token that carries no delete permission and a shutdown control a named on-call engineer can trigger, so the injected instruction fails because the action was never available to it.
  • After an agent writes a wrong value to a shared dataset, the team traces it through the action ledger to the run, the model version, and the role that owned that workflow, and closes the gap.
  • Responsibility for an agent sits with "the department" in general; when it misbehaves, no one is answerable and no one has clear authority to shut it down.
  • A new agent feature goes live for low-risk cases first, with success and go-live criteria and an accepted risk limit set in advance, before any wider rollout.
  • A personal agent connected to a researcher's mail and files is entered in the institution's agent inventory and reviewed like any deployed one, because its broad access is part of the same risk.

Sources

Change history

  • 2026-07-27: Added the EU AI Omnibus (2026) as a supporting source on bp4-a1 (technical safeguards in the system, not policy; Art 5(1a) foreseeable-misuse standard).
  • 2026-07-27: Renumbered from BP03 to BP04 on inserting the new BP01 (match the method to the task).
  • 2026-07-27: Rewrote Examples as concrete scenarios (actor, action, outcome), including anti-patterns, replacing restatements of the practice.
  • 2026-07-26: Rewritten to merge action-logging and attributable agent identity into the practice, add independent grounding (NIST, EU AI Act, OECD, ISO/IEC 42001, OpenAI, Chan et al., Kolt), and add two citation caveats: the "guidelines to guardrails / Kassorla et al." attribution in the source could not be verified, and the JRC "not automatable" wording is unconfirmed.
  • 2026-07-25: Created from the HFD distillation, grounded in the EU Expert Forum entry. New practice added because the guidelines-to-guardrails point is a separate action, not a sub-case of stating a human-in-the-loop level (BP09).

Discussion