Skip to content

About this practice

Status
draft
Endorsement
none yet
Last reviewed
2026-09-16

Best practice 4

Govern agent autonomy and accountability

Practice

When you run an agent in your own work, set its limits where they take effect: what it can access, how far it can act on its own, how much it may consume, and when it must stop and ask. Stay the decision-maker for anything that affects results or other people. A personal agent connected to your mail and files is part of the institution's risk, even if no one else set it up. A group that runs its own server, database, or storage is also its provider. Name who owns the machine, separate from whoever runs the agent. Before an agent gets write or delete access to shared storage, make sure a backup exists and that the agent's account cannot reach other people's data.

Put the limits you promise in the system, not only in the docs, so an agent cannot exceed its scope by being told to. Give operators the stop, escalation, spend-cap, and shutdown controls they need, log agent actions under attributable identities, and record which role holds each responsibility. Roll out new agent features in stages, with clear success and go-live criteria.

Written policy alone does not bind a system that acts. Move the rules you care about into guardrails and enforced permissions. Give each responsibility a named role with clear escalation and shutdown paths, keep a list of the agents in use, require that their actions are logged and attributable, and require independent evaluation where the risk is high. A budget an agent can exceed is not a limit; the cap belongs in the system that meters the spending. Decide what autonomy the organisation is willing to grant, and treat governance as ongoing, not a one-time policy.

Reasons

An agent acts. A policy document tells a person what to do. It does nothing to a system that only follows the actions it is allowed to take. As agents get more autonomous, the gap between written rules and enforced limits becomes the main risk. To close it, put the limits where they take effect: what the agent can access, how far it can act without a human, how much it may consume, and when it must stop or escalate. Consumption belongs in that list because an agent that loops does not crash. It keeps working, and it keeps billing, so an unmetered run can exhaust a budget while staying inside every permission it was granted. Accountability has to be just as concrete. If responsibility sits with "the institution" in general, no one is answerable; it has to sit with named roles, with clear paths to escalate and to shut an agent down. Accountability also needs a record: if an agent's actions are not logged under an attributable identity, no one can reconstruct what it did or hold the right role answerable. Personal agents with broad access to mail, files, and calendars sit outside all of this, so governance has to cover individual use, not only institutional deployments. Which external services an agent may send data to is one of its permission scopes; which data may go to which service at all is BP11. This is the consensus position across the main governance frameworks (NIST, the EU AI Act, OECD, ISO/IEC 42001), which agree on enforced limits, named accountability, an agent inventory, logging, and independent review.

Examples

  • A policy document says the agent must not delete production records, but nothing in the system stops it; a prompt-injectedPrompt injectionAn attack that hides instructions inside content an agent reads (a web page, a document, a tool description), so the model follows them as if they came from the user. The user never sees the injected text, which is what makes it dangerous. Tool poisoning is one form, where the hidden instructions sit in an interface's tool description. instruction gets it to do exactly that. The rule was written, not enforced.
  • The same agent runs with a scoped token that carries no delete permission and a shutdown control a named on-call engineer can trigger, so the injected instruction fails because the action was never available to it.
  • An agent with shell access on a shared lab server, running under an account that can see every group's directory, deletes another group's data while cleaning up; there is no backup, and no one owned the machine. The same server, with the agent under a scoped account that cannot delete outside its own project, a snapshot taken before write access was granted, and a named owner for the server, would have contained the loss.
  • A vendor answers a document-borne injection in a drafting assistant with a detector in front of the model and a model upgrade; the reported payloads stop working, but the class still reproduces at publication, because the assistant still has to read the document in order to judge it. Whatever closes the class has to limit what the assistant can do with what it read, since it cannot reliably be told what to ignore (demonstrated in a commercial office suite under coordinated disclosure, not observed in science).
  • After an agent writes a wrong value to a shared dataset, the team traces it through the action ledger to the run, the model version, and the role that owned that workflow, and closes the gap.
  • Responsibility for an agent sits with "the department" in general; when it misbehaves, no one is answerable and no one has clear authority to shut it down.
  • An agent runs against a token ceiling set per run and an alert at half of it; when a misconfiguration sends it looping over the same records, it stops at the ceiling instead of billing for months against a budget that existed only on paper.
  • A new agent feature goes live for low-risk cases first, with success and go-live criteria and an accepted risk limit set in advance, before any wider rollout.
  • A personal agent connected to a researcher's mail and files is entered in the institution's agent inventory and reviewed like any deployed one, because its broad access is part of the same risk.

Sources

Change history

  • 2026-10-08: Added a Reasons cross-reference to the new BP11 (where research data may go). The services an agent may send data to stay a permission scope here.
  • 2026-09-16: Challenge #28. Extended the practitioner tab to groups that run their own infrastructure (the group is also the provider; a named owner for the machine; backup and a scoped account before write or delete access to shared storage) and added a shared-lab-server Example. No new atom: the case falls under bp4-a3 and bp4-a5.
  • 2026-09-16: Added the Copilot for Word disclosure (Måløy 2026) as a supporting source on bp4-a2, with an Example: a detector model and a model upgrade closed the reported payloads but not the class, because the model has to read the content in order to judge it.
  • 2026-07-31: Extended bp4-a3 to name resource and spend caps alongside permission scopes, autonomy limits, and shutdown paths, with matching Reasons, tab, and Example text. An agent inside every permission it was granted can still exhaust a budget, because a looping agent does not crash. Prompted by the reported Amazon cost overruns, recorded in the provenance data as a qualification on bp4-a3.
  • 2026-07-27: Added the EU AI Omnibus (2026) as a supporting source on bp4-a1 (technical safeguards in the system, not policy; Art 5(1a) foreseeable-misuse standard).
  • 2026-07-27: Renumbered from BP03 to BP04 on inserting the new BP01 (match the method to the task).
  • 2026-07-27: Rewrote Examples as concrete scenarios (actor, action, outcome), including anti-patterns, replacing restatements of the practice.
  • 2026-07-26: Rewritten to merge action-logging and attributable agent identity into the practice, add independent grounding (NIST, EU AI Act, OECD, ISO/IEC 42001, OpenAI, Chan et al., Kolt), and add two citation caveats: the "guidelines to guardrails / Kassorla et al." attribution in the source could not be verified, and the JRC "not automatable" wording is unconfirmed.
  • 2026-07-25: Created from the HFD distillation, grounded in the EU Expert Forum entry. New practice added because the guidelines-to-guardrails point is a separate action, not a sub-case of stating a human-in-the-loop level (BP09).

Discussion