Recommendation: treat autonomous agents as controlled software operators, not digital employees. Start with bounded, reversible workflows where tool access, identity, approval points and recovery actions can be specified before deployment. Keep high-impact decisions and irreversible changes outside the agent’s independent authority until testing demonstrates that the controls work under adversarial conditions.
Choose processes for bounded autonomy
An agent is a good candidate for a process when the objective is clear, the required tools are known, the permitted data is limited, and incorrect actions can be detected and reversed. Useful early candidates include assembling a case file, reconciling records into a proposed update, or preparing a change for human approval.
Be cautious when a process combines ambiguous objectives, sensitive data, irreversible actions and weak observability. “Resolve the issue” is not a sufficient operating specification if the agent can alter customer records or approve payments.
Use these decision criteria:
- Bounded objective: define what success and failure look like.
- Constrained action space: prefer specific business functions over general shell, browser or API access.
- Reversibility: design compensating transactions, rollback and expiry.
- Observable state: record inputs, plans, tool calls, results, approvals and final effects.
- Human accountability: name the person or role responsible for the outcome.
- Fallback path: make it straightforward to stop the agent and complete the process manually.
This remains a design recommendation. NIST’s 18 May 2026 synthesis of responses on AI-agent security found broad agreement that agents create novel security concerns and that conventional cybersecurity practices need adaptation. It summarises stakeholder responses and is neither a production incident dataset nor a validation of any particular architecture. See NIST’s summary analysis.
Prefer orchestration over unconstrained autonomy
A production workflow should usually be an orchestrated state machine with model calls inside it, rather than a model receiving a broad goal and deciding the entire process. The orchestrator should own state transitions, retries, deadlines, idempotency, rate limits and escalation.
A practical pattern is:
- Classify and validate the request.
- Retrieve only approved context.
- Generate a proposed plan.
- Check the plan against deterministic policy.
- Execute low-risk steps through narrow tools.
- Pause for approval before defined high-impact actions.
- Verify the result independently.
- Record evidence and close or escalate.
Separate planning from execution where possible. The model can propose an action, but a policy service should decide whether the action is permitted. Downstream systems must enforce authorisation themselves. OWASP’s LLM06:2025 guidance recommends complete mediation, meaning every request to a downstream system is checked against security policy rather than trusted because an agent selected it. See OWASP’s LLM06:2025 Excessive Agency guidance.
Make tools narrow, typed and least-privileged
Tool design is a primary control surface. A tool called send_email is easier to constrain than a tool called run_command; update_invoice_status is easier to review than unrestricted database write access.
For each tool, define:
- allowed operations and objects
- input schema and validation rules
- permitted data classification
- rate and quantity limits
- required approval level
- failure and retry behaviour
Do not expose unused functionality “for convenience”. OWASP’s 2025 guidance identifies excessive functionality, excessive permissions and excessive autonomy as common causes of excessive agency. Its examples include read-only use cases backed by identities that can also modify or delete records.
The Model Context Protocol, or MCP, standardises connections between AI applications and external tools, data sources and workflows. The 28 July 2026 specification added a stateless protocol core, routable headers, authorisation hardening and a formal deprecation policy. These are protocol capabilities, and they say nothing about whether connected tools are trustworthy or correctly governed. See the MCP 2026-07-28 specification release.
Treat remote MCP servers as third-party software and third-party content as untrusted input. Review, pin and monitor servers, and test them with synthetic data before granting access to sensitive systems. The MCP roadmap published 22 August 2026 also identifies progressive tool discovery as a priority because large tool catalogues can make selection less reliable. See The New MCP Roadmap.
Design identity for the action, not just the agent
An agent may act as a user, as a workload, or as a workload delegated authority by a user. These models have different audit and containment properties.
Use user-context authorisation when the action must reflect the individual’s existing rights. Use a distinct workload identity when the process is organisational and should not inherit a person’s broad access. In either case:
- issue short-lived credentials
- separate read and write scopes
- require step-up approval for sensitive operations
- log the originating user, agent identity, tool and downstream resource
- revoke credentials independently of the model runtime
MCP’s Enterprise-Managed Authorization extension was described as stable on 18 June 2026, enabling central provisioning for connected servers. However, the MCP roadmap published on 22 August 2026 states that current authorisation is still primarily designed around a person approving access in a browser, while workload identity, delegation and standard token exchange remain active areas of work. Treat the extension as useful infrastructure that still falls short of a complete agent-identity model.
NIST’s concept paper published on 5 February 2026 frames software-agent identity and authorisation as a potential demonstration project rather than a completed standard. Organisations should still test token scope, revocation, confused-deputy scenarios, delegation chains and cross-tenant isolation in their own environment. See NIST’s concept paper on software and AI agent identity.
Use approval as a control, not a ritual
Human approval is useful when the reviewer can understand the proposed action, its consequences and the evidence supporting it. A prompt saying “Allow agent to continue?” is weak control design.
Approval requests should show:
- the intended business outcome
- the exact records or systems affected
- the proposed changes
- uncertainty or policy exceptions
- rollback or recovery method
- the identity under which execution will occur
Ask for approval once the agent has prepared a concrete, reviewable result, after any harmless investigation is done.
Attach approval to the side effect itself, in addition to any approval at the start of a workflow. OpenAI’s agent documentation, which shows no publication or update date, describes approval interruptions that pause execution, preserve resumable state and continue the same run after a decision. It also warns that agent-level guardrails do not automatically run around every nested or custom tool call, so validation should sit next to the tool that creates the side effect. See OpenAI’s guardrails and human-review guidance.
Do not assume frequent approval produces reliable oversight. In a provider account published on 25 May 2026, Anthropic reported that users approved roughly 93% of permission prompts in one product and described approval fatigue as a reason to combine review with stronger containment. This is a provider report about its own telemetry and systems; independent validation is separate work. See Anthropic’s account of agent containment.
Contain the runtime and the data path
Prompt instructions and model safeguards are probabilistic. Runtime boundaries can be deterministic. Use separate environments, network egress controls, filesystem boundaries, secret isolation, resource quotas and termination mechanisms according to the potential impact of failure.
Containment should cover three paths:
- Execution: what code, processes and tools can run.
- Network: which hosts, APIs and data stores can be reached.
- Data: which records can be read, retained, copied or transmitted.
Keep secrets outside the model context where possible, using brokered operations that perform a narrow action without exposing reusable credentials. Monitor outbound traffic for unusual destinations, volume and timing.
AISI’s sandboxing work separates isolation into tooling, host and network controls and notes that higher-stakes evaluations need stronger isolation. The toolkit is aimed at safe testing and does not prove production safety, but the principle transfers well to enterprise deployment. Apply the same separation when an agent can execute code or handle untrusted files. See AISI’s Inspect Sandboxing Toolkit, published 7 August 2025.
Test failure modes, not just task completion
An agent evaluation should measure both useful completion and unacceptable behaviour. Include malformed inputs, ambiguous requests, stale data, tool outages, duplicate events, malicious documents, poisoned memory, prompt injection, excessive retries, privilege escalation and partial downstream failure.
Test tool contracts, orchestration and policy rules, identity and authorisation, end-to-end business outcomes, adversarial scenarios, and recovery and termination.
Record the model version, prompts, tools, permissions, test budget and environment. A fixed compute budget can understate capability on longer tasks, so compare results across budgets where the production system may continue working for longer. AISI’s 2026 evaluation work found that capability often rises with test-time compute across cyber and software-engineering tasks, although the observed relationship has not been established across all enterprise processes. See AISI’s analysis of test-time compute.
What we are watching
- Whether agent identity standards mature beyond user-delegated OAuth to support workload identity, delegation chains and reliable revocation.
- Whether MCP and similar protocols develop stronger mechanisms for server provenance, tool integrity and policy enforcement.
- Independent, sustained production evidence about approval fatigue, incident rates and recovery effectiveness.
- Evaluation methods that test long-running agents, adaptive attackers, memory poisoning and cross-system tool chains.
- Whether evaluation budgets become a standard reported variable rather than a hidden implementation detail.
- Evidence that runtime containment remains effective when agents receive more tools, more context and larger execution budgets.
- Whether organisations can maintain an auditable inventory of agents, tools, permissions, prompts, models and downstream effects as these systems multiply.