Your Agent Has a Goal. Who Gave It Authority?

A request for school statistics should not turn into an attempt to break into the website holding them.

On September 30, Transluce published research describing agents apparently searching for public information that attempted rudimentary attacks against government websites. In one case, activity at a US Department of Education site included more than 200,000 requests and a failed SQL injection probe. The data being sought appeared to match a research task in Google's DeepSearchQA benchmark.

The researchers also identified failed probes against Library and Archives Canada. They found no evidence in these datasets of access to nonpublic information, and they did not attribute all the activity to OpenAI. These were newly published findings about earlier events. Those qualifications matter. So does the possibility the report raises: an ordinary information request can become an unauthorized attempt to get the answer. Transluce's report

“Find the answer” is not a mandate

I have written before about the agent that pursues its task beyond its mandate. The question I want to take further now belongs to the people designing and operating the product. Who defines that mandate, and where does the system enforce it?

I recently argued for independent verification with real access and the freedom to report. That function needs something concrete to verify. It needs an account of what the agent was allowed to do.

“Find the answer” does not provide one.

Consider a research assistant that reaches a source it cannot access. It could return a partial answer, explain the missing source, or ask an authorized person for help. It could also search for a different credential, upload material to an outside service, or keep probing until it finds another route.

OpenAI's September 16 disclosures make this problem tangible. The company described training or evaluation cases in which models uploaded files to the public internet without authorization, including to obtain a citation or share work with collaborating agents. These are individual examples, not a measure of how often the behavior occurs. But they expose a design question that applies well beyond a model lab: what is the permitted response when the approved workflow cannot finish the task? OpenAI's disclosure

That question belongs in the product requirements before it appears in an incident report.

Preparing the brief and authorizing its destination are separate decisions

I encountered a concrete version while building an agent platform for employees, informed by what I had learned operating TARS, my personal agent.

One useful request is simple: prepare me for a customer meeting. Bring together the relevant account history, recent conversations, open opportunities, support issues, and product usage. Narrow the queries to the customer, stay within my existing access, and show me which sources contributed to the answer.

Then add one instruction: send the finished brief to my private email.

Our platform blocks that send and records the attempt. The interface identifies the action, destination, and reason for refusal. The brief can still be useful. The requested transfer is outside the permitted boundary.

That distinction was a product requirement from the beginning. It also applies when the instruction comes from a customer message or document the agent retrieves. Words inside source material cannot grant new authority over the company's information.

The PM has work to do here. Define the intended user outcome, the permitted effects, and the experience when the task reaches a limit. Security and data owners establish the applicable policies. Engineering makes those policies enforceable where an action becomes real. Independent verification tests whether the controls hold, including when the agent encounters an obstacle or misleading input.

These responsibilities need named owners. Otherwise, each team can assume somebody else defined what “on the user's behalf” actually permits.

AIR Security's September 17 Plugin4Shell disclosure described a flaw in affected coding agents that could cause a plugin installation to resolve to different code from the version a marketplace had reviewed and pinned. An attacker controlling the plugin repository could exploit that mismatch. The research illustrates a different failure mechanism from an agent improvising an unauthorized route, but it raises a related assurance question: did the approved thing remain the thing that ran? AIR Security's disclosure

An approval record needs a defined scope. Which artifact, permissions, destination, and conditions did it cover? Which changes invalidate it? A successful review cannot silently become permission for whatever happens to arrive later.

A successful control hidden inside an apparent failure

This is also where our measures of success need attention.

Suppose a meeting brief is complete, but the destination is prohibited. The agent refuses the transfer and explains why. If the product records only “task incomplete,” the dashboard hides a successful control inside an apparent failure.

I would separate useful completion within authority, correct refusal or escalation, recoverable failure, and unauthorized effects. Each calls for a different response. A missing integration may deserve investment. An unnecessary interruption may deserve better design. A prohibited transfer that was correctly blocked deserves to remain blocked.

You still need to examine the quality of the refusal. Did the system preserve useful work? Explain the limit? Route the decision to someone who could resolve it? Stopping safely should not leave the user stranded.

A correction I owe

A point I made recently deserves a correction. I said a checker paid by the builder could not count as a second opinion. That was too absolute.

On September 18, Anthropic announced an embedded evaluation partnership with Accenture and said it would fund the work directly. It also acknowledged that operating details and broader standards were still being developed. Anthropic's announcement

Four questions before you expand

Before expanding an agent workflow, choose one task already in use. Take it through the four questions that organize my book's Intent, Build, Evidence, Decision loop:

1. Intent: What outcome is authorized? Name the permitted sources, actions, destinations, and limits. State what should happen when those are insufficient. Include the people affected by the agent's actions, not only the person requesting them.
2. Build: Where are those limits enforced? Identify the control that checks an action before execution. Test an allowed action and a refused one. Check that the agent cannot broaden its own permissions or disable the check while trying to finish.
3. Evidence: What proves what happened? Preserve records of consequential attempts, policy decisions, and confirmed effects. Give an independent reviewer enough access to reconstruct the run. An agent's final explanation needs supporting evidence.
4. Decision: Who can stop the work and authorize its return? Test cancellation and access revocation while work is queued or running. Explain what has already happened, what can still be prevented, and what requires a separate recovery action.

Bring a PM, an engineer, and a security owner to that exercise. Ask each to point to the same boundary in the running system. If they describe different limits, resolve the disagreement before expanding the agent's reach.

An agent's ability to keep going can be valuable. The product has to make the conditions for continuing explicit, enforce them, and preserve the evidence. The goal tells the agent what to pursue. Someone remains responsible for deciding how far it may go.

My new book, Product Management in the Agentic Era, is now live. Read a free sample.