Agents Don't Need to Be Smarter. They Need Permission Slips

New model releases keep promising higher scores and longer unattended runs. Giving an agent that much freedom can still go wrong: it can edit the wrong files, send a message too early, or state something that is not in its sources. Those failure modes are familiar in practice. How often they happen in production is not settled by the studies below.

The studies do show a capability gap on realistic tasks, not a ranking of "smarter model" against "stricter process." WebArena reported that its best GPT-4 agent completed 14.41% of web tasks, against 78.24% for people (Zhou et al., 2023). The original SWE-bench paper reported that its best model, Claude 2, resolved 1.96% of real GitHub issues (Jimenez et al., 2024). Later systems score higher. Those two figures are dated, and they are not a current leaderboard. The encyclopedia article on autonomous agents keeps the same caution.

#Structure beats cleverness

One way to deploy agents is a small hierarchy. A manager receives the task and routes it to a specialist, and each specialist has one job. This is a design choice, not a result from the papers above. The point is to keep each step inspectable. When one model plans, edits, checks and executes in the same loop, a small mistake is harder to see before the next step builds on it.

#The human approval gate

A workable sequence is: the agent prepares, a person approves, then the agent acts. Nothing is changed and nothing is sent without an explicit yes. When the agent proposes a file edit or a message, it should show the before and after. When it uses a fact, it should link the document that fact came from.

How long that review takes depends on the change. The gain is that a person sees the exact diff before it lands. The encyclopedia notes that human approval for consequential actions is a commonly recommended practice, and that no source reviewed there measures how widely teams actually use it.

#Operational transparency replaces status meetings

Several agents create a tracking problem of their own: what is blocked, what is missing, and what is waiting on a person. A short daily note can list those three things. That is a habit that fits the approval gate. It is not something the benchmarks tested, and it does not by itself remove the need to look at the work.

#What the critics say

A fair objection is that an approval gate adds friction. If every change needs a yes, the agent is slower, and some of the point of automation is lost. People who want full autonomy argue that the model should finish the task without being stopped.

#Why unconstrained autonomy fails

The 2023 numbers are the evidence for caution, not a proof that every unattended run corrupts data. Zhou et al. and Jimenez et al. showed agents failing most of the realistic tasks in those benchmarks. An agent that misreads a schema or a file path can apply that mistake across many edits before anyone looks. Whether a permission step prevents that in a given team is something that team has to measure.

Clear roles and an approval rule are a way to stay inside that gap while models improve. They are not a substitute for reading the current benchmark, and they are not evidence that a better prompt would fail where a permission slip succeeds.

#What should happen

Teams putting agents on real files or real messages should decide, before the run, which actions stop for a person. A reasonable minimum is: halt before execution, show a before-and-after diff, and attach a source for any factual claim the agent wants a person to rely on.

A starter instruction that matches this post:

"I want a team of AI agents for my work. Create one manager that receives every task and routes it to a specialist. Give each specialist one job. Rules: never change anything without my explicit yes, show every change as before and after, never send anything on my behalf, and link every item to its source."

#Bottom line

Agents do not have to be more autonomous to be useful. On the 2023 evidence, they were weak at realistic web and software tasks. A human permission step, a diff, and a source link are controls for that weakness. They are not a claim that process has been shown to beat a smarter model.

#Frequently asked questions

#Why do AI agents need permission slips?

On WebArena, the best reported GPT-4 agent in the paper completed 14.41% of tasks against 78.24% for people (Zhou et al., 2023). On the original SWE-bench, Claude 2 resolved 1.96% of issues (Jimenez et al., 2024). A permission step before edits or outbound messages is a way to limit damage from those misses. It does not by itself prove the agent will be accurate.

#What is the role of the manager agent?

In the setup this post recommends, the manager receives the task and routes it to a specialist with one job. That split is an operating choice so each step can be reviewed. It is not a benchmark result.

#How should changes be presented for approval?

As a before-and-after comparison of the file or the message, plus a link to the source of any fact the person is being asked to trust. The person then approves or rejects that specific change.

#Sources

• Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., and Neubig, G. (2023). WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv:2307.13854. https://arxiv.org/abs/2307.13854. Best GPT-4 agent in the paper: 14.41% task success, against 78.24% for people. A 2023 result.
• Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770. https://arxiv.org/abs/2310.06770. The paper states that Claude 2 resolved 1.96% of the issues. Later scores are higher and are not used here.
• Encyclopedia: Autonomous agents. Same dated benchmarks, and the same note that approval gates are recommended practice without a measured adoption rate.