要点
- Same model (GLM-5.3-flash), same 90 AppWorld tasks in 30 scenarios, one attempt each, scored in the Harbor framework. The authors' Harbor fork is public.
- Task success: CUGA 81.1% (73/90), Hermes 46.7% (42/90), Deep Agents 30.0% (27/90). Scenario success, which requires all three variants in a scenario to pass: CUGA 66.7% (20/30), Hermes 26.7% (8/30), Deep Agents 6.7% (2/30).
- A cited analysis of SWE-bench Pro, not this experiment, found the same GLM-5.2 model moving from 23% to 52% when only the harness changed.
- CUGA's AppWorld setup decomposes tasks by app, shortlists tools, lets the model write Python to call APIs inside a sandbox, checks state-changing code before it runs, and reflects after the fact.
- Policies can sit at intent, playbook, tool approval, tool-guard, and output-format points. In the Policy Playground, "Pay everything I owe" led an unguarded agent to pay 90 dollars, including a denied claim; the guarded agent waited for approval and paid the 10 dollars that was due.
On 3 November 2026 IBM Research published a Hugging Face post that treats the usual question, "which model should I buy?", as the wrong one. Community lists now track nearly 200 agent harnesses, meaning the runtimes that wrap a model with planning, tools, and checks. The authors kept the model fixed and changed only that runtime. Completion on a 90-task slice of AppWorld moved from 30.0% to 81.1%.
[1]What shipped
AppWorld is a benchmark of multi-step work across 9 consumer apps and 457 APIs, including email, payments, and file storage. The authors call it a proxy, not a full enterprise suite: it tests logging in, reading API docs, paging through results, and changing state across several apps. They ran CUGA (IBM's Configurable Generalist Agent), Hermes from Nous Research, and Deep Agents from LangChain, all with GLM-5.3-flash, on the same 90 tasks in 30 scenarios, one attempt per task.
CUGA used its AppWorld configuration: AppWorld instructions, task decomposition, a check before execution, and automatic reflection. Hermes used Harbor's default Hermes setup with AppWorld tools over MCP. Deep Agents received AppWorld tools over MCP plus LangChain middleware configured for 469 tools, including tool selection, context summarization, a to-do list, retries, and call limits. The authors also added two small fixes so that malformed GLM replies did not end Deep Agents runs.
They report two scores. TGC is the share of tasks passed. SGC is the share of scenarios in which all three variants passed, so it rewards consistency rather than a lucky run. CUGA reached 81.1% TGC and 66.7% SGC. Hermes reached 46.7% and 26.7%. Deep Agents reached 30.0% and 6.7%. The authors say this is not a verdict on Hermes or Deep Agents, which were built for always-on personal assistants and for long research-style jobs, not for multi-app API work like AppWorld. That, they argue, is the point: the runtime has to fit the job.
They then describe the CUGA loop that ran in this setup. Every agent prepares, calls the model, executes, and repeats. The same loop can scale from one agent with its own tools to a supervisor coordinating several agents, locally or remotely. In the AppWorld configuration the runtime splits a task into steps, one app at a time; each step sees that app's tools plus a few shared ones, not all 469; a search can fetch extra tools; the model writes Python that calls the APIs; the code runs in a restricted sandbox with a time limit; a check reviews each code block that would change data before it runs; afterwards the system reviews the result and can revise the plan; a pagination audit flags a full page of results when the next page was never requested.
Each of those steps is also a place to enforce a rule. Most teams put payment approvals or "keep customer data inside the organization" into the system prompt. A prompt is an instruction. The model can misread it, a user can talk the agent out of it, and an injected message can change it mid-task. CUGA policies, described in the authors' ACM CAIS 2026 paper Governance by Construction for Generalist Agents, act at four points: an intent guard can block or answer a request before the agent acts; a playbook gives a standard procedure; tool approval pauses the run for a person and shows the proposed code; a tool guide with ToolGuard runs check code before the tool, and a blocked call never reaches the tool; an output formatter shapes the final reply to an approved template. Guidance steers the model. Approvals and ToolGuards are enforced by the runtime, whatever the model decides.
The Policy Playground sends the same message to two live agents. For "Pay everything I owe," the agent without policies paid 90 dollars, including a denied claim in collections. The agent with policies held the payment for approval, then paid only the 10 dollars that was due.
The authors also list when not to pick CUGA: use a terminal coding agent such as opencode, Codex, or Claude Code for that job; use OpenClaw or Hermes for a personal assistant that lives in chat apps; use LangGraph directly if you want a blank graph framework. CUGA is built on LangGraph and is aimed at business work across APIs, apps, and web interfaces, under rules you can enforce. It is Apache 2.0, works with any OpenAI-compatible endpoint, and can run locally, on Kubernetes in a private or sovereign cloud, or in an air-gapped cluster. With hosted models, they say CUGA held first place on WebArena from February to September 2025 and on AppWorld from July 2025 to February 2026.
[1]Evidence on the table
The primary evidence is the 3 November 2026 IBM Research post, including the setup table, the 73/90, 42/90, and 27/90 counts, the scenario scores, the feature list, and the Policy Playground example. The comparison setup is public in the authors' Harbor fork. Three limits are written into the post: this is a 90-task sample on one benchmark; the setups were not identical; and the runs did not use policies, so they do not measure policy enforcement. The authors also note that the CUGA team ran the comparison, and that CUGA used an AppWorld configuration the other harnesses did not. They invite better Hermes or Deep Agents configurations to harbor-eval and say they will rerun them on the same tasks and publish the results. The SWE-bench Pro movement from 23% to 52% for GLM-5.2 is cited from another analysis collected by best-of-Agent-Harnesses, not generated in this experiment.
[1]Steelman and gaps
Steelman. Buyers still compare models first and treat the wrapper as plumbing. If the same weights can finish 30% or 81% of a multi-app workload depending on planning, tool visibility, code execution, and a check-before-write step, then the wrapper is part of the capability. Prompt-only policies fail in a predictable way: they can be ignored. A runtime that can refuse to call a payment tool is a different object. Publishing the Harbor fork and asking for rival configs is the right way to keep that claim honest.
Gaps. The comparison was run by the CUGA team with a CUGA-specific AppWorld configuration, so the headline gap mixes harness identity with extra recipe work. Hermes and Deep Agents were not designed for this benchmark, which the authors admit and which also makes the contrast easier. Ninety tasks and one attempt per task leave room for variance. The policy story is illustrated by a playground example and a paper, not by the 90-task table, because those runs did not turn policies on. Independent groups have not yet, in this post, closed or confirmed the gap.
[1]Who should care, and what to watch
Teams automating work across internal APIs, anyone writing payment or privacy rules into a system prompt and calling that governance, and builders of the nearly 200 listed harnesses. Watch over six months: whether rival configs narrow the AppWorld gap on harbor-eval; whether buyers start asking where a rule is enforced, rather than whether it is mentioned; and whether other labs repeat a same-model, different-runtime protocol on a second benchmark.
[1]Close
The useful sentence is not that CUGA won a bake-off. It is that the runtime you wrap around a model can move completion by tens of points, and that a rule which lives only in the prompt is still a suggestion. Six months on, either public same-model comparisons become a normal way to choose an agent stack, or teams will keep swapping models while the unchecked wrapper does the work that looked like intelligence.
[1]