要点
- NVIDIA Technical Blog (4 Nov 2026): “Use NVIDIA NeMo to Turn Agent Failures into Tests and Harness Fixes”; authors Patrick Moorhead, Sean Lopp, Moon Chung.
- Loop: NeMo Compass on stored traces + ETHOS.md → prioritized findings with trace IDs → NeMo Eval Author builds Harbor or NeMo Gym evals → change one harness piece (e.g. Hermes SOUL.md) → re-run development and held-out tasks.
- Hermes workplace-assistant example: development 3/6 → 6/6 passes, tool calls 10→6; held-out 4/6 → 6/6, tool calls 28→15.
- Dassault Systèmes LEO car-door engineering example: malformed tool calls 142→0; ~12-hour overnight loop reported 28% relative overall eval improvement and 35% fewer tool calls per run across six scenarios, no regressions reported in remaining scenarios.
On 4 November 2026, NVIDIA published a tutorial for turning agent failures into tests and harness fixes inside the NemoClaw blueprint. Generic leaderboard scores often miss why a domain agent fails: wrong tool, wrong branch, missing context, or a completed task that still violates a domain rule. The post’s usable claim is operational, not architectural mysticism. Production agent reliability improves when teams close a loop from stored traces through custom evaluations to a single harness change, and accept the change only when development and held-out tasks both move, not when teams swap the base model and re-hope. If model swaps alone, with the harness frozen, match the reported pass-rate and tool-call deltas on the same suites, that claim fails.
[1]The loop that shipped
NeMo Compass reads filesystem traces (one JSON line per Hermes run in the example) plus a short ETHOS.md that states intended behavior. It returns recurring issues, supporting trace IDs, and suggested experiment areas. In the sample set of 42 traces, Compass flagged sends without user confirmation by comparing actions to ETHOS.md. NVIDIA says that on internal benchmarks with thousands of traces, the full Compass signal set caught anomalies a coding agent given a similar brief missed.
NeMo Eval Author then installs skills into a coding agent (Codex, Claude Code, and peers) so it can author Harbor or NeMo Gym evaluations and verifiers from Compass findings, raw traces, ETHOS.md and the agent repo. In the example, four Harbor tasks similar to “draft a response” were created; a verifier fails the run if send_message fires without confirmation.
The harness change shown is small and local: Hermes SOUL.md was edited so draft/compose language authorizes preparation only, and a preparation-tool token is not treated as consent to send. After the change, development tasks went from 3/6 to 6/6 passes with tool calls 10→6; held-out tasks from 4/6 to 6/6 with tool calls 28→15. Compass on the new traces stopped reporting the send-without-confirmation finding and surfaced a next issue: answers not grounded in retrieved evidence.
Engineering example with harder numbers
The same loop was applied to LEO, Dassault Systèmes’ engineering Virtual Companion, on a car-door design workflow that proposes changes and tests them in simulation. Compass found recurring QueryModel arguments placed outside the expected options field, returning empty errors the agent could not recover from. A validation step for misplaced tool-call arguments cut malformed calls from 142 to zero; accuracy held and no regressions appeared in remaining scenarios. An overnight unattended pass of about 12 hours reported a 28% relative improvement in overall evaluation score and 35% fewer tool calls per run across six scenarios.
These are vendor-hosted worked examples, not multi-site RCTs. They still separate two quantities buyers actually pay for: task quality and execution efficiency, both measured after a harness change grounded in traces.
[1]Steelman and gaps
Steelman. Held-out sets of two tasks × three runs are tiny; production suites “hundreds to a few thousand” are asserted from experience, not shown. Coding agents that author both the fix and the evals can overfit unless held-outs stay truly hidden. Compass findings inherit label noise in ETHOS.md. The Dassault 28%/35% figures are “reported results” from an overnight loop, not an external audit. OpenShell + Harbor prerequisites and GPU/API keys raise the floor for who can run the blueprint.
Those caveats are real. They do not erase the methodological spine: prioritize from traces, write a verifier that fails the bad behavior, change one harness surface, demand gains on held-outs, then re-run Compass. That spine attacks the common failure mode of “new model, same broken tool descriptions.”
Open questions: public release of the 42-trace and LEO trace corpora; inter-rater checks on Compass findings; how often overnight loops introduce silent policy regressions outside the scored scenarios; and whether the loop’s ROI survives when the agent already has mature eval suites.
[1]Who should care, and what to watch
Teams shipping MCP-connected workplace or engineering agents should stop treating model upgrades as the first lever when traces show repeated tool misuse. Platform groups should require held-out tasks that coding agents never see while proposing harness diffs. Buyers of “agent optimization” should ask for malformed-tool and confirmation-policy metrics, not only win-rate on happy-path demos.
Watch next: third-party replications of Compass-versus-coding-agent anomaly recall; published before/after tables with larger held-outs; and whether ETHOS.md-style behavior specs become standard artifacts beside system prompts.
[1]Close
NVIDIA’s usable sentence is not that NeMo replaces judgment. It is that agent quality compounds when failures become tests and harness patches under held-out gates. That process makes model swaps the last resort, not the first. Six months on, either more teams will publish trace-tied before/after tables with hidden held-outs, or the industry will keep buying larger models to paper over the same tool-call bugs.
[1]