要点
- AWS AI Blog (4 Nov 2026 PT): “Use decision models with Amazon Bedrock AgentCore Evaluations”; connects Jev (TypeSafe) and Strands Decider 2B through AgentCore code-based evaluators.
- Decision models return calibrated probabilities over fixed answers (noul / choice / score), not free-form rationales; sample evaluators cover groundedness, helpfulness, goal completion.
- Cascade: screen every turn with a decision model (~hundreds of ms, no LLM inference in the illustrated path); escalate only sessions below thresholds (example: groundedness 0.75, helpfulness 0.60) to Builtin.* LLM judges; diagram claims most routine sessions skip LLM calls (illustrative 75–90% fewer).
- Before release gates: validate on labeled own traffic with three human labelers, majority vote, agreement/calibration/disagreement analysis versus built-ins; re-check after question rewording or model version changes.
Lede
On 4 November 2026, Amazon Web Services published a how-to that plugs decision models into Amazon Bedrock AgentCore Evaluations. Agent scoring still needs labels you can threshold. Free-form LLM-as-judge text drifts in confidence from run to run. The post’s usable claim is architectural. For narrow, fixed-answer agent checks, calibrated decision models should screen every turn and escalate only low-confidence sessions to LLM-as-judge; they should not replace judges wholesale. If running full LLM judges on every turn beats that cascade on cost-adjusted agreement with human majority labels on the same traffic, the claim fails.
[1]What shipped
AgentCore Evaluations already scores agents on demand, in batch against stored sessions, and online against production traffic, with 15 built-in evaluators plus custom LLM-as-judge or code-based evaluators. Code-based evaluators run in AWS Lambda and may call external systems. This post shows two decision-model backends through that path: Jev, a hosted model from TypeSafe, and Strands Decider 2B, an open ~2B-parameter model you host yourself.
A decision model takes structured state and a fixed answer set and returns a calibrated probability distribution. Question types in both backends include noul (yes/no probability), choice (up to 255 options), and score (2–10 ordered levels with a probability-weighted value). Sample flags register Groundedness, Helpfulness, and GoalCompletion evaluators for either backend, beside deterministic Lambdas (response length, regex fact checks against mock HR data) and built-ins such as Correctness, Helpfulness, and ResponseRelevance.
The cascade pattern is the operational core. Run a fast decision-model check on every turn. Invoke a full built-in LLM evaluator only when the signal falls below a threshold you set after validation. AWS contrasts typical LLM-judge latency (about 2–10 s) with decision-model latency on the order of hundreds of milliseconds on the illustrated path, and states decision models fit “reading” checks while LLM judges still handle “deriving” explanations (citing the JEV-as-a-Judge preprint framing). An accompanying cascade diagram for an HR Assistant session uses example thresholds (groundedness 0.75, helpfulness 0.60) and claims most routine sessions can skip LLM evaluator calls, with an illustrative band of 75–90% fewer calls.
[1]Evidence and limits
The sample ships deploy and evaluate scripts, OpenTelemetry span adapters, and an HR Assistant with five tools on fixed mock data so evaluator behavior is easy to inspect. That is a strength for teaching the plumbing and a weakness for external validity: mock facts are not production HR traffic. The 75–90% LLM-call reduction is illustrative on the cascade figure, not a measured fleet statistic. Jev needs a TypeSafe API key; Decider needs a reachable host (local or EC2 in-VPC). Vendor claims about calibration must be re-earned on each team’s labeled sample.
AWS’s own validation recipe is required, not optional. Pull sessions that include failures; get three independent human labels and majority vote; split unanimous versus split-vote turns; run decision-model and matching built-in evaluators on the same inputs; check agreement, calibration of reported probabilities, and where disagreements cluster; re-run after any question rewrite or model version change. Until that loop is closed, release gates on decision-model scores are premature.
Steelman. LLM judges already improve with better rubrics and caching; a second model class adds ops surface. Calibration can look good on easy unanimous turns and collapse on the ambiguous ones that matter for gating. Cascade thresholds chosen on mock data will mis-fire in production. Hosted Jev creates a third-party dependency inside the eval path. Open Decider 2B may underperform larger judges on nuanced helpfulness.
Those caveats support cascade-plus-validation, not discarding decision models. The methodological spine is: separate narrow, thresholdable checks from explanatory judging; measure both against humans on your traffic; spend LLM-judge budget where confidence is low.
[1]Who pays, and what to watch
Teams running online agent eval at high sampling rates should price LLM-judge spend per turn before adding more built-ins. Platform groups should expose decision-model questions as versioned artifacts beside prompts. Buyers of “agent eval” suites should ask for human-agreement tables and cascade skip rates, not only a list of builtin names.
Watch next: public labeled suites comparing Jev, Strands Decider, and Bedrock built-ins on the same traces; whether the 75–90% skip band holds outside the HR demo; and whether AgentCore documents standard threshold-selection tooling.
[1]Close
AWS’s usable sentence is not that decision models retire LLM-as-judge. It is that agent evaluation gets cheaper and more stable when calibrated screens run on every turn and expensive judges fire only when unsure, after human validation on your own traffic. Six months on, either more teams will publish cascade skip rates with agreement tables, or eval spend will keep scaling linearly with every production turn.
[1]