要点
- On 10 November 2026, Amazon published an AWS Machine Learning blog, "The agent evaluation playbook: Built-in, DeepEval, and AutoEval on Amazon Bedrock AgentCore Evaluations," by Bharathi Srinivasan, Manoj Selvakumar, Neha Thakur, and Visakh Madathil.
- AgentCore Evaluations offers three sources of judges: built-in evaluators authored and quality-tested by AWS and referenced by IDs such as Builtin.GoalSuccessRate; third-party managed evaluators from DeepEval and AutoEval, which AgentCore hosts, including the judge model and inference; and custom evaluators that start from a built-in or third-party base and change the model, prompt, or rubric.
- The authors tell teams to define success in business terms, break it into observable behaviors, and map each behavior to a trace-level or session-level evaluator. Starting with metric names that merely sound right is described as the most common mistake.
- A multi-dimension evaluator scores several criteria in one judge call and returns one result per dimension, with no composite score. The post's example is a support reply that must resolve the issue, follow policy, and stay professional, where those criteria can fail independently.
- The same evaluator IDs work in on-demand, online, and batch modes. Metrics that need a reference answer or an expected tool path, including Builtin.Correctness and the trajectory-match checks, run in batch evaluation only. The post does not publish independent reliability figures for these judges.
On 10 November 2026, Amazon published an AWS Machine Learning blog on how to evaluate agents on Amazon Bedrock AgentCore. The post, signed by Bharathi Srinivasan, Manoj Selvakumar, Neha Thakur, and Visakh Madathil, says teams can use AWS built-in evaluators together with DeepEval and AutoEval evaluators directly from AgentCore Evaluations. The checkable claim is not that AWS now offers a longer list of metric names. It is that AgentCore will host DeepEval and AutoEval as managed judges under the same IDs in live traffic, and that a single blended score is treated as a design mistake, because one high number can hide a failure on another behavior.
[1]What AgentCore Evaluations actually offers
The catalog has three tiers. Built-in evaluators are authored and quality-tested by AWS. A team refers to them by ID, such as Builtin.GoalSuccessRate or Builtin.Helpfulness, and the service runs the rest. The post says they are benchmarked for reliability and cover common dimensions, and it names 13 more built-in IDs besides those two, without publishing the benchmark numbers in the text we read.
Third-party managed evaluators come from DeepEval and AutoEval. In managed mode, AgentCore hosts the code, picks the judge model, and runs inference on its own capacity, in the same way it runs built-in evaluators. The authors' pitch is that a team already using one of those libraries in development can run the same metrics in production without a separate evaluation service.
Custom evaluators take a built-in or third-party evaluator as a base, so a team can swap in its own model and change the prompt or rubric. After creation, a custom evaluator is used like the others. The post also points to a ListEvaluators API, which returns the built-in, DeepEval, and AutoEval catalog plus any custom evaluators in the account, and to a prompt template library in the AgentCore documentation for definitions, scoring scales, and input requirements.
Built-in evaluators are grouped in the post as response quality, task and goal completion, tool use, skill use, and safety, plus three trajectory checks that compare the agent's tool path with one the team expected. DeepEval is grouped as task and goal completion, conversational quality, tool use, and safety, including ConversationCompleteness, KnowledgeRetention, and PIILeakage. AutoEval is described as more specialized, with Security, Possible, and Humor.
[1]Start from the business outcome, not the metric name
The authors say the usual error is to open the metric list and pick names that sound right. Helpfulness sounds good, so a team uses it. The run produces numbers, and almost no useful insight, because the metric fits its label rather than what success means for that agent.
They offer three steps instead. First, define success in business terms. For a customer support agent, that might be first-contact resolution. For a research agent, it might be a complete, well-sourced answer grounded in retrieved documents. Second, break that outcome into behaviors you can observe in a trace or a session: whether the agent finished the user's task, stayed coherent across turns, stayed inside policy, or called the right tools. Third, map each behavior to an evaluator at the right level. Trace-level metrics cover per-turn or per-tool-call behavior. Session-level metrics cover outcomes that only make sense over the whole conversation.
The post's warning about hidden failure sits here. A support agent can resolve every ticket and still expose personal information in its replies. In that case, the agent is failing even though it has a high completion rate. ConversationCompleteness, in the DeepEval set, is offered as a way to catch an agent that answers the latest message and drops earlier requests. KnowledgeRetention is offered as a way to catch an agent that forgets something the user said three turns ago.
[1]Why a single blended score is treated as a mistake
Most evaluators return one score per trace or session. That works when you measure one behavior. The authors say it fails when one reply must satisfy several criteria that can break independently: resolve the issue, follow refund and data policy, and stay professional. One judge that weighs all three returns one number, and you cannot tell which criterion drove it.
A multi-dimension evaluator is their answer. It is a custom judge with a list of dimensions instead of one scale. Each dimension has its own name, instruction, and numeric or categorical scale. The judge reads the trace once and scores every dimension in the same call. AgentCore returns one result per dimension, and no composite score. The post says a release that improves tone but regresses policy compliance should fail, and that a single blended score would hide that.
They also say when not to force those criteria into one evaluator. Keep them separate when an established built-in or third-party metric already covers the behavior, such as Builtin.Correctness or ThirdParty.DeepEval.PIILeakage. Keep them separate when they need different levels or different judge models, because the dimensions in one evaluator share a single level and model. Do the same when some criteria need ground truth, such as an expected response. Keeping the reference-free criteria in their own evaluator means that evaluator can still run on live traffic.
That last distinction matters for the product claim. On-demand evaluation passes a session to client.evaluate() during development. Online evaluation attaches to a live agent runtime and samples real traffic. Batch evaluation runs a fixed evaluator stack against a golden set of sessions. All three modes use the same evaluator IDs. Metrics that need a reference answer or an expected trajectory still run in batch only, because live traffic has no reference answer or expected path.
[1]What the post does not show
This is an AWS product blog, written by AWS authors, including a specialist whose role is described as technical go-to-market for Trustworthy AI. The built-in evaluators are said to be quality-tested and benchmarked for reliability, but the post does not give those benchmark numbers, sample sizes, or comparisons with human raters. DeepEval and AutoEval are established open-source libraries, but the managed versions run on judge models and capacity that AWS picks. The post does not report how often those hosted judges agree with the libraries as teams already run them, or with human review.
The catalogs and API names are checkable as product description. The claim that a blended score can hide a failure is an argument, illustrated with examples, not a measured result from a production fleet in this post. We did not call ListEvaluators or run an evaluation ourselves.
[1]The fair case
The fair reading is that many teams already have a metric dashboard and still cannot answer whether the agent did the job the business cares about. Putting DeepEval and AutoEval behind the same IDs as the built-in judges, and making those IDs work in development, live sampling, and batch regression, is a concrete way to stop evaluation from being a separate pile of offline scripts. Refusing a composite score, when two criteria can fail independently, is also the right default.
It is also fair to treat this as a catalog and a method, not as evidence that the judges are accurate. A team still has to choose which two or three behaviors actually show that the business outcome happened, and it still has to live with the fact that the most exact checks, the ones that need a reference answer or an expected tool path, cannot see live traffic. The post itself tells readers to start with a short stack and add a metric only when they find a failure the current set missed.
[1]Who should care, and what to watch
Teams already running agents on AgentCore, especially in support, research, or regulated content, should care, as should evaluation engineers who already use DeepEval or AutoEval in development and want the same names in production. Compliance reviewers who today see only completion rate or latency should care too.
In the next six months, the tests are whether teams actually put DeepEval or AutoEval judges on online, sampled live traffic, whether they adopt multi-dimension judges without slipping a composite score back in, and whether AWS publishes anything checkable about judge reliability. A second test is narrower: whether ground-truth metrics remain batch-only, or whether teams find a workable way to grade live traffic without a reference answer.
[1]Close
On the evidence in this post, AgentCore Evaluations is a way to run AWS, DeepEval, and AutoEval judges under the same IDs, including on live traffic, and to score behaviors separately when one high number would hide another failure. It is not, in these pages, a measured proof that those judges are reliable. The claim to watch is whether that shared-ID setup, and the refusal of a blended score, become the default way AWS-hosted agents are graded over the next six months, while the exact, reference-based checks stay in batch until someone shows they can be used live.
[1]