Back to Explore
NewsVibeFix editorialUpdated Nov 20, 2026

Did Your Agent Get Better or Worse? AWS Open-Sources UAEF to Measure Agents from Every Framework with One Ruler

On 11 November 2026, AWS open-sourced UAEF: adapters turn traces from LangGraph, LangChain, Strands and Bedrock AgentCore into one format, then 36 metrics across 7 dimensions score them, flagging regressions past 5%. The judge model is hardened too — scores must cite verbatim evidence, and never decide safety.

A developer types at a dual-monitor workstation; the left screen shows a dark code editor, the right a testing dashboard with green checkmarks and bar charts

In short: agents keep multiplying, but nobody can say whether one got better or worse

On 11 November 2026, AWS open-sourced UAEF — the Universal Agent Evaluation Framework — on its Machine Learning Blog. It answers a specific question: you changed a prompt, swapped a model, added a tool — did the agent get better or worse? Most teams cannot answer that, not because they never test, but because every framework change forces the evaluation tooling to be rebuilt, and scores from different setups cannot be compared.

UAEF works like this: whether the agent was built with LangGraph, LangChain, Strands Agents or Amazon Bedrock AgentCore, its run record is first translated into one common format, then scored with the same 36 metrics across 7 dimensions. Scores are stored, compared against the previous version, and the size and location of any change are visible at a glance. The code is open source under Apache 2.0 in the awslabs/uaef repository.

This piece covers five things: where the problem actually lies, how UAEF builds its "one ruler", what the 36 metrics measure, how it stops the judge model from being talked into a higher score, and whether builders can adopt it right away.

Where the problem lies: every framework keeps its own books

Building agents today means choosing between frameworks. LangGraph, LangChain, Strands Agents and Bedrock AgentCore each have their own way of writing things — and their own way of recording a run: how messages are logged, how tool calls are logged, how handoffs between agents are logged. Evaluation tooling tends to follow the framework: metrics, scoring logic and result storage are written against one trace format. Switch frameworks, or run agents on more than one, and the evaluation has to be rebuilt.

That makes a basic question hard to answer: after changing the prompt, model or tools, did the agent improve or regress? Answering it needs two things: one consistent set of metrics across versions of an agent, and an archive of past results to compare against. Without either, iteration runs on feel. The AWS blog adds that agent quality is not something a single accuracy number can capture either — tool calling, response quality, responsible AI, performance, multi-turn conversation, multi-agent coordination and reasoning can each go wrong on their own, and a single score hides most of that.

According to the blog, before the public release, 12 organizations in telecom, healthcare and financial services had already used UAEF to evaluate their agents and compare versions before release. That number is the vendor's own claim, with no third-party list behind it, but it does suggest UAEF ran in real teams for a while before being open-sourced, rather than arriving as fresh demo code.

How UAEF works: one handwriting first, then the scores

UAEF's core is a simple separation: normalize the format first, score second. Adapters handle step one: they read the record of an agent's run — its messages, tool calls and agent-to-agent handoffs — and convert it into one canonical representation called the AgentTrace. Every step after that — metrics, scoring, comparison, reports — only ever sees the AgentTrace, never the framework's native format. So whichever framework built the agent, the scores are measured on the same ruler.

Adapters currently cover LangGraph, LangChain, Strands Agents, Amazon Bedrock AgentCore and Langfuse, plus a generic JSON adapter for other frameworks through a custom schema. A trace reaches UAEF two ways: an existing record is handed in from a log or an observability store, or UAEF runs the agent itself — given a runtime ARN or an HTTP endpoint, it calls the agent on each dataset input, collects the response, and adapts it in one step.

One evaluation runs as a closed loop: the adapter normalizes the format, then the trace is screened for personally identifiable information — email addresses, phone numbers, credit card numbers — before anything is stored. The evaluation engine scores it, and the result is stored as an experiment in Amazon DynamoDB and Amazon S3: DynamoDB holds the per-experiment metadata used for listing and comparison, S3 holds the full results. Finally, reports explain what changed and why, and the team changes the prompt, model or tools and evaluates again. The official blog draws this as a diagram titled "closed-loop data flow".

UAEF closed-loop evaluation: adapters normalize trace formats, the engine scores with 36 metrics, results are stored, baselines are compared, reports feed the next iteration

Seven dimensions, 36 metrics: one ruler measuring seven things

The evaluation engine's 36 built-in metrics sit in 7 dimensions, and every metric returns a normalized score between 0 and 1, where 1 is best, so dimensions and the overall result share one scale. Each dimension watches one kind of failure. Tool calling checks whether the agent selected the right tools, called them in a sensible order and passed valid arguments. Response quality checks whether the final answer is relevant, complete, accurate and grounded in the provided context rather than hallucinated. Responsible AI covers safety, bias, toxicity and resistance to prompt injection. Performance covers operational cost in latency, token use and throughput. Multi-turn covers sustaining a conversation, including context retention, coherence and goal completion across turns. Multi-agent covers coordination in systems of cooperating agents, such as task delegation and division of work. Reasoning covers the internal logic of the agent's chain of thought, including consistency and freedom from fallacies.

Metrics come in two families. Deterministic metrics compute a score directly from the trace and the ground truth and reproduce exactly on repeated runs — tool calling and performance, for example. LLM-as-a-judge metrics score qualities that resist a closed-form definition, such as response relevance or safety, by prompting a judge model on Amazon Bedrock. You can also add custom metrics or draw on the RAGAS and DeepEval integrations.

One practical detail: ground truth is optional. Only metrics that compare the agent's response against a reference need it; evaluating tool calling additionally requires the ground truth to include the expected tool calls — which tools the agent should call, in what order, and with what arguments. Datasets without ground truth can still be evaluated, because metrics that need a reference answer are skipped automatically. So production traffic can be scored without labelling every line first.

The section most worth reading closely: the judge can be fooled too

The most interesting part of the whole blog is not the metric list but the paragraphs on the judge model. The problem is this: the judge reads the agent's output and scores it, so if the agent can influence its judge, it can talk its way into a higher score. AWS handles it in four layers: the rubric is sent separately from the content, and the content is escaped so it cannot break out of its delimiter and be read as instructions; each judge response is parsed strictly and must cite a verbatim span of the content it scored; a score whose evidence is not found is rejected rather than recorded.

More important is the final line: judge scores are evaluation signal, not a safety decision, and should not gate a security control. Runtime enforcement belongs in a separate Amazon Bedrock Guardrails invocation path. The full model is in the repository's SECURITY.md. The subtext is clear: scores can tell you where an agent is weak, but they must not decide "ship or block". Many teams wire evaluation scores straight into a release gate; UAEF's authors wrote a warning in advance: do not do that.

This is also where this piece differs most from older writing on "scoring agents". Scoring is not the hard part — making the scores trustworthy is. UAEF builds anti-gaming into the design instead of appending a note that "judges may be biased".

How regressions are counted: a 5% move lights up

Once scores are stored, UAEF does a second job: comparison. Each evaluation can be stored as an experiment, and an experiment contains runs, each recording the agent design under test — framework, model, tools, system prompt and parameters — plus a configuration snapshot and a Git commit. You can mark one run as the baseline, so each score is tied to the change that produced it: a new prompt, a different model, or an added tool.

The comparison itself is straightforward: for each metric two runs share, it calculates the absolute and percentage change, then classifies the metric as a regression, an improvement or unchanged, using a configurable threshold with a default of 5 percent. Regressions also get a severity: minor at 5–10 percent, moderate at 10–20 percent, severe above 20 percent. Regressions are grouped by dimension, so one glance shows whether a prompt change hurt tool calling or response quality. In the web UI, the Compare tab lists stored experiments; the first selected is the baseline, and per-metric scores appear with regressions and improvements highlighted, using the same comparison engine as the Python library, so results match on both sides.

Reports come in three layers, answering three different questions. The results view covers "how did this run perform": the overall score, the number of test cases, the framework, queries that failed to run, per-metric averages with a dimension radar chart and a metric bar chart, and a table of every test case with its query, response, expected answer and score. A multi-turn conversation shows each turn with its own scores and the tools called. The detailed statistical report covers "why did it score that way": breakdowns by dimension and metric, confidence intervals and outliers, score distributions, correlations between metrics, a responsible AI summary and failing test cases, closing with recommendations pointing at the lowest-scoring dimensions and the metrics that vary most. That report is computed from the scores alone, without a language model, so the same results produce the same report. The third layer is LLM-generated insight: a Bedrock model analyzes the report for strengths, critical weaknesses, hypotheses about root causes and prioritized recommendations — and for multi-run comparisons, the overall trend, likely causes of regressions and improvements, the risk each regression carries and next steps.

Two ways to use it, and the real costs

UAEF can be used two ways over the same engine, writing to the same stores. With the Python library you evaluate agents in code from notebooks, scripts and CI/CD pipelines. With the deployable AWS service you evaluate through a low-code web interface: upload a dataset, select a framework and metrics, run the evaluation and review the scores. Start in the interface and move to code later without changing how results are scored or stored.

But "open source" does not mean "zero cost". Three bills are worth seeing before adopting it. First, the environment needs Python 3.10 or later and the uv package manager, and judging plus persistence need AWS credentials — judges run on Bedrock, results live in DynamoDB and S3. Second, stored things need tidying up yourself: the experiment table defaults to uaef-experiments and the results bucket to uaef-results, and if you deploy the service and later tear it down, the DynamoDB tables, S3 buckets and the Cognito user pool are created with a retain policy, so deleting the stack leaves them — and their contents — in place, and the storage keeps costing you until you remove it yourself. Third, the repository is still early: it distributes a 0.2.0 wheel and, as of 13 November, had no tagged releases. The first two points come from the official blog itself; the third comes from a third-party check on 13 November.

One more official caveat deserves to be pulled out on its own: UAEF is a reference architecture, designed to run inside a platform that supplies user and tenant isolation. Direct multi-tenant deployment without that host enforcement is unsupported. Fine for personal projects and internal team use; serving it to outside customers means building the isolation layer first.

For builders: pick metrics from the scenario, do not run all 36

The official blog includes a playbook, and its message is blunt: start from your scenario, not the metric list. A travel-booking agent and a policy Q&A agent fail in different ways, and scoring both on every dimension wastes time and Bedrock invocations. The blog's table lists starting metrics per scenario: RAG and knowledge Q&A, general Q&A, tool-using agents, multi-turn conversation, reasoning, safety and compliance, multi-agent, and production monitoring each get their own starting set. Once chosen, pass the metric names as the metrics argument to evaluate() or batch_evaluate().

Datasets are lighter than expected: a CSV or XLSX file, one row per test case, with only the Question column required. Answer, context, expected_tool_calls as a JSON array, session_id and turn_id are all optional. Multi-turn datasets group rows sharing a session_id into one conversation, ordered by turn_id. Before sharing or persisting a dataset drawn from real traffic, the blog recommends running uaef.security.PIIDetector over it to detect and redact personally identifiable information first.

The pipeline has six independent choices, laid out as a table in the blog: passive or active (evaluate a trace you have, or run the agent), one trace or a dataset, single-turn or multi-turn, single-agent or multi-agent (automatic), offline or online (OnlineEvaluator for live conversations), ephemeral or persisted. Only persisted runs can be compared — worth deciding before you start, not after.

Running it from zero: one dataset, one call to evaluate

A first evaluation with UAEF is lighter than it looks. Prepare a CSV or XLSX file, one test case per row, with only the Question column required; Answer, context, expected_tool_calls, session_id and turn_id are all optional. Then call evaluate() for one record or batch_evaluate() for the whole dataset, passing your chosen metric names in the metrics argument. To keep the run for later comparison, add persist=True and an experiment name, and the results go into DynamoDB and S3.

The official repository ships a runnable notebook for each supported framework — LangGraph, LangChain, Strands Agents, Bedrock AgentCore and Langfuse — plus one for the generic JSON adapter, one for the RAGAS and DeepEval integrations, and others for experiment management, the reporting pipeline and PII detection. Working through the notebooks is enough to learn the tool. Live monitoring has a ready entry point too: OnlineEvaluator scores real-time conversations turn by turn, and offline testing and online monitoring run on the same engine, so scores never disagree between the two.

For an individual developer, the most practical way to start is to take an agent you already run, pick three to five metrics, and score a batch of real conversations you already have. You do not even need ground truth to begin — metrics that need a reference answer are skipped automatically, so you see deterministic scores for tool calling and performance first, then decide whether to add reference answers or switch on judge models. The hardest part of evaluation is always the first run; UAEF brings that first run down to one dataset and one function call.

How this relates to what this site has already published

"Scoring agents" is not a new topic on this site, so the boundaries need drawing. The 19 October Microsoft Scope piece covered Microsoft's open-source evaluation tool, anchored on "giving the agent a test paper" — testing whether an agent's product experience can be turned into repeatable experiments in a DevAx setting. This piece's anchor is different: UAEF's core is cross-framework trace normalization — converting run records from different frameworks into one format, scoring them with one set of metrics, and tracking regressions across versions. Scope is about "how to test"; UAEF is about "different classes sitting the same exam, with scores comparable across versions".

The other two are hands-on guides for readers, not news: one on building an evaluation harness for an AI agent, one on an evals system for vibe projects, both teaching the same methodology — "without evals, every model swap is a gamble". This piece is a specific open-source release from 11 November, about what AWS's tool looks like, what it can do and where it falls short. Guides teach you how to build evaluation; news tells you a ready-made evaluation tool has arrived. They are read differently and do not replace each other. Read this piece, then go back to those two guides — UAEF makes a good hands-on subject for trying them out.

Views 0Comments 0

Comments (0)

ME
0/1000
Loading comments...

Sources

Browse projectsPublish your project