feat: Third-party eval metrics adapter (DeepEval + Autoevals) with strands-evals mappers by stone-coding · Pull Request #568 · aws/bedrock-agentcore-sdk-python

stone-coding · 2026-07-06T17:07:35Z

Summary

Add DeepEvalAdapter and AutoevalsAdapter that integrate third-party evaluation metrics with
AgentCore's code-based evaluator framework.

Span mapping via strands-evals: Uses detect_otel_mapper() from strands-agents-evals for
auto-detection of Strands, OpenInference LangChain, and OpenTelemetry LangChain span formats.
Bridge function converts Session → SpanMapResult for adapter consumption.
custom_mapper parameter lets customers bypass built-in mappers for unsupported frameworks:
- DeepEval: custom_mapper: Callable[[EvaluatorInput], LLMTestCase]
- Autoevals: custom_mapper: Callable[[EvaluatorInput], Dict[str, Any]]
tools_called + expected_tools extraction from execute_tool spans — enables
ToolCorrectnessMetric and ArgumentCorrectnessMetric
assertions from reference_inputs mapped to LLMTestCase.context → should now say assertions from reference_inputs mapped to LLMTestCase.context (None when no assertions)
reference_inputs full pipeline: expectedResponse → expected_output,
expectedTrajectory → expected_tools, assertions → context
Never raises unhandled exceptions — all error paths return valid EvaluatorOutput with
structured errorCode/errorMessage
Pinned dependency: strands-agents-evals>=1.0.0,<2.0.0

Test plan

38 unit tests passing (pytest tests/.../third_party/ -v)
21 E2E tests passing across:
- DeepEval: 12 metrics (RAG, Agentic, Custom/GEval, Contextual, Non-LLM, Safety + OpenInference +
  OpenTelemetry)
- Autoevals: 6 metrics (LLM-Judge, LLM-Scorer, Deterministic + OpenInference + OpenTelemetry)
- Custom mapper: 3 tests (Google ADK, OpenAI Agents, Autoevals path)
E2E validated against live AgentCore service (evaluate() API → Lambda → metric → score)

Introduces a new integrations/deepeval/ module that adapts AgentCore Lambda evaluation events into DeepEval LLMTestCase objects, runs any BaseMetric, and returns structured score/label/explanation responses.

…leTurnParams deprecation

…d EvaluatorInput support

… layer, simplify per TJ/Irene feedback

…ion, add validate_fields to DeepEvalAdapter

…xpected_response_text property

- Rename span_parsers → span_mappers, simplify to Strands-only - Rename field_mapper → customer_mapper across all adapters - customer_mapper now returns native types directly: - DeepEval: EvaluatorInput → LLMTestCase - Autoevals: EvaluatorInput → Dict[str, Any] (eval kwargs) - Refactor BaseAdapter: remove intermediate execute(), add _run() pattern - 83 unit tests passing

chianhw · 2026-07-08T18:56:48Z

+            customer_mapper=lambda ev: {
+                "input": ev.session_spans[0]["attributes"]["question"],
+                "output": ev.session_spans[0]["attributes"]["answer"],
+                "expected": "the expected answer",
+            },


Do we need to accept aws lambda?

Replaced lambda examples with named functions in docstrings

chianhw · 2026-07-08T20:16:27Z

+    def __init__(
+        self,
+        scorer: Any,
+        customer_mapper: Optional[Callable[[EvaluatorInput], Dict[str, Any]]] = None,


We have not updated this type Dict[str, Any]?

Regarding naming, "Custom mapper" would be a better choice for describing an arbitrary mapper provided by a customer?

1.The type Dict[str, Any] is correct — Autoevals has no typed
input class like DeepEval's LLMTestCase. Scorers take plain
kwargs (input, output, expected) and the dict gets unpacked
as metric.eval(**kwargs). Different scorers need different
keys, so a generic dict is the right contract.

chianhw · 2026-07-08T21:00:07Z

+        self,
+        scorer: Any,
+        customer_mapper: Optional[Callable[[EvaluatorInput], Dict[str, Any]]] = None,
+        threshold: float = 0.5,


label: Optional[str] = None label in EvaluatorOutput is optional.
We don't need to have a default value to generate a label when a metric doesn't have a label and the user doesn't provide any threshold. Can we set default as None?

Changed threshold default to None. When no threshold is set,
no Pass/Fail judgment is made. Note: EvaluatorOutput's
validator currently requires a label for success responses —
I'm returning the score as the label string when threshold is
None. Let me know if you'd rather relax the validator to
allow label=None.

chianhw · 2026-07-08T21:00:37Z

+                mapping when provided. Expected keys: input, output, expected (optional).
+            threshold: Score threshold for Pass/Fail determination. Defaults to 0.5.
+        """
+        self.scorer = scorer


this naming is not aligned with DeepEval adaptor. Looks like you haven't updated AutoevalsAdapter.
If you haven't completed end to end tests for AutoevalsAdapter, you should not include it in your PR.

Working on E2E test plan now. Considering how to strcuture parameterized Lambda handlers and how to obtains span from strands, openinference, opentelemetry, and customer mappers.

chianhw · 2026-07-08T21:00:43Z

+            customer_mapper=lambda ev: LLMTestCase(
+                input=ev.session_spans[0]["attributes"]["user_query"],
+                actual_output=ev.session_spans[0]["attributes"]["response"],
+            ),


using lambda? same as above

chianhw · 2026-07-08T21:00:49Z

+    def __init__(
+        self,
+        metric: BaseMetric,
+        customer_mapper: Optional[Callable[[EvaluatorInput], LLMTestCase]] = None,


naming? same as above

jariy17

Lefts some comments but its a good addition to our SDK.

jariy17 · 2026-07-20T16:20:12Z

+
+from bedrock_agentcore.evaluation.custom_code_based_evaluators.third_party.autoevals.adapter import AutoevalsAdapter
+
+__all__ = ["AutoevalsAdapter"]


nit: It should be AutoEvals

Fixed — renamed to AutoEvalsAdapter throughout.

jariy17 · 2026-07-20T16:21:30Z

+            threshold: Optional score threshold for Pass/Fail label. If None, label
+                is omitted from the output.
+        """
+        self.metric = metric


Can this support custom evaluators?

Yes, metric accepts any object with an .eval(output, expected, **kwargs) method. Customers can pass their own scorer class alongside the built-in ones.

jariy17 · 2026-07-20T16:43:24Z

+        scope_name = get_scope_name(span)
+        if scope_name == SCOPE_AMAZON_OTEL_LANGCHAIN:
+            return LangChainOtelSessionMapper()
+        if scope_name == "opentelemetry.instrumentation.langchain":


nit: you can combine this and the above if statements together

Fixed, combined into if scope_name in (SCOPE_AMAZON_OTEL_LANGCHAIN, "opentelemetry.instrumentation.langchain").

jariy17 · 2026-07-20T17:05:32Z

+        metric = BiasMetric(threshold=0.5)
+        adapter = DeepEvalAdapter(
+            metric=metric,
+            field_mapper=lambda ev: {


does this exist? This should be custom_mapper.

Fixed, renamed to custom_mapper and updated return type to LLMTestCase.

jariy17 · 2026-07-20T17:06:04Z

+        from bedrock_agentcore.evaluation.custom_code_based_evaluators.third_party.autoevals import AutoevalsAdapter
+
+        scorer = Factuality()
+        adapter = AutoevalsAdapter(scorer=scorer)


I think this scorer is suppose to be metric.

Fixed, rename to metric

jariy17 · 2026-07-20T17:50:06Z

+        try:
+            self.metric.measure(test_case)
+        except Exception as e:
+            if type(e).__name__ == "MissingTestCaseParamsError":


type(e).__name__ == "MissingTestCaseParamsError" - we already hard-import deepeval at the top of this file, so just import the exception class and use isinstance. String-matching on class name is fragile.

Also, pyproject.toml pins deepeval>=2.0.0 with no upper bound. If DeepEval renames or moves this class in a future release, this check silently returns False, falls through to the base except Exception, and every MISSING_REQUIRED_FIELD becomes METRIC_ERROR.

Fixed, imported MissingTestCaseParamsError from deepeval.errors and using except MissingTestCaseParamsError directly. No more string matching.

jariy17 · 2026-07-20T17:51:39Z

+        try:
+            self.metric.measure(test_case)
+        except Exception as e:
+            if type(e).__name__ == "MissingTestCaseParamsError":


type(e).__name__ == "MissingTestCaseParamsError" - we already hard-import deepeval at the top of this file, so just import the exception class and use isinstance. String-matching on class name is fragile.

Also, pyproject.toml pins deepeval>=2.0.0 with no upper bound. If DeepEval renames or moves this class in a future release, this check silently returns False, falls through to the base except Exception, and every MISSING_REQUIRED_FIELD becomes METRIC_ERROR.

Fixed, imported MissingTestCaseParamsError from deepeval.errors and using except MissingTestCaseParamsError directly. No more string matching.

jariy17 · 2026-07-20T17:52:52Z

    "requests>=2.31.0",
 ]
+deepeval = [
+    "deepeval>=2.0.0",


registry.py imports strands_evals but neither the deepeval nor autoevals extras include strands-agents-evals. A customer following the advertised install path (pip install bedrock-agentcore[deepeval]) gets an ImportError at cold start. Add strands-agents-evals>=1.0.0,<2.0.0 to both extras.

Fixed, added strands-agents-evals>=1.0.0,<2.0.0 to both deepeval and autoevals extras.

jariy17 · 2026-07-20T17:54:35Z

+    expected_tools: Optional[List[Dict[str, Any]]] = None
+    assertions: Optional[List[str]] = None
+
+    def to_dict(self) -> Dict[str, Any]:


is this being used? Please remove if it's not

jariy17 · 2026-07-20T17:54:48Z

+        return result
+
+
+def _try_parse_json(val: str) -> Any:


is this dead code? PLease remove

- Rename AutoevalsAdapter → AutoEvalsAdapter (class + all imports) - Fix integ tests: scorer→metric, field_mapper→custom_mapper, correct return types - Combine LangChain scope if-statements into single check - context=None when no assertions (don't use tool outputs as ground truth) - Import MissingTestCaseParamsError properly, use isinstance() not string matching - Add strands-agents-evals>=1.0.0,<2.0.0 to deepeval and autoevals extras - Create FieldExtractionError(ValueError) for scoped exception handling - Remove dead code: to_dict(), _try_parse_json - Document trace-level only limitation in _session_to_span_map_result docstring

…SAdapter to use the new BaseAdapter._run() pattern with SpanMapResult, custom_mapper (replacing field_mapper), and CloudWatch split format spans. Update tests to use the new span format and API.

- Add `turns` field to SpanMapResult for multi-turn session data - Build turns from all AgentInvocationSpans across traces in registry - Branch on BaseConversationalMetric in DeepEvalAdapter to build ConversationalTestCase with Turn objects - Return FieldExtractionError when single-turn spans used with conversational metrics - Remove SCOPE_AMAZON_OTEL_LANGCHAIN workaround (no longer needed after Irene sync — not a supported path) - Set context=None instead of duplicating retrieval_context (per TJ)

haomiao037 added 14 commits June 30, 2026 11:59

Add DeepEvalHandler integration with unit tests

77d16a9

Introduces a new integrations/deepeval/ module that adapts AgentCore Lambda evaluation events into DeepEval LLMTestCase objects, runs any BaseMetric, and returns structured score/label/explanation responses.

Fix span extraction to use real AgentCore _eval_log_records structure

402ea78

Set context field from tool messages for HallucinationMetric support

e9ef47d

Use metric.success for label instead of manual threshold comparison

f97827e

Add model override and timeout enforcement to DeepEvalHandler

9d256ae

Add model override, timeout enforcement, use metric.success, fix Sing…

c142d50

…leTurnParams deprecation

Fix _get_required_params to handle GEval unmappable typing params

3ccc98c

Add .deepeval/ to gitignore

a884f91

Move model override to init to avoid per-call mutation

6ac198c

Refactor to BaseAdapter framework with DeepEval/Autoevals adapters an…

8d415e5

…d EvaluatorInput support

Major refactor: move to custom_code_based_evaluators, add span parser…

8627ab0

… layer, simplify per TJ/Irene feedback

Fix review items: add reference_inputs to model, tighten error detect…

9a9b6a7

…ion, add validate_fields to DeepEvalAdapter

Adapt to upstream ReferenceInput model: remove duplicate field, use e…

0499d4b

…xpected_response_text property

stone-coding requested a review from a team July 6, 2026 17:07

stone-coding requested a deployment to manual-approval July 6, 2026 17:07 — with GitHub Actions Waiting

chianhw reviewed Jul 8, 2026

View reviewed changes

stone-coding requested a deployment to manual-approval July 16, 2026 20:59 — with GitHub Actions Waiting

jariy17 requested changes Jul 20, 2026

View reviewed changes

stone-coding requested a deployment to manual-approval July 20, 2026 19:56 — with GitHub Actions Waiting

stone-coding requested a deployment to manual-approval July 22, 2026 00:09 — with GitHub Actions Waiting


		from bedrock_agentcore.evaluation.custom_code_based_evaluators.third_party.autoevals.adapter import AutoevalsAdapter

		__all__ = ["AutoevalsAdapter"]

Uh oh!

Conversation

stone-coding commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jariy17 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

stone-coding commented Jul 6, 2026 •

edited

Loading