LocalEvalService discards per-invocation results for any metric reporting EvalStatus.NOT_EVALUATED -- is there an intended shape for non-pass/fail (measurement) metrics?

[gaurav-gandhi-2411]

🔴 Required Information

Describe the Bug:
LocalEvalService._evaluate_metric_for_eval_case (local_eval_service.py:428-436)
discards a metric's real per-invocation PerInvocationResult (score, rubric_scores)
and substitutes an empty one whenever EvaluationResult.overall_eval_status == EvalStatus.NOT_EVALUATED:

invocation_result = (
    evaluation_result.per_invocation_results[idx]
    if evaluation_result.overall_eval_status != EvalStatus.NOT_EVALUATED
    else PerInvocationResult(actual_invocation=invocation.actual_invocation)
)

NOT_EVALUATED is also the default value of EvaluationResult.overall_eval_status
and PerInvocationResult.eval_status (confirmed via model_fields — both default to
EvalStatus.NOT_EVALUATED), so this fires for any custom Evaluator that reports a
real per-invocation measurement without opting into ADK's score >= threshold -> PASSED pass/fail convention — which is the natural, low-friction thing to do for a
metric that isn't pass/fail by nature (a cost, a token count, a latency).

Downstream, AgentEvaluator._process_metrics_and_get_failures reads only from
these now-nulled per-invocation results to decide whether to raise. Since a metric
that never reports PASSED can never produce a non-empty scores list there, it
unconditionally raises AssertionError for any real value the metric computed —
no threshold configuration avoids this, because the per-invocation score was already
discarded before the threshold comparison happens.

I ran into this building a third-party ADK metric that reports real per-invocation
dollar cost (lower-is-better, so it deliberately never participates in ADK's
higher-is-better pass/fail gate — see "Additional Context" below). It surfaced two
separate effects:

1. AgentEvaluator.evaluate() always raises, regardless of the actual computed
   value or the configured threshold.
2. adk eval (CLI) doesn't raise, but the per-invocation table and the persisted
   eval_history/*.evalset_result.json both show score: null /
   rubric_scores: null for every invocation. Only one coarse, un-persisted
   console line (the aggregate Metric: ... Score: X) carries the real number —
   nothing about why a score is present/absent, or any per-call breakdown, survives
   anywhere a user can read after the run.

Steps to Reproduce:
Minimal, self-contained repro below (no third-party packages) — a toy metric that
reports a real per-invocation number (42.0) via a custom Evaluator, registered the
documented way via DEFAULT_METRIC_EVALUATOR_REGISTRY.register_evaluator, deliberately
leaving eval_status at its default (NOT_EVALUATED, since the metric isn't pass/fail).

my_agent/agent.py:

from __future__ import annotations
from collections.abc import AsyncGenerator
from google.adk.agents.llm_agent import LlmAgent
from google.adk.evaluation.eval_case import ConversationScenario, Invocation
from google.adk.evaluation.eval_metrics import EvalMetric, Interval, MetricInfo, MetricValueInfo
from google.adk.evaluation.evaluator import EvaluationResult, Evaluator, PerInvocationResult
from google.adk.evaluation.eval_rubrics import RubricScore
from google.adk.evaluation.metric_evaluator_registry import DEFAULT_METRIC_EVALUATOR_REGISTRY
from google.adk.models.base_llm import BaseLlm
from google.adk.models.llm_request import LlmRequest
from google.adk.models.llm_response import LlmResponse
from google.genai import types as genai_types

METRIC_NAME = "toy_measurement"

class ToyMeasurementEvaluator(Evaluator):
    """Reports a real per-invocation number, but is not pass/fail (e.g. a cost)."""
    def __init__(self, *, eval_metric: EvalMetric) -> None:
        self._eval_metric = eval_metric

    def evaluate_invocations(self, actual_invocations, expected_invocations=None,
                              conversation_scenario=None) -> EvaluationResult:
        per_invocation_results = [
            PerInvocationResult(
                actual_invocation=inv, expected_invocation=None, score=42.0,
                rubric_scores=[RubricScore(rubric_id="toy", score=42.0,
                                            rationale="the real breakdown")],
            )
            for inv in actual_invocations
        ]
        return EvaluationResult(
            overall_score=sum(r.score for r in per_invocation_results),
            per_invocation_results=per_invocation_results,
            # overall_eval_status intentionally left unset -> defaults to
            # NOT_EVALUATED, which is correct: this metric is not pass/fail.
        )

DEFAULT_METRIC_EVALUATOR_REGISTRY.register_evaluator(
    metric_info=MetricInfo(
        metric_name=METRIC_NAME,
        description="Toy non-pass/fail measurement metric.",
        metric_value_info=MetricValueInfo(
            interval=Interval(min_value=0.0, max_value=1_000_000.0, open_at_max=True)),
    ),
    evaluator=ToyMeasurementEvaluator,
)

class _FakeLlm(BaseLlm):
    model: str = "fake-model"
    @classmethod
    def supported_models(cls) -> list[str]:
        return ["fake-model"]
    async def generate_content_async(self, llm_request: LlmRequest, stream: bool = False):
        yield LlmResponse(content=genai_types.Content(
            parts=[genai_types.Part(text="ok")], role="model"))

root_agent = LlmAgent(name="toy_agent", model=_FakeLlm(), instruction="Answer briefly.")

my_agent/__init__.py:

from . import agent

Driver:

import asyncio, json
from pathlib import Path
from google.adk.evaluation.eval_case import EvalCase, Invocation
from google.adk.evaluation.eval_set import EvalSet
from google.genai import types as genai_types

eval_case = EvalCase(eval_id="case_1", conversation=[
    Invocation(user_content=genai_types.Content(parts=[genai_types.Part(text="hi")], role="user"))
])
Path("eval_set.json").write_text(EvalSet(eval_set_id="repro_set", eval_cases=[eval_case]).model_dump_json())
Path("test_config.json").write_text(json.dumps({"criteria": {"toy_measurement": 999999.0}}))

async def main():
    from google.adk.evaluation.agent_evaluator import AgentEvaluator
    await AgentEvaluator.evaluate(
        agent_module="my_agent",
        eval_dataset_file_path_or_dir="eval_set.json",
        num_runs=1, print_detailed_results=True,
    )

asyncio.run(main())

1. Install google-adk[eval]==2.6.3.
2. Run the driver script above (agent_module_file_path layout: my_agent/__init__.py + my_agent/agent.py, matching the standard convention).
3. Observe the raised AssertionError.

Expected Behavior:
A metric that computes a real per-invocation value (42.0 in the repro) and
deliberately reports NOT_EVALUATED because it isn't pass/fail should be able to
surface that value through AgentEvaluator.evaluate()/adk eval — at minimum
without crashing, and ideally with the per-invocation score and rationale intact
in both the printed table and the persisted eval-history JSON.

Observed Behavior:

Summary: `EvalStatus.NOT_EVALUATED` for Metric: `toy_measurement`. Expected threshold: `999999.0`, actual value: `None`.
+----+--------------------------+---------+-------------+----------+---------------------+-------------------+-----------------------+---------------------+
|    | eval_status              | score   |   threshold | prompt   | expected_response   | actual_response   | expected_tool_calls   | actual_tool_calls   |
+====+==========================+=========+=============+==========+=====================+===================+=======================+=====================+
|  0 | EvalStatus.NOT_EVALUATED |         |      999999 | hi       |                     | ok                |                       |                     |
+----+--------------------------+---------+-------------+----------+---------------------+-------------------+-----------------------+---------------------+
RESULT: evaluate() raised AssertionError:
  Following are all the test failures.
toy_measurement for None Failed. Expected 999999.0, but got None.

The metric's evaluate_invocations() was called and did return score=42.0 per
invocation (verifiable by instrumenting ToyMeasurementEvaluator.evaluate_invocations
directly) — the value is computed correctly and then discarded before it reaches any
output.

Environment Details:

• ADK Library Version: google-adk==2.6.3
• Desktop OS: Windows 11
• Python Version: 3.13.5

Model Information:

• Are you using LiteLLM: No
• Which model is being used: N/A (repro uses a fake BaseLlm, no real model call needed)

---

🟡 Optional Information

Regression:
Not a regression as far as I can tell — local_eval_service.py:428-436's
NOT_EVALUATED branch appears intentional (it's guarding against evaluators that
return zero per-invocation results), it just also catches evaluators that return a
full, real per-invocation result set while legitimately reporting NOT_EVALUATED as
their permanent status.

Additional Context — the underlying design question:
This isn't only a bug report — I'd like to understand the intended contract here.
As far as I can tell, ADK's Evaluator/EvalStatus model has no way to express
"this metric measures something real per invocation, but that measurement isn't
pass/fail" — EvalStatus is PASSED | FAILED | NOT_EVALUATED, and
NOT_EVALUATED is treated by LocalEvalService/AgentEvaluator as "nothing to
report," not as "reported, deliberately not gated." Silently negating a
lower-is-better score to force a "pass" would misrepresent it to anyone reading the
result, so NOT_EVALUATED is the only status that doesn't lie — but that value turns
out to mean "discard everything" a few layers down, not "measured, not gated."

Is a non-pass/fail "measurement" metric (cost, token count, latency, or similar)
something ADK's Evaluator contract is meant to support today? If so, what's the
recommended shape for it — a distinct EvalStatus variant, a flag on EvalMetric/
MetricInfo that opts a metric out of the threshold-gating and per-invocation-nulling
behavior, something else? I'm happy to be pointed at existing conventions I missed,
or to help prototype once there's a direction — I just don't want to guess at API
design that maintainers would need to carry.

How often has this issue occurred?:
Always (100%) — deterministic given the repro above.