Getting OTEL-related error - Unable to find a definition for context relevance and answer faithfulness that works with my application

[emmanouilvoitsidis]

I have been trying to use TruLens for evaluation of my RAG-based application and I want to use three evaluation metrics: answer relevance (which works fine afaik), context relevance, and answer faithfulness (comparison of retrieved context with the answer).

I have tried many different definitions of these evaluation metrics (context relevance and faithfulness) and I always seem to get an error related to OTEL. I am using version trulens v2.3.1 but have also tried 2.2.0 (in the hope that OTEL won't be forced in that version). An example of an OTEL-related error can be seen here:
raise ValueError(
f"OTEL mode only supports dictionary selectors, not {type(selectors)}!"
)
ValueError: OTEL mode only supports dictionary selectors, not <class 'trulens.core.utils.serial.Lens'>!

When I try to address what seems to be the issue, e.g. "OTEL mode only supports dictionary selectors", another OTEL-related error appears. I have managed to disable OTEL, but then other issue appear, so I clearly need it.

An example of definitions for context relevance and faithfulness, that I have used is:
f_context_relevance = (
Feedback(provider.context_relevance_with_cot_reasons, name="Context Relevance")
.on_input()
.on({"context": Select.RecordCalls.retriever})
)

f_faithfulness = Feedback(
provider.relevance,
name="Answer Faithfulness"
).on({"context": Select.RecordCalls.retriever}).on_output()

I don't have much experience in TruLens and have been trying different things over the last few days to no avail. Any help would be much appreciated and I'd be happy to share more details if required.

[dosubot[bot]]

Hi @emmanouilvoitsidis! I'm Dosu and I’m helping the trulens team.

The OTEL error you’re seeing happens because, in OTEL mode which is on by default, TruLens only supports dictionary selectors (using the Selector class), not Lens objects. To define context relevance and answer faithfulness (groundedness) metrics that work with OTEL, you need to use Feedback objects with .on({...}) and Selector instances for each argument.

Here’s how you can define these metrics in a way that works with OTEL:

import numpy as np
from trulens.core import Feedback
from trulens.core.feedback.selector import Selector
from trulens.providers.openai import OpenAI

provider = OpenAI(model_engine="gpt-4.1-mini")

# Context relevance: compares the question and each context chunk
f_groundedness = (
    Feedback(
        provider.groundedness_measure_with_cot_reasons_consider_answerability,
        name="Groundedness",
    )
    .on_context(collect_list=True)
    .on_output()
    .on_input()
)

# Context relevance between question and each context chunk.
f_context_relevance = (
    Feedback(
        provider.context_relevance_with_cot_reasons, name="Context Relevance"
    )
    .on_input()
    .on_context(collect_list=False)
    .aggregate(np.mean)  # choose a different aggregation method if you wish
)

This approach avoids the Lens selectors and should resolve the OTEL error you encountered. For answer relevance, you can use a similar pattern, which you mentioned is already working for you.

Note: the use of on_context() requires you to have instrumented a retrieval method (example below) in your app.

class RAG:
    def __init__(self, model_name: str = "gpt-5"):
        self.model_name = model_name

    @instrument(
        span_type=SpanAttributes.SpanType.RETRIEVAL,
        attributes={
            SpanAttributes.RETRIEVAL.QUERY_TEXT: "query",
            SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: "return",
        },
    )
    def retrieve(self, query: str) -> list:
        results = vector_store.query(
            query_texts=query, n_results=4
        )  # Get more results
        return [doc for sublist in results["documents"] for doc in sublist]

If you want to disable OTEL mode, set the environment variable before importing any TruLens packages:

import os
os.environ["TRULENS_OTEL_TRACING"] = "0"

This is the supported way to turn off OTEL tracing in recent versions of TruLens citation.

For more details and working examples, check out the OTEL quickstart notebook and the official blog post.

If you run into another OTEL-related error, feel free to share the exact error message and your metric definition code—happy to help debug further!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}