fix: surface experiment-comparison2 fetch failures and retry transient 5xx - #640
Open
Praneeth Yenugutala (praneethy91) wants to merge 1 commit into
Conversation
…t 5xx
Experiment.summarize() swallowed any exception from the experiment-comparison2
fetch into an empty {} summary, making a transient fetch failure indistinguishable
from an experiment that genuinely has no scores. Separately,
RetryRequestExceptionsAdapter only retried on connection-level exceptions, so a
completed non-2xx response (e.g. a 502 from a proxy hiccup) never entered the
retry path at all.
- RetryRequestExceptionsAdapter now retries responses with a transient status
(429/500/502/503/504) using the same backoff as connection-level failures.
- ExperimentSummary gains scores_fetch_error, populated when the
experiment-comparison2 fetch fails, so callers gating automation on
summary.scores can distinguish "fetch failed" from "no scores".
Fixes braintrustdata#639
Abhijeet Prasad (AbhiPrasad)
self-requested a review
July 30, 2026 22:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #639.
Experiment.summarize()caught any exception from theexperiment-comparison2fetch and silently fell back to an empty{}summary, making a transient fetch failure indistinguishable from an experiment that genuinely has no scores. Meanwhile,RetryRequestExceptionsAdapteronly retried on connection-level exceptions (timeouts, connection resets), so a request that completed with a non-2xx status (e.g. a 502 from a proxy hiccup) never entered the retry path at all — it went straight toraise_for_status()and got swallowed bysummarize().RetryRequestExceptionsAdapter.send()now also retries when a completed response has a transient status code (429/500/502/503/504), using the same exponential backoff as the existing exception-based retry path. This should eliminate most one-off 5xx blips before they ever reachsummarize().ExperimentSummarygains a newscores_fetch_error: str | Nonefield, populated whenever theexperiment-comparison2fetch fails after retries are exhausted.scores/metricsstill default to{}for backwards compatibility, but callers building automation (e.g. CI gates) onsummary.scorescan now checkscores_fetch_errorto tell "fetch failed" apart from "no scores." The printedstr(summary)also surfaces aWARNINGline when this happens.Test plan
py/src/braintrust/test_http.py::TestRetryOnHttpErrorStatus— new tests covering retry-on-502, giving up afterbase_num_retries, and not retrying non-retryable statuses (404), using a real local HTTP server (no mocks), consistent with existing adapter tests in this file.py/src/braintrust/test_framework.py::test_experiment_summarize_surfaces_scores_fetch_error— new test assertingsummarize()populatesscores_fetch_error(and the printed summary contains aWARNING) when the comparison fetch raises.cd py && uv run nox -s test_core— full core suite passes (614 passed, 63 skipped, 12 xfailed).cd py && uv run nox -s pylint— clean.cd py && uv run pre-commit run --files ...on changed files — clean.