Retry transient network errors in background token monitor

[reyortiz3]

Summary

• OAuth token refresh fails when WARP VPN disconnects overnight — the refresh endpoint becomes temporarily unreachable from the home ISP IP, causing the workload to be immediately marked as unauthenticated and requiring manual intervention.
• Adds isTransientNetworkError to distinguish DNS failures, TCP-level errors, and timeouts (transient) from OAuth 4xx / invalid_grant responses (non-transient). The background monitor now retries transient failures with exponential backoff (30 s initial → 5 min max, no deadline) until the network recovers. Context cancellation exits the loop cleanly without marking the workload as unauthenticated. Non-transient errors still fail immediately as before.

Type of change

• Bug fix

Test plan

• Unit tests (task test)

Changes

File	Change
pkg/auth/monitored_token_source.go	Add isTransientNetworkError, exponential backoff retry loop in onTick, injectable newRetryBackOff for tests
pkg/auth/monitored_token_source_test.go	Tests for isTransientNetworkError and the three retry scenarios: recovery, context cancellation, transient→non-transient escalation
go.mod / go.sum	Add github.com/cenkalti/backoff/v5 (direct dep), bump otel exporters and grpc-gateway (incidental go mod tidy updates)

Does this introduce a user-facing change?

No — the workload stays authenticated through VPN reconnects instead of going unauthenticated overnight. No CLI or config changes.

Special notes for reviewers

• MaxElapsedTime is intentionally left at zero (no deadline). The retry loop runs until the context is cancelled or the network recovers. The motivation: we don't know how long a VPN disconnect will last, and a hard cutoff would reproduce the original problem.
• The newRetryBackOff field is a test seam; it is nil in production and replaced with a fast backoff in tests to avoid multi-second sleeps.

Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 56.70103% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.95%. Comparing base (04ae197) to head (2b4364d).

Files with missing lines	Patch %	Lines
pkg/auth/monitored_token_source.go	56.70%	34 Missing and 8 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4281      +/-   ##
==========================================
+ Coverage   68.77%   68.95%   +0.17%     
==========================================
  Files         473      473              
  Lines       47917    48001      +84     
==========================================
+ Hits        32956    33098     +142     
+ Misses      12300    12292       -8     
+ Partials     2661     2611      -50     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[amirejaz]

This retry logic is fully synchronous and per-request, which has important implications during transient failures:

• backoff.Retry(...) does not spawn a goroutine; it blocks the caller.
• As written, every incoming request during a transient outage will enter its own retry loop, each with its own backoff instance.

Implications

• Multiple concurrent callers will repeatedly call mts.tokenSource.Token() independently.
• There is no coordination or deduplication of token refresh attempts.
• When the network recovers, this can lead to a burst of simultaneous retries (“thundering herd”) rather than a single shared recovery.
• All callers remain blocked until retry succeeds, context is canceled, or backoff stops.