[ISSUE] `cannot reset reader of type io.nopCloserWriterTo` when retrying OIDC authentication

[orf]

Description
If the Databricks OIDC endpoint returns a retriable error, token authentication fails with cannot reset reader of type io.nopCloserWriterTo from here:

databricks-sdk-go/common/request.go

Lines 61 to 70 in 20cf572

	func (r RequestBody) Reset() error {
	if r.Reader == nil {
	return nil
	}
	if v, ok := r.Reader.(io.Seeker); ok {
	_, err := v.Seek(0, io.SeekStart)
	return err
	} else {
	return fmt.Errorf("cannot reset reader of type %T", r.Reader)
	}

This is because the google OIDC library creates a request here https://github.com/golang/oauth2/blob/acc38155b7f6f36aefcb58faff6f36d314dd915c/internal/token.go#L225 which wraps the body in a io.ReadCloser here https://github.com/golang/go/blob/b28808d838682b2911698fcb934813b02f39fa69/src/net/http/request.go#L908

Reproduction

package main

import (
	"bytes"
	"context"
	"encoding/json"
	"io"
	"net/http"
	"os"
	"path/filepath"

	"github.com/databricks/databricks-sdk-go"
)

type hc func(r *http.Request) (*http.Response, error)

func (cb hc) RoundTrip(r *http.Request) (*http.Response, error) {
	return cb(r)
}

func main() {
	fsDir, err := os.MkdirTemp("", "tree_*")
	if err != nil {
		panic(err)
	}
	defer os.RemoveAll(fsDir)
	filePath := filepath.Join(fsDir, "token")
	err = os.WriteFile(filePath, []byte("test-token"), os.ModePerm)
	if err != nil {
		panic(err)
	}

	cfg := &databricks.Config{
		AuthType: "file-oidc",
		OIDCTokenFilepath: filePath,
		Host: "https://accounts.cloud.databricks.com/",
		AccountID: "test-account",
		HTTPTransport: hc(func(r *http.Request) (*http.Response, error) {
			bodyBytes, err := json.Marshal(map[string]string{
				"error": "too many requests",
			})
			if err != nil {
				panic(err)
			}

			return &http.Response{
				StatusCode:    http.StatusTooManyRequests,
				Body:          io.NopCloser(bytes.NewReader(bodyBytes)),
				ContentLength: int64(len(bodyBytes)),
				Header: map[string][]string{
					"Content-Type": {"application/json"},
				},
				Request: r,
			}, nil
		}),
	}
	c, err := databricks.NewAccountClient(cfg)
	if err != nil{
		panic(err)
	}
	_, err = c.Config.GetTokenSource().Token(context.TODO())
	if err != nil {
		panic(err)
	}
}

Expected behavior
The request should be retried

Is it a regression?
No

Debug Logs

2025/12/31 14:13:45 [TRACE] Loading config via environment
2025/12/31 14:13:45 [TRACE] Loading config via config-file
2025/12/31 14:13:45 [TRACE] Attempting to configure auth: "file-oidc"
2025/12/31 14:13:45 [DEBUG] No ClientID provided, authenticating with Account-wide token federation
2025/12/31 14:13:45 [DEBUG] POST /oidc/accounts/test-account/v1/token
> <http.RoundTripper>
<  
< {
<   "error": "too many requests"
< }
2025/12/31 14:13:45 [DEBUG] Attempting retry because of "too many requests"
2025/12/31 14:13:45 [DEBUG] non-retriable error: cannot reset reader of type io.nopCloserWriterTo
panic: file-oidc auth: Post "https://accounts.cloud.databricks.com/oidc/accounts/test-account/v1/token": cannot reset reader of type io.nopCloserWriterTo. Config: host=https://accounts.cloud.databricks.com, account_id=test-account, databricks_id_token_filepath=/var/folders/r1/myg4s_g127d6pjrb1rbry1_80000gn/T/tree_908089712/token

Other Information

• OS: [e.g. macOS]: MacOS
• Version: [e.g. 0.1.0]: Latest

Additional context
Add any other context about the problem here.

[orf]

This also occurs if you use a m2m token with ClientID and ClientSecret

[skwon615]

Any updates or fixes here? Also hitting the same issue

[orf]

@renaudhartert-db i'm not sure the change you made fixes this issue.

We have a simple test for this bug that looks similar to the reproduction case above:

	requestCount := 0
	s.accountFetcher.accountClient.Config.HTTPTransport = hc(func(r *http.Request) (*http.Response, error) {
		s.Require().Equal("https://localhost:1234/dummy/token", r.URL.String())

		resp := s.makeResponse(http.StatusTooManyRequests, map[string]string{
			"error": "too many requests",
		})
		resp.Request = r
		requestCount++
		return resp, nil
	})
	s.Greater(requestCount, 1)

This causes IsRetriable to be called with a *url.Error error, not a oauth2.RetrieveError, which isn't handled by httpResponse or isTransientNetworkError.

This results in only 1 request being made, with no retries.

[renaudhartert-db]

Hi @orf,

You're correct that PR #1602 did not address the issue with the retry logic within the HTTP client. That is intentional, in retrospective, we believe that having implemented the retry logic at the transport level was a mistake and are progressively moving it to a higher level.

Rather than addressing the issue within the current retry logic, the PR implemented the retry logic at the application level in the auth code. Our plan is to apply a similar pattern in other places, ultimately deprecating the transport-level retry.

That being said, please let us know if you observe that auth retries are not working the way you would expect them to.