[1/6] feat: add refreshable storage credential providers

[zakariya-s]

Which issue does this PR close?

• Part of Support refreshing vended storage credentials for REST catalog tables #2931.
• Extracted from feat(rest): Support refreshing vended storage credentials #2932.

What changes are included in this PR?

This PR introduces the core API for supplying refreshable storage credentials:

• Adds StorageCredentialProvider and backend-specific credential types for S3 and GCS.
• Propagates an optional credential provider through FileIOBuilder and FileIO.
• Adds StorageFactory::build_with_credentials.
• Adds the standard S3 and GCS credential refresh property constants.

Backend integration and REST credential refreshing will be added in subsequent PRs.

Are these changes tested?

Yes.

AI Disclosure

AI helped splitting this from the reference PR.

[paleolimbot]

Thank you for this! I'm not a committer but I did recently reinvent basically this to get credential refresh working for our catalog and this API would make our lives much easier.

I don't have any notes here...this entrypoint work work for our code because at the point we construct the FileIO we also have access to our client object that can refresh credentials. The credential values are redacted in the Debug implementation and they're not serializable. The PR obviously needs a few things to go from good idea -> useful but I also think the attempt to minimize the scope is great too.

[zakariya-s]

Hey @paleolimbot, great to hear this will be useful! We also have to do workarounds at the moment to get this working for our DataFusion clients, and I've also heard that others are doing similar stuff in their internal forks of iceberg-rust.

Like you mention, this PR doesn't quite implement the entire E2E refreshing, but I do plan to split it into a few PRs to ease the review burden. Feel free to take a look at the full draft PR (#2932) which does implement it E2E and gives an indicator of how it should roughly look like when done.

[zakariya-s]

Hi @CTTY! I know that bandwidth has been tough at the moment, but could I get a first pass at this (and the reference #2932 for the bigger context) when you have time (no rush at all for this PR). Thank you!

(Also I wonder if you think I should change this to be a stacked PR, now that GH supports it)

[CTTY]

Hi, thanks for the contribution! The direction looks good and we do want this feature.

I haven't got a chance to review the full PR, but I have some questions on the general design:

• Do we really need an abstraction layer for CredentialProvider?

Since the credential refreshing logic will purely depend on the type of storage backend, isn't embedding the cred provider directly into each StorageFactory implementations an easier choice? Maybe you have some examples cases where custom storage credential providing logic will be important?

[zakariya-s]

Thanks @CTTY, the main reason for the trait is that credential acquisition/vending isn't really backend specific, and doing it this way avoids each backend storage factory reimplementing the REST auth logic (e.g. auth, parsing, prefix selection, caching, backoff, etc.). Without the trait, RestCatalog would have to know or downcast each StorageFactory to configure each specific credential loader.

This approach also opens the possibility of non-REST implementations too (like Glue/Lake Formation via AWS specific APIs) without doing what Java does with having custom credential providers for each backend.

Also Iceberg Java has the backend-specific vended credential providers in each of the cloud-specific FileIO implementations, whereas we only have a generic FileIO in iceberg-rust as OpenDAL handles most of it.