feat(object-store): mask env vars with empty-string storage options - #8098
Closed
westonpace wants to merge 2 commits into
Closed
feat(object-store): mask env vars with empty-string storage options#8098westonpace wants to merge 2 commits into
westonpace wants to merge 2 commits into
Conversation
Setting a storage option to an empty string now masks the environment
variable of the same name (case-insensitive) from the AWS credential
provider chain.
This lets users with both IRSA (AWS_WEB_IDENTITY_TOKEN_FILE + AWS_ROLE_ARN)
and pod/ECS credentials (AWS_CONTAINER_CREDENTIALS_FULL_URI) set in their
environment selectively suppress the higher-priority IRSA provider for a
specific bucket by passing e.g. `{"AWS_WEB_IDENTITY_TOKEN_FILE": ""}` in
storage_options.
Implementation:
- StorageOptions::masked_env_vars() returns the uppercase env var names
whose keys have empty-string values in storage options
- StorageOptions::as_s3_options() filters out empty-string entries so
masks are not forwarded as actual S3 config values
- build_credentials_chain_with_masks() builds a CredentialsProviderChain
that mirrors DefaultCredentialsChain order but skips providers whose
required env vars are masked
- build_aws_credential() accepts a masked_env_vars parameter and uses
the custom chain when any masking is in effect
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
github-actions
Bot
added
A-encoding
westonpace
marked this pull request as ready for review
July 30, 2026 15:43
westonpace
marked this pull request as draft
July 30, 2026 15:48
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Member
Author
|
Closing in favor of #8103 which does not reinvent the AWS credential chain |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A user had two buckets they needed to access. Each bucket was configured with a different credential mechanism. One bucket was using IRSA (AWS_WEB_IDENTITY_TOKEN_FILE + AWS_ROLE_ARN). The other bucket was using ECS credentials (AWS_CONTAINER_CREDENTIALS_FULL_URI).
When all environment variables were set lance would always choose IRSA (since it is higher precedence).
The user tried manually popping out env vars but:
This PR makes it possible to mask out environment variables using a storage option set to the empty string. In addition, it creates an environment variable snapshot, so expired tokens do not see a different environment.