Getting Aws::S3::Errors::XAmzContentSHA256Mismatch since 1.210.0

[jtreitz]

Describe the bug

We bumped aws-sdk-s3 1.209.0 to 1.210.0 and started getting
Getting Aws::S3::Errors::XAmzContentSHA256Mismatch errors
Code otherwise unchanged

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Updating the gem should not produce Aws::S3::Errors::XAmzContentSHA256Mismatch errors

Current Behavior

Updating the gem produces Aws::S3::Errors::XAmzContentSHA256Mismatch errors

Reproduction Steps

Update aws-sdk-s3 1.209.0 to 1.210.0

Possible Solution

No response

Additional Information/Context

No response

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk-s3 1.210.0

Environment details (Version of Ruby, OS environment)

Ruby 3.4.7, MacOs Tahoe 26.1

[jterapin]

Hi! Thanks for reporting this issue. To help us investigate, could you provide:

• The full stack trace showing where the error occurs
• Your aws-sdk-core version
• The S3 operation you're using (if possible, a code snippet)

Additionally, do you use a custom signer by any chance?

The recent changes primarily affect PutObject and UploadPart operations, so knowing your specific use case will help us pinpoint the issue.

[jtreitz]

Rails ActiveStorage was trying to store a variant (put an object)

-    aws-sdk-s3 (1.209.0)
-      aws-sdk-core (~> 3, >= 3.234.0)
+    aws-sdk-s3 (1.210.0)
+      aws-sdk-core (~> 3, >= 3.241.0)

seahorse/client/plugins/raise_response_errors.rb in call at line 17


aws-sdk-s3/plugins/sse_cpk.rb in call at line 24


aws-sdk-s3/plugins/dualstack.rb in call at line 21


aws-sdk-s3/plugins/accelerate.rb in call at line 43


aws-sdk-core/plugins/checksum_algorithm.rb in call at line 167


aws-sdk-core/plugins/jsonvalue_converter.rb in call at line 16


aws-sdk-core/plugins/invocation_id.rb in call at line 16


aws-sdk-core/plugins/idempotency_token.rb in call at line 19


aws-sdk-core/plugins/param_converter.rb in call at line 26


seahorse/client/plugins/request_callback.rb in call at line 89


aws-sdk-core/plugins/response_paging.rb in call at line 12


seahorse/client/plugins/response_target.rb in call at line 24


aws-sdk-core/plugins/telemetry.rb in block in Aws::Plugins::Telemetry::Handler#call at line 39


aws-sdk-core/telemetry/no_op.rb in in_span at line 29


aws-sdk-core/plugins/telemetry.rb in span_wrapper at line 53


aws-sdk-core/plugins/telemetry.rb in call at line 39


seahorse/client/request.rb in send_request at line 72


aws-sdk-s3/client.rb in put_object at line 18508


aws-sdk-s3/object.rb in block in Aws::S3::Object#put at line 3082


aws-sdk-core/plugins/user_agent.rb in Aws::Plugins::UserAgent.metric at line 92


aws-sdk-s3/object.rb in put at line 3081


active_storage/service/s3_service.rb in upload_with_single_part at line 143

...

Hope this helps

[jterapin]

Thanks for the details. Active Storage adds its own layer on top of the AWS SDK, so I'll need to investigate how that interaction might be affected.

To help debug this, could you provide a minimal code example that reproduces the XAmzContentSHA256Mismatch error? This would help isolate whether the issue is in our SDK changes or the Active Storage integration.

[chrisgavin]

I've been having similar issues with request signing since upgrading. I suspect the problem only affects S3 compatible storages like Google Cloud Storage. The problem seems to be x-amz-content-sha256 is set to STREAMING-UNSIGNED-PAYLOAD-TRAILER, rather than the actual SHA, which might work in real S3, but a lot of third party implementations do not support it.

[xadips]

Same for s3 glacier as the backend storage with version 1.210.0 gets the same mismatch error

[jterapin]

Thank you for all the additional contexts. This definitely helps with the investigation.

I do have more questions that could help me dig into further:

@jtreitz - Could you share more details about your setup? My initial testing with Active Storage/S3 works on my end, so I'm looking for what might be different in your setup.

• Do you use any third party dependencies that are S3-compatible?
• Any specific configuration that might be relevant?
• Any specific upload patterns or file types that trigger this issue?

@xadips - What's your setup and reproducible step?

• Are you using S3-compatible services or standard AWS S3?
• Could you provide a minimal code example that triggers the error?
• I tested with S3 Glacier storage class but couldn't reproduce the issue - are there other configuration details or specific scenarios I should try?

[jtreitz]

Just realized the bucket in question is indeed hosted by a S3-compatible service (storj.io) – didn't think of that yesterday, sorry.

The usecase is pretty standard, users upload a JPG through a form and we create 3 variants of it like this:

class Photo < ApplicationRecord
  has_one_attached :file do |attachment|
    attachment.variant :small,  resize_to_limit: [ 640,  640], format: :webp, saver: { quality: 50, strip: true }
    attachment.variant :large,  resize_to_limit: [3200, 3200], format: :webp
    attachment.variant :compat, resize_to_limit: [3200, 3200], format: :jpeg
  end
end

image_processing (1.14.0)
      mini_magick (>= 4.9.5, < 6)
      ruby-vips (>= 2.0.17, < 3)

activestorage (= 8.0.4)

    aws-eventstream (1.4.0)
    aws-partitions (1.1200.0)
    aws-sdk-core (3.241.0)
      aws-eventstream (~> 1, >= 1.3.0)
      aws-partitions (~> 1, >= 1.992.0)
      aws-sigv4 (~> 1.9)
      base64
      bigdecimal
      jmespath (~> 1, >= 1.6.1)
      logger
    aws-sdk-kms (1.119.0)
      aws-sdk-core (~> 3, >= 3.241.0)
      aws-sigv4 (~> 1.5)
    aws-sdk-s3 (1.209.0)
      aws-sdk-core (~> 3, >= 3.234.0)
      aws-sdk-kms (~> 1)
      aws-sigv4 (~> 1.5)
    aws-sigv4 (1.12.1)
      aws-eventstream (~> 1, >= 1.0.2)

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.