[SDK CPP] WHEN_REQUIRED drops the checksum body/trailer even when SetChecksumAlgorithm is called explicitly, producing a malformed signed request (400 InvalidRequest)

[curios-kris-coh]

Describe the bug

When S3ClientConfiguration::checksumConfig.requestChecksumCalculation is set
to RequestChecksumCalculation::WHEN_REQUIRED and the caller explicitly
calls SetChecksumAlgorithm(CRC32) on a non-multipart PutObjectRequest,
the SDK still signs the request as if a checksum algorithm was supplied
(x-amz-sdk-checksum-algorithm ends up in the signed headers), but does
not actually attach the checksum — neither as an inline
x-amz-checksum-crc32 header (non-chunked path) nor as an
x-amz-trailer/aws-chunked trailer (streaming path).

The resulting request is malformed and S3 correctly rejects it with a
400 InvalidRequest:

x-amz-sdk-checksum-algorithm specified, but no corresponding x-amz-checksum-*
or x-amz-trailer headers were found.

This defeats the documented purpose of WHEN_REQUIRED. Per AWS's own
documentation (Data Integrity Protections for Amazon S3 /
docs.aws.amazon.com/sdkref/latest/guide/feature-dataintegrity.html) and the
equivalent Java SDK docs, WHEN_REQUIRED is meant to suppress the SDK's
unsolicited default checksum for operations that don't strictly require
one — it should still honor a checksum the caller explicitly asked for via
SetChecksumAlgorithm. Right now it does neither correctly: it signs as if
the checksum were present, but omits it.

Select this option if this issue appears to be a regression: Yes —
checksumConfig (and this WHEN_REQUIRED/WHEN_SUPPORTED distinction) didn't
exist prior to 1.11.486 (the "default integrity protections" / DIP release).
Pre-486 SDKs (tested: 1.11.421) always compute and attach the checksum when
SetChecksumAlgorithm is called explicitly, regardless of any "required vs
supported" concept, so this is a regression introduced by the DIP rollout
and its ChecksumInterceptor (introduced in commit ac9296d7a0b, refactored
further in faa33a61862 / PR #3500).

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

With requestChecksumCalculation = WHEN_REQUIRED and an explicit
SetChecksumAlgorithm(CRC32) call, the SDK should still compute and attach
the CRC32 checksum (either inline or via trailer, following the same
streaming/non-streaming logic used under WHEN_SUPPORTED), because the
caller explicitly requested it. WHEN_REQUIRED should only change behavior
for the unsolicited/default checksum case (i.e., when the caller does
not call SetChecksumAlgorithm at all).

Current Behavior

• PutObject succeeds when requestChecksumCalculation is left at its
  default (WHEN_SUPPORTED) or explicitly set to WHEN_SUPPORTED, with
  SetChecksumAlgorithm(CRC32) set on the request.
• PutObject fails with a 400 InvalidRequest when
  requestChecksumCalculation is explicitly set to WHEN_REQUIRED, with the
  exact same SetChecksumAlgorithm(CRC32) call on the request. Error:

PutObject FAILED
  ExceptionName:  InvalidRequest
  Message:        x-amz-sdk-checksum-algorithm specified, but no corresponding
                  x-amz-checksum-* or x-amz-trailer headers were found.
  RequestId:      NQDF14VJ7M6SSFPY
  ResponseCode:   400

Confirmed reproducible on:

• 1.11.712
• main @ 1.11.840 (current HEAD as of this report) — not fixed
  upstream

Confirmed not reproducible / not applicable on:

• 1.11.421 (pre-DIP; checksumConfig doesn't exist, SetChecksumAlgorithm
  always results in a computed+attached checksum)

Reproduction Steps

#include <aws/core/Aws.h>
#include <aws/core/auth/AWSCredentials.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/S3ClientConfiguration.h>
#include <aws/s3/model/PutObjectRequest.h>
#include <aws/s3/model/ChecksumAlgorithm.h>

int main() {
  Aws::SDKOptions options;
  Aws::InitAPI(options);
  {
    Aws::S3::S3ClientConfiguration config;
    config.region = "us-west-1";
    config.checksumConfig.requestChecksumCalculation =
        Aws::Client::RequestChecksumCalculation::WHEN_REQUIRED;  // <-- triggers the bug

    Aws::Auth::AWSCredentials creds("<access-key>", "<secret-key>");
    Aws::S3::S3Client client(creds, nullptr, config);

    auto body = Aws::MakeShared<Aws::StringStream>("repro");
    *body << std::string(5 * 1024 * 1024, 'x');  // 5MB body
    body->seekg(0);

    Aws::S3::Model::PutObjectRequest req;
    req.SetBucket("<your-bucket>");
    req.SetKey("repro-object");
    req.SetChecksumAlgorithm(Aws::S3::Model::ChecksumAlgorithm::CRC32);  // explicit
    req.SetBody(body);

    auto outcome = client.PutObject(req);
    if (!outcome.IsSuccess()) {
      std::cerr << outcome.GetError().GetMessage() << "\n";  // 400 InvalidRequest
    }
  }
  Aws::ShutdownAPI(options);
}

Toggling only config.checksumConfig.requestChecksumCalculation between
WHEN_SUPPORTED (or leaving it unset, which defaults to WHEN_SUPPORTED)
and WHEN_REQUIRED is the only variable — everything else (SDK version,
client, bucket, algorithm, body) is held constant.

A full, version-parameterized repro harness (build script + test app that
can be linked against multiple aws-sdk-cpp versions) is available on
request / can be attached to this issue.

Possible Solution

In src/aws-cpp-sdk-core/include/smithy/client/features/ChecksumInterceptor.h,
the branch that resolves whether to attach a checksum appears to gate solely
on m_requestChecksumCalculation without independently checking whether the
caller already provided an explicit checksum algorithm/value
(checksumValueAndAlgorithmProvided / handleProvidedChecksum in the
current refactor from PR #3500). The fix likely needs
handleProvidedChecksum (or its caller) to always run when the caller
explicitly set an algorithm, independent of the
WHEN_REQUIRED/WHEN_SUPPORTED gate — that gate should only affect whether
calculateAndSetChecksum (the unsolicited/default path) runs.
Separately, whatever adds x-amz-sdk-checksum-algorithm to the
signed-headers set (likely in the signer, not this file) should be
conditioned on the same "did we actually attach a checksum" outcome, so the
signed-headers claim and the actual request body never disagree.

Additional Information/Context

• This is the same class of bug that's been reported against other AWS
  SDKs/tools for the WHEN_REQUIRED setting, just manifesting differently
  each time — e.g. boto/s3transfer#327 (CLI ignores WHEN_REQUIRED
  entirely and sends the checksum anyway) and the Akamai/OUTSCALE
  compatibility notes documenting WHEN_REQUIRED as the standard workaround
  for third-party S3-compatible endpoints that don't support the newer
  checksum trailer format. In our case it's the opposite failure mode:
  WHEN_REQUIRED is honored too aggressively and drops a checksum the
  caller explicitly asked for.

• Possibly related: Checksum verification fails for composite checkum types #3496 (composite checksum response validation failing,
  also in ChecksumInterceptor.h, also worked around via
  WHEN_REQUIRED/responseChecksumValidation) — suggests the
  WHEN_REQUIRED/WHEN_SUPPORTED resolution logic in this file may have
  more than one edge case around explicit-vs-default checksum handling.

• We hit this against a non-AWS, S3-compatible portal, initially trying to
  use WHEN_REQUIRED specifically as the documented workaround for
  compatibility with that portal — this bug means that workaround is
  unusable for any call that also sets an explicit checksum algorithm.

• AWS CPP SDK version: 1.11.712 and main @ 1.11.840 (both reproduce);
  1.11.421 does not (feature doesn't exist pre-486)

• Compiler and Version: g++ 13.3.0

• Operating System: Ubuntu 24.04

• Region: us-west-1

AWS CPP SDK version used

1.11.712 and main @ 1.11.840 (both reproduce); 1.11.421 does not (feature doesn't exist pre-486)

Compiler and Version used

gcc 11.4.0

Operating System and version

Ubuntu 22.04.5 LTS (Jammy Jellyfish)

[curios-kris-coh]

Update: precise regression range identified, with wire-level evidence

Since filing, I bisected further and can narrow the regression range precisely:

• 1.11.421 — N/A, checksumConfig doesn't exist yet (pre-DIP)
• 1.11.486 (first DIP release) — works correctly
• 1.11.712 — fails (this issue's original report)
• main @ 1.11.840 (current HEAD as of this comment) — still fails

So this is not "broken since DIP launched" — it's a regression introduced
somewhere in the ~226 releases between 1.11.486 and 1.11.712.

Wire-level evidence (curl trace, --debug/Trace log level)

At 1.11.486 (WHEN_REQUIRED + explicit SetChecksumAlgorithm(CRC32)) — correct, well-formed request:

x-amz-sdk-checksum-algorithm:CRC32
x-amz-trailer:x-amz-checksum-crc32
...
Signed Headers value: ...;transfer-encoding;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-trailer

This is the chunked/streaming/trailer path — content-length is correctly
replaced by x-amz-decoded-content-length + transfer-encoding: chunked +
content-encoding: aws-chunked, and x-amz-trailer correctly declares
where the checksum will be found. PutObject succeeds.

At 1.11.712 (identical code, only the SDK build differs) — malformed request:

x-amz-sdk-checksum-algorithm:CRC32
...
Signed Headers value: ...;content-length;...;x-amz-sdk-checksum-algorithm

No x-amz-trailer, no transfer-encoding, no content-encoding: aws-chunked — the plain content-length path was used instead, but
x-amz-sdk-checksum-algorithm: CRC32 is still signed with no
corresponding checksum data anywhere in the request. Fails with:

400 InvalidRequest: x-amz-sdk-checksum-algorithm specified, but no
corresponding x-amz-checksum-* or x-amz-trailer headers were found.

What I found while narrowing this down

I traced through ChecksumInterceptor.h's history between 486 and 712
(commits cb061dc3bd1, 382e22376e9, 1a48e8b2ade, faa33a61862/#3500)
and also AWSAuthV4Signer.cpp's history in the same range, including
PR #3635 ("refactor: centralize aws-chunked encoding in
ChunkingInterceptor"), which moved the chunked-encoding setup logic out
of the signer into a new ChunkingInterceptor. That refactor's gating
condition (ShouldApplyChunking, keying off request->GetRequestHash()
being non-null) looks functionally equivalent to what the signer checked
before the refactor — so while it's a plausible candidate given the
timing, I wasn't able to fully confirm it's the actual regression commit
by reading diffs alone. Every version of ChecksumInterceptor.h's
top-level gate I looked at ((!algo.empty() && mode==WHEN_SUPPORTED) || RequestChecksumRequired()) appears unchanged and false for this
exact case at both 486 and 712 — which doesn't fully explain why 486
succeeds. There's likely some interaction I'm missing between how/when
SetRequestHash gets attached to the request that isn't visible from
diffing this one file in isolation.

I don't have a from-source-reading-confirmed root cause at the exact
commit level, but the 486→712 empirical boundary above is solid (built
and ran both versions directly, not just read the source). A git bisect between those two tags, running the actual repro at each step,
would likely find the exact commit much faster than continued manual
diffing — happy to do that if useful, otherwise deferring to whoever has
fuller context on how ChecksumInterceptor / ChunkingInterceptor /
the signer are meant to interact.

[sbiscigl]

With requestChecksumCalculation = WHEN_REQUIRED and an explicit
SetChecksumAlgorithm(CRC32) call, the SDK should still compute and attach
the CRC32 checksum (either inline or via trailer, following the same
streaming/non-streaming logic used under WHEN_SUPPORTED), because the
caller explicitly requested it. WHEN_REQUIRED should only change behavior
for the unsolicited/default checksum case (i.e., when the caller does
not call SetChecksumAlgorithm at all).

yeah looks like officially it is "when_required - When set, a checksum will only be calculated for request payloads of operations modeled with the @httpChecksum trait where requestChecksumRequired is true OR where a requestAlgorithmMember is modeled and the user sets it."

can get a fix a out for this

Select this option if this issue appears to be a regression: Yes

i'd like to push back on calling this a regression, the when_required/when_supported configuration didnt exist until after the default data integrity change. the default behavior turned into when_supported which works and is the default. if you were to opt into when_required and specify a checksum algorithm, i don't think we would ever calculate one for you after that change. still a bug, and one that i am fixing, i just don't see the regression. if there is a version where specifying when_required and checksum worked then let me know and this is a regression. no youre right it is a regression since then, looking into a fix

[sbiscigl]

fix is merged, should be released later this afternoon with tag 1.11.842 let me know if you have any questions