Independent configuration of threshold for multipart upload and copy

[mikewatt]

Describe the feature

MultipartS3AsyncClient allows configuring the multipart threshold in bytes:

S3AsyncClient.builder()
    .multipartEnabled(true)
    .multipartConfiguration(c -> c.thresholdInBytes(...))

As far as I can tell, this threshold is used for both put object and copy object to determine when to perform the operation as a multi part operation.

Contrast this to the v1.x TransferManager that had very different default thresholds for the two operations: https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/transfer/TransferManagerConfiguration.java#L36-L42

One practical implication of this is that in multipart copy operations, metadata is not copied between objects, and must be done manually, so using a low threshold for multipart operations may require manual metadata copying for copies that otherwise would not require this. Raising the threshold though may adversely affect upload operations.

Since these two operations have different performance characteristics, is it reasonable to allow configuration of the thresholds independently?

Use Case

We perform both uploads and bucket-to-bucket copies with the same MultipartS3AsyncClient. In general we can guarantee an upper bound on object size (which is less that the single-part copy requirement), and when performing bucket-to-bucket copy we require metadata to be copied as well as object content.

This worked as expected with the v1.x TransferManager, with its thresholds of 16MB and 5GB for put and copy respectively.

We'd like to be able to configure the multipart copy threshold to something like 5GB to avoid having to manually copy metadata, without having to evaluate the effect this may have on multi part upload performance.

Proposed Solution

No response

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

AWS Java SDK version used

2.38.7

JDK version used

17

Operating System and version

MacOS 26.1

[bhoradc]

Hi @mikewatt

Thank you for this feature request. Your use-case clearly explains the need for separate threshold configurations for multipart PUT and COPY operations, especially given the metadata handling differences and performance characteristics between these operations.

I'll review this with the team to validate if there are any existing limitations with the current behavior of not providing different thresholds for copy multipart operations and assess the feasibility of this feature.

Regards,
Chaitanya