customEndpoint plugin defaults don't work

[er1c]

Describe the bug

We added the customEndpoint plugin and quickly ran into:

1. It needs an IAM permission (rds:DescribeDBClusterEndpoints) that isn't documented:

software.amazon.awssdk.services.rds.model.RdsException: User: arn:aws:sts::....:assumed-role/....-service/.... is not authorized to perform: rds:DescribeDBClusterEndpoints on resource: arn:aws:rds:us-east-1:....:cluster-endpoint:svc-....-reader because no identity-based policy allows the rds:DescribeDBClusterEndpoints action (Service: Rds, Status Code: 403, Request ID: aedfbcfc-...) 

2. After a few minutes it will start throwing other exceptions:

[ip-10-2-1-116.ec2.internal] [severe] s.a.j.p.c.CustomEndpointPlugin - Encountered an exception while monitoring custom endpoint 'svc-...-reader.cluster-custom-....us-east-1.rds.amazonaws.com'. 
[ip-10-2-1-116.ec2.internal] [error] s.a.j.p.c.CustomEndpointPlugin - Encountered an exception while monitoring custom endpoint 'svc-...-reader.cluster-custom-....us-east-1.rds.amazonaws.com'. 
software.amazon.awssdk.services.rds.model.RdsException: Rate exceeded (Service: Rds, Status Code: 400, Request ID: 3d0b5619-42bc-4039-9045-4d6bcdc32ae4) 

Expected Behavior

I would expect the default values would work, and not cause rate limiting problems.

What plugins are used? What other connection properties were set?

initialConnection,auroraConnectionTracker,failover2,efm2,customEndpoint

Current Behavior

[ip-10-2-1-116.ec2.internal] [severe] s.a.j.p.c.CustomEndpointPlugin - Encountered an exception while monitoring custom endpoint 'svc-...-reader.cluster-custom-....us-east-1.rds.amazonaws.com'. 
[ip-10-2-1-116.ec2.internal] [error] s.a.j.p.c.CustomEndpointPlugin - Encountered an exception while monitoring custom endpoint 'svc-...-reader.cluster-custom-....us-east-1.rds.amazonaws.com'. 
software.amazon.awssdk.services.rds.model.RdsException: Rate exceeded (Service: Rds, Status Code: 400, Request ID: 3d0b5619-42bc-4039-9045-4d6bcdc32ae4) 
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125) 
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82) 
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:60) 
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:41) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:50) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:38) 
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:79) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:41) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.executeRequest(RetryableStage2.java:93) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:56) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:36) 
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) 
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53) 
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32) 
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) 
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) 
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) 
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210) 
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103) 
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173) 
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80) 
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182) 
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74) 
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) 
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53) 
	at software.amazon.awssdk.services.rds.DefaultRdsClient.describeDBClusterEndpoints(DefaultRdsClient.java:4810) 
	at software.amazon.awssdk.services.rds.RdsClient.describeDBClusterEndpoints(RdsClient.java:7166) 
	at software.amazon.jdbc.plugin.customendpoint.CustomEndpointMonitorImpl.run(CustomEndpointMonitorImpl.java:128) 
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
	at java.base/java.lang.Thread.run(Thread.java:829) 
	Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Rate exceeded (Service: Rds, Status Code: 400, Request ID: 1468faf3-c24c-48f4-affe-78fae1967fed) 
	Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Rate exceeded (Service: Rds, Status Code: 400, Request ID: b4ff7e3d-ebe2-42b6-ae3c-1641f432bab2) 
	Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 3 failure: Rate exceeded (Service: Rds, Status Code: 400, Request ID: 1876ec26-6c16-4bba-9cc6-e34d762c83a4) 

Reproduction Steps

This was for about 20 ECS tasks with ~~ 40 connections per instance

Possible Solution

The interval rates might need to be updated?

Additional Information/Context

No response

The AWS Advanced JDBC Driver version used

2.5.2

JDK version used

17

Operating System and version

linux

[sergiyvamz]

Hi @er1c

We've merged a fix for the issue and would like to ask you to verify the fix if it's possible. Could you kindly checkout our snapshot build and let us know if the issue still persists?
https://github.com/aws/aws-advanced-jdbc-wrapper/blob/main/docs/using-the-jdbc-driver/UsingTheJdbcDriver.md#using-a-snapshot-of-the-driver

Thank you!

[karenc-bq]

Hi @er1c, the fix is now available in v2.6.4. We will close this issue now.

Thank you again for raising this, please feel free to reach out again if you encounter any other issues.

[er1c]

After reviewing the fix I don't think I would deploy this into my production environment: https://github.com/aws/aws-advanced-jdbc-wrapper/pull/1521/files#r2392847221

[er1c]

@karenc-bq just to followup again, that fix looks insufficient

[karenc-bq]

Hi @er1c, just wanted to provide a quick update here, we are investigating this and will update this thread when we have a fix.