API Latency is TOO DAMN HIGH

[georgeaf99]

We get latency spikes up to 60s when we have too much traffic on the API. We need to solve this issue before we can on-board new properties.

[georgeaf99]

During this period no DynamoDB requests were throttled on any table or index.

[georgeaf99]

CloudWatch metrics for the instance running the gatorapi-test

[georgeaf99]

Another example of recurring latency spikes

[georgeaf99]

The above graph seems like it could be caused by the API blocking to create a connection to stripe or dynamodb?

[xuset]

The latency was occurring in the notifier container. Mostly because of the socket.io client not behaving properly. Updating the client to the latest version seemed to fix this issue.

[xuset]

The socket.io server is still experiencing these high latencies. It seems to be caused by an open bug in the socket.io swift client socketio/socket.io-client-swift#279 .

Below is my best guess at what's happening:
The problem is the the socket.io swift client does not close the socket when the app sleeps. So the connection persists until the load balancer kills it at 60 seconds.

There is an additional bug in the socketio swift client where if the connection was closed while the app was asleep, then when the app wakes up, swift socketio still thinks the connection is open even though it is not. This leads to a bug in the delegateit ios app where the transaction updates are not getting through.

DelegateIt/DelegateIt_iOS#62 proposes a possible solution to fix that problem which in turn will fix the server latency problem.. Hopefully.

[georgeaf99]

@xuset Nice, that was some pretty impressive detective work.