Memory leak when RateLimitOverride is set with uniq ip on every call to ShouldRateLimit method #275
Description
In our service we use envoyproxy/ratelimit as sidecar and are making grpc calls to ShouldRateLimit method explicitly without envoy proxy.
Simplified config looks like
---
domain: app
descriptors:
- key: ip
rate_limit:
unit: minute
requests_per_unit: 1000
In our request we specify Limit (limit override) - https://github.com/envoyproxy/java-control-plane/blob/main/api/src/main/proto/envoy/extensions/common/ratelimit/v3/ratelimit.proto#L93 on every request which lead to this codepath getting executed - https://github.com/envoyproxy/ratelimit/blob/main/src/config/config_impl.go#L252-L261
In that if statement when unique IP is getting used on every request we are getting unique rateLimitKey and also as a result are getting new set of stats Counters this.statsManager.NewStats(rateLimitKey),. So number of stats Counters and keys always grows over time.
- Stats should not be collected at all if
USE_STATSDis set tofalse. - Stats should expire and stats cache should get cleaned up over time
UPDATE:
our simplified ratelimit request to looks like this:
{
"domain": "app",
"descriptors": [
{
"entries": {
"key": "our_entry_key",
"value": "customer1234_1.2.3.4"
},
"limit": {
"requests_per_unit": 1000,
"unit": "MINUTE"
}
}
],
"hits_addend": 1
}
^ note 2 things
- we have
limitdefined in each request and is different based on customer. In this case customer iscustomer1234 - key is built by
<customer_identifier>_<ip>
Because limit is defined this code path is getting executed
https://github.com/envoyproxy/ratelimit/blob/main/src/config/config_impl.go#L252-L261
which executes this.statsManager.NewStats(rateLimitKey) which will build whole bunch of new stats counters based on key (customer1234_1.2.3.4)