Querier active api tracker #7216

eeldaly · 2026-01-14T22:35:30Z

What this PR does:
Add an active api tracker to print api requests in a querier on OOMKill / crash

Example log:

"These API calls didn't finish in prometheus' last run:"apis="[
{\"Method\":\"GET\",
\"Path\":\"/prometheus/api/v1/series\",
\"User-Agent\":\"PostmanRuntime/7.49.1\",
\"X-Request-ID\":\"\",
\"X-Scope-OrgID\":\"12345\",
\"end\":\"\",
\"limit\":\"100\",
\"matches\"\"metric_0metric_0metric_0m,metric_1metric_1metric_1m,metric_2metric_2metric_2m,metric_3metric_3metric\",
\"number-of-matches\":5,
\"start\":\"\",
\"timestamp_sec\":1768349758
}]"

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

yeya24 · 2026-01-14T22:47:00Z

pkg/querier/api_tracker/api_tracker.go

+	"github.com/edsrzf/mmap-go"
+)
+
+type APITracker struct {


Can we make this tracker generic? Move to a better package maybe in utils. I feel it can be used in query frontend and more components as well. Maybe a generic name can be RequestsTracker. Make sure when inserting entries we can insert arbitrary entries?

yeya24 · 2026-01-15T00:30:14Z

pkg/querier/api_tracker/api_wrapper.go

+	entryMap["timestamp_sec"] = time.Now().Unix()
+	entryMap["Path"] = r.URL.Path
+	entryMap["Method"] = r.Method
+	entryMap["User-Agent"] = r.Header.Get("User-Agent")


Why do we need this?

yeya24 · 2026-01-15T00:30:31Z

pkg/querier/api_tracker/api_wrapper.go

+	entryMap["Method"] = r.Method
+	entryMap["User-Agent"] = r.Header.Get("User-Agent")
+	entryMap["X-Scope-OrgID"] = r.Header.Get("X-Scope-OrgID")
+	entryMap["X-Request-ID"] = r.Header.Get("X-Request-ID")


We shouldn't hardcode the request ID. It is extracted from the context

yeya24 · 2026-01-15T00:30:46Z

pkg/querier/api_tracker/api_wrapper.go

+	entryMap["Path"] = r.URL.Path
+	entryMap["Method"] = r.Method
+	entryMap["User-Agent"] = r.Header.Get("User-Agent")
+	entryMap["X-Scope-OrgID"] = r.Header.Get("X-Scope-OrgID")


Extract user ID from context. There is helper function to do it

pkg/querier/api_tracker/api_wrapper.go

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

justinjung04

LGTM, thank you!

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com>

eeldaly · 2026-01-28T22:01:46Z

One issue that still needs to be addressed is that the request id is not yet in the context at the time of it being added to the tracker. Getting request id from context currently returns an empty string

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

CHANGELOG.md

pkg/util/request_tracker/request_wrapper.go

pkg/util/request_tracker/request_extractor.go

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

…caped characters Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

yeya24 · 2026-02-10T06:48:39Z

pkg/util/request_tracker/request_extractor.go

+	}
+	fieldValueEncoded = fieldValueEncoded[1 : len(fieldValueEncoded)-1]
+	fieldValueEncodedTrimmed := trimStringByBytes(fieldValueEncoded, size)
+	fieldValueEncodedTrimmed = "\"" + removeHalfCutEscapeChar(fieldValueEncodedTrimmed) + "\""


This seems relevant. It seems very likely for the field value to have double quotes?

The json encode puts double quotes which I remove on 107 and add back on 109 after trimming the string to the correct size. Let me know if theres a cleaner way to trim the string.

Originally, I was not encoding the string before trimming but this caused issues where escaping characters when encoding json makes the string larger than desired.

I'll take another look at it tmrw to see if theres a cleaner way and make sure its working well as intended

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

yeya24

lint failure is also relevant

yeya24 · 2026-02-10T06:46:58Z

pkg/util/request_tracker/request_extractor.go

+	Extract(r *http.Request) []byte
+}
+
+type DefaultExtractor struct{}


Let's remove DefaultExtractor as it is not used anywhere.

yeya24 · 2026-02-10T06:48:39Z

pkg/util/request_tracker/request_extractor.go

+	}
+	fieldValueEncoded = fieldValueEncoded[1 : len(fieldValueEncoded)-1]
+	fieldValueEncodedTrimmed := trimStringByBytes(fieldValueEncoded, size)
+	fieldValueEncodedTrimmed = "\"" + removeHalfCutEscapeChar(fieldValueEncodedTrimmed) + "\""


This seems relevant. It seems very likely for the field value to have double quotes?

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

Add active api tracker

c399131

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

pull-request-size bot added the size/L label Jan 14, 2026

dosubot bot added the component/querier label Jan 14, 2026

Update changelog

0bd714e

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

yeya24 reviewed Jan 14, 2026

View reviewed changes

yeya24 reviewed Jan 15, 2026

View reviewed changes

Made request tracker more generic; added query apis to tracker

3b566eb

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

pull-request-size bot added size/XL and removed size/L labels Jan 16, 2026

justinjung04 approved these changes Jan 23, 2026

View reviewed changes

eeldaly and others added 2 commits January 23, 2026 14:23

add prometheus copyright header for copied part

83e2259

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

Merge branch 'master' into master

d06e5eb

Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com>

Add request id if not in context

575b518

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

eeldaly requested a review from yeya24 January 30, 2026 21:44

fix test assert expected msg

2b585b7

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

yeya24 reviewed Feb 3, 2026

View reviewed changes

eeldaly added 4 commits February 3, 2026 10:45

Add more logging fields

5da5eeb

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

Rename log entry keys, remove request id generation

814d788

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

Wrap request id wrapper before oomkill logger

eeeaf01

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

Add test for query truncation. Fix edge case with query truncating es…

72a8720

…caped characters Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

github-advanced-security bot found potential problems Feb 3, 2026

View reviewed changes

lint

bfbfe8d

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

eeldaly requested a review from yeya24 February 4, 2026 21:20

eeldaly added 2 commits February 4, 2026 13:36

remove debug line

f910102

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

Add extractor tests

4b9776e

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

yeya24 reviewed Feb 10, 2026

View reviewed changes

Removed default extractor

8c32cbd

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>

Querier active api tracker #7216

Are you sure you want to change the base?

Querier active api tracker #7216

Uh oh!

Conversation

eeldaly commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

justinjung04 left a comment

Choose a reason for hiding this comment

Uh oh!

eeldaly commented Jan 28, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check failure

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yeya24 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

eeldaly commented Jan 14, 2026 •

edited

Loading