Add metric and logging for activator-autoscaler connectivity #16318

prashanthjos · 2025-12-24T02:32:29Z

Description

This PR adds observability for the websocket connection between the activator and autoscaler components. When the autoscaler is not reachable, operators currently have no easy way to identify this issue, which can lead to autoscaling failures.

Changes

New Metric

Name: kn.activator.autoscaler.reachable
Type: Int64Gauge
Values: 1 (reachable), 0 (not reachable)
Description: Whether the autoscaler is reachable from the activator

New Logging

ERROR level log when autoscaler is not reachable:
- "Autoscaler is not reachable from activator. Stats were not sent." (on send failure)
- "Autoscaler is not reachable from activator." (on connection check failure)

How It Works

The metric is recorded in two scenarios:

Periodic check (every 5s):
- Uses [conn.Status()] to check if connection is established
- Catches: Connection never established, DNS failures, autoscaler not running at startup
On stat send:
- Detects [SendRaw()] failures during actual stat message transmission
- Catches: Network timeouts, connection drops, autoscaler becoming unreachable

Testing

Unit tests pass (go test ./pkg/activator/... -v)
New test TestAutoscalerConnectionStatusMonitor added

knative-prow · 2025-12-24T02:32:38Z

Welcome @prashanthjos! It looks like this is your first PR to knative/serving 🎉

knative-prow · 2025-12-24T02:32:39Z

Hi @prashanthjos. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

knative-prow · 2025-12-24T02:33:10Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: prashanthjos
Once this PR has been reviewed and has the lgtm label, please assign dsimansk for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This change adds observability for the websocket connection between the activator and autoscaler components: - Add `activator_autoscaler_reachable` gauge metric (1=reachable, 0=not reachable) - Log ERROR when autoscaler is not reachable during stat sending - Add periodic connection status monitor (every 5s) to detect connection establishment failures - Add unit tests for the new AutoscalerConnectionStatusMonitor function The metric is recorded in two scenarios: 1. When SendRaw fails/succeeds during stat message sending 2. When the periodic status check detects connection not established This helps operators identify connectivity issues between activator and autoscaler that could impact autoscaling decisions.

thiagomedina · 2025-12-24T13:07:15Z

/ok-to-test

codecov · 2025-12-24T13:13:20Z

Codecov Report

❌ Patch coverage is 95.23810% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.16%. Comparing base (a8803aa) to head (0a216b5).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/activator/metrics.go	88.23%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #16318      +/-   ##
==========================================
+ Coverage   80.09%   80.16%   +0.07%     
==========================================
  Files         215      216       +1     
  Lines       13391    13429      +38     
==========================================
+ Hits        10725    10766      +41     
  Misses       2304     2304              
+ Partials      362      359       -3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

prashanthjos · 2025-12-25T02:14:25Z

/retest

prashanthjos · 2025-12-29T19:12:42Z

Related docs PR:

knative/docs#6548

prashanthjos · 2026-01-02T19:14:38Z

@skonto @dsimansk can you please take a look at the PR when you get sometime

linkvt

Thanks for your PR! I understand the requirement but think we could simplify/change the current implementation a bit, see my comments.

I haven't worked that much with this code so far, a more experienced maintainer might have more thoughts about the changes, not sure if they are still in their winter break though.

linkvt · 2026-01-07T10:23:15Z

pkg/activator/stat_reporter.go

+
+// AutoscalerConnectionStatusMonitor periodically checks if the autoscaler is reachable
+// and emits metrics and logs accordingly.
+func AutoscalerConnectionStatusMonitor(ctx context.Context, logger *zap.SugaredLogger, conn StatusChecker, mp metric.MeterProvider) {


I don't think we need this monitor as the stats are already reported every second, see

serving/pkg/activator/handler/concurrency_reporter.go

Line 41 in 09e59ec

const reportInterval = time.Second

This means errors would be detected there already.

linkvt · 2026-01-07T10:33:01Z

pkg/activator/metrics.go

+
+	meter := provider.Meter(scopeName)
+
+	m.autoscalerReachable, err = meter.Int64Gauge(


I think we should use counters with labels here instead of a gauge. If the connection is flaky we might get the case where we always check when the gauge is 1.
If we have a counter with result=success or result=error we would:

not miss any errors anymore

could create an alert based on the success rate e.g. during the last 5 minutes if success rate is e.g. below 95%

linkvt · 2026-01-07T10:35:29Z

pkg/activator/metrics.go

@@ -0,0 +1,53 @@
+/*


Would it maybe make sense to add this to the existing metrics in https://github.com/knative/serving/blob/main/pkg/activator/handler/metrics.go ?

linkvt · 2026-01-07T10:54:49Z

pkg/activator/stat_reporter_test.go

+	}
+
+	// Give some time for the goroutine to process the error and log
+	time.Sleep(100 * time.Millisecond)


Do we actually need this?
I always try not to add Sleeps in my tests as they increase the duration of the whole test suite. Recent go versions also contain synctest to work around this but less code is always better 😀

knative-prow bot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 24, 2025

knative-prow bot requested review from dsimansk and skonto December 24, 2025 02:33

prashanthjos force-pushed the main branch from 72d0280 to dce652d Compare December 24, 2025 02:38

knative-prow bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 24, 2025

prashanthjos mentioned this pull request Dec 29, 2025

Add documentation for activator_autoscaler_reachable metric knative/docs#6548

Open

prashanthjos added 2 commits December 30, 2025 08:58

Correcting the metric name inline with knative standards

1e54a49

Increasing the code coverage

0a216b5

linkvt reviewed Jan 7, 2026

View reviewed changes


		meter := provider.Meter(scopeName)

		m.autoscalerReachable, err = meter.Int64Gauge(

Add metric and logging for activator-autoscaler connectivity #16318

Are you sure you want to change the base?

Add metric and logging for activator-autoscaler connectivity #16318

Conversation

prashanthjos commented Dec 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

New Metric

New Logging

How It Works

Testing

Uh oh!

knative-prow bot commented Dec 24, 2025

Uh oh!

knative-prow bot commented Dec 24, 2025

Uh oh!

knative-prow bot commented Dec 24, 2025

Uh oh!

thiagomedina commented Dec 24, 2025

Uh oh!

codecov bot commented Dec 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

prashanthjos commented Dec 25, 2025

Uh oh!

prashanthjos commented Dec 29, 2025

Uh oh!

prashanthjos commented Jan 2, 2026

Uh oh!

linkvt left a comment

Choose a reason for hiding this comment

Uh oh!

linkvt Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

linkvt Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

linkvt Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

linkvt Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

prashanthjos commented Dec 24, 2025 •

edited

Loading

codecov bot commented Dec 24, 2025 •

edited

Loading