fix: restore HPA behavior when paused-scale-in/out annotation is deleted

[shedimon]

Summary

When paused-scale-in or paused-scale-out annotation is deleted (not set to "false"), the HPA's SelectPolicy remains stuck at Disabled instead of being restored to the default behavior. This occurs when selectPolicy is not explicitly defined in the ScaledObject spec, even if other behavior fields like policies or stabilizationWindowSeconds are configured.

Problem Description

Affected Scenario

A user pauses scale-in by setting autoscaling.keda.sh/paused-scale-in=true on a ScaledObject. KEDA correctly sets the HPA's scaleDown.selectPolicy to Disabled. However, when the user removes this annotation to resume normal scaling, the HPA's policy remains Disabled indefinitely.

Root Cause

The HPA update logic uses DeepDerivative to compare the desired HPA spec with the existing one:

!equality.Semantic.DeepDerivative(hpa.Spec, foundHpa.Spec)

DeepDerivative treats nil as "unset" and considers it a subset of any value. So when comparing:

• Desired: Behavior.ScaleDown.SelectPolicy = nil
• Existing: Behavior.ScaleDown.SelectPolicy = Disabled

DeepDerivative returns true (considers them equal), so no update is triggered.

How to Reproduce

1. Create a ScaledObject with the paused-scale-in annotation and custom behavior where selectPolicy is not explicitly defined, along with a dummy deployment for KEDA to scale:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: my-scaledobject
  annotations:
    autoscaling.keda.sh/paused-scale-in: "true"
spec:
  scaleTargetRef:
    name: my-deployment
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
    - type: cron
      metadata:
        timezone: UTC
        start: "0 * * * *"
        end: "1 * * * *"
        desiredReplicas: "2"
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          # selectPolicy is NOT defined - this is the condition for the bug
          stabilizationWindowSeconds: 600
          policies:
            - type: Percent
              value: 5
              periodSeconds: 300
        scaleUp:
          policies:
            - type: Percent
              value: 10
              periodSeconds: 120
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-deployment
  template:
    metadata:
      labels:
        app: my-deployment
    spec:
      containers:
        - name: nginx
          image: nginx:alpine

2. Verify the HPA has Disabled scale down policy:

kubectl get hpa keda-hpa-my-scaledobject -o jsonpath='{.spec.behavior.scaleDown.selectPolicy}'
Disabled

3. Remove the annotation (expecting behavior to be restored):

kubectl annotate scaledobject my-scaledobject autoscaling.keda.sh/paused-scale-in-

4. Check HPA behavior again - BUG: still shows "Disabled"

kubectl get hpa keda-hpa-my-scaledobject -o jsonpath='{.spec.behavior.scaleDown.selectPolicy}'
Disabled  ← Should be default

Conditions for bug to manifest

• ScaledObject has paused-scale-in=true or paused-scale-out=true annotation
• The ScaledObject does not have an explicit selectPolicy defined in spec.advanced.horizontalPodAutoscalerConfig.behavior.scaleDown (or scaleUp for paused-scale-out). Other fields like stabilizationWindowSeconds or policies can be defined - the bug still occurs.
• User deletes the annotation (vs. setting it to "false")

Solution

Add an explicit DeepEqual check for the Behavior field before the DeepDerivative check:

if len(hpa.Spec.Metrics) != len(foundHpa.Spec.Metrics) ||
    !equality.Semantic.DeepEqual(hpa.Spec.Behavior, foundHpa.Spec.Behavior) ||
    !equality.Semantic.DeepDerivative(hpa.Spec, foundHpa.Spec) {

DeepEqual correctly identifies that {SelectPolicy: nil} and {SelectPolicy: Disabled} are different, triggering the HPA update.

This follows the existing pattern already used for the Metrics length check, which also works around a DeepDerivative limitation.

Changes

File	Change
controllers/keda/hpa.go	Added DeepEqual check for Behavior field with explanatory comment
controllers/keda/scaledobject_controller_test.go	Added test case that verifies HPA behavior is restored when annotation is deleted

Testing

Added integration test removes scale down disabled policy when annotation removed without custom scaledown behavior that:

1. Creates a ScaledObject with paused-scale-in=true annotation and custom ScaleUp behavior (but no ScaleDown behavior)
2. Verifies HPA has Disabled ScaleDown policy
3. Deletes the annotation
4. Verifies HPA ScaleDown is no longer Disabled while custom ScaleUp config is preserved

[github-actions]

Thank you for your contribution! 🙏

Please understand that we will do our best to review your PR and give you feedback as soon as possible, but please bear with us if it takes a little longer as expected.

While you are waiting, make sure to:

• Add an entry in our changelog in alphabetical order and link related issue
• Update the documentation, if needed
• Add unit & e2e tests for your changes
• GitHub checks are passing
• Is the DCO check failing? Here is how you can fix DCO issues

Once the initial tests are successful, a KEDA member will ensure that the e2e tests are run. Once the e2e tests have been successfully completed, the PR may be merged at a later date. Please be patient.

Learn more about our contribution guide.

[snyk-io]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[JorTurFer]

nice fix!

[JorTurFer]

/run-e2e internal
Update: You can check the progress here

[Copilot]

Pull request overview

This PR fixes a bug where removing the paused-scale-in or paused-scale-out annotation from a ScaledObject fails to restore the HPA's default scaling behavior when the ScaledObject doesn't explicitly define a selectPolicy in its behavior configuration.

Key Changes

• Added explicit DeepEqual check for HPA Behavior field to detect nil-to-value changes that DeepDerivative misses
• Added comprehensive unit test verifying HPA behavior restoration when annotation is removed without custom ScaleDown configuration
• Added e2e test validating end-to-end scaling behavior after annotation removal

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
controllers/keda/hpa.go	Added DeepEqual check for Behavior field before DeepDerivative comparison to properly detect when paused annotations are removed and Behavior should change from Disabled back to nil
controllers/keda/scaledobject_controller_test.go	Added unit test that verifies HPA's ScaleDown policy is no longer Disabled after removing paused-scale-in annotation when no custom ScaleDown behavior is defined, while preserving custom ScaleUp configuration
tests/internals/pause_scale_in_restore/pause_scale_in_restore_test.go	Added e2e test that validates deployment scales down correctly after removing paused-scale-in annotation
CHANGELOG.md	Added entry documenting the fix with PR reference

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

This e2e test verifies the end-to-end behavior (that scaling works after removing the annotation), but doesn't directly verify that the HPA's ScaleDown.SelectPolicy is no longer Disabled. Consider adding an assertion to check the HPA spec directly, similar to what the unit test does in scaledobject_controller_test.go, to make the test more explicit about what it's verifying.

[Copilot]

The fix in hpa.go addresses the issue for both paused-scale-in and paused-scale-out annotations, but the new test only covers the paused-scale-in case. Consider adding a similar test case for paused-scale-out annotation removal to ensure complete coverage of the fix for both annotations.

[aliaqel-stripe]

good find!