Vault: automated stale pending queue purge

[russell-stern]

Summary

Adds a gated recovery path when the vault KV pending queue is stuck and blocking observation quorum, so ops no longer need to leave VaultForceEmptyOCRRounds enabled globally.

Depends on chainlink-common#2104 (StoredPendingQueueIndex.written_seq_nr, CRE settings VaultPendingQueueStaleAutoEmpty and VaultPendingQueueStaleRoundThreshold).

Stale pending queue auto-empty (gated, default off)

• When VaultPendingQueueStaleAutoEmpty is on and seqNr - written_seq_nr >= VaultPendingQueueStaleRoundThreshold (default 30), Observation and ValidateObservation skip store-backed pending items for that round—same effect as force-empty for the KV queue only, so a successful StateTransition can rewrite/clear stuck entries.
• written_seq_nr is written to the pending queue index only when the gate is enabled, keeping KV index bytes identical to pre-change {length} serialization while the feature is off (avoids rollout / determinism issues).
• Backwards compatibility: if the queue is non-empty but written_seq_nr == 0, auto-empty does not run until the first post-enable WritePendingQueue sets the watermark.
• VaultForceEmptyOCRRounds is unchanged and still takes precedence when enabled (manual escape hatch).

Local queue hardening (ungated)

• prepareObservationPendingQueueBlobs skips a single oversize local-queue blob instead of failing the whole observation.

Observability

• Beholder counters for stale auto-empty and skipped local blobs.

[github-actions]

I see you updated files related to core. Please run make gocs in the root directory to add a changeset as well as in the text include at least one of the following tags:

• #added For any new functionality added.
• #breaking_change For any functionality that requires manual action for the node to boot.
• #bugfix For bug fixes.
• #changed For any change to the existing functionality.
• #db_update For any feature that introduces updates to database schema.
• #deprecation_notice For any upcoming deprecation functionality.
• #internal For changesets that need to be excluded from the final changelog.
• #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
• #removed For any functionality/config that is removed.
• #updated For any functionality that is updated.
• #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

[github-actions]

✅ No conflicts with other open PRs targeting develop

[trunk-io]

     

Failed Test	Failure Summary	Logs
TestConfidentialRelayHandler_QuorumWithRealAggregator		Logs ↗︎
TestHandler_HandleGatewayMessage/capability_execute_error_returned_in_result		Logs ↗︎
TestAggregator_quorumUnobtainable	The test 'TestAggregator_quorumUnobtainable' failed during execution.	Logs ↗︎
TestAggregator_skipsTransportErrorsFromQuorum	The test 'TestAggregator_skipsTransportErrorsFromQuorum' failed during execution.	Logs ↗︎

... and 10 more

View Full Report ↗︎ ⋅ Docs

[cl-sonarqube-production]

Quality Gate passed

Issues
2 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube