Skip to content

fix(node): handle SIGTERM across startup phases#6620

Open
paolino wants to merge 3 commits into
IntersectMBO:masterfrom
lambdasistemi:fix/sigterm-startup-shutdown
Open

fix(node): handle SIGTERM across startup phases#6620
paolino wants to merge 3 commits into
IntersectMBO:masterfrom
lambdasistemi:fix/sigterm-startup-shutdown

Conversation

@paolino

@paolino paolino commented Jul 16, 2026

Copy link
Copy Markdown

Description

Handle SIGTERM correctly while cardano-node is loading its configuration.

A SIGTERM received during startup was previously delivered as ExitSuccess. Configuration parsing could catch and wrap that exception, producing a generic parse error and making the process exit with status 1 instead of shutting down successfully.

This change introduces a phase-aware signal path:

  • During startup, SIGTERM throws a dedicated asynchronous SigTermException.
  • The top-level exception handler converts it to ExitSuccess.
  • Immediately before diffusion starts, the signal handler switches back to ExitSuccess, which the diffusion layer already recognizes as an expected runtime shutdown.

This prevents configuration parsers from treating startup termination as a parsing failure while preserving the existing runtime shutdown behavior.

Regression tests

The deterministic cardano-node-test properties cover the signal path without relying on process-start timing:

  • SigTermException is classified as SomeAsyncException, so ordinary configuration-parser exception handling cannot wrap it.
  • The top-level exception handler converts startup SigTermException to ExitSuccess.
  • Startup and runtime phases select their intended exception types.
  • On Unix, the real POSIX handler is installed before the test raises SIGTERM; the top-level handler must then exit successfully.

The installed-handler property passed 100 tests locally. The previous process-level test was removed because its fixed startup delay raced with handler installation on Linux and macOS CI.

Antithesis reproduction

Antithesis originally exposed this intermittent failure in run 75fa032ed4fd4931ff63d020d9f48208-56-17.

Failing property:

container: p4, exit code: 1

Relevant run logs:

vtime 5404.813501  p4 stderr: OtherParseException ExitSuccess
vtime 5404.813501  p4 stderr: cardano-node: Generic parse exception:
vtime 5404.813501  p4 stderr: ExitSuccess
vtime 5405.110873  containers_meta: p4 died, exit code 1

The patched image was subsequently exercised in three-hour Antithesis run ce9c380a77c6c17f77edae0f35146a08-56-17:

  • 65 properties passed, 0 failed
  • OtherParseException: 0 events
  • Generic parse exception: 0 events

The clean three-hour run is broader fault-injection regression coverage, not proof of the fix by itself: the failure is intermittent and unpatched runs also completed without reproducing it. The deterministic properties above provide red/green coverage of async classification, top-level conversion, phase selection, and actual POSIX signal delivery.

The Antithesis reports are restricted to authorized tenant users, so the stable run IDs and reviewer-safe log excerpts are included here instead of access-dependent report URLs.

Local verification

  • nix develop --quiet -c cabal test cardano-node-test passes; all five top-level SIGTERM properties pass 100 tests.
  • nix build --no-link .#hydraJobs.x86_64-linux.windows.cardano-node passes the same Windows cross-build that previously failed under -Werror.
  • nix develop --quiet -c cabal build cardano-testnet-test passes after removing the timing-based property.
  • hlint reports No hints for all touched Haskell files.

Checklist

  • Commit sequence broadly makes sense and commits have useful messages
  • New tests are added if needed and existing tests are updated
  • No additional changelog entry is needed; the existing Next-version entry already notes clean exit on SIGTERM
  • No version bounds need updating; there are no dependency changes
  • CI passes. The following CI checks are required:
    • Code is linted with hlint
    • Code is formatted with stylish-haskell
    • Code builds on Linux, macOS and Windows for ghc-9.6 and ghc-9.12
  • Self-reviewed the diff

Note on CI

This PR is from a fork, so the necessary CI jobs will need to be triggered by someone with write privileges.

@paolino
paolino requested a review from a team as a code owner July 16, 2026 15:32
@paolino
paolino marked this pull request as draft July 16, 2026 15:33
@paolino
paolino force-pushed the fix/sigterm-startup-shutdown branch from 508760c to 9bf4bfa Compare July 16, 2026 17:26
@paolino
paolino marked this pull request as ready for review July 17, 2026 06:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant