Skip to content

fix(pipe): reuse safe local progress on alter - #18423

Merged
jt2594838 merged 2 commits into
apache:masterfrom
Caideyipi:fix/pipe-alter-local-progress
Aug 11, 2026
Merged

fix(pipe): reuse safe local progress on alter#18423
jt2594838 merged 2 commits into
apache:masterfrom
Caideyipi:fix/pipe-alter-local-progress

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Summary:

  • Reuse a local committed PipeTask progress index during alter only when the old and new realtime-only modes are unchanged, the leader remains on this DataNode, and the old task is actually local.
  • Preserve ConfigNode progress monotonically and retain MinimumProgressIndex reset semantics for mode changes and leader migration.
  • Process alter's dropped and updated PipeMeta entries in one DataNode task-agent invocation, with timeout handling.

Tests:

  • mvn spotless:check -pl iotdb-core/datanode
  • 14 targeted JUnit tests passed, covering local progress, ConfigNode progress precedence, mode reset, both leader-migration directions, missing local tasks, batch handling, timeout handling, and existing multi-pipe error behavior.
  • Full Maven compile remains blocked by this workspace's existing memory pressure and generated Thrift API mismatch.

@jt2594838 jt2594838 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to add an IT that tests altering an pipe does not re-send already received data?

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Added testAlterPipeDoesNotResendCommittedData in 5fb019b. It transfers two rows, deletes them on the receiver, alters the sink configuration, asserts that the deleted rows are not sent again, then writes a third row and verifies that replication remains active. The sender heartbeat interval is set to 600 seconds so the test covers stale ConfigNode metadata instead of relying on a periodic heartbeat. The targeted IT passes locally.

@jt2594838
jt2594838 merged commit ef0a792 into apache:master Aug 11, 2026
40 of 41 checks passed
@jt2594838
jt2594838 deleted the fix/pipe-alter-local-progress branch August 11, 2026 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants