Skip to content

[Pipe] Support receiver-side async takeover for TsFile conversion - #18421

Open
Caideyipi wants to merge 5 commits into
apache:masterfrom
Caideyipi:pipe-tsfile-async-conversion
Open

[Pipe] Support receiver-side async takeover for TsFile conversion#18421
Caideyipi wants to merge 5 commits into
apache:masterfrom
Caideyipi:pipe-tsfile-async-conversion

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

What does this PR do?

Adds resilient receiver-side handling for Pipe TsFile type-conversion failures.

  • Adds connector.exception.data.convert-on-type-mismatch.tsfile.async-load (and the sink.* alias), defaulting to true, and propagates the setting with a stable conversion task ID.
  • Deduplicates seal and retry requests by deterministic task ID. Retries reuse the existing task status and context instead of starting duplicate conversions.
  • On a type-mismatch conversion failure, atomically hands the TsFile and mod file to the receiver active-load async directory and marks the task receiver-owned.
  • Treats temporary unavailability and OOM as a pause of local conversion only; input files and parser progress remain retryable.
  • Reuses Pipe parser state and checkpoints across retries, including iterators, pending tablets, page and chunk metadata, file index, and pending statements.
  • When async takeover is disabled, sender retries return the current task status and never create a second conversion task.

Tests

  • Added or updated parser, receiver, transfer-request, active-load path and scanner, load utility, and conversion-task-manager unit tests; 85 targeted tests passed.
  • DataNode compilation passed.
  • Spotless and git diff --check passed.

…onversion

# Conflicts:
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/parser/scan/TsFileInsertionEventScanParser.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/parser/table/TsFileInsertionEventTableParser.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/parser/table/TsFileInsertionEventTableParserTabletIterator.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/converter/LoadTableStatementDataTypeConvertExecutionVisitor.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/converter/LoadTreeTsFileTabletIterator.java
#	iotdb-core/datanode/src/test/java/org/apache/iotdb/db/pipe/event/TsFileInsertionEventParserTest.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants