Skip to content

Support Iceberg SparkCopyOnWriteScan in IcebergReflection.ICEBERG_SCAN_CLASSES #5319

Description

@unikdahal

What is the problem the feature request solves?

When running Copy-On-Write (CoW) table operations such as MERGE INTO, UPDATE, or DELETE against Iceberg tables in Spark Iceberg's Spark planning extension creates a
SparkCopyOnWriteScan (org.apache.iceberg.spark.source.SparkCopyOnWriteScan) for scanning the target table data files.

Found the gap while testing native iceberg writes #4487

Currently, IcebergReflection.ICEBERG_SCAN_CLASSES only includes:

• org.apache.iceberg.spark.source.SparkBatchQueryScan
• org.apache.iceberg.spark.source.SparkStagedScan

Because SparkCopyOnWriteScan is missing from ICEBERG_SCAN_CLASSES, IcebergReflection.isIcebergScanClass(...) returns false when CometScanRule evaluates a physical plan
containing a SparkCopyOnWriteScan. As a result, Comet fails to recognize SparkCopyOnWriteScan as a supported Iceberg scan and falls back to JVM Spark execution for the scan
phase of Copy-On-Write queries.

Describe the potential solution

Add SparkCopyOnWriteScan ("org.apache.iceberg.spark.source.SparkCopyOnWriteScan") to IcebergReflection.ClassNames and include it in IcebergReflection.ICEBERG_SCAN_CLASSES.

// In IcebergReflection.scala
object ClassNames {
  ...
  val SPARK_COPY_ON_WRITE_SCAN = "org.apache.iceberg.spark.source.SparkCopyOnWriteScan"
}

val ICEBERG_SCAN_CLASSES: Set[String] =
  Set(
    ClassNames.SPARK_BATCH_QUERY_SCAN,
    ClassNames.SPARK_STAGED_SCAN,
    ClassNames.SPARK_COPY_ON_WRITE_SCAN)

This enables CometScanRule to recognize SparkCopyOnWriteScan instances and convert them to native Comet scans during Iceberg Copy-On-Write MERGE, UPDATE, and DELETE
operations.

Additional context

SparkCopyOnWriteScan extends Iceberg's standard Spark scan primitives (SparkBatchQueryScan / SparkScan) and behaves identically with respect to file tasks, table schemas, and
filter pushdowns during data file reads.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:scanParquet scan / data readingenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions