Skip to content

Snapshot Changes should have a Streaming Interface #15659

@RussellSpitzer

Description

@RussellSpitzer

Feature Request / Improvement

Currently SnapshotChanges only provides a caching interface that eagerly materializes all file changes into in-memory lists and returns Iterable. There are use cases where returning a streaming CloseableIterable would be preferable see #13556 for example.

I think we probably should just add CloseableIterable interfaces to SnapshotChanges as well and just have the caching calls wrap them.

Query engine

None

Willingness to contribute

  • I can contribute this improvement/feature independently
  • I would be willing to contribute this improvement/feature with guidance from the Iceberg community
  • I cannot contribute this improvement/feature at this time

Metadata

Metadata

Assignees

No one assigned

    Labels

    improvementPR that improves existing functionality

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions