Skip to content

Add GCSToAzureBlobStorageOperator for GCS to Azure Blob transfer#64966

Open
yuseok89 wants to merge 11 commits into
apache:mainfrom
yuseok89:feat/gcs-to-azure-blob-operator
Open

Add GCSToAzureBlobStorageOperator for GCS to Azure Blob transfer#64966
yuseok89 wants to merge 11 commits into
apache:mainfrom
yuseok89:feat/gcs-to-azure-blob-operator

Conversation

@yuseok89
Copy link
Copy Markdown
Contributor

@yuseok89 yuseok89 commented Apr 9, 2026

Adds GCSToAzureBlobStorageOperator to copy objects from Google Cloud Storage to Azure Blob Storage.
There was no existing operator for this path (GCS → Azure Blob), so this fills that gap. The implementation follows the same overall patterns as S3ToAzureBlobStorageOperator.

Test Screenshots

Source files (GCS)

image

Default structure

image image

Flatten structure

image image
Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    • Cursor

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@yuseok89 yuseok89 force-pushed the feat/gcs-to-azure-blob-operator branch from 0090bd7 to 104ab09 Compare April 11, 2026 14:54
@yuseok89 yuseok89 marked this pull request as ready for review April 12, 2026 12:31
@potiuk potiuk marked this pull request as draft April 22, 2026 18:10
@potiuk
Copy link
Copy Markdown
Member

potiuk commented Apr 22, 2026

@yuseok89 Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.

  • Merge conflicts: this branch has conflicts with main and cannot be merged as-is. Rebase onto the current main to resolve: git fetch upstream main && git rebase upstream/main, resolve, force-push.

See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush.


Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

@yuseok89 yuseok89 force-pushed the feat/gcs-to-azure-blob-operator branch 2 times, most recently from 753db45 to 3809d5f Compare April 24, 2026 14:01
@yuseok89 yuseok89 marked this pull request as ready for review April 25, 2026 12:59
Copy link
Copy Markdown
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reading after the local rebase and noticed a few correctness issues worth a fixup before merge. Inline notes follow — items 1, 4, 6, 7 are real bugs (1 and 6 are the only ones I'd consider blocking), the rest are style/coverage. Items 1 and 4 have one-click suggestions; 2, 3, 5, 6, 7 are discussion comments because the fix touches non-contiguous lines.

Nice operator overall — much cleaner shape than the sibling S3ToAzureBlobStorageOperator (separate flatten_structure / keep_directory_structure semantics, OpenLineage facets, optional-dep handling via AirflowOptionalProviderFeatureException). The four issues below are mostly polish.


This review was generated by AI and may contain mistakes — please weigh it accordingly. Once you've had a look, a human maintainer will do the final review on the PR.

@potiuk potiuk force-pushed the feat/gcs-to-azure-blob-operator branch from f06b287 to 106edec Compare April 26, 2026 16:02
@yuseok89 yuseok89 force-pushed the feat/gcs-to-azure-blob-operator branch from 106edec to cb1488d Compare April 27, 2026 14:36
@yuseok89
Copy link
Copy Markdown
Contributor Author

@potiuk
Thanks again for the thorough review.
Every point landed and I learned a lot. The Windows path issue and the retry-mutation bug in particular were not on my radar. I'll be more careful with these patterns next time and think about how to catch them earlier on my side before the PR.

@yuseok89 yuseok89 force-pushed the feat/gcs-to-azure-blob-operator branch from a11f684 to 1dbd4c3 Compare April 28, 2026 14:35
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label May 5, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 9, 2026

uv.lock on main just moved via #66631 ("[main] Upgrade important CI environment"), commit 1aff9b6 and this PR currently conflicts.

Quickest fix:

git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease

Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future uv.lock bumps.

@yuseok89 yuseok89 force-pushed the feat/gcs-to-azure-blob-operator branch from 36951bc to c174e00 Compare May 9, 2026 14:17
@yuseok89 yuseok89 force-pushed the feat/gcs-to-azure-blob-operator branch from c174e00 to 042b8eb Compare May 10, 2026 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers kind:documentation provider:microsoft-azure Azure-related issues ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants