Add GCSToAzureBlobStorageOperator for GCS to Azure Blob transfer#64966
Add GCSToAzureBlobStorageOperator for GCS to Azure Blob transfer#64966yuseok89 wants to merge 11 commits into
Conversation
715aaeb to
20b3f6f
Compare
0090bd7 to
104ab09
Compare
|
@yuseok89 Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.
See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush. Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you. |
753db45 to
3809d5f
Compare
potiuk
left a comment
There was a problem hiding this comment.
Re-reading after the local rebase and noticed a few correctness issues worth a fixup before merge. Inline notes follow — items 1, 4, 6, 7 are real bugs (1 and 6 are the only ones I'd consider blocking), the rest are style/coverage. Items 1 and 4 have one-click suggestions; 2, 3, 5, 6, 7 are discussion comments because the fix touches non-contiguous lines.
Nice operator overall — much cleaner shape than the sibling S3ToAzureBlobStorageOperator (separate flatten_structure / keep_directory_structure semantics, OpenLineage facets, optional-dep handling via AirflowOptionalProviderFeatureException). The four issues below are mostly polish.
This review was generated by AI and may contain mistakes — please weigh it accordingly. Once you've had a look, a human maintainer will do the final review on the PR.
f06b287 to
106edec
Compare
106edec to
cb1488d
Compare
|
@potiuk |
a11f684 to
1dbd4c3
Compare
|
Quickest fix: git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-leaseAutomated nudge — ignore if you're not ready to rebase. This comment is updated in place on future |
36951bc to
c174e00
Compare
…e/transfers/gcs_to_wasb.py Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
…e/transfers/gcs_to_wasb.py Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
c174e00 to
042b8eb
Compare
Adds
GCSToAzureBlobStorageOperatorto copy objects from Google Cloud Storage to Azure Blob Storage.There was no existing operator for this path (GCS → Azure Blob), so this fills that gap. The implementation follows the same overall patterns as
S3ToAzureBlobStorageOperator.Test Screenshots
Source files (GCS)
Default structure
Flatten structure
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.