Add IBM MQ provider#62790
Conversation
|
Since this PR needs approval the community, maybe put in draft to avoid any accidental merge? |
|
Looks good for me. Would you add yourself as CODEOWNER for this? Do you know somebody (even non committer) which would help as steward? |
563ed98 to
55b67e5
Compare
3c85c36 to
014dd51
Compare
47f961b to
aceb1a5
Compare
There was a problem hiding this comment.
Pull request overview
This PR introduces a new IBM MQ provider to Apache Airflow, integrating IBM MQ with Airflow’s event-driven scheduling via the MessageQueueProvider abstraction and adding an IBMMQHook plus an MQ trigger.
Changes:
- Adds
apache-airflow-providers-ibm-mqprovider package (hook, trigger, message-queue provider) with unit/system tests. - Wires the new provider into the monorepo/workspace (extras, members, CI docker-compose test sources, mypy paths, labels, CODEOWNERS, issue template).
- Adds provider documentation pages and updates spelling wordlist and Breeze docs artifacts.
Reviewed changes
Copilot reviewed 62 out of 70 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py |
Implements IBMMQHook with sync/async consume/produce and reconnection/backoff. |
providers/ibm/mq/src/airflow/providers/ibm/mq/triggers/mq.py |
Adds AwaitMessageTrigger that polls IBM MQ via the hook and yields TriggerEvent. |
providers/ibm/mq/src/airflow/providers/ibm/mq/queues/mq.py |
Adds IBMMQMessageQueueProvider to integrate with common.messaging’s MessageQueueTrigger. |
providers/ibm/mq/tests/** |
Adds unit tests for hook/trigger/provider and a system-test example DAG. |
providers/ibm/mq/docs/**, providers/ibm/mq/README.rst |
Adds provider docs, connection docs, and generated README/changelog. |
pyproject.toml, uv.lock |
Registers provider in workspace + extras and locks optional ibmmq dependency. |
.github/*, scripts/ci/docker-compose/* |
CI wiring, labels, CODEOWNERS, and issue-template updates for the new provider. |
airflow-core/docs/extra-packages-ref.rst |
Adds the ibm-mq extra to docs. |
900886b to
10bf39b
Compare
…_trigger.py Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
…sent in URI when defining IBMMQMessageQueueProvider
…er due to incubation status
|
Quickest fix: git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-leaseAutomated nudge — ignore if you're not ready to rebase. This comment is updated in place on future |
…eError will be raised anyway if not an integer
With the introduction of event-driven scheduling and the MessageQueueProvider abstraction in Airflow, it has become significantly easier to trigger DAGs from external message brokers as described in Astronomer's guide on event-driven scheduling.
Many enterprises still rely heavily on IBM MQ as their primary enterprise messaging backbone. However, at the moment there is no official Airflow provider supporting IBM MQ.
This implementation consists of:
This allows IBM MQ to function similarly to Kafka, SQS, etc., within the Airflow event-driven scheduling framework.
The implementation is built on top of the open-source IBM MQ Python wrapper:
IBM has recently released and documented their modern Python binding here:
https://community.ibm.com/community/user/blogs/dylan-goode/2025/10/16/new-python-binding-for-ibm-mq
The hook supports:
The MessageQueueProvider implementation integrates with Airflow's event-driven scheduling so that DAGs can be triggered based on IBM MQ messages.
Why this might make sense:
I am willing to act as initial maintainer and code owner, of course this is purely a proposition.
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.