Skip to content

Add IBM MQ provider#62790

Open
dabla wants to merge 45 commits into
apache:mainfrom
dabla:feature/ibm-mq-provider
Open

Add IBM MQ provider#62790
dabla wants to merge 45 commits into
apache:mainfrom
dabla:feature/ibm-mq-provider

Conversation

@dabla
Copy link
Copy Markdown
Contributor

@dabla dabla commented Mar 3, 2026

With the introduction of event-driven scheduling and the MessageQueueProvider abstraction in Airflow, it has become significantly easier to trigger DAGs from external message brokers as described in Astronomer's guide on event-driven scheduling.

Many enterprises still rely heavily on IBM MQ as their primary enterprise messaging backbone. However, at the moment there is no official Airflow provider supporting IBM MQ.

This implementation consists of:

  • An IBMMQHook
  • A MessageQueueProvider implementation for IBM MQ
  • The ability to trigger DAGs from IBM MQ events
  • Standard producer/consumer patterns from within tasks

This allows IBM MQ to function similarly to Kafka, SQS, etc., within the Airflow event-driven scheduling framework.

The implementation is built on top of the open-source IBM MQ Python wrapper:

  • IBM MQ Python ibmmq library

IBM has recently released and documented their modern Python binding here:

https://community.ibm.com/community/user/blogs/dylan-goode/2025/10/16/new-python-binding-for-ibm-mq

The hook supports:

  • Secure connections (TLS) (to be done)
  • Queue get/put operations
  • Configurable polling behaviour

The MessageQueueProvider implementation integrates with Airflow's event-driven scheduling so that DAGs can be triggered based on IBM MQ messages.

Why this might make sense:

  • IBM MQ is still widely used in regulated industries (banking, insurance, government).
  • Many enterprises using Airflow also run IBM MQ.
  • This would allow IBM MQ to be a first-class citizen in Airflow's event-driven ecosystem.
  • The dependency is officially maintained by IBM and open source.

I am willing to act as initial maintainer and code owner, of course this is purely a proposition.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@vincbeck
Copy link
Copy Markdown
Contributor

vincbeck commented Mar 3, 2026

Since this PR needs approval the community, maybe put in draft to avoid any accidental merge?

@dabla dabla marked this pull request as draft March 3, 2026 15:25
@jscheffl
Copy link
Copy Markdown
Contributor

jscheffl commented Mar 4, 2026

Looks good for me. Would you add yourself as CODEOWNER for this? Do you know somebody (even non committer) which would help as steward?

Comment thread providers/ibm/mq/provider.yaml
@dabla dabla force-pushed the feature/ibm-mq-provider branch from 563ed98 to 55b67e5 Compare March 10, 2026 19:25
@dabla dabla force-pushed the feature/ibm-mq-provider branch 2 times, most recently from 3c85c36 to 014dd51 Compare March 18, 2026 20:21
Copy link
Copy Markdown
Member

@kaxil kaxil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 5 new issues from the 30 commits pushed since last review.

Comment thread providers/ibm/mq/tests/unit/ibm/mq/hooks/test_mq.py
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/triggers/mq.py Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py Outdated
Comment thread providers/ibm/mq/docs/connections/mq.rst Outdated
Comment thread providers/ibm/mq/provider.yaml
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/get_provider_info.py
Comment thread providers/ibm/mq/pyproject.toml
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/queues/mq.py Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/__init__.py
@dabla dabla force-pushed the feature/ibm-mq-provider branch 2 times, most recently from 47f961b to aceb1a5 Compare March 23, 2026 17:02
@kaxil kaxil requested a review from Copilot March 24, 2026 23:13
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new IBM MQ provider to Apache Airflow, integrating IBM MQ with Airflow’s event-driven scheduling via the MessageQueueProvider abstraction and adding an IBMMQHook plus an MQ trigger.

Changes:

  • Adds apache-airflow-providers-ibm-mq provider package (hook, trigger, message-queue provider) with unit/system tests.
  • Wires the new provider into the monorepo/workspace (extras, members, CI docker-compose test sources, mypy paths, labels, CODEOWNERS, issue template).
  • Adds provider documentation pages and updates spelling wordlist and Breeze docs artifacts.

Reviewed changes

Copilot reviewed 62 out of 70 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py Implements IBMMQHook with sync/async consume/produce and reconnection/backoff.
providers/ibm/mq/src/airflow/providers/ibm/mq/triggers/mq.py Adds AwaitMessageTrigger that polls IBM MQ via the hook and yields TriggerEvent.
providers/ibm/mq/src/airflow/providers/ibm/mq/queues/mq.py Adds IBMMQMessageQueueProvider to integrate with common.messaging’s MessageQueueTrigger.
providers/ibm/mq/tests/** Adds unit tests for hook/trigger/provider and a system-test example DAG.
providers/ibm/mq/docs/**, providers/ibm/mq/README.rst Adds provider docs, connection docs, and generated README/changelog.
pyproject.toml, uv.lock Registers provider in workspace + extras and locks optional ibmmq dependency.
.github/*, scripts/ci/docker-compose/* CI wiring, labels, CODEOWNERS, and issue-template updates for the new provider.
airflow-core/docs/extra-packages-ref.rst Adds the ibm-mq extra to docs.

Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/queues/mq.py
Comment thread providers/ibm/mq/docs/changelog.rst Outdated
Comment thread providers/ibm/mq/docs/changelog.rst Outdated
Comment thread providers/ibm/mq/tests/unit/ibm/mq/__init__.py Outdated
Comment thread providers/ibm/mq/tests/unit/ibm/mq/hooks/test_mq.py
Comment thread airflow-core/docs/extra-packages-ref.rst Outdated
Comment thread providers/ibm/mq/README.rst Outdated
@dabla dabla marked this pull request as ready for review March 25, 2026 18:58
Comment thread providers/ibm/mq/tests/system/ibm/mq/example_dag_message_queue_trigger.py Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/queues/mq.py Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/queues/mq.py Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/hooks/mq.py Outdated
Comment thread providers/ibm/mq/docs/connections/mq.rst Outdated
Comment thread airflow-core/docs/extra-packages-ref.rst Outdated
Comment thread providers/ibm/mq/src/airflow/providers/ibm/mq/queues/mq.py
Comment thread providers/ibm/mq/provider.yaml Outdated
@dabla dabla force-pushed the feature/ibm-mq-provider branch from 900886b to 10bf39b Compare May 2, 2026 16:57
@dabla dabla requested a review from kaxil May 3, 2026 09:12
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 9, 2026

uv.lock on main just moved via #66631 ("[main] Upgrade important CI environment"), commit 1aff9b6 and this PR currently conflicts.

Quickest fix:

git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease

Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future uv.lock bumps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants