Add dbt-factory bundle template and example to contrib

[mwojtyczka]

What this adds

By default, running dbt on Databricks executes your whole dbt project as a single, opaque Workflow task — one green or red box. You can't see which model failed, you can't rerun just the failed models, and independent models don't run in parallel.

This PR provides a template that turns a dbt project into a Databricks Workflow with one task per dbt object
(model, seed, snapshot, test), with dependencies wired to match your dbt DAG. That gives you:

• Faster execution — independent models run in parallel, and the notebook task type keeps
  dbt's dependencies pre-cached in the serverless environment, avoiding a cold start on every task.
• Visibility & simplified troubleshooting — pinpoint failures at the model level in the UI.
• Enhanced logging & notifications — per-task logs and precise, model-level error alerts.
• Improved retriability — retry only the failed model tasks without rerunning the whole project.
• Seamless testing — dbt data tests run as their own tasks right after each model finishes.

It ships two things:

• contrib/dbt_factory/ — a complete, deployable example you can clone and run.
• contrib/templates/dbt-factory/ — a databricks bundle init template that scaffolds a new dbt project already wired up this way (bring your own models, or migrate an existing project).

How it works

flowchart TD
    subgraph setup["One-time setup"]
      T["dbt-factory bundle template"] -->|databricks bundle init| B["Scaffolded project:<br/>dbt project + PyDABs hook + factory code"]
      X["Existing dbt project<br/>(optional)"] -.->|move models/seeds/... into src/| B
    end
    subgraph deploy["Every deploy"]
      C["make manifest<br/>(dbt parse)"] --> D["target/manifest.json"]
      D --> E["databricks bundle deploy"]
      E --> F["PyDABs load_resources reads the<br/>manifest and generates the job"]
    end
    subgraph runtime["At run time — serverless"]
      G["Databricks Workflow:<br/>one task per model / seed / snapshot / test"] --> H["Each task triggers dbt<br/>via the runner notebook"]
      H --> I[("SQL warehouse")]
    end
    B --> C
    F --> G

    classDef optional stroke:#999,stroke-dasharray:5 4,color:#888;
    class X optional;

Loading

The job is not written as YAML. It's generated at databricks bundle deploy time from the dbt manifest via the PyDABs
python.resources hook: resources/__init__.py's load_resources reads target/manifest.json and builds one Databricks task per dbt node, wiring up dependencies. No per-model YAML is checked in — the task graph tracks the dbt DAG automatically.

The generation logic comes from the databricks-dbt-factory library, whose source is included under src/databricks_dbt_factory/ (adapted from commit 29866ed, v0.3.5, MIT — see NOTICE for attribution). The only integration code is the small resources/__init__.py.

Design choices

• Serverless compute and the notebook task type (dbt runs via a small runner notebook using dbtRunner) for the fastest task start times.
• A minimal set of exposed options: bundle_tests, environment_key, extra_dbt_command_options. Target / warehouse / catalog / schema live in dbt_profiles/profiles.yml.
• The example commits a target/manifest.json so it deploys out of the box; make manifest regenerates it. The README covers migrating an existing dbt project (init, then move your dbt files into the generated project).
• Ships both an example and a template
• Use notebook task (not dbt task):
  • Performance improvements
    • Utilisation of base environments > faster task init, especially on multi layer dag's
    • pre-built partial_parse.msgpack and injecting it via dbtRunner(manifest=) > skips many file reads on shared WS, especially impactful on large production dbt setups with 1000's of models & tests
    • local write targets
  • more flexibility for customers to add functionality of their own

Testing

• Deployed and ran end-to-end on a serverless SQL warehouse: databricks bundle deploy + databricks bundle run completes SUCCESS with all model and test tasks passing.
• Offline test suite (make test) passes — including an integration test that exercises load_resources against the committed manifest with no workspace.
• databricks bundle init on the template renders cleanly; the generated project's resources/__init__.py compiles and its bundle validates.
• Passes ruff format --check (the repo's fmt CI check).

Benchmarks

• 17min single dbt task
• 29min dbt task type with dbt factory task level split
• 19min notebook task type with dbt factory task level split

Note for reviewers

The databricks_dbt_factory/ core (and its tests) appear twice: once in the example (contrib/dbt_factory/) and once in the template (contrib/templates/dbt-factory/template/{{.project_name}}/). This is intentional, not an
oversight:

• Each artifact must be self-contained. The example is meant to be cloned and run as-is, and databricks bundle init can only stamp out files that live under the template's template/ directory — a template file can't reference code outside it. So both need their own copy.
• Consistent with repo precedent. This mirrors the existing contrib/templates/data-engineering + contrib/data_engineering pairing, which likewise duplicates its shared files (e.g. scripts/, conftest.py) between template and example.

The code is owned by this repo (bundle-examples) — the NOTICE files credit the original databricks-dbt-factory source for attribution and it passes ruff format --check.

Vendored-code fidelity. The six core files (dbt_factory.py, dbt_task.py, task_factory.py, utils.py, __version__.py, notebook/run_dbt_command.py) are byte-for-byte identical between the example (contrib/dbt_factory/) and the template (contrib/templates/dbt-factory/template/{{.project_name}}/) — verified with diff. Relative to the upstream databricks-dbt-factory v0.3.5 (commit 29866ed) they are AST-identical (semantically the same); the only textual differences from upstream are cosmetic — quote style normalized to double quotes and this repo's formatting. Because the template's ruff config lives in pyproject.toml.tmpl (which ruff can't discover), a small .ruff.toml at the template root (a sibling of databricks_template_schema.json, outside template/ so bundle init never renders it) sets line-length = 120 so the root ruff format --check validates the template sources at the same width as the example.

Slight downside: a future update to the core means editing it in both places (re-synced from the pinned upstream commit). Low-cost in practice — it's stable code touched only on version bumps, not something maintained in-repo day to day.

Live testing

Example project from the template:

More complex example:

[MaximHammer]

Quick Screen capture of usage https://drive.google.com/file/d/1VcEkn6Hc3JOUYyOGFdUGQA2mLSJx4sg8

[MaximHammer]

Additional Context on why not native dbt task type but rather a notebook task type for optimal speed:

• utilisation of base environments > faster task init, especially on multi layer dag's
• pre-built partial_parse.msgpack and injecting it via dbtRunner(manifest=) > skips many file reads on shared WS, especially impactful on large production dbt setups with 1000's of models & tests
• local write targets

2 not speed related reasons:

• notebook task type also provides extreme flexibility for customers to add functionality of their own
• we can easily support both task types just as the original dbt factory code - but do not see a clear value add for the extra complexity

[pietern]

Spurious newline.

[mwojtyczka]

removed

[pietern]

If these are specific to DBT factory, I recommend using DBT_FACTORY_zzz env vars.

[mwojtyczka]

fixed

[pietern]

Triggers a warning if empty. You can fix with a .gitkeep in the template that is always generated.

[mwojtyczka]

done

[pietern]

Can you do a search/replace in these trees for "Declarative Automation Bundles"

[mwojtyczka]

done