Add dbt-factory bundle template and example to contrib

contrib/dbt_factory/.gitignore

@@ -0,0 +1,19 @@
+.venv/
+__pycache__/
+*.pyc
+.databricks/
+logs/
+# Ignore installed dbt packages but keep the directory (via .gitkeep) so the
+# `dbt_packages/**` sync.include in databricks.yml always matches at least one file.
+dbt_packages/*
+!dbt_packages/.gitkeep
+dbt_profiles/.user.yml
+uv.lock
+
+# dbt build artifacts, but keep the committed manifest that resources/__init__.py reads at
+# deploy time. Regenerate it with `make manifest`.
+target/*
+!target/manifest.json
+
+# Serverless base-environment file, regenerated on every `bundle deploy` by resources/__init__.py.
+dbt_serverless_env.yaml

contrib/dbt_factory/Makefile

@@ -0,0 +1,40 @@
+.PHONY: setup deps manifest validate deploy run test test-update-expected-tasks test-e2e
+
+# Install dependencies into the .venv used by the bundle (databricks.yml -> python.venv_path).
+setup:
+	uv sync --dev
+
+# Install dbt package dependencies declared in packages.yml / dependencies.yml (if any).
+# `dbt deps` empties dbt_packages/, so restore the .gitkeep afterwards; it keeps the directory
+# present so the `dbt_packages/**` sync.include in databricks.yml always matches at least one file.
+deps:
+	uv run dbt deps
+	touch dbt_packages/.gitkeep
+
+# Regenerate the dbt manifest that resources/__init__.py reads at deploy time.
+# `dbt parse` does not connect to a warehouse; it only reads the project files.
+manifest: deps
+	uv run dbt parse --profiles-dir dbt_profiles
+
+validate:
+	databricks bundle validate
+
+# Regenerate the manifest and deploy the generated job to the dev target.
+deploy: manifest
+	databricks bundle deploy --target dev
+
+run:
+	databricks bundle run dbt_factory_job
+
+test:
+	uv run pytest tests
+
+# Refresh the saved "expected tasks" file the tests compare against, after you intentionally
+# change the generated output.
+test-update-expected-tasks:
+	uv run python tests/update_expected_tasks.py
+
+# End-to-end test: generate a project from the template, deploy the factory-generated job to your
+# workspace, run it, verify the output, and tear it all down. Needs DBT_FACTORY_* env (see tests/e2e/run_e2e.py).
+test-e2e:
+	uv run python tests/e2e/run_e2e.py

contrib/dbt_factory/NOTICE

@@ -0,0 +1,34 @@
+This project includes code adapted from the "databricks-dbt-factory" library.
+
+    Source:       https://github.com/mwojtyczka/databricks-dbt-factory
+    Full history: https://github.com/mwojtyczka/databricks-dbt-factory/commits/main
+    Adapted from: commit 29866edb82de28c33fe888be67d451068d4d7f94 (v0.3.5)
+    Location:     src/databricks_dbt_factory/
+
+The code under src/databricks_dbt_factory/ originates from that repository (trimmed to the
+serverless + notebook-task subset used here, then reformatted to this repository's style) and is
+provided under the MIT
+license reproduced below, which this NOTICE preserves for attribution. All other files are
+part of the bundle-examples repository and are covered by that repository's license.
+
+--------------------------------------------------------------------------------
+
+MIT License
+
+Copyright (c) 2024-present mwojtyczka <wojtyczka.marcin@gmail.com>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this
+software and associated documentation files (the "Software"), to deal in the Software
+without restriction, including without limitation the rights to use, copy, modify, merge,
+publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons
+to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or
+substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
+INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE
+FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.

contrib/dbt_factory/README.md

@@ -0,0 +1,247 @@
+# dbt_factory
+
+This example runs a [dbt](https://docs.getdbt.com/) project on Databricks as a
+**Databricks Workflow with one task per dbt object** (model, seed, snapshot, test) instead of
+running the whole project as a single opaque task.
+
+It does this by combining two pieces:
+
+* **[databricks-dbt-factory](https://github.com/mwojtyczka/databricks-dbt-factory)** — a small
+  library that reads a dbt `manifest.json` and expands it into Databricks job tasks, wiring up
+  the dependencies between them. Its source is included under `src/databricks_dbt_factory/`
+  (see [`NOTICE`](NOTICE) for attribution and license).
+* **[PyDABs](https://docs.databricks.com/dev-tools/bundles/python)** — the Declarative Automation
+  Bundles Python resources hook. At `databricks bundle deploy` time the Databricks CLI calls
+  `load_resources` in [`resources/__init__.py`](resources/__init__.py), which runs the factory
+  against the manifest and returns the generated job.
+
+The result: **no per-model job YAML is checked in**. The task graph is generated on the fly from
+the dbt manifest each time you deploy.
+
+## Why one task per dbt object?
+
+By default dbt's integration with Databricks Workflows treats the whole project as a single
+task — a black box. Expanding it into one task per object gives:
+
+* **Faster execution** — independent models run in parallel, and the notebook task type runs dbt
+  from a pre-built serverless base environment, avoiding a dependency install on every task.
+* **Visibility & simplified troubleshooting** — pinpoint and fix issues at the model level right
+  in the Databricks Workflows UI.
+* **Enhanced logging & notifications** — per-task logs and precise, model-level error alerts.
+* **Improved retriability** — retry only the failed model tasks without rerunning the whole project.
+* **Seamless testing** — dbt data tests run as their own tasks right after each model finishes,
+  for faster validation and feedback.
+
+This example uses **serverless compute** and the **notebook task type** (each task triggers dbt
+through a small runner notebook using the `dbtRunner` Python API) for the fastest task start
+times. See the [databricks-dbt-factory README](https://github.com/mwojtyczka/databricks-dbt-factory#benefits)
+for more.
+
+## How it works
+
+The [`dbt-factory` template](../templates/dbt-factory) scaffolds a self-contained project.
+From then on, each `databricks bundle deploy` regenerates the Workflow from your current dbt
+manifest — add or remove a model and the task graph follows on the next deploy, with no per-model
+YAML to maintain.
+
+```mermaid
+flowchart TD
+    subgraph setup["One-time setup"]
+      T["dbt-factory bundle template"] -->|databricks bundle init| B["Scaffolded project:<br/>dbt project + PyDABs hook + factory code"]
+      X["Existing dbt project<br/>(optional)"] -.->|move models/seeds/... into src/| B
+    end
+    subgraph deploy["Every deploy"]
+      C["make manifest<br/>(dbt parse)"] --> D["target/manifest.json"]
+      D --> E["databricks bundle deploy"]
+      E --> F["PyDABs load_resources reads the<br/>manifest and generates the job"]
+    end
+    subgraph runtime["At run time — serverless"]
+      G["Databricks Workflow:<br/>one task per model / seed / snapshot / test"] --> H["Each task triggers dbt<br/>via the runner notebook"]
+      H --> I[("SQL warehouse")]
+    end
+    B --> C
+    F --> G
+
+    classDef optional stroke:#999,stroke-dasharray:5 4,color:#888;
+    class X optional;
+```
+
+## Project structure
+
+```
+dbt_factory/
+├── databricks.yml              # Bundle definition; wires up the PyDABs `load_resources` hook
+├── dbt_project.yml             # dbt project (models under src/models, etc.)
+├── dbt_profiles/profiles.yml   # dbt profiles for the deployed job (dev / prod targets)
+├── profile_template.yml        # prompts for `dbt init` (local development)
+├── resources/__init__.py       # PyDABs glue: manifest -> generated job (the only integration code)
+├── src/
+│   ├── models/                 # your dbt models (example: orders_raw, orders_daily)
+│   └── databricks_dbt_factory/ # vendored factory library (trimmed; see NOTICE)
+├── target/manifest.json        # committed dbt manifest, read at deploy time (regenerate with `make manifest`)
+├── tests/                      # tests for the vendored factory + the PyDABs integration
+├── pyproject.toml              # dependencies (installed into .venv via `uv sync`)
+└── Makefile                    # convenience targets: setup, manifest, validate, deploy, run, test, test-e2e
+```
+
+## Setup
+
+1. Install the [Databricks CLI](https://docs.databricks.com/dev-tools/cli/databricks-cli.html)
+   and the [uv](https://docs.astral.sh/uv/) package manager.
+
+2. Authenticate to your Databricks workspace:
+   ```
+   $ databricks configure
+   ```
+
+3. Install dependencies into the `.venv` the bundle uses:
+   ```
+   $ make setup      # == uv sync --dev
+   ```
+
+4. Edit `dbt_profiles/profiles.yml` and set your SQL warehouse `http_path`, `catalog`, and
+   `schema`. Set the workspace host in `databricks.yml` (and the prod `root_path` / permissions).
+
+## The dbt manifest
+
+`resources/__init__.py` reads `target/manifest.json` at deploy time to build the task graph. A
+manifest is committed so the bundle deploys out of the box. **After you change your models,
+regenerate it:**
+
+```
+$ make manifest      # == uv run dbt deps && uv run dbt parse
+```
+
+`dbt parse` only reads your project files; it does not connect to a warehouse. The manifest
+location is configurable — point at a different file via the `DBT_MANIFEST_PATH` environment
+variable or by editing `MANIFEST_PATH` in `resources/__init__.py`.
+
+> **Faster task startup (automatic).** `make manifest` also writes `target/partial_parse.msgpack`
+> next to the manifest. The bundle syncs it (`sync.include` in `databricks.yml`) and each notebook
+> task injects it to **skip dbt's parse phase** — a large win on big projects, where parsing (not
+> the SQL) dominates each task's time. No `git add -f` needed: because the runtime dbt is pinned to
+> your local version (see "dbt version and the serverless environment" below), the version-specific
+> msgpack always loads instead of being silently ignored. Deploy with `make deploy` (or run
+> `make manifest` first) so the shipped msgpack always matches your current models.
+
+## Deploy and run
+
+```
+$ make deploy      # regenerates the manifest + parse cache, then deploys
+$ make run         # == databricks bundle run dbt_factory_job
+```
+
+`make deploy` regenerates `target/manifest.json` and `target/partial_parse.msgpack` (via `dbt
+parse`) before deploying, so the task graph and the synced parse cache always match your current
+models. You can also call the CLI directly — just run `make manifest` first:
+
+```
+$ databricks bundle deploy --target dev
+$ databricks bundle run dbt_factory_job
+```
+
+Open the run URL the CLI prints to watch the generated per-model task graph execute. Deploying
+in `dev` mode prefixes resources with `[dev your_name]` and pauses the daily schedule; deploy to
+`prod` with `--target prod`.
+
+## Configuring the generated job
+
+A few knobs are exposed as constants at the top of `resources/__init__.py`:
+
+* `BUNDLE_TESTS` — when `True`, single-model tests are bundled into one `dbt test` task per
+  resource (fewer task startups; faster for test-heavy projects). Default `False` (one task per
+  test node, for maximum per-test visibility).
+* `ENVIRONMENT_KEY` — the serverless environment key (default `Default`).
+* `EXTRA_DBT_COMMAND_OPTIONS` — extra options appended to every generated dbt command.
+
+The dbt target, warehouse, catalog, and schema are configured in `dbt_profiles/profiles.yml`
+and selected per bundle target via `--target ${bundle.target}`.
+
+### dbt version and the serverless environment
+
+You don't set the runtime dbt version by hand. At deploy time `resources/__init__.py` pins the
+serverless environment to the **exact `dbt-databricks` version installed in the bundle's `.venv`** —
+the same version you use locally to generate the manifest and develop with. `pyproject.toml` is the
+single source of truth: change the version there, re-run `make setup`, and the next deploy uses it.
+This guarantees the version running in Databricks matches the one you tested with.
+
+The version is shipped as a small `dbt_serverless_env.yaml` [base environment](https://docs.databricks.com/aws/en/compute/serverless/dependencies)
+that the bundle generates and syncs on every deploy (git-ignored), so Databricks pre-builds the
+environment once instead of installing dbt on every task.
+
+## Migrating an existing dbt project
+
+Bring your own dbt project by **generating a fresh project from the template and moving your dbt
+files into it.** You don't touch dependencies, the vendored factory, or any paths — the generated
+project already ships all of that.
+
+1. Generate a new project (or copy this `dbt_factory` example):
+
+   ```
+   $ databricks bundle init https://github.com/databricks/bundle-examples --template-dir contrib/templates/dbt-factory
+   ```
+
+2. Remove the starter models and copy your dbt sources into the matching `src/` subdirectories:
+
+   ```
+   $ rm -r src/models/example
+   # Copy whichever of these your project has (skip the ones you don't use):
+   $ cp -R /path/to/your/dbt/models/*     src/models/
+   $ cp -R /path/to/your/dbt/seeds/*      src/seeds/
+   $ cp -R /path/to/your/dbt/snapshots/*  src/snapshots/
+   $ cp -R /path/to/your/dbt/macros/*     src/macros/
+   $ cp -R /path/to/your/dbt/tests/*      src/tests/
+   ```
+
+   The generated `dbt_project.yml` already points `model-paths`, `seed-paths`, etc. at these
+   `src/` folders, so your files are picked up as-is. Merge any model/seed configuration from your
+   own `dbt_project.yml` into the generated one (keep the generated `name`/`profile`), and remove
+   the leftover `models: dbt_factory: example:` block that referenced the deleted starter models —
+   otherwise `dbt parse` warns that those config paths don't apply to any resource. If you use dbt
+   packages, copy your `packages.yml` to the project root too: `make manifest` installs them
+   (`dbt deps`) and the bundle syncs the resulting `dbt_packages/` to the workspace, so the job
+   never installs packages at runtime.
+
+3. Point `dbt_profiles/profiles.yml` at your warehouse (`http_path`, `catalog`, `schema`). Leave
+   the `host`/`token` lines as they are — the runner notebook sets those at runtime.
+
+4. Generate the manifest and deploy:
+
+   ```
+   $ make setup
+   $ make manifest      # dbt parse -> target/manifest.json
+   $ databricks bundle deploy --target dev
+   ```
+
+That's the whole migration: no dependency wrangling and no path edits, because your project keeps
+the generated layout (dbt project at the bundle root, factory under `src/`). If you'd rather keep
+your project's existing directory structure instead of `src/`, edit the `*-paths` in
+`dbt_project.yml` to point at your folders — nothing else changes.
+
+## Tests
+
+```
+$ make test      # == uv run pytest tests
+```
+
+This runs the factory's unit tests plus an offline test that exercises the PyDABs integration
+against the committed manifest; no workspace is required. One test compares the generated tasks
+with a saved snapshot (`tests/test_data/expected_tasks.json`), so unintended changes to the
+generated job fail the suite. After an intentional change to the generated output, refresh the
+snapshot with `make test-update-expected-tasks`.
+
+There is also a live end-to-end test that generates a project from the template, deploys it to
+your workspace, runs the generated job, verifies the output tables, and tears everything down
+again:
+
+```
+$ make test-e2e
+```
+
+See [`tests/e2e/README.md`](tests/e2e/README.md) for the required environment variables.
+
+## Local development with dbt
+
+You can still develop the dbt project locally with the dbt CLI. Initialize your own profile with
+`dbt init` (see `profile_template.yml`), then use `dbt run`, `dbt test`, etc. as usual. See the
+[`dbt_sql`](../../dbt_sql) example for a more detailed local-dbt walkthrough.

contrib/dbt_factory/databricks.yml

@@ -0,0 +1,53 @@
+# This is a Declarative Automation Bundles definition for dbt_factory.
+# See https://docs.databricks.com/dev-tools/bundles/index.html for documentation.
+#
+# The job for this bundle is NOT defined in YAML. Instead it is generated at deploy time
+# from the dbt manifest by resources/__init__.py (see the `python.resources` hook below),
+# producing one Databricks task per dbt object (model / seed / snapshot / test).
+bundle:
+  name: dbt_factory
+  uuid: 19ecc815-cff0-449c-91c1-e68239d49ccb
+
+# PyDABs: the Databricks CLI calls `load_resources` during `bundle deploy` to build resources
+# defined in Python. See resources/__init__.py.
+python:
+  venv_path: .venv
+  resources:
+    - "resources:load_resources"
+
+# These files are generated (by resources/__init__.py / `make manifest`) and git-ignored; include
+# them explicitly so `bundle deploy` still syncs them to the workspace:
+#   - dbt_serverless_env.yaml    the base_environment the generated job points at.
+#   - target/partial_parse.msgpack   dbt's parse cache; each task injects it to skip parsing (see
+#                                    README). Regenerated by `make manifest`; usable at runtime
+#                                    because the runtime dbt is pinned to your local version.
+#   - dbt_packages/**            dbt packages installed by `make manifest` (dbt deps), so tasks
+#                                never install packages at runtime.
+sync:
+  include:
+    - dbt_serverless_env.yaml
+    - target/partial_parse.msgpack
+    - dbt_packages/**
+
+# Deployment targets. The dbt target is selected via `--target ${bundle.target}`, so these
+# names match the dbt outputs in dbt_profiles/profiles.yml.
+targets:
+  dev:
+    # The default target uses 'mode: development' to create a development copy.
+    # - Deployed resources get prefixed with '[dev my_user_name]'
+    # - Any job schedules and triggers are paused by default.
+    # See also https://docs.databricks.com/dev-tools/bundles/deployment-modes.html.
+    mode: development
+    default: true
+    workspace:
+      host: https://company.databricks.com
+
+  prod:
+    mode: production
+    workspace:
+      host: https://company.databricks.com
+      # We explicitly deploy to /Workspace/Users/user@company.com to make sure we only have a single copy.
+      root_path: /Workspace/Users/user@company.com/.bundle/${bundle.name}/${bundle.target}
+    permissions:
+      - user_name: user@company.com
+        level: CAN_MANAGE