Skip to content

pipelines: use lakeflow-pipelines template #4237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
kanterov wants to merge 1 commit into
base: main
Choose a base branch
from
Open

pipelines: use lakeflow-pipelines template #4237

kanterov wants to merge 1 commit into from

Conversation

@kanterov
Copy link
Collaborator

@kanterov kanterov commented Jan 9, 2026

Changes

Change template in "pipelines init" from "cli-pipelines" to "lakeflow-pipelines".

Remove the code for "cli-pipelines" template and point "cli-pipelines" template into "lakeflow-pipelines" in case anything relies on undocumented hidden template.

There are a few minor problems with lakeflow-pipelines template that we will address in a follow-up. For example, we create job parameters, but never use them.

Why

lakeflow-pipelines template follows recommended layout for SDP projects in DABs and is consistent with how other templates are structured.

Tests

Updated acceptance tests

@kanterov kanterov temporarily deployed to test-trigger-is January 9, 2026 12:37 — with GitHub Actions Inactive
@kanterov kanterov force-pushed the pipelines-cli-template branch from 1f111da to 6c12108 Compare January 9, 2026 12:38
@kanterov kanterov temporarily deployed to test-trigger-is January 9, 2026 12:39 — with GitHub Actions Inactive
@eng-dev-ecosystem-bot
Copy link
Collaborator

eng-dev-ecosystem-bot commented Jan 9, 2026 •
edited
Loading

Commit: 6c12108

Run: 20852196885

Env 🟨​KNOWN 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
🟨​ aws linux 9 14 2 381 678 26:37
🟨​ aws windows 17 6 2 383 676 23:32
🟨​ aws-ucws linux 5 1 17 2 532 555 39:06
🟨​ aws-ucws windows 13 10 2 534 553 36:22
🟨​ azure linux 2 16 3 381 677 26:48
🟨​ azure windows 11 7 3 383 675 25:54
🟨​ azure-ucws linux 2 16 3 528 554 37:24
🟨​ azure-ucws windows 11 7 3 530 552 42:12
🟨​ gcp linux 2 16 3 370 683 26:31
🟨​ gcp windows 11 7 3 372 681 28:31
25 interesting tests: 17 KNOWN, 7 RECOVERED, 1 SKIP
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
🟨​ TestAccept 🟨​K 🟨​K 🟨​K 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
💚​ TestAccept/bundle/deployment/bind/alert 🙈​S 🙈​S 🙈​S 🙈​S 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🟨​ TestAccept/bundle/generate/alert 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
🟨​ TestAccept/bundle/generate/alert/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
🟨​ TestAccept/bundle/generate/alert/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
💚​ TestAccept/bundle/resources/alerts/basic 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/basic/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/basic/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/with_file 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/with_file/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/with_file/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 🟨​K 🟨​K 🟨​K 🟨​K 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 🔄​f 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 🟨​K 🟨​K 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestExport 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestExportWithFileFlag 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestImportDir 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
🟨​ TestImportDirDoesNotOverwrite 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
🟨​ TestImportDirWithOverwriteFlag 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
🟨​ TestImportFileFormatAuto 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
🟨​ TestImportFileFormatSource 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
Top 32 slowest tests (at least 2 minutes):
duration env testname
9:04 aws-ucws linux TestAccept/bundle/resources/synced_database_tables/basic
7:43 azure-ucws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:49 aws-ucws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:44 aws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:36 azure-ucws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:35 aws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:33 gcp linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:32 aws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:28 gcp windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:05 gcp linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
4:36 gcp windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
4:30 aws-ucws windows TestAccept/bundle/resources/synced_database_tables/basic
4:30 azure-ucws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
3:41 aws-ucws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
3:18 azure-ucws windows TestAccept/bundle/resources/synced_database_tables/basic
3:03 azure-ucws linux TestAccept/bundle/resources/synced_database_tables/basic
2:36 aws-ucws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
2:26 aws-ucws linux TestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=yes/PY=yes/READPLAN=
2:21 aws-ucws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
2:19 aws-ucws linux TestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=direct/DLT=yes/NBOOK=no/PY=no/READPLAN=1
2:17 aws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
2:13 aws windows TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=yes/PY=yes/READPLAN=
2:12 azure linux TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.12
2:12 azure windows TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=yes/PY=yes/READPLAN=
2:12 aws linux TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=yes/NBOOK=no/PY=yes/READPLAN=1
2:11 azure linux TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=yes/PY=yes/READPLAN=
2:08 aws windows TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=yes/READPLAN=
2:08 azure windows TestAccept/bundle/resources/models/basic/DATABRICKS_BUNDLE_ENGINE=terraform
2:06 gcp windows TestAccept/bundle/resources/models/basic/DATABRICKS_BUNDLE_ENGINE=terraform
2:06 aws-ucws windows TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.11
2:05 aws windows TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=yes/PY=yes/READPLAN=
2:03 aws-ucws linux TestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=no/PY=yes/READPLAN=

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jefferycheng1 jefferycheng1 Awaiting requested review from jefferycheng1 jefferycheng1 is a code owner

@lennartkats-db lennartkats-db Awaiting requested review from lennartkats-db lennartkats-db is a code owner

@andrewnester andrewnester Awaiting requested review from andrewnester andrewnester is a code owner

@anton-107 anton-107 Awaiting requested review from anton-107 anton-107 is a code owner

@denik denik Awaiting requested review from denik denik is a code owner

@pietern pietern Awaiting requested review from pietern pietern is a code owner

@shreyas-goenka shreyas-goenka Awaiting requested review from shreyas-goenka shreyas-goenka is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kanterov @eng-dev-ecosystem-bot