bug(Databricks): databricks-sql-connector executes queries sequentially regardless of concurrent_tasks

[davem-bis]

Summary

When using the Databricks SQL Connector for SQL models and setting concurrent_tasks > 1, SQLMesh will spawn as many threads as concurrent_tasks, but they can only execute commands sequentially due to the lock in the ThreadLocalSharedConnectionPool.

Root Cause

I believe this has been caused by setting shared_connection = True in #5746, whilst this has resolved the issue of OAuth race conditions, it has done so by forcing everything to be sequential.

Proposed Solutions

There is an interim solution and a longer term solution that I believe should be implemented:

Interim Solution

Set shared_connection to be based on whether an OAuth authentication method is being used. If an access token is being used, then a non-shared connection is suitable.

In core/config/connection.py:DatabricksConnectionConfig:

shared_connection: t.ClassVar[bool] = auth_type is not None

This issue will remain for OAuth authentication flows, but this is a quick fix for those using access tokens.

'Proper' Solution

For OAuth, change the authentication flow to use a single connection, and then use a non-shared pool after that point.

Environments Tested On

• Databricks Cluster
  DBR: 17.3 LTS (Python 3.12.3)
  SQLMesh Version: 0.253.3
• Windows 11
  Python: 3.12.3
  SQLMesh Version: 0.253.3

[davem-bis]

I've implemented the interim solution in PR #5859.