fix(kernel): Thrift-parity fixes + GetArrowBatches + decimal-as-float opt-in + U2M scopes/telemetry + SPOG warehouse-id routing by mani-mathur-arch · Pull Request #416 · databricks/databricks-sql-go

mani-mathur-arch · 2026-07-20T18:37:45Z

What

SEA-via-kernel↔Thrift parity fixes surfaced by the Go comparator (two database/sql connections against the same warehouse, differing only in useKernel), plus the public Arrow batch API, U2M scope/browser fixes, an auth-logging cleanup, and SPOG routing. All kernel-path-only and entirely Go-side (no kernel/C-ABI change).

Parity fixes

TIMESTAMP out-of-nanosecond-range wrapping. Arrow's ToTime forms an int64-ns intermediate that overflows for microsecond TIMESTAMPs outside ~1678–2262 ('0001-01-01'→1754-08-30). Now uses the unit-specific time.Unix* constructors, covering 0001–9999. Thrift unaffected (gets a preformatted string).
DDL affected-rows -1 vs 0. The C ABI returns -1 for unknown/NA counts (DDL/SELECT); Thrift reports 0. Fold the -1 sentinel to 0; real DML counts pass through.
User-Agent not forwarded. The kernel's built-in UA leaked into query history. Now forwards the driver UA via set_custom_header, matching Thrift; WithUserAgentEntry still customizes.
VOID/NULL columns typed NULL vs STRING. Server stringifies VOID over the wire (even SELECT NULL reports STRING_TYPE), so arrow.NULL now maps to STRING, matching Thrift. Corrects two tests that pinned NULL_TYPE from the enum name, not live.

Arrow batch API

GetArrowBatches not exposed on kernel. kernelRows now implements it over the zero-copy next_batch pull (prefetch for exact HasNext; transfer record ownership to the caller). GetArrowIPCStreams returns ErrNotSupportedByKernel (kernel exports via C Data Interface, not IPC). Compile-time rows.Rows assertion guards the gap.

Decimal-as-float opt-in (WithKernelDecimalAsFloat)

New kernel-only connection option that scans top-level DECIMAL columns to a lossy float64 (arrow-go Num.ToFloat64) instead of the exact fixed-point string. The kernel still receives native Arrow Decimal128 over the wire — only the Go scanner changes, skipping the per-cell string materialization for a cheap scalar. Off by default (behavior byte-identical to today when unset); mirrors the Thrift driver's pre-UseArrowNativeDecimal default, and is rejected on the Thrift path (use WithArrowNativeDecimal there). Nested decimals (inside list/struct/map) still render exactly.
Why: the kernel's fetch/decode transport is already the faster of the two backends; its only large-result deficit was the per-cell decimal string render. Skipping it surfaces the transport win — on decimal-heavy large results (tpcds_sf100.catalog_sales, 15×DECIMAL(7,2), 1.3M–31M rows) the option is ~25% faster than Thrift (0.75–0.76×) and up to ~37% faster than the exact-string kernel path (0.63×), consistently across sizes and reps. Lossy beyond ~15–17 digits, so it's an opt-in speed knob, not a default.

U2M

Scope parity. Kernel U2M forwarded no scopes → kernel applied all-apis, rejected where the built-in client lacks it. resolveKernelAuth now populates Auth.Scopes from the same oauth.GetScopes Thrift uses.
Double-browser fix. The telemetry/feature-flag client wrapped the interactive authenticator, launching a second browser at connect (blocking up to ~120s). Telemetry init is now skipped on the kernel+U2M path (detected via the U2MClientID structural interface); best-effort telemetry is dropped rather than made to prompt, matching Python's default. Unauthenticated telemetry is the follow-up — PECOBLR-3839.

Auth logging

M2M/U2M/OAuth-config/token-provider paths logged through global zerolog/log (default Debug), ignoring WithLogLevel on both backends. Routed through the driver logger so they honor the configured level; auth logic untouched.

SPOG routing (WithWarehouseID path)

On a unified/SPOG host the kernel needs x-databricks-org-id, which it injects only via set_http_path → from_http_path (parses the warehouse id + ?o=). But Go calls set_warehouse (→ ConnectionConfig::new, no org id) whenever a warehouse id is set, and the kernel refuses a caller-supplied org-id header — so a WithWarehouseID SPOG session 303'd to /login. Now, when a warehouse id is set and the HTTP path is a canonical warehouses/endpoints path with a non-empty ?o=, route via set_http_path (kernel parses the same id plus the org id). Guarded by httpPathCarriesOrgRouting, so non-SPOG connections keep the existing set_warehouse routing untouched. Go-only — Python/Node/ODBC always use the http_path route.

Testing

Unit: TestScanCellTimestampOutOfNanoRange, TestNormalizeAffectedRows, TestColumnTypeInfoFor/…MatchesThriftMapping/…ScanTypeCoversScanner (VOID→STRING), TestScanCellDecimalAsFloat (float64 arm vs unchanged exact-string default), U2M scope assertions in TestValidateKernelConfig/TestResolveKernelAuthRealAuthenticators, TestInteractiveU2MAuthenticatorDetection, TestHTTPPathCarriesOrgRouting. ./auth/... green after the logger swap (no import cycle).
Live (staging kernel backend): GetArrowBatches — 100,000 rows / 32 zero-copy batches, schema + count pass; GetArrowIPCStreams returns the sentinel. U2M double-browser prompt confirmed gone. SPOG live on peco.azuredatabricks.net — the WithWarehouseID routing that previously 303'd now returns SELECT 1 == 1 / current_user() with the org-id header injected; non-SPOG queries unchanged. WithKernelDecimalAsFloat verified live: default returns DECIMAL/string, option-on returns STRING/float64 for the same value, both 99.98.
Both build paths green: pure-Go (CGO_ENABLED=0) and tagged cgo (-tags databricks_kernel).
Comparator (Thrift vs SEA): 9 diffs (from 13 pre-fix, vs main's 41). The 5 dt_interval_column metadata diffs are fixed here (INTERVAL_DAY_TIME→STRING). Of the 9 remaining, 8 are not driver bugs (NaN==NaN artifact ×3, non-deterministic EXPLAIN node ids, by-design volume-staging rejection, raw-fetch_arrow shape ×2, a permission-gated CREATE PROCEDURE sqlState detail); 1 is a deterministic DESCRIBE TABLE extra-trailing-rows diff that's server output (likely a comparator-side skip_rows filter, not this repo). Native decimal/interval on the raw-fetch_arrow path is a tradeoff, tracked as PECOBLR-3838 (needs a kernel KERNEL_REV bump).

Stacked on mani/sea-kernel-new-items (#412).

Co-authored-by: Isaac

The kernel Rows path converted arrow timestamps with arrow's ToTime, which forms an int64-nanosecond intermediate (value * unit-multiplier). For a microsecond TIMESTAMP — Databricks' wire unit — that overflows int64 outside ~1678-2262 and silently wraps a valid instant to a wrong one (TIMESTAMP '0001-01-01' scanned back as 1754-08-30, '9999-12-31' as 1816-03-30). Convert via the unit-specific time.Unix/UnixMilli/UnixMicro constructors, which split into (seconds, nanoseconds) internally and represent the full 0001-9999 range Databricks TIMESTAMP allows. The default Thrift path is unaffected — it receives TIMESTAMP as a preformatted server string and never runs this multiply — so this closes a kernel-only divergence from Thrift. TestScanCellTimestampOutOfNanoRange pins both repro endpoints plus the int64-ns boundary years; the existing unit and location tests still pass. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>

The C ABI returns -1 for "not applicable / unknown" affected-row counts — DDL, SELECT, or a warehouse that doesn't surface the counter (the kernel's num_modified_rows Option<i64> None). The Thrift path reports 0 in those cases, so Result.RowsAffected() diverged: a CREATE/DROP SCHEMA returned -1 on the kernel backend and 0 on Thrift. Fold the -1 sentinel to 0 so RowsAffected() is identical across backends; a real DML count (>= 0) passes through unchanged. The rule lives in a pure, untagged normalizeAffectedRows so it is unit-testable in the default CGO_ENABLED=0 build alongside the other kernel decision helpers. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>

…iver The kernel backend never forwarded the driver's User-Agent, so the kernel's built-in UA leaked through and query history misattributed SEA-path queries. Forward the same composed UA the Thrift path sends via set_custom_header, so both backends are attributed alike. Default-on; WithUserAgentEntry still customizes it. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>

The server stringifies VOID over the Thrift wire (verified live: even bare SELECT NULL reports STRING_TYPE), same as intervals, but hands the kernel a native arrow.NULL schema. Map arrow.NULL to STRING so both backends report identical column metadata; correct the two parity tests that wrongly pinned NULL_TYPE (derived from the enum name, never captured live). Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>

Address review of the parity fixes: assert kc.UserAgent == BuildUserAgent (non-empty) in TestBuildKernelConfig so deleting the forwarding or an empty UA fails a pure-Go run; add a CAST(NULL AS VOID) column to the live column-type parity query so the arrow.NULL→STRING mapping is guarded against the real server, not only the pure-Go tests. Trim the duplicated -1→0 rationale at the affected-rows call site to a one-liner. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>

msrathore-db · 2026-07-23T21:48:49Z

+// None). The Thrift path reports 0 in exactly those cases (TGetOperationStatusResp
+// defaults NumModifiedRows to 0), so the -1 sentinel is folded to 0 for parity.
+// A real DML count (>= 0) passes through unchanged.
+func normalizeAffectedRows(n int64) int64 {


make this change in kernel not here

changing this in the kernel will break the other drivers. other drivers have different specs for each "not applicable" row counts (python has -1, node has none). Go thrift has 0 for the database/sql spec.

msrathore-db · 2026-07-23T21:49:08Z

 	}

+	// User-Agent so query history attributes the kernel path to this driver.
+	if k.cfg.UserAgent != "" {


please verify from query history

verified, the behavior is sound and matches how the python driver passes its user agent and what the query history shows

msrathore-db · 2026-07-23T21:50:02Z

-		"INTERVAL '1' MONTH a_iv_ym, st_point(1,2) a_geom"
+		// VOID: the server declares it STRING_TYPE over Thrift; asserts the kernel
+		// arrow.NULL→STRING mapping matches live, not just in the pure-Go tests.
+		"INTERVAL '1' MONTH a_iv_ym, st_point(1,2) a_geom, CAST(NULL AS VOID) a_void"


can you please confirm this NULL to string in other drivers too?

The other drivers (node and python) also have this gap, the kernel returns nulls as void but thrift returns as string. i think this is an intentional design decision. verified by the python comparator, node one is currently broken due to dependency differences. I can remove it if that should be taken up as a separate fix for all the drivers or at the kernel level

msrathore-db

Please confirm the similar issues in other drivers over kernel like node and python so we can decide the location of these fixes

msrathore-db

Verify other drivers

kernelRows implemented only driver.Rows + the ColumnType* interfaces, so a Conn.Raw + rows.(dbsqlrows.Rows) assertion failed on the kernel backend (it succeeds on Thrift). Implement GetArrowBatches by wrapping the existing zero-copy next_batch pull, and reject GetArrowIPCStreams (the kernel C ABI exports Arrow C Data, not IPC bytes). A compile-time dbrows.Rows assertion guards the gap. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>

m2m.Authenticate logged via the global zerolog logger, whose default level is Debug, so "OAuth token fetched successfully" printed regardless of the driver's configured log level (default Warn) on both backends. Route it through the driver's logger package so it honors WithLogLevel like every other line. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>

The U2M, OAuth-config, and token-provider paths still logged via the global zerolog logger (default level Debug), so their lines printed as raw JSON regardless of the driver's configured level — the same leak the m2m fix addressed. Route all of them through the driver's logger package so every auth log honors WithLogLevel and uses the driver's format; auth logic is untouched. This closes the last global-zerolog call sites in the driver. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>

The kernel U2M path forwarded only the client id, so the kernel applied its own default scope set (all-apis + offline_access). A workspace whose built-in databricks-sql-connector client isn't granted all-apis rejects that with access_denied. resolveKernelAuth now populates Auth.Scopes from oauth.GetScopes — the same function the Thrift path uses (offline_access + sql on AWS/GCP, offline_access + <tenant>/user_impersonation on Azure) — so both backends authorize identically. The setAuth -> set_auth_u2m wiring already forwarded Scopes; only the population was missing. No kernel/C-ABI change. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>

DAY-TIME intervals can arrive from the kernel as arrow.INTERVAL_DAY_TIME (*array.DayTimeInterval), not just arrow.DURATION. ColumnTypeInfoFor lacked the case, so dt_interval columns reported ""/*interface{} vs Thrift's STRING/string (comparator diff across 5 configs); the value scanner also lacked it, so a real DT-interval row would have errored. Add both — metadata → STRING and the scanner folding {Days,Milliseconds} to ms through the same renderer — mirroring the existing DURATION/INTERVAL_MONTHS handling. Kernel-path only; no other driver and no kernel/C-ABI change. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>

Detach kernelRows.ctx from the caller's QueryContext (context2.WithoutCancel + Close-driven cancel), matching the Thrift path: a deadline that only gated statement submission no longer truncates a large CloudFetch stream. Seed iterationErr on batch-path fetch errors so a failed stream is recorded as a failed statement in telemetry, not a success. Add kernel-tagged coverage for the previously-untested batch API: white-box unit tests (IPC rejection, schema, drained-stream, sticky fetch-error → iterationErr) and a live GetArrowBatches draining e2e with an exact row count. Add a literal-assertion TestGetScopes (AWS/GCP/Azure) so a scope regression — the cause of the kernel U2M access_denied incident — is caught rather than asserted tautologically. Fix stale comments: the U2M doc now reflects that scopes are forwarded, m2m references SetLogLevel (not the nonexistent WithLogLevel), and the m2m success log moved after the error check. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>

… flow On the kernel U2M path the kernel owns the interactive browser login, but the driver also wrapped the same authenticator into its telemetry/feature-flag HTTP client, which triggered a second, redundant browser prompt (and could block connect on its ~120s callback wait). Detect the interactive U2M authenticator via a structural interface (U2MClientID, unique to it) and skip telemetry init on the kernel+U2M path. Telemetry is best-effort, so it's dropped rather than made to prompt, mirroring the Python driver's default. Unauthenticated telemetry (Python/Node parity) is tracked in PECOBLR-3839. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>

On a unified/SPOG host the workspace id rides in the HTTP path's ?o=<org>, and the kernel injects x-databricks-org-id only when the path goes through set_http_path (kernel-side ConnectionConfig::from_http_path parses both the warehouse id and ?o=). set_warehouse takes only host+id and drops the org id, so a WithWarehouseID-addressed SPOG session 303s to /login. The kernel also refuses a caller-supplied x-databricks-org-id custom header, so the org id can reach the kernel only via the path. When a warehouse id is set but the HTTP path is a canonical warehouses/endpoints path carrying a non-empty ?o=, route by the path instead: the kernel still parses the same warehouse id out of it, plus the org id it needs. Guarded by httpPathCarriesOrgRouting so non-SPOG connections (no ?o=) keep the existing set_warehouse routing untouched. Go-only: Python and Node always use the http_path route, and the ODBC driver calls set_http_path with the raw path, so all three already parse ?o=. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>

Adds a kernel-only connection option that scans top-level DECIMAL columns to a lossy float64 instead of the exact fixed-point string. The kernel still receives native Arrow Decimal128; only the Go scanner changes, skipping the per-cell string materialization for a cheap scalar. Off by default; mirrors the Thrift driver's pre-UseArrowNativeDecimal behavior. Nested decimals still render exactly. On decimal-heavy large results this is ~25% faster than the Thrift path and up to ~37% faster than the exact-string kernel path. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>

TestKernelExperimentalFieldsClassified reflects over every KernelExperimentalConfig field and requires an explicit disposition entry so no experimental knob is silently dropped. Register DecimalAsFloat (wired Go-side via kernel.Config -> kernelOp -> arrowscan). Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>

github-actions · 2026-07-27T14:49:33Z

Go integration tests triggered (sea / passthrough). View workflow runs.

Base automatically changed from mani/sea-kernel-new-items to main July 21, 2026 04:09

mani-mathur-arch added 2 commits July 21, 2026 08:57

mani-mathur-arch force-pushed the mani/sea-kernel-comparator-fixes branch from 838bb0c to 21ad6fa Compare July 21, 2026 09:16

mani-mathur-arch added 2 commits July 22, 2026 04:07

mani-mathur-arch force-pushed the mani/sea-kernel-comparator-fixes branch from db597fd to 8815db9 Compare July 22, 2026 18:51

mani-mathur-arch changed the title ~~fix(kernel): TIMESTAMP range + DDL affected-rows parity with Thrift~~ fix(kernel): Thrift parity fixes (TIMESTAMP range, DDL rows, User-Agent, VOID type) Jul 22, 2026

msrathore-db reviewed Jul 23, 2026

View reviewed changes

msrathore-db approved these changes Jul 23, 2026

View reviewed changes

msrathore-db self-requested a review July 23, 2026 21:50

msrathore-db reviewed Jul 23, 2026

View reviewed changes

mani-mathur-arch added 3 commits July 24, 2026 18:40

Merge origin/main into kernel-arrow-batches

dec28fe

mani-mathur-arch changed the title ~~fix(kernel): Thrift parity fixes (TIMESTAMP range, DDL rows, User-Agent, VOID type)~~ fix(kernel): Thrift parity fixes + GetArrowBatches (TIMESTAMP, DDL, User-Agent, VOID, Arrow batches, M2M logging) Jul 24, 2026

mani-mathur-arch added 2 commits July 24, 2026 19:52

mani-mathur-arch added 3 commits July 24, 2026 21:03

mani-mathur-arch changed the title ~~fix(kernel): Thrift-parity fixes + GetArrowBatches + U2M scopes/telemetry + SPOG warehouse-id routing~~ fix(kernel): Thrift-parity fixes + GetArrowBatches + decimal-as-float opt-in + U2M scopes/telemetry + SPOG warehouse-id routing Jul 27, 2026

mani-mathur-arch added the integration-test-kernel Preview the full passthrough kernel integration suite (live warehouse) label Jul 27, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(kernel): Thrift-parity fixes + GetArrowBatches + decimal-as-float opt-in + U2M scopes/telemetry + SPOG warehouse-id routing#416

fix(kernel): Thrift-parity fixes + GetArrowBatches + decimal-as-float opt-in + U2M scopes/telemetry + SPOG warehouse-id routing#416
mani-mathur-arch wants to merge 16 commits into
mainfrom
mani/sea-kernel-comparator-fixes

mani-mathur-arch commented Jul 20, 2026 •

edited

Loading

Uh oh!

msrathore-db Jul 23, 2026

Uh oh!

mani-mathur-arch Jul 24, 2026 •

edited

Loading

Uh oh!

msrathore-db Jul 23, 2026

Uh oh!

mani-mathur-arch Jul 24, 2026

Uh oh!

msrathore-db Jul 23, 2026

Uh oh!

mani-mathur-arch Jul 24, 2026 •

edited

Loading

Uh oh!

msrathore-db left a comment

Uh oh!

msrathore-db left a comment

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mani-mathur-arch commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Testing

Uh oh!

msrathore-db Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

mani-mathur-arch Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

msrathore-db Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

mani-mathur-arch Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

msrathore-db Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

mani-mathur-arch Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

msrathore-db left a comment

Choose a reason for hiding this comment

Uh oh!

msrathore-db left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mani-mathur-arch commented Jul 20, 2026 •

edited

Loading

mani-mathur-arch Jul 24, 2026 •

edited

Loading

mani-mathur-arch Jul 24, 2026 •

edited

Loading