Skip to content

feat(bq_driver): add AllowedProjects DSN option to scope SQLTables metadata enumeration - #1617

Open
Rigbyfab4 wants to merge 3 commits into
googleapis:mainfrom
Rigbyfab4:feature/allowed-projects
Open

feat(bq_driver): add AllowedProjects DSN option to scope SQLTables metadata enumeration#1617
Rigbyfab4 wants to merge 3 commits into
googleapis:mainfrom
Rigbyfab4:feature/allowed-projects

Conversation

@Rigbyfab4

Copy link
Copy Markdown

Add AllowedProjects DSN option to scope SQLTables metadata enumeration

Problem

Power Query for Excel opens a connection and immediately calls SQLTables with
wildcard catalog/schema arguments to populate its Navigator tree. Today
GetFilteredProjectIds answers that by enumerating every project the
authenticated principal can reach via the projects.list REST API, then calling
datasets.list for each one.

In a large organization a principal can typically see far more projects than it
holds BigQuery permissions on. datasets.list on such a project fails with a
permission error, and because GetResultSetForDatasets surfaces the first error
from its parallel fan-out, one inaccessible project fails the entire
SQLTables call
. The Navigator tree cannot be built at all, even though the
user only wanted the handful of projects they do have access to.

The enumeration is also slow and quota-heavy at that scale, but the permission
failure is what blocks the user outright.

Neither existing mechanism helps:

  • AdditionalProjects only adds projects on top of the full enumeration.
  • ListProjectsParent restricts via the Cloud Resource Manager hierarchy, which
    requires cloudresourcemanager.googleapis.com access and has no UI.

Neither is an explicit allowlist, and neither prevents the broad call.

Change

Add an AllowedProjects DSN option naming the projects to expose. When set,
GetFilteredProjectIds filters that list and skips projects.list entirely,
so no inaccessible project is ever contacted.

Existing filter semantics are preserved: entries are matched against the catalog
argument using the same ODBC LIKE-pattern / SQL_ATTR_METADATA_ID rules used
elsewhere in the file, with surrounding whitespace trimmed and blank entries
ignored.

AllowedProjects and AdditionalProjects remain orthogonal and composable: if
both are set, enumeration starts from the allowlist and still appends the
AdditionalProjects ids.

The DSN key is fully usable headlessly for automation. For non-technical users
there is also a pick list in the Advanced Options dialog: a Load Projects
button populates a checkbox list via the existing
DriverForm::GetCatalogAndDataset path, preserving ticks across reloads and
retaining saved ids the account no longer reports.

SQLColumns is unaffected — its catalog argument is always a single resolved
project, so it never triggers account-wide enumeration.

Commits

  1. feat(bq_driver): add AllowedProjects DSN option to scope SQLTables
  2. fix(bq_driver): keep the Advanced Options dialog within the screen
  3. fix(bq_driver): stop empty dialog areas acting as a documentation link

Commits 2 and 3 fix pre-existing defects, not regressions from this feature.
They are separated so they can be reviewed or dropped independently. See
"Pre-existing issues" below for why they surfaced now.

Testing

Unit tests (all passing):

  • FilterAllowedProjects — 8 cases: wildcard, LIKE patterns, _
    single-character wildcard, non-matching pattern, blank/whitespace entries,
    empty allowlist, SQL_ATTR_METADATA_ID exact match, and % treated as a
    literal under SQL_ATTR_METADATA_ID.
  • ConnectionHandleALLOWEDPROJECTS parsing, set and unset, including that
    it stays independent of AdditionalProjects.
  • AdvanceOptionsSetValues / ResetToDefaults round-trip, and a pick-list
    test that ticks two saved ids, unticks one, clicks OK and asserts only the
    remaining id is saved.

Integration test CatalogTest.SQLTables_AllowedProjects: enumerates every
reachable project, then re-connects with AllowedProjects set to a subset and
asserts the returned catalog set is exactly that subset — so a project the
principal can reach but did not allow-list must be absent. This is what proves
the enumeration was skipped rather than merely filtered.

Manual verification on Windows against a real BigQuery account:

  • Advanced Options → Load Projects populated the checkbox list from the
    account; ticking a subset and pressing OK persisted AllowedProjects to the
    DSN and re-opening the dialog restored the ticks.
  • Excel Power Query Navigator built its tree showing only the allow-listed
    project. A second project the principal could reach, but had not allow-listed,
    was correctly absent, and a table loaded successfully through Power Query.

Note: the Win32 unit tests are not built by CI (see below), so the dialog tests
above were run locally with a temporary build workaround.

Pre-existing issues encountered

All of the following sit in Win32 dialog code that no CI path compiles:
ci/gha/builds/windows-cmake-integration.sh sets -DODBC_UNIT_TESTING=OFF
(required, because fuzztest fails to configure under MSVC with
Compiler MSVC is not supported), while the Linux builds that do compile unit
tests exclude these files via the if (WIN32) guard in bq_driver.cmake. That
single gap is likely why the following accumulated unnoticed, and may be worth
addressing on its own.

Fixed here (commits 2 and 3):

  1. Advanced Options opened taller than the screen. The dialog used a fixed
    720px height centred on raw screen metrics. On a 1280x720 display at 150%
    scaling only 672px of work area is available, leaving the OK and Cancel
    buttons unreachable below the screen edge. Now clamped to the work area with
    a vertical scrollbar.
  2. Blank dialog areas acted as a hyperlink. Both dialogs hit-test
    WM_LBUTTONDOWN against the documentation hyperlink's rectangle, but the
    CreateHyperlinkLabel calls are commented out. GetDlgItem returns NULL,
    GetClientRect(NULL, &rect) fails leaving rect uninitialised, and
    PtInRect then matches arbitrary clicks — launching a browser from empty
    parts of the dialog.

Not fixed, reported for triage:

  1. driver_form_test.cc does not compile. It calls
    DriverForm::TestODBCConnection(section) with one argument at lines 172, 181
    and 191; the signature changed in commit 5a10f6b9. Error C2660.

  2. ConvertLogLevel.ValidateLogLevelConversion fails. The test expects ""
    for an invalid log level; the implementation returns "0" (kLogOff).

  3. Two dialog tests hang indefinitely. AdvanceOptionsTest.ShowWindow and a
    case in ProxyOptionsTest call a form's Show(), whose modal GetMessage
    loop never receives WM_QUIT. (The new pick-list test avoids this by posting
    WM_QUIT before calling Show().)

  4. User DSNs cannot be used to connect. GetSectionWin
    (internal/utils.cc) hardcodes RegOpenKeyEx(HKEY_LOCAL_MACHINE, ...), so a
    DSN under HKEY_CURRENT_USER is invisible when connecting — a DSN created via
    the User DSN tab of the ODBC administrator cannot be used at all.

    This looks like an oversight rather than a design decision, since the driver
    is otherwise scope-aware: the write path uses SQLWriteDSNToIni /
    SQLWritePrivateProfileString, and the config-dialog read path uses
    SQLGetPrivateProfileString. Only the connect path hardcodes the hive, so the
    driver can create and re-edit a User DSN that it then cannot connect with.

    The failure is also misleading rather than merely inconvenient: with no values
    found, stoi("") throws in CreateAuth and its catch block silently defaults
    to kServiceAccount, so the user is told
    "The path to the file can't be empty" — an error about credentials — when the
    real problem is that the DSN was never read.

    Suggested fix: have the DSN read path consult HKEY_CURRENT_USER before
    HKEY_LOCAL_MACHINE, matching the precedence ODBC itself uses, or switch it
    to SQLGetPrivateProfileString as the config dialog already does. Worth
    noting for whoever picks this up: GetSectionWin is also used to read the
    trace/logging configuration, so changing it in place affects more than DSN
    lookups — adding a DSN-specific reader may be the safer shape. Deliberately
    left out of this PR as it is unrelated to this feature and deserves its own
    review.

Checklist

  • Contributor License Agreement signed
  • Code formatted with the project's clang-format and the top-level
    .clang-format
  • ci/cloudbuild/build.sh -t checkers-pr --docker — not run locally (no
    Docker available on the development machine); formatting was applied with
    the pinned clang-format directly
  • Unit tests run locally
  • Integration tests requiring a multi-project GCP account — the new
    SQLTables_AllowedProjects test compiles but was not run in CI conditions

🤖 Generated with Claude Code

Joshua McHenry added 3 commits July 27, 2026 19:42
Power Query's Navigator opens a connection and immediately calls SQLTables
with wildcard catalog/schema arguments. GetFilteredProjectIds answered that
by enumerating every project the authenticated principal can reach via the
projects.list REST API, then listing datasets for each one.

In a large enterprise organization the principal can typically *see* far
more projects than it holds BigQuery permissions on. Listing datasets for
such a project fails with a permission error, and because
GetResultSetForDatasets surfaces the first error from its parallel fan-out,
one inaccessible project fails the whole SQLTables call. The result is that
the Navigator tree cannot be built at all, even though the user only wanted
the handful of projects they do have access to. Enumeration is also slow and
quota-heavy at that scale, but the permission failure is the blocking issue.

Add an AllowedProjects DSN option naming the projects to expose. When set,
GetFilteredProjectIds filters that list and skips projects.list entirely, so
no inaccessible project is ever contacted. The existing filter semantics are
preserved: entries are matched against the catalog argument with the same
LIKE pattern / SQL_ATTR_METADATA_ID rules used elsewhere.

AllowedProjects and AdditionalProjects stay orthogonal and composable: if
both are set, enumeration starts from the allowlist and still appends the
AdditionalProjects ids on top.

Also add a pick list to the Advanced Options dialog so non-technical users
need not hand-type project ids. "Load Projects" reuses the existing
DriverForm::GetCatalogAndDataset path to populate a checkbox list view,
preserving ticks across reloads and keeping saved ids the account no longer
reports. The DSN key remains usable headlessly for automation.

Tests: unit coverage for FilterAllowedProjects and ALLOWEDPROJECTS parsing,
a dialog round-trip test, and an integration test asserting that a project
the principal can reach but did not allow-list is absent from SQLTables.
The Advanced Options dialog asked for a fixed 720x525 window and centred it
on the raw screen metrics. On a display whose usable height is smaller than
that -- a 1280x720 panel at 150% scaling leaves 672px of work area -- the
lower part of the dialog, including the OK and Cancel buttons, fell below
the screen edge with no way to reach it. Adding the allowed-projects pick
list made this worse by growing the layout, but the dialog already did not
fit such a display before that.

Clamp the window height to the desktop work area and give the window a
vertical scrollbar, so a layout taller than the available space stays fully
reachable. Scrolling moves the child controls via ScrollWindow, which keeps
the existing absolute control positions untouched. The scroll range is
derived from the laid-out content height and refreshed on WM_SIZE; the
scrollbar, mouse wheel, page keys and thumb dragging are all handled.

Widen the frame by the scrollbar width. The scrollbar is carved out of the
client area, so without this every right-aligned control loses those pixels
and the edit boxes are clipped.

Accumulate mouse wheel movement rather than dividing each message by
WHEEL_DELTA. High-resolution wheels and precision trackpads report fractions
of a notch, which integer division rounded away to nothing, so those devices
could not scroll the dialog at all. Also honour the system
SPI_GETWHEELSCROLLLINES setting instead of assuming three lines.

Also centre the window inside the work area rather than the full screen, so
it is not positioned partly behind the taskbar.
Clicking a blank part of the DSN setup dialog or the Advanced Options dialog
could open a browser at the BigQuery ODBC/JDBC drivers page.

Both dialogs hit-test WM_LBUTTONDOWN against the documentation hyperlink's
rectangle, but the CreateHyperlinkLabel calls that would create that control
are currently commented out. GetDlgItem therefore returns NULL,
GetClientRect(NULL, &rect) fails and leaves 'rect' holding whatever was on
the stack, and PtInRect matches that uninitialised rectangle -- so clicks on
unrelated parts of the dialog launch the browser. Which clicks match varies
between runs, since it depends on stack contents.

Bail out when the control does not exist or its rectangle cannot be read, and
zero-initialise the rectangle so it can never hold garbage.
@Rigbyfab4
Rigbyfab4 requested a review from a team as a code owner July 28, 2026 17:03
@google-cla

google-cla Bot commented Jul 28, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@Rigbyfab4
Rigbyfab4 force-pushed the feature/allowed-projects branch from b8de255 to 90e0976 Compare July 28, 2026 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant