feat(bq_driver): Optimize SQLForeignKeys api impl - #1603
Closed
NeerajDwivedii wants to merge 5 commits into
Closed
Conversation
NeerajDwivedii
force-pushed
the
optimize_SQLForeignKeys_Impl
branch
2 times, most recently
from
July 24, 2026 05:47
1f2330f to
57558f5
Compare
NeerajDwivedii
force-pushed
the
optimize_SQLForeignKeys_Impl
branch
from
July 27, 2026 06:18
a1b9c72 to
62fb250
Compare
NeerajDwivedii
force-pushed
the
optimize_SQLForeignKeys_Impl
branch
from
July 27, 2026 09:03
62fb250 to
5164c47
Compare
Collaborator
Author
|
This PR is not required now, It has enhanced implementation in this pr |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Performance Improvement Summary
Previously,
SQLForeignKeysgenerated a single, complex SQL query to handle all threeSQLForeignKeysscenarios. The query accessed theINFORMATION_SCHEMAviews, performed multiple joins, and included several parameterized arguments. It was executed using thejobs.queryclient library method. Due to its complexity, it placed a significant load on the BigQuery engine, resulting in longer query execution times and increased response latency.This PR resolves the performance issue by utilizing table metadata obtained from the
GetTableAPI instead of executing the complexINFORMATION_SCHEMAquery. By using the metadata directly, the implementation reduces query overhead on BigQuery and improves the overall response time.Implementation Details
Scenario 1: Only the Foreign Key table name is provided
GetTableAPI to retrieve the metadata for the foreign key table.TableConstraintsobject, which includes both primary key and foreign key definitions.ForeignKeysstructure, where each foreign key entry contains the referenced table and referenced column details.GetTableAPI for the referenced table and extracts its primary key information.SQLForeignKeysresult set.Scenario 2: Both Primary Key and Foreign Key table names are provided
GetTableAPI to retrieve the metadata for the foreign key table and utilizes theForeignKeysstructure, as described in Scenario 1.GetTableAPI for the primary key table and extracts its primary key information.Scenario 3: Only the Primary Key table name is provided
GetTableAPI for the primary key table and extracts all primary key information from its metadata.ListAllTablesAPI to retrieve all tables within the same dataset and catalog.GetTableAPI to retrieve its metadata and extract the foreign key definitions.GetTableAPI calls for the tables are executed in parallel, significantly reducing the overall response time despite multiple metadata retrieval operations.Benchmark Results