Skip to content

feat(bq_driver): Optimize SQLForeignKeys api impl - #1603

Closed
NeerajDwivedii wants to merge 5 commits into
mainfrom
optimize_SQLForeignKeys_Impl
Closed

feat(bq_driver): Optimize SQLForeignKeys api impl#1603
NeerajDwivedii wants to merge 5 commits into
mainfrom
optimize_SQLForeignKeys_Impl

Conversation

@NeerajDwivedii

@NeerajDwivedii NeerajDwivedii commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Performance Improvement Summary

Previously, SQLForeignKeys generated a single, complex SQL query to handle all three SQLForeignKeys scenarios. The query accessed the INFORMATION_SCHEMA views, performed multiple joins, and included several parameterized arguments. It was executed using the jobs.query client library method. Due to its complexity, it placed a significant load on the BigQuery engine, resulting in longer query execution times and increased response latency.

This PR resolves the performance issue by utilizing table metadata obtained from the GetTable API instead of executing the complex INFORMATION_SCHEMA query. By using the metadata directly, the implementation reduces query overhead on BigQuery and improves the overall response time.

Implementation Details

Scenario 1: Only the Foreign Key table name is provided

  • The implementation calls the GetTable API to retrieve the metadata for the foreign key table.
  • The metadata contains a TableConstraints object, which includes both primary key and foreign key definitions.
  • The implementation utilizes the ForeignKeys structure, where each foreign key entry contains the referenced table and referenced column details.
  • For each foreign key, the implementation calls the GetTable API for the referenced table and extracts its primary key information.
  • The extracted primary keys are searched to determine whether the foreign key's referenced column is present.
  • If the referenced column is found among the primary keys, a corresponding row is added to the SQLForeignKeys result set.

Scenario 2: Both Primary Key and Foreign Key table names are provided

  • The implementation calls the GetTable API to retrieve the metadata for the foreign key table and utilizes the ForeignKeys structure, as described in Scenario 1.
  • It then calls the GetTable API for the primary key table and extracts its primary key information.
  • The extracted primary keys are searched to determine whether the foreign key's referenced column is present.
  • If a matching primary key is found, a corresponding row is added to the result set.

Scenario 3: Only the Primary Key table name is provided

  • The implementation first calls the GetTable API for the primary key table and extracts all primary key information from its metadata.
  • It then calls the ListAllTables API to retrieve all tables within the same dataset and catalog.
  • For each table, the implementation invokes the GetTable API to retrieve its metadata and extract the foreign key definitions.
  • If a foreign key's referenced column matches one of the extracted primary keys, a corresponding row is added to the result set.
  • To improve performance, all GetTable API calls for the tables are executed in parallel, significantly reducing the overall response time despite multiple metadata retrieval operations.

Benchmark Results

@NeerajDwivedii
NeerajDwivedii force-pushed the optimize_SQLForeignKeys_Impl branch from 62fb250 to 5164c47 Compare July 27, 2026 09:03
@NeerajDwivedii

Copy link
Copy Markdown
Collaborator Author

This PR is not required now, It has enhanced implementation in this pr
#1614

@NeerajDwivedii
NeerajDwivedii deleted the optimize_SQLForeignKeys_Impl branch July 28, 2026 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant