Skip to content

chore: migrate java-bigquery to monorepo#12019

Open
chingor13 wants to merge 3411 commits intomainfrom
migrate-java-bigquery
Open

chore: migrate java-bigquery to monorepo#12019
chingor13 wants to merge 3411 commits intomainfrom
migrate-java-bigquery

Conversation

@chingor13
Copy link
Contributor

@chingor13 chingor13 commented Feb 24, 2026

INTEGRATION_TEST_ARGS='-pl !google-cloud-bigquery-jdbc' TEST_WITH_EXISTING=true SKIP_TESTS=true SOURCE_REPO_URL=https://github.com/googleapis/java-bigquery ./monorepo-migration/migrate.sh

Blocked by #12034

ldetmer and others added 30 commits February 28, 2025 12:52
* Implement getArray in BigQueryResultImpl

* deps: update dependency com.google.apis:google-api-services-bigquery to v2-rev20250216-2.0.0 (#3688)

* build(deps): update dependency org.apache.maven.plugins:maven-compiler-plugin to v3.14.0 (#3689)

* deps: update ossf/scorecard-action action to v2.4.1 (#3690)

* deps: update actions/upload-artifact action to v4.6.1 (#3691)

* Remove public from BigQueryResultSet class def

* Remove INTEGER_ARRAY_FIELD_LIST from IT test file

* Add ReadApi IT test

* Try old IT test values

* Convert Time objects to strings for comparison

* Fix Formatting

---------

Co-authored-by: Mend Renovate <bot@renovateapp.com>
)

* docs: update error handling comment to be more precise in samples

* style

* style

* update comment

* ignoredUnusedDeclaredDependencies

* revert wrong commit/PR push ignoredUnusedDeclaredDependencies
#3715)

* deps: exclude io.netty:netty-common from org.apache.arrow:arrow-memory-netty

* use dep version override instead of exclusion

* use dep version override instead of exclusion

* ignoredUnusedDeclaredDependencies
* fix: retry ExceptionHandler not retrying on IOException due to exception translation

* fix: retry ExceptionHandler not retrying on IOException due to exception translation

* fix: retry ExceptionHandler not retrying on IOException due to exception translation

* fix: retry ExceptionHandler not retrying on IOException due to exception translation

* fix: retry ExceptionHandler not retrying on IOException due to exception translation

* cast to HttpBigQueryRpc

* cast to HttpBigQueryRpc

* fix BigQueryRpc mocks in unit tests

* fix clirr check and format

* fix clirr

* fix clirr

* refactor HttpBigQueryRpc.write

* refactor HttpBigQueryRpc.write

* refactor ConnectionImpl HttpBigQueryRpc call

* refactor ConnectionImpl HttpBigQueryRpc call

* refactor create table

* refactor create table

* refactor BigQueryImpl

* refactor ConnectionImpl

* add missing unit test for BigQueryImpl deleteJob

* add tests to validate retry on BIGQUERY_EXCEPTION_HANDLER

* handle exception wrapping/unwrapping with new methods

* update runWithRetries to use BigQuery ruWithRetries to correctly handle thrown IOExceptions

* fix unit test

* add unit tests for retry in TableDataWriteChannel

* remove unnecessary null check

* fix unit test where null is returned in mocks causing errors
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
…perties (#4107)

* feat(jdbc): enforce strict JDBC URL parsing and validation

* feat(jdbc): sync DataSource properties with BigQueryConnection

* test: update tests for strict parsing

* chore: fix failing tests

* chore: throw original `BigQueryJdbcRuntimeException`

* chore: add url encoding to appendProperties to prevent injection vulnerabilites

* chore: refactor `parseAndRemovePartnerTokenProperty()`

* fix: use "ProjectId" instead of "Project_Id" in tests

* chore: add error message sanitization

* chore: address pr feedback

* fix: regex for partnerToken

* chore: use `Joiner` and copy of map when setting properties
@chingor13 chingor13 force-pushed the migrate-java-bigquery branch 2 times, most recently from 7f3c89a to 9d59d3b Compare February 26, 2026 16:57
renovate-bot and others added 10 commits February 26, 2026 15:48
…storage-bom to v3.22.0 (#4117)

Co-authored-by: Lawrence Qiu <lawrenceqiu@google.com>
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
* chore: Disable testRoutineRemoteUDF test

* chore: Add check for null connection

* chore: Add check for null client
* chore: Fix short query tests

* chore: Use global otel instance for test

* chore: Refactor to account for null and invalid server mappings

* chore: Default to UNKNOWN enum

* chore: Fix nullable annotations

* chore: Update class comment

* chore: Move test into the correct package

* chore: Fix test issues
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
@chingor13 chingor13 force-pushed the migrate-java-bigquery branch from 5eecc97 to ca53ffb Compare March 5, 2026 19:12
@suztomo
Copy link
Member

suztomo commented Mar 5, 2026

Note for myself: #12034 is about https://github.com/googleapis/google-cloud-java/actions/runs/22732700304/job/65925748123?pr=12019 failing with these errors:

Error: 5:812 [ERROR] Failed to execute goal on project google-cloud-bigquery: Could not resolve dependencies for project com.google.cloud:google-cloud-bigquery:jar:2.60.1-SNAPSHOT
Error: 5:812 [ERROR] dependency: com.google.cloud:google-cloud-bigquerystorage:jar:3.21.1-SNAPSHOT (compile)
Error: 5:813 [ERROR] 	com.google.cloud:google-cloud-bigquerystorage:jar:3.21.1-SNAPSHOT was not found in https://maven-central.storage-download.googleapis.com/maven2 during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of google-maven-central-copy has elapsed or updates are forced
Error: 5:813 [ERROR] 	com.google.cloud:google-cloud-bigquerystorage:jar:3.21.1-SNAPSHOT was not found in https://repo1.maven.org/maven2 during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of maven-central has elapsed or updates are forced
Error: 5:814 [ERROR] dependency: com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:jar:3.21.1-SNAPSHOT (compile)
Error: 5:814 [ERROR] 	com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:jar:3.21.1-SNAPSHOT was not found in https://maven-central.storage-download.googleapis.com/maven2 during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of google-maven-central-copy has elapsed or updates are forced
Error: 5:815 [ERROR] 	com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:jar:3.21.1-SNAPSHOT was not found in https://repo1.maven.org/maven2 during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of maven-central has elapsed or updates are forced
Error: 5:815 [ERROR] dependency: com.google.cloud:google-cloud-datacatalog:jar:1.92.0-SNAPSHOT (test)

@suztomo
Copy link
Member

suztomo commented Mar 5, 2026

Next error: https://github.com/googleapis/google-cloud-java/actions/runs/22737730633/job/65943374908?pr=12019

Formatting only changed modules: java-bigquery/benchmark,java-bigquery/google-cloud-bigquery,java-bigquery/google-cloud-bigquery-jdbc
[INFO] Scanning for projects...
Warning:  
Warning:  Some problems were encountered while building the effective model for com.google.cloud:google-cloud-bigquery:jar:2.60.1-SNAPSHOT
Warning:  'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: com.google.cloud:google-cloud-storage:jar -> duplicate declaration of version (?) @ line 161, column 17
Warning:  'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: com.google.cloud:google-cloud-datacatalog:jar -> duplicate declaration of version 1.93.0-SNAPSHOT @ line 191, column 17
Warning:  
Warning:  It is highly recommended to fix these problems because they threaten the stability of your build.
Warning:  
Warning:  For this reason, future Maven versions might no longer support building such malformed projects.
Warning:  
Error: ] Could not find the selected project in the reactor: java-bigquery/benchmark @ 
Error:  Could not find the selected project in the reactor: java-bigquery/benchmark -> [Help 1]

This error occurs because the benchmark module is specified as a child module only when the benchmark profile is on. java-bigquery/pom.xml:

  <profiles>
    <profile>
      <id>include-samples</id>
      <modules>
        <module>samples</module>
      </modules>
    </profile>
    <profile>
      <id>benchmark</id>
      <modules>
        <module>benchmark</module>
      </modules>
    </profile>
  </profiles>

@suztomo
Copy link
Member

suztomo commented Mar 6, 2026

"Kokoro - Test: Logging Integration" showing bigquery is strange

Testing the entire monorepo
java-bigquery modified, running split integration test
Compiling and building all modules for java-bigquery
Installing modules:
java-bigquery
Installing submodules:
java-bigquery/google-cloud-bigquery,java-bigquery/google-cloud-bigquery-bom,java-bigquery/google-cloud-bigquery-jdbc

@suztomo
Copy link
Member

suztomo commented Mar 6, 2026

java-bigquery ci / lint (pull_request) failed:

Running in subdir: java-bigquery
~/work/google-cloud-java/google-cloud-java/java-bigquery ~/work/google-cloud-java/google-cloud-java
fatal: bad object 12e2e94140e2a96686f5e070846e5e4215d05b14
Error: Process completed with exit code 128.

It's BASE_SHA.

  .kokoro/build.sh
  shell: /usr/bin/bash -e {0}
  env:
    JAVA_HOME: /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/17.0.18-8/x64
    JAVA_HOME_17_X64: /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/17.0.18-8/x64
    JOB_TYPE: lint
    HEAD_SHA: 29860e9f5aee17fc69fa80414fbdd279af63decb
    BASE_SHA: 12e2e94140e2a96686f5e070846e5e4215d05b14

@suztomo suztomo force-pushed the migrate-java-bigquery branch 2 times, most recently from 953d240 to 2f17901 Compare March 6, 2026 05:39
@suztomo suztomo marked this pull request as ready for review March 6, 2026 15:18
@suztomo suztomo requested a review from a team as a code owner March 6, 2026 15:18
@snippet-bot
Copy link

snippet-bot bot commented Mar 6, 2026

Here is the summary of changes.

You are about to add 161 region tags.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

  • Refresh this comment

@suztomo
Copy link
Member

suztomo commented Mar 6, 2026

All checks passed (except the CLA that says there are too many commits.).

@suztomo
Copy link
Member

suztomo commented Mar 6, 2026

https://fusion2/ci/kokoro/prod:cloud-devrel%2Fclient-libraries%2Fjava%2Fgoogle-cloud-java%2Fpresubmit%2Fbigquery-integration/activity/c1fd6da6-ee28-42d1-bd05-798caedd46f4/log

18:18:08:264 [ERROR] ITBigQueryTest.testOpenTelemetryTracingDatasets:7789 expected: <null> but was: <EU>

@suztomo
Copy link
Member

suztomo commented Mar 6, 2026

Kokoro - Test: BigQuery Integration — Build successful

Now the bigquery integration test succeeded. It was flaky. This pull request no longer modifies the logging integration tests.

@suztomo suztomo force-pushed the migrate-java-bigquery branch from 1b59c04 to febfbe0 Compare March 6, 2026 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.