-
Notifications
You must be signed in to change notification settings - Fork 1.1k
chore(spanner): migrate grpc-gcp into google-cloud-java #13184
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
rahul2393
wants to merge
196
commits into
googleapis:main
Choose a base branch
from
rahul2393:migrate-grpc-gcp-java
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
196 commits
Select commit
Hold shift + click to select a range
ee66ca5
Add the class and tests to parse the ApiConfig.json
Jennnnny e07b256
Implement the channelpool and add unit tests/ basic integration tests
Jennnnny d23c585
Add more tests
Jennnnny e631332
Add more tests
Jennnnny 309a74b
=Add the feature of affinitykey, and add more spanner tests.
Jennnnny f9f858e
Code refactoring, bug fixes, more spanner tests
Jennnnny b8df78b
Code refactoring and Bug fixes
Jennnnny 6dcf12f
using queue to deal with previous msgs
Jennnnny c32d33e
Add gradle build method, handle synchronization of sendMessage()
Jennnnny 48ee524
Add more comments
Jennnnny 22b4b4f
apply conditional var and code refactorying
Jennnnny 6bd42e8
update method getKey() and call.request()
Jennnnny e28bb75
Try to use runnable
Jennnnny b666875
bug fixes
Jennnnny a88bdc8
Add bigtable 101 streams test
Jennnnny 8ee4ec5
Add bigtable tests
Jennnnny f6b1cfa
Merge branch 'grpc-gcp' into queue
Jennnnny 959ca76
delete pom.xml, some minor changes
Jennnnny 6e26839
some minor changes
Jennnnny 293db7f
fix getKey()
Jennnnny 65a704f
reorganization
Jennnnny c44b1d7
add folder gradle
Jennnnny 8ee0bcb
Update README.md
Jennnnny 44a41d2
Update bind/unbind and getKeyFromMessage
Jennnnny 688bc5a
Merge branch 'grpc-gcp' of github.com:Jennnnny/grpc-gcp-java into grp…
Jennnnny e4ef695
Add spanner benchmark
Jennnnny 5476785
Add spanner benchmark
Jennnnny cb257a8
bug fixes
Jennnnny 8fcc8ed
Add content to grpc-gcp/README.md
WeiranFang d490900
Change group to com.google.cloud
WeiranFang 541dd5b
move proto file to structured folder
WeiranFang eeb20db
Make json file concise
WeiranFang ed43585
restructure integration tests and benchmark tests
WeiranFang 0d10a1d
Add GcpManagedChannelBuilder with api config
WeiranFang 3549f9f
Add google-java-format plugin and apply code format
WeiranFang 056b320
Add mvn publish info, remove unused dependencies
WeiranFang 87907b1
Add bootstrap steps for spanner and cbt integration tests
WeiranFang 4dbcd61
Minor fix in spanner integration test
WeiranFang cb8b85f
Add Spanner client tests.
Jennnnny c48d3c2
Add SpannerClient benchmark.
Jennnnny b715c22
Add batch create sessions.
Jennnnny 3e99400
Add tests for spanner BatchCreateSessions
Jennnnny 1fc2957
Apply customized projrct_id in Spanner benchmark
Jennnnny ae75407
Update README.md
Jennnnny 5398ac6
Support ubind multiple keys
Jennnnny 3f57aa0
Merge pull request #24 from Jennnnny/batch-sessions
WeiranFang 6330fb5
Version vpdates
Jennnnny cf968a4
update message builder
Jennnnny 9a002ee
Revert some typos
Jennnnny 8531180
Add a new method setPoolSize to channel builder
Jennnnny 8efc384
Update version
Jennnnny 5153376
Add more tests
Jennnnny 2258266
remove log
Jennnnny 994149d
move benchmarks to a new directory
Jennnnny dea6a5f
Update README.md
Jennnnny 6eeb02a
Update grpc-gcp/README.md
nimf 444a504
Update grpc-gcp/src/test/java/com/google/grpc/gcp/SpannerIntegrationT…
nimf 1d47b71
Respect maxConcurrentStreamsLowWatermark setting.
nimf 89918b8
Merge pull request #75 from nimf/low-watermark-fix
nimf 6df586c
Add optional metrics.
nimf 33bee7b
Addressed PR comments.
nimf 1b8d15d
Add fallback functionality for non-ready channels.
nimf 1e28fc2
Addressed PR comments.
nimf 0964d24
Change ChannelRef's activeStreamsCount to AtomicInteger
nimf 7c9341e
Move activeStreamsCount changes to another PR
nimf 99e283e
Addressed PR comments.
nimf 83f9640
Merge branch 'master' into requests-fallback
nimf 9d006f0
Enable unresponsive connection detection
nimf 3ec1749
Added test. Fixed deadline exceeded status condition.
nimf a30cd3c
Change options builders to allow rebuilding
nimf 03697a2
Add more metrics.
nimf 4b4de25
Change pool_index metric label to be 1 based.
nimf 4565622
Add fallback count metric.
nimf 9d57a69
Add unresponsive connection detection metrics.
nimf ab4f2e1
Prepare for maxConcurrentStreamsLowWatermark to be 0.
nimf b9efec1
Fix authority method.
nimf 0958710
Rename package to com.google.cloud.grpc
nimf 17a4988
Add implementation version.
nimf 52690d7
Change java package for proto.
nimf c26d7ab
Configure nexus release.
nimf f8ebab4
Version 1.0.0
nimf 1dcc31e
Bump version
nimf fd89c9d
Adjust Spanner tests to use zero low watermark.
nimf 6242be7
Update readme to note zero low watermark for Spanner example
nimf 4bb7941
Adjust bind/unbind logic and remove bindLock.
nimf 5dfa569
Remake channelRefs list, add tests for fallbacks.
nimf b8bf7b2
Remove channelRefById map.
nimf 0047427
Lint/style/readability fixes.
nimf 0dff3ec
Lint/style/readability fixes.
nimf b802472
Bring back null checks to bind/unbind methods.
nimf e0a8670
Merge branch 'master' into channel-refs-remake
nimf 018d273
Merge branch 'master' into lint-fix
nimf 3b271b5
Lint/style/readability fixes.
nimf edd1d38
Add a warning for possible new connectivity states.
nimf 52223d8
Revert: Add a warning for possible new connectivity states.
nimf 57477f6
New release 1.1.0
nimf aca7229
Bump version
nimf 79227cc
Consolidate channel pool config in the GcpChannelPoolOptions.
nimf 10445c9
Add optional round-robin channel selection for affinity binding calls.
nimf eb1309d
Count fallbacks for calls without an affinity key.
nimf 63f0a67
Merge pull request #127 from nimf/rr-bind
nimf ce62964
Log metrics with fine log level.
nimf 67d8855
Bind keys on the fly.
nimf 7dbbabb
Add pool index to all logs.
nimf 6342c5c
Log metric options.
nimf 3cd00fb
fixed test comments
nimf 4806945
Merge pull request #131 from nimf/more-logs
nimf f3c6bb0
Log creation and shutdown.
nimf 0ed0733
Log active streams, calls, and affinity counts for each channel.
nimf 0311f12
Log channel creation, state change, and forcing to idle state.
nimf 93cb3c1
Log bind/unbind affinity keys.
nimf f6e1c7b
Log channel chosen for a bind call.
nimf e6524ee
Log fallback successes and failures.
nimf a27d23e
Add minSize option.
nimf fb2be2c
Merge branch 'master' into min-size
nimf 661f6c3
Add MultiEndpoint
nimf 142eb4c
Allow GcpManagedChannel state change notifications.
nimf 79d9c3f
Remove race conditions when creating new channels.
nimf ff52615
Add minSize to options toString, reorder options in toString.
nimf db16162
Add multi-endpoint feature.
nimf c2c8635
Address PR comments
nimf 91ac930
Address PR comments
nimf 2b88919
Add TODOs for endpoints credentials.
nimf 60517d5
Rename maybeFallback to maybeUpdateCurrentEndpoint.
nimf 43d0e7a
Add authorityFor method
nimf 395bfe3
Update TODOs for different channel credentials.
nimf ed65538
New release 1.2.0
nimf 13ccf57
New release 1.2.1.
nimf 1be0c80
Add switching delay.
nimf 0809957
Add switching delay to GcpMultiEndpointChannel.
nimf 0b0cdba
Prevent preemptive switching in MultiEndpoint.
nimf 7af2cd1
Reduce state checks for repeating endpoints.
nimf f64e18e
Add add/remove endpoint test case for switching delay.
nimf fb6a57c
Look for the multi-endpoint specified in gRPC context.
nimf 6b555f9
Reduce tests flakiness.
nimf 645f0f7
grpc-gcp version bump to 1.3.0
nimf 78f841e
fix: allow specifying protocol in the endpoint
nimf a55620f
bump version to 1.3.1
nimf 4857b4b
fix: Fix race condition when running and adding callbacks
nimf 022a2c2
Bump version to 1.3.2
nimf a2c8d25
MultiEndpoint metrics
nimf 77875e7
Version bump. Deps upgrade.
nimf 2a6f637
Rename type label
nimf f8805ee
Support common labels for multi-endpoint metrics
nimf 3fb64c5
Add disable affinity feature
nimf 9c088ce
fix: decrease test flakiness
nimf 056c86d
lint: fix format
nimf 757b13e
Version bump to 1.5.0
nimf 0b940e7
feat: Affinity keys cleanup. Allow setting keys lifetime (#174)
nimf 6c087ad
feat: manual affinity via call options or context (#175)
nimf 41126db
Version bump to 1.6.0 (#176)
nimf 00ffb9c
Log manual affinity and disabling affinity (#177)
nimf 84315f2
Version bump to 1.6.1 (#178)
nimf 945e2c2
Bump grpc-gcp to dev 1.6.2-SNAPSHOT version (#188)
nimf ce5dcad
feat: Dynamic channel pool scaling (#194)
nimf f598b08
Gradle update to 7.6 (#211)
nimf d3d399f
chore: switch to jreleaser and central.sonatype.com (#216)
nimf 4d2f0b8
Bump version to 1.7.0 (#217)
nimf 8153a1a
fix javadoc (#218)
nimf 0f7aff9
feat: End to end fallback (#221)
nimf faa4f95
feat: migrate opencensus metrics to opentelemetry (#222)
rahul2393 954c316
Version bump to 1.8.0 (#223)
nimf 011baf2
Bump to snapshot version. (#224)
nimf 7e0ffe3
feat: set channelID in call options for downstream interceptor to get…
rahul2393 30402e3
Version bump to 1.9.0 (#226)
nimf 53143a4
bump to snapshot version (#227)
nimf a18a3d1
fix: race for metrics values. Plus fix log metrics when reported via …
nimf 365db5e
Bump to 1.9.1 (#229)
nimf 4511def
Bump to 1.9.2-SNAPSHOT (#230)
nimf a732bbd
fix: Use daemon threads for grpc-gcp background executors by default …
rahul2393 0df15d7
fix: unbind manual affinity keys after terminal calls (#232)
rahul2393 757cdbd
Bump to 1.9.2
rahul2393 72bd49f
Bump to 1.9.3-SNAPSHOT
rahul2393 6ba5f72
perf: use power-of-two random choices for channel selection to avoid …
rahul2393 802f197
Bump to 1.10.0
rahul2393 d787372
feat: migrate grpc-gcp to maven (#237)
rahul2393 b46b7b8
chore(spanner): import grpc-gcp history
rahul2393 3603326
chore(spanner): wire grpc-gcp module into release
rahul2393 241463b
fix copyright
rahul2393 53fa96f
fix flaky test
rahul2393 1409dc3
fix pom
rahul2393 7e835e3
Merge branch 'main' into migrate-grpc-gcp-java
rahul2393 9b614f3
fix maven test file
rahul2393 9539915
add java8 profile
rahul2393 3d28ec1
move to separate module
rahul2393 642e2c2
add pom.xml
rahul2393 ac2fa9e
fix test
rahul2393 aa45641
fix build
rahul2393 d06bbc2
skip native build
rahul2393 eba6cfd
Merge branch 'main' into migrate-grpc-gcp-java
rahul2393 b0cea77
Merge branch 'main' into migrate-grpc-gcp-java
rahul2393 e094728
update Codeowners, and pom comments
rahul2393 c0da4c3
Merge branch 'main' into migrate-grpc-gcp-java
rahul2393 f60241a
Merge remote-tracking branch 'origin/main' into migrate-grpc-gcp-java
rahul2393 681cf22
rename to grpc-gcp-java
rahul2393 d350034
Merge branch 'main' into migrate-grpc-gcp-java
rahul2393 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| { | ||
| "api_shortname": "grpc-gcp", | ||
| "name_pretty": "gRPC GCP Extension", | ||
| "product_documentation": "https://github.com/GoogleCloudPlatform/grpc-gcp-java", | ||
| "api_description": "gRPC extension library for Google Cloud Platform clients.", | ||
| "client_documentation": "https://cloud.google.com/java/docs/reference/grpc-gcp/latest/overview", | ||
| "release_level": "stable", | ||
| "transport": "grpc", | ||
| "language": "java", | ||
| "repo": "googleapis/google-cloud-java", | ||
| "repo_short": "google-cloud-java", | ||
| "distribution_name": "com.google.cloud:grpc-gcp", | ||
| "library_type": "OTHER", | ||
| "codeowner_team": "@googleapis/cloud-sdk-java-team", | ||
| "recommended_package": "com.google.cloud.grpc", | ||
| "min_java_version": 8 | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,125 @@ | ||
| # Changelog | ||
|
|
||
| ## 1.10.0 (2026-04-10) | ||
|
|
||
| ### Fixes | ||
|
|
||
| * Use power-of-two random choices for channel selection to avoid thundering herd | ||
|
|
||
|
|
||
| ## 1.9.2 (2026-03-21) | ||
|
|
||
| ### Fixes | ||
|
|
||
| * Use daemon threads for grpc-gcp background executors by default | ||
|
|
||
| ### Features | ||
|
|
||
| * Support of Unbind manual affinity keys after terminal calls | ||
|
|
||
|
|
||
| ## 1.9.1 (2025-12-27) | ||
|
|
||
| ### Fixes | ||
|
|
||
| * Fix race for metrics values (#228) | ||
|
|
||
| ## 1.9.0 (2025-12-15) | ||
|
|
||
| ### Features | ||
|
|
||
| * Set channelID in call options for downstream interceptors (#225) | ||
|
|
||
| ## 1.8.0 (2025-12-12) | ||
|
|
||
| ### Features | ||
|
|
||
| * End to end fallback (#221) | ||
| * Migrate opencensus metrics to opentelemetry (#222) | ||
|
|
||
| ## 1.7.0 (2025-09-23) | ||
|
|
||
| ### Features | ||
|
|
||
| * Dynamic channel pool scaling (#194) | ||
|
|
||
| ## 1.6.1 (2024-06-17) | ||
|
|
||
| ### Features | ||
|
|
||
| * Log manual affinity and disabling affinity (#177) | ||
|
|
||
| ## 1.6.0 (2024-05-09) | ||
|
|
||
| ### Features | ||
|
|
||
| * Affinity keys cleanup. Allow setting keys lifetime (#174) | ||
| * Manual affinity via call options or context (#175) | ||
|
|
||
| ## 1.5.0 (2023-11-08) | ||
|
|
||
| ### Features | ||
|
|
||
| * Add disable affinity feature (#167) | ||
|
|
||
| ## 1.4.1 (2023-01-30) | ||
|
|
||
| ### Fixes | ||
|
|
||
| - Support common labels for multi-endpoint metrics (#160) | ||
|
|
||
| ## 1.4.0 (2023-01-27) | ||
|
|
||
| ### Features | ||
|
|
||
| - MultiEndpoint metrics (#158) | ||
|
|
||
| ## 1.3.2 (2022-12-02) | ||
|
|
||
| ### Fixes | ||
|
|
||
| * Fix race condition when running and adding state change callbacks. (#155) | ||
|
|
||
| ## 1.3.1 (2022-10-24) | ||
|
|
||
| ### Fixes | ||
|
|
||
| * MultiEndpoint: allow specifying protocol in the endpoint. (#151) | ||
|
|
||
| ## 1.3.0 (2022-10-12) | ||
|
|
||
| ### Features | ||
|
|
||
| * Add switching delay to GcpMultiEndpointChannel. (#145) | ||
| * Look for a multi-endpoint specified in gRPC context. (#149) | ||
|
|
||
| ## 1.2.0 (2022-07-18) | ||
|
|
||
| ### Features | ||
|
|
||
| * multi-endpoint (#135). | ||
| * round-robin for bind calls (#127). | ||
| * minSize of the channel pool (#134). | ||
| * detailed logs (#133). | ||
| * log metrics (#131). | ||
| * map a key to current channel when bound call arrives but no mapping for the | ||
| key exists (#132). | ||
| * consolidate channel pool config in the GcpChannelPoolOptions (#109). | ||
|
|
||
| ## 1.1.0 (2021-07-20) | ||
|
|
||
| ### Features | ||
|
|
||
| * improve bind/unbind logic. | ||
| * improve getChannelRef performance for calls without channel affinity. | ||
| * more accurate affinity counts. | ||
| * allow to rebind a key to another channel and adjust affinity counts accordingly. | ||
|
|
||
| ## 1.0.0 (2021-06-11) | ||
|
|
||
| ### Features | ||
|
|
||
| * gRPC channel pool with configurable channel affinity. | ||
| * provide a channel with minimum active streams for a new call. | ||
| * an optional fallback from non-ready channel to a ready channel. | ||
| * an option to detect unresponsive channels. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,121 @@ | ||
| # gRPC-GCP for Java | ||
|
|
||
| grpc-gcp provides gRPC support for Google Cloud Clients. | ||
|
|
||
| This module is currently used by the Google Cloud Spanner client library to provide | ||
| gRPC-GCP channel pooling and related transport behavior. It is published separately | ||
| as `com.google.cloud:grpc-gcp` because Spanner exposes grpc-gcp types in its public | ||
| API. | ||
|
|
||
| ## Usage | ||
|
|
||
| Use Spanner API as an example. | ||
|
|
||
| First, Create a json file defining API configuration, with ChannelPoolConfig and MethodConfig. | ||
|
|
||
| ```json | ||
| { | ||
| "channelPool": { | ||
| "maxSize": 3, | ||
| "maxConcurrentStreamsLowWatermark": 0 | ||
| }, | ||
| "method": [ | ||
| { | ||
| "name": [ "google.spanner.v1.Spanner/CreateSession" ], | ||
| "affinity": { | ||
| "command": "BIND", | ||
| "affinityKey": "name" | ||
| } | ||
| }, | ||
| { | ||
| "name": [ "google.spanner.v1.Spanner/GetSession" ], | ||
| "affinity": { | ||
| "command": "BOUND", | ||
| "affinityKey": "name" | ||
| } | ||
| }, | ||
| { | ||
| "name": [ "google.spanner.v1.Spanner/DeleteSession" ], | ||
| "affinity": { | ||
| "command": "UNBIND", | ||
| "affinityKey": "name" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| Initialize `GcpManagedChannel` based on this API config file. | ||
|
|
||
| ```java | ||
| String API_CONFIG_FILE = "api_config_file.json" | ||
| String SPANNER_TARGET = "spanner.googleapis.com"; | ||
| ... | ||
|
|
||
| ManagedChannelBuilder delegateChannelBuilder = ManagedChannelBuilder.forAddress(SPANNER_TARGET, 443); | ||
| GcpManagedChannelBuilder gcpBuilder = | ||
| GcpManagedChannelBuilder.forDelegateBuilder(delegateChannelBuilder) | ||
| .withApiConfigJsonFile(jsonApiConfig); | ||
| ManagedChannel gcpChannel = gcpBuilder.build(); | ||
| ``` | ||
|
|
||
| Create Cloud API stub using `GcpManagedChannel`. | ||
|
|
||
| ```java | ||
| GoogleCredentials creds = getCreds(); | ||
| SpannerBlockingStub stub = | ||
| SpannerGrpc.newBlockingStub(gcpChannel) | ||
| .withCallCredentials(MoreCallCredentials.from(creds)); | ||
| ``` | ||
|
|
||
| ## Build from source | ||
|
|
||
| Download source. | ||
|
|
||
| ```sh | ||
| git clone https://github.com/googleapis/google-cloud-java.git && cd google-cloud-java/grpc-gcp-java | ||
| ``` | ||
|
|
||
| Build project with unit tests. | ||
|
|
||
| ```sh | ||
| mvn verify | ||
| ``` | ||
|
|
||
| Install the jar to your local Maven repository. | ||
|
|
||
| ```sh | ||
| mvn install | ||
| ``` | ||
|
|
||
| ## Test | ||
|
|
||
| Setup credentials. See [Getting Started With Authentication](https://cloud.google.com/docs/authentication/getting-started) for more details. | ||
|
|
||
| ```sh | ||
| export GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json | ||
| ``` | ||
|
|
||
| ```sh | ||
| export GCP_PROJECT_ID=project_id | ||
| ``` | ||
|
|
||
| Run unit tests. | ||
|
|
||
| ```sh | ||
| mvn test | ||
| ``` | ||
|
|
||
| Integration tests are kept as source references but are not wired into the default monorepo Maven lifecycle. | ||
|
|
||
| ## Publish | ||
|
|
||
| This artifact is released by the `google-cloud-java` monorepo release pipeline. | ||
|
|
||
| ## Code Format | ||
|
|
||
| Run google-java-format | ||
|
|
||
| ```sh | ||
| mvn fmt:format | ||
| ``` | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a blurb that this module is only used by Spanner?