RATIS-2529. Bound gRPC worker EventLoopGroup thread count. by CRZbulabula · Pull Request #1466 · apache/ratis

CRZbulabula · 2026-05-20T03:00:36Z

Summary

Fixes RATIS-2529: gRPC worker threads permanently inflate to availableProcessors * 2 after follower restart catch-up.

Add raft.grpc.server.worker.event-loop.threads and raft.grpc.client.worker.event-loop.threads (default 0 = current gRPC behavior).
When > 0, build a dedicated EpollEventLoopGroup (or NioEventLoopGroup when Epoll is unavailable) of that size and wire it into both the server NettyServerBuilders and the client / server-to-server NettyChannelBuilders.
A single boss + worker group is shared across the admin / client / server NettyServerBuilders and the GrpcServerProtocolClient instances managed by one GrpcServicesImpl; both groups are shut down in closeImpl().

This lets operators cap the worker thread count (e.g. 4–8) so a follower catch-up burst can't permanently expand the shared gRPC default EventLoopGroup (which never shrinks its threads once started).

Test plan

New unit test TestGrpcEventLoops covers the helper (thread count, channel-type detection, config-key roundtrip, null shutdown).
New integration test TestGrpcWorkerEventLoopThreads brings up a 3-node MiniRaftClusterWithGrpc with capped server (2) and client (1) worker threads and asserts a client request succeeds.
Existing TestCustomGrpcServices, TestGrpcFactory, TestLeaderInstallSnapshotWithGrpc, TestLinearizableReadWithGrpc, TestGroupInfoWithGrpc all pass.

Add raft.grpc.server.worker.event-loop.threads and raft.grpc.client.worker.event-loop.threads (default 0 = gRPC default, i.e. availableProcessors * 2). When > 0, build a dedicated Epoll/Nio EventLoopGroup of that size and wire it into both the server NettyServerBuilders and the client NettyChannelBuilders so a follower catch-up burst can't permanently inflate the worker thread count.

yandrey321 · 2026-05-20T11:48:25Z

+     * a positive value caps the worker event-loop thread count.
+     */
+    String WORKER_EVENT_LOOP_THREADS_KEY = PREFIX + ".worker.event-loop.threads";
+    int WORKER_EVENT_LOOP_THREADS_DEFAULT = 0;


we need some positive number here like 16/24/32 or some function from CPU cores number.

yandrey321 · 2026-05-20T11:48:55Z

+     * a positive value caps the worker event-loop thread count.
+     */
+    String WORKER_EVENT_LOOP_THREADS_KEY = PREFIX + ".worker.event-loop.threads";
+    int WORKER_EVENT_LOOP_THREADS_DEFAULT = 0;


we need some positive number here like 16/24/32 or some function from CPU cores number.

yandrey321

Did you do any performance testing how thread pools size affect performance?

szetszwo

@CRZbulabula , thanks for working on this!

Please see the comments inlined and also https://issues.apache.org/jira/secure/attachment/13082324/1466_review.patch

szetszwo · 2026-05-20T18:14:05Z

+     * 0 (default) means use the gRPC default (i.e. {@code availableProcessors * 2});
+     * a positive value caps the worker event-loop thread count.
+     */
+    String WORKER_EVENT_LOOP_THREADS_KEY = PREFIX + ".worker.event-loop.threads";


In NettyConfigKeys, we have the following conf:

key: raft.netty.dataStream.client.use-epoll (boolean, default=true) key: raft.netty.dataStream.client.worker-group.size (int, default=28)

Let's use similar conf for gRPC.

szetszwo · 2026-05-20T18:23:21Z

+
+    EventLoopGroup getBossEventLoopGroup() {
+      if (workerEventLoopThreads > 0 && bossEventLoopGroup == null) {
+        bossEventLoopGroup = GrpcEventLoops.newEventLoopGroup(1, server.getId() + "-grpc-boss-ELG");


Let's make bossEventLoopThreads configurable?

szetszwo · 2026-05-21T18:02:25Z

  private GrpcServicesImpl(Builder b) {
-    super(b.server::getId, id -> new PeerProxyMap<>(id.toString(), b::newGrpcServerProtocolClient));
+    super(b.server::getId, id -> new PeerProxyMap<>(id.toString(),
+        peer -> b.newGrpcServerProtocolClient(peer, b.getWorkerEventLoopGroup())));


We should create only one group for all clients.

szetszwo · 2026-05-21T18:10:59Z

+  private final EventLoopGroup bossEventLoopGroup;
+  private final EventLoopGroup workerEventLoopGroup;


We also need a group for client. BTW, let's use shorter names.

private EventLoopGroup serverBosses; private EventLoopGroup serverWorkers; private EventLoopGroup clientWorkers;

szetszwo · 2026-05-21T18:11:40Z

+    GrpcEventLoops.shutdownGracefully(workerEventLoopGroup);
+    GrpcEventLoops.shutdownGracefully(bossEventLoopGroup);


We should call shutdownGracefully() for all groups and then await(..)

szetszwo · 2026-05-21T18:12:45Z

+ * into {@link NettyServerBuilder} / {@link NettyChannelBuilder} consistently
+ * with the matching channel type.
+ */
+public final class GrpcEventLoops {


Let's move NettyUtils to ratis-common and then add the new methods there.

CRZbulabula · 2026-05-25T14:55:36Z

Thanks for the review. I have pushed commit 2088cdf addressing the comments:

Moved NettyUtils to ratis-common as org.apache.ratis.util.NettyUtils and removed the separate GrpcEventLoops helper.
Added gRPC configs similar to NettyConfigKeys:
- raft.grpc.use-epoll
- raft.grpc.client.worker-group.size
- raft.grpc.server.worker-group.size
- raft.grpc.server.boss-group.size
The worker-group.size defaults are now positive and CPU-based: Math.max(1, NettyRuntime.availableProcessors() * 2).
GrpcServicesImpl now creates one serverBosses group, one serverWorkers group, and one clientWorkers group. The server builders share serverBosses/serverWorkers, and all server-to-server clients share clientWorkers.
NettyUtils.shutdownGracefully(...) now starts shutdownGracefully() for all groups first, then awaits termination.
Verified with: mvn -pl ratis-grpc,ratis-netty,ratis-test -am -DskipTests clean test-compile

For the performance question: I ran a manual MiniRaftClusterWithGrpc benchmark on my macOS arm64 host with JDK 21, using 500 warmup ops and 5000 measured async sends at concurrency 16, with 3 iterations per config. Median results for worker sizes 8/16/24/32 were about 290/285/291/313 ops/s, p50 53.4/56.0/54.3/49.8 ms, p99 75.6/75.4/71.2/77.0 ms, and 0 failures. This did not show a regression from using bounded/configurable worker groups; the gRPC worker group handles non-blocking I/O while request execution is still handled by raft.grpc.server.async.request.thread.pool.size.

szetszwo · 2026-05-25T18:02:48Z

@CRZbulabula , thanks for the update! The change looks good.

However, there are some test failures (attempted twice). I am not sure if it is related. It may be the cases that it uses too many threads in some tests and causing timeout. Let's try rerunning the failing tests one more time.

szetszwo

@CRZbulabula , thanks for the update!

Thanks also fixing the other problems in Leader and other tests. Since they are unrelated to gRPC, could file one or more JIRAs for these changes?

Please see the comments inlined and also https://issues.apache.org/jira/secure/attachment/13082486/1466_review2.patch

szetszwo · 2026-05-26T18:26:36Z

  private final int maxMessageSize;
  private final TimeDuration requestTimeoutDuration;
  private final TimeDuration watchRequestTimeoutDuration;
+  private final EventLoopGroup clientWorkers;


Let's use a MemoizedSupplier for lazy initialization.

Done in 00bd24e. GrpcClientRpc now stores the client workers as a MemoizedSupplier<EventLoopGroup>, and GrpcClientProtocolClient obtains the group lazily when building a channel.

szetszwo · 2026-05-26T18:29:19Z

  public GrpcClientRpc(ClientId clientId, RaftProperties properties,
      SslContext adminSslContext, SslContext clientSslContext) {
+    this(clientId, properties, adminSslContext, clientSslContext, newClientWorkers(clientId, properties));
+  }


Let's change it to a static method:

public static GrpcClientRpc create(ClientId clientId, RaftProperties properties, SslContext adminSslContext, SslContext clientSslContext) { final MemoizedSupplier<EventLoopGroup> eventLoopGroup = MemoizedSupplier.valueOf(() -> NettyUtils.newEventLoopGroup( clientId + "-client-workers", GrpcConfigKeys.Client.workerGroupSize(properties), GrpcConfigKeys.useEpoll(properties))); return new GrpcClientRpc(clientId, properties, adminSslContext, clientSslContext, eventLoopGroup); }

private final MemoizedSupplier<EventLoopGroup> clientWorkers; private GrpcClientRpc(ClientId clientId, RaftProperties properties, SslContext adminSslContext, SslContext clientSslContext, MemoizedSupplier<EventLoopGroup> clientWorkers) { super(new PeerProxyMap<>(clientId.toString(), p -> new GrpcClientProtocolClient(clientId, p, properties, adminSslContext, clientSslContext, clientWorkers))); this.clientWorkers = clientWorkers; this.clientId = clientId; this.maxMessageSize = GrpcConfigKeys.messageSizeMax(properties, LOG::debug).getSizeInt(); this.requestTimeoutDuration = RaftClientConfigKeys.Rpc.requestTimeout(properties); this.watchRequestTimeoutDuration = RaftClientConfigKeys.Rpc.watchRequestTimeout(properties); }

Done in 00bd24e. I added GrpcClientRpc.create(...), made the constructor private, and updated GrpcFactory to use the factory method.

szetszwo · 2026-05-26T18:35:08Z

+    try {
+      super.close();
+    } finally {
+      NettyUtils.shutdownGracefully(clientWorkers);


Check if it is initialized:

if (clientWorkers.isInitialized()) { NettyUtils.shutdownGracefully(clientWorkers.get()); }

Done in 00bd24e. close() now checks clientWorkers.isInitialized() before shutting down the shared client worker group.

szetszwo · 2026-05-26T18:41:02Z

+        clientWorkers = NettyUtils.newEventLoopGroup(id + "-client-workers",
+            GrpcConfigKeys.Client.workerGroupSize(props), useEpoll);
+        return new GrpcServicesImpl(this);
+      } catch (RuntimeException | Error e) {


Let's catch Throwable?

Done in 00bd24e. Builder#build() now catches Throwable, shuts down clientWorkers, serverWorkers, and serverBosses, then rethrows RuntimeException and Error directly.

szetszwo · 2026-05-26T18:41:32Z

      channelBuilder.negotiationType(NegotiationType.PLAINTEXT);
    }
    channelBuilder.disableRetry();
+    if (eventLoopGroup != null) {


It won't be null. Let's add requireNonNull:

this.eventLoopGroup = Objects.requireNonNull(eventLoopGroup, "eventLoopGroup == null");

Done in 00bd24e. The constructor now uses Objects.requireNonNull(eventLoopGroup, "eventLoopGroup == null"), and buildChannel(...) always sets the channel type and event-loop group.

szetszwo · 2026-05-26T18:44:27Z

    } else {
      channelBuilder.negotiationType(NegotiationType.PLAINTEXT);
    }
+    if (eventLoopGroup != null) {


Similarly, it won't be null.

Done in 00bd24e. GrpcStubPool.buildManagedChannel(...) now requires a non-null event-loop group and always wires it into the Netty channel builder.

CRZbulabula · 2026-05-28T04:19:23Z

Thanks for the review. I have pushed commit 00bd24e addressing the latest comments:

GrpcClientRpc now uses a MemoizedSupplier<EventLoopGroup> and exposes the suggested static create(...) factory.
The client worker group is initialized lazily and is shut down only when the supplier has been initialized.
GrpcServicesImpl.Builder#build() now catches Throwable, shuts down any created event-loop groups, and rethrows RuntimeException / Error directly.
GrpcServerProtocolClient and GrpcStubPool now require a non-null event-loop group and always configure the Netty channel with it.

I also created the JIRAs for the unrelated fixes:

What would you prefer as the next step for those unrelated fixes: keep them in this PR with the JIRA references, or split them into separate PRs?

yandrey321 reviewed May 20, 2026

View reviewed changes

szetszwo reviewed May 21, 2026

View reviewed changes

RATIS-2529. Address gRPC event loop review comments

2088cdf

CRZbulabula added 4 commits May 26, 2026 13:59

RATIS-2529. Fix gRPC client worker ownership

2c3aeae

RATIS-2529. Advance replied index before write replies

4fc23ac

RATIS-2529. Wait for restart threads before log checks

c429e43

RATIS-2529. Stabilize gRPC CI timing tests

1884a5b

szetszwo reviewed May 26, 2026

View reviewed changes

RATIS-2529. Address gRPC EventLoopGroup review comments

00bd24e

CRZbulabula added 2 commits May 28, 2026 14:49

RATIS-2529. Fix gRPC client RPC checkstyle

9fb727d

RATIS-2529. Stabilize install snapshot restart test

09a7eab

		private final EventLoopGroup bossEventLoopGroup;
		private final EventLoopGroup workerEventLoopGroup;

		GrpcEventLoops.shutdownGracefully(workerEventLoopGroup);
		GrpcEventLoops.shutdownGracefully(bossEventLoopGroup);

Conversation

CRZbulabula commented May 20, 2026

Summary

Test plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yandrey321 left a comment

Choose a reason for hiding this comment

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

szetszwo May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CRZbulabula commented May 25, 2026

Uh oh!

szetszwo commented May 25, 2026

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CRZbulabula commented May 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

szetszwo May 20, 2026 •

edited

Loading