[Issue #1325] fix bugs of rockset index by Rolland1944 · Pull Request #1326 · pixelsdb/pixels

Rolland1944 · 2026-05-11T14:30:05Z

No description provided.

Copilot

Pull request overview

This PR addresses multiple correctness issues in the Rockset (RocksDB Cloud) index integration by aligning Java JNI wrappers with their native implementations, improving DB open/CF discovery behavior, and updating the native build/install flow.

Changes:

Fix JNI API mismatches (argument ordering, signatures, null-return handling) between Java bindings and C++ implementations.
Improve RocksDB Cloud startup by listing column families using a CloudEnv handle and retrying DB open when column families are missing.
Update the native rocksdb-cloud build setup (pinned version, extra patching/linking) and add an install step for the JNI shared library.

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
pixels-index/pixels-index-rockset/src/test/java/io/pixelsdb/pixels/index/rockset/TestRocksetIndex.java	Disables the `testPutEntries` test by commenting it out.
pixels-index/pixels-index-rockset/src/main/java/io/pixelsdb/pixels/index/rockset/RocksetFactory.java	Uses CloudEnv for CF listing, refactors CF descriptor creation, adds retry-on-missing-CF open logic.
pixels-index/pixels-index-rockset/src/main/java/io/pixelsdb/pixels/index/rockset/jni/RocksetEnv.java	Exposes `nativeHandle()` publicly for factory/JNI usage.
pixels-index/pixels-index-rockset/src/main/java/io/pixelsdb/pixels/index/rockset/jni/RocksetDB.java	Adds null check for native open result; fixes put arg order; updates CF listing JNI signature.
pixels-index/pixels-index-rockset/pom.xml	Adds additional gRPC test dependencies.
install.sh	Copies the Rockset JNI `.so` into `$PIXELS_HOME/lib` if present.
cpp/pixels-index/pixels-index-rockset/README.md	Documents pinned rocksdb-cloud version/toolchain and updated JNI header generation steps.
cpp/pixels-index/pixels-index-rockset/lib/jni/RocksetEnv.cpp	Switches to CloudEnv-based initialization and credential validation.
cpp/pixels-index/pixels-index-rockset/lib/jni/Rockset.cpp	Fixes options handling for DBCloud open; adds manifest preload for CF listing; aligns createColumnFamily JNI.
cpp/pixels-index/pixels-index-rockset/lib/jni/ReadOptions.cpp	Fixes JNI receiver type for instance method binding.
cpp/pixels-index/pixels-index-rockset/lib/jni/memory_util.cpp	Updates includes and relies on new JNI helpers in `portal.h`.
cpp/pixels-index/pixels-index-rockset/lib/jni/LRUCache.cpp	Updates `NewLRUCache` call to match newer RocksDB API (but leaves extra JNI arg).
cpp/pixels-index/pixels-index-rockset/lib/jni/ColumnFamilyOptions.cpp	Corrects write-buffer-size bounds check using `numeric_limits<size_t>`.
cpp/pixels-index/pixels-index-rockset/lib/jni/BlockBasedTableConfig.cpp	Fixes native parameter ordering/signature to match Java.
cpp/pixels-index/pixels-index-rockset/include/jni/portal.h	Adds JNI helper utilities (pointer conversion, boxed primitives, HashMap helpers).
cpp/pixels-index/pixels-index-rockset/include/jni/io_pixelsdb_pixels_index_rockset_jni_RocksetDB.h	Updates generated JNI signature for CF listing.
cpp/pixels-index/pixels-index-rockset/CMakeLists.txt	Pins rocksdb-cloud version, adds patch step, links lz4, and tweaks build flags/toolchain.
cpp/pixels-index/pixels-index-rockset/cmake/PatchRocksDBCloud.cmake	Patches rocksdb-cloud CMake sources/libs and adjusts AWS S3 client construction.

Files not reviewed (1)

cpp/pixels-index/pixels-index-rockset/include/jni/io_pixelsdb_pixels_index_rockset_jni_RocksetDB.h: Language not supported

Comments suppressed due to low confidence (1)

cpp/pixels-index/pixels-index-rockset/lib/jni/Rockset.cpp:55

GetStringUTFChars can return nullptr (OOM). db_path_chars is used unconditionally to construct std::string, which would crash if null is returned. Please add a null check and return early (allowing the exception to propagate) before using/releasing the pointer.

  // 2. db path
  const char* db_path_chars =
      env->GetStringUTFChars(jdb_path, nullptr);
  std::string db_path(db_path_chars);
  env->ReleaseStringUTFChars(jdb_path, db_path_chars);

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    // @Test
+    // public void testPutEntries() throws SinglePointIndexException, MainIndexException
+    // {
+    //     long timestamp = 1000L;
+    //     long fileId = 1L;
+    //     int rgId = 2;
+
+    //     List<IndexProto.PrimaryIndexEntry> entries = new ArrayList<>();
+
+    //     // Create two entries
+    //     for (int i = 0; i < 2; i++)
+    //     {
+    //         byte[] key = ("testPutEntries" + i).getBytes(); // use different keys
+
+    //         long rowId = i * 1000L;
+
+    //         IndexProto.IndexKey keyProto = IndexProto.IndexKey.newBuilder()
+    //                 .setIndexId(INDEX_ID).setKey(ByteString.copyFrom(key)).setTimestamp(timestamp).build();
+
+    //         IndexProto.RowLocation rowLocation = IndexProto.RowLocation.newBuilder()
+    //                 .setFileId(fileId).setRgId(rgId).setRgRowOffset(i).build();
+


        try
        {
-            existingColumnFamilies = RocksetDB.listColumnFamilies0(rocksetPath);
+            existingColumnFamilies = RocksetDB.listColumnFamilies0(rocksetEnv.nativeHandle(), rocksetPath);


  // 4. bucket / prefix
  const char* bucket_chars =
      env->GetStringUTFChars(jbucket_name, nullptr);
  const char* prefix_chars =
      env->GetStringUTFChars(js3_prefix, nullptr);

-  cfs_options.src_bucket.SetBucketName(bucket_chars);
-  cfs_options.src_bucket.SetObjectPath(prefix_chars);
-  cfs_options.src_bucket.SetRegion(
-      getenv("AWS_DEFAULT_REGION"));
+  const std::string bucket_name(bucket_chars);
+  const std::string object_prefix(prefix_chars);
+  const std::string region(getenv("AWS_DEFAULT_REGION"));

  env->ReleaseStringUTFChars(jbucket_name, bucket_chars);
  env->ReleaseStringUTFChars(js3_prefix, prefix_chars);


+class HashMapJni {
+    public:
+    template<typename K, typename V, typename JK, typename JV>
+    using FnMapKV =
+        std::function<std::unique_ptr<std::pair<JK, JV>>(const std::pair<K, V>&)>;
+
+    static jobject construct(JNIEnv* env, uint32_t initial_capacity) {


 # 3. Configure rocksdb-cloud as external project
-set(ROCKSDB_CLOUD_GIT_REPOSITORY https://github.com/rockset/rocksdb-cloud.git)
-set(ROCKSDB_CLOUD_GIT_TAG master)
+set(ROCKSDB_CLOUD_GIT_REPOSITORY git@github.com:rockset/rocksdb-cloud.git)
+set(ROCKSDB_CLOUD_GIT_TAG v6.7.3)
 set(ROCKSDB_CMAKE_ARGS
    -DCMAKE_BUILD_TYPE=Release
+    -DCMAKE_C_COMPILER=/usr/bin/gcc-10
+    -DCMAKE_CXX_COMPILER=/usr/bin/g++-10


              static_cast<double>(jhigh_pri_pool_ratio),
              nullptr /* memory_allocator */,
              ROCKSDB_NAMESPACE::kDefaultToAdaptiveMutex,
-              ROCKSDB_NAMESPACE::kDefaultCacheMetadataChargePolicy,
-              static_cast<double>(jlow_pri_pool_ratio)));
+              ROCKSDB_NAMESPACE::kDefaultCacheMetadataChargePolicy));


 set(ROCKSDB_CMAKE_ARGS
    -DCMAKE_BUILD_TYPE=Release
+    -DCMAKE_C_COMPILER=/usr/bin/gcc-10
+    -DCMAKE_CXX_COMPILER=/usr/bin/g++-10
    -DWITH_GFLAGS=OFF


gengdy1545 · 2026-05-12T16:39:25Z

For the workflow, consider pinning the Ubuntu version and restricting permissions.
Consider incorporating actions/cache to avoid repeatedly fetching, compiling, and installing the AWS SDK during each run.
Add failure detection to the CMake patch step; specifically, consider using execute_process(COMMAND patch -p1 ...) to apply an actual .patch file. The patch utility will exit with a non-zero status code if no matches are found, and the .patch file itself is human-readable, compatible with git diff, and easier to maintain. This helps prevent silent failures when upgrading rocksdb-cloud in the future.

The suggestions above were informed by AI.

AntiO2 · 2026-05-13T15:33:39Z

For the workflow, consider pinning the Ubuntu version and restricting permissions.

Consider incorporating actions/cache to avoid repeatedly fetching, compiling, and installing the AWS SDK during each run.

Add failure detection to the CMake patch step; specifically, consider using execute_process(COMMAND patch -p1 ...) to apply an actual .patch file. The patch utility will exit with a non-zero status code if no matches are found, and the .patch file itself is human-readable, compatible with git diff, and easier to maintain. This helps prevent silent failures when upgrading rocksdb-cloud in the future.

The suggestions above were informed by AI.

https://github.com/pixelsdb/pixels/actions/runs/25803582244/job/75802943689

Done, CI execution time reduced from 12 minutes to 4 minutes.

fix: fix rocksdb-cloud bugs of rockset index

c1afde0

Rolland1944 force-pushed the ISSUE-1325 branch from 18fdd03 to c1afde0 Compare May 11, 2026 14:40

fix: fix install.sh for rockset cpp

f17b767

AntiO2 requested a review from Copilot May 12, 2026 09:09

Copilot started reviewing on behalf of AntiO2 May 12, 2026 09:10 View session

Copilot AI reviewed May 12, 2026

View reviewed changes

Add rockset CI build and runtime fixes

3961586

AntiO2 added 2 commits May 13, 2026 21:23

Improve rockset CI caching and patch safety

ecb8721

Fix rockset CI AWS SDK include paths

84bd356

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Issue #1325] fix bugs of rockset index#1326

[Issue #1325] fix bugs of rockset index#1326
Rolland1944 wants to merge 5 commits into
pixelsdb:masterfrom
Rolland1944:ISSUE-1325

Rolland1944 commented May 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

gengdy1545 commented May 12, 2026

Uh oh!

AntiO2 commented May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Rolland1944 commented May 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

gengdy1545 commented May 12, 2026

Uh oh!

AntiO2 commented May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants