feat(release): publish a portable Linux build of openshell-gateway compatible with glibc 2.35 (Ubuntu 22.04, Debian 12, RHEL 9)

[theFong]

Problem Statement

OpenShell's published Linux binaries (openshell, openshell-gateway, openshell-sandbox) are dynamically linked against glibc 2.39 because CI builds them on Fedora 44. That excludes every long-term-support Linux distro shipping today:

Distro	glibc	Runs openshell-gateway?
Ubuntu 24.04 Noble	2.39	✅
Ubuntu 22.04 Jammy	2.35	❌
Ubuntu 20.04 Focal	2.31	❌
Debian 13 Trixie	2.41	✅
Debian 12 Bookworm	2.36	❌
RHEL 10	2.39	✅
RHEL 9	2.34	❌
Amazon Linux 2023	2.34	❌

On any of the ❌ rows the binary fails at startup with:

/usr/bin/openshell-gateway: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.38' not found (required by /usr/bin/openshell-gateway)
/usr/bin/openshell-gateway: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.39' not found (required by /usr/bin/openshell-gateway)

NVIDIA Brev supports Ubuntu 22

NVIDIA Brev provisions Ubuntu 22.04 Jammy as its default base image for every Brev Launchable. That means any one-click deployment of OpenShell from Brev — including our own brevdev/dex-ui Launchable — has to override Brev's default image with a custom imageId pointing at ubuntu-2404-lts-amd64. We can't currently rely on the operator's preferred Brev image, can't piggyback on Brev's hardened defaults, and can't help operators who already have a 22.04 fleet they'd like to install OpenShell onto.

Same pain point applies to the much larger universe of operators running Ubuntu 22.04 LTS in their own VPCs (it's the most common Linux on AWS/GCP/Azure right now), RHEL 9 shops, Debian 12 servers, etc. They all hit the GLIBC error on first install and have no documented escape hatch other than "upgrade your OS."

Reproduction:

$ docker run -it ubuntu:22.04 bash -c "
    apt-get update -y && apt-get install -y curl
    curl -fsSL https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
    openshell-gateway --version
  "
…
/usr/bin/openshell-gateway: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.39' not found

Proposed Design

Publish a second binary build target per release that links against an older glibc. Two reasonable variants — either works, the first is preferred:

Option A (preferred): Build on Debian Bookworm in cargo-zigbuild

cargo-zigbuild lets us cross-compile against any libc version we choose. Targeting glibc 2.31 (Ubuntu 20.04 / Debian 11 baseline) covers every distro listed above plus everything older still in LTS. The Cargo.toml release profile and Cross.toml workflow would gain a second matrix dimension:

# .github/workflows/rust-native-build.yml — sketch
strategy:
  matrix:
    target:
      - x86_64-unknown-linux-gnu          # existing; glibc 2.39
      - x86_64-unknown-linux-gnu.2.31     # new; glibc 2.31 baseline
      - aarch64-unknown-linux-gnu
      - aarch64-unknown-linux-gnu.2.31

Release artifacts: ship both, with names that make the constraint discoverable:

• openshell-gateway-x86_64-unknown-linux-gnu.tar.gz (current; glibc 2.39+)
• openshell-gateway-x86_64-unknown-linux-gnu-glibc2.31.tar.gz (new; glibc 2.31+)

install.sh detects the host's glibc version (ldd --version parse) and picks the right artifact automatically. The current "latest glibc" build stays the default for distros that can handle it (smaller binary, more compiler optimizations, etc.); the portable build is only used when the host can't satisfy the modern variant.

Both .deb and .rpm packages would still ship — but .deb would gain a second variant (openshell_x.y.z-1_amd64_glibc2.31.deb) for Bookworm/Jammy etc. (.rpm already targets fc44; adding a .el9.x86_64.rpm from the portable build would cover RHEL 9.)

Option B (cheaper, less flexible): MUSL static binary

We already publish openshell-aarch64-unknown-linux-musl.tar.gz for the CLI on arm64. Extending the musl matrix to x86_64 and to the gateway + sandbox binaries would give us a statically-linked artifact that runs on any Linux regardless of glibc version. Cost: musl/glibc behavior differences on edge cases (DNS resolver, threading, mmap'd files), larger binary (~25% bigger), worse perf on certain workloads.

Option A is more work but ships a binary that looks/feels identical to today's just with a lower libc floor. Option B is a one-line workflow change but introduces a runtime that's less battle-tested for daemons doing heavy syscalls (which the gateway is).

Alternatives Considered

1. Just install via Docker. OpenShell already publishes ghcr.io/nvidia/openshell/gateway with glibc baked in. Works, but:

   • Forces operators into a container topology they may not want (especially mTLS rotation, log shipping, host-namespace mounts).
   • Means the .deb / .rpm + install.sh paths are de facto broken on the most common LTS distros, with no upstream acknowledgement.
   • The container image itself still has to pull compute_drivers=["podman"|"docker"] config + the appropriate host socket, which moves the integration problem rather than removing it.

2. Document "Ubuntu 24.04+ required" in install.sh and exit with a friendlier error. Necessary regardless — even with this issue resolved we should detect glibc-too-old at install time — but doesn't help operators who can't change their OS.

3. Build openshell-gateway from source on the host. A ~15-25 min cold cargo build for any operator who can't run our binary. Hostile UX, also a non-trivial host requirement (Rust toolchain + dev headers).

4. Backport newer glibc to old distros. Brittle, defeats the LTS contract, not a real option.

Agent Investigation

This came up while building a Brev Launchable that mirrors pnpm play for brevdev/dex-ui — a one-click "spin up an OpenShell + dex-ui stack" environment.

• Brev's default GCP base image is Ubuntu 22.04.5 LTS (Jammy Jellyfish), glibc 2.35. Confirmed by cat /etc/os-release && ldd --version on a fresh n1-standard-2 instance.
• install.sh from main installs the .deb cleanly; openshell-gateway --version immediately fails with the GLIBC_2.39 error above.
• We currently work around this by setting createWorkspaceRequest.imageId = "projects/ubuntu-os-cloud/global/images/family/ubuntu-2404-lts-amd64" in the Launchable JSON, which forces a 24.04 base. That works but couples our Launchable to a single OS family, and doesn't help operators using OpenShell outside of Brev.
• I inspected releases v0.0.36 → v0.0.43 with strings openshell-gateway | grep '^GLIBC_2\.' | sort -V. Every released binary in that window references GLIBC_2.34, GLIBC_2.38, GLIBC_2.39 — pinning to an older OpenShell version does not unblock Ubuntu 22.04.
• The CI builder image / runner config is the single point where this changes. The cargo-zigbuild route preserves the existing GNU dynamic-linking ABI while pushing the minimum glibc down — no source-code changes needed in any crate.

Acceptance Criteria

• CI produces a openshell-gateway-x86_64-unknown-linux-gnu-glibc<MIN>.tar.gz artifact for each tagged release.
• install.sh picks the portable artifact automatically when the host's glibc is older than the modern build's minimum.
• A .deb variant targeting Debian Bookworm / Ubuntu Jammy ships in the release (openshell_X.Y.Z-1_amd64_glibc2.31.deb or similar).
• Manual smoke: docker run -it ubuntu:22.04 → install.sh → openshell-gateway --version exits 0 and prints the version.

[johntmyers]

related: #1117

[pimlock]

I think this issue should first focus on the host-executed Linux binaries. The Python wheel work in #1117 is related, but it has a different packaging shape and should probably be discussed separately.

The current Linux binary compatibility picture looks roughly like this:

Component	Current libc variant / floor	What to do for wider compatibility
openshell-gateway	GNU/glibc, host-executed gateway binary; Docker and Podman drivers are linked into this process	Lower the glibc floor here. This covers Docker/Podman gateway deployments too.
openshell-driver-vm	GNU/glibc, host-executed subprocess used by the VM runtime	Lower the glibc floor here as well.
Standalone openshell CLI	Static musl	Already avoids host glibc.
openshell-sandbox supervisor	Runs inside sandbox containers/pods/VM guests; Docker/Podman delivery is controlled by supervisor image/bind mount	Not the main host OS glibc blocker for this issue.

There are two main implementation options for the GNU/glibc host-executed binaries:

1. Publish one portable GNU/glibc artifact
   Build openshell-gateway and openshell-driver-vm with a lower glibc floor, for example glibc 2.31 or glibc 2.28, and make that the default Linux artifact.

2. Publish multiple GNU/glibc tiers
   Keep the current builder-floor artifact and add a portable artifact, then update install.sh, package naming, docs, and release validation to choose the right one.

My preference is to start with option 1 if validation passes. For Rust, the cost of targeting an older glibc baseline should usually be smaller than it would be for a C/C++ project, and a single Linux artifact keeps the install and support story much simpler.

Some useful prior art:

Project / tool type	Linux strategy	Relevance
ripgrep and bat	Publish both GNU and musl Linux assets	Shows the dual-track model: GNU for normal Linux integration, musl as a portable fallback.
rust-analyzer	Publishes GNU Linux assets and also a musl Linux asset	Good cautionary example that GNU builds need a deliberate floor. Their changelog explicitly notes that moving the build to Ubuntu 20.04 raised the minimum supported glibc to 2.31.
Rust toolchain (rustc / cargo)	Official Linux GNU targets document a glibc 2.17+ baseline	Shows the controlled GNU baseline model for core tooling that needs broad enterprise compatibility.

For OpenShell, I think the best first step for host-executed runtime binaries is the controlled GNU baseline model rather than a full musl switch. openshell-gateway is a daemon with more host integration surface than the CLI, so keeping GNU/glibc behavior close to today while lowering the required glibc version seems like the lower-risk compatibility fix. The standalone openshell CLI already follows the other common Rust distribution strategy by publishing a static musl artifact.

Suggested acceptance criteria:

• CI builds openshell-gateway and openshell-driver-vm with an explicit glibc floor.
• CI verifies the generated binaries do not reference symbols above that floor.
• Smoke tests run on representative older distros, at minimum Ubuntu 22.04 and ideally RHEL 9 / Amazon Linux 2023. If we choose glibc 2.28, also validate RHEL 8.
• install.sh and package publishing consume the portable artifact by default, unless we decide to support multiple glibc tiers.