Ship the CUDA delegate in a CUDA-enabled ExecuTorch wheel - #21478
Draft
shoumikhin wants to merge 12 commits into
Draft
Ship the CUDA delegate in a CUDA-enabled ExecuTorch wheel#21478shoumikhin wants to merge 12 commits into
shoumikhin wants to merge 12 commits into
Conversation
Contributor
Author
|
Stack from ghstack (oldest at bottom): |
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21478
Note: Links to docs will display an error until the docs builds have been completed. ❌ 62 New Failures, 2 Pending, 1 Unrelated Failure, 42 Unclassified FailuresAs of commit 00d5527 with merge base b26b9ac ( NEW FAILURES - The following jobs have failed:
UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:
BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
shoumikhin
added a commit
that referenced
this pull request
Jul 29, 2026
## Why this is needed
The previous change ships a default Linux wheel with the core runtime as a
linkable libexecutorch.so, but no GPU delegate. To actually run a model on
an NVIDIA GPU (including a coalesced TensorRT + CUDA .pte), a C++ program
also needs the CUDA delegate as a loadable library.
This follows the way PyTorch distributes CUDA: the default wheel has the CPU
runtime, and a CUDA-specific wheel (built on a CUDA machine / installed from
a CUDA index) adds the GPU binaries. Here the CUDA-enabled ExecuTorch wheel
ships the CUDA delegate shared libraries alongside the same core runtime, so
a C++ consumer can link and load the CUDA backend with no source build.
Today the CUDA backend only exists as a static archive that is baked into
the Python module (_portable_lib.so). There is no standalone loadable
backend a C++ program can use, the way QNN already ships
libqnn_executorch_backend.so. This change adds that.
## What is inside
Added only when the wheel is built with CUDA (Linux):
- executorch/lib/libexecutorch_cuda_backend.so: a loadable CUDA delegate.
It wraps the existing static CUDA backend so that loading the library runs
its "CudaBackend" registration and registers into the one process-wide
runtime in libexecutorch.so. It is libtorch-free.
- executorch/lib/libextension_cuda.so: the CUDA caller-stream extension. It
is a single shared library on purpose so its per-thread caller-stream state
has exactly one instance across the whole process (needed so a TensorRT
delegate and the CUDA backend share one stream).
- executorch/include/executorch/extension/cuda/*.h: the caller-stream headers.
- CMake targets executorch::cuda_backend and executorch::extension_cuda in the
shipped executorch-config.cmake, defined only when the libraries are present.
The default (CPU) wheel is unchanged: none of these are added unless the
wheel is built with CUDA. Windows and macOS wheels are unchanged.
## How to use it
```bash
pip install executorch # from a CUDA-enabled index/build
```
```cmake
find_package(executorch CONFIG REQUIRED)
add_executable(my_runner main.cpp)
# Linking the CUDA backend force-loads it so it registers on startup.
target_link_libraries(my_runner PRIVATE
executorch::runtime
executorch::cuda_backend)
```
```bash
cmake -S . -B build \
-DCMAKE_PREFIX_PATH="$(python -c 'import executorch.utils as u; print(u.cmake_prefix_path)')"
cmake --build build
```
At runtime, get_backend_class("CudaBackend") returns the registered backend.
## Test plan
Verified on Linux x86_64 with an NVIDIA H100 and CUDA 12.8:
- Built the wheel with CUDA enabled and confirmed it contains
libexecutorch_cuda_backend.so and libextension_cuda.so under executorch/lib/,
the CUDA headers, and the CMake config.
- Installed the wheel into a clean virtual environment and built a standalone
C++ program via find_package(executorch) linking executorch::cuda_backend.
The program compiled, linked, ran, and get_backend_class("CudaBackend")
returned non-null, both when the backend was linked directly and when it was
dlopen'd.
- Confirmed it runs with no LD_LIBRARY_PATH: the shipped libraries resolve each
other through an $ORIGIN rpath because they are co-located in executorch/lib/.
- Confirmed the program and the CUDA backend link no libtorch/libc10 (via ldd).
- Confirmed there is a single libextension_cuda.so that both the backend and
the consumer resolve to (the one-instance caller-stream requirement).
Root-caused and fixed a load-time abort during development: the CUDA backend
static archive already carries a whole-archive link option, so also wrapping it
in an explicit whole-archive linked it twice and ran its static registration
twice, hitting a duplicate-registration check. The fix links it once.
Known follow-ups: a CUDA wheel CI job to exercise this automatically, and
running auditwheel on release so the shipped rpath is limited to $ORIGIN.
ghstack-source-id: 27c5585
ghstack-comment-id: 5124122149
Pull-Request: #21478
shoumikhin
added
ciflow/periodic
ciflow/trunk
ciflow/binaries
ciflow/binaries/all
shoumikhin
added a commit
that referenced
this pull request
Jul 29, 2026
## Why this is needed
The previous change ships a default Linux wheel with the core runtime as a
linkable libexecutorch.so, but no GPU delegate. To actually run a model on
an NVIDIA GPU (including a coalesced TensorRT + CUDA .pte), a C++ program
also needs the CUDA delegate as a loadable library.
This follows the way PyTorch distributes CUDA: the default wheel has the CPU
runtime, and a CUDA-specific wheel (built on a CUDA machine / installed from
a CUDA index) adds the GPU binaries. Here the CUDA-enabled ExecuTorch wheel
ships the CUDA delegate shared libraries alongside the same core runtime, so
a C++ consumer can link and load the CUDA backend with no source build.
Today the CUDA backend only exists as a static archive that is baked into
the Python module (_portable_lib.so). There is no standalone loadable
backend a C++ program can use, the way QNN already ships
libqnn_executorch_backend.so. This change adds that.
## What is inside
Added only when the wheel is built with CUDA (Linux):
- executorch/lib/libexecutorch_cuda_backend.so: a loadable CUDA delegate.
It wraps the existing static CUDA backend so that loading the library runs
its "CudaBackend" registration and registers into the one process-wide
runtime in libexecutorch.so. It is libtorch-free.
- executorch/lib/libextension_cuda.so: the CUDA caller-stream extension. It
is a single shared library on purpose so its per-thread caller-stream state
has exactly one instance across the whole process (needed so a TensorRT
delegate and the CUDA backend share one stream).
- executorch/include/executorch/extension/cuda/*.h: the caller-stream headers.
- CMake targets executorch::cuda_backend and executorch::extension_cuda in the
shipped executorch-config.cmake, defined only when the libraries are present.
The default (CPU) wheel is unchanged: none of these are added unless the
wheel is built with CUDA. Windows and macOS wheels are unchanged.
## How to use it
```bash
pip install executorch # from a CUDA-enabled index/build
```
```cmake
find_package(executorch CONFIG REQUIRED)
add_executable(my_runner main.cpp)
# Linking the CUDA backend force-loads it so it registers on startup.
target_link_libraries(my_runner PRIVATE
executorch::runtime
executorch::cuda_backend)
```
```bash
cmake -S . -B build \
-DCMAKE_PREFIX_PATH="$(python -c 'import executorch.utils as u; print(u.cmake_prefix_path)')"
cmake --build build
```
At runtime, get_backend_class("CudaBackend") returns the registered backend.
## Test plan
Verified on Linux x86_64 with an NVIDIA H100 and CUDA 12.8:
- Built the wheel with CUDA enabled and confirmed it contains
libexecutorch_cuda_backend.so and libextension_cuda.so under executorch/lib/,
the CUDA headers, and the CMake config.
- Installed the wheel into a clean virtual environment and built a standalone
C++ program via find_package(executorch) linking executorch::cuda_backend.
The program compiled, linked, ran, and get_backend_class("CudaBackend")
returned non-null, both when the backend was linked directly and when it was
dlopen'd.
- Confirmed it runs with no LD_LIBRARY_PATH: the shipped libraries resolve each
other through an $ORIGIN rpath because they are co-located in executorch/lib/.
- Confirmed the program and the CUDA backend link no libtorch/libc10 (via ldd).
- Confirmed there is a single libextension_cuda.so that both the backend and
the consumer resolve to (the one-instance caller-stream requirement).
Root-caused and fixed a load-time abort during development: the CUDA backend
static archive already carries a whole-archive link option, so also wrapping it
in an explicit whole-archive linked it twice and ran its static registration
twice, hitting a duplicate-registration check. The fix links it once.
Known follow-ups: a CUDA wheel CI job to exercise this automatically, and
running auditwheel on release so the shipped rpath is limited to $ORIGIN.
ghstack-source-id: 2098e23
ghstack-comment-id: 5124122149
Pull-Request: #21478
shoumikhin
added a commit
that referenced
this pull request
Jul 29, 2026
## Why this is needed
The previous change ships a default Linux wheel with the core runtime as a
linkable libexecutorch.so, but no GPU delegate. To actually run a model on
an NVIDIA GPU (including a coalesced TensorRT + CUDA .pte), a C++ program
also needs the CUDA delegate as a loadable library.
This follows the way PyTorch distributes CUDA: the default wheel has the CPU
runtime, and a CUDA-specific wheel (built on a CUDA machine / installed from
a CUDA index) adds the GPU binaries. Here the CUDA-enabled ExecuTorch wheel
ships the CUDA delegate shared libraries alongside the same core runtime, so
a C++ consumer can link and load the CUDA backend with no source build.
Today the CUDA backend only exists as a static archive that is baked into
the Python module (_portable_lib.so). There is no standalone loadable
backend a C++ program can use, the way QNN already ships
libqnn_executorch_backend.so. This change adds that.
## What is inside
Added only when the wheel is built with CUDA (Linux):
- executorch/lib/libexecutorch_cuda_backend.so: a loadable CUDA delegate.
It wraps the existing static CUDA backend so that loading the library runs
its "CudaBackend" registration and registers into the one process-wide
runtime in libexecutorch.so. It is libtorch-free.
- executorch/lib/libextension_cuda.so: the CUDA caller-stream extension. It
is a single shared library on purpose so its per-thread caller-stream state
has exactly one instance across the whole process (needed so a TensorRT
delegate and the CUDA backend share one stream).
- executorch/include/executorch/extension/cuda/*.h: the caller-stream headers.
- CMake targets executorch::cuda_backend and executorch::extension_cuda in the
shipped executorch-config.cmake, defined only when the libraries are present.
The default (CPU) wheel is unchanged: none of these are added unless the
wheel is built with CUDA. Windows and macOS wheels are unchanged.
## How to use it
```bash
pip install executorch # from a CUDA-enabled index/build
```
```cmake
find_package(executorch CONFIG REQUIRED)
add_executable(my_runner main.cpp)
# Linking the CUDA backend force-loads it so it registers on startup.
target_link_libraries(my_runner PRIVATE
executorch::runtime
executorch::cuda_backend)
```
```bash
cmake -S . -B build \
-DCMAKE_PREFIX_PATH="$(python -c 'import executorch.utils as u; print(u.cmake_prefix_path)')"
cmake --build build
```
At runtime, get_backend_class("CudaBackend") returns the registered backend.
## Test plan
Verified on Linux x86_64 with an NVIDIA H100 and CUDA 12.8:
- Built the wheel with CUDA enabled and confirmed it contains
libexecutorch_cuda_backend.so and libextension_cuda.so under executorch/lib/,
the CUDA headers, and the CMake config.
- Installed the wheel into a clean virtual environment and built a standalone
C++ program via find_package(executorch) linking executorch::cuda_backend.
The program compiled, linked, ran, and get_backend_class("CudaBackend")
returned non-null, both when the backend was linked directly and when it was
dlopen'd.
- Confirmed it runs with no LD_LIBRARY_PATH: the shipped libraries resolve each
other through an $ORIGIN rpath because they are co-located in executorch/lib/.
- Confirmed the program and the CUDA backend link no libtorch/libc10 (via ldd).
- Confirmed there is a single libextension_cuda.so that both the backend and
the consumer resolve to (the one-instance caller-stream requirement).
Root-caused and fixed a load-time abort during development: the CUDA backend
static archive already carries a whole-archive link option, so also wrapping it
in an explicit whole-archive linked it twice and ran its static registration
twice, hitting a duplicate-registration check. The fix links it once.
Known follow-ups: a CUDA wheel CI job to exercise this automatically, and
running auditwheel on release so the shipped rpath is limited to $ORIGIN.
ghstack-source-id: 2098e23
ghstack-comment-id: 5124122149
Pull-Request: #21478
shoumikhin
added a commit
that referenced
this pull request
Jul 30, 2026
## Why this is needed
The previous change ships a default Linux wheel with the core runtime as a
linkable libexecutorch.so, but no GPU delegate. To actually run a model on
an NVIDIA GPU (including a coalesced TensorRT + CUDA .pte), a C++ program
also needs the CUDA delegate as a loadable library.
This follows the way PyTorch distributes CUDA: the default wheel has the CPU
runtime, and a CUDA-specific wheel (built on a CUDA machine / installed from
a CUDA index) adds the GPU binaries. Here the CUDA-enabled ExecuTorch wheel
ships the CUDA delegate shared libraries alongside the same core runtime, so
a C++ consumer can link and load the CUDA backend with no source build.
Today the CUDA backend only exists as a static archive that is baked into
the Python module (_portable_lib.so). There is no standalone loadable
backend a C++ program can use, the way QNN already ships
libqnn_executorch_backend.so. This change adds that.
## What is inside
Added only when the wheel is built with CUDA (Linux):
- executorch/lib/libexecutorch_cuda_backend.so: a loadable CUDA delegate.
It wraps the existing static CUDA backend so that loading the library runs
its "CudaBackend" registration and registers into the one process-wide
runtime in libexecutorch.so. It is libtorch-free.
- executorch/lib/libextension_cuda.so: the CUDA caller-stream extension. It
is a single shared library on purpose so its per-thread caller-stream state
has exactly one instance across the whole process (needed so a TensorRT
delegate and the CUDA backend share one stream).
- executorch/include/executorch/extension/cuda/*.h: the caller-stream headers.
- CMake targets executorch::cuda_backend and executorch::extension_cuda in the
shipped executorch-config.cmake, defined only when the libraries are present.
The default (CPU) wheel is unchanged: none of these are added unless the
wheel is built with CUDA. Windows and macOS wheels are unchanged.
## How to use it
```bash
pip install executorch # from a CUDA-enabled index/build
```
```cmake
find_package(executorch CONFIG REQUIRED)
add_executable(my_runner main.cpp)
# Linking the CUDA backend force-loads it so it registers on startup.
target_link_libraries(my_runner PRIVATE
executorch::runtime
executorch::cuda_backend)
```
```bash
cmake -S . -B build \
-DCMAKE_PREFIX_PATH="$(python -c 'import executorch.utils as u; print(u.cmake_prefix_path)')"
cmake --build build
```
At runtime, get_backend_class("CudaBackend") returns the registered backend.
## Test plan
Verified on Linux x86_64 with an NVIDIA H100 and CUDA 12.8:
- Built the wheel with CUDA enabled and confirmed it contains
libexecutorch_cuda_backend.so and libextension_cuda.so under executorch/lib/,
the CUDA headers, and the CMake config.
- Installed the wheel into a clean virtual environment and built a standalone
C++ program via find_package(executorch) linking executorch::cuda_backend.
The program compiled, linked, ran, and get_backend_class("CudaBackend")
returned non-null, both when the backend was linked directly and when it was
dlopen'd.
- Confirmed it runs with no LD_LIBRARY_PATH: the shipped libraries resolve each
other through an $ORIGIN rpath because they are co-located in executorch/lib/.
- Confirmed the program and the CUDA backend link no libtorch/libc10 (via ldd).
- Confirmed there is a single libextension_cuda.so that both the backend and
the consumer resolve to (the one-instance caller-stream requirement).
Root-caused and fixed a load-time abort during development: the CUDA backend
static archive already carries a whole-archive link option, so also wrapping it
in an explicit whole-archive linked it twice and ran its static registration
twice, hitting a duplicate-registration check. The fix links it once.
Known follow-ups: a CUDA wheel CI job to exercise this automatically, and
running auditwheel on release so the shipped rpath is limited to $ORIGIN.
ghstack-source-id: dc8ddc4
ghstack-comment-id: 5124122149
Pull-Request: #21478
shoumikhin
added a commit
that referenced
this pull request
Jul 30, 2026
## Why this is needed
The previous change ships a default Linux wheel with the core runtime as a
linkable libexecutorch.so, but no GPU delegate. To actually run a model on
an NVIDIA GPU (including a coalesced TensorRT + CUDA .pte), a C++ program
also needs the CUDA delegate as a loadable library.
This follows the way PyTorch distributes CUDA: the default wheel has the CPU
runtime, and a CUDA-specific wheel (built on a CUDA machine / installed from
a CUDA index) adds the GPU binaries. Here the CUDA-enabled ExecuTorch wheel
ships the CUDA delegate shared libraries alongside the same core runtime, so
a C++ consumer can link and load the CUDA backend with no source build.
Today the CUDA backend only exists as a static archive that is baked into
the Python module (_portable_lib.so). There is no standalone loadable
backend a C++ program can use, the way QNN already ships
libqnn_executorch_backend.so. This change adds that.
## What is inside
Added only when the wheel is built with CUDA (Linux):
- executorch/lib/libexecutorch_cuda_backend.so: a loadable CUDA delegate.
It wraps the existing static CUDA backend so that loading the library runs
its "CudaBackend" registration and registers into the one process-wide
runtime in libexecutorch.so. It is libtorch-free.
- executorch/lib/libextension_cuda.so: the CUDA caller-stream extension. It
is a single shared library on purpose so its per-thread caller-stream state
has exactly one instance across the whole process (needed so a TensorRT
delegate and the CUDA backend share one stream).
- executorch/include/executorch/extension/cuda/*.h: the caller-stream headers.
- CMake targets executorch::cuda_backend and executorch::extension_cuda in the
shipped executorch-config.cmake, defined only when the libraries are present.
The default (CPU) wheel is unchanged: none of these are added unless the
wheel is built with CUDA. Windows and macOS wheels are unchanged.
## How to use it
```bash
pip install executorch # from a CUDA-enabled index/build
```
```cmake
find_package(executorch CONFIG REQUIRED)
add_executable(my_runner main.cpp)
# Linking the CUDA backend force-loads it so it registers on startup.
target_link_libraries(my_runner PRIVATE
executorch::runtime
executorch::cuda_backend)
```
```bash
cmake -S . -B build \
-DCMAKE_PREFIX_PATH="$(python -c 'import executorch.utils as u; print(u.cmake_prefix_path)')"
cmake --build build
```
At runtime, get_backend_class("CudaBackend") returns the registered backend.
## Test plan
Verified on Linux x86_64 with an NVIDIA H100 and CUDA 12.8:
- Built the wheel with CUDA enabled and confirmed it contains
libexecutorch_cuda_backend.so and libextension_cuda.so under executorch/lib/,
the CUDA headers, and the CMake config.
- Installed the wheel into a clean virtual environment and built a standalone
C++ program via find_package(executorch) linking executorch::cuda_backend.
The program compiled, linked, ran, and get_backend_class("CudaBackend")
returned non-null, both when the backend was linked directly and when it was
dlopen'd.
- Confirmed it runs with no LD_LIBRARY_PATH: the shipped libraries resolve each
other through an $ORIGIN rpath because they are co-located in executorch/lib/.
- Confirmed the program and the CUDA backend link no libtorch/libc10 (via ldd).
- Confirmed there is a single libextension_cuda.so that both the backend and
the consumer resolve to (the one-instance caller-stream requirement).
Root-caused and fixed a load-time abort during development: the CUDA backend
static archive already carries a whole-archive link option, so also wrapping it
in an explicit whole-archive linked it twice and ran its static registration
twice, hitting a duplicate-registration check. The fix links it once.
Known follow-ups: a CUDA wheel CI job to exercise this automatically, and
running auditwheel on release so the shipped rpath is limited to $ORIGIN.
ghstack-source-id: dd0a612
ghstack-comment-id: 5124122149
Pull-Request: #21478
shoumikhin
added a commit
that referenced
this pull request
Jul 30, 2026
## Why this is needed
The previous change ships a default Linux wheel with the core runtime as a
linkable libexecutorch.so, but no GPU delegate. To actually run a model on
an NVIDIA GPU (including a coalesced TensorRT + CUDA .pte), a C++ program
also needs the CUDA delegate as a loadable library.
This follows the way PyTorch distributes CUDA: the default wheel has the CPU
runtime, and a CUDA-specific wheel (built on a CUDA machine / installed from
a CUDA index) adds the GPU binaries. Here the CUDA-enabled ExecuTorch wheel
ships the CUDA delegate shared libraries alongside the same core runtime, so
a C++ consumer can link and load the CUDA backend with no source build.
Today the CUDA backend only exists as a static archive that is baked into
the Python module (_portable_lib.so). There is no standalone loadable
backend a C++ program can use, the way QNN already ships
libqnn_executorch_backend.so. This change adds that.
## What is inside
Added only when the wheel is built with CUDA (Linux):
- executorch/lib/libexecutorch_cuda_backend.so: a loadable CUDA delegate.
It wraps the existing static CUDA backend so that loading the library runs
its "CudaBackend" registration and registers into the one process-wide
runtime in libexecutorch.so. It is libtorch-free.
- executorch/lib/libextension_cuda.so: the CUDA caller-stream extension. It
is a single shared library on purpose so its per-thread caller-stream state
has exactly one instance across the whole process (needed so a TensorRT
delegate and the CUDA backend share one stream).
- executorch/include/executorch/extension/cuda/*.h: the caller-stream headers.
- CMake targets executorch::cuda_backend and executorch::extension_cuda in the
shipped executorch-config.cmake, defined only when the libraries are present.
The default (CPU) wheel is unchanged: none of these are added unless the
wheel is built with CUDA. Windows and macOS wheels are unchanged.
## How to use it
```bash
pip install executorch # from a CUDA-enabled index/build
```
```cmake
find_package(executorch CONFIG REQUIRED)
add_executable(my_runner main.cpp)
# Linking the CUDA backend force-loads it so it registers on startup.
target_link_libraries(my_runner PRIVATE
executorch::runtime
executorch::cuda_backend)
```
```bash
cmake -S . -B build \
-DCMAKE_PREFIX_PATH="$(python -c 'import executorch.utils as u; print(u.cmake_prefix_path)')"
cmake --build build
```
At runtime, get_backend_class("CudaBackend") returns the registered backend.
## Test plan
Verified on Linux x86_64 with an NVIDIA H100 and CUDA 12.8:
- Built the wheel with CUDA enabled and confirmed it contains
libexecutorch_cuda_backend.so and libextension_cuda.so under executorch/lib/,
the CUDA headers, and the CMake config.
- Installed the wheel into a clean virtual environment and built a standalone
C++ program via find_package(executorch) linking executorch::cuda_backend.
The program compiled, linked, ran, and get_backend_class("CudaBackend")
returned non-null, both when the backend was linked directly and when it was
dlopen'd.
- Confirmed it runs with no LD_LIBRARY_PATH: the shipped libraries resolve each
other through an $ORIGIN rpath because they are co-located in executorch/lib/.
- Confirmed the program and the CUDA backend link no libtorch/libc10 (via ldd).
- Confirmed there is a single libextension_cuda.so that both the backend and
the consumer resolve to (the one-instance caller-stream requirement).
Root-caused and fixed a load-time abort during development: the CUDA backend
static archive already carries a whole-archive link option, so also wrapping it
in an explicit whole-archive linked it twice and ran its static registration
twice, hitting a duplicate-registration check. The fix links it once.
Known follow-ups: a CUDA wheel CI job to exercise this automatically, and
running auditwheel on release so the shipped rpath is limited to $ORIGIN.
ghstack-source-id: 5946067
ghstack-comment-id: 5124122149
Pull-Request: #21478
shoumikhin
added a commit
that referenced
this pull request
Jul 30, 2026
The previous change ships a default Linux wheel with the core runtime as a
linkable libexecutorch.so, but no GPU delegate. To actually run a model on
an NVIDIA GPU (including a coalesced TensorRT + CUDA .pte), a C++ program
also needs the CUDA delegate as a loadable library.
This follows the way PyTorch distributes CUDA: the default wheel has the CPU
runtime, and a CUDA-specific wheel (built on a CUDA machine / installed from
a CUDA index) adds the GPU binaries. Here the CUDA-enabled ExecuTorch wheel
ships the CUDA delegate shared libraries alongside the same core runtime, so
a C++ consumer can link and load the CUDA backend with no source build.
Today the CUDA backend only exists as a static archive that is baked into
the Python module (_portable_lib.so). There is no standalone loadable
backend a C++ program can use, the way QNN already ships
libqnn_executorch_backend.so. This change adds that.
Added only when the wheel is built with CUDA (Linux):
- executorch/lib/libexecutorch_cuda_backend.so: a loadable CUDA delegate.
It wraps the existing static CUDA backend so that loading the library runs
its "CudaBackend" registration and registers into the one process-wide
runtime in libexecutorch.so. It is libtorch-free.
- executorch/lib/libextension_cuda.so: the CUDA caller-stream extension. It
is a single shared library on purpose so its per-thread caller-stream state
has exactly one instance across the whole process (needed so a TensorRT
delegate and the CUDA backend share one stream).
- executorch/include/executorch/extension/cuda/*.h: the caller-stream headers.
- CMake targets executorch::cuda_backend and executorch::extension_cuda in the
shipped executorch-config.cmake, defined only when the libraries are present.
The default (CPU) wheel is unchanged: none of these are added unless the
wheel is built with CUDA. Windows and macOS wheels are unchanged.
```bash
pip install executorch # from a CUDA-enabled index/build
```
```cmake
find_package(executorch CONFIG REQUIRED)
add_executable(my_runner main.cpp)
target_link_libraries(my_runner PRIVATE
executorch::runtime
executorch::cuda_backend)
```
```bash
cmake -S . -B build \
-DCMAKE_PREFIX_PATH="$(python -c 'import executorch.utils as u; print(u.cmake_prefix_path)')"
cmake --build build
```
At runtime, get_backend_class("CudaBackend") returns the registered backend.
Verified on Linux x86_64 with an NVIDIA H100 and CUDA 12.8:
- Built the wheel with CUDA enabled and confirmed it contains
libexecutorch_cuda_backend.so and libextension_cuda.so under executorch/lib/,
the CUDA headers, and the CMake config.
- Installed the wheel into a clean virtual environment and built a standalone
C++ program via find_package(executorch) linking executorch::cuda_backend.
The program compiled, linked, ran, and get_backend_class("CudaBackend")
returned non-null, both when the backend was linked directly and when it was
dlopen'd.
- Confirmed it runs with no LD_LIBRARY_PATH: the shipped libraries resolve each
other through an $ORIGIN rpath because they are co-located in executorch/lib/.
- Confirmed the program and the CUDA backend link no libtorch/libc10 (via ldd).
- Confirmed there is a single libextension_cuda.so that both the backend and
the consumer resolve to (the one-instance caller-stream requirement).
Root-caused and fixed a load-time abort during development: the CUDA backend
static archive already carries a whole-archive link option, so also wrapping it
in an explicit whole-archive linked it twice and ran its static registration
twice, hitting a duplicate-registration check. The fix links it once.
Known follow-ups: a CUDA wheel CI job to exercise this automatically, and
running auditwheel on release so the shipped rpath is limited to $ORIGIN.
ghstack-source-id: eabaea7
ghstack-comment-id: 5124122149
Pull-Request: #21478
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a loadable CUDA delegate shared library to CUDA-enabled Linux ExecuTorch wheels so that standalone C++ consumers can link/load the CUDA backend (and related CUDA stream extension) without building from source, mirroring how other delegates (e.g., QNN) are distributed.
Changes:
- Adds an “empty” translation unit to allow building a wrapper shared library that bundles the existing static CUDA backend for load-time registration.
- Introduces a new
executorch_cuda_backendshared library target inbackends/cuda/(gated onEXECUTORCH_BUILD_SHAREDand non-MSVC), with wheel-relocatable$ORIGIN-based RPATH and link options intended to prevent--as-neededfrom dropping the library. - Adds a new PR CI job to build a CUDA-enabled wheel on a GPU runner and validate that
libexecutorch_cuda_backend.soships and successfully registersCudaBackendat runtime.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| backends/cuda/runtime/executorch_cuda_backend_lib.cpp | Adds a placeholder TU to satisfy CMake for a shared-library wrapper used to retain static registration. |
| backends/cuda/CMakeLists.txt | Builds/installs libexecutorch_cuda_backend.so for shared builds, sets wheel-relocatable RPATH, and adds consumer link options to force retention under --as-needed. |
| .github/workflows/pull.yml | Adds a GPU CI job that builds a CUDA wheel and asserts the shipped CUDA backend library is present and loadable. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
shoumikhin
added a commit
that referenced
this pull request
Jul 30, 2026
The previous change ships a default Linux wheel with the core runtime as a
linkable libexecutorch.so, but no GPU delegate. To actually run a model on
an NVIDIA GPU (including a coalesced TensorRT + CUDA .pte), a C++ program
also needs the CUDA delegate as a loadable library.
This follows the way PyTorch distributes CUDA: the default wheel has the CPU
runtime, and a CUDA-specific wheel (built on a CUDA machine / installed from
a CUDA index) adds the GPU binaries. Here the CUDA-enabled ExecuTorch wheel
ships the CUDA delegate shared libraries alongside the same core runtime, so
a C++ consumer can link and load the CUDA backend with no source build.
Today the CUDA backend only exists as a static archive that is baked into
the Python module (_portable_lib.so). There is no standalone loadable
backend a C++ program can use, the way QNN already ships
libqnn_executorch_backend.so. This change adds that.
Added only when the wheel is built with CUDA (Linux):
- executorch/lib/libexecutorch_cuda_backend.so: a loadable CUDA delegate.
It wraps the existing static CUDA backend so that loading the library runs
its "CudaBackend" registration and registers into the one process-wide
runtime in libexecutorch.so. It is libtorch-free.
- executorch/lib/libextension_cuda.so: the CUDA caller-stream extension. It
is a single shared library on purpose so its per-thread caller-stream state
has exactly one instance across the whole process (needed so a TensorRT
delegate and the CUDA backend share one stream).
- executorch/include/executorch/extension/cuda/*.h: the caller-stream headers.
- CMake targets executorch::cuda_backend and executorch::extension_cuda in the
shipped executorch-config.cmake, defined only when the libraries are present.
The default (CPU) wheel is unchanged: none of these are added unless the
wheel is built with CUDA. Windows and macOS wheels are unchanged.
```bash
pip install executorch # from a CUDA-enabled index/build
```
```cmake
find_package(executorch CONFIG REQUIRED)
add_executable(my_runner main.cpp)
target_link_libraries(my_runner PRIVATE
executorch::runtime
executorch::cuda_backend)
```
```bash
cmake -S . -B build \
-DCMAKE_PREFIX_PATH="$(python -c 'import executorch.utils as u; print(u.cmake_prefix_path)')"
cmake --build build
```
At runtime, get_backend_class("CudaBackend") returns the registered backend.
Verified on Linux x86_64 with an NVIDIA H100 and CUDA 12.8:
- Built the wheel with CUDA enabled and confirmed it contains
libexecutorch_cuda_backend.so and libextension_cuda.so under executorch/lib/,
the CUDA headers, and the CMake config.
- Installed the wheel into a clean virtual environment and built a standalone
C++ program via find_package(executorch) linking executorch::cuda_backend.
The program compiled, linked, ran, and get_backend_class("CudaBackend")
returned non-null, both when the backend was linked directly and when it was
dlopen'd.
- Confirmed it runs with no LD_LIBRARY_PATH: the shipped libraries resolve each
other through an $ORIGIN rpath because they are co-located in executorch/lib/.
- Confirmed the program and the CUDA backend link no libtorch/libc10 (via ldd).
- Confirmed there is a single libextension_cuda.so that both the backend and
the consumer resolve to (the one-instance caller-stream requirement).
Root-caused and fixed a load-time abort during development: the CUDA backend
static archive already carries a whole-archive link option, so also wrapping it
in an explicit whole-archive linked it twice and ran its static registration
twice, hitting a duplicate-registration check. The fix links it once.
Known follow-ups: a CUDA wheel CI job to exercise this automatically, and
running auditwheel on release so the shipped rpath is limited to $ORIGIN.
ghstack-source-id: fb189a3
ghstack-comment-id: 5124122149
Pull-Request: #21478
shoumikhin
added a commit
that referenced
this pull request
Jul 30, 2026
The previous change ships a default Linux wheel with the core runtime as a
linkable libexecutorch.so, but no GPU delegate. To actually run a model on
an NVIDIA GPU (including a coalesced TensorRT + CUDA .pte), a C++ program
also needs the CUDA delegate as a loadable library.
This follows the way PyTorch distributes CUDA: the default wheel has the CPU
runtime, and a CUDA-specific wheel (built on a CUDA machine / installed from
a CUDA index) adds the GPU binaries. Here the CUDA-enabled ExecuTorch wheel
ships the CUDA delegate shared libraries alongside the same core runtime, so
a C++ consumer can link and load the CUDA backend with no source build.
Today the CUDA backend only exists as a static archive that is baked into
the Python module (_portable_lib.so). There is no standalone loadable
backend a C++ program can use, the way QNN already ships
libqnn_executorch_backend.so. This change adds that.
Added only when the wheel is built with CUDA (Linux):
- executorch/lib/libexecutorch_cuda_backend.so: a loadable CUDA delegate.
It wraps the existing static CUDA backend so that loading the library runs
its "CudaBackend" registration and registers into the one process-wide
runtime in libexecutorch.so. It is libtorch-free.
- executorch/lib/libextension_cuda.so: the CUDA caller-stream extension. It
is a single shared library on purpose so its per-thread caller-stream state
has exactly one instance across the whole process (needed so a TensorRT
delegate and the CUDA backend share one stream).
- executorch/include/executorch/extension/cuda/*.h: the caller-stream headers.
- CMake targets executorch::cuda_backend and executorch::extension_cuda in the
shipped executorch-config.cmake, defined only when the libraries are present.
The default (CPU) wheel is unchanged: none of these are added unless the
wheel is built with CUDA. Windows and macOS wheels are unchanged.
```bash
pip install executorch # from a CUDA-enabled index/build
```
```cmake
find_package(executorch CONFIG REQUIRED)
add_executable(my_runner main.cpp)
target_link_libraries(my_runner PRIVATE
executorch::runtime
executorch::cuda_backend)
```
```bash
cmake -S . -B build \
-DCMAKE_PREFIX_PATH="$(python -c 'import executorch.utils as u; print(u.cmake_prefix_path)')"
cmake --build build
```
At runtime, get_backend_class("CudaBackend") returns the registered backend.
Verified on Linux x86_64 with an NVIDIA H100 and CUDA 12.8:
- Built the wheel with CUDA enabled and confirmed it contains
libexecutorch_cuda_backend.so and libextension_cuda.so under executorch/lib/,
the CUDA headers, and the CMake config.
- Installed the wheel into a clean virtual environment and built a standalone
C++ program via find_package(executorch) linking executorch::cuda_backend.
The program compiled, linked, ran, and get_backend_class("CudaBackend")
returned non-null, both when the backend was linked directly and when it was
dlopen'd.
- Confirmed it runs with no LD_LIBRARY_PATH: the shipped libraries resolve each
other through an $ORIGIN rpath because they are co-located in executorch/lib/.
- Confirmed the program and the CUDA backend link no libtorch/libc10 (via ldd).
- Confirmed there is a single libextension_cuda.so that both the backend and
the consumer resolve to (the one-instance caller-stream requirement).
Root-caused and fixed a load-time abort during development: the CUDA backend
static archive already carries a whole-archive link option, so also wrapping it
in an explicit whole-archive linked it twice and ran its static registration
twice, hitting a duplicate-registration check. The fix links it once.
Known follow-ups: a CUDA wheel CI job to exercise this automatically, and
running auditwheel on release so the shipped rpath is limited to $ORIGIN.
ghstack-source-id: 4bb0dba
ghstack-comment-id: 5124122149
Pull-Request: #21478
shoumikhin
added a commit
that referenced
this pull request
Jul 30, 2026
The previous change ships a default Linux wheel with the core runtime as a
linkable libexecutorch.so, but no GPU delegate. To actually run a model on
an NVIDIA GPU (including a coalesced TensorRT + CUDA .pte), a C++ program
also needs the CUDA delegate as a loadable library.
This follows the way PyTorch distributes CUDA: the default wheel has the CPU
runtime, and a CUDA-specific wheel (built on a CUDA machine / installed from
a CUDA index) adds the GPU binaries. Here the CUDA-enabled ExecuTorch wheel
ships the CUDA delegate shared libraries alongside the same core runtime, so
a C++ consumer can link and load the CUDA backend with no source build.
Today the CUDA backend only exists as a static archive that is baked into
the Python module (_portable_lib.so). There is no standalone loadable
backend a C++ program can use, the way QNN already ships
libqnn_executorch_backend.so. This change adds that.
Added only when the wheel is built with CUDA (Linux):
- executorch/lib/libexecutorch_cuda_backend.so: a loadable CUDA delegate.
It wraps the existing static CUDA backend so that loading the library runs
its "CudaBackend" registration and registers into the one process-wide
runtime in libexecutorch.so. It is libtorch-free.
- executorch/lib/libextension_cuda.so: the CUDA caller-stream extension. It
is a single shared library on purpose so its per-thread caller-stream state
has exactly one instance across the whole process (needed so a TensorRT
delegate and the CUDA backend share one stream).
- executorch/include/executorch/extension/cuda/*.h: the caller-stream headers.
- CMake targets executorch::cuda_backend and executorch::extension_cuda in the
shipped executorch-config.cmake, defined only when the libraries are present.
The default (CPU) wheel is unchanged: none of these are added unless the
wheel is built with CUDA. Windows and macOS wheels are unchanged.
```bash
pip install executorch # from a CUDA-enabled index/build
```
```cmake
find_package(executorch CONFIG REQUIRED)
add_executable(my_runner main.cpp)
target_link_libraries(my_runner PRIVATE
executorch::runtime
executorch::cuda_backend)
```
```bash
cmake -S . -B build \
-DCMAKE_PREFIX_PATH="$(python -c 'import executorch.utils as u; print(u.cmake_prefix_path)')"
cmake --build build
```
At runtime, get_backend_class("CudaBackend") returns the registered backend.
Verified on Linux x86_64 with an NVIDIA H100 and CUDA 12.8:
- Built the wheel with CUDA enabled and confirmed it contains
libexecutorch_cuda_backend.so and libextension_cuda.so under executorch/lib/,
the CUDA headers, and the CMake config.
- Installed the wheel into a clean virtual environment and built a standalone
C++ program via find_package(executorch) linking executorch::cuda_backend.
The program compiled, linked, ran, and get_backend_class("CudaBackend")
returned non-null, both when the backend was linked directly and when it was
dlopen'd.
- Confirmed it runs with no LD_LIBRARY_PATH: the shipped libraries resolve each
other through an $ORIGIN rpath because they are co-located in executorch/lib/.
- Confirmed the program and the CUDA backend link no libtorch/libc10 (via ldd).
- Confirmed there is a single libextension_cuda.so that both the backend and
the consumer resolve to (the one-instance caller-stream requirement).
Root-caused and fixed a load-time abort during development: the CUDA backend
static archive already carries a whole-archive link option, so also wrapping it
in an explicit whole-archive linked it twice and ran its static registration
twice, hitting a duplicate-registration check. The fix links it once.
Known follow-ups: a CUDA wheel CI job to exercise this automatically, and
running auditwheel on release so the shipped rpath is limited to $ORIGIN.
ghstack-source-id: ad3687f
ghstack-comment-id: 5124122149
Pull-Request: #21478
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this is needed
The previous change ships a default Linux wheel with the core runtime as a
linkable libexecutorch.so, but no GPU delegate. To actually run a model on
an NVIDIA GPU (including a coalesced TensorRT + CUDA .pte), a C++ program
also needs the CUDA delegate as a loadable library.
This follows the way PyTorch distributes CUDA: the default wheel has the CPU
runtime, and a CUDA-specific wheel (built on a CUDA machine / installed from
a CUDA index) adds the GPU binaries. Here the CUDA-enabled ExecuTorch wheel
ships the CUDA delegate shared libraries alongside the same core runtime, so
a C++ consumer can link and load the CUDA backend with no source build.
Today the CUDA backend only exists as a static archive that is baked into
the Python module (_portable_lib.so). There is no standalone loadable
backend a C++ program can use, the way QNN already ships
libqnn_executorch_backend.so. This change adds that.
What is inside
Added only when the wheel is built with CUDA (Linux):
It wraps the existing static CUDA backend so that loading the library runs
its "CudaBackend" registration and registers into the one process-wide
runtime in libexecutorch.so. It is libtorch-free.
is a single shared library on purpose so its per-thread caller-stream state
has exactly one instance across the whole process (needed so a TensorRT
delegate and the CUDA backend share one stream).
shipped executorch-config.cmake, defined only when the libraries are present.
The default (CPU) wheel is unchanged: none of these are added unless the
wheel is built with CUDA. Windows and macOS wheels are unchanged.
How to use it
pip install executorch # from a CUDA-enabled index/buildAt runtime, get_backend_class("CudaBackend") returns the registered backend.
Test plan
Verified on Linux x86_64 with an NVIDIA H100 and CUDA 12.8:
libexecutorch_cuda_backend.so and libextension_cuda.so under executorch/lib/,
the CUDA headers, and the CMake config.
C++ program via find_package(executorch) linking executorch::cuda_backend.
The program compiled, linked, ran, and get_backend_class("CudaBackend")
returned non-null, both when the backend was linked directly and when it was
dlopen'd.
other through an $ORIGIN rpath because they are co-located in executorch/lib/.
the consumer resolve to (the one-instance caller-stream requirement).
Root-caused and fixed a load-time abort during development: the CUDA backend
static archive already carries a whole-archive link option, so also wrapping it
in an explicit whole-archive linked it twice and ran its static registration
twice, hitting a duplicate-registration check. The fix links it once.
Known follow-ups: a CUDA wheel CI job to exercise this automatically, and
running auditwheel on release so the shipped rpath is limited to $ORIGIN.