[ET Device Support] Module: allocate device memory for planned buffers

[Gasoonjia]

Stack from ghstack (oldest at bottom):

• [ET Device Support] CUDA-native Qwen 3.5 MoE inference with device tensor pipeline #18788
• [ET Device Support] Extract shared device test utilities to reduce redundancy #18762
• [ET Device Support] Add device tensor helper functions to TensorPtr API #18761
• [ET Device Support] Add ExecutorchBackendConfig flags for skipping H2D/D2H copies #18760
• [ET Device Support] PropagateDevicePass inserts H2D/D2H copy ops at delegate boundaries #18730
• [ET Device Support] Define et_copy runtime h2d and d2h copy ops #18729
• [ET Device Support] Define AOT device copy ops registry #18728
• [ET Device Support] CudaAllocator: device memory allocator for CUDA backend #18477
• -> [ET Device Support] Module: allocate device memory for planned buffers #18476
• [ET Device Support] MemoryManager: add per-buffer device metadata #18475
• [ET Device Support] MethodMeta: expose per-buffer device placement API #18474
• [ET Device Support] DeviceMemoryBuffer RAII class for device memory lifetime management #18473
• [ET Device Support] Emitter reads non_const_buffer_device from graph meta #18472
• [ET Device Support] Device-aware memory planning: separate buffers per device type #18375
• [ET Device Support] Add NonConstBufferDevice schema for per-buffer device mapping #18330
• [ET Device Support] DeviceAllocator interface and DeviceAllocatorRegistry #17535
• [ET Device Support] Annotate device attributes of CUDA backend IO tensors cuda device #18080
• [ET Device Support] Parse device info from serialized tensor in tensor_parser #18328
• [ET Device Support] Propagate device info from TensorSpec into serialized Tensor #18079
• [executorch] Propagate device metadata from partitioner result onto TensorSpecs #18078

This diff enables module API loading program memory-planed on non-cpu device. It update Module::load_method() to detect device buffers via MethodMeta and allocate device memory using the registered DeviceAllocator. Device memory is managed via DeviceMemoryBuffer RAII objects stored in PlannedMemory, ensuring proper cleanup when the Method is destroyed.

Differential Revision: D97850705

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18476

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 67 New Failures

As of commit d0b1424 with merge base b5ae0b9 ():

NEW FAILURES - The following jobs have failed:

• Apple / build-benchmark-app / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• Apple / build-demo-ios / macos-job (gh)
• Apple / build-frameworks-ios / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• Build Presets / apple (pybind) / build (gh)
• Build Presets / linux (pybind, linux.2xlarge, executorch-ubuntu-22.04-clang12) / build (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• Build Presets / linux (pybind, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• Build Presets / windows (pybind) / build (gh)
• periodic / test-models-linux (buck2, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
  RuntimeError: Command docker exec -t f5386502155d2e4f07494062c12994573d8dde825659bbaab97c8906b332c070 /exec failed with exit code 1
• periodic / test-models-linux (cmake, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
  RuntimeError: Command docker exec -t 3a9b7b1c64353a9a0f1a5c0aa1ba793c6e61b6ac055c01a8d740785bfe728ad0 /exec failed with exit code 1
• periodic / test-models-linux (cmake, vit, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
  RuntimeError: Command docker exec -t 6a756ab9b2ddcc8d4915323151a97a567cee3276eba98008a659a421807dbc47 /exec failed with exit code 1
• pull / android / build-android (gh)
  Process completed with exit code 2.
• pull / test-coreml-bc-macos (macos-m1-stable) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / test-coreml-bc-macos (macos-m2-stable) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.arm64.2xlarge, executorch-ubuntu-22.04-gc... / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04... / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu... / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu-22.04-... / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux-android / linux-job (gh)
  RuntimeError: Command docker exec -t b77baa8ebc30ca3f400459da7632e0036aa4e9a6145735f9277560e4de7c404e /exec failed with exit code 2
• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• pull / test-lora-linux / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-lora-multimethod-linux / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-models-linux (add_mul, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 75250620782b151612ed9e1ae6713e96c5d3dd60eb037fd7ba7c33e92b4d0b38 /exec failed with exit code 1
• pull / test-models-linux (add, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t bd3ae41bb3ee33c9edc9fc053fe4ae81fb0c294d7bac0e728ca34aa0bda7e276 /exec failed with exit code 1
• pull / test-models-linux (emformer_join, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t 5992ec6292ebdb1ce6c33e0da920e6663acf7d4fe5af6a1b0e2b582a8e849a97 /exec failed with exit code 1
• pull / test-models-linux (emformer_transcribe, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 3e0405e3234b4842a2231669555269f20cb1166f74bd745a3c5f1016ce1b3b76 /exec failed with exit code 1
• pull / test-models-linux (ic3, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t fbb34a2a32d9542b6cf9c35ae4ea5ea7bf96dc3f84e5d74faf902c93f31feb79 /exec failed with exit code 1
• pull / test-models-linux (ic4, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t c6304c4d41f354a092aa1398221351ec18dd82f67090d5e597f404932eaaef07 /exec failed with exit code 1
• pull / test-models-linux (linear, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 72a48061991b8c3288747e8aafea9eebfa25c606b76ef26c315fd074520398fd /exec failed with exit code 1
• pull / test-models-linux (mobilebert, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d4c2477a94b49e6038d414923e4250e3fd7707fbf8578936e2b1f62530e2e522 /exec failed with exit code 1
• pull / test-models-linux (mv2, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t f91879dcf83d7dac9d5a48db70685619db3b41f12f150b50f3fd1e59db8f7573 /exec failed with exit code 1
• pull / test-models-linux (resnet18, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t f508e865b7b7f2e70389f4d36079fa64a0631ca09c0c77180136aeed7faa08b5 /exec failed with exit code 1
• pull / test-models-linux (resnet50, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 539ee8bbdba12ab886a751837187b9a484cc26f5b5bb769604c9fb73f059e8b5 /exec failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
  RuntimeError: Command docker exec -t 3869a9daacb6ee49e57de1e04d0df2879d742b2d7ea953cb3d30aabe1397d64d /exec failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
  RuntimeError: Command docker exec -t a1ddd84c6ddc9a293b9aa3fb39d9a0737c0720108114b4a6c75dd5b48b7820e2 /exec failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
  RuntimeError: Command docker exec -t 601a1f9bc74c7f1db2aa2d98e5a7d72268c307db39d2f67f402148f853a1fa25 /exec failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
  RuntimeError: Command docker exec -t d639190b9db3e955be9b442874bd98db219bdf4992cc2dd284ece09f616de366 /exec failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
  RuntimeError: Command docker exec -t 368da9e3a0566c0d0bc66c315dcd2217269e721dcaf00285e6fdc304bc082a3a /exec failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
  RuntimeError: Command docker exec -t 3c817c1f763f1351cf060a51f1229b6d9f7071d50eea5bb195d226a2fce0b57f /exec failed with exit code 1
• pull / test-moshi-linux / linux-job (gh)
  test_mimi
• pull / test-multimodal-linux (gemma3-4b) / linux-job (gh)
  RuntimeError: Command docker exec -t 593f4b9eb4872eff63acee88cf2b81c0fc840bafb95d048ea64f36919d25229c /exec failed with exit code 1
• pull / test-openvino-linux / linux-job (gh)
  RuntimeError: Command docker exec -t df46a5c644410e2c7904a6a621c000916b0c613f7059a5bcd00607ad0a2aae6b /exec failed with exit code 1
• pull / test-parakeet-xnnpack-linux / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• pull / test-phi-3-mini-runner-linux / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• pull / test-sqnr-static-llm-qnn-linux (smollm2_135m) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• pull / test-static-llama-qnn-linux (stories_110m) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• pull / test-static-llama-qnn-linux (stories_260k_bc) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• pull / test-voxtral-realtime-xnnpack-linux / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-vulkan-models-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 4ebfe4f52cc756cadb9716c911ef1a1959324d944ca477f2fb1f7131dbfe016f /exec failed with exit code 1
• pull / test-vulkan-operators-linux / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• pull / unittest / linux / linux-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest / macos / macos-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest / windows / windows-job (gh)
  Process completed with exit code 1.
• pull / unittest-arm-backend-with-no-deps (test_pytest_models_tosa) / linux-job (gh)
  RuntimeError: Command docker exec -t 78fffe250f7b17a03672a034af5a5036ecf6cf0b91748e44cd98dbe7783b527c /exec failed with exit code 1
• pull / unittest-arm-backend-with-no-deps (test_pytest_ops_no_target) / linux-job (gh)
  RuntimeError: Command docker exec -t 773c84881082b41442335309def99f04caf78753e713492e0869d122c6f97166 /exec failed with exit code 1
• pull / unittest-arm-backend-with-no-deps (test_pytest_ops_tosa) / linux-job (gh)
  RuntimeError: Command docker exec -t c188013e2725b09468f3374c5b1b49cb348e3e60c2991ab3d9e4a738577468fd /exec failed with exit code 1
• pull / unittest-editable / linux / linux-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest-editable / macos / macos-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest-editable / windows / windows-job (gh)
  Process completed with exit code 1.
• pull / unittest-nxp-neutron / linux-job (gh)
  RuntimeError: Command docker exec -t b36e3322e1c4676be33c9a297949fb8ab85905cdd209332c143618fe855ce476 /exec failed with exit code 2
• pull / unittest-wasm-bindings (--enable-etdump) / linux-job (gh)
  RuntimeError: Command docker exec -t 9863273dbeeb8278ff8ee342f60de311ca8b35a341bfe14bc2b0a7558319872b /exec failed with exit code 2
• pull / unittest-wasm-bindings / linux-job (gh)
  RuntimeError: Command docker exec -t 13350fb930098f80679970dc846e6a4a8f35b856d05ab46b00dc897bb3e54349 /exec failed with exit code 2
• Test QNN Backend / test-qnn / test-backend-linux (qnn, models) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• Test QNN Backend / test-qnn / test-backend-linux (qnn, operators) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.