Skip to content

Integrate vendor VFIO vGPUs into the instance lifecycle - #321

Open
yummybomb wants to merge 39 commits into
hypeship/vendor-vfio-backendfrom
hypeship/vendor-vfio-vgpu
Open

Integrate vendor VFIO vGPUs into the instance lifecycle#321
yummybomb wants to merge 39 commits into
hypeship/vendor-vfio-backendfrom
hypeship/vendor-vfio-vgpu

Conversation

@yummybomb

@yummybomb yummybomb commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Top layer of the vendor VFIO vGPU stack (generalize-vgpu-device #322hypervisor-liveness #363vendor-vfio-backend #364this). The backend itself is in #364; this layer wires it into the instance lifecycle and adds the release guards that make reusable-VF assignments safe:

  • Live-instance claim guard on release — an assignment goes stale when its release succeeds but the metadata save does not (or start fails between the release and its first save). Since vendor VFIO reuses the same VF path across assignments, honoring a stale claim could clear a VF out from under the live instance that owns it now. Every release consults live instance metadata (liveness verified via Harden hypervisor process liveness checks #363's socket-ownership check) and drops stale metadata without touching the device.
  • Assignment durability — assignments are tagged with the owning instance ID, persisted before booting a started instance without carrying over the previous VMM identity, and retained in metadata when rollback release fails in create/start so later release paths can still find the device.
  • Fail-closed startup reconciliation — the protected set of live VFs is built from ListInstancesForReconcile, which fails on any unreadable metadata instead of silently skipping it. Recent assignments receive bounded startup protection when their PID is absent or stale; when the inventory is unavailable, vendor VFIO reconciliation is skipped entirely while mdev reconciliation still runs.
  • Hypervisor scope — hypervisor selection remains a caller policy, preserving the framework layer's existing mdev behavior. The production rollout continues to place vGPU instances on QEMU; the Cloud Hypervisor vendor-VFIO limitation (upstream cloud-hypervisor#7572) is documented in lib/devices/GPU.md.
  • Failure reporting — a failed create or start whose vGPU release also failed surfaces as vgpu_cleanup_pending with retained/unretained guidance; the retention record keeps identity fields (name, image, hypervisor) so it lists as a recognizable, deletable instance. The instance API exposes gpu.device_path, the assignment identity on vendor VFIO hosts where mdev_uuid is empty.

Testing

  • go build ./..., go vet ./... clean
  • go test -race passes for lib/instances targeted suites, lib/devices, lib/resources, lib/builds (TestCreateInstanceWithNetwork needs image pulls + iptables and fails in this environment on the unmodified base as well)

Note

High Risk
Touches GPU assignment persistence, release, and startup reconciliation on shared VF paths; incorrect claim or reconcile logic could detach live workloads or leak assignments.

Overview
Wires vendor VFIO vGPUs into create/start/delete and enables devices.CreateVGPU for that framework (integration tests no longer skip vendor VFIO).

Lifecycle and safety: Assignments persist GPUDevicePath, GPUAssignedAt, and optional delete-only retention stubs (GPURetainedForCleanup) when rollback cannot release a VF. Failed create/start rollbacks surface vgpu_cleanup_pending on the API (checked before wrapped errors.Is mappings). Vendor VFIO release scans other instances’ metadata and fails closed when a live claimant or ambiguous inventory exists; mdev skips that scan. Start clears stale hypervisor identity before assigning a new vGPU; fork/snapshot/start reject retention stubs.

Startup: Mdev-only reconcile is replaced by reconcileVGPUs, which builds a protected VF set via ListInstancesForReconcile (strict metadata reads), live hypervisor checks, and a bounded grace for recent assignments, then calls devices.ReconcileVGPUs with optional delayed retry.

API: Instance GPU responses add optional device_path alongside mdev_uuid.

Reviewed by Cursor Bugbot for commit 6afdcbb. Bugbot is set up for automated code reviews on this repo. Configure here.

@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 199b1d1 to a93d009 Compare July 28, 2026 21:32
@yummybomb
yummybomb changed the base branch from main to hypeship/generalize-vgpu-device July 28, 2026 21:32
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch 4 times, most recently from d893fce to de4742a Compare July 29, 2026 15:15
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from de4742a to 18e047f Compare July 29, 2026 16:09
@yummybomb

yummybomb commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Trimmed over-engineering after an architectural review against the parent task (host kernel 6.8 / Ubuntu 24.04 support). Most of the removed weight came from earlier review-round additions rather than the original design:

@yummybomb
yummybomb marked this pull request as ready for review August 5, 2026 19:47
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 7ef1f8d to bf21162 Compare August 5, 2026 19:57
Comment thread lib/instances/start.go Outdated
Comment thread integration/vgpu_test.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from bf21162 to 83c53ea Compare August 5, 2026 20:24
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 83c53ea to 79eef07 Compare August 5, 2026 20:39
Comment thread cmd/api/main.go
Comment thread lib/devices/vendor_vfio_linux.go Outdated
Comment thread lib/instances/query.go Outdated
Comment thread lib/instances/create.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from ab2b0e8 to 19b9602 Compare August 6, 2026 17:08
Comment thread lib/instances/start.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from d92201d to f177335 Compare August 6, 2026 19:06
@yummybomb
yummybomb changed the base branch from hypeship/generalize-vgpu-device to hypeship/vendor-vfio-backend August 6, 2026 19:08
@yummybomb yummybomb changed the title Support vendor VFIO vGPU devices Integrate vendor VFIO vGPUs into the instance lifecycle Aug 6, 2026
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from f177335 to 1d875f2 Compare August 6, 2026 19:26
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 1d875f2 to 23fb2f5 Compare August 6, 2026 19:40
Enable vendor VFIO dispatch in CreateVGPU now that the lifecycle
persists assignments durably and guards releases.

Protect nil-PID claims in the release guard: the hypervisor PID is only
persisted after the claimant boots, so a matching assignment without a
PID must be treated as live, matching the startup reconcile protection.

Scan raw metadata instead of hydrating instances for the claim check.
Hydration derives state through hypervisor queries for every instance
on the host, which every vendor VFIO release would pay; the guard only
needs the stored assignment, PID, and socket. Unreadable metadata still
fails the release closed.

Report pending vGPU cleanup even when retaining the rollback record
fails: the destroy already failed, so the caller must learn about the
outstanding assignment either way.
The vgpu_cleanup_pending response replaced the original create error
with cleanup guidance, leaving the cause only in server logs. Prefix
the message with the wrapped error so callers see why creation failed
as well as how to recover.
ListInstancesForReconcile hydrated every instance (socket stat, UFFD
health, /vm.info per instance) before the API served and again on each
grace retry, while the protected-set scan only reads stored metadata
fields. List raw metadata fail-closed instead, matching the release
claim scan, and drop the now-unused loadInstances parameterization.

Snapshot restore preserved the source's vGPU assignment path fields but
not GPUAssignedAt, so a retained assignment lost its crash-recovery
grace timestamp across a restore. Carry the timestamp with the rest of
the assignment.
When start's vGPU create fails with a pending device-layer cleanup, the
error was returned untyped, so the API mapped it to a generic
internal_error. Create already wraps the same condition in
VGPUCleanupPendingError and surfaces vgpu_cleanup_pending with
retained/unretained guidance.

Wrap start's pending-cleanup error the same way — Retained reflects
whether the retention record was persisted — and map it in the
StartInstance handler ahead of the errors.Is cases so the wrapped cause
cannot hide the pending cleanup.
…ures

The vgpu_cleanup_pending guidance tells callers to delete the retained
instance to retry a failed vGPU release, but no test exercised delete
against the minimal GPU-fields-only stub cleanupFailedCreate writes.
Add one.

Losing the reconcile inventory disables vendor VFIO reconciliation
host-wide while releases fail closed on the same inventory, so log it
at error level instead of warn. Also document the wholesale-restore
assumption in cleanupStartVGPU.
Vendor VFIO vGPUs boot but are non-functional on Cloud Hypervisor
(upstream cloud-hypervisor#7572), and the wedged VM then blocks the VF
release until startup reconcile. Reject the combination at create and
start after the rollback handler is registered, so the rejected device
is released through the normal cleanup path. Hypervisor selection
otherwise stays caller policy and mdev on Cloud Hypervisor keeps
working.

Retain identity fields (name, image, hypervisor, data dir) on the
failed-create retention record so it lists as a recognizable, deletable
instance instead of a nameless phantom; resource claims released by
rollback stay dropped.

Expose the assigned vGPU device_path in the instance API - on vendor
VFIO hosts mdev_uuid is empty and the sysfs path is the identity an
operator needs when a release wedges.
When a later start step failed and rollback could not destroy the
freshly created vGPU, cleanupStartVGPU retained the assignment on disk
but startInstance returned the original failure untyped, so the API
reported a generic error instead of vgpu_cleanup_pending with the
retained-assignment guidance. Mirror create's named-return wrap:
cleanupStartVGPU reports retention state and start wraps the returned
error in VGPUCleanupPendingError.
When start rollback fails to destroy a vGPU and the cleanup metadata
save also fails, the assignment may still be on disk from the mid-start
save. Reporting Retained: false then misdirects callers to wait for
startup reconcile when delete or a retried start can already release
it. Check whether the surviving record still points at the device,
matching create's retention-survives check.
Drop the vendor-VFIO-on-Cloud-Hypervisor rejection from create and
start, restoring the phase-0 decision that hypervisor selection is
caller policy: production callers pin vGPU instances to QEMU, and the
Cloud Hypervisor limitation stays documented in lib/devices/GPU.md.
retainedVGPUFromCreateError built a GPU-fields-only stub, so the
retained record from a failed device-layer cleanup listed nameless and,
with GPUProfile empty, the API hid its gpu block including device_path.
The caller now supplies the identity fields and the stub picks up the
pending device's profile.
Startup reconcile protects an assignment whose PID is absent or stale
for a bounded grace window, but the release-side claim scan treated a
dead PID as unclaimed immediately. Align the two guards: a recent
assignment whose recorded hypervisor is not running fails the scan
closed so the requester retains and retries, and past the grace window
the dead claim no longer blocks the release.
A failed create whose vGPU release also failed persists a delete-only
retention stub with no boot configuration. The stub derives as Stopped,
so start would release the retained VF and then try to boot the
incomplete record. Mark the stub with GPURetainedForCleanup and reject
start with invalid_state guidance pointing at delete, which retries the
release.
A fork or snapshot of a failed-create retention stub could never boot:
the stub has no boot configuration, and clearing the delete-only marker
on the child would only produce a startable-but-broken record that
recreates a vGPU from GPUProfile with incomplete metadata. Reject fork
and snapshot of retention stubs with the same invalid_state guidance as
start, so delete (which retries the release) is the only action on
them.
The claim guard and startup reconcile protection predate the
HypervisorProcessIdentity struct and the removal of the standalone
identity-exists helpers. Route both through resolveLiveHypervisorPID:
the claim guard keeps failing closed on unresolvable ownership, and
reconcile protection gets a fail-open HypervisorMayBeAlive wrapper so
unresolvable ownership still protects the device.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant