Skip to content

Abstract vGPU devices behind a framework dispatch - #366

Open
yummybomb wants to merge 21 commits into
mainfrom
hypeship/vgpu-framework-abstraction
Open

Abstract vGPU devices behind a framework dispatch#366
yummybomb wants to merge 21 commits into
mainfrom
hypeship/vgpu-framework-abstraction

Conversation

@yummybomb

@yummybomb yummybomb commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Bottom layer of the vendor VFIO vGPU stack (this#322#363#364#321). Behavior-preserving refactor only — no lifecycle semantics change in this layer.

Kernel 6.8 hosts assign NVIDIA vGPUs through a vendor-specific VFIO interface instead of mdev, so the mdev-shaped seams get generalized before the new backend lands above:

  • VGPUDevice / VGPUFramework abstraction — mdev moves behind a framework dispatch (CreateVGPU / DestroyVGPU / DiscoverVGPU), with assignments described by a framework + device path instead of a bare mdev UUID.
  • Hypervisor wiring — QEMU receives the generalized vGPU path through VGPUDevicePath, while Cloud Hypervisor continues to receive the existing mdev path as a passthrough device.
  • Metadata — VF allocation tracked with a single field; instance metadata carries GPUFramework / GPUDevicePath alongside the mdev UUID.
  • QEMU args — vGPU attaches via sysfsdev, and the dead mdev branch is dropped from PCI passthrough args.

Testing

  • go build ./..., go vet clean
  • go test -race ./lib/devices/ ./lib/hypervisor/... and targeted lib/instances suites pass (TestSocketCacheKeyChangesWhenSocketIsRecreated and the network/image-dependent instances tests fail identically on the unmodified stack head in this environment)

Note

Medium Risk
Touches GPU instance create/start/stop/delete and hypervisor device args; behavior should match mdev today but mistakes could leak vGPU slots or break VM boot attachment.

Overview
Introduces a framework-agnostic vGPU layer (VGPUDevice, VGPUFramework, CreateVGPU / DestroyVGPU) so mdev is one backend today and future vendor VFIO paths can plug in without rewriting instance lifecycle.

Instance create/start/stop/delete now allocate and release vGPUs through that API and persist GPUFramework, GPUDevicePath, and the mdev UUID (for legacy rows), with helpers to resolve the attach path and tear down via VGPUAssignment. SR-IOV VF state is renamed from HasMdev to Allocated to match framework-neutral wording.

Hypervisors get a dedicated VGPUDevicePath on VMConfig instead of stuffing mdev sysfs paths into PCIDevices; QEMU always attaches vGPU with vfio-pci,sysfsdev=…, and microvm validation rejects both PCI passthrough and vGPU paths.

Reviewed by Cursor Bugbot for commit 9396bc0. Bugbot is set up for automated code reviews on this repo. Configure here.

@yummybomb yummybomb changed the title hypeship/vgpu framework abstraction Abstract vGPU devices behind a framework dispatch Aug 6, 2026
@yummybomb
yummybomb marked this pull request as ready for review August 6, 2026 19:20
@yummybomb
yummybomb force-pushed the hypeship/vgpu-framework-abstraction branch 2 times, most recently from c3d6a2f to f677357 Compare August 6, 2026 19:40
Comment thread lib/instances/start.go Outdated
@yummybomb
yummybomb force-pushed the hypeship/vgpu-framework-abstraction branch from a903554 to 4499016 Compare August 7, 2026 20:52
@yummybomb
yummybomb force-pushed the hypeship/vgpu-framework-abstraction branch from 12bd9d9 to 0abb18e Compare August 10, 2026 18:26
Comment thread lib/instances/vgpu.go Outdated
@yummybomb
yummybomb force-pushed the hypeship/vgpu-framework-abstraction branch from 0abb18e to 5f9e4fd Compare August 10, 2026 18:58
Comment thread lib/instances/stop.go
Comment thread lib/instances/create.go
@yummybomb
yummybomb requested a review from sjmiller609 August 11, 2026 21:35
@yummybomb
yummybomb force-pushed the hypeship/vgpu-framework-abstraction branch from fe97bda to d86b86b Compare August 12, 2026 15:20

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d86b86b. Configure here.

Comment thread lib/hypervisor/qemu/config.go
@yummybomb
yummybomb force-pushed the hypeship/vgpu-framework-abstraction branch from d86b86b to ec4e189 Compare August 13, 2026 14:19
Stop and delete keep best-effort vGPU release (log and continue,
metadata always cleared) and start no longer releases a stale stored
assignment. Those behavior changes belong to the lifecycle-hardening
layer, not this behavior-preserving refactor.
vGPU attachment moved from PCIDevices to VGPUDevicePath, which let a
microvm config carrying only a vGPU pass the profile validation that
previously rejected the mdev path via the PCIDevices check.
@yummybomb
yummybomb force-pushed the hypeship/vgpu-framework-abstraction branch from ec4e189 to 40e61b2 Compare August 13, 2026 18:28

@sjmiller609 sjmiller609 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • consider a struct param for DestroyVGPU(ctx, framework, devicePath, mdevUUID) — three positional strings are easy to swap silently as backends multiply
  • error strings still say "create vGPU mdev for profile %s" — reword in whichever layer introduces non-mdev backends
  • stop/delete log "uuid", stored.GPUMdevUUID — will be empty for path-only assignments; log the device path instead

@yummybomb

Copy link
Copy Markdown
Contributor Author

addressed the review in 113d9e6 + 9396bc0:

  • DestroyVGPU now takes a devices.VGPUAssignment struct instead of three positional strings — pulled the exact shape down from the next PR in the stack, so its diff collapses on restack
  • stop/delete now log device_path (always populated) instead of the possibly-empty mdev uuid
  • the "create vGPU mdev" error strings stay as-is in this layer since mdev is the only backend here; the layer that introduces the vendor backend already rewords them to "create vGPU for profile"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants