Harden hypervisor process liveness checks

[yummybomb]

Summary

Layer 1 of the vendor VFIO vGPU stack (generalize-vgpu-device ← this ← vendor-vfio-backend ← vendor-vfio-vgpu). Pure hypervisor-process hardening with no vGPU-specific code; reviewable in isolation.

The upper layers guard vGPU release decisions on "is this instance's hypervisor still alive", so the liveness answer has to be trustworthy first:

• Unify liveness checks on ProcessExists — one exported, EPERM-aware, zombie-filtering definition instead of scattered bare kill(pid, 0) probes. EPERM means the process exists but cannot be signaled; treating it as dead would be wrong.
• Wait for non-child hypervisor exit before finishing kill — after a hypeman restart the hypervisor is not our child, so Wait4 returns ECHILD immediately and the kill loop finished before the process had exited. Poll for actual exit in that case.
• Verify process identity before teardown — a bare PID probe treats any process that reused a stored hypervisor PID as the owning VMM. On Linux, require confirmed socket ownership or a matching PID/start-time token scoped to the current host boot, including the graceful stop/delete wait path.
• Treat provable death as death — two disproofs unwedge stop/delete instead of failing closed forever: an identity token recorded under a different host boot cannot be the recorded VMM, and a socket that no process owns or references (both the listener scan and the full command-line scan come up empty) proves the recorded VMM is gone even when a recycled PID is live. The genuinely ambiguous case — a live process matching the socket by command line only — still fails closed, and the recycled PID is never signaled.

Failure contract

When the hypervisor can be confirmed dead or killed, delete completes synchronously (worst case ~2s graceful + SIGKILL waits). When ownership is genuinely unprovable — a live command-line-only socket match, or a process surviving SIGKILL — stop/delete return an error (API 500) with metadata retained and the restart policy blocked, so a retried delete is safe and converges once the ambiguity clears (debug process exits, host reboots). An earlier revision handled this with a background delete finalizer; it was removed in favor of the synchronous error after review — the ambiguous cases are rare, self-resolving, and better surfaced loudly than retried silently.

Testing

• go build ./..., go vet clean
• go test -race ./lib/instances/ targeted suites pass (TestCreateInstanceWithNetwork requires image pulls + iptables and fails in this environment on the unmodified base as well; TestStandbyAndRestore/TestForkCloudHypervisorFromRunningNetwork boot real VMs and cannot run in this sandbox — CI covers them)

---

Note

High Risk
Changes core stop/delete/standby and hypervisor kill semantics on Linux; incorrect ownership logic could leave VMs running or signal wrong processes, though ambiguous cases fail closed with retained metadata.

Overview
Hardens hypervisor liveness so stop, delete, and standby only tear down a VMM when ownership is provable, avoiding mistaken kills of recycled PIDs and wedged lifecycle operations.

Introduces HypervisorProcessIdentity (PID plus boot-scoped start time and boot ID) embedded in instance metadata with unchanged JSON keys. Confirmed socket owners get a full identity token; command-line-only or dead fallbacks store an unconfirmed PID only. resolveLiveHypervisorPID drives destructive paths: trust matching tokens, require confirmed /proc socket ownership, treat empty socket scans or stale boot tokens as provably dead, and fail closed on live ambiguous command-line matches.

Linux socket resolution now returns a confirmed flag, prefers listeners over accepted connections, handles multiple owners, and adds ResolveProcessPIDForOwner for expected-owner fast paths. ProcessExists is unified (EPERM-aware, zombie-filtering); killProcessAndWait replaces ad-hoc SIGKILL loops and waits for non-child hypervisors after hypeman restart.

Behavioral tightening: delete and standby abort (retain metadata) when kill/shutdown cannot confirm the hypervisor is gone; graceful stop waits on the socket owner, not the stored PID; shutdown defers unlinking the control socket until the process is confirmed dead. Startup runs BackfillHypervisorProcessIdentities for legacy instances missing identity tokens.

^{Reviewed by Cursor Bugbot for commit f9abc52. Bugbot is set up for automated code reviews on this repo. Configure here.}