Modernize the default Hyper-V enlightenment profile for Windows guests on KVM

[andrijapanicsb]

Summary

Apache CloudStack 4.22 gives all Windows guests a localtime clock and
hypervclock, but it emits a <hyperv> feature block only when the guest OS
label contains Windows PV. Even in that case, the block is limited to:

<hyperv>
  <relaxed state='on'/>
  <vapic state='on'/>
  <spinlocks state='on' retries='8096'/>
</hyperv>

This is a historical, very small subset of the Hyper-V enlightenments supported
by every KVM host family currently listed in the CloudStack 4.22 compatibility
matrix. The generic Windows PV label also does not describe a Windows version
and does not, by itself, select VirtIO storage or networking.

CloudStack should introduce an explicit, versioned Windows-on-KVM enlightenment
profile instead of relying on the substring Windows PV as the only switch.

Current behavior

In LibvirtComputingResource.enlightenWindowsVm() CloudStack checks only:

if (vmTO.getOs().contains("Windows PV")) {
    // relaxed, vapic and spinlocks only
}

LibvirtVMDef.HyperVEnlightenmentFeatureDef can currently serialize only those
three features. Separately, every OS label beginning with Windows receives a
hypervclock timer.

Consequences:

• Windows Server 2012 through Windows Server 2025 and current Windows desktop
  guests do not receive a <hyperv> block when their real version-specific OS
  type is selected.
• Selecting Windows PV loses version information while enabling only three
  old enlightenments.
• The effective behavior is hard-coded rather than represented as a documented
  compatibility profile.
• CloudStack cannot safely add newer features according to host capabilities or
  preserve a stable feature set for live migration within a cluster.

Why this can be modernized without requiring VirtIO drivers

Hyper-V enlightenments are CPU/timer/interrupt interfaces exposed by KVM and
QEMU. They are independent of the guest's disk and network device models. A
Windows VM can use these features while continuing to boot from IDE or SATA and
using an emulated E1000 NIC.

This issue therefore must not implicitly change disk controllers, NIC models,
firmware, or machine type. Those changes have separate guest-driver and boot
compatibility risks.

CloudStack 4.22 KVM compatibility floor

The 4.22 compatibility matrix lists the following KVM host OS families:

• Ubuntu 22.04 and 24.04 LTS
• Oracle Linux, Rocky Linux, AlmaLinux and RHEL 8, 9 and 10
• openSUSE Leap 15 and SLES 15

CloudStack documents major OS families and does not pin a minimum service-pack,
point-release, kernel, QEMU package build, or libvirt package build. Therefore
the table below is a conservative engineering floor based on the oldest branch
named by CloudStack, not a claim that every EOL initial point-release is tested
or should be deployed.

CloudStack 4.22 host family	Conservative oldest branch reference	QEMU family	libvirt family	Linux kernel family	Qualification
Ubuntu	22.04 LTS	6.2	8.0	5.15	Explicitly listed by CloudStack
RHEL / Oracle Linux	8.0-era baseline	2.12	4.5	4.18	RHEL is listed as expected to work; current patched minor releases are required in practice
Rocky / AlmaLinux	EL8 baseline	no lower than the EL8 floor	no lower than the EL8 floor	4.18	These distributions first shipped after EL8.0; use the lower EL8 family floor for design
openSUSE Leap / SLES	15.0-era baseline	2.11	4.0	4.12	CloudStack says Leap 15.6 is the tested Leap release; the older numbers are only a conservative branch floor

The lowest conservative host in that matrix is still substantially newer than
the minimum required for the proposed baseline below.

Proposed implementation

1. Add explicit profiles and version-based defaults

Add a saved Windows virtualization profile with four values:

• disabled: no <hyperv> block; retain the existing Windows clock handling.
• legacy: exactly the current Windows PV feature set: relaxed, vapic,
  and spinlocks retries='8096'.
• modern-safe: the CPU-vendor-neutral, cross-distro baseline described below.
• auto: modern-safe plus optional features supported by every eligible host
  in the selected KVM cluster.

Use the following defaults for newly deployed VMs:

Selected guest OS	Default profile	Reason
Windows 8/8.1/10/11	modern-safe	Version-specific modern Windows type
Windows Server 2012/2012 R2/2016/2019/2022/2025 and newer	modern-safe	Version-specific modern Windows Server type
Windows 7 and Windows Server 2008/2008 R2	legacy	Preserve the small feature set originally introduced for Windows Server 2008 and later
Windows Vista/XP, Windows Server 2003 and older	disabled	Do not advertise untested features to legacy guests
Versionless Windows PV	legacy	Preserve current CloudStack behavior
Unknown or unclassified Windows type	disabled	Prefer predictable compatibility when the Windows generation is not known

The operator can override the proposed value before deployment. Existing VMs
must keep their current generated XML when no saved profile exists: existing
Windows PV VMs continue to behave as legacy, while other existing Windows
VMs continue without a <hyperv> block. An upgrade alone must not alter a VM's
virtual CPU definition.

The selected and effective values must be visible in debug logs and in the
generated domain XML.

2. Expose a clear Windows-only UI and API setting

Show a field labelled Windows virtualization optimizations when the selected
hypervisor is KVM and the selected guest OS is Windows. Hide it for non-Windows
guests and non-KVM hypervisors.

The field is automatically populated according to the table above and offers:

• Modern (recommended) ??? portable modern optimizations supported across the
  CloudStack 4.22 KVM host baseline.
• Automatic ??? Modern plus additional optimizations supported by every host
  in the target KVM cluster.
• Legacy compatibility ??? current Windows PV behavior.
• Disabled ??? no Hyper-V feature block.

The management server must be the authoritative source for the supported
profile values and their stable identifiers. The UI must populate the drop-down
from backend-provided values, following the same general pattern used for other
VM settings such as keyboard type and root disk controller. Do not maintain an
independent frontend-only list that can drift from backend validation.

The default UI should not expose individual QEMU feature names or the raw
spinlock retry number. A tooltip or help panel must explain that the setting
controls Hyper-V-compatible CPU, timer, and interrupt optimizations exposed by
KVM; it does not install or require VirtIO storage or network drivers. Each
option must have a short description and a link to administrator documentation.

Store the template's proposed/default value in template details. At deployment,
copy the selected value to VM details. From that point, the VM value is the
authoritative setting used for every start, restart, and reschedule. Changing it
after deployment must require the VM to be stopped. This prevents a later
CloudStack upgrade, template edit, or move to another host from silently
changing the virtual CPU definition.

Expose the same drop-down in VM Settings. When the VM is stopped, an operator can
select another backend-provided profile and save it to VM details. When the VM
is running, show the effective value as read-only and explain that the VM must
be stopped before it can be changed. The newly selected profile is applied when
CloudStack next generates the domain XML and starts the VM.

The API must expose the supported values, accept the selected optional value at
deployment, and provide an update operation for a stopped VM so API and UI
behavior remain identical. Backend validation must reject unknown values and
changes attempted while the VM is running. Existing callers that omit the value
retain the compatibility behavior described above.

3. Implement a universal modern-safe baseline

Use an explicit/custom feature list rather than libvirt Hyper-V passthrough:

<hyperv mode='custom'>
  <relaxed state='on'/>
  <vapic state='on'/>
  <spinlocks state='on' retries='4095'/>
  <vpindex state='on'/>
  <runtime state='on'/>
  <synic state='on'/>
  <stimer state='on'/>
</hyperv>

The highest host requirement in this baseline is:

• QEMU 2.6 or newer
• libvirt 1.3.3 or newer

This is below the conservative QEMU/libvirt floor of every KVM distribution
family listed by CloudStack 4.22. It is independent of Intel versus AMD host CPU
and does not require VirtIO drivers in the Windows guest.

spinlocks retries is a retry count, not a time value or network port. When one
virtual CPU is waiting for a lock potentially held by another virtual CPU, the
guest retries that many times before sending the advisory
HvCallNotifyLongSpinWait hypercall. The notification helps the hypervisor make
a better scheduling decision, especially on an overcommitted host where the
lock-holding virtual CPU may not currently be running.

CloudStack introduced the decimal value 8096 in the 2018 enlightenment change,
but its commit, pull request, tests, and review discussion contain no explanation
for that value. There is no technical relationship between this retry count and
CloudStack's optional integration API port 8096; the identical decimal number
appears to be coincidental. Preserve 8096 only in the legacy profile so that
existing behavior remains reproducible. Use 4095 (0xfff) for modern-safe,
which is the current QEMU recommendation for overcommitted hosts. QEMU documents
0xffffffff as the effective "never notify" choice when this scheduling hint is
not desired.

4. Add newer features through the auto profile

Do not add the following to the unconditional baseline:

Optional feature	Minimum libvirt	Minimum QEMU	Proposed policy
frequencies	4.7.0	2.12	auto tier only; required dependency for re-enlightenment
reenlightenment	4.7.0	3.0	auto tier; validate migration behavior and CPU frequency exposure
tlbflush	4.7.0	3.0	auto tier; requires vpindex
ipi	4.10.0	3.1	auto tier; requires vpindex
stimer direct	5.7.0	4.1	Newer-host tier only
evmcs	4.10.0	3.1	Explicit opt-in only for nested Hyper-V on compatible Intel hosts; not a universal Intel/AMD feature
avic	8.10.0	6.2	Do not enable generically; hardware/topology-specific
tlbflush direct/extended	11.0.0	7.1	Future tier only; outside the oldest CloudStack 4.22 host-family floor

Feature dependencies must be encoded and tested, for example stimer depends
on synic, synic depends on vpindex, and reenlightenment depends on
frequencies.

The modern-safe profile must not perform Intel/AMD or per-host capability
checks. It is the portable baseline.

The auto profile is part of this proposal and may be delivered as a follow-up
PR after the portable baseline and UI/API setting. It must use the intersection
of libvirt/QEMU capabilities across every host eligible to run the VM. Store both
the requested value (auto) and the resolved feature list on the VM so that its
virtual CPU definition remains stable across restart and migration. If a target
host cannot provide that saved feature list, CloudStack must reject that target
rather than silently reducing the list.

mode='passthrough' must not be used because it can expose a different feature
set on different hosts and make live migration unsafe. Optional features with
additional hardware requirements, including evmcs, avic, and
reenlightenment, must not be enabled merely because the installed QEMU version
recognizes their XML.

5. Extend the domain XML model and Windows classification

• Extend HyperVEnlightenmentFeatureDef to serialize the baseline and optional
  feature set, including nested attributes only where required.
• Replace the single contains("Windows PV") decision with guest-family and
  guest-version-aware selection.
• Preserve hypervclock and existing local-time behavior.
• Persist the chosen/effective profile so rescheduling and restart generate the
  same virtual CPU definition.
• Reject an unsupported explicitly requested profile with a clear, actionable
  error; never silently reduce the requested feature set or emit XML the
  destination cluster cannot support.

Compatibility and rollout requirements

• No change to an existing VM merely because CloudStack was upgraded.
• No automatic IDE/SATA-to-VirtIO, E1000-to-VirtIO, BIOS-to-UEFI, or
  i440FX-to-Q35 conversion as part of this issue.
• No dependence on VirtIO drivers.
• The baseline must work on both Intel and AMD KVM hosts.
• The effective feature set must be stable across all hosts eligible for live
  migration of the VM.
• Apply the version-based defaults only to newly deployed VMs. Preserve the
  generated XML of existing VMs unless the operator explicitly changes it.
• Document that changing a VM's enlightenment profile changes its virtual CPU
  definition and must be performed while stopped.

Acceptance criteria

1. Unit tests verify exact XML for disabled, legacy, modern-safe, and each
   supported auto capability tier.
2. Unit tests verify guest classification for version-specific Windows types,
   Windows PV, non-Windows guests, and null/unknown OS mappings.
3. The modern-safe XML can be defined and started on the oldest supported
   CloudStack CI images or equivalent test hosts for the Ubuntu 22.04, EL8,
   and SUSE 15 families.
4. Boot, reboot, clean shutdown, timekeeping, and multi-vCPU stress are tested
   with Windows Server 2012 R2, 2016, 2019, 2022, and 2025.
5. At least one test uses only emulated storage/network devices to prove that
   VirtIO drivers are not a prerequisite.
6. Intel and AMD hosts are covered.
7. Cold and live migration are tested within a homogeneous cluster and across
   hosts with different QEMU/libvirt patch levels but the same negotiated
   effective profile.
8. An existing VM without profile metadata retains the pre-change XML.
9. Unsupported optional features produce a clear validation error; CloudStack
   does not silently reduce a saved or explicitly selected feature set.
10. Administrator documentation describes profiles, host-version boundaries,
    guest-version mapping, override behavior, and rollback.
11. UI tests verify that the field is shown only for Windows on KVM, receives the
    correct version-based default, can be overridden, and includes explanatory
    help text.
12. Tests preserve 8096 in legacy and emit 4095 in modern-safe.
13. The VM Settings drop-down is populated from backend-provided values, is
    editable for a stopped VM, is read-only for a running VM, and persists the
    selected value in VM details.
14. API tests verify supported-value discovery, deployment-time selection,
    stopped-VM updates, rejection of unknown values, and rejection of changes
    while a VM is running.

Out of scope

• Selecting VirtIO disk or NIC models
• Installing or validating Windows VirtIO drivers
• Automatically converting existing disk controllers or firmware
• Changing BIOS/i440FX versus UEFI/Q35 defaults
• Video-device modernization
• Enabling nested Hyper-V or evmcs by default

References

• CloudStack 4.22 compatibility matrix
• Current CloudStack KVM Windows feature and clock generation
• Current CloudStack Hyper-V feature XML model
• Original CloudStack Windows enlightenment change
• libvirt domain XML: Hyper-V features and minimum versions
• QEMU Hyper-V enlightenment behavior and recommendations
• Microsoft Hyper-V guest spinlock behavior
• Microsoft HvCallNotifyLongSpinWait
• Ubuntu 22.04 QEMU package
• Ubuntu 22.04 libvirt package

Decisions proposed by this issue

• Default new, version-specific Windows 8 / Server 2012 and newer VMs to
  modern-safe; default older and unknown types according to the explicit table
  above.
• Show a documented, operator-overridable Windows-only setting in both the UI
  and API.
• Populate the UI drop-down from the backend's authoritative profile list and
  allow the saved value to be changed from VM Settings while the VM is stopped.
• Save the template default and copy the selected value to the VM so subsequent
  starts use the same virtual CPU definition.
• Ship the CPU-vendor-neutral modern-safe baseline without host capability
  negotiation.
• Include auto in the overall issue scope, but allow its cluster capability
  implementation to be a follow-up PR.
• Preserve retries 8096 only for exact legacy compatibility and use QEMU's
  documented overcommit recommendation, 4095, for modern-safe.

[andrijapanicsb]

@swill @nvazquez @ustcweizhou pls chime in with opinions before I jump into implementing this 🙏