KVM: Enable HA heartbeat on ShareMountPoint by weizhouapache · Pull Request #12773 · apache/cloudstack

weizhouapache · 2026-03-09T12:38:15Z

Description

This PR improves KVM HA heartbeat to support SharedMountPoint

Tested

add shared mount point pool
kvm hosts write heartbeat to the mount point
deploy vms
set global settingforce.ha to true
force shutdown one of the kvm hosts
after 3 mins, the VMs are started on other kvm hosts.

Example of heartbeat files

[root@kvm1 ~]# ls -l /mnt/cloudstack*/KVMHA
/mnt/cloudstack1/KVMHA:
total 1
-rw-r--r--. 1 root root 11 Mar  9 12:28 hb-10.1.34.125
-rw-r--r--. 1 root root 11 Mar  9 12:09 hb-10.1.35.76

/mnt/cloudstack2/KVMHA:
total 1
-rw-r--r--. 1 root root 11 Mar  9 12:28 hb-10.1.34.125
-rw-r--r--. 1 root root 11 Mar  9 12:09 hb-10.1.35.76

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov · 2026-03-09T12:58:37Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 3.70%. Comparing base (c3d6a8c) to head (a361b28).
⚠️ Report is 5 commits behind head on 4.22.

❗ There is a different number of reports uploaded between BASE (c3d6a8c) and HEAD (a361b28). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (c3d6a8c) HEAD (a361b28)

unittests 1 0

Additional details and impacted files

@@              Coverage Diff              @@
##               4.22   #12773       +/-   ##
=============================================
- Coverage     17.61%    3.70%   -13.91%     
=============================================
  Files          5917      448     -5469     
  Lines        531402    38028   -493374     
  Branches      64971     7036    -57935     
=============================================
- Hits          93586     1409    -92177     
+ Misses       427262    36432   -390830     
+ Partials      10554      187    -10367

Flag	Coverage Δ
uitests	`3.70% <ø> (-0.01%)`	⬇️
unittests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

weizhouapache · 2026-03-09T15:43:26Z

@blueoranguran package

Copilot

Pull request overview

Enhances KVM HA heartbeat to support SharedMountPoint primary storage by introducing a new heartbeat script and extending HA support checks across KVM storage and monitor components.

Changes:

Added a new KVM “SharedMountPoint” heartbeat script (kvmsmpheartbeat.sh) that writes heartbeat files to a mountpoint.
Expanded HA-supported storage pool types to include SharedMountPoint in driver, monitor, and pool logic.
Added a setType hook to KVMStoragePool and set the pool type after creation / retrieval to ensure HA detection works.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh	New local mountpoint-based heartbeat writer/checker script for SharedMountPoint HA.
plugins/storage/volume/default/.../CloudStackPrimaryDataStoreDriverImpl.java	Allows HA support for `SharedMountPoint` in primary storage capability logic.
plugins/hypervisors/kvm/.../LibvirtStoragePool.java	Enables HA support for SharedMountPoint and selects the correct heartbeat script.
plugins/hypervisors/kvm/.../KVMStoragePoolManager.java	Sets pool type after creation; exposes `getStorageAdaptor`.
plugins/hypervisors/kvm/.../KVMStoragePool.java	Introduces a default `setType` method for backward compatibility.
plugins/hypervisors/kvm/.../LibvirtCheckVMActivityOnStoragePoolCommandWrapper.java	Ensures retrieved pool has its type set before HA checks.
plugins/hypervisors/kvm/.../KVMHAMonitor.java	Expands heartbeat monitoring to include SharedMountPoint pools.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh

...ns/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStoragePoolManager.java

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAMonitor.java

scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh

blueorangutan · 2026-03-09T17:02:56Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 17048

weizhouapache · 2026-03-09T18:19:37Z

@blueorangutan package

blueorangutan · 2026-03-09T18:20:04Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-03-09T19:26:38Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17054

weizhouapache · 2026-03-09T20:54:39Z

@blueorangutan test

blueorangutan · 2026-03-09T20:56:03Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

weizhouapache · 2026-03-10T07:45:30Z

@sureshanaparti @rajujith
added this to 4.22.1 milestone, a small improvement but may benefit several users

cc @DaanHoogland @NuxRo

blueorangutan · 2026-03-10T13:14:24Z

[SF] Trillian test result (tid-15586)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 55643 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12773-t15586-kvm-ol8.zip
Smoke tests completed. 146 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestListIdsParams>:teardown	`Error`	1.12	test_list_ids_parameter.py
test_01_snapshot_root_disk	`Error`	7.74	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	`Error`	47.53	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	`Error`	47.54	test_snapshots.py
ContextSuite context=TestSnapshotStandaloneBackup>:teardown	`Error`	31.34	test_snapshots.py
test_01_snapshot_usage	`Error`	25.61	test_usage.py
test_01_vpn_usage	`Error`	1.08	test_usage.py

…/storage/KVMStoragePoolManager.java

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

DaanHoogland

clgtm

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAMonitor.java

…d rename it

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStoragePoolManager.java:399

isPoolSupportHA() now includes SharedMountPoint, so this block will add SharedMountPoint primary pools to _haMonitor. However, deleteStoragePool(...) below still removes pools from _haMonitor only when type == NetworkFilesystem, which can leave SharedMountPoint pools registered in the HA monitor after deletion/disconnect. Update the removal condition to stay consistent with the expanded HA-support set.

    private synchronized KVMStoragePool createStoragePool(String name, String host, int port, String path, String userInfo, StoragePoolType type, Map<String, String> details, boolean primaryStorage) {
        StorageAdaptor adaptor = getStorageAdaptor(type);
        KVMStoragePool pool = adaptor.createStoragePool(name, host, port, path, userInfo, type, details, primaryStorage);
        pool.setType(type);

        // LibvirtStorageAdaptor-specific statement
        if (pool.isPoolSupportHA() && primaryStorage) {
            KVMHABase.HAStoragePool storagePool = new KVMHABase.HAStoragePool(pool, host, path, PoolType.PrimaryStorage);
            _haMonitor.addStoragePool(storagePool);
        }

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAMonitor.java:145

The debug log messages in this method are now used for both NetworkFilesystem and SharedMountPoint, but still say "Found NFS storage pool". Also, this "Found ... continuing" message is logged even after the pool is marked for removal (e.g. when storage == null). Adjust the log text/placement to reflect "libvirt storage pool" and only log "found" when the pool is actually present and running.

    private void checkForNotExistingLibvirtStoragePools(Set<String> removedPools, String uuid) {
        try {
            Connect conn = LibvirtConnection.getConnection();
            StoragePool storage = conn.storagePoolLookupByUUIDString(uuid);
            if (storage == null || storage.getInfo().state != StoragePoolState.VIR_STORAGE_POOL_RUNNING) {
                if (storage == null) {
                    logger.debug(String.format("Libvirt storage pool [%s] not found, removing from HA list.", uuid));
                } else {
                    logger.debug(String.format("Libvirt storage pool [%s] found, but not running, removing from HA list.", uuid));
                }

                removedPools.add(uuid);
            }

            logger.debug(String.format("Found NFS storage pool [%s] in libvirt, continuing.", uuid));

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh

sureshanaparti

clgtm

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

sureshanaparti · 2026-03-12T09:23:38Z

@weizhouapache any doc update required for this?

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStoragePool.java

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStoragePool.java

scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh

…/storage/LibvirtStoragePool.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStoragePoolManager.java:399

Now that pool.isPoolSupportHA() can be true for SharedMountPoint (via LIBVIRT_STORAGE_POOL_TYPES_WITH_HA_SUPPORT) and such pools will be added to _haMonitor here, the corresponding deleteStoragePool(...) methods in this class still only call _haMonitor.removeStoragePool(uuid) for StoragePoolType.NetworkFilesystem. This can leave stale entries for deleted SharedMountPoint pools and keep running unnecessary heartbeat attempts. Consider updating the delete path to remove HA pools for all HA-supported libvirt storage pool types.

        // LibvirtStorageAdaptor-specific statement
        if (pool.isPoolSupportHA() && primaryStorage) {
            KVMHABase.HAStoragePool storagePool = new KVMHABase.HAStoragePool(pool, host, path, PoolType.PrimaryStorage);
            _haMonitor.addStoragePool(storagePool);
        }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStoragePool.java

scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh

...ns/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStoragePoolManager.java

Copilot

Pull request overview

Copilot reviewed 8 out of 9 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…n it

Copilot

Pull request overview

Copilot reviewed 8 out of 9 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh

engine/components-api/src/main/java/com/cloud/ha/HighAvailabilityManager.java

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStoragePool.java

weizhouapache · 2026-03-12T12:21:34Z

@blueorangutan package

blueorangutan · 2026-03-12T12:22:05Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

weizhouapache · 2026-03-12T12:25:50Z

thanks @DaanHoogland @sureshanaparti , and Copilot for review

@NuxRo
I think this is ready for testing now

boring-cyborg bot added component:kvm component:primary-storage component:storage labels Mar 9, 2026

KVM: Enable HA heartbeat on ShareMountPoint

93b95aa

weizhouapache force-pushed the 4.22-kvm-ha-on-shared-mount-point branch from 296835a to 93b95aa Compare March 9, 2026 12:47

weizhouapache requested a review from Copilot March 9, 2026 16:02

Copilot AI reviewed Mar 9, 2026

View reviewed changes

Copilot started reviewing on behalf of weizhouapache March 9, 2026 16:35 View session

weizhouapache added this to the 4.22.1 milestone Mar 10, 2026

weizhouapache added status:needs-testing status:needs-review labels Mar 10, 2026

weizhouapache added this to Apache CloudStack 4.22.1 Mar 10, 2026

weizhouapache linked an issue Mar 10, 2026 that may be closed by this pull request

KV Host HA/instance HA with shared mountpoint #9750

Open

weizhouapache and others added 3 commits March 10, 2026 17:01

Update plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm…

f6c6a17

…/storage/KVMStoragePoolManager.java

Update scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh

a803c2f

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh

eacece1

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

rajujith moved this to In Progress in Apache CloudStack 4.22.1 Mar 11, 2026

DaanHoogland approved these changes Mar 11, 2026

View reviewed changes

sureshanaparti reviewed Mar 12, 2026

View reviewed changes

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAMonitor.java Outdated Show resolved Hide resolved

move STORAGE_POOL_TYPES_WITH_HA_SUPPORT to HighAvailabilityManager an…

4e4da44

…d rename it

Copilot AI reviewed Mar 12, 2026

View reviewed changes

scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh Outdated Show resolved Hide resolved

scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh Outdated Show resolved Hide resolved

scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh Outdated Show resolved Hide resolved

sureshanaparti approved these changes Mar 12, 2026

View reviewed changes

Update scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh

5443527

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

weizhouapache and others added 3 commits March 12, 2026 10:27

Update scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh

bca0221

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh

c9b0f97

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

minor changes

1268f5d

weizhouapache requested a review from Copilot March 12, 2026 09:29

Copilot started reviewing on behalf of weizhouapache March 12, 2026 09:30 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

weizhouapache and others added 3 commits March 12, 2026 11:01

Update plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm…

8670f25

…/storage/LibvirtStoragePool.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh

b543690

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

fix wrong logic

67d9c8f

weizhouapache requested a review from Copilot March 12, 2026 10:27

Copilot started reviewing on behalf of weizhouapache March 12, 2026 10:29 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

weizhouapache added 2 commits March 12, 2026 11:59

KVM: remove KVMHABase.s_heartBeatPath

6bcb459

small refactoring

d30ca6e

weizhouapache requested a review from Copilot March 12, 2026 11:08

Copilot started reviewing on behalf of weizhouapache March 12, 2026 11:08 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh Outdated Show resolved Hide resolved

weizhouapache and others added 2 commits March 12, 2026 12:54

Update scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh

ab4557d

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

ensure mountPoint ends with a single trailing slash when delete vms o…

a361b28

…n it

weizhouapache requested a review from Copilot March 12, 2026 11:56

Copilot AI reviewed Mar 12, 2026

View reviewed changes

weizhouapache marked this pull request as ready for review March 12, 2026 12:25

Copilot started reviewing on behalf of weizhouapache March 12, 2026 12:25 View session

Conversation

weizhouapache commented Mar 9, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

weizhouapache commented Mar 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blueorangutan commented Mar 9, 2026

Uh oh!

weizhouapache commented Mar 9, 2026

Uh oh!

blueorangutan commented Mar 9, 2026

Uh oh!

blueorangutan commented Mar 9, 2026

Uh oh!

weizhouapache commented Mar 9, 2026

Uh oh!

blueorangutan commented Mar 9, 2026

Uh oh!

weizhouapache commented Mar 10, 2026

Uh oh!

blueorangutan commented Mar 10, 2026

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sureshanaparti left a comment

Choose a reason for hiding this comment

Uh oh!

sureshanaparti commented Mar 12, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Mar 9, 2026 •

edited

Loading