Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper - #14006
Open
abh1sar wants to merge 2 commits into
Open
Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper#14006abh1sar wants to merge 2 commits into
abh1sar wants to merge 2 commits into
Conversation
…up restore Script.executeCommand returns null when the command fails, it does not throw, so the try/catch around the mount and umount of the backup repository could never fire and the return value was discarded. A repository that fails to mount was therefore treated as mounted, and the restore carried on against an empty directory until it failed later with a misleading "backup file not found". A failed umount was ignored the same way, leaking the mount. Both now go through executeCommandForExitValue and check the exit value. The same refactor also dropped the timeouts. mountTimeout was still passed into mountBackupDirectory but never used, and the rsync of the volume lost the command timeout, so both fell back to the one hour default in Script instead of the configured values. An unresponsive repository could hold a restore up for an hour rather than failing after nas.backup.restore.mount.timeout seconds.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 4.22 #14006 +/- ##
=========================================
Coverage 17.79% 17.79%
- Complexity 15995 15996 +1
=========================================
Files 5928 5928
Lines 534306 534323 +17
Branches 65383 65385 +2
=========================================
+ Hits 95069 95076 +7
- Misses 428467 428477 +10
Partials 10770 10770
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
Author
|
@blueorangutan package |
|
@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
abh1sar
changed the title
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19015 |
abh1sar
requested review from
shwstppr and
weizhouapache
and
a lite review from Copilot
August 29, 2026 16:15
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes restore-from-backup behavior for KVM/libvirt by correctly treating failed mount/umount operations as failures (via exit codes instead of relying on exceptions) and by restoring the intended command timeouts so restores don’t hang for Script’s 1-hour default.
Changes:
- Switch mount to
Script.executeCommandForExitValue(timeout, ...)and fail fast on non-zero exit codes. - Switch umount to
executeCommandForExitValue(...)and fail on non-zero exit codes (instead of silently ignoring failure). - Ensure rsync uses the configured restore command timeout; extend unit tests and add a new timeout-focused test.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtRestoreBackupCommandWrapper.java | Validates mount/umount exit codes and restores rsync timeout usage during restore operations. |
| plugins/hypervisors/kvm/src/test/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtRestoreBackupCommandWrapperTest.java | Updates mocks for timeout-aware executeCommandForExitValue and adds a test asserting the configured mount timeout is used. |
Suppressed comments (1)
plugins/hypervisors/kvm/src/test/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtRestoreBackupCommandWrapperTest.java:605
- This test attempts to capture the timeout used for the mount command, but the "mount" detection has the same issue as other varargs stubs: invocation.getArguments() contains (timeout, String[] cmd), and String.valueOf(cmd) won't equal "mount". As a result, mountTimeout[0] is never set and the assertion can fail (or the test can pass without actually checking the mount invocation, depending on defaults).
if (Arrays.stream(invocation.getArguments()).map(String::valueOf).anyMatch("mount"::equals)) {
mountTimeout[0] = invocation.getArgument(0);
return 1; // stop the restore right after the mount
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…repository The directory created for the mount is removed by the caller in a finally block, but that block is only reached once the mount has succeeded, so a repository that cannot be mounted left an empty directory behind on every attempt. It is now removed before the failure is reported, best effort and logged if it cannot be. The unmount ran without a timeout and so fell back to the one hour default in Script. Unmounting a repository that has become unreachable blocks as easily as mounting one, and this runs in the cleanup path of a restore, so it is now bounded by the configured mount timeout like the mount itself.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Script.executeCommand returns null when the command fails, it does not throw, so the try/catch around the mount and umount of the backup repository could never fire and the return value was discarded.
A repository that fails to mount was therefore treated as mounted, and the restore carried on against an empty directory until it failed later with a misleading "backup file not found". A failed umount was ignored the same way, leaking the mount. Both now go through executeCommandForExitValue and check the exit value.
The same refactor also dropped the timeouts. mountTimeout was still passed into mountBackupDirectory but never used, and the rsync of the volume lost the command timeout, so both fell back to the one hour default in Script instead of the configured values. An unresponsive repository could hold a restore up for an hour rather than failing after nas.backup.restore.mount.timeout seconds.
This is a regression from 56ad044
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
Verified that restore fails after the specified nas.backup.restore.mount.timeout with a dead address.
How did you try to break this feature and the system with this change?