Skip to content

Knowledge Base#

Reasons a Windows VM Restarted Unexpectedly

Reasons a Windows VM Restarted Unexpectedly

If you have a Windows VM that has recently restarted unexpectedly, the first place to begin is to review the logs from the VM dashboard.
You can reach the log viewer by navigating from the Main Menu > Machines > Virtual Machines > the name of the VM you are investigating > Logs.

From the Log view, search through (or filter the results) under the Message section. Several status messages will indicate why the VM stopped running. The following are the most common and a brief explanation of each:

  • Message: VM action 'kill' sent
    This indicates that a VergeOS user issued a Kill Power command from the VergeOS interface. The log will indicate the user under the Source column.
    In this scenario, consult with the system user to determine the reason for issuing this command.

  • Message: VM action 'poweroff' sent
    This indicates that a VergeOS user issued a graceful Power Off command from the VergeOS interface. This command successfully interacted with the Guest OS ACPI to gracefully stop the VM. The log will indicate the user under the Source column.
    In this scenario, consult with the system user to determine the reason for issuing this command.

  • Message: VM has shutdown
    This indicates that the shutdown command was issued from inside the guest operating system directly.
    In this scenario, consult with the guest operating system logs to determine the reason for the shutdown command.

  • Message: VM has reset
    This indicates that the restart command was issued from inside the guest operating system directly.
    In this scenario, consult with the guest operating system logs to determine the reason for the reset command.

Best Practice

A guest VM running Windows that experiences unexpected restarts is often found to be caused by the Microsoft Windows Update Services being configured to automatically apply updates that require a restart.
To investigate this further, consult with Knowledge-Base articles about the particular version of the Windows OS.
One of the best places to start investigating at a Windows Guest level is using Windows Event Viewer and reviewing the Windows Update logs for more information.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Loading Virtio Drivers in Windows Recovery Console

If a guest VM running any version of Windows OS with the Virtio-SCSI disk drivers installed is booted into Recovery Console mode, the operating system partition may not be visible as the drivers are not yet loaded.

Prerequisites

Before starting with recovery work on a guest OS, ensure that the administrator has:

  1. Access to the VM, including tested/known working credentials.
  2. The Virtio drivers ISO loaded and available in the VergeOS environment under Files.
  3. The Windows OS installation ISO available in the VergeOS environment.

Requirements

To load Windows drivers from a Windows Guest OS recovery console, administrators will need:

  1. VergeOS UI platform access to the environment running the Guest OS.
  2. A copy of the (latest) Virtio disk drivers in ISO format.
  3. A copy of the Windows operating system ISO (example: Windows boot disk).

Starting the Windows Recovery Console

  1. Login to the console and follow the Windows installation prompts to navigate to the Recovery Console and Command Prompt.
  2. Once at a command prompt, type the following command to list available disks on the VM:
wmic logicaldisk get deviceid, volumename, description

Press ENTER.

This will return a list of available disks. Typically, D: will be the EFI boot volume, E: will be the CD installation media, and X: will be the active, booted Windows recovery session.

The Virtio-SCSI disks are likely not present, and the drivers need to be loaded for them to appear.

Installing the Virtio-SCSI and Storage Drivers

From the Command Prompt in the Windows Recovery Console, follow these steps:

  1. To load the Virtio-SCSI Storage drivers, type:
Drvload e:\vioscsi\2k16\amd64\vioscsi.inf

Press ENTER.

  • E: refers to the drive letter where the Virtio ISO is loaded.
  • 2k16\amd64 refers to the path to the Virtio driver based on OS. Browse the ISO image for paths to other OS versions if needed.
  1. To load the Virtio Storage drivers, type:
Drvload e:\viostor\2k16\amd64\viosstor.inf

Press ENTER.

  • E: refers again to the drive letter where the Virtio ISO is loaded.
  • 2k16\amd64 refers to the path to the Virtio driver based on OS. Adjust as necessary.

Verifying the Disk Mount

After loading both drivers, the disk should mount, typically as an F: drive.

To verify this, run the command again:

wmic logicaldisk get deviceid, volumename, description

Press ENTER. You should now see the newly mounted disk.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Windows Disks Showing Offline After Restarting

How to Correct Disks showing Offline after a Windows VM is Restarted

After importing a VM into the VergeOS platform, some (virtual) disks may not come online after a restart of the guest VM.

Beginning in Windows 2008, Microsoft added a setting for the default state of additional (non OS) disk drives. Occasionally when restarting, Windows may detect a hardware change, which is more likely if the VM was imported from an alternate hypervisor. When windows detects a hardware change, it may not bring secondary virtual disks online automatically. To change this to the recommended setting run the following Windows PowerShell:

Set-StorageSetting -NewDiskPolicy OnlineAll

Additionally, this can be done from the command line using diskpart or from Disk Management.

Document Information

  • Last Updated: 2024-08-10
  • vergeOS Version: 4.11

UEFI Disks as Boot Devices Will Not Boot Properly

After importing VM images leveraging UEFI disks as boot devices, sometimes the VM will not boot properly inside the VergeOS platform. To resolve this, tweaks to boot order/options need to be made.

Here is a list of suggested solutions for issues with imported VMs not booting because of guest UEFI BIOS settings:

Solution 1

  1. From a fresh import of the VM (before trying to power it up inside VergeOS), edit the VM in Verge and enable the UEFI Boot option.
  2. Power on the VM.
  3. Hit ESC within 5 seconds to get into the VM BIOS (you can edit the VM settings in Verge to increase the boot timer if necessary).
  4. Enter the BIOS and navigate to Boot Manager Options (this may vary depending on the BIOS).
  5. Change the selected boot device to the Verge IO device.
  6. Exit the UEFI BIOS.
  7. Reboot the VM.

Solution 2

  1. From a fresh import of the VM (before trying to power it up inside VergeOS), edit the VM in Verge and enable the UEFI Boot option.
  2. Power on the VM.
  3. Hit ESC within 5 seconds to get into the VM BIOS (you can edit the VM settings in Verge to increase the boot timer if necessary).
  4. Enter the BIOS and navigate to Boot Manager Options (this may vary depending on the BIOS).
  5. Disable Secure Boot as an option.
  6. Exit the UEFI BIOS.
  7. Reboot the VM.

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6
  • in VM
  • 2 min read

How to TRIM your Drives

After importing a virtual machine from another hypervisor, sometimes the free space available inside the virtual machine does not match the free space reported to the VergeOS platform. This discrepancy is often due to the virtual disk being thick-provisioned from the VM source, making VergeOS unaware of the unused disk space. To resolve this, a TRIM/UNMAP operation needs to be performed on the virtual disk from within the virtual machine.

Prerequisites for Running a TRIM Command

  1. Edit the virtual drive(s) in question in the VergeOS UI and ensure Discard is enabled.
  2. Ensure the virtual drive(s) is using a drive type of virtIO-SCSI or SATA.
  3. Ensure the virtual drive(s) is assigned to a Solid State Tier (usually tier 1-3).

Trimming a Windows Drive

To perform a manual TRIM operation in a Windows environment, follow these steps:

  1. Launch PowerShell as an admin user.
  2. From the PowerShell prompt, type the following command:

Optimize-Volume -DriveLetter YourDriveLetter -ReTrim -Verbose
Example:

Optimize-Volume -DriveLetter E -ReTrim -Verbose
3. Press Enter and wait for the command to complete.

As the TRIM operation progresses, you can watch the reported free space in the VergeOS dashboard increase as the blank data on the volume is removed.

If this does not resolve the issue, TRIM may not be enabled. To check if TRIM is enabled:

  1. Run the following FSUTIL command:
fsutil behavior query disabledeletenotify

If the value is 1, TRIM is not enabled on the drive.

  1. To enable TRIM, run:
fsutil behavior set disabledeletenotify 0

After enabling, rerun the TRIM commands.

Trimming a Linux Drive

Newer Linux distros have TRIM enabled by default via a systemd service or a cron job. To check if automated TRIM is enabled, follow these steps:

  1. Ensure the prerequisite steps from above are complete.

Example: Ubuntu Server

  1. Launch a terminal.
  2. Enter the following commands to check the status:

  • Check TRIM Timer/Schedule Status:

    sudo systemctl status fstrim.timer
    - Check TRIM Service Status:

    sudo systemctl status fstrim

If TRIM is enabled, an operation will run at the next scheduled time. If TRIM is not enabled, you can run a manual TRIM using:

sudo fstrim -av

Info

It is recommended to enable automatic TRIM to ensure that data usage is reflected accurately between VergeOS and the guest OS.

To enable automatic TRIM, run:

sudo systemctl enable fstrim.timer

For more information on fstrim, visit the man page.

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6

Windows is Unable to Detect a Virtual Disk Drive

Virtual Disk Drive not detected in Windows

In most cases this is because the disk drive Interface Type is set to virtIO-SCSI (which is the default value) on any virtual machine that is being created. If that is the case, you will need to load the virtIO drivers during the Windows installation process. Windows does not natively recognize virtIO interfaces, so it cannot see the virtual disk.

Open Source virtIO drivers can be downloaded from here courtesy of Fedora Linux.

If Redhat virtIO drivers are required due to signed driver needs within Windows, a Redhat account will need to be made on their website and the drivers downloaded directly from them.

Installing virtIO Drivers during Windows Install

During a Windows installation, users can use the console interface tools to change the CD-ROM image to the newly downloaded virtIO drivers iso. Information on using the console interface tools can be found in the inline help within the category titled VDI under the section Using the Console.

Alternatively, you can choose to change the virtual disk drive Interface type to SATA and Windows will find the virtual disk and continue with the installation. !! info "This will come with a performance impact due to SATA drivers being software emulated."

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Windows - Slow to Format a New Disk

Formatting a Virtual Disk with Windows 2012 (and later) Hosts May Take Longer Than Expected

Windows Server 2012 (and later) hosts will, by default, issue SCSI TRIM and Unmap commands equivalent to the entire size of the virtual disk. This behavior is the same even if the "Perform a quick format" option is checked, which significantly slows down the format process.

It is possible to disable the SCSI TRIM and Unmap feature on the host for the duration of the format.

To Disable TRIM and Unmap

Using a Windows CMD window on the host, issue the following command:

fsutil behavior set DisableDeleteNotify 1

To Re-enable the Feature

Use the following command to re-enable the Trim and Unmap feature:

fsutil behavior set DisableDeleteNotify 0

To Verify the Current Setting

You can verify the current Trim and Unmap setting by issuing the following command:

fsutil behavior query DisableDeleteNotify

The output will show one of the following:

  • DisableDeleteNotify=0 - The 'Trim and Unmap' feature is on (enabled).
  • DisableDeleteNotify=1 - The 'Trim and Unmap' feature is off (disabled).

Affected Versions

  • Only Windows Server 2012 and later hosts are affected. All earlier versions (e.g., Windows 2008) do not exhibit the same issue.

Non-server Versions

  • Non-server versions of Windows (e.g., Windows 8.x and 10.x) do not support the DisableDeleteNotify parameter.

Document Information

  • Last Updated: 2024-08-29
  • VergeOS Version: 4.12.6

Windows Restored VM Not Bootable

After restoring a copy of a virtual machine from a recent snapshot, the restored copy may fail to boot properly. The VM may stop with a blue screen message which reads:

Your PC ran into a problem and needs to restart. We're just collecting some error info, and then we'll restart for you.

There are several guest-level issues that can cause a VM running Windows to not start successfully. Below are the most common causes and their corresponding solutions.

Common Causes and Solutions

1. Non-Quiesced Snapshots

One of the most frequent causes of a restored VM failing to boot is that the snapshot was not taken in a clean (Quiesced) state. A quiesced snapshot ensures that the VM's memory and disk I/O are in a stable state, making the restored VM more likely to boot successfully. Without quiescing, the snapshot could have captured an unstable or inconsistent state.

Solution:
  • Ensure that when snapshots are taken, they are quiesced. Quiescing allows the OS to pause I/O operations, flush memory, and ensure that no incomplete transactions are saved in the snapshot.
  • For future backups, enable the Quiesce Snapshots option when scheduling backups for Windows VMs. This feature ensures that the system is in a stable state before taking a snapshot.

2. Pending or Partially Installed Windows Updates

If Windows updates were partially installed or in progress when the snapshot was taken, the restored VM might experience issues booting due to an incomplete or corrupted update state.

Solution:
  • Boot the VM into Safe Mode and complete any pending updates.
  • You can also attempt to disable the Windows Update service temporarily to allow the VM to boot without applying incomplete updates. Once booted, manually re-enable and check for updates.
  • Review the Windows Update Logs using Event Viewer to identify any problematic updates that might need to be rolled back or reinstalled.

3. Driver Incompatibility or Missing Drivers

Sometimes, the VM's hardware configuration in VergeOS (e.g., disk controllers, network adapters) may differ from the original environment, causing issues with booting due to incompatible or missing drivers. This is especially common when restoring VMs from a different hypervisor.

Solution:
  • Boot the VM using Windows Recovery and attempt to repair the system automatically.
  • Verify that the appropriate Virtio or SCSI drivers are installed, especially if the VM is using Virtio interfaces for storage or networking.
  • If the issue persists, boot into Safe Mode and manually update the VM's drivers from the Device Manager.

4. Corrupted Boot Loader

If the Windows bootloader was corrupted in the snapshot, the restored VM will not boot properly. This could happen if the system was performing a critical task related to the boot process (like an update or disk operation) when the snapshot was taken.

Solution:
  • Use the Windows Recovery Environment (WinRE) to repair the bootloader: 1. Boot the VM using a Windows installation disk or recovery media. 2. Select Repair your computer > Troubleshoot > Advanced options > Startup Repair. 3. If Startup Repair doesn’t work, open a Command Prompt and run the following commands: 
    bootrec /fixmbr
bootrec /fixboot
bootrec /rebuildbcd
  • These commands will repair the Master Boot Record (MBR) and rebuild the Boot Configuration Data (BCD).

5. Hardware Configuration Changes

Changes to the VM’s hardware configuration, such as CPU count, memory allocation, or disk type, may cause instability or prevent the VM from booting.

Solution:
  • Verify that the VM’s hardware configuration in VergeOS matches the original configuration from when the snapshot was taken.
  • If you made any changes, such as increasing memory or changing the number of CPUs, try reverting to the original configuration to see if the VM boots properly.

Best Practice: Manage Windows Updates in Guest VMs

A guest VM running Windows OS, and experiencing an unexpected restart, is often found to be caused by the Microsoft Windows Update service being configured to automatically apply updates that frequently require a restart.

Recommendations: - Schedule snapshot creation during maintenance windows when Windows updates are not being applied. - Configure Windows Update settings to avoid automatic installations or reboots, especially on critical VMs. Instead, use a manual update process during scheduled maintenance periods. - Regularly review the Windows Update logs in Event Viewer to detect potential issues related to updates that could affect the stability of the VM.

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6

Workloads Failing to Migrate

Reasons That Workloads May Fail to Migrate

A workload is any process that is running on a node. Common workloads include Virtual Machines (VM), NAS Services, Networking, and Tenant Nodes.

The main reasons a workload fails to migrate from one node to another in the system are:

  • Insufficient available resources: There may not be enough resources (such as RAM) on the target node to run the workload you're trying to migrate. Check the amount of RAM consumed by the workload (VM or Tenant node), then review the resources available on the target node.

  • Pinned VM configuration: A VM may be pinned to a specific node. Review the VM’s settings and check the CPU Type setting. If the CPU Type is set to Host Processor, the VM will be unable to migrate. In this case, the VM must be powered off before it can be migrated successfully.

  • Tenant node migration issues: Tenant nodes may also face migration issues for the same reasons as listed above. Log into the Tenant User Interface, and check the following:

  • Verify that each Tenant node has sufficient available resources to host the migrating tenant workloads.
  • Verify that each Tenant VM is not configured with the CPU Type set to Host Processor.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Proper Power Sequence

Proper Shutdown Sequence for a VergeOS Environment

To power off a cluster (a collection of two or more nodes), follow these steps:

  1. Check any running workloads on each node of the cluster. Navigate to the node dashboard for each node and review the Running Machines section.
  2. If there are tenants running on any of the nodes, log into those tenant environments and gracefully shut down all running workloads.

  3. Power off all running workloads on each node, including VMs, tenant nodes, VMware backup services, and NAS services (if applicable).

    vNet Containers

    There is no need to manually stop any running vNet containers; they will be gracefully stopped automatically in the subsequent steps.

  4. After stopping all running workloads, navigate to the Cluster dashboard for the cluster you wish to power off.

  5. Select Power Off from the left-hand menu to begin shutting down each node in the cluster.
  6. Finally, navigate to System → Clusters and select Power Off in the left menu to power off the entire cluster.

IMPORTANT

If an environment contains multiple clusters, ALWAYS shut down the cluster containing the controller nodes (Node1 & Node2) LAST.

Proper Power-On Sequence for a VergeOS Environment

To properly power on a VergeOS environment, perform the following steps:

  1. Power on Node1.
  2. Wait 30 seconds to a minute.
    • This brief pause lets Node1 begin initializing before the rest of the cluster arrives.
  3. Power on all other nodes within the cluster. All other cluster nodes can be powered on immediately -- there is no need to delay between nodes.
  4. Multi-cluster environments: bring the controller cluster fully online before powering on additional clusters.

  5. The system will automatically perform integrity checks of each storage tier.

    Storage Tier Integrity Checks

    See vSAN Tier Status (Journal Walks) for information related to these automatic integrity checks (journal walks).

  6. Verify system health.

    • On the Main Dashboard, verify all nodes and vSAN tiers appear with a green status.
    • Check for any new alarms.

Power-On After an Uncontrolled Shutdown

Extra care should be taken when powering on a cluster after an uncontrolled shutdown (e.g. an abrupt power loss, UPS exhaustion, or facility failure). See Cluster Recovery After Full Power Outage for the full recovery procedure, including pre-power-on checks, vSAN integrity verification, and prevention guidance.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6