Skip to content

VM#

Force Power Off a VM Using the API

Overview

Key Points

  • Use the VergeOS API to force power off a stuck VM
  • Requires API/Swagger access
  • Process involves multiple API calls to ensure accurate targeting
  • Should only be used when normal power off methods fail

This guide explains how to force power off a non-responsive virtual machine (VM) using the VergeOS API when standard power-off methods are unsuccessful.

Prerequisites

  • Access to the VergeOS UI with administrative privileges
  • The name of the stuck/non-responsive VM
  • Basic understanding of API operations

Important

This procedure should only be used when standard power-off methods have failed. Forcing a VM to power off can lead to data loss or corruption if not used carefully.

Steps

1. Access the API Documentation

  1. Navigate to System in the VergeOS UI
  2. Click on API Documentation (also known as Swagger)

2. Locate the VM ID

  1. In the API interface, locate and expand the VMs table
  2. Click the blue GET button
  3. In the parameters section: - Use filter name eq your_vm_name to find a specific VM
  4. Click Execute
  5. Note the Machine number from the response

3. Get Machine Status ID

  1. Navigate to the machines table
  2. Click the blue GEt /machines/{id}
  3. In the parameters: - Set id to the Machine number from the previous response - Set fields to status
  4. Click Execute
  5. From the response, note the status value

4. Verify Machine Status

  1. Go to the machine_status table
  2. Click the blue GET /machine_stats/{id}
  3. In the parameters: - Set id to status_number (using the status value from step 3) - Set fields to most
  4. Click Execute
  5. Verify this is the correct VM by checking: - Number of cores - RAM allocation - Status information - Machine number

5. Force Power Off

  1. In the machine_status table, click PUT
  2. Enter the status number as the id resource id
  3. In the request body, enter the following JSON:
{
    "running": false,
    "migratable": true,
    "status": "stopped",
    "state": "offline"
}
  1. Click Execute

6. Verify Power Off

  1. Return to the VergeOS UI
  2. Verify that the VM shows as powered off

Troubleshooting

Common Issues

  • If the VM doesn't show as powered off after the API call, wait a few minutes for the status to update
  • If the status doesn't update, verify that all IDs were correct in the previous steps
  • In case of errors, check the API response for specific error messages

Additional Notes

  • Always document the VM's ID and status values before making changes
  • Consider taking a snapshot of the VM before forcing power off if possible
  • Monitor the VM after forcing power off to ensure it starts properly when needed

Feedback

Need Help?

If you do not feel confortable with this process, please reach out to our support team for assistance.

Document Information

  • Last Updated: 2024-01-28
  • VergeOS Version: All
  • in VM
  • 2 min read

Emulating USB Devices in VergeOS

Overview

Key Points

  • Create emulated USB devices for VMs
  • Enables legacy application support
  • Allows hotplugging storage for driver installation
  • Requires specific VM configuration

Prerequisites

Before creating an emulated USB device, ensure your VM meets these requirements:

  • VM settings:
  • Allow Hotplug must be enabled
  • Machine Type must be 9.0 or higher
  • VirtIO drivers installed in the guest OS

Machine Type Changes

Before modifying a VM's machine type, create a short-term snapshot (24-hour expiration) to enable rollback if needed.

VirtIO Driver Installation

  • Linux distributions typically include VirtIO drivers
  • For Windows, drivers are available in VergeOS custom ISOs or can be downloaded from: VirtIO Drivers

Steps to Create USB Device

  1. Access VM Settings - Navigate to the VM dashboard (Main Dashboard > Machines > Virtual Machines) - Double-click your target VM

  2. Create New Drive - Click Drives in the left menu - Select New from the left menu

  3. Configure USB Device - Set Media to either Disk or Clone Disk - Set Interface to USB - Optional: Enter a custom name (system will auto-generate if blank) - If using Clone Disk, select the appropriate *.raw file

  4. Enable Device - Click Submit to create the device - Select the new USB device from the drive list - Click HotPlug in the left menu

For additional drive configuration options, see: VM Drives

Troubleshooting

Common Issues

  • Problem: Device stays in "hot plugging" status
  • Solution:
    1. Check VM dashboard logs for errors
    2. Verify all prerequisites are met
    3. Try restarting the VM if settings were recently changed
  • Problem: Device shows as offline
  • Solution: Ensure hotplug is enabled and VirtIO drivers are installed

Document Information

  • Last Updated: 2024-11-19
  • VergeOS Version: 4.13

Device Passthrough Advanced Configuration (Manual Creation/Editing of Resource Rules)

Although allowing auto-generation of resource rules (e.g. when you select a device and use the Make Passthrough menu option) is easiest and usually recommended, there may be situations where it may be useful to manually create a resource rule or to modify an auto-generated resource rule.

It is important to read and be familiar with PCI Passthrough Risks and Precautions before making passthrough configurations.

Manually Create a New Resource Rule

  1. From the main dashboard, click Resources.
  2. Click Rules (ui card or on the left menu).
  3. Click New on the left menu.
  4. Provide a Name for the Rule; it is recommended to use a descriptive name can be helpful in future administration.
  5. Select the Resource Group to which the resource rule will apply.
  6. Select a specific Node or select --None-- to apply the rule to all nodes.
  7. Select the Type (PCI, USB, SR-IOV, or NVIDIA vGPU).
  8. Leave the default value set to --None-- in the field labeled Automatically created based on PCI Device.
  9. Configure device filters as desired; filter fields will vary depending on the device type selected; see below. (Advanced Entry 1 option also available)

Information on installed PCI devices, for use in filters, you can use the PCI devices listing: from the Main Dashboard, navigate to the Resources -> PCI Devices. To show additional fields, right-click in the heading section to select from the full list of available columns that can be displayed.

Edit an Existing Resource Rule

  1. Navigate to the Associated Resource Group dashboard (Main Dashboard > Resources > Groups > double-click the particular group).
  2. In the Rules section, locate and click the desired resource rule.
  3. Click Edit on the left menu.
  4. Node selection and PCI Filters can be modified as needed. (Advanced Entry 1 option also available)

  1. The Advanced Entry section allows you to manually input filter syntax rather than using the filter entry fields. Generally, it is preferable to allow system-generated syntax based on your filter field selections. ↩↩↩↩

How to Configure a VM Export Volume

In VergeOS, you can create a volume specifically for exporting selected virtual machines (VMs). This export volume can then be used with third-party backup software to back up the VMs. The volume contains VM snapshots from the most recent export.

Steps to Configure a VM Export Volume

1. Preparing the VMs for Export

  1. Edit the VMs you want to export:
    • Navigate to the VM settings and enable the option for Allow Export.

2. Setting Up the NAS Service

To host the VM export volume, you will need to create and configure a NAS service:

  1. Navigate to NAS > NAS Services.
  2. Click New.
  3. Provide Name, Hostname, TimeZone, and Networking for the NAS service.
  4. Click Submit to initialize the NAS service.

3. Starting the NAS Service

Once the NAS service is created:

  1. Select the NAS service from the list.
  2. Click Power On to bring the NAS online and prepare it to host the export volume.

4. Creating a NAS User

You’ll need to create a user to access the NAS:

  1. Navigate to NAS > NAS Services > The NAS Service you created.
  2. Click NAS Users > New.
  3. Provide a username and password.

5. Creating a New Volume for VM Export

  1. Navigate to NAS > Volumes > New
  2. Provide a Name for the volume and choose VergeOS VM Export as the filesystem type.
  3. Select the appropriate NAS service to host the export volume.
  4. Make sure Quiesced is checked.
  5. Adjust the number of exports to store.
  6. Click Submit.

6. Starting the VM Export

  1. Under Export VMs, select Start to initiate the VM export process.
  2. Confirm by clicking Yes at the prompt.

Setting up a CIFS Share for the Exported Data

To access the exported VM snapshots, set up a CIFS share on the NAS volume:

  1. Create a CIFS Share: - Navigate to NAS > CIFS Shares > New. - Select the export volume you created earlier as the target volume. - Provide a Share Name and assign the NAS user you created in step 4 to access the share.

  2. Access the Share: - Browse to \\IPorDNSnameoftheNAS\CIFSShareYouCreated. - Use the NAS user credentials when prompted.

!!! note "For Windows Users" You may need to edit the Group Policy (GPO) or modify the Windows Registry to connect using the Guest account if Guest mode is enabled.

Automating the VM Export

You can schedule regular exports by setting up an automated event:

  1. Navigate to the VM Export Volume.
  2. Select Events > New.
  3. Select *Scheduled as the Task Triggered by.
  4. Provide a Name for the task.
  5. Select VM Exports for section.
  6. Ensure VM Exports as the volume you created previously.
  7. Configure the event to trigger the export at your desired intervals.
  8. Click Submit

By following these steps, you'll have a properly configured VM export volume that can be used with third-party backup solutions, along with automated export scheduling.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6
  • in VM
  • 1 min read

How to Resize a Virtual Disk Drive

Note

Drives can only be increased in size; they cannot be reduced. Verify whether your guest OS supports resizing without a power cycle, particularly for Virtio-SCSI drives.

To resize a virtual disk drive within a VM, follow these steps:

  1. From the VM Dashboard, click Drives in the left menu.
  2. Select the drive to be modified.
  3. Click Edit in the left menu.
  4. Modify the drive size as desired.
  5. Click Submit.

Important Notes

Note

If the VM configuration allows hot-plugging, the disk interface is set to Virtio-SCSI, and the guest Operating System (OS) supports it, the drive size can typically be increased without power cycling the VM.

Warning

Drives cannot be reduced in size. While partitions may be resized inside the guest OS, the disk drive itself cannot be shrunk.

Warning

Modifications to drive size will most likely require corresponding changes within the guest Operating System to utilize the newly added space.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Changing Resolution on a UEFI VM

In a UEFI-based virtual machine (VM), the screen resolution is controlled by the OVMF (Open Virtual Machine Firmware) Platform Configuration. By default, only the resolution configured in the platform settings is available to the VM. If you need to change the display resolution, follow the steps outlined below.

Steps to Change the Screen Resolution

  1. Reboot the VM while connected to the console (through the VergeOS UI or any console manager you're using).

  2. As the VM starts, press ESC to enter the UEFI Settings menu.

  3. Once in the UEFI menu, navigate to Device Manager.

  4. In Device Manager, select OVMF Platform Configuration.

  5. Choose your desired Preferred Resolution from the list of available options.

  6. Save the configuration changes and reboot the VM for the new resolution to take effect.

Notes and Considerations

  • The resolution options available depend on the firmware and the display capabilities of the guest operating system.
  • If you are still experiencing resolution issues, ensure that your guest OS has the necessary display drivers installed.
  • UEFI settings may vary slightly depending on the VM and configuration, so some steps may look a little different in certain environments.

By following these steps, you can successfully adjust the screen resolution for your UEFI-based VM.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

VM Disk Discard

The Discard option on a VM Disk in VergeOS is responsible for managing unused storage blocks by issuing TRIM or DISCARD commands. When Discard is enabled, the system automatically frees up unused blocks, helping to maintain efficient storage usage on the vSAN.

Enabling or Disabling Disk Discard

When creating or editing a VM disk, you have the option to enable or disable Discard. By default, Discard is enabled, and it is highly recommended to leave it enabled for optimal storage efficiency. When Discard is disabled, deleted files on the virtual disk do not immediately free up the corresponding storage, leading to potential overuse of storage resources.

Here’s what happens when Discard is enabled:

  • The system periodically identifies and frees unused disk blocks.
  • vSAN storage remains optimized, as unused blocks are reclaimed.
  • Disk space usage more accurately reflects the actual data stored on the VM.

Only disable Discard for performance reasons

Disabling Discard can lead to storage inefficiencies and should only be done for specific performance-related reasons. Always consult with VergeOS Support before disabling this feature.

Why Use Disk Discard?

  • Efficient Storage Management: When a file is deleted from a VM, the unused blocks are immediately flagged as free, allowing the vSAN to reuse that space for other data.
  • Improved Disk Performance: Discard operations help maintain a clean and optimized storage system, reducing overhead from managing fragmented or unused blocks.
  • Space Reclamation: Particularly in environments with high storage churn (i.e., frequent file creation and deletion), Discard ensures that space is consistently reclaimed, avoiding storage bloat.

When to Disable Disk Discard

In rare circumstances, you may need to disable Discard to improve performance, particularly on certain workloads where the overhead of issuing TRIM/DISCARD commands may cause delays or slowdowns. Before making this change, it's critical to understand the trade-offs in terms of storage efficiency and consult with VergeOS Support for further guidance.

By keeping Discard enabled, you ensure that VergeOS optimizes storage for virtual machines, maintaining high efficiency and minimizing wasted space.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Bypassing TPM Requirements in Windows 11

How to Bypass Windows 11's TPM Requirement Using the Registry Editor during the installation

This only applies to versions of VergeOS previous to 4.11 (Atria).

If you have the Windows 11 install disk or ISO, you can bypass the Windows TPM and RAM requirements by making registry changes during the install.

This method only works on a clean install and does not allow you to bypass the requirement for at least a dual-core CPU.

  1. Boot off of your Windows 11 install disk. If you don't have one, one can be downloaded from here. The first screen should ask you to choose the language of your install (which should be correct). tpm-1.png
  1. Press SHIFT + F10 to launch the command prompt. tpm-2.png
  1. Type regedit and hit Enter to launch registry editor. tpm-3.png
  1. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\Setup. tpm-4.png
  1. Create a new registry key under Setup and name it LabConfig. To create a registry key, right click in the right window pane and select New->Key. Then enter the key name. tpm-5.png
  1. Within LabConfig, create 2 new DWORD values called BypassTPMCheck and BypassSecureBoot and set each to 1. To create a new DWORD value, right click in the right window and select new DWORD (32-bit) Value then name the key, double-click to open it and set it to 1. If you also want to bypass the RAM requirement, add a DWORD values for BypassRAMCheck. tpm-6.png
  1. Close regedit and exit the command prompt. You can now continue with your Windows 11 installation as normal.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.11

Troubleshooting VM Network Connectivity Issues

Before you begin, verify if other virtual machines in the environment can access the internet. If no other machines can, there may be a network issue upstream of the VergeOS platform that is preventing access to the outside world. If other VMs are still able to access the internet, the most likely cause is that a configuration step was missed.

The following are the most common configuration mistakes that cause network issues:

  • Missing NIC Configuration: The newly created VM may not have a NIC configured. To verify this, review the NICs section of the VM dashboard. Ensure at least one NIC is present. If not, add one.
  • Incorrect Network Assignment: The VM's NIC may be connected to the wrong network. In the NICs section, ensure at least one NIC is present with the status set to Up, and verify that the correct network is listed. If not, edit the NIC and assign the correct network (one used by a VM with internet access).
  • Improper IP Configuration: The VM might not have a properly configured IP address. Typically, this is resolved at the guest level. Refer to the guest operating system’s documentation to ensure the NIC is detected, installed (with drivers), and configured correctly.
  • Virtio Drivers Not Installed: If the Virtio drivers are not installed, the NIC may not function properly. For instructions on installing Virtio drivers, refer to the Product Guide.

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6
  • in Tenant, VM
  • 2 min read

How to Share a VM into a Tenant

VergeOS provides an easy way to share a virtual machine (VM) image from a parent environment into a tenant located beneath the current environment. Follow these steps to accomplish the task:

Steps to Share a VM

  1. Navigate to the VM Dashboard: - Go to the VM dashboard of the VM you want to move into a tenant.

  2. Gracefully Power Down the VM: - It is best practice to gracefully power down the VM using the guest operating system's best practices before moving it.

  3. Take a Snapshot: - In the VM dashboard, expand Snapshots in the left navigation menu to access the snapshot commands. - Click Take Snapshot to launch the Machine Snapshot creation screen.

  4. Complete the Snapshot Creation: - At the Machine Snapshot creation screen, fill in the required fields:

    • Machine: The virtual machine you are moving.
    • Name: Provide a unique name for the snapshot.
    • Expiration Date: Set the date/time when the snapshot will automatically be deleted.
    • Click Submit to create the snapshot.

  5. Share the VM Snapshot: - After clicking Submit, you will be taken to the dashboard of the newly created snapshot. - From this view, click on Share VM in the left navigation menu to launch the Shared Objects creation screen.

  6. Create the Shared Object: - At the New Shared Objects creation screen, fill in the required fields:

    • Name: Name the snapshot of the VM something unique.
    • Type: Select Virtual Machine.
    • Snapshot: This should match the name provided during the snapshot creation.
    • Recipient: Select the tenant where you want to share the VM.
    • Click Submit to create the shared object.

  7. Access the Tenant Environment: - Using a web browser, navigate to the tenant environment where the snapshot object was shared. - Log in with the proper authentication credentials.

  8. Create a New Virtual Machine in the Tenant: - In the tenant environment, navigate to Machines > Virtual Machines, and click New to begin creating a new virtual machine.

  9. Import from Shared Objects: - At the New Virtual Machine creation screen, under Select Type, choose -- Import from Shared Objects --. - In the Selections Available section, select the shared object created earlier, then click Next.

  10. Complete the Virtual Machine Settings:

    • On the Virtual Machine Settings screen, complete the required fields:
    • Shared Objects: Select the shared object created earlier.
    • Click Submit to create the new virtual machine in the tenant.

---

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

How To Efficiently Recover a Single VM from a Remote Cloud Snapshot

This is easily achieved for systems that are successfully configured to send a cloud snapshot to a remote destination tenant and have the Sync-back configured properly.

Note

For more information on configuring Sync-back, refer to the inline help Category titled 'Site Sync' under the section labeled 'Syncing Back'.

Recover a Copy of the VM on the Backup Side

  1. On the destination side (where the snapshots are sent), review all of the received remote snapshots and locate the desired snapshot that closely matches the date/time. This is accomplished from the Main Dashboard > System > Cloud Snapshots.
  2. Once the snapshot is located, select (check) the Cloud Snapshot in the list of available snapshots, and on the left navigation menu, select View VMs.
  3. Wait while the list of available VMs loads. This can take a few minutes.
  4. Once the list of virtual machines contained within the cloud snapshot loads, select (check) the desired VM to recover and then select Recover on the left navigation menu.
  5. The Recover VM Snapshot option appears. It is recommended to rename the VM to reflect the date the snapshot was taken. For example, Domain Controller recovered on 09012022.
  6. Wait while the VM is recovered.


Create a New Cloud Snapshot on the Backup Side

  1. On the destination side (where snapshots are sent), create a new cloud snapshot that will contain the newly recovered VM from the steps above. This is done from Main Dashboard > System > Cloud Snapshots.
  2. On the Cloud Snapshots page, select New on the left navigation menu.
  3. The New Cloud Snapshot creation page will load. Name the snapshot. It is recommended to name it something that is easily referenced in future steps, such as Recoveryon09012022.
  4. Set the expiration date to something logical. It should not exist forever but should be far enough into the future to allow time for the transfer back to the original system.
  5. After setting the name and expiration, select Submit to create the snapshot.


Request a Sync-Back on the Original/Source Side

  1. On the origin (sending) side, navigate to the configured outgoing site sync. This is done from Main Dashboard > Backup/DR > Outgoing Syncs, and then double-clicking into the configured outgoing sync.
  2. From the outgoing sync dashboard, click Refresh Remote Snaps on the left navigation menu. This will query the remote side for any new snapshots and list them. It should detect the snapshot created in the steps above.

  3. Once the newly created snapshot is detected, it will be listed under the Remote Snapshots section. Find the snapshot and click on the Request to Download icon to the far right of the listed snapshot.

  4. The Request Cloud Snapshot menu will load. Set a reasonable expiration date for how long the recovered snapshot will be retained on this system, and click Submit.

  5. The system will load the Sync-Back / Incoming Sync. The length of time it will take to transfer the snapshot back can vary greatly depending on several factors, including bandwidth speed and the size of the data to transfer.

  6. Wait while the synchronization completes. Once finished, a new entry will appear in the log section.

    Example

    Sync for 'Morning_20220901' complete (4m 17s) checked [78.1GB] scanned [1.76TB] sent [5.16GB] sent net [2.16GB] dirs [210] files [641] updated [31]

  7. At this point, the snapshot has been successfully transferred back to the original location. Administrators can now perform standard recovery tasks on the VM contained within the snapshot.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6
  • in VM
  • 2 min read

Making a Non-Persistent VM

How to Create a Non-Persistent VM on Reboot

A Non-Persistent VM reverts to its original state after a reboot, discarding any changes made during the session. This is useful for VDI (Virtual Desktop Infrastructure) environments where the system should reset after each use.

Steps to Create a Non-Persistent VM:

  1. Navigate to the VM dashboard.
  2. Shutdown the VM by selecting Actions > Power Off or using the Power button:

Note

This ensures the data is in a good state for cloning.

  1. Click the Copy button next to the main disk on the VM. nonpersistent-2.png
  2. Change the Media Type to Non-Persistent and click Submit at the bottom.
  3. Click the Edit icon edit icon pencil for the original Disk Media Type.

Note

The new disk will show a Media Type of Non-Persistent. Any changes made to this disk will be reverted upon a reboot of the VM. 6. Uncheck the Enabled checkbox. 7. Start the VM by selecting Power On from the left-hand menu or clicking the Play button:

This will boot the VM using the non-persistent disk. The disk is fully writable during the session, but all changes will be discarded upon reboot, reverting the VM back to its original state.

Do not delete the original disk. It will not take up additional space due to Deduplication.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6
  • in VM
  • 2 min read

How to Isolate a Virtual Machine

Isolating a virtual machine (VM) can be done in several ways, depending on the specific requirements of the environment. Below are two common methods for isolating a VM within VergeOS.

Remove the Attached Network from the VM

This method essentially simulates unplugging a network cable from the VM, making it function without any external connectivity. It’s suitable when the VM doesn’t need to communicate with any other network.

Steps to Remove the Network:

  1. Navigate to the Virtual Machine Dashboard.
  2. Click on NICs in the left navigation menu to access the VM's virtual network adapters.
  3. Select the NIC you want to edit and click Edit from the left navigation menu.
  4. In the NIC configuration window, use the drop-down list to change the Network from its current value to --None--.
  5. If the VM has multiple NICs, repeat this process for all active/enabled NICs.

By removing the network from all NICs, the VM will no longer have network access.

Create a New Internal Network

If the VM requires connectivity but still needs to be isolated from other networks, creating a new internal network with no other VMs connected is the preferred solution.

Steps to Create an Internal Network:

  1. From the VergeOS dashboard, navigate to Networking and create a new Internal Network.
  2. Set a Default Gateway for outbound access if needed.
  3. After the internal network is created, return to the VM dashboard and update the NIC to attach the VM to the newly created internal network.

For more detailed instructions on creating internal networks, refer to the VergeOS inline help under Networking in the Internal Networks section.

This method allows the VM to have restricted network access while still providing outbound connectivity through the internal network.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Exporting Virtual Machines from VergeOS

Exporting a virtual machine (VM) from VergeOS allows you to download the VM’s disk in .raw format, which is compatible with many hypervisors. This guide outlines the steps to export a VM and download its virtual disk.

Steps to Export a Virtual Machine

  1. Log in to the VergeOS platform and navigate to the dashboard of the virtual machine you wish to export.
  2. From the left-hand menu, click on Drives to view a list of the virtual disk drives attached to the VM.
  3. Select the drive you want to export, then choose Download from the left-hand menu. The virtual disk will be downloaded in .raw format.
    • The .raw disk format is widely supported by many modern hypervisors.
  4. The system will automatically begin downloading the disk image.
  5. Once the download completes, refer to the documentation for your destination hypervisor for instructions on how to import, upload, or convert the .raw disk image.

Important Considerations

  • .raw format: The exported VM drive is downloaded in .raw format. This format is compatible with most hypervisors, but some may require converting the image to another format such as .qcow2 or .vmdk. Check your destination hypervisor’s documentation for conversion tools or instructions.
  • File size: Depending on the size of the virtual disk, the download can be large. Ensure you have enough disk space available on the system where you are downloading the file.

By following these steps, you can successfully export and download virtual machines from VergeOS for use in other environments.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Reasons for Unexpected / Unexplained vSAN Growth

There are several reasons for the vSAN to start growing at a rate faster than anticipated. Administrators should first determine when the unexplained growth occurred by reviewing the vSAN Tiers' growth history, and then assess potential areas for unexpected growth.

Review vSAN Tiers for Growth History

To isolate unexplained growth, it is important to narrow down when the growth increased exponentially. Using the steps below, administrators can review storage growth and visualize normal growth from daily operations versus spikes in growth, which are typically unexpected.

  1. Navigate to the vSAN Tiers from the Main Dashboard. If vSAN Tiers is not present, then this environment is a tenant of a parent system, and the vSAN tier needs to be examined at the parent system.
  2. Open the vSAN Tier with unexpected growth (for example, vSAN Tier 0).
  3. On the left navigation menu, click on History.
  4. A new menu will appear showing history in various graphs. Modify the filter period to isolate any growth on this tier. - It is recommended to start with a custom filter of 1 day and review the Storage Usage graph.

Things to Note:

  • If you see dips and spikes every hour or once a day, this is likely the result of snapshots falling out of retention (old ones expiring, new ones being created). Note whether the total storage consumed at the start of the day is nearly equivalent to the end of the day. If so, expand the custom filter to a week.
  • When reviewing by week, check if the total storage consumed at the start of the week is similar to the end. If, for example, the growth is roughly 10%, repeat for the previous week. If the weekly growth percentage is consistent, this represents your average weekly growth rate, which can help plan for hardware expansion.
  • Filter the current month and check for any sudden spikes in storage consumption on the Storage Usage graph. Click and drag over the time in question to zoom in on the data, and hover over the graph for specific date/time information.

vsan_unexpected_growth.png

Possible Reasons for Storage Increase

Several areas in the VergeOS platform may contribute to unexpected storage growth. Common areas to check include:

  • Cloud Snapshots:
  • Navigate to System > Cloud Snapshots.
  • Are any being held past their expected expiration time?
  • Are there snapshots without a Snapshot Profile? These may have been taken manually. Investigate when and why they were taken.

  • Are any snapshots set to "Never Expire"? This can lead to large data consumption over time.

  • Virtual Machines (VMs) Snapshots:

  • Navigate to the Machines Dashboard. The Snapshots count box shows the number of machine-level snapshots present. Click this box to list all VM snapshots and their creation date/time. Review if any can be removed.

  • Navigate to Machines > Virtual Machines. Sort by the Snapshot Profile column to identify VMs with machine-level snapshots. These are included in the recurring cloud snapshots, so review whether individual snapshots are necessary or if they can be removed.

  • VMWare Backup Jobs:

  • Navigate to Backup/DR > VMware Services and review each VMware Service instance for Backup Job history.

  • On the left menu, click Backup Jobs to review each specific instance. Check the Expires column for each backup and review if it can be removed.

  • Media Images:

  • Navigate to Media Images and sort by Modified. Check if any upload dates/times match the unexplained growth period.

  • Review whether media images, especially other hypervisor formats (e.g., .ova or .vhdx), can be removed.

  • Incoming Site Syncs:

  • Navigate to Backup/DR > Incoming Syncs. Open each Incoming Sync dashboard and check the Received Snapshots count. Investigate the source (origin) site for increased storage matching the timeframe.

  • Tenant Storage:

  • Navigate to Tenants > Each Tenant Dashboard.
  • Review Total Storage Used by clicking on History in the left menu. Follow the same process listed above to review growth history.
  • If unexpected growth is found, investigate within the tenant for the possible causes of storage increase (as listed above), and within any sub-tenants if applicable.

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6

Windows - Time Shift

Description of The Issue

VergeOS Virtual Machines (VMs) running Windows OS may experience ‘time shift,’ where the guest OS will periodically adjust the time to an incorrect value. This is caused by the Windows OS, which expects time from the physical motherboard to be in local time (RTC) instead of Coordinated Universal Time (UTC).

VergeOS provides time in UTC, which has become the industry standard as it compensates for Daylight Saving Time (DST) changes. The guest OS will automatically adjust the time when comparing its clock value against an authoritative time source because it assumes the time provided by the hardware is local instead of UTC. This comparison causes periodic discrepancies because the guest OS is unable to adjust the physical node clock to match what it perceives as the correct time.

VergeOS Configuration Option: RTC Base

RTC Base is an individual VM setting that allows VergeOS administrators to set the time provided to the OS as either local or UTC. The value can be found when editing any VM.

rtcbase-utc-screenshot.png

With this configuration setting, administrators can granularly control every machine, though it is important to understand the expected behavior of each option.

Local Time

Setting RTC Base to Local Time will pass the time from the physical nodes to the virtual guest OS. This emulates the legacy behavior that Windows expects. When Windows compares the local time clock against an authoritative time service, Windows will adjust the time within the guest OS based on the time zone defined in the Windows configuration. This can result in unexpected behavior.

Things to consider with local time are:

  • The physical node time zone will be presented the same to any guest OS. If VergeOS is hosting VMs for different time zones, each Windows VM will perceive local time as the same value.
  • The physical node time and the guest OS will require proper configuration to avoid issues when Daylight Saving Time (DST) starts or stops each year. RTC clocks will need to be adjusted through software. Historically, issues have arisen when guidelines for DST have changed.

UTC Time

Setting RTC Base to UTC will pass the time from physical nodes into guest VMs as coordinated universal time. This is the industry standard for modern software applications that handle time.

When using the UTC setting in VergeOS, Windows VMs should be configured to recognize that time is presented as universal. For most Windows operating systems, the adjustment is made in the Windows Registry by adding a value to recognize “RealTimeIsUniversal.”

For 64-Bit Operating Systems

Under the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation, there needs to be a REG_QWORD entry with the following values:

  • Name: RealTimeIsUniversal
  • Value data: 1

After adjusting this setting, administrators will need to completely Power Off the VM and then Power On the VM before the change takes effect.

Important

When making software adjustments to guest OS or applications, administrators should check with production documentation for that software, including the latest KB articles, updates, and release notes.

Additional Reading on this Topic

Document Information

  • Last Updated: 2024-08-29
  • VergeOS Version: 4.12

Reasons a Windows VM Restarted Unexpectedly

Reasons a Windows VM Restarted Unexpectedly

If you have a Windows VM that has recently restarted unexpectedly, the first place to begin is to review the logs from the VM dashboard.
You can reach the log viewer by navigating from the Main Menu > Machines > Virtual Machines > the name of the VM you are investigating > Logs.

From the Log view, search through (or filter the results) under the Message section. Several status messages will indicate why the VM stopped running. The following are the most common and a brief explanation of each:

  • Message: VM action 'kill' sent
    This indicates that a VergeOS user issued a Kill Power command from the VergeOS interface. The log will indicate the user under the Source column.
    In this scenario, consult with the system user to determine the reason for issuing this command.

  • Message: VM action 'poweroff' sent
    This indicates that a VergeOS user issued a graceful Power Off command from the VergeOS interface. This command successfully interacted with the Guest OS ACPI to gracefully stop the VM. The log will indicate the user under the Source column.
    In this scenario, consult with the system user to determine the reason for issuing this command.

  • Message: VM has shutdown
    This indicates that the shutdown command was issued from inside the guest operating system directly.
    In this scenario, consult with the guest operating system logs to determine the reason for the shutdown command.

  • Message: VM has reset
    This indicates that the restart command was issued from inside the guest operating system directly.
    In this scenario, consult with the guest operating system logs to determine the reason for the reset command.

Best Practice

A guest VM running Windows that experiences unexpected restarts is often found to be caused by the Microsoft Windows Update Services being configured to automatically apply updates that require a restart.
To investigate this further, consult with Knowledge-Base articles about the particular version of the Windows OS.
One of the best places to start investigating at a Windows Guest level is using Windows Event Viewer and reviewing the Windows Update logs for more information.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Loading Virtio Drivers in Windows Recovery Console

If a guest VM running any version of Windows OS with the Virtio-SCSI disk drivers installed is booted into Recovery Console mode, the operating system partition may not be visible as the drivers are not yet loaded.

Prerequisites

Before starting with recovery work on a guest OS, ensure that the administrator has:

  1. Access to the VM, including tested/known working credentials.
  2. The Virtio drivers ISO loaded and available in the VergeOS environment under Media Images.
  3. The Windows OS installation ISO available in the VergeOS environment.

Requirements

To load Windows drivers from a Windows Guest OS recovery console, administrators will need:

  1. VergeOS UI platform access to the environment running the Guest OS.
  2. A copy of the (latest) Virtio disk drivers in ISO format.
  3. A copy of the Windows operating system ISO (example: Windows boot disk).

Starting the Windows Recovery Console

  1. Login to the console and follow the Windows installation prompts to navigate to the Recovery Console and Command Prompt.
  2. Once at a command prompt, type the following command to list available disks on the VM:
wmic logicaldisk get deviceid, volumename, description

Press ENTER.

This will return a list of available disks. Typically, D: will be the EFI boot volume, E: will be the CD installation media, and X: will be the active, booted Windows recovery session.

The Virtio-SCSI disks are likely not present, and the drivers need to be loaded for them to appear.

Installing the Virtio-SCSI and Storage Drivers

From the Command Prompt in the Windows Recovery Console, follow these steps:

  1. To load the Virtio-SCSI Storage drivers, type:
Drvload e:\vioscsi\2k16\amd64\vioscsi.inf

Press ENTER.

  • E: refers to the drive letter where the Virtio ISO is loaded.
  • 2k16\amd64 refers to the path to the Virtio driver based on OS. Browse the ISO image for paths to other OS versions if needed.
  1. To load the Virtio Storage drivers, type:
Drvload e:\viostor\2k16\amd64\viosstor.inf

Press ENTER.

  • E: refers again to the drive letter where the Virtio ISO is loaded.
  • 2k16\amd64 refers to the path to the Virtio driver based on OS. Adjust as necessary.

Verifying the Disk Mount

After loading both drivers, the disk should mount, typically as an F: drive.

To verify this, run the command again:

wmic logicaldisk get deviceid, volumename, description

Press ENTER. You should now see the newly mounted disk.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Windows Disks Showing Offline After Restarting

How to Correct Disks showing Offline after a Windows VM is Restarted

After importing a VM into the VergeOS platform, some (virtual) disks may not come online after a restart of the guest VM.

Beginning in Windows 2008, Microsoft added a setting for the default state of additional (non OS) disk drives. Occasionally when restarting, Windows may detect a hardware change, which is more likely if the VM was imported from an alternate hypervisor. When windows detects a hardware change, it may not bring secondary virtual disks online automatically. To change this to the recommended setting run the following Windows PowerShell:

Set-StorageSetting -NewDiskPolicy OnlineAll

Additionally, this can be done from the command line using diskpart or from Disk Management.

Document Information

  • Last Updated: 2024-08-10
  • vergeOS Version: 4.11

UEFI Disks as Boot Devices Will Not Boot Properly

After importing VM images leveraging UEFI disks as boot devices, sometimes the VM will not boot properly inside the VergeOS platform. To resolve this, tweaks to boot order/options need to be made.

Here is a list of suggested solutions for issues with imported VMs not booting because of guest UEFI BIOS settings:

Solution 1

  1. From a fresh import of the VM (before trying to power it up inside VergeOS), edit the VM in Verge and enable the UEFI Boot option.
  2. Power on the VM.
  3. Hit ESC within 5 seconds to get into the VM BIOS (you can edit the VM settings in Verge to increase the boot timer if necessary).
  4. Enter the BIOS and navigate to Boot Manager Options (this may vary depending on the BIOS).
  5. Change the selected boot device to the Verge IO device.
  6. Exit the UEFI BIOS.
  7. Reboot the VM.

Solution 2

  1. From a fresh import of the VM (before trying to power it up inside VergeOS), edit the VM in Verge and enable the UEFI Boot option.
  2. Power on the VM.
  3. Hit ESC within 5 seconds to get into the VM BIOS (you can edit the VM settings in Verge to increase the boot timer if necessary).
  4. Enter the BIOS and navigate to Boot Manager Options (this may vary depending on the BIOS).
  5. Disable Secure Boot as an option.
  6. Exit the UEFI BIOS.
  7. Reboot the VM.

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6
  • in VM
  • 2 min read

How to TRIM your Drives

After importing a virtual machine from another hypervisor, sometimes the free space available inside the virtual machine does not match the free space reported to the VergeOS platform. This discrepancy is often due to the virtual disk being thick-provisioned from the VM source, making VergeOS unaware of the unused disk space. To resolve this, a TRIM/UNMAP operation needs to be performed on the virtual disk from within the virtual machine.

Prerequisites for Running a TRIM Command

  1. Edit the virtual drive(s) in question in the VergeOS UI and ensure Discard is enabled.
  2. Ensure the virtual drive(s) is using a drive type of virtIO-SCSI or SATA.
  3. Ensure the virtual drive(s) is assigned to a Solid State Tier (usually tier 1-3).

Trimming a Windows Drive

To perform a manual TRIM operation in a Windows environment, follow these steps:

  1. Launch PowerShell as an admin user.
  2. From the PowerShell prompt, type the following command:

Optimize-Volume -DriveLetter YourDriveLetter -ReTrim -Verbose
Example:

Optimize-Volume -DriveLetter E -ReTrim -Verbose
3. Press Enter and wait for the command to complete.

As the TRIM operation progresses, you can watch the reported free space in the VergeOS dashboard increase as the blank data on the volume is removed.

If this does not resolve the issue, TRIM may not be enabled. To check if TRIM is enabled:

  1. Run the following FSUTIL command:
fsutil behavior query disabledeletenotify

If the value is 1, TRIM is not enabled on the drive.

  1. To enable TRIM, run:
fsutil behavior set disabledeletenotify 0

After enabling, rerun the TRIM commands.

Trimming a Linux Drive

Newer Linux distros have TRIM enabled by default via a systemd service or a cron job. To check if automated TRIM is enabled, follow these steps:

  1. Ensure the prerequisite steps from above are complete.

Example: Ubuntu Server

  1. Launch a terminal.
  2. Enter the following commands to check the status:

  • Check TRIM Timer/Schedule Status:

    sudo systemctl status fstrim.timer
    - Check TRIM Service Status:

    sudo systemctl status fstrim

If TRIM is enabled, an operation will run at the next scheduled time. If TRIM is not enabled, you can run a manual TRIM using:

sudo fstrim -av

Info

It is recommended to enable automatic TRIM to ensure that data usage is reflected accurately between VergeOS and the guest OS.

To enable automatic TRIM, run:

sudo systemctl enable fstrim.timer

For more information on fstrim, visit the man page.

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6

Windows is Unable to Detect a Virtual Disk Drive

Virtual Disk Drive not detected in Windows

In most cases this is because the disk drive Interface Type is set to virtIO-SCSI (which is the default value) on any virtual machine that is being created. If that is the case, you will need to load the virtIO drivers during the Windows installation process. Windows does not natively recognize virtIO interfaces, so it cannot see the virtual disk.

Open Source virtIO drivers can be downloaded from here courtesy of Fedora Linux.

If Redhat virtIO drivers are required due to signed driver needs within Windows, a Redhat account will need to be made on their website and the drivers downloaded directly from them.

Installing virtIO Drivers during Windows Install

During a Windows installation, users can use the console interface tools to change the CD-ROM image to the newly downloaded virtIO drivers iso. Information on using the console interface tools can be found in the inline help within the category titled VDI under the section Using the Console.

Alternatively, you can choose to change the virtual disk drive Interface type to SATA and Windows will find the virtual disk and continue with the installation. !! info "This will come with a performance impact due to SATA drivers being software emulated."

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Windows - Slow to Format a New Disk

Formatting a Virtual Disk with Windows 2012 (and later) Hosts May Take Longer Than Expected

Windows Server 2012 (and later) hosts will, by default, issue SCSI TRIM and Unmap commands equivalent to the entire size of the virtual disk. This behavior is the same even if the "Perform a quick format" option is checked, which significantly slows down the format process.

It is possible to disable the SCSI TRIM and Unmap feature on the host for the duration of the format.

To Disable TRIM and Unmap

Using a Windows CMD window on the host, issue the following command:

fsutil behavior set DisableDeleteNotify 1

To Re-enable the Feature

Use the following command to re-enable the Trim and Unmap feature:

fsutil behavior set DisableDeleteNotify 0

To Verify the Current Setting

You can verify the current Trim and Unmap setting by issuing the following command:

fsutil behavior query DisableDeleteNotify

The output will show one of the following:

  • DisableDeleteNotify=0 - The 'Trim and Unmap' feature is on (enabled).
  • DisableDeleteNotify=1 - The 'Trim and Unmap' feature is off (disabled).

Affected Versions

  • Only Windows Server 2012 and later hosts are affected. All earlier versions (e.g., Windows 2008) do not exhibit the same issue.

Non-server Versions

  • Non-server versions of Windows (e.g., Windows 8.x and 10.x) do not support the DisableDeleteNotify parameter.

Document Information

  • Last Updated: 2024-08-29
  • VergeOS Version: 4.12.6

Windows Restored VM Not Bootable

After restoring a copy of a virtual machine from a recent snapshot, the restored copy may fail to boot properly. The VM may stop with a blue screen message which reads:

Your PC ran into a problem and needs to restart. We're just collecting some error info, and then we'll restart for you.

There are several guest-level issues that can cause a VM running Windows to not start successfully. Below are the most common causes and their corresponding solutions.

Common Causes and Solutions

1. Non-Quiesced Snapshots

One of the most frequent causes of a restored VM failing to boot is that the snapshot was not taken in a clean (Quiesced) state. A quiesced snapshot ensures that the VM's memory and disk I/O are in a stable state, making the restored VM more likely to boot successfully. Without quiescing, the snapshot could have captured an unstable or inconsistent state.

Solution:
  • Ensure that when snapshots are taken, they are quiesced. Quiescing allows the OS to pause I/O operations, flush memory, and ensure that no incomplete transactions are saved in the snapshot.
  • For future backups, enable the Quiesce Snapshots option when scheduling backups for Windows VMs. This feature ensures that the system is in a stable state before taking a snapshot.

2. Pending or Partially Installed Windows Updates

If Windows updates were partially installed or in progress when the snapshot was taken, the restored VM might experience issues booting due to an incomplete or corrupted update state.

Solution:
  • Boot the VM into Safe Mode and complete any pending updates.
  • You can also attempt to disable the Windows Update service temporarily to allow the VM to boot without applying incomplete updates. Once booted, manually re-enable and check for updates.
  • Review the Windows Update Logs using Event Viewer to identify any problematic updates that might need to be rolled back or reinstalled.

3. Driver Incompatibility or Missing Drivers

Sometimes, the VM's hardware configuration in VergeOS (e.g., disk controllers, network adapters) may differ from the original environment, causing issues with booting due to incompatible or missing drivers. This is especially common when restoring VMs from a different hypervisor.

Solution:
  • Boot the VM using Windows Recovery and attempt to repair the system automatically.
  • Verify that the appropriate Virtio or SCSI drivers are installed, especially if the VM is using Virtio interfaces for storage or networking.
  • If the issue persists, boot into Safe Mode and manually update the VM's drivers from the Device Manager.

4. Corrupted Boot Loader

If the Windows bootloader was corrupted in the snapshot, the restored VM will not boot properly. This could happen if the system was performing a critical task related to the boot process (like an update or disk operation) when the snapshot was taken.

Solution:
  • Use the Windows Recovery Environment (WinRE) to repair the bootloader: 1. Boot the VM using a Windows installation disk or recovery media. 2. Select Repair your computer > Troubleshoot > Advanced options > Startup Repair. 3. If Startup Repair doesn’t work, open a Command Prompt and run the following commands: 
    bootrec /fixmbr
bootrec /fixboot
bootrec /rebuildbcd
  • These commands will repair the Master Boot Record (MBR) and rebuild the Boot Configuration Data (BCD).

5. Hardware Configuration Changes

Changes to the VM’s hardware configuration, such as CPU count, memory allocation, or disk type, may cause instability or prevent the VM from booting.

Solution:
  • Verify that the VM’s hardware configuration in VergeOS matches the original configuration from when the snapshot was taken.
  • If you made any changes, such as increasing memory or changing the number of CPUs, try reverting to the original configuration to see if the VM boots properly.

Best Practice: Manage Windows Updates in Guest VMs

A guest VM running Windows OS, and experiencing an unexpected restart, is often found to be caused by the Microsoft Windows Update service being configured to automatically apply updates that frequently require a restart.

Recommendations: - Schedule snapshot creation during maintenance windows when Windows updates are not being applied. - Configure Windows Update settings to avoid automatic installations or reboots, especially on critical VMs. Instead, use a manual update process during scheduled maintenance periods. - Regularly review the Windows Update logs in Event Viewer to detect potential issues related to updates that could affect the stability of the VM.

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6