Skip to content

Knowledge Base#

Troubleshooting VM Network Connectivity Issues

Before you begin, verify if other virtual machines in the environment can access the internet. If no other machines can, there may be a network issue upstream of the VergeOS platform that is preventing access to the outside world. If other VMs are still able to access the internet, the most likely cause is that a configuration step was missed.

The following are the most common configuration mistakes that cause network issues:

  • Missing NIC Configuration: The newly created VM may not have a NIC configured. To verify this, review the NICs section of the VM dashboard. Ensure at least one NIC is present. If not, add one.
  • Incorrect Network Assignment: The VM's NIC may be connected to the wrong network. In the NICs section, ensure at least one NIC is present with the status set to Up, and verify that the correct network is listed. If not, edit the NIC and assign the correct network (one used by a VM with internet access).
  • Improper IP Configuration: The VM might not have a properly configured IP address. Typically, this is resolved at the guest level. Refer to the guest operating system’s documentation to ensure the NIC is detected, installed (with drivers), and configured correctly.
  • Virtio Drivers Not Installed: If the Virtio drivers are not installed, the NIC may not function properly. For instructions on installing Virtio drivers, refer to the Product Guide.

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6
  • in Tenant, VM
  • 2 min read

How to Share a VM into a Tenant

VergeOS provides an easy way to share a virtual machine (VM) image from a parent environment into a tenant located beneath the current environment. Follow these steps to accomplish the task:

Steps to Share a VM

  1. Navigate to the VM Dashboard: - Go to the VM dashboard of the VM you want to move into a tenant.

  2. Gracefully Power Down the VM: - It is best practice to gracefully power down the VM using the guest operating system's best practices before moving it.

  3. Take a Snapshot: - In the VM dashboard, expand Snapshots in the left navigation menu to access the snapshot commands. - Click Take Snapshot to launch the Machine Snapshot creation screen.

  4. Complete the Snapshot Creation: - At the Machine Snapshot creation screen, fill in the required fields:

    • Machine: The virtual machine you are moving.
    • Name: Provide a unique name for the snapshot.
    • Expiration Date: Set the date/time when the snapshot will automatically be deleted.
    • Click Submit to create the snapshot.

  5. Share the VM Snapshot: - After clicking Submit, you will be taken to the dashboard of the newly created snapshot. - From this view, click on Share VM in the left navigation menu to launch the Shared Objects creation screen.

  6. Create the Shared Object: - At the New Shared Objects creation screen, fill in the required fields:

    • Name: Name the snapshot of the VM something unique.
    • Type: Select Virtual Machine.
    • Snapshot: This should match the name provided during the snapshot creation.
    • Recipient: Select the tenant where you want to share the VM.
    • Click Submit to create the shared object.

  7. Access the Tenant Environment: - Using a web browser, navigate to the tenant environment where the snapshot object was shared. - Log in with the proper authentication credentials.

  8. Create a New Virtual Machine in the Tenant: - In the tenant environment, navigate to Machines > Virtual Machines, and click New to begin creating a new virtual machine.

  9. Import from Shared Objects: - At the New Virtual Machine creation screen, under Select Type, choose -- Import from Shared Objects --. - In the Selections Available section, select the shared object created earlier, then click Next.

  10. Complete the Virtual Machine Settings:

    • On the Virtual Machine Settings screen, complete the required fields:
    • Shared Objects: Select the shared object created earlier.
    • Click Submit to create the new virtual machine in the tenant.

---

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

How To Efficiently Recover a Single VM from a Remote Cloud Snapshot

This is easily achieved for systems that are successfully configured to send a cloud snapshot to a remote destination tenant and have the Sync-back configured properly.

Note

For more information on configuring Sync-back, refer to the inline help Category titled 'Site Sync' under the section labeled 'Syncing Back'.

Recover a Copy of the VM on the Backup Side

  1. On the destination side (where the snapshots are sent), review all of the received remote snapshots and locate the desired snapshot that closely matches the date/time. This is accomplished from the Main Dashboard > System > Cloud Snapshots.
  2. Once the snapshot is located, select (check) the Cloud Snapshot in the list of available snapshots, and on the left navigation menu, select View VMs.
  3. Wait while the list of available VMs loads. This can take a few minutes.
  4. Once the list of virtual machines contained within the cloud snapshot loads, select (check) the desired VM to recover and then select Recover on the left navigation menu.
  5. The Recover VM Snapshot option appears. It is recommended to rename the VM to reflect the date the snapshot was taken. For example, Domain Controller recovered on 09012022.
  6. Wait while the VM is recovered.


Create a New Cloud Snapshot on the Backup Side

  1. On the destination side (where snapshots are sent), create a new cloud snapshot that will contain the newly recovered VM from the steps above. This is done from Main Dashboard > System > Cloud Snapshots.
  2. On the Cloud Snapshots page, select New on the left navigation menu.
  3. The New Cloud Snapshot creation page will load. Name the snapshot. It is recommended to name it something that is easily referenced in future steps, such as Recoveryon09012022.
  4. Set the expiration date to something logical. It should not exist forever but should be far enough into the future to allow time for the transfer back to the original system.
  5. After setting the name and expiration, select Submit to create the snapshot.


Request a Sync-Back on the Original/Source Side

  1. On the origin (sending) side, navigate to the configured outgoing site sync. This is done from Main Dashboard > Backup/DR > Outgoing Syncs, and then double-clicking into the configured outgoing sync.
  2. From the outgoing sync dashboard, click Refresh Remote Snaps on the left navigation menu. This will query the remote side for any new snapshots and list them. It should detect the snapshot created in the steps above.

  3. Once the newly created snapshot is detected, it will be listed under the Remote Snapshots section. Find the snapshot and click on the Request to Download icon to the far right of the listed snapshot.

  4. The Request Cloud Snapshot menu will load. Set a reasonable expiration date for how long the recovered snapshot will be retained on this system, and click Submit.

  5. The system will load the Sync-Back / Incoming Sync. The length of time it will take to transfer the snapshot back can vary greatly depending on several factors, including bandwidth speed and the size of the data to transfer.

  6. Wait while the synchronization completes. Once finished, a new entry will appear in the log section.

    Example

    Sync for 'Morning_20220901' complete (4m 17s) checked [78.1GB] scanned [1.76TB] sent [5.16GB] sent net [2.16GB] dirs [210] files [641] updated [31]

  7. At this point, the snapshot has been successfully transferred back to the original location. Administrators can now perform standard recovery tasks on the VM contained within the snapshot.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6
  • in vSAN
  • 1 min read

Preferred Tier Usage

How Preferred Tier Settings Determine Which Tier to Use

When creating or modifying a virtual machine (VM) disk drive in VergeOS, users can set a Preferred Tier. In most cases, this is left at default, which can be configured under System > System Settings > Default VM Drive Tier. However, the system's behavior when a specified tier does not exist can be unexpected. Here's how VergeOS determines which tier to use in such cases:

  • Setting a preferred tier to a non-existent higher tier:

    • Example: If a user selects Tier 3 in a system that only has Tier 1 and Tier 4 storage available, the system will attempt to pick the next higher (slower) tier. In this case, the system will default to Tier 4.

  • Setting a preferred tier to a non-existent lower tier:

    • Example: If a user selects Tier 3 in a system that only has Tier 1 and Tier 2 storage, the system will pick the next lower (faster) tier. In this case, the system will default to Tier 2.

In both scenarios, VergeOS ensures that the closest available tier is selected based on the user’s preference.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Best Practices - Running a pfSense Virtual Firewall

pfSense is a widely-used open-source firewall and router software that can be run as a virtual machine (VM) within VergeOS. Leveraging pfSense inside VergeOS allows for highly customizable and flexible firewall configurations. Below are the best practices for creating and maintaining a pfSense virtual firewall in VergeOS.

1. Configuring Disk and Network Interfaces

When deploying pfSense as a virtual machine, ensure that the disk and network interfaces are configured for optimal performance.

  • Disk and Network Interface Type: Set both the disk and network interfaces to VirtIO. This configuration provides better performance compared to the default options.
    • VirtIO for NIC: By default, VergeOS may configure the network interface to use E1000. While this can work, it's recommended to switch to VirtIO for better throughput and reduced CPU overhead. Failure to use VirtIO can lead to intermittent traffic issues or slowness, especially under high network loads.
    • VirtIO for Storage: Using VirtIO for storage ensures faster disk I/O, reducing bottlenecks in the system when dealing with large firewall logs or managing stateful connections.

2. Disabling Hardware Checksum Offloading

In certain environments, pfSense may experience network performance issues like packet loss, slowness, or connection timeouts. This is commonly due to hardware checksum offloading on virtualized NICs.

  • Disable Hardware Checksum Offloading:
    • Within the pfSense UI, navigate to System > Advanced > Networking and disable Hardware Checksum Offloading.
    • When enabled, pfSense offloads the processing of checksums to the virtual NIC. However, this feature is better suited for physical NICs, and in virtualized environments, it can cause performance degradation by generating unnecessary processing overhead on the virtual machine.

pfSense NIC Offloading Settings

3. Assigning Adequate Resources

  • CPU and RAM Allocation:

    • Depending on the size of your network and the complexity of your firewall rules, ensure you assign adequate CPU cores and RAM to your pfSense VM.
    • For small to medium environments, 2 CPU cores and 2GB of RAM is usually sufficient. For more complex configurations or higher network traffic, consider increasing these resources to ensure optimal performance.

  • Disk Space

    • Allocate enough disk space for system logs, caching, and configuration backups. Start with at least 10GB of disk space and increase based on the features in use, such as VPNs or IPS/IDS logging.

4. Snapshots and Rollbacks

  • Use VergeOS Snapshots:

    • Before making significant changes to your pfSense configuration or performing major upgrades, create a VergeOS snapshot of the pfSense VM. This allows for a quick rollback in case of misconfiguration or failure.

  • Automate Snapshots:

    • Automate your snapshots for pfSense to ensure regular backups of your firewall’s state. These snapshots can be scheduled in VergeOS and easily restored when needed.

Following these best practices ensures that pfSense operates efficiently and securely within VergeOS, providing a reliable and high-performance firewall solution for your virtualized network environment.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6
  • in VM
  • 2 min read

Making a Non-Persistent VM

How to Create a Non-Persistent VM on Reboot

A Non-Persistent VM reverts to its original state after a reboot, discarding any changes made during the session. This is useful for VDI (Virtual Desktop Infrastructure) environments where the system should reset after each use.

Steps to Create a Non-Persistent VM:

  1. Navigate to the VM dashboard.
  2. Shutdown the VM by selecting Actions > Power Off or using the Power button:

Note

This ensures the data is in a good state for cloning.

  1. Click the Copy button next to the main disk on the VM. nonpersistent-2.png
  2. Change the Media Type to Non-Persistent and click Submit at the bottom.
  3. Click the Edit icon edit icon pencil for the original Disk Media Type.

Note

The new disk will show a Media Type of Non-Persistent. Any changes made to this disk will be reverted upon a reboot of the VM. 6. Uncheck the Enabled checkbox. 7. Start the VM by selecting Power On from the left-hand menu or clicking the Play button:

This will boot the VM using the non-persistent disk. The disk is fully writable during the session, but all changes will be discarded upon reboot, reverting the VM back to its original state.

Do not delete the original disk. It will not take up additional space due to Deduplication.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6
  • in VM
  • 2 min read

How to Isolate a Virtual Machine

Isolating a virtual machine (VM) can be done in several ways, depending on the specific requirements of the environment. Below are two common methods for isolating a VM within VergeOS.

Remove the Attached Network from the VM

This method essentially simulates unplugging a network cable from the VM, making it function without any external connectivity. It’s suitable when the VM doesn’t need to communicate with any other network.

Steps to Remove the Network:

  1. Navigate to the Virtual Machine Dashboard.
  2. Click on NICs in the left navigation menu to access the VM's virtual network adapters.
  3. Select the NIC you want to edit and click Edit from the left navigation menu.
  4. In the NIC configuration window, use the drop-down list to change the Network from its current value to --None--.
  5. If the VM has multiple NICs, repeat this process for all active/enabled NICs.

By removing the network from all NICs, the VM will no longer have network access.

Create a New Internal Network

If the VM requires connectivity but still needs to be isolated from other networks, creating a new internal network with no other VMs connected is the preferred solution.

Steps to Create an Internal Network:

  1. From the VergeOS dashboard, navigate to Networking and create a new Internal Network.
  2. Set a Default Gateway for outbound access if needed.
  3. After the internal network is created, return to the VM dashboard and update the NIC to attach the VM to the newly created internal network.

For more detailed instructions on creating internal networks, refer to the VergeOS inline help under Networking in the Internal Networks section.

This method allows the VM to have restricted network access while still providing outbound connectivity through the internal network.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Exporting Virtual Machines from VergeOS

Exporting a virtual machine (VM) from VergeOS allows you to download the VM’s disk in .raw format, which is compatible with many hypervisors. This guide outlines the steps to export a VM and download its virtual disk.

Steps to Export a Virtual Machine

  1. Log in to the VergeOS platform and navigate to the dashboard of the virtual machine you wish to export.
  2. From the left-hand menu, click on Drives to view a list of the virtual disk drives attached to the VM.
  3. Select the drive you want to export, then choose Download from the left-hand menu. The virtual disk will be downloaded in .raw format.
    • The .raw disk format is widely supported by many modern hypervisors.
  4. The system will automatically begin downloading the disk image.
  5. Once the download completes, refer to the documentation for your destination hypervisor for instructions on how to import, upload, or convert the .raw disk image.

Important Considerations

  • .raw format: The exported VM drive is downloaded in .raw format. This format is compatible with most hypervisors, but some may require converting the image to another format such as .qcow2 or .vmdk. Check your destination hypervisor’s documentation for conversion tools or instructions.
  • File size: Depending on the size of the virtual disk, the download can be large. Ensure you have enough disk space available on the system where you are downloading the file.

By following these steps, you can successfully export and download virtual machines from VergeOS for use in other environments.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Accessing the Verge.io UI from a VM

Overview

Key Points

  • Access the VergeOS UI from a VM within your environment
  • Utilize hair-pinning network technique
  • Create a specific network rule on the internal network

This article guides you through the process of setting up access to the VergeOS User Interface (UI) from a virtual machine (VM) running inside the VergeOS system. This is accomplished using a networking technique known as hair-pinning, where a packet travels to an interface, goes out towards the Internet, but instead of continuing, it makes a "hairpin turn" and comes back in on the same interface.

Prerequisites

  • A running VergeOS environment
  • A virtual machine (VM) within your VergeOS environment
  • Access to the VergeOS UI
  • Basic understanding of network rules in VergeOS

Steps

  1. Navigate to the Internal Network - Log into your VergeOS environment - Go to the internal network that your target VM is connected to

  2. Create a New Rule - Locate the option to create a new rule - Configure the rule with the following settings:

    Rule: - Name: Use a reference name, such as "Allow UI" - Action: Translate - Protocol: TCP - Direction: Incoming - Interface: Auto - Pin: No

    Source: - Type: Any / None - Source Ports/Ranges: Leave blank

    Destination: - Type: My Network Address - Destination Ports/Ranges: 80, 443

    Target: - Type: Other Network DMZ IP - Target Network: Core - Target Ports/Ranges: Leave blank

  3. Submit the Rule - Click "Submit" to save the rule

  4. Apply the New Rule - Click "Apply Rules" to activate the newly created rule

  5. Access the UI from the VM - Open a web browser within your VM - Navigate to the IP address of the internal network (e.g., if the internal network IP is 192.168.0.1, use this address)

Pro Tip

Always ensure that your VM's network settings are correctly configured to use the internal network where you've set up this rule.

Visual Guide

Here's a visual representation of the rule configuration:

hairpin.png

Troubleshooting

Common Issues

  • Problem: Unable to access the UI after creating the rule
  • Solution:
    1. Verify that the rule is applied correctly
    2. Check if the VM's network interface is on the correct internal network
    3. Ensure no firewall rules are blocking the connection

Additional Resources

Feedback

Need Help?

If you encounter any issues while setting up UI access or have questions about this process, please don't hesitate to contact our support team.

Document Information

  • Last Updated: 2024-08-29
  • VergeOS Version: 4.13.3

Sharing Media Images to Tenants

Overview

Key Points

  • Service Providers can share files with Tenants
  • Files must already be uploaded to Media Images
  • Process is quick and uses a branch command

This article guides Service Providers through the process of sharing files, specifically media images, with their Tenants in the VergeOS system. This feature allows Tenants to access specific files within their own Media Images section.

Prerequisites

  • Access to the VergeOS system as a Service Provider
  • Files already uploaded to the vSAN
  • Existing Tenants in the system

Steps

  1. Navigate to the Tenants Dashboard - From the Main Dashboard, select "Tenants" on the left menu

  2. Access Tenant List - Select "Tenants" to view a listing of all your Tenants

  3. Select the Desired Tenant - Double click on the Tenant you want to share files with

  4. Access the Add File Feature - Select "Add File" in the left menu

  5. Choose File Type - Select File Type from the dropdown list

Tip

Select "ALL" to get a listing of all files available, regardless of type. This will include .raw files (VM disk images) from the parent VDC.

  1. Select Specific File - Choose the specific file you want to share from the dropdown list

  2. Submit Changes - Click the submit button at the bottom of the page

  3. Confirmation - The process is near-instant as it is done with a branch command - The file is now available to the Tenant within their own Media Images section

Troubleshooting

Common Issues

  • Problem: File not appearing in Tenant's Media Images section

  • Solution:

    1. Verify that the file was successfully uploaded to Media Images
    2. Check if the correct file type was selected
    3. Ensure that the changes were submitted properly

  • Problem: Unable to select a specific file

  • Solution:
    1. Confirm that the file exists in Media Images
    2. Try selecting "ALL" in the File Type dropdown to see if the file appears

Additional Resources

Feedback

Need Help?

If you encounter any issues while sharing media images or have questions about this process, please don't hesitate to contact our support team.

Document Information

  • Last Updated: 2023-08-24
  • VergeOS Version: 4.12.6
  • in Tenant
  • 1 min read

How to Reset a Tenant Administrative Password

If you need to change the administrative credentials for a tenant environment in your VergeOS system, follow these steps:

  1. Log in to the VergeOS environment where the tenant is hosted.
  2. Navigate to the Tenant Dashboard of the tenant for which you need to change the admin password.
  3. In the Tenant Dashboard, click the Edit button in the left navigation menu. This will open the Tenant Edit screen.
  4. In the Tenant Edit screen, type the new password in the Admin User Password field.
  5. Re-enter the password in the Confirm Admin User Password field.
  6. After entering the same password in both fields, click Submit. If the passwords do not match, an error message will appear.
  7. After successfully updating the admin password, open a web browser and navigate to the tenant environment where the password was changed.
  8. Log in as the admin. The username will be admin and the password will be the new password you saved in step 6.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6
  • in vSAN
  • 1 min read

vSAN Encryption Information

You can confirm that the vSAN has encryption enabled by navigating to Nodes> Node 1> Drives and then double-clicking on the first drive in the list.  The Encrypted checkbox is checked if the Vsan is encrypted.

  • Encryption for the vSAN is configured during the initial installation only.

  • System startup on an encrypted system can be configured two different ways:

  1. The most common method is by having encryption keys written to a USB drive during the initial installation. In this scenario, these drives are typically plugged into the first two nodes of an encrypted system to boot normally. All other nodes do not require them, as Node 1 and Node 2 are the controller nodes. The USB drive does not require much storage at all, less than 1GB.
  2. If the controller nodes do not have USB encryption keys connected, the system will prompt an operator to type the proper encryption password to complete the power-up process.
  • Default encryption is set for all snapshot synchronizations through a site-sync.

Information about encrypting a Site Synchronization can be found in the Product Guide

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6

Reasons for Unexpected / Unexplained vSAN Growth

There are several reasons for the vSAN to start growing at a rate faster than anticipated. Administrators should first determine when the unexplained growth occurred by reviewing the vSAN Tiers' growth history, and then assess potential areas for unexpected growth.

Review vSAN Tiers for Growth History

To isolate unexplained growth, it is important to narrow down when the growth increased exponentially. Using the steps below, administrators can review storage growth and visualize normal growth from daily operations versus spikes in growth, which are typically unexpected.

  1. Navigate to the vSAN Tiers from the Main Dashboard. If vSAN Tiers is not present, then this environment is a tenant of a parent system, and the vSAN tier needs to be examined at the parent system.
  2. Open the vSAN Tier with unexpected growth (for example, vSAN Tier 0).
  3. On the left navigation menu, click on History.
  4. A new menu will appear showing history in various graphs. Modify the filter period to isolate any growth on this tier. - It is recommended to start with a custom filter of 1 day and review the Storage Usage graph.

Things to Note:

  • If you see dips and spikes every hour or once a day, this is likely the result of snapshots falling out of retention (old ones expiring, new ones being created). Note whether the total storage consumed at the start of the day is nearly equivalent to the end of the day. If so, expand the custom filter to a week.
  • When reviewing by week, check if the total storage consumed at the start of the week is similar to the end. If, for example, the growth is roughly 10%, repeat for the previous week. If the weekly growth percentage is consistent, this represents your average weekly growth rate, which can help plan for hardware expansion.
  • Filter the current month and check for any sudden spikes in storage consumption on the Storage Usage graph. Click and drag over the time in question to zoom in on the data, and hover over the graph for specific date/time information.

vsan_unexpected_growth.png

Possible Reasons for Storage Increase

Several areas in the VergeOS platform may contribute to unexpected storage growth. Common areas to check include:

  • Cloud Snapshots:
  • Navigate to System > Cloud Snapshots.
  • Are any being held past their expected expiration time?
  • Are there snapshots without a Snapshot Profile? These may have been taken manually. Investigate when and why they were taken.

  • Are any snapshots set to "Never Expire"? This can lead to large data consumption over time.

  • Virtual Machines (VMs) Snapshots:

  • Navigate to the Machines Dashboard. The Snapshots count box shows the number of machine-level snapshots present. Click this box to list all VM snapshots and their creation date/time. Review if any can be removed.

  • Navigate to Machines > Virtual Machines. Sort by the Snapshot Profile column to identify VMs with machine-level snapshots. These are included in the recurring cloud snapshots, so review whether individual snapshots are necessary or if they can be removed.

  • VMWare Backup Jobs:

  • Navigate to Backup/DR > VMware Services and review each VMware Service instance for Backup Job history.

  • On the left menu, click Backup Jobs to review each specific instance. Check the Expires column for each backup and review if it can be removed.

  • Media Images:

  • Navigate to Media Images and sort by Modified. Check if any upload dates/times match the unexplained growth period.

  • Review whether media images, especially other hypervisor formats (e.g., .ova or .vhdx), can be removed.

  • Incoming Site Syncs:

  • Navigate to Backup/DR > Incoming Syncs. Open each Incoming Sync dashboard and check the Received Snapshots count. Investigate the source (origin) site for increased storage matching the timeframe.

  • Tenant Storage:

  • Navigate to Tenants > Each Tenant Dashboard.
  • Review Total Storage Used by clicking on History in the left menu. Follow the same process listed above to review growth history.
  • If unexpected growth is found, investigate within the tenant for the possible causes of storage increase (as listed above), and within any sub-tenants if applicable.

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6

Common Snapshot Synchronization Error Messages Explained

The VergeOS platform provides a feature known as Site Syncs to replicate a copy of a cloud snapshot.

For more information on Snapshots and Site Syncs, refer to our Product Guide on Sync Configuration.

Occasionally, the system may generate a system alert from a new Message Log entry related to the Site Sync functionality. Below is a list of common errors along with a brief explanation:

ybvsan: Error walking tier 3 refs: (2) No such file or directory

  • This error can occur if the VergeOS software version is mismatched between the sending side and the destination side.

Unable to delete snapshot that no longer exists: Resource '/v4/cloud_snapshots/1' not found during delete

  • This error is usually the result of a timing issue when a snapshot is being deleted, and the reference is deleted in the metadata of the system.

Error notifying client with 'notify_complete' Error (403) while communicating with server

  • The snapshot successfully synchronized, but this error appeared during the sync clean-up process. If this error occurs on multiple snapshot synchronizations, the handshake credentials between the two systems may have stopped working. In that case, consult VergeOS support for assistance.

Error - Sync back not found and no registration code supplied

  • This error occurs when requesting a snapshot back from the destination site to the source site. If this message appears, Sync Back is not configured between the two systems. Refer to the Guide on Sync Back for instructions on setting up Sync-Back between the systems.

Error - Sync Request for 'system name' Error notifying client with 'notify_start' Connection timed out

  • This error occurs when requesting a snapshot back from the destination site to the source site, similar to the previous error. Ensure that Sync Back is configured properly between the two systems.

An error has occurred while syncing 'snapshot name': Resource temporarily unavailable. Retrying 1 of 10

  • This error typically results from an interruption of the transfer connection between the source site and the destination site. The sync will automatically retry following the Retry interval setting. The retry count will increase until the connection is re-established or until the maximum Queue retry count has been reached.

Error- Unable to create tenant snapshot 'snapshotinterval_yyyymmdd': This name already exists

  • The local snapshot schedule is naming snapshots the same as the inbound snapshots from the site sync. A simple fix is to rename the origin (sending) side snapshot by editing the auto-sync configuration. Use the field Prefix the snapshot name with this on the destination and add something unique, such as remote-.

Unable to update cloud snapshot: No such file or directory

  • This error indicates a possible timing issue with snapshots. Review the Outgoing Sync configuration on the sending site for any setting mismatches.

Error notifying client with 'notify_start' Connection timed out

  • The sync task was unable to start because the connection timed out. Typically, this error occurs when requesting a snapshot back from the destination side to the original sending side. In most cases, this is caused by a firewall blocking the traffic or missing traffic rules on the destination side. Refer to the Guide on Sync Configuration for the required traffic rules.

---

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

How To Identify a Failed Disk In Your VergeOS Environment

VergeOS offers a diagnostic function that allows system administrators to turn a disk drive's LED light on or off, making it easier to physically identify a failed or problematic drive. Follow the steps below to locate a failed disk drive for replacement.

Steps to Identify a Failed Disk

  1. Log in to the VergeOS UI and navigate to the dashboard of the node where the failed disk resides.
  2. On the Node Dashboard, locate and select Diagnostics from the left-hand column.
  3. In the Diagnostics page, change the Query to LED Control (Drive).
  4. In the LED Control (Drive) details section:
    • Path: Enter the path to the drive you want to locate (e.g., /dev/sdb). If you're unsure of the path, check the system alerts and logs for recent error or warning messages.
    • State: Set the LED state to On, then click Send to activate the LED light on the drive.
  5. Locate the drive with the active LED indicator in your physical server.
  6. Once the drive has been identified and replaced, set the State to Off and click Send to deactivate the LED light.

For detailed instructions on drive replacement, refer to the Maintenance section in the inline help under Drive Replacement. This section guides you through the entire process.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6
  • in VPN
  • 1 min read

Wireguard - Adding Nameserver entries to Client Configs

Wireguard Config Entries

The following are instructions for adding a PostUp and PostDown script to the Wireguard config.
For Windows, this adds Powershell commands for adding and removing a DNS Client Rule when the client connects and disconnects.

Windows Clients

  1. In the Windows Wireguard client, edit the config.
  2. Add the following commands in the [Interface] section:
PostUp = powershell -command "Add-DnsClientNrptRule -Namespace 'domainname.com' -NameServers '10.1.10.2'"
PostDown = powershell -command "Get-DnsClientNrptRule | Where { $_.Namespace -match '.*domainname\.com' } | Remove-DnsClientNrptRule -force"
  1. Change the following entries to match your setup:
    • Namespace: A comma-separated list of domain names to add.
    • NameServers: A comma-separated list of nameserver IP addresses.

For the -match, make sure to include a backslash (\) before each period (.)

Linux Clients

This may vary based on your Linux distribution.

  1. Edit the config file on the Linux client.
  2. In the [Interface] section, add the following:
PostUp = resolvectl dns %i 10.1.10.2; resolvectl domain %i domainname.com
PreUp = iptables -A INPUT -i wg -m state --state ESTABLISHED,RELATED -j ACCEPT
PreUp = iptables -A INPUT -i wg -j REJECT
PostDown = iptables -D INPUT -i wg -m state --state ESTABLISHED,RELATED -j ACCEPT
PostDown = iptables -D INPUT -i wg -j REJECT
  1. Replace 10.1.10.2 with the correct IP of your nameserver.
  2. Replace domainname.com with your domain name.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

How to Configure Routing Between Networks

The following is a simple method to establish a route between two networks in the VergeOS platform.

Create a Network Rule on the First Network to Route Traffic to the Second Network

  1. Navigate to the first network that you would like to route traffic from.
  2. From the network dashboard, click on Rules in the left navigation menu.
  3. In the Rules menu, click on New to create a new network rule.
  4. Configure the rule with the following settings: - Rule:
    • Name: A name indicating this rule is a route to the second network.
    • Action: Route
    • Protocol: Any
    • Direction: Outgoing
    • Source:
    • Type: My Network Address
    • Destination:
    • Type: Other Network Address
    • Network: The name of the second network
    • Target:
    • Type: Other Network DMZ IP
    • Target Network: The name of the second network
  5. After completing this rule, click Submit to save the rule.

Create a Network Rule on the First Network to Allow Traffic from the Second Network

  1. From the same network dashboard, click on Rules in the left navigation menu.
  2. In the Rules menu, click on New to create a new network rule.
  3. Configure the rule with the following settings: - Rule:
    • Name: A name indicating this rule allows traffic from the second network.
    • Action: Accept
    • Protocol: Any
    • Direction: Incoming
    • Source:
    • Type: Other Network Address
    • Network: The name of the second network
    • Destination:
    • Type: My Network Address
  4. After completing this rule, click Submit to save the rule.
  5. Click Apply Rules to enable the rule.

Info

After completing the two rules on the first network, you will need to create identical rules on the second network.

Create a Network Rule on the Second Network to Route Traffic to the First Network

  1. Navigate to the second network that you would like to route traffic from.
  2. From the network dashboard, click on Rules in the left navigation menu.
  3. In the Rules menu, click on New to create a new network rule.
  4. Configure the rule with the following settings: - Rule:
    • Name: A name indicating this rule is a route to the first network.
    • Action: Route
    • Protocol: Any
    • Direction: Outgoing
    • Source:
    • Type: My Network Address
    • Destination:
    • Type: Other Network Address
    • Network: The name of the first network
    • Target:
    • Type: Other Network DMZ IP
    • Target Network: The name of the first network
  5. After completing this rule, click Submit to save the rule.

Create a Network Rule on the Second Network to Allow Traffic from the First Network

  1. From the same network dashboard, click on Rules in the left navigation menu.
  2. In the Rules menu, click on New to create a new network rule.
  3. Configure the rule with the following settings: - Rule:
    • Name: A name indicating this rule allows traffic from the first network.
    • Action: Accept
    • Protocol: Any
    • Direction: Incoming
    • Source:
    • Type: Other Network Address
    • Network: The name of the first network
    • Destination:
    • Type: My Network Address
  4. After completing this rule, click Submit to save the rule.
  5. Click Apply Rules to enable the rule.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

How to Update Your VergeOS Environment

WARNING: DO NOT SKIP MAJOR VERSIONS

For example, if you are on version 4.8, please update to version 4.9 before updating to 4.10. Skipping versions could cause major database and configuration issues.

The VergeOS platform supports 'zero downtime' updating. This means that during a routine update process, guest workloads (VMs and tenant environments) can remain on and running as normal.

For more information on the update process, please refer to our Product Guide.

The time required to complete an update varies depending on factors such as: - Number of nodes in the system - Storage consumed on the vSAN - Data change rate generated by workloads during the update - Hardware performance (processor speed, disk type, network speed)

During the update, you may see tiers and drives alternate between Online, Verifying, or Repairing statuses. This is normal as the system verifies data integrity.

Before performing an update, verify that your system memory usage is within limits. For example, in a two-node configuration, total system memory should be under 50% utilization to allow the remaining node to handle 100% of workloads.

Ensure no nodes are in Maintenance mode before proceeding with the update.

Update Procedure

  1. Log in to the VergeOS UI.

  2. Navigate to System > Updates, then select Check for Updates in the left menu.

    • A pop-up will prompt Yes or No, select Yes.
    • If a banner appears stating "A new minor version is available on a different branch," follow the prompt to change branches by selecting Change Branch in the left menu. Confirm with Yes. VergeOSupgrade-new-img2.png

  3. The packages to be downloaded will now be highlighted. VergeOSupgrade-new-img3.png

    • Select Download in the left menu.
    • A pop-up will prompt Yes or No, select Yes.
    • The download process will appear on the dashboard in the Current Update Server tile.

  4. Once the download completes, the Install action will become available.

    • Select Install when ready.
    • Confirm with Yes to begin the install.

  5. After installation, a request to reboot the system will occur. - Select Reboot in the left menu to initiate a rolling reboot across all nodes.

Note

The update will start with Node 1, putting it into maintenance mode and migrating workloads to another node. During minor version changes, you may briefly lose access to the UI as it fails back to Node 1. This is normal and should last no more than a minute or two. Workloads should not experience network issues.

Troubleshooting Steps

Workloads Failing to Migrate

  • This error is usually due to insufficient resources (RAM) in the cluster. Try migrating other workloads or adjust RAM usage. More causes and solutions are detailed here.

vSAN Taking a Long Time to Verify

  • The verification process depends on factors like network speed, disk type, and consumed data. HDDs will take longer than NVMe or SSDs. On VergeOS versions 4.9.0 and higher, check the Full Walk Progress on the tiers dashboard for an indication of how far along the verification is.

    walk-percentage.png

WARNING

This process must complete before rebooting any additional nodes. Failure to do so can result in a double failure, causing workloads to crash.

Unable to Connect to Update Server

  • Ensure the system has a working DNS server on its external (UI) network:

    1. Navigate to the external network dashboard.
    2. Select Diagnostics in the left menu.
    3. Set Query to DNS Lookup and select Send.
    4. If DNS is properly configured, the response will display Verge.io's IP address. If not, check DNS settings and retry the query.

  • Expired Update Server credentials can also cause this issue. These are tied to the system's license and should be renewed through your VergeOS sales representative.

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6

Accessing the User Interface from an Internal Network

Overview

Key Points

  • Access the vergeOS UI from a VM within your environment
  • Create a route rule on the internal network
  • Simple process involving dashboard navigation and rule creation

This article guides you through the process of setting up access to the vergeOS User Interface (UI) from a virtual machine (VM) within your vergeOS environment. This is accomplished by creating a specific route rule on the network to which your VM is connected, typically an internal network.

Prerequisites

  • A running vergeOS environment
  • A virtual machine (VM) within your vergeOS environment
  • Access to the vergeOS dashboard
  • Basic understanding of network rules in vergeOS

Steps

  1. Navigate to the Network Dashboard - Log into your vergeOS environment - Go to the dashboard of the network that your target VM is connected to

  2. Create a New Rule - Locate the option to create a new rule - Use the settings shown in the image below:

ui-access-rule.png

  1. Submit the Rule - After configuring the rule, submit it - You will be redirected to the list view of rules on the network

  2. Apply the New Rule - Click "Apply Rules" to activate the newly created rule

  3. Access the UI from the VM - Open a web browser within your VM - Navigate to the IP address of the Verge UI (e.g., https://192.168.4.1)

Pro Tip

Always ensure that your VM's network settings are correctly configured to use the internal network where you've set up this rule.

Troubleshooting

Common Issues

  • Problem: Unable to access the UI after creating the rule
  • Solution:
    1. Verify that the rule is applied correctly
    2. Check if the VM's network interface is on the correct network
    3. Ensure no firewall rules are blocking the connection

Additional Resources

Feedback

Need Help?

If you encounter any issues while setting up UI access or have questions about this process, please don't hesitate to contact our support team.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Windows - Time Shift

Description of The Issue

VergeOS Virtual Machines (VMs) running Windows OS may experience ‘time shift,’ where the guest OS will periodically adjust the time to an incorrect value. This is caused by the Windows OS, which expects time from the physical motherboard to be in local time (RTC) instead of Coordinated Universal Time (UTC).

VergeOS provides time in UTC, which has become the industry standard as it compensates for Daylight Saving Time (DST) changes. The guest OS will automatically adjust the time when comparing its clock value against an authoritative time source because it assumes the time provided by the hardware is local instead of UTC. This comparison causes periodic discrepancies because the guest OS is unable to adjust the physical node clock to match what it perceives as the correct time.

VergeOS Configuration Option: RTC Base

RTC Base is an individual VM setting that allows VergeOS administrators to set the time provided to the OS as either local or UTC. The value can be found when editing any VM.

rtcbase-utc-screenshot.png

With this configuration setting, administrators can granularly control every machine, though it is important to understand the expected behavior of each option.

Local Time

Setting RTC Base to Local Time will pass the time from the physical nodes to the virtual guest OS. This emulates the legacy behavior that Windows expects. When Windows compares the local time clock against an authoritative time service, Windows will adjust the time within the guest OS based on the time zone defined in the Windows configuration. This can result in unexpected behavior.

Things to consider with local time are:

  • The physical node time zone will be presented the same to any guest OS. If VergeOS is hosting VMs for different time zones, each Windows VM will perceive local time as the same value.
  • The physical node time and the guest OS will require proper configuration to avoid issues when Daylight Saving Time (DST) starts or stops each year. RTC clocks will need to be adjusted through software. Historically, issues have arisen when guidelines for DST have changed.

UTC Time

Setting RTC Base to UTC will pass the time from physical nodes into guest VMs as coordinated universal time. This is the industry standard for modern software applications that handle time.

When using the UTC setting in VergeOS, Windows VMs should be configured to recognize that time is presented as universal. For most Windows operating systems, the adjustment is made in the Windows Registry by adding a value to recognize “RealTimeIsUniversal.”

For 64-Bit Operating Systems

Under the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation, there needs to be a REG_QWORD entry with the following values:

  • Name: RealTimeIsUniversal
  • Value data: 1

After adjusting this setting, administrators will need to completely Power Off the VM and then Power On the VM before the change takes effect.

Important

When making software adjustments to guest OS or applications, administrators should check with production documentation for that software, including the latest KB articles, updates, and release notes.

Additional Reading on this Topic

Document Information

  • Last Updated: 2024-08-29
  • VergeOS Version: 4.12

Reasons a Windows VM Restarted Unexpectedly

Reasons a Windows VM Restarted Unexpectedly

If you have a Windows VM that has recently restarted unexpectedly, the first place to begin is to review the logs from the VM dashboard.
You can reach the log viewer by navigating from the Main Menu > Machines > Virtual Machines > the name of the VM you are investigating > Logs.

From the Log view, search through (or filter the results) under the Message section. Several status messages will indicate why the VM stopped running. The following are the most common and a brief explanation of each:

  • Message: VM action 'kill' sent
    This indicates that a VergeOS user issued a Kill Power command from the VergeOS interface. The log will indicate the user under the Source column.
    In this scenario, consult with the system user to determine the reason for issuing this command.

  • Message: VM action 'poweroff' sent
    This indicates that a VergeOS user issued a graceful Power Off command from the VergeOS interface. This command successfully interacted with the Guest OS ACPI to gracefully stop the VM. The log will indicate the user under the Source column.
    In this scenario, consult with the system user to determine the reason for issuing this command.

  • Message: VM has shutdown
    This indicates that the shutdown command was issued from inside the guest operating system directly.
    In this scenario, consult with the guest operating system logs to determine the reason for the shutdown command.

  • Message: VM has reset
    This indicates that the restart command was issued from inside the guest operating system directly.
    In this scenario, consult with the guest operating system logs to determine the reason for the reset command.

Best Practice

A guest VM running Windows that experiences unexpected restarts is often found to be caused by the Microsoft Windows Update Services being configured to automatically apply updates that require a restart.
To investigate this further, consult with Knowledge-Base articles about the particular version of the Windows OS.
One of the best places to start investigating at a Windows Guest level is using Windows Event Viewer and reviewing the Windows Update logs for more information.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Loading Virtio Drivers in Windows Recovery Console

If a guest VM running any version of Windows OS with the Virtio-SCSI disk drivers installed is booted into Recovery Console mode, the operating system partition may not be visible as the drivers are not yet loaded.

Prerequisites

Before starting with recovery work on a guest OS, ensure that the administrator has:

  1. Access to the VM, including tested/known working credentials.
  2. The Virtio drivers ISO loaded and available in the VergeOS environment under Media Images.
  3. The Windows OS installation ISO available in the VergeOS environment.

Requirements

To load Windows drivers from a Windows Guest OS recovery console, administrators will need:

  1. VergeOS UI platform access to the environment running the Guest OS.
  2. A copy of the (latest) Virtio disk drivers in ISO format.
  3. A copy of the Windows operating system ISO (example: Windows boot disk).

Starting the Windows Recovery Console

  1. Login to the console and follow the Windows installation prompts to navigate to the Recovery Console and Command Prompt.
  2. Once at a command prompt, type the following command to list available disks on the VM:
wmic logicaldisk get deviceid, volumename, description

Press ENTER.

This will return a list of available disks. Typically, D: will be the EFI boot volume, E: will be the CD installation media, and X: will be the active, booted Windows recovery session.

The Virtio-SCSI disks are likely not present, and the drivers need to be loaded for them to appear.

Installing the Virtio-SCSI and Storage Drivers

From the Command Prompt in the Windows Recovery Console, follow these steps:

  1. To load the Virtio-SCSI Storage drivers, type:
Drvload e:\vioscsi\2k16\amd64\vioscsi.inf

Press ENTER.

  • E: refers to the drive letter where the Virtio ISO is loaded.
  • 2k16\amd64 refers to the path to the Virtio driver based on OS. Browse the ISO image for paths to other OS versions if needed.
  1. To load the Virtio Storage drivers, type:
Drvload e:\viostor\2k16\amd64\viosstor.inf

Press ENTER.

  • E: refers again to the drive letter where the Virtio ISO is loaded.
  • 2k16\amd64 refers to the path to the Virtio driver based on OS. Adjust as necessary.

Verifying the Disk Mount

After loading both drivers, the disk should mount, typically as an F: drive.

To verify this, run the command again:

wmic logicaldisk get deviceid, volumename, description

Press ENTER. You should now see the newly mounted disk.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Windows Disks Showing Offline After Restarting

How to Correct Disks showing Offline after a Windows VM is Restarted

After importing a VM into the VergeOS platform, some (virtual) disks may not come online after a restart of the guest VM.

Beginning in Windows 2008, Microsoft added a setting for the default state of additional (non OS) disk drives. Occasionally when restarting, Windows may detect a hardware change, which is more likely if the VM was imported from an alternate hypervisor. When windows detects a hardware change, it may not bring secondary virtual disks online automatically. To change this to the recommended setting run the following Windows PowerShell:

Set-StorageSetting -NewDiskPolicy OnlineAll

Additionally, this can be done from the command line using diskpart or from Disk Management.

Document Information

  • Last Updated: 2024-08-10
  • vergeOS Version: 4.11

UEFI Disks as Boot Devices Will Not Boot Properly

After importing VM images leveraging UEFI disks as boot devices, sometimes the VM will not boot properly inside the VergeOS platform. To resolve this, tweaks to boot order/options need to be made.

Here is a list of suggested solutions for issues with imported VMs not booting because of guest UEFI BIOS settings:

Solution 1

  1. From a fresh import of the VM (before trying to power it up inside VergeOS), edit the VM in Verge and enable the UEFI Boot option.
  2. Power on the VM.
  3. Hit ESC within 5 seconds to get into the VM BIOS (you can edit the VM settings in Verge to increase the boot timer if necessary).
  4. Enter the BIOS and navigate to Boot Manager Options (this may vary depending on the BIOS).
  5. Change the selected boot device to the Verge IO device.
  6. Exit the UEFI BIOS.
  7. Reboot the VM.

Solution 2

  1. From a fresh import of the VM (before trying to power it up inside VergeOS), edit the VM in Verge and enable the UEFI Boot option.
  2. Power on the VM.
  3. Hit ESC within 5 seconds to get into the VM BIOS (you can edit the VM settings in Verge to increase the boot timer if necessary).
  4. Enter the BIOS and navigate to Boot Manager Options (this may vary depending on the BIOS).
  5. Disable Secure Boot as an option.
  6. Exit the UEFI BIOS.
  7. Reboot the VM.

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6
  • in VM
  • 2 min read

How to TRIM your Drives

After importing a virtual machine from another hypervisor, sometimes the free space available inside the virtual machine does not match the free space reported to the VergeOS platform. This discrepancy is often due to the virtual disk being thick-provisioned from the VM source, making VergeOS unaware of the unused disk space. To resolve this, a TRIM/UNMAP operation needs to be performed on the virtual disk from within the virtual machine.

Prerequisites for Running a TRIM Command

  1. Edit the virtual drive(s) in question in the VergeOS UI and ensure Discard is enabled.
  2. Ensure the virtual drive(s) is using a drive type of virtIO-SCSI or SATA.
  3. Ensure the virtual drive(s) is assigned to a Solid State Tier (usually tier 1-3).

Trimming a Windows Drive

To perform a manual TRIM operation in a Windows environment, follow these steps:

  1. Launch PowerShell as an admin user.
  2. From the PowerShell prompt, type the following command:

Optimize-Volume -DriveLetter YourDriveLetter -ReTrim -Verbose
Example:

Optimize-Volume -DriveLetter E -ReTrim -Verbose
3. Press Enter and wait for the command to complete.

As the TRIM operation progresses, you can watch the reported free space in the VergeOS dashboard increase as the blank data on the volume is removed.

If this does not resolve the issue, TRIM may not be enabled. To check if TRIM is enabled:

  1. Run the following FSUTIL command:
fsutil behavior query disabledeletenotify

If the value is 1, TRIM is not enabled on the drive.

  1. To enable TRIM, run:
fsutil behavior set disabledeletenotify 0

After enabling, rerun the TRIM commands.

Trimming a Linux Drive

Newer Linux distros have TRIM enabled by default via a systemd service or a cron job. To check if automated TRIM is enabled, follow these steps:

  1. Ensure the prerequisite steps from above are complete.

Example: Ubuntu Server

  1. Launch a terminal.
  2. Enter the following commands to check the status:

  • Check TRIM Timer/Schedule Status:

    sudo systemctl status fstrim.timer
    - Check TRIM Service Status:

    sudo systemctl status fstrim

If TRIM is enabled, an operation will run at the next scheduled time. If TRIM is not enabled, you can run a manual TRIM using:

sudo fstrim -av

Info

It is recommended to enable automatic TRIM to ensure that data usage is reflected accurately between VergeOS and the guest OS.

To enable automatic TRIM, run:

sudo systemctl enable fstrim.timer

For more information on fstrim, visit the man page.

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6

Windows is Unable to Detect a Virtual Disk Drive

Virtual Disk Drive not detected in Windows

In most cases this is because the disk drive Interface Type is set to virtIO-SCSI (which is the default value) on any virtual machine that is being created. If that is the case, you will need to load the virtIO drivers during the Windows installation process. Windows does not natively recognize virtIO interfaces, so it cannot see the virtual disk.

Open Source virtIO drivers can be downloaded from here courtesy of Fedora Linux.

If Redhat virtIO drivers are required due to signed driver needs within Windows, a Redhat account will need to be made on their website and the drivers downloaded directly from them.

Installing virtIO Drivers during Windows Install

During a Windows installation, users can use the console interface tools to change the CD-ROM image to the newly downloaded virtIO drivers iso. Information on using the console interface tools can be found in the inline help within the category titled VDI under the section Using the Console.

Alternatively, you can choose to change the virtual disk drive Interface type to SATA and Windows will find the virtual disk and continue with the installation. !! info "This will come with a performance impact due to SATA drivers being software emulated."

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Windows - Slow to Format a New Disk

Formatting a Virtual Disk with Windows 2012 (and later) Hosts May Take Longer Than Expected

Windows Server 2012 (and later) hosts will, by default, issue SCSI TRIM and Unmap commands equivalent to the entire size of the virtual disk. This behavior is the same even if the "Perform a quick format" option is checked, which significantly slows down the format process.

It is possible to disable the SCSI TRIM and Unmap feature on the host for the duration of the format.

To Disable TRIM and Unmap

Using a Windows CMD window on the host, issue the following command:

fsutil behavior set DisableDeleteNotify 1

To Re-enable the Feature

Use the following command to re-enable the Trim and Unmap feature:

fsutil behavior set DisableDeleteNotify 0

To Verify the Current Setting

You can verify the current Trim and Unmap setting by issuing the following command:

fsutil behavior query DisableDeleteNotify

The output will show one of the following:

  • DisableDeleteNotify=0 - The 'Trim and Unmap' feature is on (enabled).
  • DisableDeleteNotify=1 - The 'Trim and Unmap' feature is off (disabled).

Affected Versions

  • Only Windows Server 2012 and later hosts are affected. All earlier versions (e.g., Windows 2008) do not exhibit the same issue.

Non-server Versions

  • Non-server versions of Windows (e.g., Windows 8.x and 10.x) do not support the DisableDeleteNotify parameter.

Document Information

  • Last Updated: 2024-08-29
  • VergeOS Version: 4.12.6

Windows Restored VM Not Bootable

After restoring a copy of a virtual machine from a recent snapshot, the restored copy may fail to boot properly. The VM may stop with a blue screen message which reads:

Your PC ran into a problem and needs to restart. We're just collecting some error info, and then we'll restart for you.

There are several guest-level issues that can cause a VM running Windows to not start successfully. Below are the most common causes and their corresponding solutions.

Common Causes and Solutions

1. Non-Quiesced Snapshots

One of the most frequent causes of a restored VM failing to boot is that the snapshot was not taken in a clean (Quiesced) state. A quiesced snapshot ensures that the VM's memory and disk I/O are in a stable state, making the restored VM more likely to boot successfully. Without quiescing, the snapshot could have captured an unstable or inconsistent state.

Solution:
  • Ensure that when snapshots are taken, they are quiesced. Quiescing allows the OS to pause I/O operations, flush memory, and ensure that no incomplete transactions are saved in the snapshot.
  • For future backups, enable the Quiesce Snapshots option when scheduling backups for Windows VMs. This feature ensures that the system is in a stable state before taking a snapshot.

2. Pending or Partially Installed Windows Updates

If Windows updates were partially installed or in progress when the snapshot was taken, the restored VM might experience issues booting due to an incomplete or corrupted update state.

Solution:
  • Boot the VM into Safe Mode and complete any pending updates.
  • You can also attempt to disable the Windows Update service temporarily to allow the VM to boot without applying incomplete updates. Once booted, manually re-enable and check for updates.
  • Review the Windows Update Logs using Event Viewer to identify any problematic updates that might need to be rolled back or reinstalled.

3. Driver Incompatibility or Missing Drivers

Sometimes, the VM's hardware configuration in VergeOS (e.g., disk controllers, network adapters) may differ from the original environment, causing issues with booting due to incompatible or missing drivers. This is especially common when restoring VMs from a different hypervisor.

Solution:
  • Boot the VM using Windows Recovery and attempt to repair the system automatically.
  • Verify that the appropriate Virtio or SCSI drivers are installed, especially if the VM is using Virtio interfaces for storage or networking.
  • If the issue persists, boot into Safe Mode and manually update the VM's drivers from the Device Manager.

4. Corrupted Boot Loader

If the Windows bootloader was corrupted in the snapshot, the restored VM will not boot properly. This could happen if the system was performing a critical task related to the boot process (like an update or disk operation) when the snapshot was taken.

Solution:
  • Use the Windows Recovery Environment (WinRE) to repair the bootloader: 1. Boot the VM using a Windows installation disk or recovery media. 2. Select Repair your computer > Troubleshoot > Advanced options > Startup Repair. 3. If Startup Repair doesn’t work, open a Command Prompt and run the following commands: 
    bootrec /fixmbr
bootrec /fixboot
bootrec /rebuildbcd
  • These commands will repair the Master Boot Record (MBR) and rebuild the Boot Configuration Data (BCD).

5. Hardware Configuration Changes

Changes to the VM’s hardware configuration, such as CPU count, memory allocation, or disk type, may cause instability or prevent the VM from booting.

Solution:
  • Verify that the VM’s hardware configuration in VergeOS matches the original configuration from when the snapshot was taken.
  • If you made any changes, such as increasing memory or changing the number of CPUs, try reverting to the original configuration to see if the VM boots properly.

Best Practice: Manage Windows Updates in Guest VMs

A guest VM running Windows OS, and experiencing an unexpected restart, is often found to be caused by the Microsoft Windows Update service being configured to automatically apply updates that frequently require a restart.

Recommendations: - Schedule snapshot creation during maintenance windows when Windows updates are not being applied. - Configure Windows Update settings to avoid automatic installations or reboots, especially on critical VMs. Instead, use a manual update process during scheduled maintenance periods. - Regularly review the Windows Update logs in Event Viewer to detect potential issues related to updates that could affect the stability of the VM.

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6

Workloads Failing to Migrate

Reasons That Workloads May Fail to Migrate

A workload is any process that is running on a node. Common workloads include Virtual Machines (VM), NAS Services, Networking, and Tenant Nodes.

The main reasons a workload fails to migrate from one node to another in the system are:

  • Insufficient available resources: There may not be enough resources (such as RAM) on the target node to run the workload you're trying to migrate. Check the amount of RAM consumed by the workload (VM or Tenant node), then review the resources available on the target node.

  • Pinned VM configuration: A VM may be pinned to a specific node. Review the VM’s settings and check the CPU Type setting. If the CPU Type is set to Host Processor, the VM will be unable to migrate. In this case, the VM must be powered off before it can be migrated successfully.

  • Tenant node migration issues: Tenant nodes may also face migration issues for the same reasons as listed above. Log into the Tenant User Interface, and check the following:

  • Verify that each Tenant node has sufficient available resources to host the migrating tenant workloads.
  • Verify that each Tenant VM is not configured with the CPU Type set to Host Processor.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Proper Power Sequence

Proper Shutdown Sequence for a VergeOS Environment

To power off a cluster (a collection of two or more nodes) follow these steps:

  1. Check any running workloads on each node of the cluster. Navigate to the node dashboard for each node and review the Running Machines section.
  2. If there are tenants running on any of the nodes, log into those tenant environments and gracefully shut down all running workloads.
  3. Power off all running workloads on each node, including VMs, tenant nodes, VMware backup services, and NAS services (if applicable).

vNet Containers

There is no need to manually stop any running vNet containers; they will be gracefully stopped automatically in the subsequent steps.

  1. After stopping all running workloads, navigate to the Cluster dashboard for the cluster you wish to power off.
  2. Select Power Off from the left-hand menu to begin shutting down each node in the cluster.
  3. Finally, navigate to System -> Clusters and select Power Off in the left menu to power off the entire cluster.

IMPORTANT

If an environment contains multiple clusters, ALWAYS shut down the cluster containing the controller nodes (Node1 & Node2) LAST.

Proper Power On Sequence for a VergeOS Environment

To properly power on a VergeOS environment, perform the following steps:

  1. Power on Node1.
  2. Once Node1 is online, power on Node2.
  3. Power on all other nodes, waiting approximately 1 minute between power actions.
  4. On the main dashboard, verify that the environment is Green and Online.

main-dash-stoplights.png

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Managing Media Images

How To Create, Upload, and Manage Media Images

The Media Images section provides a central location to upload files for use in VergeOS. It supports ISOs, VM disk images, logos for custom branding, and general file sharing between sites and tenants.

How to Upload Files to Media Images

To upload a media image, ensure that the file is in one of the supported formats.

Other extensions can be uploaded to the server but may not be recognized as usable by VergeOS.

  1. From the Main Dashboard, select Media Images from the left menu and click Upload.
  2. You may also choose Upload from URL if you are sharing a file from another site or have a URL.

Once the file transfer completes, it will be available in the Media Images list for use.

  1. To create a Public Link, select a file and click Add Public Link from the left menu.

  2. Choose the format for sharing the file:

  • Anonymous (uuid): Appends the file's UUID to the end of the link.
  • Custom: Allows for a custom name to be used in the link.
  • Use file name: Appends the existing file name and extension to the link.
Expiration Type
  • Never Expire: The link will remain active indefinitely.
  • Set Date: Set a specific expiration date and time for the link.

The Public Link can be shared with other systems, for general file sharing, or with local tenants to provide access without requiring an internet download. However, sharing via this method uses network bandwidth. For a more efficient way to share files to a tenant, see the Add Media to Tenants guide.

In the Media Images section, you can also:

  • Manipulate Public Links
  • Download files
  • Edit file names and storage tiers
  • View and Remove References to files
  • Delete files."

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6

Customizing the VergeOS User Interface

Every layer of tenancy in VergeOS has the ability to customize its branding (if allowed). This guide will walk you through how to modify the user interface (UI) with your own branding elements, including logos, colors, and favicons.

How to Change the Branding

Follow these steps to customize your VergeOS environment:

  1. Upload the desired logo in .png or .jpg format to Media Images. - For best results, use a 144x36 image for the large logo and 44x44 for the small logo.

  2. If desired, upload a favicon in .ico format to Media Images.

  3. From the Main Dashboard, navigate to System -> Settings -> Edit UI Branding.

  4. Check the "Enabled" box if it is unchecked to reveal the settings for custom branding.

Once enabled, you'll see several UI customization options.

UI Branding Customization Options

Here are all the elements you can modify for VergeOS branding:

  • Background Color
  • Default: #303030
  • Custom Logo (Large)
  • Upload a .jpg or .png file with dimensions 144x36.
  • Custom Logo (Small)
  • Upload a .jpg or .png file with dimensions 44x44.
  • Custom Favicon
  • Upload an .ico file for the favicon.

Left Navigation

  • Background Color
  • Default: #212a31
  • Text Color
  • Default: #ffffff
  • Hover Over Background Color
  • Default: #647ee6
  • Hover Over Text Color
  • Default: #ffffff

Header/Footer

  • Background Color
  • Default: #3f5ccc
  • Text Color
  • Default: #ffffff

Pop-Up Menus

  • Title Background Color
  • Default: #303030
  • Title Text Color
  • Default: #ffffff

Buttons

  • Background Color
  • Default: #303030
  • Text Color
  • Default: #ffffff
  • Hover Over Background Color
  • Default: #647ee6
  • Hover Over Text Color
  • Default: #ffffff

Miscellaneous

  • Dashboard Panes Hover Over Text Color
  • Default: #647ee6

Best Practices for UI Customization

  • Consistency: Ensure that the colors and logos you choose for the branding align with your organization's style guide for consistency.
  • Readability: Use contrasting colors for text and background elements to ensure readability across different devices.
  • Logo Dimensions: Upload logos with the correct dimensions to prevent distortion or scaling issues.

By following these steps and using the available options, you can fully customize the VergeOS UI to reflect your organization's brand. Make sure to save your settings once you're satisfied with the changes.

Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6
  • 1 min read

KB Template


Document Information

  • Last Updated: 2024-08-29
  • vergeOS Version: 4.12.6