Skip to content

Maintenance#

vSAN Diagnostics Guide

Overview

This guide provides comprehensive information about the vSAN diagnostic options available in the user interface. These diagnostic tools enable system administrators to monitor, troubleshoot, and maintain vSAN deployments effectively.

Critical Warning

The diagnostic commands detailed in this guide are powerful administrative tools. Improper usage can result in: - System outages - Service interruptions - Potential data loss

Exercise extreme caution and ensure proper understanding before execution.

Prerequisites

To use these diagnostic tools, you must have:

  • Root-level access to your VergeIO cluster
  • Note: Tenants do not have a vSAN.

Accessing vSAN Diagnostics

  1. Navigate to vSAN Diagnostics using either method:
  • From the home screen: Select the vSAN Tiers count box → vSAN Diagnostics (left menu)

  • Alternative path: Home screen → System (left menu) → vSAN Diagnostics 2. Command execution:

  • Select desired command from the dropdown menu

  • Configure available options if applicable
  • Click SEND→ to execute

Command Visibility

Enable the "Show Command" option to view the exact command being executed. This can be valuable for: - SSH execution - BASH script integration - Advanced command automation

Diagnostic Commands

Add Drive to vSAN

Running this command allows you to manually add a drive via the UI. Drives are normally added either during the installation, or via the Nodes > Nodes Drives page. However adding them in that way does not allow for the addition of drives to Tier0.

Prerequisites:

  • Drive must be physically present in the system
  • Drive must be visible from Nodes > Nodes Drives page

Usage Parameters:

  • Selecting Add Drive to vSAN
  • From the right menu, select the Node that we will be adding the drive to.
  • Enter the appropriate path, E.G. `/dev/nvme0n1```<br>
  • You can use the "Click here to view devices" to get the path. !!! warning "The contents of this drive will be overwritten."
  • Select the Tier you want to assign the drive to.
  • Check the Swap box if you want Swap enabled on this drive. !!! info "This will use the cluster settings for the Swap size."
  • Verify. You will need to TYPE Yes I know what I'm doing in the Verify box.
  • Select SEND →

CLI Syntax:

vcmd newdevice --path=PATH [OPTIONS]
  --path=PATH    Path to target device
  --tier=NUM     Tier number assignment

Cancel Integrity Check

Terminates any active integrity check operations. See Integrity Check for additional information.

CLI Syntax:

vcmd cancelintegcheck

Clear Reference Counts

Reference counts are how the vSAN tracks the number of times a file is referenced in the vSAN. Clearing this count will force a full vSAN walk and a refresh of the Reference Counts.

Function:

  • Clears existing reference counts
  • Initiates full vSAN traversal
  • Rebuilds reference count data

Support Authorization Required

Execute only under direct support guidance.

Usage Parameters:

  • Verify. You will need to TYPE Yes I know what I'm doing in the Verify box.
  • Select SEND →

CLI Syntax:

vcmd clearrefcounts

Find Inode

Running this query will allow you to find out what an Inode (Index Node) referrences. Inode is a data structure that stores information about a file or directory, such as its owner, access rights, date and time of creation and modification, size and location on the vSAN. Each file or directory in the system has its own unique index node number (inode number), which can be used to perform various operations with a file or directory. This can be used to troubleshoot errors in the vSAN.

Purpose:

  • Retrieves inode reference information
  • Maps inode numbers to filesystem entities
  • Assists in vSAN troubleshooting

CLI Syntax:

find /vsan -inum inode_number_here -printf /%P\n

Get Cache Info

Retrieves detailed cache information for specified nodes.

Output Information:

  • Total cache capacity
  • Available cache space
  • Cache page statistics
  • Performance metrics

CLI Syntax:

vcmd getcacheinfo

Get Clients

Retrieves client connection information for specified nodes.

Output Information:

  • Connected node information
  • IP address mappings
  • Worker thread statistics

CLI Syntax:

vcmd getclients

Get Cluster Rates

Retrieves cluster-wide performance metrics.

Output Information:

  • Read/write rates
  • Throttle status
  • Performance statistics

CLI Syntax:

vcmd getclusterrates

Get Cluster Usage

Provides cluster-wide storage utilization information.

Output Information:

  • Maximum storage capacity
  • Current utilization
  • Repair operation counts

CLI Syntax:

vcmd getclusterusage

Get Current Master

Retrieves master node information from each cluster member.

Output Information:

  • Master node identification
  • Online status
  • Transaction logging information

CLI Syntax:

vcmd getcurmaster

Get Device Integrity

Retrieves integrity check results for specified nodes.

CLI Syntax:

vcmd getdeviceinteg

Get Device List

Provides comprehensive device inventory.

Output Information:

  • Device identifiers
  • System paths
  • Tier assignments

CLI Syntax:

vcmd getdevicelist

Get Device Status

Retrieves detailed device status information.

Output Information:

  • Device paths
  • Operational status
  • Capacity metrics
  • Performance statistics

CLI Syntax:

vcmd getdevicestatus

Get Device Usage

Provides device utilization metrics.

Output Information:

  • Total capacity
  • Current utilization
  • Usage trends

CLI Syntax:

vcmd getdeviceusage

Get File Status

Retrieves detailed file metadata.

Output Information:

  • Inode information
  • File type
  • Tier assignment
  • Hash key data

CLI Syntax:

vcmd stat /path/to/file.raw

Get Fuse Info

Retrieves FUSE (Filesystem in Userspace) statistics.

Output Information:

  • Mount point information
  • Thread statistics
  • Throttling metrics

CLI Syntax:

vcmd getfuseinfo

Get Integrity Check Status

Retrieves results from the most recent integrity check.

Output Information:

  • Check status
  • Path information
  • Temporal data
  • Verification results

CLI Syntax:

vcmd getintegcheckstatus

Get Journal Status

Retrieves journal system status information.

Output Information:

  • Operational status
  • Redundancy status
  • System metadata

CLI Syntax:

vcmd getjournalstatus

Get Node Device List

Retrieves detailed hardware information for storage devices.

Output Information:

  • Driver information
  • Model specifications
  • Firmware versions
  • Physical attributes

CLI Syntax:

vcmd getnodedevicelist

Get Node Info

Retrieves comprehensive node configuration data.

Output Information:

  • Node identification
  • Cluster configuration
  • System parameters
  • Operational status

CLI Syntax:

vcmd getnodeinfo

Get Node List

Provides cluster-wide node inventory.

Output Information:

  • Node identification
  • Online status
  • Version information
  • Tier utilization

CLI Syntax:

vcmd getnodelist

Get Path from Inode

Resolves filesystem paths from inode numbers.

CLI Syntax:

vcmd getpathfromino $1

Get Read Ahead

Retrieves read-ahead buffer statistics.

Output Information:

  • Queue statistics
  • Thread utilization
  • System status

CLI Syntax:

vcmd getreadahead

Get Repair Status

Monitors ongoing repair operations.

Output Information:

  • Device repair status
  • Operation progress
  • System health

CLI Syntax:

vcmd getrepairstatus

Get Running Configuration

Retrieves active system configuration.

Output Information:

  • Worker thread allocation
  • System throttles
  • Operational parameters

CLI Syntax:

vcmd getrunningconf

Get Sync List

Monitors synchronization operations.

Output Information:

  • Operation frequency
  • Start times
  • File processing status

CLI Syntax:

vcmd getsynclist

Get Tier Device Maps

Retrieves tier-to-device mapping information.

Output Information:

  • Physical device mappings
  • Tier assignments
  • System configuration

CLI Syntax:

vcmd gettierdevicemaps

Get Tier Node Maps

Retrieves tier-to-node mapping information.

Technical Details:

  • Base-0 indexing (0=Node1, 1=Node2, etc.)
  • 65536 buckets per tier map
  • Primary (tier_x.0) and redundant (tier_x.1) mappings

CLI Syntax:

vcmd gettiernodemaps

Get Tier Status

Retrieves comprehensive tier health information.

Output Information:

  • Redundancy status
  • Walk statistics
  • Transaction data
  • Health metrics

CLI Syntax:

vcmd gettierstatus

Get Top Usage Rates

Monitors real-time I/O statistics.

Real-time Data

Multiple executions may be necessary for trend analysis.

CLI Syntax:

vcmd getfhlist | grep -Eo '(ino|rrate|wrate)\b.*'

Get Volume Usage

Retrieves detailed volume utilization statistics.

Parameters:

  • Path specification (optional)
  • Recursive flag
  • Human-readable output
  • Preferred tier display

CLI Syntax:

vcmd getvolusage --path=/ --recursive=1 --human=1

Integrity Check

Initiates system integrity verification.

Parameters:

  • Path specification (required)
  • Recursive operation
  • Fix mode (destructive)
  • Meta-tier only option

Data Loss Risk

Fix mode zeros bad blocks. THIS IS DESTRUCTIVE. Use only under support guidance.

CLI Syntax:

vcmd integcheck /vol

Integrity Check Device

Performs device-level integrity verification.

Parameters:

  • Node selection
  • Device ID (-1 for all devices)

CLI Syntax:

vcmd integcheckdevice --id=x

Summarize Disk Usage

Generates storage utilization summaries.

Parameters:

  • Path specification
  • Recursive operation
  • Preferred tier display
  • Deduplication analysis
  • Fast deduplication option

CLI Syntax:

vcmd du /vol

Additional Resources

Feedback

Need Help?

If you need further assistance or have any questions about this article, please don't hesitate to reach out to our support team.

Document Information

  • Last Updated: 2024-12-27
  • VergeOS Version: 4.13.2

Virtual Wire Setup and Use

A virtual wire provides a tenant the ability to access a VLAN outside the VergeOS environment without going through routing steps.

Prerequisite Steps

  1. Add the desired VLAN(s) to the appropriate switch ports so they are accessible to the nodes running the VergeOS environment.
  2. Determine whether the tenant will need access to a single VLAN or multiple VLANs. This will determine the virtual wire configuration.

Warning

VLANs 1 & 100-102 cannot be used in a virtual wire capacity. These VLANs are reserved for internal traffic. They can, however, be remapped to another VLAN for tenant consumption.

Info

If a tenant requires access to more than 1 or 2 VLANs, it is recommended to configure the virtual wire in Trunk Mode.

Creating a 1:1 Virtual Wire

  1. Ensure the VLAN(s) have been created in the VergeOS UI. If not, follow the steps to create VLAN(s) here.
  2. From the Main Dashboard, select Networks in the left menu to open the Networks Dashboard.
  3. Select Virtual Wires in the left menu to view all virtual wires in the environment.
  4. Select New to create a new virtual wire.
  5. Enter the following settings: virtual-wire-create-settings.png

Info

  • The Network dropdown will list all networks inside the environment. Choose the network with the corresponding VLAN to pass into the tenant.
  • The Destination Wire dropdown will automatically select Empty List if no unconnected virtual wires are detected.
  • Leave the PVID field set to 1.
  1. Submit your changes and return to the virtual wires list view.
  2. Select New to create the second half of the virtual wire.
  3. Enter the following settings: virtual-wire-create-settings-tenant.png

Info

  • In the Network dropdown, select the tenant network that the VLAN will be passed to, typically named tenant_'$TENANTNAME'.
  • The Destination Wire dropdown will automatically select the other half of the virtual wire created earlier.
  • Change the PVID field to the actual VLAN ID of the network being attached.
  1. Submit your changes.
  2. Navigate to the Networks Dashboard, select Networks, and apply the rules for the networks connected by the virtual wires.

Creating a Trunk Mode Virtual Wire

Warning

To use Trunk Mode Virtual Wires, the corresponding "Physical Network" (tied to node NICs) must be set to bridge mode.

Warning

If the external network is in a VLAN and the physical NIC that the external network references is in bridge mode, trunking a virtual wire from the bridge will not work.

Setting a Physical Network to Bridge Mode

  1. Navigate to Networks in the left menu to access the Networks Dashboard.
  2. Select Networks again to view all networks in the environment.
  3. Double-click the Physical Network (NIC) that the VLANs are trunked to on the physical switch. !!! info A "Physical Network" typically has "Switch" appended and represents a physical NIC on a node.
  4. Select Edit to enter the network configuration page.
  5. In the configuration page, enable Physical Bridged to activate Bridge Mode.

The "On Power Loss" setting can remain as Last State or Power On.

  1. Submit your changes.
  2. Reboot the necessary nodes for Bridge Mode to become active.

Configuring a Trunk Mode Virtual Wire

  1. Ensure the "Physical Network" is set to Bridged Mode and is powered on.
  2. From the Main Dashboard, select Networks and then Virtual Wires.
  3. Select New to create the first half of the virtual wire.
  4. Enter the following settings: vw-trunk-host.png

Info

  • Select the corresponding Physical Network in the Network dropdown.
  • Set the PVID field to 0.
  • Enter the allowed VLANs in the Allowed VLAN List, comma-delimited and with ranges as necessary.
  1. Submit your configuration.
  2. Select New to create the second half of the virtual wire.
  3. Enter the following settings: vw-trunk-tenant.png

Info

  • Select the tenant network in the Network dropdown.
  • Set the PVID field to 0.
  • Enter the allowed VLANs in the Allowed VLAN List.
  1. Submit your changes.
  2. Apply the rules for the connected networks as described above.

Adding VLANs Inside the Tenant

  1. Navigate to the tenant UI and log in.
  2. From the Main Dashboard, navigate to Networks, then select New External.
  3. Enter the following settings: virtual-wire-network-in-tenant.png

For the interface network, select Physical.

  1. Submit your configuration.
  2. Attach workloads to the network for Layer 2 access to networks outside of Verge.io.

Troubleshooting Steps

Traffic is not reaching the virtual machine

  • Confirm firewall rules related to the virtual wire have been applied.
  • Verify the destination tenant network and VLAN network are in the "Running" state and reside on the same physical node.
  • Ensure VLANs are trunked to the correct physical node ports.

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6

How to Update Your VergeOS Environment

WARNING: DO NOT SKIP MAJOR VERSIONS

For example, if you are on version 4.8, please update to version 4.9 before updating to 4.10. Skipping versions could cause major database and configuration issues.

The VergeOS platform supports 'zero downtime' updating. This means that during a routine update process, guest workloads (VMs and tenant environments) can remain on and running as normal.

For more information on the update process, please refer to our Product Guide.

The time required to complete an update varies depending on factors such as: - Number of nodes in the system - Storage consumed on the vSAN - Data change rate generated by workloads during the update - Hardware performance (processor speed, disk type, network speed)

During the update, you may see tiers and drives alternate between Online, Verifying, or Repairing statuses. This is normal as the system verifies data integrity.

Before performing an update, verify that your system memory usage is within limits. For example, in a two-node configuration, total system memory should be under 50% utilization to allow the remaining node to handle 100% of workloads.

Ensure no nodes are in Maintenance mode before proceeding with the update.

Update Procedure

  1. Log in to the VergeOS UI.

  2. Navigate to System > Updates, then select Check for Updates in the left menu.

    • A pop-up will prompt Yes or No, select Yes.
    • If a banner appears stating "A new minor version is available on a different branch," follow the prompt to change branches by selecting Change Branch in the left menu. Confirm with Yes. VergeOSupgrade-new-img2.png

  3. The packages to be downloaded will now be highlighted. VergeOSupgrade-new-img3.png

    • Select Download in the left menu.
    • A pop-up will prompt Yes or No, select Yes.
    • The download process will appear on the dashboard in the Current Update Server tile.

  4. Once the download completes, the Install action will become available.

    • Select Install when ready.
    • Confirm with Yes to begin the install.

  5. After installation, a request to reboot the system will occur. - Select Reboot in the left menu to initiate a rolling reboot across all nodes.

Note

The update will start with Node 1, putting it into maintenance mode and migrating workloads to another node. During minor version changes, you may briefly lose access to the UI as it fails back to Node 1. This is normal and should last no more than a minute or two. Workloads should not experience network issues.

Troubleshooting Steps

Workloads Failing to Migrate

  • This error is usually due to insufficient resources (RAM) in the cluster. Try migrating other workloads or adjust RAM usage. More causes and solutions are detailed here.

vSAN Taking a Long Time to Verify

  • The verification process depends on factors like network speed, disk type, and consumed data. HDDs will take longer than NVMe or SSDs. On VergeOS versions 4.9.0 and higher, check the Full Walk Progress on the tiers dashboard for an indication of how far along the verification is.

    walk-percentage.png

WARNING

This process must complete before rebooting any additional nodes. Failure to do so can result in a double failure, causing workloads to crash.

Unable to Connect to Update Server

  • Ensure the system has a working DNS server on its external (UI) network:

    1. Navigate to the external network dashboard.
    2. Select Diagnostics in the left menu.
    3. Set Query to DNS Lookup and select Send.
    4. If DNS is properly configured, the response will display Verge.io's IP address. If not, check DNS settings and retry the query.

  • Expired Update Server credentials can also cause this issue. These are tied to the system's license and should be renewed through your VergeOS sales representative.

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6