Skip to content

Maintenance#

Network Diagnostics Guide

Overview

This guide provides comprehensive information about the network diagnostic options available in the user interface. These diagnostic tools enable system administrators to monitor, troubleshoot, and maintain Verge deployments effectively.

Critical Warning

The diagnostic commands detailed in this guide are powerful administrative tools. Improper usage can result in:

  • System outages
  • Service interruptions
  • Potential data loss

Exercise extreme caution and ensure proper understanding before execution.

Prerequisites

To use these diagnostic tools, you must have:

  • UI access to your VergeIO cluster
  • Note: Tenants will have their own networking, and therefore their own Network Diagnostics page.

Accessing Network Diagnostics / Issuing Diagnostic Commands

  1. Navigate to Network Diagnostics using either method:
  • From the home screen: Select the Networks count box → Networks (left menu) → Select a network (right side of the screen) → Diagnostics

  • Alternative path: Home screen → Networks (left menu) → Select a network (right side of the screen) → Diagnostics 2. Command execution:

  • Select desired command from the dropdown menu

  • Configure available options if applicable
  • Click SEND→ to execute

Command Visibility

Enable the "Show Command" option to view the exact command being executed. This can be valuable for:

  • SSH execution
  • BASH script integration
  • Advanced command automation

ARP Scan

Purpose: Scans the local network using ARP (Address Resolution Protocol) packets to discover active devices.

Details:

  • Sends ARP requests to all possible addresses in the specified network
  • Displays MAC and IP addresses of responding devices
  • Used for network discovery and inventory

CLI Syntax: 

lxc-attach -n vnet3 -- arp-scan -l -I [interface]

ARP Table

Purpose: Displays the current ARP cache with IP addresses instead of hostnames.

Details:

  • Shows the ARP table maintained by the kernel
  • The -n flag prevents DNS lookups (displays numeric IP addresses)
  • Lists MAC addresses associated with IP addresses the system has communicated with

CLI Syntax: 

lxc-attach -n vnet3 -- arp -n

DHCP Release/Renew

Purpose: Releases the DHCP address for the selected interface, then attempts to renew it.

Details:

  • This command sequence effectively performs a "release and renew" operation for DHCP-assigned IP addresses:
  • Release the current IP address (USR2 signal)
  • Waits for 2 seconds (sleep 2)
  • Request a new IP address (USR1 signal)

CLI Syntax: 

lxc-attach -n vnet3 -- busybox sh -c killall -USR2 udhcpc ; sleep 2 ; killall -USR1 udhcpc

DNS Lookup

Purpose: Performs DNS lookups for specified query types.

Details:

  • Used for troubleshooting DNS issues and domain information gathering
  • Common query types:
  • A: IPv4 address records
  • AAAA: IPv6 address records
  • MX: Mail exchange records
  • NS: Name server records
  • TXT: Text records
  • CNAME: Canonical name records

CLI Syntax: 

lxc-attach -n vnet3 -- host -t Query_Type DNS_Name

FRRouting / BGP/OSPF

Purpose: Allows the configuration of FRRouting.

Details: For more information on other values and variables, refer to FRR documentation & Configuring BGP Hold Down Timers.

CLI Syntax: 

lxc-attach -n vnet3 -- vtysh -c command_goes_here

IP Commands

Purpose: Allows for the configuration of IP and it's subsequent related items.

Address

Purpose: Displays and configures network interface addresses.

Details:

  • Shows IP addresses, subnet masks, and interface states
  • Displays both IPv4 and IPv6 addresses

CLI Syntax: 

lxc-attach -n vnet3 -- /sbin/ip address

Connection Tracking

Purpose: Displays the contents of Netfilter's Connection Tracking file which contains information about network connections.

Details: This file displays the current state of all tracked network connections on the system. Each line in the file represents a single tracked connection. The contents typically include entries with fields such as:

  • Protocol (tcp, udp, icmp)
  • Connection states (ESTABLISHED, TIME_WAIT, etc.)
  • Source and destination IP addresses and ports
  • Connection timeouts
  • Packet and byte counts
  • NAT information if applicable

CLI Syntax: 

lxc-attach -n vnet3 -- dd bs=262144 count=1 if=/proc/net/nf_conntrack

Purpose: Displays network interfaces at layer 2 (data link layer).

Details: - Shows interface states, MTU values, and MAC addresses

CLI Syntax: 

lxc-attach -n vnet3 -- /sbin/ip link

Multicast Address

Purpose: Displays multicast addresses assigned to interfaces.

Details: - Shows IPv4 and IPv6 multicast addresses - Displays which interfaces are subscribed to which multicast groups - Useful for debugging multicast routing and applications

CLI Syntax: 

lxc-attach -n vnet3 -- /sbin/ip maddress

Multicast Routing Cache

Purpose: Displays multicast routing table.

Details: - Shows active multicast routes - Includes source and group addresses - Displays incoming and outgoing interfaces - Useful for troubleshooting multicast routing issues

CLI Syntax: 

lxc-attach -n vnet3 -- /sbin/ip mroute

Neighbor

Purpose: Displays neighbors (ARP table) of the current device.

Details: - Similar to arp -n. - Shows IPv4 and IPv6 neighbors - Includes MAC addresses and states of neighbors

CLI Syntax: 

lxc-attach -n vnet3 -- /sbin/ip neighbor

Routing Table

Purpose: Displays the IP routing table.

Details: - Shows all routes currently configured on the system - Displays default gateway, network routes, and host routes

CLI Syntax: 

ip route show table all

Rule

Purpose: Displays and manipulates the routing policy database.

Details: - Shows policy-based routing rules - Allows for complex routing setups with multiple routing tables - Used for advanced networking configurations - Rules are evaluated in priority order (lower numbers first)

CLI Syntax: 

lxc-attach -n vnet3 -- /sbin/ip rule

Transform (xfrm) - Policy

Purpose: Displays IPSec policies.

Details: - Shows security policies for IPSec communications - Displays source, destination, protocols, and actions

CLI Syntax: 

lxc-attach -n vnet3 -- /sbin/ip xfrm policy

Transform (xfrm) - State

Purpose: Displays IPSec security associations.

Details: - Shows the current security associations (SAs) for IPSec - Displays encryption algorithms, keys, and related information - Used in conjunction with ip xfrm policy

CLI Syntax: 

lxc-attach -n vnet3 -- /sbin/ip xfrm state

IPsec

Purpose: Allows for the configuration of IPsec and it's subsequent related items.

List Cryptographic Algorithms

Purpose: Lists all algorithms supported by the IPSec stack.

Details: - Displays encryption, authentication, and compression algorithms - Shows available key lengths and other parameters

CLI Syntax: 

lxc-attach -n vnet3 -- ipsec listalgs

List IKE Counters

Purpose: Displays statistics and counters for IPSec connections.

Details: - Shows packet counts, bytes transferred, and errors - Useful for monitoring and troubleshooting IPSec tunnels

CLI Syntax: 

lxc-attach -n vnet3 -- ipsec listcounters

IPsec Show Config

Purpose: Displays the contents of StrongSwan's VPN Configuration file which contains information about network VPN connections.

** Details**: This file contains settings for your VPN tunnels including:

  • Connection definitions
  • Authentication methods
  • Encryption algorithms
  • Network settings
  • Tunnel endpoints
  • Identity information
  • Secret key references

CLI Syntax: 

lxc-attach -n vnet3 -- dd bs=262144 count=1 if=/tmp/vpn/ipsec.conf status=none

Status

Purpose: Displays the status of IPSec connections.

Details: - Shows active IPSec tunnels and their current state - Displays connection names, remote endpoints, and status - Useful for quick verification of IPSec connectivity

CLI Syntax: 

lxc-attach -n vnet3 -- ipsec status

Status All

Purpose: Displays detailed status of all IPSec connections.

Details: - Shows comprehensive information about IPSec tunnels - Includes encryption algorithms, key lifetimes, traffic statistics - Displays connection policies and security associations - Valuable for in-depth troubleshooting of IPSec issues

CLI Syntax: 

lxc-attach -n vnet3 -- ipsec statusall

NMAP

Purpose: Network exploration and security auditing tool.

Details: - Scans networks and hosts for open ports and services - Can determine operating systems and service versions - Supports various scanning techniques (SYN, TCP, UDP, etc.) - Offers script-based vulnerability scanning - Essential tool for network administrators and security professionals

CLI Syntax: 

lxc-attach -n vnet3 -- nmap 192.168.0.1 -p22-100

Ping

Purpose: Tests connectivity to a target host.

Details: - Sends ICMP Echo Request packets and waits for ICMP Echo Reply - Measures round-trip time (latency) to the target - Shows packet loss percentage - Basic but essential network troubleshooting tool - Useful for testing basic connectivity and network performance

CLI Syntax: 

lxc-attach -n vnet3 -- busybox ping -c 1 -W 5 8.8.8.8

Show Firewall Rules

Purpose: Displays the current nftables firewall ruleset.

Details: - Shows all tables, chains, and rules configured in nftables - Replacement for the older iptables command - Provides a comprehensive view of the firewall configuration - Useful for troubleshooting connectivity issues and security auditing

CLI Syntax: 

lxc-attach -n vnet3 -- nft list ruleset

TCP Connection Test

Purpose: Uses netcat to connect to checkip.dyndns.org to determine your public IP address.

Details: - Establishes a TCP connection to the specified host and port - Used to determine external IP address

CLI Syntax: 

lxc-attach -n vnet3 -- busybox nc -w5 checkip.dyndns.org 80

TCP Dump

Purpose: Captures and displays network packets on a specific interface.

Details: - There are multiple Verbose Output options - Checking Show Link-Level Header can aid in VLAN troubleshooting - Expressions can be used to filter the output. - Type qualifiers (host, net, port, portrange) - Direction qualifiers (src, dst) - Protocol qualifiers (ether, ip, ip6, tcp, udp, icmp, arp) - Logical operators (and, or, not) - Advanced filters (greater than, less than, TCP flags, byte offsets)

CLI Syntax: 

lxc-attach -n vnet3 -- busybox timeout 15 tcpdump -lni eth0 -c 100

Top CPU Usage

Purpose: Displays system process information in batch mode for a single iteration.

Details: - Shows CPU, memory, and process details - Useful for system monitoring and troubleshooting performance issues

CLI Syntax: 

lxc-attach -n vnet3 -- /usr/bin/top -b -n 1

Top Network Usage

Purpose: Displays bandwidth usage on a network interface by host.

Details: - Shows real-time bandwidth usage by connection - Useful for identifying which hosts are using the most bandwidth

CLI Syntax: 

lxc-attach -n vnet3 -- busybox timeout 10 /usr/sbin/iftop -tNi eth0 -n

Trace Route

Purpose: Traces the route packets take to a destination.

Details: - Displays each hop (router) between your computer and the destination - Shows round-trip time for each hop - Useful for diagnosing routing issues and network latency problems

CLI Syntax: 

traceroute -n -w 3 google.com

Trace/Debug Firewall Rules

Purpose: Monitors and traces packets as they traverse nftables rules.

Details: - Shows which rules packets match and the resulting actions - Extremely useful for debugging complex firewall configurations - Requires root privileges

CLI Syntax: 

lxc-attach -n vnet3 -- busybox timeout 3 nft -nnn monitor trace

What's My IP

Purpose: Queries OpenDNS to determine your public IP address.

Details: - Simple, reliable method to determine your public IP address - Works even when HTTP-based services might be blocked

CLI Syntax: 

lxc-attach -n vnet3 -- dig +short myip.opendns.com @208.67.222.222

Additional Resources

Feedback

Need Help?

If you need further assistance or have any questions about this article, please don't hesitate to reach out to our support team.

Document Information

  • Last Updated: 2024-03-21
  • VergeOS Version: 4.13.4

vSAN Diagnostics Guide

Overview

This guide provides comprehensive information about the vSAN diagnostic options available in the user interface. These diagnostic tools enable system administrators to monitor, troubleshoot, and maintain vSAN deployments effectively.

Critical Warning

The diagnostic commands detailed in this guide are powerful administrative tools. Improper usage can result in: - System outages - Service interruptions - Potential data loss

Exercise extreme caution and ensure proper understanding before execution.

Prerequisites

To use these diagnostic tools, you must have:

  • Root-level access to your VergeIO cluster
  • Note: Tenants do not have a vSAN.

Accessing vSAN Diagnostics

  1. Navigate to vSAN Diagnostics using either method:
  • From the home screen: Select the vSAN Tiers count box → vSAN Diagnostics (left menu)

  • Alternative path: Home screen → System (left menu) → vSAN Diagnostics 2. Command execution:

  • Select desired command from the dropdown menu

  • Configure available options if applicable
  • Click SEND→ to execute

Command Visibility

Enable the "Show Command" option to view the exact command being executed. This can be valuable for: - SSH execution - BASH script integration - Advanced command automation

Diagnostic Commands

Add Drive to vSAN

Running this command allows you to manually add a drive via the UI. Drives are normally added either during the installation, or via the Nodes > Nodes Drives page. However adding them in that way does not allow for the addition of drives to Tier0.

Prerequisites:

  • Drive must be physically present in the system
  • Drive must be visible from Nodes > Nodes Drives page

Usage Parameters:

  • Selecting Add Drive to vSAN
  • From the right menu, select the Node that we will be adding the drive to.
  • Enter the appropriate path, E.G. `/dev/nvme0n1```<br>
  • You can use the "Click here to view devices" to get the path. !!! warning "The contents of this drive will be overwritten."
  • Select the Tier you want to assign the drive to.
  • Check the Swap box if you want Swap enabled on this drive. !!! info "This will use the cluster settings for the Swap size."
  • Verify. You will need to TYPE Yes I know what I'm doing in the Verify box.
  • Select SEND →

CLI Syntax:

vcmd newdevice --path=PATH [OPTIONS]
  --path=PATH    Path to target device
  --tier=NUM     Tier number assignment

Cancel Integrity Check

Terminates any active integrity check operations. See Integrity Check for additional information.

CLI Syntax:

vcmd cancelintegcheck

Clear Reference Counts

Reference counts are how the vSAN tracks the number of times a file is referenced in the vSAN. Clearing this count will force a full vSAN walk and a refresh of the Reference Counts.

Function:

  • Clears existing reference counts
  • Initiates full vSAN traversal
  • Rebuilds reference count data

Support Authorization Required

Execute only under direct support guidance.

Usage Parameters:

  • Verify. You will need to TYPE Yes I know what I'm doing in the Verify box.
  • Select SEND →

CLI Syntax:

vcmd clearrefcounts

Find Inode

Running this query will allow you to find out what an Inode (Index Node) referrences. Inode is a data structure that stores information about a file or directory, such as its owner, access rights, date and time of creation and modification, size and location on the vSAN. Each file or directory in the system has its own unique index node number (inode number), which can be used to perform various operations with a file or directory. This can be used to troubleshoot errors in the vSAN.

Purpose:

  • Retrieves inode reference information
  • Maps inode numbers to filesystem entities
  • Assists in vSAN troubleshooting

CLI Syntax:

find /vsan -inum inode_number_here -printf /%P\n

Get Cache Info

Retrieves detailed cache information for specified nodes.

Output Information:

  • Total cache capacity
  • Available cache space
  • Cache page statistics
  • Performance metrics

CLI Syntax:

vcmd getcacheinfo

Get Clients

Retrieves client connection information for specified nodes.

Output Information:

  • Connected node information
  • IP address mappings
  • Worker thread statistics

CLI Syntax:

vcmd getclients

Get Cluster Rates

Retrieves cluster-wide performance metrics.

Output Information:

  • Read/write rates
  • Throttle status
  • Performance statistics

CLI Syntax:

vcmd getclusterrates

Get Cluster Usage

Provides cluster-wide storage utilization information.

Output Information:

  • Maximum storage capacity
  • Current utilization
  • Repair operation counts

CLI Syntax:

vcmd getclusterusage

Get Current Master

Retrieves master node information from each cluster member.

Output Information:

  • Master node identification
  • Online status
  • Transaction logging information

CLI Syntax:

vcmd getcurmaster

Get Device Integrity

Retrieves integrity check results for specified nodes.

CLI Syntax:

vcmd getdeviceinteg

Get Device List

Provides comprehensive device inventory.

Output Information:

  • Device identifiers
  • System paths
  • Tier assignments

CLI Syntax:

vcmd getdevicelist

Get Device Status

Retrieves detailed device status information.

Output Information:

  • Device paths
  • Operational status
  • Capacity metrics
  • Performance statistics

CLI Syntax:

vcmd getdevicestatus

Get Device Usage

Provides device utilization metrics.

Output Information:

  • Total capacity
  • Current utilization
  • Usage trends

CLI Syntax:

vcmd getdeviceusage

Get File Status

Retrieves detailed file metadata.

Output Information:

  • Inode information
  • File type
  • Tier assignment
  • Hash key data

CLI Syntax:

vcmd stat /path/to/file.raw

Get Fuse Info

Retrieves FUSE (Filesystem in Userspace) statistics.

Output Information:

  • Mount point information
  • Thread statistics
  • Throttling metrics

CLI Syntax:

vcmd getfuseinfo

Get Integrity Check Status

Retrieves results from the most recent integrity check.

Output Information:

  • Check status
  • Path information
  • Temporal data
  • Verification results

CLI Syntax:

vcmd getintegcheckstatus

Get Journal Status

Retrieves journal system status information.

Output Information:

  • Operational status
  • Redundancy status
  • System metadata

CLI Syntax:

vcmd getjournalstatus

Get Node Device List

Retrieves detailed hardware information for storage devices.

Output Information:

  • Driver information
  • Model specifications
  • Firmware versions
  • Physical attributes

CLI Syntax:

vcmd getnodedevicelist

Get Node Info

Retrieves comprehensive node configuration data.

Output Information:

  • Node identification
  • Cluster configuration
  • System parameters
  • Operational status

CLI Syntax:

vcmd getnodeinfo

Get Node List

Provides cluster-wide node inventory.

Output Information:

  • Node identification
  • Online status
  • Version information
  • Tier utilization

CLI Syntax:

vcmd getnodelist

Get Path from Inode

Resolves filesystem paths from inode numbers.

CLI Syntax:

vcmd getpathfromino $1

Get Read Ahead

Retrieves read-ahead buffer statistics.

Output Information:

  • Queue statistics
  • Thread utilization
  • System status

CLI Syntax:

vcmd getreadahead

Get Repair Status

Monitors ongoing repair operations.

Output Information:

  • Device repair status
  • Operation progress
  • System health

CLI Syntax:

vcmd getrepairstatus

Get Running Configuration

Retrieves active system configuration.

Output Information:

  • Worker thread allocation
  • System throttles
  • Operational parameters

CLI Syntax:

vcmd getrunningconf

Get Sync List

Monitors synchronization operations.

Output Information:

  • Operation frequency
  • Start times
  • File processing status

CLI Syntax:

vcmd getsynclist

Get Tier Device Maps

Retrieves tier-to-device mapping information.

Output Information:

  • Physical device mappings
  • Tier assignments
  • System configuration

CLI Syntax:

vcmd gettierdevicemaps

Get Tier Node Maps

Retrieves tier-to-node mapping information.

Technical Details:

  • Base-0 indexing (0=Node1, 1=Node2, etc.)
  • 65536 buckets per tier map
  • Primary (tier_x.0) and redundant (tier_x.1) mappings

CLI Syntax:

vcmd gettiernodemaps

Get Tier Status

Retrieves comprehensive tier health information.

Output Information:

  • Redundancy status
  • Walk statistics
  • Transaction data
  • Health metrics

CLI Syntax:

vcmd gettierstatus

Get Top Usage Rates

Monitors real-time I/O statistics.

Real-time Data

Multiple executions may be necessary for trend analysis.

CLI Syntax:

vcmd getfhlist | grep -Eo '(ino|rrate|wrate)\b.*'

Get Volume Usage

Retrieves detailed volume utilization statistics.

Parameters:

  • Path specification (optional)
  • Recursive flag
  • Human-readable output
  • Preferred tier display

CLI Syntax:

vcmd getvolusage --path=/ --recursive=1 --human=1

Integrity Check

Initiates system integrity verification.

Parameters:

  • Path specification (required)
  • Recursive operation
  • Fix mode (destructive)
  • Meta-tier only option

Data Loss Risk

Fix mode zeros bad blocks. THIS IS DESTRUCTIVE. Use only under support guidance.

CLI Syntax:

vcmd integcheck /vol

Integrity Check Device

Performs device-level integrity verification.

Parameters:

  • Node selection
  • Device ID (-1 for all devices)

CLI Syntax:

vcmd integcheckdevice --id=x

Summarize Disk Usage

Generates storage utilization summaries.

Parameters:

  • Path specification
  • Recursive operation
  • Preferred tier display
  • Deduplication analysis
  • Fast deduplication option

CLI Syntax:

vcmd du /vol

Additional Resources

Feedback

Need Help?

If you need further assistance or have any questions about this article, please don't hesitate to reach out to our support team.

Document Information

  • Last Updated: 2024-12-27
  • VergeOS Version: 4.13.2

How to Update Your VergeOS Environment

WARNING: DO NOT SKIP MAJOR VERSIONS

For example, if you are on version 4.8, please update to version 4.9 before updating to 4.10. Skipping versions could cause major database and configuration issues.

The VergeOS platform supports 'zero downtime' updating. This means that during a routine update process, guest workloads (VMs and tenant environments) can remain on and running as normal.

For more information on the update process, please refer to our Product Guide.

The time required to complete an update varies depending on factors such as: - Number of nodes in the system - Storage consumed on the vSAN - Data change rate generated by workloads during the update - Hardware performance (processor speed, disk type, network speed)

During the update, you may see tiers and drives alternate between Online, Verifying, or Repairing statuses. This is normal as the system verifies data integrity.

Before performing an update, verify that your system memory usage is within limits. For example, in a two-node configuration, total system memory should be under 50% utilization to allow the remaining node to handle 100% of workloads.

Ensure no nodes are in Maintenance mode before proceeding with the update.

Update Procedure

  1. Log in to the VergeOS UI.

  2. Navigate to System > Updates, then select Check for Updates in the left menu.

    • A pop-up will prompt Yes or No, select Yes.
    • If a banner appears stating "A new minor version is available on a different branch," follow the prompt to change branches by selecting Change Branch in the left menu. Confirm with Yes. VergeOSupgrade-new-img2.png

  3. The packages to be downloaded will now be highlighted. VergeOSupgrade-new-img3.png

    • Select Download in the left menu.
    • A pop-up will prompt Yes or No, select Yes.
    • The download process will appear on the dashboard in the Current Update Server tile.

  4. Once the download completes, the Install action will become available.

    • Select Install when ready.
    • Confirm with Yes to begin the install.

  5. After installation, a request to reboot the system will occur. - Select Reboot in the left menu to initiate a rolling reboot across all nodes.

Note

The update will start with Node 1, putting it into maintenance mode and migrating workloads to another node. During minor version changes, you may briefly lose access to the UI as it fails back to Node 1. This is normal and should last no more than a minute or two. Workloads should not experience network issues.

Troubleshooting Steps

Workloads Failing to Migrate

  • This error is usually due to insufficient resources (RAM) in the cluster. Try migrating other workloads or adjust RAM usage. More causes and solutions are detailed here.

vSAN Taking a Long Time to Verify

  • The verification process depends on factors like network speed, disk type, and consumed data. HDDs will take longer than NVMe or SSDs. On VergeOS versions 4.9.0 and higher, check the Full Walk Progress on the tiers dashboard for an indication of how far along the verification is.

    walk-percentage.png

WARNING

This process must complete before rebooting any additional nodes. Failure to do so can result in a double failure, causing workloads to crash.

Unable to Connect to Update Server

  • Ensure the system has a working DNS server on its external (UI) network:

    1. Navigate to the external network dashboard.
    2. Select Diagnostics in the left menu.
    3. Set Query to DNS Lookup and select Send.
    4. If DNS is properly configured, the response will display Verge.io's IP address. If not, check DNS settings and retry the query.

  • Expired Update Server credentials can also cause this issue. These are tied to the system's license and should be renewed through your VergeOS sales representative.

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6