Skip to content

System Administration#

Proper VergeOS System Shutdown Procedure

Overview

This document provides the step-by-step procedure for properly shutting down a VergeOS system. Following the correct shutdown sequence is critical to ensure data integrity, prevent corruption, and maintain the health of your VergeOS environment.

Prerequisites

  • Administrative access to the VergeOS UI
  • Understanding of your cluster topology
  • Identification of all running workloads and tenants
  • Knowledge of controller node locations (Node1 & Node2)

Shutdown Sequence

Step 1: Inventory Running Workloads

Before beginning the shutdown process, you must identify all active workloads across your environment.

  1. Navigate to each Node Dashboard in your cluster
  2. Review the Running Machines section on each node
  3. Document all running workloads including: - Virtual machines (VMs) - Tenant nodes - VMware backup services - NAS services - Any other active services

Step 2: Shutdown Tenant Workloads

If tenants are running on any nodes in your cluster:

  1. Log into each tenant environment that has active workloads
  2. Gracefully shut down all running workloads within each tenant
  3. Verify shutdown completion before proceeding to the next step

Important

Ensure all tenant workloads are properly shut down before proceeding. Failing to do so may result in data loss or corruption.

Step 3: Power Off Host-Level Workloads

After all tenant workloads are shut down, power off all remaining workloads on each node:

  1. Virtual Machines (VMs): Use the graceful shutdown option when possible
  2. Tenant Nodes: Ensure these are powered off after their internal workloads
  3. VMware Backup Services: Stop any active backup operations
  4. NAS Services: Safely stop all NAS-related services
  5. Other Services: Power off any remaining active services

vNet Containers

vNet containers do not need to be manually stopped. They will be gracefully stopped automatically during the cluster shutdown process.

Step 4: Shutdown Individual Nodes

Once all workloads are stopped:

  1. Navigate to the Cluster Dashboard for the cluster you wish to power off
  2. Select Power Off from the left-hand menu
  3. The system will begin shutting down each node in the cluster
  4. Monitor the shutdown progress through the cluster dashboard

Step 5: Shutdown the Entire Cluster

After individual nodes have been shut down:

  1. Navigate to System → Clusters
  2. Select the cluster you want to shut down
  3. Select Power Off from the left menu
  4. Confirm the shutdown when prompted

Multi-Cluster Environment Considerations

Critical Warning

If your environment contains multiple clusters, you must ALWAYS shut down the cluster containing the controller nodes (Node1 & Node2) LAST.

Shutdown Order for Multi-Cluster Environments:

  1. Shut down all non-controller clusters first
  2. Shut down the controller cluster (containing Node1 & Node2) last

This ensures that cluster coordination and management services remain available until all other clusters are safely shut down.

Alternative Method: API Shutdown

For advanced users or automation purposes, you can use the VergeOS API to shutdown clusters:

POST /v4/cluster_actions
{
    "cluster": [cluster_id],
    "action": "shutdown",
    "params": "{}"
}

Proper Power-On Sequence

When powering your VergeOS system back on, follow this sequence:

Single Cluster Environment:

  1. Power on Node1 first
  2. Wait for Node1 to come online completely
  3. Power on Node2
  4. Power on remaining nodes one at a time, waiting approximately 1 minute between each
  5. Verify system status on the main dashboard (should show Green and Online)

Multi-Cluster Environment:

  1. Power on the controller cluster first (Node1, then Node2, then remaining controller nodes)
  2. Wait for controller cluster to be fully online
  3. Power on other clusters following the single-cluster sequence for each

Verification and Monitoring

After completing the shutdown or startup process:

  1. Check the main dashboard for system status indicators
  2. Verify all nodes show appropriate status (Offline for shutdown, Online for startup)
  3. Monitor system logs for any errors or warnings
  4. Test critical services after startup to ensure proper operation

Troubleshooting

Common Issues During Shutdown:

Workloads Won't Shut Down Gracefully: - Check guest OS ACPI settings - Use "Hard Reset" or "Kill Power" options as last resort - Review VM power management settings

Nodes Won't Enter Shutdown: - Verify all workloads are stopped - Check for stuck or non-responsive VMs - Review node logs for error messages

Cluster Shutdown Fails: - Ensure individual nodes are properly shut down first - Check cluster status and connectivity - Verify no active migrations or maintenance operations

Getting Help:

If you encounter issues during the shutdown process: 1. Document the error messages and current system state 2. Check the VergeOS logs for detailed error information 3. Contact VergeOS support with specific details about the issue

Best Practices

  • Plan shutdown windows during low-usage periods
  • Notify users before beginning shutdown procedures
  • Document your specific environment including cluster topology and critical workloads
  • Test the shutdown process in non-production environments first
  • Maintain current backups before performing system shutdowns
  • Use maintenance mode for individual nodes when possible instead of full shutdowns

Summary Checklist

  • Inventory all running workloads across all nodes
  • Shut down tenant workloads gracefully
  • Power off all host-level workloads (VMs, services, etc.)
  • Navigate to Cluster Dashboard and select Power Off
  • Navigate to System → Clusters and power off the entire cluster
  • For multi-cluster: Shut down controller cluster (Node1 & Node2) LAST
  • Verify shutdown completion through dashboard monitoring
  • Document any issues encountered for future reference

Following this procedure ensures a safe and controlled shutdown of your VergeOS environment while maintaining data integrity and system health.

Document Information

  • Document Type: Knowledge Base Article
  • Category: System Administration
  • Tags: shutdown, power-off, cluster-management, system-maintenance
  • Applies to: VergeOS 4.12.6 and later versions

Network Diagnostics Guide

Overview

This guide provides comprehensive information about the network diagnostic options available in the user interface. These diagnostic tools enable system administrators to monitor, troubleshoot, and maintain Verge deployments effectively.

Critical Warning

The diagnostic commands detailed in this guide are powerful administrative tools. Improper usage can result in:

  • System outages
  • Service interruptions
  • Potential data loss

Exercise extreme caution and ensure proper understanding before execution.

Prerequisites

To use these diagnostic tools, you must have:

  • UI access to your VergeIO cluster
  • Note: Tenants will have their own networking, and therefore their own Network Diagnostics page.

Accessing Network Diagnostics / Issuing Diagnostic Commands

  1. Navigate to Network Diagnostics using either method:
  • From the home screen: Select the Networks count box → Networks (left menu) → Select a network (right side of the screen) → Diagnostics

  • Alternative path: Home screen → Networks (left menu) → Select a network (right side of the screen) → Diagnostics 2. Command execution:

  • Select desired command from the dropdown menu

  • Configure available options if applicable
  • Click SEND→ to execute

Command Visibility

Enable the "Show Command" option to view the exact command being executed. This can be valuable for:

  • SSH execution
  • BASH script integration
  • Advanced command automation

ARP Scan

Purpose: Scans the local network using ARP (Address Resolution Protocol) packets to discover active devices.

Details:

  • Sends ARP requests to all possible addresses in the specified network
  • Displays MAC and IP addresses of responding devices
  • Used for network discovery and inventory

CLI Syntax: 

lxc-attach -n vnet3 -- arp-scan -l -I [interface]

ARP Table

Purpose: Displays the current ARP cache with IP addresses instead of hostnames.

Details:

  • Shows the ARP table maintained by the kernel
  • The -n flag prevents DNS lookups (displays numeric IP addresses)
  • Lists MAC addresses associated with IP addresses the system has communicated with

CLI Syntax: 

lxc-attach -n vnet3 -- arp -n

DHCP Release/Renew

Purpose: Releases the DHCP address for the selected interface, then attempts to renew it.

Details:

  • This command sequence effectively performs a "release and renew" operation for DHCP-assigned IP addresses:
  • Release the current IP address (USR2 signal)
  • Waits for 2 seconds (sleep 2)
  • Request a new IP address (USR1 signal)

CLI Syntax: 

lxc-attach -n vnet3 -- busybox sh -c killall -USR2 udhcpc ; sleep 2 ; killall -USR1 udhcpc

DNS Lookup

Purpose: Performs DNS lookups for specified query types.

Details:

  • Used for troubleshooting DNS issues and domain information gathering
  • Common query types:
  • A: IPv4 address records
  • AAAA: IPv6 address records
  • MX: Mail exchange records
  • NS: Name server records
  • TXT: Text records
  • CNAME: Canonical name records

CLI Syntax: 

lxc-attach -n vnet3 -- host -t Query_Type DNS_Name

FRRouting / BGP/OSPF

Purpose: Allows the configuration of FRRouting.

Details: For more information on other values and variables, refer to FRR documentation & Configuring BGP Hold Down Timers.

CLI Syntax: 

lxc-attach -n vnet3 -- vtysh -c command_goes_here

IP Commands

Purpose: Allows for the configuration of IP and it's subsequent related items.

Address

Purpose: Displays and configures network interface addresses.

Details:

  • Shows IP addresses, subnet masks, and interface states
  • Displays both IPv4 and IPv6 addresses

CLI Syntax: 

lxc-attach -n vnet3 -- /sbin/ip address

Connection Tracking

Purpose: Displays the contents of Netfilter's Connection Tracking file which contains information about network connections.

Details: This file displays the current state of all tracked network connections on the system. Each line in the file represents a single tracked connection. The contents typically include entries with fields such as:

  • Protocol (tcp, udp, icmp)
  • Connection states (ESTABLISHED, TIME_WAIT, etc.)
  • Source and destination IP addresses and ports
  • Connection timeouts
  • Packet and byte counts
  • NAT information if applicable

CLI Syntax: 

lxc-attach -n vnet3 -- dd bs=262144 count=1 if=/proc/net/nf_conntrack

Purpose: Displays network interfaces at layer 2 (data link layer).

Details: - Shows interface states, MTU values, and MAC addresses

CLI Syntax: 

lxc-attach -n vnet3 -- /sbin/ip link

Multicast Address

Purpose: Displays multicast addresses assigned to interfaces.

Details: - Shows IPv4 and IPv6 multicast addresses - Displays which interfaces are subscribed to which multicast groups - Useful for debugging multicast routing and applications

CLI Syntax: 

lxc-attach -n vnet3 -- /sbin/ip maddress

Multicast Routing Cache

Purpose: Displays multicast routing table.

Details: - Shows active multicast routes - Includes source and group addresses - Displays incoming and outgoing interfaces - Useful for troubleshooting multicast routing issues

CLI Syntax: 

lxc-attach -n vnet3 -- /sbin/ip mroute

Neighbor

Purpose: Displays neighbors (ARP table) of the current device.

Details: - Similar to arp -n. - Shows IPv4 and IPv6 neighbors - Includes MAC addresses and states of neighbors

CLI Syntax: 

lxc-attach -n vnet3 -- /sbin/ip neighbor

Routing Table

Purpose: Displays the IP routing table.

Details: - Shows all routes currently configured on the system - Displays default gateway, network routes, and host routes

CLI Syntax: 

ip route show table all

Rule

Purpose: Displays and manipulates the routing policy database.

Details: - Shows policy-based routing rules - Allows for complex routing setups with multiple routing tables - Used for advanced networking configurations - Rules are evaluated in priority order (lower numbers first)

CLI Syntax: 

lxc-attach -n vnet3 -- /sbin/ip rule

Transform (xfrm) - Policy

Purpose: Displays IPSec policies.

Details: - Shows security policies for IPSec communications - Displays source, destination, protocols, and actions

CLI Syntax: 

lxc-attach -n vnet3 -- /sbin/ip xfrm policy

Transform (xfrm) - State

Purpose: Displays IPSec security associations.

Details: - Shows the current security associations (SAs) for IPSec - Displays encryption algorithms, keys, and related information - Used in conjunction with ip xfrm policy

CLI Syntax: 

lxc-attach -n vnet3 -- /sbin/ip xfrm state

IPsec

Purpose: Allows for the configuration of IPsec and it's subsequent related items.

List Cryptographic Algorithms

Purpose: Lists all algorithms supported by the IPSec stack.

Details: - Displays encryption, authentication, and compression algorithms - Shows available key lengths and other parameters

CLI Syntax: 

lxc-attach -n vnet3 -- ipsec listalgs

List IKE Counters

Purpose: Displays statistics and counters for IPSec connections.

Details: - Shows packet counts, bytes transferred, and errors - Useful for monitoring and troubleshooting IPSec tunnels

CLI Syntax: 

lxc-attach -n vnet3 -- ipsec listcounters

IPsec Show Config

Purpose: Displays the contents of StrongSwan's VPN Configuration file which contains information about network VPN connections.

** Details**: This file contains settings for your VPN tunnels including:

  • Connection definitions
  • Authentication methods
  • Encryption algorithms
  • Network settings
  • Tunnel endpoints
  • Identity information
  • Secret key references

CLI Syntax: 

lxc-attach -n vnet3 -- dd bs=262144 count=1 if=/tmp/vpn/ipsec.conf status=none

Status

Purpose: Displays the status of IPSec connections.

Details: - Shows active IPSec tunnels and their current state - Displays connection names, remote endpoints, and status - Useful for quick verification of IPSec connectivity

CLI Syntax: 

lxc-attach -n vnet3 -- ipsec status

Status All

Purpose: Displays detailed status of all IPSec connections.

Details: - Shows comprehensive information about IPSec tunnels - Includes encryption algorithms, key lifetimes, traffic statistics - Displays connection policies and security associations - Valuable for in-depth troubleshooting of IPSec issues

CLI Syntax: 

lxc-attach -n vnet3 -- ipsec statusall

NMAP

Purpose: Network exploration and security auditing tool.

Details: - Scans networks and hosts for open ports and services - Can determine operating systems and service versions - Supports various scanning techniques (SYN, TCP, UDP, etc.) - Offers script-based vulnerability scanning - Essential tool for network administrators and security professionals

CLI Syntax: 

lxc-attach -n vnet3 -- nmap 192.168.0.1 -p22-100

Ping

Purpose: Tests connectivity to a target host.

Details: - Sends ICMP Echo Request packets and waits for ICMP Echo Reply - Measures round-trip time (latency) to the target - Shows packet loss percentage - Basic but essential network troubleshooting tool - Useful for testing basic connectivity and network performance

CLI Syntax: 

lxc-attach -n vnet3 -- busybox ping -c 1 -W 5 8.8.8.8

Show Firewall Rules

Purpose: Displays the current nftables firewall ruleset.

Details: - Shows all tables, chains, and rules configured in nftables - Replacement for the older iptables command - Provides a comprehensive view of the firewall configuration - Useful for troubleshooting connectivity issues and security auditing

CLI Syntax: 

lxc-attach -n vnet3 -- nft list ruleset

TCP Connection Test

Purpose: Uses netcat to connect to checkip.dyndns.org to determine your public IP address.

Details: - Establishes a TCP connection to the specified host and port - Used to determine external IP address

CLI Syntax: 

lxc-attach -n vnet3 -- busybox nc -w5 checkip.dyndns.org 80

TCP Dump

Purpose: Captures and displays network packets on a specific interface.

Details: - There are multiple Verbose Output options - Checking Show Link-Level Header can aid in VLAN troubleshooting - Expressions can be used to filter the output. - Type qualifiers (host, net, port, portrange) - Direction qualifiers (src, dst) - Protocol qualifiers (ether, ip, ip6, tcp, udp, icmp, arp) - Logical operators (and, or, not) - Advanced filters (greater than, less than, TCP flags, byte offsets)

CLI Syntax: 

lxc-attach -n vnet3 -- busybox timeout 15 tcpdump -lni eth0 -c 100

Top CPU Usage

Purpose: Displays system process information in batch mode for a single iteration.

Details: - Shows CPU, memory, and process details - Useful for system monitoring and troubleshooting performance issues

CLI Syntax: 

lxc-attach -n vnet3 -- /usr/bin/top -b -n 1

Top Network Usage

Purpose: Displays bandwidth usage on a network interface by host.

Details: - Shows real-time bandwidth usage by connection - Useful for identifying which hosts are using the most bandwidth

CLI Syntax: 

lxc-attach -n vnet3 -- busybox timeout 10 /usr/sbin/iftop -tNi eth0 -n

Trace Route

Purpose: Traces the route packets take to a destination.

Details: - Displays each hop (router) between your computer and the destination - Shows round-trip time for each hop - Useful for diagnosing routing issues and network latency problems

CLI Syntax: 

traceroute -n -w 3 google.com

Trace/Debug Firewall Rules

Purpose: Monitors and traces packets as they traverse nftables rules.

Details: - Shows which rules packets match and the resulting actions - Extremely useful for debugging complex firewall configurations - Requires root privileges

CLI Syntax: 

lxc-attach -n vnet3 -- busybox timeout 3 nft -nnn monitor trace

What's My IP

Purpose: Queries OpenDNS to determine your public IP address.

Details: - Simple, reliable method to determine your public IP address - Works even when HTTP-based services might be blocked

CLI Syntax: 

lxc-attach -n vnet3 -- dig +short myip.opendns.com @208.67.222.222

Additional Resources

Feedback

Need Help?

If you need further assistance or have any questions about this article, please don't hesitate to reach out to our support team.

Document Information

  • Last Updated: 2024-03-21
  • VergeOS Version: 4.13.4

Enabling System SSH Access

Key Points

  • SSH access to a VergeOS system is generally not needed because full access is provided from the UI.
  • SSH should only be enabled for specific hardware diagnostics or other special circumstances.
  • Although VergeOS employs many safety protections, opening SSH on any system can introduce vulnerability.

Important SSH Security Procedures

  • Always use source-controlled external rules to strictly limit ssh access to trusted addresses.
  • Enable SSH access on a temporary basis; disable rules again when done with the session.

Steps to Enable SSH access

SSH Access rules are auto-created, and disabled, during system installation.

  1. Enable the core network rule: Navigate to the Core network dashboard, modify the "SSH Access" rule, select the Enabled option and Submit to save the change. ssh-rule-core.png

  2. Add source control to the external network rule: Navigate to the external network dashboard, modify the "SSH Access" rule to configure specific source IP address(es) and/or address range(s) to tightly control access.

  3. Enable the external network rule: select the Enabled option and Submit to save the change.
    Ex. External Network Rule: ssh-rule-external.png

  4. Apply Rules to both networks.

Warning

Danger

  • VergeOS is a specialized kernel, with a read-only overlay. Do not install additional Debian packages or applications as they can conflict with VergeOS operation and cause system malfunction or data loss. Additionally, extraneous programs are wiped at reboot.
  • Check with VergeOS support before making any modifications at the command line. Issues resulting from unsanctioned command-line changes are the sole responsibility of the customer.

Configuring VergeOS as an OIDC Client

Overview

Key Points

  • Configure VergeOS to use OIDC authentication
  • Connect to a VergeOS OIDC identity provider
  • Enable automatic user creation and synchronization
  • Customize login appearance and behavior

This guide explains how to configure a VergeOS system or tenant to authenticate using another VergeOS system as an OIDC identity provider.

Prerequisites

  • Access to the VergeOS OIDC provider system
  • Well Known Configuration URL from the provider
  • Client ID and Client Secret from the provider
  • Administrative access to the client VergeOS system
  • Full URL of the client VergeOS system

Steps

1. Access Authorization Settings

  • Navigate to the Main Dashboard
  • Click System in the left menu
  • Select Authorization Sources
  • Click New

2. Configure Basic Settings

  • Name: Enter an identifier for this auth source (appears on login button)
  • Driver: Select OpenID (Well Known Config)
  • Base URL: Enter the Well Known Configuration URL
  • Redirect URI: Enter the full URL of this VergeOS system
  • Client ID: Paste the client ID from the provider
  • Client Secret: Paste the client secret from the provider

3. Configure Authentication Parameters

Default values typically work best for these settings: - Token hint parameter: Leave as post_logout_redirect_uri - Redirect parameter: Leave as post_logout_redirect_uri - Scope: Leave as openid profile email groups - Group Scope: Leave as groups

Check these boxes for optimal functionality: - Decode ID Token - Update Remote User - Update User Email Address - Update User Display Name - Update Group Membership

5. Configure User Creation

Choose your preferred user creation method: - Auto-Create Users: Enter .* to create all users automatically - Auto-Create Users in Group: Specify groups for restricted auto-creation

6. Customize Login Appearance

Optionally configure: - Button background color - Button text color - Custom Font Awesome icon - Icon color (using HEX codes)

7. Save Configuration

  • Click Submit to create the authorization source

Best Practices

  • Test authentication with a test user before rolling out widely
  • Keep debug mode disabled unless troubleshooting
  • Document your configuration choices for future reference
  • Regular verify user synchronization is working as expected

Troubleshooting

Common Issues

  • Authentication Fails

    • Verify Client ID and Secret are correct
    • Check Well Known Configuration URL
    • Ensure Redirect URI matches exactly

  • User Sync Issues

    • Verify Group Scope is enabled
    • Check group membership settings
    • Enable Debug Mode temporarily

  • Login Button Missing

    • Verify authorization source is enabled
    • Check login styling settings
    • Clear browser cache

Additional Resources

Feedback

Need Help?

If you encounter any issues while configuring OIDC client settings or have questions about this process, please don't hesitate to contact our support team.

Document Information

  • Last Updated: 2024-01-22
  • VergeOS Version: 4.12 and later

Setting Up VergeOS as an Identity Provider with OIDC

Overview

Key Points

  • Create an OIDC application to establish VergeOS as an identity provider
  • Enable single sign-on for other VergeOS systems and tenants
  • Configure centralized authentication with third-party providers
  • Support multiple client systems with a single OIDC setup

This guide walks you through the process of configuring VergeOS as an identity provider using OpenID Connect (OIDC), allowing centralized authentication for multiple VergeOS systems and tenants.

Prerequisites

  • Administrative access to the VergeOS system
  • Valid SSL certificate installed on the VergeOS system
  • Basic understanding of OIDC concepts
  • URLs of client systems that will use this authentication

Steps to Create an OIDC Application

  1. Access OIDC Settings - Navigate to the Main Dashboard - Click System in the left menu - Select OIDC Applications - Click New

  2. Configure Basic Settings - Enter a descriptive Name for the application - Check the Enabled box - Add an optional Description

  3. Set Up Redirect URIs - Enter the callback URL(s) where users will be redirected after authentication - Format: https://your-system-name.example.com - Multiple URIs can be added for different client systems

!!! tip "Using Wildcards" You can use wildcards in redirect URIs: - For multiple systems: https://examplecorp-site*.example.com - For multiple subdomains: https://vergesystem.*.example.com

  1. Configure Authentication Options - Force Authorization Source: Optionally select a third-party provider - Map User: Choose if all verified users should map to a specific account - Set Scope Settings (Profile, Email, Groups) - Configure access restrictions if needed

  2. Save Configuration - Click Submit to create the OIDC application - The system will generate a Client ID and Secret

Retrieving Client Credentials

  1. Access Application Dashboard - Navigate to System > OIDC Applications - Double-click your OIDC application

  2. Copy Required Information - Client ID: Copy using the displayed value or copy icon - Client Secret: Use the copy icon (value is hidden) - Well Known Configuration URL: Copy the displayed URL

Best Practices

  • Create separate OIDC applications for different client groups
  • Regularly review and update access restrictions
  • Use specific redirect URIs instead of wildcards when possible
  • Document which systems are using each OIDC application

Troubleshooting

Common Issues

  • Authentication Fails

    • Verify SSL certificate is valid and not expired
    • Check redirect URI matches exactly
    • Ensure client ID and secret are correctly copied

  • Scope Access Denied

    • Verify required scopes are enabled
    • Check user permissions in restriction settings

  • Redirect Problems

    • Confirm URL format matches redirect URI
    • Verify wildcard patterns if used
    • Check for SSL certificate issues

Additional Resources

Feedback

Need Help?

If you encounter any issues while setting up OIDC or have questions about this process, please don't hesitate to contact our support team.

Document Information

  • Last Updated: 2024-08-29
  • VergeOS Version: 4.12 and later

vSAN Diagnostics Guide

Overview

This guide provides comprehensive information about the vSAN diagnostic options available in the user interface. These diagnostic tools enable system administrators to monitor, troubleshoot, and maintain vSAN deployments effectively.

Critical Warning

The diagnostic commands detailed in this guide are powerful administrative tools. Improper usage can result in: - System outages - Service interruptions - Potential data loss

Exercise extreme caution and ensure proper understanding before execution.

Prerequisites

To use these diagnostic tools, you must have:

  • Root-level access to your VergeIO cluster
  • Note: Tenants do not have a vSAN.

Accessing vSAN Diagnostics

  1. Navigate to vSAN Diagnostics using either method:
  • From the home screen: Select the vSAN Tiers count box → vSAN Diagnostics (left menu)

  • Alternative path: Home screen → System (left menu) → vSAN Diagnostics 2. Command execution:

  • Select desired command from the dropdown menu

  • Configure available options if applicable
  • Click SEND→ to execute

Command Visibility

Enable the "Show Command" option to view the exact command being executed. This can be valuable for: - SSH execution - BASH script integration - Advanced command automation

Diagnostic Commands

Add Drive to vSAN

Running this command allows you to manually add a drive via the UI. Drives are normally added either during the installation, or via the Nodes > Nodes Drives page. However adding them in that way does not allow for the addition of drives to Tier0.

Prerequisites:

  • Drive must be physically present in the system
  • Drive must be visible from Nodes > Nodes Drives page

Usage Parameters:

  • Selecting Add Drive to vSAN
  • From the right menu, select the Node that we will be adding the drive to.
  • Enter the appropriate path, E.G. `/dev/nvme0n1```<br>
  • You can use the "Click here to view devices" to get the path. !!! warning "The contents of this drive will be overwritten."
  • Select the Tier you want to assign the drive to.
  • Check the Swap box if you want Swap enabled on this drive. !!! info "This will use the cluster settings for the Swap size."
  • Verify. You will need to TYPE Yes I know what I'm doing in the Verify box.
  • Select SEND →

CLI Syntax:

vcmd newdevice --path=PATH [OPTIONS]
  --path=PATH    Path to target device
  --tier=NUM     Tier number assignment

Cancel Integrity Check

Terminates any active integrity check operations. See Integrity Check for additional information.

CLI Syntax:

vcmd cancelintegcheck

Clear Reference Counts

Reference counts are how the vSAN tracks the number of times a file is referenced in the vSAN. Clearing this count will force a full vSAN walk and a refresh of the Reference Counts.

Function:

  • Clears existing reference counts
  • Initiates full vSAN traversal
  • Rebuilds reference count data

Support Authorization Required

Execute only under direct support guidance.

Usage Parameters:

  • Verify. You will need to TYPE Yes I know what I'm doing in the Verify box.
  • Select SEND →

CLI Syntax:

vcmd clearrefcounts

Find Inode

Running this query will allow you to find out what an Inode (Index Node) referrences. Inode is a data structure that stores information about a file or directory, such as its owner, access rights, date and time of creation and modification, size and location on the vSAN. Each file or directory in the system has its own unique index node number (inode number), which can be used to perform various operations with a file or directory. This can be used to troubleshoot errors in the vSAN.

Purpose:

  • Retrieves inode reference information
  • Maps inode numbers to filesystem entities
  • Assists in vSAN troubleshooting

CLI Syntax:

find /vsan -inum inode_number_here -printf /%P\n

Get Cache Info

Retrieves detailed cache information for specified nodes.

Output Information:

  • Total cache capacity
  • Available cache space
  • Cache page statistics
  • Performance metrics

CLI Syntax:

vcmd getcacheinfo

Get Clients

Retrieves client connection information for specified nodes.

Output Information:

  • Connected node information
  • IP address mappings
  • Worker thread statistics

CLI Syntax:

vcmd getclients

Get Cluster Rates

Retrieves cluster-wide performance metrics.

Output Information:

  • Read/write rates
  • Throttle status
  • Performance statistics

CLI Syntax:

vcmd getclusterrates

Get Cluster Usage

Provides cluster-wide storage utilization information.

Output Information:

  • Maximum storage capacity
  • Current utilization
  • Repair operation counts

CLI Syntax:

vcmd getclusterusage

Get Current Master

Retrieves master node information from each cluster member.

Output Information:

  • Master node identification
  • Online status
  • Transaction logging information

CLI Syntax:

vcmd getcurmaster

Get Device Integrity

Retrieves integrity check results for specified nodes.

CLI Syntax:

vcmd getdeviceinteg

Get Device List

Provides comprehensive device inventory.

Output Information:

  • Device identifiers
  • System paths
  • Tier assignments

CLI Syntax:

vcmd getdevicelist

Get Device Status

Retrieves detailed device status information.

Output Information:

  • Device paths
  • Operational status
  • Capacity metrics
  • Performance statistics

CLI Syntax:

vcmd getdevicestatus

Get Device Usage

Provides device utilization metrics.

Output Information:

  • Total capacity
  • Current utilization
  • Usage trends

CLI Syntax:

vcmd getdeviceusage

Get File Status

Retrieves detailed file metadata.

Output Information:

  • Inode information
  • File type
  • Tier assignment
  • Hash key data

CLI Syntax:

vcmd stat /path/to/file.raw

Get Fuse Info

Retrieves FUSE (Filesystem in Userspace) statistics.

Output Information:

  • Mount point information
  • Thread statistics
  • Throttling metrics

CLI Syntax:

vcmd getfuseinfo

Get Integrity Check Status

Retrieves results from the most recent integrity check.

Output Information:

  • Check status
  • Path information
  • Temporal data
  • Verification results

CLI Syntax:

vcmd getintegcheckstatus

Get Journal Status

Retrieves journal system status information.

Output Information:

  • Operational status
  • Redundancy status
  • System metadata

CLI Syntax:

vcmd getjournalstatus

Get Node Device List

Retrieves detailed hardware information for storage devices.

Output Information:

  • Driver information
  • Model specifications
  • Firmware versions
  • Physical attributes

CLI Syntax:

vcmd getnodedevicelist

Get Node Info

Retrieves comprehensive node configuration data.

Output Information:

  • Node identification
  • Cluster configuration
  • System parameters
  • Operational status

CLI Syntax:

vcmd getnodeinfo

Get Node List

Provides cluster-wide node inventory.

Output Information:

  • Node identification
  • Online status
  • Version information
  • Tier utilization

CLI Syntax:

vcmd getnodelist

Get Path from Inode

Resolves filesystem paths from inode numbers.

CLI Syntax:

vcmd getpathfromino $1

Get Read Ahead

Retrieves read-ahead buffer statistics.

Output Information:

  • Queue statistics
  • Thread utilization
  • System status

CLI Syntax:

vcmd getreadahead

Get Repair Status

Monitors ongoing repair operations.

Output Information:

  • Device repair status
  • Operation progress
  • System health

CLI Syntax:

vcmd getrepairstatus

Get Running Configuration

Retrieves active system configuration.

Output Information:

  • Worker thread allocation
  • System throttles
  • Operational parameters

CLI Syntax:

vcmd getrunningconf

Get Sync List

Monitors synchronization operations.

Output Information:

  • Operation frequency
  • Start times
  • File processing status

CLI Syntax:

vcmd getsynclist

Get Tier Device Maps

Retrieves tier-to-device mapping information.

Output Information:

  • Physical device mappings
  • Tier assignments
  • System configuration

CLI Syntax:

vcmd gettierdevicemaps

Get Tier Node Maps

Retrieves tier-to-node mapping information.

Technical Details:

  • Base-0 indexing (0=Node1, 1=Node2, etc.)
  • 65536 buckets per tier map
  • Primary (tier_x.0) and redundant (tier_x.1) mappings

CLI Syntax:

vcmd gettiernodemaps

Get Tier Status

Retrieves comprehensive tier health information.

Output Information:

  • Redundancy status
  • Walk statistics
  • Transaction data
  • Health metrics

CLI Syntax:

vcmd gettierstatus

Get Top Usage Rates

Monitors real-time I/O statistics.

Real-time Data

Multiple executions may be necessary for trend analysis.

CLI Syntax:

vcmd getfhlist | grep -Eo '(ino|rrate|wrate)\b.*'

Get Volume Usage

Retrieves detailed volume utilization statistics.

Parameters:

  • Path specification (optional)
  • Recursive flag
  • Human-readable output
  • Preferred tier display

CLI Syntax:

vcmd getvolusage --path=/ --recursive=1 --human=1

Integrity Check

Initiates system integrity verification.

Parameters:

  • Path specification (required)
  • Recursive operation
  • Fix mode (destructive)
  • Meta-tier only option

Data Loss Risk

Fix mode zeros bad blocks. THIS IS DESTRUCTIVE. Use only under support guidance.

CLI Syntax:

vcmd integcheck /vol

Integrity Check Device

Performs device-level integrity verification.

Parameters:

  • Node selection
  • Device ID (-1 for all devices)

CLI Syntax:

vcmd integcheckdevice --id=x

Summarize Disk Usage

Generates storage utilization summaries.

Parameters:

  • Path specification
  • Recursive operation
  • Preferred tier display
  • Deduplication analysis
  • Fast deduplication option

CLI Syntax:

vcmd du /vol

Additional Resources

Feedback

Need Help?

If you need further assistance or have any questions about this article, please don't hesitate to reach out to our support team.

Document Information

  • Last Updated: 2024-12-27
  • VergeOS Version: 4.13.2

Updating the VergeOS System

Overview

Key Points

  • System updates should be performed during a maintenance window
  • Updates can be performed with zero downtime when adequate resources are available
  • System updates are only run from the host system (top-level parent)
  • Tenant systems are automatically updated from their host system
  • Updates can be scheduled or performed on-demand
  • The system automatically handles workload migration during updates

This guide provides detailed instructions for performing system updates in VergeOS, whether on-demand or scheduled.

Prerequisites

  • Administrative access to the VergeOS Cloud Dashboard
  • Adequate system resources to allow workload migration during updates
  • A maintenance window (recommended, though not required due to zero-downtime capability)

Performing On-Demand Updates

1. Check for Updates

  1. Navigate to System > Updates in the Cloud Dashboard
  2. Click Check For Updates in the left menu
  3. Click Yes to confirm
    • The Packages section will show available updates
    • A cloud icon indicates downloadable packages
    • Version information displays current and available versions

2. Download Updates

  1. Click Download in the left menu
  2. Click Yes to confirm
  3. Wait for the download to complete

3. Install Updates

  1. Click Install in the left menu
  2. Click Yes to confirm
  3. Wait for installation to complete
    • Status will show "Idle - Reboot Required" when ready
    • The Reboot option will become enabled

Note

Updates that don't include VergeOS package changes won't require full node reboots, but still need the Reboot option to apply changes.

4. Apply Updates

  1. Click Reboot in the left menu
  2. Click Yes to confirm - The system will process one node at a time:
    • Node enters maintenance mode
    • Workloads migrate to other nodes
    • Application restarts/node reboots
    • Node exits maintenance mode
    • Progress shows in the Status field
    • Nodes Updated status tracks completion

Tip

Use Cancel Reboot to halt automatic reboots if needed (e.g., for workload rebalancing)

Scheduling Updates

1. Create Update Task

  1. Navigate to System > Updates > Tasks
  2. Click New in the left menu

2. Configure Schedule

  1. Choose scheduling option:
    • One-time: Keep default "Does Not Repeat"
    • Recurring: Select frequency (weekly, bi-weekly, monthly)
  2. Set Start Date and time
  3. For recurring tasks, optionally set end date

3. Configure Task Details

  1. Enter required Name
  2. Add optional Description
  3. Select Task Type:
    • Choose "Download, Install, and Reboot" for complete update
  4. Optional: Enable Delete After Running
  5. Click Submit to save

Best Practices

  • Schedule updates during low-usage periods and during maintenance windows
  • Ensure adequate system resources for workload migration
  • Monitor system during update process
  • Keep regular backups before major updates
  • Review available updates before applying

Troubleshooting

Common Issues

  • Issue: Workloads fail to migrate

  • Solution: Verify adequate resources on target nodes

  • Issue: Update process hangs

  • Solution: Check system logs and contact support if needed

  • Issue: Node fails to rejoin after reboot

  • Solution: Review logs and network connectivity

Feedback

Need Help?

If you encounter any issues during the update process or have questions, please reach out to our support team.

Document Information

  • Last Updated: 2024-12-19
  • VergeOS Version: All

Device Passthrough Advanced Configuration (Manual Creation/Editing of Resource Rules)

Although allowing auto-generation of resource rules (e.g. when you select a device and use the Make Passthrough menu option) is easiest and usually recommended, there may be situations where it may be useful to manually create a resource rule or to modify an auto-generated resource rule.

It is important to read and be familiar with PCI Passthrough Risks and Precautions before making passthrough configurations.

Manually Create a New Resource Rule

  1. From the main dashboard, click Resources.
  2. Click Rules (ui card or on the left menu).
  3. Click New on the left menu.
  4. Provide a Name for the Rule; it is recommended to use a descriptive name can be helpful in future administration.
  5. Select the Resource Group to which the resource rule will apply.
  6. Select a specific Node or select --None-- to apply the rule to all nodes.
  7. Select the Type (PCI, USB, SR-IOV, or NVIDIA vGPU).
  8. Leave the default value set to --None-- in the field labeled Automatically created based on PCI Device.
  9. Configure device filters as desired; filter fields will vary depending on the device type selected; see below. (Advanced Entry 1 option also available)

Information on installed PCI devices, for use in filters, you can use the PCI devices listing: from the Main Dashboard, navigate to the Resources -> PCI Devices. To show additional fields, right-click in the heading section to select from the full list of available columns that can be displayed.

Edit an Existing Resource Rule

  1. Navigate to the Associated Resource Group dashboard (Main Dashboard > Resources > Groups > double-click the particular group).
  2. In the Rules section, locate and click the desired resource rule.
  3. Click Edit on the left menu.
  4. Node selection and PCI Filters can be modified as needed. (Advanced Entry 1 option also available)

  1. The Advanced Entry section allows you to manually input filter syntax rather than using the filter entry fields. Generally, it is preferable to allow system-generated syntax based on your filter field selections. ↩↩↩↩↩↩

Terraform VergeIO Provider

The Terraform VergeIO Provider enables the integration and automation of VergeOS infrastructure with Terraform. It allows users to define, manage, and scale VergeOS resources as part of Infrastructure as Code (IaC) workflows.

For the latest provider documentation and examples, please refer to the following:

Example Usage

For more detailed usage examples, check the docs folder in the GitHub repository.

Example Configuration

provider "vergeio" {
  host     = "https://some_url_or_ip"
  username = "my_user"
  password = "my_password"
  insecure = false  # Use true if using self-signed SSL certificates
}

resource "vergeio_vm" "new_vm" {
  name        = "NEW VM"
  description = "NEW TF VM"
  enabled     = true
  os_family   = "linux"
  cpu_cores   = 4
  machine_type = "q35"
  ram         = 8192
}

Initializing and Applying

To apply the configuration:

terraform init && terraform apply

Configuration Reference

host (Required): URL or IP address for the VergeOS system or tenant.
username (Required): Username for the VergeOS system or tenant.
password (Required): Password for the provided username.
insecure (Optional): Set to true for systems using self-signed SSL certificates.

Resources

The following VergeOS resources can be managed via Terraform:

vergeio_drive
vergeio_member
vergeio_network
vergeio_nic
vergeio_user
vergeio_vm

Data Sources

The following data sources are available for querying VergeOS resources:

vergeio_clusters
vergeio_groups
vergeio_mediasources
vergeio_networks
vergeio_nodes
vergeio_version
vergeio_vms

Testing a Sample Configuration

To test your configuration, create a main.tf file in your Terraform workspace:

terraform {
  required_providers {
    vergeio = {
      source = "vergeio/cloud/vergeio"
    }
  }
}

provider "vergeio" {
  host     = "https://someURLorIP"
  username = "username"
  password = "password"
}

resource "vergeio_vm" "new_vm" {
  name        = "NEW VM"
  description = "NEW TF VM"
  enabled     = true
  os_family   = "linux"
  cpu_cores   = 4
  machine_type = "q35"
  ram         = 8192
}

Then, run the following command:

terraform init && terraform apply

Document Information

  • Last Updated: 2024-09-03
  • VergeOS Version: 4.12.6

Updating a VergeOS System with Airgap License

Overview

Key Points

  • System updates should be performed during a maintenance window
  • This guide details the process of manually updating a VergeOS system using an air-gap license.
  • The update is performed using an ISO file, ensuring that systems without internet access can be kept up-to-date.
  • Ensure you have a valid air-gap license and the latest ISO file before starting.

This guide provides a step-by-step process to manually update your air-gapped VergeOS system using an ISO file.

Prerequisites

  • Access to the VergeOS Cloud Dashboard.
  • The latest VergeOS update ISO file.
  • A valid air-gap license.
  • A recent backup of your VergeOS system.

Steps

  1. Download the Update ISO - Visit the VergeOS updates page at https://updates.verge.io/download. - Download the latest VergeOS release ISO file.

!!! tip "Pro Tip" Ensure that the ISO file corresponds to your current VergeOS version to avoid compatibility issues.

  1. Upload the ISO to VergeOS - Log in to your VergeOS Cloud Dashboard. - Navigate to Media Images in the left-hand menu. - Upload the downloaded ISO file to the Media Images section.

!!! note The upload process may take a few minutes depending on your network speed.

  1. Configure Update Settings - Go to System > Updates > Edit Settings. - In the Update Source dropdown menu, select -- Update ISO --. - Choose the ISO file you just uploaded from the Media Images. - Click Submit to save the settings.

  2. Perform the Update - Return to the Updates section and click Check For Updates. - Once the update is detected, click Download. - After the download completes, click Install. - Follow the prompts to Reboot the system to apply the updates.

!!! warning "Important" Do not interrupt the update process. Ensure that the system remains powered on and connected during the update.

Troubleshooting

Common Issues

  • Issue: Update not detected after uploading the ISO.

  • Solution: Ensure the ISO was uploaded correctly and reselect it in the Update Source settings.

  • Issue: Errors during the update process.

  • Solution: Check system logs for detailed error messages and verify that your air-gap license is valid.

  • Issue: System fails to reboot after the update.

  • Solution: Contact Verge support for assistance.

Additional Resources

Feedback

Need Help?

If you encounter any issues during the update process or have any questions, please reach out to our support team.

Document Information

  • Last Updated: 2024-08-19
  • VergeOS Version: 4.12.6

Requesting an Airgap License for VergeOS

Air-gap licensing is not common and requires justification. Please see Licensing and Software Updates for more information.

Overview

Key Points

  • VergeOS requires a valid license for operation
  • Air-gapped environments need a special airgap license
  • The process involves generating a license request file and emailing it to Verge

This guide walks you through the process of requesting an airgap license for VergeOS systems in environments without outbound Internet access.

Prerequisites

  • Access to the VergeOS Cloud Dashboard
  • A working email client on a machine that can send external emails
  • Understanding of your system's airgapped status

Steps

  1. Navigate to System Settings

    • From the System Dashboard, click System on the left menu
    • Click Settings on the left menu

  2. Initiate License Request

    • In the License section, click the Request License button

  3. Generate License Request File

    • A popup window titled "Request Generated" will appear
    • This window displays information about the license request file

  4. Download Request File

    • Click the Download Request File button
    • Save the license request file to your local machine

  5. Prepare Email to Verge

    • Click the Email license@Verge.io button
    • This opens your default email client with a pre-addressed email

  6. Send License Request

    • Attach the downloaded license request file to the email
    • Provide additional information in the email body (e.g., company name, purpose of license)
    • Send the email to Verge's licensing team

What Happens Next

  1. Verge processes your request and generates an airgap license file
  2. You receive a reply email with the airgap license file attached
  3. Upload the received license file to your VergeOS system (covered in a separate guide)

Processing Time

If you haven't received a response within 2 business days, please follow up with Verge's support team.

Important Considerations

  • Ensure the system requesting the license is the one you intend to license
  • Keep the license request file secure
  • For multiple systems, repeat this process for each system individually

Troubleshooting

Common Issues

  • Problem: Unable to generate license request file

  • Solution: Verify your access permissions in the VergeOS Cloud Dashboard

  • Problem: Email client doesn't open automatically

  • Solution: Manually compose an email to license@Verge.io and attach the downloaded request file

Feedback

Need Help?

If you encounter any issues while requesting an airgap license or have questions about this process, please don't hesitate to contact our support team.