Skip to content

TroubleshootingπŸ”—

TopologyπŸ”—

Verity displays a live map of the devices in the network and their connections.

Topology Detection and BehaviorπŸ”—

  1. Autodiscovery
  2. Connection moves
  3. Remove connections

ReportsπŸ”—

Verity provides a comprehensive list of reports to identify current conditions in the network. Reports are viewed by navigating to the Reports window or clicking the Reports icon of a device object .

How to Download a Device SnapshotπŸ”—

To download switch device information, locate the device in the topology view. In the device's top-right corner, click Capture Download Snapshot. The button turns light green while the snapshot is being prepared, then dark green when it's ready. Click it again to download.

alt text

How to Download a Verity SnapshotπŸ”—

In addition to individual device snapshots, Verity allows for download snapshots of VNC, Observability VM, AI Ops VM, and GUI software. These are used for broader troubleshooting of the host software. Instructions are available here.

MACs / IPs / HostnamesπŸ”—

Verity provides the ability to display MAC addresses, IP's and hostnames across the entire system. To use this feature navigate to Reports/ MACs / IPs / Hostnames.

The search field defaults to MAC Address and can be changed to search for IP address or hostname.

MAC ExportsπŸ”—

MAC Exports provides a summary MAC report for all devices connected to the network's user facing ethernet ports. This tool can be automated to generate a daily report or used on a case-by-case basis as needed. To manually run a report, enable the checkbox for all services you want to include, click Run Report Now, and wait for the Download Latest Report icon to turn blue. Once it does, click the icon to download the CSV file.

Ping and Trace RouteπŸ”—

Verity provides tools to troubleshoot Layer 3 issues for the underlay and Tenant connections for services configure for Layer 3. The Ping and Trace Route feature is a diagnostic tool that sends a β€˜ping’ to and from a specified address and details the message path.

ControlsπŸ”—

The Ping and Trace Route dialog box is opened by clicking Open Ping/Traceroute Dialog button .

To use this feature you configure the following parameters and click the Start button .

SourceπŸ”—

  • Device: This is the source device of the ping.
  • VRF: This is where you select the VRF. The VRFs of Tenants, the Underlay (selected device), and Management (Device Controller) are all options.
  • IP Version: Filters the available source IP addresses. Select All to show both IPv4 and IPv6 addresses, IPv4 to show only IPv4 addresses, or IPv6 to show only IPv6 addresses.
  • IP: The targeted IP address to send the ping command from. The IP address changes depending on the VRF and IP Version settings. Source IP choices include an address-family label, such as IPv4 or IPv6.

DestinationπŸ”—

  • IP: The targeted IP address to receive the ping command.

PingπŸ”—

  • Count: The number of ping events sent before the process is complete.

Trace RouteπŸ”—

When this option is selected, the trace route is included in the Result field.

CMD StatusπŸ”—

This is a progress indicator. When the feature is ready, CMD Status says Ready; when active, CMD Status says In Progress; and when the process is complete, it says Done.

DHCP SnoopingπŸ”—

Verity provides the location of connected L3 addresses detected by DHCP snooping functions in SONiC.

To enable DHCP snooping navigate to Admin/Site Settings and check the box next to Enable DHCP Snooping .

Viewing DHCP Assigned IP'sπŸ”—

To view DHCP assigned IP addresses go to Reports/DHCP Assigned IPs .

View LicenseπŸ”—

To view the license go to Admin/Licensing.

The Licensing object displays License Usage and Physical Port Usage bar graphs. While focused on the object, the user can see the date of licensing expiration, contact information for support, and reports of license.

By clicking on the report icons for License Utilization a report for thh list will be shown. For a specific report on used, preprovisioned, stranded, or spare licenses, the user can click on each section of the bar graph. This can also be done with physical ports that are licensed, fabric or spare.

An example License Utilization report is shown below:

Monitoring TroubleshootingπŸ”—

OverviewπŸ”—

This troubleshooting script is a comprehensive diagnostic and maintenance tool for the Verity Monitoring System. It provides a menu-driven interface to check system status, perform maintenance tasks, and generate debug information for support purposes.

PurposeπŸ”—

The script helps administrators:

Diagnose Issues: Check service status, connectivity, and configuration

Perform Maintenance: Adjust logging levels, restore backups, populate data

Generate Support Data: Create comprehensive debug snapshots for technical support

Monitor Health: Verify all Monitoring components are functioning correctly

PrerequisitesπŸ”—

System RequirementsπŸ”—

Privileges: Must run as root or with sudo access

Dependencies:

  • Monitoring system previously installed via setup script
  • Active internet connection for API testing

Running the ScriptπŸ”—

Basic ExecutionπŸ”—

bash

sudo ./monitoring_admin.sh

You are presented with two options:

  1. Setup
  2. Troubleshooting

Select 2 Troubleshooting.

Option 1: Check Monitoring Services StatusπŸ”—

1. Check if the Monitoring services are running

Purpose: Verifies all Monitoring Docker containers are running properly

What it checks:

  • ETL (fabric-manager-ml-python)
  • Grafana
  • Prometheus
  • Promtail
  • Telegraf
  • Neo4j-graph
  • Loki
  • Cadvisor
  • Alertmanager
  • Node-exporter
  • AI-service
  • Neo4j-vector

Expected Output:

ETL services are running.
Grafana services are running.
Prometheus services are running.
...
--------------------------------------------
Total Services Tested: 12
Currently Running: 12
Not Running: 0
--------------------------------------------

If Services Are Down:

run the command 'sudo docker compose -f /be_monitoring/docker-compose.yml up -d' and re-run this script


Option 2: Check Time SynchronizationπŸ”—

2. Check the status of the time synchronization

Purpose: Verifies system time is properly synchronized

Expected Output:

               Local time: Wed 2024-12-18 10:30:15 UTC
           Universal time: Wed 2024-12-18 10:30:15 UTC
                 RTC time: Wed 2024-12-18 10:30:15
                Time zone: UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Key Indicators:

  • System clock synchronized: yes - Time is properly synced
  • NTP service: active - Time synchronization service is running

Option 3: Report FQDN SettingπŸ”—

3. Report the FQDN setting in the Monitoring.ini file

Purpose: Shows the configured vNetC host address

Expected Output:

FQDN setting: vnc-monitoring.company.com

Uses: Verify Monitoring knows where to connect to vNetC


Option 4: Report MAC Address SettingπŸ”—

4. Report the FABRIC_MANAGER_MONITORING_VM_MAC_ADDRESS setting in the Monitoring.ini file

Purpose: Shows the configured MAC address for VM identification

Expected Output:

FABRIC_MANAGER_MONITORING_VM_MAC_ADDRESS setting: aa:bb:cc:dd:ee:ff

Uses: Verify unique VM identification in the Monitoring system


Option 5: Report Package VersionπŸ”—

5. Report the version of the Monitoring package

Purpose: Shows the currently installed Monitoring software version

Expected Output:

Version: 2.1.64.108

Uses: Verify version for support purposes or update planning


Option 6: Perform Announcement CallπŸ”—

6. Perform an announcement call to the FQDN setting

Purpose: Tests Monitoring's ability to announce itself to vNetC

What it does: Executes the announcement process to register with vNetC

Expected Behavior: Script runs the announcement process and reports any failures.


Option 7: Check vNetC API StatusπŸ”—

7. Check vNetC API calls are working and report the HTTP status code

Purpose: Tests API connectivity to the vNetC system

Expected Output (Success):

Verity vNetC API calls are working using the FQDN/ROOT_API_URL: vnc-monitoring.company.com

Expected Output (Failure):

Verity vNetC API calls are not working. FQDN/ROOT_API_URL: vnc-monitoring.company.com | HTTP status code: 404

HTTP Status Codes:

  • 200: Success - API is working properly
  • 401: Unauthorized - Check credentials
  • 404: Not Found - Check URL/hostname
  • 500: Server Error - vNetC system issue
  • 000: Connection failed - Network/DNS issue

Option 8: Set Debug LoggingπŸ”—

8. Set logging level to DEBUG

Purpose: Enables detailed logging for troubleshooting

Effect: Changes LOG_LEVEL=DEBUG in monitoring.ini

When to use: - Investigating specific issues - Before reproducing problems for support - Detailed system analysis

Warning

Debug logging generates large log files and may impact performance


Option 9: Populate Knowledge GraphπŸ”—

9. Perform a ETL call to populate the knowledge graph

Purpose: Manually triggers data extraction, transformation, and loading

What it does: Executes the knowledge graph population process

When to use:

  • After configuration changes
  • To refresh Monitoring data
  • Testing data pipeline functionality

Expected Behavior: Process runs and populates the knowledge graph with current data


Option 10: Set Error LoggingπŸ”—

10. Set logging level to ERROR

Purpose: Reduces logging to only error messages

Effect: Changes LOG_LEVEL=ERROR in monitoring.ini

When to use:

  • After debugging is complete
  • To reduce log file sizes
  • For normal production operation

Option 11: Create Debug SnapshotπŸ”—

11. Create Monitoring debug snapshot for Verity Support

Purpose: Generates comprehensive diagnostic package for technical support

Prerequisites:

  • Must attempt steps 1-9 first
  • Specify which step failed

What it includes:

  • Complete system status check
  • All container logs (latest 10,000 lines)
  • Error logs for each container
  • System resource information
  • Docker container status and stats
  • Configuration settings
  • Time synchronization status
  • API connectivity results

Output Files: - monitoring_debug_package_YYYYMMDD_HHMMSS.tar - Complete debug package

Archive Contents:

monitoring_service_status_results_YYYYMMDD_HHMMSS.txt
<container_name>_latest.log (for each container)
<container_name>_errors.log (for each container)
system_info.txt

When to use: When you need to provide comprehensive system information to Verity Support


Option 12: Restore Docker VolumeπŸ”—

12. Restore Docker Volume

Purpose: Provides restoration of Docker volume.

Choose your docker volume from the options provided.

Options:

12.1. Prometheus: Restore Prometheus Volume from backup

Purpose: Restores the Prometheus Docker volume from a backup.

What it does:

  • Prompts the user to select one of the existing Prometheus backups
  • Performs a Docker Volume backup based on the selected backup

When to use:

  • After data corruption in Prometheus
  • To revert Prometheus data to a previous state
  • For disaster recovery

12.2. Quit


Option 13: Reinstall of Latest Monitoring Application PackageπŸ”—

13. Reinstall of Latest Monitoring Application Package
What it does: Reinstalls the latest Monitoring Application Package

Option 14: Create a Prometheus BackupπŸ”—

14. Create a Prometheus Backup
What it does: Creates a Prometheus backup that can be imported via Option 12: Restore Docker Volume.


Option 15: Perform Hard Drive CleanupπŸ”—

15. Perform Hard Drive Cleanup

Purpose: Frees disk space on the Monitoring VM when it is running low

What it does:

  • Checks free space on the root disk. If 20 GB or more is free, it does nothing and reports that no cleanup is needed
  • If less than 20 GB is free, it:
    • Keeps only the latest Prometheus backup for each volume
    • Removes unused Docker volumes (docker volume prune)
    • Deletes .log files older than 3 days
    • Removes Docker volumes that are not named be_* or tpg_*, if they are not in use
    • Reports how much space was recovered and the new free space
  • Writes its output to /be_monitoring/logs/system/cleanup_space.log

When to use:

  • When the VM is low on disk space (below 20 GB free)
  • Before reinstalling or upgrading, if free space is limited

Note

Older Prometheus backups are deleted, and only the newest one is kept. If you need older backups, copy them off the VM first. Logs older than 3 days are not recoverable.


Option 16: Restart the Monitoring ServicesπŸ”—

16. Restart the Monitoring Services

Purpose: Restarts all monitoring services (Docker containers) on the Monitoring VM

What it does:

  • Regenerates the Alertmanager configuration from the saved settings, including any email notification settings
  • Stops all monitoring containers (docker compose down)
  • Starts them again (docker compose up -d)

When to use:

  • After changing configuration settings
  • When a service is unresponsive or unhealthy
  • After editing email notification settings

Note

All monitoring services are unavailable while the containers restart. Allow about 3 minutes for them to become healthy again.


Option 17: Toggle Neo4j Gateway AccessπŸ”—

17. Toggle Neo4j Gateway Access

Purpose: Temporarily allows Neo4j Browser to connect to the knowledge graph database through the gateway for diagnostics. Run the option again to turn access off.

What it does:

When access is currently off (secured):

  • Saves the current Neo4j configuration
  • Switches Neo4j to a diagnostic configuration with no authentication
  • Opens the /neo4j/ and Bolt WebSocket routes on the gateway and reloads the gateway
  • Restarts the neo4j-graph container (takes about 30 seconds)
  • Displays the connection details:
    • Neo4j Browser: http(s)://<host>:<gateway-port>/neo4j/browser/
    • Bolt connect URL: bolt+ws://<host>:7688 (choose No authentication)

When access is currently on:

  • Restores the saved secured Neo4j configuration
  • Removes the gateway routes and reloads the gateway
  • Restarts the neo4j-graph container

When to use:

  • When Verity Support asks you to inspect the knowledge graph
  • To troubleshoot topology or data problems in Neo4j

Security Warning

While access is on, anyone who can reach the gateway can read the Neo4j database without credentials. Use this option only while troubleshooting, and run it again to restore the secured configuration when you are done.


Option 18: Toggle TLS Verification Between vNetC and MonitoringπŸ”—

18. Toggle TLS Verification Between vNetC and Monitoring

Purpose: Turns TLS certificate verification for calls from the Monitoring VM to the vNetC off for troubleshooting, and back on. Run the option again to switch back.

What it does:

When verification is currently on:

  • Displays a warning and asks you to type YES to confirm. Any other answer cancels with no change
  • Saves the current NBI_VERIFY_SSL setting, the stored vNetC CA bundle and the CA fingerprint pin
  • Sets NBI_VERIFY_SSL=false
  • Restarts the services that call the vNetC (mcp-server, sensai-prompt-service and verity-ml-python)
  • Records the change in /be_install/logs/nbi_tls_toggle.log

When verification is currently off:

  • Restores the saved NBI_VERIFY_SSL setting, CA bundle and fingerprint pin
  • Restarts the same services
  • Records the change in /be_install/logs/nbi_tls_toggle.log

When to use:

  • To confirm whether SSL errors between the Monitoring VM and the vNetC come from certificate verification
  • Only as a temporary measure while you fix the vNetC certificate or CA trust

Security Warning

While verification is off, credentials and API traffic between the Monitoring VM and the vNetC can be intercepted. Turn verification back on as soon as you finish troubleshooting. Installing a new package also turns verification back on automatically.


Option 99: ExitπŸ”—

99. Exit
Exits the menu.


Daily Health CheckπŸ”—

  1. Run Option 1 (Check services)
  2. Run Option 2 (Check time sync)
  3. Run Option 7 (Check API)

Troubleshooting WorkflowπŸ”—

  1. Set debug logging (Option 8)
  2. Check all services (Option 1)
  3. Verify connectivity (Options 6, 7)
  4. Perform ETL test (Option 9)
  5. Create debug snapshot (Option 11)
  6. Reset logging level (Option 10)

Pre-Support ContactπŸ”—

  1. Complete steps 1-9 in order
  2. Note which step fails
  3. Create debug snapshot (Option 11)
  4. Provide the generated tar file to support

Expected Output ExamplesπŸ”—

Healthy System OutputπŸ”—

Select:
    1. Check if the Monitoring services are running
    ...
Enter your choice: 1

ETL services are running.
Grafana services are running.
Prometheus services are running.
Promtail services are running.
Telegraf services are running.
Neo4j-graph services are running.
Loki services are running.
Cadvisor services are running.
Alertmanager services are running.
Node-exporter services are running.
AI-service services are running.
Neo4j-vector services are running.

--------------------------------------------
Total Services Tested: 12
Currently Running: 12
Not Running: 0
--------------------------------------------

Press Enter to continue...

Problem System OutputπŸ”—

ETL services are running.
Grafana services is not running.
Prometheus services are running.
...
--------------------------------------------
Total Services Tested: 12
Currently Running: 11
Not Running: 1
--------------------------------------------
run the command 'sudo docker compose -f /be_monitoring/docker-compose.yml up -d' and re-run this script

Troubleshooting Common IssuesπŸ”—

No Docker Containers RunningπŸ”—

Symptoms: All services show "not running"

Solution:

sudo docker compose -f /be_monitoring/docker-compose.yml up -d

API Calls Failing (HTTP 401)πŸ”—

Symptoms: Option 7 shows HTTP status code 401

Cause: Invalid API credentials

Solution: Check /be_monitoring/.env file for correct credentials

API Calls Failing (HTTP 000)πŸ”—

Symptoms: Option 7 shows HTTP status code 000

Cause: Network connectivity issue

Solution: 1. Check Option 3 for correct FQDN 2. Verify network connectivity: ping <fqdn> 3. Check DNS resolution: nslookup <fqdn>

Time Synchronization IssuesπŸ”—

Symptoms: Option 2 shows System clock synchronized: no

Solution:

sudo systemctl restart systemd-timesyncd
sudo timedatectl set-ntp true

Debug Snapshot Creation FailsπŸ”—

Symptoms: Option 11 fails to create archive

Cause: Insufficient disk space or permissions

Solution:

# Check disk space
df -h
# Ensure proper permissions
sudo chown -R $USER:$USER ./

Log File LocationsπŸ”—

Script OutputsπŸ”—

  • Debug snapshots: ./monitoring_debug_package_YYYYMMDD_HHMMSS.tar
  • Container logs: ./monitoring_logs_snapshot/

System LogsπŸ”—

  • Docker containers: sudo docker logs <container_name>
  • System logs: /var/log/syslog
  • Monitoring logs: /be_monitoring/logs/ (if exists)

Best PracticesπŸ”—

Regular MaintenanceπŸ”—

  1. Run daily health checks (Options 1, 2, 7).
  2. Monitor log levels (keep at ERROR for production).
  3. Perform periodic ETL operations (Option 9).

Before Support ContactπŸ”—

  1. Always run the complete diagnostic sequence (1-9).
  2. Create debug snapshot (Option 11).
  3. Note exact error messages and timing.
  4. Include system information and recent changes.

Safety GuidelinesπŸ”—

  1. Test changes in non-production first.
  2. Always create debug snapshots before major changes.
  3. Keep debug logging temporary (return to ERROR level).

Recovery ProceduresπŸ”—

Complete System RecoveryπŸ”—

If multiple services are down:

# Stop all containers
sudo docker compose -f /be_monitoring/docker-compose.yml down

# Start all containers
sudo docker compose -f /be_monitoring/docker-compose.yml up -d

# Wait 2 minutes, then check status
sleep 120
# Run troubleshooting script Option 1

Data RecoveryπŸ”—

If data appears corrupted:

  1. Use Option 12 to restore the Prometheus volume from backup.
  2. Re-run setup script if necessary.

Support InformationπŸ”—

When to Contact SupportπŸ”—

  • Multiple services consistently failing
  • API connectivity issues persist after basic troubleshooting
  • Data corruption or loss
  • Before using Option 999

Information to ProvideπŸ”—

  • Debug snapshot tar file from Option 11
  • Specific error messages
  • Recent system changes
  • Output from Options 1, 2, 3, 4, 5, 7

Self-Help ResourcesπŸ”—

  • Check Docker container logs: sudo docker logs <container_name>
  • Verify system resources: free -h and df -h
  • Check network connectivity to vNetC host
  • Review monitoring.ini configuration file

This troubleshooting script is designed to be comprehensive and self-contained, providing both diagnostic capabilities and maintenance functions for Monitoring.