Skip to content

Troubleshooting🔗

Topology🔗

Verity displays a live map of the devices in the network and their connections.

Topology Detection and Behavior🔗

  1. Autodiscovery
  2. Connection moves
  3. Remove connections

Reports🔗

Verity provides a comprehensive list of reports to identify current conditions in the network. Reports are viewed by navigating to the Reports window or clicking the Reports icon of a device object .

MAC Address Workbench🔗

Verity provides the ability to display MAC addresses across the entire system.

The Mac Address Workbench lets you search device by mac address. Clicking the Search button without specifying a mac address displays all devices .

From within a list of devices, clicking opens a new tab to search on the corresponding mac address .

Ping and Trace Route🔗

Verity provides tools to troubleshoot Layer 3 issues for the underlay and Tenant connections for services configure for Layer 3. The Ping and Trace Route feature is a diagnostic tool that sends a ‘ping’ to and from a specified address and details the message path.

Beware of Misconfigurations

This feature is only applicable to systems using GNMI (gRPC Network Management Interface ). Before you use this feature, ensure the chosen device has its Device Controller/Managed Device Comm Type set to GNMI.

Controls🔗

The Ping and Trace Route dialog box is opened by clicking Open Ping/Traceroute Dialog button .

To use this feature you configure the following parameters and click the Start button .

Source🔗

  • Device: This is the source device of the ping.
  • VRF: This is where you select the VRF. The VRFs of Tenants, the Underlay (selected device), and Management (Device Controller) are all options.
  • IP: The targeted IP address to send the ping command from. The IP address changes depending on the VRF setting.

Destination🔗

  • IP: The targeted IP address to receive the ping command.

Ping🔗

  • Count: The number of ping events sent before the process is complete.

Trace Route🔗

When this option is selected, the trace route is included in the Result field.

CMD Status🔗

This is a progress indicator. When the feature is ready, CMD Status says Ready; when active, CMD Status says In Progress; and when the process is complete, it says Done.

DHCP Snooping🔗

Verity provides the location of connected L3 addresses detected by DHCP snooping functions in SONiC.

To enable DHCP snooping navigate to Topology/Site Settings and check the box next to Enable DHCP Snooping .

Viewing DHCP Assigned IP's🔗

To view DHCP assigned IP addresses go to Reports/DHCP Assigned IPs .

View License🔗

To view the license go to Administration/Licensing

The Licensing object displays License Usage and Physical Port Usage bar graphs. While focused on the object, the user can see the date of licensing expiration, contact information for support, and reports of license.

By clicking on the report icons for License Utilization, License by Device Utilization, and Physical Port Utilization, a report for those lists will be shown. For a specific report on used, preprovisioned, stranded, or spare licenses, the user can click on each section of the bar graph. This can also be done with physical ports that are licensed, fabric or spare.

An example License Utilization report is shown below:

Monitoring VM🔗

Monitoring Setup Check Script🔗

This script provides a comprehensive set of tools to check and manage the monitoring system deployment. It offers various options to verify services, configurations, and perform essential monitoring tasks.

Prepare for Verity Support Snapshot🔗

Create Debug Snapshot is option 9 in the list below. Please attempt to run the previous options (1-8) before option 9.

Quicksteps🔗

  1. ssh to the monitoring VM
  2. Run the script sudo ~/check-monitoring-setup.sh
  3. Perform steps 1-8
  4. Select option 9 to create the debug package
  5. Exit the script - select option 10

  6. On a local cli instance:

    • scp the debug package to a local directory
    • example:

    scp verity@10.1.1.10:/home/verity/debug_snapshot_20241126101010.tar.gz .

This option is used to prepare the system for a Verity Support Snapshot. It creates a comprehensive debug package containing:

  • System status and configuration checks
  • Container logs and error reports
  • System information and resources
  • Docker container status

Available Options (1-10)🔗

A summary of each option is provided below.

1. Check Monitoring Services Status🔗

Performs a health check on all monitoring-related Docker services:

  • verity-ml-python (ETL services)
  • Grafana
  • Prometheus
  • Promtail
  • Telegraf
  • Neo4j-graph
  • Loki
  • Cadvisor
  • Alertmanager
  • Node-exporter

Provides a summary showing:

  • Total services tested
  • Number of running services
  • Number of non-running services

2. Check Time Synchronization Status🔗

Displays the current system time synchronization status using timedatectl. This is crucial for proper monitoring system operation and log correlation.

3. Report FQDN Setting🔗

Shows the Fully Qualified Domain Name (FQDN) setting from the Monitoring.ini file by extracting the ROOT_API_URL value. This was created as part of the setup.sh script. It is used to identify the url for the vNetC endpoint and northbound API.

4. Report MAC Address Setting🔗

Displays the VERITY_MONITORING_VM_MAC_ADDRESS setting from the Monitoring.ini file, which is used for system identification.

5. Report Monitoring Package Version🔗

Displays the current installed version of the monitoring package as specified in the Monitoring.ini file.

6. Perform Announcement Call🔗

Executes the announce_monitoring_to_vnetc.py script within the verity-ml-python container to announce the monitoring system to vNetC. This is used to register the monitoring system with the vNetC system and activates the dashboard triggers on the Verity GUI. And sets the monitoring credentials used to access the vNetC system.

7. Check vNetC API Status🔗

Verifies the Verity vNetC API connectivity by:

  • Retrieving API credentials from the .env file
  • Making a test API call
  • Reporting the HTTP status code (200 indicates success)

8. Perform ETL Call🔗

Executes an ETL (Extract, Transform, Load) process to populate the knowledge graph by running the knowledge_graph.py script within the verity-ml-python container.

9. Create Debug Snapshot🔗

This option is used to create a comprehensive debug package containing:

  • System status and configuration checks
  • Container logs and error reports
  • System information and resources
  • Docker container status

The debug package is saved as a single tar archive containing:

  • Debug snapshot with system status and checks
  • Latest logs for each container
  • Error logs for each container
  • System information and container status

10. Exit🔗

Exits the script.

Usage🔗

  1. ssh to the monitoring VM
  2. Run the script sudo ~/check-monitoring-setup.sh