Versa Director Issues

Communication Issues between CPE Devices and Director Nodes

To troubleshoot connectivity issues between customer premise equipment (CPE) devices and Director nodes:

1. To check whether Netconf is running on the CPE device, issue the netstat –ntua | grep 2022 Linux command. For example:

admin@Branch1-10-192-125-49~$ netstat -ntua | grep 2022
tcp 0   0 0.0.0.0:2022        0.0.0.0:*                LISTEN

2. To check the reachability of the CPE device's IP address and Netconf port from the Director node, issue the /opt/versa/vnms/scripts/netconf-check.sh ip-address shell command. When the command completes successfully, the following message displays:

===== SUCCESS: IPAddress and NetConf port is reachable and also able to do basic handshake====

3. To check whether routes are present on the Director node towards the CPE device, issue the route –n Linux command. For example:

Administrator@Director:~$ route -n
10.0.0.0 192.168.43.1 255.0.0.0 UG 0 0 0 eth1

4. On the Controller node, issue the show configuration system session | display set CLI command to enable tcp-adjust-mss (that is, set it to true) and disable check-tcp-syn (that is, set it false). For example:

admin@Controller1-cli> show configuration system session | display set
set system session check-tcp-syn false
set system session tcp-adjust-mss enable true

5. On the Director node, issue the show devices device device-name netconf-notifications subscription CLI command to check the Netconf notification subscription status. For example:

Administrator@VDir> show devices device Branch-1 netconf-notifications subscription status
NAME             STATUS  
--------------------------
voae_oam_subscr  running

6. Check whether the firewall service is enabled on Controller nodes. If it is enabled, make sure that objects and allow policies are in place.
7. To ensure that the default value of the IPsec tunnel MTU is configured with a value of 1336 or less, issue the show interfaces detail tvi-0/2 | grep mtu CLI command. For example:

admin@Controller1-cli> show interfaces detail tvi-0/2 | grep mtu
MTU : 1336 

Connectivity Issues between Director and Analytics Nodes

To troubleshoot connectivity issues between Director and Analytics nodes:

1. To ensure that the Director hostname is correct while installing the Director certificate on the Analytics Nodes, issue the sudo /opt/versa/scripts/van-scripts/van-vd-cert-install.sh./versa_director_client.cer director-hostname Linux command.
2. To ensure that port 9182 is listening on the Analytics interface that is communicating with the Director node, issue the netstat –ntua | grep 9182 Linux command in the Director shell. For example:

Administrator@VDir:~$ netstat -ntua | grep 9182
tcp6 0 0 10.192.125.42:9182 :::* LISTEN
Administrator@VDir:~$

HA Issues

To troubleshoot Director high availability issues:

1. Ensure that the software build is same on both the primary and secondary Director nodes, and ensure that the entries in /etc/hostname and /etc/hosts are consistent.

2. To check the current HA status on both the primary and secondary Director nodes, issue the request vnmsha actions status CLI command.
3. To check the current HA details on both the primary and secondary Director nodes, issue the request vnmsha actions get-vnmsha-details fetch-peer-vnmsha-details true CLI command. For example:

Administrator@versa-director> request vnmsha actions get-vnmsha-details fetch-peer-vnmsha-details true
peer-vnmsha- detail {
mgmt-ip- address 8.8.8.2
Administrator@versa-director:~$ cat /etc/hosts
8.8.8.2 versa-director

4. To ensure that you can establish an SSH between the primary and scondary Director nodes, issue the request vnmsha actions get-vnmsha-postgres-status command.
5. Ensure that all ports required for HA communication are open. For more information, see Firewall Requirements.
6. In rare cases, a split-brain situation make occur on the Director node, in which both nodes appear in the active state. If this occurs you might see the following message on both Director nodes:
   
   

   To resolve this issue, log in to both Director nodes and issue following CLI command:

> request vnmsha actions disable-ha

Then, you must manually reconfigure Director HA from the node that has the latest configuration.

TACACS+ Issues

To troubleshoot the Director TACACS+ issues:

1. Log in by affixing @System for system users and @orgName for tenant users.
2. Configure TACACS+ as the default connector.
3. Configure TACACS+ as an authentication connector for the organization.
4. To start a TACACS+ client fromthe CLI, issue the the $ /opt/versa/vnms/scripts/tac_plus_client $tacacs_ipaddress $tacacs_port $tacacs_secret $userName $passwd command.

Tools Not Working on Director Monitor Screen

The ping, tcpdump, and traceroute functions are not activated when you click Start/Stop from the Monitor tab in the Versa Director. To enable them, you must run the WWW service in the branch configuration. By default, the WWW service is disabled.

To enable the WWW service in a branch configuration:

1. In Director view, select a device.
2. In Appliance view, select the Configuration tab in the top menu bar.
3. Select Others > System > Configuration > Configuration in the left menu bar to view the branch configuration.
   
   
4. In the Services pane, click the Edit icon.
   
   
5. Click WWW to enable the WWW service.

If you get the Connected message in the results section followed by the Accept SSL certificate error message:

1. Click the link to accept the certificate.
2. Log out from the Versa Director.
3. Log in to the Versa Director.

The service is now connected and you can start using them.

Errors while Deploying a New Organization

When you add an interface configuration to a new Controller node during the onboarding process, the interface configuration is automatically added to the Controller node. However, when you add an interface configuration to an existing Controller node, the newly added interface is added only to new organizations, but not to the existing organizations, and the following errors display:

• Failed to on-board org for controller
• com.versa.vnms.cdbadaptor.common.exception. ProcessingFailureException: Failed to populate customer org data

To resolve these errors, add the interface configurations to site configuration:

1. Log in to Versa Director.
2. In Director view, select the device.
3. In Appliance view, select the Configuration tab in the top menu bar.
4. Select Services > SD-WAN > System > Site Configuration in the left menu bar.
5. Select Site Configuration window and click the Edit icon. The Edit Site Configuration popup window displays.
6. Click the Add icon in the WAN Interfaces table.
7. In the Add WAN Interfaces popup window, add information about the interface.

Branch Out of Sync with Director Node

A device is considered out of sync when it becomes reachable from the Director node and the "SSH host key-mismatch" error message displays.

To resolve the issue:

1. Compare the key of the Branch that is in sync with the key of the problematic branch.
2. If it does not match, issue the .ssh$ pwd command, note the path, and enter the working branch key to the out of sync branch on the same path.

admin@Versa-CPE:~/.ssh$ pwd
/home/admin/.ssh
ssh-dss AAAAB3NzaC1kc3MAAACBALP7ajSyNetN18XZrpmrhyMcqWVAjwrgiyN8mwsdfCAN0iEQr0QlRh4JeHkV+VciaS6JvzwjtcWaZiu++KVJIazei4JUsuqjVxTA5GjzfdHSaNR8XEx7g1ovfQXjTKYqI1ew+YoLcI7+FypjnKfYV/
FtKOuEqLAE25dL3QdvAAAAFQCYMXGDJcPUlv3poMJQk1gNHx1mvQAAAIAPzOG05h4BbD5ULK4XJnEnnzBkBfnjl7jwbXgLE97A1lU8MD3Yy5bosoX85qRf5RkTvgzn4gokGwErGY3/GWLH9jshx1Uz6PMKGZK/
WyvTgmwmD8jlBMma5sAZv+lr9kTza6g9tP5Mn6KlvhXGHvlRplQM0+oELUaOZAAAAIAp3HsgMWZ+g2fNj4gaYYzLxtS9CreWGxOyeec+Efh+7GxEAZh8FvL7t+dzoLb0QFvaB6xIsmjtk9NJn9Xmt0tfue0nDDPHNmTq8ak8zFJs5mBQEM4C3aYiOB/
7WQRTOYTLBzhCPQC6a3m/RHyaScm1OqV9fbhDbzOEwWQcpg== root@versa-director

Disk Utilization Problems

If the disk utilization of any of the Director partitions is greater than 70 percent, a disk utilization warning is displayed on the Director screen. To reduce disk space usage, you can do any of the following

• Clean up log files in the /var/log/vnms folder—You can delete old logs that have been archived. To do this, run the following CLI command:

user@Director:~$ sudo find /var/log/vnms -name "*.gz" -delete

You can also selectively delete log files that you do not use, but you should exercise caution so as not to delete files erroneously.

• Delete old packages that you no longer need—There may be old upgrade packages from Versa Director or VOS devices on the Director node. To delete images that were uploaded through the GUI, in the Director GUI, select the Administration tab in the top menu bar and then select Inventory > Images in the left menu bar. To delete images that were uploaded through the CLI, go to the /var/versa/packages/vnms and /var/versa/packages/device/ directories and selectively delete the old packages.
• Delete old service packages (SPacks) and OS SPacks—To delete old SPacks and OS SPacks, in the Director GUI, select the Administration tab in the top menu bar and then select Inventory > Security Packages and Inventory > OS Security Packages in the left menu bar. If the Director GUI is not available because of high disk utilization (100%), you can manually delete the unnecessary SPacks and OS SPacks from the /var/versa/packages directory.
• Delete temporary files—To delete all the contents of the /var/tmp and /tmp directories:

user@Director:~$ rm -rf /var/tmp* /tmp*

• Reduce the number of backup files stored on the Director node—You can delete all backup files that are older than 2 days, or you can copy the backup files to a remote device. Backup files are in the /var/versa/backups directory.
• Check which directories are using a lot of disk space, and remove contents from them if possible—To check the directories, run the following command:

user@Director:~$ sudo du -hd 1 path 2> /dev/null

For example:

• Reduce the amount of disk space used by the Postgres database—To check how much disk space PostgreSQL is using, run the following command:

  user@Director:~$ sudo du -kh -d1 /var/lib/postgresql/9.5/main/
  

If you determine that Postgres is using too much space on the hard drive, clean up the database:

1. Run the following command:

user@Director:~$ sudo su - postgres

2. Locate the REDO WAL file string:

user@Director:~$ /usr/lib/postgresql/9.5/bin/pg_controldata /var/lib/postgresql/9.5/main | grep "REDO WAL file"

For example:

postgres@Director:~$ /usr/lib/postgresql/9.5/bin/pg_controldata /var/lib/postgresql/9.5/main | grep "REDO WAL file"
Latest checkpoint's REDO WAL file:          000000010000000000000012

3. Use the output from the grep command to clean up the Postgres database. For example:

postgres@Director:~$ pg_archivecleanup -d /var/lib/postgresql/9.5/main/pg_xlog 000000010000000000000012

To permanently decrease the number of archives generated, you can do the following:

1. Edit the /etc/postgresql/9.5/main/postgresql.streaming.conf file, and make the following changes:
   1. Change the archive_mode entry from archive_mode = on to archive_mode = off.
   2. Change the archive_command entry from archive_command = '/bin/true' to archive_command = ''.
2. Edit the /opt/versa/vnms/etc/conf/postgre/postgresql.streaming.conf file, and make the following changes:
   1. Change the archive_mode entry from archive_mode = on to archive_mode = off.
   2. Change the archive_command entry from archive_command = '/bin/true' to archive_command = ''.

Another permanent solution is to increase the size of partitions and filesystems on the Director node. For more information, see Expand Disk Storage for Analytics Nodes. 

Supported Software Information

Releases 20.2 and later support all content described in this article.

Additional Information

Expand Disk Storage for Director Nodes