Troubleshoot Analytics Access and Certificate Issues

Last updated
Save as PDF

For supported software information, click here.

When you select the Analytics tab in the Director GUI, the Director node connects to the Analytics application on a node in an Analytics cluster. The connection is performed over a TCP port and uses a self-signed certificate for role-based access control (RBAC) validation. If you receive an error message when selecting the Analytics tab, it is likely due to one of the following issues:

Analytics application is not accessible
Analytics TCP port is not accessible
SSL certificates are not synced

This article describes how to identify and resolve these issues.

Identify Analytics Application Accessibility Issues

The Analytics application must be running on the Analytics node in order for the Director GUI to connect to it. This section describes how to verify that the Analytics application is running and suggests steps to determine whether there is a certificate or connector issue on the Director node.

To identify and resolve Analytics application accessibility issues:

Log in to a shell on the Analytics node. The default username is admin, and the password is versa123.
From the shell, issue the vsh status command. Verify that the versa-analytics-app line, for the Analytics application, displays the status Running.

admin@Analytics$ vsh status
versa-confd                  is Running                 
versa-lced                   is Running                 
versa-spackmgr               is Running                 
versa-analytics-monitor      is Running                 
versa-analytics-driver       is Running                 
versa-analytics-dataops      is Running                 
versa-analytics-app          is Running                 
versa-monit                  is Running

If the application is not running, start it by issuing the following command. After the application starts, select the Director Analytics tab again in the Director GUI to determine whether the access issue is resolved. If you still cannot access the Analytics node by select the Analytics tab in the Director GUI, continue with the next step.
```
admin@Analytics$ vsh start
```
Access the Analytics application directly from a browser window by entering the URL https://ip-address-of-analytics-node:8443. For Releases 20.2.x, you can use port 8080.
- If the login screen displays as shown above, continue with the next step.
- If an error message displays, there is likely an underlying issue with connecting to the port or the Analytics node. Check whether any firewall rule is blocking communication between the Director and Analytics nodes on port 8443.
Enter your Director username and password, and then click the Sign In icon. The Analytics application GUI interface should display on the screen.
- If the Analytics application GUI displays but the Analytics tab in the Director GUI still fails, there might be a configuration issue with the Analytics connector on the Director node. For more information, see Troubleshoot Analytics Connector Issues, below.
- If the "Invalid username or password" message displays, there might be a certificate issue between the Director and Analytics nodes. For more information, see Troubleshoot Certificate Issues, below.

Troubleshoot Analytics Connector Issues

The Director node uses the IP address and port number settings of an Analytics connector to determine how to access the Analytics application. When the Analytics tab fails to connect to a cluster, the issue might be with these settings.

If your topology contains multiple Analytics clusters, you must also select a node from the correct Analytics cluster after you select the Analytics tab. You can use any node in a cluster to administer the other nodes in a cluster.

To select a node from a specific Analytics cluster:

In Director view, select the Analytics tab.
To select a node for Releases 22.1.1 and later, hover over the Analytics tab and then select a node from the cluster. For Releases 21.2 and earlier, select a node in the drop-down menu in the horizontal menu bar.
If no nodes from the cluster display, there might be an Analytics connector issue.

To identify and resolve an Analytics connector issue:

In Director view, select the Administration tab in the top menu bar.
Select Connectors > Analytics Cluster in the left menu bar. The main pane displays the connectors that are already configured.
Select a connector in the main pane. The Edit Analytics Cluster popup window displays.
Verify that the connector is configured with the correct IP address for the nodes in the Analytics cluster and that the port number is correct. For Releases 21.1.x and later, the connector should use port 8443. For Releases 20.2.x, you can also use port 8080.
Modify the values if required.
Click OK.

Troubleshoot Certificate Issues

When you perform the initial software configuration, you copy certificates between Director and Analytics nodes. These certificates allow the Director GUI to access the Analytics application. This section describes how to verify that the certificates are correct, how to verify that the /etc/hosts file is in sync with the certificates, and how to fix an incorrect configuration.

To identify and resolve certificate issues:

admin@Director$ sudo su - versa
versa@Director$ whoami
versa

Verify that Director certificates match those on the Analytics nodes by executing the vnms-cert-sync.sh script with the --verify option.

The script prompts you to enter the name of the Analytics cluster connector, which it refers to as the VAN Cluster Name. You can find the Analytics cluster connector name in the Director GUI. In Director view, select Administration > Connectors > Analytics Cluster in the left menu bar. The connector names are displayed in the Cluster Name column in the main pane. In the example below, the connector name is Analytics.

versa@Director$ /opt/versa/vnms/scripts/vnms-cert-sync.sh --verify
Syncing Director certificates to VAN Cluster
Enter VAN Cluster Name:
Analytics

(output omitted for brevity)

verifying certificates
MD5 hash matches with Director certificate on 10.48.80.17
MD5-Director:54b5c78863a2112d5f9a927a2b2a6f58
MD5Remote:54b5c78863a2112d5f9a927a2b2a6f58

If the script returns "MD5 hash matches", continue with Step 5.
If the script returns a message stating that the MD5 sum does not match, continue with Step 4.

Optionally, to compare certificates using the md5sum command, issue the following command . Compare the checksum values between nodes to verify that they are identical.

On the Director node:

versa@Director$ md5sum /var/versa/vnms/data/certs/versa_director_client.cer
9e7c4a1eb717f567d81d69cc4dc7cc70  /var/versa/vnms/data/certs/versa_director_client.cer

On the Analytics node:

admin@Analytics$ md5sum /opt/versa/var/van-app/certificates/versa_director_client.cer
9e7c4a1eb717f567d81d69cc4dc7cc70  /opt/versa/var/van-app/certificates/versa_director_client.cer

If Step 2 or Step 3 shows an MD5 sum mismatch, re-sync the certificates by issuing the following command.

versa@Director$ /opt/versa/vnms/scripts/vnms-cert-sync.sh --sync

Display the certificates on the Director node, and then verify that their common name (CN) and subject alternative name (SAN) values match those listed in the /etc/hosts file on the Analytics node.
1. Change to the /var/versa/vnms/data/certs directory, and verify that the versa_director_client.cer file is present.

versa@Director$ cd /var/versa/vnms/data/certs
versa@Director$ ls -lrtk versa_director_client.cer
-rw-rw---- 1 versa versa 1180 Jan  7  2020 versa_director_client.cer

Execute the keytool utility on the certificate file using the options shown in the example below. Make a note of the CN and SAN values in the certificate.

versa@Director$ /usr/lib/jvm/java-11-openjdk-amd64/bin/keytool --printcert -v -file versa_director_client.cer
Owner: CN=Director1, OU=VersaDirector, O=versa-networks, ST=California, C=US
Issuer: L=Santa Clara, ST=California, C=US, OU=VersaDirector, O=versa-networks, CN=versa-director1
Serial number: 1
Valid from: Thu Jul 25 06:11:30 UTC 2019 until: Sun Jul 22 06:11:30 UTC 2029
Certificate fingerprints:
SHA1: E6:06:8F:F2:6E:77:2C:1C:88:30:60:8B:57:5A:AD:58:E2:6E:66:9F
SHA256: 68:D8:0C:EE:74:F8:DB:2E:FC:67:DE:32:C8:1E:18:FC:16:75:79:68:60:15:0D:0D:59:25:CB:BE:20:04:79:EF
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
Extensions:
#1: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:false
  PathLen: undefined
]
#2: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName:[
  DNSName: Director1
  DNSName: Director2
]

Log in to a shell on each Analytics node in the cluster, and then verify that the /etc/hosts entry for the Director node matches the values from the certificate. Use the hostname from the CN and SAN values in the certificate.

Caution: If you are using a wildcard certificate, use the full domain name when you create an entry in the /etc/hosts file. For example, if the CN value is *.utt.com, use hostnames Director1.utt.com and Director2.utt.com. Enter the same full domain name (for example, Director1.utt.com) when you register Versa Director on the Analytics Authentication page.

admin@Analytics$ cat /etc/hosts
127.0.0.1 localhost
192.10.10.56 Analytics
10.192.215.145 versa-director1 Director1
10.192.215.146 versa-director2 Director2

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Verify that the Analytics node can access ports 9182 and 9183 on the Director node by issuing the nc –zvw3 command, using the Director hostname and port number as arguments.

admin@Analytics$ nc -zvw3 Director1 9182
Connection to Director5 9182 port [tcp/*] succeeded!
admin@Analytics$ nc -zvw3 Director1 9183
Connection to Director5 9182 port [tcp/*] succeeded!

If the commands return "connection refused" or any other error, check that there are no firewalls blocking port 9182 or 9183 and that routes are present to access the Director node. Issue the route -n command to confirm the routes on Analytics and Director nodes. Ping the Director node from the Analytics node to ensure routing is performing correctly. For a list of port numbers that Analytics nodes use for communication with other devices in the network, see Firewall Requirements.

Ensure that the certificates from the Analytics nodes are pulled into the versa_analytics_truststore.ts file on the Director node.

In the Director shell, issue the following commands to check whether the versa_analytics_truststore.ts file has been created on the Director node:

versa@Director$ cd /var/versa/vnms/data/certs
versa@Director$ ls -tlr versa_analytics_truststore.ts
-rw-rw---- 1 versa versa 1274 Jul 30 05:42 versa_analytics_truststore.ts

If the versa_analytics_truststore.ts file does not exist, or if the Versa Analytics certificates were regenerated, resynchronize and import the Analytics certificates by executing the vd-van-cert-upgrade.sh script in the active Director shell. This script transfers the Analytics certificates from each of the Analytics nodes configured under the connectors and then imports them. You must restart services on the Director node for certificate changes to take effect.

Note: Enter y when the script asks if you want to postpone the restart. This allows you to restart Director services manually at a convenient time.

versa@Director$ cd /opt/versa/vnms/scripts
versa@Director$ ./vd-van-cert-upgrade.sh --pull
Pulling Analytics certificates to Director key store
Checking previous version config path
Changing port for [Analytics]
No modifications to commit.
Port Migration completed
VAN Clusters IPs: [ 10.48.189.23 ]
Removing previous analytics cert store
Getting Certificate for : 10.48.189.23
depth=0 C = US, ST = California, L = Santa Clara, O = versa-networks, OU = VersaAnalytics, CN = versa-analytics
verify error:num=18:self signed certificate
verify return:1
depth=0 C = US, ST = California, L = Santa Clara, O = versa-networks, OU = VersaAnalytics, CN = versa-analytics
verify return:1
DONE
Importing Certificate for : 10.48.189.23
Certificate was added to keystore
Certificates Imported... Requires restart.. Do you want to post pone restart (y/N): y

To restart services manually when the node is not part of an HA Director pair, issue the vsh restart command.

versa@Director$ vsh restart
Stopping VNMS service
------------------------------------
Stopping TOMCAT................[Stopped]
Stopping REDIS.................[Stopped]
Stopping NETBOX-IPAM...........[Stopped]
Stopping POSTGRE...............[Stopped]
Stopping SPRING-BOOT...........[Stopped]
Stopping SPACKMGR..............[Stopped]
Stopping NCS...................[Stopped]
* Stopping daemon monitor monit
Starting VNMS service
------------------------------------
Starting NCS...................[Started]
Starting POSTGRE...............[Started]
Starting NETBOX-IPAM...........[Started]
Starting SPRING-BOOT.......... [Started]
Starting REDIS.................[Started]
Starting TOMCAT................[Started]

To restart services manually when the node is part of an HA Director pair, issue the following commands.

On the standby node:

versa@Director-standby$ vsh stop

On the active node:

versa@Director-active$ vsh restart

On the standby node:

versa@Director-standby$ vsh start

Issue the following commands on the active and standby nodes to ensure that they are in sync:

versa@Director$ cli
versa@Director> request vnmsha actions check-sync-status 
versa@Director> exit

In the Analytics application, revoke and then register the Director nodes.
1. Access the application from a browser window by entering the URL https://ip-address-of-analytics-node:8443. For Releases 20.2.x, you can also use port 8080.
2. Select Administration > Configurations > Authentication in the left menu bar. The following screen displays, and the main pane displays the hostname of the Director node.
3. Click Revoke, and then click Register for the Director node.
If the registration fails, collect the contents of the catalina.log file and then continue with Step 9.

admin@Analytics$ cat /var/log/versa/tomcat/catalina.log

If you still cannot use the Analytics tab in the Director GUI to connect to the Analytics node, contact Versa Networks Customer Support and attach the output of the tomcat and web log files.

To collect this output, open two shell windows with logging enabled so that you can capture the command output. In the first window log in to the Analytics node, and in the second login to the Director node.

From the shell window that is logged in to the Analytics node, issue the following command:

admin@Analytics$ tail -f /var/log/versa/tomcat/*.log

From the shell window that is logged in to the Director node, issue the following commands:

admin@Director$ cd /var/log/vnms/spring-boot/
admin@Director$ tail -f vnms-spring-boot.log /var/log/vnms/web/*.log

Leave the tail command running in each of the two shell windows. Then, select the Analytics tab in the Director GUI. The goal is to produce an error message in the shell windows, so select the Analytics tab multiple times if required. Go to the shell windows and press Ctrl-C to terminate the tail commands. Collect the shell window output and attach to the Customer Support ticket.

Supported Software Information

Releases 20.2 and later support all content in this article, except:

Releases 21.1.x and later no longer support using port 8080 to access the Analytics application. Use port 8443 instead.

Additional Information

Create and Manage Certificates
Perform Initial Software Configuration