Versa Analytics Configuration Concepts

Analytics Network Connections

Analytics nodes typically use separate interfaces to connect to management, control, and internal networks.

The following figure shows a sample topology containing an Analytics cluster with four Analytics nodes connected to three networks. In this example, there are no log forwarder nodes.

The following example shows a sample topology with 8 nodes. 4 nodes are log forwader nodes. 

• Management network—The northbound interface on each Analytics node typically connects to the management network. A Versa Director node uses this network to communicate with Analytics nodes when, for example, it displays an Analytics graph or creates a report. The IP address of the interface, usually interface eth0, on the Analytics node connecting to this network, is called the RPC address during initial configuration. 
• Control network—The southbound interface on each Analytics node typically connects to the control network. This network is used to send logs from Controller nodes to Analytics nodes. The IP address of the interface, usually interface eth1, connecting to this network is called the collector address during initial configuration.
• Internal network—This network is internal to the Analytics cluster. It is used by Analytics cluster nodes to exchange information, for example, to send logs to other nodes within the cluster and to synchronize databases within the cluster. The IP address of the interface, usually interface eth2, connecting to this network is called the listen address during initial configuration.

Note: Some configurations use a single network for both the control and internal networks.

Analytics Cluster Node Types

An Analytics cluster comprises Analytics nodes that run different combinations of software components. These combinations come in three types: analytics, search, and forwarder. 

• Search-type Analytics nodes—Search-type Analytics nodes store flow logs, event data, and packet captures and are used when searching logs in the Versa Director GUI. These nodes run the search platform and contain a datastore. It is recommended that you configure at least two search-type Analytics nodes within a cluster for high availability. Search-type Analytics nodes use the internal network to communicate with other nodes in the cluster. You can configure search-type Analytics nodes as Analytics log collector nodes.
• Analytics-type Analytics nodes—Analytics-type Analytics nodes store historical data and are used when creating many of the reports and graphs in the Versa Director GUI. These nodes run a noSQL database. You normally configure two analytics-type Analytics node within a cluster for high availability. Analytics-type Analytics nodes use the internal network to communicate with other nodes in the cluster. You can configure analytics-type Analytics nodes as Analytics log collector nodes.
• Forwarder-type Analytics nodes—Forwarder-type Analytics nodes do not run the noSQL database or search engine and should be configured as Analytics log collector nodes. These nodes accept incoming connections containing logs from VOS devices, called log export functionality (LEF) connections. Forwarder-type Analytics nodes are useful when an Analytics cluster has a large number of incoming LEF connections. You can send all LEF connections to forwarder-type Analytics nodes and have these nodes process all incoming logs for an Analytics cluster. You can alternately configure some VOS devices to send logs to the forwarder-type Analytics nodes and other VOS devices to send logs to the analytics-type and search-type nodes. Both options help distribute incoming LEF connections over multiple Analytics nodes within the cluster and avoid overwhelming the search-type and analytics-type Analytics nodes.
• Lightweight-type Analytics nodes—Lightweight-type Analytics nodes accept incoming connections containing logs from VOS devices and stream the logs to third-party collectors. They do not perform database operations and do not forward logs to other Analytics nodes in the cluster.
• Aggregator-type Analytics nodes—(For Releases 22.1.1 and later.) To provide a consolidated view for tenant-level reports, you can configure an Analytics aggregator node, which can consolidate and aggregate data from multiple standard Analytics clusters. Aggregator nodes generate reports by pulling data individually from standard Analytics clusters, called child clusters. For more information, see Configure Analytics Aggregator Nodes.
• Analytics log collector nodes—Analytics nodes that run the log collector exporter program to receive logs from VOS devices are referred to as Analytics log collector nodes. Any of the search-type, analytics-type, forwarder-type, or lightweight-type nodes can be configured as Analytics log collector nodes.

The following table summarizes the components of various Analytics node types. The log collector exporter program and Versa Analytics driver are described later in this document.

See Log Collector Exporter for a description of the log collector exporter and Versa Analytics Driver for a description of the Versa Analytics driver program.

Search-type and analytics-type Analytics nodes are sometimes collectively referred to as datastore nodes.

Log Collector Exporter

The log collector exporter program accepts incoming connections from VOS devices and reads the incoming logs from those connections. It removes the IPFIX overhead from the logs and stores them in a log storage directory on the Analytics log collector node on which the log was received. You can also configure the log collector exporter to forward a copy of the logs, in syslog format, to a remote system.

For the log collector exporter to accept and store incoming logs and export them to a remote system, you configure the following:

• Local collector—A local collector consists of a unique combination of an IP address of an interface on that node and TCP port number. A local collector also includes the name of a log storage directory. Adding a local collector configures the log collector exporter to listen for incoming logs at the given TCP port at the specified IP address, and then to store the logs in the given log storage directory. If the log collector exporter stores the logs in the directory /var/tmp/log, the Versa Analytics driver processes the logs. For information about the Versa Analytics driver, see Versa Analytics Driver.
  
  For example, the following Director screen displays values for a local collector named collector1. This local collector configuration has the log collector exporter listen at TCP port 1234 at IP address 192.168.95.2 and store logs received in the local /var/tmp/log directory. 192.168.95.2 is the IP address of the network interface that connects to the control network.
  
  
   
• Remote collector—You can configure the log collector exporter to send a copy of logs, in syslog format, to another system. To do so, you create a remote collector. A remote collector destination is a system configured to accept logs in syslog format. A remote collector can stream logs on tcp/udp/tls transport. You use a remote collector, for example, when streaming logs to a Kafka cluster.
• Remote collector group—You can configure the log collector exporter to treat a set of remote collectors as a group. You export logs to a remote collector group instead of an individual remote collector. The log collector exporter sends logs to the first reachable remote collector destination in the remote collector group, starting with an indicated primary remote collector.
• Remote collector group list—You can configure the log collector exporter to export logs to multiple remote collector groups. You do this by creating a remote collector group list. The log collector exporter streams logs to the active remote collector of each remote collector group in the list.
• Log exporter rule​​s—To send logs to a remote collector, you create log exporter rules. These rules associate a local collector to a single remote collector, remote collector group, or remote collector group list, so that the local collector automatically streams incoming logs to the associated remote system or systems.

For information about configuring local collectors, remote collectors, remote collector groups, remote collector group lists, and exporter rules, see Configure Log Collectors and Log Exporter Rules.

Versa Analytics Driver

Any logs copied to the /var/tmp/log directory on an Analytics log collector node are processed by the Versa Analytics driver on that node. The driver performs extract, transform, and load (ETL) operations on the logs. Depending on the type of the logs, the driver sends log data to the analytics-type or search-type nodes where it gets inserted into the Analytics datastores, so that it is available to be used by web services. After the Versa Analytic driver has processed a log, it moves the log to a backup subdirectory.

The Versa Analytics driver processes only logs copied to /var/tmp/log. If you configure a local collector to store logs in a directory other than /var/tmp/log, the logs are not processed. This can be useful when you are required to collect logs but do not need log analysis.

After the Versa Analytics driver inserts the processed data as entries into the datastores within the cluster, the datastores store the data for a configurable period of time, called the retention time. When the retention time has expired, the data is removed from the datastore. You control the amount of data stored in the datastores by adjusting the retention time for specific types of entries. To reprocess data after it has been deleted from the datastores, you can restore logs from the archive into the /var/tmp/log directory. For information on configuring retention time, see Analytics Datastore Limits in Versa Analytics Scaling Recommendations.

Log Archival

Local collectors configured on Analytics log collector nodes store logs in a local log storage directory. The default configuration sets up the Linux cron utility to automatically archive logs in the /var/tmp/log backup subdirectories to the /var/tmp/archive directory. If you configure a local collector to store logs in a non-standard directory, one other than /var/tmp/log, the logs are not automatically archived. If required, you can create an additional cron job to archive the logs contained in the non-standard directory. Each Analytics log collector node contains its own log storage directories, log archive directories, and cron jobs. You configure log archival separately on each node. You can NFS-mount any of the directories containing archives to automatically store the archives on a remote system. For more information on managing log archives, see Manage Versa Analytics Log Archives.

LEF Collectors

When you configure LEF, you define collectors for each provider and each tenant organization on a VOS device. The collector defines where to send the logs for an organization, specifying a destination IP address and a TCP or UDP port number. You can configure the collector destination to be one of the following:

• An ADC service tuple (IP address, protocol, and port number) used by an application delivery controller (ADC) on a Versa Controller node
• Third-party system that is configured as a Netflow collector
• Third-part system that is configured as a syslog server
• Local collector configured on an Analytics node

Application Delivery Controllers

On each VOS device, LEF establishes a TCP or UDP connection between the VOS device and the destination system to which it is transferring logs. There is a separate LEF connection for each LEF collector. If a branch VOS device supports many organizations, a large number of LEF connections may be created. If the destination for these LEF connections is a single Analytics node or a limited number of Analytics nodes, the result might be a bottleneck at the Analytics node, because the Analytics node performs many functions in addition to log processing, and the log processing might slow down. To avoid this bottleneck, you can configure an intermediary on a Versa Controller node called an application delivery controller (ADC) service. The ADC service defines an IP address, protocol (TCP or UDP), and port number combination, called an ADC service tuple. The ADC service listens for LEF connections on the ADC service tuple. The ADC then maps the connections, using a load-balancing algorithm, to an Analytics node or nodes or a 3rd-party system. When you use this intermediary, you configure the ADC service as the destination of the LEF connections and the logs. 

The following example shows the LEF collectors for three organizations (Tenant1, Tenant2, and Tenant3) on a branch VOS device. Each of the three LEF collectors uses an ADC service tuple of 10.10.10.1 TCP port 1234, as its destination. The LEF collectors form LEF connections to the ADC service listening at this ADC service tuple, and then the ADC service maps the LEF connections to specific log collector nodes in the Analytics cluster.

The ADC service on the Controller node includes an ADC server for each Analytics log collector, and an ADC server monitor to check the UP or DOWN state of the log collector nodes. The monitor can use either the ICMP or TCP protocol to monitor the nodes. 

If you configure an ADC monitor to use TCP, it creates a temporary connection to the local collector connection pool. By default, the monitor uses the IP address and port number of its associated ADC server to create these connections, and the local collector is set to accept a maximum of 512 connections. To avoid exceeding the maximum, it is recommended that you configure an additional local collector using an alternate port number on each log collector node. Then, use the alternate port number when configuring the monitor. For information about adding a local collector, see Modify or Add a Local Collector in Configure Log Collectors and Log Exporter Rules. For information about configuring an ADC, see Configure an Application Delivery Controller.

The following diagram illustrates an ADC service with an ADC monitor using TCP. The monitor is associated with both ADC Server 1 and ADC Server 2.

For information about configuring an ADC, see Configure an Application Delivery Controller.

Collector Groups

You can configure LEF to treat a set of collectors as a group. Collector groups are configured per organization. The organization can be a tenant organization or a provider organization. When using a collector group, LEF still sends logs to a single destination, but it has multiple destinations to choose from. LEF checks the reachability of the destination for the first log collector in the collector group. If reachable, the VOS device sends the log to that destination. If not reachable, the VOS device checks the reachability of the next log collector's destination. It tries each log collector destination for the group in turn to determine where to send the log.

You can use collector groups to create redundancy in your topology. The example below shows three organizations (Tenant1, Tenant2, and Tenant3), each with two log collectors and a collector group. The first log collector for each collector group uses the ADC service tuple of 10.10.10.1 TCP port 1234 on Controller 1 as its destination, The second uses an ADC service tuple of 10.10.10.2 TCP port 1234 on Controller 2 as its destination. If one of the Controller nodes fails, the VOS device can continue sending logs using the other Controller node.

Collector Group Lists

For Releases 21.2.1 and later.

You can configure LEF to send logs to multiple log collector destinations. You do this by placing two or more collector groups in a collector group list. Collector group lists are defined as part of a LEF profile configuration. Logs are sent to the active collector of each of the collector groups in the list.

LEF Profiles

After configuring LEF collectors and LEF collector groups, you associate them with a LEF profile. A LEF profile configuration includes a LEF template indicating whether to export logs in IPFIX or syslog format and the names of LEF collectors and LEF collector groups. You can define a collector group list as part of the LEF profile configuration and you can assign one LEF profile to be the default. To apply log export functionality to a service or feature on a VOS device, you explicitly associate a LEF profile in the service or feature configuration except those features and services that automatically send logs to the default LEF profile. To send logs for specific types of traffic, such as specific applications, you associate the LEF profile with a traffic monitoring policy. The VOS device sends logs for the feature or service to the active collector of the LEF profile. For more information about configuring logging for specific features and services, see Apply Log Export Functionality.