Headend Deployment Options

You can deploy the typical headend architectures shown above as host OSs on physical hardware, which is called a bare-metal deployment. You can also deploy these headend architectures in a virtualized architecture, for which Versa Networks supports ESXi and KVM, and the AWS, Azure, and Google Cloud Platform public clouds.

Deploying the headend on bare-metal hardware provides better performance and scalability, typically, 20 to 30 percent over a virtualized architecture. However, deploying on physical hardware has hardware dependencies. For example, not all server hardware, such as special RAID controllers, is supported.

For both bare-metal and virtualized deployments, it is important to follow the hardware requirements described in Hardware and Software Requirements for Headend.

Firewall Dependencies for Headend Deployment in a Data Center

When you implement the headend in an existing data center, several protocols and functions are required to connect to the data center. You must follow the requirements for connectivity, reachability, and security. For more information on firewall requirements, see Firewall Requirements.

Best Practices for Hardening the Headend

If you deploy Versa Directors that are exposed to the internet so that external users can access them, you must apply common IT hardening practices. This includes installing Ubuntu OS patches, installing official certificates (Versa Networks software ships with self-signed certificates), and changing the default passwords. Note that you must use the OS patches provided by Versa and not install OS patches directly from Ubuntu.

For more information, see the following articles:

• Perform Manual Hardening for Versa Analytics
• Perform Manual Hardening for Versa Director

Analytics Cluster Redundancy Options

You can configure a primary and secondary cluster in active-active or active-standby mode, or configure the secondary cluster for log redundancy only. We recommend that primary and secondary clusters run in separate datacenters. We do not recommend you separate the nodes a single cluster across multiple regions due to latency and performance issues. Running separate clusters per region helps manage these clusters independently and avoid downtime for storing and retrieving data during maintenance and node failures.

You can configure pairs of Analytics clusters in the following modes:

• Full Redundancy in Active-Active Mode—In this configuration, primary and secondary clusters have identical numbers of analytics, search, and log forwarder nodes. VOS devices are configured to send logs to both these independent clusters simultaneously. For detailed information about configuring clusters in active-active mode, see Configure a Secondary Cluster for Log Collection.
  • Advantages:

    • Highly available. 

    • The Analytics application can access all historic data from the secondary cluster when the primary cluster is down.

  • Limitations: 

    • Running the entire database on a secondary cluster increases costs and operational complexity.

    • There is no cluster-to-cluster synchronization. If the primary cluster goes down, data may be retrieved from secondary cluster. However, when the primary cluster comes back up, there is no automatic synchronization of the data that the secondary cluster received during the downtime.

  • Sample Topology: A sample topology used for both active-active and active-standby modes is illustrated in the figure below. Note that although the topology is identical for the two modes, the software configuration differs for Controllers and VOS devices (branches).
    
    

• Full Redundancy in Active-Standby Mode—In this configuration, primary and secondary clusters have identical number of analytics, search, and log forwarder nodes. VOS devices are configured to send one copy of the logs to Controllers. ADC load balancers on the Controllers are configured to send logs to only the primary cluster when all its Analytics nodes are up. When Analytics nodes are down, the load balancers automatically send logs to the Analytics cluster in the secondary data center. For information about configuring clusters in active-standby mode, see Configure a Secondary Cluster for Log Collection.
   
  • Advantages:
    • The Analytics application can access data from the secondary cluster when the primary cluster is down. In this mode, only data collected during the primary cluster down time is displayed.

  • Limitations:

    • Running the entire database on a secondary cluster increases costs and operational complexity.

    • The secondary cluster does not have data collected before the primary cluster went down. 

    • When primary cluster operation resumes, you must switch between the clusters to get the consolidated data.

  • Topology: The topology for active-standby mode is identical to active-active mode.

• Log Redundancy Mode—In this configuration, the secondary cluster has only the log forwarders running. You configure ADC load balancers on Controllers to automatically forward logs to a secondary datacenter when all the nodes of the primary datacenter are down. The log forwarders in the secondary datacenter receive the logs and archive them. When the primary datacenter comes back up, you can run a cron script on the secondary log forwarders to transfer and populate the logs back to the primary Analytics cluster. Analytics nodes include a cluster synchronization tool to assist in the log transfer. For more information, see Cluster Synchronization Tool in What's New in Release 20.2.

  • Advantages:

    • No need to run the full Analytics cluster in the secondary datacenter and keep it in sync with the primary cluster in anticipation of the datacenter going down.

    • Saves on costs, without losing any data.

  • Limitations:

    • Until the primary cluster comes back up, Versa Analytics reports are not visible.

  • Sample Topology:
    
    

Role of the Log Forwarder

The Versa Operating System^TM (VOS^TM) edge devices generate logs destined for the Analytics database. These logs are carried in an IPFIX template and are delivered to the log forwarder. The log forwarder removes the IPFIX template overhead and stores the logs in clear text on log collector disk. The Analytics database servers can then access the logs from the log forwarder. Optionally, logs can be forwarded in syslog format to third-party log forwarders.

The log forwarder function is part of the Versa Analytics node. It can run on the Versa Analytics node itself or as a standalone log forwarder, where a separate VM or server is dedicated to this role. For larger deployments, especially for deployments with 1000 or more branches, it is recommended that you separate the log forwarder function, because it provides better scalability for the Analytics platform. When the log forwarder runs on a separate device, you typically deploy it south of the Analytics or Search nodes, as shown in the following figure.

Best Practices for Versa Analytics Sizing

The scaling of a Versa Analytics cluster is driven by the logging configuration for the individual features enabled on VOS edge devices. Most features include an optional logging configuration to log events related to a specific rule or profile. The number of features for which you enable logging and the volume of logs that each feature generates has a direct impact on the sizing and scalability of the Versa Analytics cluster.

Versa Analytics has two distinct database personalities, Analytics node and Search node. The Analytics node delivers the data for most of the dashboards shown in the Analytics GUI. These data are aggregated data, and the data aggregation is enabled by default. Therefore, the dashboards are populated without configuration on the VOS edge devices. Analytics node scaling is determined by granularity of the reported data, which is usually set between 5 and 15 minutes. The topology also affects the scaling of the Dashboard feature. A full-mesh topology with multiple underlays generates more SLA logging messages compared to a hub-spoke topology with a single underlay. This is because SLA monitoring between branches in the underlay contributes significantly to the log volume as it reports SLA measurements to the Analytics node.

The primary tasks of the Search node are to store logging and event data, and to drive the log sections and log tables on the Dashboard. These functions are heavily utilized, because every event in the network triggers a log entry on the Search node. In a minimal default configuration, the only information that is logged to the search nodes are the alarm logs. The sizing of the search node depends on the following:

• Number of features enabled with logging
• Logging data volume and log rate
• Retention period of logged data
• Packet capture—You should enable packet capture and traffic monitoring only for traffic related to specific troubleshooting. Packet captures are not stored in the database, but rather as stored as PCAP files. If these files are not processed correctly, they can quickly fill up the disk space.

Feature such as firewall logging and traffic monitor logging (Netflow) have a significant impact on the overall scaling of the Analytics cluster. Therefore, it is recommended that you avoid creating wildcard logging rules. In response to an increases analytics load, you can scale both the Analytics and Search nodes horizontally. Versa Networks has a calculator tool that can help you size an Analytics cluster based on the information described in the next section. 

Branch HA

To implement HA at a customer branch site, the site must have two VOS edge devices. To provide branch redundancy, VOS devices support two modes of operation: active–active mode and active–standby mode.

Active–active mode, which is stateless, is the more commonly used HA mode. It is simple to deploy and provides better performance during standard operations, because both the VOS edge devices are able to process traffic at all times.

For active–standby mode, which is both stateless and stateful, both underlays for each CPE device are physically connected, and so the cross-connect link is not required. You can configure each CPE device as a standalone CPE device and use VRRP on the LAN side.

There are a few drawbacks to active–standby mode:

• Without the cross-connect, each CPE device cannot take advantage of both WAN transports. As a workaround, you can introduce a Layer 2 switch on the WAN side to allow each CPE device to have access to both WAN circuits.
• During a CPE device failover, stateful connections are lost and TCP sessions are re-established. While the user may not notice the re-establishment of the TCP session, the functioning of some devices may be affected.

In stateful active–standby HA mode, only the active CPE device can forward traffic.

Stateful active–standby HA mode maintains a stateful synchronization between the edge devices. You use this mode when state is important, such as for NGFW and CGNAT traffic.

The following table compares the HA modes.

Cross-Connects in an Active–Active HA Topology

The following figure shows an active–active branch HA topology that uses two WAN transport underlays but does not have the dual underlays on the CPE devices.

The cross-connect link is a physical connection between the redundant CPE devices that emulates the missing transport domain in a branch and provides redundancy to the attached clients, as illustrated in the following figure.

For the cross-connect link, you configure VLAN tagging for each WAN transport virtual router (VR) instance and you configure IP addresses configured using Workflow templates. Because the WAN transport VRs are distinct routing instances, they allow for reuse of IP addresses.

When you enable HA, by default, the back-to-back logical interfaces on the cross-connects are assigned IP addresses from the address range 172.16.255.0/30. The primary CPE device is assigned the address 172.16.255.1, and the second CPE device is assigned 172.16.255.2. Which CPE device is the primary or secondary is determined by which device uses the primary device template that is configured in the Director Workflow. The template for the secondary device is automatically generated by the Workflow.

The following screenshot shows a default HA interface configuration for the CPE-1 device. Notice that the assigned IP address is 172.16.255.1.

The following screenshot shows a default HA interface configuration for the CPE-2 device. Here, the assigned IP address is 172.16.255.2.

You configure static routes in the transport VR of each CPE device, and you use the Workflow to configure ICMP monitoring in the transport VR associated with the cross-connect link to direct traffic destined to the WAN connection of the paired CPE device. The following figure show static route ICMP monitoring with HA.

If the cross-connect interface fails over, ir the peer CPE device goes down, or if there is any other IP reachability issue over the cross-connect interface, the static route is withdrawn from the routing table of the corresponding WAN transport VR.

To configure ICMP monitoring on the CPE-1 device that connects to the MPLS transport VR from the CLI:

admin@CPE-1-cli> show configuration | display set | match icmp
set routing-instances INET-Transport-VR routing-options static route 0.0.0.0/0 172.16.255.1 none icmp
set routing-instances INET-Transport-VR routing-options static route 0.0.0.0/0 172.16.255.1 none icmp interval 5
set routing-instances INET-Transport-VR routing-options static route 0.0.0.0/0 172.16.255.1 none icmp threshold 6

To configure ICMP monitoring on the CPE-2 device that connects to the internet transport-VR from the CLI:

admin@CPE-2-cli> show configuration | display set | match icmp
set routing-instances MPLS-Transport-VR routing-options static route 0.0.0.0/0 172.16.255.2 none icmp
set routing-instances MPLS-Transport-VR routing-options static route 0.0.0.0/0 172.16.255.2 none icmp interval 5
set routing-instances MPLS-Transport-VR routing-options static route 0.0.0.0/0 172.16.255.2 none icmp threshold 6

On the LAN side, you use VRRP to elect an active node and a standby node. The logical interface and its virtual IP address is used as the next hop or gateway on the LAN.

DIA in an Active–Active HA Topology

When you configure direct internet access (DIA) in an active–active HA scenario, there are a few things to note.

When you enable DIA, the Director Workflow automates the configuration of BGP peering between the transport VR and the LAN-VR, which is needed to propagate default route.

The ICMP monitoring is between the cross-connect logical interfaces of the CPE-1 internet transport VR and the CPE-2 device and therefore does not protect against internet WAN link failure on the CPE-2 device. This may result in a local internet black hole scenario if the internet WAN link fails on CPE-2, as show in the following figure.

To protect against the local black hole, you must take additional measures, such as monitoring the next next-hop (that is the remote next hop). Doing this may add complexity to the network design, because you may need to use NAT to allow the internet provider WAN interface to reply to ICMP echo requests. The NAT would be necessary because ICMP requests are sourced from the 172.16.255.0/30 prefix range and are not necessarily routed back by the provider router.

The recommended solution is to use dynamic routing over the cross-connect interface between transport VRs to propagate routes and the default route from main transport-VR of each CPE device.

Configure LTE Hot Standby for SD-WAN VPN Traffic

This section discusses two scenarios for using LTE hot standby for SD-WAN VPN (site-to-site) traffic:

• Use LTE as backup circuit on local and remove device
• Avoid LTE for non-business and scavenger traffic

When you configure WAN links, a circuit media attribute is associated with each link. When you configure LTE hot standby for SD-WAN VPN (site-to-site) traffic, the circuit media can be any of the following:

• Cable
• DSL
• Ethernet
• LTE
• T1
• T3

Configure LTE as a Backup for Internet Traffic

This section describes how to configure an LTE link to be a backup for local internet breakout (DIA) traffic. The scenarios in this section have either one wired link and one LTE link, or more than one wired link and one LTE link.

To use LTE as a hot standby for direct internet access (DIA) traffic, in the Workflows template, in the Tunnels tab, do not click the Load Balance option.

The Workflows template automatically creates two BGP sessions over a virtual interface pair that run between the WAN transport VR and the MSP LAN VR. The default route is advertised from each of the WAN transport VRs to the LAN VR. The default route from the BB1 transport VR has a BGP local preference value of 120 by default, and the default route from the LTE transport VR has a BGP local preference value of 119. Based on the local preference values, the LAN VR installs the default route from the BB1 transport VR as the active default route, thus making BB1 WAN link the primary WAN link for local internet-bound traffic.

To load-balance between two wired WAN links for DIA, while having an LTE link as a hot-standby backup, you enable DIA on all internet WAN and check the load balance knob. This configuration sets all WAN links to the same local preference.

Because you select the load balance option, all the default routes from the BB1, BB2, and LTE transport VRs are advertised to the LAN VR with a BGP local preference value of 120. The result is that traffic is load-balanced across all the three WAN links. To have the LTE link be the backup and be active only when the two wired WAN links are down, modify the BGP local preference of the default route advertised by the LTE VR to a value of 119 or lower.

To modify the local preference value:

1. In Appliance view, select the Configuration tab in the top menu bar.
2. Select Networking > Virtual Routers in the left menu bar, and select a virtual router instance. The Virtual Router popup window displays.
3. Select the BGP tab in the left menu bar. The main pane displays a list of the BGP instances that are already configured.
4. Select the BGP instance for the LTE link. The Edit BGP Instance popup window displays.
5. Select the Peer/Group Policy tab. The Add Add Peer/Group Policy popup window displays.
6. Select the Action tab, and change the local preference value to a value of 119 or lower.
   
   

Minimize the Sending of Management Traffic on LTE Links

By default, the VOS device randomly assigns an available link to provide connectivity to the Controller nodes and Versa Networks headend devices. This means that the LTE link could be used to send management traffic. Management traffic is all traffic towards the Controller and headend devices, including LEF logging information, which is sent to the Analytics node, or uploading software image files.

To prevent the unnecessary use of the LTE data plan, you configure management traffic so that the preferred links are the wired WAN links. To do this, you set the management priority of the LTE WAN links to a value that is lower than that of the wired WAN links. For management priorities, a value 0 indicates the highest priority and a value 15 is the lowest.

To set the management traffic priority to minimize the use of LTE links for management traffic:

1. In Appliance view, select the Configuration tab in the top menu bar.
2. Select Services > SD-WAN > Site in the left menu bar. The following screen displays.
   
   
    
3. Click the Edit icon in the main pane. The Edit Site popup window displays.
4. In the WAN Interfaces table, select the LTE interface. The Edit WAN Interfaces popup window displays.
   
   
5. In the Management Traffic > Priority field, enter a value of 15. Note that for management priorities, a value 0 indicates the highest priority and a value 15 is the lowest.
6. Click OK.

Define Per-Interface Encryption Statically

You can statically define the encryption method per WAN interface if the underlay is a private or secured circuit such as an MPLS service provided as part of a service provider Layer 3 VPN service. Traditionally, network administrators have considered these private Layer 3 VPNs as secured and did not typically encrypt data over it. In a similar manner, you can consider the MPLS Layer 3 VPN as secured, and so data can be transported using the clear-text tunnel.

A benefit of clear-text transport it that it does not use IPsec overhead on the platform

To configure static definition per WAN interface:

1. In Appliance view, select the Configuration tab in the top menu bar.
2. Select Services > SD-WAN > Site in the left menu bar.
3. In the Site pane, click the Edit icon.
4. In the WAN Interfaces tab, select a WAN interface. The Edit WAN Interfaces popup window displays.
   
   
5. In the Encryption field, select the desired encryption for the interface:

• Always—Encrypt all traffic.
• Never—Do not encrypt traffic.
• Optional—Encryption is optional.

6. Click OK.

For more information, see Configure Encryption on WAN Interfaces.

Use SD-WAN Policy To Dynamically Define Encryption

In some scenarios, you need to dynamically turn off encryption for some traffic, for example, for traffic that is already encrypted by an application (such as HTTPS or TLS/SSL secured application data) and traffic that is of no interest, from a security point of view, to the enterprise (such as recreational traffic from Facebook and YouTube). To turn off encryption for these types of traffic, you configure an SD-WAN forwarding profile, in which you set the desired encryption, and then you associate the forwarding with an SD-WAN policy that identifies the traffic to match.

To use SD-WAN policy to dynamically define encryption:

1. In Appliance view, select the Configuration tab in the top menu bar.
2. Select Services > SD-WAN > Forwarding Profiles in the left menu bar.
3. Click the Add icon. The Edit Forwarding Profile popup window displays.
   
   
4. In the Encryption field, select the desired encryption.

For more information, see Configure SD-WAN Traffic-Steering.

Spoke-to-Hub Only

In a spoke-to-hub-only topology, the only prefixes advertised, by default, are hub routes, and spokes routes are not re-advertised by the hub branch. You use this topology when spokes do not have to communicate with each other. A good example is a network of ATM cash machines in which devices communicate exclusively with resources in the customer data center. The following figures shows that spoke prefixes are accepted only by the hub and that they are rejected by other spokes based on the BGP community configuration.

The following CLI output shows that the spoke Branch-1 VRF route table contains routes only from the hub:

admin@Branch-1-cli> show route routing-instance Tenant1-LAN-VR
Routes for Routing instance : Tenant1-LAN-VR AFI: ipv4 SAFI: unicast
Codes: E1 - OSPF external type 1, E2 - OSPF external type 2
IA - inter area, iA - intra area,
L1 - IS-IS level-1, L2 - IS-IS level-2
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
RTI - Learnt from another routing-instance
+ - Active Route

Prot   Type   Dest Address/Mask   Next-hop        Age      Interface name
----   ----   -----------------   --------        ---      --------------
conn   N/A   +192.168.1.0/24      0.0.0.0         2d20h04m  vni-0/2.0
local  N/A   +192.168.1.1/32      0.0.0.0         2d20h04m  directly connected
BGP    N/A   +192.168.10.0/24     10.0.40.110     2d20h04m  Indirect
BGP    N/A    192.168.10.0/24     10.0.40.111     2d20h04m  Indirect

The route table on spoke Branch-1 shows only destinations behind hubs, again with Hub-1 being preferred, and the table shows no spokes routes. The following output shows the prefixes advertised by spoke Branch-1:

admin@Branch-1-cli> show route table l3vpn.ipv4.unicast advertising-protocol bgp
Routes for Routing instance : Tenant1-Control-VR AFI: ipv4 SAFI: unicast

Routing entry for 192.168.1.0/24
    Peer Address           : 10.0.40.1
    Route Distinguisher    : 2L:2
    Next-hop               : 10.0.40.101
    VPN Label              : 24704
    Local Preference       : 110
    A S Path                : N/A
    Origin                 : Igp
    MED                    : 0
    Community              : [ 8000:2 8001:110 8002:111 ]
    Extended community     : [ target:2L:2 ]

You use BGP import policies to filter spoke routes. For example, using spoke community 8000:2 filters out spoke routes on hubs in the LAN-VR-Export VR, and these routes are not advertised back to the spokes. Therefore, the hub route tables contains all spoke prefixes:

admin@Hub-1-cli> show route routing-instance Tenant1-LAN-VR
Routes for Routing instance : Tenant1-LAN-VR AFI: ipv4 SAFI: unicast
Codes: E1 - OSPF external type 1, E2 - OSPF external type 2
IA - inter area, iA - intra area,
L1 - IS-IS level-1, L2 - IS-IS level-2
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
RTI - Learnt from another routing-instance
+ - Active Route

Prot   Type  Dest Address/Mask   Next-hop        Age       Interface name
----   ----  -----------------   --------        ---       --------------
BGP    N/A  +192.168.1.0/24      10.0.40.101     00:21:24  Indirect
BGP    N/A  +192.168.2.0/24      10.0.40.102     00:21:27  Indirect
BGP    N/A  +192.168.3.0/24      10.0.40.103     00:21:23  Indirect
BGP    N/A  +192.168.4.0/24      10.0.40.104     00:21:26  Indirect
BGP    N/A   192.168.10.0/24     10.0.40.111     00:36:45  Indirect
conn   N/A  +192.168.10.0/24     0.0.0.0         00:51:13  vni-0/2.0
local  N/A  +192.168.10.1/32     0.0.0.0         00:51:13  directly connected 

On hubs, all spokes prefixes are installed in the corresponding VRF. You can implement redistribution policy on hubs to perform route summarization or to generate default a static route, for instance, to attract traffic from spokes.

Spoke-to-Spoke via Hub (Spoke-to-Spoke Through Another SD-WAN Edge Device)

In a spoke-to-spoke via hub topology (spoke-to-spoke through another SD-WAN edge device), spoke sites are connected to each other through hub site. The data path between two spokes travels through the hub. The following figure illustrates this topology.
 

You can use the spoke-to-spoke through a hub topology when communication between branches is not required, for example, when security and other services are centralized at the hub site or when the cost of ownership of WAN links dictates it.

The following figure shows the method that hub sites use manipulate VRF spoke routes. With this method, you change the route distinguisher on the hub for a set of VRF routes that you are advertising so that the Controller nodes can separate the routes and accept them during BGP route selection. The Controller nodes have the original routes from the spokes and the spoke routes advertised by the hubs, and they reflects them to the branches. You use route-target filtering on the spokes to perform the remainder of the route selection. With route-target filter, you import the hub-advertised spoke routes, and the Controller nodes use these routes to select the hub as the next hop towards the remote sites.

In a spoke-to-spoke via hub topology, the branches communicate only through the hub. The IP prefixes of remote branches always have the hub as the next hop.

SLA monitoring is active only on paths towards hub sites and Controller nodes, and spoke sites are not monitored. This reduces the amount of SLA probe traffic compared to a full-mesh topology and addresses the concerns of scalability in deployments that have a large number of branches.

The following CLI output shows the SLA monitoring view on Branch-1:

admin@Branch-1-cli> show orgs org Tenant1 sd-wan sla-monitor status
                                                  LOCAL  REMOTE
                                                  WAN    WAN
              PATH     FWD    LOCAL     REMOTE    LINK   LINK   ADAPTIVE     DAMP     DAMP   CONN             LAST
SITE NAME     HANDLE   CLASS  WAN LINK  WAN LINK  ID     ID     MONITORING   STATE    FLAPS  STATE  FLAPS  FLAPPED
-----------------------------------------------------------------------------------------------------------------------------------------
Controller-1  69888    fc_nc  MPLS      MPLS      1      1      disable      disable  0      up     1      3d01h39m
              74240    fc_nc  Internet  Internet  2      2      disable      disable  0      up     1      3d01h39m
Hub-1         7213316  fc_ef  MPLS      MPLS      1      1      suspend      disable  0      up     1      2d21h02m
              7217668  fc_ef  Internet  Internet  2      2      suspend      disable  0      up     1      2d21h02m
Hub-2         7278852  fc_ef  MPLS      MPLS      1      1      suspend      disable  0      up     1      2d21h00m
              7283204  fc_ef  Internet  Internet  2      2      suspend      disable  0      up     1      2d21h00m

Because the hub nodes re-advertise the spoke branch prefixes, the spoke branches learn all the spoke prefixes. The next-hop IP address for the spoke branches is the hub's loopback TVI address.

The following output from the Branch-1 VRF routes table shows a deployment with two hubs. The hub that you configure with a higher priority is the one that maintains the active route

admin@Branch-1-cli> show route routing-instance Tenant1-LAN-VR
Routes for Routing instance : Tenant1-LAN-VR AFI: ipv4 SAFI: unicast
Codes: E1 - OSPF external type 1, E2 - OSPF external type 2
IA - inter area, iA - intra area,
L1 - IS-IS level-1, L2 - IS-IS level-2
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
RTI - Learnt from another routing-instance
+ - Active Route

Prot   Type   Dest Address/Mask   Next-hop     Age        Interface name
----   ----   -----------------   --------     ---        --------------
conn   N/A   +192.168.1.0/24      0.0.0.0      1d23h16m   vni-0/2.0
local  N/A   +192.168.1.1/32      0.0.0.0      1d23h16m   directly connected
BGP    N/A   +192.168.2.0/24      10.0.40.110  03:26:28   Indirect
BGP    N/A    192.168.2.0/24      10.0.40.111  03:26:28   Indirect
BGP    N/A   +192.168.3.0/24      10.0.40.110  1d23h16m   Indirect
BGP    N/A    192.168.3.0/24      10.0.40.111  1d23h16m   Indirect
BGP    N/A   +192.168.4.0/24      10.0.40.110  1d23h16m   Indirect
BGP    N/A    192.168.4.0/24      10.0.40.111  1d23h16m   Indirect
BGP    N/A   +192.168.10.0/24     10.0.40.110  1d23h16m   Indirect
BGP    N/A    192.168.10.0/24     10.0.40.111  1d23h16m   Indirect 

The following CLI output shows an example of the spoke routes advertised by Branch-1:

admin@Branch-1-cli> show route table l3vpn.ipv4.unicast advertising-protocol bgp
Routes for Routing instance : Tenant1-Control-VR AFI: ipv4 SAFI: unicast

Routing entry for 192.168.1.0/24
    Peer Address         : 10.0.40.1
    Route Distinguisher  : 2L:2
    Next-hop             : 10.0.40.101
    VPN Label            : 24704
    Local Preference     : 110
    AS Path              : N/A
    Origin               : Igp
    MED                  : 0
    Community            : [ 8000:1 8001:110 8002:111 ]
    Extended community   : [ target:2L:2 ] 

The community string 8000:1 marks the spokes routes so that the BGP import policy on the spokes can identify them, and in this case, it rejects routes starting with this community string. The following CLI output is an example of a spoke route advertised by Hub-1:

admin@Hub-1-cli> show route table l3vpn.ipv4.unicast advertising-protocol bgp

Routing entry for 192.168.2.0/24
    Peer Address         : 10.0.40.1
    Route Distinguisher  : 16002L:110
    Next-hop             : 10.0.40.110
    VPN Label            : 24705
    Local Preference     : 100
    AS Path              : N/A
    Origin               : Incomplete
    MED                  : 0
    Community            : [ 8000:0 8000:1 8001:110 8002:111 8009:8009 ]
    Extended community   : [ target:16002L:0 target:16002L:110 ] 

The community string 8000:0 marks the spokes routes advertised by hubs so that the BGP import policy can identify them, and in this case, it accepts these routes, which have the hub as the next hop.

The community string 8009:8010, which you can see in the diagram above for Hub-1 (192.168.10.0/24), marks the direct LAN route from hubs, which the BGP import policy also accepts.

In this topology, Hub-1 is configured with a higher priority than Hub-2. This configuration explains why there are two entries in the route table for each prefix and why Hub-1 is the preferred next hop. Having two hubs provides redundancy, because Hub-2 is used when Hub-1 is not reachable.

The BGP import policy uses the extended-community attribute to accept routes from the hubs and to set a higher local preference for Hub-1. The extended target string 16002L:110 is derived from site ID 110, which is Hub-1.

Spoke-to-Spoke Direct and Partial Mesh

In a partial-mesh topology, some nodes are directly attached to each other, while other nodes are attached only to one or two nodes. You can select this topology when there are geographically dispersed sites in the same region that you want to communicate directly with each other, and when you want inter-regional traffic to transit through hub branches or when there is a high level of traffic exchanged between specific sites. The following figure illustrates a partial-mesh topology.

 

For a spoke-to-spoke–direct topology, you use spoke groups. Branches within the same spoke group can communicate directly with each other, and they use hubs to reach branches in different spoke groups. In the topology shown above, Branch-1 and Branch-2 communicate with each other directly, but to reach Branch-3 the next hop is Hub-1. The hubs are connected using a full-mesh topology.

It is recommended that you deploy a spoke-to-spoke–direct topology whenever feasible, so that you can use spoke groups to provide redundancy and flexible meshing of branches.

The following CLI output shows the SLA monitoring view on Branch-1:

admin@Branch-1-cli> show orgs org Tenant1 sd-wan sla-monitor status
                                                  LOCAL REMOTE
                                                  WAN   WAN
              PATH     FWD    LOCAL     REMOTE    LINK  LINK     ADAPTIVE    DAMP     DAMP   CONN          LAST
SITE NAME     HANDLE   CLASS  WAN LINK  WAN LINK  ID    ID       MONITORING  STATE    FLAPS  STATE  FLAPS  FLAPPED
------------------------------------------------------------------------------------------------------------------
Branch-2      6689028  fc_ef  MPLS      MPLS      1     1         active     disable  0      up      1     00:04:59
              6693380  fc_ef  Internet  Internet  2     2         active     disable  0      up      1     00:04:59
Controller-1  69888    fc_nc  MPLS      MPLS      1     1         disable    disable  0      up      5     1d22h41m
              74240    fc_nc  Internet  Internet  2     2         disable    disable  0      up      1     1d22h45m
Hub-1         7213316  fc_ef  MPLS      MPLS      1     1         suspend    disable  0      up      7     02:48:59
              7217668  fc_ef  Internet  Internet  2     2         suspend    disable  0      up      3     02:48:58
Hub-2         7278852  fc_ef  MPLS      MPLS      1     1         suspend    disable  0      up      5     02:48:41
              7283204  fc_ef  Internet  Internet  2     2         suspend    disable  0      up      3     02:48:41

SLA monitoring is performed on the paths towards Hub-1, Hub-2, and Branch-2, because these belong to the same spoke group. SLA monitoring is not performed on branches in different spoke groups.

The following CLI output shows the Branch-1 VRF route table:

admin@Branch-1-cli> show route routing-instance Tenant1-LAN-VR
Routes for Routing instance : Tenant1-LAN-VR AFI: ipv4 SAFI: unicast
Codes: E1 - OSPF external type 1, E2 - OSPF external type 2
IA - inter area, iA - intra area,
L1 - IS-IS level-1, L2 - IS-IS level-2
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
RTI - Learnt from another routing-instance
+ - Active Route

Prot   Type  Dest Address/Mask   Next-hop         Age     Interface name
----   ----  -----------------   --------         ---     --------------
conn   N/A  +192.168.1.0/24      0.0.0.0         3d18h59m vni-0/2.0
local  N/A  +192.168.1.1/32      0.0.0.0         3d18h59m directly connected
BGP    N/A  +192.168.2.0/24      10.0.40.102     00:25:29 Indirect
BGP    N/A   192.168.2.0/24      10.0.40.110     00:25:30 Indirect
BGP    N/A   192.168.2.0/24      10.0.40.111     00:25:30 Indirect
BGP    N/A  +192.168.3.0/24      10.0.40.110     00:20:49 Indirect
BGP    N/A   192.168.3.0/24      10.0.40.111     00:20:48 Indirect
BGP    N/A  +192.168.4.0/24      10.0.40.110     00:24:56 Indirect
BGP    N/A   192.168.4.0/24      10.0.40.111     00:24:56 Indirect
BGP    N/A  +192.168.10.0/24     10.0.40.110     03:09:26 Indirect
BGP    N/A   192.168.10.0/24     10.0.40.111     02:55:29 Indirect

A spoke-to-spoke direct topology incorporates additional redundancy. The CLI output above shows three route table entries for 192.168.2.0/24. These routes are advertised by Branch-2 as well as by Hub-1 and Hub-2. When there are underlay connectivity issues between Branch-1 and Branch-2, the hubs provide a redundant path to reach the Branch-2 prefixes.

Routes for branches in different spoke groups are only reachable through the hubs.

To create the expected topology, you use BGP community strings and import policies to accept or reject routes.

The following CLI output shows the prefixes advertised by Branch-2:

admin@Branch-2-cli> show route table l3vpn.ipv4.unicast advertising-protocol bgp

Routes for Routing instance : Internet-Transport-VR AFI: ipv4 SAFI: unicast

Routes for Routing instance : MPLS-Transport-VR AFI: ipv4 SAFI: unicast
Routes for Routing instance : Tenant1-Control-VR AFI: ipv4 SAFI: unicast

Routing entry for 192.168.2.0/24
    Peer Address         : 10.0.40.1
    Route Distinguisher  : 2L:2
    Next-hop             : 10.0.40.102
    VPN Label            : 24704
    Local Preference     : 110
    AS Path              : N/A
    Origin               : Igp
    MED                  : 0
    Community            : [ 8000:1 8001:110 8002:111 8010:1 ]
    Extended community   : [ target:2L:2 ] 

Here, the community string 8010:1 corresponds to spoke Group-1. The community string is a unique BGP community that is associated with the spoke group and that you assign during the Workflow configuration. Based on this community string, the import policy based is pushed to the spokes that are in the same spoke group.

In the topology shown in the figure above, Hub-1 has a higher priority. You configure an import policy that manipulates the Local-Pref attribute (Branch-2 > Hub-1 > Hub-2) to prefer the routes advertised by Hub-1. The following output shows the Local Preference configuration on Branch-1:

admin@Branch-1-cli> show route table l3vpn.ipv4.unicast receive-protocol bgp 192.168.2.0

Routes for Routing instance : Internet-Transport-VR AFI: ipv4 SAFI: unicast

Routes for Routing instance : MPLS-Transport-VR AFI: ipv4 SAFI: unicast

Routes for Routing instance : Tenant1-Control-VR AFI: ipv4 SAFI: unicast

Routing entry for 192.168.2.0/24
    Peer Address         : 10.0.40.1
    Route Distinguisher  : 2L:2
    Next-hop             : 10.0.40.102
    VPN Label            : 24704
    Local Preference     : 110
    AS Path              : N/A
    Origin               : Igp
    MED                  : 0
    Community            : [ 8000:1 8001:110 8002:111 8009:8009 8010:1 ]
    Extended community   : [ target:2L:2 ]
    Preference           : Default

Routing entry for 192.168.2.0/24
    Peer Address         : 10.0.40.1
    Route Distinguisher  : 16002L:110
    Next-hop             : 10.0.40.110
    VPN Label            : 24705
    Local Preference     : 102
    AS Path              : N/A
    Origin               : Incomplete
    MED                  : 0
    Community            : [ 8000:0 8000:1 8001:110 8002:111 8009:8009 8009:8009 8010:1 ]
    Extended community   : [ target:16002L:0 target:16002L:110 ]
    Preference           : Default

Routing entry for 192.168.2.0/24
    Peer Address         : 10.0.40.1
    Route Distinguisher  : 16002L:111
    Next-hop             : 10.0.40.111
    VPN Label            : 24705
    Local Preference     : 101
    AS Path              : N/A
    Origin               : Incomplete
    MED                  : 0
    Community            : [ 8000:0 8000:1 8001:110 8002:111 8009:8009 8009:8009 8010:1 ]
    Extended community   : [ target:16002L:0 target:16002L:111 ]
    Preference           : Default

The following CLI output shows the entries in the Branch-1 route table:

admin@Hub-1-cli> show route routing-instance Tenant1-LAN-VR

Routes for Routing instance : Tenant1-LAN-VR AFI: ipv4 SAFI: unicast

Codes: E1 - OSPF external type 1, E2 - OSPF external type 2
IA - inter area, iA - intra area,
L1 - IS-IS level-1, L2 - IS-IS level-2
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
RTI - Learnt from another routing-instance
+ - Active Route

Prot   Type  Dest Address/Mask  Next-hop        Age        Interface name
----   ----  -----------------  --------        ---        --------------
BGP    N/A  +192.168.1.0/24     10.0.40.101     1d03h20m   Indirect
BGP    N/A   192.168.1.0/24     10.0.40.111     1d03h20m   Indirect
BGP    N/A  +192.168.2.0/24     10.0.40.102     1d03h20m   Indirect
BGP    N/A   192.168.2.0/24     10.0.40.111     1d03h20m   Indirect
BGP    N/A  +192.168.3.0/24     10.0.40.103     1d03h19m   Indirect
BGP    N/A   192.168.3.0/24     10.0.40.111     1d03h19m   Indirect
BGP    N/A  +192.168.4.0/24     10.0.40.104     1d03h19m   Indirect
BGP    N/A   192.168.4.0/24     10.0.40.111     1d03h19m   Indirect
BGP    N/A   192.168.10.0/24    10.0.40.111     1w4d03h    Indirect
conn   N/A  +192.168.10.0/24    0.0.0.0         1w4d03h    vni-0/2.0
local  N/A  +192.168.10.1/32    0.0.0.0         1w4d03h    directly connected 

Spokes routes are directly reachable from hubs, and the backup is to use the remote hubs. For example, in the output above, the prefixes learned from Branch-1 (the first two entries in the roue table) show that the direct route is active.

Spoke-Hub-Hub-Spoke (Regional-Mesh) Topology

In a spoke-hub-hub-spoke (SHHS) topology, also called a regional-mesh topology, you group hub and branch devices by region. Within a particular region, the topology can be anything—full mesh, partial mesh, or hub and spoke—and branches communicate based on the selected topology. When a branch in one region wants to communicate with a branch in another region, the communication transits through regional hubs.

You can select this topology when there is geographical separation and when regional WAN transport networks are available and hubs between regions use the company backbone or high-bandwidth WAN links. The following figures shows two regional networks with different topologies: Region A uses a spoke-to-spoke–direct topology, and Region B uses a spoke-to-spoke topology through a hub. In this example, communication between Branch-1 and Branch-3 uses Hub-1 and Hub-2, but the branches can use any regional hubs in the local region. This topology demonstrates how you can use the SHHS topology in regional networks.

The following CLI output shows the SD-WAN topology view from Branch-1. Because Region A uses a full-mesh topology, the output shows only regional hubs and branches.

admin@Branch-1-cli> show orgs org Tenant1 sd-wan brief
              SITE  MANAGEMENT                           CONNECTIVITY  IS
SITE NAME     ID    IP         TYPE     UP TIME          STATUS        CTRLR
---------------------------------------------------------------------------------------
Branch-1      101   10.0.40.101 local   12d:19h:48m:37s  -             no
Branch-2      102   10.0.40.102 remote  22m:34s          Connected     no
Controller-1  1     10.0.40.1   remote  6d:21h:54m:14s   Connected     yes
Hub-1         110   10.0.40.110 remote  12d:19h:47m:28s  Connected     no

The following CLI output shows the SD-WAN topology view from Branch-3. Branch-3 is the only hub branch, which is normal in a spoke-to-spoke through a hub topology:

admin@Branch-3-cli> show orgs org Tenant1 sd-wan brief
                SITE        MANAGEMENT                          CONNECTIVITY     IS
SITE NAME       ID          IP         TYPE     UP TIME         STATUS           CTRLR
---------------------------------------------------------------------------------------
Branch-3       103         10.0.40.103 local    12d:22h:41m:34s -                 no
Controller-1   1           10.0.40.1   remote   7d:21h:59m:35s  Connected         yes
Hub-2          111         10.0.40.111 remote   3h:14m:8s       Connected

The following CLI output shows the entries in the Branch-1 VRF route table. The two prefixes in both regions are highlighted. The prefix 192.168.2.0/24, for Branch-2, is a direct route towards Branch-2 through Hub-1 for redundancy, and it is the less preferred path. The preferred route is the direct route that uses Branch-2 as the next hop and that is the active route, as indicated by the plus sign (+). The prefix 192.168.3.0/24 is for Branch-3, and the route table entries points to Hub-1.

admin@Branch-1-cli> show route routing-instance Tenant1-LAN-VR

Routes for Routing instance : Tenant1-LAN-VR AFI: ipv4 SAFI: unicast
Codes: E1 - OSPF external type 1, E2 - OSPF external type 2
IA - inter area, iA - intra area,
L1 - IS-IS level-1, L2 - IS-IS level-2
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
RTI - Learnt from another routing-instance
+ - Active Route

Prot     Type    Dest Address/Mask     Next-hop     Age     Interface name
----     ----    -----------------     --------     ---     --------------
conn     N/A     +192.168.1.0/24      0.0.0.0     1w5d22h     vni-0/2.0
local    N/A     +192.168.1.1/32      0.0.0.0     1w5d22h     directly connected
BGP      N/A     +192.168.2.0/24      10.0.40.102 03:15:57    Indirect
BGP      N/A      192.168.2.0/24      10.0.40.110 03:08:36    Indirect
BGP      N/A     +192.168.3.0/24      10.0.40.110 00:01:06    Indirect
BGP      N/A     +192.168.4.0/24      10.0.40.110 00:01:13    Indirect
BGP      N/A     +192.168.10.0/24     10.0.40.110 03:08:35    Indirect
BGP      N/A     +192.168.11.0/24     10.0.40.110 03:08:36    Indirect

The following CLI output shows the entries in the Branch-3 VRF route table. The highlighted output shows two prefixes in both regions and that there is no difference in the treatment between regions. Region B uses a spoke-to-spoke through a hub topology. The prefix 192.168.1.0/24 is behind Branch-1, and the prefix 192.168.4.0/24 prefix is behind Branch-4.

admin@Branch-3-cli> show route routing-instance Tenant1-LAN-VR

Routes for Routing instance : Tenant1-LAN-VR AFI: ipv4 SAFI: unicast

Codes: E1 - OSPF external type 1, E2 - OSPF external type 2
IA - inter area, iA - intra area,
L1 - IS-IS level-1, L2 - IS-IS level-2
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
RTI - Learnt from another routing-instance
+ - Active Route

Prot   Type  Dest Address/Mask   Next-hop         Age     Interface name
----   ----  -----------------   --------         ---     --------------
BGP    N/A  +192.168.1.0/24      10.0.40.111     02:46:42 Indirect
BGP    N/A  +192.168.2.0/24      10.0.40.111     02:46:42 Indirect
conn   N/A  +192.168.3.0/24      0.0.0.0         1w5d22h  vni-0/2.0
local  N/A  +192.168.3.1/32      0.0.0.0         1w5d22h  directly connected
BGP    N/A  +192.168.4.0/24      10.0.40.111     00:04:31 Indirect
BGP    N/A  +192.168.10.0/24     10.0.40.111     02:46:43 Indirect
BGP    N/A  +192.168.11.0/24     10.0.40.111     02:46:43 Indirect

Basic Performance-Based Breakout Configuration

This section describes how to configure performance-based application breakout that is based on active monitoring of an SaaS application, and it build on the application-based breakout scenario. Performance-based breakout leverages the infrastructure of having local and remote DIA, and it requires that you modify the NAT rule. However, for performance-based breakout, you configure the SD-WAN policy differently.
 
For performance-based breakout, you create the following additional building blocks:

• SaaS application monitor on both the local and remote breakout edge devices
• SD-WAN SLA profile for the local edge device
• SD-WAN forwarding profile for the local edge device
• SD-WAN policy for the local edge device

In the design configuration example here, the Salesforce application breaks out using the breakout point that provides the lowest latency to this SaaS. The latency is measured from the point of view of the local edge device. This configuration example has two breakout options:

• Break out Salesforce traffic locally—The latency measurement is from the local edge device to Salesforce only, over the internet.
• Break out Salesforce traffic remotely—The latency measurement is from the local edge device to the remote edge device over the SD-WAN overlay, and then to Salesforce over the internet connection to the remote edge device. This configuration adds together the internet latency and overlay latency.

For both the local and remote edge devices, you configure a SaaS application monitor that checks the performance to a specific SaaS application FQDN. Note that the example here shows only the minimum required configuration for the SaaS application monitor, and it shows the configuration of the local edge device only. You might have different requirements or more complicated use cases. Many configuration options are available, and many alternative scenarios are possible, such as more local internet breakouts, more  remote internet breakout places, and different SLA criteria. For more information, see Configure SaaS Application Monitoring, Configure SLA Profiles for SD-WAN Traffic Steering, and Configure SD-WAN Traffic Steering.

To configure performance-based monitoring for the Salesforce SaaS application:

1. Add a Saas application monitor named salesforce:
   1. In Appliance view, select the Configuration tab in the top menu bar.
   2. Select Networking > SaaS App Monitor in the left menu bar.
   3. Click the  Add icon to add a SaaS application monitor.
      
      
   4. In the Name field, enter "salesforce".
   5. In the FQDN/Host Address table, click the  Add icon to select the FQDN salesforce.com.
   6. Click OK.
2. Configure an SLA profile that references the SaaS application monitor you configured in Step 1:
   1. In Appliance view, select the Configuration tab in the top menu bar.
   2. Select Services   > SD-WAN > SLA Profiles in the left menu bar.
   3. Click the  Add icon to add an SLA profile, or select an existing SLA profile, as shown here. The Create SLA Profile or Edit SLA Profile popup window displays. (Note that the screenshot shows the incorrect SLA profile name. It should be "perf_DIA".)
   4. Select the SaaS App Monitor tab, and in the Monitor Name field, select salesforce. This example uses Low Latency, to have the network to choose the lowest latency or loss. Alternatively, you can set values for the maximum latency or maximum loss, to have all paths be considered as long as the maximum value is not reached.
      
      
   5. Click OK.
3. Configure a forwarding profile that references the SLA profile you configured in Step 2:
   1. In Appliance view, select the Configuration tab in the top menu bar.
   2. Select Services  > SD-WAN > Forwarding Profiles in the left menu bar. The Add Forwarding Profile or Edit Forwarding Profile popup window displays.
   3. Click the  Add icon to add a forwarding profile, or select an existing forwarding profile, as shown here.
   4. Select the General tab, and in the SLA Profile field, select the SLA profile.
      
       
   5.  Select the Next Hop tab and specify the breakout options for the traffic used by this forwarding profile. In this example, there is a local breakout and a remote breakout to the internet:
      1. For the local internet breakout, select the local WAN interface, and for the remote internet breakout, select the remote site where the remote breakout is available.
         
         
          
      2. Select the conditions for the breakout to use. In this example, because we have chosen to have the network automatically chooses the breakout that has the lowest latency, we select Automatic in the Next-Hop Selection Method field.
      3. Set the same priority value (here, 1) for both breakouts.
         
          
   6. Click OK. 
4. Configure an SD-WAN policy that defines the traffic classification criteria for this configuration example.
   1. In Appliance view, select a post-staging template.
   2. Select the Configuration tab in the top menu bar.
   3. Select Configuration > Services  > SD-WAN > Policies > Rules in the left menu bar
   4. Click the  Add icon to add a rule, or select an existing SD-WAN policy rule, as shown here.
   5. Select the Source/Destination tab, and in the Source Zone table, select the source zone. In this example, the source for the Saleforce traffic is a LAN zone.
      
      
   6. Select the Applications tab, and in the SaaS Application Groups table, select Salesforce-Apps. Note that this example works only with the SaaS Application Group classification.
      
      
   7. Select the Enforce tab, and in the Forwarding Profile, select the forwarding profile you configured in Step 3.
      
      
   8. Click OK.

To verify the performance-based breakout configuration: 

1. Verify the operation of the SaaS application monitor at the local and remote breakout points by issuing the show application-monitor local detail CLI command.
   
   The following output shows that the local breakout point measures the latency to Salesforce as 225 ms:

   admin@Site3-cli> show application-monitor local detail
   APPLICATION                                                                                      
   MONITOR                                        LOSS        LATENCY   LOCAL         EXPORT        
   NAME         ROUTING INSTANCE            TYPE  PERCENTAGE  MILLISEC  ORGANIZATION  ORGANIZATION  
   -------------------------------------------------------------------------------------------------
   salesforce   Internet-ISP1-Transport-VR  icmp  0.0         225.53    Enterprise1   Enterprise1 
   

   The output at the remote breakout point measures the latency to Salesforce as 215 ms:

   admin@GW1-cli> show application-monitor local detail
   APPLICATION                                                                                      
   MONITOR                                        LOSS        LATENCY   LOCAL         EXPORT        
   NAME         ROUTING INSTANCE            TYPE  PERCENTAGE  MILLISEC  ORGANIZATION  ORGANIZATION  
   -------------------------------------------------------------------------------------------------
   salesforce   Internet-ISP1-Transport-VR  icmp  0.0         215.75    Enterprise1   Enterprise1  
   

2. For the Salesforce traffic to reach the remote breakout point, you must add latency of the overlay. To verify this latency, issue the following CLI command:

   admin@Site3-cli> show orgs org Enterprise1 sd-wan sla-monitor metrics last-1m GW1
                                                       LOCAL  REMOTE                                                  
                                                       WAN    WAN     TWO    FWD    REV    PDU    FWD    REV                            
   SITE  PATH     FWD    LOCAL WAN      REMOTE WAN     LINK   LINK    WAY    DELAY  DELAY  LOSS   LOSS   LOSS   
   NAME  HANDLE   CLASS  LINK           LINK           ID     ID      DELAY  VAR    VAR    RATIO  RATIO  RATIO  
   ------------------------------------------------------------------------------------------------------------
   GW1   6627844  fc_ef  Internet-ISP1  Internet-ISP1  2      2       171    0      1      0.0    0.0    0.0   
   

3. The command output in Steps 1 and 2 shows that the SD-WAN logic is computing the best path (in this example, based on latency) and will select the local breakout point because it is better. To verify this:

   admin@Site3-cli> show orgs org-services Enterprise1 sd-wan policies Default-Policy rules nexthop application-monitor detail
                                                                         APPLICATION  APPLICATION  APPLICATION  APPLICATION  
                          NEXTHOP   NEXTHOP     NEXTHOP  NEXTHOP  HIT    MONITOR      MONITOR      MONITOR      MONITOR      
   NAME                   PRIORITY  NAME        STATUS   ACTIVE   COUNT  NAME         TYPE         LATENCY      LOSS        
   ------------------------------------------------------------------------------------------------------------------------
   Salesforce_best_DIA    1         Local_WAN   up       yes      127    salesforce   icmp         226.87       0.0          
                          1         remote_WAN  up       -        88     salesforce   icmp         389.7        0.0  
   

4. Verify whether Salesforce sessions are actually going to the local breakout:

   admin@Site3-cli> show orgs org Enterprise1 sessions brief | select application salesforce
   VSN  VSN  SESS                DESTINATION    SOURCE  DESTINATION                                       
   ID   VID  ID    SOURCE IP     IP             PORT    PORT         PROTOCOL  NATTED  SD-WAN  APPLICATION
   ------------------------------------------------------------------------------------------------------
   0    2    955   172.1.123.10  13.110.37.145  40880   443          6         No      No     salesforce 
   0    2    956   172.1.123.10  13.110.37.145  40880   443          6         Yes     No     salesforce 
   0    2    965   172.1.123.10  13.110.37.145  40886   443          6         No      No     salesforce 
   0    2    966   172.1.123.10  13.110.37.145  40886   443          6         Yes     No     salesforce 
   0    2    969   172.1.123.10  13.110.37.145  40890   443          6         No      No     salesforce 
   0    2    970   172.1.123.10  13.110.37.145  40890   443          6         Yes     No     salesforce 
   

Variants of the Basic Performance-Based Breakout Configuration

The configuration example above is based on a performance-based breakout that has a local breakout option and a remote breakout option. You can use the same approach when there are multiple local breakout possibilities, for example, two WAN transports. If these local WANs are provided by different ISPs, each may have different performance characteristics. For example, the WAN transport provided by ISP1 might be closer to the SaaS application server than ISP2. Another common use case is when a cloud security provider provides the breakout. In this case, an IPsec or GRE tunnel directs the traffic to the cloud security provider and you can apply the same logic for all such use cases.

For these scenarios, you configure the next-hop selection method to Automatic in the forwarding profile, as follows:

1. Configure a forwarding profile with two ISPs as next-hop options. For more information, see Configure SD-WAN Traffic-Steering.
2. In Appliance view, select the Configuration tab in the top menu bar.
3. Select Services  > SD-WAN > Forwarding Profiles in the left menu bar.
4. Click the  Add icon, or select an existing forwarding profile, as shown here.
5. Select the Next Hop tab.
6. In the Next-Hop Selection Method, select Automatic.
7. In the Next-Hope Failure Action field, select Wait Recover.
   
   
8. Click OK.

To verify which local breakout is the best performing one, issue the following CLI command:

admin@Site2-cli> show orgs org-services Enterprise1 sd-wan application-metrics brief
             METRIC                                    REMOTE   REMOTE                HIT    
APPLICATION  TYPE    DESTINATION IP     LOCAL CIRCUIT  CIRCUIT  SITE    METRIC  TTL   COUNT  
---------------------------------------------------------------------------------------------
Concur-Apps  VLR     0.0.0.0/0          Internet-ISP1  -        -       11      2996  2      
                                        Internet-ISP2  -        -       59      2997  1      
                     23.38.19.188/32    Internet-ISP1  -        -       7       2822  0      
                                        Internet-ISP2  -        -       4       2996  1      
                     92.123.164.163/32  Internet-ISP1  -        -       11      2996  1      
                                        Internet-ISP2  -        -       61      2997  5    

The CLI output above shows the different metrics for the local circuits toward ISP1 and ISP2. For this test, the internet circuit to ISP1 is impaired by high loss and high latency, resulting in a bad link score. Therefore, for the last entry in the output, the metric for ISP1 remains low, while the metric for ISP2 increases. The result is that ISP2 gets more sessions (a hit count of 5 compared to a hit count of 1 for ISP1).

Another example is a performance-based breakout to the best performing external cloud security provider. In this use case, you set the forwarding profile next hop to the IP address of the remote tunnel IP endpoint. For all scenarios, it is recommended that you add a monitor to the next hop. In this case, the monitor is verifying the IP endpoint of the tunnel. If the IP endpoint is not reachable, this next hop is not considered.

To configure the forwarding profile next hop to the IP address of the remote tunnel IP endpoint:

1. Configure a forwarding profile with two ISPs as next hop options. For more information, see Configure SD-WAN Traffic-Steering.
2. In Appliance view, select the Configuration tab in the top menu bar.
3. Select Services  > SD-WAN > Forwarding Profiles in the left menu bar.
4. Click the  Add icon, or select an existing forwarding profile, as shown here. The Add Forwarding Profile or Edit Forwarding Profile popup window displays.
5. Select the Next Hop tab.
6. In the Next-Hop Select Method field, select Automatic.
7. In the Next-Hop Failure Action field, select Wait Recover.
8. Click the  Add icon, and in the Add Next-Hop Priorities popup window, specify the IP address for next hop to the remote tunnel IP end point. GRE and IPsec are the most common tunnel options. For more information, see Configure Site-to-Site Tunnels and Configure Interfaces.
   
   
9. In the Monitor Field, select a monitor to use for the tunnel.
10. Click OK twice.

After you configure the tunnels, add a static route to route traffic to them. It is recommended that you monitor the tunnel endpoint is monitored so that if the tunnel fails, the static route is withdrawn. You do this by configuring an IP SLA monitor, as follows:

1. In Appliance view, select the Configuration tab in the top menu bar.
2. Select Networking  > IP SLA > Monitor in the left menu bar. The main pane displays the IP SLA monitor objects that are already configured. For more information, see Configure IP SLA Monitor Objects.
3. Click the  Add icon, or select an existing IP SLA monitor, as shown below. The Add IP SLA Monitor or Edit IP SLA Monitor popup window displays.
   
   
4. Associate this IP SLA monitor with the static route. If you want an active/standby connection to the cloud security proxy, the preference or metric can influence which route to which tunnel is active. For more information, see Associate an IP SLA Monitor Object with a Static Route.
   1. In Appliance view, select the Configuration tab in the top menu bar.
   2. Select Networking  > Virtual Routers in the left menu bar
   3. Select a virtual router instance.
   4. In the Edit VR popup window, select the Static Routing tab.
   5. Select the IPv4/IPv6 Unicast tab in the horizontal menu bar.
   6. Click the  Add icon to add a static route to associate with the monitor object. If you are adding an IP SLA monitor to an existing static route, click the address of the static route in the Destination column.
      
      

Best Practices for Performance-Based Breakout

SaaS application monitoring has many configuration options. Your SaaS application may not work for the basic measurement based on ICMP, as shown in this example, and for best results you may to ture the SaaS application monitor.

DNS proxy and web proxy may interfere with the functioning of the SaaS application, so check the following sections to understand the roles of these two types of proxies. If the functioning is affected, enhance the configuration with DNS proxy (always recommended) and web proxy chaining (required when you use a centralized web proxy).

Note that the you can optimize the SaaS applications listed in the SaaS application group list. Applications that are not listed cannot leverage the performance-based breakout functionality.

DNS Proxy Configuration

The DNS proxy configuration described here is an extension to the use case described in Performance-Based SaaS Optimization.

You configure the DNS proxy only on the local DIA edge device. No additional configuration is required on the central DIA edge device.

For performance-based SaaS optimization, you must use the DNS server of the local ISP or a global DNS provider, such as a Google DNS server. This configuration example uses the Google DNS server 8.8.4.4 for Salesforce.com.

To configure the DNS proxy:

1. Configure a DNS proxy profile. For more information, see Configure DNS Proxy Profiles.
   1. In Appliance view, select the Configuration tab in the top menu bar.
   2. Select Networking  > DNS > Proxy Profiles in the left menu bar.
   3. Click the  Add icon, or select an existing proxy profile, as shown here. The Add Proxy Profile or Edit Proxy Profile popup window displays.
   4. In the Name file, enter a name for the DNS proxy profile. Here, the name is Google-DNS.
      
      
       
   5. Select the Resolver tab to configure a DNS resolver.
   6. Click the  Add icon, or select an existing DNS resolver, as shown here. The Add Resolver or Edit Resolver popup window displays.
      
      
      
      It is recommended that you monitor the reachability of the configured DNS server. If the DNS server is not reachable, the monitor fails, the DNS proxy stops forwarding DNS queries to that DNS server, and instead, the DNS server configured one the client is used. Because the DNS server is dynamically assigned, for this example, only the specification of the DNS server is relevant. This example chooses to monitor the reachability of the DNS server by applying an IP SLA monitoring object. If the DNS server is not reachable, this DNS profile is not used, thus& avoiding the potential blackholing of DNS queries.
2. Configure DNS proxy policy rule to specify matching conditions, that is, where the DNS query originates from and to which FQDN it applies). For more information, see Configure DNS Redirection Rules.
   1. In Appliance view, select the Configuration tab in the top menu bar.
   2. Select Networking  > DNS > Policies in the left menu bar.
   3. Select the Rules tab in the horizontal menu.
      
      
       
   4. Click the  Add icon. The Add Redirection Rules popup window displays.
   5. Select the Source/Destination tab, and in the Source Zone table, select the source zone.
      
      
   6. Select the DNS Headers Match tab and enter the following information:
      
      
   7. In the Opcode field, select the value Query.
   8. In the Query Value field, select the FQDN regex notation. The exact value of regex notation is important. Because we want the query for be for all traffic designated to the Salesforce.com domain, use the regex notation ".*salesforce.com".
   9. Select the Proxy setting tab.
      
      
   10. In the Actions field, click Proxy Setting.
   11. In the Proxy Profile field, select the proxy profile you created in Step 1, here, Google-DNS.
   12. Click Apply Policy-Based Forwarding. This setting enables the looking up of SD-WAN policies to ensure that the SaaS application breaks out to the best performing DIA. If you do not select this option, the DNS server 8.8.4.4 is used, but the egress interface is now DTVI-0/51, which is the overlay tunnel to the gateway. You can also verify whether the proxy, here, follows the routing table to get to the specified DNS server. Because the SD-WAN policy may dynamically decide whether the break out is local or central, the DNS proxy must follow that decision.
   13. In the LEF Profile field, select the log profile to use, which is useful when you want to verify DNS queries.
3. Click OK.

To verify the DNS proxy configuration:

1. To verify that the current path to Salesforce.com is the best performing path, issue the following CLI command:

   admin@Site3-cli> show orgs org-services Enterprise1 sd-wan policies Default-Policy rules nexthop application-monitor detail
                                                                         APPLICATION  APPLICATION  APPLICATION  APPL             
                          NEXTHOP   NEXTHOP     NEXTHOP  NEXTHOP  HIT    MONITOR      MONITOR      MONITOR      MONITOR      
   NAME                   PRIORITY  NAME        STATUS   ACTIVE   COUNT  NAME         TYPE         LATENCY      LOSS         
   --------------------------------------------------------------------------------------------------------------------
   Salesforce_best_DIA    1         Local_WAN   up       yes      79     salesforce   icmp         326.42       0.0
                          1         remote_WAN  up       -        108    salesforce   icmp         398.2        0.0
   

The output shows that the path to Salesforce.com from the local breakout has the lowest latency (326 ms compared to 398 ms) and is the active path.

2. To verify that the configuration is functioning correctly and that the DNS query is sent to the Google DNS server from the local DIA:
   1. Select Analytics > Home > Logs > DNS Proxy.
      
       
   2. Zoom in to view details:
      
      
      
      The output above shows that for salesforce.com, the DNS server 8.8.4.4 is used (the default DNS for the client is an enterprise internal DNS) and that the egress interface is TVI-0/603, which is the internal interface for DIA. (Check the dnspDomain, destPort, destAddr, and egrlf fields in the output.)You can also verify the DIA interface by issuing the show interface brief CLI command.
3. To whether the local DIA might be impaired and so the central DIA is preferred, issue the following CLI command:

   admin@Site3-cli> show orgs org-services Enterprise1 sd-wan policies Default-Policy rules nexthop application-monitor detail
                                                                         APPLICATION  APPLICATION  APPLICATION  APPL
                          NEXTHOP   NEXTHOP     NEXTHOP  NEXTHOP  HIT    MONITOR      MONITOR      MONITOR      MONITOR    
   NAME                   PRIORITY  NAME        STATUS   ACTIVE   COUNT  NAME         TYPE         LATENCY      LOSS
   -------------------------------------------------------------------------------------------------------------------
   Salesforce_best_DIA    1         Local_WAN   up       -        191    salesforce   icmp         329.41       0.0
                          1         remote_WAN  up       yes      108    salesforce   icmp         188.72       0.0 
   

   
   In Analytics, you can display the details:
   
   This verification shows that the DNS server 8.8.4.4 is used, but the egress interface is now DTVI-0/51, which is the overlay tunnel to the gateway. You can also verify the DIA interface by issuing the show interface dynamic tunnels CLI command.

Best Practices for DNS Proxy

The configuration example here describes how the SD-WAN network can dynamically intercept the client DNS query and forwards the query to the DIA point that provides the best SaaS performance. The example shows the minimum configuration required for this use case to work.

One can image more advanced uses of DNS proxy. For example, enterprises often use their own internal DNS server. You might configure a global DNS external service, such as Google DNS, and an internal DNS for the IP endpoints, but you prefer local breakout to a global external DNS. For this type of scenarios, a DNS proxy provides a solution, intercepting DNS traffic for the internal domain (based on the internal FQDN) and having the rest of the traffic be served by the global external DNS.

Use BGP to Exchange Routes with the MPLS Provider on the MPLS WAN Interface

Using BGP to exchange routes with the MPLS provider on the MPLS WAN interface is the most common method for DIY enterprise SD-WAN deployments. For this method, you configure a BGP peering session on the MPLS-Transport-VR on the gateway to exchange routes from the MPLS Provider to the SD-WAN network. You can use Director Workflows to automate this configuration.

The advantage of this approach is that you need only a single IP interface to connect to the MPLS provider, and this is the interface over which you establish the BGP peering session. Having a singe IP interface is commonly seen in enterprise network designs in which a single MPLS provider offers MPLS VPN underlay services. This MPLS VPN network has connections to the legacy branches and provides MPLS underlay connectivity to SD-WAN branches. Whether an SD-WAN branch has an MPLS or internet underlay, the SD-WAN VPN network uses only the WAN IP addresses to establish connectivity with other SD-WAN branches and the gateway. The SD-WAN branch routes are exchanged over an encrypted overlay.

The following figure shows how the gateway is configured internally. A virtual TVI interface pair is created (using a Director Workflow) in the MPLS-Transport-VR and Tenant-LAN-VR. Over this virtual interface, an EBGP neighbor session is established with AS numbers 64513 and 64514. When the route is advertised to the SD-WAN network, the default BGP AS number of the SD-WAN overlay, 64512, is appended to the BGP AS path list. If the SD-WAN route advertised by one gateway is inadvertently learned by another gateway through the MPLS underlay provider, the route is automatically blocked as a result of the AS path check.

To configure an EBGP session on the MPLS WAN link to the MPLS PE router:

1. In Director view, select the Workflows tab in the top menu bar.
2. Select Template > Templates in the left menu bar.
3. Click the Add icon, or select an existing template. The Create/Edit Template window displays. For more information, see Create Device Templates.
   
   
4. Select the Routing tab, and configure the MPLS WAN link.
5. Select the Tunnels tab, and create the BGP session between the MPLS-Transport-VRE and Tenant-LAN-VR over a virtual interface pair (TVI interface) as shown below.
   
   
6. Click OK.

Establish a BGP Session with the MPLS Provider on a LAN Interface

In a scenario in which you establish a BGP session with the MPLS provider on a LAN interface, you use Director Workflows to configure a BGP peering session to exchange routes with the MPLS provider through an IP interface in the Tenant-LAN-VR, as illustrated below.

Enterprises hosting their own gateways need two interfaces or logical subinterfaces from the MPLS provider, one that terminates on the MPLS-Transport-VR and that is the underlay to connect to other SD-WAN-enabled sites on MPLS, and a second that terminates in the Tenant-LAN-VR and that is used to connect to non-SD-WAN sites on the MPLS network.

This configuration is commonly seen when a service provider offers a dedicated gateway service their customers and is also the MPLS service provider for their customers. In this scenario, the MPLS provider has one interface or subinterface in the customer VRF that terminates in the customer's Tenant-LAN-VR. The interface that connects to the WAN-Transport-VR is used only to connect to SD-WAN–enabled sites. For any two sites to connect over SD-WAN, only the branch WAN IP connectivity is required. Hence, the peering with the provider on the MPLS-Transport-VR can be a common VRF into which the branch WAN IP addresses of customer VRFs are placed, but the individual customer VRFs have only the WAN IP address of the gateway branch. The EBGP peering session in the Tenant-LAN-VR is with an interface on the provider's MPLS PE router that is part of the customer's VRF in which the legacy branch routes are placed. This design allows each site in the customers SD-WAN network to set up an SLA to the gateway using its underlay IP address, but sites in two different customer networks cannot establish an SLA or any communication path between them. To achieve this, the customer SD-WAN branch MPLS interface WAN IP addresses must be unique or source-NATed.

To configure this type of gateway:

1. In Director view, select the Workflows tab in the top menu bar.
2. Select Template > Templates in the left menu bar.
3. Click the Add icon to create a new template, or select an existing template, as shown here. For more information, see Create Device Templates.
4. Select the Routing tab, and configure the tenant LANs, as shown below:
   
   
5. Click OK.

You can modify the routing policy to filter sent and received routes in the device template or directly for the VOS device. In the configuration, in the Appliance view, select Networking in the left menu bar, select the MPLS-Transport-VR, and select BGP. Then select the BGP instance ID and select Peer/Group Policy. For more information, see Configure Peer and Peer Group Policy.

Packet Replication

Packet replication is the most effective option for applications that are sensitive to latency and packet loss. Packet replication, which is primarily used for real-time applications, takes an identified type of application traffic and sends a copy of each packet across multiple paths. This mechanism prevents inherent latency anomalies and packet losses from negatively impacting the applications.

One drawback of packet replication is that it can have a significant impact on the bandwidth utilization of WAN circuits. Therefore, you should use this feature mainly for low-volume UDP traffic that is sensitive to packet loss, such as VoIP traffic. Also, it is recommended that you configure both Start When SLA Violated and Stop When options in the forwarding profile that you associate with packet replication.

Packet replication works as follows: If a transmitted packet is lost in transit, a copy of the packet is forwarded to the receiver. The receiving branch discards the copies of the packet and forwards only one packet to the receiver. You must enable reordering on all branches to ensure that the receiving branch reorders packets before forwarding them to the receiver.

Packet replication is suitable for branches with multiple SD-WAN paths between branches, while FEC is also effective on a branch that has only a single access links. Packet replication works well when the amount of critical traffic that is being duplicated across the networks is far less than the capacity of the network.

By default, packet replication is disabled. To enable replication, you configure it when you configure an SD-WAN traffic-steering forwarding profile:

1. In Director view:
   1. Select the Configuration tab in the top menu bar.
   2. Select Templates > Device Templates in the horizontal menu bar.
   3. Select an organization in the left menu bar.
   4. Select a template in the main pane. The view changes to Appliance view.
2. Select the Configuration tab in the top menu bar.
3. Select Services > SD-WAN > Forwarding Profiles in the left menu bar.
4. Click the Add icon. The Add Forwarding Profile popup window displays. Enter the following information in the Replication group of fields. For more information, see Configure Replication for SD-WAN Traffic Steering.
   
   
5. To enable packet replication, click Enable. By default, replication is disabled.
6. In the Replication Factor field, enter the number of egress packets to send for each ingress packet. For example, if you configure a replication factor of 2, for each ingress packet, two egress packets (the original and one copy) are forwarded to the next hop. The default replication factor is 2. If there are more than two paths between branches, you can increase the value.
7. In the Start When field, select SLA Violation, to enable packet replication only when all available paths have violated the SLA. Otherwise, replication is enabled on all flows that use the forwarding profile. It is recommended that when you enable packet replication, you always select the SLA Violation option.
8. Optionally, click Stop When to set a circuit utilization threshold value to use to stop packet replication and thus prevent oversubscription of links. When the utilization of any of the links used for replication reaches the percentage set in the Circuit Utilization option, replication stops, and the flow passes using only one available link, as was the case before you enabled replication.
9. When you click Stop When, in the Circuit Utilization field, enter the circuit utilization threshold at which replication stops automatically if all the device's WAN circuits exceed this threshold. Specify this as a percentage of the total circuit bandwidth. This threshold applies to all circuits, even those that are set as to be avoided.
10. Click OK.

The following are best practices for packet replication:

• Use packet replication only for specific traffic, such as VoIP. Avoid enabling replication on wildcard traffic.
• Enable replication only when Start When is SLA Violated, and always set the Stop When and Circuit Utilization. For the circuit utilization calculations to be accurate, ensure that the bandwidth values are set correctly on the interface.
• A replication factor of 2 is adequate for most cases. However, you can increase the value to match the available paths for the flows that are protected. For example, four paths can benefit from a replication factor of 3.
• Bandwidth utilization increases by 100% for each unit increment in the replication factor. That is, a replication factor of 2 equates to a 100% increase in bandwidth, a factor of 3 equates to a 200% increase, and so on.
• Enable replication for real-time traffic and other business-critical traffic for which jitter and loss are important.

Forward Error Correction

FEC is a mechanism to correct bit errors at the physical layer. While FEC has traditionally been used for this purpose, it has been adapted so that it can be used to recover from packet loss at the network level.

Packet-level FEC works by adding an additional loss recovery packet for every nth packet that is sent. This additional loss recovery packet enables a VOS edge device to reconstitute lost packets at the far end of a WAN link, before the packets are delivered to TCP or other transport layers. This mechanism avoids transport layer retransmission and, in the case of TCP, prevents the TCP congestion avoidance mechanism from lowering the throughput available to the application. For the modest overhead of an additional loss recovery packet, FEC reduces packet loss dramatically, enabling applications to benefit from the maximum throughput that the WAN link can support.

You can turn on FEC along with replication at the sites that have multiple paths, to provide maximum protection and correction. You can also use FEC to recover packets independently when replication might not be useful, for example, at sites with a single path for transport traffic.

To configure FEC:

1. In Appliance view, select the Configuration tab in the top menu bar.
2. Select Services > SD-WAN > Forwarding Profiles in the left menu bar.
3. Click the Add icon. The Add Forwarding Profile popup window displays. Enter the following information in the FEC tab. For more information, see Configure SD-WAN Traffic-Steering.
   
   
4. For FEC on the sender, configure the following options in the Sender group of fields:
   1. To enable FEC, click Enable.
   2. In the Duplicate FEC Packet field, if extra protection is required, additional FEC parity data packet can be sent. The FEC parity packets could be transmitted over the same WAN link or over another available WAN circuit. By default, duplicate FEC packets are not sent.
   3. In the FEC Packet field, select the circuit on which to send the FEC packets. The default if the alternate circuit. If you configure Duplicate FEC Packets, select the same circuit that you use for the duplicate packets.
   4. In the Maximum FEC Packet Size field, enter the maximum size of the data in the packet to protect. For example, voice packets are typically less than 100 bytes, so set the maximum size to 100bytes to protect the traffic.
   5. In the Number of Packets per FEC field, nter the number of data packets after which an FECe packet is generated and sent to the peer branch. The default value is 4, which means the system generates FEC parity data after each four packets of actual data transmitted according to this forwarding profile.
   6. In the Start When field, if you do not select any value, the forwarding profile always protects traffic for the defined rule. If you select SLA Violation, the FEC is enabled only when all available paths have a violated SLA.
   7. Click the Stop When field to optionally prevent oversubscription of the links. When the utilization of the link over the FEC data is sent reaches the percentage configured in the Circuit Utilization field, the FEC operations stop.
5. For FEC on the receiver, configure the following options in the Receiver group of fields:
   1. Click the Recovery field to enable packet recovery after the receiver receives FEC packets. By default, receiver packet recovery is enabled.
   2. Click the Preserve Order to reorder out-of-order packets and forward them in their original order. By default, reordering is enabled. Note that this option is independent from Reorder option in the Forwarding Profile.
6. Click OK.

There are multiple ways of configuring Layer 3 FEC, and the following are some recommendations:

• WAN circuit utilization—When bandwidth is at a premium, FEC is more efficient than packet replication, and it generates only one parity packet for a specified number of data-carrying packets (range 1 through 32, default 4). The generated FEC parity packet can recover a packet on the peer branch only if one packet is lost in the specified number of packets per FEC.
• If packet corruption on a path has a specific pattern (corrupting the nth recovery packet and its replica), it is recommended that you replicate the FEC packet on an alternate circuit so that non-corrupted FEC packets can reach the remote site.
• Small versus large packets—Layer 3 FEC works well on flows of small packets, such as voice or point-of-sale transactions. Applications that have large packets, such as video and file transfers, are usually not business-critical applications, and performing forward error correction on fragmented packets is difficult. It is recommended that you use FEC for packets less than 500 bytes.
• LTE and FEC—In most cases, you should not use FEC on LTE interfaces, because FEC increases the amount of traffic on a network that is already limited in capacity. When packets are dropped in a wireless network, it is usually many packets in a row, and FEC is not useful in such a scenario.
• Adaptive codecs versus FEC—Many of the latest voice and video codecs support FEC within the application codec. Usually, these adaptive codecs are more efficient than Layer 3 FEC. For these cases, you must perform tests to find out whether the Layer 3 FEC feature provides any advantage if you use it on top of the voice or video codec FEC feature.
• Versa implements single-dimensional FEC, which is most effective against bit error and single packet loss within a protected block. FEC improves the probability that a stream arrives intact at a receiver. You can calculate the probability with the following formula:

P(FEC = (1 – p_x)^w+1 = ((w + 1)(1 – p_x)^w)p_x

Where:

• P(FEC) is the FEC recovery probability, as a percentage
• p_x is the packet loss, as a percentage
• w is the number of packets per parity (npp)

This formula indicates that the FEC default settings of 4 (npp) is effective only up about a 5 percent packet loss, as illustrated with the following graph. You should consider replication for higher loss percentages.

• FEC performance increases when the npp is closer to 1. There is, however, a 25 percent increase in the bandwidth utilization value for every unit decrease in x – 1, where x = npp. For example, if the default npp is 4, reducing it to 3 increases bandwidth utilization by 25 percent. When npp = 1, bandwidth utilization increases by 100 percent.

Floating Static Routes

Static routes can be conditionally withdrawn based on the state of another target IP address, as illustrated in the following figure.

The following output shows a scenario in which the next-hop IP address is monitored on the primary link, while the second static route is configured with a lower preference. The output shows that the primary static route is withdrawn because the monitor probe fails. Monitor objects can be based DNS, ICMP, or TCP.

admin@Router-cli> show route routing-instance Router-DC | grep 200.200.0.1/32
static N/A +200.200.0.1/32 192.168.105.2 00:01:42 vni-0/2.0
static N/A 200.200.0.1/32 192.168.105.3 00:04:00 vni-0/2.0
[ok][2020-06-15 07:46:04]

admin@Router-cli> show monitor brief
NAME             ADDRESS         VRF        TENANT        STATE  TYPE
---------------------------------------------------------------------
Monitor-Network1 192.168.105.2   Router-DC  Provider-Org  Up     icmp

[ok][2020-06-15 07:46:07]
admin@Router-cli> show monitor brief
NAME              ADDRESS        VRF        TENANT        STATE  TYPE
----------------------------------------------------------------------
Monitor-Network1  192.168.105.2  Router-DC  Provider-Org  Down   icmp

[ok][2020-06-15 07:46:09]
admin@Router-cli> show route routing-instance Router-DC | grep 200.200.0.1/32
static N/A +200.200.0.1/32 192.168.105.3 00:04:09 vni-0/2.0

Route Leaking

The static route next hop can be resolved to the same routing instance or to a different routing instance, as illustrated by the following figure, which shows that for traffic originating from the LAN, the next hop is resolved to the transport VR.

The following snippet shows an example of how to configure route leaking with static routes:

admin@PocBr1-cli> show configuration routing-instances PoC-Org-LAN-VR routing-options
static {
    route {
        5.5.5.5/32 MPLS-Transport-VR none {
            preference 1;
        }
    }
}

The route table shows the following static route:

admin@PocBr1-cli> show route routing-instance PoC-Org-LAN-VR | grep 5.5.5.5/32
static N/A +5.5.5.5/32 0.0.0.0 00:20:45 Indirect

The following sample output uses the tcpdump command to capture the ping traffic on the data center router:

admin@Router-cli> tcpdump vni-0/1 filter "host 5.5.5.5"
Starting capture on vni-0/1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on _vni_0_1, link-type EN10MB (Ethernet), capture size 262144 bytes
03:23:06.084351 52:54:00:f1:38:4c > 52:54:00:e7:c3:9c, ethertype IPv4 (0x0800), length 98: 169.254.0.202 > 5.5.5.5: ICMP echo request, id 38, seq 75, length 64
03:23:07.088364 52:54:00:f1:38:4c > 52:54:00:e7:c3:9c, ethertype IPv4 (0x0800), length 98: 169.254.0.202 > 5.5.5.5: ICMP echo request, id 38, seq 76, length 64
03:23:08.088510 52:54:00:f1:38:4c > 52:54:00:e7:c3:9c, ethertype IPv4 (0x0800), length 98: 169.254.0.202 > 5.5.5.5: ICMP echo request, id 38, seq 77, length 64
03:23:09.092360 52:54:00:f1:38:4c > 52:54:00:e7:c3:9c, ethertype IPv4 (0x0800), length 98: 169.254.0.202 > 5.5.5.5: ICMP echo request, id 38, seq 78, length 64
03:23:10.096368 52:54:00:f1:38:4c > 52:54:00:e7:c3:9c, ethertype IPv4 (0x0800), length 98: 169.254.0.202 > 5.5.5.5: ICMP echo request, id 38, seq 79, length 64
^C

Fast Convergence on the SD-WAN Side

For dual-homed deployments, you should configure a unique route distinguisher (RD) for each CPE LAN-VRF, to allow the dual-homed CPEs to advertise unique information for the same local prefix. In this way, the BGP best-path computation done on the Controller nodes reflects the prefixes from both CPEs in the dual-homed architecture. Having both sets of prefixes improves convergence, because it is not necessary to resend a BGP update when a failure occurs. It also helps in troubleshooting, because having a unique route distinguisher makes it much easier to map to the CPE from which the prefix originated. Director Workflows typically configure the same route distinguisher for all nodes. However, note that the Workflows automatically set unique route distinguishers when you configure an HA pair.

To configure route distinguishers:

1. In the Director view:
   1. Select the Configuration tab in the top menu bar.
   2. Select Devices > Devices in the horizontal menu bar.
   3. Select an organization in the left menu bar.
   4. Select a branch in the main pane. The view changes to Appliance view.
2. Select the Configuration tab in the top menu bar.
3. Select Networking > Virtual Routers in the left menu bar.
4. Select a LAN-VR in the main pane. The Edit LAN-VR popup window displays, and the Virtual Router Details tab is selected.
5. In the Route Distinguisher field, enter an RD value for the first CPE.
   
   
6. In the Route Distinguisher field, enter an RD value for the second CPE.
   
   

The following output shows that the Controller nodes reflect the route information that comes from both CPEs.

admin@Controller-RR-cli> show route table l3vpn.ipv4.unicast advertising-protocol bgp neighbor-address 10.0.160.101
...
Routing entry for 192.168.4.0/24
    Peer Address : 10.0.160.101
    Route Distinguisher: 3L:3
    Next-hop : 10.0.160.103
    VPN Label : 24704
    Local Preference : 110
    AS Path : N/A
    Origin : Igp
    MED : 0
    Community : [ N/A ]
    Extended community : [ target:3L:3 ]

Routing entry for 192.168.4.0/24
    Peer Address : 10.0.160.101
    Route Distinguisher: 3L:4
    Next-hop : 10.0.160.104
    VPN Label : 24704
    Local Preference : 109
    AS Path : N/A
    Origin : Igp
    MED : 0
    Community : [ N/A ]
    Extended community : [ target:3L:3 ]

As a result, on the other sites, two paths are available and installed in the route table. For example:

admin@Branch1-cli> show route routing-instance Tenant1-LAN-VR 192.168.4.0

Routes for Routing instance : Tenant1-LAN-VR AFI: ipv4 SAFI: unicast
[+] - Active Route

Routing entry for 192.168.4.0 (mask 255.255.255.0) [+]
Known via 'BGP', distance 200,
    Redistributing via BGP
    Last update from 10.0.160.103 00:01:17 ago
Routing Descriptor Blocks:
* 10.0.160.103 , via Indirect 00:01:17 ago

Routing entry for 192.168.4.0 (mask 255.255.255.0)
Known via 'BGP', distance 200,
    Redistributing via BGP
    Last update from 10.0.160.104 00:01:17 ago
Routing Descriptor Blocks:
* 10.0.160.104 , via Indirect 00:01:17 ago

It is recommended that you not use other techniques in the SD-WAN overlay, such using BFD and adjusting graceful restart timers, because they interfere with the headless operation of the Versa solution.

Fast Convergence on the LAN Side

An important aspect of fast convergence is rapid failure detection. The detection can be based on a link status failure, such as when neighbors are connected through a Layer 2 network. An alternate detection approach is to use BFD, which is a protocol that offers timer-related detection. BFD is a high-speed protocol designed to detect fast link failures in milliseconds. IGPs and BGP are BFD clients, and multiple routing protocols can piggyback a single BFD session. It is recommended that you enable BFD on the VOS CPE devices. and establish BFD sessions with customer-owned Layer 3 devices. On VOS edge device, you can use BFD with BGP, OSPF, RIP, and static routes.

The following example command output shows a BFD session that is established between two OSPF neighbors:

admin@branch8-cli> show ospf neighbor brief
State codes: atmpt - attempt, exchg - exchange, exst - exchange start,
             load - loading, 2-way - two-way, full - full
Op codes:    gdown - going down, gup - going up

Intf address     Interface     State     Neighbor ID     Pri     Op
------------     ---------     -----     -----------     ---     --
192.168.254.2    vni-0/4.10    full      1.1.1.1         1       up

admin@branch8-cli> show bfd session org Tenant1 routing-instance-name Tenant1-LAN-VR session-summary
Instance         Address       State     RxPkts         TxPkts
Tenant1-LAN-VR   192.168.254.2 up        8005           16966

Scalability

VOS edge devices support a fully fledged scalable and powerful routing software suite. This section discusses some best practices for configuring BGP and OSPF for when you are. adding. a Versa CPE to an existing network.

admin@Controller-1-cli> show bgp group brief

routing-instance: Provider-Control-VR
BGP instance: 3
Group Type: Internal                     Local AS: 64512
    Name: Branches
    Options: < PEER-AS EXPORT-POLICY >
    peer-as:
    export-policy: TO_SDWAN
    Total peers: 7         Established: 7

    10.1.192.101+37024
    10.1.192.102+43934
    10.1.192.103+44734
    10.1.192.104+41982
    10.1.192.105+34777
    10.1.192.106+36272
    10.1.192.107+33559

Loop Prevention

In many cases, routing loops appear because of mutual route redistribution between two protocols at more than one point. For dual-homed topologies in which mutual redistribution is performed between BGP and OSPF, you should configure a domain VPN tag and enable the DN bit (Down bit) option in the OSPF instance.

Let's talk more about the Down bit and the domain tag to understand how they work in regards to loop prevention. When you use OSPF and BGP in an SD-WAN overlay network, the interaction between the two protocols is similar to that between OSPF and a BGP/MPLS network, as described in RFC 4577. As a summary:

• If the OSPF route has been advertised from a PE router (that is, a VOS edge device) into an OSPF area, the Down (DN) bit is set in the Options field of an OSPF LSA header Type 3, Type 5, or Type 7 LSA to ensure that the route is ignored by any other PE routers that receive them.
• If a particular VRF in a PE is associated with an instance of OSPF, the VPN route tag, domain VPN tag, or OSPF tag is required . In this case, the tag is configured with a special OSPF route tag value that is set in the OSPF Type 5 LSA. For the route tag value, the VOS device uses the VRF instance number of the virtual router (that is, of Tenant-LAN-VR), which is assigned randomly by the Director node and which is the same value across the SD-WAN network. So when a PE router receives the route after it has traversed multiple CE routers, if the PE router has the same VRF instance number, it ignore the routes because it has the same route tag value. Configuring and including the VPN route tag is required for backwards-compatibility with deployed implementations that do not set the DN bit in Type 5 LSAs.

The topology in the following figure explains how to check whether the Down bit and domain tag are set and shows the possible issues that might occur if the VOS CPEs are not directly connected to the local LAN.

In this topology, on the single CPE site (Branch-250), the prefix of the local LAN (50.50.50/24) is redistributed into MP-BGP. On the dual-homed HA active-active site, OSPF runs between the VOS CPE devices (Branch-6 and Branch-7) and a local router.

To prevent routing loops, both the OSPF Down bit and the OSPF domain VPN tag are set when routes are redistributed from MP-BGP into OSPF redistribution. You can see this for the prefix 50.50.50/24 (which comes from Branch-250) when it is redistributed from MP-BGP into OSPF on the Branch-6 and Branch-7 routers:

admin@branch6-cli> show ospf database routing-instance Tenant1-LAN-VR external detail 
50.50.50.0         6.6.6.6         0x80000001         15         0x0000A795
    Age: 15 secs; Sequence number: 0x80000001; Checksum: 0x0000A795
    Options: DN, E
    Type: E2
    Metric: 100
    Forwarding address: 0.0.0.0 
    External Route Tag: 2 
50.50.50.0          7.7.7.7         0x80000001         18         0x000089AF
    Age: 18 secs; Sequence number: 0x80000001; Checksum: 0x000089AF
    Options: DN, E
    Type: E2
    Metric: 100
    Forwarding address: 0.0.0.0 
    External Route Tag: 2

The Down bit and VPN tag are helpful for preventing routing loops prevention. However, a problem arises when the VOS CPEs are not directly connected to the local LAN and another router provides tenant-based LAN connectivity and is an OSPF neighbor with a tenant on the VOS CPE. The following sample output shows a third router that has Branch-6 and Branch-7 as OSPF neighbors in VRF Tenant1:

router ospf 1 vrf Tenant1
  router-id 1.1.1.1
  redistribute connected subnets route-map C2OSPF
  passive-interface Ethernet1/3.10

IOU1# show ip ospf 1 neighbor

Neighbor ID         Pri     State         Dead Time         Address         Interface
7.7.7.7             0       FULL/  -      00:00:38         192.168.254.13   Ethernet0/2.10
6.6.6.6             0       FULL/  -      00:00:30         192.168.254.9    Ethernet0/1.10

Because the Down bit is set in the Type 5 LSA for 50.50.50/24 and the existing router is already VRF aware (it is acting as a PE), the LSA is not considered for SPF calculation and the prefix is not added to the router's route table. This breaks the connectivity for the dual-homed site to remote locations:

IOU1# show ip ospf 1 database external 50.50.50.0

            OSPF Router with ID (1.1.1.1) (Process ID 1)
                
                Type-5 AS External Link States
    LS age: 1524
    Options: (No TOS-capability, No DC, Downward)
    LS Type: AS External Link
    Link State ID: 50.50.50.0 (External Network Number )
    Advertising Router: 6.6.6.6
    LS Seq Number: 80000001
    Checksum: 0xA795
    Length: 36
    Network Mask: /24
        Metric Type: 2 (Larger than any link state path)
        MTID: 0
        Metric: 100
        Forward Address: 0.0.0.0 
        External Route Tag: 2

    LS age: 1525 
    Options: (No TOS-capability, No DC, Downward)
    LS Type: AS External Link
    Link State ID: 50.50.50.0 (External Network Number )
    Advertising Router: 7.7.7.7
    LS Seq Number: 80000001
    Checksum: 0x89AF
    Length: 36
    Network Mask: /24
        Metric Type: 2 (Larger than any link state path)
        MTID: 0
        Metric: 100
        Forward Address: 0.0.0.0 
        External Route Tag: 2 

However, the prefix 50.50.50/24 is not present in the route table:

IOU1# show ip route vrf Tenant1 50.50.50.0

Routing Table: Tenant1
% Network not in table 

The recommended solution is to check whether the intermediary router (here, IOU1) has the capability to ignore the Down bit set in the LSA option field and can instead use the LSA in the SPF algorithm, so that it can add the prefix to its route table. Most vendors call this feature VRF-lite. If the existing customer router does not support this capability, you should disable the Down bit setting on VOS devices. In this situation, you must be especially careful when mutual route redistribution between MP-BGP and OSPF is performed.

Routing Security

Best Practices for Dynamic Routing

• For improved convergence in a dual-homed branch, use a unique route distinguisher for each LAN-VRF to advertise unique information for the same local prefix.
• Configure the OSPF network as a point-to-point network to prevent the use of unnecessary CPU cycles.
• Configure BGP peers as part of the same BGP peer group so that the peers use the same outbound policy.
• Enable HMAC-MD5 authentication on all interfaces on which an IGP runs.
• Configure BGP authentication and TTL for BGP sessions.

Rewrite Policy

You can use a rewrite policy to remark packets and frames based on the classification done on ingress. The classification is done using a Layer 3/Layer 4 or App QoS policy that assigns the packets to a forwarding class and loss priority. The rewrite policy uses the forwarding class and loss priority information as match conditions to set a QoS value.

A typical use case is to rewrite the LAN traffic passing through a CPE device, whether the traffic is destined for another LAN port or for egress somewhere else on a remote branch. The DSCP of the traffic is modified and is carried, or propagated, with the traffic as it moves through the network.

There are three types of rewrite policies:

• DSCP rewrite policy
• DSCP6 rewrite policy
• 802.1p rewrite policy

Depending on where you apply the rewrite policy, it modifies the QoS bits of either the inner or outer header, but it does not propagate.

In a rewrite policy, the match condition is the forwarding class and the loss priority. You configure the forwarding class in the QoS profile, which is referenced in the Layer 3/Layer 4 or App QoS policy.

You enable the following options in the QoS profile depending on whether the type of rewrite policy:

• DSCP Rewrite
• Dot 1P Rewrite

If you do not select either the DSCP rewrite or Dot 1P rewrite option in the QoS profile, the rewrite policy does not take effect.

One use case for this setting is when you have more than one QoS policy that classifies traffic to the same forwarding class. However, if you do not need to remark all the traffic with the rewrite policy, you can use this setting to help differentiate which traffic is forwarding class is evaluated by the rewrite policy.

A second use case is on a multitenant VOS device. on which the rewrite policy is applied on the WAN interfaces that are owned by the provider. If there are two customer tenants on the branch and each tenant uses the same forwarding class in their QoS policies, they can choose whether to set the DSCP Rewrite and Dot1P Rewrite option, depending on whether they want their traffic to be remarked..

Rewrite Options

The rewrite options set a flag to copy the markings between the inner and outer IP headers. This is a global setting, so it affects all the traffic passing through the VOS device. You cannot apply in on a per-interface or per-traffic class basis.

You can configure the following options:

• Copy From Outer—Copy the marking of the outer IP header to the inner IP header when a branch sends packets to a remote branch.
• Copy From Inner—Copy the marking of the inner IP header to the outer IP header when a branch receives packets from a remote branch.

The Copy From Outer setting remarks the inner IP header of traffic coming from a remote site. If the outgoing network applies a rewrite policy that also remarks the inner IP header of the same packet, the marking in the rewrite policy overwrites the marking made by the Copy From Outer option. For example, on a regular branch the outgoing network could be a LAN and, on the hub the outgoing network could be a WAN interface.

With the Copy From Outer setting, the VOS edge device trusts the markings from the underlay. You should trust outer markings on packets received on a private circuit such as MPLS, but you should not trust them on packets received not from the internet. Therefore, you should use this setting only on a branch that has only an MPLS WAN circuit.

The Copy From Inner settings works on a model whereby the VOS device trusts the markings from the LAN. Because this setting remarks the outer IP header of the traffic coming from the LAN site, if you apply a rewrite policy on the WAN network that also remarks the outer IP header of the same packet, the marking made by the Copy From Inner setting overrides the marking made by the rewrite policy.

QoS Propagation Policy on Hubs

On a hub, traffic that arrives encrypted from a remote site, is decrypted, and is encrypted again before it is sent it to another site. Because the packets are decrypted on the hub, the QoS propagation policy can remark both the inner and outer headers based on the classification made on the inner packet (that is, the actual application packet). The hub can also use the rewrite option flags like Copy From Outer and Copy From Inner.