CSG5000 100Gbps Port Per-Tunnel and Pipe Level Shapers
For supported software information, click here.
This feature is for use with the Versa CSG5000 gateway or similar high-end gateway appliances that employ multi-queue (RSS/TSS) pollers. This allows VLAN or per-tunnel (pipe) shaping to be enabled on 10 Gbps or 100 Gbps ports on these appliances. The support includes per-pipe level shaping rates and a pipe’s per-traffic classes, along with committed and peak rates, and round-robin scheduling of four queues for each traffic class. It also supports port-level shaping as well as port-level, per-traffic shaping and scheduling class (eight traffic classes or queues for port level) as for the existing shaping features for the CSG5000.
The port-level shaping continues to use hardware schedulers, whereas the pipe-level shaping is scheduled in software. Each pipe/tunnel is anchored on a single thread, so the aggregate rate for a single tunnel/pipe is limited by the shaping throughput of shaping on a single thread. The aggregate rate also depends on the number of tunnels/pipes that are anchored on the same poller thread. Example throughput numbers are given at the end of this document.
The following configurations use show and VTY commands to enable, observe, and use the pipe-level shapers on CSG5000 gateways. Most are existing commands and configurations:
- For configuring adaptive shaping on branches, see Configure CoS and Configure Adaptive Shaping.
- Configure a queue depth of [192 – 255] (WRED) for the non-real time traffic classes schedulers configured for the 100 Gbps ports.
org-services Tenant1 {
class-of-service {
…
…
drop-profiles {
dp1 {
minimum 192;
maximum 255;
}
}
schedulers {
sched-AF {
…
drop-profile-map {
loss-priority low {
drop-profile dp1;
}
loss-priority high {
drop-profile dp1;
}
}
}
sched-BE {
…
drop-profile-map {
loss-priority low {
drop-profile dp1;
}
loss-priority high {
drop-profile dp1;
}
}
}
- Separating the set of 8-Rx and 8-Tx pollers for improving Tx-shaping performance is recommended. It is not necessary, but is recommended to improve the performance of the shapers:
admin@New-5000-CSG-cli(config)% show system service-options
poller-count 16; >>>> Poller count 16
port-poller-map { >>> For setting pollers for each port explicitly
port vni-0/16 {
poller-cpu-id [ 48 49 50 51 52 53 54 55 ];
}
port vni-0/18 {
poller-cpu-id [ 48 49 50 51 52 53 54 55 ];
}
}
tx-port-poller-map { >>> Separating Tx pollers for vni-0/16 to isolated one from Rx pollers and all other ports to pollers CPUs [56 – 63]
port vni-0/16 {
poller-cpu-id [ 56 57 58 59 60 61 62 63 ];
}
port vni-0/18 {
poller-cpu-id [ 48 49 50 51 52 53 54 55 ];
}
}
- A configuration has been added to change a tunnel or pipe’s re-anchoring to a different poller:
- The load on pollers can be monitored with:
vsm-vcsn0> show vsm load - The poller index can be obtained by the vty command:
vsm-vcsn0> show cos scheduler pipe-mapping vni-0/16 - Configure on the branch to which the tunnel is to be shaped, SD-WAN device site membership:
- The load on pollers can be monitored with:
admin@New-1500-CSG-cli(config)% show | compare
orgs {
org Tenant1 {
sd-wan {
site {
+ group-memberships [ sdwan-branch-grp1 ];
}
}
}
}
- Configure a policy to match the remote branch’s SD-WAN tunnel’s group membership for steering to a different poller.
admin@New-5000-CSG-cli(config)% show | compare
orgs {
org Tenant10 {
sd-wan {
site {
wan-interfaces {
vni-0/16.1200 {
+ shaper-policy shaper_pol;
}
}
+ path-policy shaper_pol {
+ term t1 {
+ match {
+ remote-group-membership [ sdwan-branch-grp1 ];
+ remote-site-type [ branch ];
+ }
+ action {
+ shaper {
+ poller-index 8;
+ }
+ }
+ }
+ }
}
}
}
}
CLI show Commands for Shaping
- To check for shaping results:
admin@New-5000-CSG-cli> show class-of-services interfaces extensive vni-0/16
- To check for shaping accuracy/efficacy:
admin@New-5000-CSG-cli> show class-of-services interfaces extensive 16 | match "q0: fc_ef" | nomore
q0: fc_ef 1 0 0 0 0 0 0 0 0
q0: fc_ef 1 24750-99000 644471800 107310586 433548912624 71961454884 255 96928592 15475776
q0: fc_ef 1 1500000-6000000 763352466 249 4425017389512 133490 0 976399360 0
q0: fc_ef 1 1500000-6000000 7636392065 0 4426780624814 0 1 974296992 0
q0: fc_ef 1 1500000-6000000 7633474270 44 4425091032164 27856 0 975612504 0
q0: fc_ef 1 1500000-6000000 7636140293 101 442663252938 55446 3 975399232 0
q0: fc_ef 1 1500000-6000000 9407948239 89717413 5453789429318 51971266282 0 1164299416 0
q0: fc_ef 1 1500000-6000000 7634549804 104 4425714565212 57128 1 973590664 0
q0: fc_ef 1 200000-800000 6100473837 3031081739 3537524195762 1755996021318 255 779209856 391953144
q0: fc_ef 1 1500000-6000000 9094455107 4276292 5272009945002 2479185952 0 1170227712 0
q0: fc_ef 1 1500000-6000000 9427754275 102542887 5465270740418 59400559886 2 1169789680 0
q0: fc_ef 1 250000-1000000 7625991932 1473164305 4421392369204 853343636486 253 974041080 196704240
q0: fc_ef 1 1500000-6000000 9486017317 51637754 5499046457006 29888967300 4 1168665400 0
q0: fc_ef 1 1500000-6000000 7606462600 3 4409440603028 1486 0 969242896 0
[ok][2025-01-29 09:05:23]
VTY Commands
$ vsh connect vsmd
To check CPU utilization or service load:
vsm-vcsn0> show vsm load
Load info of 2011ms (sampled 1984ms ago): WT EWMA avg sampled over 1ms interval
Svc Svc CPU
Thread PID Load EWMA Load
----------------------------------
WT-0 14685 2 0 100
WT-1 14686 98 97 100
WT-2 14687 62 61 100
…
…
…
WT-46 14764 51 50 100
WT-Avg ------ 55 --- 100
PT-0 14765 57 --- 100
PT-1 14766 57 --- 100
PT-2 14767 57 --- 100
PT-3 14768 57 --- 100
PT-4 14769 57 --- 100
PT-5 14770 56 --- 100
PT-6 14771 57 --- 100
PT-7 14792 57 --- 100
PT-8 14793 90 --- 100
PT-9 14794 99 --- 100
PT-10 14795 89 --- 100
PT-11 14796 80 --- 100
PT-12 14797 99 --- 100
PT-13 14798 87 --- 100
PT-14 14799 79 --- 100
PT-15 14800 99 --- 100
• To check the tunnel/pipe to poller mapping:
vsm-vcsn0> show cos scheduler pipe-mapping vni-0/16 Interface name : vni-0/16 Pipe ID poller IfIdx AcktID IsCFG Description 0 65535 0 0 1 Default ===> 65k is invalid poller index and shows pipes that are skipping pipe level shaping. 1 9 0 0 1 vni-0/16.10 2 10 0 0 1 vni-0/16.11 3 11 0 0 1 vni-0/16.12 4 12 0 0 1 vni-0/16.13 5 13 0 0 1 vni-0/16.14 6 14 0 0 1 vni-0/16.15 7 15 0 0 1 vni-0/16.16 8 9 0 0 1 vni-0/16.17 9 8 0 0 1 vni-0/16.18 10 11 0 0 1 vni-0/16.19 11 10 0 0 1 vni-0/16.2 12 13 0 0 1 vni-0/16.20 13 12 0 0 1 vni-0/16.21 14 15 0 0 1 vni-0/16.22 15 14 0 0 1 vni-0/16.23 16 8 0 0 1 vni-0/16.24 17 9 0 0 1 vni-0/16.25 18 10 0 0 1 vni-0/16.26 19 11 0 0 1 vni-0/16.27 20 12 0 0 1 vni-0/16.28 21 13 0 0 1 vni-0/16.29 22 14 0 0 1 vni-0/16.3 23 15 0 0 1 vni-0/16.30 24 9 0 0 1 vni-0/16.31 25 8 0 0 1 vni-0/16.32 26 11 0 0 1 vni-0/16.33 27 10 0 0 1 vni-0/16.34 28 13 0 0 1 vni-0/16.35 29 12 0 0 1 vni-0/16.36 30 15 0 0 1 vni-0/16.37 31 14 0 0 1 vni-0/16.38 32 8 0 0 1 vni-0/16.39 33 9 0 0 1 vni-0/16.4 34 10 0 0 1 vni-0/16.40 35 11 0 0 1 vni-0/16.41 36 12 0 0 1 vni-0/16.5 37 13 0 0 1 vni-0/16.6 38 14 0 0 1 vni-0/16.7 39 15 0 0 1 vni-0/16.8 40 9 0 0 1 vni-0/16.9 41 8 1308 34 0 WAN-105:34:10:clear 42 11 1309 34 0 WAN-105:34:10:secure 43 10 1312 34 0 WAN-105:34:14:clear 44 13 1313 34 0 WAN-105:34:14:secure 45 12 1396 34 0 WAN-105:34:4:clear 46 15 1397 34 0 WAN-105:34:4:secure 47 14 1320 34 0 WAN-105:34:8:clear 48 8 1321 34 0 WAN-105:34:8:secure 49 9 1326 34 0 WAN-105:34:15:clear 50 10 1327 34 0 WAN-105:34:15:secure 51 11 1392 34 0 WAN-105:34:17:clear 52 12 1393 34 0 WAN-105:34:17:secure 53 13 1332 34 0 WAN-105:34:21:clear 54 14 1333 34 0 WAN-105:34:21:secure 55 15 1336 34 0 WAN-105:34:19:clear 56 9 1337 34 0 WAN-105:34:19:secure 57 8 1340 34 0 WAN-105:34:16:clear 58 11 1341 34 0 WAN-105:34:16:secure 59 10 1344 34 0 WAN-105:34:18:clear 60 13 1345 34 0 WAN-105:34:18:secure 61 12 1398 34 0 WAN-105:34:3:clear 62 15 1399 34 0 WAN-105:34:3:secure 63 14 1354 34 0 WAN-105:34:13:clear 64 8 1355 34 0 WAN-105:34:13:secure 65 9 1358 34 0 WAN-105:34:12:clear 66 10 1359 34 0 WAN-105:34:12:secure 67 11 1360 34 0 WAN-105:34:11:clear 68 12 1361 34 0 WAN-105:34:11:secure 69 13 1364 34 0 WAN-105:34:7:clear 70 14 1365 34 0 WAN-105:34:7:secure 71 15 1370 34 0 WAN-105:34:5:clear 72 9 1371 34 0 WAN-105:34:5:secure 73 8 1374 34 0 WAN-105:34:9:clear 74 11 1375 34 0 WAN-105:34:9:secure 75 10 1376 34 0 WAN-105:34:6:clear 76 13 1377 34 0 WAN-105:34:6:secure 77 12 1382 34 0 WAN-105:34:1:clear 78 15 1383 34 0 WAN-105:34:1:secure 79 14 1386 34 0 WAN-105:34:2:clear 80 8 1387 34 0 WAN-105:34:2:secure
65k is an invalid poller index and shows pipes that are skipping pipe-level shaping. Traffic for these pipes will be directly enqueued to hardware queues and are not anchored. All pollers can transmit packets for these pipes. The vni-0/16.0 is the default VLAN which will not be shaped for pipes (skips pipe shaping) and is directly enqueued to hardware queues.
To see the per-poller hardware queue statistics:
vsm-vcsn0> show vsm statistics hw-cos detail vni-0/16 HW-COS TX queue stats for port : 1 [vni-0/16] | PID | TC | tx-pkts | tx-dropped | tx-bytes | Qid | ----------------------------------------------------------------------------- | 8 | TC0 | 195133 | 0 | 34955874 | 0 | | | TC1 | 0 | 0 | 0 | 0 | | | TC2 | 28713276799 | 0 | 16808087979858 | 1 | | | TC3 | 0 | 0 | 0 | 1 | | | TC4 | 0 | 0 | 0 | 2 | | | TC5 | 0 | 0 | 0 | 2 | | | TC6 | 0 | 0 | 0 | 3 | | | TC7 | 0 | 0 | 0 | 3 | | 221433 pkts skipped pipe shaping for poller idx 8 >>>Number of pkts that were directly queued to hardware queue >>>without going to pipe level shaping queue. ----------------------------------------------------------------------------- | 9 | TC0 | 64986 | 0 | 13435196 | 4 | | | TC1 | 0 | 0 | 0 | 4 | | | TC2 | 37707261578 | 0 | 21967433587764 | 5 | | | TC3 | 0 | 0 | 0 | 5 | | | TC4 | 0 | 0 | 0 | 6 | | | TC5 | 0 | 0 | 0 | 6 | | | TC6 | 0 | 0 | 0 | 7 | | | TC7 | 0 | 0 | 0 | 7 | | 90147 pkts skipped pipe shaping for poller idx 9 ----------------------------------------------------------------------------- | 10 | TC0 | 64986 | 0 | 13419516 | 8 | | | TC1 | 0 | 0 | 0 | 8 | | TC2 | 36238651269 | 0 | 21116831908338 | 9 | | | TC3 | 0 | 0 | 0 | 9 | | | TC4 | 0 | 0 | 0 | 10 | | | TC5 | 0 | 0 | 0 | 10 | | | TC6 | 0 | 0 | 0 | 11 | | | TC7 | 0 | 0 | 0 | 11 | | 68787 pkts skipped pipe shaping for poller idx 10 ----------------------------------------------------------------------------- | 11 | TC0 | 97429 | 0 | 20109314 | 12 | | | TC1 | 0 | 0 | 0 | 12 | | | TC2 | 34110709926 | 0 | 19936965137332 | 13 | | | TC3 | 0 | 0 | 0 | 13 | | | TC4 | 0 | 0 | 0 | 14 | | | TC5 | 0 | 0 | 0 | 14 | | | TC6 | 0 | 0 | 0 | 15 | | | TC7 | 0 | 0 | 0 | 15 | | 73912 pkts skipped pipe shaping for poller idx 11 ----------------------------------------------------------------------------- | 12 | TC0 | 66121 | 0 | 13636539 | 16 | | | TC1 | 0 | 0 | 0 | 16 | | | TC2 | 36239159180 | 0 | 21116935038992 | 17 | | | TC3 | 0 | 0 | 0 | 17 | | | TC4 | 0 | 0 | 0 | 18 | | | TC5 | 0 | 0 | 0 | 18 | | | TC6 | 0 | 0 | 0 | 19 | | | TC7 | 0 | 0 | 0 | 19 | | 72644 pkts skipped pipe shaping for poller idx 12 ----------------------------------------------------------------------------- | 13 | TC0 | 97414 | 0 | 20100600 | 20 | | | TC1 | 0 | 0 | 0 | 20 | | | TC2 | 33418844999 | 0 | 19536175558966 | 21 | | | TC3 | 0 | 0 | 0 | 21 | | | TC4 | 0 | 0 | 0 | 22 | | | TC5 | 0 | 0 | 0 | 22 | | | TC6 | 0 | 0 | 0 | 23 | | | TC7 | 0 | 0 | 0 | 23 | | 75374 pkts skipped pipe shaping for poller idx 13 ----------------------------------------------------------------------------- | 14 | TC0 | 65025 | 0 | 13504254 | 24 | | | TC1 | 0 | 0 | 0 | 24 | | TC2 | 36228751914 | 0 | 21110706721924 | 25 | | TC3 | 0 | 0 | 0 | 25 | | | TC4 | 0 | 0 | 0 | 26 | | | TC5 | 0 | 0 | 0 | 26 | | | TC6 | 0 | 0 | 0 | 27 | | | TC7 | 0 | 0 | 0 | 27 | | 80863 pkts skipped pipe shaping for poller idx 14 ----------------------------------------------------------------------------- | 15 | TC0 | 772631 | 0 | 474236742 | 28 | | | TC1 | 0 | 0 | 0 | 28 | | | TC2 | 36821399184 | 0 | 21506562732292 | 29 | | | TC3 | 0 | 0 | 0 | 29 | | | TC4 | 0 | 0 | 0 | 30 | | | TC5 | 0 | 0 | 0 | 30 | | | TC6 | 0 | 0 | 0 | 31 | | | TC7 | 0 | 0 | 0 | 31 | | 86503 pkts skipped pipe shaping for poller idx 15 ----------------------------------------------------------------------------- | Total | TC0 | 1423725 | 0 | 603398035 | | | TC1 | 0 | 0 | 0 | | | TC2 | 279478059328 | 0 | 163099701282670 | | | TC3 | 0 | 0 | 0 | | | TC4 | 0 | 0 | 0 | | | TC5 | 0 | 0 | 0 | | | TC6 | 0 | 0 | 0 | | | TC7 | 0 | 0 | 0 |
Limitations and Known Issues
- A maximum of eight Tx polling threads can be employed for shaping due to existing NIC firmware and ICE driver limitations.
- The IFD/port-level shaping uses a different hardware level scheduler, whereas the tunnel/pipe-level uses a different one in software. Due to this, the port scheduler does not account for pipe’s traffic class level CIR guarantees in some cases if the port’s shaping rate is reduced to a value lower than the default rate (for example, 100Gbps->40Ggbps). To avoid this drawback, the configuration of any committed rates for the pipes (for example, VLANs) should be such that: SUM{CIR of all pipes} < = port’s shaping rate.
- The CLI configuration to re-anchor the tunnel to a different poller is not correctly applied after a service restart. Hence, this configuration will need to be deleted and reapplied after service restart.
- The current (older) version of the Versa Director user interface does not support tunnel re-anchoring; it is only supported via CLI.
- Due to this limitation, the configuration sync back from the gateway (CSG5k) in which shapers for 100Gbps are enabled (and if tunnel re-anchoring is configured) will fail. To work around this, the CLI configuration for re-anchoring on the gateway should be deleted before re-syncing the configuration from the gateway to the Versa Director, and after this the configuration can be added back.
Sample Profiling for Accuracy and Performance
- 8 Rx and 8 Tx poller CPUs separated for a single 100 Gbps interface
- Hypermode enabled
- 60K sessions well distributed to all WTs
- Average packet size: 579 Bytes
- 20 SD-WAN tunnels, each shaped to 99 Mbps
- Single traffic class for each pipe and [ 255 – 192 ] queue depth for all queues
- Total traffic sent to the tunnels—8.5 Gbps with the following distribution:
- 10 tunnels sent 100 Mbps
- 5 tunnels sent 500 Mbps
- 5 tunnels sent 1 Gbps
Profiling using VLAN pipe traffic, for loading, along with the above tunnel traffic for the following cases with all pipes accurately shaped as per their shaper profiles. The offered and egress rates are measured at the Spirent. The tunnel egress rates are lower due to tunnel overheads being included at shapers on the WAN interface:
- 60 pipes: Total offered rate of 48.5 Gbps, egress shaped rate of 40.1Gbps.
- 40 VLANs: offered rate 40 Gbps. 38.4 Gbps egress shaped rate
- 20 tunnels: offered rate 8.5 Gbps. 1.7 Gbps egress shaped rate
- 40 pipes: Total offered rate of 40 Gbps, egress shaped rate of 27 Gbps.
- 20 VLANs: offered rate 31.5 Gbps. 25.3 Gbps egress shaped rate
- 20 tunnels: offered rate 8.5 Gbps. 1.7 Gbps egress shaped rate
- 28 pipes: Total offered rate of 41 Gbps, egress shaped rate of 30 Gbps.
- 8 VLANs: offered rate 32.5Gbps. 28.3 Gbps egress shaped rate
- 20 tunnels: offered rate 8.5 Gbps. 1.7 Gbps egress shaped rate
Supported Software Information
Releases 22.1.4 (Service Release dated 2025-08-18) and later support all content described in this article.
Additional Information
Configure Adaptive Shaping
Configure CoS
Configure SD-WAN Traffic Steering