Dynamic Path Selection
The CloudEOS and vEOS supports the Dynamic Path Selection that selects the path for the traffic to optimize application performance in the enterprise deployments.
The enterprise network sites like Data centers, Branches, Public Cloud (AWS VPC, Azure VNet, and others) are connected through multiple SPs (MPLS, Internet, LTE). Enterprises deploy edge routers to connect these sites over the SP WAN networks and in some cases building GRE or IPsec tunnels between sites. For high availability reasons, there are at least two WAN networks or paths available between sites.
In the above example there are 5 paths, 1 MPLS path and are four paths through ISPs: ISP1, ISP1-ISP3, ISP2-ISP3, ISP2-ISP1. Different ISP have different costs, bandwidth, WAN characteristics, SLAs, and so on. This is ideal for users wanting to use various SPs in a cost effective manner without sacrificing application performance. The traditional enterprises use MPLS VPNs which provides a very good WAN characteristics such as (latency, etc), but, at a very high costs. Internet has been gaining adoption as an alternative WAN to MPLS that offers much higher bandwidth at lower costs. Also, MPLS VPNs are not available in all geographies. While ISPs are more readily available and at a lower cost, however, maintaining application performance for traffic across sites is a big problem because ISPs don’t offer a good SLAs. The traditional routing solutions do not address the requirements to optimize routing across WAN SP networks.
Overview
This section describes the functional overview of the Dynamic Path Selection feature. The below figure shows three routers in different sites interconnected through two SPs. In this example, Site 1 is a hub site and is connected to both Site 2 and Site 3. There are two paths between site1 to site 2 and two paths from site 1 to site 3.
Path Definition
A “path” represents a pair of interfaces, a source interface and a destination interface through which traffic can flow from site to site. For example, eth1/router1 -- eth1/router2 is a path. Note, that there could be many paths through the same egress interface. The “path” does not refer to the actual network path the packet takes through the SP network. There could be multiple network paths in SP network from customer’s edge router to another edge router. Also, the network paths could change. A path is unidirectional and path characteristics is tracked in each direction.
Dynamic Load Balancing
Selects the best path (destination IP and egress interface) to a destination for a given application. The algorithm has to select the best paths based on user specified priorities or constraints, and dynamically load balance flows across selected paths.
WAN Overlay using VXLAN
Note, that the routers are connected to two SPs in the above diagram. All customer prefixes are on the overlay network and if the VTEP IP r1addr and r2addr addresses are accessible through SP networks then the VXLAN overlay would work similar to the datacenter network. However the VTEP IP address is an internal IP address and is not routable over SP networks. While it is possible to make the VTEP IP address routable over MPS network (unlike ISP), since we want to dynamically load balance across SP networks we will not advertise the VTEP IP address over MPLS.
However, the WAN interfaces have SP routable IP address. For example, r1w1 IP address is routable on WAN1 and r1w2 IP address is routable on WAN2. The forwarding engine will replace the VTEP address on the packet based on the path selected before sending it to SP network.
Therefore for router 1:
- The router VTEP IP V1 is the nexthop for all the customer prefixes and the customer prefixes p1, and others are advertised using EVPN type 5 address family.
- VTEP IP V1 is reached through the two publically routable WAN IP addresses r1w1 address and r1w2 address.
For this, the router needs to know SP routable IP addresses through which it can reach each.
DPS / Et100 Interface
Peer VTEP Reachability
In the above figure there are five paths between the two sites:
- MPLS - Ip11, ip21
- Internet Ip12 - ip22
- Internet Ip12 - ip23
- Internet Ip13 - ip22
- Internet Ip13 - ip23
Currently peer VTEP reachability needs to be configured statically, but, in future this is exchanged through BGP. BGP runs on the same loopback interface used as VXLAN source VREP interface in underlay.
The router tracks if the configured paths are available using routing updates, interface state and so on, and programs the available paths for forwarding.
Control Plane Traffic
The BGP traffic that is going between sites will all go through DPS interface and leverage path selection feature to ensure that the BGP traffic leverages all the path selection features. Different path selection policies can be setup for different control plane traffic types as for end applications.
Load Balancing Algorithm
Algorithm selects the path that meets all the criteria for an application. If there are multiple paths that meet then it load balances across the available paths. If none of the paths meet the criteria is it picks the one with the lowest loss rate.
The selected path for a given flow is then stored in flow cache. The chosen path is not reevaluated for constraints. Packets from that flow will take the same path even if the path characteristics no longer meet the user specified criteria.
Events that trigger the re-selection of path for a flow are as shown below:
- When path is no longer active.
- When the flow is remapped to a different application.
- When user has changed the constraints or priority such that the path is no longer valid for this flow.
Path Telemetry
Path Telemetry feature provides the ability to determine WAN path state and measure its characteristics including latency (one way delay), jitter, packet loss rate and throughput.
The outer IP header uses the WAN IP addresses on local and peer WAN interfaces of the path. IP header is followed by a UDP header where the destination port is set to be 4793 by default or to be the port number configured by user in CLI. When IPsec is enabled, destination port is set to be 4500. A path telemetry header is inserted in between of UDP/ESP header and the inner IP packet for path characteristics measurement purpose.
Path State Determination
Path telemetry uses keepalive and feedback packets to determine path state. It sends out keepalive periodically (once per second) and if it receives peer’s feedback packet, the path is considered as active and its characteristics is measured. Accordingly, if feedback packet is not received within a certain period of time (for 5 keepalive we sent), the path is considered as inactive and is not used for path selection.
Configuration
This section describes the commands to configure and verify the Dynamic Path Selection feature.
Defining Paths
A “path” represents a pair of interfaces (or their IP addresses), a source interface and a destination interface through which traffic can flow from site to site.
For example, in the above figure there are two paths from Router1 to Router2
- MPLS path - 172.16.1.1 -- 172.16.2.1
- 4 Internet paths
- 1.1.1.1 -- 3.3.3.3
- 1.1.1.1 -- 4.4.4.4
- 2.2.2.2 -- 3.3.3.3
- 2.2.2.2 -- 4.4.4.4
However, some of the paths are crossing ISPs, for example, 1.1.1.1 -- 4.4.4.4 is going from router1 through ISP1, ISP2 to router2. In some customer scenarios ISP2 could be an LTE SP and could be purely as a backup in case ISP1 fails. In this case the paths 1.1.1.1 -- 4.4.4.4 and 2.2.2.2 -- 3.3.3.3 should not be used.
Path-group similar to nexthop-group is used to group the paths in order to
- Restrict paths - define which paths are valid among the available paths like the LTE backup SP discussed before
- Apply specific policies to path group. Eg apply encryption for all Internet paths
Path group commands are configured under “router path-selection” as shown below. The commands are explained in the subsections.
router path-selection
path-group <group-name>
local interface <intf-name>
## more local interface commands
## that belong to the same path-group, eg Internet
peer static router-ip <ip-address>
ipv4 address <ip-addr1>
## more IP addresses through which the router can be reachedThe router-IP is the same as the VTEP-IP. local is used to configure the local WAN IP address or interface part of the path-group. Peer is used to configure the remote VTEP reachability statically.
Each combination of peer and local IP address is a potential path. If routing resolves the remote IP through a local interface then that local-remote IP pair becomes a real path that is used for forwarding.
In the topology in the above figure two groups are defined.
- mpls-group
- Internet-group
Further if paths need to be restricted through the Internet, the Internet groups can be divided into more groups. For example, the customer can define ISP1 and ISP2-ISP3 as separate groups create 2 Internet paths instead of 4.
Creating Path-Groups under Path-Selection
Syntax
router path-selection path-group <name>
name:name of the path group
Example
switch(config)#router path-selection
switch(config-dynamic-path-selection)#
switch(config-dynamic-path-selection)#path-group mplsSpecifying Local Interfaces under Path-Group Sub-Mode
Syntax
path-group <name> local interface <intf-name>
local interface: is used to configure the local WAN interface part of the path-group. The IP addresses assigned to the WAN interface is used as WAN IP. Multiple interfaces can be specified. For example, if there are two ISP connections.
Example
In the above deployment: ether1 is part of MPLS path-group.
switch(config-dynamic-path-selection)#path-group mpls
switch(config-path-group-mpls)#local interface ether1
Ethernet 2 and 3 are part of Internet path-group
switch(config-dynamic-path-selection)#path-group internet
switch(config-path-group-internet)#local interface ether2
switch(config-path-group-internet)#local interface ether3Specifying Remote VTEPs and their Reachability Statically
Syntax
path-group <name> peer static router-ip <ip-address>ipv4 address <ip-addr1>ipv4 address <ip-addr2> ## more IP addresses through which the router can be reached
peer static is used to configure the remote VTEP reachability statically via routable IP addresses over the SP network. The router-IP is the VTEP IP address. In the case of Internet, the routable IP address is a public IP address. In the case of MPLS it is Enterprise specific private IP address that the MPLS provider knows how to reach. Typically customer edge routers (CEs) are configured to exchange subnets by running eBGP to the SP’s PE router.
Example
switch(config-dynamic-path-selection)#path-group mpls
switch(config-path-group-mpls)#peer static router-ip 10.2.2.2
switch(config-peer-router-ip-10.2.2.2-mpls)#ipv4 address 172.16.2.1
For the Internet path group Router2’s router IP 10.2.2.2 is reachable via two IP addresses only via ISP1 3.3.3.3 and another through ISP2 4.4.4.4
switch(config-dynamic-path-selection)#path-group internet
switch(config-path-group-internet)#peer static router-ip 10.2.2.2
switch(config-peer-router-ip-10.2.2.2-internet)#ipv4 address 3.3.3.3
switch(config-peer-router-ip-10.2.2.2-internet)#ipv4 address 4.4.4.4It is important to note that once local and remote IP addresses are specified for a path-group then all combinations of local and remote IP address is a potential path for load balancing.
Example
switch(config)#router path-selection
switch(config-dynamic-path-selection)#path-group mpls
switch(config-path-group-mpls)#local interface et1
switch(config-path-group-mpls)#peer static router-ip 10.2.2.2
switch(config-peer-router-ip-10.2.2.2-mpls)#ipv4 address 172.16.2.1
switch(config-peer-router-ip-10.2.2.2-mpls)#path-group internet
switch(config-path-group-internet)#local interface et2
switch(config-path-group-internet)#local interface et3
switch(config-path-group-internet)#peer static router-ip 10.2.2.2
switch(config-peer-router-ip-10.2.2.2-internet)#ipv4 address 3.3.3.3
switch(config-peer-router-ip-10.2.2.2-internet)#ipv4 address 4.4.4.4
The paths defined are
MPLS path - 172.16.1.1 -- 172.16.2.1
4 Internet paths
1.1.1.1 -- 3.3.3.3
1.1.1.1 -- 4.4.4.4
2.2.2.2 -- 3.3.3.3
2.2.2.2 -- 4.4.4.4
However if ISP2 is a LTE and the customer does not want paths to cross over from ISP1 to LTE then the configuration should be
switch(config)#router path-selection
switch(config-dynamic-path-selection)#path-group mpls
switch(config-path-group-mpls)#local interface et1
switch(config-path-group-mpls)#peer static router-ip 10.2.2.2
switch(config-peer-router-ip-10.2.2.2-mpls)#ipv4 address 172.16.2.1
switch(config-peer-router-ip-10.2.2.2-mpls)#path-group internet
switch(config-path-group-internet)#local interface et2
switch(config-path-group-internet)#peer static router-ip 10.2.2.2
switch(config-peer-router-ip-10.2.2.2-internet)#ipv4 address 3.3.3.3
switch(config-peer-router-ip-10.2.2.2-internet)#path-group lte
switch(config-path-group-lte)#local interface et3
switch(config-path-group-lte)#peer static router-ip 10.2.2.2
switch(config-peer-router-ip-10.2.2.2-lte)#ipv4 address 4.4.4.4
In the above case the paths are
MPLS path - 172.16.1.1 -- 172.16.2.1
Internet path 1.1.1.1 -- 3.3.3.3
LTE path 2.2.2.2 -- 4.4.4.4.Underlay DPS Configuration
For DPS paths and EVPN routes to be exchanged we need to configure VXLAN with a private IP address of a loopback interface and DPS interface should be configured as L3 interface. Please note that the configuration for DPS interface has to be split up and configured under two interfaces VXLAN1 and et100. In future they are replaced with one single DPS interface.
DPS Interface Configuration
For the DPS interface add any private IP address to make it an Layer 3 interface. However, the assigned IP address is not used for routing.
Syntax
interface Ethernet100 no switchport ip address 11.0.0.1/24
switch(config)#interface ethernet 100
switch(config-if-Et100)#no switchport
switch(config-if-Et100)#ip address 11.0.0.1/24VXLAN Configuration
In the example below 1.1.1.1 is a private IP which is configured in loopback 0 interface is used as VXLAN source interface.
Example
switch(config)#interface loopback 0
switch(config-if-Lo0)#ip address 1.1.1.1/32
switch(config-if-Lo0)#interface vxlan1
switch(config-if-Vx1)#vxlan source-interface loopback 0
switch(config-if-Vx1)#vxlan udp-port 4789
switch(config-if-Vx1)#vxlan vrf vrf1 vni 100BGP runs on the same loopback IP as VXLAN source interface IP. In the above example BGP runs on ips 1.1.1.1, 2.2.2.2, and 3.3.3.3 on each peer.
For underlay routing add the remote peer routes via DPS interface and statically add an ARP entry for remote peer. In future versions of EOS the underlay routing also be handled by BGP.
Example
switch(config)#ip route 2.2.2.2/32 ethernet 100
switch(config)#ip route 3.3.3.3/32 ethernet 100
switch(config)#arp 2.2.2.2 00:00:33:02:00:00 arpa
switch(config)#arp 3.3.3.3 00:00:33:03:00:00 arpa
The above configuration makes the peers reachable via DPS.Applying Policies for Path Groups
The policies for the path groups are applied on all the paths in the group. The following policy is supported:
Encrypting Path-Group
Applying IPsec to the group will enable encryption on all the paths in the group as per the applied IPsec profile. This policy is used to encrypt all Internet paths. This configuration simplifies IPsec configuration as the customer does not have to specify what traffic to encrypt.
Syntax
path-group <name> ipsec profile <ipsec-profile-name> Applying IPsec profile will cause all the paths in the path group to be encrypted based on the algorithms and authentication mechanisms as per the profile.
Configuring Load Balancing Profile
Load balancing policy is configured under router path-selection as shown.
Syntax
router path-selection load-balance policy <name> latency <milliseconds> jitter <milliseconds> loss-rate <0.00-100.00 percentage> path-group <group-name> [ priority <number>] path-group <group-name> The commands are explained in the following subsections.
Specifying Path Groups to the Load Balancer
Syntax
router path-selection load-balance policy <name> path-group <group-name> path-group <group-name>
When multiple path-groups are specified flows are load balanced across all the paths in the specified path-groups.
Example
switch(config)#router path-selection
switch(config-dynamic-path-selection)#load-balance policy best-effort
switch(config-load-balance-policy-best-effort)#path-group mpls
switch(config-load-balance-policy-best-effort)#path-group internetSpecifying Constraints for Path Selection
Syntax
router path-selection load-balance policy <name> latency <milliseconds> jitter <milliseconds> loss-rate <0.00-100.00 percentage>
Latency, jitter and loss-rate constraints can be specified for path selection. There can be more than one path that meets the constraints in which case the flows are load balanced across all the selected paths. All constraints need to be met. If none of the paths meet the constraints, then the path with the lowest loss rate is chosen as the best path.
Example
switch(config-path-selection)#load-balance policy voice
switch(config-load-balance-policy-voice)#path-group mpls
switch(config-load-balance-policy-voice)#path-group internet
switch(config-load-balance-policy-voice)#latency 50
switch(config-load-balance-policy-voice)#loss-rate 1In this case, the traffic is load balanced across all the paths that meet the constraints. If none matches then the traffic is sent to the best path.
Specifying Preference to a Path-Group
Syntax
router path-selection load-balance policy <name> path-group <group-name> [ priority <number>] path-group <group-name>
Preference can be specified for path-groups. Flows are load balanced based on path group priority. The lower the number the higher the priority is given to the path group. If not specified, default policy is 1 (highest). If multiple path groups in the same load-balance profile have same priority traffic will be load balanced among them. If no paths in a path-group are available then paths from the next lower priority is considered. Paths may not be available because of the following reasons:
- Interface is down
- Route is not resolved
- Path keepalives have failed
- Specified constraints for the load balancing policy is not met
Example
switch(config-dynamic-path-selection)#load-balance policy voice
switch(config-load-balance-policy-voice)#path-group mpls
switch(config-load-balance-policy-voice)#path-group internetWhen MPLS path is down then all the existing flows are forwarded through Internet paths. When MPLS path is up again, all the new flows are forwarded through MPLS paths.
Classification - Application Profiles
The existing commands in EOS are as shown below.
Syntax
application traffic recognition application ipv4 http-8080 { protocol <proto> [ destination-port { <port_num> | <port-range> } ] } protocol tcp destination-port 8080 protocol tcp destination-port 8000 application ipv4 app2-service protocol tcp destination-port 8001-8080
Applications is specified either with custom signatures specified using the application configuration as shown above or can be imported from a DPI engine. Application configuration might have to be extended to address the path-selection use case.
Syntax
Applications can be grouped and other attributes like the traffic class can be specified using application-profile as below.
application traffic recognition application-profile <app-xyz> application <app-name-1> application <app-name-2>
Example
switch(config)#application traffic recognition
switch(config-app-recognition)#application-profile gold
switch(config-app-profile-gold)#application voice
switch(config-app-profile-gold)#traffic-policies
“bronze” profile for best effort
switch(config-app-recognition)#application-profile bronze
switch(config-app-profile-bronze)#application best-effort
switch(config-app-profile-bronze)#traffic-policiesPath Selection Policy
The load balancing policy can be specified based on the application.
Syntax
router path-selection policy <dps-policy-name> <rule key> application-profile <profile-name> load-balance <load balance policy name><rule key> application-profile <profile-name> load-balance <load balance policy name>
Sequence numbers are required since a flow can potentially match multiple application profiles. Also, we have “set load-balance” as a sub-mode so we can add other actions for “match application-profile”.
switch(config)#router path-selection
switch(config-dynamic-path-selection)#policy dynamic
switch(config-policy-dynamic)#10 application-profile voice
switch(config-policy-rule-key-10-dynamic)#load-balance voice
switch(config-policy-rule-key-10-dynamic)#20 application-profile best
switch(config-policy-rule-key-20-dynamic)#load-balance bestApplying Path Selection Policy
All traffic going from site to site will go through VTI interfaces and is VXLAN encapsulated. Different classification and path selection policies are specified for each VRF. For example, the test VRF can have simple application classification and load balancing policy.
Syntax
router path-selection vrf <vrf-name> path-selection-policy <policy-name>
VRF “all” can be specified to apply policy on all VRFs. In case both “all” and per VRF policy is specified, only the per VRF policy is applied.
The policy (classification and load balancing) needs to be applied to the datapath once it is determined that traffic is going from site to site. This is done to avoid the classification overhead for LAN to LAN traffic. When policy is applied on a VRF it is actually applied on the egress direction on the hidden SVI interface for the VTI (VXLAN tunnel interface). If there is no VTI configured then this policy is ignored.
When policy is applied on a VRF it is actually applied on the egress direction on the hidden SVI interface for the VTI (VXLAN tunnel interface) as shown below. If there is no VTI configured then this policy is ignored.
switch(config)#router path-selection
switch(config-dynamic-path-selection)#vrf red
switch(config-vrf-red)#path-selection-policy production
switch(config-vrf-red)#Path Telemetry UDP Port
By default, the path telemetry protocol uses 4793 as the destination UDP port number for encapsulation purpose. The below command is used to configure the UDP port for DPS.
Syntax
router path-selection encapsulation path-telemetry udp port <number>
switch(config)#router path-selection
switch(config-dynamic-path-selection)#encapsulation path-telemetry udp port 4794Complete Path Selection Configuration Example
switch#application traffic recognition
switch(config-app-recognition)#application-profile platinum
switch(config-app-profile-platinum)#application voice
switch(config-app-profile-platinum)#application skype-voice
switch(config-app-profile-platinum)#application-profile bronze
switch(config-app-profile-bronze)#application scp
switch(config-app-profile-bronze)#application ftp
switch(config-app-profile-bronze)#router path-selection
switch(config-dynamic-path-selection)#path-group mpls
switch(config-path-group-mpls)#local interface et1
switch(config-path-group-mpls)#peer static router-ip 10.2.2.2
switch(config-peer-router-ip-10.2.2.2-mpls)#ipv4 address 172.16.2.1
switch(config-peer-router-ip-10.2.2.2-mpls)#path-group internet
switch(config-path-group-internet)#local interface et2
switch(config-path-group-internet)#local interface et3
switch(config-path-group-internet)#peer static router-ip 10.2.2.2
switch(config-peer-router-ip-10.2.2.2-internet)#ipv4 address 3.3.3.3
switch(config-peer-router-ip-10.2.2.2-internet)#ipv4 address 4.4.4.4
switch(config-dynamic-path-selection)#load-balance policy voice
switch(config-load-balance-policy-voice)#latency 50
switch(config-load-balance-policy-voice)#path-group mpls
switch(config-load-balance-policy-voice)#path-group internet priority 2
switch(config-load-balance-policy-voice)#load-balance policy best-effort
switch(config-load-balance-policy-best-effort)#path-group mpls
switch(config-load-balance-policy-best-effort)#path-group internet
switch(config-load-balance-policy-best-effort)#load-balance policy default
switch(config-load-balance-policy-default)#path-group internet
switch(config-load-balance-policy-default)#policy dynamic
switch(config-policy-dynamic)#10 application-profile platinum
switch(config-policy-rule-key-10-dynamic)#load-balance voice
switch(config-policy-rule-key-10-dynamic)#20 application-profile bronze
switch(config-policy-rule-key-20-dynamic)#load-balance best-effort
switch(config-dynamic-path-selection)#policy dynamic
switch(config-policy-dynamic)#interface ethernet 100
switch(config-if-Et100)#no switchport
switch(config-if-Et100)#ip address 11.0.0.1/24
switch(config-if-Et100)#interface loopback 0
switch(config-if-Lo0)#ip address 10.1.1.1/32
switch(config-if-Lo0)#interface vxlan 1
switch(config-if-Vx1)#vxlan source-interface loopback 0
switch(config-if-Vx1)#vxlan udp-port 4789
switch(config-if-Vx1)#vxlan vrf vrf1 vni 100
switch(config-if-Vx1)#ip route 10.2.2.2/32 ethernet 100
switch(config)#arp 10.2.2.2 00:00:33:02:00:00 arpa
switch(config)#Site-1
switch(config)#router path-selection
switch(config-dynamic-path-selection)#path-group 1
switch(config-path-group-1)#local interface ethernet 5
!
switch(config-path-group-1)#peer static router-ip 22.22.22.22
switch(config-peer-router-ip-22.22.22.22-1)#ipv4 address
8.0.1.5
!
switch(config-peer-router-ip-22.22.22.22-1)#load-balance
policy policy-1
switch(config-load-balance-policy-policy-1)#path-group 1
!
switch(config-load-balance-policy-policy-1)#policy policy-1
switch(config-policy-policy-1)#default-match
switch(config-policy-default-rule-policy-1)#load-balance
policy-1
!
switch(config-policy-default-rule-policy-1)#vrf default
switch(config-vrf-default)#path-selection-policy policy-1
!
switch(config-dynamic-path-selection)#vrf et1
switch(config-vrf-et1)#path-selection-policy policy-1
!
switch(config-vrf-et1)#vrf instance et1
switch(config-vrf-et1)#interface ethernet 1
switch(config-if-Et1)#description LAN-interface
switch(config-if-Et1)#no switchport
switch(config-if-Et1)#ip address 4.0.1.5/24
!
switch(config)#vrf instance et1
switch(config-vrf-et1)#interface ethernet 1
switch(config-if-Et1)#description LAN-interface
switch(config-if-Et1)#no switchport
switch(config-if-Et1)#ip address 4.0.1.5/24
!
switch(config-if-Et1)#interface ethernet 5
switch(config-if-Et5)#description WAN-Interface
switch(config-if-Et5)#no switchport
switch(config-if-Et5)#ip address 5.0.1.5/24
!
switch(config-if-Et5)#interface ethernet 100
switch(config-if-Et100)#no switchport
switch(config-if-Et100)#ip address 10.0.0.2/24
!
switch(config-if-Et100)#interface loopback 1
switch(config-if-Lo1)#ip address 11.11.11.11/32
!
switch(config-if-Lo1)#interface vxlan 1
switch(config-if-Vx1)#vxlan source-interface loopback 1
switch(config-if-Vx1)#vxlan udp-port 4789
switch(config-if-Vx1)#vxlan vrf et1 vni 5
!
switch(config-if-Vx1)#ip route 22.22.22.22/32 ethernet 100
!
switch(config)#arp 22.22.22.22 22:22:22:22:22:22 arpa
!
switch(config)#ip routing
switch(config)#ip routing vrf et1
!
switch(config)#router bgp 32
switch(config-router-bgp)#neighbor 5.0.1.1 remote-as 501
switch(config-router-bgp)#neighbor 5.0.1.1 maximum-routes
12000
switch(config-router-bgp)#neighbor 22.22.22.22 remote-as 43
switch(config-router-bgp)#neighbor 22.22.22.22 update-source
loopback 1
switch(config-router-bgp)#neighbor 22.22.22.22 ebgp-multihop
switch(config-router-bgp)#neighbor 22.22.22.22 send-community
extended
switch(config-router-bgp)#neighbor 22.22.22.22 maximum-routes
12000
switch(config-router-bgp)#redistribute static
!
switch(config-router-bgp)#address-family evpn
switch(config-router-bgp-af)#neighbor 22.22.22.22 activate
!
switch(config-router-bgp-af)#exit
switch(config-router-bgp)#address-family ipv4
switch(config-router-bgp-af)#no neighbor 22.22.22.22 activate
switch(config-router-bgp-af)#exit
!
switch(config)#router bgp 32
switch(config-router-bgp)#vrf et1
switch(config-router-bgp-vrf-et1)#rd 4.0.1.5:0
switch(config-router-bgp-vrf-et1)#route-target import evpn
9.0.1.5:0
switch(config-router-bgp-vrf-et1)#route-target export evpn
4.0.1.5:0
switch(config-router-bgp-vrf-et1)#router-id 4.0.1.5
switch(config-router-bgp-vrf-et1)#network 4.0.1.0/24
switch(config-router-bgp-vrf-et1)#network 50.0.0.0/24
switch(config-router-bgp-vrf-et1)#exit
switch(config-router-bgp)#exit
switch(config)#
---------------------------------------------------------------------------------
Site-2
switch(config)#router path-selection
switch(config-dynamic-path-selection)#path-group 1
switch(config-path-group-1)#local interface ethernet 1
!
switch(config-path-group-1)#peer static router-ip 11.11.11.11
switch(config-peer-router-ip-11.11.11.11-1)#ipv4 address
5.0.1.5
!
switch(config-peer-router-ip-11.11.11.11-1)#load-balance
policy policy-1
switch(config-load-balance-policy-policy-1)#path-group 1
!
switch(config-load-balance-policy-policy-1)#policy policy-1
switch(config-policy-policy-1)#default-match
switch(config-policy-default-rule-policy-1)#load-balance
policy-1
!
switch(config-policy-default-rule-policy-1)#vrf default
switch(config-vrf-default)#path-selection-policy policy-1
!
switch(config-dynamic-path-selection)#vrf et5
switch(config-vrf-et5)#path-selection-policy policy-1
!
switch(config-vrf-et5)#vrf instance et5
switch(config-vrf-et5)#interface ethernet 1
switch(config-if-Et1)#description WAN-Interface
switch(config-if-Et1)#no switchport
switch(config-if-Et1)#ip address 8.0.1.5/24
!
switch(config)#vrf instance et5
switch(config-vrf-et5)#interface ethernet 5
switch(config-if-Et5)#description LAN-interface
switch(config-if-Et5)#no switchport
switch(config-if-Et5)#ip address 9.0.1.5/24
!
switch(config-if-Et5)#interface ethernet 100
switch(config-if-Et100)#no switchport
switch(config-if-Et100)#ip address 10.0.0.1/24
!
switch(config-if-Et100)#interface loopback 1
switch(config-if-Lo1)#ip address 22.22.22.22/32
!
switch(config-if-Lo1)#interface vxlan 1
switch(config-if-Vx1)#vxlan source-interface loopback 1
switch(config-if-Vx1)#vxlan udp-port 4789
switch(config-if-Vx1)#vxlan vrf et5 vni 5
!
switch(config-if-Vx1)#ip route 11.11.11.11/32 ethernet 100
!
switch(config)#arp 11.11.11.11 11:11:11:11:11:11 arpa
!
switch(config)#ip routing
switch(config)#ip routing vrf et5
!
switch(config)#router bgp 43
switch(config-router-bgp)#maximum-paths 16
switch(config-router-bgp)#neighbor 8.0.1.1 remote-as 701
switch(config-router-bgp)#neighbor 8.0.1.1 maximum-routes
12000
switch(config-router-bgp)#neighbor 11.11.11.11 remote-as 32
switch(config-router-bgp)#neighbor 11.11.11.11 update-source
loopback 1
switch(config-router-bgp)#neighbor 11.11.11.11 ebgp-multihop
switch(config-router-bgp)#neighbor 11.11.11.11 send-community
extended
switch(config-router-bgp)#neighbor 11.11.11.11 maximum-routes
12000
!
switch(config-router-bgp)#address-family evpn
switch(config-router-bgp-af)#neighbor 11.11.11.11 activate
switch(config-router-bgp-af)#exit
!
switch(config-router-bgp)#address-family ipv4
switch(config-router-bgp-af)#no neighbor 11.11.11.11 activate
switch(config-router-bgp-af)#exit
!
switch(config)#router bgp 40
switch(config-router-bgp)#vrf et5
switch(config-router-bgp-vrf-et5)#rd 9.0.1.5:0
switch(config-router-bgp-vrf-et5)#route-target import evpn
4.0.1.5:0
switch(config-router-bgp-vrf-et5)#route-target export evpn
9.0.1.5:0
switch(config-router-bgp-vrf-et5)#router-id 9.0.1.5
switch(config-router-bgp-vrf-et5)#network 9.0.1.0/24
switch(config-router-bgp-vrf-et5)#network 51.0.0.0/24
switch(config-router-bgp-vrf-et5)#exit
switch(config-router-bgp)#exit
switch(config)#DPS Display Commands
The following show commands are used to verify the various information of the Dynamic Path Selection application.
Path Telemetry Show Commands
These two show commands provide path telemetry status:
show monitor telemetry path characteristics [ detail ][ destination DSTIP ][ path-name NAME ][ peer PEERIP ] [ source SRCIP ] [ traffic-class TC ]
show monitor telemetry path counters [ detail ][ destination DSTIP ][ path-name NAME ][ peer PEERIP ] [ source SRCIP ][ traffic-class TC ]
Example
- The show monitor telemetry path characteristics command displays the path state, latency, jitter, and other information.
switch#show monitor telemetry path characteristics PathName TrafficClassTxStateLatency(ms)Jitter(ms)Throughput(Mbps)LossRate(%) path10 active 3.520 1.12210.000.01 path20 active 35.2202.33010.001.01 switch#show monitor telemetry path characteristics detail Peer: 10.1.10.5 PathName: path1 Source: 156.142.20.23, Destination: 156.142.40.21 Traffic Class: 0 TxState: active Latency: 3.520 ms Jitter:1.122 ms Throughput: 10.00 Mbps LossRate: 0.01 % PathName: path2 Source: 156.142.20.24, Destination: 156.142.40.22 Traffic Class: 0 TxState: active Latency: 35.220 ms Jitter:2.330ms Throughput: 1000 Mbps LossRate: 1.01 %
- The show monitor telemetry path counters displays the input output bytes and packets and flow information.
switch#show monitor telemetry path counters PathName TrafficClassInBytesInPktsInPktsDropOutBytesOutPktsOutPktsDrop path10 455330010220 5341333 7520 path20 455330010220 5341333 7520 kvs17-b10#show monitor telemetry path counters detail Peer: 10.1.10.5 PathName: path1 Source: 156.142.20.23, Destination: 156.142.40.21 Traffic Class: 0 InBytes: 4553300 InPkts: 1022 InPktsDrop: 0 OutBytes: 5341333 OutPkts: 752 OutPktsDrop: 0
Both path characteristics and path counters show results can be filtered by path name, destination IP, source IP, remote IP and traffic class. And both of them have detail version output and brief version output, default version is brief version as shown.
IPsec Show Commands
The following IPsec show commands filter IPsec connections based on path name and remote IP address. The IPsec show results are filtered using the following options like Tunnel, Detail, Path, and VRF.
- The show ip security connection path command displays all path based IP security connections.
switch#show ip security connection path NameSource Dest Status Uptime InputOutput Rekey Time Path1 ip1ip3Established22 minutes 0 bytes0 bytes34 minutes 0 pkts 0 pkts Path2 ip2ip3Established22 minutes 0 bytes0 bytes34 minutes 0 pkts 0 pkts Path2 ip5ip6Established22 minutes 0 bytes0 bytes34 minutes 0 pkts 0 pkts
- The show ip security connection path name command displays IPsec path connections based on the path name.
switch#show ip security connection path name path1 NameSource Dest Status Uptime InputOutput Rekey Time Path1 ip1ip3Established22 minutes 0 bytes0 bytes34 minutes 0 pkts 0 pkts
- Theshow ip security connection path peer command displays the IPsec path connections based on the remote router IP.
switch#show ip security connection path peer ip3 NameSource Dest Status Uptime InputOutput Rekey Time Path1 ip1ip3Established22 minutes 0 bytes0 bytes34 minutes 0 pkts0 pkts Path2 ip2ip3Established22 minutes 0 bytes0 bytes34 minutes 0 pkts0 pkts
Load balance and Application Counters
These counters display the statistics of load balancing based on application profile, overlay VRF and remote node IP:
show path-selection load-balance counter [ detail ] [ application-profile APPNAME ] [ peer PEERIP ] [ vrf VRFNAME]
show path-selection application counters[ application-profile APPNAME ] [ peer PEERIP ] [ vrf VRFNAME ]
- The show path-selection load-balance counter command displays for every ( application profile, overlay VRF and remote IP ), per path group flow count and the throughput of path group.
switch#show path-selection load-balance counters AppProfileVrfPeer PathGroupPath FlowsThroughput(Mbps) app1vrf1 11.0.1.1 transit0 path200.00 app2vrf1 11.0.1.1 transit1 path100.00 default_app default11.0.1.1 transit0 path200.00 transit1 path100.00
- The show path-selection load-balance counters detail command displays for every ( application profile, overlay VRF and remote IP ), per path group flow count, out bytes, out packets and the throughput of path group.
switch#show path-selection load-balance counters detail AppProfileVrfPeer PathGroupPath FlowsThroughput(Mbps) OutBytes OutPkts app1vrf1 11.0.1.1 transit0 path200.00 00 app2vrf1 11.0.1.1 transit1 path100.00 00 default_app default11.0.1.1 transit0 path200.00 1052 17 transit1 path100.00 1321 17
- The show path-selection application counters command displays the application profile, overlay VRF and remote IP out bytes, out packets and throughput.
switch#show path-selection application counters AppProfile VRF PeerThroughput OutBytes OutPackets SilverRed 10.0.0.1153000 15
Output of both show path-selection load-balance counters and show path-selection application counters can be filtered by application-profile name, peer IP address and vrf name.
Clear Commands
The following commands clears the DPS related counters:
Syntax
Clear load balancing and application counters:
clear path-selection counters Clear path telemetry counters:
clear monitor telemetry path counters
Troubleshooting
In order for DPS to work, the following needs to be working.
- Verify the paths are in the “Estab” or “Estab IPSec” state using “show path-selection paths” command. If the path is not in established state.
- ARP Pending - Make sure the next-hop to the path destination IP is available.
- Route Pending - Make sure a route to the path destination IP is available through the local interface for the path.
- IPSec Pending - Check IPSec connection with “show ip security connection” or other IPSec related commands between the path’s local interface and the path’s destination.
- If the paths are in Estab state, verify the paths are active and available using “show monitor telemetry path characteristics”
- If a path is inactive, make sure IP connectivity is working between the path’s source IP/interface and destination IP. Ping the path destination with the path source IP could be one of the ways to verify this. And also, to check the configuration and make sure that the paths are configured symmetrically on both sites.
- Check and make sure there are DPS communications between the source and destination IPs using TCP dump on et100.
- Paths are active but ping between loopbacks of the two sites is not working. Loopbacks should be reachable through overlay.
- Check your interface VXLAN1 configuration.
- Check and make sure you have applied a policy with default match to your “vrf default” configuration in DPS.
- Site-to-Site loopback IPs are reachable but data traffic is not going through.
- Check your EVPN configuration. Make sure the remote routes are in your VRF route table of your sites.
- Make sure your DPS configuration has proper policy, application profile, default match and load-balance profile
Limitations
- DPS currently supports VNI number from 1 to 255 only.
- The same interface cannot be configured as a local interface in different DPS path groups.
- DPS WAN interfaces/local interfaces configured in path groups should be in default VRF.
- DPS does not work with port translation currently.