End of Support

LDP Entropy Label

If a network device uses deep packet inspection for load balancing, it is recommended to use entropy label in LDP to improve load balancing in MPLS networks by providing sufficient entropy in the label stack itself.

Configuring LDP Entropy Label

Use the entropy-label command to enable the MPLS LDP entropy labelling on the switch. Use the no form of the command to disable the MPLS LDP entropy labelling. By default, the entropy label knob is disabled.

switch(config)# mpls ldp
switch(config-mpls-ldp)# entropy-label
switch(config-mpls-ldp)# exit

Show Commands

The show running-config all command displays the Entropy Label Signaling status.

switch# show running-config all
!
mpls ldp
   ...
   no entropy-label
   ...
!

The show mpls ldp bindings detail command displays whether the ELC is advertised or not for a given FEC for both local/remote bindings.

switch# show mpls ldp bindings 1.1.1.1/32 detail
1.1.1.1/32
     Local binding:    Label: imp-null, ELC: False 
          Uptime:   1:51:14
          Advertised to: 2.2.2.2:0, 3.3.3.3:0
     Remote binding: Peer ID: 2.2.2.2:0, Label: 100001, ELC: True
          Uptime:   1:51:57
     Remote binding: Peer ID: 3.3.3.3:0, Label: 100002, ELC: True
          Uptime:   1:52:05

Following debug commands are modified to incorporate Entropy label and Entropy label indicator information in label stack for debugging on ingress router.

switch> show platform fap fec <idx>
Tunnel Type: Mpop(mpls pop), Mpush(mpls push), Mswap(mpls swap),
             MoG(mpls-over-gre), T(IPv4 tunnels GRE/VXLAN)
 'CW  - Control word',
 'FL  - Flow label',
 'EL  - Entropy label',
 'ELI - Entropy label indicator',
 'D   - ECMP is divergent across switching chips',
 ' -----------------------------------------------------------------------------------',
 '|                                 FEC Entry                                         |',
 '|-----------------------------------------------------------------------------------',
 '|     |      |     |                    |     |       |                   |',
 '| ECMP|  FEC |     |                    |     |       |                   | Tunnel',
 '|Index| Index| Cmd |     Destination    | VID |Outlif |   MAC / CPU Code  |T Value',
 ' ----------------------------------------------------------------------------------',
 '|  -  |353896|ROUTE| Et1/1              |0    |20480  | 00:00:00:00:00:01 |Mpush 501 401 301 ELI EL 201'

B. show platform fap ip route
Tunnel Type: M(mpls), G(gre), MoG(mpls-over-gre),
             VXLAN-o(VXLAN outer-rewrite info), VXLAN-i(VXLAN inner-rewrite info)
 'CW  - Control word',
 'FL  - Flow label',
 'EL  - Entropy label',
 'ELI - Entropy label indicator',
 '*   - Routes in LEM',
 'D   - ECMP is divergent across switching chips',
 ' ---------------------------------------------------------------------------------------------------------',
 '|                                 Routing Table                                           |              |',
 '|---------------------------------------------------------------------------------------------------------',
 '|VRF|   Destination    |     |                    |     |       |                   | ECMP|  FEC | Tunnel',
 '| ID|      Subnet      | Cmd |     Destination    | VID |Outlif |   MAC / CPU Code  |Index| Index|T Value',
 ' ---------------------------------------------------------------------------------------------------------',
 '|0  |192.168.1.3/32    |ROUTE| Et1/2              |100  |8188   | 00:00:00:00:00:02 |  -  |353897|   -   ',
 '|0  |200.0.0.0/24      |ROUTE| Et1/2              |100  |20484  | 00:00:00:00:00:02 |  -  |353898|M 501 401 301 ELI EL 201',
 '|0  |0.0.0.0/0         |TRAP | CoppSystemL3LpmOver|0    |  -    | SlowReceive       |  -  |576815|   -   ',

Limitations

When the Transit LSR is acting as a PHP hop for a given LDP tunnel with entropy label pushed onto it by an Ingress LSR, only the LDP tunnel label will be popped at PHP whenever the Egress LSR has signaled implicit-null in LDP.
Egress LSR should be capable of handling ELI as top label in the label stack like [ ELI, EL, VPN, Payload ].

BGP PIC Edge for EVPN VXLAN Routes for Remote VTEP Failures

When a remote VTEP goes down, the IGP and BGP must recompute a new best path traffic destined to affected BGP prefixes originally reachable by the problematic VTEP. Currently, the BGP PIC is restricted to locally identifiable failures such as link failures.

To overcome such VTEP failure issues, support for EVPN-learned VTEPs improve convergence times in these scenarios by tying the liveness detection provided by the BFD sessions into existing BGP PIC support for software fast-failover. Upon detecting that a BFD session to a remote VTEP has gone down, the hardware forwarding agents will update the affected adjacencies before the corresponding underlay route has been removed from the FIB which can improve convergence times.

The diagram above outlines a scenario in which CE-2 is sending traffic bound for 10.10.10.0/24 via PE-12 to PE-11 to CE-1. PE-11 goes down, however we have BFD sessions from PE-21 to the remote VTEPs of PE-11 and PE-12 and, therefore detect that it goes down and quickly update the forwarding to send the traffic along the pre-computed backup path via PE-12 to CE-1.

Configuring BGP PIC Edge for EVPN VXLAN Routes for Remote VTEP Failures

Use the bfd vtep evpn command to configure the BGP PIC Edge for EVPN VXLAN routes for remote VTEPs. This command is configured under the VXLAN Tunnel Interface (VTI).

switch# config
switch(config)# interface VXLAN1
switch(config-if-Vx1)# bfd vtep evpn interval <interval> min-rx <min-rx> multiplier <multiplier>

This configuration uses the specified timer values to initiate BFD sessions for all VTEPs learned through EVPN VXLAN for this VTI.

interval – Transmit rate in milliseconds
min-rx – Expected minimum incoming rate in milliseconds
multiplier – BFD multiplier

Example

switch(config-if-Vx1)# bfd vtep evpn interval 100 min-rx 100 multiplier 3

In this example (assuming symmetric configuration on other PE devices), any BFD for VXLAN session initiated on the VTI would have a detect time of 300ms (interval of 100ms multiplied by 3).

To utilize these BFD sessions, traffic must have an alternate path tin the event that the session goes down. This would include other paths in an ECMP group or a backup path.

As mentioned, by default the above configuration will initiate BFD sessions for all VTEPs learned through EVPN VXLAN for the VTI.

switch# config
switch(config)# interface VXLAN1
switch(config-if-Vx1)# bfd vtep evpn prefix-list <PREFIX-LIST>

This command uses a supplied prefix list to filter and select the candidate VTEPs. By default, an empty prefix list will act as a deny-all and not initiate BFD sessions with any learned VTEPs.

Show Commands

The show interface <VTI> command is used to view if BFD is enabled on the VTI and to see the timers used for any of the BFD sessions or any prefix-list configured for filtering BFD sessions.

switch# show interface VXLAN1
VXLAN1 is up, line protocol is up (connected)
  Hardware is VXLAN
  Source interface is Loopback0 and is active with 10.1.1.1
  Replication/Flood Mode is headend with Flood List Source: CLI
  Remote MAC learning is disabled
  VNI mapping to VLANs
  Static VLAN to VNI mapping is
  Dynamic VLAN to VNI mapping for 'evpn' is
    [4092, 30000]     [4093, 20000]
  Dynamic VLAN to VNI mapping for 'vccbfd' is
    [4091, 0]
  Note: All Dynamic VLANs used by VCS are internal VLANs.
        Use 'show VXLAN vni' for details.
  Static VRF to VNI mapping is
   [vrf0, 20000]
  MLAG Shared Router MAC is 0000.0000.0000
  BFD is enabled with transmit interval 50, receive interval 50, multiplier 3, VTEP prefix list pl-example

The existing show bfd peers command is used to view the state of the BFD for VXLAN sessions.

switch# show bfd peers
VRF name: default
-----------------
DstAddr    MyDisc       YourDisc    Interface/Transport    Type        LastUp           LastDown       LastDiag       State
-------- ----------  ------------- ---------------------- ---------  --------------  ------------    -------------- -----------
10.1.1.2   1965370229   3607849318                  NA       VXLAN    01/12/21 10:45          NA       No Diagnostic       Up
10.1.1.3   1355343148   2407539267                  NA       VXLAN    01/12/21 10:45          NA       No Diagnostic       Up

The show ip route <VRF> command can be used to determine which prefixes are eligible for fast-failover.

switch# show ip route vrf example-vrf

VRF: example-vrf
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route, L - VRF Leaked,
       RC - Route Cache Route

Gateway of last resort is not set

 C        20.0.2.0/24 is directly connected, Ethernet14/1
 B E      99.99.0.0/24 [200/0] via VTEP 10.1.1.2 VNI 30000 router-mac fc:bd:67:3d:21:fd
                               via VTEP 10.1.1.3 VNI 30000 router-mac ba:ed:43:3f:ca:8e backup

In the above example there is a prefix with the primary path using a VXLAN tunnel to VTEP 10.1.1.2 and has a backup VXLAN tunnel to VTEP 10.1.1.3. Both paths are monitored via BFD.

In other use cases the prefix may have multiple paths with ECMP in which one or multiple of the paths are VXLAN tunnels to remote VTEPs monitored by these BFD for VXLAN sessions.

switch# show ip route vrf example-vrf

VRF: example-vrf
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route, L - VRF Leaked,
       RC - Route Cache Route

Gateway of last resort is not set

 C        20.0.2.0/24 is directly connected, Ethernet14/1
 B E      99.99.0.0/24 [200/0] via VTEP 10.1.1.2 VNI 30000 router-mac fc:bd:67:3d:24:fe
                               via VTEP 10.1.1.3 VNI 30000 router-mac ba:ed:43:3f:ca:8e

MLAG

This feature applies only to the scenario when remote prefix is known via two different MLAG VTEP pairs.

Prior to 4.26.0F this feature is only supported on the primary switch of an MLAG pair due to the use of Shared VTEP IP within MLAG pair as VXLAN tunnel source/destination. If BFD for VXLAN packets are received on the secondary MLAG switch, they will be forwarded to the primary MLAG switch for processing. Because only the primary MLAG switch will have BFD state for remote VTEPs, if a BFD session to a remote VTEP goes down only the primary MLAG switch will perform the fast-failover, while the secondary MLAG switch will retain current behavior. Therefore, it is not recommended to use this feature in conjunction with MLAG.

As of 4.26.0F, the primary MLAG switch will sync its BFD for VXLAN state to the secondary MLAG switch to allow the secondary to failover to an alternate path as well. To view the synced state, a new show command has been added.

switch# show bfd peers protocol VXLAN mlag primary 
Remote VTEPS for VXLAN1 on MLAG primary:
VTEP         BFD Status 
----------  ---------- 
10.1.1.2       up
10.1.1.3       up

However, because this state is synced across devices, the secondary MLAG switch will not be as performant in reacting to the BFD state transitions as the primary MLAG switch, which is natively responding to the BFD session.

Another exception is the multi-VTEP MLAG feature, which allows BFD for VXLAN to run on the secondary MLAG switch. When running with multi-VTEP MLAG both the primary and secondary switches will run independent BFD sessions to remote VTEPs and react to BFD state transitions separately. Each switch will use the local VTEP IP of the VTI as the source IP address for the BFD sessions, which must differ from the MLAG VTEP IP.

In summary, it is only recommended to use MLAG with this feature if configured with the multi-VTEP IP feature referenced above.

Troubleshooting

Ensure that BFD configuration is present on the relevant VTI and that the VTI status shows BFD as enabled using the mentioned show interface <VTI> command.
As mentioned in the prior section, BFD state transitions are Syslogged and will display if a BFD session to a remote VTEP goes down.
Upon fast-failover to a separate path, show ip route will still display FIB state that may display the original path. To view the post failover prefix state, show ip hardware ale vrf <VRF> <prefix> can be used instead.

Limitations

Support is limited to EVPN VXLAN.
IPv6 VXLAN underlay with this feature is not supported.

VXLAN DSCP Mapping

VXLAN DSCP Mapping allows the selection of Differentiated Services Code Point DSCP and Traffic Class TC values for packets at VTEPs ( VXLAN Tunnel Endpoint) along VXLAN encapsulation and decapsulation directions respectively.

An incoming packet from an edge port is encapsulated with a new IP and VXLAN header before being sent to a remote VTEP via a core facing port for encapsulation direction. When this is enabled, the DSCP field of the outgoing packet is set as the value of the DSCP of the incoming packet at the edge port.

An incoming VXLAN packet on a core port is decapsulated and forwarded via an edge port for decapsulation direction. When this is enabled, the TC value is derived from the value of the DSCP of the incoming VXLAN encapsulated packet at the core port. This is applicable for both VXLAN Bridging and Routing.

Platform Compatibility

The following platforms support VXLAN DSCP mapping:

DCS-7050SX3-48YC12
DCS-7050CX3M-32S
DCS-7050CX3-32S

Configuration

The following command enables the propagation of the DSCP field from the incoming packet to the VXLANencapsulated packet.

switch(config)# interface VXLAN 1
switch(config-if-Vx1)# VXLAN qos dscp propagation encapsulation

The following command enables the assignment of the TC value during decapsulation of a VXLAN packet.

switch(config)# interface VXLAN 1
switch(config-if-Vx1)# VXLAN qos dscp to traffic-class decapsulation

Show Commands

The show VXLAN qos command shows the VXLAN DSCP and TC configuration.

switch# show VXLAN qos 
VXLAN ECN Propagation is Disabled.
Outer header DSCP is derived from Inner DSCP.
Traffic Class is derived from Outer DSCP.

Limitation

The core ports must be in DSCP Trust mode to derive TC on the outer DSCP field of the incoming VXLAN packet.

EVPN E-Tree for MPLS and VXLAN

As part of an L2 EVPN service, E-Tree assigns the role of root or leaf to each attachment circuit (AC). After assigning the role, E-Tree enforces the following rules:

Root ACs can communicate with Leaf ACs and other Root ACs.
Leaf ACs can only communicate with Root ACs, and traffic between Leaf ACs blocked.

Configure the ACs at the VLAN level and enforce forwarding rules using a combination of locally configured leaf VLANs for local hosts and asymmetric route targets for remote hosts.

E-Tree supports the E-Tree extended community for EVPN Type-2 and Type-3 routes and allows E-Tree operation with only a single route-target configured, a regular non-E-Tree EVPN deployment. This support also allows EVPN MAC mobility to perform correctly in an E-Tree environment. Only deployments that use VXLAN encapsulation support the E-Tree extended community, and deployments using MPLS encapsulation must use asymmetric route targets.

When implementing a single route-target E-Tree, EOS supports configurable handling of remote leaf hosts in local leaf VLANs. By default, the local FDB for local leaf VLANs does not install remote local leaf hosts. Instead, a per-VLAN configuration allows the installation of explicit drop routes for remote leaf hosts, which removes unnecessary BUM traffic at the cost of local FDB space.

E-Tree Extended Community

The E-Tree extended community automatically attaches to all EVPN type-2 and type-3 routes advertised by a device associated with a VLAN locally configured as an E-Tree leaf AC. All type-2 routes that represent hosts learned on a leaf VLAN advertise the E-Tree extended community with a leaf flag set attached. All type-3 routes that represent a VLAN configured as a leaf AC advertise the E-Tree extended community with the leaf flag set attached.

After receiving an EVPN type-2 route with the E-Tree extended community flag set attached, a device checks the E-Tree role of the local VLAN to import the routes. If configured as an E-Tree root AC, the local floodlist for the VLAN adds the remote VTEP contained in the type-3 route. If the local VLAN has a configuration for an E-Tree leaf, the local floodlist filters the remote VTEP contained in the type-3 route. Leaf VLANS contain only remote VTEPs in the floodlist associated with the VLAN on the remote VTEP configured as an E-Tree root and prevents local leaf BUM traffic from flooding to the remote leaf ACs.

EVPN Active/Active Multihoming

EOS only supports EVPN Active/Active Multihoming on root VLANs. Only the peer that sees the host traffic learns the multihomed hosts if configured on leaf VLANs and using asymmetric route-target E-Tree. Because type-2 routes from other peers in the leaf VLANs do not import, an aliasing entry never installs for the hosts on the same Ethernet segment learned on other peers.

In addition, if configured on leaf VLANS, when using single route target extended community-based E-Tree, other peers in leaf VLANs import type-2 routes and associated aliasing entries do not install on the local FDB and treat them as remote leaf hosts.

In both cases, this leads to extra flooding as traffic destined for multihomed hosts that land on peers have not locally learned to treat the host as BUM traffic and floods.

EVPN IGP Cost for VTEP Reachability

In EVPN deployment with VXLAN underlay when an EVPN type-5 prefix is imported into an IP VRF, the IGP cost of the underlay VTEP reachability is not considered as part of BGP best-path selection post import. Therefore, if such a prefix is reachable via more than one VTEPs, the IGP metric step in the BGP best-path selection algorithm will not filter out any paths irrespective of the underlay’s IGP metric for the VTEP reachability. If ECMP is enabled in the overlay and multiple paths are found to be otherwise equivalent, such paths would form ECMP regardless of the IGP metric. This is the default behavior.

However, the behavior can be overridden by the following configuration command encapsulation VXLAN layer-3 set next-hop igp-cost under config-route-bgp-af mode for address-family evpn as shown.

switch(config-router-bgp)# address-family evpn
switch(config-router-bgp-af)# [no | default] encapsulation VXLAN layer-3 set next-hop igp-cost

The encapsulation VXLAN layer-3 set next-hop igp-cost command will cause the underlay IGP metric for the VTEP reachability to be considered for BGP best path selection in the IP VRF that is importing the EVPN route. An IGP protocol such as OSPF, ISIS, or static configuration could be the source of such a metric value.

Note: This feature is available only with the multi-agent routing protocol model.

Configuration Example

Let us consider a topology of four routers leaf1, leaf2, leaf3 and rtr1 as shown in the figure.

There could be IGP running among the four routers, but for simplicity let's have static routes with IGP metric on leaf1 to lo0 of border leaf2 and leaf3 (and vice-versa for reverse reachability on the respective routers, though that configuration not shown here):

leaf1#
ip route 11.0.1.1/32 10.0.0.2 metric 340
ip route 11.0.2.1/32 10.0.0.2 metric 350

Following are the IP routes in the default VRF on leaf1.

leaf1# show ip route

VRF: default
Codes: C - connected, S - static, K - kernel, 
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route, L - VRF Leaked,
       RC - Route Cache Route

Gateway of last resort:
 S        0.0.0.0/0 [1/0] via 10.0.0.2, Ethernet2

 C        10.0.0.0/24 is directly connected, Ethernet2
 C        11.0.0.1/32 is directly connected, Loopback0
 S        11.0.1.1/32 [1/340] via 10.0.0.2, Ethernet2
 S        11.0.2.1/32 [1/350] via 10.0.0.2, Ethernet2

Following are eBGP-multihop EVPN neighbor pairs with VXLAN as underlay:

leaf1 (ASN-300) ? leaf2 (ASN-301)

leaf1 (ASN-300) ? leaf3 (ASN-302)

Consider an example where a prefix 20.0.100.1/32 is reachable behind two VTEPs leaf2 and leaf3 as learnt on leaf1 via eBGP EVPN Type-5 routes.

Following EVPN paths will show for 20.0.100.1/32 on leaf1:

leaf1(config)# show bgp evpn detail
BGP routing table entry for ip-prefix 20.0.100.1/32, Route Distinguisher: 11.0.1.1:0
 Paths: 1 available
  301
    11.0.1.1 from 10.0.1.1 (0.0.2.1)
      Origin INCOMPLETE, metric -, localpref 100, weight 0, valid, external, best
      Extended Community: Route-Target-AS:64500:20000 TunnelEncap:tunnelTypeVXLAN 
      EvpnRouterMac:00:00:78:03:00:00
      VNI: 20000
BGP routing table entry for ip-prefix 20.0.100.1/32, Route Distinguisher: 11.0.2.1:0
 Paths: 1 available
  302
    11.0.2.1 from 10.0.2.1 (0.0.3.1)
      Origin INCOMPLETE, metric -, localpref 100, weight 0, valid, external, best
      Extended Community: Route-Target-AS:64500:20000 TunnelEncap:tunnelTypeVXLAN 
      EvpnRouterMac:00:00:78:04:00:00
      VNI: 20000

Show Commands

By default, since the underlay IGP cost for the VTEP reachability is not used for best path selection in the imported EVPN Type-5 routes, either of the two paths could be selected as the best path. If ECMP is configured in the importing VRF, then the two paths will form ECMP. The following example shows an ECMP that is allowed in the importing VRF vrf1.

switch1# show ip route vrf vrf1

VRF: vrf1
Codes: C - connected, S - static, K - kernel, 
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route, L - VRF Leaked,
       RC - Route Cache Route

 B E      20.0.100.1/32 [200/0] via VTEP 11.0.1.1 VNI 20000 router-mac 00:00:78:02:00:00
                                via VTEP 11.0.2.1 VNI 20000 router-mac 00:00:78:03:00:00

switch1# show ip bgp 20.0.100.1/32 vrf vrf1
BGP routing table information for VRF vrf1
Router identifier 11.0.0.1, local AS number 300
BGP routing table entry for 20.0.100.1/32
 Paths: 2 available
  302
    11.0.2.1 from 10.0.2.1 (0.0.3.1), imported EVPN route, RD 11.0.2.1:0
      Origin INCOMPLETE, metric 0, localpref 100, IGP metric 350, weight 0, tag 0
      Received 01:11:00 ago, valid, external, ECMP head, ECMP, best, ECMP contributor
      Extended Community: Route-Target-AS:64500:20000 TunnelEncap:tunnelTypeVXLAN 
      EvpnRouterMac:00:00:78:04:00:00
      Remote VNI: 20000
      Rx SAFI: Unicast
  301
    11.0.1.1 from 10.0.1.1 (0.0.2.1), imported EVPN route, RD 11.0.1.1:0
      Origin INCOMPLETE, metric 0, localpref 100, IGP metric 340, weight 0, tag 0
      Received 01:11:00 ago, valid, external, ECMP, ECMP contributor
      Not best: ECMP-Fast configured
      Extended Community: Route-Target-AS:64500:20000 TunnelEncap:tunnelTypeVXLAN 
      EvpnRouterMac:00:00:78:03:00:00
      Remote VNI: 20000
      Rx SAFI: Unicast

The configuration command encapsulation VXLAN layer-3 set next-hop igp-cost under address-family evpn will cause the underlay IGP cost to be taken into account and only VTEP 11.0.1.1 with lower IGP cost will be selected as shown below:

switch1# show ip route vrf vrf1

VRF: vrf1
Codes: C - connected, S - static, K - kernel, 
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route, L - VRF Leaked,
       RC - Route Cache Route

 B E      20.0.100.1/32 [200/0] via VTEP 11.0.1.1 VNI 20000 router-mac 00:00:78:02:00:00

switch1(config)# show ip bgp 20.0.100.1/32 vrf vrf1
BGP routing table information for VRF vrf1
Router identifier 11.0.0.1, local AS number 300
BGP routing table entry for 20.0.100.1/32
 Paths: 2 available
  301
    11.0.1.1 from 10.0.1.1 (0.0.2.1), imported EVPN route, RD 11.0.1.1:0
      Origin INCOMPLETE, metric 0, localpref 100, IGP metric 340, weight 0, tag 0
      Received 00:23:35 ago, valid, external, best
      Extended Community: Route-Target-AS:64500:20000 TunnelEncap:tunnelTypeVXLAN 
      EvpnRouterMac:00:00:78:03:00:00
      Remote VNI: 20000
      Rx SAFI: Unicast
  302
    11.0.2.1 from 10.0.2.1 (0.0.3.1), imported EVPN route, RD 11.0.2.1:0
      Origin INCOMPLETE, metric 0, localpref 100, IGP metric 350, weight 0, tag 0
      Received 00:23:35 ago, valid, external
      Not best: IGP cost
      Extended Community: Route-Target-AS:64500:20000 TunnelEncap:tunnelTypeVXLAN 
      EvpnRouterMac:00:00:78:04:00:00
      Remote VNI: 20000
      Rx SAFI: Unicast

EVPN VXLAN Single-Gateway Centralized Routing

In a traditional EVPN VXLAN centralized anycast gateway deployment, multiple L3 VTEPs serve the role of the centralized anycast gateway. For the hosts to have a consistant ARP binding for any of the individual centralized gateway VTEPs, each VTEP operating as a centralized gateway is configured with a virtual router MAC (VARP MAC).A virtual VTEP IP (VARP VTEP IP) is shared between all of the L3 VTEPs operating as centralized gateways. Each centralized gateway VTEP also advertises an EVPN type-3 route for both its primary VTEP IP and VARP VTEP IP, so both IPs end up in the overlay floodset.

The traditional configuration works fine, but in the specific case of a network with only a single L3 VTEP centralized gateway (or single MLAG pair operating as the L3 VTEP centralized gateway), this leads to unnecessary BUM traffic. When both the physical VTEP IP and the VARP IP end up in the overlay floodset, BUM traffic is duplicated to the centralized gateway, which can overhead workloads that have a lot of broadcast or multicast traffic. There is only a single centralized gateway, so it is not necessary to have a VARP VTEP IP to provide a stable ARP binding for the gateway.

The EVPN VXLAN Single-Gateway Centralized Routing feature enables a single L3 VTEP (or single MLAG pair) operating as an anycast gateway to not configure a VARP VTEP IP, thereby eliminating the duplicate BUM traffic caused by the VARP VTEP IP being in the overlay floodset. It accomplishes this with two changes:

A change to the default ARP behavior when a host ARPs for the MAC address associated with a virtual IP address (SVI IP).
Previously, ARPing for a virtual IP returned different MAC address bindings, depending on whether or not a VARP VTEP IP was configured. If a VARP VTEP IP was configured, the ARP request returns the configured VARP MAC. If one was not, the ARP request returns the switch router MAC. This feature changes the behavior to always respond with the configured VARP MAC to an ARP request for a virtual IP. This closes an exception to the rule that virtual IPs are always associated with the VARP MAC.
A new EVPN MAC VRF configuration command that generates an EVPN type-2 route for the VARP MAC with a nexthop of the physical VTEP IP.

In a traditional EVPN centralized anycast gateway development, the presence of a configured VARP VTEP IP advertises an EVPN type-2 route for the VARP MAC with a nexthop of the VARP VTEP IP. This allows TOR switches to learn the ARP binding of the centralized anycast gateway (presumably, their default gateway). With this feature, no VARP VTEP IP is configured, so an alternative method is required to advertise the appropriate EVPN route. This feature adds a new EVPN MAC VRF configuration command, redistribute router-mac next-hop vtep primary ,which when configured on a MAC VRF advertises an EVPN type-2 route for the VARP MAC with a nexthop of the primary VTEP IP. This allows TOR switches to learn the ARP binding of the centralized anycast gateway.

Configuration

On the multi-homing PEs, you must configure the Ethernet Segment (ES) to the CE. In addition, the configuration needed for asymmetric IRB or symmetric IRB must be configured on the local and remote PEs.

In the example, CE1 is a multi-homed CE in VLAN20. CE2 is a remote CE in VLAN30. Asymmetric IRB is configured for inter-VLAN traffic.

Configuration on PE1:

switch(config)# interface Port-Channel100
switch(config-if-Po100)# switchport access vlan 20
! 
switch(config-if-Po100)# evpn ethernet-segment
switch(config-evpn-es)# identifier 0033:3333:3333:3333:3333
switch(config-evpn-es)# route-target import 00:03:00:03:00:03
switch(config-evpn-es)# lacp system-id 1234.5678.0123
!
switch(config)# interface Ethernet1
switch(config-if-Et1)# switchport mode trunk
switch(config-if-Et1)# channel-group 100 mode on
!
switch(config)# interface Loopback0
switch(config-if-Lo0)# ip address 10.255.0.0/32
!
switch(config)# interface Vlan20
switch(config-if-Vl20)# ip address virtual 20.0.20.1/24
!
switch(config)# interface Vlan30
switch(config-if-Vl30)# ip address virtual 20.0.30.1/24
!
switch(config)# interface VXLAN1
switch(config=if-Vx1)# VXLAN source-interface Loopback0
switch(config=if-Vx1)# VXLAN udp-port 4789
switch(config=if-Vx1)# VXLAN vlan 20 vni 10020
switch(config=if-Vx1)# VXLAN vlan 30 vni 10030
!
switch(config)# ip virtual-router mac-address 00:00:80:00:00:00
!
switch(config)# router bgp 300
switch(config-router-bgp)# router-id 0.0.0.1
switch(config-router-bgp)# neighbor 10.0.0.1 remote-as 303
switch(config-router-bgp)# neighbor 10.0.0.1 ebgp-multihop
switch(config-router-bgp)# neighbor 10.0.0.1 send-community extended
switch(config-router-bgp)# neighbor 10.0.0.1 maximum-routes 12000
switch(config-router-bgp)# redistribute static
   !
switch(config-router-bgp)# vlan 20
switch(config-macvrf-20)# rd 10.255.0.0:20
switch(config-macvrf-20)# route-target both 64500:10020
switch(config-macvrf-20)# redistribute learned
   !
switch(config-router-bgp)# vlan 30
switch(config-macvrf-30)# rd 10.255.0.0:30
switch(config-macvrf-30)# route-target both 64500:10030
switch(config-macvrf-30)# redistribute learned
   !
switch(config-macvrf-30)# address-family evpn
switch(config-router-bgp-af)# neighbor 10.0.0.1 activate

The Ethernet segment to the multi-homed CE is configured on the port channel interface Port-Channel 100. SVI 20 and SVI 30 along with VARP IP are configured for inter-subnet routing. A VARP MAC is configured globally on PE1. The configuration on PE2 is similar to the configuration shown above. On PE3, SVI 20 and SVI 30 are configured along with VARP IP and VARP MAC.

Symmetric IRB with IPv4 example:

Configuration on PE1:

switch(config)# vrf instance red
switch(config-vrf-red)# rd 10.255.0.0:0
!
switch(config-vrf-red)# interface Port-Channel100
switch(config-if-Po100)# switchport access vlan 20
!
switch(config-if-Po100)# evpn ethernet-segment
switch(config-evpn-es)# identifier 0033:3333:3333:3333:3333
switch(config-evpn-es)# route-target import 00:03:00:03:00:03
switch(config-evpn-es)# lacp system-id 1234.5678.0123
!
switch(config-if-Po100)# interface Ethernet6/6/1
switch(config-if-Et6/6/1)# switchport mode trunk
switch(config-if-Et6/6/1)# channel-group 100 mode on
!
switch(config-if-Et6/6/1)# interface Loopback0
switch(config-if-Lo0)# ip address 10.255.0.0/32
!
switch(config-if-Lo0)# interface Vlan20
switch(config-if-Vl20)# vrf red
switch(config-if-Vl20)# ip address virtual 20.0.20.1/24
!
switch(config-if-Vl20)# interface VXLAN1
switch(config-if-Vx1)# VXLAN source-interface Loopback0
switch(config-if-Vx1)# VXLAN udp-port 4789   
switch(config-if-Vx1)# VXLAN vlan 10 vni 10010  
switch(config-if-Vx1)# VXLAN vlan 20 vni 10020
switch(config-if-Vx1)# VXLAN vrf red vni 20000
!
switch(config-if-Vx1)# ip virtual-router mac-address 00:00:80:00:00:00
!
switch(config)# router bgp 300
switch(config-router-bgp)# router-id 0.0.0.1
switch(config-router-bgp)# maximum-paths 2
switch(config-router-bgp)# neighbor 10.0.0.1 remote-as 303
switch(config-router-bgp)# neighbor 10.0.0.1 ebgp-multihop
switch(config-router-bgp)# neighbor 10.0.0.1 send-community extended
switch(config-router-bgp)# neighbor 10.0.0.1 maximum-routes 12000
switch(config-router-bgp)# redistribute static
!
switch(config-router-bgp)# vlan 20     
switch(config-macvrf-20)# rd 10.255.0.0:20
switch(config-macvrf-20)# route-target both 64500:10020
switch(config-macvrf-20)# redistribute learned
!
switch(config-macvrf-20)# address-family evpn
switch(config-router-bgp-af)# neighbor 10.0.0.1 activate
!
switch(config-router-bgp-af)# vrf red
switch(config-router-bgp-vrf-red)# rd 10.255.0.0:0
switch(config-router-bgp-vrf-red)# route-target import evpn 64500:20000
switch(config-router-bgp-vrf-red)# route-target export evpn 64500:20000
switch(config-router-bgp-vrf-red)# router-id 10.255.0.0
!

The Ethernet segment to the multi-homed CE1 is configured on the port channel interface Port-Channel 100. SVI 20 along with VARP IP and VARP MAC is configured. Also, IP VRF is configured which is needed for symmetric IRB. The configuration on PE2 is similar. On PE3, IP VRF and SVI 30 are configured for symmetric IRB.

VXLAN example

A netwok with two VRFs, red and blue has VLANs 10 and 20 in red and VLANs 30 and 40 in blue. The spines act as a route reflectors. Multicast groups are used to encapsulate traffic arriving in a VRF such that it is delivered to VTEPs that have that VRF provisioned.

interface Loopback0
   ip address 10.0.0.20/32
!
vlan 10
vlan 20
vlan 30
vlan 40
!
interface Ethernet1
   switchport access vlan 10
!
interface Ethernet2
   switchport access vlan 20
!
interface Ethernet3
   switchport access vlan 30
!
interface Ethernet4
   switchport access vlan 40
!
interface Vlan10
   vrf red 
   ip address virtual 192.168.1.0/24
   ip igmp
   pim ipv4 local-interface loopback0
!
interface Vlan20
   vrf red 
   ip address 192.168.2.0/24
   ip igmp
!
interface Vlan30
   vrf blue
   ip address 192.168.1.0/24
   ip igmp
!
interface Vlan40
   vrf blue
   ip address 192.168.2.0/24
   ip igmp
!
interface VXLAN1
   VXLAN source-interface Loopback0
   VXLAN vlan 10 vni 10
   VXLAN vlan 20 vni 20
   VXLAN vlan 30 vni 30
   VXLAN vlan 40 vni 40
   VXLAN vrf red vni 100
   VXLAN vrf blue vni 200
   VXLAN vlan 10 flood group 225.1.1.2
   VXLAN vlan 20 flood group 225.1.1.3
   VXLAN vlan 30 flood group 226.1.1.2
   VXLAN vlan 40 flood group 226.1.1.3
   VXLAN vrf red multicast group 225.1.1.1
   VXLAN vrf blue multicast group 226.1.1.1
!

Show Commands

The following examples are based on the sample topology and configuration in the previous sections.

On the remote VTEP, to display the EVPN routes to the multi-homed CE (20.0.20.2):

switch# show bgp evpn route-type mac-ip 20.0.20.2
BGP routing table information for VRF default
Router identifier 0.0.3.1, local AS number 302
Route status codes: s - suppressed, * - valid, > - active, # - not installed, E - ECMP head, 
                    e - ECMP S - Stale, c - Contributing to ECMP, b - backup
                    % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path
 * >     RD: 10.255.0.0:20 mac-ip 0000.0101.0000 20.0.20.2
                                10.255.0.0            -       100     0       303 300 i
 * >     RD: 10.255.0.1:20 mac-ip 0000.0101.0000 20.0.20.2
                                10.255.0.1            -       100     0       303 301 i

As shown above, there are two EVPN MAC-IP routes for the multi-homed CE.

On the remote PE, to display the installed routes to the multi-homed CE:

switch# show ip route vrf red 20.0.20.2/32

VRF: red
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route, L - VRF Leaked

 B E      20.0.20.2/32 [200/0] via VTEP 10.255.0.0 VNI 20000 router-mac 00:00:78:01:00:00
                               via VTEP 10.255.0.1 VNI 20000 router-mac 00:00:78:04:00:00

Two routes to the multi-homed CE are installed from the two EVPN MAC-IP routes and they form L3 ECMP.

On the remote PE, to check the details of the two routes in BGP RIB:

switch# show ip bgp 20.0.20.2/32 vrf red
BGP routing table information for VRF red
Router identifier 10.255.0.2, local AS number 302
BGP routing table entry for 20.0.20.2/32
 Paths: 2 available
  303 300
    10.255.0.0 from 10.0.2.1 (0.0.1.1), imported EVPN route, RD 10.255.0.0:20
      Origin IGP, metric 0, localpref 100, IGP metric 0, weight 0, received 00:14:43 ago, 
      valid, external, ECMP head, ECMP, best, ECMP contributor
      Extended Community: Route-Target-AS:64500:10020 Route-Target-AS:64500:20000 
      TunnelEncap:tunnelTypeVXLAN EvpnMacMobility:1 EvpnRouterMac:00:00:78:01:00:00
      Remote VNI: 20000
      Rx SAFI: Unicast
  303 301
    10.255.0.1 from 10.0.2.1 (0.0.1.1), imported EVPN route, RD 10.255.0.1:20
      Origin IGP, metric 0, localpref 100, IGP metric 0, weight 0, received 00:14:43 ago, 
      valid, external, ECMP, ECMP contributor
      Extended Community: Route-Target-AS:64500:10020 Route-Target-AS:64500:20000 
      TunnelEncap:tunnelTypeVXLAN EvpnMacMobility:1 EvpnRouterMac:00:00:78:04:00:00
      EvpnNdFlags:pflag
      Remote VNI: 20000
      Rx SAFI: Unicast

The second route has EvpnNdFlags:pflag to indicate that this is a proxy MAC-IP route.

This command shows information about the SBD instance that is created when evpn multicast is configured under an IP VRF:

switch# show bgp evpn instance sbd red
EVPN instance: SBD red
  Route distinguisher: 100:1
  Service interface: VLAN-based
  Local IP address: 10.0.0.20
  Encapsulation type: VXLAN
vtep2#show bgp evpn instance sbd blue
EVPN instance: SBD red
  Route distinguisher: 200:1
  Service interface: VLAN-based
  Local IP address: 10.0.0.20
  Encapsulation type: VXLAN

The OISM supported capability is set in IMET routes for the SBD when evpn multicast is configured for a VRF. The IGMP proxy flag is not set in IMET routes for VLANs in the VRF unlessredistribute igmp is configured in a VLAN:

switch# show bgp evpn route-type imet next-hop 10.0.0.10 detail
BGP routing table information for VRF default
Router identifier 0.0.0.1, local AS number 300
BGP routing table entry for imet 10.0.0.10, Route Distinguisher: 10:1
 Paths: 1 available
  Local
    10.0.0.10 from 10.0.0.1 (0.0.1.1)
      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
      Extended Community: Route-Target-AS:10:1 TunnelEncap:tunnelTypeVXLAN
      VNI: 10
      PMSI Tunnel: PIM-SSM Tree, MPLS Label: 10, Leaf Information Required: false, 
      Tunnel ID: 10.0.0.10, 225.1.1.2
BGP routing table information for VRF default
Router identifier 0.0.0.1, local AS number 300
BGP routing table entry for imet 10.0.0.10, Route Distinguisher: 100:1
 Paths: 1 available
  Local
    10.0.0.10 from 10.0.0.1 (0.0.1.1)
      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
      Extended Community: Route-Target-AS:100:1 TunnelEncap:tunnelTypeVXLAN Multicast 
      Flags: IGMP proxy, OISM-supported, SBD
      VNI: 100
      PMSI Tunnel: Ingress Replication, MPLS Label: 100, Leaf Information Required: false, 
      Tunnel ID: 10.0.0.10

This command shows information about the single SMET route for the group with the RD of VRF red:

switch# show bgp evpn route-type smet multicast 228.1.1.1
BGP routing table information for VRF default
Router identifier 0.0.1.1, local AS number 300
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
                    S - Stale, c - Contributing to ECMP, b - backup
                    % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
          Network                Next Hop              Metric  LocPref Weight  Path
 * >     RD: 100:1 smet (S, G): (*, 228.1.1.1) originating IP: 10.0.0.10
                                 10.0.0.10             -       100     0       i
 * >     RD: 100:1 smet (S, G): (*, 228.1.1.1) originating IP: 10.0.0.20
                                 -                     -        -      0       i
 * >     RD: 100:1 smet (S, G): (*, 228.1.1.1) originating IP: 10.0.0.30
                                 10.0.0.30             -       100     0       i
 * >     RD: 200:1 smet (S, G): (*, 228.1.1.1) originating IP: 10.0.0.20
                                 -                     -        -      0       i
 * >     RD: 200:1 smet (S, G): (*, 228.1.1.1) originating IP: 10.0.0.40
                                 10.0.0.40             -       100     0       i

This command shows information about the SPMSI route for the group:

switch# show bgp evpn route-type spmsi
BGP routing table information for VRF defaultRouter identifier 0.0.0.1, local AS number 300
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
                    S - Stale, c - Contributing to ECMP, b - backup
                    % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
          Network                Next Hop              Metric  LocPref Weight  Path
 * >     RD:  10:1 spmsi (S, G): (*, *) originating IP: 10.0.0.10
                                 10.0.0.10             -       100     0       i
 * >     RD:  10:1 spmsi (S, G): (*, *) originating IP: 10.0.0.20
                                 -                     -       -       0       i
 * >     RD:  20:1 spmsi (S, G): (*, *) originating IP: 10.0.0.20
                                 -                     -       -       0       i
 * >     RD:  30:1 spmsi (S, G): (*, *) originating IP: 10.0.0.20
                                 -                     -       -       0       i
 * >     RD:  40:1 spmsi (S, G): (*, *) originating IP: 10.0.0.20
                                 -                     -       -       0       i
 * >     RD:  10:1 spmsi (S, G): (*, *) originating IP: 10.0.0.30
                                 10.0.0.30             -       100     0       i
 * >     RD:  20:1 spmsi (S, G): (*, *) originating IP: 10.0.0.30
                                 10.0.0.30             -       100     0       i
 * >     RD:  30:1 spmsi (S, G): (*, *) originating IP: 10.0.0.30
                                 10.0.0.30             -       100     0       i
 * >     RD:  40:1 spmsi (S, G): (*, *) originating IP: 10.0.0.30
                                 10.0.0.30             -       100     0       i
 * >     RD:  30:1 spmsi (S, G): (*, *) originating IP: 10.0.0.40
                                 10.0.0.40             -       100     0       i
 * >     RD: 100:1 spmsi (S, G): (*, *) originating IP: 10.0.0.10
                                 10.0.0.10             -       100     0       i
 * >     RD: 100:1 spmsi (S, G): (*, *) originating IP: 10.0.0.20
                                 -                     -       -       0       i
 * >     RD: 200:1 spmsi (S, G): (*, *) originating IP: 10.0.0.20
                                 -                     -       -       0       i
 * >     RD: 100:1 spmsi (S, G): (*, *) originating IP: 10.0.0.30
                                 10.0.0.30             -       100     0       i
 * >     RD: 200:1 spmsi (S, G): (*, *) originating IP: 10.0.0.30
                                 10.0.0.30             -       100     0       i
 * >     RD: 200:1 spmsi (S, G): (*, *) originating IP: 10.0.0.40
                                 10.0.0.40             -       100     0       i

Limitations

The EVPN VXLAN all-active multi-homing integrated routing and bridging feature has the following limitations:

L2 loop-free protocols, such as Spanning Tree Protocol (STP) are not supported between PE-CE with EVPN VXLAN All-Active Multi-homing. The topology must be loop-free. STP must be disabled for the Multihomed VLANs on PEs and CEs.
A limit of two multi-homing destinations are installed at one time for any particular MAC address. Any other valid destinations are ignored in the context of switching unicast traffic, until one of the active destinations is removed. The multi-homing PEs operate normally regardless of the number of PEs on the ethernet segment; this limitation only affects the selection of a destination for unicast traffic.
VXLAN GPE is not supported, so packets cannot be flagged as having been broadcast. As a result, unicast packets that are known at the sender and unknown at the receiver are dropped if the receiving PE is not a designated forwarder. Similarly, unicast packets that are unknown at the sender and known at the receiver are duplicated at the receiving CE. This state automatically resolves itself as the BGP network distributes the appropriate type 1 auto-discovery and type 2 MAC/IP advertisement EVPN routes.
Fast mass withdrawal is not supported, so there may be a delay if an interface harboring a large number of MAC addresses goes down.

Flood Traffic Filtering with EVPN

The VXLAN fabric managed by EVPN does not always flood with broadcast, multicast, or unknown MAC traffic. Sometimes, ARP request broadcast and ND multicast traffic flood the VXLAN fabric as well. There may be other cases where flooding ARP plus other traffic is allowed, but not all broadcast traffic into the fabric.

Most of the ARPs are learned through EVPN, and for some cases where the ARP is not learned through EVPN, it is acceptable to flood such ARP requests. However, rate limit the ARP flooding going into the VXLAN fabric.

Configuration

The default command disables flooding of different kinds of traffic into the VXLAN fabric to restrict only ARP and ND traffic flooding.

switch(config-rtr-l2-vpn)# flooding default disabled

The following command enables flooding of ARP packets into the VXLAN fabric again.

switch(config-rtr-l2-vpn)# arp flooding

The above command starts flooding ARP traffic into the VXLAN fabric while all other traffic including ND traffic is prevented from getting flooded into the fabric. The following command enables flooding of ND traffic.

switch(config-rtr-l2-vpn)# nd flooding

Note: These commands are in effect only when EVPN is enabled. ARP & ND flooding are enabled by default.

Show Commands

The show VXLAN counters software command shows the number of ARP and ND packets that were prevented from getting flooded into the VXLAN fabric.

switch# show VXLAN counters software 
. . . . . . . . 
Tx pkts after IPv6 encapsulation                     :  0
SW pkts forwarded to remote VTEPs via HW HER         :  4
SW pkts forwarding to remote VTEPs via HW HER failed :  0
Packets suppressed from getting flooded              :  0

Inter-VRF Local Route Leaking

Inter-VRF local route leaking allows the leaking of routes from one VRF (the source VRF) to another VRF (the destination VRF) on the same router. Inter-VRF routes can exist in any VRF (including the default VRF) on the system. Routes can be leaked using the following methods:

Inter-VRF Local Route Leaking using BGP VPN
Inter-VRF Local Route Leaking using VRF-leak Agent

Inter-VRF Local Route Leaking using BGP VPN

Inter-VRF local route leaking allows the user to export and import routes from one VRF to another on the same device. This is implemented by exporting routes from a VRF to the local VPN table using the route target extended community list and importing the same route target extended community lists from the local VPN table into the target VRF. VRF route leaking is supported on VPN-IPv4, VPN-IPv6, and EVPN types.

Figure 1. Inter-VRF Local Route Leaking using Local VPN Table

Accessing Shared Resources Across VPNs

To access shared resources across VPNs, all the routes from the shared services VRF must be leaked into each of the VPN VRFs, and customer routes must be leaked into the shared services VRF for return traffic. Accessing shared resources allows the route target of the shared services VRF to be exported into all customer VRFs, and allows the shared services VRF to import route targets from customers A and B. The following figure shows how to provide customers, corresponding to multiple VPN domains, access to services like DHCP available in the shared VRF.

Route leaking across the VRFs is supported on VPN-IPv4, VPN-IPv6, and EVPN.

Figure 2. Accessing Shared Resources Across VPNs

Configuring Inter-VRF Local Route Leaking

Inter-VRF local route leaking is configured using VPN-IPv4, VPN-IPv6, and EVPN. Prefixes can be exported and imported using any of the configured VPN types. Ensure that the same VPN type that is exported is used while importing.

Leaking unicast IPv4 or IPv6 prefixes is supported and achieved by exporting prefixes locally to the VPN table and importing locally from the VPN table into the target VRF on the same device as shown in the figure titled Inter-VRF Local Route Leaking using Local VPN Table using the route-target command.

Exporting or importing the routes to or from the EVPN table is accomplished with the following two methods:

Using VXLAN for encapsulation
Using MPLS for encapsulation

Using VXLAN for Encapsulation

To use VXLAN encapsulation type, make sure that VRF to VNI mapping is present and the interface status for the VXLAN interface is up. This is the default encapsulation type for EVPN.

Example

The configuration for VXLAN encapsulation type is as follows:

switch(config)# router bgp 65001
switch(config-router-bgp)# address-family evpn
switch(config-router-bgp-af)# neighbor default encapsulation VXLAN next-hop-self source-interface Loopback0
switch(config)# hardware tcam
switch(config-hw-tcam)# system profile VXLAN-routing
switch(config-hw-tcam)# interface VXLAN1
switch(config-hw-tcam-if-Vx1)# VXLAN source-interface Loopback0
switch(config-hw-tcam-if-Vx1)# VXLAN udp-port 4789
switch(config-hw-tcam-if-Vx1)# VXLAN vrf vrf-blue vni 20001
switch(config-hw-tcam-if-Vx1)# VXLAN vrf vrf-red vni 10001

Using MPLS for Encapsulation

To use MPLS encapsulation type to export to the EVPN table, MPLS needs to be enabled globally on the device and the encapsulation method needs to be changed from default type, that is VXLAN to MPLS under the EVPN address-family sub-mode.

Example

switch(config)# router bgp 65001
switch(config-router-bgp)# address-family evpn
switch(config-router-bgp-af)# neighbor default encapsulation mpls next-hop-self source-interface Loopback0

Route-Distinguisher

Route-Distinguisher (RD) uniquely identifies routes from a particular VRF. Route-Distinguisher is configured for every VRF from which routes are exported from or imported into.

The following commands are used to configure Route-Distinguisher for a VRF.

switch(config-router-bgp)# vrf vrf-services
switch(config-router-bgp-vrf-vrf-services)# rd 1.0.0.1:1

switch(config-router-bgp)# vrf vrf-blue 
switch(config-router-bgp-vrf-vrf-blue)# rd 2.0.0.1:2

Exporting Routes from a VRF

Use the route-target export command to export routes from a VRF to the local VPN or EVPN table using the route target extended community list.

Examples

These commands export routes from vrf-red to the local VPN table.

switch(config)# service routing protocols model multi-agent
switch(config)# mpls ip
switch(config)# router bgp 65001
switch(config-router-bgp)# vrf vrf-red
switch(config-router-bgp-vrf-vrf-red)# rd 1:1
switch(config-router-bgp-vrf-vrf-red)# route-target export vpn-ipv4 10:10
switch(config-router-bgp-vrf-vrf-red)# route-target export vpn-ipv6 10:20

These commands export routes from vrf-red to the EVPN table.

switch(config)# router bgp 65001
switch(config-router-bgp)# vrf vrf-red
switch(config-router-bgp-vrf-vrf-red)# rd 1:1
switch(config-router-bgp-vrf-vrf-red)# route-target export evpn 10:1

Importing Routes into a VRF

Use the route-target import command to import the exported routes from the local VPN or EVPN table to the target VRF using the route target extended community list.

Examples

These commands import routes from the VPN table to vrf-blue.

switch(config)# service routing protocols model multi-agent
switch(config)# mpls ip
switch(config)# router bgp 65001
switch(config-router-bgp)# vrf vrf-blue
switch(config-router-bgp-vrf-vrf-blue)# rd 2:2
switch(config-router-bgp-vrf-vrf-blue)# route-target import vpn-ipv4 10:10
switch(config-router-bgp-vrf-vrf-blue)# route-target import vpn-ipv6 10:20

These commands import routes from the EVPN table to vrf-blue.

switch(config)# router bgp 65001
switch(config-router-bgp)# vrf vrf-blue
switch(config-router-bgp-vrf-vrf-blue)# rd 2:2
switch(config-router-bgp-vrf-vrf-blue)# route-target import evpn 10:1

Exporting and Importing Routes using Route Map

To manage VRF route leaking, control the export and import prefixes with route-map export or import commands. The route map is effective only if the VRF or the VPN paths are already candidates for export or import. The route-target export or import commandmust be configured first. Setting BGP attributes using route maps is effective only on the export end.

Note: Prefixes that are leaked are not re-exported to the VPN table from the target VRF.

Examples

These commands export routes from vrf-red to the local VPN table.

switch(config)# service routing protocols model multi-agent
switch(config)# mpls ip
switch(config)# router bgp 65001
switch(config-router-bgp)# vrf vrf-red
switch(config-router-bgp-vrf-vrf-red)# rd 1:1
switch(config-router-bgp-vrf-vrf-red)# route-target export vpn-ipv4 10:10
switch(config-router-bgp-vrf-vrf-red)# route-target export vpn-ipv6 10:20
switch(config-router-bgp-vrf-vrf-red)# route-target export vpn-ipv4 route-map EXPORT_V4_ROUTES_T0_VPN_TABLE
switch(config-router-bgp-vrf-vrf-red)# route-target export vpn-ipv6 route-map EXPORT_V6_ROUTES_T0_VPN_TABLE

These commands export routes to from vrf-red to the EVPN table.

switch(config)# router bgp 65001
switch(config-router-bgp)# vrf vrf-red
switch(config-router-bgp-vrf-vrf-red)# rd 1:1
switch(config-router-bgp-vrf-vrf-red)# route-target export evpn 10:1
switch(config-router-bgp-vrf-vrf-red)# route-target export evpn route-map EXPORT_ROUTES_T0_EVPN_TABLE

These commands import routes from the VPN table to vrf-blue.

switch(config)# service routing protocols model multi-agent
switch(config)# mpls ip
switch(config)# router bgp 65001
switch(config-router-bgp)# vrf vrf-blue
switch(config-router-bgp-vrf-vrf-blue)# rd 1:1
switch(config-router-bgp-vrf-vrf-blue)# route-target import vpn-ipv4 10:10
switch(config-router-bgp-vrf-vrf-blue)# route-target import vpn-ipv6 10:20
switch(config-router-bgp-vrf-vrf-blue)# route-target import vpn-ipv4 route-map IMPORT_V4_ROUTES_VPN_TABLE
switch(config-router-bgp-vrf-vrf-blue)# route-target import vpn-ipv6 route-map IMPORT_V6_ROUTES_VPN_TABLE

These commands import routes from the EVPN table to vrf-blue.

switch(config)# router bgp 65001
switch(config-router-bgp)# vrf vrf-blue
switch(config-router-bgp-vrf-vrf-blue)# rd 2:2
switch(config-router-bgp-vrf-vrf-blue)# route-target import evpn 10:1
switch(config-router-bgp-vrf-vrf-blue)# route-target import evpn route-map IMPORT_ROUTES_FROM_EVPN_TABLE

Inter-VRF Local Route Leaking using VRF-leak Agent

Inter-VRF local route leaking allows routes to leak from one VRF to another using a route map as a VRF-leak agent. VRFs are leaked based on the preferences assigned to each VRF.

Configuring Route Maps

To leak routes from one VRF to another using a route map, use the router general command to enter Router-General Configuration Mode, then enter the VRF submode for the destination VRF, and use the leak routes command to specify the source VRF and the route map to be used. Routes in the source VRF that match the policy in the route map will then be considered for leaking into the configuration-mode VRF. If two or more policies specify leaking the same prefix to the same destination VRF, the route with a higher (post-set-clause) distance and preference is chosen.

Example

These commands configure a route map to leak routes from VRF1 to VRF2 using route map RM1.

switch(config)# router general
switch(config-router-general)# vrf VRF2
switch(config-router-general-vrf-VRF2)# leak routes source-vrf VRF1 subscribe-policy RM1
switch(config-router-general-vrf-VRF2)#

EVPN Multicast VRF Leaking

Overview

EVPN Multicast virtual routing and forwarding (VRF) Leaking forwards multicast traffic from a sender in a tenant or external domain (VRF) to a different domain (VRF) with connected receivers. The source VRF refers to the VRF of the multicast sender, and the receiver VRF refers to the VRF of the multicast receiver.

Typically, a multicast receiver sends IGMP messages to a multicast stream, and a last hop router converts the IGMP message to a Protocol Independent Multicast (PIM) message and propagates them upstream on the network. When a VRF receives an IGMP or PIM message, a corresponding VRF leak occurs where the IGMP or PIM messages leak to the source VRF, which the source VRF processes. The corresponding EVPN Selective Multicast Ethernet Tag (SMET) route originates with the Supplementary Broadcast Domain (SBD) route target of the VRF and the route distinguisher (RD) where the sources connect. The receiver VRF RT and RD do not originate SMET routes.

When configuring EVPN Multicast VRF Leaking, the source VRF contains the Multicase Outgoing Interface (OIF) list of multicast routes from the receiver VRF. When the source VRF forwards traffic, the VRF sends a copy of the OIF list across the VRF.

EVPN Multicast VRF Leaking supports a per multicast group configuration to leak multicast traffic.

Note: Configure EVPN Multicast VRF Leaking with caution. Any network design or configuration issues could potentially cause traffic to blackhole, send multiple copies of traffic to the same receiver, or create traffic looping, which may impact traffic on your network.

In the following EVPN topology, two VRF tenants, red and blue, have multicase receivers and senders. The receiver, Rb1, on the blue VRF looks for group G on VRF, red, which has the resources. To enable the feature, configure VRF Leaking for group G on the PE connected to the Rb1 with VRF red as the source VRF.

Figure 1. Figure 1 - EVPN Multicast Traffic Flow with VRF Leaking

In the example topology, the control flow works as follows:

Rb1 sends an IGMP join request for (*,G), and the host device is unaware of VRF Leaking.
VRF blue has the configuration for VRF Leaking for G, so the egress Provider Edge (PE) router sends a SMET to the red VRF, and does not send a SMET to the blue VRF or the VLAN100.
The ingress PE connected to the source on VRF red, sends VXLAN-encapsulated multicast packets with the VNI of the source VLAN across the EVPN core.
The egress PE receives the VXLAN packet, decapsulates it, and routes the data packet to VRF blue.

Note: EVPN multi-homed receivers support VRF Leaking, and the configuration should be identical on both designated-forwarder (DF) and non-DF VTEPs for the multi-homed Ethernet segment.

EVPN Multicast VRF Leaking with a PIM EVPN Gateway (PEG)

On an EVPN network with a PEG role, the PEG acts as the rendezvous point (RP) for the leaked groups. The PEG also functions as the Rendezvous Point (RP) of the source VRF, which follows the PIM network model, where the PEG acts as the RP of the source and receiver VRFs. The configuration requires synchronization using a protocol such as Multicast Source Discovery Protocol (MSDP), and the source synchronization may be necessary with the source and receiver VRFs.

If an RP exists in the external PIM domain, then Multicast VRF Leaking can be configured on that RP instead of the PEG, but this requires that both the source and receiver VRFs extend into the PIM domain that includes the RP.

In this case, the sources on the EVPN tenant VRFs exist in a different VRF from the RP, and the unicast route leaks to the source from the tenant VRF into the external domain VRF to build the multicast trees and perform Reverse Path Forwarding (RPF) validation.

Figure 2. Figure 2 - Multicast Traffic Flow with VRF Leaking on the PEG

Using MLAG as a PEG on an EVPN Network

When configuring VRF Leaking on MLAG devices used as PEGS, configure the MLAG devices as a PEG in both the receiver VRF and the source VRF. Typically, the external PIM domain connects to the receiver. To enable PEG functionality on the source VRF, configure a stub VLAN and a corresponding SVI on both MLAG peers. Enable PIM on the SVIs of both peers and attach them to the source VRF. This enables PEG functionality in the source VRF as well.

Static Inter-VRF Route

The Static Inter-VRF Route feature adds support for static inter-VRF routes. This enables the configuration of routes to destinations in one ingress VRF with an ability to specify a next-hop in a different egress VRF through a static configuration.

You can configure static inter-VRF routes in default and non-default VRFs. A different egress VRF is achieved by “tagging” the next-hop or forwarding via with a reference to an egress VRF (different from the source VRF) in which that next-hop should be evaluated. Static inter-VRF routes with ECMP next-hop sets in the same egress VRF or heterogenous egress VRFs can be specified.

The Static Inter-VRF Route feature is independent and complementary to other mechanisms that can be used to setup local inter-VRF routes. The other supported mechanisms in EOS and the broader use-cases they support are documented here:

Configuration

The configuration to setup static-Inter VRF routes in an ingress (source) VRF to forward IP traffic to a different egress (target) VRF can be done in the following modes:

This command creates a static route in one ingress VRF that points to a next-hop in a different egress VRF.
ip|ipv6 route [vrf vrf-name destination-prefix [egress-vrfegress-next-hop vrf-name] next-hop

Show Commands

Use the show ip route vrf to display the egress VRF name if it differs from the source VRF.

Example

switch# show ip route vrf vrf1

VRF: vrf1
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route, L - VRF Leaked

Gateway of last resort is not set

 S        1.0.1.0/24 [1/0] via 1.0.0.2, Vlan2180 (egress VRF default)
 S        1.0.7.0/24 [1/0] via 1.0.6.2, Vlan2507 (egress VRF vrf3)

Limitations

For bidirectional traffic to work correctly between a pair of VRFs, static inter-VRF routes in both VRFs must be configured.
Static Inter-VRF routing is supported only in multi-agent routing protocol mode.

VCS to EVPN Hitless Migration

VCS to EVPN Hitless Migration enables support for migrating from only using VCS as the control plane to only using EVPN as a control plane in a hitless manner concerning L2 reachability information.

Configuration

In the example, CE1 is a multi-homed CE in VLAN20. CE2 is a remote CE in VLAN30. Asymmetric IRB is configured for inter-VLAN traffic.

Configuration on PE1:

switch(config)# interface Port-Channel100
switch(config-if-Po100)# switchport access vlan 20
! 
switch(config-if-Po100)# evpn ethernet-segment
switch(config-evpn-es)# identifier 0033:3333:3333:3333:3333
switch(config-evpn-es)# route-target import 00:03:00:03:00:03
switch(config-evpn-es)# lacp system-id 1234.5678.0123
!
switch(config)# interface Ethernet1
switch(config-if-Et1)# switchport mode trunk
switch(config-if-Et1)# channel-group 100 mode on
!
switch(config)# interface Loopback0
switch(config-if-Lo0)# ip address 10.255.0.0/32
!
switch(config)# interface Vlan20
switch(config-if-Vl20)# ip address virtual 20.0.20.1/24
!
switch(config)# interface Vlan30
switch(config-if-Vl30)# ip address virtual 20.0.30.1/24
!
switch(config)# interface VXLAN1
switch(config=if-Vx1)# VXLAN source-interface Loopback0
switch(config=if-Vx1)# VXLAN udp-port 4789
switch(config=if-Vx1)# VXLAN vlan 20 vni 10020
switch(config=if-Vx1)# VXLAN vlan 30 vni 10030
!
switch(config)# ip virtual-router mac-address 00:00:80:00:00:00
!
switch(config)# router bgp 300
switch(config-router-bgp)# router-id 0.0.0.1
switch(config-router-bgp)# neighbor 10.0.0.1 remote-as 303
switch(config-router-bgp)# neighbor 10.0.0.1 ebgp-multihop
switch(config-router-bgp)# neighbor 10.0.0.1 send-community extended
switch(config-router-bgp)# neighbor 10.0.0.1 maximum-routes 12000
switch(config-router-bgp)# redistribute static
   !
switch(config-router-bgp)# vlan 20
switch(config-macvrf-20)# rd 10.255.0.0:20
switch(config-macvrf-20)# route-target both 64500:10020
switch(config-macvrf-20)# redistribute learned
   !
switch(config-router-bgp)# vlan 30
switch(config-macvrf-30)# rd 10.255.0.0:30
switch(config-macvrf-30)# route-target both 64500:10030
switch(config-macvrf-30)# redistribute learned
   !
switch(config-macvrf-30)# address-family evpn
switch(config-router-bgp-af)# neighbor 10.0.0.1 activate

Symmetric IRB with IPv4 example:

Configuration on PE1:

switch(config)# vrf instance red
switch(config-vrf-red)# rd 10.255.0.0:0
!
switch(config-vrf-red)# interface Port-Channel100
switch(config-if-Po100)# switchport access vlan 20
!
switch(config-if-Po100)# evpn ethernet-segment
switch(config-evpn-es)# identifier 0033:3333:3333:3333:3333
switch(config-evpn-es)# route-target import 00:03:00:03:00:03
switch(config-evpn-es)# lacp system-id 1234.5678.0123
!
switch(config-if-Po100)# interface Ethernet6/6/1
switch(config-if-Et6/6/1)# switchport mode trunk
switch(config-if-Et6/6/1)# channel-group 100 mode on
!
switch(config-if-Et6/6/1)# interface Loopback0
switch(config-if-Lo0)# ip address 10.255.0.0/32
!
switch(config-if-Lo0)# interface Vlan20
switch(config-if-Vl20)# vrf red
switch(config-if-Vl20)# ip address virtual 20.0.20.1/24
!
switch(config-if-Vl20)# interface VXLAN1
switch(config-if-Vx1)# VXLAN source-interface Loopback0
switch(config-if-Vx1)# VXLAN udp-port 4789   
switch(config-if-Vx1)# VXLAN vlan 10 vni 10010  
switch(config-if-Vx1)# VXLAN vlan 20 vni 10020
switch(config-if-Vx1)# VXLAN vrf red vni 20000
!
switch(config-if-Vx1)# ip virtual-router mac-address 00:00:80:00:00:00
!
switch(config)# router bgp 300
switch(config-router-bgp)# router-id 0.0.0.1
switch(config-router-bgp)# maximum-paths 2
switch(config-router-bgp)# neighbor 10.0.0.1 remote-as 303
switch(config-router-bgp)# neighbor 10.0.0.1 ebgp-multihop
switch(config-router-bgp)# neighbor 10.0.0.1 send-community extended
switch(config-router-bgp)# neighbor 10.0.0.1 maximum-routes 12000
switch(config-router-bgp)# redistribute static
!
switch(config-router-bgp)# vlan 20     
switch(config-macvrf-20)# rd 10.255.0.0:20
switch(config-macvrf-20)# route-target both 64500:10020
switch(config-macvrf-20)# redistribute learned
!
switch(config-macvrf-20)# address-family evpn
switch(config-router-bgp-af)# neighbor 10.0.0.1 activate
!
switch(config-router-bgp-af)# vrf red
switch(config-router-bgp-vrf-red)# rd 10.255.0.0:0
switch(config-router-bgp-vrf-red)# route-target import evpn 64500:20000
switch(config-router-bgp-vrf-red)# route-target export evpn 64500:20000
switch(config-router-bgp-vrf-red)# router-id 10.255.0.0
!

VXLAN example

interface Loopback0
   ip address 10.0.0.20/32
!
vlan 10
vlan 20
vlan 30
vlan 40
!
interface Ethernet1
   switchport access vlan 10
!
interface Ethernet2
   switchport access vlan 20
!
interface Ethernet3
   switchport access vlan 30
!
interface Ethernet4
   switchport access vlan 40
!
interface Vlan10
   vrf red 
   ip address virtual 192.168.1.0/24
   ip igmp
   pim ipv4 local-interface loopback0
!
interface Vlan20
   vrf red 
   ip address 192.168.2.0/24
   ip igmp
!
interface Vlan30
   vrf blue
   ip address 192.168.1.0/24
   ip igmp
!
interface Vlan40
   vrf blue
   ip address 192.168.2.0/24
   ip igmp
!
interface VXLAN1
   VXLAN source-interface Loopback0
   VXLAN vlan 10 vni 10
   VXLAN vlan 20 vni 20
   VXLAN vlan 30 vni 30
   VXLAN vlan 40 vni 40
   VXLAN vrf red vni 100
   VXLAN vrf blue vni 200
   VXLAN vlan 10 flood group 225.1.1.2
   VXLAN vlan 20 flood group 225.1.1.3
   VXLAN vlan 30 flood group 226.1.1.2
   VXLAN vlan 40 flood group 226.1.1.3
   VXLAN vrf red multicast group 225.1.1.1
   VXLAN vrf blue multicast group 226.1.1.1
!

Show Commands

The following examples are based on the sample topology and configuration in the previous sections.

On the remote VTEP, to display the EVPN routes to the multi-homed CE (20.0.20.2):

switch# show bgp evpn route-type mac-ip 20.0.20.2
BGP routing table information for VRF default
Router identifier 0.0.3.1, local AS number 302
Route status codes: s - suppressed, * - valid, > - active, # - not installed, E - ECMP head, 
                    e - ECMP S - Stale, c - Contributing to ECMP, b - backup
                    % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path
 * >     RD: 10.255.0.0:20 mac-ip 0000.0101.0000 20.0.20.2
                                10.255.0.0            -       100     0       303 300 i
 * >     RD: 10.255.0.1:20 mac-ip 0000.0101.0000 20.0.20.2
                                10.255.0.1            -       100     0       303 301 i

As shown above, there are two EVPN MAC-IP routes for the multi-homed CE.

On the remote PE, to display the installed routes to the multi-homed CE:

switch# show ip route vrf red 20.0.20.2/32

VRF: red
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route, L - VRF Leaked

 B E      20.0.20.2/32 [200/0] via VTEP 10.255.0.0 VNI 20000 router-mac 00:00:78:01:00:00
                               via VTEP 10.255.0.1 VNI 20000 router-mac 00:00:78:04:00:00

Two routes to the multi-homed CE are installed from the two EVPN MAC-IP routes and they form L3 ECMP.

On the remote PE, to check the details of the two routes in BGP RIB:

switch# show ip bgp 20.0.20.2/32 vrf red
BGP routing table information for VRF red
Router identifier 10.255.0.2, local AS number 302
BGP routing table entry for 20.0.20.2/32
 Paths: 2 available
  303 300
    10.255.0.0 from 10.0.2.1 (0.0.1.1), imported EVPN route, RD 10.255.0.0:20
      Origin IGP, metric 0, localpref 100, IGP metric 0, weight 0, received 00:14:43 ago, 
      valid, external, ECMP head, ECMP, best, ECMP contributor
      Extended Community: Route-Target-AS:64500:10020 Route-Target-AS:64500:20000 
      TunnelEncap:tunnelTypeVXLAN EvpnMacMobility:1 EvpnRouterMac:00:00:78:01:00:00
      Remote VNI: 20000
      Rx SAFI: Unicast
  303 301
    10.255.0.1 from 10.0.2.1 (0.0.1.1), imported EVPN route, RD 10.255.0.1:20
      Origin IGP, metric 0, localpref 100, IGP metric 0, weight 0, received 00:14:43 ago, 
      valid, external, ECMP, ECMP contributor
      Extended Community: Route-Target-AS:64500:10020 Route-Target-AS:64500:20000 
      TunnelEncap:tunnelTypeVXLAN EvpnMacMobility:1 EvpnRouterMac:00:00:78:04:00:00
      EvpnNdFlags:pflag
      Remote VNI: 20000
      Rx SAFI: Unicast

The second route has EvpnNdFlags:pflag to indicate that this is a proxy MAC-IP route.

This command shows information about the SBD instance that is created when evpn multicast is configured under an IP VRF:

switch# show bgp evpn instance sbd red
EVPN instance: SBD red
  Route distinguisher: 100:1
  Service interface: VLAN-based
  Local IP address: 10.0.0.20
  Encapsulation type: VXLAN
vtep2#show bgp evpn instance sbd blue
EVPN instance: SBD red
  Route distinguisher: 200:1
  Service interface: VLAN-based
  Local IP address: 10.0.0.20
  Encapsulation type: VXLAN

switch# show bgp evpn route-type imet next-hop 10.0.0.10 detail
BGP routing table information for VRF default
Router identifier 0.0.0.1, local AS number 300
BGP routing table entry for imet 10.0.0.10, Route Distinguisher: 10:1
 Paths: 1 available
  Local
    10.0.0.10 from 10.0.0.1 (0.0.1.1)
      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
      Extended Community: Route-Target-AS:10:1 TunnelEncap:tunnelTypeVXLAN
      VNI: 10
      PMSI Tunnel: PIM-SSM Tree, MPLS Label: 10, Leaf Information Required: false, 
      Tunnel ID: 10.0.0.10, 225.1.1.2
BGP routing table information for VRF default
Router identifier 0.0.0.1, local AS number 300
BGP routing table entry for imet 10.0.0.10, Route Distinguisher: 100:1
 Paths: 1 available
  Local
    10.0.0.10 from 10.0.0.1 (0.0.1.1)
      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
      Extended Community: Route-Target-AS:100:1 TunnelEncap:tunnelTypeVXLAN Multicast 
      Flags: IGMP proxy, OISM-supported, SBD
      VNI: 100
      PMSI Tunnel: Ingress Replication, MPLS Label: 100, Leaf Information Required: false, 
      Tunnel ID: 10.0.0.10

This command shows information about the single SMET route for the group with the RD of VRF red:

switch# show bgp evpn route-type smet multicast 228.1.1.1
BGP routing table information for VRF default
Router identifier 0.0.1.1, local AS number 300
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
                    S - Stale, c - Contributing to ECMP, b - backup
                    % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
          Network                Next Hop              Metric  LocPref Weight  Path
 * >     RD: 100:1 smet (S, G): (*, 228.1.1.1) originating IP: 10.0.0.10
                                 10.0.0.10             -       100     0       i
 * >     RD: 100:1 smet (S, G): (*, 228.1.1.1) originating IP: 10.0.0.20
                                 -                     -        -      0       i
 * >     RD: 100:1 smet (S, G): (*, 228.1.1.1) originating IP: 10.0.0.30
                                 10.0.0.30             -       100     0       i
 * >     RD: 200:1 smet (S, G): (*, 228.1.1.1) originating IP: 10.0.0.20
                                 -                     -        -      0       i
 * >     RD: 200:1 smet (S, G): (*, 228.1.1.1) originating IP: 10.0.0.40
                                 10.0.0.40             -       100     0       i

This command shows information about the SPMSI route for the group:

switch# show bgp evpn route-type spmsi
BGP routing table information for VRF defaultRouter identifier 0.0.0.1, local AS number 300
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
                    S - Stale, c - Contributing to ECMP, b - backup
                    % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
          Network                Next Hop              Metric  LocPref Weight  Path
 * >     RD:  10:1 spmsi (S, G): (*, *) originating IP: 10.0.0.10
                                 10.0.0.10             -       100     0       i
 * >     RD:  10:1 spmsi (S, G): (*, *) originating IP: 10.0.0.20
                                 -                     -       -       0       i
 * >     RD:  20:1 spmsi (S, G): (*, *) originating IP: 10.0.0.20
                                 -                     -       -       0       i
 * >     RD:  30:1 spmsi (S, G): (*, *) originating IP: 10.0.0.20
                                 -                     -       -       0       i
 * >     RD:  40:1 spmsi (S, G): (*, *) originating IP: 10.0.0.20
                                 -                     -       -       0       i
 * >     RD:  10:1 spmsi (S, G): (*, *) originating IP: 10.0.0.30
                                 10.0.0.30             -       100     0       i
 * >     RD:  20:1 spmsi (S, G): (*, *) originating IP: 10.0.0.30
                                 10.0.0.30             -       100     0       i
 * >     RD:  30:1 spmsi (S, G): (*, *) originating IP: 10.0.0.30
                                 10.0.0.30             -       100     0       i
 * >     RD:  40:1 spmsi (S, G): (*, *) originating IP: 10.0.0.30
                                 10.0.0.30             -       100     0       i
 * >     RD:  30:1 spmsi (S, G): (*, *) originating IP: 10.0.0.40
                                 10.0.0.40             -       100     0       i
 * >     RD: 100:1 spmsi (S, G): (*, *) originating IP: 10.0.0.10
                                 10.0.0.10             -       100     0       i
 * >     RD: 100:1 spmsi (S, G): (*, *) originating IP: 10.0.0.20
                                 -                     -       -       0       i
 * >     RD: 200:1 spmsi (S, G): (*, *) originating IP: 10.0.0.20
                                 -                     -       -       0       i
 * >     RD: 100:1 spmsi (S, G): (*, *) originating IP: 10.0.0.30
                                 10.0.0.30             -       100     0       i
 * >     RD: 200:1 spmsi (S, G): (*, *) originating IP: 10.0.0.30
                                 10.0.0.30             -       100     0       i
 * >     RD: 200:1 spmsi (S, G): (*, *) originating IP: 10.0.0.40
                                 10.0.0.40             -       100     0       i

Limitations

The EVPN VXLAN all-active multi-homing integrated routing and bridging feature has the following limitations:

L2 loop-free protocols, such as Spanning Tree Protocol (STP) are not supported between PE-CE with EVPN VXLAN All-Active Multi-homing. The topology must be loop-free. STP must be disabled for the Multihomed VLANs on PEs and CEs.
A limit of two multi-homing destinations are installed at one time for any particular MAC address. Any other valid destinations are ignored in the context of switching unicast traffic, until one of the active destinations is removed. The multi-homing PEs operate normally regardless of the number of PEs on the ethernet segment; this limitation only affects the selection of a destination for unicast traffic.
VXLAN GPE is not supported, so packets cannot be flagged as having been broadcast. As a result, unicast packets that are known at the sender and unknown at the receiver are dropped if the receiving PE is not a designated forwarder. Similarly, unicast packets that are unknown at the sender and known at the receiver are duplicated at the receiving CE. This state automatically resolves itself as the BGP network distributes the appropriate type 1 auto-discovery and type 2 MAC/IP advertisement EVPN routes.
Fast mass withdrawal is not supported, so there may be a delay if an interface harboring a large number of MAC addresses goes down.

Page 3 of 15

End of Support

LDP Entropy Label

Configuring LDP Entropy Label

Show Commands

Limitations

BGP PIC Edge for EVPN VXLAN Routes for Remote VTEP Failures

Configuring BGP PIC Edge for EVPN VXLAN Routes for Remote VTEP Failures

Show Commands

MLAG

Troubleshooting

Limitations

VXLAN DSCP Mapping

Platform Compatibility

Configuration

Show Commands

Limitation

EVPN E-Tree for MPLS and VXLAN

E-Tree Extended Community

EVPN Active/Active Multihoming

EVPN IGP Cost for VTEP Reachability

Configuration Example

Show Commands

EVPN VXLAN Single-Gateway Centralized Routing

Configuration

Show Commands

Limitations

Flood Traffic Filtering with EVPN

Configuration

Show Commands

Inter-VRF Local Route Leaking

Inter-VRF Local Route Leaking using BGP VPN

Accessing Shared Resources Across VPNs

Configuring Inter-VRF Local Route Leaking

Using VXLAN for Encapsulation

Using MPLS for Encapsulation

Route-Distinguisher

Exporting Routes from a VRF

Importing Routes into a VRF

Exporting and Importing Routes using Route Map

Inter-VRF Local Route Leaking using VRF-leak Agent

Configuring Route Maps

EVPN Multicast VRF Leaking

Overview

EVPN Multicast VRF Leaking with a PIM EVPN Gateway (PEG)

Using MLAG as a PEG on an EVPN Network

Static Inter-VRF Route

Configuration

Show Commands

Limitations

VCS to EVPN Hitless Migration

Configuration

Show Commands

Limitations