VM Tracer

This chapter describes VM Tracer configuration and usage and contains these sections:

Introducing VM Tracer

VM Tracer determines the network configuration and requirements of connected VMware hypervisors. The switch uses the VMware SOAP XML API to discover VMware host server components, including the following:
  • Instantiated VMs with the network configuration, including VLANs and distributed or virtual switches.
  • Provide server hardware IPMI data to the network manager.

     

VM Tracer also supports adaptive auto-segmentation, which automatically provisions and prunes VLANs from server-switched ports as VMs are instantiated and moved within the data center.

VM Tracer Description

Cloud operating systems manage large virtualized computing infrastructures, including software and hardware. Cloud operating systems consist of virtual machines and hypervisors:
  • A Virtual Machine (VM) contains software that emulates a computer running on dedicated physical hardware. Multiple VMs share physical computer resources from a single physical device. The operating system controls each VM.
  • A hypervisor, a Virtual Machine Manager (VMM), manages multiple operating systems running concurrently on a physical device.

     

VM Tracer tracks the activity of VMs controlled by hypervisors connected to the switch Ethernet or LAG ports. It supports vSphere versions 6.0– 7.0. The vSphere features include Distributed Virtual Switches (DVS) and VM movement among VMware servers (VMotion).

vSphere components includethe following:
  • ESX and ESXi - Hypervisors that run on VMware host server hardware.
  • vCenter -A centralized tool that manages multiple servers running VMware hypervisors.
  • NSX for vSphere® (NSX-V) - A network virtualization platform delivering networking and security.

     

Monitoring VLAN based configurations requires vCenter access. Monitoring VXLAN-based configurations requires access to vCenter and NSX-V. The following sections describe topologies that monitor these networks:

Monitoring VLAN-based Configurations

vCenter manages ESX hosts and VMs through a central database. VM Tracer identifies interfaces connected to a specified ESX host and sends discovery packets (CDP or LLDP) on interfaces where VM Tracer is enabled. The ESX host updates the vCenter when it receives a discovery packet. VM Tracer reads this data from the vCenter through a SOAP (Simple Object Access Protocol) XML API to associate the ESX host to the connected switch ports. The following figure displays the network topology of this configuration.

Figure 1. VM Tracer Topology – Monitoring VLAN Based Configurations

 

VM Tracer connects to a maximum of four vCenters through a SOAP XML API to discover VMs in the data centers managed by the vCenters. VM Tracer maintains a list of VMs in the data center and collects network-related information about each VM, including the number of Vnics (virtual network interface card), each Vnic MAC address, the connected switch, and the host on which it resides. VM Tracer also identifies the host NICs connected to the switch through the bridge MAC address and the interface port name. VM Tracer then searches for VMs on this host and connects to the vswitch or dvswitch with the uplink mapped to the connected NIC.

VM Tracer creates a VM Table for each connected interface that lists the active VMs, sorted by Vnic MAC address. Each VM entry includes the name, Vnic name, VLAN, switch name, datacenter name, and port group. The VM Table deletes the VM entry after removing the corresponding VM, moving the VM to a different host, or the Vnic no longer exists as part of the vswitch or dvswitch. The VM Table adds an entry after creating a VM or moving a VM to a host connected to the interface. VM Tracer monitors vCenter for VM management updates. If an interface goes down, the VM Table removes all VM entries for that interface.

Monitoring VXLAN-based Configurations

Monitoring VXLAN-based configurations require access to the NSX for vSphere® (NSX-V), in addition to the configuration described in Monitoring VLAN-based Configurations. Each VM Tracer session can communicate with one NSX-V through a REST interface over XML and gathers VXLAN information by polling it on a 30-second polling cycle. VXLAN data that the switch receives from the NSX-V includes the following:
  • VNI range
  • VXLAN segment
  • Multicast address range
  • Network scope

     

The network scope specifies the virtual address space that the VXLAN segments span and the server group (cluster) collections within the segments, which in turn contain a collection of distributed virtual switches (DVS) from ESX hosts within the clusters.

VM Tracer uses this information to build a network model. Communications with NSX-V require a single polling thread that detects network connectivity and constantly updates the local data model.

The following diagram displays the network topology of this configuration.

Figure 2. VM Tracer Topology – Monitoring VXLAN Based Configurations

 

VM Tracer Configuration Procedures

The following sections describe the session configuration process, configuring the NSX-V connection for VXLAN based configurations, and the procedure for enabling VM Tracer on individual interfaces. The switch defines the vmtracer configuration mode and VMtracer mode:
  • vmtracer configuration mode is a command mode for configuring VM Tracer monitoring sessions.
  • VMtracer mode defines an interface state that sends discovery packets to attached vSwitches.

     

Configuring vCenter Monitoring Sessions

A VM Tracer session connects the switch to a vCenter server for downloading data about VMs and vSwitches managed by ESX hosts connected to the switch ports. The switch supports four VM Tracer sessions.

Place the switch in the vmtracer configuration mode to edit session parameters, including the vCenter location and dynamic VLAN usage. Changes take effect by exiting vmtracer configuration mode.

The vmtracer session command places the switch in the vmtracer configuration mode for a specified session. The command either creates a new session or loads an existing session for editing.

 

Example

This command enters the vmtracer configuration mode for the system_1 session.
switch(config)# vmtracer session system_1
switch(vmtracer-system_1)#

 

In vmtracer configuration mode, the url (vmtracer mode), username, and password commands specify the location and the account information that authenticates the switch. The URL parameter must reference a fully formed secure URL.

 

Example

These commands specify the IANA url along with the username and password that allow the switch to access the location.
switch(vmtracer-system_1)# url https://example.com/sdk
switch(vmtracer-system_1)# username a-switch_01
switch(vmtracer-system_1)# password abcde
switch(vmtracer-system_1)#

Default session settings permit auto-segmentation, or the dynamic allocation and pruning of VLANs when creating, deleting, or moving a VM managed by the ESX host to a different host. The autovlan disable command prevents auto-segmentation, regardless of VM activity. The allowed-vlan command specifies the available VLANs when adding or moving a VM. By default, all VLANs are allowed.

 

Examples
  • This command disables auto-segmentation.
    switch(vmtracer-system_1)# autovlan disable
    switch(vmtracer-system_1)#

     

  • These commands enable auto-segmentation and limit the list of allowed VLANs to VLAN 1-2000.
    switch(vmtracer-system_1)# no autovlan disable
    switch(vmtracer-system_1)# allow-vlan 1-2000
    switch(vmtracer-system_1)#

     

The exit command returns the switch to the global configuration mode and enables the VM Tracer session. The vmtracer configuration mode can be re-entered for this session to edit session parameters.

 

Example

This command exits vmtracer configuration mode.
switch(vmtracer-system_1)# exit
switch(config)#

 

The no vmtracer session command disables the session and removes it from running-config.

 

Example

This command disables and deletes the system_1 VM Tracer session.
switch(config)# no vmtracer session system_1
switch(config)#

 

Configuring vShield Monitoring Sessions

The switch must communicate with an NSX for vSphere® (NSX-V) to monitor VXLAN-based VMware configurations. The vmtracer-VXLAN configuration mode specifies the location and user account data that permits the switch to access an NSX-V within the configuration mode vmtracer session.

Place the switch in the vmtracer configuration mode to edit session parameters, including the vCenter location and dynamic VLAN usage. Changes take effect by exiting vmtracer mode.

Execute the vxlan command from the vmtracer mode for a specified session and places the switch in the vmtracer-VXLAN configuration mode for that session. Each VM Tracer session can be associated with one vShield instance.

 

Example

These commands create the vShield instance for the VMTracer session named vnet-1.
switch(config)# vmtracer session vnet-1
switch(config-vmtracer-vnet-1)# vxlan
switch(config-vmtracer-vnet-1-vxlan)#

 

In the vmtracer-vxlan configuration mode, the url, username (vmtracer-vxlan mode), and password (vmtracer-VXLAN mode) commands specify the vShield server location and the account information that authenticates the switch to the vShield server. The url parameter must reference a fully formed secure url, such as https://vcshield.democorp.com/sdk.

 

Example

These commands specify the vShield URL along with the username and password that allow the switch to access the vShield server.
switch(config-vmtracer-vnet-1-vxlan)# url https://vshieldserver.company1.org/sdk
switch(config-vmtracer-vnet-1-vxlan)# username a-shield_01
switch(config-vmtracer-vnet-1-vxlan)# password home
switch(config-vmtracer-vnet-1-vxlan)#

 

Enabling VMtracer Mode

VMtracer mode provides an interface setting that enables interfaces to send discovery packets to the connected vSwitch. The vmtracer command enables VMtracer mode on the configuration mode interface.

 

Examples
  • These commands enable VMtracer mode on the interface Ethernet3.
    switch(config)# interface Ethernet3
    switch(config-if-Et3)# vmtracer vmware-esx
    switch(config-if-Et3)#

     

    The no vmtracer command disables vmtracer mode on the configuration mode interface.

     

  • This command disables vmtracer mode on the interface ethernet 3.
    switch(config-if-Et3)# no vmtracer vmware-esx
    switch(config-if-Et3)#

     

Displaying VM Tracer Data

Displaying Session Status

The show vmtracer session command displays information about the specified session.

Without the detail parameter, the command displays connection parameters and status for the vCenter associated to the specified session.

 

Example

This command displays connection parameters for the vCenter associated with the system_1 session.
switch# show vmtracer session system_1
    vCenter URL https://vmware-vcenter1/sdk
    username arista
    password arista
    Session Status Disconnected

 

With the detail parameter, the command displays connection status and data concerning messages the vCenter previously received from ESX hosts connected to the switch.

 

Example

This command displays connection parameters and message details for the vCenter associated with the system_1 session.
switch# show vmtracer session system_1 detail
    vCenter URL https://vmware-vcenter1/sdk
    username arista
    sessionState Connected
    lastStateChange 19 days, 23:03:59 ago
    lastMsgSent CheckForUpdatesMsg
    timeOfLastMsg 19 days, 23:14:09 ago
    resonseTimeForLastMsg 0.0
    numSuccessfulMsg 43183
    lastSuccessfulMsg CheckForUpdatesMsg
    lastSuccessfulMsgTime 19 days, 23:14:19 ago
    numFailedMsg 1076
    lastFailedMsg CheckForUpdatesMsg
    lastFailedMsgTime 19 days, 23:14:09 ago
    lastErrorCode Error -1 fault: SOAP-ENV:Client [no subcode]
    "End of file or no input: Operation interrupted or timed out after 600s send 
or     600s receive delay"
    Detail: [no detail]
    CheckForUpdates:

 

Displaying VM Interfaces

The show vmtracer interface command displays the VM interfaces (Vnics) active on switch interfaces with vmtracer mode enabled. For each Vnic, the command displays the name of the attached VM, the adapter name, the VLAN, the VM power state, and the presence status of its MAC address in the switch MAC table.

This command displays the Vnics connected to all VM Tracer-enabled interfaces.

switch# show vmtracer interface

Ethernet8 : example.com
    VM Name VM Adapter VLAN Status
    esx3.aristanetworks.com vmk0 0 Up/Down
    vspheremanagement Network adapter 1 0 Up/Down

Ethernet15 : example.om
    VM Name VM Adapter VLAN Status
    Openview Network adapter 1 123 Up/Down
    VmTracerVm Network adapter 1 123 Down/Down

Ethernet23 : example.com
    VM Name VM Adapter VLAN Status

Ethernet24 : example.com
    VM Name VM Adapter VLAN Status

 

Displaying VMs

The show vmtracer vm command displays VM interfaces (Vnics) accessible to the VM Tracer-enabled interfaces. For each active listed VM, the command displays tghe name, adapter, and the connected hypervisor.
  • This command displays the VMs connected to all VM Tracer-enabled interfaces.
    switch# show vmtracer vm
        VM Name VM Adapter Interface VLAN
        Openview Network adapter 1 Et15 123
        vspheremanagement Network adapter 1 Et8 0
        VmTracerVm Network adapter 1 Et15 123
        example.com vmk0 Et8 0

     

  • This command displays connection data for the VMs connected to all VM Tracer-enabled interfaces.
    switch# show vmtracer vm detail
    VM Name Openview
       intf : Et15
       vnic : Network adapter 1
       mac : 00:0c:29:ae:7e:90
       portgroup : dvPortGroup
       vlan : 123
       switch : vds
       host : example.com

     

VM Tracer Commands

allowed-vlan

The allowed-vlan command specifies the VLANs that may be added when adding or moving a VM from the hypervisor connected to the session specified by the vmtracer mode. By default, all VLANs are allowed.

 

Command Mode

Vmtracer Configuration

 

Command Syntax

allowed-vlan [VLAN_LIST]

no allowed-vlan

default allowed-vlan vlan

 

Parameters

VLAN_LIST     The VLAN list or the edit actions to the current VLAN list. Valid v_range formats include number, or number range.
  • v_range     The list consists of the v_range VLANs.
  • add v_range     The v_range VLANs are added to the current VLAN list.
  • all     The list consists of all VLANs (1-4094).
  • except v_range     The list consists of all VLANs except for those specified by v_range.
  • none     The list of VLANs is empty.
  • remove v_range     The v_range VLANs are removed from the current VLAN list.

 

Related Command

vmtracer session vmtracer session places the switch in the vmtracer configuration mode.

 

Examples
  • This command sets the list of allowed VLANs to 1 through 2000.
    switch(vmtracer-system_1)# allow-vlan 1-2000
    switch(vmtracer-system_1)#

     

  • This command adds VLANs to 2501 through 3000.
    switch(vmtracer-system_1)# allow-vlan add 2051-3000
    switch(vmtracer-system_1)#

autovlan disable

Default VM Tracer session settings enable auto provisioning, which allows the dynamic assignment and pruning of VLANs when creating, deleting, or moving a VM attached to the ESX connected to the switch to a different ESX host. The autovlan setting controls auto provisioning.

The autovlan disable command disables auto provisioning, which prevents the creation or deletion of VLANs regardless of VM activity. The allowed-vlan command specifies the VLANs that may be added when a VM is added or moved. By default, all VLANs are allowed.

The no autovlan disable command enables the creation and deletion of VLANs caused by VM activity. This is the default setting.

 

Command Mode

Vmtracer Configuration

 

Command Syntax

autovlan disable

no autovlan disable

default autovlan disable

 

Related Command

vmtracer session places the switch in the vmtracer configuration mode.

 

Example

This command disables dynamic VLAN creation or pruning within the configuration mode VM Tracer session.
switch(vmtracer-system_1)# autovlan disable
switch(vmtracer-system_1)#

password

The password command specifies the token that authorizes the username to the vCenter associated with the VM Tracer mode session.

 

Command Mode

Vmtracer Configuration

 

Command Syntax

password [ENCRYPTION] [password]

 

Parameters
  • ENCRYPTION - Encryption level of the password.
    • no parameter - The password in a clear-text string.
    • 0 - The password in a clear-text string. Equivalent to no parameter.
    • 7 the password is an encrypted string.

       

  • password - Text that authenticates the username.
    • password is a clear-text string if ENCRYPTION specifies clear text.
    • password is an encrypted string if ENCRYPTION specifies an encrypted string.

 

Related Command

vmtracer session places the switch in the vmtracer configuration mode.

 

Example

This command configures abode as the clear-text string that authorizes the username a-switch_01 located at example.com/sdk.
switch(vmtracer-system_1)# url https://example.com/sdk
switch(vmtracer-system_1)# username a-switch_01
switch(vmtracer-system_1)# password abcde
switch(vmtracer-system_1)#

password (vmtracer-VXLAN mode)

The password command specifies the token that authorizes the username on the NSX for vSphere® (NSX-V) server located at the URL configured for the configuration mode VM Tracer. The switch uses this account to access NSX-V information.

The password statement is replaced in running-config for the configuration mode interface by a subsequent password command. The statement is removed by deleting the NSX-V instance through a vxlan command in vmtracer configuration mode.

 

Command Mode

Vmtracer-VXLAN Configuration

 

Command Syntax

password [ENCRYPTION] password

 

Parameters
  • ENCRYPTION     encryption level of the password.
    • no parameterpassword is a clear-text string.
    • 0 theis a clear-text string. Equivalent to no parameter.
    • 7 the password is an encrypted string.

       

  • password     text that autorizes the username.
    • password is a clear-text string if ENCRYPTION specifies clear text.
    • password is an encrypted string if ENCRYPTION specifies an encrypted string.

 

Related Commands

vxlan places the switch in the vmtracer-vxlan configuration mode.

 

Example

This command configures 5678 as the clear-text string that authorizes the username admin to the NSX-V located at https://example.com/sdk.
switch(config)# vmtracer session vnet-1
switch(config-vmtracer-vnet-1)# vxlan
switch(config-vmtracer-vnet-1-vxlan)# url https://example.com/sdk
switch(config-vmtracer-vnet-1-vxlan)# username admin
switch(config-vmtracer-vnet-1-vxlan)# password 5678
switch(config-vmtracer-vnet-1-vxlan)# exit
switch(config-vmtracer-vnet-1)# show active
vmtracer session vnet-1
   allowed-vlan 1-4094
   vxlan
      url https://example.com/sdk
      username admin
      password 7 s2Xq4GSBlYU=
switch(config-vmtracer-vnet-1)#

show vmtracer all

The show vmtracer all command displays VM Tracer data for all switches with the vSphere scope.

 

Command Mode

EXEC

 

Command Syntax

show vmtracer all

 

Example

This command displays data for both switches in the vSphere scope.
switch> show vmtracer all
Switch : a109(10.10.30.109)
Ethernet49     : 10.102.28.3/10G
   VM Name             VM Adapter          VLAN      Status    State
   ABCD                Network adapter 2   native    Up/--     --

Switch : a164(10.10.30.(172.22.30.164)
Ethernet49     : 10.102.28.3/10G Storage Network/dvUplink1
   VM Name             VM Adapter          VLAN      Status    State
   WXYZ                Network adapter 2   native    Up/--     --
switch>

show vmtracer interface

The show vmtracer interface command displays the VM interfaces (Vnics) active on the VM Tracer enabled interface. For each Vnic, the command displays the name of the attached VM, the adapter name, its VLAN, the VM power state, and the presence status of its MAC address in the switch MAC table.

 

Command Mode

EXEC

 

Command Syntax

show vmtracer interface [INT_NAME] [INFO_LEVEL]

 

Parameters
  • INT_NAME     the interfaces to be configured. Values include:
    • no parameter     command returns information for all interfaces.
    • ethernet e_range      Ethernet interface range.
    • port-channel p_range      Port Channel interface range.

      Valid e_range and p_range formats include number, number range, or comma-delimited list of numbers and ranges.

       

  • INFO_LEVEL     specifies information that the command returns.
    • no parameter     connection parameters and status for VM associated to specified sessions.
    • detail      connection status and data concerning messages the VM.
    • host     name of the connected host.

 

Examples
  • This command displays the Vnics connected to all VM Tracer enabled interfaces.
    switch > show vmtracer interface
    
    Ethernet8 : example.com
        VM Name                     VM Adapter          VLAN     Status
        esx3.aristanetworks.com     vmk0                0        Up/Down
        vspheremanagement           Network adapter 1   0        Up/Down
    
    Ethernet15 : example.com
        VM Name                     VM Adapter          VLAN      Status
        Openview                    Network adapter 1   123       Up/Down
        VmTracerVm                  Network adapter 1   123       Down/Down
    
    Ethernet23 : example.com
        VM Name                     VM Adapter          VLAN       Status
    switch>

     

  • This command displays the Vnics connected to the interface Ethernet8.
    switch> show vmtracer interface Ethernet8
    
    Ethernet8 : example.com
        VM Name                     VM Adapter          VLAN      Status
        example.com                 vmk0                0         Up/Down
        vspheremanagement           Network adapter 1   0         Up/Down
    switch>

show vmtracer session

The show vmtracer session command displays vCenter and vShield connection information for a specified VM Tracer session.

 

Command Mode

EXEC

 

Command Syntax

show vmtracer session [SESSION_LIST]

 

Parameters

SESSION_LIST     VM Tracer sessions for which the command returns information.
  • no parameter     all configured VM Tracer sessions.
  • session_name     name of one VM Tracer session.

 

Example

This command displays connection parameters associated to the abcde session.
switch> show vmtracer session abcde
Session abcde
vCenter URL       https://example.com/sdk 
username                        Administrator 
autovlan                        enabled 
allowed-vlans     1-4094 
sessionState      Connected 
VShield URL       https:/vmware-vshield5.1.xyz.abcde.com 
username                    admin 
sessionState      Connected

switch>

show vmtracer session vcenter

The show vmtracer session vcenter command displays vCenter information for a specified VM Tracer session.

 

Command Mode

EXEC

 

Command Syntax

show vmtracer session session_name vcenter [INFO_LEVEL]

 

Parameters
  • session_name     VM Tracer sessions for which the command returns information.
  • INFO_LEVEL     specifies information that the command returns.
    • no parameter      displays connection and status information for the specified vCenter.
    • detail      displays connection, status, and history information for the specified vCenter.

 

Examples
  • This command displays connection parameters for the vCenter associated to the abcde session.
    switch> show vmtracer session abcde vcenter
    
    Session           abcde
    vCenter URL       https://vmware-vcenter5.1/sdk
    username          Administrator
    autovlan          enabled
    allowed-vlans     1-4094
    sessionState      Connected
    switch>

     

  • This command displays connection parameters and history details from the vCenter associated to the abcde session.
    switch> show vmtracer session abcde vcenter detail
    
    Session                        abcde
    vCenter URL                    https://vmware-vcenter5.1/sdk
    username                       Administrator
    autovlan                       enabled
    allowed-vlans                  1-4094
    SessionState                   Connected
    lastStateChange                2:46:50 ago
    lastMsgSent                    Query network hint message
    timeOfLastMsg                  0:00:20 ago
    responseTimeForLastMsg         0.000102301000479
    numSuccessfulMsg               998
    lastSuccessfulMsg              Query network hint message
    lastSuccessfulMsgTime          0:00:20 ago
    numFailedMsg                   0 
    lastFailedMsg                  -- 
    lastFailedMsgTime              never
    lastErrorCode                  --
    switch>

show vmtracer session vsm

The show vmtracer session vsm command displays NSX-V information for a specified VM Tracer session.

 

Command Mode

EXEC

 

Command Syntax

show vmtracer session session_name vsm [INFO_LEVEL]

 

Parameters
  • session_name     VM Tracer sessions for which the command returns information.
  • INFO_LEVEL     specifies information that the command returns.
    • no parameter      connection and status information for the specified NSX-V.
    • detail      connection, status, and history information for the specified NSX-V.

 

Examples
  • This command displays connection parameters for the NSX-V associated to the abcde session.
    switch> show vmtracer session abcde vsm
    
    Session           abcde
    VShield URL       https://example.com/sdk
    username          admin
    sessionState      Connected
    switch>

     

  • This command displays connection parameters and history details from the vShield Manager associated to the abcde session.
    switch> show vmtracer session abcde vsm detail
    
    Session                        abcde
    VShield URL                    https://vmware-vshield5.1/
    username                       admin
    SessionState                   Connected
    LaststateChange                19 days, 23:14:19 ago
    LastMsgSent                    /api/2.0/vdn/scopes
    timeOfLastMsg                  1 days, 13:22:09 ago
    responseTimeForLastMsg         0.3 sec
    numSuccessfulMsg               3649
    lastSuccessfulMsg              /api/2.0/vdn/scopes
    lastSuccessfulMsgTime          0:00:00 ago
    numFailedMsg                   1
    lastFailedMsg                  /api/2.0/vdn/config/segments
    lastFailedMsgTime              10 days, 1:15:29 ago
    lastErrorCode                  CURLE_COULDNT_RESOLVE_HOST - Couldn't resolve host
                                   The given remote host was not resolved.
    switch>

show vmtracer vm

The show vmtracer vm command displays VMs interfaces (Vnics) accessible to VM Tracer enabled interfaces. For each active VM, the command displays the name of the VM, its adapter, and the hypervisor to which it connects.

 

Command Mode

EXEC

 

Command Syntax

show vmtracer [INT_NAME] vm [VM_LIST]

 

Parameters
  • INT_NAME     the interfaces name Values include:
    • no parameter     command returns information for all interfaces.
    • interface ethernet e_range      Ethernet interface range.
    • interface port-channel p_range      Port Channel interface range.

      Valid e_range and p_range formats include a number, number range, or comma-delimited list of numbers and ranges.

       

  • VM_LIST     The virtual machines for which the command displays information. Options include:
    • no parameter    command returns information for all present VMs.
    • vm_name     command returns information only for specified VM.

 

Related Command

The show vmtracer vm detail command displays connection information for one or more specified VMs.

 

Example

This command displays the VMs connected to all VM Tracer enabled interfaces.
switch> show vmtracer vm
VM Name             Esx Host          Interface  VLAN     Status

vCenter1            172.22.28.8       Po45       native   Down/Down
vCenter2            172.22.28.8       Po45       native   Up/Up
vCenter3            172.22.28.8       Po45       11       Down/Down
vCenter4            172.22.28.8       Po45       native   Down/Down
VMKernel                              Po43       native   Up/Up
demo vcenter 5 clone                  Po43       native   Up/Up
switch>

show vmtracer vm detail

The show vmtracer vm detail command displays connection data for VMs interfaces (Vnics) accessible to VM Tracer enabled interfaces.

 

Command Mode

EXEC

 

Command Syntax

show vmtracer vm [VM_LIST] detail

 

Parameters

VM_LIST     The virtual machines for which the command displays information. Options include:
  • no parameter     command returns information for all present VMs.
  • vm_name     command returns information only for specified VM.

 

Examples
  • This command displays connection data for the VMs connected to all VM Tracer enabled interfaces.
    switch# show vmtracer vm vmcenter1
    VM Name  vCenter1 Server App
      Interface   :     Po45
      vNIC        :     Network adapter 1
      MAC         :     00:31:22:8e:b8:41
      Portgroup   :     VM Network
      VLAN        :     native
      Switch      :     Switch2
      Status      :     Down/Down
      Host        :     10.22.18.28
      Data Center :     vcenter-5
    switch>

     

  • This command displays connection data for the VMs connected to all VM Tracer enabled interfaces.
    switch> show vmtracer vm detail
    VM Name  vCenter1 Server App
      Interface   :     Po45
      vNIC        :     Network adapter 1
      MAC         :     00:31:22:8e:b8:41
      Portgroup   :     VM Network
      VLAN        :     native
      Switch      :     Switch2
      Status      :     Down/Down
      Host        :     10.22.18.28
      Data Center :     vcenter-5
    
    VM Name  vCenter2 Server App
      Interface   :     Po45
      vNIC        :     vmk0
      MAC         :     00:33:23:3c:e1:4e
      Portgroup   :     Management Network
      VLAN        :     native
    
    switch>

show vmtracer vnic counters

The show vmtracer interface vnic counters command displays input and output packet counts for VM interfaces (Vnics) active on the specified interface or VM.

 

Command Mode

EXEC

 

Command Syntax

show vmtracer [ENTITY] vnic counters

 

Parameters

ENTITY     the virtual machine or interface over which statistics are gathered and displayed.
  • no parameter     command returns information for all active VMs.
  • interface ethernet e_range      Ethernet interface range.
  • interface port-channel p_range      Port Channel interface range.
  • vm vm_name     command returns information for specified VM.

    Valid e_range and p_range formats include a number, number range, or comma-delimited list of numbers and ranges.

 

Example

This command displays the Vnics connected to interface ethernet 24.
switch> show vmtracer interface ethernet 24 vnic counters
Physical Intf: Ethernet24
Host: 10.17.28.8/site1/dvUplink1
VM Name        vNic                  Input Pkt/Byte/%         Output Pkt/Byte/%
vCenter1       Network adapter 2     2550/   187175/  0.6     6/        360/  0.0
vCenter2       Network adapter 2     418615/ 30678024/ 99.4   1904439/ 1145654613/100.0
Summary                              421165/ 30865199/100.0   1904445/ 1145654973/100.0
switch>

show vmtracer vxlan segment

The show vmtracer vxlan segment command displays information about the VXLAN segments that are managed by the connected NSX for vSphere® (NSX-V).

 

Command Mode

EXEC

 

Command Syntax

show vmtracer segment ENTITY

 

Parameters

ENTITY     specifies the information that the command displays. Options include:
  • no parameter    displays information for VXLAN segments.
  • pool     displays resource pools available to segments.
  • pool pool_name      displays connection information about the specified pool.
  • range      displays the VNI range of the managed segments.

 

Examples
  • This command displays the VXLAN segments managed by the NSX-V.
    switch> show vmtracer vxlan segment
    Name                 VNI     Multicast IP     Network Scope 
    ------------------------------------------------------------ 
    Eng Wire             5002    237.0.0.1        abcde
    HR Wire              5000    237.0.0.2        abcde
    
    switch>

     

  • This command displays the resource pools available to the VXLANs.
    switch> show vmtracer vxlan segment pool
    Name                  Description                     Segments 
    ------------------------------------------------------------------------ 
    abcde                 Spans Cluster 1 and Cluster 2   Eng Wire, HR Wire 
    
    switch>

     

  • This command displays connection and packet information for the abcde pool.
    switch> show vmtracer vxlan segment pool abcde
    Name:          abcde
    Description:   Spans Cluster 1 and Cluster 2 
    Segments:      Eng Wire, HR Wire 
    
    VXLAN Segment  Cluster   Host                 VTEP IP           DVS       VLAN  MTU
    Eng Wire       Cluster2  test2.example.com    10.168.200.1/24  dvs-test2  200   1600
    Eng Wire       Cluster1  test2.example.com    10.168.100.1/24  dvs-test1  100   1600
    HR Wire        Cluster1  test2.example.com    10.168.100.1/24  dvs-test1  100   1600
    HR Wire        Cluster2  test2.example.com    10.168.200.1/24  dvs-test2  200   1600
    switch>

     

  • This command displays the VNI range of the VXLAN segments.
    switch> show vmtracer vxlan segment range
    
    VNI Range            Multicast IP Range
    --------------------------------------------
    5000 - 5024          237.0.0.1 - 237.0.0.117 
    
    Name                 VNI     Multicast IP     Network Scope
    ----------------------------------------------------------- 
    HR Wire              5002    237.0.0.1        abcde
    
    Eng Wire             5000    237.0.0.2        abcde
    switch>

show vmtracer vxlan vm

The show vmtracer vxlan vm command displays the VXLAN segments, their VTEP IP numbers, and their VM endpoints that are managed by the connected NSX for vSphere® (NSX-V).

 

Command Mode

EXEC

 

Command Syntax

show vmtracer vxlan vm

 

Example

This command displays the VM endpoints of the VXLAN segments managed by the NSX-V.
switch> show vmtracer vxlan vm
VXLAN Segment       VTEP IP            VLAN  VMs
Eng Wire            192.168.200.1/24   200   Eng VM3, Eng VM2
Eng Wire            192.168.100.1/24   100   Eng VM1
HR Wire             192.168.100.1/24   100   HR VM2, HR VM1
HR Wire             192.168.200.1/24   200   --
switch>

source-interface

The source-interface command allows you to connect to a remote vCenter endpoint by using the primary address of the interface as the source IP address. If the interface is not specified, the source IP address is determined by the routing table.

The no source-interface and default source-interface commands restore default behavior by removing the source-interface command from the running-config.

 

Command Mode

Vmtracer Configuration

 

Command Syntax

source-interface [INTERFACE_NAME]

no source-interface

default source-interface

 

Parameters

INTERFACE_NAME     specifies the interface for which the information is displayed. Options include:
  • Ethernet e_num     specifies the Ethernet interface number.
  • Loopback l_num     specifies the loopback interface number. Value ranges from 0 to 2100.
  • Management m_num     specifies the management interface number. The values are 1 or 2.
  • Port-Channel {lag_num | lag_num.sub_num}     specifies the port-channel interface number. Value of interface ranges from 1 to 2000. Value of sub-interface ranges from 1 to 4094.
  • Tunnel tunnel_num     specifies the tunnel interface number. Value ranges from 1 to 255.
  • UnconnectedEthernet port_num     specifies the unconnected Ethernet port number. Value ranges from 1 to 8.
  • VLAN vlan_num     specifies the VLAN interface number. Value ranges from 1 to 4094.

 

Related Commands

The vmtracer session command places the switch in the vmtracer configuration mode.

 

Examples
  • This command configures VM Tracer to use interface Ethernet 17 to derive the source address for session packets.
    switch(config)# vmtracer session system_1
    switch(config-vmtracer-session-system_1)# source-interface Ethernet 17
    switch(config-vmtracer-session-system_1)#

     

  • This command configures interface Loopback 0 for VM Tracer session.
    switch(config)# vmtracer session system_1
    switch(config-vmtracer-session-system_1)# source-interface Loopback 0
    switch(config-vmtracer-session-system_1)#

     

  • This command configures interface management 1 for VM Tracer session.
    switch(config)# vmtracer session system_1
    switch(config-vmtracer-session-system_1)# source-interface management 1
    switch(config-vmtracer-session-system_1)#

     

  • This command configures port-channel 10 for VM Tracer session.
    switch(config)# vmtracer session system_1
    switch(config-vmtracer-session-system_1)# source-interface port-channel 10
    switch(config-vmtracer-session-system_1)#

     

  • This command configures interface tunnel 25 for VM Tracer session.
    switch(config)# vmtracer session system_1
    switch(config-vmtracer-session-system_1)# source-interface tunnel 25
    switch(config-vmtracer-session-system_1)#

     

  • This command configures unconnected interface Ethernet 1 for VM Tracer session.
    switch(config)# vmtracer session system_1
    switch(config-vmtracer-session-system_1)# source-interface unconnected Ethernet 1
    switch(config-vmtracer-session-system_1)#

     

  • This command configures interface vlan 25 for VM Tracer session.
    switch(config)# vmtracer session system_1
    switch(config-vmtracer-session-system_1)# source-interface vlan 25
    switch(config-vmtracer-session-system_1)#

url (vmtracer mode)

The url command specifies the vCenter server location that is monitored by the session being edited by the current vmtracer mode. The command must reference a fully formed secure url.

 

Command Mode

Vmtracer Configuration

 

Command Syntax

url url_name

 

Parameter

url_name     location of the vCenter server. Valid formats include IP address (dotted decimal notation) and fully qualified domain name.

 

Related Commands

The vmtracer session command places the switch in the vmtracer configuration mode.

 

Example

This command specifies the location of the vCenter monitored by the system_1 VM Tracer session.
switch(vmtracer-system_1)# url https://example.com/sdk
switch(vmtracer-system_1)#

url

The url command specifies the NSX for vSphere® (NSX-V) server location monitored for VXLAN information by the configuration mode VM Tracer session. The command must reference a fully formed secure URL.

The url statement is replaced in running-config for the configuration mode session by a subsequent url command. The statement is removed by deleting the NSX-V instance through a vxlan command in vmtracer configuration mode.

 

Command Mode

Vmtracer-VXLAN Configuration

 

Command Syntax

url url_name

 

Parameter

url_name     location of the NSX-V server. Valid formats include IP address (dotted decimal notation) and fully qualified domain name.

 

Related Commands

The vxlan command places the switch in the vmtracer-vxlan configuration mode.

 

Example

This command configures the location of the NSX-V monitored by the vnet-1 VM Tracer session.
switch(config)# vmtracer session vnet-1
switch(config-vmtracer-vnet-1)# vxlan
switch(config-vmtracer-vnet-1-vxlan)# url https://example.com/sdk
switch(config-vmtracer-vnet-1-vxlan)# exit
switch(config-vmtracer-vnet-1)# show active
vmtracer session vnet-1
   allowed-vlan 1-4094
   vxlan
      url https://example.com/sdk
switch(config-vmtracer-vnet-1)#

username

The username command identifies the switch account name on the vCenter server. The switch uses this user name to access vCenter information.

 

Command Mode

Vmtracer Configuration

 

Command Syntax

username name_string

 

Parameter

name_string     vCenter account user name. Parameter must match the user name configured on the vCenter.

 

Related Commands

The vmtracer session command places the switch in the vmtracer configuration mode.

 

Example

This command configures the user name for the vCenter associated with the system_1 session. The session uses this user name to log into the vCenter server.
switch(vmtracer-system_1)# username a-switch_01
switch(vmtracer-system_1)#

username (vmtracer-vxlan mode)

The username command identifies the switch’s account name on the NSX for vSphere® (NSX-V) server located at the URL configured for the configuration mode VM Tracer. The switch uses this user name to access NSX-V information.

The username statement is replaced in running-config for the configuration mode interface by a subsequent username command. The statement is removed by deleting the NSX-V instance through a vxlan command in the vmtracer configuration mode.

 

Command Mode

Vmtracer-VXLAN Configuration

 

Command Syntax

username name_string

 

Parameter

name_string     NSX-V account user name. Parameter must match a user name configured on the NSX-V.

 

Related Commands

The vxlan command places the switch in the vmtracer-vxlan configuration mode.

 

Example

This command configures the user name of admin for the NSX-V located at the URL specified by the URL command.
switch(config)# vmtracer session vnet-1
switch(config-vmtracer-vnet-1)# vxlan
switch(config-vmtracer-vnet-1-vxlan)# url https://example.com/sdk
switch(config-vmtracer-vnet-1-vxlan)# username admin
switch(config-vmtracer-vnet-1-vxlan)# exit
switch(config-vmtracer-vnet-1)# show active
vmtracer session vnet-1
   allowed-vlan 1-4094
   vxlan
      url https://example.com/sdk
      username admin
switch(config-vmtracer-vnet-1)#

vmtracer

The vmtracer command enables vmtracer mode on the configuration mode interface. Interfaces with vmtracer mode enabled send discovery packets to the connected vSwitch.

The no vmtracer and default vmtracer commands disable vmtracer mode on the configuration mode interface by removing the corresponding vmtracer command from running-config.

 

Command Mode

Interface-Ethernet Configuration Interface-Port-channel Configuration

 

Command Syntax

vmtracer HOST_TYPE

no vmtracer HOST_TYPE

default vmtracer HOST_TYPE

 

Parameters

HOST_TYPE - the type of hypervisor that controls the vSwitch to which the interface connects.
  • vmware-esx - ESX or ESXI hypervisor (VMware).

 

Examples
  • These commands enable the vmtracer mode on the interface Ethernet 3.
    switch(config)# interface Ethernet 3
    switch(config-if-Et3)# vmtracer vmware-esx 
    switch(config-if-Et3)#

     

  • This command disables the vmtracer mode on the interface Ethernet 3.
    switch(config-if-Et3)# no vmtracer vmware-esx 
    switch(config-if-Et3)#

vmtracer session

The vmtracer session command places the switch in the vmtracer mode for the specified session. The command creates a new session or loads an existing session for editing.

A VM Tracer session connects the switch to a vCenter server at a specified location, then downloads data about VMs and vSwitches managed by ESX hosts connected to switch ports. The switch supports a maximum of four VM Tracer sessions.

Configure VM Tracer session parameters in the vmtracer mode. Parameters configured in the vmtracer mode include the vCenter location and dynamic VLAN usage.

The no vmtracer session and default vmtracer session commands disable the session and remove the configuration from running-config.

 

Command Mode

Global Configuration

 

Command Syntax

vmtracer session name

no vmtracer session name

default vmtracer session name

 

Parameter

name     The label assigned to the VM Tracer session.

 

Examples
  • This command enters vmtracer mode for the system_1 session.
    switch(config)# vmtracer session system_1
    switch(vmtracer-system_1)#

     

  • This command disables the system_1 VM Tracer session. The system_1 session and removes all of the parameters from running-config
    switch(config)#no vmtracer session system_1 
    switch(config)#

vrf

The vrf command allows the switch to communicate with a vCenter server by enabling VmTracer configuration mode. By default, VmTracer is enabled only in the default vrf command.

 

Command Mode

Vmtracer Configuration

 

Command Syntax

vrf vrf_name

 

Parameter

vrf_name     specifies information of the corresponding VRF.

 

Example

These commands place the VRF vrf1 in the vmtracer configuration mode.
switch(config)# vmtracer session system_1
switch(config-vmtracer-session-system_1)# vrf vrf1
switch(config-vmtracer-session-system_1)#

vxlan

The vxlan command places the switch in the vmtracer-vxlan configuration mode. To monitor VXLAN based VMware configurations, the switch must communicate with a NSX for vSphere® (NSX-V). The vmtracer-vxlan configuration mode specifies the location and user account data that allows the switch to access a NSX-V within the configuration mode vmtracer session. Each VM Tracer session can be associated with one NSX-V instance.

The no vxlan and default interface vxlan commands delete the NSX-V instance from the configuration mode vmtracer session by removing all of the vmtracer-vxlan mode commands from running-config.

 

Command Mode

Vmtracer Configuration

 

Command Syntax

vxlan

no vxlan

default vxlan

 

Related Command

The vmtracer session command places the switch in the vmtracer configuration mode.

 

Example

These commands create the vShield instance for the VMTracer session named vnet-1.
switch(config)# vmtracer session vnet-1
switch(config-vmtracer-vnet-1)# vxlan
switch(config-vmtracer-vnet-1-vxlan)#

Sample Configurations

EVPN VXLAN IRB Sample Configuration

In the following topology, we are connecting a Layer 2 site with a Layer 3 site using Layer 3 EVPN (type-5 route). The right side leaves are MLAG leaves and have SVI 10 in VRF-Blue. A number of directly connected hosts are simulated behind the right side leaf. The left side leaves are individual leaves that connect with a remote switch in vrf VRF-Blue to learn Layer 3 routes using BGP. The left side leaves are configured as two independent Layer 3 only VTEPs.

Figure 1. Layer 3 EVPN Configuration

To provide VXLAN routing and bridging between the two MLAG domains, each leaf switch is EVPN peering with the four spine switches via a loopback interface.

eBGP Underlay Configuration: Leaf-11

Underlay configuration is straightforward and all neighbors are eBGP. Since all leaves share the same AS number, the allowas-in command was added in the leaf.

interface Ethernet1
   description Spine-1-et1/1
   mtu 9214
   no switchport
   ip address 172.168.1.1/31

interface Ethernet8/1
   description ck428-et8/1
   speed forced 40gfull
   no switchport
   ip address 172.168.1.10/31

interface Loopback0
   ip address 1.1.1.11/32

ip prefix-list loopback
   seq 10 permit 1.1.1.0/24 ge 24
!
route-map loopback permit 10
   match ip address prefix-list loopback

router bgp 65004
neighbor SPINE peer-group
   neighbor SPINE remote-as 65001
   neighbor SPINE allowas-in 1
   neighbor SPINE soft-reconfiguration inbound all
   neighbor SPINE send-community
   neighbor 172.168.1.0 peer-group SPINE
   neighbor 172.168.1.11 remote-as 65003
   redistribute connected route-map loopback

eBGP Underlay Configuration: Spine-1

interface Ethernet1/1
   description Leaf-11-et1
   mtu 9214
   no switchport
   ip address 172.168.1.0/31

interface Loopback0
   ip address 1.1.1.1/32
!
ip prefix-list loopback
   seq 10 permit 1.1.1.0/24 ge 24
!
route-map loopback permit 10
   match ip address prefix-list loopback
!
router bgp 65001
   neighbor 172.168.1.1 remote-as 65004
   redistribute connected route-map loopback

VRF Configuration: Leaf-11

VRF-Blue is configured on all the left leaves. The left leaves have pure Layer 3 interfaces and the right side has SVI 10.

vrf instance VRF-Blue

ip routing vrf VRF-Blue

interface Ethernet36
   no switchport
   vrf VRF-Blue
   ip address 172.168.1.9/31

router bgp 65004
     vrf VRF-Blue
           neighbor 172.168.1.8 remote-as 65005

VRF Configuration: Leaf-21

vlan 10

vrf instance VRF-Blue

ip routing vrf VRF-Blue

interface Vlan10
   vrf VRF-Blue
   ip address virtual 10.10.10.1/24

ip virtual-router mac-address 00:aa:aa:aa:aa:aa

interface Port-Channel3
   switchport mode trunk
   mlag 3

VXLAN Configuration: Leaf-11

Make sure all VTEPs have unique loopback0 addresses to represent unique VTEP identifiers. For every VNI that EVPN receives, a dynamic VLAN is allocated, so it is a good practice to keep the same VNI.

interface VXLAN1
   VXLAN source-interface Loopback0
   VXLAN udp-port 4789
   VXLAN vrf VRF-Blue vni 10001

VXLAN Configuration: Leaf-21

interface VXLAN1
   VXLAN source-interface Loopback0
   VXLAN udp-port 4789
   VXLAN vrf VRF-Blue vni 10001

EVPN Configuration: Leaf-11

Leaf establishes the EVPN neighborship with all four spines for redundancy. EVPN neighborship is on the loopback address and the multihop keyword is used. Make sure to disable the IPv4 address family for EVPN neighbors.

Since the spine is acting like a route-reflector for EVPN routes, make sure to configure the next-hop-unchanged.

router bgp 65004
   neighbor SPINE_EVPN peer-group
   neighbor SPINE_EVPN remote-as 65001
   neighbor SPINE_EVPN update-source Loopback0
   neighbor SPINE_EVPN ebgp-multihop 3
   neighbor SPINE_EVPN send-community extended
   neighbor SPINE_EVPN maximum-routes 12000
   neighbor 1.1.1.1 peer-group SPINE_EVPN
   !
   address-family evpn
      neighbor SPINE_EVPN activate
   !
   address-family ipv4
      no neighbor SPINE_EVPN activate

EVPN Configuration: Leaf-21

router bgp 65002
   neighbor SPINE_EVPN peer-group
   neighbor SPINE_EVPN remote-as 65001
   neighbor SPINE_EVPN update-source Loopback0
   neighbor SPINE_EVPN allowas-in 1
   neighbor SPINE_EVPN ebgp-multihop 3
   neighbor SPINE_EVPN send-community extended
   neighbor SPINE_EVPN maximum-routes 12000
   neighbor 1.1.1.1 peer-group SPINE_EVPN
   !
   address-family evpn
      neighbor SPINE_EVPN activate
   !
   address-family ipv4
      no neighbor SPINE_EVPN activate

EVPN Configuration: Spine-1

router bgp 65004
   neighbor SPINE_EVPN peer-group
   neighbor SPINE_EVPN remote-as 65001
   neighbor SPINE_EVPN update-source Loopback0
   neighbor SPINE_EVPN ebgp-multihop 3
   neighbor SPINE_EVPN send-community extended
   neighbor SPINE_EVPN maximum-routes 12000
   neighbor 1.1.1.1 peer-group SPINE_EVPN
   !
   address-family evpn
      neighbor SPINE_EVPN activate
   !
   address-family ipv4
      no neighbor SPINE_EVPN activate

Advertise VRF Routes in EVPN: Leaf-11

By configuring VRF under router-bgp, you are advertising routes from that VRF into EVPN using the RD/RT. The remote end can install the route by importing the RT.

Leaf-11 has routes in VRF-Blue learned through eBGP with the neighbor down south. Since the routes are already in BGP VRF table, we do not want to configure the redistribute command.

router bgp 65004
   neighbor SPINE_EVPN peer-group
   neighbor SPINE_EVPN remote-as 65001
   neighbor SPINE_EVPN update-source Loopback0
   neighbor SPINE_EVPN ebgp-multihop 3
   neighbor SPINE_EVPN send-community extended
   neighbor SPINE_EVPN maximum-routes 12000
   neighbor 1.1.1.1 peer-group SPINE_EVPN
   !
   address-family evpn
      neighbor SPINE_EVPN activate
   !
   address-family ipv4
      no neighbor SPINE_EVPN activate

Advertise VRF Routes in EVPN: Leaf-21

On the other hand Leaf-21 wants to export the connected SVI into EVPN and therefore require redistribute connected command.

router bgp 65002
   neighbor SPINE_EVPN peer-group
   neighbor SPINE_EVPN remote-as 65001
   neighbor SPINE_EVPN update-source Loopback0
   neighbor SPINE_EVPN allowas-in 1
   neighbor SPINE_EVPN ebgp-multihop 3
   neighbor SPINE_EVPN send-community extended
   neighbor SPINE_EVPN maximum-routes 12000
   neighbor 1.1.1.1 peer-group SPINE_EVPN
   !
   address-family evpn
      neighbor SPINE_EVPN activate
   !
   address-family ipv4
      no neighbor SPINE_EVPN activate

Multi-Tenant EVPN VXLAN IRB Sample Configuration

The following configuration example shows a deployment using both symmetric and asymmetric IRB, with VLAN-based and VLAN-aware bundle services; and eBGP overlay and underlay.

Figure 2. Tenant-A: Symmetric IRB

Figure 3. Tenant-B: Asymmetric IRB

In the symmetric and asymmetric IRB configurations illustrated in the figures above, for Tenant-A, four subnets are stretched across the two MLAG domains with two subnets (VLAN 10, 10.10.10.0/24 and VLAN 11, 10.10.11.0/24) configured as a VLAN-based service and two other subnets (VLAN 12,10.10.12.0/24 and VLAN 13, 10.10.13.0/24) as a VLAN-aware bundle service.

For Tenant-B, four subnets are stretched across the two MLAG domains with two subnets (VLAN 210, 10.10.10.0/24 and VLAN 211,10.10.11.0/24) configured as a VLAN-based service, and two other subnets (VLAN 212,10.10.12.0/24 and VLAN 213,10.10.13.0/24) as a VLAN-aware bundle service.

In addition, each MLAG domain has a single local subnet (Rack-1 subnet 10.10.20.0/24 and Rack-2 subnet 10.10.21.0/24) for the tenant. To provide direct distributed routing, each leaf switch is configured with the same virtual IP address for the four stretched subnets. The virtual IP address is configured in both physical leaf switches of the relevant MLAG domain for the local-only subnets.

For each MLAG domain, a logical VTEP is created with the same shared loopback address. For Rack-1, the logical VTEP IP is 2.2.2.1 and for the Rack-2, the logical VTEP IP is 2.2.2.2. Directly connected to each leaf switch is a host, which is a member of one of the two IP subnets. To provide Layer 2 connectivity across the racks, VXLAN bridging is enabled by mapping VLAN to VNIs as detailed in the diagram.

To provide IP connectivity across all subnets, both stretched and directly connected, an IP-VRF is shared between the two MLAG domains for the tenant. This is used as a transit network to announce and forward the locally attached subnets. Each leaf switch is EVPN peering with the four spine switches via a loopback interface on the leaf and again on the spine switches. To provide external connectivity, Leaf-11 and Leaf-12 are eBGP peering via the tenants’ VRFs with the border routers. Both core routers are advertising external prefixes for Internet and any remote site connectivity (default route and IP prefixes from the other DC for the tenant). To provide connectivity within the EVPN domain, the leaf switches (Leaf-21 and Leaf-22) re-advertise the prefixes into the tenant’s VRF via a type-5 route advertisement, with a next-hop equal to the advertising VTEP.

MLAG Configuration: Leaf-11 and Leaf-12

Leaf-11 MLAG Configuration

spanning-tree mode mstp
no spanning-tree vlan-id 4093-4094
!
ip virtual-router mac-address mlag-peer
!
vlan 4094
   name MLAG_PEER
   trunk group MLAG
!
vlan 4093
   name LEAF_PEER_L3
   trunk group LEAF_PEER_L3
!
interface Vlan4094
   ip address 172.168.10.1/30
!
interface Port-Channel100
   description port-channel to access switch
    switchport trunk allowed vlan 10-13,20,210-213,220
   switchport mode trunk
   mlag 1
!
interface Port-Channel1000
   switchport mode trunk
   switchport trunk group LEAF_PEER_L3
   switchport trunk group MLAG
!
mlag configuration
   domain-id Rack-1
   local-interface Vlan4094
   peer-address 172.168.10.2
   peer-link Port-Channel1000

Leaf-12 MLAG Configuration

spanning-tree mode mstp
no spanning-tree vlan-id 4093-4094
!
ip virtual-router mac-address mlag-peer
!
vlan 4094
   name MLAG_PEER
   trunk group MLAG
!
vlan 4093
   name LEAF_PEER_L3
   trunk group LEAF_PEER_L3
!
interface Vlan4094
   ip address 172.168.10.2/30
!
interface Port-Channel100
   description port-channel to access switch
    switchport trunk allowed vlan 10-13,20,210-213,220
   switchport mode trunk
   mlag 1
!
interface Port-Channel1000
   switchport mode trunk
   switchport trunk group LEAF_PEER_L3
   switchport trunk group MLAG
!
mlag configuration
   domain-id Rack-1
   local-interface Vlan4094
   peer-address 172.168.10.1
   peer-link Port-Channel1000

MLAG Configuration: Leaf-21 and Leaf-22

Leaf-21 MLAG Configuration

spanning-tree mode mstp
no spanning-tree vlan-id 4093-4094
!
ip virtual-router mac-address mlag-peer
!
vlan 4094
   name MLAG_PEER
   trunk group MLAG
!
vlan 4093
   name LEAF_PEER_L3
   trunk group LEAF_PEER_L3
!
interface Vlan4094
   ip address 172.168.10.1/30
!
interface Port-Channel100
   description port-channel to access switch
  switchport trunk allowed vlan 10-13,21,210-213,220-221
   switchport mode trunk
   mlag 1
!
interface Port-Channel1000
   switchport mode trunk
   switchport trunk group LEAF_PEER_L3
   switchport trunk group MLAG
!
mlag configuration
   domain-id Rack-1
   local-interface Vlan4094
   peer-address 172.168.10.2
   peer-link Port-Channel1000

Leaf-22 MLAG Configuration

spanning-tree mode mstp
no spanning-tree vlan-id 4093-4094
!
ip virtual-router mac-address mlag-peer
!
vlan 4094
   name MLAG_PEER
   trunk group MLAG
!
vlan 4093
   name LEAF_PEER_L3
   trunk group LEAF_PEER_L3
!
interface Vlan4094
   ip address 172.168.10.2/30
!
interface Port-Channel100
   description port-channel to access switch
  switchport trunk allowed vlan 10-13,21,210-213,220-221
   switchport mode trunk
   mlag 1
!
interface Port-Channel1000
   switchport mode trunk
   switchport trunk group LEAF_PEER_L3
   switchport trunk group MLAG
!
mlag configuration
   domain-id Rack-1
   local-interface Vlan4094
   peer-address 172.168.10.1
   peer-link Port-Channel1000hannel1000

VLAN and Distributed IP Address Configuration: Leaf-11 and Leaf-21

VLAN and interface configuration for VLAN 10 (virtual IP address 10.10.10.254) and VLAN 11 (virtual IP address 10.10.11.254), along with SVIs 12, 13, and 20, are similarly configured. To provide multi-tenancy, the two tenant VLANs are placed in a dedicated VRF, named Tenant-A. A further five tenant VLANs are configured and assigned to VRF Tenant-B.

The other VLANs are for peering, MLAG, and a unique VLAN SVI. These VLANs do not use virtual IP addresses.

The tenants’ stretched subnets (Tenant-A: VLANs 10,11,12, and 13; Tenant-B: VLANs 210, 211, 211, 212, and 213) are mapped to unique overlay VXLAN VNIs. The tenants’ IP-VRF (Tenant-A and Tenant-B) is associated with a VNI using the VXLAN vrf command under the VXLAN interface. In the forwarding model for symmetric IRB, this VNI will be used as the transit VNI for routing to subnets not locally configured on the VTEP.

As a standard MLAG configuration, both leaf switches in each MLAG domain share the same logical VTEP IP address. Thus MLAG domain, Rack-1 (Leaf-11 + Leaf-12) has a shared logical VTEP IP of 2.2.2.1 and Rack-2 (Leaf-21 + Leaf-22) has a shared logical VTEP IP of 2.2.2.2.

Leaf-11 VLAN and Distributed IP Address Configuration

!
ip virtual-router mac-address 00:aa:aa:aa:aa:aa
!
vlan 10-11,20,210-211,220,111,2111
!
vlan 12-13
   name VLAN-AWARE-BUNDLE-TENANT-A
!
vlan 212-213
   name VLAN-AWARE-BUNDLE-TENANT-B
!
vrf instance tenant-a
!
vrf instance tenant-b
!
interface lan10
   mtu 9164
   vrf tenant-a
   ip address virtual 10.10.10.254/24
!
interface Vlan11
   mtu 9164
   vrf tenant-a
   ip address virtual 10.10.11.254/24
!
interface Vlan12
   mtu 9164
   vrf tenant-a
   ip address virtual 10.10.12.254/24
!
interface Vlan13
   mtu 9164
   vrf tenant-a
   ip address virtual 10.10.13.254/24
!
interface Vlan20
   mtu 9164
   vrf tenant-a
   ip address virtual 10.10.20.254/24
!
interface Vlan210
   mtu 9164
   vrf tenant-b
   ip address virtual 10.10.10.254/24
!
interface Vlan211
   mtu 9164
   vrf tenant-b
   ip address virtual 10.10.11.254/24
!
interface Vlan212
   mtu 9164
   vrf tenant-b
   ip address virtual 10.10.12.254/24
!
interface Vlan213
   mtu 9164
   vrf tenant-b
   ip address virtual 10.10.13.254/24
!
interface Vlan220
   mtu 9164
   vrf tenant-b
   ip address virtual 10.10.20.254/24
!
interface Vlan1111
   description Unique-highest-IP-in-each-IP-Vrf
   mtu 9164
   vrf tenant-a
   ip address 223.255.255.249/30
!
interface Vlan2111
   description Unique-highest-IP-in-each-IP-Vrf
   mtu 9164
   vrf tenant-b
   ip address 223.255.255.249/30
!
interface Vlan4093
   ip address 172.168.11.1/30

Leaf-21 VLAN and Distributed IP Address Configuration

!
ip virtual-router mac-address 00:aa:aa:aa:aa:aa
!
vlan 10-11,20,210-211,220,111,2111
!
vlan 12-13
   name VLAN-AWARE-BUNDLE-TENANT-A
!
vlan 212-213
   name VLAN-AWARE-BUNDLE-TENANT-B
!
vrf instance tenant-a
!
vrf instance tenant-b
!
interface Vlan10
   mtu 9164
   vrf tenant-a
   ip address virtual 10.10.10.254/24
!
interface Vlan11
   mtu 9164
   vrf tenant-a
   ip address virtual 10.10.11.254/24
!
interface Vlan12
   mtu 9164
   vrf tenant-a
   ip address virtual 10.10.12.254/24
!
interface Vlan13
   mtu 9164
   vrf tenant-a
   ip address virtual 10.10.13.254/24
!
interface Vlan21
   mtu 9164
   vrf tenant-a
   ip address virtual 10.10.21.254/24
!
interface Vlan210
   mtu 9164
   vrf tenant-b
   ip address virtual 10.10.10.254/24
!
interface Vlan211
   mtu 9164
   vrf tenant-b
   ip address virtual 10.10.11.254/24
!
interface Vlan212
   mtu 9164
   vrf tenant-b
   ip address virtual 10.10.12.254/24
!
interface Vlan213
   mtu 9164
   vrf tenant-b
   ip address virtual 10.10.13.254/24
!
interface Vlan221
   mtu 9164
   vrf tenant-b
   ip address virtual 10.10.21.254/24
!
interface Vlan1111
   description Unique-highest-IP-in-each-IP-Vrf
   mtu 9164
   vrf tenant-a
   ip address 223.255.255.253/30
!
interface Vlan2111
   description Unique-highest-IP-in-each-IP-Vrf
   mtu 9164
   vrf tenant-b
   ip address 223.255.255.253/30
!
interface Vlan4093
   ip address 172.168.11.1/30
!

VXLAN Interface Configuration: Leaf-11 and Leaf-21

The tenants’ VLANs are mapped to unique overlay VXLAN VNIs. VLAN 10 is mapped to VNI 1010 on both MLAG domains, and VLAN 11 is mapped to VNI 1011. As standard MLAG configuration, both leaf switches in each MLAG domain share the same logical VTEP IP address. Thus MLAG domain Rack-1 (Leaf-11 + Leaf-12) has a shared logical VTEP IP of 2.2.2.1 and Rack-2 (Leaf-21 + Leaf-22) has a shared logical VTEP IP of 2.2.2.2. Also configured is the VRF-to-VXLAN mapping for Tenant-A.

Leaf-11 VXLAN Interface Configuration

!
interface Loopback1
   ip address 2.2.2.1/32
!
interface VXLAN1
   VXLAN source-interface Loopback1
   VXLAN udp-port 4789
   VXLAN vlan 10 vni 1010
   VXLAN vlan 11 vni 1011
   VXLAN vlan 12 vni 1012
   VXLAN vlan 13 vni 1013
   VXLAN vlan 20 vni 1020
   VXLAN vlan 210 vni 1210
   VXLAN vlan 211 vni 1211
   VXLAN vlan 212 vni 1212
   VXLAN vlan 213 vni 1213
   VXLAN vlan 220 vni 1220
   VXLAN vrf tenant-a vni 1000
   VXLAN vrf tenant-b vni 1001

Leaf-21 VXLAN Interface Configuration

!
interface Loopback1
   ip address 2.2.2.2/32
!
interface VXLAN1
   VXLAN source-interface Loopback1
   VXLAN udp-port 4789
   VXLAN vlan 10 vni 1010
   VXLAN vlan 11 vni 1011
   VXLAN vlan 12 vni 1012
   VXLAN vlan 13 vni 1013
   VXLAN vlan 21 vni 1021
   VXLAN vlan 210 vni 1210
   VXLAN vlan 211 vni 1211
   VXLAN vlan 212 vni 1212
   VXLAN vlan 213 vni 1213
   VXLAN vlan 221 vni 1221
   VXLAN vrf tenant-a vni 1000
   VXLAN vrf tenant-b vni 1001
Note: This configuration uses VXLAN routing. For single-chip T2 and TH platforms, recirculation must be enabled. For R-Series platforms, the following configuration commands must be added:

hardware tcam

system profile VXLAN-routing

Refer to diagrams for VLAN and SVI assignment to tenant; Leaf-11 also has peering out to the border router in addition to the connected SVIs.

eBGP Underlay Configuration on the Leaf Switches

The leaf switches for the underlay network peer with each spine on the physical interface. For EVPN route advertisement, the BGP EVPN session is between loopback addresses.

In this case, the underlay is all eBGP, and peering is on the physical interfaces. The MLAG leaves also peer with each other in the underlay to retain BGP EVPN connectivity (loopback reachability) in the unlikely case that all spine links are down. This is a failover configuration that can be implemented if there is ever the chance a leaf could be “core isolated.” The configuration can be viewed on each leaf using the command show running-configuration section bgp.

The following examples show the underlay configuration on all four leaf switches, and also on two of the spine switches as an example of the underlay configuration on the spine.

The configuration uses the following peer groups:

SPINE     configuration inherited for underlay (eBGP) peering to the spines

SPINE_EVPN     overlay eBGP peering between spine and leaf, using loopbacks

Figure 4. Physical Underlay Topology


eBGP Underlay Configuration: Leaf-11

route-map loopback permit 10
   match ip address prefix-list loopback
!
route-map dont_advertise_loopbacks deny 10
   match ip address prefix-list loopback
!
route-map dont_advertise_loopbacks permit 20
!
ip prefix-list loopback
    seq 10 permit 1.1.1.11/32
    seq 20 permit 1.1.1.12/32
    seq 30 permit 1.1.1.22/32
    seq 40 permit 1.1.1.21/32
    seq 50 permit 2.2.2.1/32
    seq 60 permit 2.2.2.2/32
!
router bgp 65002
   router-id 1.1.1.11
   maximum-paths 8 ecmp 16
   neighbor SPINE peer-group
   neighbor SPINE remote-as 65001
   neighbor SPINE allowas-in 1
   neighbor SPINE soft-reconfiguration inbound all
   neighbor SPINE route-map loopback out
   neighbor SPINE send-community
   neighbor 172.168.1.1 peer-group SPINE
   neighbor 172.168.1.5 peer-group SPINE
   neighbor 172.168.1.9 peer-group SPINE
   neighbor 172.168.1.13 peer-group SPINE
   neighbor 172.168.11.2 remote-as 65004
   neighbor 172.168.11.2 local-as 65002 no-prepend replace-as
   neighbor 172.168.11.2 allowas-in 1
   neighbor 172.168.11.2 maximum-routes 12000
   redistribute connected route-map loopback

eBGP Underlay Configuration: Leaf-12

route-map loopback permit 10
   match ip address prefix-list loopback
!
route-map dont_advertise_loopbacks deny 10
   match ip address prefix-list loopback
!
route-map dont_advertise_loopbacks permit 20
!
ip prefix-list loopback
    seq 10 permit 1.1.1.11/32
    seq 20 permit 1.1.1.12/32
    seq 30 permit 1.1.1.22/32
    seq 40 permit 1.1.1.21/32
    seq 50 permit 2.2.2.1/32
    seq 60 permit 2.2.2.2/32
!
router bgp 65002
   router-id 1.1.1.12
   maximum-paths 8 ecmp 16
   neighbor SPINE peer-group
   neighbor SPINE remote-as 65001
   neighbor SPINE allowas-in 1
   neighbor SPINE soft-reconfiguration inbound all
   neighbor SPINE route-map loopback out
   neighbor SPINE send-community
   neighbor 172.168.2.1 peer-group SPINE
   neighbor 172.168.2.5 peer-group SPINE
   neighbor 172.168.2.9 peer-group SPINE
   neighbor 172.168.2.13 peer-group SPINE
   neighbor 172.168.11.1 remote-as 65002
   neighbor 172.168.11.1 local-as 65004 no-prepend replace-as
   neighbor 172.168.11.1 allowas-in 1
   neighbor 172.168.11.1 maximum-routes 12000
   redistribute connected route-map loopback

eBGP Underlay Configuration: Leaf-21

route-map loopback permit 10
   match ip address prefix-list loopback
!
ip prefix-list loopback
    seq 10 permit 1.1.1.11/32
    seq 20 permit 1.1.1.12/32
    seq 30 permit 1.1.1.22/32
    seq 40 permit 1.1.1.21/32
    seq 50 permit 2.2.2.1/32
    seq 60 permit 2.2.2.2/32
!
router bgp 65002
   router-id 1.1.1.21
   maximum-paths 8 ecmp 16
   neighbor SPINE peer-group
   neighbor SPINE remote-as 65001
   neighbor SPINE allowas-in 1
   neighbor SPINE soft-reconfiguration inbound all
   neighbor SPINE route-map loopback out
   neighbor SPINE send-community
   neighbor SPINE maximum-routes 20000
   neighbor 172.168.3.1 peer-group SPINE
   neighbor 172.168.3.5 peer-group SPINE
   neighbor 172.168.3.9 peer-group SPINE
   neighbor 172.168.3.13 peer-group SPINE
   neighbor 172.168.11.2 remote-as 65004
   neighbor 172.168.11.2 local-as 65002 no-prepend replace-as
   neighbor 172.168.11.2 allowas-in 1
   neighbor 172.168.11.2 maximum-routes 12000
   redistribute connected route-map loopback

eBGP Underlay Configuration: Leaf-22

route-map loopback permit 10
   match ip address prefix-list loopback
!
ip prefix-list loopback
    seq 10 permit 1.1.1.11/32
    seq 20 permit 1.1.1.12/32
    seq 30 permit 1.1.1.22/32
    seq 40 permit 1.1.1.21/32
    seq 50 permit 2.2.2.1/32
    seq 60 permit 2.2.2.2/32
!
router bgp 65002
   router-id 1.1.1.22
   maximum-paths 8 ecmp 16
   neighbor SPINE peer-group
   neighbor SPINE remote-as 65001
   neighbor SPINE allowas-in 1
   neighbor SPINE soft-reconfiguration inbound all
   neighbor SPINE route-map loopback out
   neighbor SPINE send-community
   neighbor SPINE maximum-routes 20000
   neighbor 172.168.4.1 peer-group SPINE
   neighbor 172.168.4.5 peer-group SPINE
   neighbor 172.168.4.9 peer-group SPINE
   neighbor 172.168.4.13 peer-group SPINE
   neighbor 172.168.11.1 remote-as 65002
   neighbor 172.168.11.1 local-as 65004 no-prepend replace-as
   neighbor 172.168.11.2 allowas-in 1
   neighbor 172.168.11.1 maximum-routes 12000
   redistribute connected route-map loopback

EVPN BGP Configuration on the Spine Switches

The EVPN BGP configuration on two of the spine switches is summarized below. Note that only the EVPN BGP sessions are listed for the two spine switches: the BGP underlay configuration is not included.

EVPN BGP Configuration: Spine-1

route-map loopback permit 10
   match ip address prefix-list loopback
!
ip prefix-list loopback
    seq 10 permit 1.1.1.11/32
    seq 20 permit 1.1.1.12/32
    seq 30 permit 1.1.1.22/32
    seq 40 permit 1.1.1.21/32
    seq 50 permit 2.2.2.1/32
    seq 60 permit 2.2.2.2/32
!
router bgp 65001
   router-id 1.1.1.1
   distance bgp 20 200 200
   maximum-paths 8 ecmp 16
   neighbor LEAF peer-group
   neighbor LEAF remote-as 65002
   neighbor LEAF maximum-routes 20000
   neighbor 172.168.1.2 peer-group LEAF
   neighbor 172.168.2.2 peer-group LEAF
   neighbor 172.168.3.2 peer-group LEAF
   neighbor 172.168.4.2 peer-group LEAF
   redistribute connected route-map loopback

EVPN BGP Configuration: Spine-2

route-map loopback permit 10
   match ip address prefix-list loopback
!
ip prefix-list loopback
    seq 10 permit 1.1.1.11/32
    seq 20 permit 1.1.1.12/32
    seq 30 permit 1.1.1.22/32
    seq 40 permit 1.1.1.21/32
    seq 50 permit 2.2.2.1/32
    seq 60 permit 2.2.2.2/32
!
router bgp 65001
   router-id 1.1.1.2
   distance bgp 20 200 200
   maximum-paths 8 ecmp 16
   neighbor LEAF peer-group
   neighbor LEAF remote-as 65002
   neighbor LEAF maximum-routes 20000
   neighbor 172.168.1.6 peer-group LEAF
   neighbor 172.168.2.6 peer-group LEAF
   neighbor 172.168.3.6 peer-group LEAF
   neighbor 172.168.4.6 peer-group LEAF
   redistribute connected route-map loopback

eBGP Overlay on Leaf Switches

The MAC VRFs and IP VRF for the tenants’ subnets are created in the BGP router context with unique Route-Distinguishers (RD) and Route-Targets (RT) attached to each MAC-VRF and IP-VRF. The RDs provide support for overlapping MAC and IP addresses across tenants, while the RTs allow control of the routes imported and exported between MAC VRFs.

To ensure all routes are correctly imported between VTEPs sharing the same Layer-2 domain, the import and export RTs are equal across the two MLAG domains. The redistribute learned statement under each MAC VRF ensures any locally learned MACs in the VLAN are automatically announced as type-2 routes.

The IP VRF (Tenant-A) is created on all leaf switches which have subnets attached to the tenant’s VRF with the same route target ensuring that routes are correctly imported and exported between VTEPs in the VRF. On Leaf-21 and Leaf-22, to import the external routes an eBGP session with the BGP peering router is created under the IP VRF (Tenant-A) context, and a peering from each to the other is created on the overlay.

Note: All MAC VRFs are unique, and each has its own RT, matched by the other leaves in the DC. The “tenants” as such are defined at layer 3 by assigning SVIs to the appropriate VRF. To view this assignment, use the show ip route vrf <tenant> connected command. Note below that VLANs 12-13 and 212-213 (shown in bold) are configured as a bundle-aware EVPN service. Also note the peering from Leaf-11 to the BGP border router in each tenant VRF.

EVPN BGP Overlay Configuration for the Tenants’ MAC VRFs and IP VRF: Leaf-11

route-map loopback permit 10
   match ip address prefix-list loopback
!
route-map dont_advertise_loopbacks deny 10
   match ip address prefix-list loopback
!
route-map dont_advertise_loopbacks permit 20
!
ip prefix-list loopback
    seq 10 permit 1.1.1.11/32
    seq 20 permit 1.1.1.12/32
    seq 30 permit 1.1.1.22/32
    seq 40 permit 1.1.1.21/32
    seq 50 permit 2.2.2.1/32
    seq 60 permit 2.2.2.2/32
!
router bgp 65002
   router-id 1.1.1.11
   maximum-paths 4
   neighbor SPINE_EVPN peer-group
   neighbor SPINE_EVPN remote-as 65001
   neighbor SPINE_EVPN update-source Loopback0
   neighbor SPINE_EVPN allowas-in 2
   neighbor SPINE_EVPN ebgp-multihop 5
   neighbor SPINE_EVPN send-community extended
   neighbor SPINE_EVPN maximum-routes 12000
   neighbor 1.1.1.1 peer-group SPINE_EVPN
   neighbor 1.1.1.2 peer-group SPINE_EVPN
   redistribute connected route-map loopback
   !
  vlan 10
      rd 1.1.1.11:1010
      route-target both 1010:1010
      redistribute learned
   !
   vlan 11
      rd 1.1.1.11:1011
      route-target both 1011:1011
      redistribute learned
   !
   vlan 20
      rd 1.1.1.11:1020
      route-target both 1020:1020
      redistribute learned
   !
   vlan 210
      rd 1.1.1.11:1210
      route-target both 1210:1210
      redistribute learned
      no redistribute host-route
   !
   vlan 211
      rd 1.1.1.11:1211
      route-target both 1211:1211
      redistribute learned
      no redistribute host-route
   !
   vlan 220
      rd 1.1.1.11:1220
      route-target both 1220:1220
      redistribute learned
      no redistribute host-route
   !
  vlan-aware-bundle Tenant-A-VLAN-12-13
      rd 1.1.1.11:1213
      route-target both 12:13
      redistribute learned
      vlan 12-13
   !
   vlan-aware-bundle Tenant-B-VLAN-212-213
       rd 1.1.1.11:21213
       route-target both 212:213
       redistribute learned
       no redistribute host-route
       vlan 212-213
   !
   address-family evpn
      neighbor SPINE_EVPN activate
   !
   address-family ipv4
      no neighbor SPINE_EVPN activate
   !
   vrf tenant-a
      rd 1.1.1.11:1000
      route-target import 1000:1000
      route-target export 1000:1000
      neighbor 192.168.168.9 remote-as 64512
      neighbor 192.168.168.9 local-as 65002 no-prepend replace-as
      neighbor 192.168.168.9 maximum-routes 12000
      neighbor 223.255.255.250 peer-group LEAF_PEER_OVERLAY
      neighbor 223.255.255.250 remote-as 65004
      neighbor 223.255.255.250 local-as 65002 no-prepend replace-as
      redistribute connected route-map dont_advertise_loopbacks
   !
   vrf tenant-b
      rd 1.1.1.11:1001
      route-target import 1001:1001
      route-target export 1001:1001
      neighbor 192.168.168.21 remote-as 64513
      neighbor 192.168.168.21 local-as 65002 no-prepend replace-as
      neighbor 192.168.168.21 maximum-routes 12000
      neighbor 223.255.255.249 peer-group LEAF_PEER_OVERLAY
      neighbor 223.255.255.249 remote-as 65004
      neighbor 223.255.255.249 local-as 65002 no-prepend replace-as
      redistribute connected route-map dont_advertise_loopbacks

EVPN BGP Overlay Configuration for the Tenants’ MAC VRFs and IP VRF: Leaf-12

route-map loopback permit 10
   match ip address prefix-list loopback
!
route-map dont_advertise_loopbacks deny 10
   match ip address prefix-list loopback
!
route-map dont_advertise_loopbacks permit 20
!
ip prefix-list loopback
    seq 10 permit 1.1.1.11/32
    seq 20 permit 1.1.1.12/32
    seq 30 permit 1.1.1.22/32
    seq 40 permit 1.1.1.21/32
    seq 50 permit 2.2.2.1/32
    seq 60 permit 2.2.2.2/32
!
router bgp 65002
   router-id 1.1.1.12
   maximum-paths 4
   neighbor SPINE_EVPN peer-group
   neighbor SPINE_EVPN remote-as 65001
   neighbor SPINE_EVPN update-source Loopback0
   neighbor SPINE_EVPN allowas-in 2
   neighbor SPINE_EVPN ebgp-multihop 5
   neighbor SPINE_EVPN send-community extended
   neighbor SPINE_EVPN maximum-routes 12000
   neighbor 1.1.1.1 peer-group SPINE_EVPN
   neighbor 1.1.1.2 peer-group SPINE_EVPN
   redistribute connected route-map loopback
!
  vlan 10
      rd 1.1.1.12:1010
      route-target both 1010:1010
      redistribute learned
   !
   vlan 11
      rd 1.1.1.12:1011
      route-target both 1011:1011
      redistribute learned
   !
   vlan 20
      rd 1.1.1.12:1020
      route-target both 1020:1020
      redistribute learned
   !
   vlan 210
      rd 1.1.1.12:1210
      route-target both 1210:1210
      redistribute learned
      no redistribute host-route
   !
   vlan 211
      rd 1.1.1.12:1211
      route-target both 1211:1211
      redistribute learned
      no redistribute host-route
   !
   vlan 220
      rd 1.1.1.12:1220
      route-target both 1220:1220
      redistribute learned
      no redistribute host-route
   !
   vlan-aware-bundle Tenant-A-VLAN-12-13
      rd 1.1.1.12:1213
      route-target both 12:13
      redistribute learned
      vlan 12-13
   !
   vlan-aware-bundle Tenant-B-VLAN-212-213
      rd 1.1.1.12:21213
      route-target both 212:213
      redistribute learned
      no redistribute host-route
      vlan 212-213
   !
   address-family evpn
      neighbor SPINE_EVPN activate
   !
   address-family ipv4
      no neighbor SPINE_EVPN activate
   !
   vrf tenant-a
      rd 1.1.1.12:1000
      route-target import 1000:1000
      route-target export 1000:1000
      neighbor 192.168.168.13 remote-as 64512
      neighbor 192.168.168.13 local-as 65002 no-prepend replace-as
      neighbor 192.168.168.13 maximum-routes 12000
      neighbor 223.255.255.249 peer-group LEAF_PEER_OVERLAY
      neighbor 223.255.255.249 remote-as 65002
      neighbor 223.255.255.249 local-as 65004 no-prepend replace-as
      redistribute connected route-map dont_advertise_loopbacks
   !
   vrf tenant-b
      rd 1.1.1.12:1001
      route-target import 1001:1001
      route-target export 1001:1001
      neighbor 192.168.168.23 remote-as 64513
      neighbor 192.168.168.23 local-as 65002 no-prepend replace-as
      neighbor 192.168.168.23 maximum-routes 12000
      neighbor 223.255.255.249 peer-group LEAF_PEER_OVERLAY
      neighbor 223.255.255.249 remote-as 65002
      neighbor 223.255.255.249 local-as 65004 no-prepend replace-as
      redistribute connected route-map dont_advertise_loopbacks

EVPN BGP Overlay Configuration for the Tenants’ MAC VRFs and IP VRF: Leaf-21

route-map loopback permit 10
   match ip address prefix-list loopback
!
route-map dont_advertise_loopbacks deny 10
   match ip address prefix-list loopback
!
route-map dont_advertise_loopbacks permit 20
!
router bgp 65002
   router-id 1.1.1.21
   maximum-paths 4
   neighbor SPINE_EVPN peer-group
   neighbor SPINE_EVPN remote-as 65001
   neighbor SPINE_EVPN update-source Loopback0
   neighbor SPINE_EVPN allowas-in 2
   neighbor SPINE_EVPN ebgp-multihop 5
   neighbor SPINE_EVPN send-community extended
   neighbor SPINE_EVPN maximum-routes 12000
   neighbor 1.1.1.1 peer-group SPINE_EVPN
   neighbor 1.1.1.2 peer-group SPINE_EVPN
   redistribute connected route-map loopback
   !
   vlan 10
      rd 1.1.1.21:1010
      route-target both 1010:1010
      redistribute learned
   !
   vlan 11
      rd 1.1.1.21:1011
      route-target both 1011:1011
      redistribute learned
   !
   vlan 21
      rd 1.1.1.21:1021
      route-target both 1021:1021
      redistribute learned
   !
   vlan 210
      rd 1.1.1.21:1210
      route-target both 1210:1210
      redistribute learned
      no redistribute host-route
   !
   vlan 211
      rd 1.1.1.21:1211
      route-target both 1211:1211
      redistribute learned
      no redistribute host-route
   !
   vlan 221
      rd 1.1.1.21:1221
      route-target both 1221:1221
      redistribute learned
      no redistribute host-route
   !
   vlan-aware-bundle Tenant-A-VLAN-12-13
      rd 1.1.1.21:1213
      route-target both 12:13
      redistribute learned
      vlan 12-13
   !
   vlan-aware-bundle Tenant-B-VLAN-212-213
      rd 1.1.1.21:21213
      route-target both 212:213
      redistribute learned
      redistribute host-route
      vlan 212-213
   !
   address-family evpn
      neighbor SPINE_EVPN activate
   !
   address-family ipv4
      no neighbor SPINE_EVPN activate
   !
   vrf tenant-a
      rd 1.1.1.21:1000
      route-target import 1000:1000
      route-target export 1000:1000
      neighbor 223.255.255.254 remote-as 65002
      neighbor 223.255.255.254 next-hop-self
      neighbor 223.255.255.254 update-source Vlan1111
      neighbor 223.255.255.254 allowas-in 1
      neighbor 223.255.255.254 maximum-routes 12000
      redistribute connected route-map dont_advertise_loopbacks
   !
   vrf tenant-b
      rd 1.1.1.21:1001
      route-target import 1001:1001
      route-target export 1001:1001
      neighbor 223.255.255.254 remote-as 65002
      neighbor 223.255.255.254 next-hop-self
      neighbor 223.255.255.254 update-source Vlan2111
      neighbor 223.255.255.254 allowas-in 1
      neighbor 223.255.255.254 maximum-routes 12000
      redistribute connected route-map dont_advertise_loopbacks

EVPN BGP Overlay Configuration for the Tenants’ MAC VRFs and IP VRF: Leaf-22

route-map loopback permit 10
   match ip address prefix-list loopback
!
route-map dont_advertise_loopbacks deny 10
   match ip address prefix-list loopback
!
route-map dont_advertise_loopbacks permit 20
!
router bgp 65002
   router-id 1.1.1.22
   maximum-paths 4
   neighbor SPINE_EVPN peer-group
   neighbor SPINE_EVPN remote-as 65001
   neighbor SPINE_EVPN update-source Loopback0
   neighbor SPINE_EVPN allowas-in 2
   neighbor SPINE_EVPN ebgp-multihop 5
   neighbor SPINE_EVPN send-community extended
   neighbor SPINE_EVPN maximum-routes 12000
   neighbor 1.1.1.1 peer-group SPINE_EVPN
   neighbor 1.1.1.2 peer-group SPINE_EVPN
   redistribute connected route-map loopback
   !
   vlan 10
      rd 1.1.1.22:1010
      route-target both 1010:1010
      redistribute learned
   !
   vlan 11
      rd 1.1.1.22:1011
      route-target both 1011:1011
      redistribute learned
   !
   vlan 21
      rd 1.1.1.22:1021
      route-target both 1021:1021
      redistribute learned
   !
   vlan 210
      rd 1.1.1.22:1210
      route-target both 1210:1210
      redistribute learned
      no redistribute host-route
   !
   vlan 211
      rd 1.1.1.22:1211
      route-target both 1211:1211
      redistribute learned
      no redistribute host-route
   !
   vlan 221
      rd 1.1.1.22:1221
      route-target both 1221:1221
      redistribute learned
      no redistribute host-route
   !
   vlan-aware-bundle Tenant-A-VLAN-12-13
      rd 1.1.1.22:1213
      route-target both 12:13
      redistribute learned
      vlan 12-13
   !
   vlan-aware-bundle Tenant-B-VLAN-212-213
      rd 1.1.1.22:21213
      route-target both 212:213
      redistribute learned
      no redistribute host-route
      vlan 212-213
   !
   address-family evpn
      neighbor SPINE_EVPN activate
   !
   address-family ipv4
      no neighbor SPINE_EVPN activate
   !
   vrf tenant-a
      rd 1.1.1.22:1000
      route-target import 1000:1000
      route-target export 1000:1000
      neighbor 223.255.255.253 remote-as 65002
      neighbor 223.255.255.253 next-hop-self
      neighbor 223.255.255.253 update-source Vlan1111
      neighbor 223.255.255.253 allowas-in 1
      neighbor 223.255.255.253 maximum-routes 12000
      redistribute connected route-map dont_advertise_loopbacks
   !
   vrf tenant-b
      rd 1.1.1.22:1001
      route-target import 1001:1001
      route-target export 1001:1001
      neighbor 223.255.255.253 remote-as 65002
      neighbor 223.255.255.253 next-hop-self
      neighbor 223.255.255.253 update-source Vlan2111
      neighbor 223.255.255.253 allowas-in 1
      neighbor 223.255.255.253 maximum-routes 12000
      redistribute connected route-map dont_advertise_loopbacks

eBGP Overlay on Spine Switches

The EVPN BGP configuration on the spine switches is summarised in the following examples. Note that only the EVPN BGP sessions are listed for two spine switches; the BGP underlay configuration is not included.

EVPN BGP Overlay Configuration: Spine-1

!
router bgp 65001
   router-id 1.1.1.1
   distance bgp 20 200 200
   maximum-paths 8 ecmp 16
   neighbor LEAF_EVPN peer-group
   neighbor LEAF_EVPN remote-as 65002
   neighbor LEAF_EVPN update-source Loopback0
   neighbor LEAF_EVPN ebgp-multihop 5
   neighbor LEAF_EVPN send-community extended
   neighbor LEAF_EVPN next-hop-unchanged 
   neighbor LEAF_EVPN maximum-routes 12000  
   neighbor 1.1.1.11 peer-group LEAF_EVPN
   neighbor 1.1.1.12 peer-group LEAF_EVPN
   neighbor 1.1.1.21 peer-group LEAF_EVPN
   neighbor 1.1.1.22 peer-group LEAF_EVPN
   !
   address-family evpn
      neighbor LEAF_EVPN activate
   !
   address-family ipv4
      no neighbor LEAF_EVPN activate
!
   address-family ipv6
      no neighbor LEAF_EVPN activate
!

EVPN BGP Overlay Configuration: Spine-2

!
router bgp 65001
   router-id 1.1.1.2
   distance bgp 20 200 200
   maximum-paths 8 ecmp 16
   neighbor LEAF_EVPN peer-group
   neighbor LEAF_EVPN remote-as 65002
   neighbor LEAF_EVPN update-source Loopback0
   neighbor LEAF_EVPN ebgp-multihop 5
   neighbor LEAF_EVPN send-community extended
   neighbor LEAF_EVPN next-hop-unchanged 
   neighbor LEAF_EVPN maximum-routes 12000 
   neighbor 1.1.1.11 peer-group LEAF_EVPN
   neighbor 1.1.1.12 peer-group LEAF_EVPN
   neighbor 1.1.1.21 peer-group LEAF_EVPN
   neighbor 1.1.1.21 peer-group LEAF_EVPN 
   !
   address-family evpn
      neighbor LEAF_EVPN activate
   !
   address-family ipv4
      no neighbor LEAF_EVPN activate
!
   address-family ipv6
      no neighbor LEAF_EVPN activate
!

Symmetric IRB Configuration (Tenant-A)

In symmetric IRB, the host routes are generated by advertising type-2 routes with both the MAC VRF VNI and the routing (or VRF) VNI. On Leaf-11, the MAC VRFs for Tenant-A are left in their default configuration (i.e., redistributing host routes). The following example shows the configuration for the MAC VRF.

MAC VRF Configuration for Tenant-A: Leaf-11

The redistribute learned commands below cause type-2 routes to be advertised with two labels: in VLAN 10, 1010 and 1000; in VLAN 11, 1011 and 1000; in VLAN 21, 1021 and 1000.

vlan 10
      rd 1.1.1.11:1010
      route-target both 1010:1010
      redistribute learned
!
   vlan 11
      rd 1.1.1.11:1011
      route-target both 1011:1011
      redistribute learned
!
   vlan 21
      rd 1.1.1.11:1021
      route-target both 1021:1021
      redistribute learned
   !

With this configuration, any locally learned MAC-IP binding on a leaf switch will be advertised as a type-2 route with two labels. For example, on switches Leaf-21 and Leaf-22, any MAC-IP binding locally learned on subnets 10.10.10.0/24, 10.10.11.0/24, or 10.10.21.0/24 will be advertised as type-2 routes with two labels (the MAC VRF of 1010, 1011, or 1021 and the IP VRF of 1000) and two route targets equal to the relevant MAC VRF for the host and IP VRF for the tenant (1000:1000). The remote leaf switches (Leaf-11 and Leaf-12), will now learn the host route in the IP VRF.

In addition to advertising the type-2 routes with dual labels, the switch will still advertise type-5 routes. This ensures connectivity to the remote subnet even when no host on the subnet has been learned. With both a layer-2 route and layer-3 host route for Server-3 learned on the MAC VRF(1010) and the IP VRF (1000) on Leaf-11, traffic ingressing on Leaf-11 from the local subnet 10.10.10.103 (i.e., VLAN 10) will be VXLAN bridged based on the MAC VRF entry. Traffic ingressing from outside the subnet (i.e., VLAN 11, 12, 13, or 20) will be routed to the host via the IP VRF host route.

The VLAN-aware bundle VLAN type-2 routes are advertised with the VNI ID within the update.

The type-5 routes are advertised with the IP VRF Route Distinguisher and the VNI label, signifying that the forwarding path for the prefix would be the IP VRF. The imported routes from the eBGP peering with the BGP border router in Leaf-11 and Leaf-12 are imported by both switches, and redistributed via type-5 advertisements to Leaf-21 and Leaf-22.

Asymmetric IRB Configuration (Tenant-B)

In asymmetric IRB, the host routes are generated by advertising type-2 routes with just the MAC VRF VNI. On leaf 11, the MAC VRFs for Tenant-B are configured with no redistribute host route within the MAC VRF configuration. The following example shows the configuration for the MAC VRF.

MAC VRF Configuration for Tenant-B: Leaf-11

The no redistribute host-route commands below cause type-2 routes to be advertised with a single label: in VLAN 210, 1110; in VLAN 211, 1211; in VLAN 220, 1220; and in the VLAN-aware bundle (Tenant-B-VLAN-212-213), 1212 and 1213.

vlan 210
      rd 1.1.1.11:1210
      route-target both 1210:1210
      redistribute learned
      no redistribute host-route
   !
   vlan 211
      rd 1.1.1.11:1211
      route-target both 1211:1211
      redistribute learned
      no redistribute host-route
   !
   vlan 220
      rd 1.1.1.11:1220
      route-target both 1220:1220
      redistribute learned
      no redistribute host-route
!
     vlan-aware-bundle Tenant-B-VLAN-212-213
      rd 1.1.1.11:21213
      route-target both 212:213
      redistribute learned
      no redistribute host-route
      vlan 212-213 
   !

With this configuration, any locally learned MAC-IP binding on a leaf switch will be advertised as a type-2 route with a single label. For example, on Leaf-11 and Leaf-12, any MAC-IP binding locally learned on subnets 10.10.10.0/24, 10.10.11.0/24, or 10.10.21.0/24 will be advertised as type-2 routes with a single label, the MAC VRF (1210, 1211, 1220, 1212, 1213, or 21111). The IP VRF (1001) still advertises the type-5 prefix routes. This ensures connectivity to the remote subnet even when no host on the subnet has been learned.

The VLAN-aware bundle VLAN type-2 routes are advertised with the VNI ID within the update.

EVPN MPLS Sample Configuration

This section describes configuring and verifying BGP VPN, which has steps similar to the EVPN VXLAN demonstration. Here, we examine BGP EVPN layer 3 VPN over LDP, Segment Routing (ISIS-SR), and BGP-SR transport LSPs. This highlights the difference between the transport and the VPN overlay service.

Layer 3 VPN Over ISIS-SR

The following figures illustrate the overview of combined control and data planes.

Figure 5. Control Plane Tenant-A Over ISIS-SR

Figure 6. Control Plane Tenant-B over ISIS-SR

Figure 7. Control Plane and Forwarding Tenant-A Over ISIS-SR

The North Edge router has an eBGP peering session out to Leaf-11 and Leaf-12 in DC1, while the South Edge router has peerings to Leaf-11 andLeaf-12 in DC2. Tenant-a has few additional local interfaces used for testing.

Example

The show ip route vrf tenant-a connected command displays the interfaces assigned to the tenant-a of North Edge router.

north-edge# show ip route vrf tenant-a connected

VRF: tenant-a
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route

 C      192.168.168.8/30 is directly connected, Ethernet6/3.1
 C      192.168.168.12/30 is directly connected, Ethernet6/2.1

Activating EVPN

In all scenarios, the EVPN must be activated under BGP and neighbors configured to exchange Layer 2 VPN/EVPN NLRI. The tenant’s VRF (tenant-a and tenant-b) is associated with a dynamically assigned label by BGP.

An activated EVPN provides the following functionalities:
  • Enables the multi-agent routing protocol model, which is required for EVPN support.
  • Sets the local autonomous system number to 64512 and configures IBGP neighbors that are activated for the Layer 2 VPN/EVPN address family.
  • Sets the EVPN encapsulation type to MPLS.
  • Specifies that Loopback0 will be used as the next-hop for all advertised EVPN routes. The underlay configuration must provide MPLS LSPs from remote PEs to this loopback interface address

    .

Example

The service routing protocols model multi-agent command activates EVPN on the north edge router.
service routing protocols model multi-agent

router bgp 64512
   router-id 1.1.1.111
   maximum-paths 128 ecmp 128
   neighbor 2.2.2.222 remote-as 64512
   neighbor 2.2.2.222 update-source Loopback0
   neighbor 2.2.2.222 bfd
   neighbor 2.2.2.222 send-community extended
   !
   address-family evpn
     neighbor default encapsulation mpls next-hop-self source-interface Loopback0
       neighbor default graceful-restart
       neighbor 2.2.2.222 activate
   !

Layer 3 Overlay Configuration

Distribution of layer 3 routes over BGP is enabled by configuring one or more IP VRFs under the router bgp configuration mode. Additionally, IP routing must be enabled in the VRF.

The VRF is assigned a unique Route-Distinguisher (RD). The RD allows the PE to advertise EVPN routes for the same IP prefix that have been exported by different VRFs. The NLRI RouteKey of a route exported from the VRF’s IPv4 table into EVPN consists of both the RD and the original IP prefix.

The Route-Target (RT) extended communities for the VRF. The RTs are associated with all routes exported from the VRF. Received EVPN type-5 routes carrying at least one RT matching the VRFs configuration are imported into the VRF. The route target directives are configured under the IPv4 or IPv6 address- family.

Example

The vrf tenant-a and vrf tenant-a commands define overlay VRFs (tenant-a and tenant-b) on the VTEP of North Edge router and enables IPv4 routing within them.
vrf tenant-a
      rd 1.1.1.1:64512
      route-target import evpn 64512:11
      route-target export evpn 64512:11
      router-id 1.1.1.111
      neighbor 192.168.168.10 remote-as 65002
      neighbor 192.168.168.10 local-as 64512 no-prepend replace-as
      neighbor 192.168.168.10 default-originate
      neighbor 192.168.168.10 maximum-routes 12000
      neighbor 192.168.168.14 remote-as 65002
      neighbor 192.168.168.14 local-as 64512 no-prepend replace-as
      neighbor 192.168.168.14 default-originate
      neighbor 192.168.168.14 maximum-routes 12000
      redistribute connected
      redistribute static
   !
   vrf tenant-b
      rd 1.1.1.1:64513
      route-target import evpn 64513:11
      route-target export evpn 64513:11
      router-id 1.1.1.111
      neighbor 192.168.168.20 remote-as 65002
      neighbor 192.168.168.20 local-as 64513 no-prepend replace-as
      neighbor 192.168.168.20 maximum-routes 12000
      neighbor 192.168.168.22 remote-as 65002
      neighbor 192.168.168.22 local-as 64513 no-prepend replace-as
      neighbor 192.168.168.22 maximum-routes 12000
      redistribute connected
      redistribute static
   !

Verifying BGP EVPN Layer 3 VPN

Show commands are executed in the North Edge router to view routes to the South Edge router. Execute the same commands in the South Edge router to view vice-versa routes.

Examples
  • The show bgp evpn summary command displays the status of EVPN peers in North Edge router.
    north-edge# show bgp evpn summary
    BGP summary information for VRF default
    Router identifier 1.1.1.111, local AS number 64512
    Neighbor Status Codes: m - Under maintenance
      Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  
    PfxRcd PfxAcc
      2.2.2.222        4  64512            195       127    0    0 01:13:31 Estab  78     78

  • The show bgp evpn route-type ip-prefix ipv4 next-hop 6.6.6.6 command displays all BGP EVPN ip prefix routes received from the South Edge router (6.6.6.6). Not all are advertised via the RR 2.2.2.222.

    Note: Each entry in the table represents a BGP path. The path specific information includes Route-Distinguisher and IP prefix. Paths are either received from EVPN peers or exported from local VRFs.
    north-edge# show bgp evpn route-type ip-prefix ipv4 next-hop 6.6.6.6
    BGP routing table information for VRF default
    Router identifier 1.1.1.111, local AS number 64512
    Route status codes: s - suppressed, * - valid, > - active, # - not installed, E - ECMP head, e - ECMP
                        S - Stale, c - Contributing to ECMP, b - backup
                        % - Pending BGP convergence
    Origin codes: i - IGP, e - EGP, ? - incomplete
    AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
    
             Network             Next Hop         Metric  LocPref Weight Path
     * >     RD: 6.6.6.6:64512 ip-prefix 0.0.0.0/0
                                 6.6.6.6          0       100     0       ? Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     RD: 6.6.6.6:64513 ip-prefix 0.0.0.0/0
                                 6.6.6.6          0       100     0       ? Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     RD: 6.6.6.6:64514 ip-prefix 10.255.255.0/30
                                 6.6.6.6          -       100     0      65010 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     RD: 6.6.6.6:64512 ip-prefix 100.10.10.0/24
                                 6.6.6.6          -       100     0      65006 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     RD: 6.6.6.6:64513 ip-prefix 100.10.10.0/24
                                 6.6.6.6          -       100     0      65006 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     RD: 6.6.6.6:64512 ip-prefix 100.10.10.103/32
                                 6.6.6.6          -       100     0      65006 65005 65006 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     RD: 6.6.6.6:64512 ip-prefix 100.10.10.104/32
                                 6.6.6.6          -       100     0      65006 65005 65006 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     RD: 6.6.6.6:64512 ip-prefix 100.10.11.0/24
                                 6.6.6.6          -       100     0      65006 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     RD: 6.6.6.6:64513 ip-prefix 100.10.11.0/24
                                 6.6.6.6          -       100     0      65006 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     RD: 6.6.6.6:64512 ip-prefix 100.10.11.103/32
                                 6.6.6.6          -       100     0      65006 65005 65006 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     RD: 6.6.6.6:64512 ip-prefix 100.10.11.104/32
                                 6.6.6.6          -       100     0      65006 65005 65006 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222

  • The show bgp evpn route-type ip-prefix 100.10.11.0/24 detail command displays a detailed view of the IP prefix route for 100.10.11.0/24. The output again includes the RD and IP prefix identifying the route. As seen above the route is received from the route reflector, and the VPN label for tenant-a is 958810.
    north-edge# show bgp evpn route-type ip-prefix 100.10.11.0/24 detail
    BGP routing table information for VRF default
    Router identifier 1.1.1.111, local AS number 64512
    BGP routing table entry for ip-prefix 100.10.11.0/24, Route Distinguisher: 6.6.6.6:64512
     Paths: 1 available
      65006
        6.6.6.6 from 2.2.2.222 (2.2.2.222)
          Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
          Extended Community: Route-Target-AS:64512:11 TunnelEncap:tunnelTypeMpls
          MPLS label: 958810
    BGP routing table entry for ip-prefix 100.10.11.0/24, Route Distinguisher: 6.6.6.6:64513
     Paths: 1 available
      65006
        6.6.6.6 from 2.2.2.222 (2.2.2.222)
          Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
          Extended Community: Route-Target-AS:64513:11 TunnelEncap:tunnelTypeMpls
          MPLS label: 953372

    Note: Tenant-a and tenant-b share the same route. Therefore, both route with RD 6.6.6.6:64513 and RT 64513:11.

  • The show ip bgp vrf tenant-a command displays the BGP table for VRF in tenant-a containing imported EVPN routes. Each entry in the table represent a BGP path that is either locally redistributed / received into the VRF or imported from the EVPN table.
    north-edge# show ip bgp vrf tenant-a
    BGP routing table information for VRF tenant-a
    Router identifier 1.1.1.111, local AS number 64512
    Route status codes: s - suppressed, * - valid, > - active, # - not installed, E - ECMP head, e - ECMP
                        S - Stale, c - Contributing to ECMP, b - backup, L - labeled-unicast
                        % - Pending BGP convergence
    Origin codes: i - IGP, e - EGP, ? - incomplete
    AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
    
             Network             Next Hop         Metric  LocPref Weight Path
     * >     0.0.0.0/0           6.6.6.6          0       100     0      ? Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >Ec   10.10.10.0/24       192.168.168.14   -       100     0      65002 i
     *  ec   10.10.10.0/24       192.168.168.10   -       100     0      65002 i
     * >Ec   10.10.10.103/32     192.168.168.14   -       100     0      65002 i
     *  ec   10.10.10.103/32     192.168.168.10   -       100     0      65002 i
     * >Ec   10.10.10.104/32     192.168.168.14   -       100     0      65002 i
    
     * >Ec   10.10.44.1/32       192.168.168.14   -       100     0      65002 i
     *  ec   10.10.44.1/32       192.168.168.10   -       100     0      65002 i
     * >     100.10.10.0/24      6.6.6.6          -       100     0      65006 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     100.10.10.103/32    6.6.6.6          -       100     0      65006 65005 65006 i Or-ID: 6.6.6.6 
    C-LST: 2.2.2.222
     * >     100.10.10.104/32    6.6.6.6          -       100     0      65006 65005 65006 i Or-ID: 6.6.6.6 
    C-LST: 2.2.2.222
    C-LST: 2.2.2.222
     * >     100.10.21.102/32    6.6.6.6          -       100     0      65006 65005 65006 i Or-ID: 6.6.6.6 
    C-LST: 2.2.2.222
     * >     100.10.30.0/24      6.6.6.6          -       100     0      65006 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     100.10.32.0/24      6.6.6.6          -       100     0      65006 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     192.168.168.0/30    6.6.6.6          -       100     0      i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     192.168.168.4/30    6.6.6.6          -       100     0      i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     192.168.168.8/30    -                -       -       0      i
     *  Ec   192.168.168.8/30    192.168.168.14   -       100     0      65002 i
     *  ec   192.168.168.8/30    192.168.168.10   -       100     0      65002 i
     * >     192.168.168.12/30   -                -       -       0      i
     *  Ec   192.168.168.12/30   192.168.168.14   -       100     0      65002 i
     *  ec   192.168.168.12/30   192.168.168.10   -       100     0      65002 i
     * >     223.255.254.248/30  6.6.6.6          -       100     0      65006 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     223.255.254.252/30  6.6.6.6          -       100     0      65006 65005 65006 i Or-ID: 6.6.6.6 
    C-LST: 2.2.2.222
     * >Ec   223.255.255.248/30  192.168.168.14   -       100     0      65002 i
     *  ec   223.255.255.248/30  192.168.168.10   -       100     0      65002 i
     * >Ec   223.255.255.252/30  192.168.168.14   -       100     0      65002 i
     *  ec   223.255.255.252/30  192.168.168.10   -       100     0      65002 i
    

    Note: EVPN routes are received from router 2.2.2.222 C-List (cluster list - basically identifying this route as from a route-reflector) with originating router being 6.6.6.6.

  • The show ip route vrf tenant-b command displays the BGP table for VRF in tenant-b containing imported EVPN routes.
    north-edge# show ip route vrf tenant-b
    
    VRF: tenant-b
    Codes: C - connected, S - static, K - kernel,
           O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
           E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
           N2 - OSPF NSSA external type2, B I - iBGP, B E - eBGP,
           R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
           O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
           NG - Nexthop Group Static Route, V - VXLAN Control Service,
           DH - DHCP client installed default route, M - Martian,
           DP - Dynamic Policy Route
    
    Gateway of last resort:
     B I    0.0.0.0/0 [200/0] via 6.6.6.6/32, IS-IS SR tunnel index 6, label 953372
                                 via 192.168.58.12, Ethernet1/1, label 408006
                                 via 192.168.59.12, Ethernet2/1, label 408006
    
     B E    10.10.10.0/24 [200/0] via 192.168.168.22, Ethernet6/2.2
                                  via 192.168.168.20, Ethernet6/3.2
    
     B E    10.10.21.0/24 [200/0] via 192.168.168.22, Ethernet6/2.2
                                  via 192.168.168.20, Ethernet6/3.2
     B I    100.10.10.0/24 [200/0] via 6.6.6.6/32, IS-IS SR tunnel index 6, label 953372
                                      via 192.168.58.12, Ethernet1/1, label 408006
                                      via 192.168.59.12, Ethernet2/1, label 408006
    
     C      192.168.168.20/31 is directly connected, Ethernet6/3.2
     C      192.168.168.22/31 is directly connected, Ethernet6/2.2
     B I    223.255.254.248/30 [200/0] via 6.6.6.6/32, IS-IS SR tunnel index 6, label 953372
                                          via 192.168.58.12, Ethernet1/1, label 408006
                                          via 192.168.59.12, Ethernet2/1, label 408006
     B I    223.255.254.252/30 [200/0] via 6.6.6.6/32, IS-IS SR tunnel index 6, label 953372
                                          via 192.168.58.12, Ethernet1/1, label 408006
                                          via 192.168.59.12, Ethernet2/1, label 408006
     B E    223.255.255.248/30 [200/0] via 192.168.168.22, Ethernet6/2.2
                                       via 192.168.168.20, Ethernet6/3.2
     B E    223.255.255.252/30 [200/0] via 192.168.168.22, Ethernet6/2.2
                                       via 192.168.168.20, Ethernet6/3.2

    Note: If we look at the routes in the VRF for tenant-b, we see that the VPN label has now changed, whilst the transport label for NH 6.6.6.6 is the same. The only difference seen in tenant-b, aside from the different VPN label, is that there are no host-routes in tenant-b because within each DC tenant-b is running in asymmetric mode, therefore no host routes are generated/installed in the IP VRF.

Layer 3 EVPN Over LDP

The following figures illustrate an overview of the combined control and data planes.

Figure 8. Control Plane Tenant-A Over LDP

Figure 9. Control Plane Tenant-B over LDP

Figure 10. Control Plane & Forwarding Tenant-A Over LDP


To switch to using the MPLS LDP transport, change the next-hop advertised for EVPN routes. As illustrated above, the next hop needs to be set to loopback 200 to use the LDP LSP.

This is achieved by configuring the next-hop for EVPN routes on both North Edge and South Edge routes. The output again includes the RD and IP prefixes identifying the route. As seen in the output, we now have the NH set to 6.6.6.200 for tenant-a and tenant-b.

router bgp 64512
   !
   address-family evpn
     neighbor default encapsulation mpls next-hop-self source-interface Loopback200

Once this is configured, we can check the BGP updates and the routes in the VRF.

north-edge# show bgp evpn route-type ip-prefix 100.10.11.0/24 detail
BGP routing table information for VRF default
Router identifier 1.1.1.111, local AS number 64512
BGP routing table entry for ip-prefix 100.10.11.0/24, Route Distinguisher: 6.6.6.6:64512
 Paths: 1 available
  65006
    6.6.6.200 from 2.2.2.222 (2.2.2.222)
      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
      Extended Community: Route-Target-AS:64512:11 TunnelEncap:tunnelTypeMpls
      MPLS label: 958810
BGP routing table entry for ip-prefix 100.10.11.0/24, Route Distinguisher: 6.6.6.6:64513
 Paths: 1 available
  65006
    6.6.6.200 from 2.2.2.222 (2.2.2.222)
      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
      Extended Community: Route-Target-AS:64513:11 TunnelEncap:tunnelTypeMpls
      MPLS label: 953372

Note: We have the same route in tenant-a and tenant-b in DC2. Therefore, VPN label has not changed in the two other routes with RD 6.6.6.6:64513 and RT 64513:11, reinforcing the fact that the BGP VPN label is orthogonal to the transport label.

Finally, look at the routes in the VRF tenant-a.

north-edge# show ip route vrf tenant-a

VRF: tenant-a
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS ----level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route

Gateway of last resort:
 B I    0.0.0.0/0 [200/0] via 6.6.6.200/32, LDP tunnel index 1, label 958810
                             via 192.168.58.12, Ethernet1/1, label 904097
                             via 192.168.59.12, Ethernet2/1, label 904098

 B E    10.10.10.103/32 [200/0] via 192.168.168.14, Ethernet6/2.1
                                via 192.168.168.10, Ethernet6/3.1
 B E    10.10.10.104/32 [200/0] via 192.168.168.14, Ethernet6/2.1

                              via 192.168.168.10, Ethernet6/3.1
 B I    100.10.10.103/32 [200/0] via 6.6.6.200/32, LDP tunnel index 1, label 958810
                                    via 192.168.58.12, Ethernet1/1, label 904097
                                    via 192.168.59.12, Ethernet2/1, label 904098

 B I    192.168.168.4/30 [200/0] via 6.6.6.200/32, LDP tunnel index 1, label 958810
                                    via 192.168.58.12, Ethernet1/1, label 904097
                                    via 192.168.59.12, Ethernet2/1, label 904098
 C      192.168.168.8/30 is directly connected, Ethernet6/3.1
 C      192.168.168.12/30 is directly connected, Ethernet6/2.1
 B I    223.255.254.248/30 [200/0] via 6.6.6.200/32, LDP tunnel index 1, label 958810
                                      via 192.168.58.12, Ethernet1/1, label 904097
                                      via 192.168.59.12, Ethernet2/1, label 904098
 B I    223.255.254.252/30 [200/0] via 6.6.6.200/32, LDP tunnel index 1, label 958810
                                      via 192.168.58.12, Ethernet1/1, label 904097
                                      via 192.168.59.12, Ethernet2/1, label 904098
 B E    223.255.255.248/30 [200/0] via 192.168.168.14, Ethernet6/2.1
                                   via 192.168.168.10, Ethernet6/3.1
 B E    223.255.255.252/30 [200/0] via 192.168.168.14, Ethernet6/2.1
                                   via 192.168.168.10, Ethernet6/3.1

Note: As can be seen from the highlighted route above the label stack, the route has the same VPN route 958810, but the transport labels are now 904097 and 904098 on top (this is the ECMP label path to reach NH 6.6.6.200).

As a comparison, let us look at the routes for tenant-b.

north-edge# show ip route vrf tenant-b

VRF: tenant-b
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route

Gateway of last resort:
 B I    0.0.0.0/0 [200/0] via 6.6.6.200/32, LDP tunnel index 1, label 953372
                             via 192.168.58.12, Ethernet1/1, label 904097
                             via 192.168.59.12, Ethernet2/1, label 904098

 B E    10.10.10.0/24 [200/0] via 192.168.168.22, Ethernet6/2.2
                              via 192.168.168.20, Ethernet6/3.2

                              via 192.168.168.20, Ethernet6/3.2
 B I    100.10.10.0/24 [200/0] via 6.6.6.200/32, LDP tunnel index 1, label 953372
                                  via 192.168.58.12, Ethernet1/1, label 904097
                                  via 192.168.59.12, Ethernet2/1, label 904098

                                     via 192.168.59.12, Ethernet2/1, label 904098
 B I    192.168.168.18/31 [200/0] via 6.6.6.200/32, LDP tunnel index 1, label 953372
                                     via 192.168.58.12, Ethernet1/1, label 904097
                                     via 192.168.59.12, Ethernet2/1, label 904098
 C      192.168.168.20/31 is directly connected, Ethernet6/3.2
 C      192.168.168.22/31 is directly connected, Ethernet6/2.2
 B I    223.255.254.248/30 [200/0] via 6.6.6.200/32, LDP tunnel index 1, label 953372
                                      via 192.168.58.12, Ethernet1/1, label 904097
                                      via 192.168.59.12, Ethernet2/1, label 904098
 B I    223.255.254.252/30 [200/0] via 6.6.6.200/32, LDP tunnel index 1, label 953372
                                      via 192.168.58.12, Ethernet1/1, label 904097
                                      via 192.168.59.12, Ethernet2/1, label 904098
 B E    223.255.255.248/30 [200/0] via 192.168.168.22, Ethernet6/2.2
                                   via 192.168.168.20, Ethernet6/3.2
 B E    223.255.255.252/30 [200/0] via 192.168.168.22, Ethernet6/2.2

Note: The only difference apart from the missing host routes (no host-route inject for this tenant), is the VPN label.

Layer 3 EVPN Over BGP-SR

The following figures illustrate an overview of the combined control and data planes.

Figure 11. Control Plane Tenant-A Over BGP-SR

Figure 12. Control Plane Tenant-B Over BGP-SR

Figure 13. Control Plane and Forwarding Tenant-A Over BGP-SR

To switch to using the MPLS BGP-SR transport, we need to change the next-hop advertised for the EVPN routes. As shown in Control Plane tenant-b Over BGP-SR, the next hop needs to be set to loopback 1 for using the BGP-SR LSP, by configuring the next-hop for the EVPN routes.

router bgp 64512
   !
   address-family evpn
     neighbor default encapsulation mpls next-hop-self source-interface Loopback1

Once the next-hop for the EVPN routes are configured, we can check the BGP updates and the routes in the VRF. The output again includes the RD and IP prefix identifying the route. As seen in the output, we now have the NH set to 6.6.6.66 for tenant-a and tenant-b.

North Edge.17:52:30# show bgp evpn route-type ip-prefix 100.10.11.0/24 detail

north-edge(config-if-Et2/1)#show bgp evpn route-type ip-prefix 100.10.11.0/24 detail
BGP routing table information for VRF default
Router identifier 1.1.1.111, local AS number 64512
BGP routing table entry for ip-prefix 100.10.11.0/24, Route Distinguisher: 6.6.6.6:64512
 Paths: 1 available
  65006
    6.6.6.66 from 2.2.2.222 (2.2.2.222)
      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
      Extended Community: Route-Target-AS:64512:11 TunnelEncap:tunnelTypeMpls
      MPLS label: 958810
BGP routing table entry for ip-prefix 100.10.11.0/24, Route Distinguisher: 6.6.6.6:64513
 Paths: 1 available
  65006
    6.6.6.66 from 2.2.2.222 (2.2.2.222)
      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
      Extended Community: Route-Target-AS:64513:11 TunnelEncap:tunnelTypeMpls
      MPLS label: 953372

Note: Again, we have the same route in tenant-a and tenant-b in DC2. Therefore, the two other routes with RD 6.6.6.6:64513 and RT 64513:11. The VPN label has not changed, reinforcing the fact that the BGP VPN label is orthogonal to the transport label.

Finally, let us look at the routes in the VRF tenant-a.

North Edge.17:55:01# show ip route vrf tenant-a

VRF: tenant-a
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route

Gateway of last resort:
 B I    0.0.0.0/0 [200/0] via 6.6.6.66/32, BGP LU tunnel index 8, label 958810
                             via 192.168.58.12, Ethernet1/1, label 200066
                             via 192.168.59.12, Ethernet2/1, label 200066

 B E    10.10.10.103/32 [200/0] via 192.168.168.14, Ethernet6/2.1
                                via 192.168.168.10, Ethernet6/3.1
 B E    10.10.10.104/32 [200/0] via 192.168.168.14, Ethernet6/2.1
                                via 192.168.168.10, Ethernet6/3.1

                              via 192.168.168.10, Ethernet6/3.1
 B I    100.10.10.103/32 [200/0] via 6.6.6.66/32, BGP LU tunnel index 8, label 958810
                                    via 192.168.58.12, Ethernet1/1, label 200066
                                    via 192.168.59.12, Ethernet2/1, label 200066

 B I    192.168.168.4/30 [200/0] via 6.6.6.66/32, BGP LU tunnel index 8, label 958810
                                    via 192.168.58.12, Ethernet1/1, label 200066
                                    via 192.168.59.12, Ethernet2/1, label 200066
 C      192.168.168.8/30 is directly connected, Ethernet6/3.1
 C      192.168.168.12/30 is directly connected, Ethernet6/2.1
 B I    223.255.254.248/30 [200/0] via 6.6.6.66/32, BGP LU tunnel index 8, label 958810
                                      via 192.168.58.12, Ethernet1/1, label 200066
                                      via 192.168.59.12, Ethernet2/1, label 200066
 B I    223.255.254.252/30 [200/0] via 6.6.6.66/32, BGP LU tunnel index 8, label 958810
                                      via 192.168.58.12, Ethernet1/1, label 200066
                                      via 192.168.59.12, Ethernet2/1, label 200066
 B E    223.255.255.248/30 [200/0] via 192.168.168.14, Ethernet6/2.1
                                   via 192.168.168.10, Ethernet6/3.1
 B E    223.255.255.252/30 [200/0] via 192.168.168.14, Ethernet6/2.1
                                   via 192.168.168.10, Ethernet6/3.1

As can be seen from the highlighted route above the label stack, the route are the transport labels 958810 and 200066 on top (this is the ECMP label path to reach NH 6.6.6.66), with the tenant-a VPN label 958810 next in the stack, identifying the route as belonging to tenant-a.

As a comparison, look at the routes for tenant-b. As seen in the output, the VPN label assigned to tenant-b is 953372.

north-edge# show bgp evpn route-type ip-prefix 100.10.11.0/24 detail
BGP routing table information for VRF default
Router identifier 1.1.1.111, local AS number 64512
BGP routing table entry for ip-prefix 100.10.11.0/24, Route Distinguisher: 6.6.6.6:64512
 Paths: 1 available
  65006
    6.6.6.66 from 2.2.2.222 (2.2.2.222)
      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
      Extended Community: Route-Target-AS:64512:11 TunnelEncap:tunnelTypeMpls
      MPLS label: 958810
BGP routing table entry for ip-prefix 100.10.11.0/24, Route Distinguisher: 6.6.6.6:64513
 Paths: 1 available
  65006
    6.6.6.66 from 2.2.2.222 (2.2.2.222)
      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
      Extended Community: Route-Target-AS:64513:11 TunnelEncap:tunnelTypeMpls
      MPLS label: 953372
north-edge#

If we now look at the routes in the VRF for tenant-b, we see that the VPN label has now changed, while the transport label (for NH 6.6.6.66 is the same). The only difference seen in tenant-b, aside from the different VPN label, is that there are no host-routes in tenant-b because within each DC tenant-b is running in asymmetric mode; therefore, no host routes are generated/installed in the IP VRF.

north-edge# show ip route vrf tenant-b

VRF: tenant-b
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route

Gateway of last resort:
 B I    0.0.0.0/0 [200/0] via 6.6.6.66/32, BGP LU tunnel index 8, label 953372
                             via 192.168.58.12, Ethernet1/1, label 200066
                             via 192.168.59.12, Ethernet2/1, label 200066

 B E    10.10.10.0/24 [200/0] via 192.168.168.22, Ethernet6/2.2
                              via 192.168.168.20, Ethernet6/3.2

 B E    10.10.21.0/24 [200/0] via 192.168.168.22, Ethernet6/2.2
                              via 192.168.168.20, Ethernet6/3.2
 B I    100.10.10.0/24 [200/0] via 6.6.6.66/32, BGP LU tunnel index 8, label 953372
                                  via 192.168.58.12, Ethernet1/1, label 200066
                                  via 192.168.59.12, Ethernet2/1, label 200066

 B I    192.168.168.18/31 [200/0] via 6.6.6.66/32, BGP LU tunnel index 8, label 953372
                                     via 192.168.58.12, Ethernet1/1, label 200066
                                     via 192.168.59.12, Ethernet2/1, label 200066
 C      192.168.168.20/31 is directly connected, Ethernet6/3.2
 C      192.168.168.22/31 is directly connected, Ethernet6/2.2
 B I    223.255.254.248/30 [200/0] via 6.6.6.66/32, BGP LU tunnel index 8, label 953372
                                      via 192.168.58.12, Ethernet1/1, label 200066
                                      via 192.168.59.12, Ethernet2/1, label 200066
 B I    223.255.254.252/30 [200/0] via 6.6.6.66/32, BGP LU tunnel index 8, label 953372
                                      via 192.168.58.12, Ethernet1/1, label 200066
                                      via 192.168.59.12, Ethernet2/1, label 200066
 B E    223.255.255.248/30 [200/0] via 192.168.168.22, Ethernet6/2.2
                                   via 192.168.168.20, Ethernet6/3.2
 B E    223.255.255.252/30 [200/0] via 192.168.168.22, Ethernet6/2.2
                                   via 192.168.168.20, Ethernet6/3.2  

EVPN VXLAN IPv6 Overlay

The EVPN VXLAN L3 Gateway using EVPN IRB supports routing traffic from one IPv6 host to another IPv6 host on a stretched VXLAN VLAN on platforms that support ND Proxy and ND suppression. The ipv6 address virtual command enables the use of one MAC address for all SVI instead of one per SVI. Both EVPN IRB and VXLAN tunnel interface are required for the feature to work. The VXLAN must be configured with a VNI or the VRF for the VLAN must be configured with a VRF/VNI mapping.

Configuring for Overlay

The following configures the switches for global IPv6 unicast routing and IPv6 unicast routing for each VRF.
switch(config)# ipv6 unicast-routing
switch(config)# ipv6 unicast-routing vrf tenant-c

The following configures the switches with a virtual MAC address, which is used for mapping all virtual router IP addresses. For VARP configs, the address is receive-only; the switch never sends packets with this address as the source. For ip address virtual, the address is also used as the source for ARP packets.
switch(config)# ipv6 virtual-router mac-address <mac>

The following shows the switch with IPv6 configured where one SVI uses one physical IP address.
switch# show run int vlan 501
interface Vlan501
      vrf forwarding tenant-c
      ipv6 enable
ipv6 address 2004:220::1:2/112
ipv6 virtual-router address 2004:220::1:10

The following shows configuration for the switch such that all SVI use the virtual MAC address and only one physical IP address.
switch# show run int vlan 501
interface Vlan501
      vrf forwarding tenant-c
      ipv6 enable
ipv6 address virtual 2004:220::1:10/112

Limitations

Any topology that requires a VXLAN Virtual VTEP address configuration is not supported.

Example Configurations

VRF-TO-VNI MAP and VLAN-TO-VNI MAP

Under VXLAN1 interface:

switch(config)# 
interface VXLAN1
   VXLAN vrf tenant-c vni 4001
   VXLAN vlan 501 vni 10501

MAC-VRF

Under BGP router configuration mode:

switch(config)# 
Router bgp 65000
vlan 501
      rd 20.1.1.1:10501
      route-target both 1:10501
      redistribute learned

IPv6 VRF BGP

switch(config)# 
router bgp 65000
vrf tenant-c
   rd 2.0.0.1:4001
   router-target import evpn 4001:4001
   router-target export evpn 4001:4001

! configure IPv4 router ID under the BGP VRF configuration
! for activating V6-only VRF
!
router-id 4.0.0.1

The selective installation configuration is the same for ARP and IPv6 ND.

switch(config)# router l2-vpn
switch(config-rtr-l2-vpn)#arp ?
 proxy              Proxy ARP
 selective-install  Install ARP entries for remote hosts on demand
switch(config-rtr-l2-vpn)#arp selective-install

The following disables the ND proxy reply to an NS for the specified target IPv6 address(es).

switch(config)# 
ipv6 prefix-list list-test
seq 10 deny 2000:0:0:69::19/64
! do not perform ND proxy on 2000:0:0:69::19/64

switch(config)# router l2-vpnswitch(config-rtr-l2-vpn)#nd proxy prefix-list list-test

The following restores the proxy behavior.

switch(config)# router l2-vpn
switch(config-rtr-l2-vpn)# no nd proxy prefix-list list-test

The following disables router solicitation packets sent by a host from getting flooded to all VTEPs.

switch(config)# router l2-vpn
switch(config-rtr-l2-vpn)# nd rs flooding disabled

The following restores the default behavior.

switch(config)# router l2-vpn
switch(config-rtr-l2-vpn)# no nd rs flooding disabled

The following disables Duplicate-Address-Detection (DAD) multicast packets from getting flooded to all VTEPs when there is no matching IP to MAC binding found in EVPN published IP to MAC bindings. When a match is found, a DAD frame is flooded to all VTEPs (instead of doing a proxy reply) to confirm that host liveliness.

switch(config)# router l2-vpn
switch(config-rtr-l2-vpn)# nd dad flooding disabled

The following restores the default behavior.

switch(config)# router l2-vpn
switch(config-rtr-l2-vpn)# no nd dad flooding disabled

The following disables Neighbor Advertisement (NA) multicast packets from the SVI configured as a virtual router from getting flooded to all VTEPs.

switch(config)# router l2-vpn
switch(config-rtr-l2-vpn)# virtual-router neighbor advertisement flooding disabled

The following restores the default behavior.

switch(config)# router l2-vpn
switch(config-rtr-l2-vpn)# no virtual-router neighbor advertisement flooding disabled

The following disables Gratuitous ARP multicast packets from the SVI configured as a virtual router from getting flooded to all VTEPs.

switch(config)# router l2-vpn
switch(config-rtr-l2-vpn)# virtual-router arp advertisement flooding disabled

The following restores the default behavior.

switch(config)# router l2-vpn
switch(config-rtr-l2-vpn)# no virtual-router arp advertisement flooding disabled

Checking the Status of the Switches

IPv6 Local Host

The following displays the ND bindings for a given VRF. The output shows that the local host 002c.0100.0001 has an IPv6 link local address fe80::22c:1ff:fe00:1 and a global IPv6 address 2004:220::1:50. The host is connected to the MLAG port-channel 20.

switch# show ipv6 neighbors vrf tenant-c vlan 501 | i 002c.0100.0001
2004:220::1:50          N/A 002c.0100.0001   REACH Vl501, Port-Channel20
fe80::22c:1ff:fe00:1    N/A 002c.0100.0001   REACH Vl501, Port-Channel20

EVPN IRB redistributes all the local hosts in VLAN 501. The MAC address of the host is advertised as EVPN Type 2 MAC-only route advertisement. The global IPv6 to MAC binding is advertised using MAC-IP route.

Note: By default, the IPv6 link local binding is not advertised by EVPN.

The following displays the two MAC-only routes and two MAC-IP routes. In both cases, one route is locally originated and the second one advertised by the MLAG peer with the same VTEP IP 10.0.0.1.

switch# show bgp evpn route-type mac-ip 002c.0100.0001
BGP routing table information for VRF default
Router identifier 1.0.1.1, local AS number 65000
Route status codes: s - suppressed, * - valid, > - active, # - not installed, E - ECMP head, e - ECMP
                    S - Stale, c - Contributing to ECMP, b - backup
                    % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

         Network                Next Hop            Metric  LocPref Weight  Path
 * >     RD: 20.1.1.1:10501 mac-ip 002c.0100.0001
                                -                     -       -       0        i
         RD: 20.1.1.2:10501 mac-ip 002c.0100.0001
                                10.0.0.1               -       100     0       65002 65003 i
 * >     RD: 20.1.1.1:10501 mac-ip 002c.0100.0001 2004:220::1:50
                                -                     -       -       0        i
         RD: 20.1.1.2:10501 mac-ip 002c.0100.0001 2004:220::1:50
                                10.0.0.1               -       100     0       65002 65003 i

IPv6 Link Local Redistribution

The following configures link-local redistribution command under BGP router MAC-VRF configuration mode to redistribute IPv6 link local binding.

vlan 501
rd 20.1.1.1:10501
route-target both 1:10501
redistribute learned
redistribute link-local ipv6

When this is configured, NS from a local host for a link local target will get proxy-replied by the ingress VTEP if the binding is published to EVPN by a remote VTEP. The NS in that case will not get replicated to other VTEPs.

IPv6 Remote Host

The following displays the MAC-only and MAC-IP routes for remote host 002d.0100.0001. These two routes originated from VTEP 10.0.0.2.

switch# show bgp evpn route-type mac-ip 002d.0100.0001 detail
BGP routing table information for VRF default
Router identifier 1.0.1.1, local AS number 65000

BGP routing table entry for mac-ip 002d.0100.0001, Route Distinguisher: 20.1.1.3:10501
 Paths: 1 available
  65002 65004
    10.0.0.2 from 1.0.1.111 (1.0.1.111)
      Origin IGP, metric -, localpref 100, weight 0, valid, external, best
      Extended Community: Route-Target-AS:1:10501 TunnelEncap:tunnelTypeVXLAN
      VNI: 10501 ESI: 0000:0000:0000:0000:0000

BGP routing table entry for mac-ip 002d.0100.0001 2004:220::1:151, Route Distinguisher: 20.1.1.3:10501
 Paths: 1 available
  65002 65004
    10.0.0.2 from 1.0.1.111 (1.0.1.111)
      Origin IGP, metric -, localpref 100, weight 0, valid, external, best
      Extended Community: Route-Target-AS:1:10501 Route-Target-AS:4001:4001 TunnelEncap:tunnelTypeVXLAN 
EvpnRouterMac:28:99:3a:be:53:42
      VNI: 10501 L3 VNI: 4003 ESI: 0000:0000:0000:0000:0000

IPv6 Remote Binding for Asymmetric IRB

The following displays the local MAC-VRF vlan 501 is configured to import RT two octets ASN RT 1:10501. The MAC-IP route is imported into remote binding for vlan 501.

switch# show ipv6 neighbors remote vlan 501
ARP remote bindings
VLAN IP Address      MAC Address
---- --------------- --------------
501  2004:220::1:151 002d.0100.0001

Without ARP Selective install, always install the remote IPv6 ND binding.

The following displays the ND bindings installed in the IPv6 cache. The interface for remote hosts is always VXLAN1 501 and is displayed with a '-'.

switch# show ipv6 neighbors vrf tenant-c vlan 501 2004:220::1:151
IPv6 Address          Age Hardware Addr    State Interface
2004:220::1:151         - 002d.0100.0001   REACH Vl501, VXLAN1

IPv6 Remote Host for Symmetric IRB

The following displays the BGP information for a specific IPv6 prefix in a VRF.

switch# show ipv6 bgp  2004:220::1:151 vrf tenant-c
BGP routing table information for VRF tenant-c
Router identifier 100.52.7.254, local AS number 65000
BGP routing table entry for 2004:220::1:151/128
  Paths: 2 available
  65002 65004
  10.0.0.2 from 1.0.1.111 (1.0.1.111), imported EVPN route, RD 20.1.1.3:10501
  Origin IGP, metric -, localpref 100, weight 0, valid, external, best
Extended Community: Route-Target-AS:1:10501 Route-Target-AS:4001:4001 TunnelEncap:tunnelTypeVXLAN 
EvpnRouterMac:28:99:3a:be:53:42
    Remote VNI: 4003
    65000 65002 65004
 2005:951:1:1::1:2 from 2005:951:1:1::1:2 (100.52.7.254)
 Origin IGP, metric -, localpref 100, weight 0, valid, external
 Not best: As path length

The following displays the route for a specific IPv6 prefix in a VRF.

switch# show ipv6 route vrf tenant-c 2004:220::1:151
VRF: tenant-c
Routing entry for 2004:220::1:151
Codes: C - connected, S - static, K - kernel, O3 - OSPFv3, B - BGP, R - RIP, A B - BGP Aggregate, 
I L1 - IS-IS level 1, I L2 - IS-IS level 2, DH - DHCP, NG - Nexthop Group Static Route, M - Martian, 
DP - Dynamic Policy Route, L - VRF Leaked

B      2004:220::1:151/128 [200/0]
         via VTEP 10.0.0.2 VNI 4003 router-mac 28:99:3a:be:53:42

The following displays the VXLAN SW counters for IPv6 Neighbor Discovery Packets.

switch# show VXLAN counters software | egrep ‘ND|neighbor’
ND NS pkts skipped HER as target Ip matched SVI IP   :  0
ND NS proxy errors during transmit                   :  0
ND NS proxy neighbor remote binding misses           :  0
ND NS proxy neighbor cache misses                    :  0
ND NS proxy denied due to ACL                        :  0
ND NS proxy not applied as neighbor entry is dynamic :  0
ND NS proxy not applied as target link is local      :  0
ND NS proxy not applied as target IP is local        :  0
ND NS proxy not applied as sender link not in fdb    :  0
ND NS proxy not applied as pkt is invalid            :  0
ND NS proxy DAD frames suppressed                    :  0
ND NS proxy neighbor advt sent                       :  0
ND NS pkts from unspecified source                   :  9
ND NS pkts total suppressed                          :  0
ND NS pkts total received                            :  9
ND NA pkts total suppressed                          :  0
ND NA pkts total received                            :  0
ND NA pkts invalid                                   :  0
ND NA pkts not suppressed as source is SVI           :  0
ND NA pkts suppressed as source is SVI               :  0
ND RS pkts total suppressed                          :  0
total dynamic neighbor cache entries added in error  :  0

The following displays the VXLAN VARP packets for IPv6 ipv6 address virtual configurations.

switch# show VXLAN counters varp | grep 'neighbor'
neighbor advertisements received                     :  0
neighbor advertisements received in error            :  0
neighbor advertisements not headend replicated       :  0
neighbor sync msgs sent to mlag-peer                 :  0
neighbor cache installed                             :  0
neighbor cache install err                           :  0
neighbor cache install conflicts                     :  0
neighbor sync msgs received from mlag-peer           :  0
neighbor cache synced install err                    :  0
neighbor cache synced install conflicts              :  0

IP VPNs Sample Configuration

Here, we examine BGP EVPN layer 3 VPN over an LDP, ISIS-SR, and BGP-SR transport LSPs. This highlights the separation between the transport and the VPN overlay service.

The following figures illustrate the sample VPN Physical Topology.

Figure 14. IPv4 VPN Physical Topology

Figure 15. IPv6 VPN Physical Topology

IP VPN over ISIS-SR

The following figure illustrates an overview of the combined control and data planes.

Figure 16. IPv4 VPN and IPv6 VPN Over ISIS-SR MPLS

The next two figures illustrate the forwarding path and control plane for both IP traffic over ISIS MPLS segment routing.

Figure 17. IPv4 VPN Forwarding Over ISIS-SR MPLS

Figure 18. IPv6 VPN Forwarding Over ISIS-SR MPLS

View IPv4 and IPv6 Routes in the VRF

North Edge and South Edge routers have an eBGP peering session out to the CE; and learning routes from CE and remote PE.

  • The show ip route vrf tenant-d command displays IPv4 Routes in the VRF of North Edge.

    north-edge# show ip route vrf tenant-d
    
    VRF: tenant-d
    Codes: C - connected, S - static, K - kernel,
           O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
           E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
           N2 - OSPF NSSA external type2, B I - iBGP, B E - eBGP,
           R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
           O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
           NG - Nexthop Group Static Route, V - VXLAN Control Service,
           DH - DHCP client installed default route, M - Martian,
           DP - Dynamic Policy Route
    
    Gateway of last resort is not set
    
     B I    10.255.255.0/30 [200/0] via 6.6.6.6/32, IS-IS SR tunnel index 6, label 967920
                                       via 192.168.58.12, Ethernet1/1, label 408006
     C      10.255.255.4/30 is directly connected, Ethernet6/1.120
     B E    201.0.0.0/24 [200/0] via 10.255.255.6, Ethernet6/1.120
     B I    206.0.0.0/24 [200/0] via 6.6.6.6/32, IS-IS SR tunnel index 6, label 967920
                                    via 192.168.58.12, Ethernet1/1, label 408006

  • The show ip route vrf tenant-d command displays IPv4 Routes in the VRF of South Edge.

    south-edge# show ip route vrf tenant-d
    
    VRF: tenant-d
    Codes: C - connected, S - static, K - kernel,
           O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
           E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
           N2 - OSPF NSSA external type2, B I - iBGP, B E - eBGP,
           R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
           O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
           NG - Nexthop Group Static Route, V - VXLAN Control Service,
           DH - DHCP client installed default route, M - Martian,
           DP - Dynamic Policy Route
    
    Gateway of last resort is not set
    
     C      10.255.255.0/30 is directly connected, Ethernet6/1.620
     B I    10.255.255.4/30 [200/0] via 1.1.1.111/32, IS-IS SR tunnel index 5, label 951536
                                       via 192.168.68.11, Ethernet2/1, label 408001
     B I    201.0.0.0/24 [200/0] via 1.1.1.111/32, IS-IS SR tunnel index 5, label 951536
                                    via 192.168.68.11, Ethernet2/1, label 408001
     B E    206.0.0.0/24 [200/0] via 10.255.255.2, Ethernet6/1.620

  • The show ipv6 route vrf tenant-d command displays IPv6 Routes in the VRF of North Edge.

    north-edge# show ipv6 route vrf tenant-d
    VRF: tenant-d
    Displaying 4 of 7 IPv6 routing table entries
    Codes: C - connected, S - static, K - kernel, O3 - OSPFv3, B - BGP, R - RIP, A B - BGP Aggregate, I L1 - 
    IS-IS level 1, I L2 - IS-IS level 2, DH - DHCP, NG - Nexthop Group Static Route, M - Martian, DP - Dynamic 
    Policy Route
    
     B    2010::/126 [200/0]
           via 6.6.6.6/32, IS-IS SR tunnel index 6, label 965242
              via 192.168.58.12, Ethernet1/1, label 408006
     C    2010::4/126 [0/0]
           via Ethernet6/1.120, directly connected
     B    2201::/64 [200/0]
           via 2010::6, Ethernet6/1.120
     B    2206::/64 [200/0]
           via 6.6.6.6/32, IS-IS SR tunnel index 6, label 965242
              via 192.168.58.12, Ethernet1/1, label 408006

  • The show ipv6 route vrf tenant-d command displays IPv6 Routes in the VRF of South Edge.

    south-edge# show ipv6 route vrf tenant-d
     
    VRF: tenant-d
    Displaying 4 of 7 IPv6 routing table entries
    Codes: C - connected, S - static, K - kernel, O3 - OSPFv3, B - BGP, R - RIP, A B - BGP Aggregate, I L1 - 
    IS-IS level 1, I L2 - IS-IS level 2, DH - DHCP, NG - Nexthop Group Static Route, M - Martian, DP - Dynamic 
    Policy Route
    
     C    2010::/126 [0/0]
           via Ethernet6/1.620, directly connected
     B    2010::4/126 [200/0]
           via 1.1.1.111/32, IS-IS SR tunnel index 5, label 948858
              via 192.168.68.11, Ethernet2/1, label 408001
     B    2201::/64 [200/0]
           via 1.1.1.111/32, IS-IS SR tunnel index 5, label 948858
              via 192.168.68.11, Ethernet2/1, label 408001
     B    2206::/64 [200/0]
           via 2010::2, Ethernet6/1.620

Activating IP VPN

In all scenarios, the IP VPN must be activated under BGP and neighbors configured to exchange the IP VPN NLRIs.The tenant’s VRF (tenant-d) is associated with a dynamically assigned label by BGP.

North Edge

service routing protocols model multi-agent

router bgp 64512
   router-id 1.1.1.111
   maximum-paths 128 ecmp 128
   neighbor 2.2.2.222 remote-as 64512
   neighbor 2.2.2.222 update-source Loopback0
   neighbor 2.2.2.222 bfd
   neighbor 2.2.2.222 send-community extended
   neighbor 2.2.2.222 maximum-routes 12000
   !
   address-family vpn-ipv4
      neighbor 2.2.2.222 activate
      neighbor default encapsulation mpls next-hop-self source-interface Loopback0
   !
   address-family vpn-ipv6
      neighbor 2.2.2.222 activate
      neighbor default encapsulation mpls next-hop-self source-interface Loopback0
   !

South Edge

service routing protocols model multi-agent

router bgp 64512
   router-id 6.6.6.6
   maximum-paths 128 ecmp 128
   neighbor 2.2.2.222 remote-as 64512
   neighbor 2.2.2.222 update-source Loopback0
   neighbor 2.2.2.222 bfd
   neighbor 2.2.2.222 send-community extended
   neighbor 2.2.2.222 maximum-routes 12000
   !
   address-family vpn-ipv4
      neighbor 2.2.2.222 activate
      neighbor default encapsulation mpls next-hop-self source-interface Loopback0
   !
   address-family vpn-ipv6
      neighbor 2.2.2.222 activate
      neighbor default encapsulation mpls next-hop-self source-interface Loopback0
   !

The configuration above provides the following:

  • It enables the multi-agent routing protocol model, which is required for BGP VPN support.
  • It sets the local autonomous system number to 64512 and configured the route-reflector for both IPv4 VPN and IPv6 VPN capabilities.
  • It sets the IP VPN encapsulation type to MPLS (default).
  • It specifies that Loopback0 will be used as the next-hop for all advertised VPN routes. The underlay configuration must provide MPLS LSPs from remote PEs to this loopback interface address.

Layer 3 Overlay Configuration

Distribution of Layer 3 routes over BGP is enabled by configuring one or more IP VRFs under the router bgp configuration mode. Additionally, either IPv4 or IPv6 routing must be enabled in the VRF.

  • Configure IP VRF in the North Edge router.

    vrf instance tenant-d
    ip routing vrf tenant-d
    ipv6 unicast-routing vrf tenant-d
    !
    router bgp 64512
        vrf tenant-d
          rd 1.1.1.1:64514
          route-target import vpn-ipv4 64512:4364
          route-target import vpn-ipv6 64512:4364
          route-target export vpn-ipv4 64512:4364
          route-target export vpn-ipv6 64512:4364
          neighbor 10.255.255.6 remote-as 65011
          neighbor 10.255.255.6 maximum-routes 12000
          neighbor 2010::6 remote-as 65011
          neighbor 2010::6 maximum-routes 12000
          !
          address-family ipv6
             neighbor 2010::6 activate
          redistribute connected
          !

  • Configure IP VRF in the South Edge router.

    vrf instance tenant-d
    ip routing vrf tenant-d
    ipv6 unicast-routing vrf tenant-d
    !
    router bgp 64512
       vrf tenant-d
          rd 6.6.6.6:64514
          route-target import vpn-ipv4 64512:4364
          route-target import vpn-ipv6 64512:4364
          route-target export vpn-ipv4 64512:4364
          route-target export vpn-ipv6 64512:4364
          neighbor 10.255.255.2 remote-as 65010
          neighbor 10.255.255.2 maximum-routes 12000
          neighbor 2010::2 remote-as 65010
          neighbor 2010::2 maximum-routes 12000
          !
          address-family ipv6
             neighbor 2010::2 activate
           redistribute connected
          !

These IP VRF configurations provide the following functionalities:
  • It defines overlay VRFs (tenant-d) on the PE and enables IP unicast routing.
  • The VRF is assigned a unique Route-Distinguisher (RD). The RD allows the PE to advertise VPN routes for the same IP prefix that have been exported by different VRFs. The NLRI RouteKey of a route exported from the VRFs IPv4 table into VPN consists of both the RD and the original IP prefix.
  • The Route-Target (RT) extended communities for the VRF. The RTs are associated with all routes exported from the VRF. Received VPN routes carrying at least one RT matching the VRFs configuration are imported into the VRF.

Verifying IP VPNs over ISIS-SR

  • The show bgp vpn-ipv4 summary command displays the status of the VPN IP peers in the North Edge router with the BGP VPN enabled.

    north-edge# show bgp vpn-ipv4 summary
    BGP summary information for VRF default
    Router identifier 1.1.1.111, local AS number 64512
    Neighbor Status Codes: m - Under maintenance
      Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  
    PfxRcd PfxAcc
      2.2.2.222        4  64512            172        45    0    0 00:17:16 Estab  2      2
    north-edge#  show bgp vpn-ipv6 summary
    BGP summary information for VRF default
    Router identifier 1.1.1.111, local AS number 64512
    Neighbor Status Codes: m - Under maintenance
      Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  
    PfxRcd PfxAcc
      2.2.2.222        4  64512            172        45    0    0 00:17:20 Estab  2      2

  • The show bgp vpn-ipv4 command displays routes sent and received through IP VPN.

    north-edge# show bgp vpn-ipv4
    BGP routing table information for VRF default
    Router identifier 1.1.1.111, local AS number 64512
    Route status codes: s - suppressed, * - valid, > - active, # - not installed, E - ECMP head, e - ECMP
                        S - Stale, c - Contributing to ECMP, b - backup
                        % - Pending BGP convergence
    Origin codes: i - IGP, e - EGP, ? - incomplete
    AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
    
             Network             Next Hop         Metric  LocPref Weight Path
     * >     RD: 6.6.6.6:64514 IPv4 prefix 10.255.255.0/30
                                 6.6.6.6          -       100     0      65010 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     RD: 1.1.1.1:64514 IPv4 prefix 10.255.255.4/30
                                 -                -       100     0      65011 i
     * >     RD: 1.1.1.1:64514 IPv4 prefix 201.0.0.0/24
                                 -                -       100     0      65011 i
     * >     RD: 6.6.6.6:64514 IPv4 prefix 206.0.0.0/24
                                 6.6.6.6          -       100     0      65010 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
    
    north-edge# show bgp vpn-ipv6
    BGP routing table information for VRF default
    Router identifier 1.1.1.111, local AS number 64512
    Route status codes: s - suppressed, * - valid, > - active, # - not installed, E - ECMP head, e - ECMP
                        S - Stale, c - Contributing to ECMP, b - backup
                        % - Pending BGP convergence
    Origin codes: i - IGP, e - EGP, ? - incomplete
    AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
    
             Network             Next Hop         Metric  LocPref Weight Path
     * >     RD: 6.6.6.6:64514 IPv6 prefix 2010::/126
                                 6.6.6.6          -       100     0      65010 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     RD: 1.1.1.1:64514 IPv6 prefix 2010::4/126
                                 -                -       100     0      65011 i
     * >     RD: 1.1.1.1:64514 IPv6 prefix 2201::/64
                                 -                -       100     0      65011 i
     * >     RD: 6.6.6.6:64514 IPv6 prefix 2206::/64
                                 6.6.6.6          -       100     0      65010 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222

    Note: Each entry in the table represents a BGP path. The path specific information includes the Route-Distinguisher and the IP prefix. Paths are either received from VPN peers or exported from local VRFs.

  • The show bgp vpn-ipv4 206.0.0.0/24 detail and show bgp vpn-ipv6 2206::/64 detail commands display detailed view of the IP prefix route for 206.0.0.0/24 and 2206.::/64 of the North Edge router.

    north-edge# show bgp vpn-ipv4 206.0.0.0/24 detail
    BGP routing table information for VRF default
    Router identifier 1.1.1.111, local AS number 64512
    BGP routing table entry for IPv4 prefix 206.0.0.0/24, Route Distinguisher: 6.6.6.6:64514
     Paths: 1 available
      65010
        6.6.6.6 from 2.2.2.222 (2.2.2.222)
          Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
          Extended Community: Route-Target-AS:64512:4364
          MPLS label: 967920
    
    north-edge# show bgp vpn-ipv6 2206::/64 detail
    BGP routing table information for VRF default
    Router identifier 1.1.1.111, local AS number 64512
    BGP routing table entry for IPv6 prefix 2206::/64, Route Distinguisher: 6.6.6.6:64514
     Paths: 1 available
      65010
        6.6.6.6 from 2.2.2.222 (2.2.2.222)
          Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
          Extended Community: Route-Target-AS:64512:4364
          MPLS label: 965242  

    Note: The output includes the RD and IP prefixes identifying the route. As seen in the output, the IPv4 VPN route is received from 2.2.2.222 because it is set-up to be a route-reflector, but the next hop is 6.6.6.6. Both are advertised with tenant VPN label 967920 and 965242 and an RT.

  • The show ip bgp vrf tenant-d command displays the BGP table for the VRF containing the imported EVPN routes.

    north-edge# show ip bgp vrf tenant-d
    BGP routing table information for VRF tenant-d
    Router identifier 1.1.1.1, local AS number 64512
    Route status codes: s - suppressed, * - valid, > - active, # - not installed, E - ECMP head, e - ECMP
                        S - Stale, c - Contributing to ECMP, b - backup, L - labeled-unicast
                        % - Pending BGP convergence
    Origin codes: i - IGP, e - EGP, ? - incomplete
    AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
             Network             Next Hop         Metric  LocPref Weight Path
     * >Ec   10.255.255.0/30     6.6.6.6          -       100     0      65010 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     *  ec   10.255.255.0/30     6.6.6.6          -       100     0      65010 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     * >     10.255.255.4/30     10.255.255.6     -       100     0      65011 i
     * >     201.0.0.0/24        10.255.255.6     -       100     0      65011 i
     * >Ec   206.0.0.0/24        6.6.6.6          -       100     0      65010 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222
     *  ec   206.0.0.0/24        6.6.6.6          -       100     0      65010 i Or-ID: 6.6.6.6 C-LST: 2.2.2.222

    Note: Each entry in the table represent a BGP path that is either locally redistributed and received into the VRF or imported from the IPv4 VPN table. VPN routes are received from router 2.2.2.222 C-List (cluster list - basically identifying this route as from a route-reflector) with originating router being 6.6.6.6.

Finally, let us look at the routes in the VRF tenant-d.

VRF: tenant-d
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route

Gateway of last resort is not set

 B I    10.255.255.0/30 [200/0] via 6.6.6.6/32, IS-IS SR tunnel index 6, label 967920
                                   via 192.168.58.12, Ethernet1/1, label 408006
 C      10.255.255.4/30 is directly connected, Ethernet6/1.120
 B E    201.0.0.0/24 [200/0] via 10.255.255.6, Ethernet6/1.120
 B I    206.0.0.0/24 [200/0] via 6.6.6.6/32, IS-IS SR tunnel index 6, label 967920
                                via 192.168.58.12, Ethernet1/1, label 408006

Note: As displayed in the highlighted route above the label stack, the route is the transport label 408006 on top (this is the label to reach NH 6.6.6.6), with the tenant-a VPN label 967920 next in the stack, identifying the route as belonging to tenant-d.

A check of the Tunnel FIB confirms that 408006 is the ISIS-SR LSP.

north-edge# show mpls tunnel fib
! 'show mpls tunnel fib' has been deprecated. Please use 'show tunnel fib [options]' moving forward.
   Tunnel Type         Index       Endpoint           Nexthop             Interface          Labels        Forwarding
------------------- --------- ------------------ ------------------- ------------------ ---------------- 
   IS-IS SR IPv4       9           2.2.2.22/32        192.168.58.12       Ethernet1/1        [ 3 ]         None
   LDP                 4           2.2.2.200/32       192.168.58.12       Ethernet1/1        [ 3 ]         None
   IS-IS SR IPv4       2           2.2.2.222/32       192.168.58.12       Ethernet1/1        [ 3 ]         None
   IS-IS SR IPv4       4           3.3.3.3/32         192.168.58.12       Ethernet1/1        [ 408003 ]    None
   BGP LU              5           3.3.3.33/32        192.168.58.12       Ethernet1/1        [ 200033 ]    None
   LDP                 5           3.3.3.200/32       192.168.58.12       Ethernet1/1        [ 904099 ]    None
   IS-IS SR IPv4       8           4.4.4.4/32         192.168.58.12       Ethernet1/1        [ 408004 ]    None
   IS-IS SR IPv4       5           4.4.4.44/32        192.168.58.12       Ethernet1/1        [ 408044 ]    None
   LDP                 2           4.4.4.200/32       192.168.58.12       Ethernet1/1        [ 904098 ]    None
   IS-IS SR IPv4       3           5.5.5.5/32         192.168.58.12       Ethernet1/1        [ 408005 ]    Primary
   BGP LU              7           5.5.5.55/32        192.168.58.12       Ethernet1/1        [ 200055 ]    None
   LDP                 3           5.5.5.200/32       192.168.58.12       Ethernet1/1        [ 904100 ]    None
   IS-IS SR IPv4       6           6.6.6.6/32         192.168.58.12       Ethernet1/1        [ 408006 ]    Primary
   BGP LU              8           6.6.6.66/32        192.168.58.12       Ethernet1/1        [ 200066 ]    None
   LDP                 1           6.6.6.200/32       192.168.58.12       Ethernet1/1        [ 904097 ]    None
   IS-IS SR IPv4       1           23.1.1.11/32       192.168.1.154       Ethernet36/1       [ 3 ]         Primary
   IS-IS SR IPv4       7           23.1.1.33/32       192.168.1.174       Ethernet23/1       [ 3 ]         Primary

IP VPNs Over LDP

The following figures illustrate an overview of the combined control and data planes.

Figure 19. IPv4 VPN and IPv6 VPN Over LDP MPLS

Figure 20. IPv4 VPN Forwarding Over LDP MPLS

Figure 21. IPv6 VPN Forwarding Over LDP MPLS

To switch to using the MPLS LDP transport, we just need to change the next-hop we advertised for the VPN routes. As shown, the next hop needs to be set to loopback 200 for using the LDP LSP.

This is achieved by configuring the next-hop for the EVPN routes on both north and south edge routers.

router bgp 64512
   !
   address-family evpn
     neighbor default encapsulation mpls next-hop-self source-interface Loopback200

After this is configured, check the BGP updates and the routes in the VRF. The output again includes the RD and IP prefix identifying the route. Now the NH is set to 6.6.6.200 for tenant-d.

north-edge# show bgp vpn-ipv4 206.0.0.0/24 detail
BGP routing table information for VRF default
Router identifier 1.1.1.111, local AS number 64512
BGP routing table entry for IPv4 prefix 206.0.0.0/24, Route Distinguisher: 6.6.6.6:64514
 Paths: 1 available
  65010
    6.6.6.200 from 2.2.2.222 (2.2.2.222)
      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
      Extended Community: Route-Target-AS:64512:4364
      MPLS label: 967920
north-edge#

north-edge# show bgp vpn-ipv6 2206::/64 detail
BGP routing table information for VRF default
Router identifier 1.1.1.111, local AS number 64512
BGP routing table entry for IPv6 prefix 2206::/64, Route Distinguisher: 6.6.6.6:64514
 Paths: 1 available
  65010
    6.6.6.200 from 2.2.2.222 (2.2.2.222)
      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
      Extended Community: Route-Target-AS:64512:4364
      MPLS label: 965242
north-edge#

Note: The VPN label has not changed from the ISIS-SR case above (967920 and 965242), reinforcing the fact that the BGP VPN label is orthogonal to the transport label.

north-edge# show ip route vrf tenant-d

VRF: tenant-d
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route

Gateway of last resort is not set

 B I    10.255.255.0/30 [200/0] via 6.6.6.200/32, LDP tunnel index 1, label 967920
                                   via 192.168.58.12, Ethernet1/1, label 904097
 C      10.255.255.4/30 is directly connected, Ethernet6/1.120
 B E    201.0.0.0/24 [200/0] via 10.255.255.6, Ethernet6/1.120
 B I    206.0.0.0/24 [200/0] via 6.6.6.200/32, LDP tunnel index 1, label 967920
                                via 192.168.58.12, Ethernet1/1, label 904097

north-edge(config-router-bgp)# show ipv6 route vrf tenant-d

VRF: tenant-d
Displaying 4 of 7 IPv6 routing table entries
Codes: C - connected, S - static, K - kernel, O3 - OSPFv3, B - BGP, R - RIP, A B - BGP Aggregate, 
I L1 - IS-IS level 1, I L2 - IS-IS level 2, DH - DHCP, NG - Nexthop Group Static Route, 
M - Martian, DP - Dynamic Policy Route

 B    2010::/126 [200/0]
       via 6.6.6.6/32, IS-IS SR tunnel index 6, label 965242
          via 192.168.58.12, Ethernet1/1, label 408006
 C    2010::4/126 [0/0]
       via Ethernet6/1.120, directly connected
 B    2201::/64 [200/0]
       via 2010::6, Ethernet6/1.120
 B    2206::/64 [200/0]
       via 6.6.6.6/32, IS-IS SR tunnel index 6, label 965242
          via 192.168.58.12, Ethernet1/1, label 408006

Note: As seen from the highlighted route above the label stack, the route are the transport label 904097 on top (this is the label path to reach NH 6.6.6.200), with the tenant-d VPN label 967920 next in the stack, and identifying the route as belonging to tenant-a.

A capture of the dataplane on North-Edge matching on the LDP transport label confirms the encapsulated traffic on the wire. 904097:976920:[Source IP Address][Destination IP Address].

IP VPNs Over BGP-SR

The following figures illustrate an overview of the combined control and data planes.

Figure 22. IPv4 VPN and IPv6 VPN Over BGP-SR MPLS

Figure 23. IPv4 VPN Forwarding Over BGP-SR MPLS

Figure 24. IPv6 VPN Forwarding Over BGP-SR MPLS

To switch to using the MPLS BGP-SR transport, we just need to change the next-hop we advertised for the VPN routes. As shown, the next hop needs to be set to loopback 1 for using the BGP-SR LSP.

This is achieved by configuring the next-hop for EVPN routes.

router bgp 64512
   !
   address-family evpn
     neighbor default encapsulation mpls next-hop-self source-interface Loopback1

Once this is configured, we can check the BGP updates and the routes in the VRF. The output again includes the RD and IP prefix identifying the route. As seen in the output, we now have the NH set to 6.6.6.66 for tenant-d.

north-edge# show bgp vpn-ipv4 206.0.0.0/24 detail
BGP routing table information for VRF default
Router identifier 1.1.1.111, local AS number 64512
BGP routing table entry for IPv4 prefix 206.0.0.0/24, Route Distinguisher: 6.6.6.6:64514
 Paths: 1 available
  65010
    6.6.6.66 from 2.2.2.222 (2.2.2.222)
      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
      Extended Community: Route-Target-AS:64512:4364
      MPLS label: 967920
north-edge#
north-edge#show bgp vpn-ipv6 2206::/64 detail
BGP routing table information for VRF default
Router identifier 1.1.1.111, local AS number 64512
BGP routing table entry for IPv6 prefix 2206::/64, Route Distinguisher: 6.6.6.6:64514
 Paths: 1 available
  65010
    6.6.6.66 from 2.2.2.222 (2.2.2.222)
      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
      Extended Community: Route-Target-AS:64512:4364
      MPLS label: 965242
north-edge#

Note: The VPN label has not changed from the ISIS-SR case above (967920 and 965242), reinforcing the fact that the BGP VPN label is orthogonal to the transport label.

The output again includes the RD and IP prefix identifying the route. As seen in the output, we now have the NH set to 6.6.6.66 for tenant-d.

north-edge# show bgp vpn-ipv4 206.0.0.0/24 detail
BGP routing table information for VRF default
Router identifier 1.1.1.111, local AS number 64512
BGP routing table entry for IPv4 prefix 206.0.0.0/24, Route Distinguisher: 6.6.6.6:64514
 Paths: 1 available
  65010
    6.6.6.66 from 2.2.2.222 (2.2.2.222)
      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
      Extended Community: Route-Target-AS:64512:4364
      MPLS label: 967920
north-edge#
north-edge# show bgp vpn-ipv6 2206::/64 detail
BGP routing table information for VRF default
Router identifier 1.1.1.111, local AS number 64512
BGP routing table entry for IPv6 prefix 2206::/64, Route Distinguisher: 6.6.6.6:64514
 Paths: 1 available
  65010
    6.6.6.66 from 2.2.2.222 (2.2.2.222)
      Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
      Extended Community: Route-Target-AS:64512:4364
      MPLS label: 965242
north-edge#

Note: The VPN label has not changed from the ISIS-SR case above (967920 and 965242), reinforcing that the BGP VPN label is orthogonal to the transport label.

As displayed in the highlighted route above the label stack, the route are the transport label 200066 on top (this is the label path to reach NH 6.6.6.66), with the tenant-d VPN label 967920 next in the stack, and identifying the route as belonging to tenant-a.

north-edge# show ip route vrf tenant-d

VRF: tenant-d
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route

Gateway of last resort is not set

 B I    10.255.255.0/30 [200/0] via 6.6.6.66/32, BGP LU tunnel index 8, label 967920
                                   via 192.168.58.12, Ethernet1/1, label 200066
                                   via 192.168.59.12, Ethernet2/1, label 200066
 C      10.255.255.4/30 is directly connected, Ethernet6/1.120
 B E    201.0.0.0/24 [200/0] via 10.255.255.6, Ethernet6/1.120
 B I    206.0.0.0/24 [200/0] via 6.6.6.66/32, BGP LU tunnel index 8, label 967920
                                via 192.168.58.12, Ethernet1/1, label 200066
                                via 192.168.59.12, Ethernet2/1, label 200066

north-edge(config-router-bgp)# show ipv6 route vrf tenant-d

VRF: tenant-d
Displaying 4 of 7 IPv6 routing table entries
Codes: C - connected, S - static, K - kernel, O3 - OSPFv3, B - BGP, R - RIP, A B - BGP Aggregate, I L1 - 
IS-IS level 1, I L2 - IS-IS level 2, DH - DHCP, NG - Nexthop Group Static Route, M - Martian, DP - Dynamic 
Policy Route

 B    2010::/126 [200/0]
       via 6.6.6.66/32, BGP LU tunnel index 8, label 965242
          via 192.168.58.12, Ethernet1/1, label 200066
          via 192.168.59.12, Ethernet2/1, label 200066
 C    2010::4/126 [0/0]
       via Ethernet6/1.120, directly connected
 B    2201::/64 [200/0]
       via 2010::6, Ethernet6/1.120
 B    2206::/64 [200/0]
       via 6.6.6.66/32, BGP LU tunnel index 8, label 965242
          via 192.168.58.12, Ethernet1/1, label 200066
          via 192.168.59.12, Ethernet2/1, label 200066

A capture of the data-plane on North-Edge matching on the BGP-SR transport label confirms the encapsulated traffic on the wire. 200066:976920:[Source IP Address][Destination IP Address].

monitor session 1 source Ethernet1/1 tx
monitor session 1 destination Cpu

north-edge(config-router-bgp)# bash tcpdump -nei mirror0 -q -c 10 mpls 200066
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on mirror0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:37:15.074916 28:99:3a:4d:3e:f1 > 28:99:3a:4d:3a:f3, MPLS unicast, length 122: MPLS (label 200066, exp 0, 
ttl 63) (label 967920, exp 0, [S], ttl 63) 10.255.255.6 > 206.0.0.1: ICMP echo request, id 22573, seq 1, 
length 80

16:37:15.075088 28:99:3a:4d:3e:f1 > 28:99:3a:4d:3a:f3, MPLS unicast, length 122: MPLS (label 200066, exp 0, 
ttl 63) (label 967920, exp 0, [S], ttl 63) 10.255.255.6 > 206.0.0.1: ICMP echo request, id 22573, seq 2, length 80