VXLAN Description

These sections describe VXLAN architecture, the data objects that comprise a VXLAN network, and process of bridging packets through a VXLAN network.

VXLAN Architecture

The VXLAN architecture extends a Layer 2 network by connecting VLANs from multiple hosts through UDP tunnels called VXLAN segments. VXLAN segments are identified by a 24-bit Virtual Network Identifier (VNI). Within a host, each VLAN whose network is extended to other hosts is associated with a VNI. An extended Layer 2 network comprises the devices attached to VLANs from all hosts that are on VLANs that are associated with the same VNI.

The following figure displays the data objects that comprise a VXLAN implementation on a local host.

Figure 1. VXLAN Architecture
  • VXLAN Tunnel End Point (VTEP): a host with at least one VXLAN Tunnel Interface (VTI).
  • VXLAN Tunnel Interface (VTI): a switchport linked to a UDP socket that is shared with VLANs on various hosts. Packets bridged from a VLAN to the VTI are sent out the UDP socket with a VXLAN header. Packets arriving on the VTI through the UDP socket are demuxed to VLANs for bridging.
  • Virtual Network Identifier (VNI): a 24-bit number that distinguishes between the VLANs carried on a VTI. It facilitates the multiplexing of several VLANs over a single VTI.

VNIs can be expressed in digital or dotted decimal formats. VNI values range from 1 to 16777215 or from 0.0.1 to 255.255.255.

The network in the figure above has the following assignments:
  • VTEP IP address of 10.10.1.1.
  • UDP port of 4789.
  • One VTI that supports three VXLAN segments (UDP tunnels): VNI 200, VNI 2000, and VNI 20000
  • Five VLANs, of which three VLANs can communicate with remote devices over Layer 2.

VXLAN Gateway

A VXLAN gateway is a service that exchanges VXLAN data and packets with devices connected to different network segments. VXLAN traffic must pass through a VXLAN gateway to access services on physical devices in a distant network.

A VXLAN gateway requires the following information:
  • An IP address that is designated as the VXLAN interface source.
  • VLAN to VNI mapping.
  • VTEP list for each VNI.
  • A method for handling broadcast, unknown unicast, and multicast (BUM) packets.

Arista switches manually perform VXLAN gateway services. The switch connects to VXLAN gateways that serve other network segments. MAC address learning is performed in hardware from inbound VXLAN packets.

VXLAN Processes

When a packet enters a VLAN from a member (ingress) port, the VLAN learns the source address by adding an entry to the MAC address table that associates the source to the ingress-port. The VLAN then searches the table for destination address. If the MAC address table lists the address, the packet is sent out the corresponding port. If the MAC address table does not lists the address, the packet is flooded to all ports except the ingress port.

VXLANs extend VLANs through the addition of a VXLAN address table that correlates remote MAC addresses to their port and resident host IP address. Packets that are destined to a remote device are sent to the VXLAN tunnel interface (VTI), which is the switchport that is linked to the UDP socket. The packet is encapsulated with a VXLAN header which includes the VNI associated with the VLAN and the IP mapping of the destination host. The packet is sent through a UDP socket to the destination VTEP IP. The VTI on the remote host extracts the original packet and bridges it to the VLAN associated with the VNI on the remote host.

UDP port 4789 is recognized as the VXLAN socket and listed as the destination port on the UDP packets. The UDP source port field is filled with a hash of the inner header to facilitate load balancing.

Figure 2. VXLAN Implementation
The figure above displays a configuration that includes three VTEPs. The VXLAN defines three inter-host L2 networks. The VLANs that comprise the networks include:
  • VNI 200: VTEP 10.20.2.2: VLAN 1200 and VTEP 10.30.3.3: VLAN 200
  • VNI 2000: VTEP 10.10.1.1: VLAN 300, VTEP 10.20.2.2: VLAN 1400, and VTEP 10.30.3.3: VLAN 300
  • VNI 20000: VTEP 10.10.1.1: VLAN 200, and VTEP 10.20.2.2: VLAN 1600

VXLAN Routing

VXLAN routing is enabled by creating a VLAN interface on the VXLAN-enabled VLAN and assigning an IP address to the VLAN interface. The IP address serves as VXLAN gateway for devices that are accessible from the VXLAN-enabled VLAN.

Multicast and Broadcast over VXLAN

These sections describe multicast and broadcast over VXLANs. Multicast packet flooding describes broadcast and multicast transmission by associating a multicast group to a VTI through a configuration command.

Head-end Replication (HER) optimizes flooding of inter VTEP broadcast, unknown unicast and broadcast (BUM) traffic by using hardware and flood lists to perform replication on the supported platform.

Multicast Packet Flooding

Multicast packet flooding is supported with VXLAN bridging without MLAG. A VTI is associated with a multicast group through a configuration command.

VXLAN and Broadcast

When a VLAN receives or sends a broadcast packet the VTI is treated as a bridging domain L2 interface. The packet is sent from this interface on the multicast group associated with the VTI. The VTIs on remote VTEPs that receive this packet extract the original packet, which is then handled by the VLAN associated with the packet’s VNI. The VLAN floods the packet, excluding the VTI. When the broadcast results in a response, the resulting packet can be unicast back to the originating VTEP because the VXLAN address table obtained the host MAC to VTEP association from the broadcast packet.

VXLAN and Multicast

A VTI is treated as an L2 interface in the VLAN for handling multicast traffic, which is mapped from the VLAN to the multicast group associated with the VTI. All VTEPs join the configured multicast group for inter-VTEP communication within a VXLAN segment; this multicast group is independent of any other multicast groups that the hosts in the VLAN join.

The IP address space for the inter-host VXLAN communication may be sourced from a different VRF than the address space of the hosts in the VLAN. The multicast group for inter-VTEP transmissions must not be used for other purposes by any device in the VXLAN segment space.

Head-end Replication

Head-end replication uses a flood list to support broadcast, unknown unicast, and multicast (BUM) traffic over VXLAN. The flood list specifies a list of remote VTEPs. The switch replicates BUM data locally for bridging across the remote VTEPs specified by the flood list. This data flooding facilitates remote MAC address learning by forwarding data with unknown MAC addresses.

Head-end replication is required for VXLAN routing and to support VXLANs over MLAG.

VXLAN and MLAG

VXLAN over MLAG provides redundancy in hardware VTEPs. VTI configuration must be identical on each MLAG peer for them to act as a single VTEP. This also prevents the remote MAC from flapping between the remote VTEPs by ensuring that the rest of the network sees a host that is connected to the MLAG interface as residing behind a single VTEP.

Differences between VXLAN bridging and routing implementations over MLAG are applicable for the DCS-7050X series platform.
  • VXLAN routing recirculates a packet twice, with the first iteration performing the routing action involving an L2 header rewrite, and the second recirculation performing VXLAN encap and decap operations. Recirculation is achieved by MAC loopback on dedicated loopback interfaces.
  • The configuration for VXLAN routing on an MLAG VTEP includes separate Recirc-Channel configuration on both peers. The virtual IP, virtual MAC, and virtual VARP VTEP IP addresses are identical on both peers.
The following VTI elements must be configured identically on both MLAG peers:
  • VLAN-VNI mappings
  • VTEP IP address of the source loopback interface
  • Flood VTEP list used for head-end replication

If OSPF is also in use, configure the OSPF router ID manually to prevent the switch from using the common VTEP IP address as the router ID.

The following rules are observed by MLAG switches so that they behave as a single VXLAN VTEP:
  • Only the MLAG peer that receives a packet performs VXLAN encapsulation on it.
  • Packets are not VXLAN encapsulated if they are received from the peer link.
  • If a packet is decapsulated and sent over the peer link, it should not be flooded to active MLAG interfaces.
  • If a packet is sent over the peer link to the CPU, it is not head-end replicated to other remote VTEPs.
  • If a packet’s destination is the VTEP IP address, it is terminated by the MLAG peer that receives it.
Examples
  • These commands complete the configuration required for a VXLAN routing deployment.
    switch(config)# interface VXLAN1
switch(config-if-Vx1)# vxlan source-interface Loopback0
switch(config-if-Vx1)# vxlan udp-port 4789
switch(config-if-Vx1)# vxlan vlan 2417 vni 8358534
switch(config-if-Vx1)# vxlan flood vtep 1.0.1.1 1.0.2.1
switch(config-if-Vx1)# interface Vlan2417
switch(config-if-Vl2417)# ip address 1.0.4.1/24
switch(config-if-Vl2417)# interface Loopback0
switch(config-if-Lo0)# ip address 1.0.1.1/32
switch(config-if-Lo0)# ip routing
switch(config)# interface Recirc-Channel627
switch(config-if-Re627)# switchport recirculation features vxlan
switch(config-if-Re627)# interface Ethernet 1
switch(config-if-Et1)# traffic-loopback source system device mac
switch(config-if-Et1)# channel-group recirculation 627
switch(config-if-Et1)# exit
switch(config)# interface Ethernet 2
switch(config-if-Et2)# traffic-loopback source system device mac
switch(config-if-Et2)# channel-group recirculation 627
switch(config-if-Et2)#
  • show running interface Loopback allows remote VTEP tunnels to be routed over L3 interfaces.
    switch# show running interface Loopback298
interface Loopback298
   ip address 1.0.1.1/32
switch#
switch# show running interface Ethernet54/1.4095
interface Ethernet54/1.4095
   mtu 9214
   encapsulation dot1q vlan 267
   ip address 1.0.4.1/24
switch#
switch# show running interface Port-Channel1.4095
interface Port-Channel1.4095
   mtu 9214
   encapsulation dot1q vlan 1043
   ip address 1.0.88.1/24
switch#
switch# show running interface vxlan1
interface Vxlan1
   vxlan source-interface Loopback298
   vxlan udp-port 4789
   vxlan vlan 2156 vni 15613244
   vxlan vlan 2393 vni 3610141
   vxlan vlan 2156 flood vtep 1.0.2.1 1.0.3.1
   vxlan vlan 2393 flood vtep 1.0.2.1 1.0.3.1
switch#
switch# show port-channel
Port Channel Port-Channel1:
  Active Ports: Ethernet51/1
switch#
switch# show ip route
VRF: default
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
     . . .
     . . .
 S        1.0.2.1/32 [1/0] via 1.0.4.2, Ethernet54/1.4095
 S        1.0.3.1/32 [1/0] via 1.0.88.2, Port-Channel1.4095

    There are 2 remote VTEPs configured 1.0.2.1, and 1.0.3.1. The remote VTEP 1.0.2.1 is reachable through Ethernet54/1.4095 and remote VTEP 1.0.3.1 is reachable through port-channel1.4095.

Configuring Unconnected Ethernet Interfaces for Recirculation

On systems where bandwidth is not fully used by the front panel ports, unused bandwidth is used for recirculation.

The following example is applicable to the DCS-7050X series platform.

Example

These commands expose unconnected Ethernet interfaces which are used for recirculation, in order to use them to replace or use along with front panel Ethernet interfaces.

switch(config)# service interface unconnected expose
switch(config)# interface UnconnectedEthernet 2
switch(config-if-Ue2)# traffic-loopback source system device mac
switch(config-if-Ue2)# channel-group recirculation 627

The following example enables display of the inactive interfaces using the show command.

Example

switch(config)# service interface inactive expose

Running a show command generates the following output:

switch(config)# switch(config)#show int et21/1-4 stat
Port   	Name   Status   	Vlan 	Duplex Speed  Type     	Flags Encapsulation
Et21/1        	connected	1    	full   100G   100GBASE-CR4
Et21/2        	inactive 	1    	full   25G    100GBASE-CR4
Et21/3        	inactive 	1    	full   25G    100GBASE-CR4
Et21/4        	inactive 	1    	full   25G    100GBASE-CR4

On previous releases, Ethernet 21/2, 21/4 do not exist and the output would be the following:

switch(config)# switch(config)#show int et21/1-4 stat
Port   	Name   Status   	Vlan 	Duplex Speed  Type     	Flags Encapsulation
Et21/1        	connected	1    	full   100G   100GBASE-CR4
Et21/3        	inactive 	1    	full   25G    100GBASE-CR4

VXLAN Bridging and Routing Support

Describes the support of VXLAN Bridging and Routing on the R3 series of DCS 7280, 7500, and 7800 Arista switches.

Differences with DCS-7500R2 Implementation

The following are notable differences with respect to implementation of VXLAN on the R3 Series of switches.
  • There is no need to configure the VXLAN-routing TCAM profile to enable VXLAN routing on the R3 Series switches. The command is still accepted for backward compatibility reasons.
  • CPU bound traffic after VXLAN decapsulation (such as routing protocol packets) use the same CoPP queues used by the non VXLAN decapsulated packets. This is an improvement over the R2 series behavior where the CPU bound traffic after VXLAN decapsulation took a different CoPP queue that was shared with other IP Unicast packets.

Limitations

There is no EVPN VXLAN Multicast (Type 6/7/8 NLRI) support.

Data Structures

VXLAN implementation requires two VXLAN tables and a MAC address table accommodation.

MAC Address Table VXLAN Support

MAC address table entries correlate MAC addresses with the port upon which packets arrive. In addition to Ethernet and port channels, the port column may specify a VTI for packets that arrive on a VLAN from a remote port through the VXLAN segment.

VTEP-MAC Address Table

VTEP-MAC address table entries correlate MAC address with the IP address of the VTEP from where packets bearing the MAC address arrive. The VTI uses this table to determine the destination address for packets that are sent to remote hosts.

VNI-VLAN Map

The VNI-VLAN map displays the one-to-one correspondence between the VNIs assigned on the switch and the VLANs to which they are assigned. Each VNI can be assigned to only one VLAN; each VLAN can be assigned a maximum of one VNI. Each VNI-VLAN assignment constitutes a VXLAN segment.