BGP/MPLS L3 VPN
- Maintaining privacy of each customer.
- Allowing for overlapping IP addresses among customers.
- Having constrained route distribution – such as, only those routers in the Service Provider’s network that need the customer routes, have them.
Achieve the above through the extensions to BGP as defined in RFC 4364 for IPv4 and RFC 4659 for IPv6, and the use of VPN Routing and Forwarding Tables (VRFs), Route Distinguishers (RDs), and Route Targets (RTs).
Operation
A Virtual Private Network, or VPN, is a set of geographically dispersed sites attached to the Service Provider’s (SP) backbone, with IP interconnectivity amongst the sites. Using this scenario, the customer can obtain “VPN service” from the SP. The SP will provide a VPN service to multiple customers, using this common backbone network infrastructure. The sample VPN topology diagram below illustrates three sites where a customer is being interconnected over the SP backbone network.
At each site, the Customer Edge router (CE) attaches to the Provider’s Edge router (PE). The CE can attach to more than one PE, and in these cases the CE is said to be “multi-homed”. The routers in the SP core network, those which do not attach to any CE, are referred to as “P” routers. The “P” routers do not need to know about the customer routes.
The CE attaches to PE in a VRF. The routes learned from the CE are programmed in the corresponding VRF and the PE then distributes the routes to other PEs as “VPN routes” using MP-BGP. This is done by using two new BGP address families, VPN-IPv4 (AFI=1, SAFI=128) and VPN-IPv6 (AFI=2, SAFI=128).
On the PEs, corresponding to the VPN, VRFs configure with the RD and one or more import and export RTs. The RD attaches to the customer route to create a VPN route. By picking a unique RD for each VRF and attaching it to the customer route, the VPN route is unique. This allows for customers with overlapping IP addresses to be managed by the Service Provider. This unique VPN route is advertised to other PEs along with the configured export RT and the VPN label. The PE router allocates a VPN label per VRF and address family. The PE router programs its Label FIB (LFIB) with this label information. When the PE router receives an incoming MPLS packet with the VPN label as the topmost label, it pops the label and does an IP lookup in the associated VRF. The PE router also maintains a mapping of import RTs to the corresponding VRFs. This mapping is used later when deciding into which VRFs a received VPN route should be imported into.
The PE learns customer routes from the CE through the PE-CE routing protocol, which could be Static routing, EBGP, OSPF, or ISIS. The PE will install those customer routes in the associated VRF, with the CE as the nexthop. The customer routes in the VRF are exported into the BGP VPN table as VPN routes, along with the VPN label and the configured export RTs as BGP path attributes. These VPN routes are then advertised to other PEs which have been activated for the VPN address-families (VPN-IPv4/VPN-IPv6). The PE then sets itself as the nexthop while advertising the VPN routes. The PE which receives those VPN routes strips out the RD and import them as IPv4 or IPv6 routes into the VRFs. The list of VRFs into which the route is imported is determined based on the mapping of import RTs to VRFs. These routes will be programmed into the FIB along with the VPN label and the remote PE as the nexthop. Once these routes are installed in the VRF, the CE connected to that VRF will learn those routes (based on the PE-CE routing protocol) and make it available it in the customer network that it is attached to. This way the geographically dispersed customer sites learn of each other’s networks and IP reachability is established between the them.
Forwarding
In the Service Provider’s core network there should be MPLS LSPs between the PEs. That is, the connection to the VPN next-hop should be over an MPLS tunnel. The MPLS LSPs in the core could be setup using RSVP-TE or LDP in conjunction with OSPF/ISIS or ISIS-SR.
When the PE receives an IP packet from the CE destined to the remote site, it does an IP lookup in the VRF to which the CE is connected. This lookup provides the VPN label to be used and the nexthop, which would be the remote-PE. The VPN label is imposed on the IP packet and the resulting MPLS packet is then tunneled through the MPLS LSP to the remote PE. As result of penultimate hop popping, the MPLS packet arrives at the remote PE with the VPN label as the topmost label. The label lookup results in the VPN label being popped and an IP lookup will be done in the associated VRF. Note that the PE would have programmed this label action when it allocated the label to start with. That IP lookup will result in the IP packet being forwarded out to the CE.
Configuration
Configuring BGP/MPLS L3 VPN involves enabling the SP core for MPLS and then configuring the PEs with the required BGP configuration.
It is assumed that the SP core network is enabled for MPLS. This involves configuring an IGP (OSPF/ISIS) followed by a label distribution protocol such as LDP, RSVP-TE or ISIS-SR. Typically, loopback interfaces are configured on all the PE and the P routers and the IGPs exchange reachability to those loopback interfaces. And then the MPLS Label Distribution Protocol will set up MPLS LSPs/tunnels between all those loopbacks.
Enabling MPLS and LDP on the PE involves the following steps:
LDP Hello Redundancy
- Devices must have a loopback interface configured to serve as the transport address interface, which must be routable through all interfaces on the device.
- Devices must be reachable via a redundant path through other devices on the network.
- Both devices must have Hello Redundancy configured or the targeted Hello messages will be ignored.
After a Hello adjacency is established using LDP Basic Discovery, devices with Hello Redundancy will start sending Targeted Hello messages to the Transport Address found in the received Link Hello message of Basic Discovery. The Targeted Hello adjacency can support the session established between peers even when all Link Hello adjacencies have timed out. The FEC label bindings between two peers with no Link Hello adjacency will not be active because the Interior Gateway Protocol will not use the other peer as the next hop. Maintaining the FEC label bindings and the session between the two peers can save significant time when the Link Hello adjacency is reestablished.
The neighbor hello-redundancy command configures Hello Redundancy on all platforms under the LDP configuration mode. If a Link Hello adjacency is restored within 600 seconds of being lost, the Target Hello adjacency and the session associated with it will be dropped. The timeout can be configured using the duration option of the command. An infinite value for the duration disables the timeout.
Configuration for LDP Hello Redundancy
The following example is applicable to all platforms.
These commands enable Targeted Hello redundancy with a duration of 300 seconds.
switch(config-mpls-ldp)# neighbor hello-redundancy
switch(config-mpls-ldp)# neighbor hello-redundancy duration 300Either of the following commands in the example below disable Hello Redundancy.
switch(config-mpls-ldp)# neighbor hello-redundancy none
switch(config-mpls-ldp)# default neighbor hello-redundancyThe following command shows the Targeted Hello adjacencies established.
switch(config)# show mpls ldp discovery detail
LDP MD5 Password Not Set
Local LDP Identifier: 2.2.2.2:0
Discovery Sources:
Interfaces:
Ethernet1 (ldp):
Hello interval: 5 sec; Source IP addr: 192.168.2.1
LDP ID: 3.3.3.3:0
Source IP addr: 192.168.2.2; Transport IP addr: 3.3.3.3
Hold time: 15 sec; Proposed local/peer: 15/15 sec; Expires in: 11.96 sec
Ethernet2 (ldp):
Hello interval: 5 sec; Source IP addr: 192.168.1.2
LDP ID: 1.1.1.1:0
Source IP addr: 192.168.1.1; Transport IP addr: 1.1.1.1
Hold time: 15 sec; Proposed local/peer: 15/15 sec; Expires in: 12.00 sec
Targeted Hellos:
Targeted neighbor 1.1.1.1:
Hello interval: 15 sec; Source IP addr: 2.2.2.2
LDP ID: 1.1.1.1:0
Source IP addr: 1.1.1.1; Transport IP addr: 1.1.1.1
Hold time: 45 sec; Proposed local/peer: 45/45 sec; Expires in: 40.78 sec
Target configuration source: Hello Redundancy
Targeted neighbor 3.3.3.3:
Hello interval: 15 sec; Source IP addr: 2.2.2.2
LDP ID: 3.3.3.3:0
Source IP addr: 3.3.3.3; Transport IP addr: 3.3.3.3
Hold time: 45 sec; Proposed local/peer: 45/45 sec; Expires in: 35.89 sec
Target configuration source: Hello RedundancyEnabling BGP to Exchange the Routing Tables with the Peer
- Activating the peer (the remote PE) under the address-family VPN-IPv4 and address-family VPN-IPv6 modes enables BGP to negotiate the MPLS L3 VPN address families.
- Specify the address for the VPN next-hop using the command, neighbor default encapsulation mpls next-hop-self source-interface Loopback0.
In the preceding configuration example, the system uses the address 11.0.0.1 from interface Loopback0 as the nexthop in the VPN route advertisements.
Configuring the VRF Information
First, the VRF must be configured in the global mode and IPv4 and IPv6 routing must be enabled in the VRF. After that, under the router bgp mode, we need to configure the VRF and provide the information related to RD and import and export RTs.
Configuring the Route Distinguisher
The Route Distinguisher (RD) is structured such that it can be easily configured and managed. It is configured by specifying two fields separated by a colon, as in administrative-subfield: assigned-number-subfield.
The administrative subfield could contain either an IP address, as shown in the example (the PEs loopback address), or the AS number. The assigned-number-subfield can be any number which is determined by the SP. The primary requirement of the RD is that it must be unique per VRF. It is possible to have overlapping address space between VRFs, the intent of the RD is to ensure that a VPN route can be uniquely identified as it is received by a remote PE. However, the identification of which VRF(s) the received VPN route should be imported into is handled by the import RTs configured on the remote PE.
Configuring Route Targets
- Received VPN routes with the import RT as path attributes will be imported into the VRF will have RTs has extended-community BGP path attributes. They will be imported into VRFs which import RT configuration for RTs in the received VPN route.
- And while advertising the routes from the VRF as VPN routes, the export RT
will be attached to the route as an extended-community BGP attribute.
- In the example, the connected and static routes in the VRF are redistributed into BGP. And these routes will be exported as VPN routes.
- It is also possible to have a BGP session with the CE. In that case, routes received over that session are exported as VPN routes. And imported routes are advertised to the CE.
Configuring Import / Export Route-maps
The routes that are imported into a VRF can be further controlled by applying an import route-map. And the routes that are exported from the VRF can be controlled by applying an export route-map.
Complete the following steps:
switch# configure terminal
switch(config)# router bgp 4274781899
switch(config-router-bgp)# vrf vrf1
switch(config-router-bgp-vrf-vrf1)# rd 36351:268450419
switch(config-router-bgp-vrf-vrf1)# route-target import vpn-ipv4 36351:1001
switch(config-router-bgp-vrf-vrf1)# route-target import vpn-ipv4 36351:268450419
switch(config-router-bgp-vrf-vrf1)# route-target export vpn-ipv4 36351:268450419
switch(config-router-bgp-vrf-vrf1)# route-target import vpn-ipv4 route-map BGP-IMPORT-VRF-SERVICES
switch(config-router-bgp-vrf-vrf1)# route-target export vpn-ipv4 route-map BGP-EXPORT-VRF-VRF1
switch(config-router-bgp-vrf-vrf1)# redistribute connected route-map BGP-ANNOUNCE-CONNECTED
switch(config-router-bgp-vrf-vrf1)# redistribute static route-map BGP-ANNOUNCE-STATIC
switch(config-router-bgp-vrf-vrf1)# route-map BGP-IMPORT-VRF-SERVICES permit 10
switch(config-route-map-BGP-IMPORT-VRF-SERVICES)# match extcommunity SERVICES
switch(config-route-map-BGP-IMPORT-VRF-SERVICES)# match ip address prefix-list SERVICESVPN Next-hop
By default, the system uses the source address of the BGP session as the nexthop in the VPN advertisements. This source address is determined by the system while establishing the TCP session between the PEs. If the SP want to use a different address as the VPN next-hop, then an interface with that address must be specified under the BGP address-family VPN-IPv4 or address-family VPN-IPv6 configuration modes. For example,
switch(config-router-bgp)# address-family vpn-ipv4
switch(config-router-bgp-af)# neighbor 10.0.0.2 activate
switch(config-router-bgp-af)# neighbor default encapsulation mpls next-hop-self source-interface Loopback0
switch(config-router-bgp)# address-family vpn-ipv6
switch(config-router-bgp-af)# neighbor 10.0.0.2 activate
switch(config-router-bgp-af)# neighbor default encapsulation mpls next-hop-self source-interface Loopback0In the previous example, the interface Loopback 0 has been specified. The system picks the VPN next-hop address from the interface based on the following rules:
-
For VPN-IPv4, the IPv4 address from the interface is picked.
-
For VPN-IPv6,
- For an IPv4 peer,
- Pick the IPv4 address from the interface.
- If the interface does not have a IPv4 address, pick the IPv6 address.
- For an IPv6 peer,
- Pick the IPv6 address from the interface.
- If the interface does not have an IPv6 address, pick the IPv4 address.
- For an IPv4 peer,
For VPN-IPv6, when an IPv4 address is picked as the next-hop, it is encoded as a IPv4-mapped IPv6 address in the VPN route advertisement. This is as per RFC 4659.
Show Commands
Use the show bgp instance vrf vrf1 command to show the BGP instance status for a specific VRF to verify the route-targets and import/export route-maps being used. The command also displays the locally allocated MPLS label that the system has allocated for IPv4 and IPv6.
switch# show bgp instance vrf vrf1
BGP instance information for VRF vrf1
BGP Local AS: 4274781899, Router ID: 169.254.156.10
Total peers: 0
Static peers: 0
Dynamic peers: 0
Disabled peers: 0
Established peers: 0
Four Octet ASN mode enabled
Graceful restart helper mode disabled
Graceful restart mode disabled
Graceful restart timer timeout: 00:05:00
End of rib timer timeout: 00:05:00
Attributes of the reflected routes are not preserved
UCMP mode: disabled
Peer mac resolution timeout: 00:00:00
BGP IPv4 Listen Port Status: listening on port 179
BGP IPv6 Listen Port Status: listening on port 179
BGP Convergence information:
BGP has converged: yes, Time taken to converge: 00:00:31
Outstanding EORs: 0, Outstanding Keepalives: 0
Convergence timeout: 00:05:00
BGP Convergence timer is inactive
BGP Convergence based update synchronization is disabled
BGP Convergence slow-peer timeout: 00:01:30
Address-family IPv4 Unicast:
Redistributed routes into BGP:
Static
Connected
Route Distinguisher: 36351:268450419
Route targets to import:
VPN-IPv4:
36351:1001
36351:268450419
Route targets to export:
VPN-IPv4:
36351:268450419
Route maps to apply on import:
VPN-IPv4: BGP-IMPORT-VRF-SERVICES
Route maps to apply on export:
VPN-IPv4: BGP-EXPORT-VRF-DI-0056
Local IP lookup MPLS VRF label: 135275
Additional-paths installation is disabled
Extended next-hop capability is disabled
Address-family IPv6 Unicast:
Redistributed routes into BGP:
Static
Connected
Route Distinguisher: 36351:268450419
Route targets to import:
VPN-IPv6:
36351:1001
Route targets to export:
VPN-IPv6:
36351:268450419
Local IP lookup MPLS VRF label: 135896
Additional-paths installation is disabledUse the show bgp neighbors command to verify that the VPN address families have negotiated with the neighbor.
switch# show bgp neighbors
BGP neighbor is 10.0.0.2, remote AS 300, internal link
BGP version 4, remote router ID 0.0.1.1, VRF default
Last read 00:00:15, last write 00:00:31
Hold time is 180, keepalive interval is 60 seconds
Configured hold time is 180, keepalive interval is 60 seconds
Hold timer is active, time left: 00:02:02
Keepalive timer is active, time left: 00:00:16
Connect timer is inactive
Idle-restart timer is inactive
BGP state is Established, up for 00:44:18
Number of transitions to established: 1
Last state was OpenConfirm
Last event was HoldTime
Neighbor Capabilities:
Multiprotocol IPv4 Unicast: advertised and received and negotiated
Multiprotocol VPN-IPv4: advertised and received and negotiated
Multiprotocol VPN-IPv6: advertised and received and negotiated
Four Octet ASN: advertised and received and negotiated
Route Refresh: advertised and received and negotiated
Send End-of-RIB messages: advertised and received and negotiated
Additional-paths recv capability:
IPv4 Unicast: advertised
VPN-IPv4: advertised
VPN-IPv6: advertised
Additional-paths send capability:
IPv4 Unicast: received
VPN-IPv4: received
VPN-IPv6: received
Restart timer is inactive
End of rib timer is inactive
IPv4 Unicast End-of-RIB received: Yes
VPN-IPv4 End-of-RIB received: Yes
VPN-IPv6 End-of-RIB received: Yes
Message Statistics:
Sent Rcvd
Opens: 1 1
Notifications: 0 0
Updates: 6 6
Keepalives: 53 54
Route-Refresh: 0 0
Total messages: 60 61
Prefix Statistics:
Sent Rcvd
IPv4 Unicast: 1 1
IPv6 Unicast: 0 0
Configured maximum total number of routes is 12000
Inbound updates dropped by reason:
AS path loop detection: 0
Malformed MPBGP routes: 0
Originator ID matches local router ID: 0
Nexthop matches local IP address: 0
Local AS is 300, local router ID 0.0.0.1
Local TCP address is 10.0.0.1, local port is 179
Remote TCP address is 10.0.0.2, remote port is 47400Use the show bgp vpn-ipv4 summary command to show the status of VPN-IPv4 peers. While the examples below are with respect to VPN-IPv4, the same set of commands are applicable to VPN-IPv6.
switch# show bgp vpn-ipv4 summary
BGP summary information for VRF default
Router identifier 0.0.0.1, local AS number 300
Neighbor Status Codes: m - Under maintenance
Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
10.0.0.2 4 300 3379 60 0 0 1d23h Estab 1 1Use the show bgp vpn-ipv4 command to show how the VPN-IPv4 routes sent and received.
switch# show bgp vpn-ipv4
BGP routing table information for VRF default
Router identifier 0.0.0.1, local AS number 300
Route status codes: s - suppressed, * - valid, > - active, # - not installed, E -
ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes:Or-ID - Originator ID,C-LST - Cluster List,LL Nexthop-Link
Local Nexthop
Network Next Hop Metric LocPref Weight Path
* > RD: 11.0.0.1:0 IPv4 prefix 20.0.0.0/24
- - - 0 i
* > RD: 11.0.1.1:0 IPv4 prefix 20.0.1.0/24
11.0.1.1 - 100 0 iEach entry in the output represent a VPN path in the VPN table. For each VPN path, the RD and actual prefix along with the nexthop information is shown. Paths in the VPN table are either received from other VPN-IPv4 peers (other PEs) or exported from local VRFs.
In the above output, 20.0.0.0/24 is a local route that has been exported. Notice that it has been prepended with the RD 11.0.0.1:0 to make it a VPN-IPv4 route. And the prefix of 20.0.1.0/24 has been received from another PE. It has the RD of 11.0.1.1:0 with a nexthop of 11.0.1.1. Looking at each of those prefixes in detail:
20.0.0.0/24 is a local route from one of the VRFs that has been exported. Notice that along with the prefix, the RD, the export RT and the MPLS label information is displayed. In this case, the MPLS label is a locally allocated label.
switch# show bgp vpn-ipv4 20.0.1.0/24
BGP routing table information for VRF default
Router identifier 0.0.0.1, local AS number 300
BGP routing table entry for IPv4 prefix 20.0.1.0/24, Route Distinguisher:
11.0.1.1:0
Paths: 1 available
Local
11.0.1.1 from 10.0.0.2 (0.0.1.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Extended Community: Route-Target-AS:300:0
MPLS label: 10012320.0.1.0/24 is a prefix that has been received from the VPN-IPv4 peer, 10.0.0.2. The next-hop in this case is 11.0.1.1. This VPN route is imported into a VRF based on the import RT configuration matching the RT received in the VPN route (300:0).
The route is installed in the VRF only when the VPN next-hop is reachable through an MPLS tunnel. The presence of such an MPLS tunnel can be verified using the show tunnel fib command. The output below shows that there is an MPLS tunnel setup by LDP to the VPN nexthop 11.0.1.1.
switch# show tunnel fib
Type 'LDP', index 1, endpoint 11.0.1.1/32, forwarding None
via 10.0.0.2, 'Ethernet6'
label stack 3
Use the show ip bgp vrf vrf1 command to show the BGP table for the VRF which contains the imported VPN-IPv4 route.
switch# show ip bgp vrf vrf1
BGP routing table information for VRF vrf1
Router identifier 11.0.0.1, local AS number 300
Route status codes:s-suppressed,*-valid,>-active,#-not installed, E-ECMP head,e-ECMP
S - Stale, c - Contributing to ECMP, b - backup, L - labeled-unicast
% - Pending BGP convergence
Origin codes: i-IGP, e-EGP, ?-incomplete
AS Path Attributes: Or-ID-Originator ID, C-LST-Cluster List, LL Nexthop-Link Local Nexthop
Network Next Hop Metric LocPref Weight Path
* > 20.0.0.0/24 - - - 0 i
* > 20.0.1.0/24 11.0.1.1 - 100 0 iEach entry in the table represents a path either locally redistributed/received into the VRF (from a BGP peer) or imported from the VPN table.
Use the show ip bgp 20.0.1.0/24 vrf vrf1 command for a more detailed view of the imported IP prefix 20.0.1.0/24:
switch# show ip bgp 20.0.1.0/24 vrf vrf1
BGP routing table information for VRF vrf1
Router identifier 11.0.0.1, local AS number 300
BGP routing table entry for 20.0.1.0/24
Paths: 1 available
Local
11.0.1.1 from 10.0.0.2 (0.0.1.1), imported VPN-IPv4 route, RD 11.0.1.1:0
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Extended Community: Route-Target-AS:300:0
Remote MPLS label: 100123Use the show ip route vrf vrf1 command to view the prefix installed in route table of the VRF:
switch# show ip route vrf vrf1
VRF: vrf1
Codes: C - connected, S - static, K - kernel,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type2, B I - iBGP, B E - eBGP,
R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
NG - Nexthop Group Static Route, V - VXLAN Control Service,
DH - DHCP client installed default route, M - Martian,
DP - Dynamic Policy Route
Gateway of last resort is not set
C 20.0.0.0/24 is directly connected, Ethernet5
B I 20.0.1.0/24 [200/0] via 11.0.1.1/32,LDP tunnel index 1,label 100123
via 10.0.0.2,Ethernet3,label imp-null(3)The output displays both the VPN label, as well as the underlay tunnel (LDP) information.
Limitations
- While configuring a BGP Route-Reflector for the VPN-IPv4/VPN-IPv6 address families, the route-reflector must have transport MPLS LSPs to reach the PE nexthop addresses. Even though the route-reflector may not be in the data path and does not use the transport LSPs to forward traffic, the LSPs are required in order for the BGP nexthops to be considered as reachable valid candidates in the bestpath computation.
- When configuring eBGP peers, which receive routes over an eBGP session and re-advertise the same to a different eBGP peer, next-hop-unchanged knob must be configured, so that the original nexthop is retained.
- With iBGP route-reflector topology, next-hop-self knob must not be configured.
- Even with directly connected PEs, the VPN
nexthops should be the loopback on the directly
connected PEs with an MPLS tunnel between them.
- For a VPN route to be installed in the VRF, the VPN nexthop must resolve over a MPLS tunnel.
- With OSPF as the PE-CE protocol, RFC 4576, setting of the DN bit in the LSA, is not supported.
- Internal BGP (iBGP) as the PE-CE protocol, RFC 6368, is not supported.