Theoretical foundations and architecture
In data centres the EVPN data plane is more often built on VXLAN than on MPLS: a leaf-spine fabric provides an IP underlay with ECMP and MPLS labels are not needed. RFC 8365 describes how the same EVPN routes (Type 1-5) serve VXLAN: the service identifier is carried as a VNI, tunnel endpoints are called VTEPs, and BUM traffic is delivered with ingress replication or underlay multicast. For a provider the junction of two worlds matters: a data centre with VXLAN and a WAN with EVPN-MPLS. DCI (Data Center Interconnect) can be built by stretching VXLAN across the WAN or with a gateway that terminates one domain and re-advertises routes into the other (RFC 9014). The lesson covers the VXLAN header (RFC 7348), the 50-byte overhead and its effect on MTU, compares EVPN-VXLAN with EVPN-MPLS and shows gateway/stitching options at the data centre edge. It assumes Type 2/3/5, ESI and IRB from the previous lessons of the module.
Why VXLAN in a data centre
A leaf-spine fabric consists of switches with routed links (often eBGP in the underlay, RFC 7938); every leaf connects to every spine, and several equal-cost paths exist between two leaves. An MPLS core is excessive in such a fabric: it needs LDP or SR and labels, which data-centre hardware often does not support. VXLAN (RFC 7348) encapsulates an Ethernet frame in UDP/IP, so the underlay needs only IP routing with ECMP; path balancing comes from a hash of the inner headers that the VTEP writes into the UDP source port.
The VXLAN header, VTEP and VNI
After the outer Ethernet, IP and UDP (destination port 4789) comes the 8-byte VXLAN header: flags (the I bit set means a valid VNI), reserved bits, the 24-bit VNI and one more reserved byte. The VNI splits the network into up to 16 million segments (against 4094 for VLANs). A VTEP (VXLAN Tunnel Endpoint) is a device that encapsulates and decapsulates frames; its IP address (usually a loopback) is used as the source and the next hop. A device where a VNI is tied to a VLAN and local ports is an L2 gateway; where a VNI is tied to a VRF it is an L3 gateway (the L3 VNI for symmetric IRB).
EVPN for VXLAN (RFC 8365)
Plain VXLAN uses flood-and-learn: unknown MACs are flooded to a multicast group and MACs are learned from traffic. EVPN removes this mechanism: Type 2 announces MAC and IP, Type 3 builds the VTEP list for a VNI, Type 5 carries prefixes, and Type 1 and Type 4 provide multihoming. Differences from EVPN-MPLS: the label field carries the VNI (24 bits) instead of an MPLS label; the BGP Encapsulation extended community states the VXLAN tunnel type; the next hop is the VTEP address; RT and RD are often generated automatically from the AS number and the VNI (RFC 8365 allows auto-derivation, the behaviour depends on the platform). For an L3 VNI the Router's MAC is used: the inner destination MAC in a packet between VTEPs is the MAC of the receiving VTEP.
BUM traffic
There are two delivery methods. Ingress replication (head-end replication): Type 3 carries a PMSI Tunnel attribute of type 6, each VTEP builds a neighbour list per VNI and sends N-1 unicast copies. The underlay needs only unicast routing, but the load lands on the ingress and grows with the number of VTEPs. Underlay multicast: the VNI maps to a group, the underlay builds a PIM tree (Sparse Mode or Bidir) and the spines replicate. This saves leaf resources, but the underlay must support PIM and group distribution, and the number of groups is limited by device resources. ARP/ND suppression and limiting unknown unicast reduce the BUM volume in both variants.
Overhead and MTU
The outer headers add 14 bytes of Ethernet, 20 bytes of IPv4, 8 bytes of UDP and 8 bytes of VXLAN: 50 bytes in total for an IPv4 underlay (70 for IPv6, plus 4 more with an outer 802.1Q tag). The inner frame keeps its Ethernet header, so for a 1500-byte IP packet the underlay IP MTU must be at least 1550. VXLAN packets cannot be fragmented because a VTEP usually sets the DF bit and such packets are dropped; the right solution is jumbo frames (9000-9216) on all fabric links and the gateway. With MPLS the overhead is 4 bytes per label, typically 8-12 bytes (transport label, EVI label, sometimes an ESI label), so the MTU problem is less acute there.
Split horizon: local bias instead of the ESI label
In EVPN-MPLS a BUM frame is protected by the ESI label. VXLAN has no labels, so local bias is used (RFC 8365, section 8.3.1): the ingress VTEP copies a BUM frame from a CE to all other VTEPs, including the other VTEP of the same ES, but the receiver does not send such a frame to its local interface of that ES if the source IP is in the list of VTEPs of the segment (learned from Type 4 routes). Correct Type 4 routes and an up-to-date neighbour list are therefore critical for multihoming in VXLAN.
DCI: VXLAN stretch and gateway
The first option is to stretch VXLAN between data centres over the IP underlay of the WAN: one VNI and one EVPN BGP domain. It is simple, but the failure domain becomes shared: an error in one data centre (for example a BUM storm or a MAC avalanche) affects both. The second option is a DCI gateway at the edge of each data centre (RFC 9014): the gateway terminates VXLAN and re-advertises routes into the WAN as EVPN-MPLS (or as EVPN-VXLAN with a different VNI), changing the label or VNI, the RT and the next hop. For a WAN with an MPLS core over SR or LDP this is natural: the gateway acts as a PE and a VTEP at once. Gateways are grouped into an ES for redundancy; the DF decides who forwards BUM between the domains. For L3 DCI Type 5 routes or a junction of the IP-VRF with an L3VPN (VPNv4) at the edge are usually enough; the exact mechanism depends on the platform.
