iBGP, next hops and full mesh

Theoretical foundations and architecture

Internal BGP (iBGP, RFC 4271) underpins the control plane of service provider networks, interconnecting edge PE routers within an autonomous system. Unlike eBGP, which increments the AS_PATH attribute across domain boundaries, iBGP retains the existing AS_PATH. To prevent routing loops within an AS, iBGP strictly enforces the Split Horizon Rule: a route learned from an iBGP peer must NEVER be advertised to another iBGP peer. Consequently, every PE router requires a direct logical iBGP peering with every other PE in the network — an architecture known as iBGP Full Mesh. The session scaling formula N*(N-1)/2 illustrates the quadratic growth (10 PEs require 45 sessions; 100 PEs demand 4,950 sessions). iBGP sessions peer between stable Loopback0 interfaces (update-source). next-hop-self is used on PEs for routes learned from eBGP neighbors (otherwise the next hop is a CE address unknown to the core); in VPNv4/VPNv6 the PE sets itself as next hop automatically.

iBGP and eBGP: what actually differs

It is the same protocol (RFC 4271, TCP/179), but its behaviour depends on whether the peers are in the same AS. These are the differences that matter when designing a service provider network:

The last row is the iBGP split horizon: a route learned from an iBGP peer must not be sent to another iBGP peer. Inside an AS the AS_PATH does not grow, so a loop cannot be detected from it, and this rule replaces loop prevention.

Full mesh math

Every iBGP speaker needs a session with every other one, otherwise some routes will never reach it. N routers need N*(N-1)/2 sessions, N-1 on each router:

• 4 PEs: 6 sessions, 3 per node;
• 10 PEs: 45 sessions, 9 each;
• 20 PEs: 190 sessions, 19 each;
• 50 PEs: 1225 sessions, 49 each;
• 100 PEs: 4950 sessions, 99 each.

It is not only the number of TCP sessions that grows. Every PE stores a path from every neighbour: with 900 thousand prefixes and 99 neighbours the Adj-RIB-In can hold tens of millions of paths, and adding one PE means changing the configuration of all the others. This is the number that forces the move to a Route Reflector (next lesson). Full mesh remains a fine solution for 3-8 PEs and as a mini-mesh inside a cluster.

Loopback peering and update-source

The iBGP session is built between Loopback0 addresses, not physical interface addresses, for two reasons: a loopback does not go down with a single link (the TCP session survives an IGP reroute over a backup path), and the PE loopback address is exactly what LDP/LSPs and the BGP next hop of VPN routes use. Requirements:

1. IGP reachability of the /32 loopback: it must be a /32 route (no summarization), otherwise LDP will not have a label for the FEC of this address (LDP matches the FEC exactly) and the LSP to the next hop will not be built.
2. update-source (Cisco), connect-interface (Huawei), local-address (Junos): the outgoing SYN leaves with the loopback address. Without it the router uses the address of the outgoing interface, and the neighbouring PE, which expects 10.255.0.1, finds no match in its configuration and resets the connection: the session stays in Active.
3. The address must match on both sides: PE2 has 10.255.0.1 in its neighbor statement, and PE1 must source exactly that address.

If both PEs initiate TCP at the same time, a collision occurs: per RFC 4271 section 6.8 the connection initiated by the router with the higher BGP Identifier is kept and the other one is closed. That is why single resets at start-up are sometimes seen in the logs and are not an error.

Next hop and next-hop-self

When it receives a route over eBGP, PE1 gets 198.51.100.0/24 with NEXT_HOP 172.16.1.2 (the CE address on the link). iBGP carries this attribute unchanged. PE2 receives the route with next hop 172.16.1.2, which is not in the core IGP, and marks the path as unreachable (Cisco: inaccessible, Junos: Unusable/hidden, Huawei: the route will not become active), because BGP cannot recursively resolve the next hop. Two solutions:

• next-hop-self on the PE when sending to iBGP (Cisco next-hop-self, Huawei next-hop-local, Junos export policy then next-hop self, MikroTik nexthop-choice=force-self): the next hop becomes the PE1 loopback 10.255.0.1, which is in the IGP and in LDP. For VPN routes the PE substitutes itself automatically, but for plain IPv4 in the global table it must be configured explicitly.
• Put the 172.16.1.0/30 link network into the IGP as a passive interface. It works, but bloats the IGP and breaks when the link changes, which is why provider networks use next-hop-self.

The correct next-hop resolution for transit is: BGP next hop -> /32 in the IGP -> LDP label (transport label). If there is no label for 10.255.0.1, the packet goes out without an LSP and a P router that does not know external prefixes (BGP-free core) drops it.