The effectiveness of AS path prepending

▼ In a recent blog post The Effectiveness of AS Path Prepending (1) Russ White asks:

Just about everyone prepends AS’ to shift inbound traffic from one provider to another—but does this really work?

(AS path prepending means making the network path as BGP sees it longer to make a path less attractive so traffic will flow over another, shorter path.)

That's an interesting question, as I've been telling people for a long time that it often works too well. A single prepend can flip more than half your traffic from one link to another. In part because of that, Rolf Winter and I worked on a new Origin Preference Attribute that would provide a finer-grained tool for BGP traffic engineering. (Slides, old Internet-Draft, SAC'12 publication.)

But the answer that Russ cites from research into AS path prepending makes sense:

We observe that the effectiveness of prepending can strongly depend on the location (for around 20% of cases, ASPP has moved no targets, while for another 20% , it moved almost all targets).

The the fact that different networks get such different results when performing AS path prepending can be explained by looking at the type of network and the type of connectivity they have. Suppose a small network X has connections to big ISPs A and B. A and B have direct peering relations with most of the other large ISPs. So to the rest of the world, the two paths to X are the same length:

A X = 2 hops
B X is 2 hops

When X now prepends towards A, it's A X X = 3 hops vs B X = 2 hops so most traffic will now flow through the B X path. Prepending was extremely effective.

But what if instead of X doing prepending, A prepends. Now for traffic to X, it's still:

A A X = 3 hops
B X = 2 hops

But other networks that directly connect to A and have traffic for A itself, it doesn't matter if they see A = 1 hop or A A = 2 hops: the traffic needs to go to A, so the path length isn't considered. Even when network C is a customer of B and C sees a direct path and a path over B:

A A A = 3 hops
B A = 2 hops

C will use the direct path, even though it's longer, because the rule is: paths to customers are prioritized over paths to peers and paths to peers over paths to transit providers.

So: will AS path prepending be effective? As with most things in life, it depends.

Permalink - posted 2021-05-13

Hunting down the stuck BGP routes

▼ Ben Cox (Benjojo) has an interesting post about stuck BGP routes and a flaw in many BGP implementations where they hang when their neighbor stops accepting data over TCP: Hunting down the stuck BGP routes

A stuck BGP route means that a prefix was advertised at some point, and then it's withdrawn but the withdrawal somehow gets lost somewhere, so part of the internet still sees the withdrawn route.

There doesn't seem to be an obvious way to get these prefixes unstuck, although I would try advertising them again and withdrawing them again. The AS in the middle where the prefixes are stuck should probably clear (reset) BGP sessions until the problem is fixed, although I would be tempted to just reboot the whole router just to be safe.

The hanging issue happens when a router gets behind on processing incoming BGP updates. At some point its TCP receive buffer fills up, so it sets its TCP window size to 0. Then the buffer on the sending BGP neighbor starts to fill up with updates and keepalives. When that buffer is full, the sending BGP process can't write data to the TCP socket anymore... and bad things start to happen.

Ben feels that the BGP spec needs to be updated to address this issue. However, this is not a matter of interoperation that needs to be standardized, but simply an implementation issue that each vendor can fix on their own, in my opinion.

Permalink - posted 2021-04-22

No joke: running BGP on a $100 home router / Wi-Fi access point

▼ For some time, I've been hearing about Mikrotek routers, which couple being quite capable with being affordable. But I never got my hands on one. I'm now in the process of upgrading my home network, and learned about the Mikrotik hAP ac³. The ac³ defies easy classification, but I think it's mostly a home router and/or Wi-Fi access point. I paid € 95 and shipping, and I believe it's available in the US for about $100.

I was somewhat disappointed to learn that "5 gigabit ports" doesn't mean ports that are capable of 5 gigabit, but 5 ports that just ordinary 1 Gbps Ethernet. Initially it seemed the box didn't support IPv6, but it turns you have to enable that under "packages" and then reboot. (Not shutdown.)

However, I wasn't disappointed to learn that the ac³ supports RIP, OSPF and BGP, both for IPv4 and IPv6. So I made a configuration on the ac³ that fits with the lab network that I use for my training courses. Initially it didn't seem to work, but then after a reboot of the ac³ the BGP sessions came up. Pretty cool!

The hardest part was setting up VLAN IP interfaces. For some reason I needed to use the command line to create the VLANs:

/interface vlan add interface=bridge name=vlan1401 vlan-id=1401
/interface ethernet switch vlan add ports=ether3,switch1-cpu independent-learning=no switch=switch1 vlan-id=1401


After that, I could use the web interface to add IP addresses. I also set up BGP through the web interface, but the command line export of that looks like this:

> /routing bgp export
# apr/01/2021 20:50:37 by RouterOS 6.48.1
# software id = C3HH-K6NW
#
# model = RBD53iG-5HacD2HnD
# serial number = E7290D96999E
/routing bgp instance
set default disabled=yes
add as=65082 client-to-client-reflection=no name=Router82 router-id=203.0.113.82
/routing bgp network
add network=10.0.0.0/8 synchronize=no
/routing bgp peer
add address-families=ip,ipv6 hold-time=1m30s instance=Router82 name=ISP30 out-filter=out-as-filter
remote-address=192.0.2.21 remote-as=65030
add instance=Router82 name=ISP40 remote-address=192.0.2.41 remote-as=65040


Initially it seems the Mikrotik won't show you any BGP info, but with a bit more digging you'll get there:

Way back in 1996 the global BGP table was 30,000 prefixes and that just about fit in the 16 MB that was the maximum a Cisco 2501 router would take. The ac³ has 128 MB storage (using less than half of that) and 256 MB RAM, which is probably enough for 200k - 500k prefixes but not a full table. And of course the ac³ is not an obvious choice for such a task in the first place.

Permalink - posted 2021-04-01

When the BGP table hits 1 million prefixes, will history repeat itself?

▼ On the APNIC blog, Danny Pinto asks What will happen when the routing table hits 1024k? Back in 2014, the IPv4 BGP table reached 512k, a common limit in many routers at the time, and some bad things happened. See my post BGP table hitting 512k limit in older routers. And pretty much the same thing happened in 2008, when the BGP table hit 256k.

Interestingly, reaching the 1024k limit was predicted for 2019, but now in March 2019, we're still well below 900k: BGP table growth has leveled off from an average 10% per year in the 2010s to 5% in 2020. My prediction is that we won't reach 1024k IPv4 prefixes in the global BGP table until 2024.

However, asking when we'll reach 1024k IPv4 prefixes is probably asking the wrong question. (But, do read the APNIC blog article for a few relevant pointers.) Back in 2014, there were less than 20k IPv6 prefixes in BGP, and many networks didn't run IPv6 yet. Well, actually, there are 71k networks (autonomous systems) advertising one or more IPv4 prefixes, but only 22k networks announcing one or more IPv6 prefixes.

Still, many networks run IPv4 and IPv6 side-by-side, which means 874k IPv4 prefixes + 114k IPv6 prefixes = 988k total. Add to that internal prefixes, which can add up to many thousands in larger networks, and I'm pretty sure many networks are already over the 1024k limit today, or will be soon.

So I'm sure a few networks will be caught unprepared and run into trouble when the IPv4 table goes over 1024k, but to a much lesser degree than in 2008 and 2014.

In any event, when buying a new router, check if it can handle 2 million total prefixes in the main routing table (RIB) and the forwarding information base (FIB). Also read the fine print, sometimes IPv6 prefixes can count double, and it's relatively common that the 2M limit applies to a combination of firewall rules as well as IPv4 and IPv6 prefixes.

There is usually a separate limit for the BGP RIB, which needs to be much larger, because unlike the main RIB and the FIB, the BGP RIB keeps multiple copies of every prefix: one for each BGP neighbor. So for the BGP RIB you'll want something like 8 million. However, the BGP RIB is usually only limited by available RAM.

Permalink - posted 2021-03-23