Hunting down the stuck BGP routes

▼ Ben Cox (Benjojo) has an interesting post about stuck BGP routes and a flaw in many BGP implementations where they hang when their neighbor stops accepting data over TCP: Hunting down the stuck BGP routes

A stuck BGP route means that a prefix was advertised at some point, and then it's withdrawn but the withdrawal somehow gets lost somewhere, so part of the internet still sees the withdrawn route.

There doesn't seem to be an obvious way to get these prefixes unstuck, although I would try advertising them again and withdrawing them again. The AS in the middle where the prefixes are stuck should probably clear (reset) BGP sessions until the problem is fixed, although I would be tempted to just reboot the whole router just to be safe.

The hanging issue happens when a router gets behind on processing incoming BGP updates. At some point its TCP receive buffer fills up, so it sets its TCP window size to 0. Then the buffer on the sending BGP neighbor starts to fill up with updates and keepalives. When that buffer is full, the sending BGP process can't write data to the TCP socket anymore... and bad things start to happen.

Ben feels that the BGP spec needs to be updated to address this issue. However, this is not a matter of interoperation that needs to be standardized, but simply an implementation issue that each vendor can fix on their own, in my opinion.

Permalink - posted 2021-04-22

No joke: running BGP on a $100 home router / Wi-Fi access point

▼ For some time, I've been hearing about Mikrotek routers, which couple being quite capable with being affordable. But I never got my hands on one. I'm now in the process of upgrading my home network, and learned about the Mikrotik hAP ac³. The ac³ defies easy classification, but I think it's mostly a home router and/or Wi-Fi access point. I paid € 95 and shipping, and I believe it's available in the US for about $100.

I was somewhat disappointed to learn that "5 gigabit ports" doesn't mean ports that are capable of 5 gigabit, but 5 ports that just ordinary 1 Gbps Ethernet. Initially it seemed the box didn't support IPv6, but it turns you have to enable that under "packages" and then reboot. (Not shutdown.)

However, I wasn't disappointed to learn that the ac³ supports RIP, OSPF and BGP, both for IPv4 and IPv6. So I made a configuration on the ac³ that fits with the lab network that I use for my training courses. Initially it didn't seem to work, but then after a reboot of the ac³ the BGP sessions came up. Pretty cool!

The hardest part was setting up VLAN IP interfaces. For some reason I needed to use the command line to create the VLANs:

/interface vlan add interface=bridge name=vlan1401 vlan-id=1401
/interface ethernet switch vlan add ports=ether3,switch1-cpu independent-learning=no switch=switch1 vlan-id=1401


After that, I could use the web interface to add IP addresses. I also set up BGP through the web interface, but the command line export of that looks like this:

> /routing bgp export
# apr/01/2021 20:50:37 by RouterOS 6.48.1
# software id = C3HH-K6NW
#
# model = RBD53iG-5HacD2HnD
# serial number = E7290D96999E
/routing bgp instance
set default disabled=yes
add as=65082 client-to-client-reflection=no name=Router82 router-id=203.0.113.82
/routing bgp network
add network=10.0.0.0/8 synchronize=no
/routing bgp peer
add address-families=ip,ipv6 hold-time=1m30s instance=Router82 name=ISP30 out-filter=out-as-filter
remote-address=192.0.2.21 remote-as=65030
add instance=Router82 name=ISP40 remote-address=192.0.2.41 remote-as=65040


Initially it seems the Mikrotik won't show you any BGP info, but with a bit more digging you'll get there:

Way back in 1996 the global BGP table was 30,000 prefixes and that just about fit in the 16 MB that was the maximum a Cisco 2501 router would take. The ac³ has 128 MB storage (using less than half of that) and 256 MB RAM, which is probably enough for 200k - 500k prefixes but not a full table. And of course the ac³ is not an obvious choice for such a task in the first place.

Permalink - posted 2021-04-01

When the BGP table hits 1 million prefixes, will history repeat itself?

▼ On the APNIC blog, Danny Pinto asks What will happen when the routing table hits 1024k? Back in 2014, the IPv4 BGP table reached 512k, a common limit in many routers at the time, and some bad things happened. See my post BGP table hitting 512k limit in older routers. And pretty much the same thing happened in 2008, when the BGP table hit 256k.

Interestingly, reaching the 1024k limit was predicted for 2019, but now in March 2019, we're still well below 900k: BGP table growth has leveled off from an average 10% per year in the 2010s to 5% in 2020. My prediction is that we won't reach 1024k IPv4 prefixes in the global BGP table until 2024.

However, asking when we'll reach 1024k IPv4 prefixes is probably asking the wrong question. (But, do read the APNIC blog article for a few relevant pointers.) Back in 2014, there were less than 20k IPv6 prefixes in BGP, and many networks didn't run IPv6 yet. Well, actually, there are 71k networks (autonomous systems) advertising one or more IPv4 prefixes, but only 22k networks announcing one or more IPv6 prefixes.

Still, many networks run IPv4 and IPv6 side-by-side, which means 874k IPv4 prefixes + 114k IPv6 prefixes = 988k total. Add to that internal prefixes, which can add up to many thousands in larger networks, and I'm pretty sure many networks are already over the 1024k limit today, or will be soon.

So I'm sure a few networks will be caught unprepared and run into trouble when the IPv4 table goes over 1024k, but to a much lesser degree than in 2008 and 2014.

In any event, when buying a new router, check if it can handle 2 million total prefixes in the main routing table (RIB) and the forwarding information base (FIB). Also read the fine print, sometimes IPv6 prefixes can count double, and it's relatively common that the 2M limit applies to a combination of firewall rules as well as IPv4 and IPv6 prefixes.

There is usually a separate limit for the BGP RIB, which needs to be much larger, because unlike the main RIB and the FIB, the BGP RIB keeps multiple copies of every prefix: one for each BGP neighbor. So for the BGP RIB you'll want something like 8 million. However, the BGP RIB is usually only limited by available RAM.

Permalink - posted 2021-03-23

RPKI tools and deployment

▼ Recently, Cloudflare launched Is BGP safe yet?. And they immediately answer their own question: No.

What they're getting at is RPKI deployment. RPKI is a mechanism that lets the owner of a block of IP addresses specify which network gets to use those addresses. (Which AS gets to originate a prefix, in BGP speak.) RPKI protects to some forms of (mostly accidental) address hijacking. But for RPKI to work, the address owner needs to publish a "route origination authorization" (ROA) and networks around the globe need to filter based on these ROAs.

Five years ago, I wrote that RPKI is ready for real-world deployment. So where are we now? The US National Institute of Standards and Technology (NIST) has a very nice RPKI deployment monitor, showing the following graph:

So globally, for just about 20% of all prefixes (address blocks) RPKI checks out (valid). For a little less than a percent, the way the address block is routed is not in agreement with the ROA (invalid). For the remaining nearly 80% of prefixes, no ROA is published (not-found). However, these numbers are different in various parts of the world:

• North America: 9% valid, 0.3% invalid, 91% not-found
• Europe: 34% valid, 1% invalid, 64% not-found
• Asia-Pacific: 18% valid, 1% invalid, 81% not-found
• Latin America: 30% valid, 1% invalid, 69% not-found
• Africa: 5% valid, 0.5% invalid, 95% not-found

These numbers show how many address owners are publishing ROAs. This is very easy to do. Here in the RIPE region, it's just a few clicks in the LIR portal. The harder part is filtering based on RPKI. For this you need validator software, for which there are now several choices, and then you need to hook up the validator to your routers, explained here for Cisco and here for Juniper.

For some time, I feared networks would hesitate to filter out prefixes with the RPKI state "invalid", because there's still several thousand prefixes that are invalid. However, it now looks like there are enough big networks doing this that the onus of working around the resulting breakage is correctly put on the address owners / networks that cause the "invalid" state rather than the networks doing the filtering.

Is BGP safe yet? as well as a RIPE labs RPKI test tell you if your ISP is filtering RPKI invalid prefixes, with Cloudflare also naming and shaming the big ones that don't.

The RPKI Observatory has a list of prefixes that have the RPKI invalid state and are therefore unreachable with RPKI filtering enabled. The Route Views collector now also has RPKI, letting you check the state of individual prefixes (telnet route-views.oregon-ix.net). Or use the NLNOG RING looking glass.

Permalink - posted 2020-05-04