iljitsch.com - bgp (en)

→ BGP handling bug causes widespread internet routing instability

Wed, 28 May 2025 13:06:04 GMT

benjojo reports:

At 7AM (UTC) on Tuesday May 20th 2025 a BGP message was propagated that triggered surprising (to many) behaviours with two major BGP implementations that are often used for carrying internet traffic.
This caused a large number of “internet facing” BGP sessions to automatically shut down, causing at the very least some routing instability, and at worst brief loss of connectivity for some networks.

BGP implementations failing in various ways after seeing something incorrect/unexpected is definitely not a new thing...

BGP training course at NL-ix in Zoetermeer October 7

Mon, 15 Jul 2024 09:54:40 GMT

NL-ix brings back its famous BGP (Border Gateway Protocol) course. The course is scheduled to happen on October 7. It will be conducted by Iljitsch van Beijnum. In one day (09.30 - 16.00) course participants will learn the internals of BGP protocol. This will help them with a much better understanding of their current BGP work.

In English or Dutch depending on the wishes/needs of the participants.

More information and registration.

BGP training course at NL-ix in Zoetermeer June 3

Mon, 27 May 2024 13:35:06 GMT

NL-ix brings back its famous BGP (Border Gateway Protocol) course. The course is scheduled to happen on 3rd June. It will be conducted by Iljitsch van Beijnum. In one day (09.30 - 16.00) course participants will learn the internals of BGP protocol. This will help them with a much better understanding of their current BGP work.

In English. Hopelijk binnenkort ook weer in het Nederlands.

Meer informatie en inschrijven

Upgrading Fiber To The Home to terabit speeds

Tue, 09 Apr 2024 11:16:41 GMT

Last week, Jaap van Till asked me if BGP would be capable of supporting the terabit class interconnectivity that he foresees we’ll need in the future, possibly due to the rise of artificial intelligence. He explains his reasoning in the blog post What Link speeds will we need for AI, where he quotes VAN TILL’s CONJECTURE:

The network connection Wide Area access speed will grow in time until it matches the internal device BUS speed of the more and more complex processors and datastores.

And then concludes that 14 Tbps external links will be required in 2039. Today I can get 4 Gbps where I live. So that means a 70% speed increase per year.

Let’s first get that BGP question out of the way: I see no problems. 25 years ago I ran BGP over 64 and 128 kbps links without trouble. Six orders of magnitude later, BGP is still fine, and there is no reason to believe that even faster speeds will be a problem, just as long as the packet loss rates remain minimal.

But what would terabit class network connectivity at home look like?

Actually, I think we have all the parts to build this today. With Wavelength Devision Multiplexing (WDM), it’s possible to transmit multiple data streams through a single fiber by using slightly different wavelengths/frequencies of infrared laser light. Coarse WDM (CWDM) is relatively cheap and appropriate over shorter distances, with 18 wavelengths standardized over high performance fiber. (Fewer over most existing fiber.) For long distances, dense WDM (DWDM) can use as many as 160 wavelengths over a single fiber pair.

Bandwidth per wavelength is now 100 or 200 Gbps, and expected to increase in the future. So anything between say 10 x 100 Gbps = 1 Tbps and the 20 Tbps used by modern seacables should be possible. The catch is of course the cost.

The difficulty is with the transmitting side, as this requires a tuned laser per wavelength. On the receiving side, the wavelengths can be split using a prism and hit a set of wideband receivers. As someone who is definitely not in the business of building this equipment, it seems to me that a system with one or a small number of transmitters, a passive optical bus, and a large(r) set of receivers is definitely something that could enjoy radical performance vs cost improvements over time. And it fits perfectly with the most efficient / high speed way to connect homes to the internet that we have today: PON (passive optical network). So just add additional wavelengths to existing PON installations to gain more bandwidth in the downstream direction.

However, now we have a new challenge: TCP/IP is not a good fit for sending the massive data streams that would make good use of such a network. The problem is that TCP tries to adjust its end-to-end data transmission rate to available bandwidth. This means it needs to wait for acknowledgments from the receiving side to know it can increase its transmission rate, maintain it, or slow the transmission rate down. Downloading 100 MB over a 1 Tbps link takes less than a millisecond. But even over PON, the round-trip-time is a millisecond or two. This means that the bottleneck is the number of round trip TCP requires to reach that full terabit speed. Even if that’s an extremely unrealistic 10 RTTs, that means the total transmission time is now 11 ms, effectively only using a tenth of the available bandwidth.

So we need to overhaul TCP/IP for the super high speed stuff and instead use something more like circuit switching / time division multiplexing / token passing. Yes, everything old is new again! So for instance reserve ten 100 μs timeslots and transmit ten 10 MB “megapackets”.

So I think all of this is highly doable!

Well, there is the slight challenge of how to pipe all that bandwidth into your laptop without connecting/disconnecting that fiber all the time. Maybe use eight Thunderbolt 5 interfaces in parallel to reach 960 Gbps?

→ Enforcing First AS in BGP

Sun, 08 Oct 2023 14:00:49 GMT

The BGP RFCs state that external BGP peers should insert their own AS into the AS PATH advertised to eBGP peers. Some peers strip their AS, generally for commercial gain. Juniper and Cisco have opposite default behaviors for handling this. Make sure you set bgp enforce-first-as on Juniper routers. Caveats apply.

The annoying part here is that you want to disable this check for internet exchange route servers, but keep it enabled for everything else for security reasons. But that's not universally possible, as on some routers this is a global setting, rather than a per-neighbor one.