5 minutes of BGP instability after leap second

▼ This July 30th, at 23:59:60, a leap second was added to Coordinated Universal Time (UTC). Dyn Research posted the following graph on Twitter that shows there was significant BGP update instability for five minutes after the leap second occurred:

Unfortunately, it's not clear why this happened. However, leap seconds have triggered all kinds of mishaps in the past. They're basically miniature Y2K problems. Time and time again, software engineers show that they can't be trusted to take corner cases into account properly.

This does remind me of a situation about a decade ago, where I had a customer that experienced BGP instability every night at the same time. They used Quagga running on Linux machines. We couldn't figure out what the problem was, until we realized that at that very moment, the ntpdate command was run from the cron. ntpdate synchronizes the system clock with an NTP server. As the machine in question had a very poor system clock, this meant that the system's time was adjusted a lot every night, I think a minute or more, but definitely more than 30 seconds.

Which meant that if Quagga had gotten a BGP keepalive message 8 seconds earlier, it now thought that was 38 seconds ago. If BGP is configured with a hold time of 30 seconds, this means that Quagga now thinks the other side has been quiet for longer than the hold time and it'll tear down the BGP session. This is what happened every night for a bunch of BGP sessions. We solved this by running the NTP daemon continuously, so there was never a big adjustment in system time. (Alternatively, just letting the time drift would also have worked.)

The minimum BGP hold time is 3 seconds, so adjusting for an (improperly handled) leap second shouldn't be able to make BGP think the hold time for a session is expired. However, there could be bug somewhere else that impacted BGP.

I'm not sure whether these kinds of issues are a good argument in favor of abandoning leap seconds, as the bugs won't go away, they'll just show up at a less predictable time. But I don't like the current leap second practice, as they're unpredictable, and you can't calculate the time difference in seconds between two dates without taking the entire list of leap seconds into account. I think it would be better to save the leap seconds up and apply them all at the end of a century.

Permalink - posted 2015-07-06

RPKI is ready for real-world deployment

▼ For some years now, the Regional Internet Registries have been rolling out RPKI. The Resource Public Key Infrastructure allows holders of IP addresses to authorize an autonomous system to inject those addresses in BGP. (See here for an overview of how RPKI works and more links.)

I've always thought it would be hard to deploy RPKI in the real world, because it's just way too easy for a certificate or ROA (route origination authorization) to expire. If that then leads to routes becoming invalid and the addresses in question being unreachable, that would be a good example of the cure being worse than the disease.

Fortunately, that's not the case: RPKI is ready for real-world deployment today.

The way to deploy RPKI that's suggested in RFC 6483 as well as the relevant Cisco and Juniper documentation is to assign different local preference values to the three possible RPKI states, such as:

• Valid (RPKI checks out): local preference of 200 (highest)
• Unknown (no RPKI for this prefix): local preference of 100 (normal)
• Invalid (RPKI present but doesn't check out): local preference of 50 (lowest)

So packets will follow a path that is RPKI-validated if available. If not, they follow a path that isn't covered by RPKI if that's available. Only if there's no "valid" or "unknown" paths, the packets will be sent over an "invalid" path that is covered by RPKI, but validation failed. The trouble with this approach is that it still allows for invalid more specific prefixes to hijack traffic. For instance:

RIPE has a ROA for prefix 193.0.0.0/21 that allows AS 3333 to originate that prefix, with a maximum prefix length of /21. So if AS 4444 originates 193.0.0.0/21, that will result in the following BGP table:

    Network       Next Hop       Metric LocPrf Weight Path
>*  193.0.0.0/21  19.11.111.244      10    200      0 3333 i
 *                29.249.178.10      10     50      0 4444 i

So effectively, the path through AS 4444 is ignored. However, AS 4444 could also do this:

    Network       Next Hop       Metric LocPrf Weight Path
>*  193.0.0.0/21  19.11.111.244      10    200      0 3333 i
>*  193.0.0.0/24  29.249.178.10      10     50      0 4444 i
>*  193.0.1.0/24  29.249.178.10      10     50      0 4444 i
>*  193.0.2.0/24  29.249.178.10      10     50      0 4444 i
>*  193.0.3.0/24  29.249.178.10      10     50      0 4444 i
>*  193.0.4.0/24  29.249.178.10      10     50      0 4444 i
>*  193.0.5.0/24  29.249.178.10      10     50      0 4444 i
>*  193.0.6.0/24  29.249.178.10      10     50      0 4444 i
>*  193.0.7.0/24  29.249.178.10      10     50      0 4444 i

So even though the path towards the /21 is still routed to AS 3333, the packets flow to AS 4444 because of the longest match first rule. Solution: filter out "invalid" prefixes completely.

But then, what happens when RIPE forgets to renew their certificate or ROA in time? If their prefix would then revert to "invalid", it would disappear from routing tables everywhere, and RIPE would be unreachable:

    Network       Next Hop       Metric LocPrf Weight Path

In this scenario, it would be very dangerous to filter "invalid" prefixes, as RPKI is still relatively immature and mistakes will happen.

However, it turns out that the results of expired certificates and ROAs aren't actually problematic. In a post to the NANOG list, Alex Band points out:

If ARIN (or another other RIR) went offline or signed broken data, all signed prefixes that previously has the RPKI status "Valid", would fall back to the state "Unknown", as if they were never signed in the first place. The state would NOT be "Invalid".

So what would happen is this:

    Network       Next Hop       Metric LocPrf Weight Path
>*  193.0.0.0/21  19.11.111.244      10    100      0 3333 i

Obviously, in this case the protection against unauthorized origination of the prefixes in question would go away, but in the normal situation where nobody tries to hijack those prefixes, they would still be reachable and a mistake with certificate or ROA expiration wouldn't immediately lead to a network disappearing off of the internet.

In other words: deploy RPKI today. It doesn't protect against all forms of malicious address hijacking, but it does offer very robust protection against accidental unauthorized route origination, such as the infamous Youtube/Pakistan incident. Also, you can run an RPKI validator locally without the need for your upstream ISPs or peers to do the same.

Permalink - posted 2015-04-30

Documentation ASNs and IP prefixes

▼ As you may have noticed, I write about BGP from time to time. When coming up with example configurations, there's always the challenge of which AS numbers and IP addresses/prefixes to use. Although it's unlikely people will simply copy numbers and addresses from examples into their own BGP configurations, experience with NTP has shown that this can be a real problem, so it's a good idea to avoid "real" addresses and numbers in examples.

One obvious choice for IPv4 addresses in examples is the RFC 1918 space: 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0. For IPv6, you could use the unique (site) local addresses (ULA, RFC 4193: fc00::/7, or, more precisely, the ones you get to generate yourself in fd00::/8. I wouldn't recommend using the original site local IPv6 addresses (fec0::/10), as these are "deprecated" in RFC 3879.

For AS numbers, there's the private range 64512 - 65534 (16 bit) and 4200000000 - 4294967294 (32 bit) in the IANA registry.

However, there are also address and AS number ranges specifically set aside for example and documentation use. These have the advantage that they can easily be recognized as being intended for documentation, and won't clash with ranges used in private networks. They are:

• 16-bit AS numbers: 64496 - 64511 (RFC 5398)
• 32-bit AS numbers: 65536 - 65551 (RFC 5398)
• IPv4 addresses: 192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24 (RFC 5737)
• IPv6 addresses:2001:db8::/32 (as in "debate") (RFC 3849)

I actually didn't know about the documentation AS number ranges and the second and third IPv4 documentation ranges. The extra IPv4 ranges will be very useful, as just 192.0.2.0/24 often isn't enough in more complex BGP examples, especially as I don't want to give the impression that it's possible to deaggregate a /24 into smaller parts. The 16-bit documentation AS numbers will also be useful. Unfortunately, the 32-bit ones aren't really useful as they look too much like 16-bit numbers. However, the 65552 - 131071 range is "reserved" so I guess I'll continue to use AS numbers in the 9xxxx range as examples of 32-bit ASNs.

Permalink - posted 2015-04-24

"no synchronization" in Cisco configurations

▼ This text used to be on the BGPexpert.com home page, but Cisco now includes "no syncronization" in the default configuration, so it's unlikely anyone is still going to run into trouble because of this, so I've moved this to a separate page out of the way.

When you run BGP on two or more routers, you need to configure internal BGP (iBGP) between all of them. If those routers are Cisco routers, they won't work very well unless you configure them with no synchronization.

The no synchronization configuration command tells the routers that you don't want them to "synchronize" iBGP and the internal routing protocol such as OSPF. The idea behind synchronizing is that when you have two iBGP speaking routers with another router in between that doesn't speak BGP, the non-BGP router in the middle needs to have the same routing information as the BGP routers, or there could be routing loops. The way to make sure that the non-BGP router is aware of the routing information in BGP, is to redistribute the BGP routing information into the internal routing protocol.

By default, Cisco routers expect you to do this, and wait for the BGP routing information to show up in an internal routing protocol before they'll use any routes learned through iBGP. However, these days redistributing full BGP routing into another protocol isn't really done any more, because it's easier to simply run BGP on any routers in the middle.

But if you don't redistribute BGP into internal routing, the router will still wait for the BGP routes to show up in an internal routing protocol, which will never happen, so the iBGP routes are never used.

The "no synchronization" configuration command tells the routers they shouldn't wait for this synchronization, but just go ahead and use the iBGP routes.

Permalink - posted 2015-02-04

Comparing open source routing platforms

▼ Over the weekend, I wrote posts about trying out OpenBGPD and BIRD.

Here's a bunch more information about these two and other open source routing software:

• Moving Away From OpenBGPd To BIRD? (PDF), by Jimmy Halim (Equinix)
• Route server implementations performance (PDF)
• Open Source Roundtable (PDF), by Ondřej Filip, Martin Winter and David Lamparter

Basically, OpenBGPD's filtering system has scalability issues, and it uses a lot of CPU in large deployments. And it concerns me that these mention OpenBGPD 4.8 and 5.0, while the website only has 4.6.

And I guess I have some homework: install Xorp, a Juniper-like open source routing platform.

Permalink - posted 2015-02-02