Comparing open source routing platforms

▼ Over the weekend, I wrote posts about trying out OpenBGPD and BIRD.

Here's a bunch more information about these two and other open source routing software:

• Moving Away From OpenBGPd To BIRD? (PDF), by Jimmy Halim (Equinix)
• Route server implementations performance (PDF)
• Open Source Roundtable (PDF), by Ondřej Filip, Martin Winter and David Lamparter

Basically, OpenBGPD's filtering system has scalability issues, and it uses a lot of CPU in large deployments. And it concerns me that these mention OpenBGPD 4.8 and 5.0, while the website only has 4.6.

And I guess I have some homework: install Xorp, a Juniper-like open source routing platform.

Permalink - posted 2015-02-02

The BIRD Internet Routing Daemon

▼ Yesterday, I talked about my first experience with OpenBGPD. Which of course raised the question: why not use BIRD instead? It's a lot more mature. So I gave it a try.

The BIRD Internet Routing Daemon supports RIP, OSPF, BGP and BFD. Unlike Zebra/Quagga, all the protocols are combined in a single daemon with a single configuration file. I once again made good use of the FreeBSD ports, although BIRD has so many dependencies that the 2 GB drive that I had configured for my VM wasn't big enough...

I edited the included example bird.conf file and got a BGP session working. However, I had somewhat of a hard time: BIRD's configuration system isn't like anything I've used before. But worse, the birdc tool that lets you interact with the daemon is very non-obvious. And although BIRD's documentation is pretty decent for an open source project, it doesn't discuss the operational aspects of BGP. To add insult to injury, birdc doesn't have any built-in knowledge of BGP. Rather, BGP is handled as a generic protocol instance. This means that the built-in tab-to-complete and ?-to-show-options don't show you what you can do. The wiki has a few examples but is still not very helpful.

As such, BIRD is definitely not recommended for casual experimentation with BGP. For that, Quagga or even Zebra (which I can run on my Mac!) are much better, as the way they are configured is more Cisco-like, which means the skills gained are relatively easy to transfer to Cisco or Brocade routers. Or, if that's not important, try OpenBGPD.

On the other hand, BIRD does indeed seem to be much more mature and well-supported than OpenBGPD or Quagga (while Zebra has been dead for a decade): BIRD is currently being used as a route server at several internet exchanges. So for production use, BIRD is probably a good choice. However, expect a shallow learning curve.

(Pet peeve: a steep learning curve implies that you improve a lot in a short amount of time. So something that's hard has a shallow learning curve, not a steep one.)

Permalink - posted 2015-02-01

OpenBGPD

▼ As part of my BGP training course, I explain to the participants that they can get BGP either by buying a router from the likes of Cisco or Juniper, or by running routing software such as Zebra, Quagga or OpenBGPD on a Unix (-like) operating system. Then I always mention that I haven't tried OpenBGPD yet, but I really should.

There's no time like the present, so I decided to take the plunge today.

OpenBGPD is a daemon implementing the BGP protocol that was developed for OpenBSD. So I first tried installing an OpenBSD 5.6 VM, which was pretty painless. Unfortunately, I couldn't get the OpenBSD package manager to install OpenBGPD for me. So I went to www.openbgpd.org (where I borrowed the image above). The latest version of OpenBGPD is 4.6, which is five years old—and it wouldn't compile. I never know whether it's my mediocre Unix skills or an actual show stopping issue in these cases. Update: the mediocre skills. As @tvlooy points out: OpenBGPD is included in the OpenBSD base install.

I then installed a FreeBSD 10 VM, which is also relatively painless but requires a bit more configuration during the installation process. However, FreeBSD has OpenBGPD in its ports system, and it installed from there without issue.

So now for the interesting part: configuring OpenBGPD.

It's really interesting to see a more Unix-like take on running BGP. The de facto standard way to configure and run BGP is the one invented by Cisco: you type configuration commands which are immediately applied to the active configuration. Zebra/Quagga uses the same model, even letting you interact with the daemon using telnet. Juniper, on the other hand, uses a very different model, where you use commands to modify the configuration and then "commit" that configuration (or roll back if necessary, very old school DBMS style).

With OpenBGPD, you simply edit the /usr/local/etc/bgpd.conf file after reading the man page. When you're done, you tell the bgpd to reload the configuration. I like this a lot. Monitoring the state of BGP sessions and the BGP table is done using the bgpctl utility, which lets you run clear and show commands that aren't entirely unfamiliar.

I plan on doing the BGP training course exercises using OpenBGPD myself next week, and see how well everything works in practice. The filtering/policy system seems to be inspired by the PF firewall with its insane "the last match is the one that counts" system, and the link between a filter rule and a neighbor is much less direct than with other systems. But I'm trying to keep an open mind.

OpenBGPD looks fairly mature from what I've seen so far, with one small but annoying omission: although you can configure 32-bit AS numbers in both ASDOT and ASPLAIN formats in the configuration file, bgpctl only shows them in ASDOT format. That means that if you have a BGP session with DE-CIX which has AS 196610, bgpctl will show that session as being towards AS 3.2. I.e., ASDOT notation is two 16-bit numbers separated by a dot (3.2) while ASPLAIN is simply a 32-bit number in decimal (196610). ASDOT was in vogue for a while when 32-bit AS numbers were introduced, but quickly fell out of favor as people realized that making regular expressions that match AS numbers with a dot in them would be very annoying. There's even a three-page RFC explaining all of this.

Permalink - posted 2015-01-31

IPv6 addresses and AS numbers in 2014

▼ After yesterday's final yearly IPv4 address report, I thought that today, I'd look at the other numbers the five Regional Internet Registries give out: IPv6 addresses and autonomous system (AS) numbers.

I have pages that show the latest numbers for these based on the most recently imported RIR data: IPv6 and AS numbers.

On January first, 2015, the total number of IPv6 addresses given out was 12626804692343330164921678734295040. The number of IPv6 addresses that are currently set aside for regular "global unicast" use is 42535295865117307932921825928971026432, so 0.0296% of the IPv6 address space has been used up so far. (And we can increase the global unicast space by at least a factor six if we need to.)

Of course the number of individual IPv6 addresses is rather meaningless, as each Ethernet or Wi-Fi network uses up 18446744073709551616 of those, even if there's only a single IPv6 system connected to that network. A more meaningful number is the number of /32s given out. A /32 is the size of the address block given to small-to-mediumsize ISPs. The equivalent of 159,372.68 /32s had been given out on January first, 17,916.90 of which in 2014. That number varies a bit from year to year, as some large ISPs or government entities happen to get particularly large blocks of addresses. However, the number of IPv6 address blocks has been growing fairly steadily from 2510 in 2010 to 4536 in 2014, with a total of 22,118 given out on January first. (For IPv4 the total is 145,917, which is 9039 higher than the total on January first, 2014.) Split out per RIR region:

              2014              Totals
RIR      Blocks   In /32s   Blocks   In /32s
 
AfriNIC     62      51.00     449    4617.00
APNIC      528    2663.00    3730   46210.15
ARIN       512    5232.68    4193   41465.94
LACNIC    1208    1363.20    3370    7989.54
RIPENCC   2226    8607.00   10376   59090.03

As every BGP expert knows, in order to advertise those IPv4 and IPv6 address blocks into the global routing system, you need an autonomous system number. Originally, those were 16 bits, allowing for 64512 usable AS numbers. 58,800 of those had been given out by January first, 2015, with the yearly rate being between 3500 and 4000 in recent years and 3056 in 2014.

However, BGP has been modified to work with 32-bit AS numbers. 9475 of those have been given out; 2052 in 2011, 1706 in 2012, 1577 in 2013 and 2988 in 2014. So the total number of AS numbers given out went pas 6000 for the first time last year. The mix is rather different at the different RIRs (ASNs given out in 2014):

RIR        16+32      32    % 32
 
AfriNIC      141     114     81%
APNIC       1299     722     56%
ARIN        1587     421     27%
LACNIC       964     570     59%
RIPENCC     2053    1161     57%

Unlike with the transition from IPv4 to IPv6, where you need to adopt IPv6 in order to talk to other IPv6 users, existing 16-bit AS numbers can be used to talk to routers that have a 32-bit AS number, making for a much easier transition.

Permalink - posted 2015-01-09

Internet exchange renumbering: everything old is new again

▼ This week, the Amsterdam Internet Exchange is renumbering its peering LAN.

An internet exchange (IX) is simply a very big Ethernet. Members connect a router port to that Ethernet, and can then exchange packets with each other. When you want to exchange traffic with many other networks, obviously this is more efficient than setting up dedicated connections with all these other networks.

Until this week, AMS-IX used a /22 prefix, allowing for about a thousand connected routers. That was no longer enough, so they got a new /21 prefix, which can accommodate two thousand connected routers. This means that all the currently connected routers must get a new address. No big deal. This is why search-and-replace was invented.

However, sometimes someone makes a mistake. Like configuring <new address>/22 instead of <new address>/21. And then letting that /22 propagate to other networks over BGP. Suppose:

• 192.0.0.0/21 is the prefix used on the IX peering LAN
• 192.0.2.1 peers with 192.0.2.2
• 192.0.2.3 advertises 192.0.0.0/22
• 192.0.2.1 wants to send a BPG keepalive packet to 192.0.2.2. Normally this packet would go to the port where 192.0.0.0/21 is configured. But the 192.0.0.0/22 prefix is "more specific", so the packet is sent on its way to the source of the 192.0.0.0/22 prefix: router 192.0.2.3.
• 192.0.2.3 forwards the packet to router 192.0.2.2
• All's well that ends well? Not so fast. In BGP, there are not supposed to be any intermediate routers between two BGP routers. 192.0.2.2 sees the packet traversed and extra hop, rejects it and resets the BGP session.
• 192.0.2.1 can no longer send traffic to 192.0.2.2

(A more specific prefix is a smaller range of IP addresses. 192.0.0.0/21 is BGP talk for the address range 192.0.0.0 - 192.0.7.255. 192.0.0.0/22 is the range 192.0.0.0 - 192.0.3.255. Because the latter identifies a smaller range of IP addresses, the packets are sent in that direction, just like you'd follow a sign "Paris" rather than a sign "France" if you were going to Paris, even though Paris is part of France so presumably following the sign "France" would also get you to Paris.)

The sad thing is that the exact same thing happened in 2003, when the AMS-IX renumbered from a /24 to a /23. I always warn against this issue during my training courses, and tell students to filter the IX prefixes of internet exchanges they're connected to, as well as all possible subprefixes (more specifics) that fall within that IX prefix. For instance:

!
ip prefix-list import deny 172.16.0.0/12 le 32
ip prefix-list import deny 80.249.208.0/21 le 32
ip prefix-list import permit 0.0.0.0/0 le 24
!

This prefix list will reject incoming updates with your own prefix and all possible more specifics (assuming your prefix is 172.16.0.0/12) as well as the AMS-IX prefix 80.249.208.0/21 and all possible subprefixes. It then allows all prefixes with a prefix length of no more than /24, which is common practice for IPv4.

"le" means "less or equal" so "172.16.0.0/12 le 32" means:

172.16.0.0/12
172.16.0.0/13
172.24.0.0/13
...
172.31.255.254/31
172.31.255.254/32
172.31.255.255/32

Hopefully, by the time AMS-IX connects more than 2000 routers, the issue is moot because we no longer use IPv4. But for now: happy renumbering!

Permalink - posted 2014-10-15