Full BGP table in Linux with FRR

For a project, we need to be able to simulate sending full Internet BGP routes to devices in our lab. While we could create a simulated table ourselves in a vacuum (i.e. 500k+ static routes in Linux then resdistribute with BGP), my natural inclination was to use a live Internet feed.

How to get the feed

Both Vultr and Neptune Networks offer low-cost VMs with BGP, however I think they require a real ASN and IP resources to announce. Vultr has options to provide default or full routes, and Neptune provides full routes to all BGP customers. There are probably other ways to get a feed – message me directly if you have a need.

Debian and FRR setup

RAM Note: 2GB seems to be (barely) sufficient, 1GB is not. The memory usage is significant since the routes are stored both in bgpd and zebra daemons. There might be a solution to not put them into zebra but FRR doesn’t seem to let you *not* run zebra

root@rtr-ny1:~# free -m
               total        used        free      shared  buff/cache   available
Mem:            1962        1896          76           0         128          66
Swap:            975         289         686

I’m using Debian 12. Setup steps:

apt install frr which installs version 8.4.4-1.1~deb12u1
edit /etc/frr/daemons and set bgpd to yes
systemctl enable frr ; systemctl start frr

I don’t want to actually *use* the BGP-learned routes for routing on the host, so I put them in a route table (aka VRF). To make it official, edit /etc/iproute2/rt_tables and add a line like this:

100     bgp-rt

FRR Configuration

You could probably configure /etc/frr/frr.conf directly but I prefer to build it by hand in the vtysh CLI. Running vtysh gives you the FRR CLI which is roughly Cisco-esque, where you can show run and config to get into config mode.

One note about using BGP with FRR: when you define a neighbor and peering is established, the default behavior is for no prefixes to be sent or received. A prefix list OR route-map must be specified for routes to be exchanged with the peer (in and out). Here is a working config example, a route-map is used to put the routes into table 100

Current configuration:
!
frr version 8.4.4
frr defaults traditional
hostname rtr-ny1
log syslog informational
no ip forwarding
no ipv6 forwarding
service integrated-vtysh-config
!
router bgp 961
 neighbor 172.82.22.129 remote-as 21700
 neighbor 172.82.22.129 description Neptune_NY
 !
 address-family ipv4 unicast
  neighbor 172.82.22.129 soft-reconfiguration inbound
  neighbor 172.82.22.129 route-map BGP-IN in
 exit-address-family
exit
!
ip prefix-list ipv4-all seq 1 permit 0.0.0.0/0 le 24
ip prefix-list ipv4-all seq 99 deny any
!
route-map BGP-IN permit 1
 match ip address prefix-list ipv4-all
 set table 100
exit
!
end

Useful BGP status commands:

show bgp summary
show ip bgp neighbor 172.82.22.129
show ip bgp neighbor 172.82.22.129 advertised-routes
show ip bgp neighbor 172.82.22.129 received-routes

Use wr to save the config:

rtr-ny1# wr
Note: this version of vtysh never writes vtysh.conf
Building Configuration...
Integrated configuration saved to /etc/frr/frr.conf

exiting back to Linux shell, we can see the routes are not in the default route table (ip -4 r) but can be seen in table 100:

root@rtr-ny1:~# ip -4 route list table 100  | more
1.0.0.0/24 nhid 12 via 172.82.22.129 dev ens3 proto bgp metric 20 
1.0.4.0/24 nhid 12 via 172.82.22.129 dev ens3 proto bgp metric 20 
1.0.5.0/24 nhid 12 via 172.82.22.129 dev ens3 proto bgp metric 20 
root@rtr-ny1:~# ip -4 route list table 100  | wc -l
971664

Success?

Trimming it down

I actually wanted to trim the route count down to be above 512k but below 768k. Excluding /24s was the first thing I considered, however that would remove too many routes.

I did get an as-path regex working that will block given intermediate ASNs:

bgp as-path access-list BLOCK_ASNS seq 5 permit .* (3257|5511) .*
...
route-map BGP-IN deny 10
 match as-path BLOCK_ASNS
exit
!
route-map BGP-IN permit 20
 match ip address prefix-list ANY
 set table 100
exit

In this particular test, blocking those two ASNs brought the route count down from ~970k to ~668k.

packetastic on discord helped get a regex working for AS path length:

bgp as-path access-list LONG_AS_PATH6 seq 5 permit ^[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_.*

With this applied, the route count is down to ~600k.

Sending routes downstream

Fortunately, nothing else is really needed to be figured out at this point. Even though the BGP-learned routes are in a route table, they will be advertised to another eBGP peer. Example config for my eBGP downstream:

router bgp 65002
 bgp router-id 192.168.1.2
 neighbor 10.254.255.50 remote-as 65006
 neighbor 10.254.255.50 description Cisco6506
 !
 address-family ipv4 unicast
  neighbor 10.254.255.50 soft-reconfiguration inbound
  neighbor 10.254.255.50 prefix-list ipv4-all out
!
ip prefix-list ipv4-all seq 1 permit 0.0.0.0/0 le 24
ip prefix-list ipv4-all seq 99 deny any