New ASBR with MikroTik

It took ages, however the replacement of our autonomous system border router (ASBR) is done. In addition we managed to converge our backbone L2 and L3 equipment and finally get a rid of two single points of failure (SPOF) in sequence.

Although we are RIPE member since 2011 and have been slowly moving to our allocated IP ranges 91.146.120.0/21 and 2a03:a780::/32 in our autonomous system number (ASN) AS198031 ever since, finally getting rid of all the legacy infrastructure took quite long.

After finishing the move, we finally had the opportunity to use a better hardware than old Cisco 2960G L2 switch and SuperMicro x86_64-based PC router with BIRD. We have done some market research and ended up purchasing MikroTik CloudCoreRouter CCR2116-12G-4S+.

As our usual traffic is just around 10Mbps, this little beast should have no problems handling our normal and peak traffic without any issues. There are some interesting aspects to the way MikroTik RouterOS configuration works with respect to hardware-accelerated packet switching and routing. Especially it is necessary to add the bridge itself as one of its ports when using 802.1q VLANs. But other than that the configuration is pretty straightforward.

Some preparations were required. Before we have started, we have configured the switching bridges, VLANs and VLAN interfaces:

  /interface bridge
  add name=brcluster vlan-filtering=yes
  /interface vlan
  add interface=brcluster name=VLAN67 vlan-id=67
  add interface=brcluster name=VLAN140 vlan-id=140
  add interface=brcluster name=VLAN141 vlan-id=141
  add interface=brcluster name=VLAN142 vlan-id=142
  add interface=brcluster name=VLAN144 vlan-id=144
  add interface=brcluster name=VLAN145 vlan-id=145
  /interface bridge port
  add bridge=brcluster interface=ether1
  add bridge=brcluster interface=ether2
  add bridge=brcluster interface=ether3
  add bridge=brcluster interface=ether4
  add bridge=brcluster interface=ether9 pvid=67
  add bridge=brcluster interface=ether10 pvid=67
  /interface bridge vlan
  add bridge=brcluster tagged=ether1,ether2,ether3,ether4,brcluster vlan-ids=140,141,142,144,145,146,802
  add bridge=brcluster tagged=brcluster,ether1,ether2,ether3,ether4 untagged=ether9,ether10 vlan-ids=67

Four physical ports are used to connect two nodes of our main virtualization cluster and two ports are used for remote management of the respective server nodes.

Firstly we have configured the basic networking for both IPv4 and IPv6, including local VLAN interfaces and static routing to other locations. This was, of course, setup on top of a working VLAN configuration that was a drop-in replacement for our current network layout.

  /ip address
  add address=91.146.121.1/24 interface=VLAN140 network=91.146.121.0
  add address=91.146.122.1/24 interface=VLAN141 network=91.146.122.0
  add address=91.146.127.1/24 interface=VLAN142 network=91.146.127.0
  add address=91.146.120.209/28 interface=VLAN144 network=91.146.120.208
  add address=91.146.120.225/28 interface=VLAN145 network=91.146.120.224
  add address=81.0.255.217/29 interface=ether12 network=81.0.255.216
  add address=91.146.120.1/27 interface=VLAN67 network=91.146.120.0
  add address=91.146.120.241/28 interface=VLAN146 network=91.146.120.240
  /ip route
  add blackhole dst-address=91.146.120.0/21
  /ipv6 route
  add dst-address=2a03:a780:1ab::/48 gateway=2a03:a780:1ab::2
  add blackhole dst-address=2a03:a780::/32
  /ipv6 address
  add address=2a03:a780:6667::1 advertise=no interface=VLAN140
  add address=2a03:a780:1ab::1 advertise=no interface=VLAN140
  add address=2a03:a780:1:3::1 advertise=no interface=VLAN140
  add address=2a03:a780:5aba::1 advertise=no interface=VLAN140
  add address=2a03:a780:3:f155::1 advertise=no interface=VLAN141
  add address=2a03:a780:1:4::1 advertise=no interface=VLAN141
  add address=2a03:a780:d20e::1 advertise=no interface=VLAN142
  add address=2a03:a780:1:5::1 advertise=no interface=VLAN142
  add address=2a03:a780:1:6::1 advertise=no interface=VLAN144
  add address=2a03:a780:1:2::1 advertise=no interface=VLAN145
  add address=2001:1528:1:ab00::3 advertise=no interface=ether12
  add address=2a03:a780:b10::1 advertise=no interface=VLAN140

Secondly we added two Border Gateway Protocol (BGP) peers for each of two IP protocols in question having properly configured two redundant upstream routers for both. Testing the failover was easy, however we wanted to make sure that we can operate with all the BGP routes loaded in routing tables. Therefore we have asked our upstream ISP to send us the full tables and yes, everything worked as advertised by the manufacturer. Less than 30 seconds for receiving the full BGP routing tables for both IP protocols!

  /routing bgp connection
  add address-families=ip as=198031 connect=yes input.filter=in listen=yes local.address=81.0.255.217 \
      .role=ebgp name=bgp1v4 output.filter-chain=out4 .keep-sent-attributes=no remote.address=81.0.255.221 \
      .as=15685 router-id=81.0.255.217 templates=default
  add address-families=ip as=198031 connect=yes input.filter=in listen=yes local.address=81.0.255.217 \
      .role=ebgp name=bgp2v4 output.filter-chain=out4 .keep-sent-attributes=no remote.address=81.0.255.222 \
      .as=15685 router-id=81.0.255.217 templates=default
  add address-families=ipv6 as=198031 connect=yes input.filter=in listen=yes local.address=\
      2001:1528:1:ab00::3 .role=ebgp name=bgp1v6 output.filter-chain=out6 remote.address=\
      2001:1528:1:ab00::1 .as=15685 router-id=81.0.255.217 templates=default
  add address-families=ipv6 as=198031 connect=yes input.filter=in listen=yes local.address=\
      2001:1528:1:ab00::3 .role=ebgp name=bgp1v6 output.filter-chain=out6 remote.address=\
      2001:1528:1:ab00::2 .as=15685 router-id=81.0.255.217 templates=default
  /routing filter rule
  add chain=out disabled=no rule=accept
  add chain=in disabled=no rule=accept
  add chain=out6 disabled=no rule="if (dst-len==32) { accept }"
  add chain=out4 disabled=no rule="if (dst-len==21) { accept }"

All in all although there were some hiccups during the deployment process, the overall result is exactly what we wanted. And as we are resuming publishing more articles about what we do these days, always remember to keep an eye on our RSS feed. See you next time!