www.iparchitechs.

com

ISP Design – Using a full table

RR to improve eBGP
performance with MikroTik
routers
P R E S E N T E D B Y:

KEVIN MYERS,
NETWORK ARCHITECT
Profile: About Kevin Myers
Background:
• 20+ years in Networking
• Designed/Built Networks on 6 continents
• MikroTik Certified Trainer
• MikroTik, Cisco and Microsoft Certified

Community Involvement:
Packet Pushers (Podcast Guest / Blogger)

Group Contributor (RouterOS / WISP Talk and others)

Delegate/Roundtable contributor (NFD14)

MT Forum (Forum Veteran – Member since 2012)

Network Collective (Podcast Guest)

Profile: About IP ArchiTechs

Expert Networking
Whitebox | ISP | Data Center | Enterprise

✓ Global Consulting
✓ Managed Networks
✓ Monitoring Locations in: US | Canada | South America

✓ Load Testing Call us at: +1 855-645-7684

E-mail: consulting@iparchitechs.com
✓ Development Web: www.iparchitechs.com
Profile: About IP ArchiTechs

Now in Europe!
IPA Opened an office in Nis, Serbia in 2018
 Design: Why use a route reflector for full tables

Goal of this presentation: When the

presentation is finished, hopefully you will
have walked away with a few key concepts:
• Performance limitations of using full mesh
peering between BGP border routers
• How to leverage open source software to create
a high performance BGP RR for MikroTik border
routers
• Design benefits of using a BGP full table RR
 Design: CHR vs. Hardware for full tables?

• Which platform is better?

• Throughput capabilities?
• x86 CPU vs. ARM/Tilera?

vs.
 Design: CHR vs. Tilera/ARM for BGP Border?

Platform

CPU x86 Tilera ARM

MPLS router CPU Better for heavy Optimized for packet In between x86 and
requirements computational work. transfer. Designed to be Tilera for performance.
depend on load and Higher power draw. low power draw.
explicit/implicit null
Throughput x86 Tilera ARM
At 1530 bytes (L2), More CPU and power is Handles throughput at Handles throughput at
and 8970 bytes (L2) required to move data at different frame sizes different frame sizes
the same speed as a CCR slightly better than x86 similar to Tilera

Performance for full x86 Tilera ARM

tables x86 hardware with Lowest performance for Performance is in
HyperV is the best full tables between Tilera and x86
RouterOS based solution but the best hardware
for full BGP tables platform from MikroTik
for rapid convergence
 Design: What is Free Range Routing?

• FRRouting (FRR) is an open source IP routing protocol suite for

Linux and Unix platforms
• includes protocol daemons for BGP, IS-IS, LDP, OSPF, PIM, and
RIP.Use MikroTik CCRs or CHRs for BGP border routers to
handle throughput
• FRR has its roots in the Quagga project. In fact, it was started
by many long-time Quagga developers who combined their
efforts to improve on Quagga's well-established foundation.
 Design: Why use Free Range Routing?

• FRR is multithreaded for BGP and can improve

the performance of MikroTik BGP border
routers by acting as a route reflector
• Free software since it is open source – can be
compiled from the source or downloaded as
VM using CumulusVX.
• Has been traditionally used as an IX route
server and can handle a large volume of Full
Tables
 Design: Which design for full table route reflection?

• Free Range Routing for BGP RR – Open source routing package that supports BGP
across multiple CPUs. Doesn’t need throughput – out of path
• Use MikroTik CCRs or CHRs for BGP border routers to handle throughput
• Leverage the strength of Open Source software to increase the performance and
scalability of RouterOS as a BGP border router

Free Range
Routing
+
 Design: BGP RR - logical lab setup

Test Network for full table RR

tutone.net bgp1.tutone.net
CCR CHR
.1 100.76.1.0/29
VLAN 761
AS 5459
BGP AS 15459
Full Table
BGP RR bgp2.tutone.net
CCR CHR
.2
.11 100.126.0.0/24 100.76.2.0/29
VLAN 762
AS 6695
BGP AS 16695

Free Range bgp3.tutone.net

Routing CCR CHR
.3
100.76.3.0/29
VLAN 763
AS 1200
BGP AS 8675309 BGP AS 11200
 Design: BGP RR – switch–centric lab setup

Physical
Test Network for
BGP RR
HP 1U x86 Server
500W

94%
CHR BGP Full Feeds
4 1 UID iLO 1 4

Core switch Baltic Networks Vengeance HV

CHR

CHR Border routers

BGP Border routers

 Design: Full Table BGP RR performance

• Concept of testing
• Performance with 3 full tables on CCRs & BGP full table RR
• Performance with 3 full tables on CCRs & full mesh peerings

• Performance with 4 full tables on CCRs + CHR & BGP full table
RR
• Performance with 4 full tables on CCRs + CHR & full mesh
peerings

• Performance with 3 full tables on CHR & BGP full table RR

• Performance with 3 full tables on CHR & full mesh peerings
 Design: Full Table BGP RR performance

BGP Border BGP RR Hypervisor

CCR1036 (2) Free Range Routing ESXi 6.0
CCR1072 (1)
Total routes: 1,757,631 IP Transit Convergence: 2:55 Full Convergence: 8:51
 Design: Full Table BGP performance (Full Mesh Peerings)

BGP Border BGP RR Hypervisor

CCR1036 (2) None ESXi 6.0
CCR1072 (1)
Total routes: 1,757,631 IP Transit Convergence: 6:58 Full Convergence: 8:59
 Design: BGP RR - logical lab setup

Test Network for full table RR – 4 peers (CCR+CHR)

tutone.net bgp1.tutone.net
CCR CHR
.1 100.76.1.0/29
VLAN 761
AS 5459
BGP AS 15459
Full Table
BGP RR bgp2.tutone.net
CCR CHR
.2
.11 100.126.0.0/24 100.76.2.0/29
VLAN 762
AS 6695
BGP AS 16695

Free Range bgp3.tutone.net

Routing CCR CHR
.3
100.76.3.0/29
VLAN 763
AS 1200
BGP AS 11200

bgp4.tutone.net
CHR
CHR

100.76.4.0/29
VLAN 764
AS 13538
BGP AS 8675309 BGP AS 23538
 Design: Full Table BGP RR performance

BGP Border BGP RR Hypervisor

CCR1036 (2) Free Range Routing ESXi 6.0
CCR1072 (1)
CHR (1) HyperV
Total routes: 2,272,474 IP Transit Convergence: 5:33 Full Convergence: 16:51
 Design: Full Table BGP performance (Full Mesh Peerings)

BGP Border BGP RR Hypervisor

CCR1036 (2) None ESXi 6.0
CCR1072 (1)
CHR (1) HyperV
Total routes: 2,272,474 IP Transit Convergence: 7:27 Full Convergence: 17:01
 Design: BGP RR - logical lab setup

Test Network for full table RR – CHR Border

tutone.net bgp4.tutone.net
CHR CHR
.1 100.76.4.0/29
VLAN 764
AS 5459
BGP AS 15459
Full Table
BGP RR bgp5.tutone.net
CHR CHR
.2
.11 100.126.0.0/24 100.76.5.0/29
VLAN 765
AS 6695
BGP AS 16695

Free Range bgp6.tutone.net

Routing CHR CHR
.3
100.76.6.0/29
VLAN 766
AS 1200
BGP AS 8675309 BGP AS 11200
 Design: Full Table BGP RR performance

BGP Border BGP RR Hypervisor

CHR (3) HyperV Free Range Routing ESXi 6.0

Total routes: 2,272,474 IP Transit Convergence: 00:17 Full Convergence: 00:41

 Design: Full Table BGP performance (Full Mesh Peerings)

BGP Border BGP RR Hypervisor

CHR (3) HyperV Free Range Routing ESXi 6.0

Total routes: 2,272,474 IP Transit Convergence: 00:35 Full Convergence: 2:00

 Design: Full Table BGP RR performance

• Performance Conclusions
• BGP RR can help improve the performance of existing
CCRs in the load time of the route table from the
upstreams
• Does not significantly impact overall convergence time
• As the number of IX and IP Transit peers increases, RR
becomes more useful for performance and scale
• CHR + BGP RR achieves the best performance and
scale options
Design: BGP RR design

Design Advantages of
BGP RR
 Design: BGP RR design – Customer peering

Full table RR to provide IP Transit

tutone.net bgp4.tutone.net
CHR CHR
.1 100.76.4.0/29
VLAN 764
AS 5459
BGP AS 15459
Full Table
BGP RR bgp5.tutone.net
CHR CHR
.2
.11 100.126.0.0/24 100.76.5.0/29
VLAN 765
AS 6695
BGP AS 16695

Free Range bgp6.tutone.net

Routing CHR CHR
.3
100.76.6.0/29
VLAN 766

RR Client Peering
AS 1200
BGP AS 8675309 BGP AS 11200

BGP AS 1 – Downstream Customer

Public /30

Customer Aggregation Customer Upstream

 Design: BGP RR design – Customer peering

Use BGP Communities to select upstreams

tutone.net bgp4.tutone.net 65000:5459
CHR CHR
.1 100.76.4.0/29
VLAN 764
AS 5459
BGP AS 15459
Full Table
BGP RR bgp5.tutone.net
CHR 65000:6695 CHR
.2
.11 100.126.0.0/24 100.76.5.0/29
VLAN 765
AS 6695
BGP AS 16695

Free Range bgp6.tutone.net

Routing CHR 65000:1200
CHR
.3
100.76.6.0/29
VLAN 766

RR Client Peering
AS 1200
BGP AS 8675309 BGP AS 11200

Filter 65000:5459 at the PE BGP AS 1 – Downstream Customer

Public /30

Customer Aggregation Customer Upstream

Design: Questions?

Questions??