Interactions between Congestion Control Algorithms

Belma Turkovic∗ , Fernando A. Kuipers† and Steve Uhlig ‡

∗ †
Delft University of Technology, Delft, The Netherlands
‡Queen Mary University of London, London, United Kingdom
Email: {∗ B.Turkovic-2, † F.A.Kuipers}@tudelft.nl, ‡ Steve.Uhlig@qmul.ac.uk,

Abstract—Congestion control algorithms are crucial in achiev- RFC 793

ing high utilization while preventing overloading the network.
Over the years, many different congestion control algorithms
have been developed, each trying to improve over others in Tahoe
specific situations. However, their interactions and co-existence
has, to date, not been thoroughly evaluated, which is the focus Reno DUAL Veno
of this paper. Through head-to-head comparisons of loss-based,
delay-based and hybrid types of congestion control algorithms,
NewReno Vegas Vegas+
we reveal that fairness in resources claimed is often not achieved,
especially when flows sharing a link have different round-trip
times or belong to different groups. SCTP VegasA TCP AR

I. I NTRODUCTION HS-TCP NewVegas Africa

In the wake of the growing demand for higher bandwidth,
higher reliability, and lower latency, novel congestion control H-TCP FAST Compound
algorithms have been developed. For example, in 2016, Google
published its bottleneck bandwidth and round-trip time (BBR) Hybla VFAST Fusion
congestion control algorithm, claiming it was able to oper-
ate without filling buffers [1]. Around the same time, TCP BIC LoLa YeAH
LoLa [2] and TIMELY [3] were proposed, focusing on low
latency and bounding of the queuing delay. Moreover, new Cubic TIMELY AReno
transport protocols such as QUIC allow the implementation
Westwood Libra
of algorithms directly in user space, which facilitates quick
development of new transport features. However, congestion
Westwood+ Illinois
control algorithms have been typically developed in isolation,
without thoroughly investigating their behaviour in the pres- TCPW-A BBR
ence of other congestion control algorithms, which is the goal
of this paper. LogWestwood+ PCC
In this paper, we first divide existing congestion control
algorithms into three groups: loss-based, delay-based, and Loss-based Delay-based Hybrid
hybrid. Based on experiments in a testbed, we study the algorithms algorithms algorithms
interactions over a bottleneck link among flows of the same
Fig. 1: Classification of different congestion control algo-
group, across groups, as well as when flows have different
rithms. Dotted arrows indicate that one was based on the other.
Round-Trip Times (RTTs). We find that flows using loss-based
algorithms are over-powering flows using delay-based, as well our measurement setup, and (3) present our measurement
as hybrid algorithms. Moreover, when flows using loss-based results. Additional measurements are given in an extended
algorithms fill the queues, increase in queuing delay of all version [4].
the other flows sharing the bottleneck is determined by their
presence. Non-loss-based groups thus cannot be used in a II. BACKGROUND
typical network, where flows typically rely on a loss-based Since the original TCP specification (RFC 793 [5]), numer-
algorithm. In addition, we observe that convergence times ous congestion control algorithms have been developed. In
can be large, which may surpass the flow duration for many this paper, we focus mostly on algorithms designed for wired
applications. Finally, we find that hybrid algorithms, such as networks. The algorithms we consider can be used both by
BBR, not only favour flows with a higher RTT, but they also QUIC and TCP and can be divided into three main groups
cannot maintain a low queuing delay as promised. (see Fig. 1): (1) loss-based algorithms that detect congestion
In Section II, we provide an overview and classification of when buffers are already full and packets are dropped, (2)
congestion control mechanisms. In Section III, we (1) identify delay-based algorithms that rely on RTT measurements and
a set of key performance metrics to compare them, (2) describe detect congestion by an increase in RTT, indicating buffering,
and (3) hybrid algorithms that use some combination of the Recently, as low latency became important, several new
previous two methods. algorithms have been proposed. Hock et al. designed LoLa [2],
focusing on low latency and convergence to a fair share be-
A. Loss-based algorithms tween flows. To improve performance in datacenter networks,
Google proposed TIMELY [3], which relies on very precise
The original congestion control algorithms from [5] were
RTT measurements. Since Vegas is used as the base algorithm
loss-based algorithms with TCP Reno being the first widely
by many other delay-based and hybrid algorithms, we use it
deployed one. With the increase in network speeds, Reno’s
as a reference for delay-based algorithms.
conservative approach of halving the congestion window
became an issue. TCP connections were unable to fully C. Hybrid algorithms
utilize the available bandwidth, so that other loss-based al-
Hybrid algorithms use both loss and delay as congestion
gorithms were proposed, such as NewReno [6], Highspeed-
indicators. The first hybrid algorithm was Veno [24]. It is a
TCP (HS-TCP [7]), Hamilton-TCP (H-TCP [8]), Scalable TCP
modification of the Reno congestion control that extends the
(STCP [9]), Westwood (TCPW [10]), TCPW+ (TCP West-
additive increase and multiplicative decrease functions by also
wood+ [11]), TCPW-A [12], and LogWestwood+ [13]. They
using queuing delay as the secondary metric. To efficiently
all improved upon Reno by including additional mechanisms
utilize the available bandwidth in high-speed networks, many
to probe for network resources more aggressively. However,
algorithms use similar modifications based on the Vegas or
they also react more conservatively to loss detection events,
Dual network state estimations. Some of the most important
and discriminate between different causes of packet loss.
ones are Africa [25], Compound [26], and YeAH [27]. Other
However, these improvements did not address any of the
algorithms modify the congestion window increase function
existing RTT-fairness issues, but introduced new ones [14],
to follow a function of both the RTT and the bottleneck link
[15]. Indeed, when two flows with different RTTs share the
capacity, such as Illinois [28], AR [29], Fusion [30], TCP-
same bottleneck link, the flow with the lowest RTT is likely
Adaptive Reno (AReno) [31], and TCP Libra [32].
to obtain more resources than other flows. To resolve this
In 2016, Google developed the bottleneck bandwidth and
issue, BIC [14] and Hybla [15] were proposed. Hybla modified
round-trip time (BBR) algorithm. However, several problems,
NewReno’s Slow Start and Congestion Avoidance phases and
mostly related to the Probe RTT phase, were discovered: (1)
made them semi-independent of RTT. However, the achieved
bandwidth can be shared unfairly depending on the timing of
RTT-fairness meant that flows with higher RTTs behaved more
new flows and their RTT, and (2) unfairness towards other
aggressively. The main idea of BIC was to use a binary search
protocols, especially Cubic [33], [34], [35].
algorithm to approach the optimal congestion window size.
At the same time, a new approach to congestion control
However, later evaluations showed that BIC can still have
using online learning was proposed in PCC [36]. We use BBR
worse RTT-fairness than Reno [16]. In response, Cubic was
as our representative for hybrid algorithms, since it is actually
proposed in [16]. Since Cubic is the current default algorithm
deployed (in Google’s network) and implemented in the Linux
in the Linux kernel, we will use it as a reference for loss-based
kernel (since v4.9).
algorithms throughout this paper.
III. E VALUATION
B. Delay-based algorithms
Using the metrics described in Sec. III-A and via the set-up
In contrast to loss-based algorithms, delay-based algorithms described in Sec. III-B, in Sections III-C and III-D we evaluate
are proactive. They try to find the point when the queues in the the representatives of the three algorithm groups (Cubic, Vegas
network start to fill, by monitoring the variations in RTT. An and BBR). Additional measurements and results of all other
increase in RTT, or a packet drop, causes them to reduce their algorithms that have been implemented in the Linux kernel
sending rate, while a steady RTT indicates a congestion-free can be found in the extended version of this paper [4].
state. Unfortunately, RTT estimates can be inaccurate due to
delayed ACKs, cross traffic, routing dynamics, and queues in A. Performance metrics
the network [3], [17]. Sending rate represents the bit-rate (incl. data-link layer
The first algorithm that used queuing delay as a congestion overhead) of a flow generated by the source, per time unit.
indicator was TCP Dual. The first improvement to this algo- Throughput measures the number of bits (incl. the data-
rithm was Vegas [18]. It focuses on estimating the number of link layer overhead) received at the receiver, per time unit.
packets in the queues and keeping it under a certain threshold. RTT (round-trip time) represents the time between send-
However, several issues were identified. First, when competing ing a packet and receiving an acknowledgement of that packet.
with existing loss-based algorithms, Vegas flows suffer from Goodput measures the amount of useful data (i.e., excl.
a huge decrease in performance [19], [20]. Second, it has a overhead) delivered by the network between specific hosts,
bias towards new flows and, finally, interprets rerouting as per time unit. This value is an indicator of the application-
congestion [20]. To address these issues several modifications level QoS experienced by the end-users. Additionally, we use
to Vegas were proposed, including VegasA [20], Vegas+ [19], the goodput ratio, i.e., the amount of useful data transmitted
FAST [21], VFAST [22], and NewVegas [23]. divided by the total amount of data transmitted.
 Fairness describes how the available bandwidth is shared C. Results: BW scenario
among multiple users. We consider three different types of
fairness: (1) intra-fairness describes the resource distribution Intra-Fairness. Delay-based and loss-based algorithms
between flows running the same congestion control algorithm; have the best intra-fairness properties, with an average fairness
(2) inter-fairness describes the resource distribution between index within 0.94 − 0.95 (Table I). Fig. 3 shows that Jain’s
flows running different congestion control algorithms, and index is always close to 1, indicating that all present flows
(3) RTT-fairness describes the resource distribution between receive an equal share of the resources. In addition, delay-
flows having different RTTs. Fairness is represented by Jain’s based algorithms operate without filling the buffers, in contrast
index [37]. This index is based on the throughput and indicates to the loss-based algorithms that periodically fill the buffers
how fair the available bandwidth at the bottleneck is shared and drop packets (Fig. 3). Further, the convergence time of
between all flows present. This fairness index ranges from 1/n loss-based algorithms is higher (≈ 20 s, compared to 5s
(worst case) to 1 (best case), where n is the number of flows. needed for 2 Vegas flows) and their throughput oscillates the
most from all the evaluated approaches (Fig. 3). When the
B. Experiment setup number of Cubic flows increases to 4, bandwidth oscillations
increase as well, and fairness decreases to 0.82 [4].
Each server in our testbed has a 64-bit Quad-Core Intel
Xeon CPU running at 3GHz with 4GB of main memory and In contrast, hybrid-based algorithms (BBR) unexpectedly
has 6 independent 1 Gbps NICs. Each server can play the had the worst intra-fairness properties. Fig. 3 shows that they
role of a 6-degree networking node. All nodes run Linux rarely converge to the same bandwidth, but oscillate between
with kernel version 4.13 with the txqueuelen set to 1000, and 30 M bps and 70 M bps (every probeRTT phase), even in
were connected as shown in Fig. 2 with degree 1 ≤ n ≤ 4 scenarios in which they claim a similar share of the available
(consequence of the limited number of NICs per server in resources on average. The flow that measures a higher RTT
the testbed). Given that the performance of congestion control adopts a more aggressive approach and claims more resources,
algorithms is affected by the bottleneck link on the path, even if the measured RTT difference is very small (≤ 0.5ms).
such a simple topology is sufficient for our purposes. The Hence, they are not particularly stable. Unexpectedly, when the
maximum bandwidth and the bottleneck (between s1 and s2) number of flows increases to 4, the fairness index improves,
was limited to a pre-configured value (100M bps in the case and although oscillations go down they are still present.
of TCP and 10M bps in the case of QUIC to make sure Inter-Fairness. As expected, flows that use delay-based
that the sending rate of the end-user applications is enough algorithms experience a huge decrease in throughput if they
to saturate the bottleneck link) with the use of ethtool. To share the bottleneck with loss-based flows (Fig. 4). This is
because they detect congestion earlier, at the point when
Clients Servers the queues start to fill. Loss-based algorithms on the other
Bandwidth of the bottleneck hand continue to increase their sending rate as no loss is
Cn Sn detected. This increases the observed RTT (Fig. 3) of all flows,
.. .. triggering the delay-based flow to back off [19], [20].
. 1 2 .
A similar behaviour is observed when a bottleneck is
C1 S1
shared between flows from a hybrid and a delay-based al-
gorithm: BBR outperforms Vegas. However, the difference in
Fig. 2: Experiment topology. the throughput is less significant than the one observed in
the previous scenario, with the Vegas flow claiming almost
perform measurements, we rely on tshark, iperf, QUIC client 40M bps on average (Table I). When we increase the number
and server (available in the Chromium project [38]) and socket of Vegas or BBR flows at the bottleneck to four, the new
statistics. From traffic traces (before and after the bottleneck), flows increase their bandwidth at the expense of the BBR
we calculate the metrics described in Sec. III-A. All the values flow, reducing its share from 50M bps down to 20M bps, and
are averaged per flow, using a configurable time interval. We increasing the fairness index to 0.9 − 0.94 [4]. This is a
consider the following two scenarios: consequence of the fact that BBR tries to operate without
BW scenario. Each analyzed algorithm is compared to itself filling the queues, allowing the delay-based algorithm to grow
and all others. Host Ci generates TCP flows towards servers and claim more bandwidth. Thus, we conclude that, in contrast
running at Si using different congestion control algorithms. to loss-based algorithms, delay-based algorithms can co-exist
RTT scenario with flows having different RTTs. The with hybrid-based ones.
purpose of this scenario is to test the RTT-fairness of different When the bottleneck is shared between a hybrid and a
congestion control algorithms. In addition to the setup of the loss-based algorithm, Cubic outperforms BBR, reducing its
previous scenario, the delay at links between Si and node 2 share of resources to as little as 8% on average (Table I),
is artificially increased using Linux TC (adding 0 − 400ms). confirming results from [39]. The fairness index at the start of
We ran these scenarios five times. For all of them, the the connection is very low as Cubic claims all the available
results we observe lead to qualitatively similar interactions, bandwidth at the expense of the BBR flow. After the Cubic
as presented in Sections III-C and III-D. flow fills the buffers, BBR measures an increased RTT and
 200 ·108 1
1

Throughput [bps]
Cubic Cubic
150 0.8

Jain index
RTT [ms]

0.6 0.8
100
0.4
50 0.2 0.6
Cubic Cubic 2xCubic
0 0
0 20 40 60 0 20 40 60 0 20 40 60
t [s] t [s] t [s]
4 ·108 1
1

Throughput [bps]
Vegas Vegas
3 0.8

Jain index
RTT [ms]

0.6 0.8
2
0.4
1 0.6
Vegas Vegas 0.2 2xVegas
0 0
0 20 40 60 0 20 40 60 0 20 40 60
t [s] t [s] t [s]
8 ·108 1
1
Throughput [bps]

BBR BBR BBR BBR

6 0.8

Jain index
RTT [ms]

0.6 0.8
4
0.4
2 0.2 0.6 2xBBR
0 0
0 20 40 60 0 20 40 60 0 20 40 60
t [s] t [s] t [s]
Fig. 3: BW scenario: Comparison of average RTT, average throughput, and fairness index for representatives of the congestion
control algorithm classes groups in case the link is shared by 2 flows using the same algorithm (time unit 300ms).

TABLE I: BW scenario with 2 flows: Different metrics for representatives of the three congestion control algorithm groups
(calculated for 5 different runs).
Protocol Group Algorithm Average Average Average Average Average Average
goodput goodput ratio RTT sending rate throughput Jain index
[M bps] [%] [#packets] [ms] [M bps] [M bps]
Cubic 44.98 93.57 76.65 48.77 46.59
Loss- vs. Loss-based 0.95
Cubic 43.15 93.78 78.32 50.98 46.59
Vegas 43.81 94.81 1.66 48.65 45.47
Delay- vs. Delay-based 0.94
Vegas 42.72 94.76 1.68 49.79 44.38
BBR 44.98 92.32 3.21 52.18 46.70
Hybrid vs. Hybrid 0.86
BBR 42.72 94.39 3.24 46.89 44.36
TCP
Cubic 82.29 94.27 70.37 90.91 85.05
Loss-based vs. Hybrid 0.59
BBR 7.56 88.86 174.38 8.87 7.89
Cubic 87.73 94.34 67.16 97.30 90.66
Loss- vs. Delay-based 0.52
Vegas 1.74 91.57 139.79 2.00 1.82
Vegas 38.37 94.34 4.55 37.31 39.83
Delay-based vs. Hybrid 0.84
BBR 48.56 94.68 4.25 61.65 50.37

adopts, as a consequence, a more aggressive approach (Fig. 3). bottleneck, the observed delay is determined by it, nullifying
However, packet loss triggers Cubic’s back-off mechanism, al- the advantages of delay-based and hybrid algorithms, namely
lowing BBR to measure a lower RTT estimate. Consequently, the prevention of the queue buildup. BBR, as well as Vegas,
BBR reduces its rate, allowing the Cubic flow to claim more which claim to be able to operate with a small RTT, suffer
bandwidth again. Moreover, when we increase the number of from a huge increase in average RTT (by more than 100 ms,
Cubic flows to three, the throughput of the BBR flow drops Table I) when competing with Cubic (compared to 1 − 5ms
close to zero. Similarly, even three BBR flows are not able without Cubic). However, when a link is shared between a
to compete with one Cubic flow, with each of them claiming hybrid and a delay-based flow, both of them are able to
approximately 5% of the total bandwidth on average [4]. maintain a low RTT. In such scenarios, hybrid algorithms, such
as BBR, due to their more aggressive approach compared to
Delay. Even if one loss-based algorithm is present at the
 300 ·108 1
1

Throughput [bps]
BBR Cubic BBR & Cubic
0.8

Jain index
RTT [ms]

200 0.8
0.6
100 0.4 BBR Cubic
0.2 0.6
0 0
0 20 40 60 0 20 40 60 0 20 40 60
t [s] t [s] t [s]
300 ·108 1
1

Throughput [bps]
Vegas Cubic Vegas & Cubic
0.8

Jain index
RTT [ms]

200 0.8
0.6
100 0.4 Vegas Cubic
0.2 0.6
0 0
0 20 40 60 0 20 40 60 0 20 40 60
t [s] t [s] t [s]
8 ·108 1
1
Throughput [bps]

Vegas BBR Vegas BBR

6 0.8

Jain index
RTT [ms]

0.6 0.8
4
0.4
2 0.2 0.6 BBR & Vegas
0 0
0 20 40 60 0 20 40 60 0 20 40 60
t [s] t [s] t [s]
Fig. 4: BW scenario: Comparison of average RTT, average throughput and fairness index for representatives of the congestion
control algorithm groups in case the link is shared by 2 flows using different algorithms (time unit 300ms).

delay-based algorithms, determine the RTT. Vegas flows, as However, current kernel implementations do not capture these
a consequence, suffer from a small increase in RTT (from improvements.
1.68ms to 4.55ms, Table I). The fairness index for delay-based algorithms slowly in-
Summary. In terms of fairness, the only combination that creases over time, but due to a very conservative congestion
works well together is delay and hybrid algorithms. In such avoidance approach of Vegas, even after 60s, flows do not
a scenario, delay is low and the throughput fairly shared, the converge (Fig. 6). When we increase the number of Vegas
more flows the fairer the distribution of resources. Hybrid, as flows to four, the dynamics at the bottleneck becomes more
well as delay-based algorithms, suffer from a huge increase in complex with the newest flow (with the highest RTT) claiming
the observed delay if even one loss-based algorithm is present the largest share of resources at the end (Fig. 6). Moreover,
at the bottleneck making them unusable in typical networks contrary to the previous scenarios, in the slow start phase,
consisting of many different flows. We observe that the most Vegas flows fill the bottleneck queue and the observed queuing
popular TCP flavour, Cubic, is prone to oscillation and has a delay increases to 70ms. However, after 30s the queues are
high convergence time (≈ 20s). Further, we observe that BBR drained, fairness improves, and the observed queuing delay is
is not stable, reacting to very small changes in the observed very low for all flows (2 − 3ms, Fig. 6).
RTT, which was not previously reported in the literature. Hybrid-based algorithms, such as BBR, favour the flow
with the higher RTT, confirming results from [33], [39].
D. Results: RTT scenario The flow with a higher RTT overestimates the bottleneck
We observe RTT-fairness issues for all three groups of link, claiming all the available resources and increasing the
algorithms. Even though loss-based algorithms such as Cubic queuing delay (Fig. 5) by a factor of more than 10 (from
claim good RTT-fairness properties, they favour the flow with ≈ 4ms to ≈ 50ms). Moreover, when we increase the number
a lower RTT [40]. This is most noticeable when analyzing of BBR flows to four, contrary to expectations, the average
two Cubic flows in Fig. 5. Even when the number of flows in- RTT increases significantly (by a factor of almost 30) reaching
creases to 4 (Fig. 6), the flow with the lowest RTT immediately values comparable to the ones observed by the loss-based
claims all the available resources, leaving less than half to the algorithms in the same scenario although only BBR flows were
other flows in the first 30 s. Several improvements addressing present at the bottleneck (Fig. 6, Table III).
this problem, such as TCP Libra [32] have been proposed. Summary. We observe that RTT-fairness is poor for all
 2 Cubic flows ·108 2 Cubic flows 2 Cubic flows
600 1 1

Throughput [bps]
0ms 200ms
0.8

Jain index
RTT [ms]

400 0.6 0.8

0.4
200
0.2 0.6
0ms 200ms
0 0
0 20 40 60 0 20 40 60 0 20 40 60
t [s] t [s] t [s]
2 Vegas flows ·108 2 Vegas flows 2 Vegas flows
1 1

Throughput [bps]
0ms 200ms
0.8
200

Jain index
RTT [ms]

0.6 0.8
0.4
100 0ms 200ms
0.2 0.6
0 0
0 20 40 60 0 20 40 60 0 20 40 60
t [s] t [s] t [s]
2 BBR flows ·108 2 BBR flows 2 BBR flows
300 1 1
Throughput [bps]

0.8

Jain index
RTT [ms]

200 0.6 0.8

0ms 200ms 0ms 200ms
0.4
100
0.2 0.6
0 0
0 20 40 60 0 20 40 60 0 20 40 60
t [s] t [s] t [s]
Fig. 5: RTT scenario: Comparison of average RTT, average throughput, and fairness index for representatives of the congestion
control algorithm groups in the case the link is shared by 2 flows using the same algorithm (time unit 300ms).

TABLE II: RTT scenario: Different metrics for representatives of the congestion control algorithm groups in case the link is
shared by two flows using the same algorithm (calculated for 5 different runs).
Protocol Group Algorithm Average Average Average Average Average Average
goodput goodput ratio RTT sending rate throughput Jain index
[M bps] [%] [#packets] [ms] [M bps] [M bps]
Cubic(0ms) 65.67 94.07 233.09 75.47 67.88
Loss- vs. Loss-based 0.76
Cubic(200ms) 21.88 93.80 435.53 25.36 22.92
Vegas(0ms) 14.99 94.21 32.03 18.91 15.62
TCP Delay- vs. Delay-based 0.66
Vegas(200ms) 72.60 94.31 228.96 81.48 75.08
BBR(0ms) 8.90 91.98 50.08 9.87 9.24
Hybrid vs. Hybrid 0.56
BBR(200ms) 79.54 94.39 249.56 90.97 82.1

groups of algorithms. Delay-based algorithms are the only E. Results: QUIC

ones that can maintain a low delay compared to the other two
groups. However, they still do not converge towards their fair When QUIC is used with different congestion control algo-
share. Loss-based algorithms such as Cubic perform poorly, rithms, we observe similar interactions as earlier. With BBR,
contrary to expectations and their own claims, favouring flows we observe the same RTT-unfairness properties as with the
with lower RTTs. When loss-based algorithms converge to a TCP BBR, which always favours the flows with a higher RTT
fair share, the convergence time is so slow that the average (with an average fairness index of 0.59). Similarly, QUIC with
fairness index is still low (0.69 on average). Finally, hybrid Cubic always favours the flow with a lower RTT. However, the
algorithms such as BBR suffer from significant dynamics in difference between the throughput of the two QUIC Cubic
the sharing among its own flows, favoring those with higher flows is much smaller than the one observed for the TCP
RTT and significantly increasing the queuing delay. Hence, we equivalent, with an average fairness index of 0.93. In all our
observe that even when only BBR flows are present on the QUIC scenarios where hybrid (BBR) and loss-based (Cubic)
bottleneck, the claim of being able to operate without filling flows compete, Cubic outperforms BBR. Over time, as QUIC
the buffers is not true. BBR flows detect a higher RTT and adopt a more aggressive
 4 Cubic flows ·108 4 Cubic flows 4 Cubic flows
1 1

Throughput [bps]
50ms 100ms
400 0.8 150ms 200ms 0.8

Jain index
RTT [ms]

0.6
200 0.4 0.6
50ms 100ms
150ms 200ms
0.2 0.4
0 0
0 20 40 60 0 20 40 60 0 20 40 60
t [s] t [s] t [s]
4 Vegas flows ·108 4 Vegas flows 4 Vegas flows
1 1

Throughput [bps]
50ms 100ms 50ms 100ms
400 150ms 200ms 0.8 150ms 200ms 0.8

Jain index
RTT [ms]

0.6
200 0.4 0.6
0.2 0.4
0 0
0 20 40 60 0 20 40 60 0 20 40 60
t [s] t [s] t [s]
4 BBR flows ·108 4 BBR flows 4 BBR flows
1 1
Throughput [bps]

50ms 100ms
400 0.8 150ms 200ms 0.8

Jain index
RTT [ms]

0.6
200 0.4 0.6
50ms 100ms
150ms 200ms
0.2 0.4
0 0
0 20 40 60 0 20 40 60 0 20 40 60
t [s] t [s] t [s]
Fig. 6: RTT scenario: Comparison of average RTT, average throughput, and fairness index for representatives of the congestion
control algorithm classes in case the link is shared by 4 flows using the same algorithm (time unit 300ms).

TABLE III: RTT scenario: Different metrics for representatives of the congestion control algorithm classes in case the link is
shared by four flows using the same algorithm (calculated for 5 different runs).
Protocol Group Algorithm Average Average Average Average Average Average
goodput goodput ratio RTT sending rate throughput Jain index
[M bps] [%] [#packets] [ms] [M bps] [M bps]
Cubic(50ms) 47.48 93.86 216.60 53.66 49.59
Cubic(100ms) 15.32 92.39 264.99 17.71 16.09
Loss-based 0.69
Cubic(150ms) 11.70 91.62 316.87 13.62 12.32
Cubic(200ms) 13.68 92.33 368.14 15.78 14.39
Vegas(50ms) 27.32 92.98 94.50 31.09 28.54
Vegas(100ms) 41.85 93.88 144.11 47.13 43.63
TCP Delay-based 0.62
Vegas(150ms) 7.50 90.80 196.87 8.62 7.90
Vegas(200ms) 11.57 91.47 245.18 13.22 12.16
BBR(50ms) 7.11 88.01 203.63 42.56 7.44
BBR(100ms) 15.23 91.61 253.49 21.43 16.06
Hybrid 0.63
BBR(150ms) 22.20 93.59 302.70 18.81 23.45
BBR(200ms) 42.39 94.18 353.22 15.97 44.70

approach, BBR grabs more bandwidth at the expense of the We observed multiple fairness issues, among flows of the
Cubic flows. However, this process is slow and the throughput same group, across different groups, as well as when flows
of the BBR flow remains low. Detailed measurements of QUIC having different RTTs were sharing a bottleneck link. We
can be found in the extended version of this paper [4]. found that delay-based, as well as hybrid algorithms, suffer
from a decrease in performance when competing with flows
IV. C ONCLUSION from the loss-based group, making them unusable in a typical
network where the majority of flows will rely on a loss-based
After dividing existing congestion control algorithms into algorithm. Not only do they get an unfair share of the available
three groups (loss-based algorithms, delay-based algorithms, bandwidth, but they also suffer from a huge increase in the
and hybrid algorithms), we studied their interactions. observed delay when the loss-based algorithms fill the queues.
The only combination that worked well together was delay and [18] L. S. Brakmo, S. W. O’Malley, and L. L. Peterson, TCP Vegas: New
hybrid algorithms: the observed RTT was low and resources techniques for congestion detection and avoidance. ACM, 1994,
vol. 24, no. 4.
shared fairly (the more flows the fairer the distribution of [19] G. Hasegawa, K. Kurata, and M. Murata, “Analysis and improvement of
resources). Finally, we found that hybrid algorithms, such as fairness between TCP Reno and Vegas for deployment of TCP Vegas to
BBR, are very sensitive to changes in the RTT, even if that the Internet,” in Proceedings 2000 International Conference on Network
Protocols, Nov 2000, pp. 177–186.
difference is very small (≤ 0.5ms). They not only favour the [20] K. Srijith, L. Jacob, and A. L. Ananda, “TCP Vegas-A: Improving the
flow with a higher RTT at the expense of the other flows, but performance of TCP Vegas,” Computer communications, vol. 28, no. 4,
they also cannot maintain a low queuing delay as promised pp. 429–440, 2005.
[21] C. Jin, D. Wei, S. H. Low, J. Bunn, H. D. Choe, J. C. Doylle,
even if they are the only flows present in the network. H. Newman, S. Ravot, S. Singh, F. Paganini, G. Buhrmaster, L. Cottrell,
Our work therefore shows that to support applications that O. Martin, and W. chun Feng, “FAST TCP: from theory to experiments,”
require low latency, a good congestion control algorithm on IEEE Network, vol. 19, no. 1, pp. 4–11, Jan 2005.
[22] S. Belhaj and M. Tagina, “VFAST TCP: An improvement of FAST
its own won’t be enough. Indeed, guaranteeing that flows of TCP,” in Computer Modeling and Simulation, 2008. UKSIM 2008. Tenth
a given group (in terms of type of congestion control) will International Conference on. IEEE, 2008, pp. 88–93.
receive their expected share of resources, requires that resource [23] J. Sing and B. Soh, “TCP New Vegas: improving the performance
of TCP Vegas over high latency links,” in Network Computing and
isolation be provided between the different groups. Applications, Fourth IEEE International Symposium on. IEEE, 2005,
pp. 73–82.
R EFERENCES [24] C. P. Fu and S. C. Liew, “TCP Veno: TCP enhancement for transmission
[1] N. Cardwell, Y. Cheng, C. S. Gunn, S. H. Yeganeh, and V. Jacobson, over wireless access networks,” IEEE Journal on selected areas in
“BBR: Congestion-Based Congestion Control,” Queue, vol. 14, no. 5, communications, vol. 21, no. 2, pp. 216–228, 2003.
pp. 20–53, 2016. [25] R. King, R. Baraniuk, and R. Riedi, “TCP-Africa: An adaptive and fair
[2] M. Hock, F. Neumeister, M. Zitterbart, and R. Bless, “TCP LoLa: rapid increase rule for scalable TCP,” in INFOCOM 2005. 24th Annual
Congestion Control for Low Latencies and High Throughput,” in 2017 Joint Conference of the IEEE Computer and Communications Societies.
IEEE 42nd Conference on Local Computer Networks (LCN), 2017, pp. Proceedings IEEE, vol. 3. IEEE, 2005, pp. 1838–1848.
215–218. [26] K. Tan, J. Song, Q. Zhang, and M. Sridharan, “A Compound TCP
[3] R. Mittal, V. T. Lam, N. Dukkipati, E. Blem, H. Wassel, M. Ghobadi, Approach for High-Speed and Long Distance Networks,” in Proceed-
A. Vahdat, Y. Wang, D. Wetherall, and D. Zats, “TIMELY: RTT-based ings IEEE INFOCOM 2006. 25TH IEEE International Conference on
Congestion Control for the Datacenter,” pp. 537–550, 2015. Computer Communications, 2006, pp. 1–12.
[4] B. Turkovic, F. A. Kuipers, and S. Uhlig, “Fifty shades of congestion [27] A. Baiocchi, A. P. Castellani, and F. Vacirca, “YeAH-TCP: yet another
control: A performance and interactions evaluation,” arXiv preprint highspeed TCP,” in Proc. PFLDnet, vol. 7, 2007, pp. 37–42.
arXiv:1903.03852, 2019. [28] S. Liu, T. Başar, and R. Srikant, “TCP-Illinois: A loss-and delay-based
[5] J. Postel, “Transmission control protocol specification,” RFC 793, 1981. congestion control algorithm for high-speed networks,” Performance
[6] M. Allman, V. Paxson, and E. Blanton, “TCP Congestion Control,” RFC Evaluation, vol. 65, no. 6-7, pp. 417–440, 2008.
5681 (Draft Standard), Internet Engineering Task Force, September [29] H. Shimonishi and T. Murase, “Improving efficiency-friendliness trade-
2009. [Online]. Available: http://www.ietf.org/rfc/rfc5681.txt offs of TCP congestion control algorithm,” in Global Telecommunica-
[7] S. Floyd, “HighSpeed TCP for large congestion windows,” Tech. Rep., tions Conference, 2005. GLOBECOM’05. IEEE, vol. 1. IEEE, 2005,
2003. pp. 5–pp.
[8] D. Leith and R. Shorten, “H-TCP: TCP for high-speed and long-distance [30] K. Kaneko, T. Fujikawa, Z. Su, and J. Katto, “TCP-Fusion: a hybrid con-
networks,” in Proceedings of PFLDnet, vol. 2004, 2004. gestion control algorithm for high-speed networks,” in Proc. PFLDnet,
[9] T. Kelly, “Scalable TCP: Improving performance in highspeed wide area vol. 7, 2007, pp. 31–36.
networks,” ACM SIGCOMM computer communication Review, vol. 33, [31] H. Shimonishi, T. Hama, and T. Murase, “TCP-adaptive reno for
no. 2, pp. 83–91, 2003. improving efficiency-friendliness tradeoffs of TCP congestion control
[10] S. Mascolo, C. Casetti, M. Gerla, M. Y. Sanadidi, and R. Wang, “TCP algorithm,” in Proc. PFLDnet. Citeseer, 2006, pp. 87–91.
westwood: Bandwidth estimation for enhanced transport over wireless [32] G. Marfia, C. Palazzi, G. Pau, M. Gerla, M. Sanadidi, and M. Roccetti,
links,” in Proceedings of the 7th annual international conference on “Tcp libra: Exploring rtt-fairness for tcp,” in International Conference
Mobile computing and networking. ACM, 2001, pp. 287–297. on Research in Networking. Springer, 2007, pp. 1005–1013.
[11] L. A. Grieco and S. Mascolo, “Performance evaluation and comparison [33] D. Scholz, B. Jaeger, L. Schwaighofer, D. Raumer, F. Geyer, and
of Westwood+, New Reno, and Vegas TCP congestion control,” ACM G. Carle, “Towards a Deeper Understanding of TCP BBR Congestion
SIGCOMM Computer Communication Review, vol. 34, no. 2, pp. 25–38, Control,” in IFIP Networking 2018, Zurich, Switzerland, May 2018.
2004. [34] M. Hock, R. Bless, and M. Zitterbart, “Experimental evaluation of BBR
[12] K. Yamada, R. Wang, M. Y. Sanadidi, and M. Gerla, “TCP westwood congestion control,” in 2017 IEEE 25th International Conference on
with agile probing: dealing with dynamic, large, leaky pipes,” in Network Protocols (ICNP), Oct 2017, pp. 1–10.
2004 IEEE International Conference on Communications (IEEE Cat. [35] S. Ma, J. Jiang, W. Wang, and B. Li, “Towards RTT Fairness of
No.04CH37577), vol. 2, June 2004, pp. 1070–1074 Vol.2. Congestion-Based Congestion Control,” CoRR, vol. abs/1706.09115,
[13] D. Kliazovich, F. Granelli, and D. Miorandi, “Logarithmic window 2017. [Online]. Available: http://arxiv.org/abs/1706.09115
increase for TCP Westwood+ for improvement in high speed, long [36] M. Dong, Q. Li, D. Zarchy, P. B. Godfrey, and M. Schapira, “{PCC}:
distance networks,” Computer Networks, vol. 52, no. 12, pp. 2395–2410, Re-architecting congestion control for consistent high performance,”
2008. in 12th {USENIX} Symposium on Networked Systems Design and
[14] L. Xu, K. Harfoush, and I. Rhee, “Binary increase congestion control Implementation ({NSDI} 15), 2015, pp. 395–408.
(BIC) for fast long-distance networks,” in INFOCOM 2004. Twenty-third [37] R. K. Jain, D.-M. W. Chiu, and W. R. Hawe, “A Quantitative Measure
AnnualJoint Conference of the IEEE Computer and Communications of Fairness and Discrimination,” Eastern Research Laboratory, Digital
Societies, vol. 4. IEEE, 2004, pp. 2514–2524. Equipment Corporation, Hudson, MA, 1984.
[15] C. Caini and R. Firrincieli, “TCP Hybla: a TCP enhancement for het- [38] “The chromium projects: Chromium,” https://www.chromium.org/
erogeneous networks,” International journal of satellite communications Home, accessed: 04-03-2019.
and networking, vol. 22, no. 5, pp. 547–566, 2004. [39] M. Hock, R. Bless, and M. Zitterbart, “Experimental evaluation of bbr
[16] S. Ha, I. Rhee, and L. Xu, “CUBIC: a new TCP-friendly high-speed congestion control,” in 2017 IEEE 25th International Conference on
TCP variant,” SIGOPS Oper. Syst. Rev., vol. 42, no. 5, pp. 64–74, 2008. Network Protocols (ICNP). IEEE, 2017, pp. 1–10.
[17] A. Afanasyev, N. Tilley, P. Reiher, and L. Kleinrock, “Host-to-host [40] T. Kozu, Y. Akiyama, and S. Yamaguchi, “Improving rtt fairness on
congestion control for TCP,” IEEE Communications surveys & tutorials, cubic tcp,” in 2013 First International Symposium on Computing and
vol. 12, no. 3, pp. 304–342, 2010. Networking, Dec 2013, pp. 162–167.