Web Performance Pitfalls

Web performance is widely studied in terms of load times, numbers of objects, object sizes, and total page sizes. However, for all these metrics, there are various definitions, data sources, and measurement tools. These often lead to different results and almost all studies do not provide sufficient details about the definition of metrics and the data sources they use. This hinders reproducibility as well as comparability of the results. This paper revisits the various definitions and quantifies their impact on performance results. To do so we assess Web metrics across a large variety of Web pages.

Amazingly, even for such “obvious” metrics as load times, differences can be huge. For example, for more than 50% of the pages, the load times vary by more than 19.1% and for 10% by more than 47% depending on the exact definition of load time. Among the main culprits for such difference are the in-/exclusion of initial redirects and the choice of data source, e.g., Resource Timings API or HTTP Archive (HAR) files. Even “simpler” metrics such as the number of objects per page have a huge variance. For the Alexa 1000, we observed a difference of more than 67 objects for 10% of the pages with a median of 7 objects. This highlights the importance of precisely specifying all metrics including how and from which data source they are computed.

1. 1.

   See Fig. 2 for an overview and Appendix A for more explanation.

2. 2.

   For details regarding the methodology and the corresponding dataset see Sect. 4.

3. 3.

   Our tools are publicly available at https://github.com/theri/web-measurement-tools.

4. 4.

   See http://gs.statcounter.com/browser-market-share/desktop/worldwide.

5. 5.

   Chrome and Opera support it, Firefox is still validating their implementation.

6. 6.

   https://chromedevtools.github.io/devtools-protocol/.

7. 7.

   https://firefox-source-docs.mozilla.org/testing/marionette/marionette.

8. 8.

   Note that for HTTP/2, logged header sizes do not correspond to bytes on the wire anymore due to HTTP/2 header compression.

9. 9.

   Our scripts instrument browser automation frameworks directly to give us more control and avoid the overhead of an integrated framework such as WebPagetest.

10. 10.

    For realistic browser behavior, which includes the rendering engine, we open Web browsers including the graphical user interface rather than using them in headless mode.

11. 11.

    Close in terms of network distance.

12. 12.

    18. September 2018 for Alexa 1000 and 30. September 2018 for Alexa 10001–11000.

13. 13.

    We instruct the browser automation tool to wait for the onLoad event.

14. 14.

    See, e.g., https://support.mozilla.org/en-US/kb/search-web-address-bar.

15. 15.

    Server-side redirects use HTTP status 301 or 302. Client-side redirects use status 200 and contain the redirection URL in the response content, which we do not log.

16. 16.

    For Navigation Timings, redirects are the time between navigationStart and fetchStart. For HAR files, we use the time before the first HTTP 200 response.

17. 17.

    In September and early October 2018, TLS 1.3 was still not deployed.

18. 18.

    See analysis script eval/validate_object_sizes.py in our repository.

19. 19.

    Unless the ‘Timing-Allow-Origin’ header is set, see [5].

20. 20.

    In Firefox, only HTTP/2 Server Push objects lack body size and timings.

21. 21.

    See the examples in Sect. 4.2 of [5].

22. 22.

    Using the Page.frameStoppedLoading event instead did not resolve this problem.

23. 23.

    Examples include microsoftonline.com and googleusercontent.com.

24. 24.

    After a redirect, the browser overwrites the old fetchStart value before it fetches the new URL using a GET request. This implies that once the page load is finished, fetchStart is the start time of the loading of the final base page, as all previous values related to redirects are overwritten.

Thanks to Dominik Strohmeier for the discussion and the pointers to resources, to our shepherd Jelena Mirkovic, as well as our anonymous reviewers.

Authors and Affiliations

1. TU Berlin, Berlin, Germany

   Theresa Enghardt & Thomas Zinner

2. Max-Planck Institute for Informatics, Saarbrücken, Germany

   Anja Feldmann

Corresponding author

Correspondence to Theresa Enghardt .

Editors and Affiliations

1. Northeastern University, Boston, MA, USA

   David Choffnes

2. Federal University of Rio Grande do Sul, Porto Alegre, Rio Grande do Sul, Brazil

   Marinho Barcellos

A Web Page Load Explained

In this section, we explain a Web page load in more detail. See also Fig. 2 and the processing models in the Navigation Timings specifications [2, 3].

The starting point for a new page load, also called navigation, of a particular URL, is called navigationStart in [2]. Initially, fetchStart is set to the same value, but if a redirect occurs, fetchStart is overwritten before the new URL is loaded.

If another page has been previously loaded by the browser, e.g., in the same browser tab, this document has to be first unloaded. Then, the browser checks its cache to see whether the page is already there. If the page is not in the cache, the browser usually resolves the hostname (resulting in a DNS query and usually answer), establishes a TCP connection, and performs a TLS handshake if the scheme of the URL is https. Then, the browser issues an HTTP GET request for the URL. As soon as it receives an HTTP reply, which always contains a status line, headers, and body, the browser processes the reply.

If the reply contains an HTTP status code of 3xx, such as “301 Moved Permanently” or “302 Found”, this means that the server redirects the browser to a different URL, which is given in the “Location” header in the HTTP response. This redirect may be a same-origin redirect, which roughly means that both the old and the new URL have the same scheme (http or https), hostname, and port (see RFC 6454 [29] for details), or it may be a cross-origin redirect. For same-origin redirects, the start and end time of the redirect are recorded as Navigation Timings redirectStart and redirectEnd [2], while for cross-origin redirects they are not. Unfortunately, nearly all redirects we observed are cross-origin, as the purpose of the redirect is to use a different scheme (HTTPS instead of HTTP) or hostname (www.example.com instead of example.com). The same-origin policy is an important security and privacy feature in the Web, so information access is often restricted to, e.g., the same hostname.

Given the new URL to be fetched, the browser records the current time as fetchStart, potentially overwriting the old value^{Footnote 24}. It then checks its application cache again, resolves the host name if needed, establishes a new TCP connection, performs a new TLS handshake, and sends an HTTP request for the new URL. If it gets an HTTP reply, this may be another redirect, an error code such as “404 Not Found” or“503 Internal Server Error”, or the request may succeed with a “200 OK”. In the latter case, the body of the HTTP response usually contains the base document of the Web page in HyperText Markup Language (HTML). As soon as the browser starts receiving this document, it parses it and starts constructing the Document Object Model (DOM) of the page. For example, the document may reference additional resources, such as JavaScript, Cascading Stylesheets (CSS), or images. Typically, for each of these additional resources, the browser has to issue a new HTTP request, unless the resource is proactively sent by the server using HTTP/2 Server Push. Each new HTTP request may involve an additional name resolution, TCP handshake, and TLS handshake, because resources are often hosted on different servers than the base page. The browser now simultaneously fetches new resources, continues to parse the HTML base page, and processes the CSS and Javascripts, even though these processes may block each other. See Wang et al. [16] for a detailed explanation of this complex process.

At some point, the browser flushes the current state of the DOM to the rendering engine. The time at which this happens corresponds to Time To First Paint (TTFP). The point at which all resources in the DOM have been loaded is called DOMContentLoaded and recorded in the Navigation Timings and HAR file. However, processing of the page usually continues, until, eventually, the browser fires the onLoad event for the page which is recorded in the Navigation Timings and HAR file. The onLoad Time is usually taken as Page Load Time (PLT). At this point, the page load is considered finished. However, onLoad usually triggers the execution of one or more javascripts, which may result in loading more resources, sending data, e.g., to third parties, or other network traffic. In fact, most modern Web pages load resources continuously long after the onLoad event. Thus, Related Work usually stops counting objects after onLoad.

B Details of Lessons Learned

Next, we outline additional details regarding our lessons learned, which led to our guidelines for Web performance measurement, recall Sect. 6.

Software Versions: The Debian Linux distribution includes a version of the Firefox browser which is usually quite dated. This can have a major impact on load times. For instance, in Firefox version 61 (“Firefox Quantum”), parts of the code have been rewritten and optimized, which makes the browser much faster than previous versions. Consequently, carrying out Web page loads using an older version results in unrealistically long load times. However, updating Firefox frequently to the newest version can result in incompatibilities with measurement tools. For instance, not every version of the HAR Export Trigger extension works with every version of Firefox, so it has to be updated along with the browser. However, the upside is that in newer versions of Firefox, HAR Export Trigger is supposed to work without having the developer panel open.

Browser Traffic Unrelated to Page Loads: Modern browsers usually issue a significant number of requests that are not directly related to the page load that a user has requested. For instance, Firefox by default loads blocklists for “safe browsing”, to protect users from malware or phishing. It also automatically checks for updates and may even automatically download and install these updates for the entire browser or for individual browser extensions. These queries can involve substantial data transfers: For example, we observed the automatic download of a binary related to an H264 media component which we never activated or requested: 500 KB were downloaded in the background. Worse yet, the state of such updates is often stored in the browser profile. This may cause such downloads to be triggered for every fresh browser profile, i.e., each of our page loads. Additionally, the Chrome browser by default issues queries to various Google servers, e.g., it tries to connect each browsing session to a Google account. We provide configurations for Firefox and Chrome to turn off most features that generate such traffic, see our repository https://github.com/theri/web-measurement-tools.

Logging a Trace and Client-Side SSL Keys: To be able to better debug and validate measurement setups and tools, we recommend capturing packet traces that include at least ports 53 (DNS), 80 (HTTP), and 443 (HTTPS). Encrypted traffic can be decrypted after logging the SSL session keys within the browser: Firefox and Chrome log keys into a specified SSLKEYLOGFILE. Note that this option must be compiled into Firefox. It, e.g., does not work with the Firefox binary in the Debian repositories.

C Artifacts Related to This Paper

The following artifacts are available:

Our Tools, Such as Measurement and Evaluation Scripts: See https://github.com/theri/web-measurement-tools. This repository includes the scripts to automatically load Web pages using Firefox with Selenium and Marionette, and using Chrome with DevTools. Furthermore, it includes the analysis scripts we used to generate our plots.

Data Set of Web Page Loads: See http://dx.doi.org/10.14279/depositonce-8100. This dataset includes data from all of our experiment runs, see Sect. 4. It can be used along with our evaluation scripts to reproduce the plots in this paper, see https://github.com/theri/web-measurement-tools for details.

Reprints and permissions

© 2019 Springer Nature Switzerland AG

Policies and ethics