Nothing Special   »   [go: up one dir, main page]
Next Article in Journal
Human Activity Recognition Based on Two-Channel Residual–GRU–ECA Module with Two Types of Sensors
Previous Article in Journal
Proton Radiation Effects of CMOS Image Sensors on Different Star Map Recognition Algorithms for Star Sensors
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Electronics
Volume 12
Issue 7
10.3390/electronics12071630
electronics-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Mobile Broadband Adoption, Performance Measurements and Methodology: A Review

by
Vincent Umoh
1,*,
Unwana Ekpe
2,
Innocent Davidson
3 and
Joseph Akpan
4
1
Department of Electrical Power Engineering, Durban University of Technology, Durban 4001, South Africa
2
Department of Electrical and Electronic Engineering, Akwa Ibom State University, Mkpat Enin 524106, Nigeria
3
Department of Electrical, Electronic and Computer Engineering, Cape Peninsula University of Technology, Bellville 7535, South Africa
4
Department of Industrial Engineering, Durban University of Technology, Durban 4001, South Africa
*
Author to whom correspondence should be addressed.
Electronics 2023, 12(7), 1630; https://doi.org/10.3390/electronics12071630
Submission received: 12 March 2023 / Revised: 27 March 2023 / Accepted: 28 March 2023 / Published: 30 March 2023
(This article belongs to the Section Microwave and Wireless Communications)
Browse Figures
Figure 1
<p>Active mobile broadband subscription per 100 inhabitants: (<b>a</b>) by development status; (<b>b</b>) by region.</p> "> Figure 2
<p>The four QoS viewpoints.</p> "> Figure 3
<p>A typical methodology for MBB drive test [<a href="#B40-electronics-12-01630" class="html-bibr">40</a>].</p> "> Figure 4
<p>Smartphones mounted in moving vehicles during drive test [<a href="#B40-electronics-12-01630" class="html-bibr">40</a>].</p> "> Figure 5
<p>Bag pack with six mounted smartphones.</p> "> Figure 6
<p>Mobile applications during measurement: (<b>a</b>) Speedtest; (<b>b</b>) SamKnows.</p> "> Figure 7
<p>The overall system architecture of the NNE platform [<a href="#B52-electronics-12-01630" class="html-bibr">52</a>].</p> "> Figure 8
<p>NNE node with 4 modems connected [<a href="#B52-electronics-12-01630" class="html-bibr">52</a>].</p> "> Figure 9
<p>Overview of the MONROE platform.</p> "> Figure 10
<p>Overview of the Simplified Raspberry Pi platform [<a href="#B50-electronics-12-01630" class="html-bibr">50</a>].</p> "> Figure 11
<p>MBB measurements challenges.</p> ">
Versions Notes

Abstract

:
Mobile broadband (MBB) penetration has deepened globally over the last twenty years. This is largely due to the adoption of smart devices, improved mobile communications network coverage, and the perpetual drive to develop ever faster mobile and wireless communication technologies. However, information on the quality of service (QoS) delivered by MBB operators to the end users remains an issue of concern. This has driven independent researchers and mobile communication industry regulators to develop methodologies for independent and unbiased evaluation of the QoS offered by MBB networks. This paper provides a detailed review of MBB adoption and penetration across several regions of the world. It also includes the existing methodologies for evaluating the performance of MBB systems as experienced by the end user. Specifically, methodologies such as the drive and walk tests, crowd-sourced mobile device-based methods and the software applications they employ, and the dedicated measurement testbeds are reviewed. Based on this, the challenges of adopting each of the methods are discussed in order to make a case for the development of more robust, partially autonomous and scalable MBB measurement platforms for the future.

1. Introduction

Access to mobile broadband (MBB) has caused a phenomenal increase in the number of Internet subscribers over the years [1]. This is because faster internet access through MBB allows for an increasing number of services that were hitherto rendered by physical contact to be provided virtually. This growth was recently accelerated with the emergence of the COVID-19 pandemic, where some countries witnessed up to a 13.3% uptake in Internet connectivity, while some other countries saw a marginal decline [2]. From data presented in [1], the increase in Internet access is mostly driven by an expansion of MBB coverage and this has impacted positively the social and economic wellbeing of the global society [1]. Several studies including [3,4,5,6,7,8] have assessed this impact and reported that there is a direct correlation between access to broadband and the human development index in the countries and territories where such access is provided.
MBB is delivered to the end users mostly by terrestrial mobile network operators (MNOs), who are responsible for building adequate infrastructure for optimum quality of service (QoS) delivery. However, some of these MNOs do not deliver the speeds they advertise nor the theoretical speeds defined by the respective standards of the 3rd Generation Partnership Project (3GPP) [9]. Given this scenario and the importance of MBB networks, it is necessary for independent and unbiased performance analyses of the services of existing MNOs as experienced by the end users, to be conducted periodically. Such information would assist the MNOs to improve the capabilities of their MBB networks to be able to deliver service at an acceptable QoS. The evaluation of end user experience QoS is usually carried out through systematic end-to-end measurements using dedicated infrastructure and standard methodologies. In spite of numerous analyses (MBB) conducted over the years across different parts of the world, there is still a dearth of published literature that examines the infrastructure used and the methodologies adopted for such MBB performance assessments. The purpose of this paper is to present a current global MBB penetration and a review of existing infrastructure and methodologies developed around the world for MBB performance analysis. General challenges associated with MBB performance assessment are discussed, and the conclusion provides a direction for future research efforts. It is worth noting that the context of this paper is limited to measurement and evaluation of the performance of end-to-end MBB networks viewed from a user centric perspective. The balance of the paper examines the challenges associated with MBB measurements. The paper is structured as follows: Section 2 discusses the evolution of MBB. Section 3 examines the global penetration of MBB in order to make a case for the need for frequent assessment of MBB performance. Section 4 discusses QoS and customer quality of experience (QoE). Section 5 presents performance metrices, methodologies, testbeds and tools used in determining QoS and QoE for MBB performance evaluation. Section 6 reviews existing MBB performance measurement studies and projects, while Section 7 briefly discusses the challenges associated with MBB measurement. Section 8 is the conclusion.

2. Evolution of Mobile Broadband

Mobile broadband was birthed when the 2nd generation (2G) mobile communication systems evolved to 2.5G (3GPP Release 98 [9]) from the initial iteration deployed circa 1991 [10,11]. This was when the theoretical gross bit rate of the system exceeded the International Telecommunication Union (ITU) definition for broadband [12], achieving a maximum rate of 384 kpbs in the downlink direction. Ever since, the 3rd, 4th and 5th generation (3G, 4G and 5G) mobile communication systems have progressively achieved faster speeds, as shown in Table 1.
Table 1 shows the technologies used by various generations of mobile communication technologies. For example, 2G systems include General Packet Radio Service (GPRS) and Enhanced Data Rate for GSM Evolution (EDGE) [13,14]. Other technologies considered as part of 2G include the CDMA-based IS-95 and the digital version of the Advanced Mobile Phone System (D-AMPS) [15,16]. These additional technologies set the motion for 2G-based Internet services in 2G by offering enhanced data rates, enabling wireless devices to access the Internet and delivering improved QoS for voice and data services [17,18].
Due to the demand for better features and services, and industry drive for improved spectral and energy efficiency, newer generation of mobile communication networks have become a necessity. In Table 1, these newer systems have been classified as 3G, 4G and 5G. 3G consist of technologies such as Wideband Code Division Multiple Access (WCDMA), High-Speed Packet Access (HSPA) and Evolved HSPA (HSPA+). Please note that the standard for each technology is defined by the 3GPP and the ITU. 3G also uses CDMA in standardized family format as CDMA2000 and CDMA200 1xEv-DO (Evolution-Data Optimize) [19,20,21,22].
The 4G network was developed to deliver greater capacity for a faster and better MBB experience. It has the Long-Term Evolution (LTE), Advanced LTE (LTE-A) and Worldwide Interoperability for Microwave Access (WiMAX 2) as its main technologies along with others [23,24,25,26] listed in Table 1. The 5G mobile network, which has recently been deployed in many countries and territories offers enhanced mobile broadband (eMBB), massive machine-type communications (mMTC), and ultra-reliable and low-latency communications (URLLC) [27,28,29,30]. 5G uses Orthogonal Frequency Division Multiple Access (OFDMA) and Non-Orthogonal Multiple Access (NOMA) as its core technology in addition to the technologies used by the previous generations [28,31,32]. This is in line with the ITU requirements for the International Mobile Telecommunications for 2020 and beyond (IMT-2020) [31,32,33,34,35].
A defining characteristic of eMBB, being one of the three core 5G features, is its theoretical potential to achieve a gross throughput of up to 20 Gbps as shown in Table 1. According to [33], eMBB would be able to support data intensive applications such as ultra-high definition video streaming, and virtual and augmented reality applications.

3. Mobile Broadband Adoption

According to ITU [36], the deployment and utilization of MBB have witnessed significant growth over the years. As summarized in Table 2, the number of active MBB subscriptions worldwide has grown by up to 99% when data from year 2015 and 2021 are compared. This growth is even more significant in least developed countries (LDCs), being up to 198%. For the country groupings in Table 2, the other acronyms LLDC and SIDS represent land-locked developing countries and small island developing states, respectively. The growth trend per 100 inhabitants is graphically illustrated in Figure 1a, where it can be observed that there is a huge gap between broadband penetration in developed countries and the rest of the world.
Table 3 uses regional groupings according to the ITU’s Telecommunication Development Bureau [37] to show the growth in active MBB subscriptions. The table reveals that the largest growth, occurring between years 2015 and 2021 was 148% and this was in Africa. The Asia-Pacific region at 142% came second while the Commonwealth of Independent States (CIS) saw a 61% growth.
Figure 1b bar charts display the active MBB subscriptions per 100 inhabitants by region. It should be noted that even with the massive disruptions in broadband usage brought about by the COVID-19 pandemic [2], Africa recorded the least number of subscriptions per 100 inhabitants. In numerical terms, the Asia-Pacific and the Americas region have the highest numbers (Table 3) but Europe still dwarfs these regions in terms of penetration (Figure 1b).
The Organization for Economic Co-operation and Development (OECD) also collates and presents yearly MBB data for its 38 member countries [38]. Published data show that the group has a total of over 1.7 billion active MBB subscriptions as of December 2021 (see Table 4). It also indicates a remarkable year-on-year penetration increase for year 2020 to 2021, averaged at 6.15, as shown in Table 4. Since MBB is mostly accessed through smart phones, a report by GSMA [39] has provided an informative projection of figures regarding smart phone adoption for the years, 2021 to 2025. The report predicts a 9% increase in global adoption, and this is equivalent to 7.5 billion smart phone connections, which would mostly be driven by frontier markets in sub-Saharan Africa and the Asia-Pacific regions.
Consequently, this progressive MBB penetration and the proliferation of mobile devices across the globe make a case for a regular systematic end-to-end performance assessment of MBB networks. In doing so, the parameters used to characterize the MBB performance need to be understood.

4. Quality of Service and Quality of Experience

MBB services and the underlying terrestrial telecommunications networks are gradually being designed to consider the end-to-end performance needed by the user’s application. In measuring the quality of MBB services, QoS and QoE are the two predominant measures that are often adopted. However, in recent times, the mean opinion score MOS has also been given more attention [40]. This section reviews the definition of the two most widely adopted terms for end-to-end performance assessment of MBB networks, being the quality of service (QoS) and the quality of experience (QoE).
Quality of Service (QoS) is a term that is broadly used in the telecommunication world but increasingly gaining traction with regards to MBB. QoS is the overall quality of the applications experienced by the network users. According to the ITU [41,42], “Quality of Service (QoS) is the totality of characteristics of a telecommunications service that bear on its ability to satisfy stated and implied needs of the user of the service”. Therefore, end-to-end latency, jitter, packet loss and throughput are key QoS parameters adopted to measure the performance of MBB networks [43]. From Figure 2, QoS has four points of view when adopted within the concept of service quality. The four points of view include QoS offered by provider, QoS achieved by provider, QoS perceived by subscriber, and the subscriber’s QoS requirements [44]. The subscriber’s QoS requirements state the level of quality that a specific service requires, which may be expressed in non-technical languages with the subscribers particularly concerned about the resulting end-to-end service quality and not the internal design of the network. QoS offered by the service provider accounts for the level of quality expected to be offered to the customer by the service provider, which maybe expressed in non-technical terms for customers to understand, and in technical terms for use within the business. QoS perceived by the subscribers expresses the level of quality experienced that the subscribers believe they have experienced, which is usually expressed in terms of degrees of satisfaction and not in technical terms. QoS achieved by the service provider accounts for the level of quality truly achieved and delivered to the subscribers, which is used by the industry and sometimes by regulators, for publication in the interests of subscribers [44].
QoE involves the end-user in the overall quality assessment and satisfaction in terms of usability, accessibility and service integrity. ITU in [41,45] defines QoE as “the degree of delight or annoyance of the user of an application or service”. QoE is not only limited to technical network performance but also non-technical aspects that affect the perception and satisfaction of the user as highlighted in Table 5 [43,46]. Recent studies such as [47] have also shown how QoE and QoS can depend on each other in the overall telecommunication network system (OTS). Four normalization techniques were used to predict the QoE parameters based on only the QoS indicators in the OTS.

5. Mobile Broadband Performance Evaluation Metrics, Methodologies and Tools

This section is concerned with reviewing the available evaluation metrics and the methodologies and tools used in assessing mobile broadband performance from a user-centric perspective.

5.1. Performance Evaluation Metrics

There are different metrics for measuring and assessing the performance of MBB network operators. The key metrics include throughput, latency, packet loss and jitter. Throughput is the actual amount of data that are successfully sent or received over a communication network or link. It is measured both in the uplink and downlink and presented in kbps, Mbps or even higher values such as Gbps [46,48].
Latency, also called delay, is the time it takes for a data packet to be transmitted from its source to its expected destination. Latency is measured in milliseconds and depends on the type of network access, protocol, packet loss as well as network configuration of the MNOs. Latency portrays how responsive a network is [46,49]. Jitter is related to latency. It is described as the difference or inconsistencies in latency between end-to-end packet flow. Jitter is measured in milliseconds [50].
Packet loss is a QoS metric that describes the percentage of packets of data not reaching their destination after they have been sent across a network. Packet loss is mostly caused by network congestion and is expressed as a percentage [50].
In analyzing different MBB networks and use cases, inferences should not be drawn in isolation based on individual results of the listed performance metrics. This is due to the complex relationship between the different performance metrics and the physical layer parameters such as distance between access points and user equipment, prevailing channel fading conditions, modulation and channel coding schemes employed, signal-to-interference and noise ratio, and the time of measurements. These effects individually and collectively influence the results of the performance evaluation.

5.2. Performance Evaluation Methodologies

To measure the MBB performance delivered to the end user, a systematic end-to-end approach is usually preferred [51]. In some cases, a dedicated testbed is developed while in other cases, simplified methods that use software and mobile applications are adopted. Regulatory agencies, operators or independent researchers can carry out drive tests and walk tests to identify coverage gaps and performance problems. The major disadvantages of these kinds of tests are high costs and poor scalability [52,53]. Another approach is to depend on the end user to initiate performance measurement by visiting a website or by running performance measurement applications preinstalled on their mobile device [54]. Even though this method is scalable, there can arise issues of privacy, and bias in measurements by participants. Additionally, this method often lacks vital context information and metadata that is essential in putting the MBB measurement result in the right context.
The challenges with the aforementioned methods birthed the adoption of testbed-based MBB measurement platforms. These testbeds are robust enough to allow for controlled, scalable measurements that can span over a long period of time. Whichever approach is adopted depends on what metrics are to be assessed and what data are required, as all the approaches have their advantages and disadvantages. This subsection therefore provides an overview of the methodologies and approaches that can be adopted for MBB performance assessment.

5.2.1. Drive Test

A drive test is usually carried out to evaluate and objectively compare the capacity, coverage and quality of service (QoS) provided by mobile networks [40]. Such tests involve mounting measurement equipment inside moving vehicles to systematically collect measurements [40]. Important features of drive testing are the need for consistency in the collection of measurement data and minimizing the variation in as many factors as possible that might influence the measurement results [53]. Additionally, performance impacting variables such as collection of samples that reflects consistent speed, time-of-day, application use or device configuration are controlled to improve the ability of researchers to draw inferences. Drive tests provide a common picture of the QoS of the mobile user over a given geographical area. There are fundamentally two main formats for performing drive tests.
The first is a user equipment-based testing where measurements are performed using a typical user-equipment such as smart phones. The second method engages specialized receivers for measurement and benchmarking [40,53]. There is no defined methodology or standard for conducting a drive test. However, Figure 3 shows a typical configuration of drive test [40]. Usually, an MBB testing application is installed in identical smartphones, each dedicated to a particular MNO as shown in Figure 4 [40]. The smartphones are mounted in a moving vehicle and programmed to take measurements while the vehicle is in motion and to evaluate several performance metrics relating to user experience. The data collected are stored in the mobile device, which are retrieved and analyzed in real time or at the end of the measurement campaign. The major disadvantage of this kind of MBB measurement is that it is expensive and may not be scalable.

5.2.2. Walk Test

The walk test is another type of MBB measurement that is setup in a similar way to drive testing. However, walk tests involve walking along typical locations such as subways, pedestrians, sports stadiums, airports and malls and taking network measurements. The test equipment for a walk test is affixed to the back of the test taker or placed in a carrying harness during measurement periods [53]. Figure 5 shows an example of such a carrying case with mounting prepared to firmly carry six mobile phones. The cell phones are programmed to take performance measurements for the different MNOs through an application installed in them, as the carrier walks around defined areas. Walk tests are more expensive than drive tests as the cost of collecting data per unit area is high. The test is also limited by the walk speed and distance covered by the carrier of the test device.

5.2.3. Crowdsourced Method

The crowdsourced software application-based approach to MBB performance evaluation is a large-scale measurement method that typically depends on voluntary participation by the end user [49,55]. Mobile apps are installed on the participant’s smartphones for MBB performance assessment. They are two basic approaches to this type of MBB performance measurement. The first is the app-based panel testing, where participants known as the panel of testers have the app installed and running on their smartphones. An autonomous agent then initiates the test and collects the data in the background without the knowledge of the user [53,55]. The Federal Communication Commission (FCC) in the US is one regulator that uses this panel-based approach. The second approach relies on users to voluntarily install the app and also, initiate the test.
The crowdsourced app-based approach is cost-effective, scalable and can cover a larger geographic area than other test methods. However, different MNOs cannot be tested at the same time, and relying on the user to initiate the test may not be very effective. Privacy is also an issue as the data collected by real participants must be encrypted to be completely anonymous. Additionally, important context information such as location, type of subscription and mode of connection is often missing in this type of measurement approach [40,52,53,54,55,56]. Industry regulators often adopt this approach for national scale measurement campaigns. Some of such notable measurements have been carried out by USA’s FCC [57], the UK’s Office of Communications (OFCOM) [58,59,60], the Canadian Radio-Television and Telecommunications Commission (CRTC) [61] and the Australian Competition and Consumer Commission (ACCC) [62].
Generally, it is important to observe that each methodology has its unique benefits and weaknesses as shown in Table 6. Hence, adopting more than one approach allows for cross-validating of measurement results. In recognizing this, regulators from many countries and private researchers are gradually employing multiple approaches for MBB network assessment. This is evident in a draft report released by the Body of European Regulators for Electronic Communications (BEREC) that summarizes responses received from its EU member countries about their MBB measurement activities [63]. A section of the report presented in Table 7 shows the diverse methodologies adopted within each country, with drive test being the most commonly used approach in twenty-seven of the thirty-four countries.

5.3. Mobile Broadband Measurement Software Tools

Most MBB performance evaluation methodologies make use of software either installed as applications in smartphones, measurement nodes or accessed online. Speedtest by Ookla [64] is one of the most popularly used applications developed for crowdsourced based MBB measurement. Speedtest maintains a free testing website where visitors can measure the performance of their network using any of the available public Ookla test servers around the world [64]. Each measurement taken with the Speedtest app is initiated by the user and this consists of upload and download throughput, jitter, packet loss and latency, which can be aggregated at the provider, state or country level [53]. Over 43 billion tests have been made using the Speedtest engine since its inception [65]. The test run by Ookla on its website is between a flash-based applet embedded in a web page and a server hosted on a web browser while the mobile version runs on smart phones [66]. Figure 6a shows the interface of the mobile application which is freely available in 17 languages on Google play store for both Android and iOS-based smartphones [67].
SamKnows is also a well-known network performance company that develops MBB crowdsourced measurement applications as one of its products [53]. The SamKnows mobile application app supports MBB performance in six main categories: download speed, upload speed, latency, packet loss, web browsing and YouTube streaming MBB measurement. It can also record passive metrics such as signal strength of the connection and device manufacturer, type and model. However, for this app, the test can be conducted anonymously on its own at timed intervals [53,68]. SamKnows’ crowdsourced app is modular, allowing regulators to easily customize the app for quick MBB speed assessment. SamKnows also collaborate to develop custom-made apps for regulators such as the FCC speed test app [57]. Figure 6b shows the interface of the mobile application, which is supported by both Android and iOS-based smartphones.
Other mobile applications for MBB assessment include MBPerf [69,70], the Meteor app by OpenSignal, [71], MobiPerf app [72], RTR-NetTest [73], Netradar and the SpeedSmart speedtest app [74].

6. Review of Existing Mobile Broadband Performance Measurements

This section reviews existing user-centric mobile broadband performance measurements studies and testbed-based projects.

6.1. User-Centric Performance Evaluation Works

The research objective of most MBB studies is conducted to comparatively evaluate the MBB performance delivered by different MNOs. The research reported in [69,70] adopted a host and crowdsourced based approached using MBperf as the mobile application to measure the performance of 2G and 3G MBB networks, while [75] used a simplified Raspberry Pi testbed for measurement of the performance of 3G and 4G MBB networks over an extended period. The results of this research carried out in Nigeria reveal variations in MBB speeds delivered by four major MNOs in the country. Similarly, studies reported in [76,77,78] also used a panel-based crowdsourced approach for a comparative assessment of 3G and 4G MBB networks in Nepal, Pakistan and South Africa, respectively. They identified that the MBB speeds delivered to end users do not meet the values advertised by the MNOs. The behavior of these crowdsourced MBB measurement datasets can be analyzed using machine learning for more accurate estimations [79,80].
Apart from comparative analysis, other types of MBB performance evaluation have been carried out. For instance, [81] used a panel-based crowdsourced method for performance assessment of MBB services offered by different Internet service providers during defined peak periods and off-peak periods in major Canadian metropolitan areas. They defined peak periods as the time between 7 pm and 11 pm from Monday to Friday and off-peak periods as any hours or days exclusive of peak periods. Additionally, [18,82,83] used a dedicated testbed and drive tests approach to study the performance of different MBB networks under mobility, while [84,85] adopted the walk test methodology to perform coverage and capacity measurement and characterize the performance of MBB networks during peak periods and off-peak period.
Furthermore, when designing future technologies, MBB measurement can be valuable for benchmarking and planning network upgrades. The MBB performance measurement of the 4G networks reported in [83,86] are studies conducted to determine the baseline for 5G capabilities and assess the inefficiencies that should be addressed in the 5G network. Some of the points highlighted and the benchmarks estimated were considered in the 5G pilot MBB measurement reported in [87,88,89,90].

6.2. Testbed-Based Measurement Projects

The limitations posed by using the aforementioned methodologies have driven institutions and private researchers to develop more robust infrastructure for testbed-based experiments on MBB performance. Although some of these testbeds are expensive to build, they allow for a controlled and scalable measurement over a long period and thus, eliminate many limitations of the other methodologies. This section introduces some testbed-based MBB performance evaluation platforms and projects that already exist. It goes further to explain the network tools used for these testbeds.

6.2.1. The Nornet Edge (NNE) Platform

The Nornet Edge (NNE) platform is a testbed dedicated to the measurement and study of MBB networks and is presented in [52,91,92]. Figure 7 shows the overview of the testbed for MBB experiments. Renowned as one of the largest infrastructures in the world for MBB measurements, the NNE has over 400 fully programmable and multi-homed nodes shown in Figure 8, placed at different locations in Norway. The NNE measurement nodes comprise custom-made single-board computers running a standard Linux operating system that allow 2–5 MNOs to be connected to it using MBB modems. The node is equipped with a Samsung S5PV210 Cortex A8 1 GHz processor with 512 MB RAM, 512 MB NAND flash memory and a 16 GB SD card for storage. Sets of servers form a central backend system for collection and storing data on the NNE platform. There is also an algorithm designed to manage the nodes and run measurements for a long time on a national scale. The platform allows for the collection of status information from the modems on mobile broadband cell ID, connection mode and signal strength.
Since the platform is able to simultaneously connect to multiple networks, it is possible to directly compare QoS metrics across different MNOs. The NNE platform is built for future compatibility with new systems as its designs makes it seamless to install new measurement applications to gather new or additional data. A website is also created for real-time viewing of the status of all NNE nodes, including the status of each MBB connection.
NNE is well suited for national scale measurements and experiments that require a large number of geographically distributed measurement nodes, simultaneous connections to multiple operators, information regarding the context in which measurements are taken, and continuous measurements that span long. One such experiment and research is reported in [18].

6.2.2. The MONROE Platform

The MONROE testbed and its operation presented in [51,54,93,94] is the first open access European transnational hardware-based platform for independent, multihomed, large-scale experimentation in MBB measurements. Figure 9 shows the overview of the MONROE MBB performance evaluation platform. MONROE has a set of 150 nodes, both mobile and stationary, which are multihomed to 5 different MNOs with the aid of commercial grade subscriptions across numerous European countries. The MONROE MBB measurement node shown in Figure 9 is based on Debian GNU/Linux “stretch” distribution integrating two small programmable computers. The computers are made of PC engines APU2 board interfacing with three 3G/4G MC7455 miniPCI express modems using LTE CAT6 and one WiFi modem. Each of the nodes gathers metadata such as carrier, technology, signal strength, GPS location and sensor data from the different modems. MONROE runs its MBB experiments using Docker containers (lightweight virtualized environment) to provide agile reconfiguration. Only users who are authenticated can access resources on the platform through a web portal, and also have access to the MONROE scheduler to deploy experiments. After each experiment on the MONROE platform, the results are periodically transferred from the nodes to a repository at a back-end server, while the MONROE scheduler also sets data quotas to ensure fairness among users. Some of the vast experiments run with the MONROE testbed have been reported in [95,96,97].
Three vital features of MONROE make the platform unique. It allows measurements to be repeated and controlled for precise and scientifically verifiable results for both fixed and mobile scenarios, enables support for demanding applications such as web and video services and supports protocol and service innovation.

6.2.3. The Simplified Raspberry Pi Platform

A simplified testbed for MBB performance evaluation that follows the setup of the NNE albeit using easily sourced commercial-off-the-shelf (COTS) devices is presented in [50,75]. Figure 10 shows the overall system architecture with the Raspberry Pi forming the core of the remote MBB measurement node. The Raspberry Pi 4 with 64 quad-core Cortex-A72 processors and 2GB Low-Power Double Data Rate (LPDDRA) RAM on its board is used for the node. The testbed uses USB modems and retrofitted WiFi to connect up to 4 MNOs for 3G and 4G MBB networks, respectively. The Raspberry Pi nodes are configured with the 4-way 5V relay modules mounted and an executable script written in python to achieve multihoming for 3G and 4G MBB measurements. The node autonomously initiates the measurement at regular intervals and stores the information, which an authorized user can access remotely at the testbed core for evaluation. This simplified MBB testbed is not as sophisticated as NNE; however, it can measure the key MBB performance metrics over an extended period.
The aforementioned testbeds have been dedicated mostly to 3G and 4G MBB experiments, albeit allowing compatibility with future mobile communication networks like the recently deployed 5G network. To the best of our knowledge, there is no dedicated testbed to assess the QoS delivered to end users on the 5G MBB network from a user-centric perspective. However, as part of the 5G Public Private Partnership (5G-PPP) initiative, the EU funded 5GENESIS [98] project has been developed as a flexible and open experimentation testbed for validating the end-to-end key performance indicators (KPIs) of 5G networks. The 5GENESIS architecture is designed to provide an integrated and open experimentation framework that facilitates interactions between the experimenters and the testing facilities. A detailed description of the experimentation suite is presented in [99], while pilot 5G experiments using the testbed have been reported in [100,101].
Furthermore, there are other testbed federations such as Fed4FIRE+ [102,103] and 5TONIC [104], developed to carry out experiments on numerous aspects of 4G and 5G. Fed4FIRE+ was the largest federation of internet testbeds in Europe consisting of 23 testbeds equipped with numerous user-friendly tools that enabled remote testing in different areas of interest. The Fed4FIRE+ project, which was a successor to the Fed4FIRE project, came to an end in June 2022 and its legacy will be taken by Scientific LargeScale Infrastructure for Computing/Communication Experimental Studies. (SLICES-RI) [102,103]. 5TONIC is an open research and innovation laboratory developed to create an open global environment for industry experts and members of academia to work together on specific projects that focus on 5G technologies [104]. Some studies that utilized the 5TONIC platform have been reported in [105,106].
Table 8 presents a summary of extensively reviewed user-centric MBB performance evaluation studies, highlighting the method adopted, the QoS metrics considered, the type of access network and a summary of each study. Table 9 compares the different MBB performance measurement methods already discussed.

7. Challenges of Mobile Broadband Measurements

Different MBB measurements have their unique challenges, hence the adoption of more than one measurement approach to allow opportunities for cross-validation of results. This section briefly discusses some of the general challenges that are associated with MBB measurement shown in Figure 11.

7.1. Variety of Performance

There are infinite indices that can be used to characterize the performance of MBB during measurements. These indices include geographic locations and measurement orientations, the type of user equipment, and the context surrounding the user equipment. Small changes along the dimensions of their parameters most often result in a significant impact on the measured performance [53]. Therefore, the challenge of a proper definition of the context of measurements, collection and handling of the numerous data sets will arise.

7.2. Privacy and User Engagement

Inducing several participants across the MBB measurement ecosystem to share infrastructure or data provokes difficult challenges [134]. Privacy is one of such issue that places significant constraints on data collection due to the prevalence of single users or participants. Capturing important metadata like location related data from the user equipment during measurement poses a huge threat to individual privacy [53]. Sometimes, there are agreement terms with the participants to assure them of no privacy breach. However, such an agreement does not translate to the users not interfering with the measurement process due to differing sentiments on privacy. This could significantly impede the authenticity of data collection. Furthermore, recruiting participants and getting them engaged and motivated during the measurement process is challenging. If methods are not devised to tackle this, the participants could simply abandon their device or cite economic challenges [78]. Similarly, personnel operating dedicated measurement testbeds may interfere with the measurement processes if not properly guided.

7.3. Mobility

Conducting MBB measurement in mobility is challenging when considering the mobility of the end-user’s location in time and space and the mobility of all network resources. For instance, for a more accurate result, when conducting drive test and walk test, measurement samples are collected in as many locations as possible but with differing physical speeds of the devices. The mobility during such measurement can easily be dictated by natural or unforeseen occurrences such as existing vehicular or pedestrian traffic in the area. This will cause the exact model of geographic mobility being tested to vary, and achieving the exact details for the different locations would prove to be a challenging task [53]. Additionally, the need for shorter duration of tests, better accuracy, and analysis mechanism to handle performances during handover between base stations, roaming and wireless technologies at different speeds, is very important for how the performance data are aggregated and reported [53,96].

7.4. Lack of Methodological Transparency

It is observed that information which describes the overall structure of the measurements and reports on the various methodologies is usually available, whereas there is a dearth of details regarding the adopted measurement methodology and how the raw data are analyzed to produce the final results. This dearth of methodological transparency makes it difficult for researchers and others to critically analyze and distinguish between good and bad implementations of MBB measurement methodologies. The lack of transparency could translate to a lack of trust among researchers for future research in the evolving MBB measurement ecosystem [53].

7.5. Need for Proactive Measurements

Standard objectives during MBB measurement are needed. For example, a measurement objective may be to seek broadband performance in locations where actual users have not yet attempted to use mobile devices. Another objective may be intentionally trying to find the locations and conditions under which mobile service expectations are not being met. The aim is usually to identify coverage/availability gaps or dead spots within already covered areas or to find the boundaries of the existing coverage zones [53]. However, identifying the scope and appropriate methodologies for such mobile broadband measurement objectives is challenging.

7.6. Sustainability and Management of Infrastructure

It is viable to develop methodologies and build appropriate infrastructure for MBB measurement. However, ensuring that the infrastructure is sustainable for a foreseeable time will present a continuous challenge that needs to be considered before deployments. Adequate resources required to cater for the sustainability of the measurement infrastructure should be available or sourced [129]. There is also a prevailing challenge with managing large testbeds, especially keeping the hardware up to date with emerging technologies [96]. In terms of QoS/QoE, the lack of well-defined scalable QoS and QoE indicators limit the proper estimation of various service compositions as a part of the overall telecommunication network.

8. Conclusions

This paper has used available data to evaluate the global importance and impact of MBB. It provides an understanding of end-to-end MBB performance assessment and presented existing methodologies, approaches and testbeds for evaluating the performance of MBB networks. The challenges associated with MBB measurement are highlighted.
However, the emergence of 5G and future technologies with higher gigabit data rates and the projected increase in simultaneously connected devices would translate to more data consumption. Such development may result in significant variations in peak periods and off-peak periods data consumption patterns. These possible challenges need to be identified and addressed. Therefore, it is important that future research be geared toward optimizing present MBB measurement methods and infrastructure to accommodate these imminent changes. Furthermore, it has been highlighted that MBB performance evaluation is complex for one evaluation method to be applicable in all situations. For example, the measurement of MBB performance under static and mobile conditions will require different methods. As a result, future research should consider the development of distributed MBB measurement platforms that are capable of supporting a range of MBB experiments. Intensive research in this direction would result in the development and deployment of more robust, semi-autonomous, interlinkable measurement platforms.

Author Contributions

Conceptualization, V.U. and U.E.; methodology, V.U., J.A. and U.E.; formal analysis, V.U. and J.A.; resources, V.U., U.E. and I.D.; writing—original draft preparation, V.U.; writing—review and editing, V.U., I.D., U.E. and J.A.; visualization, V.U.; supervision, I.D. and U.E.; All authors have read and agreed to the published version of the manuscript.

Funding

Article publishing fee was funded by The Research Directorate, Durban University of Technology, South Africa.

Data Availability Statement

No new data were created or analyzed in this study. Data sharing is not applicable to this article.

Acknowledgments

This project was supported by the Nigerian Communications Commission (NCC) and the Space Science Research Centre, Durban University of Technology.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Edquist, H. The economic impact of mobile broadband speed. Telecommun. Policy 2022, 46, 102351. [Google Scholar] [CrossRef]
  2. International Telecommunication Union. Measuring Digital Development Facts and Figures 2021; International Telecommunication Union: Geneva, Switzerland, 2021. [Google Scholar]
  3. Edquist, H.; Goodridge, P.; Haskel, J.; Li, X.; Lindquist, E. How important are mobile broadband networks for the global economic development? Inf. Econ. Policy 2018, 45, 16–29. [Google Scholar] [CrossRef]
  4. International Telecommunication Union. The Impact of Broadband on the Economy: Research to Date and Policy Issues, in Broadband Series; International Telecommunication Union: Geneva, Switzerland, 2012. [Google Scholar]
  5. Kongaut, C.; Bohlin, E. Impact of broadband speed on economic outputs: An empirical study of OECD countries. Econ. Bus. Rev. 2017, 3, 12–32. [Google Scholar] [CrossRef] [Green Version]
  6. Mayer, W.; Madden, G.; Wu, C. Broadband and economic growth: A reassessment. Inf. Technol. Dev. 2020, 26, 128–145. [Google Scholar] [CrossRef]
  7. Minges, M. Exploring the relationship between broadband and economic growth. In World Development Report 2016: Digital Dividends; World Bank: Washington, DC, USA, 2016. [Google Scholar]
  8. Ugboma, M.U. Bridging the digital divide: With special reference to Nigeria. Inf. Impact J. Inf. Knowl. Manag. 2012, 3, 1–2. [Google Scholar]
  9. 3gpp. Specifications Home. Available online: https://www.3gpp.org/specifications/specifications (accessed on 2 August 2022).
  10. Chitrapu, P.; Aghili, B. Evolution of GSM into the next generation wireless world. In Proceedings of the 2007 IEEE Long Island Systems, Applications and Technology Conference, Farmingdale, NY, USA, 4 May 2007; IEEE: New York, NY, USA, 2007; pp. 1–10. [Google Scholar]
  11. Pawar, D.S.; Deshpande, A. Evolution of Wireless Technology. Int. J. Comput. Sci. Mob. Comput. IJCSMC 2020, 9, 91–94. [Google Scholar]
  12. Venkatachalam, S.; McDowell, S.D. What is broadband? Where is rural? Gov. Inf. Q. 2003, 20, 151–166. [Google Scholar] [CrossRef]
  13. Molisch, A.F. GSM–Global System for Mobile Communications. In Wireless Communications; John Wiley & Sons: Hoboken, NJ, USA, 2012; Chapter 24; p. 587. [Google Scholar]
  14. Sauter, M. General Packet Radio Service (GPRS) and EDGE. In From GSM to LTE-Advanced Pro and 5G: An Introduction to Mobile Networks and Mobile Broadband; John Wiley & Sons: Hoboken, NJ, USA, 2017; Chapter 2; pp. 73–128. [Google Scholar]
  15. Schwartz, M. Second-generation, digital, wireless systems. In Mobile Wireless Communications; Cambridge University Press, The Edinburgh Building: Cambridge, UK, 2005; Volume 25, Chapter 8; pp. 199–245. [Google Scholar]
  16. Li, X.; Gani, A.; Salleh, R.; Zakaria, O. The Future of Mobile Wireless Communication Networks. In Proceedings of the 2009 International Conference on Communication Software and Networks, Chengdu, China, 27–28 February 2009; pp. 554–557. [Google Scholar] [CrossRef]
  17. Sauter, M. From GSM to LTE-Advanced Pro and 5G an Introduction to Mobile Networks and Mobile Broadband; John Wiley & Sons Ltd.: Hoboken, NJ, USA, 2017. [Google Scholar]
  18. Siwakota, Y.R. Measuring Performance of Mobile Broadband Netwrok under Moblity; Department of Informatic, University of Oslo: Oslo, Norway, 2014. [Google Scholar]
  19. Sauter, M. Universal Mobile Telecommunications System (UMTS) and High-Speed Packet Access (HSPA). In From GSM to LTE-Advanced Pro and 5G: An Introduction to Mobile Networks and Mobile Broadband; John Wiley & Sons: Hoboken, NJ, USA, 2017; Chapter 3; pp. 129–232. [Google Scholar]
  20. Shukla, S.; Khare, V.; Garg, S.; Sharma, P. Comparative Study of 1G, 2G, 3G and 4G. J. Eng. Comput. Appl. Sci 2013, 2, 55–63. [Google Scholar]
  21. Churi, J.R.; Surendran, T.S.; Tigdi, S.A.; Yewale, S. Evolution of networks (2G-5G). In Proceedings of the International Conference on Advances in Communication and Computing Technologies (ICACACT), Kochi, India, 9–11 August 2012; Volume 51, pp. 8–13. [Google Scholar]
  22. Molisch, A.F. WCDMA/UMTS. In Wireless Communications; John Wiley & Sons: Hoboken, NJ, USA, 2012; pp. 635–663. [Google Scholar]
  23. Sauter, M. Long Term Evolution (LTE) and LTE-Advanced. In From GSM to LTE-Advanced Pro and 5G: An Introduction to Mobile Networks and Mobile Broadband; John Wiley & Sons: Hoboken, NJ, USA, 2017; Chapter 4; pp. 235–325. [Google Scholar]
  24. Tabbane, S. 4G to 5G Networks and Standard Releases. Available online: https://www.itu.int/en/ITU-D/Regional-Presence/AsiaPacific/SiteAssets/Pages/Events/2019/ITU-ASP-CoE-Training-on-/3GPP_4G%20to%205G%20networks%20evolution%20and%20releases.pdf (accessed on 1 January 2022).
  25. Ezhilarasan, E.; Dinakaran, M. A Review on Mobile Technologies: 3G, 4G and 5G. In Proceedings of the 2017 Second International Conference on Recent Trends and Challenges in Computational Models (ICRTCCM), Tindivanam, India, 3–4 February 2017; pp. 369–373. [Google Scholar] [CrossRef]
  26. Arshad, Q.K.U.D.; Kashif, A.U.; Quershi, I.M. A Review on the Evolution of Cellular Technologies. In Proceedings of the 2019 16th International Bhurban Conference on Applied Sciences and Technology (IBCAST), Islamabad, Pakistan, 8–12 January 2019; pp. 989–993. [Google Scholar] [CrossRef]
  27. Arshad, Q.K.U.D.; Kashif, A.U.; Quershi, I.M. Mobility management solutions for 5G networks: Architecture and services. Comput. Netw. 2020, 169, 107082. [Google Scholar] [CrossRef]
  28. Ghafoor, U.; Ali, M.; Khan, H.Z.; Siddiqui, A.M.; Naeem, M. NOMA and future 5G & B5G wireless networks: A paradigm. J. Netw. Comput. Appl. 2022, 204, 103413. [Google Scholar] [CrossRef]
  29. Yang, C.; Liang, P.; Fu, L.; Cui, G.; Huang, F.; Teng, F.; Bangash, Y.A. Using 5G in smart cities: A systematic mapping study. Intell. Syst. Appl. 2022, 14, 200065. [Google Scholar] [CrossRef]
  30. Zhu, Z.; Chu, Z.; Li, X. Chapter 2-Three major operating scenarios of 5G: eMBB, mMTC, URLLC. In Intelligent Sensing and Communications for Internet of Everything; Zhu, Z., Chu, Z., Li, X., Eds.; Academic Press: Cambridge, MA, USA, 2022; pp. 15–76. [Google Scholar]
  31. International Telecommunication Union. Setting the Scene for 5G: Opportunities & Challenges; International Telecommunication Union: Geneva, Switzerland, 2018. [Google Scholar]
  32. Li, Z.; Uusitalo, M.A.; Shariatmadari, H.; Singh, B. 5G URLLC: Design challenges and system concepts. In Proceedings of the 2018 15th International Symposium on Wireless Communication Systems (ISWCS), Lisbon, Portugal, 28–31 August 2018; IEEE: New York, NY, USA, 2018; pp. 1–6. [Google Scholar]
  33. ITU-R. IMT Vision–Framework and Overall Objectives of the Future Development of IMT for 2020 and Beyond; Recommendation ITU: Geneva, Switzerland, 2015. [Google Scholar]
  34. Ateya, A.A.; Muthanna, A.; Makolkina, M.; Koucheryavy, A. Study of 5G services standardization: Specifications and requirements. In Proceedings of the 2018 10th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT), Moscow, Russia, 5–9 November 2018; IEEE: New York, NY, USA, 2018; pp. 1–6. [Google Scholar]
  35. GSMA. 5G Implementation Guidelines; CK Hutchison: Hongkong, China, March 2019. [Google Scholar]
  36. ITU. Statistics. Available online: https://www.itu.int/en/ITU-D/Statistics/Pages/stat/default.aspx (accessed on 20 July 2022).
  37. ITU. Economy Classification. Available online: http://www.itu.int/en/ITU-D/Statistics/Pages/definitions/regions.aspx (accessed on 21 July 2022).
  38. OECD. Broadband Portal. Available online: https://www.oecd.org/sti/broadband/broadband-statistics/ (accessed on 14 July 2022).
  39. GSMA. The Mobile Economy; GSMA: London, UK, 2022. [Google Scholar]
  40. El-Saleh, A.A.; Alhammadi, A.; Shayea, I.; Alsharif, N.; Alzahrani, N.M.; Khalaf, O.I.; Aldhyani, T.H.H. Measuring and Assessing Performance of Mobile Broadband Networks and Future 5G Trends. Sustainability 2022, 14, 829. Available online: https://www.mdpi.com/2071-1050/14/2/829 (accessed on 1 January 2022). [CrossRef]
  41. ITU. Recommendation -T P. 10/G. 100, Vocabulary for Performance, Quality of Service and Quality of Experience; ITU: Geneva, Switzerland, 2017. [Google Scholar]
  42. ITU-T. E.800-Series–Guidelines on Regulatory Aspects of QoS. 2021. Available online: https://www.itu.int/ITU-T/recommendations/rec.aspx?rec=14832&lang=en (accessed on 31 July 2022).
  43. Malisuwan, S.; Milindavanij, D.; Kaewphanuekrungsi, W. Kaewphanuekrungsi, Quality of service (QoS) and quality of experience (QoE) of the 4G LTE perspective. Int. J. Future Comput. Commun. 2016, 5, 158. [Google Scholar] [CrossRef] [Green Version]
  44. ITU-T Recommendation, G. 1000, Communications Quality of Service: A Framework and Definitions, 11/2001. Available online: https://www.itu.int/itu-t/recommendations/rec.aspx?rec=G.1000 (accessed on 31 July 2022).
  45. ITU-T. Quality of Experience (QoE) Requirements for Real-Time Multimedia Services over 5G Networks. 2022. Available online: https://www.itu.int/pub/T-TUT-QOS-2022-1 (accessed on 31 July 2022).
  46. Budiman, E.; Moeis, D.; Soekarta, R. Broadband quality of service experience measuring mobile networks from consumer perceived. In Proceedings of the 2017 3rd International Conference on Science in Information Technology (ICSITech), Bandung, Indonesia, 25–26 October 2017; IEEE: New York, NY, USA, 2017; pp. 423–428. [Google Scholar]
  47. Poryazov, S.A.; Saranova, E.T.; Andonov, V.S. Overall Model Normalization towards Adequate Prediction and Presentation of QoE in Overall Telecommunication Systems. In Proceedings of the 2019 14th International Conference on Advanced Technologies, Systems and Services in Telecommunications (TELSIKS), Nis, Serbia, 20–22 October 2019; IEEE: New York, NY, USA, 2019; pp. 360–363. [Google Scholar]
  48. Feamster, N.; Livingood, J. Measuring internet speed: Current challenges and future recommendations. Commun. ACM 2020, 63, 72–80. [Google Scholar] [CrossRef]
  49. ITU. Quality of service guaranteed mechanisms and performance model for public packet telecommunication data networks. In Recommendation ITU-T Y.2617; ITU: Geneva, Switzerland, 2016. [Google Scholar]
  50. Umoh, V.B.; Ukommi, U.S.; Ekpe, U.M. A Comparative Study of User Experienced Mobile Broadband Performance. Niger. J. Technol. 2022, 41, 560–568. [Google Scholar] [CrossRef]
  51. Alay, O.; Lutu, A.; Garcia, R.; Peon-Quiros, M.; Mancuso, V.; Hirsch, T.; Dely, T.; Werme, J.; Evensen, K.; Hansen, A.; et al. Measuring and assessing mobile broadband networks with MONROE. In Proceedings of the 2016 IEEE 17th International Symposium on A World of Wireless, Mobile and Multimedia Networks (WoWMoM), Coimbra, Portugal, 21–24 June 2016; pp. 1–3. [Google Scholar] [CrossRef]
  52. Kvalbein, A.; Baltrūnas, D.; Evensen, K.; Xiang, J.; Elmokashfi, A.; Ferlin-Oliveira, S. The Nornet Edge platform for mobile broadband measurements. Comput. Netw. 2014, 61, 88–101. [Google Scholar] [CrossRef] [Green Version]
  53. Bauer, S.; Lehr, W. Measuring Mobile Broadband Performance. In Proceedings of the 29th European regional Conference of the International Telecommunications Society (ITS): “Towards a Digital Future: Turning Technology into Markets”? Trento, Italy, 1–4 August 2018; International Telecommunications Society (ITS): Calgary, AL, Canada, 2018. [Google Scholar]
  54. Alay, Ö.; Lutu, A.; Peón-Quirós, M.; Mancuso, V.; Hirsch, T.; Evensen, K.; Hansen, A.; Alfredsson, S.; Karlsson, J.; Brunstrom, A.; et al. Experience: An open platform for experimentation with commercial mobile broadband networks. In Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking, Snowbird, UT, USA, 16–20 October 2017; pp. 70–78. [Google Scholar]
  55. Akinlabi, A.A.; Dahunsi, F.M. Mobile broadband quality of service analysis using host-based performance measurements. Afr. J. Sci. Technol. Innov. Dev. 2021, 14, 1–19. [Google Scholar] [CrossRef]
  56. Midoglu, C.; Svoboda, P. Opportunities and challenges of using crowdsourced measurements for mobile network benchmarking a case study on RTR open data. In Proceedings of the 2016 SAI computing conference (SAI), London, UK, 13–15 July 2016; IEEE: New York, NY, USA, 2016; pp. 996–1005. [Google Scholar]
  57. Federal Communications Commission (FCC). Eleventh Measuring Broadband America Fixed Broadband Report. 2021. Available online: https://www.citizenscience.gov/assets/files/fcc-speed-test.pdf (accessed on 1 January 2022).
  58. OFCOM. Measuring 4G mobile broadband and voice performance. In Smartphone Cities; OFCOM: London, UK, 2016. [Google Scholar]
  59. OFCOM. Measuring mobile broadband and voice performance. In Smartphone Cities; OFCOM: London, UK, 2016. [Google Scholar]
  60. OFCOM. Measuring Mobile Broadband Performance in the UK 4G and 3G Network Performance; OFCOM: London, UK, 2014. [Google Scholar]
  61. The Canadian Radio-television and Telecommunications Commission (CRTC). In Measuring Broadband Canada; CRTC: Ottawa, ON, Canada, 2020.
  62. Australian Competition and Consumer Commission (ACCC). Measuring Broadband Australia. 2021. Available online: https://www.accc.gov.au/regulated-infrastructure/telecommunications-and-internet/telecommunications-monitoring/measuring-broadband-australia-program (accessed on 1 January 2022).
  63. BEREC. Common Position on Monitoring Mobile Coverage; BEREC: Rīga, Republic of Latvia, 2018. [Google Scholar]
  64. Bauer, S.; Clark, D.D.; Lehr, W. Understanding Broadband Speed Measurements. 2010. Available online: https://www.semanticscholar.org/paper/Understanding-Broadband-Speed-Measurements-Bauer-Clark/7157455aad24d88bdcaf1961977ec64911680e16 (accessed on 31 July 2022).
  65. SPEEDTEST. Available online: https://www.speedtest.net/about (accessed on 29 July 2022).
  66. Goel, U.; Wittie, M.P.; Claffy, K.C.; Le, A. Survey of end-to-end mobile network measurement testbeds, tools, and services. IEEE Commun. Surv. Tutor. 2015, 18, 105–123. [Google Scholar] [CrossRef] [Green Version]
  67. Speedtest. Speedtest Apps for Mobile. Available online: https://www.speedtest.net/apps/mobile (accessed on 31 July 2022).
  68. SamKnows. Cellular. Available online: https://www.samknows.com/products/cellular (accessed on 29 July 2022).
  69. Dahunsi, F.; Akinlabi, A. Measuring mobile broadband performance in Nigeria: 2G and 3G. Niger. J. Technol. 2019, 38, 422–436. [Google Scholar] [CrossRef]
  70. Folasade, D.; Ayokunle, A.; Olumide, O.; Jide, P. Performance Monitoring of Mobile Broadband in a Developing Country. In Proceedings of the IST-Africa 2019 Conference, Nairobi, Kenya, 8–10 May 2019. [Google Scholar]
  71. OPENSIGNAL. Available online: https://www.opensignal.com/apps (accessed on 31 July 2022).
  72. MobPerf. Available online: http://www.mobiperf.com/ (accessed on 31 July 2022).
  73. RTR. RTR-NetTest. Available online: https://www.netztest.at/en/ (accessed on 1 August 2022).
  74. SpeedSmart. Powerful Speed Test. Available online: https://speedsmart.net/app (accessed on 31 July 2022).
  75. Umoh, V.B.; Ukommi, U.S.; Ekpe, U.M. A Simplified Method for Extended Duration Measurements of Mobile Broadband Performance. In Proceedings of the 2022 IEEE Nigeria 4th International Conference on Disruptive Technologies for Sustaina-ble Development (NIGERCON), Lagos, Nigeria, 5–7 April 2022; pp. 1–5. [Google Scholar] [CrossRef]
  76. Karn, N.K.; Hongli, Z.; Shafiq, M. Measuring broadband internet performance in Nepal: A comparative study. Procedia Comput. Sci. 2017, 107, 64–69. [Google Scholar] [CrossRef]
  77. Awan, M.F.; Ahmad, T.; Qaisar, S.; Feamster, N.; Sundaresan, S. Measuring broadband access network performance in Pakistan: A comparative study. In Proceedings of the 2015 IEEE 40th Local Computer Networks Conference Workshops (LCN Workshops), Clearwater Beach, FL, USA, 26–29 October 2015; IEEE: New York, NY, USA, 2015; pp. 595–602. [Google Scholar]
  78. Chetty, M.; Sundaresan, S.; Muckaden, S.; Feamster, N.; Calandro, E. Measuring broadband performance in South Africa. In Proceedings of the 4th Annual Symposium on Computing for Development, Cape Town, South Africa, 6–7 December 2013; pp. 1–10. [Google Scholar]
  79. Kousias, K.; Alay, O.; Argyriou, A.; Lutu, A.; Riegler, M. Estimating downlink throughput from end-user measurements in mobile broadband networks. In Proceedings of the 2019 IEEE 20th International Symposium on A World of Wireless, Mobile and Multimedia Networks(WoWMoM), Washington, DC, USA, 10–12 June 2019; IEEE: New York, NY, USA, 2019; pp. 1–10. [Google Scholar]
  80. Kousias, K.; Midoglu, C.; Alay, O.; Lutu, A.; Argyriou, A.; Riegler, M. The same, only different: Contrasting mobile operator behavior from crowdsourced dataset. In Proceedings of the 2017 IEEE 28th Annual International Symposium on Personal, Indoor, and Mobile Radio Communications (PIMRC), Montreal, QC, Canada, 8–13 October 2017; IEEE: New York, NY, USA, 2017; pp. 1–6. [Google Scholar]
  81. Samknows. Analysis of Broadband Performance in Canada March & April 2016; The Canadian Radio-television and Telecommunications Commission (CRTC): Ottawa, ON, Canada, 2016. [Google Scholar]
  82. Ahmad, S.; Musleh, S.; Nordin, R. The gap between expectation & reality: Long term evolution (LTE) & third generation (3G) network performance in campus with test mobile system (TEMS). In Proceedings of the 2015 9th Asia Modelling Symposium (AMS), Kuala Lumpur, Malaysia, 7–9 September 2015; IEEE: New York, NY, USA, 2015; pp. 164–168. [Google Scholar]
  83. Parichehreh, A.; Moosavi, R.; Ramachandra, P.; Alfredsson, S.; Brunstrom, A. LTE as a Road Toward 5G: QoS Analysis in Mobility Scenario Using the Monroe Platform. In Proceedings of the 2019 IEEE Wireless Communications and Networking Conference (WCNC), Marrakesh, Morocco, 15–18 April 2019; IEEE: New York, NY, USA, 2019; pp. 1–7. [Google Scholar]
  84. Shayea, I.; Ergen, M.; Azmi, M.H.; Nandi, D.; El-Salah, A.A.; Zahedi, A. Indoor network signal coverage of mobile telecommunication networks in West Malaysia: Selangor and Johor Bahru. In Proceedings of the 2017 IEEE 13th Malaysia International Conference on Communications (MICC), Johor Bahru, Malaysia, 28–30 November 2017; pp. 288–293. [Google Scholar] [CrossRef]
  85. Turniski, F.; Lackovic, S.; Pilinsky, S.Z.; Tekovic, A. Analysis of 3G and 4G download throughput in pedestrian zones. In Proceedings of the 2016 International Symposium ELMAR, Zadar, Croatia, 12–14 September 2016; IEEE: New York, NY, USA, 2016; pp. 9–12. [Google Scholar]
  86. Daengsi, T.; Wuttidittachotti, P. Quality of Service as a Baseline for 5G: A Recent Study of 4G Network Performance in Thailand. In Proceedings of the 2020 IEEE International Conference on Communication, Networks and Satellite (Comnetsat), Batam, Indonesia, 17–18 December 2020; IEEE: New York, NY, USA, 2020; pp. 395–399. [Google Scholar]
  87. Daengsi, T.; Ungkap, P.; Wuttidittachotti, P. A Study of 5G Network Performance: A Pilot Field Trial at the Main Skytrain Stations in Bangkok. In Proceedings of the 2021 International Conference on Artificial Intelligence and Computer Science Technology (ICAICST), Yogyakarta, Indonesia, 29–30 June 2021; IEEE: New York, NY, USA, 2021; pp. 191–195. [Google Scholar]
  88. Daengsi, T.; Ungkap, P.; Wuttidittachotti, P. 5G Network Performance: A Study using Stationary and Mobility Tests on Sukhumvit Line–BTS Skytrain in Bangkok. In Proceedings of the 2021 4th International Conference of Computer and Informatics Engineering (IC2IE), Depok, Indonesia, 14–15 September 2021; IEEE: New York, NY, USA, 2021; pp. 447–450. [Google Scholar]
  89. Daengsi, T.; Ungkap, P.; Pornpongtechavanich, P.; Wuttidittachotti, P. QoS Measurement: A Comparative Study of Speeds and Latency for 5G Network Using Different Speed Test Applications for Mobile Phones. In Proceedings of the 2021 IEEE 7th International Conference on Smart Instrumentation, Measurement and Applications (ICSIMA), Virtual, 23–25 August 2021; pp. 206–210. [Google Scholar] [CrossRef]
  90. Narayanan, A.; Ramadan, E.; Carpenter, J.; Liu, Q.; Liu, Y.; Qian, F.; Zhang, Z.-L. A first look at commercial 5G performance on smartphones. Proc. Web Conf. 2020, 2020, 894–905. [Google Scholar]
  91. Gran, E.G.; Dreibholz, T.; Kvalbein, A. NorNet Core–A multi-homed research testbed. Comput. Netw. 2014, 61, 75–87. [Google Scholar] [CrossRef] [Green Version]
  92. Dreibholz, T.; Gran, E.G. Design and Implementation of the NORNET CORE Research Testbed for Multi-homed Systems. In Proceedings of the 2013 27th International Conference on Advanced Information Networking and Applications Workshops, Barcelona, Spain, 25–28 March 2013; pp. 1094–1100. [Google Scholar] [CrossRef]
  93. Alay, O.; Lutu, A.; García, R.; Peón-Quirós, M.; Mancuso, V.; Hirsch, T.; Dely, T.; Werme, J.; Evensen, K.; Hansen, A.; et al. MONROE, a distributed platform to measure and assess mobile broadband networks: Demo. In Proceedings of the Tenth ACM International Workshop on Wireless Network Testbeds, Experimental Evaluation, and Characterization, New York, NY, USA, 3–7 October 2016. [Google Scholar] [CrossRef] [Green Version]
  94. Alay, O.; Lutu, A.; García, R.; Peón-Quirós, M.; Mancuso, V.; Hirsch, T.; Dely, T.; Werme, J.; Evensen, K.; Hansen, A.; et al. MONROE: Measuring Mobile Broadband Networks in Europe. 2017. Available online: https://dspace.networks.imdea.org/handle/20.500.12761/947 (accessed on 1 January 2022).
  95. Mancuso, V.; Quirós, M.P.; Midoglu, C.; Moulay, M.; Comite, V.; Lutu, A.; Alay, Ö.; Alfredsson, S.; Rajiullah, M.; Brunström, A.; et al. Results from running an experiment as a service platform for mobile broadband networks in Europe. Comput. Commun. 2019, 133, 89–101. [Google Scholar] [CrossRef]
  96. Midoglu, C.; Kousias, K.; Alay, Ö.; Lutu, A.; Argyriou, A.; Riegler, M.; Griwodz, C. Large scale speedtest experimentation in Mobile Broadband Networks. Comput. Netw. 2021, 184, 107629. [Google Scholar] [CrossRef]
  97. Midoglu, C.; Wimmer, L.; Lutu, A.; Alay, Ö.; Griwodz, C. MONROE-Nettest: A configurable tool for dissecting speed measurements in mobile broadband networks. In Proceedings of the IEEE INFOCOM 2018-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), Honolulu, HI, USA, 15–19 April 2018; IEEE: New York, NY, USA, 2018; pp. 342–347. [Google Scholar]
  98. 5GENESIS. Available online: https://5genesis.eu/ (accessed on 24 October 2022).
  99. 5GENESIS. Deliverable D2.4: Final Report on Facility Design and Experimentation Planning. 2020. Available online: https://5genesis.eu/wp-content/uploads/2020/07/5GENESIS_D2.4_v1.0.pdf (accessed on 1 January 2022).
  100. Zayas, A.D.; Caso, G.; Alay, Ö.; Merino, P.; Brunstrom, A.; Tsolkas, D.; Koumaras, H. A modular experimentation methodology for 5G deployments: The 5GENESIS approach. Sensors 2020, 20, 6652. [Google Scholar] [CrossRef]
  101. Christopoulou, M.; Xilouris, G.; Sarlas, A.; Koumaras, H.; Kourtis, M.-A.; Anagnostopoulos, T. 5g experimentation: The experience of the athens 5genesis facility. In Proceedings of the 2021 IFIP/IEEE International Symposium on Integrated Network Management (IM), Bordeaux, France, 18–20 May 2021; IEEE: New York, NY, USA, 2021; pp. 783–787. [Google Scholar]
  102. FED4FIRE+. Available online: https://www.fed4fire.eu/ (accessed on 14 January 2023).
  103. Demeester, P.; Van Daele, P.; Wauters, T.; Hrasnica, H. Fed4FIRE–The Largest Federation of Testbeds in Europe. In Building the Future Internet through FIRE; River Publishers: Denmark, The Netherlands, 2022; pp. 87–109. [Google Scholar]
  104. 5TONIC. Available online: https://www.5tonic.org/ (accessed on 14 January 2023).
  105. Nogales, B.; Vidal, I.; Lopez, D.R.; Rodriguez, J.; Garcia-Reinoso, J.; Azcorra, A. Design and deployment of an open management and orchestration platform for multi-site nfv experimentation. IEEE Commun. Mag. 2019, 57, 20–27. [Google Scholar] [CrossRef]
  106. Azcorra, A. Advanced 5G trials with verticals in 5TONIC laboratory. In International Ericsson All Employees Meeting of R&D; Ericsson: Madrid, Spain, 2017. [Google Scholar]
  107. Khatouni, A.S.; Mellia, M.; Marsan, M.A.; Alfredsson, S.; Karlsson, J.; Brunstrom, A.; Alay, O.; Lutu, A.; Midoglu, C.; Mancuso, V. Speedtest-like measurements in 3g/4g networks: The monroe experience. In Proceedings of the 2017 29th International Teletraffic Congress (ITC 29), Genoa, Italy, 4–8 September 2017; IEEE: New York, NY, USA, 2017; pp. 169–177. [Google Scholar]
  108. Shayea, I.; Azmi, M.H.; Ergen, M.; El-Saleh, A.A.; Han, C.T.; Arsad, A.; Rahman, T.A.; Alhammadi, A.; Daradkeh, Y.I.; Nandi, D. Performance analysis of mobile broadband networks with 5g trends and beyond: Urban areas scope in Malaysia. IEEE Access 2021, 9, 90767–90794. [Google Scholar] [CrossRef]
  109. Rajiullah, M.; Khatouni, A.S.; Midoglu, C.; Alay, Ö.; Brunstrom, A.; Griwodz, C. Mobile network performance during the COVID-19 outbreak from a testbed perspective. In Proceedings of the 14th International Workshop on Wireless Network Testbeds, Experimental evaluation & Characterization, London, UK, 21 September 2020; pp. 110–117. [Google Scholar]
  110. Shayea, I.; Ergen, M.; Azmi, M.H.; Nandi, D.; El-Salah, A.A.; Zahedi, A. Performance Analysis of Mobile Broadband Networks With 5G Trends and Beyond: Rural Areas Scope in Malaysia. IEEE Access 2020, 8, 65211–65229. [Google Scholar] [CrossRef]
  111. Al Jahdhami, M.A.; El-Saleh, A.; Alhammadi, A.; Shayea, I. Performance Analysis of Mobile Broadband Networks in Ibra City, Oman. In Proceedings of the 2021 International Conference on Artificial Intelligence and Big Data Analytics, Xi’an, China, 27–29 October 2021; pp. 1–6. [Google Scholar] [CrossRef]
  112. Fresolone, F.; Kloibhofer, R.; Ralbovsky, A.; Farkas, P.; Rakus, M.; Palenik, T. Throughput and one-way latency measurements in a 3G/4G live-network hi-mobility uplink. In Proceedings of the 2016 39th International Conference on Telecommunications and Signal Processing (TSP), Vienna, Austria, 27–29 June 2016; IEEE: New York, NY, USA, 2016; pp. 44–49. [Google Scholar]
  113. Albaladejo, M.B.; Leith, D.J.; Manzoni, P. Measurement-based modelling of lte performance in dublin city. In Proceedings of the 2016 IEEE 27th Annual International Symposium on Personal, Indoor, and Mobile Radio Communications (PIMRC), Valencia, Spain, 4–8 September 2016; IEEE: New York, NY, USA, 2016; pp. 1–6. [Google Scholar]
  114. Engiz, B.K.; Kurnaz, Ç. Comparison of Signal Strengths of 2G/3G/4G services on a University Campus. Int. J. Appl. Math. Electron. Comput. 2016, 4, 37–42. [Google Scholar] [CrossRef] [Green Version]
  115. Koprivica, L.Đ.M.; Nešković, N.; Nešković, A. Experimental performance analysis of THE 2G/3G/4G public mobile network. In Proceedings of the 2016 24th Telecommunications Forum (TELFOR), Belgrade, Serbia, 22–23 November 2016; pp. 1–4. [Google Scholar] [CrossRef]
  116. Skocir, P.; Katusic, D.; Novotni, I.; Bojic, I.; Jezic, G. Data rate fluctuations from user perspective in 4G mobile networks. In Proceedings of the 2014 22nd International Conference on Software, Telecommunications and Computer Networks (SoftCOM), Split, Croatia, 17–19 September 2014; IEEE: New York, NY, USA, 2014; p. 180185. [Google Scholar]
  117. Imoize, A.; Adegbite, O. Measurements-based performance analysis of a 4G LTE network in and around shopping malls and campus environments in Lagos Nigeria. Arid. Zone J. Eng. Technol. Environ. 2018, 14, 208. [Google Scholar]
  118. Nikravesh, A.; Choffnes, D.R.; Katz-Bassett, E.; Mao, Z.M.; Welsh, M. Mobile Network Performance from User Devices: A Longitudinal, Multidimensional Analysis. In Passive and Active Measurement; Cham, M.F., Kuzmanovic, A., Eds.; Springer International Publishing: Berlin/Heidelberg, Germany, 2014; pp. 12–22. [Google Scholar]
  119. Elmokashfi, A.; Kvalbein, A.; Xiang, J.; Evensen, K.R. Characterizing delays in Norwegian 3G networks. In International Conference on Passive and Active Network Measurement; Springer: Berlin/Heidelberg, Germany, 2012; pp. 136–146. [Google Scholar]
  120. Wang, X.; Xu, C.; Jin, W.; Zhao, G. A First Look at Cellular Network Latency in China. In Communications and Networking; Chen, Q., Meng, W., Zhao, L., Eds.; Springer International Publishing: Berlin/Heidelberg, Germany, 2018; pp. 339–348. [Google Scholar]
  121. Xu, Y.; Wang, Z.; Leong, W.K.; Leong, B. An End-to-End Measurement Study of Modern Cellular Data Networks. In Passive and Active Measurement; Faloutsos, M., Kuzmanovic, A., Eds.; Springer International Publishing: Berlin/Heidelberg, Germany, 2014; pp. 34–45. [Google Scholar]
  122. Schwind, A.; Midoglu, C.; Alay, Ö.; Griwodz, C.; Wamser, F. Dissecting the performance of YouTube video streaming in mobile networks. Int. J. Netw. Manag. 2020, 30, e2058. [Google Scholar] [CrossRef]
  123. Schwind, A.; Seufert, M.; Alay, Ö.; Casas, P.; Tran-Gia, P.; Wamser, F. Concept and implementation of video QoE measurements in a mobile broadband testbed. In Proceedings of the 2017 Network Traffic Measurement and Analysis Conference (TMA), Dublin, Ireland, 21–23 June 2017; IEEE: New York, NY, USA, 2017; pp. 1–6. [Google Scholar]
  124. Casas, P.; Seufert, M.; Wamser, F.; Gardlo, B.; Sackl, A.; Schatz, R. Next to you: Monitoring quality of experience in cellular networks from the end-devices. IEEE Trans. Netw. Serv. Manag. 2016, 13, 181–196. [Google Scholar] [CrossRef]
  125. Daengsi, T.; Chatchalermpun, S.; Praneetpolgrang, P.; Wuttidittachotti, P. A study of 4G network performance in Thailand referring to download speed. In Proceedings of the 2020 IEEE 10th Symposium on Computer Applications & Industrial Electronics (ISCAIE), Penang, Malaysia, 18–19 April 2020; IEEE: New York, NY, USA, 2020; pp. 160–163. [Google Scholar]
  126. Budiman, E.; Wicaksono, O. Measuring quality of service for mobile internet services. In Proceedings of the 2016 2nd International Conference on Science in Information Technology (ICSITech), Balikpapan, Indonesia, 26–27 October 2016; IEEE: New York, NY, USA, 2016; pp. 300–305. [Google Scholar]
  127. Soós, G.; Ficzere, D.; Varga, P.; Szalay, Z. Practical 5G KPI measurement results on a non-standalone architecture. In Proceedings of the Noms 2020–2020 IEEE/IFIP Network Operations and Management Symposium, Budapest, Hungary, 20–24 April 2020; IEEE: New York, NY, USA, 2020; pp. 1–5. [Google Scholar]
  128. Makino, I.; Wang, Z.; Terai, J.; Miki, N. Throughput and Delay Performance Measurements in Multi-Floor Building Employing Private LTE. IEEE Access 2022, 10, 24288–24301. [Google Scholar] [CrossRef]
  129. Narayanan, A.; Zhang, X.; Zhu, R.; Hassan, A.; Jin, S.; Zhu, X.; Zhang, X.; Rybkin, D.; Yang, Z.; Mao, Z.; et al. A variegated look at 5G in the wild: Performance, power, and QoE implications. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference, Virtual, 23–27 August 2021; pp. 610–625. [Google Scholar]
  130. Xu, D.; Zhou, A.; Zhang, X.; Wang, G.; Liu, X.; An, C.; Shi, Y.; Liu, L.; Ma, H. Understanding operational 5G: A first measurement study on its coverage, performance and energy consumption. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and Protocols for Computer Communication, New York, NY, USA, 10–14 August 2020; pp. 479–494. [Google Scholar]
  131. Pan, Y.; Li, R.; Xu, C. The first 5G-LTE comparative study in extreme mobility. Proc. ACM Meas. Anal. Comput. Syst. 2022, 6, 1–22. [Google Scholar]
  132. Fiandrino, C.; Martínez-Villanueva, D.J.; Widmer, J. Uncovering 5G Performance on Public Transit Systems with an App-based Measurement Study. In Proceedings of the ACM/IEEE International Conference on Modelling, Analysis and Simulation of Wireless and Mobile Systems, Montreal, QC, Canada, 24 October 2022; pp. 65–73. [Google Scholar]
  133. Rochman, M.I.; Sathya, V.; Nunez, N.; Fernandez, D.; Ghosh, M.; Ibrahim, A.S.; Payne, W. A comparison study of cellular deployments in Chicago and Miami using apps on smartphones. In Proceedings of the 15th ACM Workshop on Wireless Network Testbeds, Experimental evaluation & CHaracterization, New Orleans, LA, USA, 31 January–4 February 2022; pp. 61–68. [Google Scholar]
  134. Bauer, S.; Clark, D.; Lehr, W. Gigabit broadband measurement workshop report. SIGCOMM Comput. Commun. Rev. 2020, 50, 60–65. [Google Scholar] [CrossRef] [Green Version]
Electronics 12 01630 g001 550
Figure 1. Active mobile broadband subscription per 100 inhabitants: (a) by development status; (b) by region.
Figure 1. Active mobile broadband subscription per 100 inhabitants: (a) by development status; (b) by region.
Electronics 12 01630 g001
Electronics 12 01630 g002 550
Figure 2. The four QoS viewpoints.
Figure 2. The four QoS viewpoints.
Electronics 12 01630 g002
Electronics 12 01630 g003 550
Figure 3. A typical methodology for MBB drive test [40].
Figure 3. A typical methodology for MBB drive test [40].
Electronics 12 01630 g003
Electronics 12 01630 g004 550
Figure 4. Smartphones mounted in moving vehicles during drive test [40].
Figure 4. Smartphones mounted in moving vehicles during drive test [40].
Electronics 12 01630 g004
Electronics 12 01630 g005 550
Figure 5. Bag pack with six mounted smartphones.
Figure 5. Bag pack with six mounted smartphones.
Electronics 12 01630 g005
Electronics 12 01630 g006 550
Figure 6. Mobile applications during measurement: (a) Speedtest; (b) SamKnows.
Figure 6. Mobile applications during measurement: (a) Speedtest; (b) SamKnows.
Electronics 12 01630 g006
Electronics 12 01630 g007 550
Figure 7. The overall system architecture of the NNE platform [52].
Figure 7. The overall system architecture of the NNE platform [52].
Electronics 12 01630 g007
Electronics 12 01630 g008 550
Figure 8. NNE node with 4 modems connected [52].
Figure 8. NNE node with 4 modems connected [52].
Electronics 12 01630 g008
Electronics 12 01630 g009 550
Figure 9. Overview of the MONROE platform.
Figure 9. Overview of the MONROE platform.
Electronics 12 01630 g009
Electronics 12 01630 g010 550
Figure 10. Overview of the Simplified Raspberry Pi platform [50].
Figure 10. Overview of the Simplified Raspberry Pi platform [50].
Electronics 12 01630 g010
Electronics 12 01630 g011 550
Figure 11. MBB measurements challenges.
Figure 11. MBB measurements challenges.
Electronics 12 01630 g011
Table
Table 1. Summary of mobile broadband technologies.
Table 1. Summary of mobile broadband technologies.
GenerationModes/StandardsCommercial DeploymentMultiple AccessChannel BandwidthGross Bit RateMax Cell RangeTypical Capabilities
1GAMPS, TACS, etc.Circa 1979FDMA25 kHzNA Analogue voice
2GGSM 9001990FDMA/TDMA/FDD200 kHz14.4 kbps to 384 kbpsUp to 35 kmVoice, Short Message Service
GSM 1800
GSM 1900
IS-95CDMA/FDMA/FDD50 kHz
D-AMPS, GPRS, EDGE 30 kHz
3G
(IMT 2000)		WCDMA/UMTSCirca 2003FDD and TDD modeMultiples of 200 kHz up to 5 MHz384 kbps to 2 MbpsDepends on many variables. Up to 150 km possibleAudio and video streaming, web browsing, etc.
CDMA 2000, HSPA HSPA+FDMA/TDMA
4G
(IMT 2010)		LTE, LTE-A2010OFDMA, Multi-carrier CDMA5 MHz to 20 MHz2 Mbps to 1 GbpsHD video conferencing, gaming etc.
5G
(IMT 2020)		5G NR2019OFDMA, NOMA5 MHz to 100 MHz and beyond1 Gbps to 20 GbpsTens to a few thousand metersUltra HD and low latency applications
Table
Table 2. Growth in active mobile broadband (3G and above) subscription in millions by development status.
Table 2. Growth in active mobile broadband (3G and above) subscription in millions by development status.
Development Status2015201620172018201920202021% Growth
World328238634723531257456023654499%
Developed112612291381148515841625167849%
Developing2156263333423827416243984866125%
LDCs141192258292343384420198%
LLDCs93118156169191213236154%
SIDS2226313537394186%
Table
Table 3. Growth in active mobile broadband (3G and above) subscription in millions by region.
Table 3. Growth in active mobile broadband (3G and above) subscription in millions by region.
Regions2015201620172018201920202021% Growth
Africa180213253303356409448148%
Americas7718448949499961021106037%
Arab States16818422224526327929777%
Asia-Pacific1554193225882970322133743755142%
CIS14315517418620921423061%
Europe465 53358862967069672355%
Table
Table 4. Growth in active mobile broadband (3G and above) subscription in millions by region.
Table 4. Growth in active mobile broadband (3G and above) subscription in millions by region.
S/NCountryMobile Broadband SubscriptionsPenetration Increase
2020–2021		S/NCountryMobile Broadband SubscriptionsPenetration Increase 2020–2021
1Australia31,795,0000.2720Japan239,052,3825.12
2Austria10,583,3144.3621Korea60,721,1561.14
3Belgium10,822,3493.8322Latvia2,665,4941.37
4Canada28,647,8793.2923Lithuania3,412,5597.59
5Chile21,261,4868.4924Luxembourg713,5683.76
6Colombia36,767,0419.9825Mexico 108,835,9225.16
7Costa Rica4,501,028−3.6526Netherlands23,445,5526.04
8Czech Republic10,707,4785.6227New Zealand5,146,7039.29
9Denmark8,295,1733.9328Norway5,692,2092.90
10Estonia2,392,40715.1929Poland50,094,6806.68
11Finland8,700,0000.9630Portugal9,113,7288.51
12France67,728,0004.1531Slovak Republic4,833,9580.35
13Germany78,729,0003.2632Slovenia1,923,9644.46
14Greece9,875,4056.3733Spain 50,955,9643.71
15Hungary7,982,2696.7634Sweden13,253,7181.40
16Iceland442,4504.3935Switzerland8,827,222−0.17
17Ireland5,417,1624.3036Turkey70,029,0034.52
18Israel13,100,0006.6237United Kingdom76,230,2983.91
19Italy57,359,1012.2638United States558,699,87711.88
OECD TOTAL 1,708,754,4996.15
Table
Table 5. Technical and non-technical aspect of QoE.
Table 5. Technical and non-technical aspect of QoE.
Quality of Experience (QoE)
Technical AspectsNon-Technical Aspects
End-to-end network qualityDetermining the price according to service
Coverage areaSupport to customers
Equipment flexibility and functionalityService availability
Ease in the installation of service set-up
Table
Table 6. Advantages and disadvantages of mobile broadband measurement techniques.
Table 6. Advantages and disadvantages of mobile broadband measurement techniques.
MethodsAdvantagesDisadvantages
Drive Test
  • Consistency in data collection
  • Ability to control performance impacting variables
  • Identifies coverage holes and performance problems
  • Fairly expensive on a per measurement basis
  • It does not scale well
  • May not provide a complete end-to-end measurement of MBB performance
Walk Test
  • Provides a quick view of the QoS delivered to the user within the test locale
  • Useful where access to MBB is deemed to be very important
  • Cost of data collection is huge
  • Poor scalability
  • Limited by walk speed of the carrier
  • Places less stringent conditions on mobile networks
Crowdsourced
  • Very cost-effective on a per measurement basis
  • Highly scalable
  • Suitable for extended duration measurements
  • Ability to collect more data at different times of the day
  • Possibility of bias and error for tests initiated by humans
  • Issues of privacy signify that it may not provide some detailed context information and metadata
  • Users can operate the devices in an uncontrolled manner
Table
Table 7. Different methodologies adopted for BEREC countries.
Table 7. Different methodologies adopted for BEREC countries.
Countries and RegulatorsDrive TestingWalk TestingApp-Based
(Panel)(Crowdsourced)
Austria (RTR) NoNoYesYes
Belgium (BIPT) YesNoNot yetNot yet
Bulgaria (CRC) YesNoNoNo
Switzerland (BAKOM) NoNoNoNo
Cyprus (OCECPR) NoNoNoNo
Czech Republic (CTU) YesNoNoYes
Germany (BNetzA) YesNoNoNo
Denmark (DBA) NoNoNoNo
Estonia (ETRA) YesNoNoNo
Finland (FICORA)YesNoNoNo
France (Arcep)YesNoNoNo
Macedonia (AEC)YesNoNoNo
Greece (EETT)YesNoNoNo
Croatia (HAKOM) YesNoNoYes
Hungary (NMHH)YesNoNoNo
Ireland (ComReg)YesYesNo-
Iceland (PFS)YesNoNoNo
Italy (AGCOM)YesNoNoYes
Lithuania (RRT)YesNoNoNo
Latvia (SPRK) NoNoNoNo
Montenegro (EKIP)YesNoNoNo
Malta (MCA)YesNoNoNo
Netherlands (ACM)YesNoNoNo
Norway (NKOM) YesNoNoYes
Poland (UKE)YesNoNoNo
Portugal (ANACOM)YesNoNoYes
Romania (ANCOM)YesNoNoNo
Sweden (PTS)NoYesNoNo
Serbia (RATEL) YesYesNoYes
Slovakia (RU)YesNoNoNo
Slovenia (AKOS) YesYesNoYes
Turkey (ICTA)YesNoNoNo
United Kingdom (OFCOM)YesYesNoNo
Table
Table 8. Summary of existing mobile broadband performance evaluation works.
Table 8. Summary of existing mobile broadband performance evaluation works.
S/NRefs.Methodology AdoptedQoS MetricsAccess NetworksStudy Summary
1[69,70]CrowdsourcedDownload and upload throughput, latency and DNS lookup2G and 3GDeveloped a mobile phone application (MBPerf) and adopted a host and crowdsourced based approach to carry out a comparative analysis of the performance of four MNOs that offer MBB services in a developing country.
2[50,75]TestbedDownload and upload throughput and latency3G and 4GUsed Raspberry Pi to develop a simplified testbed and conducted a comparative analysis of the MBB performance offered by four MNOs.
3[81]CrowdsourcedDownload and upload speed, latency, packet loss and web loading time4GConducted MBB performance assessment during defined peak periods for Internet service providers using a panel-based crowdsourced method.
4[76,77]CrowdsourcedDownload and upload throughput, latency, packet loss, jitter and DNS resolution time4GProvides a comparative investigation of MBB performance using a panel-based crowdsourced method.
5[78]CrowdsourcedUpload and download throughput and latency4GPresented a comprehensive comparative study of user-centric MBB performance using a panel-based crowdsourced method.
6[18]TestbedLatency, packet loss and connectivity3G and 4GStudied the performance of MBB networks under mobility using a dedicated testbed for measurement
7[40]Drive testSignal quality, downlink and uplink throughput, ping and handover3G and 4GDeveloped and used a drive test method to evaluate and understand MBB performance in different locations
8[107]TestbedDownload speed3G and 4GPresented a “speedtest like” measurement to estimate the download speed offered by MBB networks to users.
9[108]Drive testSpeed, coverage, satisfaction and latency3G and 4GConducted performance analysis of MBB networks to enable planning for 5G network upgrade in a rural area
10[109]TestbedWeb QoE, throughput, latency, and signal coverage4GExamined the performance and response of nine mobile networks across Europe at different times during the COVID-19 pandemic
11[110]Drive testSpeed, coverage, satisfaction and latency3G and 4GConducted performance analysis of MBB networks to enable planning of 5G network upgrade urban area
12[84]Walk testReceived signal strength3G and 4GMeasured and characterized MBB performance through an indoor walk test
13[111]Drive testSignal quality, Downlink and uplink throughput and Ping3G and 4GEvaluated the MBB performance and coverage of existing MBB networks of different MNOs.
14[112]Drive testThroughput and latency3G and 4GMeasured real characteristics and coverage of MBB networks using a custom in-house made software tool.
15[113]TestbedLatency and signal quality4GAdapted data from extensive MBB measurement campaigns to develop a model suitable for realistic performance evaluation of applications and services.
16[114]Drive testSignal strength2G, 3G and 4GDetermined and compared the signal strength of MBB networks of two MNOs.
17[115]Drive testThroughput2G, 3G and 4GMeasured the performance of MBB networks using a custom mobile phone application.
18[82]Drive testThroughput and latency3G and 4GPresented a comparative analysis of real MBB networks under mobility with expected theoretical expectations in order to identify the gaps between both.
19[116]CrowdsourcedThroughput rates4GExamined user perceived data rate fluctuations in 4G networks during different periods as well as compared the performance of MNOs
20[85]Drive test and walk testThroughput and RTT3G and 4GPerformed coverage and capacity measurement of MBB networks in pedestrian zones during both busy hour and non-busy hour
21[117]Drive testSignal quality and downlink and uplink throughput4GConducted performance analysis of 4G MBB networks and observed the propagation measurement of key performance indicators using drive test
22[79]Crowdsourced and testbedThroughput4GProposed a supervise machine learning solution for a more accurate throughput estimation.
23[80]CrowdsourcedDownload and upload data rate, latency, signal strength4G Developed a Machine Learning (ML) based framework used to define and determine different behavior of MNOs from crowdsourced datasets.
24[118]CrowdsourcedThroughput, latency and DNS lookup4GConducted a longitudinal and multidimensional analysis of the extensive MBB measurement data collected to diagnose the cause behind observed performance variations.
25[119]TestbedLatency3GExamined delay characteristics in 3G networks from long-term MBB measurement data.
26[120]CrowdsourcedLatency2G to 4GAnalyzed latency of MBB networks from measurement data obtained using the crowdsourced method.
27[121]CrowdsourcedThroughput and latency3G and 4GUsed crowdsourced measurement data to study the characteristics of MBB network of three MNOs.
28[122]TestbedVideo streaming4GInvestigated the influence of different factors on YouTube streaming performance with different network configurations in four countries.
29[123]TestbedVideo streaming4GPresented the design and implementation of a large-scale measurement tool for QoE when live streaming with MBB networks.
30[124]CrowdsourcedThroughput and latency of mobile apps4GExtensively addressed the problem of QoE provisioning in smartphones from a double perspective, combining the results obtained from subjective laboratory tests with end-device passive MBB measurements and QoE crowd-sourced feedback obtained.
31[87]Drive testUpload and download throughput, latency and packet loss5GConducted a pilot MBB measurement on the 5G network to investigate the QoS parameters of two MNOs.
32[86]CrowdsourcedUpload and download throughput, latency, jitter and packet loss4GPerformed a comparative assessment of the QoS parameters obtained from 4G MBB network and used it to establish a baseline for 5G MBB evaluation.
33[88]Drive testUpload and download speed5GPerformed both stationary and mobility field tests to study the efficiency and performance of 5G networks.
34[89]Drive testUpload and download speeds and latency5GConducted stationary field test to assess the MBB QoS performance of the 5G network using three popular mobile phone speedtest applications.
35[125]CrowdsourcedDownload speed4GConducted a comparative study of download speeds on the 4G networks.
36[126]TestbedLatency, jitter, packet loss, and throughput3GConducted a study to obtain the end-to-end parameters of the QoS for internet usage from a user perspective.
37[83]TestbedLatency, upload throughput, handover4GMeasured the key MBB performance metrics of 4G networks under mobility, highlighting inefficiencies that need to be considered when designing the mobility features in 5G networks.
38[127]TestbedThroughput, latency, jitter4G and 5GPresents initial MBB measurement results of the key performance indicators on the 5G network.
39[128]Walk TestThroughput, latency 4G and 5GUsed private LTE and 5G networks to measure MBB performance metrics.
40[90]CrowdsourcedHandover, Mobile app performance5GPerformed extensive field tests of 5G network performance in different urban areas.
41[129]Crowdsourced Throughput, latency, handover 5GUsed a custom measurement tool to conduct a comprehensive measurement of numerous key aspects of commercial end-to-end 5G network performance.
42[130]TestbedThroughput, latency, coverage5GConducted a full-fledged, end-to-end measurement study of the first commercial 5G networks.
43[131]Testbed and Drive testThroughput, RTT, loss rate, signal quality4G and 5GPerformed a comparative study of the key performance metrics of 5G in extreme mobility.
44[132]Drive testThroughput, latency, video streaming 5GConducted an in-depth measurement study of 5G network performance in transit.
45[133]Crowdsourced Throughput, latency4G and 5GPerformed a comparative study of 4G and 5G network deployment in two cities using a panel-based approach.
46[100]TestbedThroughput, latency5GDeveloped a modular and flexible experimentation methodology for validating 5G network KPIs.
Table
Table 9. Comparison of the different MBB measurement methods.
Table 9. Comparison of the different MBB measurement methods.
FeaturesDrive TestWalk TestCrowdsourcedTestbed
ComplexityModerateSimpleModerateComplex
AccuracyAccurateAccurate (for the given purpose)Accurate (relying on users)Accurate
Scalability ModerateLowHighVery High
Reliability and sensitivityModerate Low ModerateHigh
Per measurement cost HighHighModerateModerate
Set up costModerateLow High (relative to the coverage) Very High
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Umoh, V.; Ekpe, U.; Davidson, I.; Akpan, J. Mobile Broadband Adoption, Performance Measurements and Methodology: A Review. Electronics 2023, 12, 1630. https://doi.org/10.3390/electronics12071630

AMA Style

Umoh V, Ekpe U, Davidson I, Akpan J. Mobile Broadband Adoption, Performance Measurements and Methodology: A Review. Electronics. 2023; 12(7):1630. https://doi.org/10.3390/electronics12071630

Chicago/Turabian Style

Umoh, Vincent, Unwana Ekpe, Innocent Davidson, and Joseph Akpan. 2023. "Mobile Broadband Adoption, Performance Measurements and Methodology: A Review" Electronics 12, no. 7: 1630. https://doi.org/10.3390/electronics12071630

APA Style

Umoh, V., Ekpe, U., Davidson, I., & Akpan, J. (2023). Mobile Broadband Adoption, Performance Measurements and Methodology: A Review. Electronics, 12(7), 1630. https://doi.org/10.3390/electronics12071630

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop