Abstract
This paper presents the main challenges and issues faced when collecting and analyzing a large volume of network data measurements. We refer in particular to data collected by means of Neubot, an open source project that uses active probes on the client side to measure the evolution of key network parameters over time to better understand the performance of end-users’ Internet connections. The measured data are already freely accessible and stored on Measurement Lab (M-Lab), an organization that provides dedicated resources to perform network measurements and diagnostics in the Internet. Given the ever increasing amount of data collected by the Neubot project as well as other similar projects hosted by M-Lab, it is necessary to improve the platform to efficiently handle the huge amount of data that is expected to come in the very near future, so that it can be used by researchers and end-users themselves to gain a better understanding of network behavior.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Palfrey, J., Zittrain, J.: Better data for a better Internet. Science 334(6060), 1210–1211 (2011)
Sundaresan, S., de Donato, W., Feamster, N., Teixeira, R., Crawford, S., Pescapè, A.: Broadband internet performance: a view from the gateway. ACM SIGCOMM Computer Communication Review 41(4), 134–145 (2011)
Speedtest.net: The global broadband speed test, http://speedtest.net
Carlson, R.: Network diagnostic tool, http://e2epi.internet2.edu/ndt/
Neubot: The network neutrality bot, http://neubot.org/
Basso, S., Servetti, A., De Martin, J.C.: Rationale, design, and implementation of the network neutrality bot. In: Proc. of AICA, L’Aquila, Italy (September 2010)
Basso, S., Servetti, A., De Martin, J.C.: The network neutrality bot architecture: a preliminary approach for self-monitoring of Internet access QoS. In: Proc. of IEEE 16th International Symposium on Computers and Communications (ISCC), Corfu, Greece (July 2011)
Measurement Lab, http://www.measurementlab.net/
Dovrolis, C., Gummadi, K., Kuzmanovic, A., Meinrath, S.D.: Measurement lab: Overview and an invitation to the research community. ACM SIGCOMM Computer Communication Review 40(3), 53–56 (2010)
Meeker, M.: Internet trends. In: D10 Conference, Rancho Palos Verdes, CA, USA (May 2012)
Google: Cloud Storage, https://developers.google.com/storage/
Google: Big Query, https://developers.google.com/bigquery/
Web 100 project, http://www.web100.org/
Asymco: When will Android reach one billion users? (February 2012), http://www.asymco.com/2012/02/29/when-will-android-reach-one-billion-users/
Spring, N., Wetherall, D., Anderson, T.: Reverse engineering the Internet. ACM SIGCOMM Computer Communication Review 34(1), 8 (2004)
Kanuparthy, P., Dovrolis, C.: ShaperProbe: end-to-end detection of ISP traffic shaping using active methods. In: Proceedings of the 2011 ACM SIGCOMM Conference on Internet Measurement Conference, pp. 473–482. ACM (2011)
Apache: Mahout project (2013), http://mahout.apache.org
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Masala, E., Servetti, A., Basso, S., De Martin, J.C. (2014). Challenges and Issues on Collecting and Analyzing Large Volumes of Network Data Measurements. In: Catania, B., et al. New Trends in Databases and Information Systems. Advances in Intelligent Systems and Computing, vol 241. Springer, Cham. https://doi.org/10.1007/978-3-319-01863-8_23
Download citation
DOI: https://doi.org/10.1007/978-3-319-01863-8_23
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-01862-1
Online ISBN: 978-3-319-01863-8
eBook Packages: EngineeringEngineering (R0)