Abstract
The Agri sector has shown an exponential growth in both the requirement for and the production and availability of data. In parallel with this growth, Agri organisations often have a need to integrate their in-house data with international, web-based datasets. Generally, data is freely available from official government sources but there is very little unity between sources, often leading to significant manual overhead in the development of data integration systems and the preparation of reports. While this has led to an increased use of data warehousing technology in the Agri sector, the issues of cost in terms of both time to access data and the financial costs of generating the Extract-Transform-Load layers remain high. In this work, we examine more lightweight data marts in an infrastructure which can support on-demand queries. We focus on the construction of data marts which combine both enterprise and web data, and present an evaluation which verifies the transformation process from source to data mart.
Research funded by Science Foundation Ireland under grant number SFI/12/RC/2289.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
DIT Agriculture Analytics Research Group (2018). http://www.agrianalytics.org/
Bruckner, R.M., List, B., Schiefer, J.: Striving towards near real-time data integration for data warehouses. In: Kambayashi, Y., Winiwarter, W., Arikawa, M. (eds.) DaWaK 2002. LNCS, vol. 2454, pp. 317–326. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-46145-0_31
UN Comtrade (2018). https://comtrade.un.org//
Eurostat: Your key to European statistics (2018). http://ec.europa.eu/eurostat/about/overview
Kargın, Y., Pirk, H., Ivanova, M., Manegold, S., Kersten, M.: Instant-on scientific data warehouses. In: Castellanos, M., Dayal, U., Rundensteiner, E.A. (eds.) BIRTE 2012. LNBIP, vol. 154, pp. 60–75. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39872-8_5
Kepak Group (2018). https://www.kepak.com/
MCR Agri Analytics (2018). http://www.mcragrianalytics.com/
McCarren, A., McCarthy, S., Sullivan, C.O., Roantree, M.: Anomaly detection in agri warehouse construction. In: Proceedings of the ACSW, pp. 1–17. ACM Press (2017)
McCarthy, S., McCarren, A., Roantree, M.: An Architecture and Services for Constructing Data Marts from Online Data Sources. Insight Report 2018–1, April 2018. http://doras.dcu.ie/22386/1/DEXA-TechnicalReport-2018.pdf
Skoutas, D., Simitsis, A., Sellis, T.: Ontology-driven conceptual design of ETL processes using graph transformations. J. Data Sem. 13, 120–146 (2009)
Statistics Canada (2018). https://www.statcan.gc.ca/eng/start
United States Department of Agriculture (2018). http://www.ers.usda.gov/data-products/chart-gallery/detail.aspx?chartId=40037
Xie, N., Wang, W., Ma, B., Zhang, X., Sun, W., Guo, F.: Research on an agricultural knowledge fusion method for big data. In: Data Science Journal (2015)
Zhu, Y., An, L., Liu, S.: Data updating and query in real-time data warehouse system. In: 2008 International Conference on Computer Science and Software Engineering, CSSE 2008, Wuhan, China, pp. 1295–1297 (2008)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
McCarthy, S., McCarren, A., Roantree, M. (2018). Combining Web and Enterprise Data for Lightweight Data Mart Construction. In: Hartmann, S., Ma, H., Hameurlain, A., Pernul, G., Wagner, R. (eds) Database and Expert Systems Applications. DEXA 2018. Lecture Notes in Computer Science(), vol 11030. Springer, Cham. https://doi.org/10.1007/978-3-319-98812-2_10
Download citation
DOI: https://doi.org/10.1007/978-3-319-98812-2_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-98811-5
Online ISBN: 978-3-319-98812-2
eBook Packages: Computer ScienceComputer Science (R0)