Abstract
The volumes of data collected have been increasing in recent years as a result of the emergence of new data sources such as sensors, social networks, among others. The data collected has a varied format (semi-structured, unstructured and structured) and, due to the speed at which they are generated, organisations cannot derive value from their data, as the data in its original form has little or no value. As a result of this new reality, the need has arisen for organisations to be able to process the large volumes of data generated, regardless of the type of format (structured or unstructured) they have, and to analyse them to obtain crucial information for the business. Additionally, some organisations need to be able to access information in real-time for later take informed decisions in good time. Thus, the focus of this article is on developing a refresh mechanism capable of synchronising different types of data and thereby standardising a global mechanism. The time it takes for the mechanism to go through all the dimensions and make the changes to a specific dimension does not total one hour - which gives a time difference of approximately 96%. It can be seen that there is a rather significant improvement with the data refresh mechanism developed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Simmhan, Y., Perera, S.: Big data analytics platforms for real-time applications in IoT. In: Pyne, S., Rao, B.L.S.P., Rao, S.B. (eds.) Big Data Analytics, pp. 115–135. Springer, New Delhi (2016). http://link.springer.com/10.1007/978-81-322-3628-3_7. Obtido Janeiro 10 2020
Krishnan, K.: Data Warehousing in the Age of Big Data. Morgan Kaufmann is an imprint of Elsevier, Amsterdam (2013)
Hurwitz, J., Nugent, A., Halper, F., Kaufman, M.: Big Data for Dummies. Wiley, Hoboken (2013)
Zikopoulos, P., Eaton, C., Deroos, D., Deutsch, T., Lapis, G.: Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data, p. 176. McGraw-Hill Osborne Media, New York (2011)
Loukides, M.: What Is Data Science? O’Reilly Media, Inc., New York (2011)
Stanton, J.M.: Introduction to Data Science, p. 197. Morgan Kaufmann, Cambridge (2013)
Baier, D., Decker, R., Schmidt-Thieme, L.: Data Analysis and Decision Support. Springer, Heidelberg (2006)
Steele, B., Chandler, J., Reddy, S.: Algorithms for Data Science. Springer, Cham (2016). http://link.springer.com/10.1007/978-3-319-45797-0. Obtido Novembro 12, 2019
Loukides, M.K.: ProQuest: What Is Data Science? O’Reilly Media, Sebastopol (2012). https://VH7QX3XE2P.search.serialssolutions.com/?V=1.0&L=VH7QX3XE2P&S=JCs&C=TC0001454060&T=marc&tab=BOOKS. Obtido Novembro 12, 2019
Kurkovsky, S.: Pervasive Computing: past, present and future. In: 2007 ITI 5th International Conference on Information and Communications Technology. Presented at 2007 ITI 5th International Conference on Information and Communications Technology, pp. 65–71. IEEE, Cairo (2007). https://ieeexplore.ieee.org/document/4475619/. Obtido Dezembro 26, 2019
Fernandes, G., Portela, F., Santos, M.F.: PWA and pervasive information system – a new era. In: Rocha, Á., Adeli, H., Reis, L.P., Costanzo, S., Orovic, I., Moreira, F. (eds.) Trends and Innovations in Information Systems and Technologies. Advances in Intelligent Systems and Computing, vol. 1161, pp. 334–343. Springer, Cham (2020). http://link.springer.com/10.1007/978-3-030-45697-9_33. Obtido Novembro 15, 2020
Reddy, Y.: Pervasive computing: implications, opportunities and challenges for the society. In: 2006 First International Symposium on Pervasive Computing and Applications. Presented at 2006 First International Symposium on Pervasive Computing and Applications, p. 5. IEEE, Urumqi (2006). https://ieeexplore.ieee.org/document/4079026/. Obtido Dezembro 26, 2019
Davies, N., Clinch, S.: Pervasive data science. IEEE Pervasive Comput. 16(3), 50–58 (2017)
Humphries, M., Hawkins, M.W., Dy, M.C.: Data Warehousing: Architecture and Implementation. Harris Kern’s Enterprise Computing Institute. Prentice Hall PTR, Upper Saddle River (1999)
Kimball, R., Ross, M.: The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling, 3rd edn. Wiley, Hoboken (2013)
Fernandes, G., Portela, F., Santos, M.F.: Towards the development of a Data Science Modular Solution, p. 8 (2019)
Lin, Y., Jun, Z., Hongyan, M., Zhongwei, Z., Zhanfang, F.: A method of extracting the semi-structured data implication rules. Procedia Comput. Sci. 131, 706–716 (2018)
Gokalp, M.O., Kocyigit, A., Eren, P.E.: A cloud-based architecture for distributed real-time processing of continuous queries. In: 2015 41st Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 459–462. IEEE, Madeira (2020). https://ieeexplore.ieee.org/document/7302489/. Obtido Janeiro 8, 2020
Loyola, R.C., Sepulveda, A.U., Hernandez, M.W.: Optimisation slowly changing dimensions of a data warehouse using object-relational. In: 2015 34th International Conference of the Chilean Computer Science Society (SCCC), pp. 1–6. IEEE, Santiago (2015). https://ieeexplore.ieee.org/document/7416593/. Obtido Abril 14, 2020
Acknowledgements
This work has been supported by FCT - Fundação para a Ciência e Tecnologia within the R&D Units Project Scope: UIDB/00319/2020. This project was also supported by IOTech - Innovation on Technology.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ferreira, V., Portela, F., Santos, M.F. (2021). A Practical Solution to Synchronise Structured and Non-structured Repositories. In: Rocha, Á., Adeli, H., Dzemyda, G., Moreira, F., Ramalho Correia, A.M. (eds) Trends and Applications in Information Systems and Technologies. WorldCIST 2021. Advances in Intelligent Systems and Computing, vol 1368. Springer, Cham. https://doi.org/10.1007/978-3-030-72654-6_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-72654-6_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-72653-9
Online ISBN: 978-3-030-72654-6
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)