Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

A Workload-Aware Change Data Capture Framework for Data Warehousing

  • Conference paper
  • First Online:
Big Data Analytics and Knowledge Discovery (DaWaK 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12925))

Included in the following conference series:

  • 875 Accesses

Abstract

Today’s data warehousing requires continuous or on-demand data integration through a Change-Data-Capture (CDC) process to extract data deltas from Online Transaction Processing Systems. This paper proposes a workload-aware CDC framework for on-demand data warehousing. This framework adopts three CDC strategies, namely trigger-based, timestamp-based and log-based, which allows capturing data deltas by taking into account the workloads of source systems. This paper evaluates the framework comprehensively, and the results demonstrate its effectiveness in terms of quality of service, including throughput, latency and staleness.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Hamdi, I., Bouazizi, E., Alshomrani, S., Feki, J.: Improving QoS in real-time data warehouses by using feedback control scheduling. J. Inf. Deci. Sci. 10(3), 181–211 (2018)

    Google Scholar 

  2. Kozielski, S., Wrembel, R. (eds): New Trends in Data Warehousing and Data Analysis. Springer (2009). https://doi.org/10.1007/978-0-387-87431-9

  3. Liu, X.: Data warehousing technologies for large-scale and right-time data. Aalborg University, Defensed on June (2012)

    Google Scholar 

  4. Liu, X., Iftikhar, N.: An ETL optimization framework using partitioning and parallelization. In: Proceedings of the 30th SAC, pp. 1015–1022 (2015)

    Google Scholar 

  5. Shi, J., Guo, S., Luan, F., Sun, L.: Qos-ls: Qos-based load scheduling algorithm in real-time data warehouse. In: Proceedings of the 5th ICMMCT (2017)

    Google Scholar 

  6. Thiele, M., Fischer, U., Lehner, W.: Partition-based workload scheduling in living data warehouse environments. Inf. Syst. 34(4–5), 382–399 (2009)

    Article  Google Scholar 

  7. Thomsen, C., Pedersen, T.B., Lehner, W.: Rite: providing on-demand data for right-time data warehousing. In: Proceedings of the 24th ICDE, pp. 456–465 (2008)

    Google Scholar 

  8. Vassiliadis, P., Simitsis, A.: Near real time ETL. In: New Trends in Data Warehousing and Data Analysis, pp. 1–31. Springer (2009). https://doi.org/10.1007/978-0-387-87431-9

Download references

Acknowledgement

This research was supported by the HEAT 4.0 project (8090-00046b) funded by Innovationsfonden.

Author information

Authors and Affiliations

  1. Heterogeneous Information Systems Group, University of Kaiserslautern, Kaiserslautern, Germany

    Weiping Qu & Stefan Dessloch

  2. Department of Technology, Management and Economics, Technical University of Denmark, 2800, Lyngby, Denmark

    Xiufeng Liu

Authors
  1. Weiping Qu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiufeng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefan Dessloch
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiufeng Liu .

Editor information

Editors and Affiliations

  1. University of Bologna, Bologna, Forli/Cesena, Italy

    Matteo Golfarelli

  2. Poznań University of Technology, Poznan, Poland

    Robert Wrembel

  3. Johannes Kepler University Linz, Linz, Austria

    Gabriele Kotsis

  4. TU Wien, Vienna, Austria

    A Min Tjoa

  5. Johannes Kepler University Linz, Linz, Austria

    Ismail Khalil

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Qu, W., Liu, X., Dessloch, S. (2021). A Workload-Aware Change Data Capture Framework for Data Warehousing. In: Golfarelli, M., Wrembel, R., Kotsis, G., Tjoa, A.M., Khalil, I. (eds) Big Data Analytics and Knowledge Discovery. DaWaK 2021. Lecture Notes in Computer Science(), vol 12925. Springer, Cham. https://doi.org/10.1007/978-3-030-86534-4_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-86534-4_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-86533-7

  • Online ISBN: 978-3-030-86534-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation