Abstract
General-purpose library application programming interfaces (APIs) for self-describing hierarchical scientific data storage, such as the HDF5 and NetCDF libraries, are traditionally of runtime nature. Runtime errors for entry existence and data types are typically caught later in the development process of higher-level application-specific APIs. In this paper, we propose exploiting modern C++ metaprogramming features to add compile-time type-safety to improve the interaction with a well-defined metadata-rich scientific schema in domain-specific hierarchical datasets. We tackle two aspects of common use: (i) direct data access, (ii) flexible “in-memory” index models for efficient search and data processing. The proposed APIs use C++17’s template type auto deduction features, C++11’s enum class for type-safety and C-style preprocessor macros for generative templated code. We showcase the pros and cons of our initial work on the standard NeXus schema used for annotating and storing experimental neutron scattering data at several facilities around the world on top of HDF5. Extendable compile-time type-safe APIs are a desirable feature that could be indexed by any modern integrated development environment (IDE). Hence, such APIs can help ease the learning curve for domain scientists using a less error-prone software interaction to enhance the findability of their data without resorting to a domain-specific language (DSL).
The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a nonexclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan). Work at Oak Ridge National Laboratory was sponsored by the Division of Scientific User Facilities, Office of Basic Energy Sciences, US Department of Energy, under Contract no. DE-AC05-00OR22725 with UT-Battelle, LLC.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Antcheva I, Ballintijn M, Bellenot B, Biskup M, Brun R, Buncic N, Canal P, Casadei D, Couet O, Fine V, Franco L, Ganis G, Gheata A, Maline DG, Goto M, Iwaszkiewicz J, Kreshuk A, Segura DM, Maunder R, Moneta L, Naumann A, Offermann E, Onuchin V, Panacek S, Rademakers F, Russo P, Tadel M (2009) ROOT—a C++ framework for petabyte data storage, statistical analysis and visualization. Comput Phys Commun 180(12):2499–2512. https://doi.org/10.1016/j.cpc.2009.08.005. 40 YEARS OF CPC: A celebratory issue focused on quality software for high performance, grid and novel computing architectures
Byna S, Breitenfeld MS, Dong B, Koziol Q, Pourmal E, Robinson D, Soumagne J, Tang H, Vishwanath V, Warren R (2020) ExaHDF5: delivering efficient parallel I/O on exascale computing systems. J Comput Sci Technol 35(1). https://doi.org/10.1007/s11390-020-9822-9
Eclipse Foundation: Eclipse CDT (C/C++ development tooling). https://www.eclipse.org/cdt/
Folk M, Heber G, Koziol Q, Pourmal E, Robinson D (2011) An overview of the HDF5 technology suite and its applications. In: Proceedings of the EDBT/ICDT 2011 workshop on array databases, AD ’11. Association for Computing Machinery, New York, NY, USA, pp 36–47. https://doi.org/10.1145/1966895.1966900
Gaur N, Mahammad S, Sharma V, Dhar D, Ramakrishnan R (2012) Template based HDF5 satellite digital data product generation software. In: 2012 international conference on data science and engineering (ICDSE), pp 160–165. https://doi.org/10.1109/ICDSE.2012.6281903
Godoy WF, Peterson PF, Hahn SE, Billings JJ (2020) Efficient data management in neutron scattering data reduction workflows at ORNL. In: 2020 IEEE international conference on big data (big data), pp 2674–2680. https://doi.org/10.1109/BigData50022.2020.9377836
Godoy WF, Peterson PF, Hahn SE, Hetrick J, Doucet M, Billings JJ (2020) Performance improvements on SNS and HFIR instrument data reduction workflows using Mantid. In: Nichols J, Verastegui B, Maccabe AB, Hernandez O, Parete-Koon S, Ahearn T (eds) Driving scientific and engineering discoveries through the convergence of HPC, big data and AI. Springer, Cham, pp 175–186
Godoy WF, Podhorszki N, Wang R, Atkins C, Eisenhauer G, Gu J, Davis P, Choi J, Germaschewski K, Huck K, Huebl A, Kim M, Kress J, Kurc T, Liu Q, Logan J, Mehta K, Ostrouchov G, Parashar M, Poeschel F, Pugmire D, Suchyta E, Takahashi K, Thompson N, Tsutsumi S, Wan L, Wolf M, Wu K, Klasky S (2020) ADIOS 2: the adaptable input output system. A framework for high-performance data management. SoftwareX 12:100561. https://doi.org/10.1016/j.softx.2020.100561
Godoy WF, Savici AT, Hahn SE, Peterson PF (2021) Efficient loading of reduced data ensembles produced at ORNL SNS/HFIR neutron time-of-flight facilities. In: (2021) IEEE International Conference on Big Data (Big Data), 2021, pp. 2949–2955. https://doi.org/10.1109/BigData52589.2021.9671354
Golasowski M, Bispo J, Martinovič J, Slaninová K, Cardoso JMP (2017) Expressing and applying C++ code transformations for the HDF5 API through a DSL. In: Saeed K, Homenda W, Chaki R (eds) Computer information systems and industrial management. Springer, Cham, pp 303–314
Gosink L, Shalf J, Stockinger K, Wu K, Bethel W (2006) HDF5-FastQuery: accelerating complex queries on HDF datasets using fast bitmap indices. In: 18th international conference on scientific and statistical database management (SSDBM’06), pp 149–158. https://doi.org/10.1109/SSDBM.2006.27
Gray J, Liu DT, Nieto-Santisteban M, Szalay A, DeWitt DJ, Heber G (2005) Scientific data management in the coming decade. SIGMOD Rec 34(4):34–41. https://doi.org/10.1145/1107499.1107503
Guennebaud G, Jacob B et al (2010) Eigen v3. https://eigen.tuxfamily.org
Hinsen K (2013) Software development for reproducible research. Comput Sci Eng 15(4):60–63. https://doi.org/10.1109/MCSE.2013.91
Ishwarappa Anuradha J (2015) A brief introduction on big data 5Vs characteristics and Hadoop technology. Procedia Comput Sci 48:319–324. https://doi.org/10.1016/j.procs.2015.04.188
Könnecke M, Akeroyd FA, Bernstein HJ, Brewster AS, Campbell SI, Clausen B, Cottrell S, Hoffmann JU, Jemian PR, Männicke D, Osborn R, Peterson PF, Richter T, Suzuki J, Watts B, Wintersberger E, Wuttke J (2015) The NeXus data format. J Appl Crystallogr 48(1):301–305. https://doi.org/10.1107/S1600576714027575
Lofstead J, Ross R (2013) Insights for exascale IO APIs from building a petascale IO API. In: SC ’13: proceedings of the international conference on high performance computing, networking, storage and analysis, pp 1–12. https://doi.org/10.1145/2503210.2503238
Nam B, Sussman A (2003) Improving access to multi-dimensional self-describing scientific datasets. In: CCGrid 2003. 3rd IEEE/ACM international symposium on cluster computing and the grid, 2003. Proceedings, pp 172–179. https://doi.org/10.1109/CCGRID.2003.1199366
Rew R, Davis G (1990) NetCDF: an interface for scientific data access. IEEE Comput Graph Appl 10(4):76–82. https://doi.org/10.1109/38.56302
Ritchie DM, Thompson K (1974) The UNIX time-sharing system. Commun ACM 17(7):365–375. https://doi.org/10.1145/361011.361061
Sanderson C, Curtin R (2016) Armadillo: a template-based C++ library for linear algebra. J Open Source Soft 1(2):26. https://doi.org/10.21105/joss.00026
Silvano C, Agosta G, Bartolini A, Beccari AR, Benini L, Besnard L, Bispo J, Cmar R, Cardoso JM, Cavazzoni C, Cesarini D, Cherubin S, Ficarelli F, Gadioli D, Golasowski M, Libri A, Martinovič J, Palermo G, Pinto P, Rohou E, Slaninová K, Vitali E (2019) The ANTAREX domain specific language for high performance computing. Microprocessors Microsyst 68:58–73. https://doi.org/10.1016/j.micpro.2019.05.005
Spreckelsen F, Rüchardt B, Lebert J, Luther S, Parlitz U, Schlemmer A (2020) Guidelines for a standardized filesystem layout for scientific data. Data 5(2). https://doi.org/10.3390/data5020043
Stroustrup B (2013) The C++ programming language, 4th edn. Addison-Wesley Professional
Sutton A, Holeman R, Maletic JI (2009) Abstracting the template instantiation relation in C++. In: 2009 IEEE international conference on software maintenance, pp 559–562. https://doi.org/10.1109/ICSM.2009.5306392
The HDF Group. Hierarchical data format, version 5 (1997-NNNN). http://www.hdfgroup.org/HDF5/
Touton J, Spertus M (2016) Declaring non-type template parameters with auto. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0127r2.html
Tratt L (2008) Domain specific language implementation via compile-time meta-programming. ACM Trans Program Lang Syst 30(6). https://doi.org/10.1145/1391956.1391958
Varga S: Easy to use HDF5 C++ templates for serial and parallel HDF5. https://github.com/steven-varga/h5cpp
Wells D, Greisen E, Harten R (1981) FITS—a flexible image transport system. Astron Astrophys Suppl 44:363
Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, Blomberg N, Boiten JW, da Silva Santos LB, Bourne PE et al (2016) The FAIR guiding principles for scientific data management and stewardship. Sci Data 3
Zhang W, Byna S, Niu C, Chen Y (2019) Exploring metadata search essentials for scientific data management. In: 2019 IEEE 26th international conference on high performance computing, data, and analytics (HiPC), pp 83–92
Zhang W, Byna S, Tang H, Williams B, Chen Y (2019) Miqs: metadata indexing and querying service for self-describing file formats. In: Proceedings of the international conference for high performance computing, networking, storage and analysis, SC ’19. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3295500.3356146
Acknowledgements
Work at Oak Ridge National Laboratory was sponsored by the Division of Scientific User Facilities, Office of Basic Energy Sciences, US Department of Energy, under Contract no. DE-AC05-00OR22725 with UT-Battelle, LLC.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 This is a U.S. government work and not under copyright protection in the U.S.; foreign copyright protection may apply
About this paper
Cite this paper
Godoy, W.F., Thakur, A.M., Hahn, S.E. (2022). Domain-Specific Type-Safe APIs for Hierarchical Scientific Data with Modern C++. In: Mathew, J., Santhosh Kumar, G., P., D., Jose, J.M. (eds) Responsible Data Science. Lecture Notes in Electrical Engineering, vol 940. Springer, Singapore. https://doi.org/10.1007/978-981-19-4453-6_14
Download citation
DOI: https://doi.org/10.1007/978-981-19-4453-6_14
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-19-4452-9
Online ISBN: 978-981-19-4453-6
eBook Packages: Computer ScienceComputer Science (R0)