SASH: Enabling continuous incremental analytic workflows on Hadoop
M Sethi, N Sachindran… - 2013 IEEE 29th …, 2013 - ieeexplore.ieee.org
M Sethi, N Sachindran, S Raghavan
2013 IEEE 29th International Conference on Data Engineering (ICDE), 2013•ieeexplore.ieee.orgThere is an emerging class of enterprise applications in areas such as log data analysis,
information discovery, and social media marketing that involve analytics over large volumes
of unstructured and semi-structured data. These applications are leveraging new analytics
platforms based on the MapReduce framework and its open source Hadoop
implementation. While this trend has engendered work on high-level data analysis
languages, NoSQL data stores, workflow engines etc., there has been very little attention to …
information discovery, and social media marketing that involve analytics over large volumes
of unstructured and semi-structured data. These applications are leveraging new analytics
platforms based on the MapReduce framework and its open source Hadoop
implementation. While this trend has engendered work on high-level data analysis
languages, NoSQL data stores, workflow engines etc., there has been very little attention to …
There is an emerging class of enterprise applications in areas such as log data analysis, information discovery, and social media marketing that involve analytics over large volumes of unstructured and semi-structured data. These applications are leveraging new analytics platforms based on the MapReduce framework and its open source Hadoop implementation. While this trend has engendered work on high-level data analysis languages, NoSQL data stores, workflow engines etc., there has been very little attention to the challenges of deploying analytic workflows into production for continuous operation. In this paper, we argue that an essential platform component for enabling continuous production analytic workflows is an analytics store. We highlight five key requirements that impact the design of such a store: (i) efficient incremental operations, (ii) flexible storage model for hierarchical data, (iii) snapshot support (iv) object-level incremental updates, and (v) support for handling change sets. We describe the design of SASH, a scalable analytics store that we have developed on top of HBase to address these requirements. Using the workload from a production workflow that powers search within IBM's intranet and extranet, we demonstrate orders of magnitude improvement in IO performance using SASH.
ieeexplore.ieee.org
Showing the best result for this search. See all results