research-article

Data mining using high performance data clouds: experimental studies using sector and sphere

Authors:

Robert Grossman,

Yunhong GuAuthors Info & Claims

KDD '08: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining

Pages 920 - 927

https://doi.org/10.1145/1401890.1402000

Published: 24 August 2008 Publication History

Abstract

We describe the design and implementation of a high performance cloud that we have used to archive, analyze and mine large distributed data sets. By a cloud, we mean an infrastructure that provides resources and/or services over the Internet. A storage cloud provides storage services, while a compute cloud provides compute services. We describe the design of the Sector storage cloud and how it provides the storage services required by the Sphere compute cloud. We also describe the programming paradigm supported by the Sphere compute cloud. Sector and Sphere are designed for analyzing large data sets using computer clusters connected with wide area high performance networks (for example, 10+ Gb/s). We describe a distributed data mining application that we have developed using Sector and Sphere. Finally, we describe some experimental studies comparing Sector/Sphere to Hadoop.

References

[1]

Amazon. Amazon Simple Storage Service (Amazon S3). www.amazon.com/s3.

[2]

Jay Beale, Andrew R Baker, and Joel Esler. Snort IDS and IPS Toolkit. Syngress, 2007.

[3]

Dhruba Borthaku. The hadoop distributed file system: Architecture and design. retrieved from lucene.apache.org/hadoop, 2007.

[4]

Leo Breiman, Jerome H. Friedman, Richard A. Olshen, and Charles J. Stone. Classification and Regression Trees. Chapman and Hall, New York, 1984.

[5]

Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI'04: Sixth Symposium on Operating System Design and Implementation, 2004.

Digital Library

[6]

National Center for Data Mining at the University of Illinois at Chicago. The large data archives project.

[7]

Ian Foster and Carl Kesselman. The Grid 2: Blueprint for a New Computing Infrastructure. Morgan Kaufmann, San Francisco, California, 2004.

Digital Library

[8]

Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. The Google File System. In SOSP, 2003.

Digital Library

[9]

Jim Gray and Alexander S. Szalay. The world-wide telescope. Science, 293:2037--2040, 2001.

[10]

William Gropp, Ewing Lusk, and Anthony Skjellum. Using MPI: Portable Parallel Programming with the Message Passing Interface, 2nd Edition. MIT Press, 1999.

Digital Library

[11]

Robert L. Grossman and Yunhong Gu. Sc 2006 bandwidth challenge: National center for data mining - udt. retrieved from https://scinet.supercomp.org/2006/bwc/graphs/challengencdm.png, 2006.

[12]

Robert L Grossman, Michael Sabala, Yunhong Gu, Anushka Anand, Matt Handley, Rajmonda Sulo, and Lee Wilkinson. Distributed discovery in e-science: Lessons from the angle project. In Next Generation Data Mining (NGDM '07), 2008.

[13]

Yunhong Gu and Robert L. Grossman. UDT: UDP-based data transfer for high-speed wide area networks. Computer Networks, 51(7):1777--1799, 2007.

Digital Library

[14]

Yunhong Gu, Robert L. Grossman, Alex Szalay, and Ani Thakar. Distributing the sloan digital sky survey using udt and sector. In Proceedings of e-Science 2006, 2006.

Digital Library

[15]

Hillol Kargupta. Proceedings of Next Generation Data Mining 2007. Taylor and Francis, 2008.

Digital Library

[16]

Amazon Web Services LLC. Amazon web services developer connection. retrieved from developer.amazonwebservices.com on November 1, 2007.

[17]

John D. Owens, David Luebke, Naga Govindaraju, Mark Harris, Jens Kruger, Aaron E. Lefohn, and Timothy J.Purcell. A survey of general-purpose computation on graphics hardware. In Eurographics 2005, pages 21--51, 2005.

[18]

The Sector Project. Sector, a distributed storage and computing infrastructure, version 1.4.

[19]

I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H Balakrishnana. Chord: A scalable peer to peer lookup service for internet applications. In Proceedings of the ACM SIGCOMM '01, pages 149--160, 2001.

Digital Library

[20]

Hbase Development Team. Hbase: Bigtable-like structured storage for hadoop hdfs. http://wiki.apache.org/lucene-hadoop/Hbase, 2007.

Cited By

Santosh Kumar Jha (2023)Technological Prospects of Cloud Computing in Web Mining: Recent Trends and Opportunitiesinternational journal of engineering technology and management sciences10.46647/ijetms.2023.v07i01.0177:1(98-104)Online publication date: 28-Feb-2023
https://doi.org/10.46647/ijetms.2023.v07i01.017
Liu FShen GZhang M(2022)Privacy-Preserving Multi-party Neural Network Learning Over Incomplete Data2022 IEEE 24th Int Conf on High Performance Computing & Communications; 8th Int Conf on Data Science & Systems; 20th Int Conf on Smart City; 8th Int Conf on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys)10.1109/HPCC-DSS-SmartCity-DependSys57074.2022.00191(1216-1221)Online publication date: Dec-2022
https://doi.org/10.1109/HPCC-DSS-SmartCity-DependSys57074.2022.00191
Alangaram SBalakannan S(2022)A Taxonomy on Strategic Viewpoint and Insight Towards Multi-Cloud EnvironmentsComputational Vision and Bio-Inspired Computing10.1007/978-981-16-9573-5_51(713-719)Online publication date: 31-Mar-2022
https://doi.org/10.1007/978-981-16-9573-5_51
Show More Cited By

Index Terms

Data mining using high performance data clouds: experimental studies using sector and sphere
1. Information systems
  1. Information systems applications
    1. Data mining
2. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        Memory management
        Distributed memory
        Process management
        Multiprocessing / multiprogramming / multitasking
    2. Software system structures
      1. Distributed systems organizing principles
        Organizing principles for web applications

Recommendations

Monitoring-based auto-scalability across hybrid clouds
SAC '18: Proceedings of the 33rd Annual ACM Symposium on Applied Computing

Cloud computing is a relatively new type of Internet-based computing that becomes more and more popular. Using methods like virtualization, adopting architectures based on microservices, automation of building and deployment processes, Cloud could ...
A Trust Management Solution in the Context of Hybrid Clouds
WETICE '14: Proceedings of the 2014 IEEE 23rd International WETICE Conference

Cloud computing is a revolutionary paradigm which enables on-demand provisioning of computing resources. Resources are delivered to cloud consumers in the form of infrastructure, platform and software services. These resources are deployed on three ...
Empirical Performance Assessment of Public Clouds Using System Level Benchmarks

Amazon's Elastic Compute Cloud EC2 Service is one of the leading public cloud service providers and offers many different levels of service. This paper looks into evaluating the memory, central processing unit CPU, and input/output I/O performance of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '08: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining

August 2008

1116 pages

ISBN:9781605581934

DOI:10.1145/1401890

General Chair:
Ying Li
Microsoft adCenter Labs
,
Program Chairs:
Bing Liu
University of Illinois at Chicago
,
Sunita Sarawagi
Indian Institute of Technology, Bombay

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 August 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD08

Sponsor:

KDD08: The 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 24 - 27, 2008

Nevada, Las Vegas, USA

Acceptance Rates

KDD '08 Paper Acceptance Rate 118 of 593 submissions, 20%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

106
Total Citations
View Citations
4,144
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)2

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Santosh Kumar Jha (2023)Technological Prospects of Cloud Computing in Web Mining: Recent Trends and Opportunitiesinternational journal of engineering technology and management sciences10.46647/ijetms.2023.v07i01.0177:1(98-104)Online publication date: 28-Feb-2023
https://doi.org/10.46647/ijetms.2023.v07i01.017
Liu FShen GZhang M(2022)Privacy-Preserving Multi-party Neural Network Learning Over Incomplete Data2022 IEEE 24th Int Conf on High Performance Computing & Communications; 8th Int Conf on Data Science & Systems; 20th Int Conf on Smart City; 8th Int Conf on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys)10.1109/HPCC-DSS-SmartCity-DependSys57074.2022.00191(1216-1221)Online publication date: Dec-2022
https://doi.org/10.1109/HPCC-DSS-SmartCity-DependSys57074.2022.00191
Alangaram SBalakannan S(2022)A Taxonomy on Strategic Viewpoint and Insight Towards Multi-Cloud EnvironmentsComputational Vision and Bio-Inspired Computing10.1007/978-981-16-9573-5_51(713-719)Online publication date: 31-Mar-2022
https://doi.org/10.1007/978-981-16-9573-5_51
Guo DGuo D(2022)Introduction of Data CenterData Center Networking10.1007/978-981-16-9368-7_1(3-24)Online publication date: 24-Feb-2022
https://doi.org/10.1007/978-981-16-9368-7_1
Pranav DPunj DDubey TChawla P(2021)Data mining in Cloud Computing2021 5th International Conference on Computing Methodologies and Communication (ICCMC)10.1109/ICCMC51019.2021.9418489(71-78)Online publication date: 8-Apr-2021
https://doi.org/10.1109/ICCMC51019.2021.9418489
Liu HZhang HGuo LYu JLin J(2021)Privacy-Preserving Cloud-Aided Broad Learning SystemComputers & Security10.1016/j.cose.2021.102503(102503)Online publication date: Oct-2021
https://doi.org/10.1016/j.cose.2021.102503
Kavitha K(2021)Assessing Teacher’s Performance Evaluation and Prediction Model Using Cloud Computing Over Multi-dimensional DatasetWireless Personal Communications10.1007/s11277-021-08394-3Online publication date: 8-Apr-2021
https://doi.org/10.1007/s11277-021-08394-3
Sood ISharma V(2020)A comparative study of Distributed Large Scale Data Mining AlgorithmsBSSS Journal of Computer10.51767/jc1102Online publication date: 25-May-2020
https://doi.org/10.51767/jc1102
Barua HMondal K(2019)A Comprehensive Survey on Cloud Data Mining (CDM) Frameworks and AlgorithmsACM Computing Surveys10.1145/334926552:5(1-62)Online publication date: 13-Sep-2019
https://dl.acm.org/doi/10.1145/3349265
Lee SJo JKim Y(2019)Survey of Data Locality in Apache Hadoop2019 IEEE International Conference on Big Data, Cloud Computing, Data Science & Engineering (BCD)10.1109/BCD.2019.8885148(46-53)Online publication date: May-2019
https://doi.org/10.1109/BCD.2019.8885148
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents