Article

The "DGX" distribution for mining massive, skewed data

Authors:

Christos Faloutsos,

Flip KornAuthors Info & Claims

KDD '01: Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining

Pages 17 - 26

https://doi.org/10.1145/502512.502521

Published: 26 August 2001 Publication History

Abstract

Skewed distributions appear very often in practice. Unfortunately, the traditional Zipf distribution often fails to model them well. In this paper, we propose a new probability distribution, the Discrete Gaussian Exponential (DGX), to achieve excellent fits in a wide variety of settings; our new distribution includes the Zipf distribution as a special case. We present a statistically sound method for estimating the DGX parameters based on maximum likelihood estimation (MLE). We applied DGX to a wide variety of real world data sets, such as sales data from a large retailer chain, us-age data from AT&T, and Internet clickstream data; in all cases, DGX fits these distributions very well, with almost a 99% correlation coefficient in quantile-quantile plots. Our algorithm also scales very well because it requires only a single pass over the data. Finally, we illustrate the power of DGX as a new tool for data mining tasks, such as outlier detection.

References

[1]

L. Adamic. Zipf, power-laws, and pareto - a ranking tutorial, http://www.parc.xerox-com/istl/groups/iea /papers/ranking/ranking_html.

[2]

B. Berry and W. Garrison, Alternate explanations of urban rank size relations" Annals of the Association of American Geographers, 48:83-91, 1958.

[3]

L. Breslau, P. Cao, L. Fan, G. Phillips, and S. Shenker. Web caching and zipf-like distributions: Evidence and implications. In IEEE Infocom '9g, New York, NY, Mar. 1999.

[4]

A. B. C. Cunha and M. CroveUa. Characteristics of www client-based traces. Tr-95-010, Boston University, April 1995. http://www.cs.bu.edu/groups /oceans/papers/Home.ht ml.

Digital Library

[5]

C. Faloutsos and H. Jagadish. On B-tree indices for skewed distributions. In 18th VLDB Conference, pages 363-374, Vancouver, British Columbia, Aug. 23-27 1992.

Digital Library

[6]

W. Fox and W. Lasker. The distribution of surname frequencies. International Statistical Review, 51:81-87, 1983.

[7]

Galton. The geometric mean in vital and social statistics. Proceedings of the Royal Society of London, 29:365-367, 1879.

[8]

S, Glassman. A caching relay for the world wide web. In Proc, of First International Confer, nee on the World Wide Web, CERN, Geneva, Switzerland, May 1994. http://wwwl.cern,ch/WWW94/P reUmProcs .html.

Digital Library

[9]

P. Halmos. Random Mms. Annals of Mathematical Statistics, 15:182-189, 1944.

[10]

G. Herdan. Small Particle Statistics. Butterworth's, London, 2 edition, 1060.

[11]

B. Hill. The rank-frequency form of zipf's law. Journal of the American Statistical Association, 69(348):1017--1026, 1974.

[12]

The MathWorks Inc. Matlab user's guide.

[13]

J. Laherr6re. "parabolic fraetal" distributions in nature, ht tp://www.hubber t peak.com/laherrere /fractal.htm.

[14]

B. Laherrere and D. Sornette. Stretched exponential distributions in nature and economy: "fat tails" with characteristic scales. European Physical Journal, B(2):525-539, 1998.

[15]

S. K. N.I. Johnson and N. Baiakrishnan. Continuous Univaviate Distribution, Volume 1. John Wiley & Sons, Inc., U.S.A, 1994.

[16]

V. Pareto. Cours d'Economie Politique. Rouge and Cie, Lausanne and Paris, 1897.

[17]

H. Simon. On a class of skew distribution functions. Biometrika, 42:425-440, 1955.

[18]

M. C. V. Almeida, A. Bestavros and A. de Oliveira. Characterizing reference locality in the www. In Proc. of IBBE International Conference in Parallel and Distributed Information Systems, Miami Beach, Florida, U.S.A., December 1996. http: //www.es.bu.edu /groups /oeeans /papers /Home,html.

Digital Library

[19]

G. Yule. A mathematical theory of evolution, based on conclusions of dr. i.e. willis, Lr.s. PhilosophieaI dimnsaetions of the Royal Society of London, 213:21-87, 1923.

[20]

G. Zipf. Human Behavior and Principle of Least Effort: An Introduction to Human Ecology. Addison Wesley, Cambridge, Massachusetts, 1949.

Cited By

Griva AZampou EStavrou VPapakiriakopoulos DDoukidis G(2022)A two-stage business analytics approach to perform behavioural and geographic customer segmentation using e-commerce delivery dataJournal of Decision Systems10.1080/12460125.2022.215107133:1(1-29)Online publication date: 8-Dec-2022
https://doi.org/10.1080/12460125.2022.2151071
Griva ABardaki CPramatari KDoukidis G(2021)Factors Affecting Customer Analytics: Evidence from Three Retail CasesInformation Systems Frontiers10.1007/s10796-020-10098-124:2(493-516)Online publication date: 4-Jan-2021
https://doi.org/10.1007/s10796-020-10098-1
Duarte-López APérez-Casany MValero J(2021)Randomly stopped extreme Zipf extensionsExtremes10.1007/s10687-021-00410-wOnline publication date: 23-Mar-2021
https://doi.org/10.1007/s10687-021-00410-w
Show More Cited By

Index Terms

The "DGX" distribution for mining massive, skewed data

Recommendations

Towards finding the best-fit distribution for OSN data
Abstract
Currently, all online social networks (OSNs) are considered to follow a power-law distribution. In this paper, the degree distribution for multiple OSNs has been studied. It is seen that the degree distributions of OSNs differ moderately from a ...
A skewed truncated t distribution

Skewed symmetric distributions have attracted a great deal of attention in the last few years. One of them, the skewed t distribution suffers from limited applicability because of the lack of finite moments. This note proposes an alternative to the ...
Estimation of parameters for the MarshallOlkin generalized exponential distribution based on complete data

This paper derives some properties of the MarshallOlkin generalized exponential distribution and shows that this distribution is more flexible than the exponentiated exponential distribution. Then we discuss estimation of the distribution parameters by ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '01: Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining

August 2001

493 pages

ISBN:158113391X

DOI:10.1145/502512

Conference Chair:
Doheon Lee
Chonnam National University, Korea
,
General Chair:
Mario Schkolnick
SGI
,
Program Chairs:
Foster Provost
New York University
,
Ramakrishnan Srikant
IBM Almaden Research Center

Copyright © 2001 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data
SIGKDD: ACM Special Interest Group on Knowledge Discovery in Data
AAAI: American Association for Artificial Intelligence

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 August 2001

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

KDD01

Sponsor:

KDD01: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 26 - 29, 2001

California, San Francisco

Acceptance Rates

KDD '01 Paper Acceptance Rate 31 of 237 submissions, 13%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

80
Total Citations
View Citations
731
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)1

Reflects downloads up to 29 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Griva AZampou EStavrou VPapakiriakopoulos DDoukidis G(2022)A two-stage business analytics approach to perform behavioural and geographic customer segmentation using e-commerce delivery dataJournal of Decision Systems10.1080/12460125.2022.215107133:1(1-29)Online publication date: 8-Dec-2022
https://doi.org/10.1080/12460125.2022.2151071
Griva ABardaki CPramatari KDoukidis G(2021)Factors Affecting Customer Analytics: Evidence from Three Retail CasesInformation Systems Frontiers10.1007/s10796-020-10098-124:2(493-516)Online publication date: 4-Jan-2021
https://doi.org/10.1007/s10796-020-10098-1
Duarte-López APérez-Casany MValero J(2021)Randomly stopped extreme Zipf extensionsExtremes10.1007/s10687-021-00410-wOnline publication date: 23-Mar-2021
https://doi.org/10.1007/s10687-021-00410-w
Drakopoulos GGiannoukou IMylonas PSioutas S(2020)A Graph Neural Network For Assessing The Affective Coherence Of Twitter Graphs2020 IEEE International Conference on Big Data (Big Data)10.1109/BigData50022.2020.9378492(3618-3627)Online publication date: 10-Dec-2020
https://doi.org/10.1109/BigData50022.2020.9378492
Meena KTayal DCastillo OJain A(2020)Handling data-skewness in character based string similarity join using HadoopApplied Computing and Informatics10.1016/j.aci.2018.11.00118:1/2(22-44)Online publication date: 4-Aug-2020
https://doi.org/10.1016/j.aci.2018.11.001
de Oliveira Rde Oliveira Peres MMartinez EAchcar J(2019)Use of a discrete Sushila distribution in the analysis of right-censored lifetime dataModel Assisted Statistics and Applications10.3233/MAS-19046514:3(255-268)Online publication date: 18-Jul-2019
https://doi.org/10.3233/MAS-190465
Pandey PSingh M(2019)MSP-N: Multiple selection procedure with ‘N’ possible growth mechanismsPLOS ONE10.1371/journal.pone.022438314:12(e0224383)Online publication date: 12-Dec-2019
https://doi.org/10.1371/journal.pone.0224383
Aksu HKorpeoglu IUlusoy O(2019)An Analysis of Social Networks Based on Tera-Scale Telecommunication DatasetsIEEE Transactions on Emerging Topics in Computing10.1109/TETC.2016.26270347:2(349-360)Online publication date: 1-Apr-2019
https://doi.org/10.1109/TETC.2016.2627034
Tayal DMeena K(2019)A new mapreduce solution for associative classification to handle scalability and skewness in vertical data structureFuture Generation Computer Systems10.1016/j.future.2019.09.040Online publication date: Sep-2019
https://doi.org/10.1016/j.future.2019.09.040
Yu CXia FQian WZhou A(2019)A parallel data generator for efficiently generating "realistic" social streamsFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-018-8022-z13:5(1072-1101)Online publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1007/s11704-018-8022-z
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents