research-article

Exactly Computing the Tail of the Poisson-Binomial Distribution

Authors:

Andrew Ray Lee,

Uri KeichAuthors Info & Claims

ACM Transactions on Mathematical Software (TOMS), Volume 47, Issue 4

Article No.: 35, Pages 1 - 19

https://doi.org/10.1145/3460774

Published: 28 September 2021 Publication History

Abstract

We present ShiftConvolvePoibin, a fast exact method to compute the tail of a Poisson-binomial distribution (PBD). Our method employs an exponential shift to retain its accuracy when computing a tail probability, and in practice we find that it is immune to the significant relative errors that other methods, exact or approximate, can suffer from when computing very small tail probabilities of the PBD. The accompanying R package is also competitive with the fastest implementations for computing the entire PBD.

References

[1]

William Biscarri, Sihai Dave Zhao, and Robert J. Brunner. 2018. A simple and fast method for computing the Poisson binomial distribution function. Computational Statistics & Data Analysis 122 (2018), 92–100.

[2]

Nicolas Brisebarre, Mioara Joldes, Jean-Michel Muller, Ana-Maria Naneş, and Joris Picot. 2019. Error analysis of some operations involved in the Cooley-Tukey Fast Fourier Transform. ACM Transactions on Mathematical Software 46, 2 (2019), 1–34.

Digital Library

[3]

J. W. Cooley and J. W. Tukey. 1965. An algorithm for the machine calculation of complex fourier series. Mathematics of Computation 19, 90 (1965), 297–301.

[4]

Matteo Frigo and Steven G. Johnson. 2005. The design and implementation of FFTW3. Proceedings of the IEEE 93, 2 (2005), 216–231.

[5]

Yili Hong. 2013. On computing the distribution function for the Poisson binomial distribution. Computational Statistics & Data Analysis 59 (2013), 41–51.

Digital Library

[6]

Uri Keich. 2005. sFFT: A faster accurate computation of the -value of the entropy score.Journal of Computational Biology 12, 4 (May 2005), 416–30.

[7]

U. Keich and N. Nagarajan. 2006. A fast and numerically robust method for exact multinomial goodness-of-fit test. Journal of Computational and Graphical Statistics 15, 4 (2006), 779–802.

[8]

Tobias Madsen, Asger Hobolth, Jens Ledet Jensen, and Jakob Skou Pedersen. 2017. Significance evaluation in factor graphs.BMC Bioinformatics 18, 1 (March 2017), 199.

[9]

Alexander Mukhin. 2019. minFFT: A Minimalist Fast Fourier Transform Library. Retrieved from https://github.com/aimukhin/minfft?files=1

[10]

Niranjan Nagarajan and Uri Keich. 2009. Reliability and efficiency of algorithms for computing the significance of the Mann–Whitney test. Computational Statistics 24, 4 (2009), 605.

Digital Library

[11]

J. C. Schatzman. 1996. Accuracy of the discrete fourier transform and the fast fourier transform. SIAM Journal on Scientific Computing 17, 5 (1996), 1150–1166.

Digital Library

[12]

T. G. Stockham, Jr.1966. High-speed convolution and correlation. In Proceedings of the 1966 Spring Joint Computer Conference. ACM, New York, NY, 229–233.

Digital Library

[13]

Huon Wilson and Uri Keich. 2016. Accurate pairwise convolutions of non-negative vectors via FFT. Computational Statistics & Data Analysis 101 (2016), 300–315.

Digital Library

[14]

Huon Wilson and Uri Keich. 2017. Accurate small tail probabilities of sums of iid lattice-valued random variables via FFT. Journal of Computational and Graphical Statistics 26, 1 (2017), 223–229.

[15]

Man Zhang, Yili Hong, and Narayanaswamy Balakrishnan. 2018. The generalized Poisson-binomial distribution and the computation of its distribution function. Journal of Statistical Computation and Simulation 88, 8 (2018), 1515–1527.

Cited By

Zhang GChen KTang JSong HWen HMumtaz S(2024)Distributed Decision Fusion in Wireless Sensor Networks for Contextual Experience Sensing and Recommendation Forming in E-CommerceIEEE Transactions on Consumer Electronics10.1109/TCE.2024.341162570:4(6817-6829)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1109/TCE.2024.3411625
Yemini MGil SGoldsmith A(2023)Cloud-Cluster Architecture for Detection in Intermittently Connected Sensor NetworksIEEE Transactions on Wireless Communications10.1109/TWC.2022.319941522:2(903-919)Online publication date: Feb-2023
https://doi.org/10.1109/TWC.2022.3199415
Ferrer-Bonsoms JJareno LRubio A(2021)Rediscover: an R package to identify mutually exclusive mutationsBioinformatics10.1093/bioinformatics/btab70938:3(844-845)Online publication date: 19-Oct-2021
https://doi.org/10.1093/bioinformatics/btab709

Index Terms

Exactly Computing the Tail of the Poisson-Binomial Distribution
1. Mathematics of computing
  1. Mathematical software
    1. Mathematical software performance
    2. Statistical software

Recommendations

Learning poisson binomial distributions
STOC '12: Proceedings of the forty-fourth annual ACM symposium on Theory of computing

We consider a basic problem in unsupervised learning: learning an unknown Poisson Binomial Distribution. A Poisson Binomial Distribution (PBD) over {0,1,...,n} is the distribution of a sum of n independent Bernoulli random variables which may have ...
On computing the distribution function for the Poisson binomial distribution

The Poisson binomial distribution is the distribution of the sum of independent and non-identically distributed random indicators. Each indicator follows a Bernoulli distribution and the individual probabilities of success vary. When all success ...
A generalized Conway-Maxwell-Poisson distribution which includes the negative binomial distribution

The Conway-Maxwell-Poisson (COM-Poisson) distribution with two parameters was originally developed as a solution to handling queueing systems with state-dependent arrival or service rates. This distribution generalizes the Poisson distribution by adding ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Mathematical Software

ACM Transactions on Mathematical Software Volume 47, Issue 4

December 2021

242 pages

ISSN:0098-3500

EISSN:1557-7295

DOI:10.1145/3485138

Editors:
Zhaojun Bai
University of California at Davis, USA
,
Wolfgang Bangerth
Colorado State University, USA

Issue’s Table of Contents

Copyright © 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 September 2021

Accepted: 01 April 2021

Revised: 01 December 2020

Received: 01 June 2020

Published in TOMS Volume 47, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
207
Total Downloads

Downloads (Last 12 months)22
Downloads (Last 6 weeks)1

Reflects downloads up to 06 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang GChen KTang JSong HWen HMumtaz S(2024)Distributed Decision Fusion in Wireless Sensor Networks for Contextual Experience Sensing and Recommendation Forming in E-CommerceIEEE Transactions on Consumer Electronics10.1109/TCE.2024.341162570:4(6817-6829)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1109/TCE.2024.3411625
Yemini MGil SGoldsmith A(2023)Cloud-Cluster Architecture for Detection in Intermittently Connected Sensor NetworksIEEE Transactions on Wireless Communications10.1109/TWC.2022.319941522:2(903-919)Online publication date: Feb-2023
https://doi.org/10.1109/TWC.2022.3199415
Ferrer-Bonsoms JJareno LRubio A(2021)Rediscover: an R package to identify mutually exclusive mutationsBioinformatics10.1093/bioinformatics/btab70938:3(844-845)Online publication date: 19-Oct-2021
https://doi.org/10.1093/bioinformatics/btab709

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Issue’s Table of Contents