research-article

The Continued Prevalence of Dichotomous Inferences at CHI

Authors:

Lonni Besançon,

Pierre DragicevicAuthors Info & Claims

CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

Paper No.: alt14, Pages 1 - 11

https://doi.org/10.1145/3290607.3310432

Published: 02 May 2019 Publication History

Abstract

Dichotomous inference is the classification of statistical evidence as either sufficient or insufficient. It is most commonly done through null hypothesis significance testing (NHST). Although predominant, dichotomous inferences have proven to cause countless problems. Thus, an increasing number of methodologists have been urging researchers to recognize the continuous nature of statistical evidence and to ban dichotomous inferences. We wanted to see whether they have had any influence on CHI. Our analysis of CHI proceedings from the past nine years suggests that they have not.

Supplementary Material

MP4 File (alt14.mp4)

Download
229.15 MB

References

[1]

Valentin Amrhein, Fränzi Korner-Nievergelt, and Tobias Roth. 2017. The earth is flat (p>0.05): Significance thresholds and the crisis of unreplicable research. PeerJ Preprints 5 (June 2017), e2921v2.

[2]

Valentin Amrhein, David Trafimow, and Sander Greenland. 2018. Inferential Statistics as Descriptive Statistics: There is No Replication Crisis if We Don't Expect Replication. The American Statistician (2018).

[3]

Thomas Baguley. 2012. Serious stats: A guide to advanced statistics for the behavioral sciences. Red Globe Press.

[4]

Louise Barkhuus and Jennifer A. Rode. 2007. From Mice to Men - 24 Years of Evaluation in CHI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). ACM, New York, NY, USA, Article 1.

Digital Library

[5]

Daniel J Benjamin, James O Berger, Magnus Johannesson, Brian A Nosek, E-J Wagenmakers, and others. 2018. Redefine statistical significance. Nature Human Behaviour 2, 1 (2018), 6.

[6]

Kelly Caine. 2016. Local Standards for Sample Size at CHI. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 981--992.

Digital Library

[7]

Melissa Coulson, Michelle Healey, Fiona Fidler, and Geoff Cumming. 2010. Confidence intervals permit, but don't guarantee, better inference than statistical significance testing. Frontiers in psychology 1 (2010), 26.

[8]

Geoff Cumming. 2012. Understanding the new statistics: effect sizes, confidence intervals and meta-analysis. Routledge Taylor & Francis Group.

[9]

Geoff Cumming. 2014. The new statistics: Why and how. Psychological Science 25, 1 (Jan. 2014), 7--29.

[10]

Peter Dixon. 2003. The p-value fallacy and how to avoid it. Canadian Journal of Experimental Psychology 57, 3 (2003), 189.

[11]

Pierre Dragicevic. 2016. Fair Statistical Communication in HCI. In Modern Statistical Methods for HCI, Judy Robertson and Maurits Kaptein (Eds.). Springer International Publishing, Cham, Switzerland, Chapter 13, 291--330.

[12]

Andrew Gelman. 2017. No to inferential thresholds. Online. Last visited 04 January 2019. (2017).

[13]

Andrew Gelman and Hal Stern. 2006. The difference between "significant" and "not significant" is not itself statistically significant. The American Statistician 60, 4 (2006), 328--331.

[14]

Gerd Gigerenzer. 2004. Mindless statistics. The Journal of Socio-Economics 33, 5 (2004), 587--606.

[15]

Gerd Gigerenzer. 2018. Statistical Rituals: The Replication Delusion and How We Got There. Advances in Methods and Practices in Psychological Science (2018), 2515245918771329.

[16]

Frank Harrell. 2018. Language for communicating frequentist results about treatment effects. Online. Last visited 04 January 2019. (2018).

[17]

Stuart H Hurlbert and Celia M Lombardi. 2009. Final collapse of the Neyman-Pearson decision theoretic framework and rise of the neoFisherian. In Annales Zoologici Fennici, Vol. 46. BioOne, 311--349.

[18]

Matthew Kay, Gregory L Nelson, and Eric B Hekler. 2016. Researcher-centered design of statistics: Why Bayesian statistics better fit the culture and incentives of HCI. In Proceedings of the CHI 2016. ACM, 4521--4532.

Digital Library

[19]

Lisa Koeman. 2018. How many participants do researchers recruit? A look at 678 UX/HCI studies. Online. Last visited 06 January 2019. (2018).

[20]

John K Kruschke and Torrin M Liddell. 2018. The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin & Review 25, 1 (2018), 178--206.

[21]

Daniel Lakens. 2016. Dance of the Bayes factors. Online. Last visited 03 January 2019. (2016).

[22]

Joe Marshall, Conor Linehan, Jocelyn Spence, and Stefan Rennick Egglestone. 2017. Throwaway citation of prior work creates risk of bad HCI research. In Proc. CHI Extended Abstracts. ACM, 827--836.

Digital Library

[23]

Anna Mauranen. 1997. Hedging in Language Reviser's Hands. Vol. 24. Walter de Gruyter. 115 pages.

[24]

Alison McCook. 2016. We're using a common statistical test all wrong. Statisticians want to fix that. Online. Last visited 03 January 2019. (2016).

[25]

Blakeley B. McShane and David Gal. 2017. Statistical Significance and the Dichotomization of Evidence. J. Amer. Statist. Assoc. 112, 519 (2017), 885--895.

[26]

Luana Micallef and Peter Rodgers. 2014. eulerAPE: drawing area-proportional 3-Venn diagrams using ellipses. PloS one 9, 7 (2014), e101717.

[27]

Greg Myers. 1989. The pragmatics of politeness in scientific articles. Applied Linguistics 10, 1 (1989), 1--35.

[28]

Robert Rosenthal and John Gaito. 1963. The Interpretation of Levels of Significance by Psychological Researchers. The Journal of Psychology 55, 1 (1963), 33--38.

[29]

Jeffrey C. Valentine, Ariel M. Aloe, and Timothy S. Lau. 2015. Life After NHST: How to Describe Your Data Without "p-ing" Everywhere. Basic and Applied Social Psychology 37, 5 (2015), 260--273.

[30]

Kees van Deemter. 2010. Not Exactly: in Praise of Vagueness. Oxford University Press.

Digital Library

[31]

Gary R. VandenBos (Ed.). 2009. Publication Manual of the American Psychological Association (6th ed.). American Psychological Association, Washington, DC.

[32]

Ignacio Vázquez Orta and Diana Giner. 2008. Beyond mood and modality: epistemic modality markers as hedges in research articles. A cross-disciplinary study. In Revistas - Revista Alicantina de Estudios Ingleses. Vol. 21. Universidad de Alicante. Departamento de Filología Inglesa.

Cited By

Rey BSakamoto YSin JIrani P(2024)Understanding User Preferences of Voice Assistant Answer Structures for Personal Health Data QueriesProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665552(1-15)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665552
Vega GMartinez-Missir VWittchen DSabnis NGirouard ACochrane KStrohmeier P(2024)vARitouch: Back of the Finger Device for Adding Variable Compliance to Rigid ObjectsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642828(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642828
Ding YSabnis NStrohmeier P(2024)Motionless Movement: Towards Vibrotactile Kinesthetic DisplaysProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642499(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642499
Show More Cited By

Index Terms

The Continued Prevalence of Dichotomous Inferences at CHI
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

The Probability that Your Hypothesis Is Correct, Credible Intervals, and Effect Sizes for IR Evaluation
SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

Using classical statistical significance tests, researchers can only discuss P(D⁺|H), the probability of observing the data D at hand or something more extreme, under the assumption that the hypothesis H is true (i.e., the p-value). But what we usually ...
Statslator: Interactive Translation of NHST and Estimation Statistics Reporting Styles in Scientific Documents
UIST '23: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

Inferential statistics are typically reported using p-values (NHST) or confidence intervals on effect sizes (estimation). This is done using a range of styles, but some readers have preferences about how statistics should be presented and others have ...
Conducting Laboratory Experiments Properly with Statistical Tools: An Easy Hands-On Tutorial
WSDM '19: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining

This hands-on half-day tutorial consists of two sessions. Part~I covers the following topics: Preliminaries; Paired and two-sample t-tests, confidence intervals; One-way ANOVA and two-way ANOVA without replication; Familiwise error rate. Part~II covers ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

May 2019

3673 pages

ISBN:9781450359719

DOI:10.1145/3290607

General Chairs:
Stephen Brewster
University of Glasgow, Scotland, UK
,
Geraldine Fitzpatrick
TU Wien, Austria
,
Program Chairs:
Anna Cox
University College London, UK
,
Vassilis Kostakos
University of Melbourne, Australia

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 May 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '19

Sponsor:

SIGCHI

CHI '19: CHI Conference on Human Factors in Computing Systems

May 4 - 9, 2019

Glasgow, Scotland Uk

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
400
Total Downloads

Downloads (Last 12 months)46
Downloads (Last 6 weeks)7

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Rey BSakamoto YSin JIrani P(2024)Understanding User Preferences of Voice Assistant Answer Structures for Personal Health Data QueriesProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665552(1-15)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665552
Vega GMartinez-Missir VWittchen DSabnis NGirouard ACochrane KStrohmeier P(2024)vARitouch: Back of the Finger Device for Adding Variable Compliance to Rigid ObjectsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642828(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642828
Ding YSabnis NStrohmeier P(2024)Motionless Movement: Towards Vibrotactile Kinesthetic DisplaysProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642499(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642499
Oral BDragicevic PTelea ADimara E(2024)Decoupling Judgment and Decision Making: A Tale of Two TailsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.334664030:10(6928-6940)Online publication date: Oct-2024
https://doi.org/10.1109/TVCG.2023.3346640
He TZhong YIsenberg PIsenberg T(2024)Design Characterization for Black-and-White Textures in VisualizationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332694130:1(1019-1029)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3326941
Zielasko DWeissker T(2023)Stay Vigilant: The Threat of a Replication Crisis in VR Locomotion ResearchProceedings of the 29th ACM Symposium on Virtual Reality Software and Technology10.1145/3611659.3615697(1-10)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3611659.3615697
Masson DMalacria SCasiez GVogel D(2023)Statslator: Interactive Translation of NHST and Estimation Statistics Reporting Styles in Scientific DocumentsProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606762(1-14)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606762
Fruchard BMalacria SCasiez GHuot S(2023)User Preference and Performance using Tagging and Browsing for Image LabelingProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580926(1-13)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580926
Blascheck TBesançon LBezerianos ALee BIslam AHe TIsenberg P(2023)Studies of Part-to-Whole Glanceable Visualizations on Smartwatch Faces2023 IEEE 16th Pacific Visualization Symposium (PacificVis)10.1109/PacificVis56936.2023.00028(187-196)Online publication date: Apr-2023
https://doi.org/10.1109/PacificVis56936.2023.00028
Huth FBlascheck TKoch SErtl T(2023)Studies and design considerations for animated transitions between small-scale visualizationsJournal of Visualization10.1007/s12650-023-00937-z26:6(1421-1443)Online publication date: 28-Aug-2023
https://doi.org/10.1007/s12650-023-00937-z
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents