research-article

Hypothesis Formalization: Empirical Findings, Software Limitations, and Design Implications

Authors:

Melissa Birchfield,

Nicole De Moura,

René JustAuthors Info & Claims

ACM Transactions on Computer-Human Interaction (TOCHI), Volume 29, Issue 1

Article No.: 6, Pages 1 - 28

https://doi.org/10.1145/3476980

Published: 07 January 2022 Publication History

Abstract

Data analysis requires translating higher level questions and hypotheses into computable statistical models. We present a mixed-methods study aimed at identifying the steps, considerations, and challenges involved in operationalizing hypotheses into statistical models, a process we refer to as hypothesis formalization. In a formative content analysis of 50 research papers, we find that researchers highlight decomposing a hypothesis into sub-hypotheses, selecting proxy variables, and formulating statistical models based on data collection design as key steps. In a lab study, we find that analysts fixated on implementation and shaped their analyses to fit familiar approaches, even if sub-optimal. In an analysis of software tools, we find that tools provide inconsistent, low-level abstractions that may limit the statistical models analysts use to formalize hypotheses. Based on these observations, we characterize hypothesis formalization as a dual-search process balancing conceptual and statistical considerations constrained by data and computation and discuss implications for future tools.

References

[1]

Sara Alspaugh, Nava Zokaei, Andrea Liu, Cindy Jin, and Marti A. Hearst. 2018. Futzing and moseying: Interviews with professional data analysts on exploration practices. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2018), 22–31.

Digital Library

[2]

Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2014. Fitting linear mixed-effects models using lme4. arXiv:1406.5823 (2014). Retrieved from https://arxiv.org/abs/1701.00133.

[3]

Douglas Bates, Martin Mächler, Ben Bolker, Steve Walker, Rune H. B. Christensen, Henrik Singmann, Bin Dai, Fabian Scheipl, Gabor Grothendieck, Peter Green, and John Fox. 2019. Package ‘lme4‘. CRAN (2019). Retrieved from https://cran.r-project.org/web/packages/lme4/lme4.pdf.

[4]

Leilani Battle and Jeffrey Heer. 2019. Characterizing exploratory visual analysis: A literature review and evaluation of analytic provenance in tableau. Computer Graphics Forum 38, 3 (2019), 145–159. Retrieved from http://idl.cs.washington.edu/papers/exploratory-visual-analysis.

[5]

Jacques Bertin. 2011. Graphics and Graphic Information Processing. Walter de Gruyter.

[6]

Michael Betancourt. 2020. Towards a Principled Bayesian Workflow. Psychological Methods 26, 1 (2020), 103-126. Retrieved from https://betanalpha.github.io/assets/case_studies/principled_bayesian_workflow.html.

[7]

Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman. 2019. Pyro: Deep universal probabilistic programming. The Journal of Machine Learning Research 20, 1 (2019), 973–978.

Digital Library

[8]

Igor Bobriakov. 2017. Top 15 python libr aries for data science in 2017. ActiveWizards in Medium (2017). Retrieved January 7, 2020 from https://medium.com/activewizards-machine-learning-company/top-15-python-libraries-for-data-science-in-in-2017-ab61b4f9b4a7.

[9]

Igor Bobriakov. 2018. Top 20 python libraries for data science in 2018. ActiveWizards in Medium (2018). Retrieved December 16, 2019 from https://medium.com/activewizards-machine-learning-company/top-20-python-libraries-for-data-science-in-2018-2ae7d1db8049.

[10]

Leo Breiman, Adele Cutler, Andy Liaw, and Matthew Wiener. 2018. Package “randomForest”. (2018). Retrieved September 16, 2020 from https://cran.r-project.org/web/packages/randomForest/randomForest.pdf.

[11]

Mollie E. Brooks, Kasper Kristensen, Koen J. van Benthem, Arni Magnusson, Casper W. Berg, Anders Nielsen, Hans J. Skaug, Martin Machler, and Benjamin M. Bolker. 2017. glmmTMB balances speed and flexibility among packages for zero-inflated generalized linear mixed modeling. The R Journal 9, 2 (2017), 378–400.

[12]

Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake Vanderplas, Arnaud Joly, Brian Holt, and Gaël Varoquaux. 2013. API design for machine learning software: Experiences from the scikit-learn project. arXiv:1309.0238. Retrieved from https://arxiv.org/abs/1309.0238.

[13]

Paul-Christian Bürkner. 2017. brms: An R package for Bayesian multilevel models using stan. Journal of Statistical Software 80, 1 (2017), 1–28.

[14]

Bob Carpenter, Andrew Gelman, Matthew D. Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. 2017. Stan : A probabilistic programming language. Journal of Statistical Software 76, 1 (2017), 1–32. DOI:https://doi.org/10.18637/jss.v076.i01

[15]

Robert Carver, Michelle Everson, John Gabrosek, Nicholas Horton, Robin Lock, Megan Mocko, Allan Rossman, Ginger Holmes Roswell, Paul Velleman, Jeffrey Witmer, and Beverly Wood. 2016. Guidelines for assessment and instruction in statistics education (GAISE) college report 2016. AMSTAT (2016).

[16]

Yunshun Chen, Aaron T. L. Lun, Davis J. McCarthy, Matthew E. Ritchie, Belinda Phipson, Yifang Hu, Xiaobei Zhou, Mark D. Robinson, and Gordon K. Smyth. 2020. Empirical analysis of digital gene expression data in R (v3.30.3). (2020). Retrieved September 16, 2020 from https://bioconductor.org/packages/release/bioc/html/edgeR.html.

[17]

Yunshun Chen, David McCarthy, Matthew Ritchie, Mark Robinson, and Gordon Smyth. 2020. edgeR: Differential analysisof sequence read count data. (2020). Retrieved September 16, 2020 from https://bioconductor.org/packages/release/bioc/vignettes/edgeR/inst/doc/edgeRUsersGuide.pdf.

[18]

François Chollet. 2015. Keras. keras.io Retrieved from https://keras.io.

[19]

Alexander Eiselmayer, Chatchavan Wacharamanotham, Michel Beaudouin-Lafon, and Wendy Mackay. 2019. Touchstone2: An interactive environment for exploring trade-offs in HCI experiment design. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. (2019).

Digital Library

[20]

Janet Feigenspan, Christian Kästner, Jörg Liebig, Sven Apel, and Stefan Hanenberg. 2012. Measuring programming experience. In Proceedings of the 2012 20th IEEE International Conference on Program Comprehension. IEEE, 73–82.

[21]

Jerome Friedman, Trevor Hastie, Rob Tibshirani, Balasubramanian Narasimhan, Kenneth Tay, Noah Simon, and Junyang Qian. 2020. Package “glmnet”. (2020). Retrieved September 16, 2020 from https://cran.r-project.org/web/packages/glmnet/index.html.

[22]

Jonah Gabry, Daniel Simpson, Aki Vehtari, Michael Betancourt, and Andrew Gelman. 2019. Visualization in bayesian workflow. Journal of the Royal Statistical Society: Series A (Statistics in Society) 182, 2 (2019), 389–402.

[23]

Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. 2013. Bayesian Data Analysis. CRC press.

[24]

Andrew Gelman and Eric Loken. 2013. The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. Department of Statistics, Columbia University (2013).

[25]

Andrew Gelman, Aki Vehtari, Daniel Simpson, Charles C. Margossian, Bob Carpenter, Yuling Yao, Lauren Kennedy, Jonah Gabry, Paul-Christian Bürkner, and Martin Modrák. 2020. Bayesian workflow. arXiv:2011.01808. Retrieved from https://arxiv.org/abs/2011.01808.

[26]

LLC. GraphPad Software. 2020. GraphPad prism 8 user guide. (2020). Retrieved fromhttps://www.graphpad.com/guides/prism/8/user-guide/index.htm.

[27]

Garrett Grolemund. 2019. Quick list of useful R packages. R Studio Support (2019). Retrieved April 28, 2020 fromhttps://support.rstudio.com/hc/en-us/articles/201057987-Quick-list-of-useful-R-packages.

[28]

Garrett Grolemund and Hadley Wickham. 2014. A cognitive interpretation of data analysis. International Statistical Review 82, 2 (2014), 184–204.

[29]

Jarrod Hadfield. 2020. Package “MCMCglmm”. (2020). Retrieved September 16, 2020 from https://cran.r-project.org/web/packages/MCMCglmm/MCMCglmm.pdf.

[30]

Jarrod D. Hadfield et al. 2010. MCMC methods for multi-response generalized linear mixed models: The MCMCglmm R package. Journal of Statistical Software 33, 2 (2010), 1–22.

[31]

Trevor Hastie and Junyang Qian. 2014. Glmnet vignette. (2014). Retrieved September 16, 2020 from https://web.stanford.edu/hastie/glmnet/glmnet_alpha.html.

[32]

Brian Hempel, Justin Lubin, and Ravi Chugh. 2019. Sketch-n-sketch: Output-directed programming for SVG. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 281–292.

Digital Library

[33]

John H. Holland, Keith J. Holyoak, Richard E. Nisbett, and Paul R. Thagard. 1989. Induction: Processes of Inference, Learning, and Discovery. MIT press.

Digital Library

[34]

Bradley Jones and John Sall. 2011. JMP statistical discovery software. Wiley Interdisciplinary Reviews: Computational Statistics 3, 3 (2011), 188–194.

Digital Library

[35]

Eric Jones, Travis Oliphant, and Pearu Peterson. 2001–2020. SciPy: Open source scientific tools for Python. Retrieved August 6, 2020 from http://www.scipy.org/.

[36]

Eric Jones, Travis Oliphant, and Pearu Peterson. 2001–2020. Statistical functions (scipy.stats). Retrieved September 14, 2020 from https://docs.scipy.org/doc/scipy/reference/stats.html.

[37]

Eric Jones, Travis Oliphant, and Pearu Peterson. 2001–2020. Optimization and root finding (scipy.optimize). Retrieved September 14, 2020 from https://docs.scipy.org/doc/scipy/reference/optimize.html.

[38]

Eunice Jun, Maureen Daum, Jared Roesch, Sarah E. Chasins, Emery D. Berger, Rene Just, and Katharina Reinecke. 2019. Tea: A high-level language and runtime system for automating statistical analysis. In Proceedings of the 32nd Annual Symposium on User Interface Software and Technology. ACM.

Digital Library

[39]

Alex Kale, Matthew Kay, and Jessica Hullman. 2019. Decision-making under uncertainty in research synthesis: Designing for the garden of forking paths. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–14.

Digital Library

[40]

Sean Kandel, Andreas Paepcke, Joseph M. Hellerstein, and Jeffrey Heer. 2012. Enterprise data analysis and visualization: An interview study. IEEE Transactions on Visualization and Computer Graphics 18, 12 (2012), 2917–2926.

Digital Library

[41]

Norbert L. Kerr. 1998. HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review 2, 3 (1998), 196–217.

[42]

David Klahr and Kevin Dunbar. 1988. Dual space search during scientific reasoning. Cognitive Science 12, 1 (1988), 1–48.

[43]

David Klahr and Herbert A. Simon. 1999. Studies of scientific discovery: Complementary approaches and convergent findings. Psychological Bulletin 125, 5 (1999), 524.

[44]

Gary Klein, Jennifer K. Phillips, Erica L. Rall, and Deborah A. Peluso. 2007. A data–frame theory of sensemaking. In Proceedings of theExpertise out of Context. Psychology Press, 118–160.

[45]

Joos Korstanje. 2019. “ANOVA’s three types of estimating sums of squares: Don’t make the wrong choice! Towards Data Science, Medium (2019). Retrieved September 14, 2020 from https://towardsdatascience.com/anovas-three-types-of-estimating-sums-of-squares-don-t-make-the-wrong-choice-91107c77a27a.

[46]

Max Kuhn, Davis Vaughan, and RStudio. 2020. parsnip: A Common API to Modeling and Analysis Functions. Retrieved from https://parsnip.tidymodels.org/.

[47]

Max Kuhn and Hadley Wickham. 2020. Tidymodels: A Collection of Packages for Modeling and Machine Learning Using Tidyverse Principles. Retrieved from https://www.tidymodels.org.

[48]

Michael D. Lee, Amy H. Criss, Berna Devezer, Christopher Donkin, Alexander Etz, Fábio P. Leite, Dora Matzke, Jeffrey N. Rouder, Jennifer S. Trueblood, Corey N. White, and Joachim Vandekerckhove. 2019. Robust modeling in cognitive science. Computational Brain and Behavior 2, 3 (2019), 141–153.

[49]

Jiali Liu, Nadia Boukhelifa, and James R. Eagan. 2019. Understanding the role of alternatives in data analysis practices. IEEE Transactions on Visualization and Computer Graphics 26, 1 (2019), 66–76.

[50]

Yang Liu, Tim Althoff, and Jeffrey Heer. 2020. Paths explored, paths omitted, paths obscured: Decision points & selective reporting in end-to-end data analysis. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.

Digital Library

[51]

StataCorp LLC. 2020. Language syntax. Retrieved September 16, 2020 fromhttps://www.stata.com/manuals13/u11.pdf.

[52]

StataCorp LLC. 2020. Stata 16 Documentation. Retrieved fromhttps://www.stata.com/features/documentation/.

[53]

David J. Lunn, Andrew Thomas, Nicky Best, and David Spiegelhalter. 2000. WinBUGS - A bayesian modelling framework: Concepts, structure, and extensibility. Statistics and Computing 10, 4 (01 Oct 2000), 325–337. DOI:https://doi.org/10.1023/A:1008929526011

Digital Library

[54]

Arni Magnusson, Hans Skaug, Anders Nielsen, Casper Berg, Kasper Kristensen, Martin Maechler, Koen van Bentham, Ben Bolker, Nafis Sadat, Daniel Lüdecke, Russ Lenth, Joseph O’Brien, and Mollie Brooks. 2020. Package “glmmTMB”. (2020). Retrieved September 16, 2020 from https://cran.r-project.org/web/packages/glmmTMB/index.html.

[55]

Richard McElreath. 2020. Statistical Rethinking: A Bayesian Course with Examples in R and Stan. CRC press.

[56]

Allen Newell and Herbert Alexander Simon. 1972. Human Problem Solving. Vol. 104. Prentice-Hall Englewood Cliffs, NJ.

Digital Library

[57]

Chi T. Ngo, Aidan J. Horner, Nora S. Newcombe, and Ingrid R. Olson. 2019. Development of holistic episodic recollection. Psychological Science 30, 12 (2019), 1696–1706.

[58]

Donald A. Norman. 1986. User centered system design: New perspectives on human-computer interaction. CRC Press.

Digital Library

[59]

University of Amsterdam. 2020. JASP: A Fresh Way to do Statistics. Retrieved September 16, 2020 from https://jasp-stats.org/.

[60]

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine learning in python. The Journal of Machine Learning Research 12 (2011), 2825–2830.

Digital Library

[61]

Josef Perktold, Skipper Seabold, Jonathan Taylor, and statsmodels developers. 2020. Statsmodels v0.10.2 reference guide. (2020). Retrieved April 1, 2021 from https://www.statsmodels.org/stable.

[62]

M. Pfannkuch. 1997. Statistical thinking: One statistician’s perspective. Research Papers on Stochastics Education (1997), 171–178.

[63]

Maxine Pfannkuch and Chris J. Wild. 2000. Statistical thinking an statistical practice: Themes gleaned from professional statisticians. Statistical Science 15, 2 (2000), 132–152.

[64]

José Pinheiro, Douglas Bates, Saikat DebRoy, Deepayan Sarkar, EISPACK authors, Siem Heisterkamp, Bert Van Willigen, and R-core. 2020. Package “nlme”. (2020). Retrieved September 16, 2020 from https://cran.r-project.org/web/packages/nlme/nlme.pdf.

[65]

Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of International Conference on Intelligence Analysis, Vol. 5. McLean, VA, 2–4.

[66]

Tanu N. Prabhu. 2019. Top python libraries used in data science. Towards Data Science, Medium (2019). Retrieved December 16, 2019 from https://towardsdatascience.com/top-python-libraries-used-in-data-science-a58e90f1b4ba.

[67]

Brian Ripley, Bill Venables, Douglas M. Bates, Kurt Hornik, Albrecht Gebhardt, and David Firth. 2020. Package “MASS”. (2020). Retrieved September 16, 2020 from https://cran.r-project.org/web/packages/MASS/MASS.pdf.

[68]

Daniel M. Russell, Mark J. Stefik, Peter Pirolli, and Stuart K. Card. 1993. The cost structure of sensemaking. In Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems. ACM, 269–276.

Digital Library

[69]

John Salvatier, Thomas V. Wiecki, and Christopher Fonnesbeck. 2016. Probabilistic programming in python using PyMC3. PeerJ Computer Science 2 (2016), e55.

[70]

SAS. 2020. JMP. Retrieved September 16, 2020 fromhttps://www.jmp.com/en_us/home.html.

[71]

Christian D. Schunn and David Klahr. 1995. A 4-space model of scientific discovery. In Proceedings of the 17th Annual Conference of the Cognitive Science Society. 106–111.

[72]

Christian D. Schunn and David Klahr. 1996. When and how to go beyond a 2-space model of scientific discovery. In Proceedings of the 18th Annual Conference of the Cognitive Science Society: July 12–15, 1996, University of California, San Diego, Vol. 18. Psychology Press, 25.

[73]

scikit-learn developers. 2020. Scikit-learn v0.23.2 documentation. (2020). Retrieved November 20, 2020 from https://scikit-learn.org/stable/.

[74]

Skipper Seabold and Josef Perktold. 2010. Statsmodels: Econometric and statistical modeling with python. In Proceedings of the 9th Python in Science Conference, Vol. 57. Scipy, 61.

[75]

IBM SPSS. [n.d.]. SPSS Software. Retrieved August 18, 2020 fromhttps://www.ibm.com/analytics/spss-statistics-software.

[76]

Stata. [n.d.]. Stata Software. Retrieved September 14, 2020 fromhttps://www.stata.com/.

[77]

Michael Suh. 2014. Higher Education, Gender & Work Dataset. Retrieved September 16, 2020 from https://www.pewsocialtrends.org/category/datasets/?download=20041.

[78]

R Core Team and contributors worldwide. 2020. Package ‘stats’ v4.1.0. CRAN (2020). Retrieved September 16, 2020 from https://stat.ethz.ch/R-manual/R-devel/library/stats/html/00Index.html.

[79]

Inc. The MathWorks. 2020. Matlab. Retrieved from https://www.mathworks.com/.

[80]

Inc. The MathWorks. 2020. Statistics and machine learning toolbox. (2020). Retrieved fromhttps://www.mathworks.com/help/stats/index.html.

[81]

Lea Verou, Tarfah Alrashed, and David Karger. 2018. Extending a reactive expression language with data update actions for end-user application authoring. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 379–387.

Digital Library

[82]

Chat Wacharamanotham, Krishna Subramanian, Sarah Theres Volkel, and Jan Borchers. 2015. Statsplorer: Guiding novices in statistical analysis. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2693–2702.

Digital Library

[83]

Hadley Wickham. 2014. Tidy data. Journal of Statistical Software 59, 1 (2014), 1–23.

[84]

Hadley Wickham, Mara Averick, Jennifer Bryan, Winston Chang, Lucy D’Agostino McGowan, Romain François, Garrett Grolemund, Alex Hayes, Lionel Henry, Jim Hester, Max Kuhn, Thomas Lin Pedersen, Evan Miller, Stephan Milton Bache, Kirill Müller, Jeroen Ooms, David Robinson, Dana Paige Seidel, Vitalie Spinu, Kohske Takahashi, Davis Vaughan, Claus Wilke, Kara Woo, and, Hiroaki Yutani. 2019. Welcome to the tidyverse. Journal of Open Source Software 4, 43 (2019), 1686.

[85]

Chris J. Wild and Maxine Pfannkuch. 1999. Statistical thinking in empirical enquiry. International Statistical Review 67, 3 (1999), 223–248.

[86]

Kanit Wongsuphasawat, Yang Liu, and Jeffrey Heer. 2019. Goals, process, and challenges of exploratory data analysis: An interview study. arXiv:1911.00568. Retrieved from https://arxiv.org/abs/1911.00568.

[87]

Bin Yu and Karl Kumbier. 2020. Veridical data science. In Proceedings of the National Academy of Sciences 117, 8 (2020), 3920–3929.

Cited By

Lam MTeoh JLanday JHeer JBernstein M(2024)Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642830(1-28)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642830
Jun EMisback EHeer JJust R(2024)rTisane: Externalizing conceptual models for data analysis prompts reconsideration of domain assumptions and facilitates statistical modelingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642267(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642267
Gu KGrunde-McLaughlin MMcNutt AHeer JAlthoff T(2024)How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz StudyProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641891(1-22)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641891
Show More Cited By

Index Terms

Hypothesis Formalization: Empirical Findings, Software Limitations, and Design Implications
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Comparing empirical likelihood and bootstrap hypothesis tests
Robust Statistical Methods for Empirical Software Engineering

There have been many changes in statistical theory in the past 30 years, including increased evidence that non-robust methods may fail to detect important results. The statistical advice available to software engineering researchers needs to be updated ...
Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power

Experimental analysis of the performance of a proposed method is a crucial and necessary task in an investigation. In this paper, we focus on the use of nonparametric statistical inference for analyzing the results obtained in an experiment design in ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Computer-Human Interaction

ACM Transactions on Computer-Human Interaction Volume 29, Issue 1

February 2022

354 pages

ISSN:1073-0516

EISSN:1557-7325

DOI:10.1145/3505201

Editor:
Kristina Höök
KTH Royal Institute of Technology, Sweden

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 January 2022

Accepted: 01 July 2021

Revised: 01 July 2021

Received: 01 February 2021

Published in TOCHI Volume 29, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
575
Total Downloads

Downloads (Last 12 months)162
Downloads (Last 6 weeks)22

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lam MTeoh JLanday JHeer JBernstein M(2024)Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642830(1-28)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642830
Jun EMisback EHeer JJust R(2024)rTisane: Externalizing conceptual models for data analysis prompts reconsideration of domain assumptions and facilitates statistical modelingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642267(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642267
Gu KGrunde-McLaughlin MMcNutt AHeer JAlthoff T(2024)How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz StudyProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641891(1-22)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641891
Xia TCarayannis ESindakis SShowkat SKanellos N(2023)Technology transfer for sustainable rural development: evidence from homestead withdrawal with compensation in Chengdu–ChongqingThe Journal of Technology Transfer10.1007/s10961-023-10019-949:1(303-333)Online publication date: 3-Oct-2023
https://doi.org/10.1007/s10961-023-10019-9
Pang YReinecke KJust R(2022)Apéritif: Scaffolding Preregistrations to Automatically Generate Analysis Code and Methods DescriptionsProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517707(1-15)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517707

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents