Abstract
When learning a new technology, programmers often have to sift through multiple online resources to find information that addresses their questions. Prior work has reported that information seekers use a number of different strategies, including following scents, or indicators, to locate appropriate resources. We present a qualitative and quantitative investigation of how programmers learning a new technology employ these strategies to navigate between online resources and evaluate the pertinence of these resources. We performed a diary and interview study with ten programmers learning a new technology, to study how users navigate from the question they have to the resource that satisfies this need. Based on our observations, we propose a resource-seeking model that represents the online resource seeking behaviour of programmers when learning a new technology. The model is comprised of six components that can be divided into two groups: Need-oriented components, i.e. Questions, Preferences, and Beliefs, and Resource-oriented components, i.e. Resources, Cues, and Impression Factors. We identified nine relations between these components and studied how the components are associated. We report on the characteristics of the components and the relationships between them, and discuss the importance of search customization and other implications of our observations for resource creators and search tools.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Code Availability
Not applicable
References
Abdi H et al (2007) Bonferroni and šidák corrections for multiple comparisons. In: Encyclopedia of measurement and statistics, vol 3, pp 103–107
Arya DM, Guo JLC, Robillard MP (2020) Information correspondence between types of documentation for APIs. Empir Softw Eng 25(5):4069–4096
Arya DM, Nassif M, Robillard MP (2021) A data-centric study of software tutorial design. IEEE Softw
Bai GR, Kayani J, Stolee KT (2020) How graduate computing students search when using an unfamiliar programming language. In: Proceedings of the 28th international conference on program comprehension, ICPC, Association for computing machinery, pp 160–171
Brandt J, Guo PJ, Lewenstein J, Dontcheva M, Klemmer SR (2009) Two studies of opportunistic programming: interleaving web foraging, learning, and writing code. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ’09. Association for Computing Machinery, pp 1589–1598
Carroll J (1990) An overview of minimalist instruction. In: Annual Hawaii international conference on system sciences, vol 4. IEEE Computer Society, pp 210–219
Chattopadhyay S, Nelson N, Au A, Morales N, Sanchez C, Pandita R, Sarma A (2020) A tale from the trenches: cognitive biases and software development. In: Proceedings of the 42nd ACM/IEEE international conference on software engineering, ICSE ’20. Association for Computing Machinery, New York, pp 654–665
Dondio P, Shaheen S (2019) Is Stack Overflow an effective complement to gaining practical knowledge compared to traditional computer science learning?. In: Proceedings of the international conference on education technology and computers (ICETC), pp 132–138
Duala-Ekoko E, Robillard MP (2012) Asking and answering questions about unfamiliar apis: an exploratory study. In: 2012 34th International conference on software engineering (ICSE). IEEE, pp 266–276
Earle RH, Rosso MA, Alexander KE (2015) User preferences of software documentation genres. In: Proceedings of the 33rd annual international conference on the design of communication, SIGDOC ’15. Association for Computing Machinery
Erdem A, Marsella S, Johnson W (1998) Task oriented software understanding. In: Proceedings of international conference on automated software engineering. IEEE Computer Society, p 230
Erdos K, Sneed HM (1998) Partial comprehension of complex programs (enough to perform maintenance). In: Proceedings. 6th International workshop on program comprehension. IWPC’98, pp 98–105
Escobar-Avila J, Venuti D, Di Penta M, Haiduc S (2019) A survey on online learning preferences for computer science and programming. In: Proceedings of international conference on software engineering: software engineering education and training (ICSE-SEET), pp 170–181
Gallardo-Valencia RE, Sim SE (2011) What kinds of development problems can be solved by searching the web?: a field study. In: Proceedings—international conference on software engineering, pp 41–44
Jääskeläinen R (2010) Think-aloud protocol. In: Handbook of translation studies, vol 1, pp 371–374
Ko AJ, DeLine R, Venolia G (2007) Information needs in collocated software development teams. In: 29th International conference on software engineering, pp 344–353
Landis JR, Koch G (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174
Lawrance J, Bellamy R, Burnett M, Rector K (2008) Using information scent to model the dynamic foraging behavior of programmers in maintenance tasks. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ’08. Association for Computing Machinery, pp 1323–1332
Lazar J, Feng JH, Hochheiser H Lazar J, Feng J H, Hochheiser H (eds) (2017a) Chapter 11—analyzing qualitative data. Morgan Kaufmann, Boston
Lazar J, Feng JH, Hochheiser H (2017b) Chapter 6—diaries. In: Lazar J, Feng J H, Hochheiser H (eds) Research methods in human computer interaction, 2nd edn. Morgan Kaufmann, Boston, pp 135–152
Liu MX, Kittur A, Myers BA (2021) To reuse or not to reuse? A framework and system for evaluating summarized knowledge. In: Proceedings of the ACM on human-computer interaction (CSCW1)
Lu Y, Hsiao IH (2017) Personalized information seeking assistant (pisa): from programming information seeking to learning. Inf Retr 20:433–455
Mack RL, Lewis CH, Carroll JM (1983) Learning to use word processors: problems and prospects. ACM Trans Inf Syst 1(3):254–271
Marques A, Bradley NC, Murphy GC (2020) Characterizing task-relevant information in natural language software artifacts. In: IEEE international conference on software maintenance and evolution (ICSME), pp 476–487
Mehta CR, Patel NR (2011) IBM SPSS exact tests. IBM Corporation, Armonk
Meng M, Steinhardt S, Schubert A (2019) How developers use API documentation: an observation study. Commun Des Q Rev 7:40–49
Nadi S, Treude C (2020) Essential sentences for navigating stack overflow answers. In: International conference on software analysis, evolution and reengineering (SANER). IEEE, pp 229–239
Piorkowski D, Fleming SD, Scaffidi C, Burnett M, Kwan I, Henley AZ, Macbeth J, Hill C, Horvath A (2015) To fix or to learn? How production bias affects developers’ information foraging during debugging. In: IEEE International conference on software maintenance and evolution (ICSME), pp 11–20
Pirolli P, Card S (1999) Information foraging. Psychol Rev 106:643–675
Pirolli P, Fu WT (2003) SNIF-ACT: a model of information foraging on the World Wide Web
Rao N, Bansal C, Zimmermann T, Awadallah AH, Nagappan N (2019) Analyzing web search behavior for software engineering tasks. arXiv:1912.09519
Robillard MP, Marcus A, Treude C, Bavota G, Chaparro O, Ernst N, Gerosa MA, Godfrey M, Lanza M, Linares-vásquez M, Murphy GC, Moreno L, Shepherd D, Wong E (2017) On-demand developer documentation. In: International conference on software maintenance and evolution (ICSME). IEEE, pp 479–483
Sadowski C, Stolee KT, Elbaum S (2015) How developers search for code: a case study. In: Proceedings of the 10th joint meeting on foundations of software engineering, ESEC/FSE. Association for Computing Machinery, pp 191–201
Sharpe D (2015) Chi-square test is statistically significant: now what? Pract Assess Res Eval 20(1):8
Sillito J, Murphy GC, De Volder K (2006) Questions programmers ask during software evolution tasks. In: Proceedings of the SIGSOFT international symposium on foundations of software engineering (FSE). Association for Computing Machinery, pp 23–34
Sprent P (2011) Fisher exact test. Springer, Berlin, pp 524–525
Srinivasa Ragavan S, Kuttal SK, Hill C, Sarma A, Piorkowski D, Burnett M (2016) Foraging among an overabundance of similar variants. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ’16. Association for Computing Machinery, New York, pp 3509–3521
Teevan J, Alvarado C, Ackerman MS, Karger DR (2004) The perfect search engine is not enough: a study of orienteering behavior in directed search. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 415–422
Treude C, Aniche M (2018) Where does google find API documentation?. In: Proceedings of international conference on software engineering. ACM, pp 23–26
Xia X, Bao L, Lo D, Kochhar PS, Hassan A, Xing Z (2017) What do developers search for on the web? Empir Softw Eng 22:3149–3185
Xie I, Joo S (2012) Factors affecting the selection of search tactics: tasks, knowledge, process, and systems. Inf Process Manag 48(2):254–270
Funding
This work has been funded by the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant.
Author information
Authors and Affiliations
Contributions
Jin L.C. Guo and Martin P. Robillard contributed equally to the research and manuscript.
Corresponding author
Ethics declarations
Ethics Approval
This study is approved by the Research Ethics Board Office at McGill University (file number: 20-07-039).
Conflict of Interests
The authors have no conflicts of interests to declare that are relevant to the content of this article.
Competing Interests
The authors have no competing interests to declare that are relevant to the content of this article.
Additional information
Communicated by: Venera Arnaoudova
Availability of Data and Material
This article is complemented by an appendix containing the documents required to replicate the study as well as our data set to reproduce our results. The appendix has been uploaded as supplementary material. Please refer to the supplementary file README.md for the list of appendix documents and a description of their content. The appendix will be made publicly available upon eventual publication of the article.
Informed Consent
All participants were provided with an informed consent form which needed to be duly signed prior to beginning the study. We proceeded with the study only if we received their signed informed consent. The informed consent form has been uploaded as a supplementary form as “Informed Consent Form.pdf” for reference.
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Arya, D.M., Guo, J.L.C. & Robillard, M.P. How programmers find online learning resources. Empir Software Eng 28, 23 (2023). https://doi.org/10.1007/s10664-022-10246-y
Accepted:
Published:
DOI: https://doi.org/10.1007/s10664-022-10246-y