Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3406522.3446056acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
research-article
Open access

A Manifesto on Resource Re-Use in Interactive Information Retrieval

Published: 14 March 2021 Publication History

Abstract

This perspective paper on resource re-use intends to draw the attention of the interactive information retrieval (IIR) community to the challenges of research documentation and archiving for future use. Resources are understood as encompassing research designs, research data and research infrastructures. It proposes eight principles for improving the re-use of resources in the IIR community and presents concrete steps on how to achieve them. A five-level system for data archiving and documentation envisions increasingly open and stable documentation and access infrastructures.

References

[1]
ACM. 2020. Artifact Review and Badging. https://www.acm.org/publications/policies/artifact-review-badging.
[2]
Alice Allen, Cecilia Aragon, Christoph Becker, Jeffrey Carver, Andrei Chis, Benoit Combemale, Mike Croucher, Kevin Crowston, Daniel Garijo, Ashish Gehani, Carole Goble, Robert Haines, Robert Hirschfeld, James Howison, Kathryn Huff, Caroline Jay, Daniel S. Katz, Claude Kirchner, Katie Kuksenok, Ralf Lämmel, Oscar Nierstrasz, Matt Turk, Rob van Nieuwpoort, Matthew Vaughn, and Jurgen J. Vinju. 2017. Engineering Academic Software (Dagstuhl Perspectives Workshop 16252). Dagstuhl Manifestos 6, 1 (2017), 1--20. https://doi.org/10.4230/DagMan.6.1.1
[3]
J. Andreoni, W. Harbaugh, and L. Vesterlund. 2003. The Carrot or the Stick: Rewards, Punishments, and Cooperation. American Economic Review 93, 3 (2003), 893--902.
[4]
Toine Bogers, Samuel Dodson, Luanne Freund, Maria Gäde, Mark Hall, Marijn Koolen, Vivien Petras, Nils Pharo, and Mette Skov. 2019. Overview of the CHIIR 2019 Workshop on Barriers to InteractiveIR Resources Re-use (BIIRRR 2019). In CEUR Workshop Proceedings, Vol. 2337. CEUR-WS.org, Aachen, Germany, 1--6. http://ceur-ws.org/Vol-2337/overview.pdf
[5]
Toine Bogers, Maria Gäde, Mark M. Hall, Vivien Petras, and Mette Skov. 2017. Lessons Learned from the CHiC and SBS Interactive Tracks: A Wishlist for Interactive IR Evaluation. In Proceedings of the Second Workshop on Supporting Complex Search Tasks co-located with the ACM SIGIR Conference on Human Information Interaction & Retrieval (CHIIR 2017), Oslo, Norway, March 11, 2017 (CEUR Workshop Proceedings, Vol. 1798), Marijn Koolen, Jaap Kamps, Toine Bogers, Nicholas J. Belkin, Diane Kelly, and Emine Yilmaz (Eds.). CEUR-WS.org, Aachen, Germany, 11--14. http://ceur-ws.org/Vol-1798/paper2.pdf
[6]
Toine Bogers, Maria Gäde, Mark Michael, Luanne Freund, Marijn Koolen, Vivien Petras, and Mette Skov. 2018. Report on the workshop on barriers to interactive IR resources re-use (BIIRRR 2018). ACM SIGIR Forum 52, 1 (2018), 119--128.
[7]
Christine L Borgman. 2012. The Conundrum of Sharing Research Data. Journal of the American Society for Information Science and Technology 63, 6 (2012), 1059-- 1078.
[8]
Christine L Borgman. 2015. Big data, little data, no data: Scholarship in the networked world. MIT press, Cambridge, MA.
[9]
Pia Borlund. 2003. The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Information research 8, 3 (2003), 8--3.
[10]
Julianne Cheek. 2008. Research design. In The SAGE encyclopedia of qualitative research methods. SAGE Publications, Inc., Thousand Oaks, CA, 762--764.
[11]
Henry Chesbrough and Jim Spohrer. 2006. A Research Manifesto for Services Science. Commun. ACM 49, 7 (July 2006), 35--40.
[12]
Jon F. Claerbout and Martin Karrenbach. 1992. Electronic Documents give Reproducible Research a New Meaning. In SEG Technical Program Expanded Abstracts 1992. Society of Exploration Geophysicists, Tulsa, OK, 601--604.
[13]
Rosaria Conte, Nigel Gilbert, Giulia Bonelli, Claudio Cioffi-Revilla, Guillaume Deffuant, Janos Kertesz, Vittorio Loreto, Suzy Moat, J-P Nadal, Anxo Sanchez, et al. 2012. Manifesto of computational social science. The European Physical Journal Special Topics 214, 1 (2012), 325--346.
[14]
Joenio Costa, Paulo Meirelles, and Christina Chavez. 2018. On the Sustainability of Academic Software: The Case of Static Analysis Tools. In Proceedings of the XXXII Brazilian Symposium on Software Engineering (Sao Carlos, Brazil) (SBES '18). Association for Computing Machinery, New York, NY, USA, 202--207. https: //doi.org/10.1145/3266237.3266243
[15]
John W. Creswell and J. David Creswell. 2017. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Sage Publications, Thousand Oaks, CA.
[16]
Giorgio De Michelis, Eric Dubois, Matthias Jarke, Florian Matthes, John Mylopoulos, Mike Papazoglou, Klaus Pohl, Joachim Schmidt, Carson Woo, and Eric Yu. 1997. Cooperative information systems: a manifesto. Accademic-Press, Cambridge, MA.
[17]
Taylor & Francis. 2020. Open and FAIR data sharing policy. https://authorservices.taylorandfrancis.com/data-sharing-policies/openand-fair/.
[18]
Luanne Freund and Barbara M. Wildemuth. 2014. Documenting and Studying the Use of Assigned Search Tasks: RepAST. Proceedings of the American Society for Information Science and Technology 51, 1 (2014), 1--4.
[19]
Maria Gäde, Mark Michael Hall, Hugo C. Huurdeman, Jaap Kamps, Marijn Koolen, Mette Skov, Toine Bogers, and David Walsh. 2016. Overview of the SBS 2016 Interactive Track. In Working Notes of the CLEF 2016 Conference (CEUR Workshop Proceedings, Vol. 1609), Krisztian Balog, Linda Cappellato, Nicola Ferro, and Craig Macdonald (Eds.). CEUR-WS.org, Aachen, Germany, 1024--1038.
[20]
Maria Gäde, Mark Michael Hall, Hugo C. Huurdeman, Jaap Kamps, Marijn Koolen, Mette Skov, Elaine Toms, and David Walsh. 2015. Overview of the SBS 2015 Interactive Track. In Working Notes of the CLEF 2015 Conference (CEUR Workshop Proceedings, Vol. 1391), Linda Cappellato, Nicola Ferro, Gareth J. F. Jones, and Eric SanJuan (Eds.). CEUR-WS.org, Aachen, Germany, 14.
[21]
Teresa Gomez-Diaz and Tomas Recio. 2019. On the evaluation of research software: the CDUR procedure. F1000Research 8 (2019). https://doi.org/10.12688/ f1000research.19994.2
[22]
Mark M Hall. 2019. To re-use is to re-write: experiences with re-using IIR experiment software. In CEUR Workshop Proceedings, Vol. 2337. CEUR-WS.org, Aachen, Germany, 19--23.
[23]
Mark Michael Hall and Toine Bogers. 2020. A Standardised Format for Exchanging User Study Instruments. In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval. ACM, New York City, NY, 457--461.
[24]
Mark Michael Hall, Hugo C. Huurdeman, Marijn Koolen, Mette Skov, and David Walsh. 2014. Overview of the INEX 2014 Interactive Social Book Search Track. In Working Notes of the CLEF 2014 Conference (CEUR Workshop Proceedings, Vol. 1180), Linda Cappellato, Nicola Ferro, Martin Halvey, and Wessel Kraaij (Eds.). CEUR-WS.org, Aachen, Germany, 480--493.
[25]
Mark M Hall, Spyros Katsaris, and Elaine Toms. 2013. A Pluggable Interactive IR Evaluation Work-bench. In European Workshop on Human-Computer Interaction and Information Retrieval. Springer, Heidelberg, Germany, 35--38.
[26]
Diana Hicks, Paul Wouters, Ludo Waltman, Sarah De Rijcke, and Ismael Rafols. 2015. Bibliometrics: the Leiden Manifesto for research metrics. Nature 520, 7548 (2015), 429--431.
[27]
Stephen Hilgartner and Sherry I. Brandt-Rauf. 1994. Data Access, Ownership, and Control: Toward Empirical Studies of Access Practices. Knowledge 15, 4 (1994), 355--372. https://doi.org/10.1177/107554709401500401 arXiv:https://doi.org/10.1177/107554709401500401
[28]
Timothy Holborn. 2014. What is 5 Star Linked Data? https://www.w3.org/community/webize/2014/01/17/what-is-5-star-linkeddata/.
[29]
Hugo C. Huurdeman, Jaap Kamps, and Max L. Wilson. 2019. The Multi-Stage Experience: the Simulated Work Task Approach to Studying Information Seeking Stages. In Proceedings of the CHIIR 2019 Workshop on Barriers to Interactive IR Resources Re-use co-located with the ACM SIGIR Conference on Human Information Interaction and Retrieval, BIIRRR@CHIIR 2019, Glasgow, UK, March 14, 2019 (CEUR Workshop Proceedings, Vol. 2337), Toine Bogers, Samuel Dodson, Maria Gäde, Luanne Freund, Mark M. Hall, Marijn Koolen, Vivien Petras, Nils Pharo, and Mette Skov (Eds.). CEUR-WS.org, Aachen, Germany, 7--13. http://ceur-ws.org/Vol2337/paper1.pdf
[30]
Diane Kelly. 2009. Methods for Evaluating Interactive Information Retrieval Systems with Users. Foundations and Trends in Information Retrieval 3, 1--2 (2009), 1--224.
[31]
Florian G Kern and Kristian Skrede Gleditsch. 2017. Exploring pre-registration and pre-analysis plans for qualitative inference. Preprint ahead of publication (2017), 1--15.
[32]
Youngseek Kim and Melissa Adler. 2015. Social scientists? data sharing behaviors: Investigating the roles of individual motivations, institutional pressures, and data repositories. International journal of information management 35, 4 (2015), 408--418.
[33]
Effie Lai-Chong Law, Arnold POS Vermeeren, Marc Hassenzahl, and Mark Blythe. 2007. Towards a UX manifesto. In Proceedings of HCI 2007 The 21st British HCI Group Annual Conference University of Lancaster, UK 21. BCS, The Chartered Institute for IT, Swindon, UK, 1--2.
[34]
Sara Mannheimer, Amy Pienta, Dessislava Kirilova, Colin Elman, and Amber Wutich. 2019. Qualitative data sharing: Data repositories and academic libraries as key partners in addressing challenges. American Behavioral Scientist 63, 5 (2019), 643--664.
[35]
Edward Miguel, Colin Camerer, Katherine Casey, Joshua Cohen, Kevin M Esterling, Alan Gerber, Rachel Glennerster, Don P Green, Macartan Humphreys, Guido Imbens, et al. 2014. Promoting transparency in social science research. Science 343, 6166 (2014), 30--31.
[36]
Geoffrey Miller. 2012. The Smartphone Psychology Manifesto. Perspectives on Psychological Science 7, 3 (2012), 221--237. https://doi.org/10.1177/1745691612441215 arXiv:https://doi.org/10.1177/1745691612441215 26168460.
[37]
Marcus R. Munafò, Brian A. Nosek, Dorothy V. M. Bishop, Katherine S. Button, Christopher D. Chambers, Nathalie Percie Du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J. Ware, and John P. A. Ioannidis. 2017. A Manifesto for Reproducible Science. Nature Human Behaviour 1, 1 (2017), 1--9.
[38]
Ragnar Nordlie and Nils Pharo. 2012. Seven years of INEX interactive retrieval experiments--lessons and challenges. In International Conference of the CrossLanguage Evaluation Forum for European Languages. Springer, Heidelberg, Germany, 13--23.
[39]
Paul Over. 2001. The TREC Interactive Track: An Annotated Bibliography. Information Processing & Management 37, 3 (2001), 369--381.
[40]
Heinz Pampel, Paul Vierkant, Frank Scholze, Roland Bertelmann, Maxi Kindling, Jens Klump, Hans-Jürgen Goebelbecker, Jens Gundlach, Peter Schirmbacher, and Uwe Dierolf. 2013. Making research data repositories visible: the re3data. org registry. PloS one 8, 11 (2013), e78080.
[41]
Irene V. Pasquetto, Christine L. Borgman, and Morgan F. Wofford. 2019. Uses and Reuses of Scientific Data: The Data Creators? Advantage. Harvard Data Science Review 1, 2 (15 11 2019). https://doi.org/10.1162/99608f92.fc14bf2d https://hdsr.mitpress.mit.edu/pub/jduhd7og.
[42]
Irene V Pasquetto, Bernadette M Randles, and Christine L Borgman. 2017. On the reuse of scientific data. Data Science Journal 16 (2017), 8. https://doi.org/10. 5334/dsj-2017-008
[43]
Ray Pawson. 2013. The science of evaluation: a realist manifesto. SAGE Publications, Los Angeles, CA.
[44]
Vivien Petras, Toine Bogers, and Maria Gaede. 2019. Elements of IIR Studies: A Review of the 2006--2018 IIiX and CHIIR Conferences. In Proceedings of the CHIIR 2019 Workshop on Barriers to Interactive IR Resources Re-use (BIIRRR 2019). CEUR-WS.org, Aachen, Germany, 37--41.
[45]
Vivien Petras, Toine Bogers, Elaine G. Toms, Mark M. Hall, Jacques Savoy, Piotr Malak, Adam Pawlowski, Nicola Ferro, and Ivano Masiero. 2013. Cultural Heritage in CLEF (CHiC 2013). In CLEF '13: Proceedings of the 4th International Conference of the CLEF Initiative (Lecture Notes in Computer Science, Vol. 8138), Pamela Forner, Henning Müller, Roberto Paredes, Paolo Rosso, and Benno Stein (Eds.). Springer, Heidelberg, Germany, 192--211.
[46]
Vivien Petras, Marijn Koolen, Maria Gaede, and Toine Bogers. 2019. Experiences with the 2013--2016 CLEF Interactive Information Retrieval Tracks. In Proceedings of the CHIIR 2019 Workshop on Barriers to Interactive IR Resources Re-use (BIIRRR 2019). CEUR-WS.org, Aachen, Germany, 29--36.
[47]
Nils Pharo, Thomas Beckers, Ragnar Nordlie, and Norbert Fuhr. 2011. Overview of the INEX 2010 Interactive Track. In INEX '10: Proceedings of the Ninth International Workshop of the Initiative for the Evaluation of XML Retrieval, Shlomo Geva, Jaap Kamps, Ralf Schenkel, and Andrew Trotman (Eds.). Springer, Berlin, Heidelberg, 227--235.
[48]
Rosalind W Picard, Seymour Papert, Walter Bender, Bruce Blumberg, Cynthia Breazeal, David Cavallo, Tod Machover, Mitchel Resnick, Deb Roy, and Carol Strohecker. 2004. Affective learning?a manifesto. BT technology journal 22, 4 (2004), 253--269. https://doi.org/10.1023/B:BTTJ.0000047603.37042.33
[49]
Graham Pryor. 2012. Managing Research Data. Facet Publishing, London, UK.
[50]
Karsten Boye Rasmussen and Grant Blank. 2007. The data documentation initiative: a preservation standard for research. Archival Science 7, 1 (2007), 55--71.
[51]
Gareth Renaud and Leif Azzopardi. 2012. SCAMP: A Tool for Conducting Interactive Information Retrieval Experiments. In IIiX '12: Proceedings of the 4th Information Interaction in Context Symposium. ACM, New York City, NY, 286--289.
[52]
Christof Schöch. 2017. Wiederholende Forschung in den digitalen Geisteswissenschaften. In DHd2017: Digital Nachhaltigkeit (DHd2017), Bern, Switzerland, 13--18 February 2017. Zenodo, Genève, Swizterland, 207--212. https://doi.org/10. 5281/zenodo.277113
[53]
Nicholas Smale, Kathryn Unsworth, Gareth Denyer, and Daniel Barr. 2018. The History, Advocacy and Efficacy of Data Management Plans. bioRxiv (2018), 30. https://doi.org/10.1101/443499 arXiv:https://www.biorxiv.org/content/early/2018/10/17/443499.full.pdf
[54]
Diana Soltani, Matthew Mitsui, and Chirag Shah. 2019. Coagmento v3. 0: Rapid prototyping of web search experiments. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval. ACM, New York City, NY, 367--371.
[55]
Susan E Swogger. 2013. PsycTESTS. Journal of the Medical Library Association: JMLA 101, 3 (2013), 234.
[56]
Anastasios Tombros, Birger Larsen, and Saadia Malik. 2005. The Interactive Track at INEX 2004. In INEX '04: Proceedings of the Third International Workshop of the Initiative for the Evaluation of XML Retrieval, Norbert Fuhr, Mounia Lalmas, Saadia Malik, and Zoltán Szlávik (Eds.). Springer, Berlin, Heidelberg, 410--423.
[57]
Elaine G Toms, Luanne Freund, and Cara Li. 2004. WiIRE: the Web interactive information retrieval experimentation system prototype. Information Processing & Management 40, 4 (2004), 655--675.
[58]
Elaine G. Toms and Mark M. Hall. 2013. The CHiC Interactive Task (CHiCi) at CLEF 2013. In Working Notes of the CLEF 2013 Conference (CEUR Workshop Proceedings, Vol. 1179), Pamela Forner, Roberto Navigli, Dan Tufis, and Nicola Ferro (Eds.). CEUR-WS.org, Aachen, Germany, 12.
[59]
U.S. National Science Board. 2005. Long-lived digital data collections: Enabling research and education in the 21st century (No. US NSF-NSB-05--40). https://www.nsf.gov/pubs/2005/nsb0540/.
[60]
Stephanie van de Sandt, Artemis Lavasa, Sünje Dallmeier-Tiessen, and Vivien Petras. 2019. The Definition of Reuse. Data Science Journal 18 (2019), 22.
[61]
Anna Elisabeth van?t Veer and Roger Giner-Sorolla. 2016. Pre-registration in social psychology?A discussion and suggested template. Journal of Experimental Social Psychology 67 (2016), 2--12.
[62]
Katrin Weller and Katharina E. Kinder-Kurlanda. 2016. A Manifesto for Data Sharing in Social Media Research. In Proceedings of the 8th ACM Conference on Web Science (Hannover, Germany) (WebSci '16). Association for Computing Machinery, New York, NY, USA, 166--172. https://doi.org/10.1145/2908131.2908172
[63]
Mark Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gaby Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Olavo Bonino da Silva Santos, Philip Bourne, Jildau Bouwman, Anthony Brookes, Tim Clark, Merce Crosas, Ingrid Dillo, Olivier Dumon, Scott Edmunds, Chris Evelo, Richard Finkers, and Barend Mons. 2016. The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data 3 (03 2016). https://doi.org/10.1038/sdata.2016.18
[64]
Paul Willis and Mats Trondman. 2000. Manifesto for Ethnography. Ethnography 1, 1 (2000), 5--16. https://doi.org/10.1177/14661380022230679

Cited By

View all
  • (2024)The Impact of CHIIR Publications: A Study of Eight Years of CHIIRProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638338(34-44)Online publication date: 10-Mar-2024
  • (2024)Evaluating Digital Library Search Systems by Using Formal Process ModellingProceedings of the 2023 ACM/IEEE Joint Conference on Digital Libraries10.1109/JCDL57899.2023.00058(1-12)Online publication date: 26-Jun-2024
  • (2023)A Systematic Review of Cost, Effort, and Load Research in Information Search and Retrieval, 1972–2020ACM Transactions on Information Systems10.1145/358306942:1(1-39)Online publication date: 18-Aug-2023
  • Show More Cited By

Index Terms

  1. A Manifesto on Resource Re-Use in Interactive Information Retrieval

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHIIR '21: Proceedings of the 2021 Conference on Human Information Interaction and Retrieval
    March 2021
    384 pages
    ISBN:9781450380553
    DOI:10.1145/3406522
    This work is licensed under a Creative Commons Attribution-ShareAlike International 4.0 License.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 14 March 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Badges

    • Best Paper

    Author Tags

    1. data documentation
    2. data sharing practices
    3. re-use
    4. research data
    5. research design
    6. research software

    Qualifiers

    • Research-article

    Conference

    CHIIR '21
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 55 of 163 submissions, 34%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)157
    • Downloads (Last 6 weeks)19
    Reflects downloads up to 23 Sep 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)The Impact of CHIIR Publications: A Study of Eight Years of CHIIRProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638338(34-44)Online publication date: 10-Mar-2024
    • (2024)Evaluating Digital Library Search Systems by Using Formal Process ModellingProceedings of the 2023 ACM/IEEE Joint Conference on Digital Libraries10.1109/JCDL57899.2023.00058(1-12)Online publication date: 26-Jun-2024
    • (2023)A Systematic Review of Cost, Effort, and Load Research in Information Search and Retrieval, 1972–2020ACM Transactions on Information Systems10.1145/358306942:1(1-39)Online publication date: 18-Aug-2023
    • (2023)Toward A Two-Sided Fairness Framework in Search and RecommendationProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578332(236-246)Online publication date: 19-Mar-2023
    • (2023)How we Work, Share, and Re-use at CHIIRProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578305(351-356)Online publication date: 19-Mar-2023
    • (2023)Collaboration Patterns and Impact of Sharing at CHIIRProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578276(313-320)Online publication date: 19-Mar-2023
    • (2023)Back to the Fundamentals: Extend the Rational AssumptionsA Behavioral Economics Approach to Interactive Information Retrieval10.1007/978-3-031-23229-9_5(131-152)Online publication date: 18-Feb-2023
    • (2023)From Rational Agent to Human with Bounded RationalityA Behavioral Economics Approach to Interactive Information Retrieval10.1007/978-3-031-23229-9_3(65-89)Online publication date: 18-Feb-2023
    • (2022)Third Workshop on Building towards Information Interaction and Retrieval Resources Re-use (BIIRRR 2022)Proceedings of the 2022 Conference on Human Information Interaction and Retrieval10.1145/3498366.3505838(374-376)Online publication date: 14-Mar-2022
    • (2022)ir_metadata: An Extensible Metadata Schema for IR ExperimentsProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531738(3078-3089)Online publication date: 6-Jul-2022
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Get Access

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media