article

How users assess web pages for information seeking

Authors:

Anastasios Tombros,

Ian Ruthven,

Joemon M. JoseAuthors Info & Claims

Journal of the American Society for Information Science and Technology, Volume 56, Issue 4

Pages 327 - 344

https://doi.org/10.1002/asi.20106

Published: 15 February 2005 Publication History

Abstract

In this article, we investigate the criteria used by online searchers when assessing the relevance of Web pages for information-seeking tasks. Twenty-four participants were given three tasks each, and they indicated the features of Web pages that they used when deciding about the usefulness of the pages in relation to the tasks. These tasks were presented within the context of a simulated work-task situation. We investigated the relative utility of features identified by participants (Web page content, structure, and quality) and how the importance of these features is affected by the type of information-seeking task performed and the stage of the search. The results of this study provide a set of criteria used by searchers to decide about the utility of Web pages for different types of tasks. Such criteria can have implications for the design of systems that use or recommend Web pages.

References

[1]

Barry, C.L. (1994). User-defined relevance criteria: An exploratory study. Journal of the American Society for Information Science, 45(3), 149-159.

Digital Library

Google Scholar

[2]

Barry, C.L. (1998). Document representations and clues to document relevance. Journal of the American Society for Information Science, 49(14), 1293-1303.

Digital Library

Google Scholar

[3]

Borlund, P. (2000). Experimental components for the evaluation of interactive information retrieval systems. Journal of Documentation, 56(1), 71-90.

Crossref

Google Scholar

[4]

Chi, E.H., Pirolli, P., Chen, K., & Pitkow, J. (2001). Using information scent to model user information needs and actions on the Web. Proceedings of the ACM SIGCHI Conference (pp. 490-497).

Crossref

Google Scholar

[5]

Cool, C., Belkin, N.J., & Kantor, P.B. (1993). Characteristics of text affecting relevance judgments. In Proceedings of the 14th National Online Meeting (pp. 77-84). Medford, NJ: Learned Information.

Google Scholar

[6]

Cothey, V. (2002). A longitudinal study of World Wide Web users' information-seeking behavior. Journal of the American Society for Information Science and Technology, 53(2), 67-78.

Digital Library

Google Scholar

[7]

Czerwinski, M.P., van Dantzich, M., Robertson, G., & Hoffmann, H. (1999). The contribution of thumbnail image, mouse-over text and spatial location memory to Web page retrieval in 3D. In Proceedings of Interact '99 (pp. 163-170).

Google Scholar

[8]

Dziadosz, S., & Chandrasekar, R. (2002). Do thumbnail previews help users make better relevance decisions about Web search results? In Proceedings of the ACM SIGIR Conference (pp. 365-366).

Crossref

Google Scholar

[9]

Fleming, J. (1998). Web navigation: Designing the user experience. Sebastopol, CA: O'Reilly & Associates.

Crossref

Google Scholar

[10]

Flick, U. (1998). An introduction to qualitative research. Thousand Oaks, CA: Sage.

Google Scholar

[11]

Hosmer, D.W., & Lemeshow, S. (1989). Applied logistic regression. New York: Wiley.

Google Scholar

[12]

Ingwersen, P. (1992). Information retrieval interaction. London: Taylor Graham.

Crossref

Google Scholar

[13]

Ivory, M.Y., Sinha, R.R., & Hearst, M.A. (2001). Empirically validated Web page metrics. Proceedings of the ACM CHI Conference on Human Factors in Computing Systems 3, 53-60.

Crossref

Google Scholar

[14]

Janes, J.W. (1991). Relevance judgments and the incremental presentation of document representations. Information Processing & Management, 27(6), 629-646.

Crossref

Google Scholar

[15]

Jansen, B.J., Spink, A., & Saracevic, T. (2000). Real life, real users and real information needs: A study and analysis of user queries on the Web. Information Processing & Management, 36(2), 207-227.

Digital Library

Google Scholar

[16]

Kelly, D., Murdock, V., Yuan, X., Croft, W.B., & Belkin, N.J. (2002). Features of documents relevant to task and fact-oriented questions. Proceedings of the 11th ACM CIKM Conference (pp. 645-647).

Crossref

Google Scholar

[17]

Kim, K.S., & Allen, B. (2002). Cognitive and task influences on Web searching behavior. Journal of the American Society for Information Science and Technology, 53(2), 109-119.

Digital Library

Google Scholar

[18]

Lazonder, A.W., Biemans, H.J.A., & Wopereis, I.G.J.H. (2000). Differences between novice and expert users in searching information on the World Wide Web. Journal of the American Society for Information Science, 51(6), 576-581.

Digital Library

Google Scholar

[19]

Maglaughlin, K.L., & Sonnenwald, D.H. (2002). User perspectives on relevance criteria: A comparison among relevant, partially relevant, and not-relevant judgments. Journal of the American Society for Information Science and Technology, 53(5), 327-342.

Digital Library

Google Scholar

[20]

Mizzaro, S. (1997). Relevance: The whole history. Journal of the American Society for Information Science, 48(9), 810-832.

Crossref

Google Scholar

[21]

Preece, J. (1994). Human computer interaction. Wokingham, UK: Addison Wesley.

Crossref

Google Scholar

[22]

Saracevic, T. (1969). Comparative effects of titles, abstracts and full text on relevance judgments. Journal of the American Society for Information Science, 22, 126-139.

Crossref

Google Scholar

[23]

Saracevic, T. (1970). The concept of "relevance" in information science: A historical review. In T. Saracevic (Ed.), Introduction to information science (pp. 111-151). New York: R.R. Bowker.

Google Scholar

[24]

Schamber, L. (1991). Users' criteria for evaluation in a multimedia environment. Proceedings of the 54th Annual Meeting of the American Society for Information Science, 28, 126-133.

Google Scholar

[25]

Schamber, L. (1994). Relevance and information behavior. Annual Review of Information Science and Technology, 29, 3-48.

Google Scholar

[26]

Schamber, L., Eisenberg, M.B., & Nilan, M.S. (1990). A reexamination of relevance: Toward a dynamic, situational definition. Information Processing & Management, 26, 755-776.

Digital Library

Google Scholar

[27]

Search Engine Heaven. (2003). http://www.richeast.org/SEHeaven.html

Google Scholar

[28]

Tang, R., & Solomon, P. (1998). Toward an understanding of the dynamics of relevance judgment: An analysis of one person's search behavior. Information Processing & Management, 34(2/3), 237-256.

Digital Library

Google Scholar

[29]

Vakkari, P. (2000). Relevance and contributing information types of searched documents in task performance. In Proceedings of the 23rd ACM SIGIR Conference (pp. 2-9).

Crossref

Google Scholar

[30]

Vakkari, P., & Hakala, N. (2000). Changes in relevance criteria and problem stages in task performance. Journal of Documentation, 56(5), 540-562.

Crossref

Google Scholar

[31]

White, M.D., & Iivonen, M. (2001). Questions as a factor in Web search strategy. Information Processing & Management, 37, 721-740.

Crossref

Google Scholar

[32]

White, R.W., Jose, J.M., & Ruthven, I. (2002). The use of implicit evidence for relevance feedback in Web retrieval. In Proceedings of the 24th European Colloquium on IR Research (pp. 93-109).

Crossref

Google Scholar

[33]

White, R.W., Jose, J.M., & Ruthven, I. (2003). A task-oriented study on the influencing effects of query-biased summarisation in Web searching. Information Processing and Management, 39(5), 707-733.

Digital Library

Google Scholar

[34]

Woodruff, A., Aoki, P.M., Brewer, E., Gauthier, P., & Rowe, L.A. (1996). An investigation of documents from the World Wide Web. Proceedings of the Fifth International World Wide Web Conference.

Crossref

Google Scholar

[35]

Woodruff, A., Rosenholtz, R., Morrison, J.B., Faulring, A., & Pirolli, P. (2002). A comparison of the use of text summaries, plain thumbnails, and enhanced thumbnails for Web search tasks. Journal of the American Society for Information Science and Technology, 53(20), 172-185.

Digital Library

Google Scholar

[36]

WWW Digital Library. (2003). http://www.vlib.org/.

Google Scholar

[37]

Wynblatt, M., & Benson, D. (1998). Web page caricatures: Multimedia summaries for WWW documents. Proceedings of the IEEE International Conference on Multimedia Computing and Systems (pp. 194-199).

Digital Library

Google Scholar

[38]

Yahoo. (2003). http://www.yahoo.com/.

Google Scholar

[39]

Yuan, W. (1997). End-user searching behavior in information retrieval: A longitudinal study. Journal of the American Society for Information Science, 48(3), 218-234.

Digital Library

Google Scholar

[40]

ZDNET. (2003). http://www.zdnet.com/yil.

Google Scholar

[41]

Zhu, X., & Gauch, S. (2000). Incorporating quality metrics in centralized distributed information retrieval on the World Wide Web. Proceedings of the 23rd ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 288-295).

Crossref

Google Scholar

Cited By

View all

Caldiroli CGasparini FCorchs SMangiatordi AGarbo RAntonietti AMantovani F(2022)Comparing online cognitive load on mobile versus PC-based devicesPersonal and Ubiquitous Computing10.1007/s00779-022-01707-827:2(495-505)Online publication date: 28-Dec-2022
https://dl.acm.org/doi/10.1007/s00779-022-01707-8
Kiesel JMeyer LPotthast MStein B(2021)Meta-Information in Conversational SearchACM Transactions on Information Systems10.1145/346886839:4(1-44)Online publication date: 16-Aug-2021
https://dl.acm.org/doi/10.1145/3468868
Crescenzi ACapra RChoi BLi YScholer FThomas PElsweiler DJoho HKando NSmith C(2021)Adaptation in Information Search and Decision-Making under Time ConstraintsProceedings of the 2021 Conference on Human Information Interaction and Retrieval10.1145/3406522.3446030(95-105)Online publication date: 14-Mar-2021
https://dl.acm.org/doi/10.1145/3406522.3446030
Show More Cited By

Index Terms

How users assess web pages for information seeking
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Relevance assessment
    2. Users and interactive retrieval
      1. Personalization
  2. World Wide Web
    1. Web searching and information discovery
      1. Personalization

Recommendations

Older Adults' Usage of Web Pages: Investigating Effects of Information Structure on Performance
Proceedings, Part I, of the First International Conference on Human Aspects of IT for the Aged Population. Design for Aging - Volume 9193

This study focuses on older adults' usage of web pages. An experiment consisted of three information structures the net structure, the tree structure, and the linear structure was conducted to investigate effects of information structure IS on older ...
A field study characterizing Web-based information-seeking tasks

Previous studies have examined various aspects of user behavior on the Web, including general information-seeking patterns, search engine use, and revisitation habits. Little research has been conducted to study how users navigate and interact with ...
Investigating the Roles of Knowledge and Cognitive Abilities in Older Adult Information Seeking on the Web

This study investigated the influences of knowledge, particularly Internet, Web browser, and search engine knowledge, as well as cognitive abilities on older adult information seeking on the Internet. The emphasis on aspects of cognition was informed by ...

Reviews

Reviewer: Jaroslav Pokorny

This paper attempts to explore the criteria used by online searchers when assessing the relevance of Web pages for information seeking tasks. After a deep analysis of related work, the authors summarize the contributions of their paper: identifying the features that users employ during an information seeking process, examination of the effects of various tasks on the criteria employed by searchers, and a discussion of how the criteria evolved during the different stages of tasks. A remarkable fact is that the features were not predefined, as is common in similar research; instead, they were discovered during the tasks. The methodology used for the study involved 24 participants, and classical tools such as predefined tasks, questionnaires, and extracting information from search sessions. The obtained data was analyzed, and follow-up statistical processing was performed. Advanced methods, like Web mining of users' sessions, were not used. In the last two sections, the authors present and discuss their results. Although the results are not too surprising, and could be influenced by the limitations of the methodology used, they could be considered during the design of Web retrieval systems. Online Computing Reviews Service

Access critical reviews of Computing literature here

Become a reviewer for Computing Reviews.

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

Publisher

John Wiley & Sons, Inc.

United States

Publication History

Published: 15 February 2005

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

62
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Caldiroli CGasparini FCorchs SMangiatordi AGarbo RAntonietti AMantovani F(2022)Comparing online cognitive load on mobile versus PC-based devicesPersonal and Ubiquitous Computing10.1007/s00779-022-01707-827:2(495-505)Online publication date: 28-Dec-2022
https://dl.acm.org/doi/10.1007/s00779-022-01707-8
Kiesel JMeyer LPotthast MStein B(2021)Meta-Information in Conversational SearchACM Transactions on Information Systems10.1145/346886839:4(1-44)Online publication date: 16-Aug-2021
https://dl.acm.org/doi/10.1145/3468868
Crescenzi ACapra RChoi BLi YScholer FThomas PElsweiler DJoho HKando NSmith C(2021)Adaptation in Information Search and Decision-Making under Time ConstraintsProceedings of the 2021 Conference on Human Information Interaction and Retrieval10.1145/3406522.3446030(95-105)Online publication date: 14-Mar-2021
https://dl.acm.org/doi/10.1145/3406522.3446030
Sa NYuan X(2021)User Assessment of Webpage UsefulnessHuman-Computer Interaction. Design and User Experience Case Studies10.1007/978-3-030-78468-3_30(442-457)Online publication date: 24-Jul-2021
https://dl.acm.org/doi/10.1007/978-3-030-78468-3_30
Qu JArguello JWang YHuang JChang YCheng XKamps JMurdock VWen JLiu Y(2020)Towards Explainable Retrieval Models for Precision Medicine Literature SearchProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401277(1593-1596)Online publication date: 25-Jul-2020
https://dl.acm.org/doi/10.1145/3397271.3401277
Uprety SSong DSong DLiu TSun LBruza PMelucci MSebastiani FYang G(2018)Investigating Order Effects in Multidimensional Relevance Judgment using Query LogsProceedings of the 2018 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3234944.3234972(191-194)Online publication date: 10-Sep-2018
https://dl.acm.org/doi/10.1145/3234944.3234972
Uprety SSu YSong DLi JCollins-Thompson KMei QDavison BLiu YYilmaz E(2018)Modeling Multidimensional User Relevance in IR using Vector SpacesThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval10.1145/3209978.3210130(993-996)Online publication date: 27-Jun-2018
https://dl.acm.org/doi/10.1145/3209978.3210130
Arguello JChoi BCapra R(2018)Factors Influencing Users’ Information RequestsACM Transactions on Information Systems10.1145/320962436:4(1-37)Online publication date: 12-Jul-2018
https://dl.acm.org/doi/10.1145/3209624
Zhang GWang JZhou GLiu JWei CEmrouznejad AQian Z(2018)Scientific Data Relevance Criteria Classification and UsageProceedings of the 2nd International Conference on Computer Science and Application Engineering10.1145/3207677.3278010(1-7)Online publication date: 22-Oct-2018
https://dl.acm.org/doi/10.1145/3207677.3278010
Brennan KKelly DShah CBelkin NByström KHuang JScholer F(2018)Strategies for Finding and Evaluating Information about Personal Finance TopicsProceedings of the 2018 Conference on Human Information Interaction & Retrieval10.1145/3176349.3176883(229-232)Online publication date: 1-Mar-2018
https://dl.acm.org/doi/10.1145/3176349.3176883
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Index Terms

Recommendations

Older Adults' Usage of Web Pages: Investigating Effects of Information Structure on Performance

A field study characterizing Web-based information-seeking tasks

Investigating the Roles of Knowledge and Cognitive Abilities in Older Adult Information Seeking on the Web

Reviews

Access critical reviews of Computing literature here

Comments

Information

Published In

Publisher

Publication History

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

Get Access

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations