research-article

Voyant: generating structured feedback on visual designs using a crowd of non-experts

Authors:

Shih-Wen Huang,

Brian BaileyAuthors Info & Claims

CSCW '14: Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing

Pages 1433 - 1444

https://doi.org/10.1145/2531602.2531604

Published: 15 February 2014 Publication History

Abstract

Feedback on designs is critical for helping users iterate toward effective solutions. This paper presents Voyant, a novel system giving users access to a non-expert crowd to receive perception-oriented feedback on their designs from a selected audience. Based on a formative study, the system generates the elements seen in a design, the order in which elements are noticed, impressions formed when the design is first viewed, and interpretation of the design relative to guidelines in the domain and the user's stated goals. An evaluation of the system was conducted with users and their designs. Users reported the feedback about impressions and interpretation of their goals was most helpful, though the other feedback types were also valued. Users found the coordinated views in Voyant useful for analyzing relations between the crowd's perception of a design and the visual elements within it. The cost of generating the feedback was considered a reasonable tradeoff for not having to organize critiques or interrupt peers.

Supplementary Material

suppl.mov (cscw0105-file3.mp4)

Supplemental video

Download
63.67 MB

References

[1]

Baldonado, M.Q.W., Woodruff, A., & Kuchinsky, A., Guidelines for Using Multiple Views in Information Visualization. In AVI, (2000), 110--119.

Digital Library

[2]

Bernstein, M.S., Brandt, J., Miller, R.C., & Karger, D.R., Crowds in Two Seconds: Enabling Realtime Crowd-Powered Interfaces. In UIST, (2011), 33--42.

Digital Library

[3]

Bernstein, M.S., Little, G., Miller, R.C., Hartmann, B., Ackerman, M.S., Karger, D.R., Crowell, D., et al., Soylent: A Word Processor with a Crowd Inside. In UIST, (2010), 313--322.

Digital Library

[4]

Cross, N. Designerly Ways of Knowing. Design Studies, 3, 5, (1982), 221--227.

[5]

Dannels, D.P. and Martin, K.N. Critiquing Critiques: A Genre Analysis of Feedback Across Novice to Expert Design Studios. Journal of Business and Technical Communication, 22, 2, (2008), 135--159.

[6]

Diehl, M. and Stroebe, W. Productivity Loss in Brainstorming Groups: Towards the Solution of a Riddle. Journal of Personality and Social Psychology, 53, 3, (1987), 497--509.

[7]

Dow, S.P., Fortuna, J., Schwartz, D., Altringer, B., & Klemmer, S., Prototyping Dynamics: Sharing Multiple Designs Improves Exploration, Group Rapport, and Results. In CHI, (2011), 2807--2816.

Digital Library

[8]

Dow, S.P., Gerber, E., & Wong, A., A Preliminary Study of Using Crowds in the Classroom. In CHI, (2013), 227--236.

Digital Library

[9]

Dow, S.P., Heddleston, K., & Klemmer, S.R., The Efficacy of Prototyping Under Time Constraints. In Creativity & Cognition, (2009), 165--174.

Digital Library

[10]

Dow, S.P., Kulkarni, A., Klemmer, S., & Hartmann, B., Shepherding the Crowd Yields Better Work. In CSCW, (2012), 1013--1022.

Digital Library

[11]

Dunne, C., Riche, N.H., Lee, B., Metoyer, R.A., & Robertson, G.G., GraphTrail: Analyzing Large Multivariate, Heterogeneous Networks while Supporting Exploration History. In CHI, (2012), 1663--1672.

Digital Library

[12]

Elkins, J. Art Critiques: A Guide. New Academia Publishing, Washington DC, 2012.

[13]

Feldman, E.B. The Teacher as Model Critic. Journal of Aesthetic Education, 7, 1, (1973), 50--57.

[14]

Feldman, E.B. Varieties of Visual Experience: Art as Image and Idea. Prentice-Hall, Englewood Cliffs, N.J., 1981.

[15]

Fischer, G., Nakakoji, K., Ostwald, J., Stahl, G., & Sumner, T., Embedding Computer-Based Critics in the Contexts of Design. In CHI, (1993), 157--164.

Digital Library

[16]

Fogarty, J., Hudson, S.E., Atkeson, C.G., Avrahami, D., Forlizzi, J., Kiesler, S., Lee, J.C., et al. Predicting Human Interruptibility with Sensors. TOCHI, 12, 1, (2005), 119--146.

Digital Library

[17]

Heer, J., Viégas, F.B., & Wattenberg, M., Voyagers and Voyeurs: Supporting Asynchronous Collaborative Information Visualization. In CHI, (2007), 1029--1038.

Digital Library

[18]

Hundhausen, C.D., Fairbrother, D., & Petre, M. An Empirical Study of the "Prototype Walkthrough": A Studio-Based Activity for HCI Education. TOCHI, 19, 4, (2012), 1--36.

Digital Library

[19]

Kittur, A., Chi, E.H., & Suh, B., Crowdsourcing User Studies With Mechanical Turk. In CHI, (2008), 453--456.

Digital Library

[20]

Kittur, A., Nickerson, J.V., Bernstein, M.S., Gerber, E.M., Shaw, A., Zimmerman, J., Lease, M., et al., The Future of Crowd Work. In CSCW, (2013), 1301--1318.

Digital Library

[21]

Komarov, S., Reinecke, K., & Gajos, K.Z., Crowdsourcing Performance Evaluations of User Interfaces. In CHI, (2013), 207--216.

Digital Library

[22]

Kress, G.R. and Leeuwen, T.v. Reading Images: the Grammar of Visual Design. Routledge, 1996.

[23]

Kulkarni, C. and Klemmer, S. Learning Design Wisdom by Augmenting Physical Studio Critique With Online Self-Assessment, Stanford University technical report, 2012.

[24]

Nijstad, B.A. and Stroebe, W. How the Group Affects the Mind: A Cognitive Model of Idea Generation in Groups. Personality and Social Psychology Review, 10, 3, (2006), 186--213.

[25]

Park, C.H., Son, K., Lee, J.H., & Bae, S.-H., Crowd vs. Crowd: Large-Scale Cooperative Design through Open Team Competition. In CSCW, (2013), 1275--1284.

Digital Library

[26]

Paulus, P. and Yang, H.-C. Idea Generation in Groups: A Basis for Creativity in Organizations. Organizational Behavior and Human Decision Processes, 82, 1, (2000), 76--87.

[27]

Reinecke, K., Yeh, T., Miratrix, L., Mardiko, R., Zhao, Y., Liu, J., & Gajos, K.Z., Predicting Users' First Impressions of Website Aesthetics With a Quantification of Perceived Visual Complexity and Colorfulness. In CHI, (2013), 2049--2058.

Digital Library

[28]

Rosenholtz, R., Twarog, N.R., Schinkel-Bielefeld, N., & Wattenberg, M., An Intuitive Model of Perceptual Grouping for HCI Design. In CHI, (2009), 1331--1340.

Digital Library

[29]

Schön, D.A. Designing as Reflective Conversation with the Materials of a Design Situation. Knowledge-Based Systems, 5, 1, (1992), 3--14.

Digital Library

[30]

Strauss, A.L. Qualitative Analysis for Social Scientists. Cambridge University Press, 1987.

[31]

Tohidi, M., Buxton, W., Baecker, R., & Sellen, A., Getting the Right Design and the Design Right. In CHI, (2006), 1243--1252.

Digital Library

[32]

Wang, H.C., Fussell, S.R., & Cosley, D., From Diversity to Creativity: Stimulating Group Brainstorming with Cultural Differences and Conversationally-Retrieved Pictures. In CSCW, (2011), 265--274.

Digital Library

[33]

Willett, W., Heer, J., & Agrawala, M., Strategies for Crowdsourcing Social Data Analysis. In CHI, (2012), 227--236.

Digital Library

[34]

Williams, R. The Non-designer's Design Book. Peachpit Press, 2008.

Digital Library

[35]

Xu, A. and Bailey, B.P., A Crowdsourcing Model for Receiving Design Critique. In CHI Extended Abstracts, (2011), 1183--1188.

Digital Library

[36]

Xu, A. and Bailey, B.P., A Reference-Based Scoring Model for Increasing the Findability of Promising Ideas in Innovation Pipelines. In CSCW, (2012), 1183--1186.

Digital Library

[37]

Xu, A. and Bailey, B.P., What Do You Think? A Case Study of Benefit, Expectation, and Interaction in a Large Online Critique Community. In CSCW, (2012), 295--304.

Digital Library

[38]

Yu, L. and Nickerson, J.V., Cooks or Cobblers? Crowd Creativity through Combination. In CHI, (2011), 1393--1402.

Digital Library

[39]

Zheng, X.S., Chakraborty, I., Lin, J.J., & Rauschenberger, R., Correlating Low-level Image Statistics with Users' Rapid Aesthetic and Affective Judgments of Web Pages. In CHI, (2009), 1--10.

Digital Library

[40]

Does This JC Penney Tea Kettle Look Like Hitler?, NBC's Today.com, http://www.today.com/news/does-j-c-penney-tea-kettle-look-hitler-6C10100642, Retrieved on May 28 2013.

[41]

Core77. http://www.core77.com.

[42]

Dribbble. http://dribbble.com.

[43]

Feedbackarmy. http://www.feedbackarmy.com.

[44]

Fivesecondtest. http://fivesecondtest.com.

[45]

Usabilla. http://www.usabilla.com.

Cited By

Wang DEng JTurpitka MEpstein D(2024)Exploring Activity-Sharing Response Differences Between Broad-Purpose and Dedicated Online Social PlatformsProceedings of the ACM on Human-Computer Interaction10.1145/36868988:CSCW2(1-37)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686898
Huh MPavel A(2024)DesignChecker: Visual Design Support for Blind and Low Vision Web DevelopersProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676369(1-19)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676369
Park JKo EPark YYim JKim J(2024)DynamicLabels: Supporting Informed Construction of Machine Learning Label Sets with Crowd FeedbackProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645157(209-228)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645157
Show More Cited By

Index Terms

Voyant: generating structured feedback on visual designs using a crowd of non-experts
1. Human-centered computing
  1. Collaborative and social computing

Recommendations

Structuring, Aggregating, and Evaluating Crowdsourced Design Critique
CSCW '15: Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing

Feedback is an important component of the design process, but gaining access to high-quality critique outside a classroom or firm is challenging. We present CrowdCrit, a web-based system that allows designers to receive design critiques from non-expert ...
Social Network, Web Forum, or Task Market?: Comparing Different Crowd Genres for Design Feedback Exchange
DIS '16: Proceedings of the 2016 ACM Conference on Designing Interactive Systems

Increasingly, designers seek feedback on their designs from crowd platforms such as social networks, Web forums, and paid task markets which demand different amounts of social capital, financial resources, and time. Yet it is unknown how the choice of ...
Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques
CSCW '16: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing

Expert feedback is valuable but hard to obtain for many designers. Online crowds can provide fast and affordable feedback, but workers may lack relevant domain knowledge and experience. Can expert rubrics address this issue and help novices provide ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CSCW '14: Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing

February 2014

1600 pages

ISBN:9781450325400

DOI:10.1145/2531602

General Chairs:
Susan Fussell
Cornell University
,
Wayne Lutters
University of Maryland, Baltimore County
,
Program Chairs:
Meredith Ringel Morris
Microsoft Research
,
Madhu Reddy
Penn State University

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 February 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CSCW'14

Sponsor:

SIGCHI

CSCW'14: Computer Supported Cooperative Work

February 15 - 19, 2014

Maryland, Baltimore, USA

Acceptance Rates

CSCW '14 Paper Acceptance Rate 134 of 497 submissions, 27%;

Overall Acceptance Rate 2,235 of 8,521 submissions, 26%

Upcoming Conference

CSCW '25

Sponsor:
sigchi

Computer-Supported Cooperative Work and Social Computing

October 18 - 22, 2025

Bergen , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

128
Total Citations
View Citations
1,852
Total Downloads

Downloads (Last 12 months)106
Downloads (Last 6 weeks)18

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang DEng JTurpitka MEpstein D(2024)Exploring Activity-Sharing Response Differences Between Broad-Purpose and Dedicated Online Social PlatformsProceedings of the ACM on Human-Computer Interaction10.1145/36868988:CSCW2(1-37)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686898
Huh MPavel A(2024)DesignChecker: Visual Design Support for Blind and Low Vision Web DevelopersProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676369(1-19)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676369
Park JKo EPark YYim JKim J(2024)DynamicLabels: Supporting Informed Construction of Machine Learning Label Sets with Crowd FeedbackProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645157(209-228)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645157
E JYen YPan ILin GLi MJin HChen MXia HDow S(2024)When to Give Feedback: Exploring Tradeoffs in the Timing of Design FeedbackProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656183(292-310)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3635636.3656183
Nouraei FSiu ARossi RLipka N(2024)Thinking Outside the Box: Non-Designer Perspectives and Recommendations for Template-Based Graphic Design ToolsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650967(1-9)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650967
Son KChoi DKim TKim J(2024)Demystifying Tacit Knowledge in Graphic Design: Characteristics, Instances, Approaches, and GuidelinesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642886(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642886
Hou YYang MCui HWang LXu JZeng W(2024)C2Ideas: Supporting Creative Interior Color Design Ideation with a Large Language ModelProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642224(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642224
Jang SJeong YJung HNam T(2024)Coherent visual design through attribute-specific feedback: a hybrid approach to intelligent design agentsDigital Creativity10.1080/14626268.2023.230135534:4(311-330)Online publication date: 7-Jan-2024
https://doi.org/10.1080/14626268.2023.2301355
Messerschmidt MCortes JNanayakkara S(2024)CoTacs: A Haptic Toolkit to Explore Effective On-Body Haptic Feedback by Ideating, Designing, Evaluating and Refining Haptic Designs Using Group CollaborationInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2358460(1-21)Online publication date: 7-Jun-2024
https://doi.org/10.1080/10447318.2024.2358460
Lee LYau YHui P(2024)Perceived User Reachability in Mobile UIs Using Data Analytics and Machine LearningInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2327199(1-24)Online publication date: 25-Mar-2024
https://doi.org/10.1080/10447318.2024.2327199
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents