Nothing Special   »   [go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/a/inm/orisre/v32y2021i3p688-712.html
   My bibliography  Save this article

Dynamic, Multidimensional, and Skillset-Specific Reputation Systems for Online Work

Author

Listed:
  • Marios Kokkodis

    (Carroll School of Management, Boston College, Chestnut Hill, Massachusetts 02467)

Abstract
Reputation systems in digital workplaces increase transaction efficiency by building trust and reducing information asymmetry. These systems, however, do not yet capture the dynamic multidimensional nature of online work. By uniformly aggregating reputation scores across worker skills, they ignore skillset-specific heterogeneity (reputation attribution), and they implicitly assume that a worker’s quality does not change over time (reputation staticity). Even further, reputation scores tend to be overly positive (reputation inflation), and, as a result, they often fail to differentiate workers efficiently. This work presents a new augmented intelligence reputation framework that combines human input with machine learning to provide dynamic, multidimensional, and skillset-specific worker reputation. The framework includes three components: The first component maps skillsets into a latent space of finite competency dimensions (word embedding), and, as a result, it directly addresses reputation attribution. The second builds dynamic competency-specific quality assessment models (hidden Markov models) that solve reputation staticity. The final component aggregates these competency-specific assessments to generate skillset-specific reputation scores. Application of this framework on a data set of 58,459 completed tasks from a major online labor market shows that, compared to alternative reputation systems, the proposed approach (1) yields more appropriate rankings of workers that form a closer-to-normal reputation distribution, (2) better identifies “nonperfect” workers who are more likely to underperform and are harder to predict, and (3) improves the ranking of within-opening choices and yields significantly better outcomes. Additional analysis of 77,044 restaurant reviews shows that the proposed framework successfully generalizes to alternative contexts, where assigned feedback scores are overly positive and service quality is multidimensional and dynamic.

Suggested Citation

  • Marios Kokkodis, 2021. "Dynamic, Multidimensional, and Skillset-Specific Reputation Systems for Online Work," Information Systems Research, INFORMS, vol. 32(3), pages 688-712, September.
    • Handle: RePEc:inm:orisre:v:32:y:2021:i:3:p:688-712
    DOI: 10.1287/isre.2020.0972
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/isre.2020.0972
    Download Restriction: no
    File URL: https://libkey.io/10.1287/isre.2020.0972?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Liran Einav & Chiara Farronato & Jonathan Levin, 2016. "Peer-to-Peer Markets," Annual Review of Economics, Annual Reviews, vol. 8(1), pages 615-635, October.
    2. Mark Graham & Isis Hjorth & Vili Lehdonvirta, 2017. "Digital labour and development: impacts of global digital labour platforms and the gig economy on worker livelihoods," Transfer: European Review of Labour and Research, , vol. 23(2), pages 135-162, May.
    3. Anindya Ghose & Panagiotis G. Ipeirotis & Beibei Li, 2014. "Examining the Impact of Ranking on Consumer Behavior and Search Engine Revenue," Management Science, INFORMS, vol. 60(7), pages 1632-1654, July.
    4. Peter Kuhn & Mikal Skuterud, 2004. "Internet Job Search and Unemployment Durations," American Economic Review, American Economic Association, vol. 94(1), pages 218-232, March.
    5. Ajay K. Agrawal & Nicola Lacetera & Elizabeth Lyons, 2013. "Does Information Help or Hinder Job Applicants from Less Developed Countries in Online Markets?," NBER Working Papers 18720, National Bureau of Economic Research, Inc.
    6. Sundararajan, Arun, 2016. "The Sharing Economy: The End of Employment and the Rise of Crowd-Based Capitalism," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262034573, April.
    7. Guest Editors: Hemant Jain & Balaji Padmanabhan & Paul A. Pavlou & Raghu T. Santanam, 2018. "all for Papers—Special Issue of Information Systems Research —Humans, Algorithms, and Augmented Intelligence: The Future of Work, Organizations, and Society," Information Systems Research, INFORMS, vol. 29(1), pages 250-251, March.
    8. Davide Proserpio & Georgios Zervas, 2017. "Online Reputation Management: Estimating the Impact of Management Responses on Consumer Reviews," Marketing Science, INFORMS, vol. 36(5), pages 645-665, September.
    9. Irfan Kanat & Yili Hong & T. S. Raghu, 2018. "Surviving in Global Online Labor Markets for IT Services: A Geo-Economic Analysis," Information Systems Research, INFORMS, vol. 29(4), pages 893-909, December.
    10. Wendy W. Moe & David A. Schweidel, 2012. "Online Product Opinions: Incidence, Evaluation, and Evolution," Marketing Science, INFORMS, vol. 31(3), pages 372-386, May.
    11. Dellarocas, Chrysanthos, 2003. "The Digitization of Word-of-mouth: Promise and Challenges of Online Feedback Mechanisms," Working papers 4296-03, Massachusetts Institute of Technology (MIT), Sloan School of Management.
    12. David H. Autor, 2001. "Wiring the Labor Market," Journal of Economic Perspectives, American Economic Association, vol. 15(1), pages 25-40, Winter.
    13. Paul A. Pavlou & David Gefen, 2004. "Building Effective Online Marketplaces with Institution-Based Trust," Information Systems Research, INFORMS, vol. 15(1), pages 37-59, March.
    14. Xianghua Lu & Sulin Ba & Lihua Huang & Yue Feng, 2013. "Promotional Marketing or Word-of-Mouth? Evidence from Online Restaurant Reviews," Information Systems Research, INFORMS, vol. 24(3), pages 596-612, September.
    15. Chrysanthos Dellarocas, 2003. "The Digitization of Word of Mouth: Promise and Challenges of Online Feedback Mechanisms," Management Science, INFORMS, vol. 49(10), pages 1407-1424, October.
    16. Hema Yoganarasimhan, 2013. "The Value of Reputation in an Online Freelance Marketplace," Marketing Science, INFORMS, vol. 32(6), pages 860-891, November.
    17. Marios Kokkodis & Panagiotis G. Ipeirotis, 2016. "Reputation Transferability in Online Labor Markets," Management Science, INFORMS, vol. 62(6), pages 1687-1706, June.
    18. John J. Horton, 2017. "The Effects of Algorithmic Labor Market Recommendations: Evidence from a Field Experiment," Journal of Labor Economics, University of Chicago Press, vol. 35(2), pages 345-385.
    19. Anindya Ghose & Panagiotis G. Ipeirotis & Beibei Li, 2012. "Designing Ranking Systems for Hotels on Travel Search Engines by Mining User-Generated and Crowdsourced Content," Marketing Science, INFORMS, vol. 31(3), pages 493-520, May.
    20. Antonio Moreno & Christian Terwiesch, 2014. "Doing Business with Strangers: Reputation in Online Service Marketplaces," Information Systems Research, INFORMS, vol. 25(4), pages 865-886, December.
    21. Kinshuk Jerath & Peter S. Fader & Bruce G. S. Hardie, 2011. "New Perspectives on Customer "Death" Using a Generalization of the Pareto/NBD Model," Marketing Science, INFORMS, vol. 30(5), pages 866-880, September.
    22. Michael Luca, 2017. "Designing Online Marketplaces: Trust and Reputation Mechanisms," Innovation Policy and the Economy, University of Chicago Press, vol. 17(1), pages 77-93.
    23. J. Yannis Bakos, 1997. "Reducing Buyer Search Costs: Implications for Electronic Marketplaces," Management Science, INFORMS, vol. 43(12), pages 1676-1692, December.
    24. Rajiv D. Banker & Iny Hwang, 2008. "Importance of Measures of Past Performance: Empirical Evidence on Quality of e†Service Providers," Contemporary Accounting Research, John Wiley & Sons, vol. 25(2), pages 307-337, June.
    25. David H. Autor & Lawrence F. Katz & Alan B. Krueger, 1998. "Computing Inequality: Have Computers Changed the Labor Market?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 113(4), pages 1169-1213.
    26. Marios Kokkodis & Theodoros Lappas, 2020. "Your Hometown Matters: Popularity-Difference Bias in Online Reputation Platforms," Information Systems Research, INFORMS, vol. 31(2), pages 412-430, June.
    27. Yi-Chun (Chad) Ho & Junjie Wu & Yong Tan, 2017. "Disconfirmation Effect on Online Rating Behavior: A Structural Model," Information Systems Research, INFORMS, vol. 28(3), pages 626-642, September.
    28. Steven Tadelis, 2016. "Reputation and Feedback Systems in Online Platform Markets," Annual Review of Economics, Annual Reviews, vol. 8(1), pages 321-340, October.
    29. Amanda Pallais, 2014. "Inefficient Hiring in Entry-Level Labor Markets," American Economic Review, American Economic Association, vol. 104(11), pages 3565-3599, November.
    30. Michael Luca & Georgios Zervas, 2016. "Fake It Till You Make It: Reputation, Competition, and Yelp Review Fraud," Management Science, INFORMS, vol. 62(12), pages 3412-3427, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Hemant Jain & Balaji Padmanabhan & Paul A. Pavlou & T. S. Raghu, 2021. "Editorial for the Special Section on Humans, Algorithms, and Augmented Intelligence: The Future of Work, Organizations, and Society," Information Systems Research, INFORMS, vol. 32(3), pages 675-687, September.
    2. Marios Kokkodis, 2023. "Adjusting Skillset Cohesion in Online Labor Markets: Reputation Gains and Opportunity Losses," Information Systems Research, INFORMS, vol. 34(3), pages 1245-1258, September.
    3. Rense Corten & Judith Kas & Timm Teubner & Martijn Arets, 2023. "The role of contextual and contentual signals for online trust: Evidence from a crowd work experiment," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-17, December.
    4. Lingfeng Dong & Ting Ji & Jie Zhang, 2022. "Effects of Conversation Politeness on Hiring Decision in Online Labor Markets: An Inverted U-Shaped Relationship Exploration," Sustainability, MDPI, vol. 14(22), pages 1-11, November.
    5. Mirko Duradoni & Stefania Collodi & Serena Coppolino Perfumi & Andrea Guazzini, 2021. "Reviewing Stranger on the Internet: The Role of Identifiability through “Reputation” in Online Decision Making," Future Internet, MDPI, vol. 13(5), pages 1-12, April.
    6. Marios Kokkodis & Sam Ransbotham, 2023. "Learning to Successfully Hire in Online Labor Markets," Management Science, INFORMS, vol. 69(3), pages 1597-1614, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dominik Gutt & Jürgen Neumann & Steffen Zimmermann & Dennis Kundisch & Jianqing Chen, 2018. "Design of Review Systems - A Strategic Instrument to shape Online Review Behavior and Economic Outcomes," Working Papers Dissertations 42, Paderborn University, Faculty of Business Administration and Economics.
    2. Marios Kokkodis, 2023. "Adjusting Skillset Cohesion in Online Labor Markets: Reputation Gains and Opportunity Losses," Information Systems Research, INFORMS, vol. 34(3), pages 1245-1258, September.
    3. Jason Chan & Jing Wang, 2018. "Hiring Preferences in Online Labor Markets: Evidence of a Female Hiring Bias," Management Science, INFORMS, vol. 64(7), pages 2973-2994, July.
    4. Apostolos Filippas & John J. Horton & Richard J. Zeckhauser, 2020. "Owning, Using, and Renting: Some Simple Economics of the “Sharing Economy”," Management Science, INFORMS, vol. 66(9), pages 4152-4172, September.
    5. Marios Kokkodis & Theodoros Lappas, 2020. "Your Hometown Matters: Popularity-Difference Bias in Online Reputation Platforms," Information Systems Research, INFORMS, vol. 31(2), pages 412-430, June.
    6. Marios Kokkodis & Panagiotis G. Ipeirotis, 2021. "Demand-Aware Career Path Recommendations: A Reinforcement Learning Approach," Management Science, INFORMS, vol. 67(7), pages 4362-4383, July.
    7. Yili Hong & Paul A. Pavlou, 2017. "On Buyer Selection of Service Providers in Online Outsourcing Platforms for IT Services," Information Systems Research, INFORMS, vol. 28(3), pages 547-562, September.
    8. Marios Kokkodis & Sam Ransbotham, 2023. "Learning to Successfully Hire in Online Labor Markets," Management Science, INFORMS, vol. 69(3), pages 1597-1614, March.
    9. Marios Kokkodis & Theodoros Lappas & Gerald C. Kane, 2022. "Optional purchase verification in e‐commerce platforms: More representative product ratings and higher quality reviews," Production and Operations Management, Production and Operations Management Society, vol. 31(7), pages 2943-2961, July.
    10. Hui, Xiang & Klein, Tobias & Stahl, Konrad, 2022. "Learning from Online Ratings," CEPR Discussion Papers 17006, C.E.P.R. Discussion Papers.
    11. Yili Hong & Jing Peng & Gordon Burtch & Ni Huang, 2021. "Just DM Me (Politely): Direct Messaging, Politeness, and Hiring Outcomes in Online Labor Markets," Information Systems Research, INFORMS, vol. 32(3), pages 786-800, September.
    12. Kazakova, E. & Sandomirskaia, M. & Suvorov, A. & Khazhgerieva, A. & Shavshin, R., 2023. "Platforms, online labor markets, and crowdsourcing. Part 1. Traditional online labor market," Journal of the New Economic Association, New Economic Association, vol. 60(3), pages 120-148.
    13. Jürgen Neumann, 2021. "When Biased Ratings Benefit the Consumer - An Economic Analysis of Online Ratings in Markets with Variety-Seeking Consumers," Working Papers Dissertations 77, Paderborn University, Faculty of Business Administration and Economics.
    14. Christoph Schneider & Markus Weinmann & Peter N.C. Mohr & Jan vom Brocke, 2021. "When the Stars Shine Too Bright: The Influence of Multidimensional Ratings on Online Consumer Ratings," Management Science, INFORMS, vol. 67(6), pages 3871-3898, June.
    15. Gordon Burtch & Yili Hong & Senthil Kumar, 2021. "When Does Dispute Resolution Substitute for a Reputation System? Empirical Evidence from a Service Procurement Platform," Production and Operations Management, Production and Operations Management Society, vol. 30(6), pages 1565-1582, June.
    16. Cristiano Codagnone & Fabienne Abadie & Federico Biagi, 2016. "The Future of Work in the ‘Sharing Economy’. Market Efficiency and Equitable Opportunities or Unfair Precarisation?," JRC Research Reports JRC101280, Joint Research Centre.
    17. Lingfang (Ivy) Li & Steven Tadelis & Xiaolan Zhou, 2020. "Buying reputation as a signal of quality: Evidence from an online marketplace," RAND Journal of Economics, RAND Corporation, vol. 51(4), pages 965-988, December.
    18. Ana Babić Rosario & Kristine Valck & Francesca Sotgiu, 2020. "Conceptualizing the electronic word-of-mouth process: What we know and need to know about eWOM creation, exposure, and evaluation," Journal of the Academy of Marketing Science, Springer, vol. 48(3), pages 422-448, May.
    19. Xiang Hui & Tobias J. Klein & Konrad Stahl, 2021. "When and Why Do Buyers Rate in Online Markets?," CRC TR 224 Discussion Paper Series crctr224_2021_267v1, University of Bonn and University of Mannheim, Germany.
    20. T. Ravichandran & Chaoqun Deng, 2023. "Effects of Managerial Response to Negative Reviews on Future Review Valence and Complaints," Information Systems Research, INFORMS, vol. 34(1), pages 319-341, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:orisre:v:32:y:2021:i:3:p:688-712. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.