article

Expecting the unexpected: effects of data collection design choices on the quality of crowdsourced user-generated content

Authors:

Roman Lukyanenko,

Jeffrey Parsons,

Yolanda F. Wiersma,

Mahed MaddahAuthors Info & Claims

MIS Quarterly, Volume 43, Issue 2

Pages 623 - 648

https://doi.org/10.25300/MISQ/2019/14439

Published: 01 June 2019 Publication History

Abstract

As crowdsourced user-generated content becomes an important source of data for organizations, a pressing question is how to ensure that data contributed by ordinary people outside of traditional organizational boundaries is of suitable quality to be useful for both known and unanticipated purposes. This research examines the impact of different information quality management strategies, and corresponding data collection design choices, on key dimensions of information quality in crowdsourced user-generated content. We conceptualize a contributor-centric information quality management approach focusing on instance-based data collection. We contrast it with the traditional consumer-centric fitness-for- use conceptualization of inf or ma tio n quality that emphasizes class-based data collection. We present laboratory and field experiments conducted in a citizen science domain that demonstrate trade-offs between the quality dimensions of accuracy, completeness (including discoveries), and precision between the two information management approaches and their corresponding data collection designs. Specifically, we show that instance-based data collection results in higher accuracy, dataset completeness, and number of discoveries, but this comes at the expense of lower precision. We further validate the practical value of the instance-based approach by conducting an applicability check with potential data consumers (scientists, in our context of citizen science). In a follow-up study, we show, using human experts and supervised machine learning techniques, that substantial precision gains on instance-based data can be achieved with post-processing. We conclude by discussing the benefits and limitations of different information quality and data collection design choices for information quality in crowdsourced user-generated content.

References

[1]

Abbasi, A., Zhou, Y., Deng, S., and Zhang, P. 2018. "Text Analytics to Support Sense-Making in Social Media: A Language-Action Perspective," MIS Quarterly (42:2), pp. 1-38.

[2]

Allen, G., and Parsons, J. 2010. "Is Query Reuse Potentially Harmful? Anchoring and Adjustment in Adapting Existing Database Queries," Information Systems Research (21:1), pp. 56-77.

Digital Library

[3]

Andriole, S. J. 2010. "Business Impact of Web 2.0 Technologies," Communications of the ACM (53:12), pp. 67-79.

Digital Library

[4]

Batini, C., Cappiello, C., Francalanci, C., and Maurino, A. 2009. "Methodologies for Data Quality Assessment and Improvement," Computing Surveys (41:3), pp. 1-52.

Digital Library

[5]

Bell, B. F. 1981. "When Is an Animal, Not an Animal?," Journal of Biological Education (15:3), pp. 213-218.

[6]

Berlin, B., Breedlove, D. E., and Raven, P. H. 1966. "Folk Taxonomies and Biological Classification," Science (154:3746), pp. 273-275.

[7]

Boakes, E. H., McGowan, P. J., Fuller, R. A., Chang-Qing, D., Clark, N. E., O'Connor, K., and Mace, G. M. 2010. "Distorted Views of Biodiversity: Spatial and Temporal Bias in Species Occurrence Data," PLoS Biology (8:6), pp. 1-11.

[8]

Bonney, R., Cooper, C. B., Dickinson, J., Kelling, S., Phillips, T., Rosenberg, K. V., and Shirk, J. 2009. "Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy," BioScience (59:11), pp. 977-984.

[9]

Brabham, D. C. 2013. Crowdsourcing, Cambridge, MA: MIT Press.

Digital Library

[10]

Browne, G. J., and Ramesh, V. 2002. "Improving Information Requirements Determination: A Cognitive Perspective," Information & Management (39:8), pp. 625-645.

Digital Library

[11]

Brynjolfsson, E., Geva, T., and Reichman, S. 2016. "Crowd-Squared: Amplifying the Predictive Power of Search Trend Data," MIS Quarterly (40:4), pp. 941-961.

Digital Library

[12]

Bunge, M. 1977. Treatise on Basic Philosophy: Ontology I: The Furniture of the World, Boston: Reidel.

[13]

Burgess, H., DeBey, L., Froehlich, H., Schmidt, N., Theobald, E., Ettinger, A., HilleRisLambers, J., Tewksbury, J., and Parrish, J. 2017. "The Science of Citizen Science: Exploring Barriers to Use as a Primary Research Tool," Biological Conservation (108), pp. 113-120.

[14]

Burton-Jones, A., and Volkoff, O. 2017. "How Can We Develop Contextualized Theories of Effective Use? A Demonstration in the Context of Community-Care Electronic Health Records," Information Systems Research (28:3), pp. 468-489.

Digital Library

[15]

Callaghan, C. T., and Gawlik, D. E. 2015. "Efficacy of EBird Data as an Aid in Conservation Planning and Monitoring," Journal of Field Ornithology (1:1), pp. 1-7.

[16]

Chen, P. 1976. "The Entity-Relationship Model—Toward a Unified View of Data," ACM Transactions on Database Systems (1:1), pp. 9-36.

Digital Library

[17]

Chittilappilly, A. I., Chen, L., and Amer-Yahia, S. 2016. "A Survey of General-Purpose Crowdsourcing Techniques," IEEE Transactions on Knowledge and Data Engineering (28:9), pp. 2246-2266.

Digital Library

[18]

Clarke, R., Burton-Jones, A., and Weber, R. 2016. "On the Onto-logical Quality and Logical Quality of Conceptual-Modeling Grammars: The Need for a Dual Perspective," Information Systems Research (27:2), pp. 365-382.

[19]

Clery, D. 2011. "Galaxy Zoo Volunteers Share Pain and Glory of Research," Science (333:6039), pp. 173-175.

[20]

Clow, D., and Makriyannis, E. 2011. ISpot Analysed: Participatory Learning and Reputation, in Proceedings of the 1^st International Conference on Learning Analytics and Knowledge, New York: ACM Press, pp. 34-43.

Digital Library

[21]

Cooper, A. K., Rapant, P., Hjelmager, J., Laurent, D., Iwaniak, A., Coetzee, S., Moellering, H., and Düren, U. 2011. "Extending the Formal Model of a Spatial Data Infrastructure to Include Volunteered Geographical Information," in Proceedings of the 25^th International Cartographic Conference, Paris, July 3-8, Paper ID CO-120.

[22]

Culnan, M. J., McHugh, P. J., and Zubillaga, J. I. 2010. "How Large U.S. Companies Can Use Twitter and Other Social Media to Gain Business Value.," MIS Quarterly Executive (9:4), pp. 243-259.

[23]

Daniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., and Allahbakhsh, M. 2018. "Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques, and Assurance Actions," ACM Computing Surveys (51:1), Article 7.

Digital Library

[24]

Deng, X., Joshi, K., and Galliers, R. D. 2016. "The Duality of Empowerment and Marginalization in Microtask Crowdsourcing: Giving Voice to the Less Powerful through Value Sensitive Design," MIS Quarterly (40:2), pp. 279-302.

Digital Library

[25]

Devillers, R., Stein, A., Bédard, Y., Chrisman, N., Fisher, P., and Shi, W. 2010. "Thirty Years of Research on Spatial Data Quality: Achievements, Failures, and Opportunities," Transactions in GIS (14:4), pp. 387-400.

[26]

Dewan, S., and Ramaprasad, J. 2012. "Research Note—Music Blogging, Online Sampling, and the Long Tail," Information Systems Research (23:3, Part 2), pp. 1056-1067.

Digital Library

[27]

Doan, A., Ramakrishnan, R., and Halevy, A. Y. 2011. "Crowd-sourcing Systems on the World-Wide Web," Communications of the ACM (54:4), pp. 86-96.

Digital Library

[28]

Engel, S., and Voshell, R. 2002. "Volunteer Biological Monitoring: Can It Accurately Assess the Ecological Condition of Streams?," American Entomologist (48:3), pp. 164-177.

[29]

Erickson, L., Petrick, I., and Trauth, E. 2012. "Hanging with the Right Crowd: Matching Crowdsourcing Need to Crowd Characteristics," in Proceedings of the 19^th Americas Conference on Information Systems, Seattle, Washington, August 9-12.

[30]

Fielden, M. A., Chaulk, A. C., Bassett, K., Wiersma, Y. F., Erbland, M., Whitney, H., and Chapman, T. W. 2015. "Aedes Japonicus Japonicus (Diptera: Culicidae) Arrives at the Most Easterly Point in North America," The Canadian Entomologist (147:6), pp. 737-740.

[31]

Galloway, A. W. E., Tudor, M. T., and Haegen, W. M. V. 2006. "The Reliability of Citizen Science: A Case Study of Oregon White Oak Stand Surveys," Wildlife Society Bulletin (34:5), pp. 1425-1429.

[32]

Gao, G. G., Greenwood, B. N., Agarwal, R., and McCullough, J. S. 2015. "Vocal Minority and Silent Majority: How Do Online Ratings Reflect Population Perceptions of Quality?," MIS Quarterly (39:3), pp. 565-589.

Digital Library

[33]

Garcia-Molina, H., Joglekar, M., Marcus, A., Parameswaran, A., and Verroios, V. 2016. "Challenges in Data Crowdsourcing," IEEE Transactions on Knowledge and Data Engineering (28:4), pp. 901-911.

Digital Library

[34]

Gebauer, J., and Schober, F. 2006. "Information System Flexibility and the Cost Efficiency of Business Processes," Journal of the AIS (7:3), p. 8.

[35]

Gigerenzer, G., and Todd, P. M. 1999. Simple Heuristics That Make Us Smart, New York: Oxford University Press.

[36]

Gilovich, T., Griffin, D., and Kahneman, D. 2002. Heuristics and Biases: The Psychology of Intuitive Judgment, Cambridge, UK: Cambridge University Press.

[37]

Goes, P. B. 2014. "Editor's Comments: Design Science Research in Top Information Systems Journals," MIS Quarterly (38:1), iii-viii.

Digital Library

[38]

Good, P. I. 2001. Resampling Methods: A Practical Guide to Data Analysis. Second Edition, Berlin, Germany: Springer Science & Business Media.

Digital Library

[39]

Goodman, J. K., and Paolacci, G. 2017. "Crowdsourcing Consumer Research," Journal of Consumer Research (44:1), pp. 196-210.

[40]

Gray, M. L., Suri, S., Ali, S. S., and Kulkarni, D. 2016. "The Crowd Is a Collaborative Network," in Proceedings of the 19^th ACM Conference on Computer-Supported Cooperative Work & Social Computing, New York: ACM Press, pp. 134-147.

Digital Library

[41]

Hackshaw, A. 2008. "Small Studies: Strengths and Limitations," European Respiratory Journal (32), pp. 1141-1143.

[42]

Harnad, S. 2005. "To Cognize Is to Categorize: Cognition Is Categorization," in Handbook of Categorization in Cognitive, H. Cohen and C. Lefebvre (eds.), Amsterdam: Elsevier Science, pp. 20-45.

[43]

Hemsley, J., and Mason, R. M. 2012. "The Nature of Knowledge in the Social Media Age: Implications for Knowledge Management Models," in Proceedings of the 45^th Hawaii International Conference on System Sciences, Los Alamitos, CA: IEEE Computer Society Press, pp. 3928-3937.

Digital Library

[44]

Ipeirotis, P. G., Provost, F., and Wang, J. 2010. "Quality Management on Amazon Mechanical Turk," in Proceedings of the ACM SIGKDD Workshop on Human Computation, New York: ACM Press, pp. 64-67.

Digital Library

[45]

Johnson, S. L., Faraj, S., and Kudaravalli, S. 2014. "Emergence of Power Laws in Online Communities: The Role of Social Mechanisms and Preferential Attachment," MIS Quarterly (38:3), pp. 795-808.

Digital Library

[46]

Kallinikos, J., and Tempini, N. 2014. "Patient Data as Medical Facts: Social Media Practices as a Foundation for Medical Knowledge Creation," Information Systems Research (25:4), pp. 817-833.

Digital Library

[47]

Khatib, F., DiMaio, F., Cooper, S., Kazmierczyk, M., Gilski, M., Krzywda, S., Zabranska, H., Pichova, I., Thompson, J., and Popovi?, Z. 2011. "Crystal Structure of a Monomeric Retroviral Protease Solved by Protein Folding Game Players," Nature Structural & Molecular Biology (18:10), pp. 1175-1177.

[48]

Kilian, D. 2008. "When it Absolutely Has to Be Accurate, Don't Trust the Crowd," ECT News Network.

[49]

Kosmala, M., Wiggins, A., Swanson, A., and Simmons, B. 2016. "Assessing Data Quality in Citizen Science," Frontiers in Ecology and the Environment (14:10), pp. 551-560.

[50]

Krogh, B., Levy, S., Dutoit, A., and Subrahmanian, E. 1996. "Strictly Class-Based Modelling Considered Harmful," in Proceedings of the 29^th Hawaii International Conference on System Sciences, Los Alamitos, CA: IEEE Computer Society Press, pp. 242-250.

Digital Library

[51]

Landis, J. R., and Koch, G. G. 1977. "The Measurement of Observer Agreement for Categorical Data," Biometrics (33:1), pp. 159-174.

[52]

Larsen, K., and Bong, C. H. 2016. "A Tool for Addressing Construct Identity in Literature Reviews and Meta-Analyses," MIS Quarterly (40:3), pp. 529-552.

Digital Library

[53]

Lassaline, M. E., Wisniewski, E. J., and Medin, D. L. 1992. "Basic Levels in Artificial and Natural Categories: Are All Basic Levels Created Equal?," in Percepts, Concepts and Categories: The Representation and Processing of Information (Volume 93), Advances in Psychology, B. Barbara (ed.), Amsterdam: North-Holland, pp. 328-378.

[54]

Lee, Y. W. 2003. "Crafting Rules: Context-Reflective Data Quality Problem Solving," Journal of Management Information Systems (20:3), pp. 93-119.

Digital Library

[55]

Lee, Y. W., Pipino, L. L., Funk, J. D., and Wang, R. Y. 2006. Journey to Data Quality, Cambridge, MA: MIT Press.

Digital Library

[56]

Leonardi, P. 2011. "When Flexible Routines Meet Flexible Technologies: Affordance, Constraint, and the Imbrication of Human and Material Agencies," MIS Quarterly (35:1), pp. 147-167.

Digital Library

[57]

Levy, M., and Germonprez, M. 2017. "The Potential for Citizen Science in Information Systems Research," Communications of the AIS (40), Article 2.

[58]

Lewandowski, E., and Specht, H. 2015. "Influence of Volunteer and Project Characteristics on Data Quality of Biological Surveys," Conservation Biology (29:3), pp. 713-723.

[59]

Li, G., Wang, J., Zheng, Y., and Franklin, M. 2016. "Crowd-sourced Data Management: A Survey," IEEE Transactions on Knowledge and Data Engineering (28:9), pp. 2296-2319.

Digital Library

[60]

Louv, R., Dickinson, J. L., and Bonney, R. 2012. Citizen Science: Public Participation in Environmental Research, Ithaca, NY: Cornell University Press.

[61]

Love, J., and Hirschheim, R. 2017. "Crowdsourcing of Information Systems Research," European Journal of Information Systems (26:4), pp. 315-332.

[62]

Lukyanenko, R., Parsons, J., and Samuel, B. M. 2019. "Representing Instances: The Case for Reengineering Conceptual Modeling Grammars," European Journal of Information Systems (28:1), pp. 68-90.

[63]

Lukyanenko, R., Parsons, J., and Wiersma, Y. 2014. "The IQ of the Crowd: Understanding and Improving Information Quality in Structured User-Generated Content," Information Systems Research (25:4), pp. 669-689.

Digital Library

[64]

Lukyanenko, R., Parsons, J., and Wiersma, Y. 2016. "Emerging Problems of Data Quality in Citizen Science," Conservation Biology (30:3), pp. 447-449.

[65]

Lukyanenko, R., Parsons, J., Wiersma, Y. F., Wachinger, G., Huber, B., and Meldt, R. 2017. "Representing Crowd Knowledge: Guidelines for Conceptual Modeling of User-Generated Content," Journal of the AIS (18:4), pp. 297-339.

[66]

Lukyanenko, R., Parsons, J., Wiersma, Y., Sieber, R., and Maddah, M. 2016. "Participatory Design for User-Generated Content: Understanding the Challenges and Moving Forward," Scandinavian Journal of Information Systems (28:1), pp. 37-70.

[67]

Madnick, S. E., Wang, R. Y., Lee, Y. W., and Zhu, H. 2009. "Overview and Framework for Data and Information Quality Research," Journal of Data and Information Quality (1:1), pp. 1-22.

Digital Library

[68]

Markus, M. L., and Silver, M. S. 2008. "A Foundation for the Study of IT Effects: A New Look at DeSanctis and Poole's Concepts of Structural Features and Spirit," Journal of the AIS (9:10/11), pp. 609-632.

[69]

Martinez, W. L., Martinez, A., and Solka, J. 2004. Exploratory Data Analysis with MATLAB, New York, NY: Taylor & Francis.

Digital Library

[70]

Mayden, R. L. 2002. "On Biological Species, Species Concepts and Individuation in the Natural World," Fish and Fisheries (3:3), pp. 171-196.

[71]

McKinley, D. C., Miller-Rushing, A. J., Ballard, H. L., Bonney, R., Brown, H., Cook-Patton, S. C., Evans, D. M., French, R. A., Parrish, J. K., and Phillips, T. B. 2016. "Citizen Science Can Improve Conservation Science, Natural Resource Management, and Environmental Protection," Biological Conservation (208), pp. 15-28.

[72]

Meijer, A., Burger, N., and Ebbers, W. 2009. "Citizens4Citizens: Mapping Participatory Practices on the Internet," Electronic Journal of E-Government (7:1), pp. 99-112.

[73]

Mervis, C. B., and Rosch, E. 1981. "Categorization of Natural Objects," Annual Review of Psychology (32:1), pp. 89-115.

[74]

Newman, G., Wiggins, A., Crall, A., Graham, E., Newman, S., and Crowston, K. 2012. "The Future of Citizen Science: Emerging Technologies and Shifting Paradigms," Frontiers in Ecology and the Environment (10:6), pp. 298-304.

[75]

Norman, D. A. 1983. "Some Observations on Mental Models," Mental Models (7:112), pp. 7-14.

[76]

Nov, O., Arazy, O., and Anderson, D. 2014. "Scientists@ Home: What Drives the Quantity and Quality of Online Citizen Science Participation," PloS One (9:4), pp. 1-11.

[77]

Olivé, A. 2007. Conceptual Modeling of Information Systems, Berlin: Springer Science & Business Media.

Digital Library

[78]

Parreiras, F. S., and Staab, S. 2010. "Using Ontologies with UML Class-Based Modeling: The TwoUse Approach," Data & Knowledge Engineering (69:11), pp. 1194-1207.

Digital Library

[79]

Parsons, J., Lukyanenko, R., and Wiersma, Y. 2011. "Easier Citizen Science Is Better," Nature (471:7336), pp. 37-37.

[80]

Parsons, J., and Wand, Y. 2000. "Emancipating Instances from the Tyranny of Classes in Information Modeling," ACM Transactions on Database Systems (25:2), pp. 228-268.

Digital Library

[81]

Petter, S., DeLone, W., and McLean, E. R. 2013. "Information Systems Success: The Quest for the Independent Variables," Journal of Management Information Systems (29:4), pp. 7-62.

[82]

Pipino, L. L., Lee, Y. W., and Wang, R. Y. 2002. "Data Quality Assessment," Communications of the ACM (45:4), pp. 211-218.

Digital Library

[83]

Prestopnik, N. R., and Tang, J. 2015. "Points, Stories, Worlds, and Diegesis: Comparing Player Experiences in Two Citizen Science Games," Computers in Human Behavior (52), pp. 492-506.

Digital Library

[84]

Provost, F., and Fawcett, T. 2013. Data Science for Business: What You Need to Know about Data Mining and Data-Analytic Thinking, Sebastopol, CA: O'Reilly Media, Inc.

Digital Library

[85]

Prpi?, J., Shukla, P. P., Kietzmann, J. H., and McCarthy, I. P. 2015. "How to Work a Crowd: Developing Crowd Capital through Crowdsourcing," Business Horizons (58:1), pp. 77-85.

[86]

Rai, A. 2016. "Editor's Comments: Synergies Between Big Data and Theory," MIS Quarterly (40:1), iii-ix.

Digital Library

[87]

Redman, T. C. 1996. Data Quality for the Information Age, Norwood, MA: Artech House.

Digital Library

[88]

Rosch, E., Mervis, C. B., Gray, W. D., Johnson, D. M., and Boyesbraem, P. 1976. "Basic Objects in Natural Categories," Cognitive Psychology (8:3), pp. 382-439.

[89]

Rosemann, M., and Vessey, I. 2008. "Toward Improving the Relevance of Information Systems Research to Practice: The Role of Applicability Checks," MIS Quarterly (32:1), pp. 1-22.

Digital Library

[90]

Saghafi, A. 2016. "Ontological and Cognitive Principles on Information Systems Modelling," Doctoral Dissertation, University of British Columbia.

[91]

Samuel, B. M., Khatri, V., and Ramesh, V. 2018. "Exploring the Effects of Extensional Versus Intentional Representations on Domain Understanding," MIS Quarterly (42:4), pp. 1186-1209.

[92]

Sawilowsky, S. S. 2009. "New Effect Size Rules of Thumb," Journal of Modern Applied Statistical Methods (8:2), pp. 467-474.

[93]

Sekhavat, Y., and Parsons, J. 2012. "Sliced Column-Store (SCS): Ontological Foundations and Practical Implications," in Proceedings of the 31^st International Conference on Conceptual Modeling, Florence, Italy, October 15-18, pp. 102-115.

Digital Library

[94]

Shankaranarayanan, G., and Blake, R. 2017. "From Content to Context: The Evolution and Growth of Data Quality Research," Journal of Data and Information Quality (8:2).

Digital Library

[95]

Shanks, G., Tansley, E., Nuredini, J., Tobin, D., and Weber, R. 2008. "Representing Part-Whole Relations in Conceptual Modeling: An Empirical Evaluation," MIS Quarterly (32:3), pp. 553-573.

Digital Library

[96]

Sheng, V. S., Provost, F., and Ipeirotis, P. G. 2008. "Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers," in Proceedings of the 14^th International Conference on Knowledge Discovery and Data Mining, New York: ACM Press, pp. 614-622.

Digital Library

[97]

Silvertown, J. 2009. "A New Dawn for Citizen Science," Trends in Ecology & Evolution (24:9), pp. 467-471.

[98]

Strong, D. M., Lee, Y. W., and Wang, R. Y. 1997. "Data Quality in Context," Communications of the ACM (40:5), pp. 103-110.

Digital Library

[99]

Sullivan, B. L., Wood, C. L., Iliff, M. J., Bonney, R. E., Fink, D., and Kelling, S. 2009. "EBird: A Citizen-Based Bird Observation Network in the Biological Sciences," Biological Conservation (142:10), pp. 2282-2292.

[100]

Surowiecki, J. 2005. The Wisdom of Crowds, New York: Anchor Books.

Digital Library

[101]

Swanson, A., Kosmala, M., Lintott, C., Simpson, R., Smith, A., and Packer, C. 2015. "Snapshot Serengeti, High-Frequency Annotated Camera Trap Images of 40 Mammalian Species in an African Savanna," Scientific Data (2), Article 150026.

[102]

Tax, S. S., and Brown, S. W. 1998. "Recovering and Learning from Service Failure," MIT Sloan Management Review (40:1), pp. 75-88.

[103]

Tilly, R., Posegga, O., Fischbach, K., and Schoder, D. 2016. "Towards a Conceptualization of Data and Information Quality in Social Information Systems," Business & Information Systems Engineering (59:1), pp. 3-21.

[104]

Trevino, L. K., and Victor, B. 1992. "Peer Reporting of Unethical Behavior: A Social Context Perspective," Academy of Management Journal (35:1), pp. 38-64.

[105]

Tucker, A. L., and Edmondson, A. C. 2003. "Why Hospitals Don't Learn from Failures: Organizational and Psychological Dynamics That Inhibit System Change," California Management Review (45:2), pp. 55-72.

[106]

Tversky, A., and Kahneman, D. 1974. "Judgment under Uncertainty: Heuristics and Biases," Science (185:4157), pp. 1124-1131.

[107]

Venkatesh, V., Morris, M. G., Davis, G. B., and Davis, F. D. 2003. "User Acceptance of Information Technology: Toward a Unified View," MIS Quarterly (27:3), pp. 425-478.

Digital Library

[108]

Wallis, G., and Bülthoff, H. 1999. "Learning to Recognize Objects," Trends in Cognitive Sciences (3:1), pp. 22-31.

[109]

Wand, Y., and Wang, R. Y. 1996. "Anchoring Data Quality Dimensions in Ontological Foundations," Communications of the ACM (39:11), pp. 86-95.

Digital Library

[110]

Wang, R. Y., Reddy, M. P., and Kon, H. B. 1995. "Toward Quality Data: An Attribute-Based Approach," Decision Support Systems (13:3-4), pp. 349-372.

Digital Library

[111]

Wang, R. Y., and Strong, D. M. 1996. "Beyond Accuracy: What Data Quality Means to Data Consumers," Journal of Management Information Systems (12:4), pp. 5-33.

Digital Library

[112]

Wells, J. D., Valacich, J. S., and Hess, T. J. 2011. "What Signals Are You Sending? How Website Quality Influences Perceptions of Product Quality and Purchase Intentions," MIS Quarterly (35:2), pp. 373-396.

Digital Library

[113]

Wiggins, A., Bonney, R., Graham, E., Henderson, S., Kelling, S., LeBuhn, G., Litauer, R., Lots, K., Michener, W., and Newman, G. 2013. "Data Management Guide for Public Participation in Scientific Research," DataOne Working Group.

[114]

Wiggins, A., and Crowston, K. 2011. "From Conservation to Crowdsourcing: A Typology of Citizen Science," in Proceedings of the 44^th Hawaii International Conference on System Sciences, Los Alamitos, CA: IEEE Computer Society Press.

Digital Library

[115]

Wiggins, A., Newman, G., Stevenson, R. D., and Crowston, K. 2011. "Mechanisms for Data Quality and Validation in Citizen Science," in Proceedings of the Computing for Citizen Science Workshop, IEEE eScience Conference, Stockholm, pp. 14-19.

Digital Library

Cited By

Abbasi AParsons JPant GSheng OSarker S(2024)Pathways for Design Research on Artificial IntelligenceInformation Systems Research10.1287/isre.2024.editorial.v35.n235:2(441-459)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1287/isre.2024.editorial.v35.n2
Cheng DCh’ng E(2022)Harnessing Collective Differences in Crowdsourcing Behaviour for Mass Photogrammetry of 3D Cultural HeritageJournal on Computing and Cultural Heritage 10.1145/356909016:1(1-23)Online publication date: 24-Dec-2022
https://dl.acm.org/doi/10.1145/3569090
Wallace SPapoutsaki ATan NGuo HHuang J(2021)Case Studies on the Motivation and Performance of Contributors Who Verify and Maintain In-Flux Tabular DatasetsProceedings of the ACM on Human-Computer Interaction10.1145/34795925:CSCW2(1-25)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3479592
Show More Cited By

Recommendations

The IQ of the Crowd: Understanding and Improving Information Quality in Structured User-Generated Content
Special Section: Information, Technology, and the Changing Nature of Work

User-generated content UGC is becoming a valuable organizational resource, as it is seen in many cases as a way to make more information available for analysis. To make effective use of UGC, it is necessary to understand information quality IQ in this ...
Citizen Science: An Information Quality Research Frontier
Abstract
The rapid proliferation of online content producing and sharing technologies resulted in an explosion of user-generated content (UGC), which now extends to scientific data. Citizen science, in which ordinary people contribute information for ...
Tag suggestion and localization in user-generated videos based on social knowledge
WSM '10: Proceedings of second ACM SIGMM workshop on Social media

Nowadays, almost any web site that provides means for sharing user-generated multimedia content, like Flickr, Facebook, YouTube and Vimeo, has tagging functionalities to let users annotate the material that they want to share. The tags are then used to ...

Comments

Information & Contributors

Information

Published In

cover image MIS Quarterly

MIS Quarterly Volume 43, Issue 2

June 2019

544 pages

ISSN:0276-7783

Editor:
Arun Rai
Robinson College of Business, Georgia State University

Issue’s Table of Contents

Publisher

Society for Information Management and The Management Information Systems Research Center

United States

Publication History

Published: 01 June 2019

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Abbasi AParsons JPant GSheng OSarker S(2024)Pathways for Design Research on Artificial IntelligenceInformation Systems Research10.1287/isre.2024.editorial.v35.n235:2(441-459)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1287/isre.2024.editorial.v35.n2
Cheng DCh’ng E(2022)Harnessing Collective Differences in Crowdsourcing Behaviour for Mass Photogrammetry of 3D Cultural HeritageJournal on Computing and Cultural Heritage 10.1145/356909016:1(1-23)Online publication date: 24-Dec-2022
https://dl.acm.org/doi/10.1145/3569090
Wallace SPapoutsaki ATan NGuo HHuang J(2021)Case Studies on the Motivation and Performance of Contributors Who Verify and Maintain In-Flux Tabular DatasetsProceedings of the ACM on Human-Computer Interaction10.1145/34795925:CSCW2(1-25)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3479592
Haug SRietz TMaedche A(2021)Accelerating Deductive Coding of Qualitative Data: An Experimental Study on the Applicability of CrowdsourcingProceedings of Mensch und Computer 202110.1145/3473856.3473873(432-443)Online publication date: 5-Sep-2021
https://dl.acm.org/doi/10.1145/3473856.3473873
Lukyanenko RStorey VPastor O(2021)Foundations of information technology based on Bunge’s systemist philosophy of realitySoftware and Systems Modeling (SoSyM)10.1007/s10270-021-00862-520:4(921-938)Online publication date: 1-Aug-2021
https://dl.acm.org/doi/10.1007/s10270-021-00862-5

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents