News Ninja: Gamified Annotation of Linguistic Bias in Online News
Authors: 
Smi Hinterreiter, Timo Spinde, Sebastian Oberdörfer, Isao Echizen, 
Marc Erich LatoschikAuthors Info & Claims
Smi Hinterreiter, Timo Spinde, Sebastian Oberdörfer, Isao Echizen, Marc Erich LatoschikAuthors Info & ClaimsArticle No.: 327, Pages 1 - 29
Abstract
Recent research shows that visualizing linguistic bias mitigates its negative effects. However, reliable automatic detection methods to generate such visualizations require costly, knowledge-intensive training data. To facilitate data collection for media bias datasets, we present News Ninja, a game employing data-collecting game mechanics to generate a crowdsourced dataset. Before annotating sentences, players are educated on media bias via a tutorial. Our findings show that datasets gathered with crowdsourced workers trained on News Ninja can reach significantly higher inter-annotator agreements than expert and crowdsourced datasets with similar data quality. As News Ninja encourages continuous play, it allows datasets to adapt to the reception and contextualization of news over time, presenting a promising strategy to reduce data collection expenses, educate players, and promote long-term bias mitigation.
Supplemental Material
MP4 File - Video Figure
Video figure for "News Ninja: Gamified Annotation of Linguistic Bias in Online News"
- Download
- 21.22 MB
MP4 File
Supplemental video
- Download
- 32.23 MB
References
[1]
Alexander Dallmann et al. 'Media Bias in German Online Newspapers'. In: Proceedings of the 26th ACM Conference on Hypertext & Social Media.HT '15. NewYork, NY, USA: Association for Computing Machinery, 2015, pp. 133--137. isbn: 9781450333955. url: https://doi.org/10.1145/2700171.2791057.
[2]
William P. Eveland Jr. and Dhavan V. Shah. 'The impact of individual and interpersonal factors on perceived news media bias'. In: Political Psychology 24.1 (2003), pp. 101--117. url: https://onlinelibrary.wiley.com/doi/abs/10. 1111/0162--895X.00318.
[3]
Pippa Norris. A Virtuous Circle: Political Communications in Postindustrial Societies. Communication, Society and Politics. Cambridge: Cambridge University Press, 2000. isbn: 978-0- 521--79015--4. url: https://www.cambridge.org/core/books/ virtuous-circle/93623037EA261D4CA3AE0CB41E41A46A.
[4]
Timo Spinde et al. 'Automated identification of bias inducing words in news articles using linguistic and context-oriented features?. In: Information Processing & Management 58.3 (Jan. 2021), p. 102505. issn: 0306--4573. url: https://www.sciencedirect.com/science/article/pii/S0306457321000157/pdfft?md5= 64e81212b3bfa861d01a6fe3d5b979c3%5C&pid=1-s2.0-S0306457321000157-main.pdf.
[5]
David Domke, Kelly McCoy, and Marcos Torres. 'News media, racial perceptions, and political cognition'. In: Communication Research 26.5 (1999), pp. 570--607. 009365099026005003. url: https://doi.org/10.1177/009365099026005003.
[6]
Alberto Ardèvol-Abreu and Homero Gil de Zúñiga. 'Effects of editorial media bias perception and media trust on the use of traditional, citizen, and social media news'. In: Journalism & Mass Communication Quarterly 94.3 (2017), pp. 703--724. url: https://doi.org/10.1177/1077699016654684.
[7]
Stefano DellaVigna and Ethan Kaplan. 'The fox news effect: Media bias and voting'. In: The Quarterly Journal of Economics 122.3 (Aug. 2007), pp. 1187--1234. issn: 0033--5533. url: https://doi.org/10.1162/qjec.122.3.1187.
[8]
Jakob-Moritz Eberl, Hajo G. Boomgaarden, and Markus Wagner. 'One bias fits all' Three types of media bias and their effects on party preferences?. In: Communication Research 44.8 (2017), pp. 1125--1148. url: https://doi.org/10.1177/ 0093650215614364.
[9]
Timo Spinde et al. 'The Media Bias Taxonomy: A Systematic Literature Review on the Forms and Automated Detection of Media Bias'. In: arXiv (2023). arXiv: 2312.16148 [cs.CL].
[10]
Ceren Budak, Sharad Goel, and Justin M. Rao. 'Fair and balanced' Quantifying media bias through crowdsourced content analysis?. In: Public Opinion Quarterly 80.S1 (Apr. 2016), pp. 250--271. issn: 0033--362X. url: https://doi.org/10.1093/poq/ nfw007.
[11]
Christoph Hube and Besnik Fetahu. 'Neural based statement classification for biased language'. In: Proceedings of the twelfth ACM international conference on web search and data mining.WSDM'19. NewYork, NY, USA: Association for Computing Machinery, 2019, pp. 195--203. isbn: 978--1--4503--5940--5. url: https://doi.org/10.1145/3289600.3291018.
[12]
Sendhil Mullainathan and Andrei Shleifer. 'Media Bias'. In: NBER Working Paper Series. Cambridge, MA, Oct. 2002. url: http://www.nber.org/papers/w9295.pdf.
[13]
Timo Spinde et al. 'Enabling News Consumers to View and Understand Biased News Coverage: A Study on the Perception and Visualization of Media Bias'. In: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020. JCDL '20. Virtual Event, China: Association for Computing Machinery, Jan. 2020, pp. 389--392. isbn: 978--1--4503--7585--6. url: https://doi.org/10.1145/3383583.3398619.
[14]
Filipe N. Ribeiro et al. 'Media Bias Monitor : Quantifying Biases of Social Media News Outlets at Large-Scale'. In: Twelfth International AAAI Conference on Web and Social Media. Palo Alto, California: AAAI Press, Jan. 2018, pp. 290--299. isbn: 978--1--57735--798--8. url: https://aaai.org/ocs/index.php/ICWSM/ICWSM18/paper/view/17878.
[15]
Astrid Kause, Tarlise Townsend, and Wolfgang Gaissmaier. 'Framing climate uncertainty: Frame choices reveal and influence climate change beliefs'. In: Weather, Climate, and Society 11.1 (2019), pp. 199--215. url: https://journals.ametsoc.org/ view/journals/wcas/11/1/wcas-d-18-0002_1.xml.
[16]
Timo Spinde et al. 'Howdowe raise media bias awareness effectively' Effects of visualizations to communicate bias?. In: PLOS ONE 17.4 (Jan. 2022). Publisher: Public Library of Science, pp. 1--14. url: https://doi.org/10.1371/journal.pone. 0266204.
[17]
Timo Spinde, Felix Hamborg, and Bela Gipp. 'An Integrated Approach to Detect Media Bias in German News Articles'. In: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020. JCDL '20. Virtual Event, China: Association for Computing Machinery, Jan. 2020, pp. 505--506. isbn: 978--1--4503--7585--6. url: https://doi.org/10.1145/3383583.3398585.
[18]
Yuanyuan Lei et al. 'Sentence-level Media Bias Analysis Informed by Discourse Structures'. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 10040--10050. url: https://aclanthology.org/2022.emnlp-main.682.
[19]
Timo Spinde et al. 'Neural Media Bias Detection Using Distant Supervision With BABE - Bias Annotations By Experts'. In: Findings of the Association for Computational Linguistics: EMNLP 2021. Dominican Republic, Nov. 2021. url: https://media-bias-research.org/wp-content/uploads/2022/01/Neural_Media_Bias_Detection_ Using_Distant_Supervision_With_BABE___Bias_Annotations_By_Experts_MBG.pdf.
[20]
Martin Wessel et al. 'Introducing MBIB - the first media bias identification benchmark task and dataset collection'. In: Proceedings of 46th international ACM SIGIR conference on research and development in information retrieval (SIGIR23). New York, NY, USA: ACM, July 2023.
[21]
Felix Hamborg, Karsten Donnay, and Bela Gipp. 'Automated identification of media bias in news articles: an interdisciplinary literature review'. In: International Journal on Digital Libraries 20.4 (Dec. 2019), pp. 391--415. issn: 1432--5012, 1432--1300. 0261-y. url: http://link.springer.com/10.1007/s00799-018-0261-y.
[22]
Luis von Ahn. 'Games with a Purpose'. In: Computer 39.6 (June 2006), pp. 92--94. issn: 0018--9162. url: http://ieeexplore.ieee.org/document/1642623/.
[23]
Christopher Madge et al. 'Incremental Game Mechanics Applied to Text Annotation'. In: Proceedings of the Annual Symposium on Computer-Human Interaction in Play. Barcelona Spain: ACM, Oct. 2019, pp. 545--558. isbn: 978--1--4503--6688--5. url: https://dl.acm.org/doi/10.1145/3311350.3347184.
[24]
Massimo Poesio et al. 'Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation'. In: ACM Transactions on Interactive Intelligent Systems 3.1 (Apr. 2013), pp. 1--44. issn: 2160--6455, 2160--6463. url: https ://dl.acm.org/doi/10.1145/2448116.2448119.
[25]
Tarja Susi, Mikael Johannesson, and Per Backlund. Serious Games : An Overview. Institutionen för kommunikation och information, 2007. url: https://urn.kb.se/resolve?urn=urn:nbn:se:his:diva-1279.
[26]
Melisa Basol, Jon Roozenbeek, and Sander van der Linden. 'Good News about Bad News: Gamified Inoculation Boosts Confidence and Cognitive Immunity Against Fake News'. In: Journal of cognition 3.1 (Jan. 2020). Publisher: Ubiquity Press, pp. 2--2. issn: 2514--4820. url: https:// .ncbi.nlm.nih.gov/31934684.
[27]
Jeroen J. G. van Merriënboer Kirschner Paul A. Ten Steps to Complex Learning: A Systematic Approach to Four-Component Instructional Design. 3rd ed. New York: Routledge, Oct. 2017. isbn: 978--1--315--11321-0.
[28]
Sebastian Oberdorfer and Marc Erich Latoschik. 'Gamified Knowledge Encoding: Knowledge Training Using Game Mechanics'. In: 2018 10th International Conference on Virtual Worlds and Games for Serious Applications (VS-Games). ISSN: 2474-0489. Sept. 2018, pp. 1--2.
[29]
Yu-kai Chou. Actionable Gamification: Beyond Points, Badges, and Leaderboards. 2019th ed. Octalysis Media, Apr. 2015. isbn: 978--1511744041.
[30]
Sylvester Arnab et al. 'Mapping learning and game mechanics for serious games analysis'. In: British Journal of Educational Technology 46.2 (2015), pp. 391--411. issn: 1467--8535. url: https://onlinelibrary.wiley.com/doi/abs/10.1111/bjet.12113.
[31]
Rosa Lilia Segundo Díaz et al. 'Building blocks for creating enjoyable games'A systematic literature review?. In: International Journal of Human-Computer Studies 159 (Mar. 2022), p. 102758. issn: 1071--5819. url: https://www.sciencedirect. com/science/article/pii/S1071581921001762.
[32]
Camiel J. Beukeboom and Christian Burgers. Linguistic Bias. July 2017. 9780190228613.013.439.
[33]
Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 'Linguistic Models for Analyzing and Detecting Biased Language?. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Sofia, Bulgaria: Association for Computational Linguistics, Aug. 2013, pp. 1650--1659. url: https://aclanthology.org/P13- 1162.
[34]
Hannah Rashkin, Sameer Singh, and Yejin Choi. 'Connotation Frames: A Data-Driven Investigation'. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 311--321.
[35]
Timo Spinde et al. MBIC -- A Media Bias Annotation Dataset Including Annotator Characteristics. arXiv:2105.11910. May 2021. url: http://arxiv.org/abs/2105. 11910.
[36]
Carl-AntonWerner Axelsson, Mona Guath, and Thomas Nygren. 'Learning How to Separate Fake from Real News: Scalable Digital Tutorials Promoting Students' Civic Online Reasoning?. In: Future Internet 13.3 (Mar. 2021), p. 60. url: https://www.mdpi. com/1999--5903/13/3/60.
[37]
John Cook, Stephan Lewandowsky, and Ullrich K. H. Ecker. 'Neutralizing misinformation through inoculation: Exposing misleading argumentation techniques reduces their influence?. In: PLOS ONE 12.5 (May 2017). Publisher: Public Library of Science. issn: 1932--6203. url: https://journals.plos.org/plosone/article?id=10.1371/ journal.pone.0175799.
[38]
Patricia Aufderheide. 'Media Literacy: From a Report of the National Leadership Conference on Media Literacy'. In: Media Literacy in the Information Age (2018).
[39]
Sarah McGrew. 'Learning to evaluate: An intervention in civic online reasoning'. In: Computers & Education 145 (Feb. 2020), p. 103711. issn: 0360--1315. 103711. url: https://www.sciencedirect.com/science/article/pii/S0360131519302647.
[40]
David Buckingham. 'Teaching media in a 'post-truth' age: fake news, media bias and the challenge for media/digital literacy education?. In: Cultura y Educación 31 (May 2019), pp. 1-- 19.
[41]
Hendrik Heuer and Elena Leah Glassman. 'A comparative evaluation of interventions against misinformation: Augmenting the WHO checklist'. In: Proceedings of the 2022 CHI conference on human factors in computing systems. CHI '22. New York, NY, USA: Association for Computing Machinery, 2022. isbn: 978--1--4503--9157--3. url: https://doi.org/10.1145/3491102.3517717.
[42]
Smilla Hinterreiter. 'A Gamified Approach To Automatically Detect Biased Wording And Train Critical Reading'. In: 2021 IEEE International Conference on Data Mining Workshops (ICDMW). Oct. 2021. url: https://media- biasresearch. org/wp-content/uploads/2021/10/hinterreiter2021a.pdf.
[43]
Edith Law and Luis von Ahn. Human computation. ISSN: 1939--4616. Springer International Publishing, 2011. isbn: 978--3-031-01555--7. 3- 031- 01555- 7. url: http://dx.doi.org/10.1007/978--3-031-01555--7.
[44]
Luis von Ahn and Laura Dabbish. 'Designing games with a purpose'. In: Communications of the ACM 51.8 (Aug. 2008), pp. 58--67. issn: 0001-0782, 1557--7317. url: https://dl.acm.org/doi/10.1145/1378704.1378719.
[45]
Luis von Ahn and Laura Dabbish. 'Labeling images with a computer game'. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI '04. Vienna, Austria: Association for Computing Machinery, 2004, pp. 319--326. isbn: 1581137028. url: https://doi.org/10.1145/985692.985733.
[46]
Edith L. M. Law et al. 'TagATune: A Game for Music and Sound Annotation'. In: Proceedings of the 8th International Conference on Music Information Retrieval, ISMIR 2007, Vienna, Austria, September 23--27, 2007. Ed. by Simon Dixon, David Bainbridge, and Rainer Typke. Austrian Computer Society, 2007, pp. 361--364. url: http://ismir2007.ismir.net/proceedings/ ISMIR2007%5C_p361%5C_law.pdf.
[47]
Edith Law and Luis von Ahn. 'Input-agreement: a new mechanism for collecting data using human computation games'. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI '09. Boston, MA, USA: Association for Computing Machinery, 2009, pp. 1197--1206. isbn: 9781605582467. url: https://doi.org/10. 1145/1518701.1518881.
[48]
Brent Lance et al. 'Towards Serious Games for Improved BCI'. In: Defense Technical Information Center, Jan. 2015, pp. 1--28. isbn: 978--981--4560--52--8.
[49]
Frederik De Grove, Peter Mechant, and Jan Van Looy. 'UnchartedWaters' Exploring Experts? Opinions on the Opportunities and Limitations of Serious Games for Foreign Language Learning?. In: Proceedings of the 3rd International Conference on Fun and Games. Fun and Games '10. Leuven, Belgium: Association for Computing Machinery, 2010, pp. 107--115. isbn: 9781605589077. url: https://doi.org/10.1145/1823818.1823830.
[50]
Richard Van Eck. 'Digital Game Based Learning: It's Not Just the Digital Natives Who Are Restless?. In: EDUCAUSE 41 (Jan. 2006).
[51]
James Gee. 'What Video Games Have to Teach Us About Learning and Literacy?. In: Computers in Entertainment 1 (Oct. 2003), p. 20.
[52]
Frank Greitzer, Olga Kuchar, and Kristy Huston. 'Cognitive science implications for enhancing training effectiveness in a serious gaming context?. In: ACM Journal of Educational Resources in Computing 7 (Nov. 2007).
[53]
Sebastian Deterding et al. 'From Game Design Elements to Gamefulness: Defining "Gamification"?. In: Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments. MindTrek '11. Tampere, Finland: Association for Computing Machinery, 2011, pp. 9--15. isbn: 9781450308168. url: https://doi.org/10.1145/2181037.2181040.
[54]
Jeanine Krath, Linda Schürmann, and Harald F. O. von Korflesch. 'Revealing the theoretical basis of gamification: A systematic review and analysis of theory in research on gamification, serious games and game-based learning?. In: Computers in Human Behavior 125 (Dec. 2021), p. 106963. issn: 0747--5632. url: https://www.sciencedirect. com/science/article/pii/S0747563221002867.
[55]
Kathleen Tuite. 'GWAPs: Games with a problem?. In: International Conference on Foundations of Digital Games. 2014. url: https://api.semanticscholar.org/CorpusID:10959426.
[56]
Chris Madge et al. 'Testing TileAttack with Three Key Audiences?. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), May 2018. isbn: 979--10--95546--10--8.
[57]
M. Jordan Raddick et al. 'Galaxy Zoo: Exploring the Motivations of Citizen Science Volunteers?. In: Astronomy Education Review 9.1 (2010). Publisher: Portico. url: https://doi.org/10.3847/aer2009036.
[58]
Doruk Kicikoglu et al. 'Wormingo: a ?true gamification? approach to anaphoric annotation?. In: Proceedings of the 14th International Conference on the Foundations of Digital Games. San Luis Obispo California USA: ACM, Aug. 2019, pp. 1--7. isbn: 978--1--4503--7217--6. url: https://dl.acm.org/doi/10.1145/3337722.3341868.
[59]
Federico Bonetti and Sara Tonelli. 'A 3D Role-Playing Game for Abusive Language Annotation'. In: Workshop on Games and Natural Language Processing. Ed. by Stephanie M. Lukin. Marseille, France: European Language Resources Association, May 2020, pp. 39--43. isbn: 979--10--95546--40--5. url: https://aclanthology.org/2020.gamnlp-1.6 (visited on 05/21/2024).
[60]
Alice Millour et al. 'Katana and Grand Guru: a Game of the Lost Words (DEMO)'. In: 9th Language&Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics (LTC'19). Pozna?, Poland, May 2019. url: https://hal.science/hal- 02106757 (visited on 05/21/2024).
[61]
Fatima Althani, Chris Madge, and Massimo Poesio. 'Less Text, More Visuals: Evaluating the Onboarding Phase in a GWAP for NLP'. In: Proceedings of the 9th Workshop on Games and Natural Language Processing within the 13th Language Resources and Evaluation Conference. Ed. by Chris Madge. Marseille, France: European Language Resources Association, June 2022, pp. 17--27. url: https://aclanthology.org/2022.games-1.3 (visited on 05/09/2024).
[62]
François Bry and Marti Matthias. 'Newsroom: A GWAP to Study Public Opinion?. In: 2020. url: https://api.semanticscholar.org/CorpusID:232216378.
[63]
Ali Aghelmaleki. 'Generating a non-sexist corpus through gamification for automatic sexism detection?. PhD thesis. Mar. 2019. url: https://kola.opus.hbz-nrw.de/frontdoor/index/index/ year/2019/docId/1814 (visited on 05/21/2024).
[64]
Daniele Vannella et al. 'Validating and Extending Semantic Knowledge Bases using Video Games with a Purpose'. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Ed. by Kristina Toutanova and Hua Wu. Baltimore, Maryland: Association for Computational Linguistics, June 2014, pp. 1294--1304. url: https://aclanthology.org/P14--1122 (visited on 05/21/2024).
[65]
Sarah Ita Levitan, Xinyue Tan, and Julia Hirschberg. 'LieCatcher: Game Framework for Collecting Human Judgments of Deceptive Speech'. In: Proceedings of the 2020 International Conference on Multimodal Interaction. Virtual Event Netherlands: ACM, Oct. 2020, pp. 762-- 763. isbn: 978--1--4503--7581--8. url: https://dl.acm.org/doi/10. 1145/3382507.3421166 (visited on 05/09/2024).
[66]
Chris Madge et al. 'Making Text Annotation Fun with a Clicker Game'. In: Proceedings of the 14th International Conference on the Foundations of Digital Games. FDG '19. New York, NY, USA: Association for Computing Machinery, 2019. isbn: 978--1--4503--7217--6. url: https://doi.org/10.1145/3337722.3341869.
[67]
Luis von Ahn. 'Duolingo: learn a language for free while helping to translate the web'. In: Proceedings of the 2013 international conference on Intelligent user interfaces. IUI '13. New York, NY, USA: Association for Computing Machinery, Mar. 2013, pp. 1--2. isbn: 978--1--4503--1965--2. url: https://doi.org/10.1145/2449396.2449398.
[68]
Amy L. Baylor and Yanghee Kim. 'Pedagogical Agent Design: The Impact of Agent Realism, Gender, Ethnicity, and Instructional Role'. In: Intelligent Tutoring Systems. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer, 2004, pp. 592--603. isbn: 978--3--540--30139--4.
[69]
Ati Suci Dian Martha and Harry B. Santoso. 'The Design and Impact of the Pedagogical Agent: A Systematic Literature Review'. In: Journal of Educators Online 16.1 (Jan. 2019). url: https://eric.ed.gov/'id=EJ1204376.
[70]
Robert K. Atkinson, Richard E. Mayer, and Mary Margaret Merrill. 'Fostering social agency in multimedia learning: Examining the impact of an animated agent's voice'. In: Contemporary Educational Psychology 30.1 (2005). Publisher: Elsevier Science, pp. 117--139. issn: 1090--2384.
[71]
Chris Madge et al. 'LingoTowns: A Virtual World For Natural Language Annotation and Language Learning'. In: Extended Abstracts of the 2022 Annual Symposium on Computer- Human Interaction in Play. CHI PLAY '22. New York, NY, USA: Association for Computing Machinery, Nov. 2022, pp. 57--62. isbn: 978--1--4503--9211--2. url: https://doi.org/10.1145/3505270.3558323.
[72]
Sebastian Oberdörfer. 'Better Learning with Gaming: Knowledge Encoding and Knowledge Learning Using Gamification?. PhD thesis. Jan. 2021, p. 198. url: https://www.researchgate.net/publication/349140377_Better_Learning_with_Gaming_ Knowledge_Encoding_and_Knowledge_Learning_Using_Gamification.
[73]
Raph Koster. Theory of Fun for Game Design. 2nd. O'Reilly Media, Inc., 2013. isbn: 1--4493- 6321-0.
[74]
Gustavo F. Tondello et al. The Gamification User Types Hexad Scale. CHI PLAY '16. Austin, Texas, USA: Association for Computing Machinery, 2016, pp. 229--243. isbn: 9781450344562. url: https://doi.org/10.1145/2967934.2968082.
[75]
Gustavo F. Tondello, Alberto Mora, and Lennart E. Nacke. 'Elements of Gameful Design Emerging from User Preferences?. In: Proceedings of the Annual Symposium on Computer- Human Interaction in Play. CHI PLAY '17. New York, NY, USA: Association for Computing Machinery, Oct. 2017, pp. 129--142. isbn: 978--1--4503--4898-0. url: https://doi.org/10.1145/3116595.3116627.
[76]
Mihaly Csikszentmihalyi. Flow: The Psychology of Optimal Experience. Harper & Row, Jan. 1990.
[77]
Tim Draws et al. 'A Checklist to Combat Cognitive Biases in Crowdsourcing?. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 9 (Oct. 2021), pp. 48--59. issn: 2769--1349. url: https://ojs.aaai.org/index.php/HCOMP/ article/view/18939.
[78]
Jakub Piskorski et al. News Categorization, Framing and Persuasion Techniques: Annotation Guidelines. Tech. rep. JRC-132862. Ispra (Italy): European Commission Joint Research Centre, Mar. 2023. url: https://knowledge4policy.ec.europa.eu/text-mining/news-categorizationframing- persuasion-techniques-annotation-guidelines_en.
[79]
Patricia L. Moravec, Antino Kim, and Alan R. Dennis. 'Appealing to sense and sensibility: System 1 and system 2 interventions for fake news on social media?. In: Information Systems Research 31.3 (2020), pp. 987--1006. url: https://doi.org/10.1287/ isre.2020.0927.
[80]
Amos Tversky and Daniel Kahneman. 'Judgment under uncertainty: Heuristics and biases?. In: Science (New York, N.Y.) 185.4157 (1974), pp. 1124--1131. url: https://www.science.org/doi/abs/10.1126/science.185.4157.1124.
[81]
Juho Hamari. Gamification - Motivations & Effects. ISSN: 1799--4942 (electronic). Aalto University, 2015. isbn: 978--952--60--6056--9. url: https://aaltodoc.aalto.fi/handle/123456789/15037 (visited on 05/07/2024).
[82]
Colin M. Gray et al. 'The Dark (Patterns) Side of UX Design?. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Apr. 2018, pp. 1--14. isbn: 978--1--4503--5620--6. url: https://dl.acm.org/doi/10.1145/3173574.3174108.
[83]
Timo Spinde et al. 'TASSY - A Text Annotation Survey System?. In: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries (JCDL). Sept. 2021. 2021.00052. url: https://media-bias-research.org/wp-content/uploads/2022/01/Spinde2021c. pdf.
[84]
Jeff Sauro and Joseph S. Dumas. 'Comparison of three one-question, post-task usability questionnaires?. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI '09. New York, NY, USA: Association for Computing Machinery, Apr. 2009, pp. 1599--1608. isbn: 978--1--60558--246--7. url: https://doi.org/10.1145/1518701.1518946.
[85]
E. McAuley, T. Duncan, and V. V. Tammen. 'Psychometric properties of the Intrinsic Motivation Inventory in a competitive sport setting: a confirmatory factor analysis?. In: Research Quarterly for Exercise and Sport 60.1 (Mar. 1989), pp. 48--58. issn: 0270--1367.
[86]
Andrew F. Hayes and Klaus Krippendorff. 'Answering the Call for a Standard Reliability Measure for Coding Data'. In: Communication Methods and Measures 1.1 (Apr. 2007), pp. 77-- 89. issn: 1931--2458. / 19312450709336664. url: https : / / doi.org / 10.1080 / 19312450709336664.
[87]
Jahna Otterbacher. 'Crowdsourcing Stereotypes: Linguistic Bias in Metadata Generated via GWAP'. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. CHI '15. New York, NY, USA: Association for Computing Machinery, Apr. 2015, pp. 1955--1964. isbn: 978--1--4503--3145--6. url: https://doi.org/10.1145/2702123.2702151.
[88]
Gianluca Demartini, Stefano Mizzaro, and Damiano Spina. 'Human-in-the-loop artificial intelligence for fighting online misinformation: Challenges and opportunities'. In: The Bulletin of the Technical Committee on Data Engineering 43 (Sept. 2020), pp. 65--74.
[89]
Lora Aroyo et al. 'Crowdsourcing Subjective Tasks: The Case Study of Understanding Toxicity in Online Discussions'. In: Companion Proceedings of The 2019 World Wide Web Conference. WWW '19. New York, NY, USA: Association for Computing Machinery, May //doi.org/10.1145/3308560.3317083.
[90]
Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju. 'Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments'. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. CHI '19. New York, NY, USA: Association for Computing Machinery, May 2019, pp. 1--12. isbn: 978--1--4503--5970--2. url: https://doi.org/10.1145/3290605.3300637.
[91]
A. Kimball Romney, Susan C. Weller, and William H. Batchelder. 'Culture as Consensus: A Theory of Culture and Informant Accuracy'. In: American Anthropologist 88.2 (1986). Publisher: American Anthropological Association, Wiley, pp. 313--338. issn: 0002--7294. url: https://www.jstor.org/stable/677564.
[92]
Peter Welinder et al. 'The multidimensional wisdom of crowds'. In: Advances in neural information processing systems. Vol. 23. Curran Associates, Inc., 2010. url: https://proceedings. neurips.cc/paper_files/paper/2010/file/0f9cafd014db7a619ddb4276af0d692c-Paper.pdf.
[93]
Yan Yan et al. 'Active learning from crowds'. In: Proceedings of the 28th International Conference on International Conference on Machine Learning. ICML'11. Madison, WI, USA: Omnipress, June 2011, pp. 1161--1168. isbn: 978--1--4503-0619--5.
[94]
Chris Madge et al. 'Experiment-Driven Development of a GWAP for Marking Segments in Text'. In: Extended Abstracts Publication of the Annual Symposium on Computer-Human Interaction in Play. CHI PLAY '17 Extended Abstracts. New York, NY, USA: Association for Computing Machinery, 2017, pp. 397--404. isbn: 9781450351119. url: https://doi.org/10.1145/3130859.3131332.
Index Terms
- News Ninja: Gamified Annotation of Linguistic Bias in Online News
Recommendations
A Multidimensional Dataset Based on Crowdsourcing for Analyzing and Detecting News Bias
CIKM '20: Proceedings of the 29th ACM International Conference on Information & Knowledge ManagementThe automatic detection of bias in news articles can have a high impact on society because undiscovered news bias may influence the political opinions, social views, and emotional feelings of readers. While various analyses and approaches to news bias ...
Newsalyze: Enabling News Consumers to Understand Media Bias
JCDL '20: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020News is a central source of information for individuals to inform themselves on current topics. Knowing a news article's slant and authenticity is of crucial importance in times of "fake news," news bots, and centralization of media ownership. We ...
An Integrated Approach to Detect Media Bias in German News Articles
JCDL '20: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020Media bias may often affect individuals' opinions on reported topics. Many existing methods that aim to identify such bias forms employ individual, specialized techniques and focus only on English texts. We propose to combine the state-of-the-art in ...
Comments
Information & Contributors
Information
Published In
October 2024
1726 pages
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 15 October 2024
Published in PACMHCI Volume 8, Issue CHI PLAY
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Hanns-Seidel Foundation
- German Academic Exchange Service (DAAD)
- project XR Hub from the Bavarian State Ministry for Digital Affairs
- JST CREST Grant Japan
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 107Total Downloads
- Downloads (Last 12 months)107
- Downloads (Last 6 weeks)44
Reflects downloads up to 22 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in