Measuring Hate: : Does a Definition Affect Self-Reported Levels of Perpetration and Exposure to Online Hate in Surveys?
Authors: 
James Hawdon, Ashley Reichelmann, Matthew Costello, Vicente J. Llorent, Pekka Räsänen, Izabela Zych, Atte Oksanen, Catherine BlayaAuthors Info & Claims
James Hawdon, Ashley Reichelmann, Matthew Costello, Vicente J. Llorent, Pekka Räsänen, Izabela Zych, Atte Oksanen, Catherine BlayaAuthors Info & ClaimsPages 812 - 831
Abstract
The purpose of this research is to test the validity of commonly used measures of exposure to and production of online extremism. Specifically, we investigate if a definition of hate influences survey responses about the production of and exposure to online hate. To explore the effects of a definition, we used a split experimental design on a sample of 18 to 25-year-old Americans where half of the respondents were exposed to the European Union’s definition of hate speech and the other half were not. Then, all respondents completed a survey with commonly used items measuring exposure to and perpetration of online hate. The results reveal that providing a definition affects self-reported levels of exposure and perpetration, but the effects are dependent on race. The findings provide evidence that survey responses about online hate may be conditioned by social desirability and framing biases. The findings that group differences exist in how questions about hate are interpreted when definitions of it are not provided mean we must be careful when using measures that try to capture exposure to and the production of hate. While more research is needed, we recommend providing a clear, unambiguous definition when using surveys to measure online hate.
References
[1]
Anne E. Casey Foundation (February 14, 2021). Social Issues that Matter to Generation Z. Retreived from. https://www.aecf.org/blog/generation-z-social-issues
[2]
Averdijk M. and Elffers H. (2012). The discrepancy between survey-based victim accounts and police reports revisited. International Review of Victimology, 18(2), 91–107. https://doi.org/10.1177/0269758011432955
[3]
Belli R. F., Traugott M. W., and Beckmann M. N. (2001). What leads to voting overreports? Contrasts of overreporters to validated votes and admitted nonvoters in the American national election studies. J. Off. Stat, 21(3), 287–308.
[4]
Bernatzky C., Costello M., and Hawdon J. (2022). Who produces online hate? An Examination of the effects of self-control, social structure, & social Learning. American Journal of Criminal Justice, 47(3), 421–440. https://doi.org/10.1007/s12103-020-09597-3
[5]
Bhutkar G., Raghvani V., and Juikar S. (2021). User survey about exposure of hate speech among Instagram users in India. International Journal Computer Applications, 183(19), 24–29. https://doi.org/10.5120/ijca2021921536
[6]
Blankenship A. (1942). Psychological difficulties in measuring consumer preference. Journal of Marketing, 6(4, part 2), 66–75. https://doi.org/10.1177/002224294200600420.1
[7]
Blaya C., Audrin C., and Skrzypiec G. (2020). School bullying, perpetration, and cyberhate: Overlapping issues. Contemporary School Psychology, Advanced Online Publication.
[8]
Brenner P. S. and DeLamater J. (2016). Measurement directiveness as a cause of response bias: Evidence from two survey experiments. Sociological Methods & Research, 45(2), 348–371. https://doi.org/10.1177/0049124114558630
[9]
Celuch M., Oksanen A., Räsänen P., Costello M., Blaya C., Zych I., Llorent V., Reichelmann A., and Hawdon J. (2022). Factors associated with online hate acceptance: A cross-national six-country study among young adults. International Journal of Environmental Research and Public Health, 19(1), 534. https://doi.org/10.3390/ijerph19010534
[10]
Costello M. and Hawdon J. (2018). Who are the online extremists among us? Sociodemographic characteristics, social networking, and online experiences of those who produce online hate materials. Violence and Gender, 5(1), 55–60. https://doi.org/10.1089/vio.2017.0048
[11]
Costello M., Hawdon J., Bernatzky C., and Mendes K. (2019). Social group identity and perceptions of online hate. Sociological Inquiry, 89(3), 427–452. https://doi.org/10.1111/soin.12274
[12]
Costello M., Hawdon J., Ratliff T., and Grantham T. (2016). Who views online extremism? Individual attributes leading to exposure. Computers in Human Behavior, 63(3), 311–320. https://doi.org/10.1016/j.chb.2016.05.033
[13]
Costello M., Restifo S. J., and Hawdon J. (2021). Viewing anti-immigrant hate online: An application of routine activity and Social Structure-Social Learning Theory. Computers in Human Behavior, 124(2), 106927. https://doi.org/10.1016/j.chb.2021.106927
[14]
Costello M., Rukus J., and Hawdon J. (2018). We don’t like your type around here: Regional and residential differences in exposure to online hate material targeting sexuality. Deviant Behavior, 40(3), 385–401. https://doi.org/10.1080/01639625.2018.1426266
[15]
Couper M. P. (2000). Web surveys: A review of issues and approaches. Public Opinion Quarterly, 64(4), 464–494. https://doi.org/10.1086/318641
[16]
Couper M. P., Conrad F. G., and Tourangeau R. (2007). Visual context effects in web surveys. Public Opinion Quarterly, 71(4), 623–634. https://doi.org/10.1093/poq/nfm044
[17]
Couper M. P., Tourangeau R., and Kenyon K. (2004). Picture this! Exploring visual effects in web surveys. Public Opinion Quarterly, 68(2), 255–266. https://doi.org/10.1093/poq/nfh013
[18]
Couper M. P., Traugott M. W., and Lamias M. J. (2001). Web survey design and administration. Public Opinion Quarterly, 65(2), 230–253. https://doi.org/10.1086/322199
[19]
Cowan G. and Mettrick J. (2002). The effects of target variables and settting on perceptions of hate speech1. Journal of Applied Social Psychology, 32(2), 277–299. https://doi.org/10.1111/j.1559-1816.2002.tb00213.x
[20]
Dillman D. (1978). Mail and telephone surveys: The total design method. Wiley & Sons.
[21]
Dynata. (2022). The foundation of accurate insightsand better decision-making. https://www.dynata.com/dynatas-world-class-quality/.
[22]
Fan W. and Yan Z. (2010). Factors affecting response rates of the web survey: A systematic review. Computers in Human Behavior, 26(2), 132–139. https://doi.org/10.1016/j.chb.2009.10.015
[23]
Farrington D. P. (2005). Childhood origins of antisocial behavior. Clinical Psychology & Psychotherapy: An International Journal of Theory & Practice, 12(3), 177–190. https://doi.org/10.1002/cpp.448
[24]
Federal Bureau of Investigation (2011). Domestic terrorism: Focus on militia extremism. https://www.fbi.gov/news/stories/2011/september/militia_092211
[25]
Feldt L. S. and Brennan R. L. (1989). Reliability. In Linn R. L. (Ed.), Educational measurement (3rd ed., pp. 105–146). Macmillan.
[26]
Foxman A. H. and Wolf C. (2013). Viral hate: Containing its spread on the internet. Macmillan.
[27]
Franklin R. A. (2010). The hate directory. Retrieved from. http://www.hatedirectory.com/hatedir.pdf
[28]
Freilich J., Belli R., and Chermak S. (2011). United States extremist crime database (ECDB) 1990-2010. START: National Consortium for the Study of Terrorism and Responses to Terrorism. Retrieved 7 August 2018).
[29]
Frissen T. (2021). Internet, the great radicalizer? Exploring relationships between seeking for online extremist materials and cognitive radicalization in young adults. Computers in Human Behavior, 114, 106549. https://doi.org/10.1016/j.chb.2020.106549
[30]
Furnham A. (1986). Response bias, social desirability and dissimulation. Personality and Individual Differences, 7(3), 385–400. https://doi.org/10.1016/0191-8869(86)90014-0
[31]
Grewenig E., Lergetporer P., Simon L., Werner K., and Woessmann L. (2018). Can online surveys represent the entire population? IZA discussion papers, No. 11799. Institute of Labor Economics (IZA).
[32]
Hassan G., Brouillette-Alarie S., Alava S., Frau-Meigs D., Lavoie L., Fetiu A., Varela W., Borokhovski E., Venkatesh V., Rousseau C., and Sieckelinck S. (2018). Exposure to extremist online content could lead to violent radicalization: A systematic review of empirical evidence. International Journal of Developmental Science, 12(1-2), 71–88. https://doi.org/10.3233/dev-170233
[33]
Hawdon J. (2017). January 24). Perpetrators and victims of online extremism: Status and vulnerability [presentation]. Les jeunes et l’incitation à la haine sur internet: Victimes, témoins, agresseurs? Comparaisons internationals. In International Conference “The young people and cyberhate: victims, witnesses, perpetrators? International domparisons.” Nice, Théâtre -. France: Grand Château - Campus Valrose - Université Nice Sophia Antipolis.
[34]
Hawdon J., Bernatzky C., and Costello M. (2019). Cyber-routines, political attitudes, and exposure to violence-advocating online extremism. Social Forces, 98(1), 329–354. https://doi.org/10.1093/sf/soy115
[35]
Hawdon J. and Costello M. (2022). Confronting online extremism: Strategies, promises, and pitfalls. In Perry B., Gruenwald J., and Scrivens R. (Eds.), Right-wing extremism in Canada and the United States (pp. 469–489). Palgrave Macmillan. https://doi.org/10.1007/978-3-030-99804-2_18
[36]
Hawdon J., Oksanen A., and Räsänen P. (2014). Victims of online hate groups: American youth’s exposure to online hate speech. In Hawdon J., Ryan J., and Lucht M. (Eds.), The causes and consequences of group violence: From bullies to terrorists (p. 165). Lexington Books.
[37]
Hawdon J., Oksanen A., and Räsänen P. (2017). Exposure to online hate in four nations: A cross-national consideration. Deviant Behavior, 38(3), 254–266. https://doi.org/10.1080/01639625.2016.1196985
[38]
Heckert D. A. and Gondolf E. W. (2000). Assessing assault self-reports by batterer program participants and their partners. Journal of Family Violence, 15(2), 181–197. https://doi.org/10.1023/a:1007594928605
[39]
Hollewell G. F. and Longpré N. (2022). Radicalization in the social media Era: Understanding the relationship between self-radicalization and the internet. International Journal of Offender Therapy and Comparative Criminology, 66(8), 896–913. https://doi.org/10.1177/0306624X211028771
[40]
Johnson T. J. and van de Vijver F. J. (2002). Social desirability in cross-cultural research. In Harness J., van de Vijver F. J., and Mohler P. (Eds.), Cross-cultural survey methods (pp. 193–202). Wiley.
[41]
Kalton G. and Schuman H. (1982). The Effect of the Question on Survey Responses: A Review(PDFSeries A (General). Journal of the Royal Statistical Society. 145(1), 42–73. https://doi.org/10.2307/2981421.hdl:2027.42/146916
[42]
Keipi T., Koiranen I., Koivula A., and Räsänen P. (2018). Assessing the social media landscape: Online relational use-purposes and life satisfaction among Finns. First Monday.
[43]
Koehler D. (2014). The radical online: Individual radicalization processes and the role of the Internet. Journal for Deradicalization, 12(1), 116–134.
[44]
Lavrakas P. J. (2008). Encyclopedia of survey research methods (pp. 664-665). Sage Publications, Inc
[45]
Lehdonvirta V., Oksanen A., Räsänen P., and Blank G. (2021). Social media, web, and panel surveys: Using non-probability samples in social and policy research. Policy & Internet, 13(1), 134–155. https://doi.org/10.1002/poi3.238
[46]
Manfreda K. L., Bosnjak M., Berzelak J., Haas I., and Vehovar V. (2008). Web surveys versus other survey modes: A meta-analysis comparing response rates. International Journal of Market Research, 50(1), 79–104. https://doi.org/10.1177/147078530805000107
[47]
Marquis K. H., Marquis M. S., and Polich J. M. (1986). Response bias and reliability in sensitive topic surveys. Journal of the American Statistical Association, 81(394), 381–389. https://doi.org/10.1080/01621459.1986.10478282
[48]
Mason R., Carlson J. E., and Tourangeau R. (1994). Contrast effects and subtraction in part- whole questions. Public Opinion Quarterly, 58(4), 569–578. https://doi.org/10.1086/269448
[49]
Mathew B., Dutt R., Goyal P., and Mukherjee A. (2019, June). Spread of hate speech in online social media [conference proceeding]. In 11th ACM Conference on Web Science (WebSci ’19). June, 2019. https://doi.org/10.1145/3292522.3326034
[50]
McCurdy E. (2021). Radicalizing Online: An overview of the role the internet plays in radicalization into extremism. MacEwan University Student eJournal, 5(1), 273. https://doi.org/10.31542/muse.v5i1.2008
[51]
Miller-Idriss C. (2022). Hate in the homeland. In Hate in the homeland. Princeton University Press. https://doi.org/10.2307/j.ctv20dsbtf
[52]
Mohajan H. K. (2017). Two criteria for good measurements in research: Validity and reliability, Economic Series, 17(4), 59–82. https://doi.org/10.26458/1746
[53]
Mondal M., Silva L. A., and Benevenuto F. (2017). A measurement study of hate speech in social media [conference proceedings]. In 28th ACM Conference on Hypertext and Social Media, Prague, Czech Republic, July 4-7, 2017. https://doi.org/10.1145/3078714.3078723
[54]
Näher A. F. and Krumpal I. (2012). Asking sensitive questions: The impact of forgiving wording and question context on social desirability bias. Quality and Quantity, 46(5), 1601–1616.
[55]
Näsi M., Räsänen P., Hawdon J., Holkeri E., and Oksanen A. (2015). Exposure to online hate material and social trust among Finnish youth. Information, Technology and People, 28(3), 607–622.
[56]
Nederhof A. J. (1985). Methods of coping with social desirability bias: A review. European Journal of Social Psychology, 15(3), 263–280. https://doi.org/10.1002/ejsp.2420150303
[57]
Obermaier M. and Schmuck D. (2022). Youths as targets: Factors of online hate speech victimization among adolescents and young adults. Journal of Computer-Mediated Communication, 27(4), zmac012. https://doi.org/10.1093/jcmc/zmac012
[58]
Oksanen A., Hawdon J., Holkeri E., Näsi M, and Räsänen P. (2014). Exposure to online hate among young social media users. Sociological Studies of Children and Youth, 18, 253–273. https://doi.org/10.1108/s1537-466120140000018021
[59]
Parker K., Graf N., and Igielnik R. (2019, January 17). Generation Z looks a lot more like Millennials on key social and political issues. Pew Research Center. https://www.pewresearch.org/social-trends/2019/01/17/generation-z-looks-a-lot-like-millennials-on-key-social-and-political-issues/
[60]
Perry B. (2000). “Button-down Terror”: The metamorphosis of the hate movement. Sociological Focus, 33(2), 113–131. https://doi.org/10.1080/00380237.2000.10571161
[61]
Potok M. (2017). The Year in Hate and extremism. Intelligence report, southern poverty Law center. Retrieved. https://www.splcenter.org/fighting-hate/intelligence-report/2018/year-hate-and-extremism
[62]
Räsänen P., Hawdon J., Holkeri E., Keipi M., Näsi T., and Oksanen A. (2016). Targets of online hate: Examining determinants of victimization among young Finnish Facebook users. Violence & Victims, 31(4), 708–726. https://doi.org/10.1891/0886-6708.VV-D-14-00079
[63]
Reichelmann A., Hawdon J., Costello M., Ryan J., Blaya C., Llorent V., Oksanen A., Räsänen P., and Zych I. (2021). Hate knows No boundaries: Online hate in six nations. Deviant Behavior, 42(9), 1100–1111. https://doi.org/10.1080/01639625.2020.1722337
[64]
Scherpenzeel A. C. (2018). How representative are online panels? Problems of coverage and selection and possible solutions. In Social and behavioral research and the internet (pp. 105–132). Routledge. https://doi.org/10.4324/9780203844922-5
[65]
Schwarz N. and Bles H. (1992). Scandals and the public trust in politicians: Assimilation and contrast effects. Personal. Soc. Psychol. Bull, 18(5), 574–579. https://doi.org/10.1177/0146167292185007
[66]
Schwarz N. and Bless H. (2007). Mental construal processes: The inclusion/exclusion model. In Stapel D. A. and Suls J. (Eds.), Assimilation and contrast in social psychology (pp. 119–141). Psychology Press.
[67]
Sedgwick D., Hawdon J., Räsänen P., and Koivula A. (2022). The role of collaboration in complying with COVID-19 health protective behaviors: A cross-national study. Administration & Society, 54(1), 29–56. https://doi.org/10.1177/00953997211012418
[68]
Shropshire K., Witte J. C., and Hawdon J. (2009). Web-survey design: Balancing measurement, response, and topical interest. Sociological Methods & Research, Advanced Online Publication.
[69]
Simmons A. D. and Bobo L. D. (2015). Can non-full-probability internet surveys yield useful data? A comparison with full-probability face-to-face surveys in the domain of race and social inequality attitudes. Sociological Methodology, 45(1), 357–387. https://doi.org/10.1177/0081175015570096
[70]
Subrahmanyam K. and Šmahel D. (2011). Constructing identity online: Identity exploration and self-presentation. In Digital youth (pp. 59–80). Springer. https://doi.org/10.1007/978-1-4419-6278-2_4
[71]
The New America Foundation International Security Program. (2015). Homegrown extremists. New America. http://securitydata.newamerica.net/extremists/deadly-attacks.html
[72]
The Southern Poverty Law Center (2021). The year in hate and extremism: 2020. Available at: https://www.splcenter.org/sites/default/files/yih_2020-21_final.pdf
[73]
Timmons S., McGinnity F., Belton C., Barjaková M., and Lunn P. (2021). It depends on how you ask: Measuring bias in population surveys of compliance with COVID-19 public health guidance. Journal of Epidemiology & Community Health, 75(4), 387–389. https://doi.org/10.1136/jech-2020-215256
[74]
Tourangeau R., Couper M. P., and Conrad F. (2004). Spacing, position, and order: Interpretive heuristics for visual features of survey questions. Public Opinion Quarterly, 68(3), 368–393. https://doi.org/10.1093/poq/nfh035
[75]
Tourangeau R. and Rasinski K. A. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103(3), 299–314. https://doi.org/10.1037/0033-2909.103.3.299
[76]
Tourangeau R., Rasinski K. A., and Bradburn N. (1991). Measuring happiness in surveys: A test of the subtraction hypothesis. Public Opinion Quarterly, 55(2), 255–266. https://doi.org/10.1086/269256
[77]
Tourangeau R., Rasinski K. A., Bradburn N., and D'Andrade R. (1989). American association for public opinion research. Public Opinion Quarterly, 53(4), 495–524. https://doi.org/10.1086/269169
[78]
Tourangeau R. and Smith T. W. (1996). Asking sensitive questions: The impact of data collection mode, question format, and question context. Public Opinion Quarterly, 60(2), 275–304. https://doi.org/10.1086/297751
[79]
Tourangeau R. and Yan T. (2007). Sensitive questions in surveys. Psychological Bulletin, 133(5), 859–883. https://doi.org/10.1037/0033-2909.133.5.859
[80]
Tynes B. (2006). Children, adolescents, and the culture of hate online. In Dowd N., Singer D., and Wilson R. F. (Eds.), Handbook of children, culture, and violence (pp. 267–289). Sage.
[81]
Tynes B., Reynolds L., and Greenfield P. M. (2004). Adolescence, race, and ethnicity on the internet: A comparison of discourse in monitored vs. unmonitored chat rooms. Journal of Applied Developmental Psychology, 25(6), 667–684. https://doi.org/10.1016/j.appdev.2004.09.003
[82]
Van de Mortel T. F. (2008). Faking it: Social desirability response bias in self-report research. Australian Journal of Advanced Nursing, 25(4), 40–48.
[83]
Wachs S., Wright M. F., and Vazsonyi A. T. (2019). Understanding the overlap between cyberbullying and cyberhate perpetration: Moderating effects of toxic online disinhibition. Criminal Behaviour and Mental Health, 29(3), 179–188. https://doi.org/10.1002/cbm.2116
[84]
Weinberg J. D., Freese J., and McElhattan D. (2014). Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsourced-recruited sample. Sociological Science, 1, 292–310. https://doi.org/10.15195/v1.a19
[85]
Witte J. C., Pargas R. P., Mobley C., and Hawdon J. (2004). Instrument effects of images in web surveys: A research note. Social Science Computer Review, 22(3), 363–369. https://doi.org/10.1177/0894439304264531
[86]
Wright M. (2008). Technology & terrorism: How the internet facilitates radicalization. Forensic Examiner, 17(4), 278.
Recommendations
Measuring Extremism: Validating an Alt-Right Twitter Accounts Dataset
Intelligent Data Engineering and Automated Learning – IDEAL 2018AbstractTwitter is one of the most commonly used Online Social Networks in the world and it has consequently attracted considerable attention from different political groups attempting to gain influence. Among these groups is the alt-right; a modern far-...
Measuring and Characterizing Hate Speech on News Websites
WebSci '20: Proceedings of the 12th ACM Conference on Web ScienceThe Web has become the main source for news acquisition. At the same time, news discussion has become more social: users can post comments on news articles or discuss news articles on other platforms like Reddit. These features empower and enable ...
Contagion dynamics of extremist propaganda in social networks
Recent terrorist attacks carried out on behalf of ISIS on American and European soil by lone wolf attackers or sleeper cells remind us of the importance of understanding the dynamics of radicalization mediated by social media communication channels. In ...
Comments
Information & Contributors
Information
Published In
© The Author(s) 2023.
Publisher
Sage Publications, Inc.
United States
Publication History
Published: 01 June 2024
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 01 Feb 2025