research-article

Open access

Simplistic Collection and Labeling Practices Limit the Utility of Benchmark Datasets for Twitter Bot Detection

Authors:

Zachary Schutzman,

Manish Raghavan,

Philipp ZimmerAuthors Info & Claims

WWW '23: Proceedings of the ACM Web Conference 2023

Pages 3660 - 3669

https://doi.org/10.1145/3543507.3583214

Published: 30 April 2023 Publication History

All formats PDF

Abstract

Accurate bot detection is necessary for the safety and integrity of online platforms. It is also crucial for research on the influence of bots in elections, the spread of misinformation, and financial market manipulation. Platforms deploy infrastructure to flag or remove automated accounts, but their tools and data are not publicly available. Thus, the public must rely on third-party bot detection. These tools employ machine learning and often achieve near-perfect performance for classification on existing datasets, suggesting bot detection is accurate, reliable and fit for use in downstream applications. We provide evidence that this is not the case and show that high performance is attributable to limitations in dataset collection and labeling rather than sophistication of the tools. Specifically, we show that simple decision rules — shallow decision trees trained on a small number of features — achieve near-state-of-the-art performance on most available datasets and that bot detection datasets, even when combined together, do not generalize well to out-of-sample datasets. Our findings reveal that predictions are highly dependent on each dataset’s collection and labeling procedures rather than fundamental differences between bots and humans. These results have important implications for both transparency in sampling and labeling procedures and potential biases in research using existing bot detection tools for pre-processing.

References

[1]

Eiman Alothali, Motamen Salih, Kadhim Hayawi, and Hany Alashwal. 2022. Bot-MGAT: A Transfer Learning Model Based on a Multi-View Graph Attention Network to Detect Social Bots. Applied Sciences 16 (2022). https://doi.org/10.3390/app12168117

[2]

Adam Badawy, Emilio Ferrara, and Kristina Lerman. 2018. Analyzing the Digital Traces of Political Manipulation: The 2016 Russian Interference Twitter Campaign. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). 258–265. https://doi.org/10.1109/ASONAM.2018.8508646

[3]

Marco T. Bastos and Dan Mercea. 2019. The Brexit Botnet and User-Generated Hyperpartisan News. Social Science Computer Review 37, 1 (2019), 38–54. https://doi.org/10.1177/0894439317734157 arXiv:https://doi.org/10.1177/0894439317734157

[4]

Alessandro Bessi and Emilio Ferrara. 2016. Social bots distort the 2016 US Presidential election online discussion.First Monday 21, 11 (2016). https://doi.org/10.5210/fm.v21i11.7090

[5]

Christopher Bouzy. 2018. Bot sentinel, Platform developed to detect and track political bots, trollbots, and untrustworthy accounts.https://botsentinel.com/

[6]

David A. Broniatowski, Amelia M. Jamison, SiHua Qi, Lulwah AlKulaib, Tao Chen, Adrian Benton, Sandra C. Quinn, and Mark Dredze. 2018. Weaponized Health Communication: Twitter Bots and Russian Trolls Amplify the Vaccine Debate. American Journal of Public Health 108, 10 (2018), 1378–1384. https://doi.org/10.2105/AJPH.2018.304567 arXiv:https://doi.org/10.2105/AJPH.2018.304567PMID: 30138075.

[7]

Daejin Choi, Selin Chun, Hyunchul Oh, Jinyoung Han, and Ted Kwon. 2020. Rumor Propagation is Amplified by Echo Chambers in Social Media. Scientific Reports 10, 310 (2020). https://doi.org/10.1038/s41598-019-57272-3

[8]

Zi Chu, Steven Gianvecchio, Haining Wang, and Sushil Jajodia. 2012. Detecting Automation of Twitter Accounts: Are You a Human, Bot, or Cyborg¿IEEE Transactions on Dependable and Secure Computing 9, 6 (2012), 811–824. https://doi.org/10.1109/TDSC.2012.75

[9]

Stefano Cresci. 2020. A Decade of Social Bot Detection. Commun. ACM 63, 10 (sep 2020), 72–83. https://doi.org/10.1145/3409116

[10]

Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2017. The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race. In 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 963–972. https://doi.org/10.1145/3041021.3055135

[11]

Stefano Cresci, Fabrizio Lillo, Daniele Regoli, Serena Tardelli, and Maurizio Tesconi. 2019. $ FAKE: Evidence of Spam and Bot Activity in Stock Microblogs on Twitter. (2019).

[12]

Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2015. Fame for sale: Efficient detection of fake Twitter followers. Decision Support Systems 80 (2015), 56–71. https://doi.org/10.1016/j.dss.2015.09.003

[13]

Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2016. DNA-Inspired Online Behavioral Modeling and Its Application to Spambot Detection. IEEE Intelligent Systems 31, 5 (2016), 58–64. https://doi.org/10.1109/MIS.2016.29

[14]

Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2017. Exploiting digital DNA for the analysis of similarities in Twitter behaviour. In IEEE International Conference on Data Science and Advanced Analytics (DSAA). 686–695. https://doi.org/10.1109/DSAA.2017.57

[15]

Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2018. Social Fingerprinting: Detection of Spambot Groups Through DNA-Inspired Behavioral Modeling. IEEE Transactions on Dependable and Secure Computing 15, 4 (2018), 561–576. https://doi.org/10.1109/TDSC.2017.2681672

[16]

Stefano Cresci, Maurizio Tesconi, Andrea Cimino, and Felice Dell’Orletta. 2015. A Linguistically-Driven Approach to Cross-Event Damage Assessment of Natural Disasters from Social Media Messages. In Proceedings of the 24th International Conference on World Wide Web (Florence, Italy) (WWW ’15 Companion). Association for Computing Machinery, New York, NY, USA, 1195–1200. https://doi.org/10.1145/2740908.2741722

[17]

Ilias Dimitriadis, Konstantinos Georgiou, and Athena Vakali. 2021. Social Botomics: A Systematic Ensemble ML Approach for Explainable and Multi-Class Bot Detection. Applied Sciences 11, 21 (2021). https://doi.org/10.3390/app11219857

[18]

Juan Echeverria, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, and Shi Zhou. 2018. LOBO: Evaluation of Generalization Deficiencies in Twitter Bot Classifiers. In 34th Annual Computer Security Applications Conference. Association for Computing Machinery, 137–146. https://doi.org/10.1145/3274694.3274738

[19]

Tuğrulcan Elmas, Rebekah Overdorf, and Karl Aberer. 2022. Characterizing Retweet Bots: The Case of Black Market Accounts. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 16. 171–182.

[20]

Shangbin Feng, Zhaoxuan Tan, and Minnan Luo Rui L and. 2022. Heterogeneity-aware Twitter Bot Detection with Relational Graph Transformers. In AAAI Conference on Artificial Intelligence. Association for the Advancement of Artificial Intelligence, 3977–3985. https://doi.org/10.1609/aaai.v36i4.20314

[21]

Shangbin Feng, Herun Wan, Ningnan Wang, Jundong Li, and Minnan Luo. 2021. SATAR: A Self-supervised Approach to Twitter Account Representation Learning and its Application in Bot Detection. In 30th ACM International Conference on Information & Knowledge Management (CIKM). Association for Computing Machinery, 3808–3817. https://doi.org/10.1145/3459637.3481949

[22]

Shangbin Feng, Herun Wan, Ningnan Wang, Jundong Li, and Minnan Luo. 2021. TwiBot-20: A Comprehensive Twitter Bot Detection Benchmark. (2021), 4485–4494.

[23]

Shangbin Feng, Herun Wan, Ningnan Wang, and Minnan Luo. 2021. BotRGCN: Twitter Bot Detection with Relational Graph Convolutional Networks. In 2021 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining. Association for Computing Machinery, 236–239. https://doi.org/10.1145/3487351.3488336

[24]

Emilio Ferrara. 2017. Disinformation and Social Bot Operations in the Run Up to the 2017 French Presidential Election. First Monday 22, 8 (2017). https://doi.org/10.5210/fm.v22i18.8005

[25]

Emilio Ferrara. 2020. What types of COVID-19 conspiracies are populated by Twitter bots¿First Monday (05 2020). https://doi.org/10.5210/fm.v25i6.10633

[26]

Florian Gallwitz and Michael Kreil. 2021. The Rise and Fall of ’Social Bot’ Research. https://ssrn.com/abstract=3814191

[27]

Florian Gallwitz and Michael Kreil. 2022. Investigating the Validity of Botometer-based Social Bot Studies. arxiv:2207.11474 [cs.SI]

[28]

Andres Garcia-Silva, Cristian Berrio, and José Manuel Gómez-Pérez. 2019. An Empirical Study on Pre-trained Embeddings and Language Models for Bot Detection. In 4th Workshop on Representation Learning for NLP (RepL4NLP). Association for Computational Linguistics, 148–155. https://doi.org/10.18653/v1/W19-4317

[29]

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021. Datasheets for datasets. Commun. ACM 64, 12 (2021), 86–92.

[30]

Zafar Gilani, Reza Farahbakhsh, Gareth Tyson, and Jon Crowcroft. 2019. A Large-scale Behavioural Analysis of Bots and Humans on Twitter. ACM Transactions on the Web 13, 1 (2019). https://doi.org/10.1145/3298789

[31]

Zafar Gilani, Reza Farahbakhsh, Gareth Tyson, Liang Wang, and Jon Crowcroft. 2017. Of Bots and Humans (on Twitter). In IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining. Association for Computing Machinery, 349–354. https://doi.org/10.1145/3110025.3110090

[32]

Zafar Gilani, Ekaterina Kochmar, and Jon Crowcroft. 2017. Classification of Twitter Accounts into Automated Agents and Human Users. In 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). 489–496.

[33]

Zafar Gilani, Ekaterina Kochmar, and Jon Crowcroft. 2020. Classification of twitter accounts into automated agents and human users. In IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). 489–496. https://doi.org/10.1145/3110025.3110091

[34]

Salvatore Giorgi, Lyle Ungar, and H. Andrew Schwartz. 2021. Characterizing Social Spambots by their Human Traits. In The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 5148–5158. https://doi.org/10.18653/v1/2021.findings-acl.457

[35]

Sandra González-Bailón, Valeria d’Andrea, Deen Freelon, and Manlio De Domenico. 2022. The advantage of the right in social media news sharing. PNAS Nexus 1, 3 (07 2022). https://doi.org/10.1093/pnasnexus/pgac137 arXiv:https://academic.oup.com/pnasnexus/article-pdf/1/3/pgac137/45484944/pgac137.pdfpgac137.

[36]

Yuriy Gorodnichenko, Tho Pham, and Oleksandr Talavera. 2018. Social Media, Sentiment and Public Opinions: Evidence from #Brexit and #USElection. NBER Working Papers 24631. National Bureau of Economic Research, Inc. https://ideas.repec.org/p/nbr/nberwo/24631.html

[37]

Qinglang Guo, Haiyong Xie, Yangyang Li, Wen Ma, and Chao Zhang. 2022. Social Bots Detection via Fusing BERT and Graph Convolutional Networks. Symmetry 14, 1 (2022). https://doi.org/10.3390/sym14010030

[38]

Maryam Heidari and James H Jones. 2020. Using BERT to Extract Topic-Independent Sentiment Features for Social Media Bot Detection. In 11th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON). 542–547. https://doi.org/10.1109/UEMCON51285.2020.9298158

[39]

Loukas Ilias and Ioanna Roussaki. 2021. Detecting malicious activity in Twitter using deep learning techniques. Applied Soft Computing 107 (2021).

[40]

S. Mo Jang, Tieming Geng, Jo-Yun Queenie Li, Ruofan Xia, Chin-Tser Huang, Hwalbin Kim, and Jijun Tang. 2018. A computational approach for examining the roots and spreading patterns of fake news: Evolution tree analysis. Computers in Human Behavior 84 (2018), 103–113. https://doi.org/10.1016/j.chb.2018.02.032

[41]

Tobias Keller and Ulrike Klinger. 2018. Social Bots in Election Campaigns: Theoretical, Empirical, and Methodological Implications. Political Communication 36, 1 (2018), 171–189. https://doi.org/10.1080/10584609.2018.1526238

[42]

Maria Kouvela, Ilias Dimitriadis, and Athena Vakali. 2020. Bot-Detective: An Explainable Twitter Bot Detection Service with Crowdsourcing Functionalities. In Proceedings of the 12th International Conference on Management of Digital EcoSystems (Virtual Event, United Arab Emirates) (MEDES ’20). Association for Computing Machinery, New York, NY, USA, 55–63. https://doi.org/10.1145/3415958.3433075

[43]

Sneha Kudugunta and Emilio Ferrara. 2018. Deep neural networks for bot detection. Information Sciences 467 (2018), 312–322. https://doi.org/10.1016/j.ins.2018.08.019

[44]

Kyumin Lee, Brian Eoff, and James Caverlee. 2011. A Long-Term Study of Content Polluters on Twitter. In Fifth International AAAI Conference on Weblogs and Social Media. Association for the Advancement of Artificial Intelligence, 185–192.

[45]

Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 4765–4774. http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf

[46]

Linhao Luo, Xiaofeng Zhang, Xiaofei Yang, and Weihuang Yang. 2019. Deepbot: A Deep Neural Network based approach for Detecting Twitter Bots. IOP Conference Series: Materials Science and Engineering 719, 1 (2019). https://doi.org/10.1088/1757-899x/719/1/012063

[47]

Franziska Martini, Paul Samula, Tobias R Keller, and Ulrike Klinger. 2021. Bot, or not¿ Comparing three methods for detecting social bots in five political discourses. Big Data & Society 8, 2 (2021), 20539517211033566. https://doi.org/10.1177/20539517211033566 arXiv:https://doi.org/10.1177/20539517211033566

[48]

David Martín-Gutiérrez, Gustavo Hernández-Peñaloza, Alberto Belmonte Hernández, Alicia Lozano-Diez, and Federico Álvarez. 2021. A Deep Learning Approach for Robust Detection of Bots in Twitter Using Transformers. IEEE Access 9 (2021), 54591–54601. https://doi.org/10.1109/ACCESS.2021.3068659

[49]

Michele Mazza, Stefano Cresci, Marco Avvenuti, Walter Quattrociocchi, and Maurizio Tesconi. 2019. RTbust: Exploiting Temporal Patterns for Botnet Detection on Twitter. In 10th ACM Conference on Web Science. Association for Computing Machinery, 183–192. https://doi.org/10.1145/3292522.3326015

[50]

Guanyi Mou and Kyumin Lee. 2020. Malicious bot detection in online social networks: arming handcrafted features with deep learning. In International Conference on Social Informatics. Springer, 220–236.

[51]

Elon Musk. 2022. Bot Percentage Thread. https://twitter.com/elonmusk/status/1555950698252181507

[52]

Leonardo Nizzoli, Serena Tardelli, Marco Avvenuti, Stefano Cresci, Maurizio Tesconi, and Emilio Ferrara. 2020. Charting the Landscape of Online Cryptocurrency Manipulation. IEEE Access 8 (2020), 113230–113245. https://doi.org/10.1109/ACCESS.2020.3003370

[53]

Gordon Pennycook, Ziv Epstein, Mohsen Mosleh, Antonio A. Arechar, Dean Eckles, and David G. Rand. 2021. Shifting attention to accuracy can reduce misinformation online. Nature 592, 6380 (2021), 590–595. https://doi.org/10.1038/s41586-021-03344-2

[54]

Francesco Pierri, Alessandro Artoni, and Stefano Ceri. 2020. Investigating Italian disinformation spreading on Twitter in the context of 2019 European elections. PloS one 15, 1 (2020), e0227821. https://doi.org/10.1371/journal.pone.0227821

[55]

Francisco Rangel and Paolo Rosso. 2015. Overview of the 7th Author Profiling Task at PAN 2019: Bots and Gender Profiling in Twitter. In CLEF Evaluation Labs and Workshop Working Notes Papers.

[56]

Adrian Rauchfleisch and Jonas Kaiser. 2020. The False positive problem of automatic bot detection in social science research. PLOS ONE 15, 10 (10 2020), 1–20. https://doi.org/10.1371/journal.pone.0241045

[57]

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You¿": Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (San Francisco, California, USA) (KDD ’16). Association for Computing Machinery, New York, NY, USA, 1135–1144. https://doi.org/10.1145/2939672.2939778

[58]

Luigi Rovito, Lorenzo Bonin, Luca Manzoni, and Andrea De Lorenzo. 2022. An Evolutionary Computation Approach for Twitter Bot Detection. Applied Sciences 12, 12 (2022). https://doi.org/10.3390/app12125915

[59]

Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1, 5 (01 May 2019), 206–215. https://doi.org/10.1038/s42256-019-0048-x

[60]

Mohsen Sayyadiharikandeh, Onur Varol, Kai-Cheng Yang, Alessandro Flammini, and Filippo Menczer. 2020. Detection of Novel Social Bots by Ensembles of Specialized Classifiers. In 29th ACM International Conference on Information & Knowledge Management. Association for Computing Machinery, 2725–2732. https://doi.org/10.1145/3340531.3412698

[61]

Chengcheng Shao, Giovanni Luca Ciampaglia, Onur Varol, Kai-Cheng Yang, Alessandro Flammini, and Filippo Menczer. 2018. The spread of low-credibility content by social bots. Nature Communications 9, 4787 (2018), 1–9. https://doi.org/10.1038/s41467-018-06930-7

[62]

Chengcheng Shao, Pik-Mai Hui, Lei Wang, Xinwen Jiang, Alessandro Flammini, Filippo Menczer, and Giovanni Luca Ciampaglia. 2018. Anatomy of an online misinformation network. PloS one 13, 4 (2018), e0196087. https://doi.org/10.1371/journal.pone.0196087

[63]

Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. 2020. FakeNewsNet: A Data Repository with News Content, Social Context, and Spatiotemporal Information for Studying Fake News on Social Media. Big Data 8, 3 (2020), 171–188. https://doi.org/10.1089/big.2020.0062

[64]

Massimo Stella, Emilio Ferrara, and Manlio De Domenico. 2018. Bots increase exposure to negative and inflammatory content in online social systems. Proceedings of the National Academy of Sciences 115, 49 (2018), 12435–12440. https://doi.org/10.1073/pnas.1803470115

[65]

M. Buğra Torusdağ, Mucahid Kutlu, and Ali Aydın Selçuk. 2020. Are We Secure from Bots¿ Investigating Vulnerabilities of Botometer. In 2020 5th International Conference on Computer Science and Engineering (UBMK). 343–348. https://doi.org/10.1109/UBMK50275.2020.9219433

[66]

Twitter. 2021. FORM 10-K. https://www.sec.gov/Archives/edgar/data/1418091/000141809121000031/twtr-20201231.htm

[67]

Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science 359, 6380 (2018), 1146–1151. https://doi.org/10.1126/science.aap9559 arXiv:https://www.science.org/doi/pdf/10.1126/science.aap9559

[68]

Xiujuan Wang, Qianqian Zheng, Kangfeng Zheng, Yi Sui, Siwei Cao, and Yutong Shi. 2021. Detecting social media bots with variational autoencoder and k-nearest neighbor. Applied Sciences 11, 12 (2021), 5482.

[69]

Magdalena Wischnewski, Rebecca Bernemann, Thao Ngo, and Nicole Krämer. 2021. Disagree¿ You Must Be a Bot! How Beliefs Shape Twitter Profile Perceptions. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 160, 11 pages. https://doi.org/10.1145/3411764.3445109

[70]

Liang Wu, Xia Hu, Fred Morstatter, and Huan Liu. 2017. Adaptive Spammer Detection with Sparse Group Modeling. In Eleventh International AAAI Conference on Web and Social Media (ICWSM). Association for the Advancement of Artificial Intelligence, 319–326.

[71]

Harry Yaojun Yan, Kai-Cheng Yang, Filippo Menczer, and James Shanahan. 2021. Asymmetrical perceptions of partisan political bots. New Media & Society 23, 10 (2021), 3016–3037. https://doi.org/10.1177/1461444820942744 arXiv:https://doi.org/10.1177/1461444820942744

[72]

Chao Yang, Robert Harkreader, and Guofei Gu. 2013. Empirical evaluation and new design for fighting evolving twitter spammers.IEEE Transactions on Information Forensics and Security 8, 8 (2013), 1280–1293. https://doi.org/10.1109/TIFS.2013.2267732

[73]

Kai‐Cheng Yang, Onur Varol, Clayton A. Davis, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. 2019. Arming the public with artificial intelligence to counter social bots.Human Behavior and Emerging Technologies 1 (2019), 48–68. https://doi.org/10.1002/hbe2.115

[74]

Kai-Cheng Yang, Emilio Ferrara, and Filippo Menczer. 2022. Botometer 101: Social bot practicum for computational social scientists. arXiv preprint arXiv:2201.01608 (2022).

[75]

Kai-Cheng Yang, Onur Varol, Pik-Mai Hui, and Filippo Menczer. 2020. Scalable and Generalizable Social Bot Detection through Data Selection. In AAAI Conference on Artificial Intelligence. Association for the Advancement of Artificial Intelligence, 1096–1103. https://doi.org/10.1609/aaai.v34i01.5460

Cited By

Kolomeets MTushkanova ODesnitsky VVitkova LChechulin A(2024)Experimental Evaluation: Can Humans Recognise Social Media Bots?Big Data and Cognitive Computing10.3390/bdcc80300248:3(24)Online publication date: 26-Feb-2024
https://doi.org/10.3390/bdcc8030024
Liu HDas ABoltz AZhou DPinaroc DLease MLee M(2024)Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AIProceedings of the ACM on Human-Computer Interaction10.1145/36869628:CSCW2(1-44)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686962
Tse AVattam SErcolani VStetson DZurko M(2024)A Testbed for Operations in the Information EnvironmentProceedings of the 17th Cyber Security Experimentation and Test Workshop10.1145/3675741.3675751(83-90)Online publication date: 13-Aug-2024
https://dl.acm.org/doi/10.1145/3675741.3675751
Show More Cited By

Index Terms

Simplistic Collection and Labeling Practices Limit the Utility of Benchmark Datasets for Twitter Bot Detection
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
    1. Redundancy
  2. Embedded and cyber-physical systems
    1. Embedded systems
    2. Robotics
2. Networks
  1. Network properties
    1. Network reliability

Recommendations

TwiBot-20: A Comprehensive Twitter Bot Detection Benchmark
CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management

Twitter has become a vital social media platform while an ample amount of malicious Twitter bots exist and induce undesirable social effects. Successful Twitter bot detection proposals are generally supervised, which rely heavily on large-scale ...
On-Demand Bot Detection and Archival System
WWW '17 Companion: Proceedings of the 26th International Conference on World Wide Web Companion

Unusually high correlation in activities among users in social media is an indicator of bot behavior. We have developed a system, called DeBot, that identifies such bots in Twitter network. Our system reports and archives thousands of bot accounts every ...
Bot Detection in Twitter: An Overview
Flexible Query Answering Systems
Abstract
Bot detection in social media, particularly on Twitter, has become a crucial issue in recent years due to the increasing use of bots for malicious uses such as the spreading of false information in order to manipulate public opinion. In this paper,...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '23: Proceedings of the ACM Web Conference 2023

April 2023

4293 pages

ISBN:9781450394161

DOI:10.1145/3543507

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 April 2023

Check for updates

Badges

Artifacts Available / v1.1

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '23

Sponsor:

SIGWEB

WWW '23: The ACM Web Conference 2023

April 30 - May 4, 2023

TX, Austin, USA

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
1,463
Total Downloads

Downloads (Last 12 months)901
Downloads (Last 6 weeks)55

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kolomeets MTushkanova ODesnitsky VVitkova LChechulin A(2024)Experimental Evaluation: Can Humans Recognise Social Media Bots?Big Data and Cognitive Computing10.3390/bdcc80300248:3(24)Online publication date: 26-Feb-2024
https://doi.org/10.3390/bdcc8030024
Liu HDas ABoltz AZhou DPinaroc DLease MLee M(2024)Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AIProceedings of the ACM on Human-Computer Interaction10.1145/36869628:CSCW2(1-44)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686962
Tse AVattam SErcolani VStetson DZurko M(2024)A Testbed for Operations in the Information EnvironmentProceedings of the 17th Cyber Security Experimentation and Test Workshop10.1145/3675741.3675751(83-90)Online publication date: 13-Aug-2024
https://dl.acm.org/doi/10.1145/3675741.3675751
Pacheco DChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Bots, Elections, and Controversies: Twitter Insights from Brazil's Polarised ElectionsProceedings of the ACM Web Conference 202410.1145/3589334.3645651(2651-2659)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645651
Silverstein PPennington CBranney PO'Connor DLawlor EO'Brien ELynott D(2024) A registered report survey of open research practices in psychology departments in the UK and Ireland British Journal of Psychology10.1111/bjop.12700115:3(497-534)Online publication date: 22-Mar-2024
https://doi.org/10.1111/bjop.12700
Boahen ESosu ROcansey SXu QWang C(2024)ASRL: Adaptive Swarm Reinforcement Learning for Enhanced OSN Intrusion DetectionIEEE Transactions on Information Forensics and Security10.1109/TIFS.2024.348850619(10258-10272)Online publication date: 2024
https://doi.org/10.1109/TIFS.2024.3488506
Dimitriadis IDialektakis GVakali A(2024)CALEBData & Knowledge Engineering10.1016/j.datak.2023.102245149:COnline publication date: 14-Mar-2024
https://dl.acm.org/doi/10.1016/j.datak.2023.102245
Reiche SCohen SSimonov KFriedrich T(2024)Integrating higher-order relations for enhanced twitter bot detectionSocial Network Analysis and Mining10.1007/s13278-024-01372-014:1Online publication date: 21-Oct-2024
https://doi.org/10.1007/s13278-024-01372-0
Yang SLi XDu YHuang DChen XFan YWang S(2024)A hierarchical dual-view model for fake news detection guided by discriminative lexiconsInternational Journal of Machine Learning and Cybernetics10.1007/s13042-024-02322-0Online publication date: 23-Aug-2024
https://doi.org/10.1007/s13042-024-02322-0
Peng HZhang YBai XDai Q(2024)Coarse-to-fine label propagation with hybrid representation for deep semi-supervised bot detectionWireless Networks10.1007/s11276-024-03821-2Online publication date: 14-Aug-2024
https://doi.org/10.1007/s11276-024-03821-2
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten