research-article

Open access

Web Scraping of COVID-19 News Stories to Create Datasets for Sentiment and Emotion Analysis

Authors:

Poojitha Thota,

Elmasri RamezAuthors Info & Claims

PETRA '21: Proceedings of the 14th PErvasive Technologies Related to Assistive Environments Conference

Pages 306 - 314

https://doi.org/10.1145/3453892.3461333

Published: 29 June 2021 Publication History

All formats PDF

Abstract

ABSTRACT

Over the past few years, the ubiquitous usage of internet to broadcast information worldwide, has proved to be one of the best methods in making people aware about their surroundings. This has also led towards storage of vast amount of data in user interactive websites. Several news channels apart from live streaming, are using internet in such ways to convey their information. And these methods have not only benefited people to acquire regular updates but have also impacted their lives in many ways during the world awakening pandemic like COVID-19. Currently, in this pandemic situation, many leaders including federal and state government officials have shown great concern towards society by providing statements such as preventive measures, which were regularly recorded and elaborated in the form of stories by the news channels in their own websites. But, in general, the type of statement given by a leader will always affect people's opinion and behavior, which also happened during this pandemic. There were variations in number of cases of COVID-19 based upon a region leader's statements as per timeline and the people's modulating lifestyle there. This kind of relationship has motivated us towards analyzing the COVID-19 data based on sentiment and emotion involved in leader's statements, giving rise to an idea of web scraping to obtain their statements and stories from news channels. The main aim of this paper is to enlighten about different techniques and libraries used for web scraping, some challenges while designing a web scraper and finally arrive at an efficient methodology using them to extract news headlines and stories and create a new dataset that can be used in sentiment and emotion analysis. In future work, we will present the results of our experiments with the downloaded datasets.

References

[1]

SCM de S Sirisuriya. 2015. A Comparative Study on Web Scraping. In Proceedings of 8th International Research Conference, KDU., http://ir.kdu.ac.lk/handle/345/1051

Google Scholar

[2]

Daniel Glez-Pena., Analia Lourenco., Hugo Lopez-Fernandez., Miguel Reboiro-Jato., Florentino Fdez-Riverola., 2013. Web scraping technologies in an API world. Briefings in Bioinformatics, Volume 15, Issue 5, September 2014, Pages 788–797., https://doi.org/10.1093/bib/bbt026

Crossref

Google Scholar

[3]

Saurkar., Anand V., Kedar G. Pathare., and Shweta A. Gode., 2018. An Overview on Web Scraping Techniques and Tools. International Journal on Future Revolution in Computer Science & Communication Engineering 4.4 (2018): 363-367., http://www.ijfrcsce.org/download/browse/Volume_4/April_18_Volume_4_Issue_4/1524638955_25-04-2018.pdf

Google Scholar

[4]

Emil Persson., 2019. Evaluating tools and techniques for web scraping. Master's Thesis, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology

Google Scholar

[5]

Hamza moh., Salem and Manuel Mazzara., 2020. Pattern Matching-based scraping of news websites. Journal of Physics: Conference Series., https://iopscience.iop.org/volume/1742-6596/1694

Google Scholar

[6]

Hadley Wickham., RStudio. 2015. Easily Harvest (Scrape) Web Pages. https://rvest.tidyverse.org/ ., https://github.com/tidyverse/rvest

Google Scholar

[7]

John Harrison., Ju Yeong Kim., 2020. R Bindings for 'Selenium WebDriver'. http://docs.ropensci.org/RSelenium

Google Scholar

[8]

D. M. Thomas and S. Mathur., 2019. Data Analysis by Web Scraping using Python. In 3rd International conference on Electronics, Communication and Aerospace Technology (ICECA), Coimbatore, India, 2019, pp. 450-454.

Crossref

Google Scholar

[9]

Kenneth Reitz., 2015. Requests: HTTP for Humans., https://requests.readthedocs.io/en/master/

Google Scholar

[10]

Ryan Mitchell., Revised in 2018. Web Scraping with Python, 2nd Edition. Book published by O'Reilly, Inc. https://www.oreilly.com/library/view/web-scraping-with/9781491985564/

Google Scholar

[11]

Scrapy developers. Revised in 2020. Scrapy Tutorial. https://docs.scrapy.org/en/latest/intro/tutorial.html

Google Scholar

[12]

Hadley Wickham, RStudio. 2019. stringr, Simple, Consistent Wrappers for Common String Operations. https://www.rdocumentation.org/packages/stringr/versions/1.4.0

Google Scholar

Cited By

View all

Fahsyah Nurzaman AUbaidah Alexander DRombot O(2024)The Use of Data Mining to Identify Primary Teacher Qualification Demand in Indonesia2024 2nd International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT)10.1109/IDCIoT59759.2024.10468006(722-727)Online publication date: 4-Jan-2024
https://doi.org/10.1109/IDCIoT59759.2024.10468006
Valova IMladenova TKanev GHalacheva T(2023)Web Scraping - State of Art, Techniques and Approaches2023 31st National Conference with International Participation (TELECOM)10.1109/TELECOM59629.2023.10409723(1-4)Online publication date: 16-Nov-2023
https://doi.org/10.1109/TELECOM59629.2023.10409723
Tanasescu LVines ABologa AVaida C(2022)Big Data ETL Process and Its Impact on Text Mining Analysis for Employees’ ReviewsApplied Sciences10.3390/app1215750912:15(7509)Online publication date: 26-Jul-2022
https://doi.org/10.3390/app12157509
Show More Cited By

Recommendations

Web Scraping using Natural Language Processing: Exploiting Unstructured Text for Data Extraction and Analysis
Abstract
In recent years, combining web scraping techniques with Natural Language Processing (NLP) has emerged as a powerful approach to unlock deeper insights from unstructured textual data. This research study presents a detailed exploration of web ...
An End-User Pipeline for Scraping and Visualizing Semi-Structured Data over the Web
Web Engineering
Abstract
The Web is a vast source of semi-structured datasets that are made readily available to support the construction of new knowledge. Information visualization techniques have been demonstrated as a suitable alternative for allowing users to analyze ...
Sentiment analysis of COVID-19 cases in Greece using Twitter data
Abstract Background
Syndromic surveillance with the use of Internet data has been used to track and forecast epidemics for the last two decades, using different sources from social media to search engine records. More recently, studies have addressed how ...

Comments

Information & Contributors

Information

Published In

PETRA '21: Proceedings of the 14th PErvasive Technologies Related to Assistive Environments Conference

June 2021

593 pages

ISBN:9781450387927

DOI:10.1145/3453892

Conference Chair:
Fillia Makedon

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 June 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

PETRA '21

PETRA '21: The 14th PErvasive Technologies Related to Assistive Environments Conference

June 29 - July 2, 2021

Corfu, Greece

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
3,467
Total Downloads

Downloads (Last 12 months)634
Downloads (Last 6 weeks)94

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Fahsyah Nurzaman AUbaidah Alexander DRombot O(2024)The Use of Data Mining to Identify Primary Teacher Qualification Demand in Indonesia2024 2nd International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT)10.1109/IDCIoT59759.2024.10468006(722-727)Online publication date: 4-Jan-2024
https://doi.org/10.1109/IDCIoT59759.2024.10468006
Valova IMladenova TKanev GHalacheva T(2023)Web Scraping - State of Art, Techniques and Approaches2023 31st National Conference with International Participation (TELECOM)10.1109/TELECOM59629.2023.10409723(1-4)Online publication date: 16-Nov-2023
https://doi.org/10.1109/TELECOM59629.2023.10409723
Tanasescu LVines ABologa AVaida C(2022)Big Data ETL Process and Its Impact on Text Mining Analysis for Employees’ ReviewsApplied Sciences10.3390/app1215750912:15(7509)Online publication date: 26-Jul-2022
https://doi.org/10.3390/app12157509
Faria LSilva KFernandes DFernandes MSoares F(2022)Tweet and News Sentiment Indicators and the Behavior of the Brazilian Stock Market2022 IEEE 20th International Conference on Industrial Informatics (INDIN)10.1109/INDIN51773.2022.9976149(578-583)Online publication date: 25-Jul-2022
https://doi.org/10.1109/INDIN51773.2022.9976149
Sultan NAbdullah D(2022)Scraping Google Scholar Data Using Cloud Computing Techniques2022 8th International Conference on Contemporary Information Technology and Mathematics (ICCITM)10.1109/ICCITM56309.2022.10032044(14-19)Online publication date: 31-Aug-2022
https://doi.org/10.1109/ICCITM56309.2022.10032044
Silva KFaria LFernandes DFernandes MFonseca ASoares F(2022)Statistical relationship between internet data indicators and the Brazilian stock exchange2022 International Conference on Computational Science and Computational Intelligence (CSCI)10.1109/CSCI58124.2022.00127(695-700)Online publication date: Dec-2022
https://doi.org/10.1109/CSCI58124.2022.00127
Christian JValiveti SJain S(2022)Profiling Cyber Crimes from News Portals Using Web ScrapingFuturistic Trends in Networks and Computing Technologies10.1007/978-981-19-5037-7_72(1007-1016)Online publication date: 16-Nov-2022
https://doi.org/10.1007/978-981-19-5037-7_72

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Recommendations

Web Scraping using Natural Language Processing: Exploiting Unstructured Text for Data Extraction and Analysis

An End-User Pipeline for Scraping and Visualizing Semi-Structured Data over the Web

Sentiment analysis of COVID-19 cases in Greece using Twitter data

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

HTML Format

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations