skip to main content
research-article

OutdoorSent: Sentiment Analysis of Urban Outdoor Images by Using Semantic and Deep Features

Published: 21 April 2020 Publication History

Abstract

Opinion mining in outdoor images posted by users during different activities can provide valuable information to better understand urban areas. In this regard, we propose a framework to classify the sentiment of outdoor images shared by users on social networks. We compare the performance of state-of-the-art ConvNet architectures and one specifically designed for sentiment analysis. We also evaluate how the merging of deep features and semantic information derived from the scene attributes can improve classification and cross-dataset generalization performance. The evaluation explores a novel dataset—namely, OutdoorSent—and other publicly available datasets. We observe that the incorporation of knowledge about semantic attributes improves the accuracy of all ConvNet architectures studied. Besides, we found that exploring only images related to the context of the study—outdoor, in our case—is recommended, i.e., indoor images were not significantly helpful. Furthermore, we demonstrated the applicability of our results in the United States city of Chicago, Illinois, showing that they can help to improve the knowledge of subjective characteristics of different areas of the city. For instance, particular areas of the city tend to concentrate more images of a specific class of sentiment, which are also correlated with median income, opening up opportunities in different fields.

References

[1]
Unaiza Ahsan, Munmun De Choudhury, and Irfan Essa. 2017. Towards using visual attributes to infer image sentiment of social events. In Proceedings of the International Joint Conference on Neural Networks (IJCNN’17). IEEE, 1372--1379.
[2]
Oscar Araque, Ignacio Corcuera-Platas, J. Fernando Sánchez-Rada, and Carlos A. Iglesias. 2017. Enhancing deep learning sentiment analysis with ensemble techniques in social applications. Exp. Syst. Applic. 77 (2017), 236--246.
[3]
Ramy Baly, Roula Hobeica, Hazem Hajj, Wassim El-Hajj, Khaled Bashir Shaban, and Ahmad Al-Sallab. 2016. A meta-framework for modeling the human reading process in sentiment analysis. ACM Trans. Inf. Syst. 35, 1, Article 7 (Aug. 2016), 21 pages.
[4]
Fabrício Benevenuto, Matheus Araújo, and Filipe Ribeiro. 2015. Sentiment analysis methods for social media. In Proceedings of the Brazilian Symposium on Multimedia and the Web (WebMedia’15). ACM, 11--11.
[5]
Damian Borth, Rongrong Ji, Tao Chen, Thomas Breuel, and Shih-Fu Chang. 2013. Large-scale visual sentiment ontology and detectors using adjective noun pairs. In Proceedings of the ACM International Conference on Multimedia. ACM, 223--232.
[6]
Guoyong Cai and Binbin Xia. 2015. Convolutional neural networks for multimedia sentiment analysis. In Natural Language Processing and Chinese Computing. Springer International Publishing, Cham, 159--167.
[7]
Víctor Campos, Brendan Jou, and Xavier Giró i Nieto. 2017. From pixels to sentiment: Fine-tuning CNNs for visual sentiment prediction. Image Vis. Comput. 65 (2017), 15--22.
[8]
Fuhai Chen, Rongrong Ji, Jinsong Su, Donglin Cao, and Yue Gao. 2018. Predicting microblog sentiments via weakly supervised multimodal deep learning. IEEE Trans. Multim. 20, 4 (Apr. 2018), 997--1007.
[9]
Tao Chen, Damian Borth, Trevor Darrell, and Shih-Fu Chang. 2014. DeepSentiBank: Visual sentiment concept classification with deep convolutional neural networks. CoRR abs/1410.8586 (2014), 1--7.
[10]
Yu-Hsiu Chen, Ting-Hsuan Chao, Sheng-Yi Bai, Yen-Liang Lin, Wen-Chin Chen, and Winston H. Hsu. 2015. Filter-invariant image classification on social media photos. In Proceedings of the ACM International Conference on Multimedia (MM’15). ACM, 855--858.
[11]
Justin Cranshaw, Raz Schwartz, Jason I. Hong, and Norman M. Sadeh. 2012. The livehoods project: Utilizing social media to understand the dynamics of a city. In Proceedings of the International AAAI Conference on Weblogs and Social Media. AAAI Press, 1--8.
[12]
CrowdFlower. 2015. Image Sentiment Polarity Classification (Dataset). CrowdFlower. Retrieved from https://data.world/crowdflower/image-sentiment-polarity.
[13]
Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. 2013. Predicting depression via social media. In Proceedings of the International AAAI Conference on Weblogs and Social Media. AAAI Press, 1--10.
[14]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 248--255.
[15]
Abhinav Dhall, Roland Goecke, and Tom Gedeon. 2015. Automatic group happiness intensity analysis. IEEE Trans. Affect. Comput. 6, 1 (Jan. 2015), 13--26.
[16]
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the International Conference on Knowledge Discovery and Data Mining. AAAI Press, 226--231. Retrieved from http://dl.acm.org/citation.cfm?id=3001460.3001507.
[17]
Fuhai Chen, Yue Gao, Donglin Cao, and Rongrong Ji. 2015. Multimodal hypergraph learning for microblog sentiment prediction. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME’15). IEEE, 1--6.
[18]
Scott A. Golder and Michael W. Macy. 2011. Diurnal and seasonal mood vary with work, sleep, and daylength across diverse cultures. Science 333, 6051 (2011), 1878--1881.
[19]
Alan Hanjalic, Christoph Kofler, and Martha Larson. 2012. Intent and its discontents: The user at the wheel of the online video search engine. In Proceedings of the ACM International Conference on Multimedia. ACM, 1239--1248.
[20]
K. He, X. Zhang, S. Ren, and J. Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). IEEE, 770--778.
[21]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput. 9, 8 (1997), 1735--1780.
[22]
Steven Hoffman, Renu Sharma, and Arun Ross. 2018. Convolutional neural networks for iris presentation attack detection: Toward cross-dataset and cross-sensor generalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18) Workshops. IEEE, 1620--1628.
[23]
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). IEEE, 4700--4708.
[24]
Minlie Huang, Qiao Qian, and Xiaoyan Zhu. 2017. Encoding syntactic knowledge in neural networks for sentiment classification. ACM Trans. Inf. Syst. 35, 3, Article 26 (June 2017), 27 pages.
[25]
Rongrong Ji, Donglin Cao, Yiyi Zhou, and Fuhai Chen. 2016. Survey of visual sentiment prediction for social media analysis. Front. Comput. Sci. 10, 4 (01 Aug. 2016), 602--611.
[26]
Jia Jia, Sen Wu, Xiaohui Wang, Peiyun Hu, Lianhong Cai, and Jie Tang. 2012. Can we understand Van Gogh’s mood?: Learning to infer affects from images in social networks. In Proceedings of the ACM International Conference on Multimedia. ACM, 857--860.
[27]
M. Katsurai and S. Satoh. 2016. Image sentiment analysis using latent correlations among visual, textual, and sentiment views. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’16). IEEE, 2837--2841.
[28]
George L. Kelling and Catherine M. Coles. 1997. Fixing Broken Windows: Restoring Order and Reducing Crime in Our Communities. Simon and Schuster, New York.
[29]
Adam D. I. Kramer, Jamie E. Guillory, and Jeffrey T. Hancock. 2014. Experimental evidence of massive-scale emotional contagion through social networks. Proc. Nat. Acad. Sci. 111, 24 (2014), 8788--8790.
[30]
Géraud Le Falher, Aristides Gionis, and Michael Mathioudakis. 2015. Where is the Soho of Rome? Measures and algorithms for finding similar neighborhoods in cities. In Proceedings of the International AAAI Conference on Weblogs and Social Media. AAAI Press, 1--23.
[31]
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (May 2015), 436--444. Retrieved from http://dx.doi.org/10.1038/nature14539.
[32]
Bing Li, Songhe Feng, Weihua Xiong, and Weiming Hu. 2012. Scaring or pleasing: Exploit emotional impact of an image. In Proceedings of the ACM International Conference on Multimedia. ACM, 1365--1366.
[33]
Dazhen Lin, Lingxiao Li, Donglin Cao, Yanping Lv, and Xiao Ke. 2018. Multi-modality weakly labeled sentiment learning based on explicit emotion signal for Chinese microblog. Neurocomputing 272 (2018), 258--269.
[34]
Jana Machajdik and Allan Hanbury. 2010. Affective image classification using features inspired by psychology and art theory. In Proceedings of the ACM International Conference on Multimedia. ACM, 83--92.
[35]
Nicolas Maisonneuve, Matthias Stevens, Maria E. Niessen, and Luc Steels. 2009. NoiseTube: Measuring and mapping noise pollution with mobile phones. In Information Technologies in Environmental Engineering. Springer, 215--228.
[36]
Stuart E. Middleton and Vadims Krivcovs. 2016. Geoparsing and geosemantics for social media: Spatiotemporal grounding of content propagating rumors to support trust and veracity analysis during breaking news. ACM Trans. Inf. Syst. 34, 3, Article 16 (Apr. 2016), 26 pages.
[37]
Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of the Conference of the International Speech Communication Association (INTERSPEECH’10), Takao Kobayashi, Keikichi Hirose, and Satoshi Nakamura (Eds.). ISCA, 1045--1048. Retrieved from http://dblp.uni-trier.de/db/conf/interspeech/interspeech2010.html#MikolovKBCK10.
[38]
Willi Mueller, Thiago H. Silva, Jussara M. Almeida, and Antonio A. F. Loureiro. 2017. Gender matters! Analyzing global cultural gender preferences for venues using social sensing. EPJ Data Sci. 6, 1 (2017), 5.
[39]
Alessandro Ortis, Giovanni M. Farinella, Giovanni Torrisi, and Sebastiano Battiato. 2018. Visual sentiment analysis based on objective text description of images. In Proceedings of the International Conference on Content-based Multimedia Indexing (CBMI’18). IEEE, 1--6.
[40]
George Parrett. 2016. 3.5 Million Photos Shared Every Minute in 2016. Deloitte. Retrieved from https://goo.gl/uwF81P.
[41]
Genevieve Patterson and James Hays. 2012. SUN attribute database: Discovering, annotating, and recognizing scene attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). IEEE, 2751--2758.
[42]
Genevieve Patterson, Chen Xu, Hang Su, and James Hays. 2014. The SUN attribute database: Beyond categories for deeper scene understanding. Int. J. Comput. Vis. 108, 1--2 (2014), 59--81.
[43]
Soujanya Poria, Haiyun Peng, Amir Hussain, Newton Howard, and Erik Cambria. 2017. Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis. Neurocomputing 261 (2017), 217--230.
[44]
Daniele Quercia, Rossano Schifanella, and Luca Maria Aiello. 2014. The shortest path to happiness: Recommending beautiful, quiet, and happy routes in the city. In Proceedings of the ACM Conference on Hypertext and Social Media. ACM, 116--125.
[45]
Daniele Quercia, Rossano Schifanella, Luca Maria Aiello, and Kate McLean. 2015. Smelly maps: The digital life of urban smellscapes. In Proceedings of the International Conference on Web and Social Media (ICWSM’15), Vol. 1. AAAI Press, 327--336.
[46]
Joseph Redmon and Ali Farhadi. 2017. YOLO9000: Better, faster, stronger. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). IEEE, 7263--7271.
[47]
Filipe N. Ribeiro, Matheus Araújo, Pollyanna Gonçalves, Marcos André Gonçalves, and Fabrício Benevenuto. 2016. SentiBench—A benchmark comparison of state-of-the-practice sentiment analysis methods. EPJ Data Sci. 5, 1 (2016), 1--29.
[48]
Darshan Santani, Salvador Ruiz-Correa, and Daniel Gatica-Perez. 2018. Looking south: Learning urban perception in developing cities. ACM Trans. Soc. Comput. 1, 3, Article 13 (Dec. 2018), 23 pages.
[49]
Frances A. Santos, Thiago H. Silva, Antonio A. F. Loureiro, and Leandro A. Villas. 2018. Uncovering the perception of urban outdoor areas expressed in social media. In Proceedings of the International Conference on Web Intelligence (WI’18), Vol. 1. IEEE/WIC/ACM, 120--127.
[50]
Stefan Siersdorfer, Enrico Minack, Fan Deng, and Jonathon Hare. 2010. Analyzing and predicting sentiment of images on the social web. In Proceedings of the ACM International Conference on Multimedia. ACM, 715--718.
[51]
Thiago H. Silva, Pedro O. S. Vaz de Melo, Jussara M. Almeida, Mirco Musolesi, and Antonio A. F. Loureiro. 2017. A large-scale study of cultural differences using urban data about eating and drinking preferences. Inf. Syst. 72, Supplement C (2017), 95--116.
[52]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014), 1--14.
[53]
Kaikai Song, Ting Yao, Qiang Ling, and Tao Mei. 2018. Boosting image sentiment analysis with visual attention. Neurocomputing 312 (2018), 218--228.
[54]
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). IEEE, 2818--2826.
[55]
Antonio Torralba and Alexei A. Efros. 2011. Unbiased look at dataset bias. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11). IEEE Computer Society, 1521--1528.
[56]
Lucia Vadicamo, Fabio Carrara, Andrea Cimino, Stefano Cresci, Felice Dell’Orletta, Fabrizio Falchi, and Maurizio Tesconi. 2017. Cross-media learning for image sentiment analysis in the wild. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW’17), Vol. 1. IEEE, 308--317.
[57]
Jianlong Wu, Zhouchen Lin, and Hongbin Zha. 2016. Multi-view common space learning for emotion recognition in the wild. In Proceedings of the ACM International Conference on Multimodal Interaction. ACM, 464--471.
[58]
Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang. 2015. Joint visual-textual sentiment analysis with deep neural networks. In Proceedings of the ACM International Conference on Multimedia (MM’15). ACM, 1071--1074.
[59]
Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang. 2015. Robust image sentiment analysis using progressively trained and domain transferred deep networks. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’15). AAAI Press, 381--388. Retrieved from http://dl.acm.org/citation.cfm?id=2887007.2887061.
[60]
Jianbo Yuan, Sean Mcdonough, Quanzeng You, and Jiebo Luo. 2013. Sentribute: Image sentiment analysis from a mid-level perspective. In Proceedings of the International Workshop on Issues of Sentiment Discovery and Opinion Mining. ACM, Article 10, 8 pages.
[61]
Xueying Zhan, Yaowei Wang, Yanghui Rao, and Qing Li. 2019. Learning from multi-annotator data: A noise-aware classification framework. ACM Trans. Inf. Syst. 37, 2, Article 26 (Feb. 2019), 28 pages.
[62]
Liang Zhao, Feng Chen, Chang-Tien Lu, and Naren Ramakrishnan. 2016. Online spatial event forecasting in microblogs. ACM Trans. Spatial Algor. Syst. 2, 4, Article 15 (Nov. 2016), 39 pages.
[63]
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2018. Places: A 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 40, 6 (June 2018), 1452--1464.
[64]
Guang-You Zhou and Jimmy Xiangji Huang. 2017. Modeling and mining domain shared knowledge for sentiment analysis. ACM Trans. Inf. Syst. 36, 2, Article 18 (Aug. 2017), 36 pages.

Cited By

View all

Index Terms

  1. OutdoorSent: Sentiment Analysis of Urban Outdoor Images by Using Semantic and Deep Features

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Transactions on Information Systems
        ACM Transactions on Information Systems  Volume 38, Issue 3
        July 2020
        311 pages
        ISSN:1046-8188
        EISSN:1558-2868
        DOI:10.1145/3394096
        Issue’s Table of Contents
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 21 April 2020
        Accepted: 01 February 2020
        Revised: 01 December 2019
        Received: 01 June 2019
        Published in TOIS Volume 38, Issue 3

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. Sentiment analysis
        2. deep learning
        3. image processing
        4. information retrieval
        5. location-based social networks

        Qualifiers

        • Research-article
        • Research
        • Refereed

        Funding Sources

        • Coordenação de Aperfeiçoamento de Pessoal de Nível Superior
        • Conselho Nacional de Desenvolvimento Científico e Tecnológico
        • Fundação de Amparo à Pesquisa do Estado de São Paulo

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)49
        • Downloads (Last 6 weeks)8
        Reflects downloads up to 28 Dec 2024

        Other Metrics

        Citations

        Cited By

        View all

        View Options

        Login options

        Full Access

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media