Evaluation of Pooling Operations and Regularization Parameters in Neural Networks for Drug-drug Interaction Extraction
Authors: 
Feng Meishan, Liu Zhi, Wei Hao, Qu Wen, Chen Fei, Lu MingYuAuthors Info & Claims
Feng Meishan, Liu Zhi, Wei Hao, Qu Wen, Chen Fei, Lu MingYuAuthors Info & ClaimsPages 112 - 117
Abstract
Recently, deep neural networks have been widely used in biomedical relation extraction, especially CNN and LSTM. Relation extraction is a classification task which uses pooling operations in neural networks to reduce dimensions and integration features and it uses regularization to prevent over-fitting. This paper evaluates the performance of CNN and LSTM in Drug-drug interaction extraction. We discuss the models' performance differences using different pooling operations and regularization parameters. Firstly, regularization can prevent over-fitting effectively, but it is important to ensure that the regularization parameters are set within the correct ranges. Secondly, the max pooling is better than other single pooling methods. Max pooling outperforms the others alternatives because is the only one which is invariant to the special pad tokens that are appending to the shorter sentences known as padding.
References
[1]
Segura-Bedmar I, Martínez P and Herrero-Zazo M. 2014. Lessons learnt from the DDIExtraction-2013 shared task. Journal of biomedical informatics, 152--164. DOI= http://doi.org/10.1016/j.jbi.2014.05.007.
[2]
Pyysalo S, Airola A and Heimonen J. 2008. Comparative analysis of five protein-protein interaction corpora. BMC bioinformatics. 9 Suppl 3(Suppl 3):S6. DOI= http://doi.org/10.1186/1471-2105-9-S3-S6
[3]
Donaldson I, Martin J and De Bruijn B. 2003. PreBIND and Textomy-mining the biomedical literature for protein-protein interactions using a support vector machine, BMC bioinformatics, 4(1):11--0. DOI= http://doi.org/10.1186/1471-2105-4-11
[4]
Berger A L, Pietra V J D and Pietra S A D. 1996. A maximum entropy approach to natural language processing. Computational linguistics, 39--71. DOI= http://doi.org/10.1016/0169-7439(95)00072-0
[5]
Lecun Y, Bottou L and Bengio Y. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 2278--2324.
[6]
Hochreiter S and Schmidhuber J. 1997. Long short-term memory. Neural computation, 9(8):1735--1780. DOI= http://doi.org/10.1162/neco.1997.9.8.1735.
[7]
Zhao Z, Yang Z and Luo L. 2016. Drug drug interaction extraction from biomedical literature using syntax convolutional neural network. Bioinformatics, 3444--3453. DOI= http://doi.org/10.1093/bioinformatics/btw486
[8]
Miyao Y and Tsujii J. 2008. Feature forest models for probabilistic HPSG parsing. Computational linguistics, 35--80.
[9]
Mikolov T, Yih W and Zweig G. 2013. Linguistic regularities in continuous space word representations. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 746--751.
[10]
Liu S, Tang B and Chen Q. 2016. Drug-drug interaction extraction via convolutional neural networks. Computational and mathematical methods in medicine, 1--8. DOI= http://doi.org/10.1155/2016/6918381
[11]
Liu S, Chen K, Chen Q and Tang B. 2017. Dependency-based convolutional neural network for drug-drug interaction extraction. In: Bioinformatics and Biomedicine (BIBM). IEEE international conference on: IEEE, 1074--1080. DOI=http://doi.org/10.1109/BIBM.2016.7822671
[12]
Quan C, Hua L, Sun X and Bai W. 2016. Multichannel convolutional neural network for biological relation extraction. BioMed Research International. 1--10. DOI=http://doi.org/10.1155/2016/1850404
[13]
Asada M, Miwa M and Sasaki Y. 2017. Extracting Drug-Drug Interactions with Attention CNNs. Bionlp, 9--18.
[14]
Y. Zhang, W. Zheng, H. Lin, J. Wang, Z. Yang and M. Dumontier. 2017. Drug-drug interaction extraction via hierarchical rnns on sequence and shortest dependency paths. Bioinformatics, 828--835. DOI= http://doi.org/10.1093/bioinformatics/btx659
[15]
W. Wang, X. Yang, C. Yang, X. Guo, X. Zhang and C. Wu. 2017. Dependency-based long short term memory network for drug-drug interaction extraction. BMC Bioinformatics, 18(S16):578. DOI= http://doi.org/10.1186/s12859-017-1962-8
[16]
R. Kavuluru and A. Rios. 2017. Extracting drug-drug interactions with word and character-level recurrent neural networks. IEEE International Conference on Healthcare Informatics (ICHI), 5--12.
[17]
Lim S, Lee K and Kang J. 2018. Drug drug interaction extraction from the literature using a recursive neural network. PloS one, 13(1): e0190926. DOI= http://doi.org/10.1371/journal.pone.0190926
[18]
Tai K S, Socher R and Manning C D. 2015. Improved semantic representations from tree-structured long short-term memory networks. arXiv:1503.00075. Retrieved from https://arxiv.org/abs/1503.00075
[19]
Sahu S K and Anand A. 2018. Drug-drug interaction extraction from biomedical texts using long short-term memory network. Journal of biomedical informatics, 15--24. DOI=http://doi.org/10.1016/j.jbi.2018.08.005
[20]
Suárez-Paniagua V and Segura-Bedmar I. 2018. Evaluation of pooling operations in convolutional architectures for drug-drug interaction extraction. BMC bioinformatics, 209.
[21]
Ng A Y. 2004. Feature selection, L 1 vs. L 2 regularization, and rotational invariance. Proceedings of the twenty-first international conference on Machine learning. ACM, 78. DOI=http://doi.org/10.1145/1015330.1015435
[22]
Srivastava N, Hinton G and Krizhevsky A. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929--1958.
[23]
J Pennington, R Socher and C Manning. 2014. GloVe: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1532--1543.
[24]
Kruse R, Borgelt C and Klawonn F. 2013. Multi-layer perceptrons. Computational Intelligence, 47--81.
[25]
Graves A and Schmidhuber J. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18 (5): 602--610. DOI= http://doi.org/10.1016/j.neunet.2005.06.042
[26]
Bahdanau D, Cho K and Bengio Y. 2014. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473. Retrieved from https://arxiv.org/abs/1409.0473
[27]
Luong M T, Pham H and Manning C D. 2015. Effective approaches to attention-based neural machine translation. arXiv:1508.04025. Retrieved from https://arxiv.org/abs/1508.04025
[28]
Xu K, Ba J and Kiros R. 2015. Show, attend and tell: Neural image caption generation with visual attention. International conference on machine learning, 2048--2057.
[29]
Vinyals O, Kaiser L and Koo T. 2015. Grammar as a foreign language. Processing Systems, 2773--2781.
[30]
Hermann K M, Kocisky T and Grefenstette E. 2015. Teaching machines to read and comprehend. Advances in Neural Information Processing Systems, 1693--1701.
Index Terms
- Evaluation of Pooling Operations and Regularization Parameters in Neural Networks for Drug-drug Interaction Extraction
Recommendations
Mixed Pooling for Convolutional Neural Networks
Rough Sets and Knowledge TechnologyAbstractConvolutional Neural Network (CNN) is a biologically inspired trainable architecture that can learn invariant features for a number of applications. In general, CNNs consist of alternating convolutional layers, non-linearity layers and feature ...
Rank-based pooling for deep convolutional neural networks
Pooling is a key mechanism in deep convolutional neural networks (CNNs) which helps to achieve translation invariance. Numerous studies, both empirically and theoretically, show that pooling consistently boosts the performance of the CNNs. The ...
A fully trainable network with RNN-based pooling
Highlights- A trainable RNN based pooling method is proposed.
- CNNs are turned into fully ...
AbstractPooling is an important component in convolutional neural networks (CNNs) for aggregating features and reducing computational burden. Compared with other components such as convolutional layers and fully connected layers which are ...
Comments
Information & Contributors
Information
Published In
December 2019
370 pages
ISBN:9781450376273
DOI:10.1145/3374587
Copyright © 2019 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
In-Cooperation
- Shenzhen University: Shenzhen University
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 March 2020
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- the National Natural Science Foundation of China
Conference
CSAI2019
CSAI2019: 2019 3rd International Conference on Computer Science and Artificial Intelligence
December 6 - 8, 2019
IL, Normal, USA
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 70Total Downloads
- Downloads (Last 12 months)3
- Downloads (Last 6 weeks)0
Reflects downloads up to 14 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in