short-paper

Open access

Deep Recurrent Conditional Random Field Network for Protein Secondary Prediction

Authors:

Alexander Rosenberg Johansen,

Casper Kaae Sønderby,

Søren Kaae Sønderby,

Ole WintherAuthors Info & Claims

ACM-BCB '17: Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology,and Health Informatics

Pages 73 - 78

https://doi.org/10.1145/3107411.3107489

Published: 20 August 2017 Publication History

Abstract

Deep learning has become the state-of-the-art method for predicting protein secondary structure from only its amino acid residues and sequence profile. Building upon these results, we propose to combine a bi-directional recurrent neural network (biRNN) with a conditional random field (CRF), which we call the biRNN-CRF. The biRNN-CRF may be seen as an improved alternative to an auto-regressive uni-directional RNN where predictions are performed sequentially conditioning on the prediction in the previous time-step. The CRF is instead nearest neighbor-aware and models for the joint distribution of the labels for all time-steps. We condition the CRF on the output of biRNN, which learns a distributed representation based on the entire sequence. The biRNN-CRF is therefore close to ideally suited for the secondary structure task because a high degree of cross-talk between neighboring elements can be expected. We validate the model on several benchmark datasets. For example, on CB513, a model with 1.7 million parameters, achieves a Q8 accuracy of 69.4 for single model and 70.9 for ensemble, which to our knowledge is state-of-the-art.

References

[1]

M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Józefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Gordon Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. A. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. 2016. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. CoRR abs/1603.04467 (2016). https://arxiv.org/abs/1603.04467 Software available from tensorflow.org.

[2]

D. Amodei, R. Anubhai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, J. Chen, M. Chrzanowski, A. Coates, G. Diamos, E. Elsen, J. Engel, L. Fan, C. Fougner, T. Han, A. Hannun, B. Jun, P. LeGresley, L. Lin, S. Narang, A. Ng, S. Ozair, R. Prenger, J. Raiman, S. Satheesh, D. Seetapun, S. Sengupta, Y. Wang, Z. Wang, C. Wang, B. Xiao, D. Yogatama, J. Zhan, and Z. Zhu. 2015. Deep Speech 2: End-to- End Speech Recognition in English and Mandarin. CoRR abs/1512.02595 (2015). http://arxiv.org/abs/1512.02595

[3]

J. Ba, J. Kiros, and G. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016).

[4]

D. Bahdanau, K. Cho, and Y. Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. CoRR abs/1409.0473 (2014). https://arxiv.org/abs/1409.0473

[5]

C. Bishop. 2006. Pattern recognition and machine learning. Springer.

Digital Library

[6]

A. Busia, J. Collins, and N. Jaitly. 2016. Protein Secondary Structure Prediction Using Deep Multi-scale Convolutional Neural Networks and Next-Step Conditioning. CoRR abs/1611.01503 (2016). http://arxiv.org/abs/1611.01503

[7]

K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP 2014).

[8]

F. Chollet. 2015. Keras: Theano-based deep learning library. (2015). https://github.com/fchollet/keras

[9]

J. Chung, Ç. Gülçehre, K. Cho, and Y. Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. CoRR abs/1412.3555 (2014). http://arxiv.org/abs/1412.3555

[10]

X. Glorot, A. Bordes, and Y. Bengio. 2011. Deep Sparse Rectifier Neural Networks. In Aistats, Vol. 15. 275.

[11]

A. Graves. 2012. Supervised sequence labelling with recurrent neural networks. Springer (2012).

[12]

K. He, X. Zhang, S. Ren, and J. Sun. 2015. Deep Residual Learning for Image Recognition. CoRR abs/1512.03385 (2015). http://arxiv.org/abs/1512.03385

[13]

S. Hochreiter and J. Schmidhuber. 1997. Long Short-Term Memory. Neural Comput. 9, 8 (Nov. 1997), 1735--1780.

Digital Library

[14]

D. Jones. 1999. Protein secondary structure prediction based on position-specific scoring matrices. Journal of Molecular Biology 292, 2 (1999), 195 -- 202.

[15]

D. Kingma and J. Ba. 2015. Adam: A method for stochastic optimization. International Conference on Learning Representation.

[16]

A. Krizhevsky, I. Sutskever, and G. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 1097--1105. http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf

Digital Library

[17]

J. Lafferty, A. McCallum, and F. Pereira. 2001. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML '01). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 282--289. http://dl.acm.org/citation.cfm?id=645530.655813

Digital Library

[18]

Y. LeCun, Y. Bengio, and G. Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436--444.

[19]

Z. Li and Y. Yu. 2016. Protein Secondary Structure Prediction Using Cascaded Convolutional and Recurrent Neural Networks. arXiv preprint arXiv:1604.07176 (2016).

Digital Library

[20]

B. Monastyrskyy, A. Kryshtafovych, J. Moult, A. Tramontano, and K. Fidelis. 2014. Assessment of protein disorder region predictions in CASP10. Proteins: Structure, Function, and Bioinformatics 82, S2 (2014), 127--137.

[21]

J. Moult, K. Fidelis, A. Kryshtafovych, T. Schwede, and A. Tramontano. 2014. Critical assessment of methods of protein structure prediction (CASP)round x. Proteins: Structure, Function, and Bioinformatics 82, S2 (2014), 1--6.

[22]

D. Ruck, S. Rogers, M. Kabrisky, M. Oxley, and B. Suter. 1990. The multilayer perceptron as an approximation to a Bayes optimal discriminant function. Neural Networks, IEEE Transactions on 1, 4 (1990), 296--298.

Digital Library

[23]

M. Schuster and K. Paliwal. 1997. Bidirectional Recurrent Neural Networks. Trans. Sig. Proc. 45, 11 (Nov. 1997), 2673--2681.

Digital Library

[24]

M. Singh. 2005. Predicting protein secondary and supersecondary structure. Handbook of Computational Molecular Biology, hapman & Hall CRC Computer and Information Science Series (2005).

[25]

S. Sønderby, C. Sønderby, H. Nielsen, and O. Winther. 2015. Convolutional LSTM networks for subcellular localization of proteins. In International Conference on Algorithms for Computational Biology. Springer, 68--80.

Digital Library

[26]

S. Sønderby and O. Winther. 2014. Protein secondary structure prediction with long short term memory networks. arXiv preprint arXiv:1412.7828 (2014).

[27]

N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. J. Mach. Learn. Res. (2014). http://dl.acm.org/citation.cfm?id=2627435.2670313

Digital Library

[28]

A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. 2016. WaveNet: A Generative Model for Raw Audio. CoRR abs/1609.03499 (2016). http://arxiv.org/abs/1609.03499

[29]

S. Wang, J. Peng, J. Ma, and J. Xu. 2016. Protein secondary structure prediction using deep convolutional neural fields. Scientific reports 6 (2016).

[30]

. Wu, M. Schuster, Z. Chen, Q. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, and others. 2016. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv preprint arXiv:1609.08144 (2016). https://arxiv.org/abs/1609.08144

[31]

J. Zhou and O. Troyanskaya. 2014. Deep Supervised and Convolutional Generative Stochastic Network for Protein Secondary Structure Prediction. In ICML. 745--753.

Digital Library

Cited By

Yang SChen X(2024)Prediction of Protein Secondary Structure Using a Hybrid Convolutional Blocks with GRU UnitsFrontiers in Computing and Intelligent Systems10.54097/k8rmxp2710:3(23-30)Online publication date: 29-Dec-2024
https://doi.org/10.54097/k8rmxp27
Kayser TShill P(2024)Protein Secondary Structure Prediction Without Alignment Using Graph Neural NetworkAdvanced Computing, Machine Learning, Robotics and Internet Technologies10.1007/978-3-031-47224-4_4(31-45)Online publication date: 16-Apr-2024
https://doi.org/10.1007/978-3-031-47224-4_4
de Oliveira GPedrini HDias Z(2021)Ensemble of Template-Free and Template-Based Classifiers for Protein Secondary Structure PredictionInternational Journal of Molecular Sciences10.3390/ijms22211144922:21(11449)Online publication date: 23-Oct-2021
https://doi.org/10.3390/ijms222111449
Show More Cited By

Index Terms

Deep Recurrent Conditional Random Field Network for Protein Secondary Prediction
1. Computing methodologies
  1. Machine learning

Recommendations

A Conditional Random Field with Loop and Its Inference Algorithm
ISDEA '12: Proceedings of the 2012 Second International Conference on Intelligent System Design and Engineering Application

A new algorithm for human motion Recognition based on Conditional Random Fields (CRFs) and Hidden Markov Models (HMM) -- HMCRF is proposed. Most existing approaches to human motion recognition with hidden states employ a Hidden Markov Model or suitable ...
A conditional random field-based model for joint sequence segmentation and classification

In this paper, we consider the problem of joint segmentation and classification of sequences in the framework of conditional random field (CRF) models. To effect this goal, we introduce a novel dual-functionality CRF model: on the first level, the ...
Bidirectional segmented-memory recurrent neural network for protein secondary structure prediction

The formation of protein secondary structure especially the regions of β-sheets involves long-range interactions between amino acids. We propose a novel recurrent neural network architecture called segmented-memory recurrent neural network (SMRNN) and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ACM-BCB '17: Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology,and Health Informatics

August 2017

800 pages

ISBN:9781450347228

DOI:10.1145/3107411

General Chairs:
Nurit Haspel
University of Massachusetts Boston, USA
,
Lenore J. Cowen
Tufts University, USA
,
Program Chairs:
Amarda Shehu
George Mason University, USA
,
Tamer Kahveci
University of Florida, USA
,
Giuseppe Pozzi
Politecnico di Milano, Italy

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGBio: ACM Special Interest Group on Bioinformatics

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 August 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

BCB '17

Sponsor:

SIGBio

BCB '17: 8th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics

August 20 - 23, 2017

Massachusetts, Boston, USA

Acceptance Rates

ACM-BCB '17 Paper Acceptance Rate 42 of 132 submissions, 32%;

Overall Acceptance Rate 254 of 885 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
845
Total Downloads

Downloads (Last 12 months)60
Downloads (Last 6 weeks)7

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang SChen X(2024)Prediction of Protein Secondary Structure Using a Hybrid Convolutional Blocks with GRU UnitsFrontiers in Computing and Intelligent Systems10.54097/k8rmxp2710:3(23-30)Online publication date: 29-Dec-2024
https://doi.org/10.54097/k8rmxp27
Kayser TShill P(2024)Protein Secondary Structure Prediction Without Alignment Using Graph Neural NetworkAdvanced Computing, Machine Learning, Robotics and Internet Technologies10.1007/978-3-031-47224-4_4(31-45)Online publication date: 16-Apr-2024
https://doi.org/10.1007/978-3-031-47224-4_4
de Oliveira GPedrini HDias Z(2021)Ensemble of Template-Free and Template-Based Classifiers for Protein Secondary Structure PredictionInternational Journal of Molecular Sciences10.3390/ijms22211144922:21(11449)Online publication date: 23-Oct-2021
https://doi.org/10.3390/ijms222111449
de Oliveira GPedrini HDias ZHung CHong JBechini ASong E(2021)Protein secondary structure prediction based on fusion of machine learning classifiersProceedings of the 36th Annual ACM Symposium on Applied Computing10.1145/3412841.3442067(26-29)Online publication date: 22-Mar-2021
https://dl.acm.org/doi/10.1145/3412841.3442067
Nahid TJui FShill P(2021)Protein Secondary Structure Prediction using Graph Neural Network2021 5th International Conference on Electrical Information and Communication Technology (EICT)10.1109/EICT54103.2021.9733590(1-6)Online publication date: 17-Dec-2021
https://doi.org/10.1109/EICT54103.2021.9733590
de Oliveira GPedrini HDias Z(2021)MMEC: Multi-Modal Ensemble Classifier for Protein Secondary Structure PredictionComputer Analysis of Images and Patterns10.1007/978-3-030-89128-2_17(175-184)Online publication date: 31-Oct-2021
https://doi.org/10.1007/978-3-030-89128-2_17
Görmez YAydın Z(2021)ROSE: A Novel Approach for Protein Secondary Structure PredictionTrends in Data Engineering Methods for Intelligent Systems10.1007/978-3-030-79357-9_45(455-464)Online publication date: 6-Jul-2021
https://doi.org/10.1007/978-3-030-79357-9_45
Jing XDong QHong DLu R(2020)Amino Acid Encoding Methods for Protein Sequences: A Comprehensive Review and AssessmentIEEE/ACM Transactions on Computational Biology and Bioinformatics10.1109/TCBB.2019.291167717:6(1918-1931)Online publication date: 1-Nov-2020
https://doi.org/10.1109/TCBB.2019.2911677
Oliveira GPedrini HDias Z(2020)Fusion of BLAST and Ensemble of Classifiers for Protein Secondary Structure Prediction2020 33rd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI)10.1109/SIBGRAPI51738.2020.00049(308-315)Online publication date: Nov-2020
https://doi.org/10.1109/SIBGRAPI51738.2020.00049
de Oliveira GPedrini HDias Z(2020)Ensemble of Bidirectional Recurrent Networks and Random Forests for Protein Secondary Structure Prediction2020 International Conference on Systems, Signals and Image Processing (IWSSIP)10.1109/IWSSIP48289.2020.9145437(311-316)Online publication date: Jul-2020
https://doi.org/10.1109/IWSSIP48289.2020.9145437
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents