Extracting and learning an unknown grammar with recurrent neural networks
Pages 317 - 324
Abstract
Simple second-order recurrent networks are shown to readily learn small known regular grammars when trained with positive and negative strings examples. We show that similar methods are appropriate for learning unknown grammars from examples of their strings. The training algorithm is an incremental real-time, recurrent learning (RTRL) method that computes the complete gradient and updates the weights at the end of each string. After or during training, a dynamic clustering algorithm extracts the production rules that the neural network has learned. The methods are illustrated by extracting rules from unknown deterministic regular grammars. For many cases the extracted grammar outperforms the neural net from which it was extracted in correctly classifying unseen strings.
References
[1]
N. Alon, A.K. Dewdney, and T.J. Ott, 'Efficient Simulation of Finite Automata by Neural Nets, Journal of the ACM, Vol. 38, p. 495 (1991).
[2]
D. Angluin, C.H. Smith, Inductive Inference: Theory and Methods, ACM Computing Surveys, Vol. 15, No. 3, p. 237, (1983).
[3]
A. Cleeremans, D. Servan-Schreiber, J. McClelland, Finite State Automata and Simple Recurrent Recurrent Networks, Neural Computation, Vol. 1, No. 3, p. 372 (1989).
[4]
J.L. Elman, Distributed Representations, Simple Recurrent Networks, and Grammatical Structure, Machine Learning, Vol. 7, No. 2/3, p. 91 (1991).
[5]
K.S. Fu, Syntactic Panern Recognition and Applications, Prentice-Hall, Englewood Cliffs, NJ. Ch.10 (1982).
[6]
C.L. Giles, G.Z. Sun, H.H. Chen, Y.C. Lee, D. Chen, Higher Order Recurrent Networks & Grammatical Inference, Advances in Neural Information Systems 2, D.S. Touretzky (ed), Morgan Kaufmann, San Mateo, Ca, p. 380 (1990).
[7]
C.L. Giles, D. Chen, C.B. Miller, H.H. Chen, G.Z. Sun, Y.C. Lee, Grammatical Inference Using Second-Order Recurrent Neural Networks, Proceedings of the International Joint Conference on Neural Networks, IEEE 91CH3049-4, Vol. 2, p. 357 (1991).
[8]
C.L. Giles, C.B. Miller, D. Chen, H.H. Chen, G.Z. Sun, Y.C. Lee, Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks, Neural Computation, accepted for publication (1992).
[9]
J. Hertz, A. Krogh, R.G. Palmer, Introduction to the Theory of Neural Computation, Addison-Wesley, Redwood City, Ca., Ch. 7 (1991).
[10]
J.E. Hopcroft, J.D. Ullman, Introduction to Automata Theory, Languages, and Computation, Addison Wesley, Reading, Ma. (1979).
[11]
Y.C. Lee, G. Doolen, H.H. Chen, G.Z. Sun, T. Maxwell, H.Y. Lee, C.L. Giles, Machine Learning Using a Higher Order Correlational Network, Physica D, Vol. 22-D, No. 1-3, p. 276 (1986).
[12]
J.B. Pollack, The Induction of Dynamical Recognizers, Machine Learning, Vol. 7, No. 2/3, p. 227 (1991).
[13]
G.Z. Sun, H.H. Chen, C.L. Giles, Y.C. Lee, D. Chen, Connectionist Pushdown Automata that Learn Context-Free Grammars, Proceedings of the International Joint Conference on Neural, Washington D.C., Lawrence Erlbaum Pub., Vol. I, p. 577 (1990).
[14]
R.L. Watrous, G.M. Kuhn, Induction of Finite-State Languages Using Second-Order Recurrent Networks, Neural Computation, accepted for publication (1992a) and these proceedings, (1992b).
[15]
R.J. Williams, D. Zipser, A Learning Algorithm for Continually Running Fully Recurrent Neural Networks, Neural Computation, Vol. 1, No. 2, p. 270, (1989).
Index Terms
- Extracting and learning an unknown grammar with recurrent neural networks
Recommendations
Simultaneous perturbation learning rule for recurrent neural networks and its FPGA implementation
Recurrent neural networks have interesting properties and can handle dynamic information processing unlike ordinary feedforward neural networks. However, they are generally difficult to use because there is no convenient learning scheme. In this paper, ...
A machine learning method for extracting symbolic knowledge from recurrent neural networks
Neural networks do not readily provide an explanation of the knowledge stored in their weights as part of their information processing. Until recently, neural networks were considered to be black boxes, with the knowledge stored in their weights not ...
Natural language learning by recurrent neural networks: a comparison with probabilistic approaches
NeMLaP3/CoNLL '98: Proceedings of the Joint Conferences on New Methods in Language Processing and Computational Natural Language LearningWe present preliminary results of experiments with two types of recurrent neural networks for a natural language learning task. The neural networks, Elman networks and Recurrent Cascade Correlation (RCC), were trained on the text of a first-year primary ...
Comments
Information & Contributors
Information
Published In
Publisher
Morgan Kaufmann Publishers Inc.
San Francisco, CA, United States
Publication History
Published: 02 December 1991
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 01 Jan 2025