abstract

Evolving modular neural sequence architectures with genetic programming

Authors:

David Dohan,

David So,

Quoc LeAuthors Info & Claims

GECCO '18: Proceedings of the Genetic and Evolutionary Computation Conference Companion

Pages 37 - 38

https://doi.org/10.1145/3205651.3208782

Published: 06 July 2018 Publication History

Get Access

Abstract

Automated architecture search has demonstrated significant success for image data, where reinforcement learning and evolution approaches now outperform the best human designed networks ([12], [8]). These successes have not transferred over to models dealing with sequential data, such as in language modeling and translation tasks. While there have been several attempts to evolve improved recurrent cells for sequence data [7], none have achieved significant gains over the standard LSTM. Recent work has introduced high performing recurrent neural network alternatives, such as Transformer [11] and Wavenet [4], but these models are the result of manual human tuning.

References

[1]

Félix-Antoine Fortin, François-Michel De Rainville, Marc-André Gardner, Marc Parizeau, and Christian Gagné. 2012. DEAP: Evolutionary Algorithms Made Easy. Journal of Machine Learning Research 13 (jul 2012), 2171--2175.

Digital Library

Google Scholar

[2]

Frederic Gruau et al. 1994. Neural Network Synthesis using Cellular Encoding and the Genetic Algorithm. (1994).

Google Scholar

[3]

Julian F Miller. 2011. Cartesian genetic programming. In Cartesian Genetic Programming. Springer, 17--34.

Google Scholar

[4]

Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 (2016).

Google Scholar

[5]

Aäron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. 2016. Conditional image generation with pixelcnn decoders. In Proceedings of the 30th International Conference on Neural Information Processing Systems. Curran Associates Inc., 4797--4805.

Digital Library

Google Scholar

[6]

Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, and Alexander Ku. 2018. Image Transformer. arXiv preprint arXiv:1802.05751 (2018).

Google Scholar

[7]

A. Rawal and R. Miikkulainen. 2018. From Nodes to Networks: Evolving Recurrent Neural Networks. ArXiv e-prints (March 2018). arXiv:1803.04439

Google Scholar

[8]

E. Real, A. Aggarwal, Y. Huang, and Q. V Le. 2018. Regularized Evolution for Image Classifier Architecture Search. ArXiv e-prints (Feb. 2018). arXiv:1802.01548

Google Scholar

[9]

Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Highway networks. arXiv preprint arXiv:1505.00387 (2015).

Google Scholar

[10]

Kenneth O Stanley and Risto Miikkulainen. 2002. Evolving neural networks through augmenting topologies. Evolutionary computation 10, 2 (2002), 99--127.

Digital Library

Google Scholar

[11]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. 6000--6010.

Google Scholar

[12]

Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. 2017. Learning transferable architectures for scalable image recognition. arXiv preprint arXiv:1707.07012 (2017).

Google Scholar

Cited By

View all

Kotyan SVargas DCoello Coello C(2020)Towards evolving robust neural architectures to defend from adversarial attacksProceedings of the 2020 Genetic and Evolutionary Computation Conference Companion10.1145/3377929.3389962(135-136)Online publication date: 8-Jul-2020
https://dl.acm.org/doi/10.1145/3377929.3389962

Index Terms

Evolving modular neural sequence architectures with genetic programming
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Bio-inspired approaches
        Genetic programming
      2. Neural networks

Recommendations

Recurrent Cartesian Genetic Programming of Artificial Neural Networks

Cartesian Genetic Programming of Artificial Neural Networks is a NeuroEvolutionary method based on Cartesian Genetic Programming. Cartesian Genetic Programming has recently been extended to allow recurrent connections. This work investigates applying ...
Modular neural network programming with genetic optimization

This study proposes a modular neural network (MNN) that is designed to accomplish both artificial intelligent prediction and programming. Each modular element adopts a high-order neural network to create a formula that considers both weights and ...
Evolving neural networks for decomposable problems using genetic programming
PRICAI'00: Proceedings of the 6th Pacific Rim international conference on Artificial intelligence

Many traditional methods for training neural networks using genetic algorithms and genetic programming do not have any special provisions for taking advantage of decomposable problems which can be solved by combining solutions to each subproblem. This ...

Comments

Information & Contributors

Information

Published In

GECCO '18: Proceedings of the Genetic and Evolutionary Computation Conference Companion

July 2018

1968 pages

ISBN:9781450357647

DOI:10.1145/3205651

Editor:
Hernan Aguirre
Shinshu University
,
General Chair:
Keiki Takadama
The University of Electro-Communications

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 July 2018

Check for updates

Author Tags

Qualifiers

Abstract

Conference

GECCO '18

Sponsor:

SIGEVO

GECCO '18: Genetic and Evolutionary Computation Conference

July 15 - 19, 2018

Kyoto, Japan

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
202
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Kotyan SVargas DCoello Coello C(2020)Towards evolving robust neural architectures to defend from adversarial attacksProceedings of the 2020 Genetic and Evolutionary Computation Conference Companion10.1145/3377929.3389962(135-136)Online publication date: 8-Jul-2020
https://dl.acm.org/doi/10.1145/3377929.3389962

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Recurrent Cartesian Genetic Programming of Artificial Neural Networks

Modular neural network programming with genetic optimization

Evolving neural networks for decomposable problems using genetic programming

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations