research-article

Open access

Preliminary tests of an anticipatory classifier system with experience replay

Authors:

Norbert Kozłowski,

Łukasz ŚmierzchałaAuthors Info & Claims

GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference Companion

Pages 2095 - 2103

https://doi.org/10.1145/3520304.3533996

Published: 19 July 2022 Publication History

Abstract

The paper describes the first attempts toward designing and evaluating Anticipatory Classifier System ACS2 in conjunction with Experience Replay (ER). Promising results verified by statistical tests are obtained both on single- and multi-step problems, albeit limited to deterministic and discrete tasks. The analysis indicates that ACS2 using memorized experiences has the potential for significant improvements in learning efficiency and knowledge generality.

References

[1]

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. 2017. Hindsight experience replay. Advances in neural information processing systems 30 (2017).

Digital Library

[2]

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. Openai gym. arXiv preprint arXiv:1606.01540 (2016).

[3]

Martin V Butz. 2002. Anticipatory learning classifier systems. Vol. 4. Springer Science & Business Media.

[4]

Martin V. Butz and Wolfgang Stolzmann. 2002. An Algorithmic Description of ACS2. In Advances in Learning Classifier Systems, Pier Luca Lanzi, Wolfgang Stolzmann, and Stewart W. Wilson (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 211--229.

Digital Library

[5]

Pawel Cichosz. 1999. An analysis of experience replay in temporal difference learning. Cybernetics & Systems 30, 5 (1999), 341--363.

[6]

B. Derrick and P. White. 2016. Why Welch's test is Type I error robust. The Quantitative Methods for Psychology 12, 1 (2016), 30--38.

[7]

Tim Hansmeier and Marco Platzner. 2021. An experimental comparison of explore/exploit strategies for the learning classifier system XCS. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. 1639--1647.

Digital Library

[8]

Joachim Hoffmann. 2016. Vorhersage und erkenntnis. (2016).

[9]

Joachim Hoffmann and Albrecht Sebald. 2000. Lernmechanismen zum Erwerb verhaltenssteuernden Wissens. Psychologische Rundschau (2000).

[10]

John H Holland and Judith S Reitman. 1978. Cognitive systems based on adaptive algorithms. In Pattern-directed inference systems. Elsevier, 313--329.

[11]

Norbert Kozlowski and Olgierd Unold. 2018. Integrating anticipatory classifier systems with OpenAI gym. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. ACM, 1410--1417.

Digital Library

[12]

Norbert Kozłowski and Olgierd Unold. 2021. Anticipatory Classifier System with Average Reward Criterion in Discretized Multi-Step Environments. Applied Sciences 11, 3 (2021), 1098.

[13]

Norbert Kozlowski and Olgierd Unold. 2022. Internalizing Knowledge for Anticipatory Classifier Systems in Discretized Real-Valued Environments. IEEE Access (2022).

[14]

Norbert Kozlowski and Olgierd Unold. 2022. Internalizing Knowledge for Anticipatory Classifier Systems in Discretized Real-Valued Environments. IEEE Access 10 (2022), 33816--33828.

[15]

Sang Gyu Kwak and Jong Hae Kim. 2017. Central limit theorem: the cornerstone of modern statistics. Korean journal of anesthesiology 70, 2 (2017), 144.

[16]

Long-Ji Lin. 1992. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning 8, 3 (1992), 293--321.

[17]

Yunzhe Liu, Marcelo G Mattar, Timothy EJ Behrens, Nathaniel D Daw, and Raymond J Dolan. 2021. Experience replay is associated with efficient nonlocal learning. Science 372, 6544 (2021), eabf1357.

[18]

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013).

[19]

Romain Orhand, Anne Jeannin-Girardon, Pierre Parrend, and Pierre Collet. 2020. PEPACS: Integrating probability-enhanced predictions to ACS2. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion. 1774--1781.

Digital Library

[20]

David Pätzel, Anthony Stein, and Jörg Hähner. 2019. A survey of formal theoretical advances regarding XCS. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. 1295--1302.

Digital Library

[21]

Lukas Rosenbauer, Anthony Stein, Roland Maier, David Pätzel, and Jörg Hähner. 2020. XCS as a reinforcement learning approach to automatic test case prioritization. In Proceedings of the 2020 genetic and evolutionary computation conference companion. 1798--1806.

Digital Library

[22]

Lukas Rosenbauer, Anthony Stein, David Pätzel, and Jöorg Hähner. 2020. XCSF with experience replay for automatic test case prioritization. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE, 1307--1314.

[23]

Anthony Stein, Roland Maier, Lukas Rosenbauer, and Jörg Hähner. 2020. XCS classifier system with experience replay. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference. 404--413.

Digital Library

[24]

Anthony Stein, Simon Menssen, and Jörg Hähner. 2018. What about interpolation? A radial basis function approach to classifier prediction modeling in XCSF. In Proceedings of the Genetic and Evolutionary Computation Conference. 537--544.

Digital Library

[25]

Wolfgang Stolzmann. 1997. Antizipative classifier systems. Ph.D. Dissertation. Fachbereich Mathematik/Informatik, University of Osnabrück.

[26]

Wolfgang Stolzmann. 2000. An Introduction to Anticipatory Classifier Systems. In Learning Classifier Systems, Pier Luca Lanzi, Wolfgang Stolzmann, and Stewart W. Wilson (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 175--194.

Digital Library

[27]

Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.

Digital Library

[28]

Ryan J. Urbanowicz and Will N. Browne. 2017. Introduction to Learning Classifier Systems (1st ed.). Springer Publishing Company, Incorporated.

Digital Library

[29]

Zhaoxiang Zang, Dehua Li, and Junying Wang. 2015. Learning classifier systems with memory condition to solve non-Markov problems. Soft Computing 19, 6 (2015), 1679--1699.

Digital Library

Cited By

Siddique AHeider MIqbal MShiraishi HLi XHandl J(2024)A Survey on Learning Classifier Systems from 2022 to 2024Proceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3664165(1797-1806)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638530.3664165
Schönberner CTomforde SLi XHandl J(2024)XCS: Is Covering All You Need?Proceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3664146(1788-1796)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638530.3664146
Smierzchała ŁKozłowski NUnold O(2023)Anticipatory Classifier System With Episode-Based Experience ReplayIEEE Access10.1109/ACCESS.2023.326987911(41190-41204)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3269879

Index Terms

Preliminary tests of an anticipatory classifier system with experience replay
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Rule learning
2. Software and its engineering

Recommendations

Investigating exploration techniques for ACS in discretized real-valued environments
GECCO '20: Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion

One way of dealing with the real-valued input signal is to discretize it. This might influence the process of learning the environmental model by the ACS2 agent. A more sophisticated method of selecting action can be applied to increase the speed of ...
Preliminary tests of a real-valued anticipatory classifier system
GECCO '19: Proceedings of the Genetic and Evolutionary Computation Conference Companion

The paper describes the first attempts toward designing and evaluating anticipatory classifier systems working in a real-valued input domain using interval predicates representation. Promising results are obtained by testing two environments - real-...
Hindsight-Combined and Hindsight-Prioritized Experience Replay
Neural Information Processing
Abstract
Reinforcement learning has proved to be of great utility; execution, however, may be costly due to sampling inefficiency. An efficient method for training is experience replay, which recalls past experiences. Several experience replay techniques, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference Companion

July 2022

2395 pages

ISBN:9781450392686

DOI:10.1145/3520304

Editor:
Jonathan E. Fieldsend
University of Exeter
,
General Chair:
Markus Wagner
The University of Adelaide

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 July 2022

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Wroclaw University of Science and Technology, Poland

Conference

GECCO '22

Sponsor:

SIGEVO

GECCO '22: Genetic and Evolutionary Computation Conference

July 9 - 13, 2022

Massachusetts, Boston

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
178
Total Downloads

Downloads (Last 12 months)82
Downloads (Last 6 weeks)23

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Siddique AHeider MIqbal MShiraishi HLi XHandl J(2024)A Survey on Learning Classifier Systems from 2022 to 2024Proceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3664165(1797-1806)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638530.3664165
Schönberner CTomforde SLi XHandl J(2024)XCS: Is Covering All You Need?Proceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3664146(1788-1796)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638530.3664146
Smierzchała ŁKozłowski NUnold O(2023)Anticipatory Classifier System With Episode-Based Experience ReplayIEEE Access10.1109/ACCESS.2023.326987911(41190-41204)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3269879

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents