research-article

Open access

Research and Practice of Delivering Tabletop Exercises

Authors:

Valdemar Švábenský,

Martin Hofbauer,

Martin HorákAuthors Info & Claims

ITiCSE 2024: Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1

Pages 220 - 226

https://doi.org/10.1145/3649217.3653642

Published: 03 July 2024 Publication History

Abstract

Tabletop exercises are used to train personnel in the efficient mitigation and resolution of incidents. They are applied in practice to support the preparedness of organizations and to highlight inefficient processes. Since tabletop exercises train competencies required in the workplace, they have been introduced into computing courses at universities as an innovation, especially within cybersecurity curricula. To help computing educators adopt this innovative method, we survey academic publications that deal with tabletop exercises. From 140 papers we identified and examined, we selected 14 papers for a detailed review. The results show that the existing research deals predominantly with exercises that follow a linear format and exercises that do not systematically collect data about trainees' learning. Computing education researchers can investigate novel approaches to instruction and assessment in the context of tabletop exercises to maximize the impact of this teaching method. Due to the relatively low number of published papers, the potential for future research is immense. Our review provides researchers, tool developers, and educators with an orientation in the area, a synthesis of trends, and implications for further work.

References

[1]

Giddeon N. Angafor, Iryna Yevseyeva, and Ying He. 2020a. Bridging the Cyber Security Skills Gap: Using Tabletop Exercises to Solve the CSSG Crisis. In Serious Games. Springer International Publishing, Cham, 117--131. https://doi.org/10.1007/978--3-030--61814--8_10

[2]

Giddeon N. Angafor, Iryna Yevseyeva, and Ying He. 2020b. Game-based learning: A review of tabletop exercises for cybersecurity incident response training. Security and Privacy, Vol. 3, 6 (2020), e126. https://doi.org/10.1002/spy2.126

[3]

Giddeon N. Angafor, Iryna Yevseyeva, and Leandros Maglaras. 2023. Scenario-based incident response training: lessons learnt from conducting an experiential learning virtual incident response tabletop exercise. Information & Computer Security, Vol. 31, 4 (2023). https://doi.org/10.1108/ICS-05--2022-0085

[4]

Avalias. 2023. Avalanche TTX. https://www.avalias.com/products/avalanche-ttx Retrieved January 17, 2024 from

[5]

Agn? Brilingait?, Linas Bukauskas, Virgilijus Krinickij, and Eduardas Kutka. 2017. Environment for Cybersecurity Tabletop Exercises. In 11th European Conference on Games Based Learning. Graz, Austria, 47--55. https://www.researchgate.net/publication/320244434_Environment_for_Cybersecurity_Tabletop_Exercises

[6]

Cinten. 2023. Crisis Management. https://www.cinten.com/crisis-management-digital-simulations-performance-analytics Retrieved January 17, 2024 from

[7]

Cyber Security Operations Consulting. 2023. Cybersecurity TTX and Incident Response. CyberSecOp. https://cybersecop.com/tabletop-exercise/cybersecurity-tabletop-exercise-services Retrieved January 17, 2024 from

[8]

CrowdStrike. 2023. CrowdStrike TTX. CrowdStrike. https://www.crowdstrike.com/services/prepare/tabletop-exercise/ Retrieved January 17, 2024 from

[9]

Svante Edzén. 2014. Table-Top Exercises for Emergency Management: Tame Solutions for Wicked Problems. In 2014 47th Hawaii International Conference on System Sciences. IEEE, USA, 1978--1985. https://doi.org/10.1109/HICSS.2014.250

Digital Library

[10]

eeedo inc. 2023. Crisis & Preparedness Cooperation Exercise Simulator. eeedo inc. https://eee.do/crisis-and-preparedness-cooperation-exercise-simulation-system/ Retrieved January 17, 2024 from

[11]

Elsevier. 2024. Scopus. Retrieved January 17, 2024. https://www.scopus.com

[12]

ENISA. 2009. Good Practice Guide on National Exercises. Technical Report. European Union Agency for Cybersecurity. https://www.enisa.europa.eu/publications/national-exercise-good-practice-guide

[13]

Cathleen A. Evans. 2019. Tabletop exercises in the nursing classroom: An introduction for nurse educators. Nursing Forum, Vol. 54, 4 (2019), 669--674. https://doi.org/10.1111/nuf.12394

[14]

Federal Emergency Management Agency. 2020. Homeland Security Exercise and Evaluation Program (HSEEP). Technical Report. Federal Emergency Management Agency. https://www.fema.gov/sites/default/files/2020-04/Homeland-Security-Exercise-and-Evaluation-Program-Doctrine-2020-Revision-2--2-25.pdf

[15]

Filigran. 2024. OpenEx Platform. Filigran. https://github.com/OpenEx-Platform/openex Retrieved January 17, 2024 from

[16]

ENISA (European Union Agency for Cybersecurity). 2023. Cyber Europe. Online, accessed January 17, 2024. https://www.enisa.europa.eu/topics/training-and-exercises/cyber-exercises/cyber-europe-programme

[17]

CC2020 Task Force. 2020. Computing Curricula 2020: Paradigms for Global Computing Education. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3467967

Digital Library

[18]

Amélie Frégeau, Alexis Cournoyer, Marc-André Maheu-Cadotte, Massimiliano Iseppon, Nathalie Soucy, Julie St-Cyr Bourque, Sylvie Cossette, Véronique Castonguay, and Richard Fleet. 2020. Use of tabletop exercises for healthcare education: a scoping review protocol. BMJ Open, Vol. 10, 1 (2020). https://doi.org/10.1136/bmjopen-2019-032662

[19]

Tim Grance, Tamara Nolan, Kristin Burke, Rich M. Dudley, Gregory C. White, and Travis Good. 2006. Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities: Recommendations of the National Institute of Standards and Technology. Technical Report. National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800--84

[20]

Philip Huff, Sandra Leiterman, and Jan P. Springer. 2023. Cyber Arena: An Open-Source Solution for Scalable Cybersecurity Labs in the Cloud. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 221--227. https://doi.org/10.1145/3545945.3569828

Digital Library

[21]

Emergency Solutions International Inc. 2023. ESI Exercises. ESI. https://esintl.ca/exercise-and-evaluation/ Retrieved January 17, 2024 from

[22]

ISO. 2013. Societal security - Guidelines for exercises. Standard. International Organization for Standardization, Geneva, CH.

[23]

Astrid Janssen and Hanneke Vreugdenhil. 2015. Objective oriented exercise evaluation with TARCK-it. In 12th Proceedings of the International Conference on Information Systems for Crisis Response and Management, Krystiansand, Norway, May 24--27, 2015. ISCRAM Association, Kristiansand, Norway. http://idl.iscram.org/files/astridjanssen/2015/1223_AstridJanssenHannekeVreugdenhil2015.pdf

[24]

Barbara Kitchenham and Stuart Charters. 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering. Technical Report. EBSE.

[25]

Vytis Kopustinskas, Bogdan Vamanu, Marcelo Masera, Rimantas vS ikas, Julia Vainio, Romualdas Petkevivc ius, and Lawrence Walzer. 2020. Tabletop Exercise as a Tool to Foster Resilience in the Energy Sector: Lessons Learned in the Baltic States. In Proceedings of the 30th European Safety and Reliability Conference and 15th Probabilistic Safety Assessment and Management Conference. Research Publishing Services, Italy, 255--262. https://doi.org/10.3850/978--981--14--8593-0_4019-cd

[26]

Klaus Krippendorff. 2004. Reliability in content analysis: Some common misconceptions and recommendations. Human communication research, Vol. 30, 3 (2004), 411--433. https://doi.org/10.1111/j.1468--2958.2004.tb00738.x

[27]

Clifton Kussmaul. 2012. Process Oriented Guided Inquiry Learning (POGIL) for Computer Science. In Proceedings of the 43rd ACM Technical Symposium on Computer Science Education (SIGCSE '12). Association for Computing Machinery, New York, NY, USA, 373--378. https://doi.org/10.1145/2157136.2157246

Digital Library

[28]

Immersive Labs. 2023. Cyber Crisis Simulator. Immersive Labs. https://www.immersivelabs.com/platform/cyber-crisis-simulator/ Retrieved January 17, 2024 from

[29]

Maria B. Line and Nils B. Moe. 2015. Understanding Collaborative Challenges in IT Security Preparedness Exercises. In ICT Systems Security and Privacy Protection. Springer, Cham, 311--324. https://doi.org/10.1007/978--3--319--18467--8_21

[30]

Micro Minder Ltd. 2023. Cybersecurity Tabletop Exercise Services. MCS. https://www.micromindercs.com/cybersecuritytabletopexercise Retrieved January 17, 2024 from

[31]

Syed S. Mahdi, Hafsa A. Jafri, Raheel Allana, Gopi Battineni, Mariam Khawaja, Syeda Sakina, Daniyal Agha, Kiran Rehman, and Francesco Amenta. 2023. Systematic review on the current state of disaster preparation Simulation Exercises (SimEx). BMC Emergency Medicine, Vol. 23, 1 (2023), 52. https://doi.org/10.1186/s12873-023-00824--8

[32]

Georgios Makrodimitris and Christos Douligeris. 2015. Towards a Successful Exercise Implementation -- A Case Study of Exercise Methodologies. In Human Aspects of Information Security, Privacy, and Trust. Springer International Publishing, Cham, 207--218. https://doi.org/10.1007/978--3--319--20376--8_19

[33]

Jim Marshall. 2009. The Cyber Scenario Modeling and Reporting Tool (CyberSMART). In 2009 Cybersecurity Applications & Technology Conference for Homeland Security, Vol. 1. IEEE, USA, 305--309. https://doi.org/10.1109/CATCH.2009.46

Digital Library

[34]

NATO Cooperative Cyber Defence Centre of Excellence. 2023. Locked Shields. Online, accessed January 17, 2024. https://ccdcoe.org/exercises/locked-shields

[35]

Natural Language Toolkit (NLTK) Project. 2023. Source code for textttnltk.metrics.agreement. Online, accessed January 17, 2024. http://www.nltk.org/_modules/nltk/metrics/agreement.html

[36]

Yuitaka Ota, Erika Mizuno, Tomomi Aoyama, Yoshihiro Hashimoto, Ichiro Koshijima, Haruna Asai, and Shiho Taniuchi. 2022. Designing Framework for Tabletop Exercise to Promote Resilience Against Cyber Attacks. In 14th International Symposium on Process Systems Engineering. Elsevier, Japan, 1471--1476. https://doi.org/10.1016/B978-0--323--85159--6.50245--1

[37]

Rain Ottis. 2014. Light Weight Tabletop Exercise for Cybersecurity Education. Journal of Homeland Security and Emergency Management, Vol. 11 (12 2014), 579--592. Issue 4. https://doi.org/10.1515/jhsem-2014-0031

[38]

Kai Petersen, Robert Feldt, Shahid Mujtaba, and Michael Mattsson. 2008. Systematic Mapping Studies in Software Engineering. In Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering (Italy) (EASE'08). BCS Learning & Development Ltd., Swindon, UK, 68--77. https://doi.org/10.14236/ewic/EASE2008.8

[39]

Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. 2015. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology, Vol. 64 (2015), 1 -- 18. https://doi.org/10.1016/j.infsof.2015.03.007

Digital Library

[40]

LLC PreparedEx. 2023. PreparedEx Services. PreparedEx, LLC. https://preparedex.com/services/ Retrieved January 17, 2024 from

[41]

RedLegg. 2023. RedLegg TTX. GLW Specialty. https://www.redlegg.com/advisory-services/tabletop-exercise Retrieved January 17, 2024 from

[42]

Richards, Chet. 2020. Boyd's OODA loop. Necesse, Vol. 5 (2020), 142--165. https://hdl.handle.net/11250/2683228

[43]

Kate Sanders, Jonas Boustedt, Anna Eckerdal, Robert McCartney, and Carol Zander. 2017. Folk Pedagogy: Nobody Doesn't Like Active Learning. In Proceedings of the 2017 ACM Conference on International Computing Education Research (Tacoma, Washington, USA) (ICER '17). Association for Computing Machinery, New York, NY, USA, 145--154. https://doi.org/10.1145/3105726.3106192

Digital Library

[44]

Andrea Skytterholm and Guro Hotvedt. 2023. Criteria for Realistic and Expedient Scenarios for Tabletop Exercises on Cyber Attacks Against Industrial Control Systems in the Petroleum Industry. In Proceedings of the International Conference on Cybersecurity, Situational Awareness and Social Media. Springer Nature, Singapore, 39--54. https://doi.org/10.1007/978--981--19--6414--5_3

[45]

Shiho Taniuchi, Tomomi Aoyama, Haruna Asai, and Ichiro Koshijima. 2019. Training Cyber Security Exercise Facilitator: Behavior Modeling Based on Human Error. In Advances in Human Factors in Cybersecurity. Springer International Publishing, Cham, 138--148. https://doi.org/10.1007/978--3--319--94782--2_14

[46]

Valdemar vSvábenský, Jan Vykopal, Pavel v Celeda, and Lydia Kraus. 2022. Applications of educational data mining and learning analytics on data from cybersecurity training. Education and Information Technologies, Vol. 27, 9 (2022), 12179--12212. https://doi.org/10.1007/s10639-022--11093--6

Digital Library

[47]

Valdemar vSvábenský, Jan Vykopal, Martin Horák, Martin Hofbauer, and Pavel vCeleda. 2024. From Paper to Platform: Evolution of a Novel Learning Environment for Tabletop Exercises. In Proceedings of the 29th Conference on Innovation and Technology in Computer Science Education (ITiCSE '24). Association for Computing Machinery, New York, NY, USA, bibinfonumpages7 pages. https://doi.org/10.1145/3649217.3653639

Digital Library

[48]

Jan Vykopal, Pavel vCeleda, Valdemar vSvábenský, Martin Hofbauer, and Martin Horák. 2024. Dataset: Research and Practice of Delivering Tabletop Exercises. https://gitlab.fi.muni.cz/inject/papers/2024-iticse-research-practice.

[49]

Jan Vykopal, Martin Vizvary, Radek Oslejsek, Pavel Celeda, and Daniel Tovarnak. 2017. Lessons Learned From Complex Hands-on Defence Exercises in a Cyber Range. In 2017 IEEE Frontiers in Education Conference (FIE). IEEE, Indianapolis, USA, 1--8. https://doi.org/10.1109/FIE.2017.8190713

Digital Library

[50]

Muhammad Mudassar Yamin, Basel Katt, and Vasileios Gkioulos. 2020. Cyber ranges and security testbeds: Scenarios, functions, tools and architecture. Computers & Security, Vol. 88 (2020), 101636. https://doi.org/10.1016/j.cose.2019.101636

Digital Library

[51]

Alexandros Zacharis and Constantinos Patsakis. 2023. AiCEF: an AI-assisted cyber exercise content generation framework using named entity recognition. International Journal of Information Security, Vol. 22, 5 (2023), 1333--1354. https://doi.org/10.1007/s10207-023-00693-z

Digital Library

[52]

Grethe Østby, Kieren N. Lovell, and Basel Katt. 2019. EXCON Teams in Cyber Security Training. In 2019 International Conference on Computational Science and Computational Intelligence (CSCI). IEEE, Las Vegas, USA, 14--19. https://doi.org/10.1109/CSCI49370.2019.00010

[53]

Conducttr ?. 2023. Crisis simulation platform. https://www.conducttr.com/ Retrieved January 17, 2024 from https://www.conducttr.com/

Cited By

Badea DHalmaghi EGorski ABucoveţchi O(2024)Building the Urban Organizational Capability for Resilience as a Matter of Updating Security Education of CommunityInternational conference KNOWLEDGE-BASED ORGANIZATION10.2478/kbo-2024-002830:1(201-207)Online publication date: 4-Jul-2024
https://doi.org/10.2478/kbo-2024-0028
Švábenský VVykopal JHorák MHofbauer MČeleda PMonga MLonati VBarendsen ESheard JPaterson J(2024)From Paper to Platform: Evolution of a Novel Learning Environment for Tabletop ExercisesProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653639(213-219)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653639

Index Terms

Research and Practice of Delivering Tabletop Exercises
1. General and reference
  1. Document types
    1. Surveys and overviews
2. Social and professional topics
  1. Professional topics
    1. Computing education

Recommendations

From Paper to Platform: Evolution of a Novel Learning Environment for Tabletop Exercises
ITiCSE 2024: Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1

For undergraduate students of computing, learning to solve complex practical problems in a team is an essential skill for their future careers. This skill is needed in various fields, such as in cybersecurity and IT governance. Tabletop exercises are an ...
AutoCAD Exercises
CAD Exercises

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ITiCSE 2024: Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1

July 2024

776 pages

ISBN:9798400706004

DOI:10.1145/3649217

General Chairs:
Mattia Monga
University of Milan, Italy
,
Violetta Lonati
University of Milan, Italy
,
Erik Barendsen
Radboud University, The Netherlands
,
Program Chairs:
Judithe Sheard
Monash University, Australia
,
James Paterson
Glasgow Caledonian University, Scotland

Copyright © 2024 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 July 2024

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Ministry of the Interior of the Czech Republic

Conference

ITiCSE 2024

Sponsor:

SIGCSE

ITiCSE 2024: Innovation and Technology in Computer Science Education

July 8 - 10, 2024

Milan, Italy

Acceptance Rates

Overall Acceptance Rate 552 of 1,613 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
312
Total Downloads

Downloads (Last 12 months)312
Downloads (Last 6 weeks)86

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Badea DHalmaghi EGorski ABucoveţchi O(2024)Building the Urban Organizational Capability for Resilience as a Matter of Updating Security Education of CommunityInternational conference KNOWLEDGE-BASED ORGANIZATION10.2478/kbo-2024-002830:1(201-207)Online publication date: 4-Jul-2024
https://doi.org/10.2478/kbo-2024-0028
Švábenský VVykopal JHorák MHofbauer MČeleda PMonga MLonati VBarendsen ESheard JPaterson J(2024)From Paper to Platform: Evolution of a Novel Learning Environment for Tabletop ExercisesProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653639(213-219)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653639

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents