research-article

Brain Computer Interface for Neuro-rehabilitation With Deep Learning Classification and Virtual Reality Feedback

Authors:

Tamás Karácsony,

John Paulin Hansen,

Helle Klingenberg Iversen,

Sadasivan PuthusserypadyAuthors Info & Claims

AH2019: Proceedings of the 10th Augmented Human International Conference 2019

Article No.: 22, Pages 1 - 8

https://doi.org/10.1145/3311823.3311864

Published: 11 March 2019 Publication History

Abstract

Though Motor Imagery (MI) stroke rehabilitation effectively promotes neural reorganization, current therapeutic methods are immeasurable and their repetitiveness can be demotivating. In this work, a real-time electroencephalogram (EEG) based MI-BCI (Brain Computer Interface) system with a virtual reality (VR) game as a motivational feedback has been developed for stroke rehabilitation. If the subject successfully hits one of the targets, it explodes and thus providing feedback on a successfully imagined and virtually executed movement of hands or feet. Novel classification algorithms with deep learning (DL) and convolutional neural network (CNN) architecture with a unique trial onset detection technique was used. Our classifiers performed better than the previous architectures on datasets from PhysioNet offline database. It provided fine classification in the real-time game setting using a 0.5 second 16 channel input for the CNN architectures. Ten participants reported the training to be interesting, fun and immersive. "It is a bit weird, because it feels like it would be my hands", was one of the comments from a test person. The VR system induced a slight discomfort and a moderate effort for MI activations was reported. We conclude that MI-BCI-VR systems with classifiers based on DL for real-time game applications should be considered for motivating MI stroke rehabilitation.

References

[1]

David Achanccaray, Kevin Acuna, Erick Carranza, and Javier Andreu-Perez. 2017. A virtual reality and brain computer interface system for upper limb rehabilitation of post stroke patients. In 2017 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE). IEEE.

[2]

Roxana Aldea and Monica Fira. 2014. Classifications of motor imagery tasks in brain computer interface using linear discriminant analysis. International Journal of Advanced Research in Artificial Intelligence 3, 7 (2014), 5--9.

[3]

Kai Keng Ang, Cuntai Guan, K S G Chua, Beng Ti Ang, C Kuah, Chuanchu Wang, Kok Soon Phua, Zheng Yang Chin, and Haihong Zhang. 2010. Clinical study of neurorehabilitation in stroke using EEG-based motor imagery brain-computer interface with robotic feedback. In 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology. IEEE.

[4]

Derek B. Archer, Nyeonju Kang, Gaurav Misra, Shannon Marble, Carolynn Patten, and Stephen A. Coombes. 2018. Visual feedback alters force control and functional activity in the visuomotor network after stroke. NeuroImage: Clinical 17 (2018), 505--517.

[5]

Pouya Bashivan, Irina Rish, Mohammed Yeasin, and Noel Codella. 2015. Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks. CoRR (2015). arXiv:cs.LG/1511.06448v3

[6]

M. S. Cameirao, S. B. i. Badia, E. Duarte, A. Frisoli, and P. F. M.J. Verschure. 2012. The Combined Impact of Virtual Reality Neurorehabilitation and Its Interfaces on Upper Extremity Functional Recovery in Patients With Chronic Stroke. Stroke 43, 10 (aug 2012), 2720--2728.

[7]

Antonio Maria Chiarelli, Pierpaolo Croce, Arcangelo Merla, and Filippo Zappasodi. 2018. Deep learning for hybrid EEG-fNIRS brain-computer interface: application to motor imagery classification. Journal of Neural Engineering 15, 3 (apr 2018), 036028.

[8]

Michael A. Dimyan and Leonardo G. Cohen. 2011. Neuroplasticity in the context of motor rehabilitation after stroke. Nature Reviews Neurology 7, 2 (jan 2011), 76--85.

[9]

Bruce H. Dobkin. 2007. Brain-computer interface technology as a tool to augment plasticity and outcomes for neurological rehabilitation. The Journal of Physiology 579, 3 (mar 2007), 637--642.

[10]

Bruce H. Dobkin and Andrew Dorsch. 2013. New Evidence for Therapies in Stroke Rehabilitation. Current Atherosclerosis Reports 15, 6 (apr 2013).

[11]

Hauke Dose, Jakob Skadkær Møller, Helle K. Iversen, and Sadasivan Puthusserypady. 2018. An End-to-end Deep Learning Approach to MI-EEG Signal Classification for BCIs. Expert Systems with Applications 114 (2018), 532--542.

[12]

Bogdan Draganski, Christian Gaser, Volker Busch, Gerhard Schuierer, Ulrich Bogdahn, and Arne May. 2004. Changes in grey matter induced by training. Nature 427, 6972 (jan 2004), 311--312.

[13]

A L Goldberger, L A Amaral, L Glass, J M Hausdorff, P C Ivanov, R G Mark, J E Mietus, G B Moody, C K Peng, and H E Stanley. 2000. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation 101 (June 2000), E215-E220. Issue 23.

[14]

Vikram Shenoy Handiru and Vinod A. Prasad. 2016. Optimized Bi-Objective EEG Channel Selection and Cross-Subject Generalization With Brain--Computer Interfaces. IEEE Transactions on Human-Machine Systems 46, 6 (dec 2016), 777--786.

[15]

Steven G. Kernie and Jack M. Parent. 2010. Forebrain neurogenesis after focal Ischemic and traumatic brain injury. Neurobiology of Disease 37, 2 (feb 2010), 267--274.

[16]

Jasmin Kevric and Abdulhamit Subasi. 2017. Comparison of signal decomposition methods in classification of EEG signals for motor-imagery BCI system. Biomedical Signal Processing and Control 31 (jan 2017), 398--406.

[17]

Youngjoo Kim, Jiwoo Ryu, Ko Keun Kim, Clive C. Took, Danilo P. Mandic, and Cheolsoo Park. 2016. Motor Imagery Classification Using Mu and Beta Rhythms of EEG with Strong Uncorrelating Transform Based Complex Common Spatial Patterns. Computational Intelligence and Neuroscience 2016 (2016), 1--13.

Digital Library

[18]

Wonjun Ko, Jeeseok Yoon, Eunsong Kang, Eunji Jun, Jun-Sik Choi, and Heung-Il Suk. 2018. Deep recurrent spatio-temporal neural network for motor imagery based BCI. In 2018 6th International Conference on Brain-Computer Interface (BCI). IEEE.

[19]

Shiu Kumar, Alok Sharma, Kabir Mamun, and Tatsuhiko Tsunoda. 2016. A Deep Learning Approach for Motor Imagery EEG Signal Classification. In 2016 3rd Asia-Pacific World Congress on Computer Science and Engineering (APWC on CSE). IEEE.

[20]

Peter Langhorne, Fiona Coupar, and Alex Pollock. 2009. Motor recovery after stroke: a systematic review. The Lancet Neurology 8, 8 (aug 2009), 741--754.

[21]

Robert Leeb, Doron Friedman, Gernot R. MÃijller-Putz, Reinhold Scherer, Mel Slater, and Gert Pfurtscheller. 2007. Self-Paced (Asynchronous) BCI Control of a Wheelchair in Virtual Environments: A Case Study with a Tetraplegic. Computational Intelligence and Neuroscience 2007 (2007), 1--8.

Digital Library

[22]

Xiang Li, Shuo Chen, Xiaolin Hu, and Jian Yang. 2018. Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift. arXiv:1801.05134v1 (2018). arXiv:cs.LG/http://arxiv.org/abs/1801.05134v1

[23]

Ana Loboda, Alexandra Margineanu, Gabriela Rotariu, and Anca Mihaela. 2014. Discrimination of EEG-Based Motor Imagery Tasks by Means of a Simple Phase Information Method. International Journal of Advanced Research in Artificial Intelligence 3, 10 (2014).

[24]

Keith R. Lohse, Courtney G. E. Hilderman, Katharine L. Cheung, Sandy Tatla, and H. F. Machiel Van der Loos. 2014. Virtual Reality Therapy for Adults Post-Stroke: A Systematic Review and Meta-Analysis Exploring Virtual Environments and Commercial Games in Therapy. PLoS ONE 9, 3 (mar 2014), e93318.

[25]

Na Lu, Tengfei Li, Xiaodong Ren, and Hongyu Miao. 2017. A Deep Learning Scheme for Motor Imagery Classification based on Restricted Boltzmann Machines. IEEE Transactions on Neural Systems and Rehabilitation Engineering 25, 6 (jun 2017), 566--576.

[26]

Kip A Ludwig, Rachel M Miriani, Nicholas B Langhals, Michael D Joseph, David J Anderson, and Daryl R Kipke. 2009. Using a common average reference to improve cortical neuron recordings from microelectrode arrays. Journal of neurophysiology 101 (March 2009), 1679--1689. Issue 3.

[27]

F. Jović M. Tolić. 2013. Classification of wavelet transformed EEG signals with neural network for imagined mental and motor tasks. Kinesiology 45, 1 (2013), 130--138. https://hrcak.srce.hr/104591

[28]

Th. Mulder. 2007. Motor imagery and action observation: cognitive tools for rehabilitation. Journal of Neural Transmission 114, 10 (jun 2007), 1265--1278.

[29]

Cheolsoo Park, Clive Cheong Took, and Danilo P. Mandic. 2014. Augmented Complex Common Spatial Patterns for Classification of Noncircular EEG From Motor Imagery Tasks. IEEE Transactions on Neural Systems and Rehabilitation Engineering 22, 1 (jan 2014), 1--10.

[30]

Floriana Pichiorri, Giovanni Morone, Manuela Petti, Jlenia Toppi, Iolanda Pisotta, Marco Molinari, Stefano Paolucci, Maurizio Inghilleri, Laura Astolfi, Febo Cincotti, and Donatella Mattia. 2015. Brain-computer interface boosts motor imagery practice during stroke recovery. Annals of Neurology 77, 5 (mar 2015), 851--865.

[31]

Floriana Pichiorri, Natalie Mrachacz-Kersting, Marco Molinari, Sonja Kleih, Andrea KÃijbler, and Donatella Mattia. 2016. Brain-computer interface based motor and cognitive rehabilitation after stroke -- state of the art, opportunity, and barriers: summary of the BCI Meeting 2016 in Asilomar. Brain-Computer Interfaces 4, 1-2 (oct 2016), 53--59.

[32]

Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. 2018. On the Convergence of Adam and Beyond. In International Conference on Learning Representations. https://openreview.net/forum?id=ryQu7f-RZ

[33]

G. Schalk, D.J. McFarland, T. Hinterberger, N. Birbaumer, and J.R. Wolpaw. 2004. BCI2000: A General-Purpose Brain-Computer Interface (BCI) System. IEEE Transactions on Biomedical Engineering 51, 6 (jun 2004), 1034--1043.

[34]

Robin Tibor Schirrmeister, Lukas Gemein, Katharina Eggensperger, Frank Hutter, and Tonio Ball. 2017. Deep learning with convolutional neural networks for decoding and visualization of EEG pathology. Human Brain Mapping 38, 11 (aug 2017), 5391--5420. arXiv:cs.LG/1708.08012v1

[35]

N. Sharma, V. M. Pomeroy, and J.-C. Baron. 2006. Motor Imagery: A Backdoor to the Motor System After Stroke? Stroke 37, 7 (jun 2006), 1941--1952.

[36]

Yurun Shen, Hongtao Lu, and Jie Jia. 2017. Classification of Motor Imagery EEG Signals with Deep Learning Models. In Lecture Notes in Computer Science. Springer International Publishing, 181--190.

[37]

Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. ICLR (2014). arXiv:cs.CV/1409.1556v6

[38]

Ryan Spicer, Julia Anglin, David M. Krum, and Sook-Lei Liew. 2017. RELNVENT: A low-cost, virtual reality brain-computer interface for severe stroke upper limb motor recovery. In 2017 IEEE Virtual Reality (VR). IEEE.

[39]

Yousef Rezaei Tabar and Ugur Halici. 2016. A novel deep learning approach for classification of EEG motor imagery signals. Journal of Neural Engineering 14, 1 (nov 2016), 016003.

[40]

Zhichuan Tang, Chao Li, and Shouqian Sun. 2017. Single-trial EEG classification of motor imagery using deep convolutional neural networks. Optik - International Journal for Light and Electron Optics 130 (feb 2017), 11--18.

[41]

Wei-Peng Teo, Makii Muthalib, Sami Yamin, Ashlee M. Hendy, Kelly Bramstedt, Eleftheria Kotsopoulos, Stephane Perrey, and Hasan Ayaz. 2016. Does a Combination of Virtual Reality, Neuromodulation and Neuroimaging Provide a Comprehensive Platform for Neurorehabilitation? -- A Narrative Review of the Literature. Frontiers in Human Neuroscience 10 (jun 2016).

[42]

L.E.H. van Dokkum, T. Ward, and I. Laffont. 2015. Brain computer interfaces for neurorehabilitation -- its current status as a rehabilitation strategy post-stroke. Annals of Physical and Rehabilitation Medicine 58, 1 (feb 2015), 3--8.

[43]

Athanasios Vourvopoulos, André Ferreira, and Sergi Bermudez i Badia. 2016. NeuRow: An Immersive VR Environment for Motor-Imagery Training with the Use of Brain-Computer Interfaces and Vibrotactile Feedback. In Proceedings of the 3rd International Conference on Physiological Computing Systems. SCITEPRESS - Science and Technology Publications.

[44]

Yijun Wang, Shangkai Gao, and Xiaornog Gao. 2005. Common Spatial Pattern Method for Channel Selektion in Motor Imagery Based Brain-computer Interface. Conference proceedings: ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual Conference 5 (2005), 5392--5395.

[45]

Jin Zhang, Chungang Yan, and Xiaoliang Gong. 2017. Deep convolutional neural network for decoding motor imagery based brain computer interface. In 2017 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC). IEEE.

Cited By

Shin DLee DHan JSon YKam T(2025)Population-based evolutionary search for joint hyperparameter and architecture optimization in brain-computer interfaceExpert Systems with Applications10.1016/j.eswa.2024.125832264(125832)Online publication date: Mar-2025
https://doi.org/10.1016/j.eswa.2024.125832
Arif AWang YYin RZhang XHelmy A(2024)EF-Net: Mental State Recognition by Analyzing Multimodal EEG-fNIRS via CNNSensors10.3390/s2406188924:6(1889)Online publication date: 15-Mar-2024
https://doi.org/10.3390/s24061889
Martín-Chinea KGómez-González JAcosta L(2024)Brain–Computer Interface Based on PLV-Spatial Filter and LSTM Classification for Intuitive Control of AvatarsElectronics10.3390/electronics1311208813:11(2088)Online publication date: 27-May-2024
https://doi.org/10.3390/electronics13112088
Show More Cited By

Index Terms

Brain Computer Interface for Neuro-rehabilitation With Deep Learning Classification and Virtual Reality Feedback
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Improving Classification Performance by Combining Feature Vectors with a Boosting Approach for Brain Computer Interface (BCI)
ICCBB '17: Proceedings of the 2017 International Conference on Computational Biology and Bioinformatics

Brain-computer interfaces (BCI) are an interesting emerging technology providing an efficient communication system between human brain and external devices like computers or neuroprosthesis. Among assorts of neuroimaging techniques, electroencephalogram ...
Design of Wearable Brain Computer Interface Based on Motor Imagery
IIH-MSP '14: Proceedings of the 2014 Tenth International Conference on Intelligent Information Hiding and Multimedia Signal Processing

Brain computer interface (BCI) is a communication system to establish a communication interface between the brain and the device. Motor imagery-based brain computer interface is one of asynchronous BCIs, and have been widely developed in recent years. ...
Development of a Wearable Motor-Imagery-Based Brain---Computer Interface

A motor-imagery-based brain---computer interface (BCI) is a translator that converts the motor intention of the brain into a control command to control external machines without muscles. Numerous motor-imagery-based BCIs have been successfully proposed ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

AH2019: Proceedings of the 10th Augmented Human International Conference 2019

March 2019

301 pages

ISBN:9781450365475

DOI:10.1145/3311823

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 March 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

AH2019

AH2019: Augmented Human International Conference 2019

March 11 - 12, 2019

Reims, France

Acceptance Rates

AH2019 Paper Acceptance Rate 26 of 53 submissions, 49%;

Overall Acceptance Rate 121 of 306 submissions, 40%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

48
Total Citations
View Citations
878
Total Downloads

Downloads (Last 12 months)118
Downloads (Last 6 weeks)15

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Shin DLee DHan JSon YKam T(2025)Population-based evolutionary search for joint hyperparameter and architecture optimization in brain-computer interfaceExpert Systems with Applications10.1016/j.eswa.2024.125832264(125832)Online publication date: Mar-2025
https://doi.org/10.1016/j.eswa.2024.125832
Arif AWang YYin RZhang XHelmy A(2024)EF-Net: Mental State Recognition by Analyzing Multimodal EEG-fNIRS via CNNSensors10.3390/s2406188924:6(1889)Online publication date: 15-Mar-2024
https://doi.org/10.3390/s24061889
Martín-Chinea KGómez-González JAcosta L(2024)Brain–Computer Interface Based on PLV-Spatial Filter and LSTM Classification for Intuitive Control of AvatarsElectronics10.3390/electronics1311208813:11(2088)Online publication date: 27-May-2024
https://doi.org/10.3390/electronics13112088
Tadayyoni HRamirez Campos MQuevedo AMurphy B(2024)Biomarkers of Immersion in Virtual Reality Based on Features Extracted from the EEG Signals: A Machine Learning ApproachBrain Sciences10.3390/brainsci1405047014:5(470)Online publication date: 7-May-2024
https://doi.org/10.3390/brainsci14050470
Sangbamrung IKanjanawattana SBhakdisongkhram G(2024)Determination of the Optimal Electrode Selection for Motor Imaginary Classification with EEGNetProceedings of the 2024 10th International Conference on Computer Technology Applications10.1145/3674558.3674567(57-65)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3674558.3674567
Mishra AGupta ISrivastava SAlfarhood SSafran M(2024) Empowering EEG motor imagery classification with deep transfer learning approach Expert Systems10.1111/exsy.1353041:4Online publication date: 9-Jan-2024
https://doi.org/10.1111/exsy.13530
Xue ZZhang YLi HChen HShen SDu H(2024)Instrumentation, Measurement, and Signal Processing in Electroencephalography-Based Brain-Computer Interfaces: Situations and ProspectsIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.341759873(1-28)Online publication date: 2024
https://doi.org/10.1109/TIM.2024.3417598
Zhu HHieu NHoang DNguyen DLin C(2024)A Human-Centric Metaverse Enabled by Brain-Computer Interface: A SurveyIEEE Communications Surveys & Tutorials10.1109/COMST.2024.338712426:3(2120-2145)Online publication date: Nov-2025
https://doi.org/10.1109/COMST.2024.3387124
Taghizadeh MVaez FFaezipour M(2024)EEG Motor Imagery Classification by Feature Extracted Deep 1D-CNN and Semi-Deep Fine-TuningIEEE Access10.1109/ACCESS.2024.343083812(111265-111279)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3430838
Hadjiaros MShimi AAvraamides MNeokleous KPattichis C(2024)Virtual Reality Brain–Computer Interfacing and the Role of Cognitive SkillsIEEE Access10.1109/ACCESS.2024.342662012(129240-129261)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3426620
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents