A Multi-Step Neural Control for Motor Brain-Machine Interface by Reinforcement Learning

Abstract:

Brain-machine interfaces (BMIs) decode cortical neural spikes of paralyzed patients to control external devices for the purpose of movement restoration. Neuroplasticity induced by conducting a relatively complex task within multistep, is helpful to performance improvements of BMI system. Reinforcement learning (RL) allows the BMI system to interact with the environment to learn the task adaptively without a teacher signal, which is more appropriate to the case for paralyzed patients. In this work, we proposed to apply Q(λ)-learning to multistep goal-directed tasks using users neural activity. Neural data were recorded from M1 of a monkey manipulating a joystick in a center-out task. Compared with a supervised learning approach, significant BMI control was achieved with correct directional decoding in 84.2% and 81% of the trials from naïve states. The results demonstrate that the BMI system was able to complete a task by interacting with the environment, indicating that RL-based methods have the potential to develop more natural BMI systems.

• Periodical: Applied Mechanics and Materials (Volume 461)
• Pages: 565-569
• DOI: https://doi.org/10.4028/www.scientific.net/AMM.461.565
• Online since: November 2013
• Authors: Fang Wang, Kai Xu, Qiao Sheng Zhang, Yi Wen Wang, Xiao Xiang Zheng
• Keywords: Brain-Machine Interface (BMI), Multi-Step, Neural Control, Reinforcement Learning

References

[1] J. Wessberg, et al., Real-time prediction of hand trajectory by ensembles of cortical neurons in primates, Nature. 408 (2000) 361-365.

DOI: 10.1038/35042582

[2] S. Kim, et al., A comparison of optimal MIMO linear and nonlinear models for brain–machine interfaces. J. Neural Eng. 3 (2006) 145-161.

[3] Y.W. Wang, et al., Sequential Monte Carlo point-process estimation of kinematics from neural spiking activity for brain-machine interfaces. Neural Comput. 21 (2009) 2894-2930.

DOI: 10.1162/neco.2009.01-08-699

[4] G.J. Gage, K.A. Ludwig, D. Kipke, Naive coadaptive cortical control, J. Neural Eng. 2 (2005) 52-63.

DOI: 10.1088/1741-2560/2/2/006

[5] M. Velliste, et al., Cortical control of a prosthetic arm for self-feeding. Nature, 453 (2008) 1098-1101.

DOI: 10.1038/nature06996

[6] J. DiGiovanna, et al., Coadaptive brain-machine interface via reinforcement learning. IEEE Trans. Biomed. Eng. 56 (2009) 54-64.

DOI: 10.1109/tbme.2008.926699

[7] R.S. Sutton, A.G. Barto, Reinforcement learning: An introduction. Cambridge Univ Press, (1998).

[8] B. Mahmoudi, J.C. Sanchez, A symbiotic brain-machine interface through value-based decision making. Plos One, 6 (2011) e14760.

DOI: 10.1371/journal.pone.0014760

[9] J.C. Sanchez, et al. Control of a center-out reaching task using a reinforcement learning Brain-Machine Interface. Neural Eng. (NER), 2011 5th International IEEE/EMBS Conference on. (2011) 525-528.

DOI: 10.1109/ner.2011.5910601

[10] R.S. Sutton, Learning to predict by the methods of temporal differences. Machine learning, 3(1) (1988) 9-44. 1The total number of firing rate samples is 216, using 3-fold cross validation.