Brain-Computer Interface for home-use application
There are several diseases, such as amyotrophic lateral sclerosis (ALS), which can lead to a loss of the ability to communicate. As a healthy person one cannot imagine what it feels like to be trapped in one’s own body - mentally present, but unable to communicate with relatives, this is called locked-in syndrome. However, a distinction must be made between locked-in and complete locked-in syndrome (CLIS). For the former, those affected can still voluntarily control certain muscles, above all the eye muscles, which in turn can be used for communication. Brain-computer interfaces (BCIs), i.e. systems that allow to control a computer by pure brain activity, have proven to be a helpful method for restoring the ability to communicate. However, all recent BCI systems are almost exclusively used in research, since all previous methods are not suitable for real-world applications. This is mainly due to the fact that for a meaningful and independent use, the recognition of the user’s intention (to control the system or not) must be highly accurate. Otherwise, this leads to random classifications/commands, which can be dangerous depending on the application, for example when controlling an electric wheelchair.
We have tackled this issue and developed a self-paced BCI based on complex visual stimulation patterns (see publications below). It was tested in the context of a spelling application, for this a virtual keyboard is shown on a computer monitor. While the user gazes a specific target (letter) which is modulated with its own unique blinking (black-white) stimulation pattern, the underlying model predicts the stimulation pattern based on the recorded Electroencephalogram (EEG). The predicted stimulation pattern is compared to all possible stimulation patterns using a similarity measure to classify the correct letter. This approach allows to spell up to 49 error-free letters per minute, whereby a target classification accuracy of almost constant 100% was achieved. In average, the method is 3 times faster than the previously fastest method. In addition, the intention to control the system was always recognized for all subjects. The intention not to control the system could always be detected for 80% of all subjects. Compared to the previously best method, the recognition rate is improved by an average factor of 6.5. Furthermore, the method allows an almost infinite number of choices to be distinguished, whereby this was shown for up to 500,000 choices by a simulation. Even with this enormous classification problem, accuracies of up to 100% were achieved. These results could only be achieved by a deep learning approach, which illustrates the importance of machine learning in this research area.
However, there are still three key points for further optimization required to make the BCI suitable for home-use, (1) subject specific adaptivity, (2) elimination of unintended commands, and (3) distinct stimulation patterns. These points should be addressed in the scope of this project:
- (1) Subject specific adaptivity is necessary for a BCI to react to changing conditions, like the fact that brain activities are non-stationary over time, even within the same user. Without an adaptive learning approach, the performance would drop over time. Due to the very high classification accuracy, the underlying model could be trained adaptively in a so-called semi-supervised procedure. This means that the brain signal that accumulates over time are continuously used for training by labeling it with the predicted classes.
- (2) Elimination of unintended commands: As mentioned, the state when the user does not intent to control the BCI, called non-control state, is as important as the control state or even more important for the reasons given. For this, a user specific threshold is required, which has so far been more or less arbitrarily determined. Although the results outperform recent methods by far, the error-rate must be reduced to 0 in order to ensure safe use. To achieve this, the threshold value should be determined using a machine learning approach which also adapts over time. In the scope of this project, several approaches should be evaluated.
- (3) Distinct stimulation patterns: The BCI is based on complex (but random) visual stimulation patterns. In a recent work [2] it was shown that the stimulation patterns can be optimized to improve the performance. In the scope of this work, an automatic machine learning approach should be developed which automatically “finds” a set of optimal stimulation patterns. Whereby the resultant set of stimulation pattern should be continuously improved.
By solving these points, a robust, accurate and user friendly BCI for communication in daily live application can be achieved. In a later phase of the project, the resultant BCI shall be evaluated with at least one potential end-user, like an ALS patient, over a period of 6 months to ensure a robust and reliable use.
Participating Team Members
Publications
2019
Towards a home-use BCI: fast asynchronous control and robust non-control state detection
by Sebastian NagelPhD thesis. University of Tübingen, 2019. [BIB] [DOI] [ABSTRACT]
Abstract: Brain-Computer Interfaces (BCIs) enable users to control a computer by using pure brain activity. Their main purpose is to restore several functionalities of motor disabled people, for example, to restore the communication ability. Recent BCIs based on visual evoked potentials (VEPs), which are brain responses to visual stimuli, have shown to achieve high-speed communication. However, BCIs have not really found their way out of the lab yet. This is mainly because all recent high-speed BCIs are based on synchronous control, which means commands can only be executed in time slots controlled by the BCI. Therefore, the user is not able to select a command at his own convenience, which poses a problem in real-world applications. Furthermore, all those BCIs are based on stimulation paradigms which restrict the number of possible commands. To be suitable for real-world applications, a BCI should be asynchronous, or also called self-paced, and must be able to identify the user’s intent to control the system or not. Although there some asynchronous BCI approaches, none of them achieved suitable real-world performances. In this thesis, the first asynchronous high-speed BCI is proposed, which allows using a virtually unlimited number of commands. Furthermore, it achieved a nearly perfect distinction between intentional control (IC) and non-control (NC), which means commands are only executed if the user intends to. This was achieved by a completely different approach, compared to recent methods. Instead of using a classifier trained on specific stimulation patterns, the presented approach is based on a general model that predicts arbitrary stimulation patterns. The approach was evaluated with a "traditional" as well as a deep machine learning method. The resultant asynchronous BCI outperforms recent methods by a multi-fold in multiple disciplines and is an essential step for moving BCI applications out of the lab and into real life. With further optimization, discussed in this thesis, it could evolve to the very first end-user suitable BCI, as it is effective (high accuracy), efficient (fast classifications), ease of use, and allows to perform as many different tasks as desired.
@phdthesis{SNT2019,
author = {Nagel, Sebastian},
title = {Towards a home-use BCI: fast asynchronous control and robust non-control state detection},
school = {University of Tübingen},
year = {2019},
month = {dec},
doi = {10.15496/publikation-37739},
month_numeric = {12}
}
World’s fastest brain-computer interface: Combining EEG2Code with deep learning
by Sebastian Nagel and Martin SpülerIn PLOS ONE 14(9): e0221909, 2019. [BIB] [DOI] [ABSTRACT]
Abstract: We present a novel approach based on deep learning for decoding sensory information from non-invasively recorded Electroencephalograms (EEG). It can either be used in a passive Brain-Computer Interface (BCI) to predict properties of a visual stimulus the person is viewing, or it can be used to actively control a BCI application. Both scenarios were tested, whereby an average information transfer rate (ITR) of 701 bit/min was achieved for the passive BCI approach with the best subject achieving an online ITR of 1237 bit/min. Further, it allowed the discrimination of 500,000 different visual stimuli based on only 2 seconds of EEG data with an accuracy of up to 100%. When using the method for an asynchronous self-paced BCI for spelling, an average utility rate of 175 bit/min was achieved, which corresponds to an average of 35 error-free letters per minute. As the presented method extracts more than three times more information than the previously fastest approach, we suggest that EEG signals carry more information than generally assumed. Finally, we observed a ceiling effect such that information content in the EEG exceeds that required for BCI control, and therefore we discuss if BCI research has reached a point where the performance of non-invasive visual BCI control cannot be substantially improved anymore.
@article{SM092019,
author = {Nagel, Sebastian and Spüler, Martin},
title = {World’s fastest brain-computer interface: Combining EEG2Code with deep learning},
journal = {PLOS ONE},
year = {2019},
month = {sep},
volume = {14},
number = {9},
pages = {e0221909},
doi = {10.1371/journal.pone.0221909},
month_numeric = {9}
}
Asynchronous non-invasive high-speed BCI speller with robust non-control state detection
by Sebastian Nagel and Martin SpülerIn Scientific Reports 9(1): 8269, 2019. [BIB] [DOI] [ABSTRACT]
Abstract: Brain-Computer Interfaces (BCIs) enable users to control a computer by using pure brain activity. Recent BCIs based on visual evoked potentials (VEPs) have shown to be suitable for high-speed communication. However, all recent high-speed BCIs are synchronous, which means that the system works with fixed time slots so that the user is not able to select a command at his own convenience, which poses a problem in real-world applications. In this paper, we present the first asynchronous high-speed BCI with robust distinction between intentional control (IC) and non-control (NC), with a nearly perfect NC state detection of only 0.075 erroneous classifications per minute. The resulting asynchronous speller achieved an average information transfer rate (ITR) of 122.7 bit/min using a 32 target matrix-keyboard. Since the method is based on random stimulation patterns it allows to use an arbitrary number of targets for any application purpose, which was shown by using an 55 target German QWERTZ-keyboard layout which allowed the participants to write an average of 16.1 (up to 30.7) correct case-sensitive letters per minute. As the presented system is the first asynchronous high-speed BCI speller with a robust non-control state detection, it is an important step for moving BCI applications out of the lab and into real-life.
@article{SM062019,
author = {Nagel, Sebastian and Spüler, Martin},
title = {Asynchronous non-invasive high-speed BCI speller with robust non-control state detection},
journal = {Scientific Reports},
year = {2019},
month = {jun},
volume = {9},
number = {1},
pages = {8269},
doi = {10.1038/s41598-019-44645-x},
month_numeric = {6}
}
2018
Modelling the brain response to arbitrary visual stimulation patterns for a flexible high-speed Brain-Computer Interface
by Sebastian Nagel and Martin SpülerIn PLoS one 13(10): e0206107, 2018. [BIB] [DOI] [ABSTRACT]
Abstract: Visual evoked potentials (VEPs) can be measured in the EEG as response to a visual stimulus. Commonly, VEPs are displayed by averaging multiple responses to a certain stimulus or a classifier is trained to identify the response to a certain stimulus. While the traditional approach is limited to a set of predefined stimulation patterns, we present a method that models the general process of VEP generation and thereby can be used to predict arbitrary visual stimulation patterns from EEG and predict how the brain responds to arbitrary stimulation patterns. We demonstrate how this method can be used to model single-flash VEPs, steady state VEPs (SSVEPs) or VEPs to complex stimulation patterns. It is further shown that this method can also be used for a high-speed BCI in an online scenario where it achieved an average information transfer rate (ITR) of 108.1 bit/min. Furthermore, in an offline analysis, we show the flexibility of the method allowing to modulate a virtually unlimited amount of targets with any desired trial duration resulting in a theoretically possible ITR of more than 470 bit/min.
@article{SM102018,
author = {Nagel, Sebastian and Spüler, Martin},
title = {Modelling the brain response to arbitrary visual stimulation patterns for a flexible high-speed Brain-Computer Interface},
journal = {PLoS one},
year = {2018},
month = {oct},
volume = {13},
number = {10},
pages = {e0206107},
doi = {10.1371/journal.pone.0206107},
month_numeric = {10}
}
Finding optimal stimulation patterns for BCIs based on visual evoked potentials
by Sebastian Nagel, Wolfgang Rosenstiel, and Martin SpülerIn Proceedings of the 7th International BCI Meeting 2018, pages 164-165, 2018. [BIB] [PDF]
@inproceedings{SWM052018,
author = {Nagel, Sebastian and Rosenstiel, Wolfgang and Spüler, Martin},
title = {Finding optimal stimulation patterns for BCIs based on visual evoked potentials},
booktitle = {Proceedings of the 7th International BCI Meeting 2018},
year = {2018},
month = {may},
pages = {164-165},
address = {Asilomar, CA},
month_numeric = {5}
}
The effect of monitor raster latency on VEPs, ERPs and Brain-Computer Interface performance
by Sebastian Nagel, Werner Dreher, Wolfgang Rosenstiel, and Martin SpülerIn Journal of Neuroscience Methods 295: 45-50, 2018. [BIB] [DOI] [ABSTRACT]
Abstract: BACKGROUND: Visual neuroscience experiments and Brain–Computer Interface (BCI) control often require strict timings in a millisecond scale. As most experiments are performed using a personal computer (PC), the latencies that are introduced by the setup should be taken into account and be corrected. As a standard computer monitor uses a rastering to update each line of the image sequentially, this causes a monitor raster latency which depends on the position, on the monitor and the refresh rate. NEW METHOD: We technically measured the raster latencies of different monitors and present the effects on visual evoked potentials (VEPs) and event-related potentials (ERPs). Additionally we present a method for correcting the monitor raster latency and analyzed the performance difference of a code-modulated VEP BCI speller by correcting the latency. COMPARISON WITH EXISTING METHODS: There are currently no other methods validating the effects of monitor raster latency on VEPs and ERPs. RESULTS: The timings of VEPs and ERPs are directly affected by the raster latency. Furthermore, correcting the raster latency resulted in a significant reduction of the target prediction error from 7.98% to 4.61% and also in a more reliable classification of targets by significantly increasing the distance between the most probable and the second most probable target by 18.23%.
@article{SWWM012018,
author = {Nagel, Sebastian and Dreher, Werner and Rosenstiel, Wolfgang and Spüler, Martin},
title = {The effect of monitor raster latency on VEPs, ERPs and Brain-Computer Interface performance},
journal = {Journal of Neuroscience Methods},
year = {2018},
month = {jan},
volume = {295},
pages = {45-50},
doi = {10.1016/j.jneumeth.2017.11.018},
month_numeric = {1}
}
2017
Random Visual Evoked Pontentials (RVEP) for Brain-Computer Interface (BCI) Control
by Sebastian Nagel, Wolfgang Rosenstiel, and Martin SpülerIn Proceedings of the 7th Graz Brain-Computer Interface Conference 2017, pages 349-354. Verlag der TU Graz, 2017. [BIB] [DOI] [ABSTRACT]
Abstract: Brain-Computer Interfaces (BCIs) enable users to control devices or communicate by using brain activity only. While BCIs based on visual evoked potentials (VEPs) have been shown to achieve high performance, we present a different paradigm for BCI control: random VEP (rVEP). We designed a regression model, trained on VEPs of fully random bit codes. Afterwards, the model is able to perform a bit-wise prediction of a previously unseen stimulation sequence, which in turn can be used for BCI control. In an offline study, the model predicts unknown stimulation sequences with an average ITR of 94.5 bits per minute (bpm) and up to 281 bpm on a single-trial level. In a copy-spelling task, the model achieved an average ITR of 64.3 bpm and up to 115.5 bpm.
@inproceedings{SWM092017,
author = {Nagel, Sebastian and Rosenstiel, Wolfgang and Spüler, Martin},
title = {Random Visual Evoked Pontentials (RVEP) for Brain-Computer Interface (BCI) Control},
booktitle = {Proceedings of the 7th Graz Brain-Computer Interface Conference 2017},
publisher = {Verlag der TU Graz},
year = {2017},
month = {sep},
pages = {349-354},
doi = {10.3217/978-3-85125-533-1-64},
month_numeric = {9}
}