Three-Class EEG-fTCD Brain-Computer Interfaces

Most of the existing brain-computer interfaces (BCIs) afford limited degrees of freedom since they usually rely on binary selection. However, it is essential, especially for BCI control applications, to design systems that allow multiple degrees of freedom. In this paper, we investigate the feasibility of increasing the possible number of selections that can be obtained through our EEGfTCD hybrid BCI. This BCI utilizes two different presentation paradigms including motor imagery (MI) and flickering mental rotation (MR)/ word generation (WG). These paradigms present two tasks to the BCI users as well as a fixation cross that represents the case when the user does not intend to perform any action. Here, we solve the 3-class problem of both the MI and flickering MR/WG presentation paradigms. Wavelet decomposition was used to analyze fTCD data of both paradigms while basic binary common spatial pattern (CSP) algorithm was extended to analyze 3class EEG MI data. Template matching was used to analyze EEG data collected using MR/WG paradigm. Average accuracies of 90.60% and 96.58% were achieved using the flickering MR/WG paradigm and MI paradigm, respectively while average information transfer rates (ITRs) of 25.70 bits/min and 43.59 bits/min were obtained using the same paradigms, respectively. In terms of accuracy and speed, the 3-class EEG-fTCD BCI employing the MI paradigm outperforms EEGfNIRS systems in comparison. Moreover, such analysis results suggest that the MI hybrid BCI is a more viable candidate for real-time BCI applications compared to the flickering MR/WG one and EEG-fNIRS systems in literature. Keywords—Electroencephalogram, Functional Transcranial Doppler Ultrasound, Hybrid Brain-Computer Interfaces, Common Spatial Pattern, Template Matching, Wavelet Decomposition, Probabilistic Fusion. *Corresponding author’s email: afk17@ pitt.edu.


Introduction
Brain computer interfaces (BCIs) aim to infer user intent through measuring, analyzing, and interpreting certain brain activities [1]. Such systems are of great importance for the users experiencing limited speech and physical abilities as they provide them with means of communication and control without requiring any muscular input [2]. Depending on how the brain activity is measured, BCIs can be categorized into invasive or non-invasive BCIs. Due to the low risk associated with noninvasive BCIs, they are more common compared to invasive ones [3].
Through such non-invasive systems, the BCI users can communicate with the surrounding environment without any surgical interventions [4].
One of the main components of a BCI system is the signal acquisition modality that is used to sense a certain brain activity [1]. Examples of these modalities include electroencephalography (EEG) [5], functional near infrared spectroscopy (fNIRS) [6], and functional magnetic resonance imaging (fMRI) [7]. EEG is the most common neuroimaging modality used to design noninvasive BCIs due to its low cost, high temporal resolution, and portability [4], [8]. However, performance of single modal BCIs decreases significantly when used in complex environments, controlled by motor-impaired patients, or used for long periods [9], [10], [11].
To overcome the limitations of single modal BCIs as well as improving their performance, multimodal BCIs have been extensively investigated [12]. Multimodal BCIs can be categorized into two categories including BCIs measuring brain and non-brain signals and BCIs measuring different brain signals using different neuroimaging modalities. Examples on the first category include systems combining EEG and Electrooculography (EOG) or EEG and electromyogram (EMG) [13]. As for the second category, the most commonly used modality with EEG for hybrid BCI design is functional near-infrared spectroscopy (fNIRS). However, fNIRS lacks the high temporal resolution required for real-time BCI applications [14].
In contrast, we have shown previously that functional transcranial Doppler (fTCD) can be used to design real-time BCIs [15]. Inspired by such a finding, we designed a multi-modal BCI that uses EEG and fTCD simultaneously to measure electrical activity and cerebral blood velocity, respectively. This multimodal BCI was designed and tested using two different presentation paradigms including motor imagery (MI) and flickering mental rotation (MR)/ word generation (WG) paradigms [16], [17]. Moreover, it was found that the proposed EEG-fTCD BCI outperforms all EEG-fNIRS systems in literature in terms of accuracy and speed [18], [19].
In this paper, to investigate the feasibility of increasing the possible number of the hybrid BCI selections, we solve the 3-class problems of both the MI and flickering MR/WG paradigm. In particular, for the MI paradigm, we analyzed the 3-class fTCD data using wavelet decomposition and extended the basic binary CSP algorithm to multi-class CSP to extract features from 3-class EEG data. As for the flickering MR/WG paradigm, we used template matching to analyze the 3class EEG data while features were extracted from the fTCD data using wavelet decomposition.
Wilcoxon rank sum test, which is a binary feature selection technique, was extended to select the significant features calculated for the 3 classes of each classification problem. For both paradigms, EEG and fTCD feature vectors were projected into 3 EEG and 3 fTCD scalar scores, respectively using support vector machines (SVM) classifier. A Bayesian fusion approach was used to fuse EEG and fTCD scores under the assumption that these evidences are independent.

Materials and Methods
In this section, we describe the EEG-fTCD system as well as our experimental design. In addition, we explain our feature extraction, selection, and fusion approaches to solve 3-class problems of both the MI and flickering MR/WG paradigms.

Data Acquisition
A g.tec EEG system was employed for EEG data acquisition using 16 EEG electrodes placed at positions Fp1, Fp2, F3, F4, Fz, Fc1, Fc2, Cz, P1, P2, C1, C2, Cp3, Cp4, P5, and P6. Reference electrode was placed over left mastoid. The collected data were sampled with a sampling rate of 256 samples/s. Moreover, data were filtered using the g.tec amplifier's 8th order bandpass filter fTCD data collection was performed using a SONARA TCD system with two 2 MHz transducers placed on the left and right sides of the transtemporal window located above the zygomatic arch [20]. Given that middle cerebral arteries (MCAs) are responsible of approximately 80% of brain blood perfusion [21], the fTCD depth was set to the depth of the mid-point of the MCAs which is 50 mm [22].

Presentation Paradigms
We designed two different presentation paradigms to be used with the proposed hybrid BCI. The first paradigm employed motor imagery (MI) tasks while the second paradigm used flickering mental rotation (MR) and word generation (WG) tasks as shown in Fig. 1.a and Fig. 1.b, respectively. For both paradigms, while acquiring EEG and fTCD simultaneously, two tasks and a fixation cross that represents the baseline were presented on the screen. Total of 150 trials were presented to each user and during each trial, a vertical arrow randomly selected one of the 3 visual icons representing the 2 tasks and the baseline. The vertical arrow pointed to the selected icon for duration of 10 s and the user was asked to perform the mental task specified by that arrow until the arrow points to another visual icon.
During the MI-based presentation scheme, a basic MI task was presented the users as shown in Fig. 1.a. In particular, a horizontal arrow pointing to the right represented right arm MI while a horizontal arrow pointing to the left represented left arm MI [17].
During MR/WG presentation paradigm, since MR and WG task are known to be differentiated using fTCD only, to make them differentiable in terms of EEG, the visual icons representing MR and WG tasks were textured with a flickering checkerboard pattern as seen in Fig. 1.b and they flickered with frequencies of 7 and 17 Hz to induce different SSVEPs in EEG [16]. During WG task, the user was asked to silently generate words starting with the letter shown on the screen while during MR task, the user was given two 3D shapes and was asked to mentally rotate one of these shapes and decide if the shapes were identical or mirrored.

Participants
Research procedures were approved by local Institutional Review Board (IRB) under the University of Pittsburgh IRB number of PRO16080475. All participants signed a written informed consent before starting the experiment. A total of 21 healthy individuals participated in this study.
In particular, to assess flickering MR/WG paradigm, data were collected from 11 individuals (3 females and 8 males) with ages ranging from 25 to 32 years while, to test MI paradigm, data were collected from10 subjects (4 males and 6 females) with ages ranging from 23 to 32 years. None of the participants had a history of migraines, concussions, strokes, heart murmurs, or other brain related injuries. Each participant attended one session that included 150 trials and each trial lasted for 10 seconds.

MI Paradigm
Common spatial pattern (CSP) was used to extract features from the 3-class EEG data of the MI paradigm. Due to its ability to find EEG spatial patterns characteristic to different MI tasks, CSP is one of the most efficient feature extraction techniques for MI EEG data [23]. CSP aims at learning the optimal spatial filters that make the observations from two different MI tasks more discriminative in terms of variance [24], [25]. Such spatial filters can be obtained through solving the following optimization problem: where Σ c is the average trial covariance matrix for class {+, −} and W is the transformation matrix.
Assume each trial data are represented as a matrix where is the number of EEG channels and is the number of EEG samples. Sample covariance matrix of each trial can be estimated as follows: Average trial covariance matrix is calculated as follows: where is the number of trials belonging to class .
(1) is solved by simultaneous diagonalization of the covariance matrices Σ c which can be written as follows: where Λ is a diagonal matrix with eigenvalues , j = 1,2, … N on its diagonal.
Solution of (4) is equivalent to solution of the generalized eigenvalue problem below: where w is the j ℎ generalized eigenvector and = (+) (4) is satisfied when the transformation matrix is equivalent to W = [w 1 , w 2 , … . w ] and is given by: where are the elements on diagonal of Λ .
Higher value of (+) reflects higher variance in the positive class when filtered using the spatial filter w . Since a high value of (+) yields low value of (−) , the same spatial filter w will give a low variance when filtering the negative class. In this paper, we solved the 3-class MI problem by considering all possible numbers of eigenvectors.
Given that CSP is a binary feature extraction technique while the classification problem to be solved is a 3-class problem, we extended basic CSP algorithm to multi-class CSP using one versus one approach [26] which decomposes a k-class problem into −1 2 binary problems that include all possible pairs of the k classes. Therefore, the 3-class problem was decomposed into 3 binary problems. In particular, the training trials of the 3 classes were arranged into 3 different groups such that each group contained the training trials corresponding to each pair of classes.
CSP was applied to the training trials of each group separately and the corresponding eigenvectors were computed. During testing a trial , the EEG data of that trial were projected on the set of eigenvectors of each group separately. In particular, for each group, we spatially filtered EEG data of trial using = 1, 2, 3, …., and 8 eigenvectors from both ends of W matrix.

fTCD Feature Extraction
Wavelet analysis was employed to analyze the 3-class fTCD data collected using both MI and flickering MR/WG paradigms. In particular, 5-level wavelet decomposition [27] was applied to the 2 fTCD data segments corresponding to each trial (one segment per fTCD channel). Four statistical features including mean, variance, skewness, and kurtosis [28], [29] were calculated for each of the 6 wavelet bands obtained from the decomposition leading to computation of 24 features per segment and 48 features per trial.

Feature Selection and Reduction
The Wilcoxon rank-sum test [30]  Given that the Wilcoxon rank-sum test is a binary feature selection technique while the classification problems to be solved are 3-class problems, we employed a one versus one approach [26] which decomposes a k-class problem into −1 2 binary problems that include all possible pairs of the k classes. In particular, each 3-class classification problem was decomposed into 3 binary problems. For each binary problem, the significant features were evaluated separately using the Wilcoxon rank-sum test. The set of selected features for each binary problem was used as an input to one support vector machine (SVM) classifier.
SVM was used to reduce EEG and fTCD feature vectors of each trial separately into scalar SVM scores. As mentioned above, each of the 3-class problems was decomposed into 3 binary problems. Each binary problem has its own significant features and its own SVM classifier where each classifier is trained using data from different pair of classes. During testing a trial belonging to class ( = 1, 2, 3) the selected EEG/fTCD feature vector of that trial is used as input to the 3 binary classifiers. Therefore, for each trial under test, the selected features from the EEG and fTCD feature vectors of that trial are reduced into 3 scalar EEG SVM scores and 3 scalar fTCD SVM scores, respectively corresponding to EEG and fTCD evidences of that trial.

Feature Fusion and Decision Making
In our previous studies [18], [19], we developed a Bayesian fusion approach of EEG and fTCD evidences to infer user intent of a given test trial under 3 different assumptions. In particular, under the first assumption ( 1), EEG and fTCD evidences are assumed to be jointly distributed while under the second assumption ( 2 ), EEG and fTCD evidences are assumed to be independent.
Under assumption ( 3), evidences of EEG and fTCD are assumed to be independent but with unequal contribution towards making a correct decision. Here, we extended that approach to solve the 3-class problem of the MI and flickering MR/WG paradigms. For each of the 3-class problems, 10-fold cross validation was used to define training and testing trials.
Given that trials are presented to each BCI user, these trials are represented by a set of EEG and fTCD evidences Y = {y 1 , … y } where y = {e , f }, e and f are EEG and fTCD evidences respectively for a test trial . The user intent x k for the test trial can be inferred through joint state estimation using EEG and fTCD evidences which can be found through solving the following optimization problem.
x k = arg max x k p(x k |Y = y ) In (7), p(x k |Y) is the posterior distribution of the state x k conditioned on the observations Y where x k = 1, 2, 3. Using Bayes rule, (7) can be rewritten as: where p(Y|x k ) is the state conditional distribution of Y and p(x k ) is the prior distribution of user intent x k . Since the trials are randomized, p(x k ) is assumed to be uniform. Therefore, (8) can be reduced to: To reduce computational complexity, we solved (9) under 2 that assumes that EEG and fTCD evidences are independent. Therefore, (9) can be written as: where p(e|x k ) and p(f|x k ) are the distributions of EEG and fTCD evidences conditioned on the state x k respectively.
SVM EEG and fTCD scores (evidences) of training trials were used separately to compute two 3-dimentional distributions [ (e|x k ) and p(f|x k )] that are assumed to be multivariate Gaussian distributions. To estimate these distributions, EEG and fTCD evidences of the training trials were fitted to a 3-dimentional Gaussian mixture model [31]. For evidences under test y = {e , f }, e and f are plugged in (10) and the user intent x k that yields the maximum likelihood is selected.

Performance Evaluation
For both the MI and flickering MR/WG paradigms, to assess the performance of the hybrid combination compared to EEG only and fTCD only, for each participant, accuracy and information transfer rate (ITR) [32] were calculated and compared. ITR is calculated as follows: where P is the classification accuracy, N is the number of BCI classes, and B is the information transfer rate per trial.
In particular, for each paradigm, accuracy and ITR were calculated at time points 1, 2….,10 s. Maximum accuracy and ITR across the 10-s period (trial length) were reported for the hybrid combination, EEG only, and fTCD only. Moreover, average ITRs of the EEG-fTCD combination, EEG only, and fTCD only were computed across the 10-s trial length.

Results
As shown in Table 1, for the MI paradigm, EEG-fTCD combination under the independence assumption (A2) obtained higher accuracies compared to accuracies obtained using EEG only and fTCD only. In particular, 96.58%, 85.50%, and 43.89% average accuracies were achieved using EEG-fTCD, EEG only, and fTCD only. On the other hand, EEG-fTCD, under A2, obtained average accuracy of 90.60% using the flickering MR/WG paradigm while EEG only and fTCD only yielded 57.54% and 44.66% average accuracies as seen in Table 2.
To assess if the classification model is balanced, for both paradigms, we calculated the accuracy of identifying each task (task sensitivity) as well as the accuracy of identifying the baseline (specificity). As seen in Fig. 2 (a) and Fig. 2 (b), considering the error bars representing variability in sensitivities and specificities across participants, it can be concluded that the EEG-fTCD classification models of the MI and MR/WG paradigms are balanced since the average sensitivities and specificities show very similar values. Moreover, for the MI paradigm, the variability in sensitivities and specificities obtained using the hybrid system is much lower compared to those obtained using fTCD only, while for the MR/WG paradigm the variability in sensitivities and specificities obtained using the hybrid system is much lower than those obtained using EEG only and fTCD only.

Figure 2 Sensitivities and specificities (mean and standard deviation) calculated using EEG-fTCD, EEG only, and fTCD only for the MI paradigm (a), and the flickering MR/WG paradigm (b).
In terms of ITRs, for each participant, we calculated ITRs corresponding to the maximum accuracies obtained using EEG-fTCD, EEG only, and fTCD only. As shown in Fig. 3, ITRs due to the hybrid combination are significantly higher than those obtained using EEG only and fTCD only. In particular, for the MI paradigm, EEG-fTCD obtained average ITR of 19.16 bits/min while EEG only and fTCD only obtained 5.57 and 0.66 bits/min, respectively while for the b)

Discussion
As shown in Tables 1 and 2, the accuracies of the 3-class problems obtained using the hybrid combination are significantly higher than those obtained using EEG only and fTCD only for both the MI and flickering MR/WG paradigms. In particular, for the MI paradigm, EEG-fTCD obtained an average accuracy of 96.58% which is 10% higher than the accuracy obtained using EEG only (the primary modality in the EEG-fTCD multimodal BCI). As for the MR/WG paradigm, the combination achieved 90.60% average accuracy which is 33% higher than that obtained using EEG only.
In line with these results, for both paradigms, ITRs of EEG-fTCD combination significantly outperformed ITRs obtained using EEG only and fTCD only as shown in Fig. 4 especially at 1-second trial length. More specifically, for the MI paradigm, EEG-fTCD achieved approximately an average ITR that is 2.5 times the average ITR obtained using EEG only while for the MR/WG paradigm, the average ITR achieved using EEG-fTCD combination is 1.5 times the average ITR of EEG only.
In terms of both accuracy and ITR, it can be noted that the MI paradigm achieved higher performance compared to the flickering MR/WG system. In particular, the combination achieved 96.58% average accuracy using the MI paradigm and 90.60% average accuracy using the MR/WG paradigm. Moreover, at 1-second trial length, ITRs of 43.59 and 25.70 bits/min were achieved using the MI and MR/WG paradigms, respectively. Such findings in addition to the performance measures obtained in our previous studies [18], [19] suggest that the MI paradigm is a more promising candidate for real-time BCI applications.
Since we introduced our multimodal EEG-fTCD BCI as a faster alternative to EEG-fNIRS BCIs, to show the significance of the proposed system, the performance of this system is compared to the performance of the BCIs in literature that combine EEG and fNIRS in terms of accuracy and trial length. As seen in Table 3, the EEG-fTCD system utilizing the MI paradigm outperforms all the studies in comparison in terms of accuracy. As for the flickering MR/WG paradigm, the achieved accuracy is comparable to many of the studies in literature. However, three out of the four studies that achieved higher accuracy than that achieved by the MR/WG paradigm employed motor execution tasks [33], [35], [36] which requires muscular input from the users unlike the tasks employed by our system. In addition, in terms of the number of possible selections, the proposed system utilizing both the MI and MR/WG paradigms is superior compared to the BCIs in literature since our system can afford 3 different selections while the BCIs in comparison are binary.
In terms of trial length, the proposed system utilizing both presentation paradigms has shorter trial length leading to faster inference of the user intent since the proposed system does not require baseline/rest periods before/after each task unlike the BCIs in comparison. Instead, the visual icon representing the baseline in our system is selected randomly and independent of the visual icons selected before or after it. This is because the baseline is not presented to stabilize the blood flow after each task, but it is considered as a separate task that represents the case when the user does not intend to produce a specific command.

Conclusion
In this paper, we studied the possibility of increasing the number of BCI commands that can be produced through our EEG-fTCD BCI through solving the 3-class problems of both the MI and flickering MR/WG paradigms. In particular, we solved 2 classification problems including right