A novel energy-motion model for continuous sEMG decoding: from muscle energy to motor pattern

At present, sEMG-based gesture recognition requires vast amounts of training data; otherwise it is limited to a few gestures. Objective. This paper presents a novel dynamic energy model that decodes continuous hand actions by training small amounts of sEMG data. Approach. The activation of forearm muscles can set the corresponding fingers in motion or state with movement trends. The moving fingers store kinetic energy, and the fingers with movement trends store potential energy. The kinetic energy and potential energy in each finger are dynamically allocated due to the adaptive-coupling mechanism of five-fingers in actual motion. Meanwhile, the sum of the two energies remains constant at a certain muscle activation. We regarded hand movements with the same direction of acceleration for five-finger as the same in energy mode and divided hand movements into ten energy modes. Independent component analysis and machine learning methods were used to model associations between sEMG signals and energy modes and expressed gestures by energy form adaptively. This theory imitates the self-adapting mechanism in actual tasks. Thus, ten healthy subjects were recruited, and three experiments mimicking activities of daily living were designed to evaluate the interface: (1) the expression of untrained gestures, (2) the decoding of the amount of single-finger energy, and (3) real-time control. Main results. (1) Participants completed the untrained hand movements (100/100, p < 0.0001). (2) The interface performed better than chance in the experiment where participants pricked balloons with a needle tip (779/1000, p < 0.0001). (3) In the experiment where participants punched a hole in the plasticine on the balloon, the success rate was over 95% (97.67 ± 5.04%, p < 0.01). Significance. The model can achieve continuous hand actions with speed or force information by training small amounts of sEMG data, which reduces learning task complexity.


Introduction
Hand loss is a highly disabling event that markedly affects the quality of life [3]. In order to replace the capabilities lost, the replacement should be designed by mimicking the native hands faithfully and provide users with intuitive control, sufficient feedback, and multiple functions (figure 1) [3][4][5][6]. Sixty years ago, the advent of a myoelectric prosthetic hand, an externally powered prosthesis extracting motion intent from electromyogram (EMG) signals, brought hope to hand amputees [7]. However, myoelectric prosthetic hands with intuitive control and multiple functions have not been widely utilized in business.
The first or earliest type has only two motions, e.g. hand open or close [8][9][10]. Two bipolar sEMG electrodes are placed at the flexors and extensors of the residual limb to record information about the neuromuscular activity. The second type is based on motion classification that assigns sEMG features to a discrete set of motions [11][12][13]. Tremendous success has been achieved in this area of research. Recently, Furui et al demonstrate a mechanism that classifies untrained combined motions from a dataset of trained single motions [13]. Despite its success in research, in its basic form, classification only allows for the sequential activation of motions, precluding intuitive control of smooth movements. An error may lead to an utterly Figure 1. Example of the primary mechanism of human hand mimicked in the present study. Active movement at one finger may result in some movements at other fingers unless we activate the corresponding muscles to keep other fingers static. Humans rarely move one finger alone but multiple fingers simultaneously. Multiple gestures are accomplished by the adaptive conversion of fingers' motion and forces according to the task itself [1,2]. We sought to regard the whole manual task as the energy transfer to mimic the human hand's adaptive mechanism. If the five-fingers' accelerations are in the same direction for some states, we regard these states as the same energy mode. P ki expresses the kinetic energy of the i th finger, while P pi expresses the potential energy of the i th finger.
unwanted action that may compromise the full task [5].
Recently, to overcome the limitations of classification-based approaches, regression-based approaches have been proposed. Most researches focus on the estimation of kinematic joint angles of the wrist [14][15][16]. Recently, Zhuang et al succeed in the estimation of the force of the single finger [17]. However, this technique requires considerable training and experience due to the mechanical coupling and physiological coupling of five-fingers [1,2,11,18,19]. This paper proposes a myoelectric interface based on the conservation of energy (figures 2 and 3). For an instant, muscular activation associated with hand movements is constant, the five-fingers are in motion, the state with movement trends or relaxation. From an energy point of view, at some point of muscular activation, the energy of the whole hand is constant. However, the form of energy in five-fingers may be kinetic energy or potential energy. If the fivefingers' accelerations are in the same direction for some states, we regard these states as the same energy mode (figure 1). Therefore, one energy mode can express multiple motor patterns. In this way, we can map massive motor patterns from sEMG by small sample training.
This paper consists of the introduction of our myoelectric interface, three exploratory operational experiments, and off-line analysis. Section 2 introduces an overview of the bionic interface. Section 3 describes the details of the interface and experiments. Section IV shows the results of operational experiments and off-line analysis. In section 5, we discuss results, limitations, and future work and give our conclusions. Note that we provide supplementary materials and movies, and 'figure S' can be found in supplementary materials (available online at stacks.iop.org/JNE/18/016019/mmedia).

Bionic inspiration and interface overview
Inspired by the neuromuscular system of the human hand, we propose a myoelectric interface based on energy allocation. The scheme is shown in figure 2. We imitate two stages that the central nervous system (CNS) activates muscles and that muscles control fingers to complete the task adaptively (figure 2(a)). Strong evidence from EMG of frogs [20], cats [21], primates [22], and humans [23] demonstrate the existence of muscle synergy in the neuromuscular system. In the first stage, a group of synergetic muscles instead of one is activated to perform a certain task, which allows the CNS to activate muscles by controlling a few synergy patterns. In the second stage, the energy of muscles is transferred through mechanical coupling to five-fingers, and manual tasks are performed by converting kinetic energy and potential energy adaptively [2,24].
In light of this, the myoelectric interface is modularized ( figure 2(b)). Firstly, control signals from CNS It includes muscle synergy, mechanical coupling, and the conversion of kinetic energy and potential energy adaptively. The process consists of two stages. In the first stage, synergetic commands from CNS is decoded into activation commands of each muscle. In the second stage, the muscle's biological energy is transferred to the finger through a mechanical coupling. Then the energy is allocated as kinetic energy and potential energy adaptively according to the task with feedback. (b) Schematic diagram of the proposed myoelectric interface. The essence of the myoelectric interface is the allocation of total energy among five-fingers. Furthermore, the size of the colored area on a single finger expresses the amount of energy. The aim is to solve for the pink area of five-fingers (allocation of total energy). are obtained using the filtered sEMG recordings as input to a synergy matrix representing muscle activation strategies from the individual muscles to muscle groups, which highlights the information of synergies from CNS. Secondly, controlling fingers through the muscles is regarded as an energy transfer process based on energy conservation. To unify the condition of static force and motion, according to Newton's Second Law, we assume that a fictitious resistance exists in fingertip during flexion and extension of each finger, and muscle activation aims to overcome the resistance. Thus, according to energy conservation, the total energy of a finger can be regarded as a sum of kinetic energy and potential energy and consists of three forms: kinetic energy, potential energy, or coexistence of kinetic energy and potential energy. At a certain moment of muscle activation, the energy allocation of five fingers is determined accordingly.
For a single finger, the total energy (kinetic energy plus potential energy) is also determined. Resistance catalyzes the interconversion between kinetic energy and potential energy, but the total amount is constant.
The essence of the myoelectric interface is the total energy allocation of five-fingers. Therefore, we solved this problem by two extreme conditions of energy transfer. Figure 3 shows an overview of the proposed myoelectric interface. To simplify the model, we utilize the conditions of wholly transferred kinetic energy and potential energy. The operator's five-fingers energy is estimated by the bionic model using independent component analysis (ICA) and conservation of energy theory, which allows the expression of the untrained hand tasks via a few trained energy modes adaptively [2,25]. Overview of the myoelectric interface based on the conservation of energy. The myoelectric interface (a) is composed of training and application stages. Both stages consist of signals measurement, signal processing, and virtual/bionic hand control. Note that, in the training stage, the sensor signals can be measured on the intact side of the user. Besides, independent component analysis (ICA) can solve for a synergy matrix that represents muscle activation. The synergy matrix is used to extract synergy patterns (b). Furthermore, each energy mode can express multiple hand tasks adaptively (c, an example, the direction of the arrow represents the direction of acceleration of single finger, and all adaptive motor patterns are the same energy mode.). P ki expresses the kinetic energy of the i th finger, while P pi expresses the potential energy of the i th finger.

Scheme for the bionic myoelectric interface
From the perspective of information and energy transfer, the neuromuscular system of human hand consists of muscle synergy, mechanical coupling, and the adaptive conversion of kinetic energy and potential energy (figures 2 and 1). Therefore, we designed our model by mimicking these parts.

ICA to mimic muscle synergy
Evidence from a large number of animal experiments demonstrates that the CNS activates muscles by controlling a small number of synergy patterns rather than individual muscles [20-22, 26, 27]. In other words, a muscle synergy pattern can be expressed by the activation of individual muscles [m 1 (t) , m 2 (t) , · · · , m n (t)] ∈ ℜ n ( n is the number of muscle) Where u (t) = [u 1 (t) , u 2 (t) , · · · , u m (t)] ∈ ℜ m ( m is the number of synergy patterns and m ⩽ n ) and F trans (· · · ) is a function that transforms muscle activation into a smaller number of synergies. Previous experiments of frogs [26] and rats [27] demonstrate that we could use ICA to extract these muscle synergy patterns. Therefore, muscle synergy pattern u (t) is extracted from the time-series sEMG pattern [s 1 (t) , s 2 (t) , · · · , s c (t)] ∈ ℜ c ( c is the number of sEMG electrodes) u (t) + e (t) = F ICAtrans (s 1 (t) , s 2 (t) , · · · , s c (t)) (2) Where F ICAtrans (· · · ) is a function that transforms the time-series sEMG pattern into a smaller number of synergies, and e (t) is the noise of the system (figure 3(b)). u (t) is regarded as the information from CNS.

Energy transfer applied to the human hand
Both mechanical coupling and neuromuscular control limit finger independence [1,2]. For example, active movement at one finger may lead to some movements at another finger (figure 1). To implement the fluid decoding of a single finger, we sought to regard manual tasks as an energy transfer process by mimicking the adaptive mechanism of the human hand. Applying the conservation of energy in the human hand can be expressed as the following formula (figure 2(b)).
Where P is the energy from muscles to hand, P K is the kinetic energy of hand, P P is the potential energy of hand (It usually exists in the form of strain energy), ∆P K is the amount of change in kinetic energy, and ∆P P is the amount of change in potential energy. Also, the energy transfer can be expressed at the level of single fingers as follows.
Here, according to the fingers' adaptive mechanism in task [1,2], to unify the static force and motion condition (Newton's Second Law of motion), we assume that a fictitious resistance exists in fingertip during flexion and extension of each finger. Muscle activation aims to overcome resistance. Thus, the energy transfer in a single finger is as follows.
In other words, in a certain muscle activation condition, although the form of energy is uncertain, the total energy of a single finger is constant. Concretely, according to conservation of energy [28], at some point in time of muscular activation, the neuromuscular system of that time is certain. Thus, each finger's total energy is certain, and other additional systems only change the form of energy. Therefore, we do not care about the form of energy. This process recasts the decoding problem as the problem of energy allocation of the five-fingers (energy mode; the adaptive and expansive expression are shown in figure 3(c)).
To simplify the model, we utilize the conditions of wholly transferred external kinetic energy P ext K or external potential energy P ext P .
Also, the form of a single finger is as follows.
Where P ext k,f ∈ { P ext k,1 , P ext k,2 , · · · , P ext k,5 } ∈ ℜ 5 and } ∈ ℜ 5 are the wholly transferred kinetic energy or potential energy in a single finger. Furthermore, with the help of an additional system, according to the principle of virtual work, the wholly transferred potential energy should be equal to the amount of change of internal energy in the additional system (figures 2(b) and 3(a)).
Where W ext p,f is the work of the energy transfer from finger to the additional system, F ext f is the force applied to the additional system, and δ is the virtual displacement. Besides, compared with equation (1), the energy of the five-fingers is expressed as the following formula.
Then, we utilize the two extreme conditions of energy transfer simultaneously to solve for each finger's total energy, as shown in figure 3(a).
Energy mode: If the five-fingers' accelerations are in the same direction for some states, we regard these states as the same energy mode. For example, the fivefingers' acceleration directions in state 1 are thumb flexion, index finger extension, middle finger extension, ring finger extension, and little finger extension. The five-fingers' acceleration directions in state 2 are thumb flexion, index finger extension, middle finger extension, ring finger extension, and little finger extension. State 1 and state 2 can be different gesture tasks (figures 1 or 3(c)). However, both states are same in energy mode.
The two extreme energy transfer conditions: 1) The wholly transformed potential energy is assured by fixing the fingers (orange square in figure 3(a)). The energy transferred from the muscles to the fingers was wholly converted into potential energy (without kinetic energy). Thus, the potential energy of each finger was equal to the total energy. 2) The wholly transformed kinetic energy is assured by the virtual hand (green square in figure 3(a)). The energy transferred from the muscles to the fingers was wholly converted into kinetic energy (without potential energy from the fingers' coupling).
The detailed description of the calculation in figure 3(a): In the training stage, the collected EMG signals are used as ICA's input to calculate the synergy matrix and converted into independent components. Then, the independent components' features are extracted as the model's input, and the recorded five-finger energy is used as the real label to train the model. In the application stage, the EMG signal is converted into independent components through the synergy matrix. Then, independent components' features are used to estimate the five-finger energy through the trained model. Finally, five-finger energy is used to control a virtual or bionic hand. Additionally, the square of speed (kinetic energy form) is Although the learning method that we selected, such as ANN or SVM, was similar to previous researches that recorded motions or force of the wrist, the difference of our interface was that we had users perform tasks that represented energy modes in the level of the whole hand. We found it very convenient to achieve massive manual tasks by training a few specific energy modes based on virtual hand visualization about the entire hand, rather than a lot of gestures or single-finger force. Notice that the gesture in the yellow box was without mechanical coupling, so the wholly transferred potential energy was recorded. We selected some energy modes with a higher usage frequency (movie S6) [24,29].
proportional to finger energy. The range limit is set in accordance with specific conditions of each finger (potential energy form).

Experimental protocol
We performed off-line analysis and three operational experiments in which the finger energy of subjects was decoded using sEMG signals recorded from their forearms. Ten able-bodied subjects (Subjects 1, 2, 3, …, 10) were recruited.

Subjects and sEMG recording
In this study, ten able-bodied subjects (two females, eight males, aged 26.4 ± 1.43 years) gave informed consent to participate in the experiment protocol. All participants were right-handed. All subjects used their left hands during training stages for convenient operations. For the right hands, the electrodes were placed in the same positions as the left hand in operational experiments. Also, these recruited subjects' hands were similar in size due to the fixed device or forces capture device (figure 4). The study protocol was approved by the ethics review board of Xi'an Jiaotong University. All of the procedures were performed in accordance with the Declaration of Helsinki and relevant policies in China.
According to anatomy and kinesiology of hand [30], the sEMG activity was recorded from eight extrinsic muscles of the forearm for all subjects. As shown in figure S3, four flexor muscles and four extensor muscles related to hand or finger movements were selected.

Training stages 3.2.2.1. System setup
The system was mainly composed of a multichannel surface electromyography device, a fixed device or force capture device, and a personal computer (figure 4). Surface EMG signals, as well as the continuous five-fingers forces with visual feedback from virtual hand in both flexion and extension directions, were recorded simultaneously. Among them, proportional five-fingers forces indicated the potential energy that was converted into the internal energy of the device. This device had two functions. One was to fix the fingers. Thus, the energy transferred from the muscles to the fingers was wholly converted into potential energy (without kinetic energy). At this time, the potential energy of each finger was equal to the total energy. Another function was to record fivefingers forces that indicate potential energy. Visual feedback from the virtual hand represented kinetic energy that was completely converted by potential energy in each finger (kinetic energy was in direct proportion to the square of the speed of virtual fingers). Formula derivation was as follows.
For potential energy of five-fingers: For kinetic energy of five-fingers: Where the subscript expressed the finger. Besides, for a certain hand, m 1 : m 2 : m 3 : m 4 : m 5 was constant. Thus, to solve for allocation of total energy, the formula was as follows.
The formula was applied to figure 4, and the proportional constant was selected by the visualization effect. The contribution lies in providing the method to select representative energy modes (see movie S6). Additional information about the devices can be found in the supplementary materials and movies.

Data collection
Participants were seated on a comfortable chair and were asked to place left hands on the table and watch the LCD monitor. A spongy cushion supported the arm with sEMG electrodes, and the fingers were fixed in the fixed device. All subjects participated in the data collection and were included in off-line analyses. Besides, the sEMG signals from Subject 10 were contaminated with noise due to electrode shifts and one lift-off electrode intentionally [31,32]. The subject was tasked with moving virtual fingers simultaneously to achieve the target gestures (contain ten energy modes in movie S6). The task consisted of the target movements of five-fingers, single-finger, two-fingers, and three-fingers flexion and extension simultaneously.

Online experiment stages
The interface mainly originates from the adaptive mechanism of fingers in life, so the design of feasibility experiments should regard activities of daily living (ADLs) as a reference as far as possible. Three elements are essential to grab an object: gesture (e.g. make a fist), the force between hand and object, and real-time hand control [24,29]. Referring to other studies [33,34], we designed three experiments to verify our interface. The feasibility was assessed in three aspects: (1) the expression of untrained continuous hand motions, (2) the decoding of the amount of finger energy, and (3) real-time control of finger energy.
After training, some artificial neural network (ANN) models were obtained (see Data analysis part). We performed the following experiments using these models. With the evolution of limb in humans, hands have developed into a highly sophisticated system used for manipulative activities, such as tool use and preparing food [24,35]. Today, as a result of cultural pressure, the complexity of the human hand motions has increased tremendously. Some tasks require different hand motions, such as turning a door handle and grabbing a car key. Besides, some tasks require a more differentiated role for each finger, such as sewing, clicking the keyboard, and playing musical instruments.

a. Experiment 1: the expression of untrained continuous hand motions
Some myoelectric interfaces depend on large amounts of training datasets of target motions to implement various hand motions classification, resulting in an increased burden on users [11,18]. Therefore, we sought to achieve the expression of multiple untrained hand motions by only training a few energy modes.
Considering the adaptive adjustment in ADLs, we asked participants to perform these hand tasks in figure 5(e) as fast as possible from figures 5e1 to 5e13 continuously rather than individually. Referring to the study [36], we recorded two outcome measures: completion rate and completion time. If the participant gave up the trial voluntarily, the trial was recorded as a failure. This procedure was analogous to giving up by amputees after trying manual tasks in ADLs.
Six subjects (Subject 5, 6, 7, 8, 9, 10) participated in this experiment. A total of four different experimental conditions were studied (table S1 and figure 5). Tests under each condition were performed five times. Participants were asked to perform experiments first with their trained hand and then with their untrained hand. The experiments with and without ICA were alternated. Additionally, in the trained hand test, we did not re-position the electrodes relative to the training phase, and in the untrained hand test, we re-positioned the electrodes. Also, at the beginning of the experiment, sEMG signals were recorded during the consecutive maximum contractions for both hands to eliminate the effects of individual differences. When manipulating objects, our native hands are good at exerting just enough fingertip force on it [37]. When the object is light and 'fragile,' such as a grape, our hands manipulate a 'gentle' enough pinch force to avoid damage. In contrast, when the object is heavy and slippery, such as a hammer, our hands can clamp the object to avoid slipping free from a stable grasp [38].

b. Experiment 2: the amount of finger energy
Some manual dexterity tests do not benefit from force sensitivity-no penalty is incurred for exerting too much force on an object, such as the Box [Note that one continuous energy mode could adaptively extend to untrained multiple motions or forces according to the task itself. In terms of motions, untrained motions included e3, e4, e5, e6, e7, e12, and e13. Motion e3 and e7 were same in the energy mode. The untrained energy modes were expressed by fundamental energy mode (e.g. e12).] P ki expresses the kinetic energy of the i th finger, while P pi expresses the potential energy of the i th finger. * p < 0.05, * * p < 0.01. Data show means ± SD. and Blocks Test and Action Research Arm Test. Nevertheless, many tasks in ADLs are highly dependent on force sensitivity. In our test, we designed an experiment where participants pricked balloons with a needle tip. The balloon was 'fragile' , it was popped if participants exerted finger energy too much; while the balloon was suspended, it slipped away if participants applied finger energy too little.
To test whether the energy-based interface distinguishes the amount of finger energy, we had participants repeatedly perform these selected hand motions by controlling a bionic hand while breaking/non-breaking the balloon. Note that the fingertips of the bionic hand were fitted with steel needles (figure 6(e) and movie S4).
Five subjects (Subject 5, 6,7,8,9) participated in this experiment. Four different experimental conditions were studied (table S2 and figure 6). Tests under each condition were performed ten times. Participants were asked to perform experiments first with their trained hand and then with their untrained hand. Also, the experiments with and without popping the balloon were alternated. In the 'non-break' task, if participants popped the balloon or could not accomplish the selected gesture, the trial was recorded as a failure. In the 'break' task, if participants could not accomplish the selected gesture and pop the balloon, the trial was recorded as a failure. Additionally, the balloon's circumference was about 66 cm, and the length of the hanging rope was about 7 cm. Each balloon was filled with 36 grams of plasticine (figure S5). Our native hands are exquisitely proficient at flexing the finger to a just right position and performing manual tasks precisely, such as combing hair and applying lipstick, depending on the real-time control of single finger energy [24].

c. Experiment 3: the real-time control of finger energy
To assess the degree of real-time control, we had participants repeatedly punch a hole in the thin plasticine (∼1 mm thickness) attached to the fixed balloon by using the index, middle, and ring fingers while ensuring the balloon intact. If participants could not control the finger in real-time, the steel needles would pop the balloon.
Five subjects (Subject 5, 6, 7, 8, 9) participated in this experiment. Both hands were tested (table S3 and figure 7). Tests under each condition were performed ten times. Success was defined as punching a

Data analysis
The analysis scheme consisted of several steps: data processing, feature extraction, and the relationship learning between sEMG features and five-fingers energy features by machine learning method.

Data processing
Data processing included two preprocessing steps. After sEMG recording, 16-channel sEMG data were band-pass filtered from 10 Hz to 450 Hz to remove movement artifacts, high-frequency noise, and were notch filtered (50 Hz) to remove power line noise and its harmonics [39]. Firstly, according to the clinical relevance of single-differential sEMG, bipolar channels are more tolerant of noise than monopolar ones [40]. Thus, 16-channel sEMG data were further processed to generate eight bipolar channels by subtracting each pair of adjacent channels along the muscle fibers. After the above step, ICA could be selected. During the training stage, the filtered sEMG signals were used as the input of ICA to calculate the synergy matrix and were converted into independent components. During the application stage, the filtered sEMG signals were converted into independent components through the synergy matrix obtained in the training stage. Besides, the finger power was calculated as the difference between the pressure of finger pulp and the finger dorsum. Thus, power data from ten channels were processed to five channels data wherein the signs of the power represented the flexion and extension of fingers, and the absolute values represented the magnitude of power. Secondly, according to a previous study [41], the response time of a control system should not introduce a delay that is perceivable by the user. This threshold is generally regarded as roughly 300 ms. Also, the previous study [42] found that the performance gradually improves as the length of the sliding window increases. Thus, like another study [43], a 200 ms sliding window with a 50 ms overlap was used to down-sampled to 6.67 Hz for the sEMG data and power data. The sEMG data in the sliding window were prepared for feature extraction. The power data in the sliding window were averaged to improve movement smoothness during online control (more information in supplementary materials).

Feature extraction
The fundamental purpose of feature extraction is to emphasize critical information while removing noise and irrelevant data. We chose two groups.
Over the past two decades, some sEMG features have been widely used in research and clinical practice. As in other studies [39,40], six timedomain features and two frequency-domain features were extracted from each sEMG channel with a 200 ms sliding windows to generate 64 sEMG features (8 features × 8 channels). [E-T: mean of absolute values, variance, waveform length, root-mean-square value, Willison amplitude, zero crossing, median frequency, and mean frequency] Additionally, sEMG amplitude is a simple and useful feature, as evidenced by commercial prostheses [10]. To further improve the robustness, we used the frequency-domain power (F-P) as 88 features (11 features × 8 channels). F-P was extracted by a short-time Fourier transform. The information of F-P was similar to the amplitude in the different frequency bands (figure S4).

Learning methods
As two typical examples, we explored two learning methods. Firstly, a multi-layer feedforward ANN was used to learn the mapping between the sEMG signals and the five-fingers energy. The functional relationship predicted by the ANN was expressed as follows.
Where P pre (t) ∈ ℜ 5×1 were the predicted fivefingers power, e (t) ∈ ℜ 64×1 (E-T) or e (t) ∈ ℜ 88×1 (F-P) represented the sEMG features, w were the weight parameters which represented the links between neurons. The network was made up of an input layer, a hidden layer with a tanh activation function (the number of neurons: 10), and a single linear output layer. The training algorithm was Levenberg-Marquardt back-propagation. Secondly, the excellent performance of the support vector machine (SVM) applied to regression problems is known. SVM regression is statistical learning machines [44] that build an approximated map between samples drawn from an input space (under the standard i.i.d. sampling hypothesis) and a set of real value. As a standard, we used the radial basis function for regression.

Operational experiments
In operational experiments, we used a 200 ms sliding window to extract F-P features and the ANN learning method to predict the five-fingers' energy online. The online instruction update rate was kept at 200 Hz (5 ms interval).

Experiment 1: energy-based interface achieved the expression of untrained continuous hand motions
All participants completed tasks successfully under normal circumstances (100 of 100 times from Subject 5-9, binomial test, p < 0.0001, figure 5(a) and movie S1-3).
The interface requires the sensor signals from one hand in training stages and can be used for unilateral amputees [42]. To test the performance of the proposed interface for the untrained hand, we had participants respectively perform the above tasks with the trained hand and the untrained hand. As might be expected, there was no significant difference in completion time between the untrained hand and the trained hand by either ICA or non-ICA algorithms (ICA:81.96 ± 8.95 s versus 79.47 ± 9.15 s, paired t test, p = 0.210, figure 5(a); non-ICA: 80.10 ± 11.48 s versus 78.27 ± 9.80 s, paired t test, p = 0.435, figure 5(a)). This may be due to the similarity among both hands from one man in neuromuscular patterns. However, there were significant differences among participants (one-way ANOVA test, p < 0.01, figure 5(c)), and the operation became significantly faster with the number of times (correlation analysis, r = −0.445, p < 0.01, figure 5(d)).
Additionally, there was no significant difference between algorithms with and without ICA under normal circumstances (untrained hand: 81.96 ± 8.95 s versus 80.10 ± 11.48 s, paired t test, p = 0.429, figure 5(a)). However, it was difficult for Subject 10 to achieve these tasks with the untrained hand by non-ICA algorithms (success times: 1 of 5 times, binomial test, p = 0.375, figure 5(b)). It was easy for Subject 10 to perform these tasks with the untrained hand by ICA algorithms (5 of 5 times, binomial test, p < 0.0001, figure 5(b)), and no significant difference was observed between the untrained hand and the trained hand (75.73 ± 9.69 s versus 77.90 ± 4.69 s, paired t test, p = 0.621, figure 5(b)).
Contrary to expectation, the 'break' task where participants popped suspended balloons with a needle was difficult relative to the 'non-break' task (trained hand: 7.48 ± 1.08 times versus 8.32 ± 1.11 times, paired t test, p < 0.01; untrained hand: 7.16 ± 0.85 times versus 8.20 ± 1.19 times, paired t test, p < 0.01; figure 6(a)). The reason may be the great quality of the balloons. Additionally, there were quality differences among balloons. Fortunately, each balloon was used for both 'break' and 'non-break' trials. We explored the property of finger energy sensitivity by incorporating these two trials to reduce or eliminate potential confound of balloons. There were significant differences among fingers and subjects (two-way ANOVA test; fingers: P < 0.05, figure 6(b); subjects: p < 0.05, figure 6(c)), and different subjects were adept in different fingers (Two-way ANOVA test, p < 0.05, figure 6(d)).

Off-line analysis
In the studies described above, the energy-based interface has been shown to confer functional benefits through three sets of operational experiments. Our purpose of this analysis was to explore the characteristics of the energy-based interface further and explain why the interface showed great functional benefits. Two performance indices were chosen to evaluate the accuracy of the estimation in each finger energy. Pearson's correlation coefficient (R) was calculated to assess the total variation between the estimated and actual energy, and the root-mean-square error (RMSE) described the total residual error.

Energy-based interface achieved a continuous estimation of finger energy in real-time
Even an element of sEMG datasets contains massive properties. Better features should be the properties that are similar in the group but different among groups, and better learning methods should be adopted to distinguish these groups by features as far as possible. To some extent, the fitting model is the classification model whose classes is infinite and continuous.
In order to assess which combination of features and learning methods could apply to the energy-based interface, a ten-fold cross-validation procedure was used to evaluate the overall statistical performance among features (E-T and F-P) and learning methods (ANN and SVM). Figure 8 showed the continuous estimation results from ten subjects in ten test trials. The signs of value represented the flexion or extension of fingers, and the absolute values represented the amount of finger energy. As an example, the equivalent state of kinetic energy and potential energy was shown from the data of Subject 4. Figure S6 showed the confusion matrix for the energy estimation of Subject 4. Although the confusion matrix showed some deviations, the user could correct the deviations in real-time, which was similar to the native hand [45,46]. Furthermore, we found ANN outperformed SVM, whether in the total variation or the total residual error (R: 0.699 ± 0.124 versus 0.653 ± 0.125, Three-way ANOVA test, p < 0.01, figure S7(C); RMSE: 0.209 ± 0.040 versus 0.223 ± 0.041, Three-way ANOVA test, p < 0.01, figure S7(F)). Also, there were significant differences among fingers (R: Three-way ANOVA test, p < 0.01, figure S7(B); RMSE: Three-way ANOVA test, p < 0.01, figure S7(E)). The lowly individuated fingers (ring and middle fingers) likely performed better in the total variation [2].
Relative to the classification-based model, another way to assess the trait of the energy-based interface is to characterize the accuracy in a specific range of energy [13]. To test this extraordinary capability for single finger energy, we divided the amount of energy from zero to maximum voluntary energy (MVE) into five ranges (normalized energy; figures 9(a) and (c)). There was a significant difference among ranges, whether in the total variation or the total residual error [R: One-way ANOVA test, p < 0.01, figure 9(a); RMSE: One-way ANOVA test, p < 0.01(logarithm of RMSE), figure 9(a)]. Also, the total residual error increased with the increase of energy (correlation analysis, r = 0.961, p < 0.01, figure 9(a)), which was similar to the native hand [45,46]. More interestingly, the distribution of relative energy (ratio of voluntary energy to MVE for the finger) was likely consistent with the usage frequency of the finger ( figure 9(c)). For instance, the finger that unconsciously exerted higher relative energy might signify more frequent use in ADLs. Furthermore, the finger flexion showed better performance than extension in total variation (0.622 ± 0.034 versus 0.495 ± 0.041, paired t test, p < 0.01, figure 9(b)).

The generalization among subjects was explored
Previous studies demonstrate that humans have a similar anatomical structure and synergy [47,48]. To assess whether the interface applies to unlearned subjects, we used another ten-fold cross-validation procedure. In this procedure, testing datasets were from one subject totally while training datasets were from other subjects, relative to the previous test. Firstly, we found significant differences among subjects, whether this procedure (one-way ANOVA test; R: p < 0.01, figure S8(A); RMSE: p <0.01, figure S8(F)) or the previous procedure (one-way ANOVA test; R: p < 0.01, figure S8(A); RMSE: p < 0.01, figure S8(F)). Secondly, the training data including data from the testing subject outperformed without (paired t test; R: 0.697 ± 0.008 versus 0.680 ± 0.007, p < 0.01, figure S8(D); RMSE: 0.215 ± 0.041 versus 0.230 ± 0.047, p < 0.01, figure S8(I)), which presumably reflected personalized anatomical structure [47]. Furthermore, as expected, the model for Subject 10 showed a more pronounced performance degradation, when the training datasets did not contain the data from the testing subject (figures S8(C) and S8(H)). We also observed a significant increase in the coefficient of variation (paired t test; R:0.044 ± 0.009 versus 0.092 ± 0.021, p < 0.01, figure S8(E); RMSE: 0.007 ± 0.004 versus 0.024 ± 0.013, p <0.01, figure  S8(J)), which indicated the huge difference in performance among features and learning methods for Subject 10.

ICA mimicking muscle synergy was explored
ICA has been applied to extract synergies from the muscles of frogs [26] and rats [27]. We rebuilt a model using the synergy matrix solved for by data from Subject 1-9. The model evaluation was accomplished through ten-fold cross-validation. We found that the ICA model showed no advantage for normal conditions in the off-line analysis (paired t test; R: 0.680 ± 0.098 versus 0.674 ± 0.099, p = 0.023; RMSE: 0.230 ± 0.047 versus 0.232 ± 0.046, p = 0.141; figures S9(B) and (D). Additionally, for Subject 10, the ICA model did not perform better than the non-ICA model in total variation (paired t test; R: 0.286 ± 0.148 versus 0.257 ± 0.171, p = 0.057; RMSE: 0.274 ± 0.052 versus 0.290 ± 0.060, p < 0.01; figure S9; Noise still existed in independent components of Subject 10 because the synergy matrix was solved for by data from Subject 1-9).

Discussion and conclusion
In the present study, we demonstrated that the dynamic energy model decoded continuous hand actions with speed or force information by training small amounts of sEMG data. This theory imitated the self-adapting mechanism about the coupling of the five-fingers in actual motion. For example, to gesture 'V' in figure 1, humans usually make thumb press ring and little fingers unconsciously and adaptively according to real-time feedback (the energy of these fingers was translated into potential energy). Thus, the feasibility study should regard ADLs as a reference as far as possible.
5.1. Experiment 1: first of all, performing with a certain task such as turning a door handle or grabbing a car key, the user must complete the correct grasping motions Considering the continuously adaptive adjustment of the human hand in ADLs, we had participants perform a series of uninterrupted actions. The results showed that a few continuous energy modes could adaptively combine into multiple motions of the virtual hand according to the task itself according to realtime feedback, which was similar to the human hand [45,46]. Also, some untrained energy modes could be expressed by trained energy modes. Besides, the correlation coefficients of off-line analysis were relatively low compared to previous researches about the wrist due to less training data and more DOFs [14][15][16]. Some studies conclude that it is unnecessary to achieve high accuracy in the mapping for reliable simultaneous and proportional myoelectric control. Instead, good online myoelectric control is achieved by the continuous interaction and adaptation of the user with the myoelectric controller through feedback [49,50]. In this study, the method was designed by imitating the self-adapting mechanism about the fingers' coupling, and the whole manual task was regarded as an energy transfer. Thus, subjects could adaptively adjust the energy mode in real-time, and results were excellent.

Experiment 2:
second, the amount of energy involved in the grasp must be controlled so that it is possible to grab, e.g. both a hammer without letting it slip and an egg without breaking it In experiment 2, participants were asked to perform two tasks. In the first task, participants used small energy to complete finger flexion and ensured that the needle of a fingertip did not pop the balloon. In the second task, participants used enough energy to complete finger flexion and ensured that the balloon was popped. Considering the influence of the quality of the balloon, we tested both conditions for each balloon. After the overall consideration of both tasks, we proved that the presented model could exert just enough finger energy on 'fragile' and 'heavy' objects.

Experiment 3: third, real-time control of finger energy is essential so that it is possible to perform manual tasks precisely, e.g. applying lipstick
We had participants control finger energy in real-time to punch a hole in the thin plasticine attached to the fixed balloon. Participants needed to stop in time after punching the hole and ensured that the balloon was not popped. In actual operation, it was easy for participants to complete this task, and the success rate was over 95%.
Overall, the study involved the main lines of research about the myoelectric interface. One line of research is to recognize hand gestures based on pattern recognition. These related works focus on improving classification accuracy and increasing the number of hand motions [11][12][13]. With adequately designed feature extraction and classifiers, it is possible to achieve extremely high classification accuracy (90%-95%) in a vast repertoire of hand motions (21 classes) [12]. In 2019, a combined hand-motions viewpoint emerged [13]. The literature [13] shows that trained single motions could express some untrained combined motions. However, this is a combination of discrete motions. Here, inheriting previous work, we proposed a different way to extend hand tasks by using continuous energy modes. Some different hand movements were divided into the same energy modes at the five-fingers level. In actual operation, the user could accomplish a mass of hand movements by adjusting a few continuous energy modes adaptively according to feedback. This means that one may consider tagging the training data by energy mode instead of gesture for later research about sEMG gesture recognition.
Another line of research is to estimate a proportional activation of each DOF. These related works mainly focus on the wrist or hand close and open [15,48,49,51,52]. However, the estimation of forces and kinematics in a single finger has rarely been investigated. Unlike the wrist, finger independence was limited by mechanical coupling and neuromuscular control [1,2]. For instance, active movement at one finger may lead to some movements at another finger. Previous approaches that estimate a proportional activation for each DOF might not be applicable to five-fingers unless training by vast amounts of data. In our work, we regarded the five-fingers as a whole and divided massive manual tasks into several specific energy modes in terms of energy; thus, this problem was overcome. Compared with the results of other studies using sEMG for continuous motion recognition [15,16,48,49,51,52], this method was extended to the single fingers of the whole hand from the wrist or hand close and open. Although the correlation coefficients of offline analysis were relatively low realtered to other studies with above 0.8 R square due to less training data and more DOFs [15,48,49,51,52], the online performances were similar. This may be due to the good stability of the energy-based interface's signal and good interaction based on energy mode [7,49,50].
In terms of signal selection and processing, as with previous studies [20][21][22][23], the present model used ICA to extract muscle synergy patterns. In contrast to the model whose features were directly extracted from the sEMG signals, the model with ICA showed no significant advantage for normal conditions. The machine learning methods, such as ANNs, might learn the mapping consistent with ICA. Furthermore, for Subject 10, we gave one possible explanation from the previous study and algorithm itself [53]. The sEMG training signals from Subject 10 contain a lot of noise from electrode shifts and one lift-off electrode. When the model was without ICA, the noise component also participated in the model's learning, such as the weight optimization of the neural network. Therefore, after changing the environment or the noise component, the trained model cannot describe the new system. However, when the model was with ICA, the noise component was removed before training, and the model only learned the remaining features. After changing the environment, although the noise component had changed, it was directly filtered by the ICA matrix. The remaining features were similar to the previous features, so the model performed better (schematic was shown in figure 2(b)). Additionally, a myoelectric signal comprises two states: a transient state emanating from a burst of fibers, as a muscle goes from rest to a voluntary contraction level, and a steady-state emanating during a consistently maintained contraction in a muscle [7]. The latter component dwarfs the former in robustness [54], and the energy-based interface used the latter.

Limitations and future work
The main limitation of the current study was that no amputee was recruited. However, previous studies demonstrate that amputees are similar to the non-disabled subjects in muscle activation [15,48]. It is not likely that the amputees' motor learning ability would be significantly affected by limb deficiency. Besides, our operational experiments showed that the performance degradation for the untrained hand was not significant when the model was trained by data of contralateral limb from the same person. Of course, we excluded the amputee in muscular atrophy or without measurable muscles. The present study emphasized the concept of hand action division by energy mode, and the user was not limited to amputees, such as neurorehabilitation after stroke [55][56][57].
Biological hand grasping and manipulating capabilities are much complex than the existing myoelectric prosthesis. Like a recent study [13], palmar arching or fingers' abduction, especially thumb abduction, was not considered. It is a recognized difficulty in sEMG recognition and is worthy of further study. Additionally, the current design of experiments only approached ADLs from the perspective of sEMG decoding as far as possible, which was a small step towards the ADLs of prosthetics. The prosthetic hand matching our interface will be developed in the future.
Amputees express a desire for intuitive myoelectric control [58]. The adaptive property of the energy-based interface underlines the importance of the user's capacity to interact with the machine and learn a new task in which the user is within the loop and can adapt to the control system. Before concluding, it is worth mentioning that this study showed that one might consider tagging the training data by energy mode instead of gesture to reduce learning task complexity for later research about sEMG gesture recognition.