An approach towards data change rule based classification of driving maneuver with LSTM network

The proposed work develops a Long Short Term Memory (LSTM) model for multi class classification of driving maneuver from sensor fusion time series dataset. The work also analyzes the significance of sensor fusion data change rule and utilized the idea with deep learning time series multi class classification of driving maneuver. We also proposed some hypotheses which are proven by the experimental results. The proposed model provides Train Accuracy: 99.98, Test Accuracy: 97.2021, Precision: 0.974848, Recall: 0.960154 and F1 score: 0.967028. The Mean Per Class Error (MPCE) is 0.01386. The significant rules can accelerate the feature extraction process of driving data. Moreover, it helps in automatic labeling of unlabeled dataset. Our future approach is to develop a tool for generating categorical label for unlabeled dataset. Besides, we have plan to optimize the proposed classifier using grid search.


Introduction
With the accelerated industrial revolution, the increasing amount of vehicles forming threats to road safety [1]. Since drivers are responsible for more than 80% of the traffic accident [2,3] there has a great influence of driver behavior on road traffic security [4]. Advanced driver assistance systems (ADAS) are installed in exclusive vehicles by automotive manufacturers to stabilize safety and competency [5]. Due to the high cost, the deployment of ADAS has been restricted. However, the focus of ADAS is assisting drivers rather than monitoring the driving style. These systems have been developed to rescue and protect people after the accident and successfully lots of lives have been saved, nonetheless cannot able to avoid the accident to take place [6]. Therefore, driver behavior should be considered as one of the most significant aspects for enhance road traffic safety [7]. It is important for monitoring new drivers and performance evaluation of drivers throughout the training sessions [8]. Besides, driver's behavior can influence the reduction of fuel i.e., energy consumption and gas emissions.
Alike ADAS, various approaches have proposed that used inertial signals, GPS, video to monitor drivers, and driving behaviors. However, being processed these input through the system exacerbate the system performance and increase computational complexity though it is verified in previous related works that inertial sensors absolutely have the ability to recognize different driving style [9]. Few researchers utilized vehicle information collected through a Controlled Area Network (CAN) using On-Board Diagnostic (OBD) port or Micro Electro Mechanical System (MEMS). Nowadays, MEMS is getting more popular over OBD port because of its compact size, lightweight and lower in energy consumption nature. So, the integration of different sensor units such as accelerometers, gyroscopes, magnetometers becomes easy [10]. Since collecting vehicle information through CAN using OBD port depends on the ability of private protocol of particular vehicle and also poses threat to the vehicle and human security [10], hence, MEMS is more appropriate for driving maneuver recognition.
Earlier many researchers follow different fuzzy inference and machine learning techniques to classify driving maneuvers. However, they ignore the analysis of the theoretical base of collected sensors data, hence, prevent the advanced optimization of driving maneuver classification research. While moving, the change of the motion sensor data can be explained with the help of rigid body kinematics [1]. We believe that these data change rules can reinforce to differentiate different maneuver i.e., events and therefore, amend the classification algorithm.
The paper introduces the significance of driving maneuver classification of sensor data in Section 1. A compressed discussion on the existing approaches is presented in Section 2. The theoretical analysis of the physical model with multi-class hypotheses, including deep learning architecture is described in Section 3. The experimental environment is being described and results are analyzed in Section 4. Finally, Section 5 concludes the paper with future directions.

Existing Approach
From the literature review, it is obvious that several solutions for driving maneuver recognition or classification have been proposed and developed. We have partitioned the literature review section into two part. The first part contains the approaches that are not related to deep learning time series classification and the second part includes deep learning time series classification approaches.
In [8] the authors proposed a system to classify driving style as aggressive and nonaggressive and type of driving maneuver based on Dynamic Time Warping (DTW) algorithm and smartphone based sensor-fusion. In [11] a Fuzzy Inference System (FIS) based mobile tool has been developed that utilized accelerometer, magnetometer, gravity sensor, GPS embedded in drivers' smartphones, and evaluated the overall driving behavior by combining different fuzzy sensing data. However, it was a web based tool that processes all sensor data in a remote database to score all the driving events based on a few predefined threshold values of each possible event. Ly et al. in [9] proposed a system that used vehicle's inertial sensors from the CAN bus using SVM and k mean clustering. They imposed few thresholds for lateral, longitudinal acceleration, yaw angular velocity sensors to create an individual driver profile of the driver to ultimately provide proper feedback to reduce the number of dangerous car maneuvers. In [7] drivers were categorized according to four different driving events namely acceleration, braking, turning, and lane changing using smartphone sensory data in a bidirectional vehicle to infrastructure communication system. They established a driver safety index to categorize each driver during the journey. Symbolic Aggregate Approximation (SAX) and Matrix profile method have been used to categorized patterns that appear in time series naturalistic driving data included maneuvers like turning, stopping at intersections, parking, and leaving parking spaces [12]. In [13] the authors concentrated four driving states namely stopped, driving, parking, and parked. They applied Random Forests (RF), Support Vector Machines (SVM), and fuzzy rule-based classifiers on data collected by accelerometer sensor installed in a smartphone inside the vehicle. In [14] they evaluated the performance and comparative analysis of multiple combinations of machine learning algorithms such as SVM. They used sensors e.g., accelerometer, gyroscope, and magnetometer installed in android smartphones to collect driving data for detecting aggressive driving events. However, only a labeled event dataset has been utilized for classification though there is a lot of unlabeled data that has a potential driving event pattern. In our experiment, we have used the dataset.
A work using the same driving dataset like [14] has been introduced in [15]. In this work time series of accelerometer data is employed as inputs of Recurrent Neural Network (RNN) to accomplish driver behavior profiling and investigated the performance of three different types of Recurrent Neural Networks which are Recurrent Neural Network (RNN), Long Short Term Memory (LSTM) and Gated Recurrent Network (GRU). However, they classified drivers' behavior only from accelerometer data, not driving maneuver. By plotting the collected sample dataset, it is clearly shown that only accelerometer sensor data change rule is not sufficient to classify all the events. Moreover, drivers' behavior is closely related to driving maneuvers. Another researcher in [16] focused on an approach to implement supervised time series classification with Random Forest (RF) and Recurrent Neural Network separately. They introduced RF to classify aggressive events and the result of RF transferred to recognize the type of maneuver. However, a limitation of the implementation is that the labels were selected manually by one person and depend on the labeler's danger perception. We agree with the authors that the assignation of labels should be extended to more criteria to reduce potential bias. We believe that our work will extend a way to find the correct label without manual labeling. In [17] the authors developed a model that consist of three major computational components, distance based representation of driving context including features of vehicle trajectory and visual features. The model took as input of videos of the road scene ahead and vehicle sensor data from OBD sensors and the video images of the front view and classified temporal dependencies of five different classes of driving maneuvers namely left turn, right turn, left lane change, right lane change, driving straight using LSTM.
The main contribution made in this research is that (i) We develop a Long Short Term Memory (LSTM) network for classification of driving maneuver from sensor fusion time series data, (ii) Establish hypotheses based on data change rules of time series data during the performance of various driving maneuver discussed in [1], (iii) Investigate a novel way to reducing manual labeling of unlabeled time series driving dataset.

Proposed Methodology
In this work, we model time series sequence data with Long Short Term Memory (LSTM) considering the classification as a multi-class supervised deep learning classification problem. For the multi-class classification purpose, we analyze the physical model of moving vehicles during the performance of different maneuvers and establish hypotheses for particular class which can be explained by the data change rule of sensor fusion dataset and fit well with the output classes of maneuver.

Analysis of Theoretical Physical Model
Many researchers have studied the dynamic model of vehicles related to the measurement of slip angle, velocity, tire model, etc. In the present research, we only focus on moving subject vehicles disregarding its inner model. During the movement of a vehicle in real traffic conditions, the most common driving maneuvers are moving forward, left and right turning, left, and right lane change. The physical model has been established from the theory of rigid body kinematics [1]. Movement of a vehicle causes changes in its kinematic states such as acceleration, deceleration, angular velocity, etc. which follow specific rules. The acceleration in time, t and angular velocity, ⃗⃗ of a vehicle with radius, can be defined by equations (1) and (2) and decomposed in x, y and z dimension. So, we get three acceleration data , , and three angular velocity data , , in three dimensions.
To establish the driving data change rule, Wu at el. [1] proposed a physical model shown in Fig 1. As magnetometer sensor data is related to the position of the geographic location of the vehicle and no direct relation with common driving maneuvers has been proved, it is out of our scope of interest. On the basis of the physical model of vehicle motion, the change of sensors' time series data which represents different driving maneuvers is inferred and illustrated in  as well. Hence, we focus on , and data only. In Eq. (1), if > , therefore > 0; hence, during the event of acceleration the time series data initially increases from zero to greater value and finally decreases to a lower value. When < , then < 0. As a result, while deceleration, commonly known as braking in the transportation system the time series data diminishes from zero to negative values and then increases to zero again.
From Eq. (2) it is clear that ⃗⃗ α , therefore, change of angular velocity is persistent to acceleration. As at the time of lane change both lateral and angular displacement happen, for left lane change both time series data of and increases from zero and gradually decreases until a negative value, then increases again. The opposite incident occurs for the right lane change.
We plot some sample time series data of labeled maneuvers to validate the data change rules of moving vehicles. The description of the used dataset is provided in Section 3.3. When a vehicle moves in longitudinal direction non-aggressively, the and data range from -2 to 2 and as angular rotation does not involve here, the angular velocity remains very negligible. In Fig 3 'non-aggressive' labeled data is shown where the highest and lowest peak of are 0.24 and -1.25, respectively. In Fig 4 Non-aggressive time series data peak ranges from -2 to +2 shown by red dotted line. By plotting more samples of non-aggressive labeled data, the same range is found. An 'aggressive acceleration' labeled time series data is plotted in Fig 5 shows that few peaks exceeded +2 and reaches +6 shown by the red dotted line. The statistical features e.g. mean, variance, and standard deviation reflect substantial change during the maneuver. At the same time, the slopes of the data points are very sharp. This pattern indicates a high gradient and energy at the time of aggressive acceleration. The amount of slope and energy is defined by Eq. (4) where, k = size of sliding window From Eq. (3) it is clear that slope increases and decreases according to two adjacent data points where Eq. (4) implies that energy increases only in the positive axis. During aggressive braking (in Fig 6), the value of drops significantly to a negative value less than -2 and almost -6. Hence, because of change in the opposite direction the slope decreases and energy, variance, and standard deviation shown a significant increase. The variance remains 0 to at most 1.5 and standard deviation follows variance at less value. As the amount of energy and variance increases both during aggressive acceleration and brake, it cannot make any difference between aggressive acceleration and braking rather indicate an aggressive event. So, we consider , and slope of to differentiate aggressive acceleration and braking to avoid redundancy.
For aggressive Left Turn (LT) and Right Turn (RT), lateral displacement and angular rotation occur at the same time, so data change in and represents these two events. During the left turn, changes from 0 to 1 and for the right turn it ranges from -1 to 0 where during a non-aggressive maneuver, change of ranges from -0.2 to 0.2. In both events the slope of increase from 0 to 0.6 and 0.3 during a non-aggressive event (see Fig 9 and Fig 10). Another significant difference between left and right turn is the slope. At the moment of left turn, the slope remains positive while for right turn it becomes negative. There is no significant effect of variance. Hence, we consider , slope and energy of ; , slope and energy of to differentiate aggressive left turn and right turn.
At the moment of Left Lane Change (LLC) and Right Lane Change (RLC), the slope of increases and decreases in greater value than left and right turn, and for both event data point move to one axis to another axis (see Fig 7 and Fig 8). There is no significant pattern of energy and variance as time series data moves to positive to a negative value and vise-versa for left and right lane change, respectively; but energy and variance increase for both maneuvers. Therefore, we consider data and slope of and .
While performing one particular maneuver, sometimes the data change pattern can follow patterns of other maneuvers but manual labeling does not show the difference because manual labeling relies on human perceptions. For instance, before or after aggressive braking, the data change follows aggressive acceleration but through the human perception the maneuver is being labeled as aggressive braking only. Many researchers proposed various classification methods where the time series data change rules have been ignored. We investigate the significance and figure out that the data change rule of a particular maneuver is valuable during feature extraction of sensor fusion time series data. So, we develop some hypotheses from the observations of labeled time series data.

Development of Hypotheses for Driving Maneuver Classes
We develop some hypotheses from the observations of time series data changing patterns for driving maneuver classes on the basis of the physical model. Time series data that do not fall into any of the classes will be non-aggressive and belong to Class 7.

Development of Time Series Classification with Deep Neural Network
Deep learning model such as recurrent neural network, long short term memory network is being designed for classification of sequential data. Since time series classification requires restoring functional dependencies between the sequences of time series and the finite set of classes after training with a set of known classes [18] we preferred to use LSTM networks for classification of driving maneuvers.

Time Series Data Collection and Preparation.
We perform our experiment using a set of real world sensor fusion time series data [19]. When two drivers with 15 years of experience performed the different driving maneuvers, a smartphone application has recorded the sensor fusion time series data. During the collection of sensor fusion time series dataset, the smartphone was sable on the windshield of the subject vehicle. While collecting sensor data, video of the front view of subject vehicle has been recorded from which later the label of different driving maneuvers had been labeled manually.
The dataset was distributed in multiple files and the labels with start and end time of events were listed in different files for four of each journey with nanosecond timestamp. We prepare the , and data with second. Besides, we add the label for each time step as ground truth label. Among 156512 time series observations only 11077 observations are manually labeled and 145435 unlabeled observations were unlabeled. We set a rolling window with 20 sample size and extract significant features from a valid window using the hypotheses of section 3.2.
Developing LSTM Model Architecture. The proposed LSTM network architecture consists of one input layer with two LSTM hidden layers and a dense output layer. Th e architecture of our LSTM is presented in Fig 11.  Fig. 11. Architecture of LSTM_128_128_Dense Model for multi class classification Wang at el. in [20] evaluated different deep learning models for time series classif ication and among those this LSTM architecture was one of the outperformed models. A dropout layer with dropout rate 0.2 is used after every layer except for the input laye r. The batch size is (1/10) of the training dataset. The activation function for every lay er is Rectified Linear Unit (ReLU) except for dense layer. The Softmax activation fun ction is used for dense layer. As this is a multiclass classification problem, the model u ses categorical cross entropy for loss optimization. The LSTM model is trained during 500 epochs with Adam optimizer.

4
Experimental Results and Analysis

Experimental Environment Setup
The architecture is implemented using Keras 2.3.0 framework with Tensorflow 2.0 backend in Python 3.8.3. The model is trained on CPU instances with RAM 8 GB and core i-5 processor.

Results and Discussion
In [20] Wang at el. used Mean-Per-Class Error (MPCE) for multi class time serie s classification. We also use MPCE to evaluate our model performance. MPCE is the average of Per Class Error (PCE). PCE of each class is calculated by Eq. 5.
The calculated MPCE of our model is 0.01386 applying hypotheses and 0.0494 witho ut applying hypotheses. Wang et al. in [20] found MPCE = 0.0407 using the 85 UCR t ime series datasets. The evaluation scores and Per Class Error is shown in Table 1 and comparison betwee n other related work the proposed work is shown in the Table 2 that indicates LSTM o utperforms over other classifiers and deep learning models. The Accuracy and Loss is plotted in Fig 12 and 13

Conclusion
The proposed work discusses the significance of sensor fusion data change rules and utilized the idea with deep learning time series multi-class classification of driving maneuvers. We used an LSTM model because it is one of the most appropriate neural network models for the classification of time series data. We propose a few hypotheses which are proven by the experimental results. This significant rules accelerate the feature extraction process from driving data. Moreover, it helps in the automatic labeling of the unlabeled dataset. Our future approach is to develop a tool for generating categorical labels for unlabeled datasets. Besides, we have a plan to optimize the proposed classifier using a grid search.