CN108491858A - Method for detecting fatigue driving based on convolutional neural networks and system - Google Patents

Abstract

The invention discloses a kind of method for detecting fatigue driving and system based on convolutional neural networks, belongs to image processing and pattern recognition field.First, two-dimentional face-image of the acquisition driver under driving condition, and degree of fatigue hierarchical classification is pressed, establish fatigue driving image library；Secondly, convolutional neural networks of the structure one containing data Layer, convolutional layer, pond layer, articulamentum and layer of classifying；Then, using image data in fatigue driving image library and label as the input of convolutional neural networks, using back-propagation algorithm to the network repetitive exercise of structure, network output loss function value is made gradually to decline and restrain；Finally, driver's driving condition lower face image measurement sample is inputted, it is identified using the convolutional neural networks model after training, realizes the detection classification of driver's face-image degree of fatigue.The present invention is compared to conventional machines learning method, hence it is evident that improves identification classifying quality, monitoring provides a kind of feasible thinking in real time for fatigue driving.

Description

Fatigue driving detection method and system based on convolutional neural network

technical field

The invention relates to a fatigue driving detection method based on a convolutional neural network, which belongs to the technical field of image processing and pattern recognition.

Background technique

With the rapid development of the economy, the automobile industry has achieved rapid growth, and people's travel methods have become richer, and automobiles are one of the most important travel methods. At the same time, the traffic order and safety conditions on the road have become very complicated, and the occurrence of traffic accidents has become more and more frequent. Fatigue driving is one of the most likely causes of traffic accidents. If the driver's fatigue driving can be detected in time and reminded in time, such traffic accidents can be prevented at the source and the incidence of traffic accidents can be reduced. Fatigue driving detection technology not only guarantees the safety of the driver's life and property, but also improves road traffic conditions. Therefore, it is of great significance to evaluate fatigue driving.

The reasons for driver fatigue driving mainly include driving for a long time, lack of sleep or poor sleep quality, poor physical condition, etc., which make the driver fatigued, and the fatigued state is most easily reflected on the driver's facial expression. Under normal conditions, the driver's facial expression is serious and relaxed, while under fatigued conditions, it appears dull and fatigued. Therefore, it is feasible to reflect the driver's fatigue state by studying the driver's facial expression state.

At present, facial expression recognition technology based on traditional machine learning methods has a relatively good recognition effect. However, facial expression recognition technology is generally aimed at anger, disgust, fear, happiness, sadness, surprise, and neutrality. In facial expression recognition, the data volume requirements are not very high, and part of it is based on the texture features of facial expressions. recognized classification. In the face of abstract facial expressions such as fatigue, a database with a small amount of data cannot fully reflect the characteristics of facial fatigue, and it is very difficult to achieve a good recognition effect. Therefore, it is necessary to establish a large fatigue driving database and a recognition algorithm that can handle a large amount of data in order to achieve a good recognition effect.

Contents of the invention

The technical problem to be solved by this invention is:

A convolutional neural network-based fatigue driving detection and recognition method and system are provided to solve the problem that traditional methods cannot accurately realize fatigue driving recognition, thereby providing a feasible and efficient method for real-time monitoring of fatigue driving.

The present invention adopts the following technical solutions for solving the problems of the technologies described above:

The present invention proposes a method for detecting and identifying fatigue driving based on a convolutional neural network, comprising the following steps:

Step 1. Establish a fatigue driving image library: collect the two-dimensional facial images of the driver in the driving state as a sample, divide the sample images into different fatigue driving states according to the actual fatigue level corresponding to the sample image, and establish a fatigue driving image library ;

Step 2. Divide samples: According to the training method based on convolutional neural network, divide the samples of each type of fatigue driving state into a training set and a verification set in a ratio of 8:2, and divide each sample image according to the fatigue driving state category mark label;

Step 3. Construct a convolutional neural network structure: the constructed convolutional neural network contains 1 data layer, 2 convolutional layers, 2 pooling layers, 2 connection layers and 1 classification layer;

Step 4, training convolutional neural network: use the training set and verification set described in step 2 together with their labels as the input of the convolutional neural network data layer, and then use the back propagation algorithm to iteratively train the convolutional neural network , optimize the training global parameters, and finally reduce and converge the loss function value output by the convolutional neural network, and generate a classification model for identifying fatigue driving;

Step 5, recognition and classification of convolutional neural network: use the new driving state facial image as a test sample to verify the accuracy of the convolutional neural network; if the accuracy is verified, apply the convolutional neural network to fatigue Recognition and classification work for driving detection.

Furthermore, in the method for detecting and identifying fatigued driving based on convolutional neural network proposed by the present invention, the different states of fatigued driving mentioned in Step 1 include: normal state, mild fatigue, and severe fatigue.

Further, in the fatigue driving detection and recognition method based on convolutional neural network proposed by the present invention, the network structure of the convolutional neural network in step 2 includes the following:

Data layer: This layer is the first layer of the network, which stores the sample images in the fatigue driving image database and the label data of the corresponding sample categories;

The first convolutional layer: this layer is the second layer of the network, using k ₁ m ₁ *m ₁ convolution kernels with a step size of l ₁ pixels, and convolving the image as the input of the next layer;

The first pooling layer: This layer is the third layer of the network, which is the downsampling of the first convolutional layer;

The second convolutional layer: this layer is the fourth layer of the network, using k ₂ m ₂ *m ₂ convolution kernels, the interval step is l ₂ pixels, and the image is convolved as the input of the next layer;

The second pooling layer: This layer is the fifth layer of the network, which is the downsampling of the second convolutional layer;

The first connection layer: This layer is the sixth layer of the network, which is the process of changing the two-dimensional features output by the upper layer into one-dimensional features;

The second connection layer: This layer is the seventh layer of the network and is the fully connected output of the first connection layer;

Classification layer: This layer is the last layer of the network, which outputs the probability distribution vector P, which is the probability value P _i belonging to the jth class, and finds the maximum value in P _i , and takes the category corresponding to i with the highest probability as the detection result , i=1,2,...,n, n is the number of categories.

Further, in the fatigue driving detection and recognition method based on convolutional neural network proposed by the present invention, the loss function described in step 4 is defined as follows:

Among them, F(θ) represents the loss function, m is the total number of training samples, n is the number of classification categories; 1{y ⁽ⁱ⁾ = j} is the indicator function, when the value of the bracket is true, the function value is 1, otherwise the function value is 0; x ⁽ⁱ⁾ represents the vector formed by the output nodes of the fully connected layer, θ ₁ , θ ₂ ,…, θ _n represent the model parameters, and T represents the transpose of the matrix.

Further, in the method for detecting and identifying fatigued driving based on the convolutional neural network proposed by the present invention, the definition of the probability value in step 4 is as follows:

identity(y)=argmax(P _j )

Among them, P _j represents the probability value belonging to the jth category, x ⁽ⁱ⁾ represents the vector formed by the output nodes of the connection layer 2, and identity(y) represents the classification result of the test sample y.

The present invention also proposes a fatigue driving detection and identification system based on a convolutional neural network, including:

The fatigue driving image library module is used to collect the two-dimensional facial images of the driver in the driving state as samples, divide the sample images into different fatigue driving states according to the actual fatigue degree corresponding to the sample images, and establish the fatigue driving image library;

The sample division module is used to divide the samples of each type of fatigue driving state into a training set and a verification set in a ratio of 8:2 according to the training method based on the convolutional neural network, and divide each sample image according to the fatigue driving state category mark label;

Convolutional neural network construction and training module: used to construct a convolutional neural network containing 1 layer of data layer, 2 layers of convolutional layer, 2 layers of pooling layer, 2 layers of connection layer and 1 layer of classification layer; and, divide the samples The training set and verification set of the module, together with their labels, are used as the input of the convolutional neural network data layer, and then the convolutional neural network is iteratively trained using the back propagation algorithm, the global parameters of the training are optimized, and finally the convolutional neural network outputs The value of the loss function decreases and converges to generate a classification model for identifying fatigue driving;

Recognize and classify module, be used for using new driving status facial image as test sample, verify the accuracy of described convolutional neural network; If verify by accuracy, then apply described convolutional neural network to the recognition classification of fatigue driving detection Work.

Compared with the prior art, the present invention adopts the above technical scheme and has the following technical effects:

1. The convolutional neural network method used in the present invention is a recognition and classification algorithm that can automatically learn a large amount of data features, and does not need manual intervention for feature extraction and classification. The quality and quantity of data samples theoretically determine the final recognition and classification effect. Therefore, compared with traditional machine learning methods, the recognition and classification effect can be significantly improved;

2. The network model trained by the present invention using the convolutional neural network algorithm has good robustness in terms of image differences such as noise, occlusion, and shooting angle of fatigue driving images.

Description of drawings

FIG. 1 is a flow chart of the method for detecting and identifying fatigued driving based on a convolutional neural network in the present invention.

Fig. 2 is a picture example of various states in the fatigue driving image library of the present invention.

Fig. 3 is a schematic diagram of the convolutional neural network structure of the present invention.

Detailed ways

Below in conjunction with accompanying drawing, technical scheme of the present invention is described in further detail:

Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It should also be understood that terms such as those defined in commonly used dictionaries should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted in an idealized or overly formal sense unless defined as herein explain.

Deep learning is a new hotspot in computer vision research and a new direction in machine learning methods. Its core is to establish a neural network model that simulates the learning and analysis of the human brain, so as to imitate the brain mechanism to interpret and analyze data, such as common text, sound, image and video data. First of all, the fatigue driving detection using deep learning method can get a higher recognition rate, which is also one of the most important requirements for fatigue driving detection and recognition; second, the advantages of deep learning can automatically learn expression features from data samples, through iteration Learning to automatically obtain similarities and differences of the same kind is not like traditional machine learning methods to manually design feature extraction algorithms; finally, deep learning can be perfectly combined with big data. The most lacking of deep learning methods is data, focusing on data quality and In the context of big data, the collection of data becomes simple, and the processing of these data by deep learning becomes more efficient. To sum up, for the detection and recognition of fatigue driving, the method of using deep learning is the most suitable.

Convolutional neural network is a deep neural network model and also a multi-layer supervised learning neural network. Compared with the ordinary neural network model, on the one hand, the connection between the neurons of the convolutional neural network is not fully connected; on the other hand, the weight of the connection between the neurons in the same layer of the convolutional neural network is shared. A non-fully connected and weight-sharing network structure reduces the complexity of the network model and reduces the number of weights. In this way, the hierarchical structure and local perception of the neural network are realized, the network structure becomes insensitive to operations such as translation, rotation, tilt, and scaling of two-dimensional images, and the image features are highly invariant. The most important network structure of the convolutional neural network for feature extraction is the hidden layer in the network, that is, the convolutional layer and the pooling layer. Through these two layers, the dimensionality reduction and feature extraction of data image features are realized. At the same time, the gradient descent method is used to minimize the loss function to reversely adjust the weight parameters in the network layer by layer, and improve the accuracy of the network through certain iterative training to achieve accurate extraction of image features.

As shown in Figure 1, the realization of the fatigue driving detection method based on the convolutional neural network of the present invention mainly comprises the following steps:

Step 1, build a fatigue driving image library:

Continuously observe the driver's facial expression and mark it according to the driver's subjective feelings during the long-term driving process, from normal state, mild fatigue to severe fatigue. And unify the attributes of the collected driver's facial images, including resolution, size, format, etc. There are problems such as shooting angle, exposure strength, and color saturation in practical applications. Note that the shooting angle, exposure strength, and color saturation of the collected images are not changed here, so that the sample data has a certain diversity. Finally, the collected image is calibrated to a color image with a pixel size of 128*128 and a format of jpg as shown in Figure 2, and a fatigue driving image library with more than 6,000 samples is established, including 2,000 in normal state and 2,000 in mild fatigue. Severe fatigue 2000 cards. Perform data enhancement on the database established above, including image rotation, reflection, flip, translation, scale, contrast, noise, etc., so that the number of enhanced data samples reaches 120,000, of which 40,000 are in normal state, 40,000 for mild fatigue and 40,000 for severe fatigue.

Step 2, divide the fatigue driving image library:

After completing the establishment of the driver's fatigue driving image database, according to the training method based on convolutional neural network, each type of expression in the fatigue driving image database is divided into training samples and verification samples according to the ratio of 8:2, that is, each type of training samples There are 32,000 images, 8,000 images of each type of verification samples, and the label numbers corresponding to each type of expression are marked.

Step 3, construct the convolutional neural network structure:

Construct the convolutional neural network structure containing 1-layer data layer, 2-layer convolutional layer, 2-layer pooling layer, 2-layer connection layer and 1-layer classification layer, the convolutional neural network structure schematic diagram of the present invention, as shown in Figure 3 .

Data layer: This layer takes the training samples, verification samples and sample labels after dividing the fatigue driving image library as the convolutional neural network data input, which needs to be converted into the data type specified by the deep learning platform;

Convolutional layer 1: This layer is the convolutional layer 1 in the convolutional neural network. It uses 50 7*7 convolution kernels with a step interval of 4 pixels to realize the multi-layer convolution function of the input sample image, and serves as input to the next layer;

Pooling layer 1: This layer is the pooling layer 1 in the convolutional neural network, and the average pooling with a pooling window of 2*2 is used to reduce the data dimension and serve as the input of the next layer;

Convolutional layer 2: This layer is the convolutional layer 2 in the convolutional neural network. It uses 128 3*3 convolution kernels with a step interval of 1 pixel to realize the multi-layer convolution function of the output data of the pooling layer 1. And as the input of the next layer;

Pooling layer 2: This layer is the pooling layer 2 in the convolutional neural network, and the average pooling with a pooling window of 2*2 is used to reduce the data dimension and serve as the input of the next layer;

Connection layer 1: This layer is the connection layer 1 in the convolutional neural network and is the fully connected input of the pooling layer 2. This layer contains 1024 nodes. And as the input of the next layer;

Connection layer 2: This layer is the connection layer 2 in the convolutional neural network, which is the fully connected input of the connection layer 1. This layer contains 1024 nodes. And as the input of the next layer;

Classification layer: This layer is the last classification layer of the network, which is the fully connected input of the connection layer 2, and the output is the probability value P _j of the jth class, and the maximum value is found in P _j , and the j corresponding to the highest probability category as the identification classification result;

Step 4, train the convolutional neural network:

Take the training samples, verification samples and their corresponding data labels divided in step 2 as the input of the convolutional neural network, use the backpropagation algorithm to iteratively train the designed neural network, and optimize the training global parameters so that the network output loss function value decreases and converges .

Step 5, recognition and classification of convolutional neural network:

The network model is used to recognize and classify the facial images of the driver in the driving state. Input the face image test sample of the driver in the driving state to verify the accuracy of the network, and apply it to the identification and classification of fatigue driving detection, and develop an automatic assessment system for fatigue driving.

The above descriptions are only part of the embodiments of the present invention. It should be pointed out that those skilled in the art can make some improvements and modifications without departing from the principles of the present invention. It should be regarded as the protection scope of the present invention.

Claims (6)

1. a method for detecting and identifying fatigued driving based on convolutional neural network, is characterized in that, comprises the steps: Step 1. Establish a fatigue driving image library: collect the two-dimensional facial images of the driver in the driving state as a sample, divide the sample images into different fatigue driving states according to the actual fatigue level corresponding to the sample image, and establish a fatigue driving image library ; Step 2. Divide samples: According to the training method based on convolutional neural network, divide the samples of each type of fatigue driving state into a training set and a verification set in a ratio of 8:2, and divide each sample image according to the fatigue driving state category mark label; Step 3. Construct a convolutional neural network structure: the constructed convolutional neural network contains 1 data layer, 2 convolutional layers, 2 pooling layers, 2 connection layers and 1 classification layer; Step 4, training convolutional neural network: use the training set and verification set described in step 2 together with their labels as the input of the convolutional neural network data layer, and then use the back propagation algorithm to iteratively train the convolutional neural network , optimize the training global parameters, and finally reduce and converge the loss function value output by the convolutional neural network, and generate a classification model for identifying fatigue driving; Step 5, recognition and classification of convolutional neural network: use the new driving state facial image as a test sample to verify the accuracy of the convolutional neural network; if the accuracy is verified, apply the convolutional neural network to fatigue Recognition and classification work for driving detection. 2. A method for detecting and identifying fatigued driving based on a convolutional neural network as claimed in claim 1, wherein the different states of fatigued driving in step 1 include: normal state, mild fatigue, and severe fatigue. 3. according to the described fatigue driving detection method based on convolutional neural network of claim 1, it is characterized in that, the network structure of convolutional neural network described in step 2 comprises as follows: Data layer: This layer is the first layer of the network, which stores the sample images in the fatigue driving image database and the label data of the corresponding sample categories; The first convolutional layer: this layer is the second layer of the network, using k ₁ m ₁ *m ₁ convolution kernels with a step size of l ₁ pixels, and convolving the image as the input of the next layer; The first pooling layer: This layer is the third layer of the network, which is the downsampling of the first convolutional layer; The second convolutional layer: this layer is the fourth layer of the network, using k ₂ m ₂ *m ₂ convolution kernels, the interval step is l ₂ pixels, and the image is convolved as the input of the next layer; The second pooling layer: This layer is the fifth layer of the network, which is the downsampling of the second convolutional layer; The first connection layer: This layer is the sixth layer of the network, which is the process of changing the two-dimensional features output by the upper layer into one-dimensional features; The second connection layer: This layer is the seventh layer of the network and is the fully connected output of the first connection layer; Classification layer: This layer is the last layer of the network, which outputs the probability distribution vector P, which is the probability value P _i belonging to the jth class, and finds the maximum value in P _i , and takes the category corresponding to i with the highest probability as the detection result , i=1,2,...,n, n is the number of categories. 4. according to the described fatigue driving detection method based on convolutional neural network of claim 1, it is characterized in that, the loss function described in step 4 is defined as follows: Among them, F(θ) represents the loss function, m is the total number of training samples, n is the number of classification categories; 1{y ⁽ⁱ⁾ = j} is the indicator function, when the value of the bracket is true, the function value is 1, otherwise the function value is 0; x ⁽ⁱ⁾ represents the vector formed by the output nodes of the fully connected layer, θ ₁ , θ ₂ ,…, θ _n represent the model parameters, and T represents the transpose of the matrix. 5. according to the described fatigue driving detection method based on convolutional neural network of claim 4, it is characterized in that, the definition of probability value described in step 4 is as follows: identity(y)=argmax(P _j ) Among them, P _j represents the probability value belonging to the jth category, x ⁽ⁱ⁾ represents the vector formed by the output nodes of the connection layer 2, and identity(y) represents the classification result of the test sample y. 6. A fatigue driving detection and recognition system based on convolutional neural network, characterized in that, comprising: The fatigue driving image library module is used to collect the two-dimensional facial images of the driver in the driving state as samples, divide the sample images into different fatigue driving states according to the actual fatigue degree corresponding to the sample images, and establish the fatigue driving image library; The sample division module is used to divide the samples of each type of fatigue driving state into a training set and a verification set in a ratio of 8:2 according to the training method based on the convolutional neural network, and divide each sample image according to the fatigue driving state category mark label; Convolutional neural network construction and training module: used to construct a convolutional neural network containing 1 layer of data layer, 2 layers of convolutional layer, 2 layers of pooling layer, 2 layers of connection layer and 1 layer of classification layer; and, divide the samples The training set and verification set of the module, together with their labels, are used as the input of the convolutional neural network data layer, and then the convolutional neural network is iteratively trained using the back propagation algorithm, the global parameters of the training are optimized, and finally the convolutional neural network outputs The value of the loss function decreases and converges to generate a classification model for identifying fatigue driving; Recognize and classify module, be used for using new driving status facial image as test sample, verify the accuracy of described convolutional neural network; If verify by accuracy, then apply described convolutional neural network to the recognition classification of fatigue driving detection Work.