Distracted Driving Detection using Python.

Used an XBOX controller with the game "Track Mania" into a Extremely Randomized Trees (ERT) model to detect distracted driving.

To use this prototype check out the prototype folder.

Demo Video found here

The final prototype consists of two models: a Convolutional Neural Network (CNN) as well as the Extreme Random Forest model similarly to the initial prototype.

This project uses the Logitech G920 Racing Wheel and acompanying pedals to capture inputs. However, any racing wheel or other form of input can be used as long as the event codes are properly mapped using device_config.py.

To start collecting data, use Inputs_G920.py in the Tier 1 folder. This file will output a folder of images as well as a csv file containg image paths and corrosponding angles.

Sample data can be found here: images and csv

These inputs are then loaded into Tier_1_Training.ipynb in the Tier 1 folder. Once the notebook is completed, the last output is a model file containing the weights and structure of the CNN. An example is Tier_1_Model.h5.

The .h5 file is then used in Inputs_G920_Distraction.py in the Tier 2 folder. This will generate a csv file containing actual and predicted angles as well as gas and brake values. An example csv is here

This csv file is usedin Tier_2_Training.ipynb to create another model file in a .pkl format. An example is Tier_2_Model.pkl

Both model files are used in Real_Time_Prediction.py to predict distraction levels real time.

Demo video of final prototype found here

The next level of the models development transitions from using Euro Truck Simulator to real world test data. The data is 300 hours of commuting along California highways and can be found here

The data was originally in video formatt with 15 frames per second. The data was manipulated from video to jpeg fromatt and adjusted to insure the driving data (steering, brakes, acceleration, etc) was present accross all frames. Outliers in the steering angle data were removed as the focus of the model is highway driving distraction where large changes in steering angle are not relevant. Sample data can be found here A Demo video can be viewed here

In order to combat Catastrophic interference the totality of the dataset was randomly broken into 10 segements to improve ease of use. Then the datapoints were scrambled to improve overall model performance. A total of 3.2 million frames and data points were used to train the model.

After adjusting the model weights a final correlation between raw and predicted angles reached 95% providing a very accurate prediction which can then be fed into the distraction model.

The first prototype of the model simply used the python library sk-learn, the following models were tested:

Due to the simple nature and limited amount of data the was trained on a MSI GE72 2QD Apache Pro using the i7-4700HQ CPU for full specs click here

With the addition of image data and the implimentation of CNN training the model on a laptop would not be feasible. Due to limited resources the free tool Google Colab, which initalizes a jupyter notebook running on a NVIDIA TESLA K80 GPU was used. This tool was very helpful in the testing of the model as it also allows colaboration between team members however it also had frequent crashes and other issues.

Moving from a smaller dataset using cloud tools to a 3.2 million image dataset needed a drastic change to our training workflow. A new system was purchased to locally train the model, initally the model was trained using basic tensorflow using the keras API on an Intel Core i7-9700K CPU @ 3.60GHz. However training the model using a CPU was very time extensive, after further research into tensorflow-gpu and Nvidia CUDA drivers this method was implimented.

Please note setting up a non-linux based tensorflow GPU enviroment in this case Windows 10 is a very convoluted process proceed with caution.

Once the tensorflow GPU process was configured properly we were able to run batches of 75-100k images at a time dratically improving our training workflow. The GPU used was the Nvidia Geforce RTX 2070.

Without any further data manipulation the correlation for ~100,000 samples is 89.7% when comparing prediction and actual steering angles. When looking at a 15 frame rolling average the correlation goes to to 95%.

A Demo video can be viewed here

The results can be seen below:

Updating the version of the tier 2 model poses a difficult challenge because acquiring real world distracted driving in an enthical manner is difficult. The best course of action seems to be simulating the distraction data using our past knowledge from the previous models and then iterating on the model until real world outputs are accurate. Through the use of driving simulators and controlled distraction experiments a dataset has been combined with the upgraded dataset in order to make a model capable of detecting real world distraction events while driving.

Below is a graph showing the distribution between positive distraction events and non-distracted driving.

Balancing the dataset is very important to aquiring an accurate prediction model especially when the output of distracted driving is a fuzzy concept which is hard to accurately define. This prevents the model from simply guessing and reduces the likelyhood of overfitting. The data has been scaled and manipulated to be within the range of zero and one to improve performance and the outliers of the model have been scrapped to remove imporerly recorded data. This model was trained on an approximate even split of 90,000 samples of distracted driving as well as focused driving.

After applying the changes the results of the updated model in accordance to the comma.ai dataset improved from ~0.72 to 0.805556 with a standard deviation of 0.313532.

There are three critical steps in the development of this project, firstly updating the distraction model, secondly implimenting physical data extraction from a car, and finally creating a dashboard for visualization of the data.

At this point the plan of action is to utilze the panda OBD2 interface which is a state of the art OBD2 extraction tool that can live stream data via USB or wifi at rates much higher then industry standards. From the OBD2 port we will use a Raspberry Pi 4 which is an inexpensive micro computer that can run both the tier 1 and 2 models. Finally a forward facing Pi Camera will live stream the image data to the Pi 4.

In order to analize the data a dashboard will be created. At this point an minimal viable product using Tableau will be made, in future a webapp using the python API dash will be implimented.