Data Augmentation-based Novel Deep Learning Method for Deepfaked Images Detection

Digital image, video, audio recognition, and tampering tools are quickly evolving [1, 2]. Digital content creation, manipulation methods, and technological expertise are also readily available [3]. Hyper-realistic digital graphics may now be created with few resources and simple how-to-do instructions accessible online [4, 5] hence leading to fake audios, videos, and images [6, 7]. With the help of the deepfake method, anyone may replace the face of someone you want with the face of someone else [8]. Splicing a synthetic face into the original picture creates this effect. Using Deep Learning, deepfake produces faked photos in which one person’s face is substituted for another’s. Celebrities’ reputations might be jeopardized if their likenesses are manipulated to disseminate misinformation. On a photograph of Abraham Lincoln, the oldest known attempt at a face-to-face trade was reported [9]. John Calhoun’s body was lithographed with his face on top. Due to advances in artificial neural networks, ML classifiers, and cyber thieves can now manipulate digital media to produce very realistically difficult-to-detect changed picture data, spreading false information. deepfake generation has become much simpler due to recent advances in disciplines like Generative Adversarial Networks (GANs) [10, 11], which need a reference image and a set of intended defects to generate realistically changed photographs.

Furthermore, FaceApp [12], MTCNN, and other tools have made creating deepfake photos quite simple. Consequently, automated systems for identifying deepfake photos have become critical, considering the worldwide impact and the scale to which these images may harm the security and stability of any society. However, since the forged images include prominent visual properties, these auto-generated fake images might be quickly assessed using Convolutional Neural Networks (CNNs). One of CNN’s many applications is picture classification. Objects that occupy the bulk of a photograph are given a class number, which is used to classify the photograph [13]. CNN is a component of the deep learning approach used for image identification and analysis and is built to handle pixelated input. The picture is used as an input and assigns priority to unique image features, allowing it to identify one from another [14]. CNN needs substantially less pre-processing than other classification techniques. While filters in other DL systems are created by hand, CNN can learn these features or filters with appropriate training [13].

Compared with other standard Neural Networks, in which layers make up the structure, the neurons in CNN’s layers are constructed in three dimensions: height, width, and depth. The most often utilized layers are convolutional/Pooling/Relu and fully connected (FC). Each layer has a basic API that employs a discrete function that may or may not have arguments to turn a 3D input into a 3D output. As previously stated, a simple CNN is a series of layers in which each layer translates a single set of operations to the other using a differentiable function. The types of layers used to build a CNN structure are the Conv/Pooling Layer and the FC Layer.

A complete CNN structure is built by stacking these layers. Input will contain the raw pixel values, and the picture’s raw pixel values represent an image with its height, width, and color. For example, if we decide to use 12 filters, we may end up with an input size of 224 \(\times\) 224 \(\times\) 12 by calculating the dot product between the input size and the weights of the neurons linked to it. In addition, the RELU layer assigns an element-based activation function. The pooling layer performs a spatial dimension downsampling process. The FC layer, calculates the final class grade. A CNN model transforms an original picture by layering values from the initial pixel values to the final class scores. Various approaches have been used to address problems such as a low detection rate for deep false images, a high error rate, a lengthy processing time, and improper data access. This study employs the concept of transfer learning to improve the detection performance of both deepfake and real images, emphasizing data augmentation approaches to improve the performance of the models during training [15, 16, 17].

While deepfake technology is still in its infancy, there has been much study into it. According to statistics from (https://app.dimensions.ai) at the end of 2020, the number of deepfake articles has increased dramatically in recent years, which is why Nguyen et al. and his collaborators conducted research [26]. Research on deep learning’s ability to represent complex and multidimensional data is rising. Despite this, the number of deepfake articles gathered is probably lower than the precise number. It has been frequently employed for dimensionality reduction and image compression by deep autoencoders, a form of deep network with such an ability [27, 28, 29].

Authors in [30] employed a neural network that uses binary classification to differentiate between real and fake photos and then offers an accurate result. One of the most often used deep learning models to identify between false and genuine images is the CNN. To compare multiple Deep Learning-based state-of-the-art models, the author suggested the deepfake-Stack, a deep ensemble learning technique. Incorporating the predictions from various level-0 models, we have constructed this model. In order to train the level-1 model, we must use out-of-sample data predictions supplied by the base models. A CNN model called deepfake Classifier was created using the InceptionResNetV2 and InceptionV3 base learners and ResNet101, DenseNet121, and Mobile-Net. Accuracy was 99.65%, and AUC-ROC was 1.0 in the experiment, which was achieved using these models. To identify deepfakes, Xu Chang et al. developed an algorithm that combines picture noise and image augmentation. The SRM filter method is used to get the image noise characteristics from the RGB pictures that must be recognized to emphasize the image noise. A big dataset and low picture quality necessitated Celeb-DF as the dataset for their experiment, which yielded significant results in identifying deepfake face photographs [31]. AUC was only 73.2% in this experiment using SRM filter and VGG-16, whereas their NA-VGG technique on the same dataset reached an AUC-ROC of 85.7%.

Badale et al. proposed a Neural Network technique to identify the fake video content reported in this article. For both original and deepfake frames, the video was used to extract them. Every image/frame has been classified as either “actual” or “fake” based on the source footage. A neural network’s first layer receives input from a flattened set of 256 \(\times\) 256 \(\times\) 3 dimensions [14]. Thus, the data is ready for input.SGD obtained 88% accuracy in categorical cross entropy, whereas Adam achieved 91%. Cross-entropy binary analysis showed that Adam was 90% accurate, whereas SGD was 86% accurate, but the mean square was 80% accurate in both cases. Another technique to identify fake videos is with Shraddha Suratkar et al. CNN architecture, and Transfer Learning methodology [8]. The use of CNN is offered as a way to extract characteristics from a video’s frames using CNN. It was determined that InceptionV3 had a 91.56% test accuracy rate, ResNet50 had an 84.04% score, ResNet18 had an 84.68% test accuracy score, NasNetMobile had a 91.97% score, and Xception had a 90.08% score [32]. The model based on Inceptionv3 had the greatest testing accuracy of all the other models in the research cited above. Rezende et al. [33] suggest using a pre-trained ResNet-50 to recognize computer-generated pictures is possible. SVM classifiers were used to replace the original top layer, which was replaced with a fully linked layer. The SVM classifier attained an accuracy rate of 94.05%, compared to a softmax layer’s 92.28% identification rate. AlexNet and VGG16 were also employed by Sengur et al. [34] to detect fraudulent content evidence by extracting facial characteristics. SVMs are used instead of thick layers to classify fake and legitimate faces in this suggested method uses transfer learning to import learned weights. The authors offered more information and better prediction accuracy and suggested merging the traits gleaned from both networks. On the CASIA dataset, the model’s accuracy was 94.01% when using the integrated features. Three sets of conv/max-pooling layers were suggested by Mo et al. to identify fraudulent faces. They employed a collection of spatially high-band-pass filters, which execute spatial operations for emphasizing minute features on pictures, increasing the image’s noise. The suggested CNN architecture uses residual noise as input characteristics. A GAN-based technique was used to enhance legitimate photos of the CELEBA-HQ dataset, resulting in a 99.4% accuracy rate [35, 36]. Regarding face picture forgery detection, authors in [37, 38] proposed comparing multiple CNN architectures over the Real and Fake Face datasets. The study includes picture normalization and preprocessing utilizing Error Level Analysis for further training and fine-tuning several deep learning models.

Existing methods have many drawbacks, including a low detection rate for deep fake images, a high mistake rate, a long processing time, and inaccurate data access. This work focuses on data augmentation approaches to improve the model performance by generating new image samples while training the models and also uses the concept of transfer learning that increases the detection performance of deepfake and real images. The survey results for identifying fake images are presented in Table 1.

The use of deep learning for detecting deepfake images is crucial as the prevalence of these hoaxes becomes a greater social problem. deepfakes are faked visual or audio content made with the help of deep learning algorithms to give the impression that a certain person or group said or did something they did not [50]. The harmful use of this technology includes disseminating misleading information and propaganda and producing fake evidence for use in the courts. As a result, it is crucial to detect deepfakes. To protect the public from misinformation and help maintain our information ecosystem’s integrity, deep learning algorithms must be developed to identify deepfakes properly.

This section explains the suggested method for real and fake image detection. The proposed methodology used to detect real and fake images is shown in Figure 1. The proposed method used a deep CNN and its various variants (VGG16, inceptionV3, and VGG19) by applying transfer learning for better classification of images. Data augmentation provides new and varied instances for training datasets to enhance model performance and results. We separated the data into three sections: training validation, testing, and evaluation. The training and validation datasets are fed into a CNN model for feature extraction. We trained CNN, VGG16, InceptionV3, and VGG19 models to predict if an image is real or fake. This research is based on two classes, Real or Fake. We evaluated our suggested models based on training accuracy, testing precision, recall, F1-score, confusion matrix, and ROC_AUC score.

We used four different CNN models for deepfake and real image classification (custom CNN model, VGG16, VGG19, and InceptionV3). To evaluate the DL model’s capabilities, six assessment measures were used (accuracy, precision, recall, F1-score, confusion matrix, and Auc_Roc score). The dataset was separated into three sections: 140,002 images were used to train the models, 39,428 images were used for validation, and 10,905 were used to test the CNN models. Python is used to conduct the experiments for this research. Python code was written and executed in this analysis using the Kaggle framework, a web-based Python editor. Table 2 depicts the experimental setup used. The experiments also used several useful programming tools (including pandas, numpy, Keras, OpenCV, TensorFlow, matplotlib, and sklearn) to aid analysis and visualization.

Artificial intelligence has progressed to the point that we can now create “deepfake pictures,” which can be used to swap out a user’s face for a fake one. In recent years, people have used deepfake photos to smear celebrities, politicians, and ordinary people. Convincingly deep-faked images have been used to incite political unrest, exact blackmail, spread fake news, and even stage hoaxed terrorist acts. Therefore, it is more difficult to distinguish genuine from fake photographs. Our research uses a transfer learning and data augmentation technique to circumvent these problems to determine which photos are real and which deepfake. This research uses 190,335 deepfake and real photos of RGB resolution for testing purposes. In addition, the dataset is pre-processed using image augmentation techniques. Models are trained using features based on CNNs. In order to determine if the proposed method was effective, this study employed standard assessment metrics. The studies use a transfer learning strategy with deep learning models (CNN, Inception V3, VGG19, and VGG16). 10,905 fresh photos were used to evaluate the deep learning models. Our improved VGG16 model outperformed other DL models in spotting genuine and fake photographs, and the results showed that the suggested method achieved 90% accuracy, recall, F1-score, and AUC-ROC, with 91% precision.

The lack of comprehensive data is the primary shortcoming of this study, which will be addressed in further research. In the future, we intend to work on video forensics. A video may also be used to identify deepfake images. The AI and deep learning model might be used to build a mobile app that makes it simpler to identify deepfake and expands the technology’s reach. Other than that, it may be possible to use it on Android.