CN108377387A - Virtual reality method for evaluating video quality based on 3D convolutional neural networks - Google Patents

Abstract

The invention relates to a 3D CNN-based stereoscopic image quality evaluation method, comprising the following steps: video preprocessing: using the left view video and the right view video of the VR video to obtain a VR differential video, uniformly extracting frames from the differential video, and giving each Frame non-overlapping blocks, video blocks at the same position in each frame constitute a VR video patch to generate enough data for 3D CNN training; build 3D CNN model; train 3D CNN model: use stochastic gradient descent method, Taking VR video patches as input, each patch is labeled with the original video quality score, and input into the network in batches. After multiple iterations, the weights of each layer of the network are fully optimized, and finally a volume that can be used to evaluate the quality of virtual reality video is obtained. product neural network model; get the final result. The invention improves the accuracy rate of the objective evaluation method.

Description

Virtual Reality Video Quality Evaluation Method Based on 3D Convolutional Neural Network

technical field

The invention belongs to the field of video processing and relates to a virtual reality video quality evaluation method.

Background technique

As a new simulation and interaction technology-virtual reality (VR) technology is used in many fields such as architecture, games and military, it can create a virtual environment consistent with the rules of the real world, or create a simulation completely out of reality environment, which will bring people a more realistic audio-visual experience and on-the-spot experience [1]. As an important carrier of virtual reality, panoramic stereoscopic video is currently closest to the definition of VR video, which plays a huge role. However, in the process of capturing, storing, and transmitting VR videos, due to equipment and processing methods, some distortion will inevitably be introduced, which will affect the quality of VR videos. Therefore, it is very important to study an evaluation method that can effectively evaluate the quality of virtual reality video. However, the subjective evaluation method is easily disturbed by many factors, and it is time-consuming and laborious, and the evaluation results are not stable enough. Compared with subjective evaluation, objective evaluation uses software to evaluate image quality, does not require participants and a large number of subjective experiments, is easy to operate, and is highly correlated with subjective evaluation, and has attracted more and more attention from relevant researchers.

Since virtual reality technology has just emerged in recent years, there is no standard and objective evaluation system for VR video [2]. VR video has the characteristics of realism, immersion, and stereoscopic effect [3]. In traditional multimedia types, stereoscopic video has the closest characteristics to VR video. Therefore, the evaluation of VR video needs to refer to the current idea of stereoscopic video quality evaluation. There are mainly three types of objective evaluation methods for stereoscopic video at present. The first type is an evaluation method based on the Human Visual System (HVS). The second category is an evaluation method based on image features combined with machine learning. The third category is the evaluation method using deep learning. All of the above methods have good reference significance for the objective evaluation of VR videos.

[1]Minderer M, Harvey C D, Donato F, et al. Neuroscience: Virtual reality explored. [J]. Nature, 2016, 533(7603): 324.

[2] X.Ge, L.Pan, Q.Li. Multi-Path Cooperative Communications Networks for Augmented and Virtual Reality Transmission. IEEE Transactions on Multimedia, vol.19, no.10, pp.2345-2358, 2017.

[3]Hosseini M, Swaminathan V.Adaptive 360VR Video Streaming: Divide and Conquer[C]//IEEE International Symposium on Multimedia.IEEE,2017:107-110.

Contents of the invention

The purpose of the present invention is to establish a VR video quality evaluation method that fully considers the characteristics of virtual reality. The VR video objective quality evaluation method proposed by the present invention uses the deep learning model 3D convolutional neural network (3D CNN) to allow the machine to extract VR video features instead of traditional manual feature extraction, and the latest deep learning model 3D CNN can fully consider Temporal motion information of the video. At the same time, the present invention designs a score fusion strategy that fits the characteristics of VR video production and playback, so as to make accurate and objective evaluations. The technical solution is as follows:

A method for evaluating the quality of stereoscopic images based on 3D CNN, the evaluation method comprising the following steps:

1) Video preprocessing: Use the left-view video and right-view video of the VR video to obtain the VR differential video, uniformly extract frames from the differential video, and cut each frame into non-overlapping blocks, and the video blocks at the same position in each frame form a VR video patches to generate enough data for 3D CNN training;

2) Establish a 3D CNN model: it includes two convolutional layers, two pooling layers and two fully connected layers, the activation function uses rectified linear unit (ReLU), and the Dropout strategy is used to prevent overfitting; then adjust the inner layers of the network structure and training parameters to achieve better classification results;

3) Training 3D CNN model: using the random gradient descent method, taking VR video patches as input, each patch is labeled with the original video quality score, and input it into the network in batches. After multiple iterations, the weights of each layer of the network are fully obtained. Optimization, and finally a convolutional neural network model that can be used to evaluate the quality of virtual reality videos;

4) Get the final result: use 3D CNN to get the score of the VR video patch, and then use the score fusion strategy to assign different weights to the VR video patches at different positions, and obtain the final objective evaluation score of the virtual reality video quality after weighting.

The VR video objective quality evaluation method proposed by the present invention uses the latest deep learning model to extract higher-dimensional features of VR videos, not only does not need to manually extract video features, but also uses machine learning to extract required features, and fully considers the video Motion information in the time domain. In addition, the present invention combines the production and playback characteristics of VR videos, assigns different weights to different video patch scores, and then uses the score fusion strategy to comprehensively express the objective quality of VR videos. The video preprocessing method adopted by the invention is simple and has strong practicability, and the proposed test model consumes less time and is easy to operate. The objective evaluation results of VR video quality obtained by this method are highly consistent with the subjective evaluation results, and can reflect the quality of VR video more accurately.

Description of drawings

Figure 1 VR video preprocessing flow chart.

Figure 2 3D CNN network framework diagram.

Figure 3 3D convolution map.

Figure 4 Scatter diagram of the relationship between subjective and objective scores: (a) symmetric distortion, (b) asymmetric distortion, (c) H.264 distortion, (d) JPEG2000 distortion.

Detailed ways

The stereoscopic image quality evaluation method based on 3D CNN provided by the present invention, each distorted VR video pair is made up of left video V _l and right video V _r , and evaluation method comprises the following steps:

Step 1: Construct difference video V _d according to the principle of stereo perception. First, each frame of the original VR video and the distorted VR video is grayscaled, and then the left video V _l and the right video V _r are used to obtain the required difference video. The value of the sum value video V _d calculated at the video position (x, y, z) is shown in formula (1):

V _d (x,y,z)=|V _l (x,y,z)-V _r (x,y,z)| (1)

Step 2: Divide the VR differential video into blocks to form video patches, thereby expanding the capacity of the dataset. Specifically, one frame is extracted every eight frames from all VR difference videos, and a total of N frames are extracted. A square image block with a size of 32×32 pixels is cut out at the same position of the extracted frame, and then the image blocks of the same video and the same position form a VR video patch. In order to fully extract the spatial information of the video, each frame should be cut evenly without overlapping image blocks, and M image blocks should be cut out of each frame of video. A total of M video patches with a size of 32×32×N can be extracted from each VR video according to different resolutions.

Step 3: Construct and train a 3D CNN deep learning model. The model of the present invention consists of two 3D convolutional layers, two 3D pooling layers and two fully connected layers. On the basis of 2D CNN, 3D CNN can effectively extract the motion information in the time domain of video by considering the information between multiple inputs. Therefore, it is necessary to explain the convolution and pooling process of 3D CNN. The formula for 3D convolution is:

where k represents the index of the feature map in the (l-1) layer connected to the current convolution kernel, Represents the kth 3D feature map in the (l-1)th layer, is the i-th 3D convolution kernel in the l-th layer, and its convolution is in superior. An additive bias term and a nonlinear activation function are performed to obtain the final feature map. is an additive bias term, and f( ) is a nonlinear activation function, such as a sigmoid function, a hyperbolic tangent function, and an integer linear function.

The formula for 3D pooling is:

Among them, m, n, j represent the size of the selected point area of the feature map.

In the present invention, the 3D CNN structure uses stochastic gradient descent and ReLU as the activation function. In order to prevent overfitting, the present invention adopts a dropout strategy, that is, a dropout strategy with a parameter of 0.5 is used after each pooling layer. After concatenating the layers, we use a dropout strategy with a parameter of 0.25. The minibatch size in the network is 128, and the learning rate of model training is set to 0.001. Additionally, batch normalization is used between each convolution and subsequent activation to speed up network training. The objective function adopted by this model is as follows:

where λ represents the regularization parameter, y _i represents the ground-truth region quality score, and f(xi ₎ represents the predicted score. After the model is built, 80% of the data is used for training, and 20% of the data is used for testing.

Step 4: After obtaining the video patch score through the depth model, the final score of the VR video is obtained through the score fusion strategy. The score fusion strategy used in the present invention assigns different weights to VR video patches at different positions according to the Equirectangular projection method of VR video, so as to obtain the final objective evaluation quality score. The equirectangular projection method will cause the polar parts of the video to be greatly stretched during the projection process, which will affect the spatial distribution of the VR video under the planar model. Since the objective quality evaluation method uses planar video as input, and the subjective evaluation score is based on the viewing experience of spherical video, the present invention designs a score fusion strategy as shown in formula (4):

where _Sf denotes the final score, _Sxy denotes the predicted score of the video patch at the video frame position (x,y), x denotes the width location, y denotes the height location, _Wxy denotes the weight of the corresponding location, h denotes the vertical direction of the VR video Height, h' indicates the vertical distance from the center of the video patch to the center of the VR video.

Step 5: Select the database. In order to prove that the predicted objective quality score of the image obtained by the method of the present invention has a high consistency with the subjective quality score, and the predicted objective quality score can accurately reflect the quality of the image, the method of the present invention is tested on the VRQ-TJU database. The database contains 13 original VR videos and 364 distorted VR videos. The distortion types include H.264 and JPEG2000, and includes both symmetrical and asymmetrical distortions. There are 104 symmetrically distorted videos and 260 asymmetrically distorted videos.

Get 4 indicators commonly used in the world to measure the performance of the objective image quality evaluation algorithm to evaluate the performance of the method of the present invention, 4 indicators are respectively Pearson linear correlation coefficient (Pearson linear correlation coefficient, PLCC), Spearman rank correlation coefficient (Spearman rank -order correlation coefficient, SRCC), Kendall rank-order correlation coefficient (Kendallrank-order correlation coefficient, KROCC) and root mean square error (Root Mean SquaredError, RMSE). The closer the value of the above three correlation coefficients is to 1, the smaller the RMSE value is, indicating that the algorithm is more accurate.

Step 6: Analyze and compare algorithm performance. In order to verify the pertinence and effectiveness of the present invention for VR video quality evaluation, the present invention refers to a method for image quality evaluation IQA, stereoscopic image quality evaluation SIQA, video quality evaluation VQA, and stereoscopic video quality evaluation SVQA for comparison in the database Verification, respectively corresponding to [1], [2], [3] and [4].

Table 1 Comprehensive data indicators

Table 2 Different distortion type indexes of the present invention

[1] A. Liu, W. Lin, and M Narwaria. Image quality assessment based on gradient similarity. IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society, 21(4):1500,2012.

[2] Alexandre Benoit, Patrick Le Callet, Patrizio Campisi, and Romain Cousseau. Using disparity for quality assessment of stereoscopic images. In IEEE International Conference on Image Processing, pages 389–392, 2008.

[3] Kalpana Seshadrinathan, Rajiv Soundararajan, Alan Conrad Bovik, and Lawrence K Cormack. Study of subjective and objective quality assessment of video. IEEE Transactions on Image Processing, 19(6):1427–1441, 2010.

[4] Nukhet Ozbek and A. Murat Tekalp. Unequal inter-view rate allocation using scalable stereo video coding and an objective stereo video quality measure. In IEEE Intern.

Claims (1)

1. A method for evaluating the quality of stereo images based on 3D CNN, the evaluation method may further comprise the steps: 1) Video preprocessing: Use the left-view video and right-view video of the VR video to obtain the VR differential video, uniformly extract frames from the differential video, and cut each frame into non-overlapping blocks, and the video blocks at the same position in each frame form a VR video patches to generate enough data for 3D CNN training; 2) Establish a 3D CNN model: it includes two convolutional layers, two pooling layers and two fully connected layers, the activation function uses rectified linear unit (ReLU), and the Dropout strategy is used to prevent overfitting; then adjust the inner layers of the network structure and training parameters to achieve better classification results; 3) Training 3D CNN model: using the random gradient descent method, taking VR video patches as input, each patch is labeled with the original video quality score, and input it into the network in batches. After multiple iterations, the weights of each layer of the network are fully obtained. Optimization, and finally a convolutional neural network model that can be used to evaluate the quality of virtual reality videos; 4) Get the final result: use 3D CNN to get the score of the VR video patch, and then use the score fusion strategy to assign different weights to the VR video patches at different positions, and obtain the final objective evaluation score of the virtual reality video quality after weighting.