WO2024139166A1 - Video coding method and apparatus, and electronic device and storage medium - Google Patents

Abstract

Provided in the present disclosure are a video coding method and apparatus, and an electronic device and a storage medium. The method comprises: acquiring a video clip to be coded, wherein said video clip comprises one or more video frames; calculating spatio-temporal domain feature information of said video clip on the basis of the one or more video frames; determining a plurality of predicted bitrate factors for said video clip under a plurality of preset video coding standards on the basis of the spatio-temporal domain feature information; determining a plurality of target bitrate factors for said video clip under the plurality of preset video coding standards on the basis of a first predicted bitrate factor among the plurality of predicted bitrate factors that corresponds to a first preset video coding standard among the plurality of preset video coding standards, and a preset video coding score interval, wherein the time for performing video coding by using the first preset video coding standard among the plurality of preset video coding standards is the shortest; and coding said video clip according to the plurality of target bitrate factors, so as to obtain a plurality of target video clips.

Description

Video encoding method and device, electronic device and storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This disclosure claims priority to Chinese patent application 202211690147.9 filed on December 27, 2022, the entire contents of which are incorporated by reference in their entirety into this disclosure.

Technical Field

The present disclosure relates to the field of computer technology, and in particular to a video encoding method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

Background technique

With the development of Internet technology, video platforms have also developed rapidly, with more and more users sharing and watching videos. For the platform, as the number of users continues to grow, the bandwidth cost for video transmission is also increasing. In order to reduce this cost, a transcoding system with better compression performance is needed. While ensuring the quality of the video, it can significantly reduce the size of the compressed video, thereby reducing the network traffic required for transmission.

The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art.

Summary of the invention

The present disclosure provides a video encoding method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

According to one aspect of the present disclosure, a video encoding method is provided, comprising: obtaining a video segment to be encoded, the video segment to be encoded comprising one or more video frames; calculating spatiotemporal feature information of the video segment to be encoded based on the one or more video frames; determining multiple predicted bitrate factors of the video segment to be encoded under multiple preset video encoding standards based on the spatiotemporal feature information; determining multiple target bitrate factors of the video segment to be encoded under multiple preset video encoding standards based on a first predicted bitrate factor corresponding to a first preset video encoding standard among multiple preset video encoding standards and a preset video encoding score interval among the multiple predicted bitrate factors, wherein, among the multiple preset video encoding standards, a first predicted bitrate factor corresponding to a first preset video encoding standard among the multiple preset video encoding standards is used. A preset video encoding standard performs video encoding in the shortest time; and the video segments to be encoded are encoded respectively according to a plurality of target bit rate factors to obtain a plurality of target video segments.

According to another aspect of the present disclosure, a video encoding device is also provided, including: an acquisition module, configured to acquire a video segment to be encoded, the video segment to be encoded including one or more video frames; a calculation module, configured to calculate the spatiotemporal feature information of the video segment to be encoded based on the one or more video frames; a prediction rate factor determination module, configured to determine a plurality of prediction rate factors of the video segment to be encoded under a plurality of preset video encoding standards based on the spatiotemporal feature information; a target rate factor determination module, configured to determine a plurality of target rate factors of the video segment to be encoded under a plurality of preset video encoding standards based on a first prediction rate factor corresponding to a first preset video encoding standard among a plurality of preset video encoding standards among the plurality of prediction rate factors and a preset video encoding score interval, wherein, among the plurality of preset video encoding standards, the time for performing video encoding using the first preset video encoding standard is the shortest; and an encoding module, configured to encode the video segments to be encoded respectively according to the plurality of target rate factors to obtain a plurality of target video segments.

According to another aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and at least one memory communicatively connected to the at least one processor, wherein the at least one memory stores a computer program, and the computer program implements the above-mentioned video encoding method when executed by the at least one processor.

According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the above-mentioned video encoding method when executed by a processor.

According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, wherein the computer program implements the above-mentioned video encoding method when executed by a processor.

According to one or more embodiments of the present disclosure, the rate factors under multiple preset video coding standards are predicted for the first time based on the spatiotemporal feature information of the video clip to be encoded, and the rate factors under multiple preset video coding standards are predicted for the second time based on the predicted rate factors and the preset video coding scores, thereby improving the accuracy of the rate factors, thereby effectively improving the accuracy of video coding and ensuring the video quality. In addition, since the rate factors under multiple preset video coding standards can be predicted at the same time, the encoding system can simultaneously output multiple channels of encoded videos of different video coding standards but all meeting the video quality requirements, avoiding repeated prediction operations for multiple video coding standards, thereby saving time costs. Further, the second prediction of the rate factor is based on the rate factor of the video coding standard with the shortest encoding time in the first prediction result. Since the computational complexity of the video coding standard with the shortest encoding time is the lowest, the rate factor under this video coding standard can reflect the accuracy of the rate factors under other video coding standards with longer encoding time to a certain extent, thereby associating different video coding standards, which is conducive to further reducing the amount of calculation required to predict the rate factor.

It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

FIG1 shows a flowchart of a video encoding method according to some embodiments of the present disclosure;

FIG2 shows a flowchart of a method for training a first rate factor prediction model according to some embodiments of the present disclosure;

FIG3 shows a structural block diagram of a first rate factor prediction model according to some embodiments of the present disclosure;

FIG4 shows a flowchart of determining multiple target bit rate factors of a video segment to be encoded under multiple preset video coding standards according to some embodiments of the present disclosure;

FIG5 shows a structural block diagram of a video encoding device according to an embodiment of the present disclosure;

FIG. 6 shows a block diagram of an exemplary electronic device that can be used to implement an embodiment of the present disclosure.

Detailed ways

The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and/or" used in this disclosure covers any one of the listed items and all possible combinations.

With the development of Internet technology, video platforms have also developed rapidly, with more and more users sharing and watching videos. For the platform, as the number of users continues to grow, the bandwidth cost for video transmission is also increasing. In order to reduce this cost, a transcoding system with better compression performance is needed to significantly reduce the size of the compressed video while ensuring the video quality, thereby reducing the network traffic required for transmission.

At present, common encoding modes of transcoding systems include fixed quantization parameter mode, constant bit rate mode, variable bit rate mode, etc. Among them, the most commonly used encoding mode is constant bit rate mode. Although the video stream output in this mode has a stable bit rate, which can improve the problem of playback jams, in this mode, since all videos are encoded using the same bit rate factor, it is impossible to ensure the picture quality experience of all videos for videos with rich scene content, and it is also impossible to avoid bit rate waste. For example, for videos with complex scenes, using the average bit rate factor to encode the video will make the bit rate of the output video insufficient and cause a significant decrease in picture quality. For videos with simple scenes, using the average bit rate factor to encode the video may cause unnecessary bit rate waste.

In order to ensure the picture quality experience and avoid bit rate waste, a constant quality encoding mode can also be used. Specifically, a target quality can be set, and the encoder can be used to encode the video to obtain a video that meets the target quality. The goal of this encoding mode is to make the quality of the encoded video as close to the set target quality as possible, and the closer the quality of the encoded video is to the set target quality, the better the encoding effect. When the actual quality is lower than the target quality, the video quality experience may be reduced, and when the actual quality exceeds the target quality, it may cause bit rate waste.

However, before using this encoding mode to encode a video, the relationship between the bit rate factor used in the encoding and the actual quality of the encoding result is unknown, that is, it is impossible to directly obtain the accurate bit rate factor by setting the target quality. Common solutions are:

(1) Encode the video multiple times and search for the optimal bit rate factor in a posteriori manner, but this method consumes huge computing resources;

(2) Predicting the bit rate factor, for example, the bit rate factor can be predicted based on the spatiotemporal complexity information of the video before encoding the video through machine learning. However, this solution is usually a one-time prediction and encoding method, that is, according to a preset goal, the bit rate factor is predicted based on the spatiotemporal complexity information of the video, and the video is encoded based on the predicted bit rate factor. This may result in a low accuracy of the prediction result of the bit rate factor, especially for videos with complex scenes or hot videos, which may not be able to stably ensure the picture quality experience and avoid bit rate waste at the same time.

In addition, the inventors have also found that in order to adapt to various terminal players, the transcoding system needs to output video streams that meet different video coding standards. However, as mentioned above, in the prior art, a one-time prediction and encoding method is usually adopted, which will result in the transcoding system needing to predict the bit rate factor and encode the video once for each video coding standard, thereby increasing the amount of calculation and time cost.

In view of this, an embodiment of the present disclosure provides a video encoding method, which performs a first prediction of the bit rate factors under multiple preset video encoding standards based on the spatiotemporal feature information of the video clip to be encoded, and performs a second prediction of the bit rate factors under multiple preset video encoding standards based on the predicted bit rate factors and the preset video encoding scores, thereby improving the accuracy of the bit rate factors, thereby effectively improving the accuracy of video encoding, and ensuring the video quality. In addition, since the bit rate factors under multiple preset video encoding standards can be predicted at the same time, the encoding system can simultaneously output multiple channels of encoded videos of different video encoding standards but all meeting the video quality requirements, avoiding repeated prediction operations for multiple video encoding standards, thereby saving time costs.

Furthermore, in the method, the second prediction of the rate factor is based on the rate factor corresponding to the video coding standard with the shortest encoding time in the first prediction result. Since the computational complexity of the video coding standard with the shortest encoding time is the lowest, the rate factor under this video coding standard can reflect the accuracy of the rate factor under other video coding standards with longer encoding time to a certain extent, thereby associating different video coding standards, which is conducive to further reducing the amount of calculation required for predicting the rate factor.

The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

FIG1 shows a flow chart of a video encoding method 100 according to some embodiments of the present disclosure. As shown in FIG1 , the method 100 may include: step S110, obtaining a video segment to be encoded, the video segment to be encoded including one or more video frames; step S120, calculating the spatiotemporal feature information of the video segment to be encoded based on the one or more video frames; step S130, determining multiple predicted rate factors of the video segment to be encoded under multiple preset video encoding standards based on the spatiotemporal feature information; step S140, determining multiple target rate factors of the video segment to be encoded under multiple preset video encoding standards based on the first predicted rate factor corresponding to the first preset video encoding standard among the multiple preset video encoding standards and the preset video encoding score interval among the multiple predicted rate factors, wherein the time for video encoding using the first preset video encoding standard is the shortest among the multiple preset video encoding standards; and step S150, encoding the video segments to be encoded respectively according to the multiple target rate factors to obtain multiple target video segments.

By using the spatiotemporal feature information of the video clip to be encoded to determine the predicted rate factors under multiple preset video coding standards, and using the predicted rate factors and preset video coding scores to determine the target rate factors under multiple preset video coding standards, the rate factors can be corrected, thereby improving the accuracy of the rate factors, thereby effectively improving the accuracy of video coding and ensuring video quality. In addition, since the rate factors under multiple preset video coding standards can be predicted at the same time, the encoding system can simultaneously output multiple encoded videos of different video coding standards but all meeting the video quality requirements, avoiding repeated prediction operations for multiple video coding standards, thereby saving time costs. Furthermore, the correction of the rate factor is based on the rate factor of the video coding standard with the shortest encoding time in the first prediction result. Since the rate factor of the video coding standard with the shortest encoding time The computational complexity is the lowest, so the bit rate factor under this video coding standard can, to a certain extent, reflect the accuracy of the bit rate factors under other video coding standards with longer encoding time, thereby associating different video coding standards and realizing the joint prediction of bit rate factors for different video coding standards, further reducing the amount of calculation required to predict the bit rate factor.

According to some embodiments of the present disclosure, step S110, obtaining a video segment to be encoded may include: obtaining a video to be encoded; performing scene detection on the video to be encoded; dividing the video to be encoded into one or more sub-video segments based on the scene detection result; and identifying each video sub-segment in the one or more video segments as a video segment to be encoded.

According to some embodiments, the stored or cached video to be encoded can be read from an appropriate storage device (local and/or remote). Alternatively, the video to be encoded can also be received from other external devices via a wired or wireless communication link. The video to be encoded can refer to any complete video file. For example, it can be a video file recorded by the user himself, or a video file captured by the user from other video files, or a video file created by the user based on multiple video files, and the scope of the subject matter claimed for protection in the present disclosure is not limited in this respect.

After the video to be encoded is obtained, scene detection may be performed on the video to be encoded, and the video to be encoded may be segmented according to the detection result, and each segmented video segment may be used as a video segment to be encoded.

According to some embodiments, scene detection can be performed on the video to be encoded based on an open source encoder. For example, in a scenario where a video V is determined to be a video to be encoded in response to receiving a video V uploaded by a user, the open source encoder x264 can be called to encode the video V, and the scene detection switch can be turned on to obtain a corresponding scene switching detection result. According to the detection result, the video V can be divided into t independent single-lens video segments (V1, V2...Vt), and each single-lens video segment can be used as a video segment to be encoded.

According to other embodiments, the video to be encoded can be segmented based on a frame operation. Continuing with the above example where the video to be encoded is V, the video to be encoded V can be subjected to a frame operation to obtain a plurality of video frames, and then the similarity between the video frame and the adjacent video frame is determined for each of the video frames, and based on the calculated similarity, it is determined whether the video frame is a boundary video frame for distinguishing adjacent shots of the video to be processed. It should be understood that any suitable decoder or other technology (e.g., Open CV, FFmpeg, etc.) can be used to perform the frame operation, and any suitable calculation method can also be used to determine the similarity between adjacent video frames (e.g., calculating frame differences, grayscale histograms, etc.), and the scope of the subject matter claimed for protection in the present disclosure is not limited in this respect.

Compared with a complete video file, the content shown in a single-shot video clip is usually similar and more coherent. Therefore, when encoding the video, it is more reasonable to use a specific and identical bitrate factor for each single-shot video clip, thereby ensuring a stable picture quality experience in all scenarios.

According to some embodiments of the present disclosure, the spatiotemporal domain feature information may include spatiotemporal complexity information, and wherein step S120, calculating the spatiotemporal domain feature information of the video segment to be encoded based on one or more video frames may include: calculating the grayscale symbiosis information and normalization information of the video segment to be encoded based on the one or more video frames as the spatiotemporal complexity information, wherein the grayscale symbiosis information is used to describe the spatial complexity of the video segment to be encoded, and the normalization information is used to describe the temporal complexity of the video segment to be encoded; and combining the grayscale symbiosis information and the normalization information to generate the spatiotemporal complexity information of the video segment to be encoded.

Gray level co-occurrence information (also known as gray level co-occurrence matrix, Gray Level Co-occurrence Matrix, GLCM) is a feature used to describe the complexity of video spatial texture. In some embodiments, for each video frame in one or more video frames of the video segment to be encoded, an m×m two-dimensional GLCM matrix can be calculated, where the value range of each element in the GLCM matrix is 0-1. Then, five eigenvalues such as information entropy, contrast, correlation, energy, and autocorrelation can be further solved for the matrix, and four statistical values such as mean, variance, skewness, and kurtosis are calculated for these eigenvalues corresponding to all video frames. For example, the mean, variance, skewness, and kurtosis of the information entropy of each video frame to be processed are calculated. Finally, a total of 20 (5×4) eigenvalues will be obtained. Further, in order to consider the multi-scale characteristics of the video, each video frame can be reduced by 4 times and 16 times, and the above steps are repeated respectively (i.e., GLCM feature extraction is performed twice). Therefore, 60 feature values, that is, 60 GLCM feature values, will be finally obtained for the video frame of the video segment to be encoded.

Normalized information (also known as the Normalized Correlation Coefficient, NCC) is a feature used to describe time domain complexity. In some embodiments, for each pair of adjacent video frames in one or more video frames of the video segment to be encoded, the NCC can be calculated to obtain a two-dimensional correlation coefficient matrix with the same length and width as the original video frame, wherein the value range of each element in the matrix is 0-1, indicating the time domain correlation of the corresponding positions of two adjacent video frames. For a video segment to be encoded containing n video frames, there will be n-1 adjacent video frame pairs, so n-1 correlation coefficient matrices can be obtained. Then, the mean, variance, kurtosis, skewness and information entropy of these n-1 correlation coefficient matrices are calculated, totaling 5 eigenvalues, and for each eigenvalue, the mean and variance are calculated from the time domain, and finally 10 (5×2) eigenvalues can be obtained, that is, 10 NCC eigenvalues.

After the GLCM eigenvalue and the NCC eigenvalue are calculated by, for example, the method described above, the two may be combined to obtain 70 eigenvalues as the spatiotemporal complexity information of the video segment to be encoded, wherein the spatiotemporal complexity information may be represented in a vector form.

Since grayscale co-occurrence information and normalization information can characterize the complexity of the video to be encoded from the spatial and temporal perspectives respectively, predicting multiple bit rate factors of the video fragments to be encoded under different video coding standards based on these spatial complexity information can make the predicted bit rate factors more accurate, which is beneficial to ensure the quality of the output video.

According to some embodiments of the present disclosure, the spatiotemporal feature information may include coding feature information, and wherein step S120, calculating the spatiotemporal feature information of the video segment to be encoded based on one or more video frames may include: preprocessing one or more video frames to generate a new video frame sequence, wherein the new video frame sequence includes a set of video frame pairs; and performing intra-frame encoding and inter-frame encoding for each video frame pair in the set of video frame pairs, respectively, to obtain the coding feature information of the video segment to be encoded.

By performing intra-frame coding or inter-frame coding on each video frame in the video to be coded, the spatiotemporal domain feature information of the video segment to be coded can be reflected from another dimension, namely, the spatial coding complexity and the temporal coding complexity. This is because intra-frame coding uses the internal image data of the video frame to be coded, while inter-frame coding uses the image data of the video frame to be coded and its adjacent video frames, thereby more comprehensively representing the video segment to be coded. The combination of this coding feature information and the spatiotemporal complexity information composed of the GLCM eigenvalue and the NCC eigenvalue can be used to predict multiple bit rate factors of the video segment to be coded under different video coding standards, which can further improve the accuracy of the prediction results, which will be particularly beneficial for hot videos that expect higher quality and more refined coding.

According to some embodiments of the present disclosure, in step S120, preprocessing one or more video frames to generate a new video frame sequence may include: copying video frames other than the first frame and the last frame of the one or more video frames to generate one or more copied video frames; and sorting the one or more video frames and the one or more copied video frames so that each of the one or more copied video frames is located after the corresponding video frame.

According to some embodiments, after generating one or more duplicate frames corresponding to video frames other than the first frame and the last frame, each duplicate frame may be placed after the corresponding original video frame to generate a new video frame sequence. For example, assuming that the video clip to be encoded includes n video frames and the sequence composed of the n video frames is ( _f1 , _f2 , _f3 , …, fn _-2 , fn _-1 , _fn ), then except the first frame _f1 and the last frame _fn , each video frame from video frames _f2 to fn _-1 is copied to generate a new video frame sequence ( _f1 , _f2 , _f2 , _f3 , _f3 , …, fn _-2 , fn _-2 , fn _{-1, fn-1 , fn} ), where ( _f1 , _f2 ), ( _f2 , _f3 ), … ( _fn-2 , fn _-1 ), ( _fn-1 , _fn ) are video frame pairs.

Compared with a video frame sequence without generating duplicate frames, by copying the video frames except the first frame and the last frame and generating one or more ordered video frame pairs, it can ensure that the data within each video frame and between adjacent video frames can be utilized, retaining data integrity, and facilitating more accurate intra-frame encoding and inter-frame encoding of each video frame of the encoded video fragment, thereby improving the accuracy of the predicted bit rate factor.

According to other embodiments of the present disclosure, preprocessing one or more video frames may also include, for example, cropping each video frame to meet size requirements, removing unnecessary (interference) features in the video frame, etc. Thus, the calculation and prediction speed can be accelerated, the encoding feature information can be quickly obtained, and the bit rate factor of the video to be encoded can be predicted, which is also conducive to improving the accuracy of the predicted bit rate factor.

After the video segment to be encoded is preprocessed to obtain a set of video frame pairs, it can be pre-encoded using, for example, a hardware H.264 encoder built into a graphics card. Pre-encoding can use, for example, a fixed quantization parameter mode, and optionally, the quantization parameter can be any suitable value such as 28, 29, 30, etc. The encoding frame structure can use, for example, an IP structure, where I represents intra-frame coding and P represents inter-frame coding. In this encoding mode, except for the first frame and the last frame, each video frame of the video segment to be encoded is encoded twice (i.e., once using intra-frame coding I and once using inter-frame coding P), where intra-frame coding can reflect the spatial coding complexity of each video frame, and inter-frame coding can reflect the temporal coding complexity of each video frame. After IP encoding is performed on each set of video frame pairs, the encoding feature information of the video segment to be encoded can be obtained.

According to some embodiments of the present disclosure, performing intra-frame coding and inter-frame coding respectively on each video frame pair in a video frame pair set to obtain coding feature information of a video segment to be encoded may include: performing intra-frame coding on the first video frame in each video frame pair in the video frame pair set to obtain intra-frame coding information; performing inter-frame coding on each video frame pair in the video frame pair set to obtain inter-frame coding information; and obtaining coding feature information based on the intra-frame coding information and the inter-frame coding information.

There are two video frames in each video frame pair of the video segment to be encoded. In this application, the first video frame refers to the previous video frame in the video frame pair. For example, for the video frame pair (f ₁ , f ₂ ), f ₁ is the first video frame; for the video frame pair to be encoded (f ₂ , f ₃ ), f ₂ is the first video frame, and so on.

According to some examples, intra-coding the first video frame in each video frame pair to obtain intra-coding information may include: extracting characteristic values of each intra-coding after intra-coding each first video frame, such as 34 characteristic values such as the number of coding bits and the proportion of intra-prediction mode; and calculating five statistical values of the mean, variance, skewness, kurtosis and information entropy of the corresponding characteristic values of all intra-coding to obtain 170 (i.e., 34×5) intra-coding information.

For each video pair, the inter-frame coding information of each video frame pair can also be extracted. For example, for a video frame pair (f ₁ , f ₂ ), the inter-frame coding information of f ₁ and f ₂ is calculated; for a video frame pair (f ₂ , f ₃ ), the inter-frame coding information of f ₂ and f ₃ is calculated, and so on.

According to some examples, performing inter-frame coding on each video frame pair to obtain inter-frame coding information may include: extracting a feature value of each inter-frame coding after performing inter-frame coding on each video frame pair, such as the number of coding bits, the number of inter-frame operations, and the number of inter-frame operations. 26 eigenvalues such as motion vector amplitude distribution, and 5 statistical values including mean, variance, skewness, kurtosis and information entropy are calculated for the corresponding eigenvalues of all inter-frame coding to obtain 130 (i.e. 26×5) inter-frame coding information.

Continuing the above example, 170 intra-frame coding information and 130 inter-frame coding information are concatenated to obtain coding feature information of 300 video segments to be coded. Similar to the spatial complexity information, the coding feature information can also be represented in vector form.

It will be understood that in the above embodiments, although the intraframe coding step is described as being performed after the interframe coding step, this should not be interpreted as requiring that the intraframe coding and the interframe coding must be performed in this particular order. For example, the intraframe coding step can be performed in parallel with the interframe coding step, or the intraframe coding step can be performed after the interframe coding step.

It will also be understood that, in the above-mentioned embodiments, although the intra-frame coding information and the inter-frame coding information are described as the mean, variance, skewness, kurtosis and information entropy of characteristic values such as the number of coding bits, the proportion of intra-frame prediction modes, the distribution of inter-frame motion vector amplitudes, etc., these characteristic values can also be directly determined as the intra-frame coding information or the inter-frame coding information.

According to some embodiments of the present disclosure, step S130, determining multiple predicted rate factors of the video segments to be encoded under multiple preset video coding standards based on the spatiotemporal domain feature information may include: inputting the spatiotemporal domain feature information of the video segments to be encoded into a first rate factor prediction model, so as to determine multiple predicted rate factors of the video segments to be encoded under multiple preset video coding standards via the first rate factor prediction model.

Rate Factor (RF) refers to the encoding parameter in the constant rate factor encoding mode, which is used to control the bit rate and quality of the encoded file. The value of the rate factor is usually in the range of 0-50. The larger the value of the rate factor, the lower the bit rate and the worse the encoding quality. Conversely, the smaller the value of the rate factor, the higher the bit rate and the better the encoding quality.

Examples of multiple preset video coding standards may include H.264 video coding standard, H.265 video coding standard, H.266 video coding standard, AV1 video coding standard. It will be understood that the preset video standard may also include any other existing video coding standard or video coding standard generated as technology develops. It will also be understood that the number of multiple preset video coding standards may be any integer value that meets actual needs. The scope of the subject matter claimed in the present disclosure is not limited in both aspects.

According to some embodiments of the present disclosure, when the spatiotemporal domain feature information only includes spatiotemporal complexity information, the feature information obtained by combining the grayscale co-occurrence information and normalization information of the video segment to be encoded can be input into a first rate factor prediction model, wherein the first rate factor prediction model is trained to determine the predicted rate factor of each video segment to be encoded under multiple preset video coding standards based on the spatiotemporal complexity information.

According to some other embodiments of the present disclosure, when the spatiotemporal domain feature information only includes coding feature information, feature information obtained by combining intra-frame coding information and inter-frame coding information of the video segment to be encoded can be input into a first rate factor prediction model, wherein the first rate factor prediction model is trained to determine the predicted rate factor of each video segment to be encoded under multiple preset video coding standards based on the coding feature information.

According to some other embodiments of the present disclosure, when the spatiotemporal domain feature information includes both spatiotemporal complexity information and coding feature information, the spatiotemporal complexity information and the coding feature information can be combined to generate feature information represented by a set of feature vectors. Continuing with the above example where the spatiotemporal complexity information is a vector containing 70 eigenvalues and the coding feature information is a vector containing 300 eigenvalues, a feature vector containing 370 eigenvalues will be obtained after combining these two vectors, thereby more completely characterizing the spatiotemporal characteristics of the video segment to be encoded. In this case, the generated set of feature vectors can be input into a first rate factor prediction model, wherein the first rate factor prediction model is trained to determine the predicted rate factor of each video segment to be encoded under multiple preset video coding standards based on both the spatiotemporal complexity information and the coding feature information.

By using the trained first rate factor prediction model to predict the predicted rate factor of each to-be-encoded video segment under multiple preset video encoding standards, the rate factor that meets the quality requirements (for example, the video encoding score of the encoded video segment is higher) can be directly determined without having to obtain it through multiple encodings and searches each time, thereby saving a lot of time costs. At the same time, when the sample data is sufficient, the accuracy of the predicted rate factor can also be ensured to a certain extent.

FIG2 shows a flow chart of a method 200 for training a first rate factor prediction model according to some embodiments of the present disclosure. As shown in FIG2, the first rate factor prediction model can be trained by the following steps of the method 200: step S210, for each of a plurality of preset video coding standards, obtaining sample spatiotemporal feature information of a sample video clip and a sample rate factor of the sample video clip under the preset video coding standard; step S220, inputting the sample spatiotemporal feature information into the first rate factor prediction model to determine the sample predicted rate factor of the sample video clip under the preset video coding standard via the first rate factor prediction model; step S230, calculating a first model loss value based on the sample rate factor and the sample predicted rate factor; and step S240, adjusting the parameters of the first rate factor prediction model based on the first model loss value under each preset video coding standard until the first model training stop condition is reached.

By pre-training and adjusting the first rate factor prediction model using sample data, the accuracy of the predicted rate factor can be improved.

According to some embodiments, in step S210, a sample video segment may be obtained by using a method similar to that of obtaining the video segment to be encoded in step S110, and the temporal and spatial characteristics of the video segment to be encoded may be calculated by using the method similar to that of calculating the temporal and spatial characteristics of the video segment to be encoded in step S120. In order to ensure the accuracy of the first rate factor prediction model after training, more sample video clips can be obtained under the premise of taking into account computing resources and time costs. The number of sample video clips can be, for example, at least 50,000, at least 100,000, etc.

According to some embodiments, in step S210, the sample rate factor may be a real rate factor, that is, the quality of the encoded video obtained after encoding the sample video segment using the rate factor meets the target quality requirement. The quality of the encoded video may be evaluated, for example, by a video coding score (also known as a video multimethod assessment fusion, VMAF). The larger the value of VMAF, the higher the quality of the encoded video. Conversely, the smaller the value of VMAF, the lower the quality of the encoded video. The value range of VMAF may be 0-100 under normal circumstances. The real bit rate factor can be obtained by the following steps: for the video coding standard, selecting an initial sample bit rate factor and a preset sample video coding score range; based on the initial sample bit rate factor, using a corresponding encoder in a real transcoding system (for example, H.264 encoder, H.265 encoder, H.266 encoder, AV1 encoder, etc.) to encode the sample video clip to obtain a sample pre-encoded video clip under the video coding standard; comparing the sample video coding score of the obtained sample pre-encoded video clip with the preset sample video coding score range; if the sample video coding score is within If the sample video coding score is within the preset sample video coding score range (i.e., the encoded video meets the target quality requirements), it indicates that the initial sample rate factor under the video coding standard is feasible, and the initial sample rate factor can be used as the real rate factor; if the sample video coding score is outside the preset sample video coding score range (i.e., the encoded video does not meet the target quality requirements), it indicates that the initial sample rate factor under the video coding standard is not feasible, and a search method such as binary search can be used to adjust the initial sample rate factor until the sample video coding score is within the preset sample video coding score range. Usually, for the same sample video clip, the rate factors under different video coding standards are generally different. Therefore, the above steps will be repeated for each video coding standard to search for sample rate factors (real rate factors) of sample video clips under multiple video coding standards.

In the above embodiment, the preset sample video coding score interval may be, for example, a value range based on VMAF and customized by the user according to actual needs (e.g., [91, 93], [90, 95]), or an interval set for each VMAF integer value in order to adapt the first rate factor prediction model to multiple target quality requirements (e.g., [91.5, 92.5], [92.5, 93.5] ... [95.5, 96.5], etc.). The scope of the subject matter claimed by the present disclosure is not limited in this respect.

According to some embodiments, in step S220, in the initial stage, initial parameters of a first rate factor prediction model may be selected for a plurality of video coding standards, and based on the initial parameters, a plurality of sample prediction rate factors of sample video clips under the plurality of video coding standards may be determined using the acquired sample spatiotemporal domain feature information.

According to some embodiments, in step S230, for each preset video coding standard, a model loss value may be calculated based on the corresponding sample prediction rate factor and the sample prediction rate factor, for example, via a loss function. Examples of loss functions include, but are not limited to, a cross entropy loss function, a maximum loss function, an average loss function, a 0-1 loss function, etc. It will be understood that in the present disclosure, other suitable methods may also be used to calculate the model loss value, and the scope of the subject matter claimed in the present disclosure is not limited in this respect.

According to some embodiments, in step S240, after calculating the model loss value under each preset video coding standard, these model loss values can be back-propagated, and the parameters of the first rate factor prediction model can be adjusted, and then steps S220-S240 are repeated until the model training stop condition is reached. Specifically, the model training stop condition can be that the model loss value is lower than a preset threshold, and/or the number of rounds of model training reaches a preset number of rounds. It will be understood that the present disclosure does not limit the model training stop condition, and it can be adjusted according to actual needs.

Fig. 3 shows a structural block diagram of a first rate factor prediction model 300 according to some embodiments of the present disclosure. As shown in Fig. 3, the first rate factor prediction model 300 may include a batch normalization module 310, a first attention mechanism module 320, a residual module 330, a second attention mechanism module 340 and a fully connected module 350, and the first rate factor prediction model 300 takes the spatiotemporal domain features of the video segment to be encoded as input, and outputs the predicted rate factor of the video segment to be encoded under different video coding standards.

The batch normalization module 310 is used to normalize the original input features into features with a mean of 0, a variance of 1, and a Gaussian distribution, thereby eliminating the impact of the dimensional differences between the features. Then, the normalized features are input to the first attention mechanism module 320, which can calculate a weight for each feature, wherein a larger weight is assigned to features with higher recognition and a smaller weight is assigned to features with lower information content, thereby automatically screening the features, thereby improving the accuracy of the prediction results. Then, the screened features are input to the residual module 330, which may include multiple convolutional layers, which change the screened features as input, and the unchanged screened features will skip these convolutional layers and be transmitted to the subsequent layers, and then be processed as a whole to obtain the output result of the residual module, thereby reducing information loss. The output of the residual module 330 can be further input to the second attention mechanism module 340 to achieve further screening of the features, thereby further improving the accuracy of the prediction results. Finally, the input features may be aggregated via the fully connected module 350 to generate and output a predicted bit rate factor.

FIG4 shows a flowchart of determining multiple target rate factors of a video segment to be encoded under multiple preset video coding standards according to some embodiments of the present disclosure. As shown in FIG4, step S140, based on a first predicted rate factor of a first preset video coding standard among multiple preset video coding standards and a preset video coding score interval, determines multiple target rate factors of a video segment to be encoded under multiple preset video coding standards. The method comprises: step S442, performing a first encoding on a video segment to be encoded based on a first predicted bit rate factor to obtain a first pre-encoded video segment under a first preset video coding standard; step S444, determining whether a video coding score of the first pre-encoded video segment is within a preset video coding score range; and step S446, in response to determining that the video coding score of the first pre-encoded video segment is within the preset video coding score range: determining a plurality of predicted bit rate factors as a plurality of target bit rate factors.

In the present disclosure, the first preset video coding standard refers to the video coding standard that takes the shortest time to perform video coding. For example, among the video coding standards H.264, H.265, H.266, and AV1, the video coding standard H.264 takes the shortest time to perform video coding, so the video coding standard H.264 is the first preset video coding standard.

In step S442, in addition to the first predicted bit rate factor, the encoding parameters used when encoding the video to be encoded may also include but are not limited to preset, GOP (Group of Picture), size, etc. Since the bit rate factor plays an important role in the quality of the encoded video, when encoding the video using the method of the present disclosure, the values of these encoding parameters can be set to fixed constants (for example, default preset values).

In step S444, similar to the setting of the preset sample video encoding score interval, the preset video encoding score interval can also be a numerical range customized by the user according to actual needs, so it will not be described in detail here.

Since the computational complexity of the video coding standard with the shortest encoding time is the lowest, the rate factor under this video coding standard can, to a certain extent, reflect the accuracy of the rate factors under other video coding standards with longer encoding times. In other words, if the video coding score of the video clip encoded based on the first predicted rate factor corresponding to the first preset video coding standard (for example, the video coding standard H.264 in the above example) is within the preset video coding score range (meeting the video quality requirement), it indicates that the accuracy of other predicted rate factors corresponding to other preset video coding standards (for example, the video coding standards H.265, H.266 and AV1 in the above example) is also high (for example, if the video coding score of the encoded video under the video coding standard H.264 has a probability of 99% being within the preset video coding score range, the video coding scores of the encoded videos under the video coding standards H.265, H.266 and AV1 also have a probability of more than 95% being within the preset video coding score range). The video coding scores of the video clips encoded using these predicted bitrate factors are usually also within the preset video coding score range (meeting the video quality requirements), so that different video coding standards can be associated, which is conducive to reducing the amount of calculation required for the predicted bitrate factors. In some examples, after multiple test data, it is shown that when the predicted bitrate factors under the video coding standard H.264 meet the video quality requirements, the probability that the predicted bitrate factors under the video coding standards H.265, H.266 and AV1 meet the video quality requirements is 95%, so these predicted bitrate factors can be directly used as target bitrate factors for encoding the video to be encoded.

Continuing to refer to Figure 4, according to other embodiments of the present disclosure, step S140, determining multiple target rate factors for the video segment to be encoded under multiple preset video coding standards based on a first predicted rate factor among multiple predicted rate factors corresponding to a first preset video coding standard among multiple preset video coding standards and a preset video coding score range, may also include: step S448, in response to determining that the video coding score of the first pre-encoded video segment is not within the preset video coding score range: based on the spatiotemporal domain feature information of the video segment to be encoded, the multiple predicted rate factors and the video coding score of the first pre-encoded video segment, updating the multiple predicted rate factors to obtain multiple target rate factors.

Continuing with the above example of multiple preset video coding standards being H.264, H.265, H.266 and AV1, when the video coding score of the pre-encoded video clip obtained based on the predicted bitrate factor under the video coding standard H.264 is not within the preset video coding score range, the respective predicted bitrate factors under the video coding standards H.265, H.266 and AV1 and the video coding score of the pre-encoded video clip under the video coding standard H.264 can be used as feedback information, together with the spatiotemporal domain feature information of the video clip to be encoded, to perform a second prediction of the bitrate factors under these video coding standards to obtain multiple target bitrate factors.

A large amount of experimental data shows that after the first prediction and encoding of the video clip to be encoded using the predicted bit rate factor, the probability that the video coding score meets the video quality requirements is 40%. Therefore, 60% of the video clips to be encoded will undergo the above-mentioned second prediction and update bit rate factor operation. When the video coding score of the first pre-encoded video clip is not within the preset video coding score range, by performing a second prediction of the bit rate factor under different video coding standards, the accuracy of the target bit rate factor can be further improved, avoiding the large error of the bit rate factor predicted for the above 60% of the video clips to be encoded and failing to stably guarantee the picture quality experience.

In addition, the update of the predicted bit rate factor obtained by the first prediction is based on the video coding score of the first pre-encoded video clip, so there is no need to use the predicted bit rate factors under other preset video coding standards to encode the video clips to be encoded separately, which will greatly save the time cost and computing resources required for encoding.

According to some embodiments of the present disclosure, step S448, based on the spatiotemporal domain feature information of the video segment to be encoded, multiple predicted rate factors and the video coding score of the first pre-encoded video segment, updating multiple predicted rate factors to obtain multiple target rate factors may include: inputting the spatiotemporal domain feature information of the video segment to be encoded, multiple predicted rate factors and the video coding score of the first pre-encoded video segment into a second rate factor prediction model, so as to determine multiple target rate factors of the video segment to be encoded under multiple preset video coding standards via the second rate factor prediction model.

According to some embodiments of the present disclosure, similar to the training method of the first rate factor prediction model, the second rate factor prediction model can be trained by the following operations: inputting the sample spatiotemporal feature information, the sample prediction rate factor under each preset video coding standard, and the first sample video coding score corresponding to the sample prediction rate factor under the first preset video coding standard into the second rate factor prediction model, so as to determine the second rate factor prediction model through the second rate factor prediction model. Determine multiple sample target rate factors of sample video clips under multiple preset video coding standards; for each preset video coding standard in the multiple preset video coding standards, calculate a second model loss value based on the sample rate factor and the sample target rate factor corresponding to the preset video coding standard; and adjust the parameters of the second rate factor prediction model based on the second model loss value until the second model training stop condition is reached.

It will be understood that, except for adding the sample prediction rate factor under each preset video coding standard and the first sample video coding score as inputs of the second rate factor prediction model, the steps in the above method for training the second rate factor prediction model are similar to the method 200 for training the first rate factor prediction model, so they will not be repeated here.

Correspondingly, compared with the first rate factor prediction model, in addition to adding the sample prediction rate factor under each preset video coding standard and the first sample video coding score as input, the structural framework of the second rate factor prediction model is also the same as the structural framework of the first rate factor prediction model, so it will not be repeated here.

It will also be understood that the training of the second rate factor prediction model is dependent on the training results of the first rate factor prediction model. Therefore, the training of the second rate factor prediction model is performed after the training of the first rate factor prediction model is completed.

After determining multiple target bit rate factors, the corresponding encoders in the transcoding system (based on the corresponding preset video coding standards) are used to encode the video segments to be encoded according to the multiple bit rate factors. After multiple test data show that after encoding the video segments using the target bit rate factors obtained by the second prediction, there is a 99% probability that the video coding score of the encoded video segments is within the preset video coding score range. Therefore, the encoding result of the video segments based on the target bit rate factors obtained by the second prediction can be directly trusted.

After encoding each video segment to be encoded using multiple target bit rate factors, method 100 may also include: for each preset video coding standard among multiple preset video coding standards, obtaining all target video segments under the preset video coding standard; and combining all target video segments based on the order of one or more sub-video segments in the video to be encoded to obtain the target video corresponding to the video to be encoded under the preset video coding standard.

After performing the above encoding operation on each video segment to be encoded in a video to be encoded, multiple target video segments for each video segment to be encoded can be obtained. Then, for each preset video encoding standard, the corresponding target video segment among the multiple target video segments for each video segment to be encoded can be written into the final video bitstream file, and the written target video segments are spliced according to the order of each video segment to be encoded in the video to be encoded, so as to obtain the target video corresponding to the video to be encoded under the preset video encoding standard.

As described above, each video segment to be encoded is a single-shot video segment showing similar and coherent content. Therefore, encoding each video segment to be encoded and obtaining the corresponding target video segment can ensure that each encoded video segment is The quality of the video clips to be encoded is improved, thereby ensuring a stable picture quality experience in various scenarios. In addition, the target video clips are spliced and combined based on the order of these video clips to be encoded in the video to be encoded, so that the complete video file can be restored, avoiding the discontinuity of the video file caused by the splicing errors of the target video clips.

FIG5 shows a structural block diagram of a video encoding device 500 according to an embodiment of the present disclosure. As shown in FIG5, the device 500 may include: an acquisition module 510, configured to acquire a video segment to be encoded, the video segment to be encoded including one or more video frames; a calculation module 520, configured to calculate the spatiotemporal feature information of the video segment to be encoded based on the one or more video frames; a prediction rate factor determination module 530, configured to determine a plurality of prediction rate factors of the video segment to be encoded under a plurality of preset video encoding standards based on the spatiotemporal feature information; a target rate factor determination module 540, configured to determine a plurality of target rate factors of the video segment to be encoded under a plurality of preset video encoding standards based on a first prediction rate factor corresponding to a first preset video encoding standard among the plurality of preset video encoding standards and a preset video encoding score interval among the plurality of prediction rate factors, wherein the time for performing video encoding using the first preset video encoding standard is the shortest among the plurality of preset video encoding standards; and an encoding module 550, configured to encode the video segments to be encoded respectively according to the plurality of target rate factors to obtain a plurality of target video segments.

According to some embodiments of the present disclosure, the spatiotemporal feature information may include spatiotemporal complexity information, and wherein the calculation module 530 may include: a module configured to calculate the grayscale symbiosis information and normalization information of the video segment to be encoded as the spatiotemporal complexity information based on one or more video frames, wherein the grayscale symbiosis information is used to describe the spatial complexity of the video segment to be encoded, and the normalization information is used to describe the temporal complexity of the video segment to be encoded; and a module configured to combine the grayscale symbiosis information and the normalization information to generate the spatiotemporal complexity information of the video segment to be encoded.

According to some embodiments of the present disclosure, the spatiotemporal domain feature information may include coding feature information, and wherein the computing module 530 may include: a preprocessing module, configured to preprocess one or more video frames to generate a new video frame sequence, the new video frame sequence including a set of video frame pairs; and an intra-frame and inter-frame coding module, configured to perform intra-frame coding and inter-frame coding, respectively, for each video frame pair in the set of video frame pairs to obtain coding feature information of the video segment to be encoded.

According to some embodiments of the present disclosure, the preprocessing module may include: a module configured to copy video frames other than the first frame and the last frame of one or more video frames to generate one or more copied video frames; and a module configured to sort the one or more video frames and the one or more copied video frames so that each of the one or more copied video frames is located after the corresponding video frame.

According to some embodiments of the present disclosure, the intra-frame and inter-frame coding module may include: a module configured to perform intra-frame coding on the first video frame in each video frame pair in the video frame pair set to obtain intra-frame coding information; A module configured to perform inter-frame coding on each video frame pair in the video frame pair set to obtain inter-frame coding information; and a module configured to obtain coding feature information based on the intra-frame coding information and the inter-frame coding information.

According to some embodiments of the present disclosure, the coding feature information may be based on one or more of the following items: the number of coding bits, the proportion of intra-frame prediction modes, and the distribution information of the amplitude of inter-frame motion vectors.

According to some embodiments of the present disclosure, the module 530 for determining the predicted bit rate factor may include: a module configured to input the spatiotemporal domain feature information of the video segment to be encoded into a first bit rate factor prediction model, so as to determine multiple predicted bit rate factors of the video segment to be encoded under multiple preset video coding standards via the first bit rate factor prediction model.

According to some embodiments of the present disclosure, the target bit rate factor determination module 540 may include: a module configured to perform a first encoding on a video segment to be encoded based on a first predicted bit rate factor to obtain a first pre-encoded video segment under a first preset video coding standard; a module configured to determine whether the video coding score of the first pre-encoded video segment is within a preset video coding score range; and a module configured to determine multiple predicted bit rate factors as multiple target bit rate factors in response to determining that the video coding score of the first pre-encoded video segment is within the preset video coding score range.

According to some embodiments of the present disclosure, the module 540 for determining the target bit rate factor may further include: an updating module, configured to, in response to determining that the video coding score of the first pre-encoded video clip is not within a preset video coding score range, update multiple predicted bit rate factors to obtain multiple target bit rate factors based on spatiotemporal domain feature information of the video clip to be encoded, multiple predicted bit rate factors and the video coding score of the first pre-encoded video clip.

According to some embodiments of the present disclosure, the update module may include: a module configured to input the spatiotemporal domain feature information of the video segment to be encoded, multiple predicted bit rate factors, and the video coding score of the first pre-encoded video segment into a second bit rate factor prediction model, so as to determine multiple target bit rate factors of the video segment to be encoded under multiple preset video coding standards via the second bit rate factor prediction model.

According to some embodiments of the present disclosure, the first rate factor prediction model can be trained by the following operations: for each of a plurality of preset video coding standards: obtaining sample spatiotemporal domain feature information of a sample video clip and a sample rate factor of the sample video clip under the preset video coding standard; inputting the sample spatiotemporal domain feature information into the first rate factor prediction model to determine the sample predicted rate factor of the sample video clip under the preset video coding standard via the first rate factor prediction model; and calculating a first model loss value based on the sample rate factor and the sample predicted rate factor; and adjusting the parameters of the first rate factor prediction model based on the first model loss value under each preset video coding standard until the first model training stop condition is reached.

According to some embodiments of the present disclosure, the second rate factor prediction model is trained by the following operations: combining the sample spatiotemporal feature information, the sample prediction rate factor under each preset video coding standard, and the first preset video coding standard with the sample prediction rate factor; A first sample video coding score corresponding to a sample predicted rate factor under a coding standard is input into a second rate factor prediction model to determine multiple sample target rate factors of sample video clips under multiple preset video coding standards through the second rate factor prediction model; for each of the multiple preset video coding standards, a second model loss value is calculated based on the sample rate factor and the sample target rate factor corresponding to the preset video coding standard; and parameters of the second rate factor prediction model are adjusted based on the second model loss value under each preset video coding standard until the second model training stop condition is reached.

According to some embodiments of the present disclosure, the acquisition module 510 may include: a module configured to acquire a video to be encoded; a module configured to perform scene detection on the video to be encoded; a module configured to divide the video to be encoded into one or more sub-video segments based on the scene detection results; and a module configured to identify each video sub-segment in one or more video segments as a video segment to be encoded.

According to some embodiments of the present disclosure, the device 500 may also include: a module configured to obtain all target video segments under each preset video coding standard among multiple preset video coding standards; and a module configured to combine all target video segments based on the order of one or more sub-video segments in the video to be encoded to obtain a target video corresponding to the video to be encoded under the preset video coding standard.

It should be understood that the modules 510-550 of the apparatus 500 shown in FIG5 may correspond to the steps S110-S150 in the method 100 described with reference to FIG1. Thus, the operations, features and advantages described above for the method 100 are also applicable to the apparatus 500 and the modules included therein. For the sake of brevity, some operations, features and advantages are not described in detail herein.

It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. The various modules described above with respect to FIG. 5 may be implemented in hardware or in hardware in combination with software and/or firmware. For example, these modules may be implemented as computer program codes/instructions configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules may be implemented as hardware logic/circuits. For example, in some embodiments, one or more of the acquisition module 510, the calculation module 520, the determination prediction rate factor module 530, the determination target rate factor module 540, and the encoding module 550 may be implemented together in a system on chip (System on Chip, SoC). SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and/or one or more components in other circuits), and may optionally execute the received program code and/or include embedded firmware to perform functions.

According to another aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and at least one memory communicatively connected to the at least one processor; wherein the at least one memory stores a computer program, and the computer program implements the above-mentioned video encoding method when executed by the at least one processor.

According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the above-mentioned video encoding method when executed by a processor.

According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, wherein the computer program implements the above-mentioned video encoding method when executed by a processor.

Referring to Fig. 6, the structural block diagram of the electronic device 600 that can be used as the server of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device can be a computer device of different types, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present disclosure described herein and/or required.

As shown in FIG. 6 , the electronic device 600 may include at least one processor 610 , a working memory 620 , an input unit 640 , a display unit 650 , a speaker 660 , a storage unit 670 , a communication unit 680 , and other output units 690 that can communicate with each other through a system bus 630 .

The processor 610 may be a single processing unit or multiple processing units, all of which may include a single or multiple computing units or multiple cores. The processor 610 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and/or any device that manipulates signals based on operating instructions. The processor 610 may be configured to obtain and execute computer-readable instructions stored in a working memory 620, a storage unit 670, or other computer-readable media, such as program codes of an operating system 620a, program codes of an application 620b, and the like.

The working memory 620 and the storage unit 670 are examples of computer-readable storage media for storing instructions that are executed by the processor 610 to implement the various functions described above. The working memory 620 may include both volatile memory and non-volatile memory (e.g., RAM, ROM, etc.). In addition, the storage unit 670 may include a hard disk drive, a solid-state drive, a removable medium, including external and removable drives, memory cards, flash memory, a floppy disk, an optical disk (e.g., a CD, DVD), a storage array, a network attached storage, a storage area network, and the like. The working memory 620 and the storage unit 670 may all be collectively referred to herein as memory or computer-readable storage media, and may be a non-transitory medium capable of storing computer-readable, processor-executable program instructions as computer program code, a computer program code, and a computer program code. The computer program code may be executed by the processor 610 as a specific machine configured to implement the operations and functions described in the examples herein.

The input unit 660 may be any type of device capable of inputting information to the electronic device 600. The input unit 660 may receive input digital or character information and generate key signal inputs related to user settings and/or function control of the electronic device, and may include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone and/or a remote controller. The output unit may be any type of device capable of presenting information, and may include but is not limited to a display unit 650, a speaker 660, and other output units 690, which may include but are not limited to video/audio output terminals, vibrators and/or printers. The communication unit 680 allows the electronic device 600 to exchange information/data with other devices via a computer network such as the Internet and/or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and/or a chipset, such as a Bluetooth ^™ device, an 802.11 device, a Wi-Fi device, a WiMAX device, a cellular communication device and/or the like.

The application 620b in the working register 620 can be loaded to execute the various methods and steps thereof described above, such as steps S110-S150 in FIG. 1, steps S210-S240 in FIG. 2, and steps S442-S448 in FIG. 4. For example, in some embodiments, the various methods described above can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 670. In some embodiments, part or all of the computer program can be loaded and/or installed on the electronic device 600 via the storage unit 670 and/or the communication unit 680. When the computer program is loaded and executed by the processor 610, one or more steps of the methods 100 and 200 described above can be executed. Alternatively, in other embodiments, the processor 610 can be configured to execute the methods 100 and 200 in any other appropriate manner (for example, by means of firmware).

Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and/or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions specified in the flowchart and/or block diagram are implemented. /operations are performed. The program code may execute entirely on the machine, partly on the machine, partly on the machine as a stand-alone software package and partly on a remote machine or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document is not limited here.

Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. In addition, each step may be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that with the evolution of technology, many of the elements described herein may be replaced by equivalent elements that appear after the present disclosure.

Claims (18)

1. A video encoding method, comprising:

   Acquire a video segment to be encoded, where the video segment to be encoded includes one or more video frames;

   Calculating spatiotemporal feature information of the video segment to be encoded based on the one or more video frames;

   Determining a plurality of predicted bit rate factors of the to-be-encoded video segment under a plurality of preset video encoding standards based on the spatiotemporal feature information;

   Determining, based on a first predicted rate factor corresponding to a first preset video coding standard among the multiple preset video coding standards and a preset video coding score interval among the multiple predicted rate factors, multiple target rate factors of the video segment to be encoded under the multiple preset video coding standards, wherein, among the multiple preset video coding standards, the time for performing video encoding using the first preset video coding standard is the shortest; and

   The to-be-encoded video segments are respectively encoded according to the multiple target bit rate factors to obtain multiple target video segments.
2. The method according to claim 1, wherein the spatiotemporal feature information includes spatiotemporal complexity information, and wherein calculating the spatiotemporal feature information of the video segment to be encoded based on the one or more video frames comprises:

   Calculating grayscale symbiosis information and normalization information of the video segment to be encoded based on the one or more video frames as the spatiotemporal complexity information, wherein the grayscale symbiosis information is used to describe the spatial complexity of the video segment to be encoded, and the normalization information is used to describe the temporal complexity of the video segment to be encoded; and

   The grayscale symbiosis information and the normalization information are combined to generate the spatiotemporal complexity information of the video segment to be encoded.
3. The method according to claim 1 or 2, wherein the spatiotemporal feature information includes coding feature information, and wherein calculating the spatiotemporal feature information of the video segment to be encoded based on the one or more video frames comprises:

   preprocessing the one or more video frames to generate a new video frame sequence, wherein the new video frame sequence includes a set of video frame pairs; and

   For each video frame pair in the video frame pair set, intra-frame coding and inter-frame coding are performed respectively to obtain the coding feature information of the video segment to be encoded.
4. The method according to claim 3, wherein preprocessing the one or more video frames to generate a new video frame sequence comprises:

   Copying video frames other than the first frame and the last frame of the one or more video frames to generate one or more copied video frames; and

   The one or more video frames and the one or more duplicate video frames are ordered such that each of the one or more duplicate video frames is located after a corresponding video frame.
5. The method according to claim 3, wherein for each video frame pair in the set of video frame pairs, intra-frame coding and inter-frame coding are respectively performed to obtain the coding feature information of the video segment to be encoded, comprising:

   Performing intra-frame coding on a first video frame in each video frame pair in the set of video frame pairs to obtain intra-frame coding information;

   Performing inter-frame coding on each video frame pair in the set of video frame pairs to obtain inter-frame coding information; and

   The encoding feature information is obtained based on the intra-frame encoding information and the inter-frame encoding information.
6. The method according to any one of claims 3 to 5, wherein the coding feature information is based on one or more of the following items: the number of coding bits, the proportion of intra-frame prediction modes, and the amplitude distribution information of inter-frame motion vectors.
7. The method according to any one of claims 1 to 6, wherein determining a plurality of predicted bit rate factors of the to-be-encoded video segment under a plurality of preset video coding standards based on the spatiotemporal feature information comprises:

   The spatiotemporal domain feature information of the to-be-encoded video segment is input into a first rate factor prediction model, so as to determine the multiple predicted rate factors of the to-be-encoded video segment under the multiple preset video coding standards through the first rate factor prediction model.
8. The method according to claim 7, wherein, based on a first predicted bit rate factor corresponding to a first preset video coding standard among the multiple preset video coding standards and a preset video coding score interval among the multiple predicted bit rate factors, determining a plurality of target bit rate factors for the to-be-encoded video segment under the multiple preset video coding standards comprises:

   Performing a first encoding on the to-be-encoded video segment based on the first predicted bit rate factor to obtain a first pre-encoded video segment under the first preset video encoding standard;

   determining whether the video coding score of the first pre-encoded video segment is within the preset video coding score range; and

   In response to determining that the video coding score of the first pre-encoded video segment is within the preset video coding score range:

   The plurality of predicted rate factors are determined as the plurality of target rate factors.
9. The method according to claim 8, wherein, based on a first predicted bit rate factor corresponding to a first preset video coding standard among the multiple preset video coding standards and a preset video coding score interval among the multiple predicted bit rate factors, determining a plurality of target bit rate factors for the to-be-encoded video segment under the multiple preset video coding standards further comprises:

   In response to determining that the video coding score of the first pre-encoded video segment is not within the preset video coding score range:

   Based on the spatiotemporal feature information of the to-be-encoded video segment, the multiple predicted rate factors and the video coding score of the first pre-encoded video segment, the multiple predicted rate factors are updated to obtain the multiple target rate factors.
10. The method according to claim 9, wherein, based on the spatiotemporal feature information of the to-be-encoded video segment, the multiple predicted rate factors and the video coding score of the first pre-encoded video segment, updating the multiple predicted rate factors to obtain the multiple target rate factors comprises:

    The spatiotemporal domain feature information of the video segment to be encoded, the multiple predicted bit rate factors and the video coding score of the first pre-encoded video segment are input into the second bit rate factor prediction model to determine the multiple target bit rate factors of the video segment to be encoded under the multiple preset video coding standards through the second bit rate factor prediction model.
11. The method according to claim 10, wherein the first rate factor prediction model is trained by:

    For each preset video coding standard of the multiple preset video coding standards:

    Acquire sample spatiotemporal feature information of a sample video segment and a sample bit rate factor of the sample video segment under the preset video coding standard;

    Inputting the sample spatiotemporal feature information into the first rate factor prediction model to determine a sample prediction rate factor of the sample video clip under the preset video coding standard via the first rate factor prediction model; and

    Calculating a first model loss value based on the sample rate factor and the sample predicted rate factor; and

    The parameters of the first rate factor prediction model are adjusted based on the first model loss value under each preset video encoding standard until the first model training stop condition is reached.
12. The method according to claim 11, wherein the second rate factor prediction model is trained by:

    Inputting the sample spatiotemporal feature information, the sample prediction rate factor under each of the preset video coding standards, and a first sample video coding score corresponding to the sample prediction rate factor under the first preset video coding standard into the second rate factor prediction model, so as to determine a plurality of sample target rate factors of the sample video clips under the plurality of preset video coding standards through the second rate factor prediction model;

    For each of the plurality of preset video coding standards, calculating a second model loss value based on the sample rate factor and a sample target rate factor corresponding to the preset video coding standard; and

    The parameters of the second rate factor prediction model are adjusted based on the second model loss value under each preset video encoding standard until the second model training stop condition is reached.
13. The method according to claim 1, wherein obtaining the video segment to be encoded comprises:

    Get the video to be encoded;

    Performing scene detection on the video to be encoded;

    Dividing the to-be-encoded video into one or more sub-video segments based on the scene detection result; and

    Each video sub-segment in the one or more video segments is identified as a video segment to be encoded.
14. The method according to claim 13, further comprising:

    For each preset video coding standard among the plurality of preset video coding standards, obtaining all target video segments under the preset video coding standard; and

    Based on the order of the one or more sub-video segments in the video to be encoded, all the target video segments are combined to obtain a target video corresponding to the video to be encoded under the preset video encoding standard.
15. A video encoding device, comprising:

    An acquisition module is configured to acquire a video segment to be encoded, where the video segment to be encoded includes one or more video frames;

    A calculation module, configured to calculate the spatiotemporal feature information of the video segment to be encoded based on the one or more video frames;

    A prediction rate factor determination module is configured to determine a plurality of prediction rate factors of the to-be-encoded video segment under a plurality of preset video encoding standards based on the spatiotemporal feature information;

    a target rate factor determination module, configured to determine a plurality of target rate factors of the video segment to be encoded under the plurality of preset video coding standards based on a first predicted rate factor corresponding to a first preset video coding standard among the plurality of preset video coding standards among the plurality of predicted rate factors and a preset video coding score interval, wherein, among the plurality of preset video coding standards, the time for performing video encoding using the first preset video coding standard is the shortest; and

    The encoding module is configured to encode the to-be-encoded video segments respectively according to the multiple target bit rate factors to obtain multiple target video segments.
16. An electronic device, comprising:

    at least one processor; and

    at least one memory communicatively coupled to the at least one processor,

    The at least one memory stores a computer program, and when the computer program is executed by the at least one processor, the method according to any one of claims 1 to 14 is implemented.
17. A non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 14.
18. A computer program product comprises a computer program, wherein the computer program implements the method according to any one of claims 1 to 14 when executed by a processor.