CN112929667B - Encoding and decoding method, device, equipment and readable storage medium - Google Patents

Abstract

The invention provides a coding and decoding method, a device, equipment and a readable storage medium, wherein the coding method comprises the following steps: determining at least two consecutive image frames from a current group of pictures GOP in response to a received instantaneous decoding refresh IDR request; determining a first partial image region from the image regions of each of the at least two consecutive image frames, respectively, as intra-coded image regions of each of the at least two consecutive image frames; wherein, the positions of the respective intra-frame coding image areas in the corresponding image areas can jointly cover the whole image area of a single image frame; in encoding the at least two consecutive image frames, performing: and encoding the intra-frame encoded image area by adopting an intra-frame encoding mode. The scheme well solves the problem that the encoding and decoding scheme aiming at the IDR request in the prior art is easy to cause image quality degradation.

Description

A codec method, device, equipment and readable storage medium

technical field

The present invention relates to the field of communication technology, in particular to a codec method, device, equipment and readable storage medium.

Background technique

In the prior art, when there is a packet loss error in the network (that is, the video appears to be blurred), usually an IDR (Instantaneous Decoding Refresh, Instant Decoding Refresh) refresh request is sent from the receiving end to the sending end, and the receiving end is received at the sending end. After the IDR request sent by the terminal, the transmitting terminal will immediately encode a video sequence with the IDR video frame as the initial frame, and send the video sequence to the receiving terminal. In this way, after the IDR video frame is completely received and decoded by the receiving terminal, the blurred screen The phenomenon will disappear.

However, since in the prior art, the code rate is usually controlled and the video sequence is sent in the form of GOP (Group of Pictures), and the bandwidth occupied by the IDR frame is generally the bandwidth occupied by the P frame (forward predictive coding frame) In this way, if a new IDR frame is suddenly inserted, it may cause a sudden increase in the bit rate, causing network jitter, packet loss, congestion, and freeze. Not only cannot solve the problem of blurred screen, but also introduce network packet loss Problems such as sharp increase and sudden drop in image quality lead to further freezes in subsequent videos and more decoding failures.

It can be seen from the above that the existing codec scheme for the IDR request has problems such as easy degradation of image quality.

Contents of the invention

The purpose of the present invention is to provide a codec method, device, device and readable storage medium to solve the problem in the prior art that the codec scheme for IDR requests easily causes image quality degradation.

In order to solve the above technical problems, an embodiment of the present invention provides an encoding method, including:

In response to the received immediate decoding refresh IDR request, from the current group of pictures GOP, determine at least two consecutive image frames;

From the respective image regions of the at least two consecutive image frames, respectively determine the first partial image regions as the respective intra-coded image regions of the at least two consecutive image frames; wherein, the respective intra-frame The position of the coded image area in the corresponding image area can jointly cover the entire image area of a single said image frame;

When encoding the at least two consecutive image frames, perform: encoding the intra-frame encoded image area in an intra-frame encoding manner.

Optionally, the method also includes:

From the respective image areas of the specified image frames, respectively determine the second partial image areas as the respective interframe coded image areas of the specified image frames; the specified image frame is one of the at least two consecutive image frames Other image frames except the first image frame; the inter-coded image area of each specified image frame satisfies: the image area corresponding to the intra-coded image area of the previous image frame of the specified image frame in the same position;

When encoding the designated image frame, perform: encoding the inter-coded image area by adopting a forward predictive encoding manner.

Optionally, in response to the received immediate decoding refresh IDR request, determine at least two consecutive image frames from the current group of pictures GOP, including:

In response to the IDR request, determine an index value representing network quality;

The at least two consecutive image frames are determined from the current GOP according to the index value.

Optionally, in response to the IDR request, determine an index value representing network quality, including:

In response to the IDR request, determine whether the current image frame meets the condition for judging the restricted image frame;

If so, determine the index value.

Optionally, judging whether the current image frame meets the criteria for judging the restricted image frame includes:

According to the arrangement position of the current image frame in the current GOP, it is determined whether the current image frame to be encoded meets the determination condition for the limited image frame.

Optionally, according to the arrangement position of the current image frame in the current GOP, determining whether the current image frame to be encoded meets the determination condition for the restricted image frame includes:

judging whether the arrangement position of the current image frame in the current GOP is relatively front or relatively behind;

If so, then determine that the current image frame meets the condition for judging the restricted image frame;

If not, it is determined that the current image frame does not meet the restricted image determination condition.

Optionally, the index value includes a value of round-trip delay rtt;

According to the index value, determining the at least two consecutive image frames from the current GOP includes:

Determine the size of the target time window according to the rtt value and the rtt threshold;

According to the size of the target time window, determining a number of consecutive image frames matching the size of the target time window from the GOP as the at least two consecutive image frames.

Optionally, determining the first partial image area from the respective image areas of the at least two consecutive image frames as the respective intra-coded image areas of the at least two consecutive image frames includes:

According to the size of the target time window and the height value of the image frame, respectively determine a first partial image area from the respective image areas of the at least two consecutive image frames as the at least two consecutive images The frame's respective intra-coded picture area.

The embodiment of the present invention also provides a decoding method, including:

Send instant decoding refresh IDR request;

receiving an image frame obtained by encoding at least two consecutive image frames by the encoding end in response to the IDR request;

When encoding the at least two consecutive image frames, the encoding end performs: encoding the respective intra-coded image areas of the at least two consecutive image frames by using an intra-frame encoding method;

The respective intra-coded image areas of the at least two consecutive image frames are the first partial image areas of the respective image areas of the at least two consecutive image frames; the respective intra-frame coded image areas are in the corresponding The positions in the image area can collectively cover the entire image area of a single image frame.

The embodiment of the present invention also provides an encoding device, including:

The first determining module is configured to determine at least two consecutive image frames from the current group of pictures GOP in response to the received immediate decoding refresh IDR request;

The second determination module is configured to respectively determine the first partial image area from the respective image areas of the at least two consecutive image frames as the respective intra-coded image areas of the at least two consecutive image frames; wherein , the positions of the respective intra-coded image areas in the corresponding image areas can jointly cover the entire image area of a single image frame;

The first encoding module is configured to, when encoding the at least two consecutive image frames, perform: encoding the intra-frame encoded image area by using an intra-frame encoding manner.

Optionally, the device also includes:

The third determination module is used to determine the second partial image area respectively from the respective image areas of the specified image frame as the respective inter-frame coded image areas of the specified image frame; the specified image frame is the at least two other image frames except the first image frame in consecutive image frames; the inter-coded image area of each designated image frame satisfies: The image areas are at the same position in the corresponding image area;

The second encoding module is configured to, when encoding the specified image frame, perform: encoding the inter-frame encoded image area by adopting a forward predictive encoding manner.

Optionally, in response to the received immediate decoding refresh IDR request, determine at least two consecutive image frames from the current group of pictures GOP, including:

In response to the IDR request, determine an index value representing network quality;

The at least two consecutive image frames are determined from the current GOP according to the index value.

Optionally, in response to the IDR request, determine an index value representing network quality, including:

In response to the IDR request, determine whether the current image frame meets the condition for judging the restricted image frame;

If so, determine the index value.

Optionally, judging whether the current image frame meets the criteria for judging the restricted image frame includes:

According to the arrangement position of the current image frame in the current GOP, it is determined whether the current image frame to be encoded meets the determination condition for the limited image frame.

Optionally, according to the arrangement position of the current image frame in the current GOP, determining whether the current image frame to be encoded meets the determination condition for the restricted image frame includes:

judging whether the arrangement position of the current image frame in the current GOP is relatively front or relatively behind;

If so, then determine that the current image frame meets the condition for judging the restricted image frame;

If not, it is determined that the current image frame does not meet the restricted image determination condition.

Optionally, the index value includes a value of round-trip delay rtt;

According to the index value, determining the at least two consecutive image frames from the current GOP includes:

Determine the size of the target time window according to the rtt value and the rtt threshold;

According to the size of the target time window, determining a number of consecutive image frames matching the size of the target time window from the GOP as the at least two consecutive image frames.

Optionally, determining the first partial image area from the respective image areas of the at least two consecutive image frames as the respective intra-coded image areas of the at least two consecutive image frames includes:

According to the size of the target time window and the height value of the image frame, respectively determine a first partial image area from the respective image areas of the at least two consecutive image frames as the at least two consecutive images The frame's respective intra-coded picture area.

The embodiment of the present invention also provides a decoding device, including:

The first sending module is used to send an instant decoding refresh IDR request;

A first receiving module, configured to receive an image frame obtained by encoding at least two consecutive image frames by the encoding end in response to the IDR request;

When encoding the at least two consecutive image frames, the encoding end performs: encoding the respective intra-coded image areas of the at least two consecutive image frames by using an intra-frame encoding method;

The respective intra-coded image areas of the at least two consecutive image frames are the first partial image areas of the respective image areas of the at least two consecutive image frames; the respective intra-frame coded image areas are in the corresponding The positions in the image area can collectively cover the entire image area of a single image frame.

An embodiment of the present invention also provides an encoding device, including a memory, a processor, and a program stored in the memory and operable on the processor; when the processor executes the program, the above encoding method is realized.

An embodiment of the present invention also provides a decoding device, including a memory, a processor, and a program stored in the memory and operable on the processor; when the processor executes the program, the above decoding method is realized.

The embodiment of the present invention also provides a readable storage medium, on which a program is stored, and when the program is executed by a processor, the steps in the above-mentioned encoding method are realized; or, when the program is executed by a processor, the steps in the above-mentioned decoding method are realized step.

The beneficial effects of above-mentioned technical scheme of the present invention are as follows:

In the above solution, the encoding method determines at least two consecutive image frames from the current group of pictures GOP in response to the received immediate decoding refresh IDR request; from the respective image areas of the at least two consecutive image frames In, the first partial image area is respectively determined as the respective intra-coded image areas of the at least two consecutive image frames; wherein, the positions of the respective intra-frame coded image areas in the corresponding image areas can be collectively covering the entire image area of a single image frame; when encoding the at least two consecutive image frames, perform: encoding the intra-frame encoded image area in an intra-frame encoding manner. Through the solution of the embodiment of the present application, after receiving the IDR request, it is possible to perform intra-frame encoding on the partial image areas of multiple image frames to achieve the effect of a complete IDR frame, instead of directly encoding at least one image frame into a complete IDR frame. IDR frames are transmitted. In this way, an IDR frame that occupies a large bandwidth is divided into multiple local intra-frame coding frames that occupy a small bandwidth, which can reduce the bandwidth occupied by image frames, thereby avoiding more packet loss and network jitter. As well as the problem of image quality degradation, it well solves the problem that the codec scheme for IDR request in the prior art easily causes image quality degradation.

Description of drawings

FIG. 1 is a schematic flow chart of an encoding method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of coverage of the entire image area in an embodiment of the present invention;

FIG. 3 is a schematic flowchart of a decoding method according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of an implementation framework of an encoding method according to an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of an encoding device according to an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of a decoding device according to an embodiment of the present invention.

Detailed ways

In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will describe in detail with reference to the drawings and specific embodiments.

The present invention aims at the problem that the encoding and decoding schemes for IDR requests in the existing technology can easily cause image quality degradation, and provides an encoding method, as shown in Figure 1, including:

Step 11: In response to the received immediate decoding refresh IDR request, determine at least two consecutive image frames from the current group of pictures GOP.

Wherein, the current GOP refers to the GOP to which the first image frame to be encoded belongs when the IDR request is received. The determined image frames in "at least two consecutive image frames" may include frames before the first image frame, the first image frame and/or frames after the first image frame, which are not limited herein.

Step 12: From the respective image regions of the at least two consecutive image frames, respectively determine the first partial image regions as the respective intra-coded image regions of the at least two consecutive image frames; wherein the respective The positions of the intra-frame coded image areas in the corresponding image area can collectively cover the entire image area of a single image frame.

Wherein "covering the entire image area of a single image frame" can be understood as: corresponding to the entire image area of an image frame; the entire image area of different image frames can be the same; the common coverage can be that the image area overlaps or does not There is overlap and common coverage. Taking the case of non-overlapping and common coverage as an example, assuming that two consecutive frames of A frame and B frame are selected, as shown in Figure 2, the intra-frame coded image area a of A frame and the intra-frame coded image area b of B frame The entire image area c of an image frame can be covered together.

Step 13: When encoding the at least two consecutive image frames, perform: encoding the intra-frame encoded image area by using an intra-frame encoding manner.

In this step, the intra-frame coding part of the IDR frame fed back after the IDR request is split into the intra-frame coding parts of multiple consecutive frames and encoded separately, so that an IDR frame that occupies a large bandwidth is divided into multiple frames that occupy a large bandwidth. Partial intra-frame coding with small bandwidth reduces bandwidth requirements and avoids packet loss and stuttering.

From the above, the encoding method provided by the embodiment of the present invention determines at least two consecutive image frames from the current group of pictures GOP in response to the received immediate decoding refresh IDR request; from the at least two consecutive image frames In the respective image areas of the frames, the first partial image areas are respectively determined as the respective intra-coded image areas of the at least two consecutive image frames; wherein, the respective intra-frame coded image areas are in the corresponding image areas The location can jointly cover the entire image area of a single image frame; when encoding the at least two consecutive image frames, perform: encoding the intra-frame encoded image area by using an intra-frame encoding method . Through the solution of the embodiment of the present application, after receiving the IDR request, it is possible to perform intra-frame encoding on the partial image areas of multiple image frames to achieve the effect of a complete IDR frame, instead of directly encoding at least one image frame into a complete IDR frame. IDR frames are transmitted. In this way, an IDR frame that occupies a large bandwidth is divided into multiple local intra-frame coded frames that occupy a small bandwidth, which can reduce the bandwidth occupied by image frames, thereby avoiding more packet loss, network jitter, and Problems such as image quality degradation; well solved the problem in the prior art that the encoding and decoding schemes for IDR requests easily cause image quality degradation.

Further, the method further includes: respectively determining the second partial image area from the respective image areas of the specified image frame as the respective inter-frame coded image areas of the specified image frame; the specified image frame is the Image frames other than the first image frame in at least two consecutive image frames; the inter-coded image area of each specified image frame satisfies: The positions of the intra-coded image areas in the corresponding image areas are the same; when encoding the specified image frame, it is performed: encoding the inter-coded image area by using a forward predictive encoding method.

In this way, starting from the first frame of at least two consecutive image frames determined above (ie, the first frame of restricted coding), there is an intra-frame coding frame that can be referred forward, and thus, starting from the second frame, it is possible to use local Forward predictive coding is used for coding. In the above process, the image area of the inter-frame encoding generally corresponds to the image area of the previous frame it refers to, but it is not limited thereto.

In the implementation of the present invention, further, the encoding of the inter-frame coded image area may refer to the first partial image area and the second partial image area in the previous image frame.

Specifically, in the embodiment of the present invention, the above-mentioned operation of determining the first partial image area and the second partial image area may include:

Determine the restricted area (including the first partial image area and the second partial image area) of each image frame in the above-mentioned at least two consecutive image frames; wherein, the N+1th frame has one more restricted area than the Nth frame, N is an integer greater than or equal to 0;

Encoding the last restricted area in the restricted area corresponding to each image frame (ie, the above-mentioned first partial image area) by means of intra-frame coding, and when there are remaining restricted areas (ie, the above-mentioned second partial image area) In the case of an image area), the remaining restricted area is encoded by means of inter-frame coding, wherein the inter-frame coding refers to all restricted areas in the previous image frame.

Wherein, the number of restricted areas of the first image frame (ie the above-mentioned current image frame) in at least two consecutive image frames can be 1, and the number of restricted areas of the next image frame of the first image frame can be 2, and so on, the quantity of the restricted area of the subsequent image frame can be increased sequentially; about the "previous image frame" refers to the previous image frame (the image frame corresponding to the remaining restricted area) one frame.

In this way, this scheme can realize block refresh (intra-frame coding), and this coding method can ensure that the coded data can be displayed normally; in addition, for the intra-frame coding method, data refresh can be guaranteed to ensure subsequent normal display; Encoding can ensure that the reference is the refreshed data, so as to ensure subsequent normal display.

In the embodiment of the present invention, whether to use the method shown in FIG. 1 may be determined according to network quality. This is because, to a certain extent, the quality of the network determines whether the problems described in the background art exist in the traditional IDR encoding.

Based on the above considerations, in one embodiment, in response to the received immediate decoding refresh IDR request, determining at least two consecutive image frames from the current group of pictures GOP may include:

In response to the IDR request, determine an index value representing network quality;

The at least two consecutive image frames are determined from the current GOP according to the index value.

Specifically, before determining the index value, in response to the IDR request, it may be determined whether the current image frame meets the condition for determining a restricted image frame (described later). If so, determine the index value. Wherein, the index value may include the value of rtt (round-trip time, round trip delay).

In the embodiment of the present invention, when the network quality is relatively good, the value of rtt is relatively small, and a relatively small number of image frames requiring limited processing can be determined according to the value of rtt; in the case of relatively poor network quality Next, the value of rtt is relatively large, and a relatively large number of image frames that need to be processed can be determined according to the value of rtt; thereby ensuring that no matter how the network quality is, the frame refresh is as good as possible; that is, by the above, it can be Determine the image frames that need to be processed according to the network quality, so as to ensure better frame refresh and avoid problems such as stuttering.

Wherein, the current image frame is the first image frame mentioned above, and "determining the index value" can also be understood as obtaining the index value.

In the embodiment of the present invention, judging whether the current image frame meets the condition for judging the restricted image frame includes: determining whether the current image frame to be encoded meets the restricted image frame according to the arrangement position of the current image frame in the current GOP. Image frame judgment condition.

In the embodiment of the present invention, according to the arrangement position of the current image frame in the current GOP, determining whether the current image frame to be encoded meets the determination condition of the restricted image frame includes: determining whether the current image frame is in the current GOP Whether the arrangement position in the current GOP is relatively front or relatively backward; if so, then determine that the current image frame meets the condition for determining the restricted image frame; if not, then determine that the current image frame does not meet the restricted image frame Limited image judgment conditions.

About whether it is relatively front or relatively back refers to whether the position of the current image frame in the entire current GOP is relatively front or back. For example, if the current image frame is at the front in the entire current GOP, it may be considered as relatively front; or if the current image frame is at the end in the entire current GOP, it may be considered as relatively behind.

The arrangement position can be represented by the position ratio, which is to determine whether the above determination condition is met according to the position of the current image frame in the current GOP. The position ratio refers to the ratio between the position ranking value (namely number M) of the current image frame in the current GOP and (the duration T and frame frequency fps corresponding to the GOP).

Specifically, it may be determined whether the position ratio corresponding to the arrangement position is within a preset value range, and the preset value range is 0-0.2, or 0.8-1. That is, it is judged whether the arrangement position is in the top 20% or bottom 20% of the entire GOP. It is to perform special processing on the front frame or the back frame - this is because the front (or back) will be closer to the IDR frame of the previous (or next) picture group, which occupies a large bandwidth and will cause the card Frame, this special treatment can reduce the occupied bandwidth to a certain extent and avoid Frame. It can also be understood as: in the embodiment of the present invention, the reason why it is considered to take 20% before and after the GOP to carry out restricted coding is because if the frame that is too far ahead is used as IDR, the problem that occurs: an IDR has just been sent, and it is sent again. An IDR will take up too much bandwidth. Similarly, if a frame that is too late is used as an IDR, there will be a problem: the latter IDR will be affected. Therefore, it is necessary to select these two parts of frames for restricted coding.

In the embodiment of the present invention, the absolute value of the difference between the two ends of the preset value range multiplied by the total number of frames corresponding to the GOP is greater than or equal to 3 frames, but it is not limited thereto.

In the embodiment of the present invention, the index value includes the value of rtt (round-trip time, round-trip delay); according to the index value, determining the at least two continuous image frames from the current GOP includes: According to the value of the rtt and the rtt threshold value, determine the size of the target time window; according to the size of the target time window, determine from the GOP the number of consecutive image frames that match the size of the target time window, as The at least two consecutive image frames.

In this way, it can be controlled to achieve the effect of refreshing the I frame with the fewest frames. The rtt threshold is greater than or equal to 100ms and less than or equal to 300ms, specifically 200ms.

Specifically, when the rtt value is less than or equal to the rtt threshold, the first formula is used to determine the size of the target time window (specifically, it may be the corresponding frame number); when the rtt value is greater than the rtt threshold In the case of the rtt threshold, the second formula is used to determine the size of the target time window; wherein, the first formula is: W=min(3,max(500/rtt,5)); the second The formula is: W=max(3,500/rtt); W represents the size of the target time window, and rtt represents the value of the rtt. The first formula means to take the maximum value of 500/rtt and 5, and then take the maximum value and the minimum value of 3; the second formula means to take the maximum value of 3 and 500/rtt.

In this way, if rtt≤rtt threshold, W is equal to 3; if rtt>rtt threshold, W is at least equal to 3, and when 500/rtt is greater than 3, W is decreasing as rtt increases; in addition, W The smaller the value, the larger the restricted area, and the larger the W, the smaller the restricted area; therefore, to a certain extent, rtt is inversely proportional to W, and rtt is proportional to the restricted area, so that when the network transmission delay is large, Try to use the least number of frames to achieve the effect of I frame refresh.

In the embodiment of the present invention, determining the first partial image area respectively from the respective image areas of the at least two consecutive image frames as the respective intra-coded image areas of the at least two consecutive image frames includes: According to the size of the target time window and the height value of the image frame, respectively determine a first partial image area from the respective image areas of the at least two consecutive image frames as the at least two consecutive images The frame's respective intra-coded picture area.

In this way, the intra-coded image area of each image frame can be accurately determined, and finally the entire image area of a single image frame can be completely covered.

Specifically, the height of the first partial image area may be Height/W lines, the Height represents the height value of the image frame, and W represents the size of the target time window; the N+1 involved above is less than or equal to W . The height of the second partial image area may also be Height/W lines. The width of the first and second partial image areas may specifically be the width of an image frame.

In this way, it can be ensured that there is no overlap between the partial image areas, thereby minimizing the intra-frame coding area to reduce the occupied width.

The embodiment of the present invention also provides a decoding method, as shown in Figure 3, including:

Step 31: Send an instant decoding refresh IDR request;

Step 32: Receive the image frame obtained by encoding at least two consecutive image frames by the encoding end in response to the IDR request; when the encoding end encodes the at least two continuous image frames, perform: using An intra-frame encoding method, encoding the respective intra-frame coded image regions of the at least two consecutive image frames; the respective intra-frame coded image regions of the at least two consecutive image frames are the at least two consecutive The first partial image area of each image area of the image frame; the positions of the respective intra-coded image areas in the corresponding image area can jointly cover the entire image area of a single image frame.

The encoding end is the encoding device.

The decoding method provided by the embodiment of the present invention sends an instant decoding refresh IDR request; receives an image frame obtained by encoding at least two consecutive image frames in response to the IDR request at the encoding end; When encoding the at least two consecutive image frames, perform: encoding the respective intra-coded image areas of the at least two consecutive image frames by using an intra-frame coding method; each of the at least two consecutive image frames The intra-frame coded image area is the first partial image area of the respective image areas of the at least two consecutive image frames; the positions of the respective intra-frame coded image areas in the corresponding image area can jointly cover The entire image area of a single said image frame. Through the solution of the embodiment of the present application, it is possible to support the implementation of intra-frame encoding for partial image areas of multiple image frames after receiving the IDR request at the encoding end to achieve the effect of a complete IDR frame, instead of directly dividing at least one image frame into Encoded into a complete IDR frame for transmission, so that an IDR frame that occupies a large bandwidth is divided into multiple partial intra-frame coding frames that occupy a small bandwidth, which can reduce the bandwidth occupied by the image frame, thereby avoiding more packet loss , network jitter, image quality degradation, and other problems; it solves the problem that the encoding and decoding scheme for IDR requests in the prior art easily causes image quality degradation.

The encoding method and decoding method provided by the embodiments of the present invention are further described below. The following IDR requests are sent by a decoding end as an example, the encoding end is an example of a sending end, and the decoding end is an example of a receiving end.

In view of the above technical problems, the embodiment of the present invention provides an encoding method and a decoding method, which can be implemented as a fast refresh method of real-time audio and video. The end requests a fast IDR frame refresh (specifically, sends an IDR request to the sender). The sender chooses one of the two methods according to the preset conditions: one is to encode a complete IDR frame; the other is to send a restricted P frame or B frame according to the network situation; in order to avoid the bandwidth occupied by huge IDR frames, Causes more packet loss, network jitter, and image quality degradation.

First of all, the specific examples of the content involved in the embodiments of the present invention are introduced as follows:

(1) The sending end A initiates an audio and video call to the receiving end B.

(2) Receiver B accepts the invitation from sender A and establishes a communication link.

(3) The camera at the sending end A collects YUV data (color-coded data), encodes it into an ES stream (Elementary Stream, basic stream) according to the video coding standard, and encapsulates it into an RTP (Real-time Transport Protocol, real-time transport protocol) package The video data is sent to the receiving end B through RTP-RTCP (Real-time Transport Control Protocol), and the audio data is processed in the same way.

(4) After the receiver B receives the audio and video stream from the transmitter A, it decodes and renders it. At the same time, the sending end of the device belonging to the receiving end B also collects YUV data and microphone data, etc., encodes, compresses, encapsulates, and transmits the stream to the receiving end of the device belonging to the sending end A.

(5) When the network is normal, the above (3) and (4) will always be executed. However, suppose that in a mobile environment, the receiver B network suddenly goes bad. Receiver B found a large number of packet loss, bit errors, and out-of-sequence frames, which caused receiver B to fail to decode and appear blurred screens and freezes. This can also be understood as a situation of poor network quality. Receiver B sends an IDR request to sender A.

(6) The sending end A processes the IDR request of the receiving end B, and can immediately encode the current frame as an IDR frame, and send it to the receiving end B; also refer to the processing framework shown in Figure 4 (the A21 coding shown in Figure 4 refers to A11, A31 and A32 encoding refer to A21 and A22, N in the figure is an integer greater than 0), in the case of a weak network (that is, low network quality), the receiving end has blurred screen, freeze, packet loss, decoding failure, etc., need The sender refreshes quickly to adapt to weak network conditions. This solution provides a stable and effective refresh mechanism, which can quickly solve blurred screens and freezes without causing network jitter and more packet loss, which can greatly improve user experience. The details can be as follows:

(1) The sender A processes the IDR request of the receiver B.

In this step, the value of the current network rtt packet (that is, the value of the above-mentioned rtt) can be obtained through the server. Alternatively, the value of rtt may be determined after the following operation (3), that is, when it is judged that the current image frame meets the determination condition of the restricted image frame.

(2) Start the network bandwidth estimation module and the restricted reference frame module.

(3) The network bandwidth estimation module determines whether the current frame enters the receiving frame according to the position of the current frame (the frame to be encoded and sent at present, that is, the above-mentioned current image frame) in the entire GOP (that is, the above-mentioned arrangement position) and preset conditions. Only reference frame modules. Among them, the preset condition is: the position ratio corresponding to the arrangement position=M/(T×fps)≤0.2, or the position ratio=M/(T×fps)≥0.8; wherein, T represents the duration of the GOP (that is, the above-mentioned The corresponding duration of the GOP), such as T seconds; M represents the numbering of the current image frame in the GOP (that is, the numbering of the above-mentioned current image frame in the GOP); fps represents the frame rate; if calculated according to the above formula If the result is less than or equal to 0.2, or greater than or equal to 0.8, enter the restricted reference frame module. Otherwise, the current frame can be immediately encoded as an IDR frame and sent to the receiving end B.

In this way, special processing can be performed on the front frame or the back frame; because the front (or back) will be closer to the larger IDR frame of the previous (or next) picture group, which occupies a large bandwidth and will cause Caton, this special treatment can reduce the occupied bandwidth to a certain extent, and avoid Caton.

In the embodiment of the present invention, it may be: 0.2×T×fps≧3. The GOP involved is customized by the encoder, containing 30 frames or 100 frames, etc.

(4) Restricted reference frame module: determine (restricted) the sliding window size W of the reference frame (ie the size of the target time window) according to the value of rtt (ie the value of the above-mentioned rtt packet). If rtt≤Tm, the sliding window size is W=min(3,max(500/rtt,5)). Wherein Tm is a preset threshold (that is, the above-mentioned rtt threshold, which may be greater than or equal to 100ms and less than or equal to 300ms), indicating the delay time, which is generally 200ms. If rtt>Tm, then the sliding window size is W=max(3,500/rtt). In this way, when the network transmission delay is large, the minimum number of frames can be used to achieve the effect of I frame refresh:

Therefore, if rtt≤Tm, then W is equal to 3; if rtt>Tm, then W is at least equal to 3, and when 500/rtt is greater than 3, W is decreasing as rtt increases; A smaller value means a larger restricted area, and a larger W means a smaller restricted area. Therefore, to a certain extent, the value of rtt is inversely proportional to W, and the value of rtt is directly proportional to the restricted area, so that when the network transmission delay is large, the effect of I frame refresh can be realized with the fewest frames as possible.

(5) Calculate the reference frame (P frame) restricted area of each frame in the sliding window (a specific implementation of the restricted area of the above-mentioned image frame), and the image height of the restricted area is (Height/W) row. It can also be understood that in this solution, the size of the restricted area of the reference frame is determined according to the value of rtt.

(6) From the current 0th frame (i.e. the above-mentioned current image frame) to the W-1th frame, continue to be encoded as a P or B frame (bidirectional predictive interpolation encoding frame), the following involves special encoding of the restricted area:

1) There is only one restricted area in the 0th frame, specifically the area whose height is from 0 to (Height/W) lines and whose width is the width of the image frame; the encoding method of this area in the current frame is intra-frame encoding. Other areas are normally coded (normally coded according to coding requirements). For example, the 0th frame is originally a B frame or a P frame, and the area outside the restricted area is coded as a B frame or a P frame according to the original principle; among them, the P frame belongs to the inter-frame coding of forward prediction, and only refers to the front closest frame. Its I frame or P frame; B frame is a bidirectional predictive coding frame, and inter-frame coding is performed with the previous I or P frame and the following P frame as reference frames.

2) There are two restricted areas in the first frame, specifically two areas with heights from 0 to (Height/W) lines and from (Height/W) to 2×(Height/W) lines. A region whose width is the width of the image frame. The first restricted area of the current frame adopts inter-frame coding, and refers to the first restricted area of the 0th frame (inter-frame coding needs to refer to the previous frame, and here the restricted area referred to in the previous frame is limited). The second restricted area of the current frame is intra-coded. Other regions are coded normally.

3) There are 3 restricted areas in the second frame, specifically, the height is from 0 to (Height/W) lines, from (Height/W) to 2×(Height/W) lines, and from 2×(Height/W) lines Three areas to 3×(Height/W) lines, the width of these three areas is the area of the image frame width. The first and second restricted areas of the current frame adopt inter-frame coding, referring to the first and second restricted areas of the first frame. The third restricted area of the current frame is intra-coded. Other regions are coded normally.

By analogy, the restricted area and encoding of the subsequent frame 3 to frame W-2 can be similarly processed by referring to the above content;

4) There are W restricted areas in frame W-1, specifically, the height is from 0 to (Height/W) rows, from (Height/W) to 2×(Height/W) rows, from 2×(Height/W) ) line to 3×(Height/W) line, from (W-1)×(Height/W) line to W×(Height/W) line, the width of these W areas is the area of the image frame width . The first, second ... to W-1 restricted areas of the current frame adopt inter-frame coding, and refer to the first, second ... to W-1 restricted areas of the W-2th frame. The W th restricted area of the current frame adopts intra-frame coding. Other regions are coded normally.

In this way, block refresh (intra-frame encoding) can be realized. This encoding method can ensure that the encoded data can be displayed normally; for intra-frame encoding, it can ensure data refresh and subsequent normal display; while inter-frame encoding can ensure reference What is displayed is the refreshed data, so as to ensure the subsequent normal display.

5) Frames beyond the sliding window are coded normally.

From the above, the solution provided by the embodiment of the present invention can realize: under the weak network, fast refresh can be realized without causing network jitter; the restricted reference frame only performs intra-frame encoding on a specific restricted area, which can ensure the weak network. Image quality is better.

The embodiment of the present invention also provides an encoding device, as shown in Figure 5, including:

The first determining module 51 is configured to determine at least two consecutive image frames from the current group of pictures GOP in response to the received immediate decoding refresh IDR request;

The second determination module 52 is configured to respectively determine the first partial image area from the respective image areas of the at least two consecutive image frames as the respective intra-coded image areas of the at least two consecutive image frames; Wherein, the positions of the respective intra-coded image areas in the corresponding image areas can jointly cover the entire image area of a single image frame;

The first encoding module 53 is configured to, when encoding the at least two consecutive image frames, perform: encoding the intra-frame encoded image area by using an intra-frame encoding manner.

The encoding device provided in the embodiment of the present invention determines at least two consecutive image frames from the current group of pictures GOP in response to the received immediate decoding refresh IDR request; from each of the at least two consecutive image frames In the image area, the first partial image area is respectively determined as the respective intra-frame coded image areas of the at least two consecutive image frames; wherein, the positions of the respective intra-frame coded image areas in the corresponding image areas , which can jointly cover the entire image area of a single image frame; when encoding the at least two consecutive image frames, perform: encoding the intra-frame encoded image area in an intra-frame encoding manner. Through the solution of the embodiment of the present application, after receiving the IDR request, it is possible to perform intra-frame encoding on the partial image areas of multiple image frames to achieve the effect of a complete IDR frame, instead of directly encoding at least one image frame into a complete IDR frame. IDR frames are transmitted. In this way, an IDR frame that occupies a large bandwidth is divided into multiple local intra-frame coding frames that occupy a small bandwidth, which can reduce the bandwidth occupied by image frames, thereby avoiding more packet loss and network jitter. As well as the problem of image quality degradation, it well solves the problem that the codec scheme for IDR request in the prior art easily causes image quality degradation.

Further, the device further includes: a third determining module, configured to respectively determine the second partial image area from the respective image areas of the specified image frame as the respective inter-frame coded image areas of the specified image frame; The designated image frame is an image frame other than the first image frame in the at least two consecutive image frames; the interframe coded image area of each designated image frame satisfies: the same as the previous one of the designated image frame The intra-coded image areas of the frame image frame are at the same position in the corresponding image area; the second encoding module is used to perform when encoding the specified image frame: adopting a forward predictive encoding method to The inter-coded picture regions are coded.

In the embodiment of the present invention, in response to the received immediate decoding refresh IDR request, determining at least two consecutive image frames from the current group of pictures GOP includes: determining an index value representing network quality in response to the IDR request; The at least two consecutive image frames are determined from the current GOP according to the index value.

Wherein, in response to the IDR request, determining an index value representing network quality includes: in response to the IDR request, judging whether the current image frame meets a condition for judging a restricted image frame; if so, determining the index value.

In the embodiment of the present invention, judging whether the current image frame meets the condition for judging the restricted image frame includes: determining whether the current image frame to be encoded meets the restricted image frame according to the arrangement position of the current image frame in the current GOP. Image frame judgment condition.

Wherein, according to the arrangement position of the current image frame in the current GOP, determining whether the current image frame to be encoded meets the determination condition of the restricted image frame includes: determining that the current image frame is in the current GOP Whether the arrangement position is relatively front or relatively back; if so, then determine that the current image frame meets the limited image frame determination condition; if not, then determine that the current image frame does not meet the limited image determination condition .

In the embodiment of the present invention, the index value includes a value of round-trip delay rtt; according to the index value, determining the at least two consecutive image frames from the current GOP includes: according to the value of rtt and The rtt threshold is used to determine the size of the target time window; according to the size of the target time window, determine from the GOP a number of consecutive image frames that match the size of the target time window as the at least two consecutive image frame.

In the embodiment of the present invention, determining the first partial image area respectively from the respective image areas of the at least two consecutive image frames as the respective intra-coded image areas of the at least two consecutive image frames includes: According to the size of the target time window and the height value of the image frame, respectively determine a first partial image area from the respective image areas of the at least two consecutive image frames as the at least two consecutive images The frame's respective intra-coded picture area.

Wherein, the implementation embodiments of the above encoding method are all applicable to the embodiment of the encoding device, and can also achieve the same technical effect.

The embodiment of the present invention also provides a decoding device, as shown in Figure 6, including:

The first sending module 61 is configured to send an instant decoding refresh IDR request;

The first receiving module 62 is configured to receive an image frame obtained by encoding at least two consecutive image frames by the encoding end in response to the IDR request;

When encoding the at least two consecutive image frames, the encoding end performs: encoding the respective intra-coded image areas of the at least two consecutive image frames by using an intra-frame encoding method;

The respective intra-coded image areas of the at least two consecutive image frames are the first partial image areas of the respective image areas of the at least two consecutive image frames; the respective intra-frame coded image areas are in the corresponding The positions in the image area can collectively cover the entire image area of a single image frame.

The decoding device provided in the embodiment of the present invention sends an immediate decoding refresh IDR request; receives an image frame obtained by encoding at least two consecutive image frames in response to the IDR request at the encoding end; When encoding the at least two consecutive image frames, perform: encoding the respective intra-coded image areas of the at least two consecutive image frames by using an intra-frame coding method; each of the at least two consecutive image frames The intra-frame coded image area is the first partial image area of the respective image areas of the at least two consecutive image frames; the positions of the respective intra-frame coded image areas in the corresponding image area can jointly cover The entire image area of a single said image frame. Through the solution of the embodiment of the present application, it is possible to support the implementation of intra-frame encoding for partial image areas of multiple image frames after receiving the IDR request at the encoding end to achieve the effect of a complete IDR frame, instead of directly dividing at least one image frame into Encoded into a complete IDR frame for transmission, so that an IDR frame that occupies a large bandwidth is divided into multiple partial intra-frame coding frames that occupy a small bandwidth, which can reduce the bandwidth occupied by the image frame, thereby avoiding more packet loss , network jitter, image quality degradation, and other problems; it solves the problem that the encoding and decoding scheme for IDR requests in the prior art easily causes image quality degradation.

Wherein, the implementation embodiments of the above decoding method are all applicable to the embodiment of the decoding device, and can also achieve the same technical effect.

An embodiment of the present invention also provides an encoding device, including a memory, a processor, and a program stored in the memory and operable on the processor; when the processor executes the program, the above encoding method is realized.

Wherein, the implementation embodiments of the above encoding method are all applicable to the embodiments of the encoding device, and can also achieve the same technical effect.

An embodiment of the present invention also provides a decoding device, including a memory, a processor, and a program stored in the memory and operable on the processor; when the processor executes the program, the above decoding method is implemented.

Wherein, the implementation embodiments of the above-mentioned decoding method are all applicable to the embodiment of the decoding device, and can also achieve the same technical effect.

The embodiment of the present invention also provides a readable storage medium, on which a program is stored, and when the program is executed by a processor, the steps in the above encoding method are implemented; or,

When the program is executed by the processor, the steps in the above decoding method are realized.

Wherein, the implementation embodiments of the above-mentioned encoding method or decoding method are all applicable to the embodiment of the readable storage medium, and can also achieve the corresponding same technical effect.

It should be noted that many functional components described in this specification are called modules, in order to more particularly emphasize the independence of their implementation.

In the embodiments of the present invention, the modules may be implemented by software so as to be executed by various types of processors. An identified module of executable code may, by way of example, comprise one or more physical or logical blocks of computer instructions which may, for example, be structured as an object, procedure, or function. Notwithstanding, the executable code of an identified module need not be physically located together, but may include distinct instructions stored in different bits which, when logically combined, constitute the module and implement the specified Purpose.

Indeed, a module of executable code may be a single instruction, or many instructions, and may even be distributed over several different code segments, among different programs and across multiple memory devices. Likewise, operational data may be identified within modules, and may be implemented in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed in different locations (including on different storage devices), and may exist, at least in part, only as electronic signals on a system or network.

When the module can be realized by software, considering the level of the existing hardware technology, the module that can be realized by software, regardless of the cost, those skilled in the art can build the corresponding hardware circuit to realize the corresponding function. The hardware circuit includes conventional very large scale integration (VLSI) circuits or gate arrays as well as existing semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, and the like.

What has been described above is a preferred embodiment of the present invention. It should be pointed out that for those skilled in the art, some improvements and modifications can also be made without departing from the principle of the present invention. These improvements and modifications should also be considered as Be the protection scope of the present invention.

Claims (13)

1. A coding method, characterized in that, comprising: In response to the received immediate decoding refresh IDR request, from the current group of pictures GOP, determine at least two consecutive image frames; From the respective image regions of the at least two consecutive image frames, respectively determine the first partial image regions as the respective intra-coded image regions of the at least two consecutive image frames; wherein, the respective intra-frame The position of the coded image area in the corresponding image area can jointly cover the entire image area of a single said image frame; When encoding the at least two consecutive image frames, perform: encoding the intra-frame encoded image area by using an intra-frame encoding method; Wherein, in response to the received immediate decoding refresh IDR request, from the current group of pictures GOP, determine at least two consecutive image frames, including: In response to the IDR request, determine whether the current image frame meets the condition for judging the restricted image frame; if so, determine the index value representing the network quality; determining the at least two consecutive image frames from the current GOP according to the index value; Wherein, judging whether the current image frame meets the condition for judging the restricted image frame includes: judging whether the arrangement position of the current image frame in the current GOP is relatively front or relatively behind; If so, then determine that the current image frame meets the condition for judging the restricted image frame; If not, it is determined that the current image frame does not meet the condition for judging the limited image frame. 2. The method according to claim 1, characterized in that the method further comprises: From the respective image areas of the specified image frames, respectively determine the second partial image areas as the respective interframe coded image areas of the specified image frames; the specified image frame is one of the at least two consecutive image frames Other image frames except the first image frame; the inter-coded image area of each specified image frame satisfies: the image area corresponding to the intra-coded image area of the previous image frame of the specified image frame in the same position; When encoding the designated image frame, perform: encoding the inter-coded image area by adopting a forward predictive encoding manner. 3. The method according to claim 1, wherein the index value comprises the value of round-trip delay rtt; According to the index value, determining the at least two consecutive image frames from the current GOP includes: Determine the size of the target time window according to the rtt value and the rtt threshold; According to the size of the target time window, determining a number of consecutive image frames matching the size of the target time window from the GOP as the at least two consecutive image frames. 4. The method according to claim 3, characterized in that, from the respective image regions of the at least two consecutive image frames, the first partial image regions are respectively determined as the respective image regions of the at least two consecutive image frames Intra-coded image regions, including: According to the size of the target time window and the height value of the image frame, respectively determine a first partial image area from the respective image areas of the at least two consecutive image frames as the at least two consecutive images The frame's respective intra-coded picture area. 5. A decoding method, characterized in that, comprising: Send instant decoding refresh IDR request; receiving an image frame obtained by encoding at least two consecutive image frames by the encoding end in response to the IDR request; When encoding the at least two consecutive image frames, the encoding end performs: encoding the respective intra-coded image areas of the at least two consecutive image frames by using an intra-frame encoding method; The respective intra-coded image areas of the at least two consecutive image frames are the first partial image areas of the respective image areas of the at least two consecutive image frames; the respective intra-frame coded image areas are in the corresponding The position in the image area can jointly cover the entire image area of a single said image frame; The at least two consecutive image frames are determined by the encoding end in the following manner: In response to the IDR request, determine whether the current image frame meets the condition for judging the restricted image frame; if so, determine the index value representing the network quality; determining the at least two consecutive image frames from the current GOP according to the index value; Wherein, judging whether the current image frame meets the condition for judging the restricted image frame includes: judging whether the arrangement position of the current image frame in the current GOP is relatively front or relatively behind; If so, then determine that the current image frame meets the condition for judging the restricted image frame; If not, it is determined that the current image frame does not meet the condition for judging the limited image frame. 6. An encoding device, characterized in that it comprises: The first determining module is configured to determine at least two consecutive image frames from the current group of pictures GOP in response to the received immediate decoding refresh IDR request; The second determination module is configured to respectively determine the first partial image area from the respective image areas of the at least two consecutive image frames as the respective intra-coded image areas of the at least two consecutive image frames; wherein , the positions of the respective intra-coded image areas in the corresponding image areas can jointly cover the entire image area of a single image frame; The first encoding module is configured to, when encoding the at least two consecutive image frames, perform: encoding the intra-frame encoded image area by using an intra-frame encoding method; Wherein, in response to the received immediate decoding refresh IDR request, from the current group of pictures GOP, determine at least two consecutive image frames, including: In response to the IDR request, determine whether the current image frame meets the condition for judging the restricted image frame; if so, determine the index value representing the network quality; determining the at least two consecutive image frames from the current GOP according to the index value; Wherein, judging whether the current image frame meets the condition for judging the restricted image frame includes: judging whether the arrangement position of the current image frame in the current GOP is relatively front or relatively behind; If so, then determine that the current image frame meets the condition for judging the restricted image frame; If not, it is determined that the current image frame does not meet the condition for judging the limited image frame. 7. The device according to claim 6, further comprising: The third determination module is used to determine the second partial image area respectively from the respective image areas of the specified image frame as the respective inter-frame coded image areas of the specified image frame; the specified image frame is the at least two other image frames except the first image frame in consecutive image frames; the inter-coded image area of each designated image frame satisfies: The image areas are at the same position in the corresponding image area; The second encoding module is configured to, when encoding the specified image frame, perform: encoding the inter-frame encoded image area by adopting a forward predictive encoding manner. 8. The device according to claim 6, wherein the index value comprises a value of round-trip delay rtt; According to the index value, determining the at least two consecutive image frames from the current GOP includes: Determine the size of the target time window according to the rtt value and the rtt threshold; According to the size of the target time window, determining a number of consecutive image frames matching the size of the target time window from the GOP as the at least two consecutive image frames. 9. The device according to claim 8, characterized in that, from the respective image regions of the at least two consecutive image frames, the first partial image regions are respectively determined as the respective image regions of the at least two consecutive image frames Intra-coded image regions, including: According to the size of the target time window and the height value of the image frame, respectively determine a first partial image area from the respective image areas of the at least two consecutive image frames as the at least two consecutive images The frame's respective intra-coded picture area. 10. A decoding device, characterized in that, comprising: The first sending module is used to send an instant decoding refresh IDR request; A first receiving module, configured to receive an image frame obtained by encoding at least two consecutive image frames by the encoding end in response to the IDR request; When encoding the at least two consecutive image frames, the encoding end performs: encoding the respective intra-coded image areas of the at least two consecutive image frames by using an intra-frame encoding method; The respective intra-coded image areas of the at least two consecutive image frames are the first partial image areas of the respective image areas of the at least two consecutive image frames; the respective intra-frame coded image areas are in the corresponding The position in the image area can jointly cover the entire image area of a single said image frame; The at least two consecutive image frames are determined by the encoding end in the following manner: In response to the IDR request, determine whether the current image frame meets the condition for judging the restricted image frame; if so, determine the index value representing the network quality; determining the at least two consecutive image frames from the current GOP according to the index value; Wherein, judging whether the current image frame meets the condition for judging the restricted image frame includes: judging whether the arrangement position of the current image frame in the current GOP is relatively front or relatively behind; If so, then determine that the current image frame meets the condition for judging the restricted image frame; If not, it is determined that the current image frame does not meet the condition for judging the limited image frame. 11. A coding device, comprising a memory, a processor, and a program stored on the memory and operable on the processor; it is characterized in that, when the processor executes the program, the program according to claims 1 to 1 is realized. The encoding method described in any one of 4. 12. A decoding device, comprising a memory, a processor, and a program stored on the memory and operable on the processor; it is characterized in that, when the processor executes the program, the program described in claim 5 is implemented. The decoding method described above. 13. A readable storage medium on which a program is stored, characterized in that, when the program is executed by a processor, the steps in the encoding method according to any one of claims 1 to 4 are realized; or, the program When executed by a processor, the steps in the decoding method as claimed in claim 5 are realized.

Images