JP2023124387A - Bit rate determination apparatus, encoding apparatus, and program - Google Patents

Abstract

To provide a bit rate determination apparatus configured to properly determine a bit rate for efficient spatial scalable encoding, and provide an encoding apparatus, and a program.SOLUTION: A preceding spatial scalable encoding unit performs encoding and decoding in advance and outputs a pair of an encoded image and a corresponding bit rate. A definition area information derivation unit determines definition area information which is information representing an area related to definition in an input image. A distortion calculation unit calculates a degree of distortion in an area indicated by the determined definition area information, in the encoded image output from the preceding spatial scalable encoding unit. An allocation bit rate calculation unit calculates an allocation bit rate which is determined based on a bit rate corresponding to a position having a predetermined inclination, in an RD curve defined based on a pair of the bit rate and a corresponding degree of distortion.SELECTED DRAWING: Figure 1

Description

The present invention relates to a bitrate determination device, an encoding device, and a program.

Spatial scalable coding is a method for efficiently coding videos with different resolutions. In the spatial scalable coding method, a base layer (BL), which is coded information for relatively low-resolution video, and an enhancement layer (EL), which is coded information for the difference between high-resolution video and low-resolution video, are layered. encoding. When encoding the enhancement layer, inter-layer processing based on the encoding of the base layer is performed. When decoding, the low resolution video can be decoded from the base layer and the high resolution video can be decoded from both the base layer and the enhancement layer.

Using spatial scalable coding, as an example, 2K video is encoded with a base layer and 4K video is encoded with a base layer and an enhancement layer. Generally, there may be multiple enhancement layers. In the latest coding method VVC (Versatile Video Coding), the Multilayer main 10 profile for performing spatial scalable coding has been defined since the first edition, and its use in video services through broadcasting and communication is expected. In contrast to spatial scalable coding, normal coding with one layer is called single layer coding.

Generally, in video coding, coding control is performed in order to perform high-quality coding in a limited transmission band. It is performed by two processes, bit rate allocation process and RD optimization (rate distortion optimization). Bit rate allocation processing is processing (bit allocation) for appropriately allocating bit rates in units such as GOP (group of pictures) units, frame units, and block units. In the bit rate allocation process, a bit rate is allocated for each predetermined unit while monitoring the consumption bit rate so as not to exceed the overall target bit rate. RD optimization is a process of selecting a prediction mode that maximizes image quality within the range of bitrates allocated by the bitrate allocation process. That is, the RD optimization process selects a prediction mode that minimizes distortion at the assigned bit rate.

In the RD optimization of HM (HEVC test model) and VTM (VVC test model), which are reference software for recent video coding standards such as HEVC (High Efficiency Video Codec) and VVC, prediction that minimizes the cost function J = D + λR Select mode. where D is the distortion, R is the bitrate, and λ is the Lagrange multiplier. The Lagrangian multiplier λ is a value calculated depending on the image and quantization parameter (QP, quantization parameter). In VTM, the value of λ is given by the following relation using the quantization parameter QP and bit depth b.

However, in this formula, a is a constant determined for each prediction mode. Once the value of λ is determined, the cost J is calculated based on D and R for each prediction mode, and the prediction mode with the smallest J is used for encoding.

By the encoding control method described above, efficient encoding can be performed so as not to exceed the target bit rate.

In Non-Patent Document 1, H. H.265/HEVC is described.

Non-Patent Document 2 describes VVC.

Supervised by Sakae Okubo, Impress Standard Textbook Series H. 265/HEVC Textbook, First Edition, Impress Japan, October 21, 2013. Benjamin Bross et al., Overview of the Versatile Video Coding (VVC) Standard and Its Applications, IEEE Transactions on Circuits and Systems for Video Technology, VOL. 31, NO. 10, October 2021.

In single-layer encoding, if a target bit rate is determined, efficient encoding can be performed by the method described above. On the other hand, spatial scalable coding requires a process of appropriately allocating a target bitrate to each layer.

If the bit rates (R _BL , R _EL ) allocated to the base layer and the enhancement layer can be determined appropriately, it is possible to perform coding control similar to single layer coding. At this time, the encoding quality fluctuates according to the video to be encoded and the values of the assigned bit rates _RBL and _REL . Therefore, it is necessary to determine R _BL and R _EL so as to maximize the quality in a limited band.

However, it is difficult to evaluate the quality of the base layer and enhancement layer with different resolutions using an objective index such as PSNR (peak signal-to-noise ratio). As an example, when the base layer and the enhancement layer are encoded at 20 Mbps (megabits per second) in total, R _BL =15 Mbps, PSNR _BL =42, R _EL =5 Mbps, PSNR _EL =40 and R _BL =18 Mbps. , PSNR _BL =45, R _EL =2 Mbps, and PSNR _EL =37, which one has higher quality cannot be determined simply. Therefore, it is required to develop a method of determining the bit rate (R _BL , R _EL ) of each layer for efficient spatial scalable coding.

The present invention has been made based on the recognition of the above problems, and provides a bit rate determination device, an encoding device, and a program capable of appropriately determining a bit rate for performing efficient spatial scalable encoding. is intended to provide

[1] In order to solve the above problems, a bit rate determination device according to one aspect of the present invention performs spatial scalable encoding on a plurality of layers at a predetermined bit rate based on an input image, a preceding spatial scalable encoder that decodes a code resulting from encoding and outputs a pair of an encoded image that is the result of decoding and a bit rate corresponding to the encoded image; A definition region information derivation unit that obtains definition region information, which is information representing a region related to definition, from the input video, and a coded image output from the preceding spatial scalable coding unit. In the RD curve determined based on the pair of the distortion calculation unit that calculates the degree of distortion in the area indicated by the fineness area information, the bit rate output by the preceding spatial scalable encoding unit, and the corresponding degree of distortion and an allocated bit rate calculation unit that obtains an allocated bit rate determined based on a bit rate corresponding to a position having a predetermined inclination.
According to this configuration, the bitrate determination device acquires the relationship between bitrate and distortion based on the results of encoding and decoding performed by the advance spatial scalable encoding unit prior to actual encoding. Also, the bitrate determination device determines the optimal (or nearly optimal) bitrate based on this relationship.

[2] Also, in one aspect of the present invention, in the bit rate determination device described above, the allocated bit rate calculation unit is for each of one or more layers other than the lowest base layer in spatial scalable coding. is to obtain the allocated bit rate of .
According to this configuration, the bitrate determination device can determine the bitrates of one or more layers (enhancement layers described in the embodiments) other than the lowest base layer.

[3] In one aspect of the present invention, in the bit rate determination device described above, the definition area information derivation unit includes an image obtained by enlarging the input image after reducing it, and a differential image creating unit for creating a differential image of the difference image created by the differential image creating unit; and a binarization unit for generating area information.

[4] Further, according to one aspect of the present invention, in the bit rate determination device described above, the fineness area information derivation unit further includes a noise removal unit that removes noise in the difference image, and the binarization unit is to generate the binarized image as the fineness area information based on the differential image from which noise has been removed by the noise removal unit.

[5] Further, according to one aspect of the present invention, in the bit rate determination device described above, the definition region information derivation unit is obtained by binarizing an image obtained by subjecting the input image to a predetermined band-pass filter process. A binarized image is generated as the fineness area information.

[6] Further, according to one aspect of the present invention, in the bit rate determination device described above, the allocated bit rate calculation unit calculates the value of the predetermined slope in the RD curve as the lowest base layer in spatial scalable coding. It is calculated based on the value of the quantization parameter.

[7] In one aspect of the present invention, the bit rate determination device described above further includes a bit rate correction unit that corrects the allocation bit rate calculated by the allocation bit rate calculation unit, and the bit rate correction unit (1) corrects the allocated bitrate so as to ensure a lower limit as the bitrate for the lowest base layer, or (2) one other than the lowest base layer, or The allocated bitrate is corrected so that the allocated bitrate for multiple layers does not exceed an upper limit.
According to this configuration, the bit rate determination device can correct the allocated bit rate calculated by the allocated bit rate calculator based on the lower limit value or the upper limit value.

[8] Further, an encoding device according to an aspect of the present invention includes a bit rate determination device according to any one of [1] to [7] above, and a space for performing spatial scalable encoding of the input image. and a scalable coding unit, wherein the spatial scalable coding unit, for one or more layers other than the lowest base layer in spatial scalable coding, the bit rate determination device is Encoding is performed with the obtained assigned bit rate as a target.
According to this configuration, the encoding device can perform spatial scalable encoding using the bit rate determined by the bit rate determination device.

[9] Further, according to one aspect of the present invention, based on an input image, spatial scalable encoding is performed on a plurality of layers at a predetermined bit rate, and the code resulting from the spatial scalable encoding is decoded and decoded. A preceding spatial scalable encoding unit that outputs a pair of an encoded image that is the result of the encoding and a bit rate corresponding to the encoded image, and information that represents an area related to the sense of detail in the input image. A definition area information derivation unit that obtains the definition area information from the input image, and a degree of distortion in the area indicated by the obtained definition area information in the encoded image output from the preceding spatial scalable encoding unit. A bit corresponding to a position having a predetermined slope in an RD curve determined based on a pair of a distortion calculating unit that calculates the bit rate output from the preceding spatial scalable encoding unit and the corresponding degree of distortion A program for causing a computer to function as a bit rate determination device including an allocation bit rate calculation unit that determines an allocation bit rate determined based on the rate.

According to the present invention, the bit rate can be determined based on the results of encoding and decoding performed by the advanced spatial scalable encoder prior to actual encoding.

1 is a block diagram showing a schematic functional configuration of an encoding device (including a bit rate determining device inside) according to an embodiment of the present invention; FIG. 4 is a block diagram showing a detailed functional configuration of a fineness area information deriving unit in the encoding device according to the same embodiment; FIG. FIG. 4 is a schematic diagram showing an example of a frame image (original image) input to a fineness area information derivation unit in the same embodiment; FIG. 4 is a schematic diagram showing an example of an image (fineness area information) output as a result of processing the image shown in FIG. 3 by a fineness area information derivation unit in the same embodiment; 4 is a block diagram showing a schematic functional configuration of an allocation bit rate calculation unit according to the same embodiment; FIG. It is an example of the graph which shows the RD curve in a fineness area|region for calculating|requiring a bit rate in the same embodiment. FIG. 12 is a block diagram showing a schematic functional configuration of an encoding device according to a sixth modification of the same embodiment; FIG. 4 is a block diagram showing an example of an internal configuration of an encoding device according to the same embodiment (including modifications);

An embodiment of the present invention will now be described with reference to the drawings. The bit rate determination device of this embodiment determines the bit rate to be assigned to each layer in order to efficiently perform spatial scalable coding. Also, the encoding device is a device that performs spatial scalable encoding at the determined bit rate.

In this embodiment, the bit rate _{R_EL} of the enhancement layer is properly determined first, and then the bit rate _{R_BL} of the base layer is determined. Efficient encoding can be achieved by this procedure. The reason is as follows.

Assuming the case of transmitting spatially scalable encoded video using only broadcast signals, the overall bit rate can be fixed. That is, (R _BL +R _EL ) can be assumed to be constant. At this time, the larger the _REL , the higher the quality of the enhancement layer and the higher the quality of the high-resolution video can be coded, but the lower the quality of the base layer. Conversely, if the minimum required bit rate can be determined as R _EL , all remaining bit rates can be R _BL . As a result, the quality of the base layer is prevented from deteriorating due to wasteful allocation of a large bit rate to _REL , and the limited band can be used efficiently.

Assuming the case of transmitting video using communication, the overall bit rate can be variable (variable bit rate). In the case of high resolution video such as 4K video or 8K video, or video that has a lot of movement and is difficult to encode, a huge amount of data is generated and bandwidth is wasted even though there is almost no difference in image quality. Otherwise, the amount of data may become enormous, causing the network to become unstable. At this time, if the _REL can be determined as a bit rate that can sufficiently improve the quality of the enhancement layer, it will be possible to efficiently use the bandwidth and provide stable video services without unnecessary transmission of a large amount of data. becomes possible.

As described above, if the _REL can be appropriately determined, efficient encoding can be achieved regardless of whether the overall transmission rate is a fixed bit rate or a variable bit rate. Below, the functional configuration of the device for appropriately determining the bit rate will be described.

In spatial scalable coding, the base layer video loses high frequency components. For this reason, no matter how high the _RBL is, it is impossible to reproduce the high frequency components (definition) originally held by the enhancement layer. In this embodiment, the bit rate _{R_EL} required for the enhancement layer is derived based on the enhancement layer signal that cannot be reproduced in the base layer. Based on the derived values, the bit rate R _BL to be assigned to the base layer and the bit rate R _EL to be assigned to the enhancement layer are determined.

The important points in the configuration of this embodiment are to extract a signal that reproduces the sense of detail from the image, and to use the RD curve (rate distortion curve) created by using the distortion of the signal. and calculating an appropriate bit rate using the In the present embodiment, extracting a signal that reproduces the sense of detail from the image is based on the sense of detail region information (information indicating the degree of presence or absence of sense of detail for each pixel) derived by the sense of detail region information deriving unit 21. can be done based on The slope calculator 231 creates the RD curve using the signal distortion. A bit rate calculator 232 calculates an appropriate bit rate based on the RD curve. As a modified example, the bit rate correction unit 24 may correct the bit rate as described later.

When the horizontal axis is the bit rate and the vertical axis is the degree of distortion (for example, the MSE value of the amount of distortion), the characteristics of the RD curve are as follows. That is, the first feature is that it is a monotonically decreasing curve in which the degree of distortion monotonously decreases with increasing bit rate (the first derivative is negative over the entire bit rate region of interest). . The second feature is that the curve is downwardly convex (the second derivative is non-negative over the entire bit rate region of interest). One method of finding the optimum bit rate in such an RD curve is to find the point of contact when a straight line with a predetermined slope touches the RD curve, and the bit rate corresponding to that point of contact is the optimum bit rate. reasonable (or nearly optimal) bitrate.
In other words, in the RD curve as described above, the bit rate corresponding to the position having the predetermined slope is the optimum (or nearly optimum) bit rate. Note that when an image quality index other than the MSE value of the amount of distortion is used, the RD curve of the value representing the degree of distortion with respect to an increase in bit rate may not be a monotonically decreasing curve. For example, when PSNR or the like is used as an image quality index (degree of distortion), it becomes a monotonically increasing curve (the first order differential coefficient is positive over the entire bit rate region of interest). Also, depending on the image quality index used, the RD curve may not necessarily be a downwardly convex curve. Also in these cases, the optimum bit rate can be obtained according to the slope of the RD curve. It may also be possible to determine the bitrate based on the intersection of the RD curve and the straight line.

FIG. 1 is a block diagram showing a schematic functional configuration of an encoding device according to this embodiment. As illustrated, the encoding device 1 includes an encoding unit 11, a decoding unit 12, a fineness area information derivation unit 21, a distortion calculation unit 22, an allocation bit rate calculation unit 23, and a spatial scalable encoding unit. 31 and. At least part of these functional units can be realized by, for example, a computer and a program. In addition, each functional unit has storage means as necessary. The storage means are, for example, program variables and memory allocated by program execution. Also, if necessary, non-volatile storage means such as a magnetic hard disk drive or a solid state drive (SSD) may be used. Also, at least part of the function of each functional unit may be realized as a dedicated electronic circuit instead of a program.

An encoding device 1 performs scalable spatial encoding at an appropriate bit rate. A video input to the encoding device 1 is a video to be encoded and is a high-resolution video. A video is a series of frame images arranged in chronological order. In other words, the encoding of video in this embodiment is reduced to the process of encoding the frame images that make up the video.

In the configuration of the encoding device 1, the configuration of the portion including the encoding unit 11, the decoding unit 12, the detail area information derivation unit 21, the distortion calculation unit 22, and the allocation bit rate calculation unit 23 is It can be regarded as the rate determination device 2 . That is, the bitrate determination device 2 determines an appropriate bitrate based on the input video.

As part of the process for determining the bit rate, the bit rate determination device 2 performs encoding and decoding of the input video in advance. That is, of the configuration of the encoding device 1 , the configuration of the portion including the encoding section 11 and the decoding section 12 may be called the advanced spatial scalable encoding section 3 .

The encoding unit 11 performs spatial scalable encoding on the input video. The encoding unit 11 can perform spatial scalable encoding using existing technology.

The decoding unit 12 decodes the video encoded by the encoding unit 11 . Decoding by the decoding unit 12 can also be performed by existing technology.

The preceding spatial scalable encoding unit 3 outputs a pair of bit rate and encoded video (video decoded by the decoding unit 12) as encoding result information (delivers to the distortion calculation unit 22).

The fineness area information derivation unit 21 derives information on the fineness area in the input image (high-resolution image). Here, the fineness region is a region having fineness in the image, and is a region having enhancement layer information that cannot be reproduced in the base layer.
That is, the fineness region information derivation unit 21 obtains fineness region information, which is information representing a region related to fineness in the input image, based on the input video.
In other words, the fineness area information derivation unit 21 derives information (fineness area information) about an area in which there is enhancement layer information that cannot be reproduced by the code of the base layer alone.

The distortion calculator 22 calculates the distortion of the definition area in the input image (input video). Specifically, the distortion calculating unit 22 calculates the degree of distortion in the area indicated by the fineness area information obtained by the fineness area information deriving unit 21 in the encoded image output from the preceding spatial scalable encoding unit 3. do.

Specifically, the distortion calculator 22 performs the following processing. The distortion calculator 22 changes the bit rate _{R_EL} of the enhancement layer and creates an encoded image for each of a plurality of _{R_ELs} . At this time, the base layer bit rate _{R_BL} is arbitrary. Next, the distortion calculator 22 calculates the distortion in the fineness area for the encoded image corresponding to each _REL . Specifically, the distortion calculator 22 calculates indicators such as the mean squared error (MSE) and PSNR (peak signal-to-noise ratio) of distortion. At this time, the distortion calculator 22 refers to the image of the fineness region information output by the fineness region information derivation unit 21 to grasp the fineness region in the image. The distortion calculation unit 22 passes a set of pairs of the enhancement layer bit rate _{R_EL} and the calculated distortion numerical value to the allocation bit rate calculation unit 23 .

The allocated bit rate calculator 23 analyzes the relationship between the rate and the distortion based on the encoded video and the detail area information, and calculates the allocated bit rate from the analysis result. Note that the allocated bit rate calculation unit 23 calculates the allocated bit rate for each of one or more layers (that is, one or more enhancement layers) other than the lowest base layer in spatial scalable coding. can be anything.

More specifically, the allocated bit rate calculator 23 determines the RD curve based on a pair of the bit rate output from the preceding spatial scalable encoder 3 and the degree of distortion corresponding to the bit rate. Then, the allocated bit rate calculator 23 obtains the allocated bit rate determined based on the bit rate (r) corresponding to the position having the predetermined slope (λ _dif ). The allocation bit rate calculator 23 determines the bit rate _{R_EL} of the enhancement layer by the method described above. The allocated bit rate calculator 23 may arbitrarily determine the base layer bit rate _RBL . However, the allocated bit rate calculator 23 may determine the R _BL based on the determined R _EL , for example, when the bit rate of the entire transmission line is a fixed rate.

A more detailed functional configuration of the allocated bit rate calculator 23 will be described later with reference to FIG.

The spatial scalable encoding unit 31 performs spatial scalable encoding on the input image at the bit rate determined by the allocation bit rate calculation unit 23 . The spatial scalable encoding unit 31 targets the assigned bit rate obtained by the assigned bit rate calculation unit 23, particularly for one or more layers (that is, enhancement layers) other than the lowest base layer in spatial scalable encoding. Encoding is performed as follows.

However, in the case of the sixth modification described later, the bit rate determined by the allocation bit rate calculator 23 is corrected by the bit rate corrector 24 .

In the encoding device 1, the aforementioned bit rate determination device 2 determines the bit rate based on the input video (image).

In the encoding device 1, the preceding spatial scalable encoding unit 3 performs spatial scalable encoding on a plurality of layers at a predetermined bit rate based on the input image, and performs spatial scalable encoding. The resulting code is decoded, and a pair of the resulting encoded image and the bit rate corresponding to the encoded image is output.

FIG. 2 is a block diagram showing a schematic functional configuration of the fineness area information derivation unit 21 according to this embodiment. As illustrated, the fineness region information derivation unit 21 includes a reduction processing unit 211, an enlargement processing unit 212, a difference image generation unit 213, a noise removal unit 214, and a binarization unit 215. be done. The aforementioned input image (high-resolution image) is input to the definition area information derivation unit 21 .

The reduction processing unit 211 reduces each frame image (original image) of the input video. That is, the reduction processing unit 211 converts the input high-resolution video into a relatively low-resolution video. Specifically, the reduction processing unit 211 converts the high-resolution image into an image having the number of pixels of the base layer. The reduction processing unit 211 passes the reduced image to the enlargement processing unit 212 .

The enlargement processing unit 212 enlarges the video transferred from the reduction processing unit 211 . Specifically, the enlargement processing unit 212 interpolates the pixels of the transferred image to convert the image to the resolution before reduction (the resolution of the input image). The enlargement processing unit 212 passes the image after enlargement to the difference image creation unit 213 .

Filters used by the reduction processing unit 211 and the enlargement processing unit 212 for reduction processing and enlargement processing, respectively, may be based on existing technology and can be arbitrarily selected. For example, a Bi-Cubic filter, a Lanczos filter, or the like can be used. Since the reduction processing unit 211 and the enlargement processing unit 212 perform processing to reduce and then enlarge the image, the definition of the input image is lost.

The difference image creation unit 213 calculates the difference between the input image (original image) and the image (image enlarged after reduction) passed from the enlargement processing unit 212 to create a difference image. In other words, the difference image creation unit 213 creates a difference image between the input image (original image) and the image (output from the enlargement processing unit 212) obtained by enlarging the input image after reducing it. Specifically, the difference image creation unit 213 calculates the difference in pixel value between corresponding pixels in both videos. The difference image may be considered to be an image that expresses the degree of fineness as the pixel value of each pixel. The difference image creation unit 213 passes the generated difference image to the noise removal unit 214 .

The noise removal unit 214 performs processing for removing noise contained in the difference image passed from the difference image creation unit 213 . Since the difference image passed from the difference image generation unit 213 is considered to contain high-frequency component noise such as camera noise, noise is removed by the noise removal unit 214 . The noise removal unit 214 may use any filter for noise removal for noise removal processing. As a filter for removing noise, a filter based on existing technology can be used. The noise removal unit 214 can perform noise removal processing using, for example, a median filter, a low-pass filter, or the like. Further, the noise removal unit 214 may remove noise by combining a plurality of filters, such as applying a median filter after a low-pass filter. The noise removal unit 214 may also perform other image processing during the noise removal process. The noise removal unit 214 passes the processed video to the binarization unit 215 .

Note that when noise removal is unnecessary, the noise removal processing by the noise removal unit 214 may be omitted.

The binarization unit 215 generates a binarized image by binarizing the pixel value of each pixel based on the difference image created by the difference image creation unit 213 based on a predetermined threshold value. This binarized image is the fineness area information. In other words, the binarized image represents information about the area related to fineness. When the noise removal unit 214 performs noise removal processing, the binarization unit 215 generates the binarized image based on the difference image from which the noise removal unit 214 has removed noise.

That is, the binarization unit 215 converts the image passed from the noise removal unit 214 (or, if the processing by the noise removal unit 214 is omitted, the video directly transferred from the difference image generation unit 213) to Perform binarization processing. The binarization unit 215 may, for example, binarize the luminance signal of a 10-bit video into 0 and 1023 based on a threshold arbitrarily determined for each pixel. Alternatively, the binarization unit 215 may apply a predetermined filter to the difference image passed from the noise removal unit 214 and then generate a binarized image based on a predetermined threshold value. The image output by the binarization unit 215 is information representing an area in the original image that has a sense of detail that cannot be reproduced in the base layer in spatial scalable coding.

That is, the image generated by the binarization unit 215 is the image output by the fineness area information deriving unit 21 (FIG. 1). Details area information will be described with reference to FIGS. 3 and 4 below.

FIG. 3 is a schematic diagram showing an example of a frame image (original image) input to the fineness area information derivation unit 21. As shown in FIG. The original image here is an image of the starting point of the marathon, and is originally a color image.

FIG. 4 is a schematic diagram showing an image output as a result of processing the image shown in FIG. Here, the binarization unit 215 in the fineness region information derivation unit 21 binarizes the luminance signal (pixel values range from 0 to 1023) of the differential image passed from the noise removal unit 214 with a threshold value of 20. , 3×3 median filter processing was performed three times and binarized with a threshold value of 700. The binarized image in FIG. 4 is represented by binary values of white and black for each pixel. As can be seen from both figures, the fineness region information deriving unit 21 successfully extracts the fineness region in the original image.

Next, the details of the processing of the allocation bit rate calculation unit 23 that determines the bit rate allocated to each layer based on the derived definition area information will be described.

FIG. 5 is a block diagram showing a schematic functional configuration of an allocation bit rate calculator according to this embodiment. As illustrated, the allocated bit rate calculator 23 includes a slope calculator 231 and a bit rate calculator 232 .

The slope calculator 231 obtains the slope of the curve at the position of the predetermined bit rate R _EL based on the bit rate R _EL of the enhancement layer and the distortion in the definition area corresponding to the bit rate. Specifically, the slope calculator 231 receives a set of pairs of the bit rate _{R_EL} and the corresponding mean squared error value of the distortion from the distortion calculator 22 described above. Then, the slope calculator 231 obtains a specific curve by plotting the corresponding points based on the set of these value pairs. This curve is the RD curve in the desired definition area. Note that the RD curve may be obtained using a regression method. An example of this curve is described later with reference to FIG.

An example of calculating the slope using the encoding result of the base layer is as follows. The quantization parameter (QP _BL ) in the base layer is known from the base layer encoding result when creating a plurality of encoded images. It is known from prior research on RD optimization that the Lagrangian multiplier λ _BL of the base layer for efficient encoding can be obtained by the following equation using QP _BL . Note that a and b are as described above.

The slope λ _dif to be obtained is given by the following formula.

Note that w is a coefficient representing weight. The value of w may be determined in advance, or may be adaptively determined for each video. Since the function of the RD curve is generally different between the base layer and the enhancement layer, it is appropriate to multiply the weighting factor w to obtain λ _dif . Note that, for example, w within the range of 0.80≦w≦1.20 may be used. Also, as an example, w=1.00.

As described above, the gradient calculator 231 of the allocated bitrate calculator 23 converts the predetermined gradient value (λ _dif ) in the RD curve to the quantization parameter value (QP _BL ).

The bit rate calculator 232 calculates the bit rate r at the point where the slope of the tangent line is λ _dif on the RD curve of the definition area obtained above. The bit rate r calculated by the bit rate calculator 232 will be described next with reference to FIG.

FIG. 6 is an example of a graph showing an RD curve in the fineness area. As described above, the slope calculator 231 of this embodiment obtains the RD curve based on distortion calculated at a plurality of bit rates _REL . In the figure, the horizontal axis is the enhancement layer bit rate R _EL (in units of Mbps (megabits per second)), and the vertical axis is the mean squared error (MSE) corresponding to the distortion in the definition area. The RD curve 301 shown in the figure is determined based on a set of pairs of bit rate _REL and mean squared error. A tangent line 302 is a straight line that has a slope of λ _dif determined as described above and is tangent to the RD curve 301 . The value in the horizontal axis direction of the point of contact P between the RD curve 301 and the tangent line 302 is the bit rate r to be calculated by the bit rate calculator 232 . This bit rate r is a bit rate obtained as an optimum value under the conditions represented by the RD curve 301 .

Although the embodiments have been described above, the present invention can also be implemented in the following modifications. A plurality of modifications will be described here. In addition, you may implement combining two or more modifications. In addition, below, only the characteristic structure and function in each modification are demonstrated.

[First modification]
The configuration described above assumes that the video input to the encoding device 1 is a high-resolution video. When the input video is a low-resolution video, as a first modified example, up-converter processing is performed before input to the encoding device 1 to increase the number of pixels of the input video. Any existing technology can be used for the upconverter process.

[Second modification]
In the above embodiment, the fineness area information derivation unit 21 derives the fineness area information by taking the difference between the images (the process of the difference image creation unit 213). In the second modified example, the fineness region information derivation unit 21 derives information about the fineness region using a band-pass filter instead. In general, the information related to fineness is the high frequency components of the original image. Therefore, in this modification, the fineness region information derivation unit 21 performs band-pass filter processing on the original image instead of performing reduction processing and enlargement processing on the original image to generate a difference image from the original image. , to create a filtered image. The band-pass filter may be a high-pass filter that passes frequency components higher than a predetermined threshold frequency. After the fineness area information derivation unit 21 according to this modification outputs the filtered image, noise removal processing (noise removal unit 214) and binarization processing (binarization unit 215) are performed in the same manner as the method already described. By doing so, the fineness area information can be created.

That is, in the second modification, the fineness region information derivation unit 21 generates a binarized image obtained by binarizing an image obtained by performing a predetermined band-pass filter process on an input image as fineness region information. do.

[Third Modification]
In the above embodiment, the method of determining the bit rate _{R_EL} of the enhancement layer has been shown, and the case where the bit rate _{R_BL} of the base layer may be determined arbitrarily has been described. On the other hand, in this third modification, the method of determining the bit rate _RBL is as follows. For example, if the base layer bit rate target value is predetermined, that bit rate is determined as the _RBL . As a result, more efficient enhancement layer bitrate allocation can be expected. Alternatively, make the base layer bitrate high enough. In other words, the bit rate of the base layer is made equal to or higher than the predetermined lower limit. As a result, the influence of coding deterioration in the base layer can be suppressed to a negligible level. For example, for 2K video, the base layer bit rate _RBL is set to 20 Mbps or higher. Alternatively, for 4K video, the base layer bit rate _RBL is set to 50 Mbps or higher.

[Fourth Modification]
In the above embodiment, the slope calculator 231 performs calculation based on the quantization parameter QP _BL in the base layer in order to obtain the slope λ _dif . Also, the slope λ _dif was obtained by multiplying λ _BL by a predetermined weighting factor w. On the other hand, in the fourth modified example, the value of the slope λ _dif is set to a predetermined constant. Even with this method, a good bit rate r may be obtained.

[Fifth Modification]
In the above embodiment, the bit rate calculator 232 obtains the bit rate r by obtaining the point of contact between the RD curve of the fineness area and the straight line with the slope λ _dif . However, there may be cases where there is no point on the RD curve of the fineness area where the slope of the tangent line is λ _dif . Therefore, in the fifth modification, the bit rate calculator 232 predetermines a default value of r (which may be called an initial value). If there is no point on the RD curve where the slope of the tangent line is λ _dif , the bit rate calculator 232 determines the default value as the value of the bit rate r. Also, if there is a point on the RD curve where the tangent slope is λ _dif , the bit rate calculator 232 determines the point corresponding to the tangent point with the slope of λ _dif as described above. Determine the bitrate r.

[Sixth Modification]
In the sixth modification, the encoding device has a bit rate corrector 24 . The bit rate correction unit 24 performs processing for correcting the bit rate calculated by the allocation bit rate calculation unit 23 .

FIG. 7 is a block diagram showing a schematic functional configuration of an encoding device 4 according to this modification. As illustrated, the encoding device 4 according to this modification includes an encoding unit 11, a decoding unit 12, a detail area information derivation unit 21, a distortion calculation unit 22, an allocation bit rate calculation unit 23, a bit It is configured to include a rate correction unit 24 and a spatial scalable coding unit 31 . That is, the encoding device 4 further has the function of the bit rate corrector 24 in addition to the functions of the encoding device 1 described with reference to FIG. In this modification, the spatial scalable encoding unit 31 performs spatial scalable encoding while using the bit rate calculated by the allocation bit rate calculation unit 23 after correction by the bit rate correction unit 24. process. Regarding other functions, the encoding device 4 functions in the same manner as the encoding device 1 described above.

The bit rate correction unit 24 having a function specific to this modification performs processing for correcting the bit rate according to the available bandwidth. That is, the bit rate calculated by the allocation bit rate calculator 23 is corrected by the bit rate corrector 24 . The spatial scalable encoding unit 31 performs spatial scalable encoding using the corrected bit rate. Specific correction processing is as described below.

Correction processing by the bit rate correction unit 24 is as follows.

When the overall bitrate used for transmission is a fixed bitrate _{R_t} , if the determined enhancement layer bitrate _{R_EL} is too large, the base layer bitrate _{R_BL} is reduced. That is, the quality of the base layer may deteriorate. In order to prevent such quality deterioration of the base layer, R _BL ^min is determined as the lower limit of R _BL , or R _EL ^max is determined as the upper limit of R _EL . When the allocated bit rate calculation unit 23 determines r as the R _EL value, the bit rate correction unit 24 performs correction processing so that the R _EL value does not exceed the upper limit value R _EL ^max . _REL after correction by the bit rate correction unit 24 is determined by the following formula (1).

This allows R _EL to be determined based on R _BL ^min or R _EL ^max , thereby ensuring a sufficient value for R _BL .

When the overall bit rate used for transmission is a variable bit rate, if the bit rate _{R_EL} of the enhancement layer is too large, it may cause instability of the transmission path (communication network). Also, even with a variable bit rate, there may be an upper limit to the bandwidth that can be used. Therefore, R _EL ^max is determined as the upper limit of R _EL . When the allocated bit rate calculation unit 23 determines r as the R _EL value, the bit rate correction unit 24 performs correction processing so that the R _EL value does not exceed the upper limit value R _EL ^max . _REL after correction by the bit rate correction unit 24 is determined by the following formula (2).

That is, in the case of the sixth modified example, the bit rate correction unit 24 corrects the allocation bit rate calculated by the allocation bit rate calculation unit 23 . The correction method is as follows. That is, the bit rate correction unit 24 corrects the bit rate _{R_EL} assigned to the enhancement layer so that the lower limit value R _BL ^min as the bit rate for the lowest base layer (BL) can be secured. Alternatively, the bit rate correction unit 24 corrects the allocated bit rate so that the allocated bit rate for one or more layers (EL) other than the lowest base layer does not exceed the upper limit value R _EL ^max . .

[Seventh Modification]
For the embodiments described above, the frequency of bitrate calculation and determination was not specified. As a seventh modification, the frequency with which the allocation bit rate calculation unit 23 calculates the bit rate (in the case of the sixth modification, the frequency with which the bit rate correction unit 24 corrects the bit rate is the same) is appropriately let it be determined. For example, the frequency of calculating the bit rate (the same applies to the frequency of correction; the same applies hereinafter) may be set for each frame of the input video. Alternatively, the frequency of calculating the bit rate may be every predetermined number of multiple frames of the input video (for example, every 30 frames).

Alternatively, the encoding device 1 may detect a scene change point in the input video, and calculate the bit rate based on the frames belonging to the new scene when the scene change is detected. Scene change detection itself can be performed using existing technology. An example of the timing of the scene change is the timing of switching from the video in the studio to the video at the relay destination in a program recorded in a studio.

The scene change described above is detected based on a predetermined feature amount of consecutive frame images. This is generalized not only to scene changes, but also to frame-by-frame feature amounts (time-dependent feature amounts) in the input video, or for each area (space) within the image (area-dependent feature amounts within the image). amount), the timing for recalculating the bit rate may be determined. In this case, the bit rate can be recalculated at appropriate timing based on not only the scene change but also the features of the video.

Further, as a further modified example of the seventh modified example, the following may be done. That is, the allocated bit rate calculation unit 23 (and the bit rate correction unit 24 when correcting the bit rate) does not determine the bit rate based on one frame image, but determines the bit rate based on a plurality of frame images. A bit rate (for example, an average value of those bit rates) may be determined based on the bit rates determined respectively by the

[Eighth modification]
In the embodiments described above, two layers of information are encoded, a base layer and one enhancement layer. Therefore, two bit rates, _RBL and _REL , were determined. In the eighth modified example, the encoding device 1 performs similar processing in encoding of three or more layers, and determines the bit rate of each layer. When the number of layers increases, similar processing is performed for each higher layer to allocate the bit rate of each layer.

That is, in the eighth modified example, the RD curve is obtained by deriving the fineness area information for each of the plurality of enhancement layers. Then, for each layer, a tangent line tangent to the RD curve with a predetermined slope is obtained, and the coding bit rate of that layer is determined based on the bit rate at the tangent point.

By using the eighth modification, it becomes possible to determine an appropriate bit rate in spatial scalable coding with an arbitrary number of layers (two or more).

[Ninth Modification]
In the embodiments described above, the case of transmitting encoded information has been described. Not only when transmitting encoded video, but also when recording video encoded by spatial scalable coding on recording media (e.g., Blu-ray discs, semiconductor memories, magnetic hard disks, etc.) can apply the bitrate determination technique described in .

[Tenth Modification]
As a tenth modification, only the function of the bit rate determination device 2 inside the encoding device 1 may be implemented as an independent device. In this case, the bit rate determination device 2 can determine the bit rate and output information on the determined bit rate to the outside. The function of performing spatial scalable encoding can refer to the bit rate information determined by the bit rate determination device 2 and use the bit rate as a target rate to perform encoding of the corresponding layer.

FIG. 8 is a block diagram showing an example of the internal configuration of the encoding device 1 or encoding device 4 of the above embodiment (including modifications). The encoding device 1 and the encoding device 4 can be realized using a computer. As shown, the computer includes a central processing unit 901 , RAM 902 , input/output ports 903 , input/output devices 904 and 905 and the like, and bus 906 . The computer itself can be implemented using existing technology. The central processing unit 901 executes instructions included in programs read from the RAM 902 or the like. The central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic operations and logical operations according to each instruction. A RAM 902 stores data and programs. Each element contained in RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory". The input/output port 903 is a port for the central processing unit 901 to exchange data with an external input/output device or the like. Input/output devices 904 and 905 are input/output devices. The input/output devices 904 and 905 exchange data with the central processing unit 901 via the input/output port 903 . Bus 906 is a common communication path used inside the computer. For example, central processing unit 901 reads and writes data in RAM 902 via bus 906 . Also, for example, central processing unit 901 accesses input/output ports via bus 906 .

At least part of the functions of the encoding device 1 and the encoding device 4 in the above-described embodiments (including modifications) can be realized by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in this recording medium may be read into a computer system and executed. It should be noted that the "computer system" referred to here includes hardware such as an OS and peripheral devices. In addition, “computer-readable recording media” refers to portable media such as flexible discs, magneto-optical discs, ROMs, CD-ROMs, DVD-ROMs, USB memories, and storage devices such as hard disks built into computer systems. Say things. In other words, the "computer-readable recording medium" may be a non-transitory computer-readable recording medium. In addition, "computer-readable recording medium" means a medium that temporarily and dynamically retains a program, such as a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line. , it may also include something that holds the program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in that case. Further, the program may be for realizing part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in a computer system.

As described above, according to the above-described embodiments (including modifications), the preceding spatial scalable coding unit performs coding in advance, and the relationship between the bit rate and the distortion of the coded image at that time is Obtainable. Then, the bit rate to be assigned can be optimized based on the relationship between the bit rate and the distortion of the encoded image. That is, efficient codes can be generated in advance spatial scalable coding.

The embodiments (including modifications) of the present invention have been described in detail above with reference to the drawings, but the specific configuration is not limited to this embodiment, and can be designed without departing from the gist of the present invention. etc. are also included.

INDUSTRIAL APPLICABILITY The present invention can be used, for example, in the work of encoding and transmitting or recording video. However, the scope of application of the present invention is not limited to those exemplified here.

1 encoding device 2 bit rate determining device 3 preceding spatial scalable encoding unit 4 encoding device 5 bit rate determining device 11 encoding unit 12 decoding unit 21 detail area information derivation unit 22 distortion calculation unit 23 allocation bit rate calculation unit 24 bit rate correction unit 31 spatial scalable coding unit 211 reduction processing unit 212 enlargement processing unit 213 difference image generation unit 214 noise removal unit 215 binarization unit 231 inclination calculation unit 232 bit rate calculation unit 301 RD curve 302 tangent (inclination λ _dif )
901 central processing unit 902 RAM
903 input/output ports 904, 905 input/output device 906 bus

Claims (9)

Based on an input image, perform spatial scalable encoding on a plurality of layers at a predetermined bit rate, decode the code resulting from the spatial scalable encoding, and decode the resulting encoded image and the encoded image. a preceding spatial scalable encoder that outputs a pair of a bit rate corresponding to an image;
a fineness region information derivation unit that obtains fineness region information, which is information representing a region related to fineness in the input image, from the input image;
a distortion calculation unit that calculates the degree of distortion in the region indicated by the obtained fineness region information of the encoded image output from the preceding spatial scalable encoding unit;
An assigned bit rate determined based on a bit rate corresponding to a position having a predetermined slope in an RD curve determined based on a pair of the bit rate output from the preceding spatial scalable encoding unit and the corresponding degree of distortion. an allocation bit rate calculation unit for obtaining
A bitrate determination device comprising: The allocated bitrate calculation unit obtains the allocated bitrate for each of one or more layers other than the lowest base layer in spatial scalable coding,
2. A bitrate determination device according to claim 1. The definition area information derivation unit,
an image obtained by enlarging the input image after it has been reduced, and a difference image creating unit that creates a difference image between the input image and the input image;
a binarization unit that generates a binarized image obtained by binarizing the pixel value of each pixel based on the difference image created by the difference image creation unit based on a predetermined threshold value as the fineness region information;
comprising
3. A bitrate determination device according to claim 1 or 2. The definition area information derivation unit,
a noise removal unit that removes noise in the difference image;
further comprising
The binarization unit generates the binarized image as the fineness region information based on the difference image after noise has been removed by the noise removal unit.
4. A bitrate determination device according to claim 3. The definition area information derivation unit generates a binarized image obtained by binarizing an image obtained by performing a predetermined band-pass filter process on the input image as the definition area information.
3. A bitrate determination device according to claim 1 or 2. The allocated bit rate calculation unit calculates the value of the predetermined slope in the RD curve based on the value of the quantization parameter of the lowest base layer in spatial scalable coding,
Bitrate determination device according to any one of claims 1 to 5. a bit rate correction unit that corrects the allocated bit rate calculated by the allocated bit rate calculation unit;
further comprising
The bit rate correction unit
(1) correcting the allocated bit rate so as to ensure a lower limit as the bit rate for the lowest base layer;
or,
(2) correcting the allocated bitrate so that the allocated bitrate for one or more layers other than the lowest base layer does not exceed an upper limit;
3. A bitrate determination device according to claim 2. a bitrate determination device according to any one of claims 1 to 7;
a spatial scalable encoding unit that performs spatial scalable encoding of the input image;
An encoding device comprising:
The spatial scalable encoding unit performs encoding targeting the assigned bit rate obtained by the bit rate determination device for one or more layers other than the lowest base layer in spatial scalable encoding.
Encoding device. Based on an input image, perform spatial scalable encoding on a plurality of layers at a predetermined bit rate, decode the code resulting from the spatial scalable encoding, and decode the resulting encoded image and the encoded image. a preceding spatial scalable encoder that outputs a pair of a bit rate corresponding to an image;
a fineness region information derivation unit that obtains fineness region information, which is information representing a region related to fineness in the input image, from the input image;
a distortion calculation unit that calculates the degree of distortion in the region indicated by the obtained fineness region information of the encoded image output from the preceding spatial scalable encoding unit;
An assigned bit rate determined based on a bit rate corresponding to a position having a predetermined slope in an RD curve determined based on a pair of the bit rate output from the preceding spatial scalable encoding unit and the corresponding degree of distortion. an allocation bit rate calculation unit for obtaining
A program for making a computer act as a bitrate determination device, comprising

Images