FR2929797A1 - METHOD AND DEVICE FOR ENCODING AN IMAGE SEQUENCE - Google Patents

Abstract

The invention relates to a method of encoding an image sequence in the form of a scalable stream comprising a base layer and an enhancement layer. The method comprises the following steps: determining (E101, E102) in each image of the sequence, called the first image sequence, a cutting window (W1); subsampling (E103), for each image of the first sequence, the image portion delimited by the clipping window (W1) so as to construct a second sequence of images; and- coding (E12, E14, E16) the second image sequence as a base layer of the scalable flow and the first image sequence as a scalable flux enhancement layer.According to the invention, the clipping window (W1 ) is determined in an image of the first sequence according to the following steps: - determining (E101) in the image a first cutting window (W0); anddetermining (E102) a second clipping window surrounding the first clipping window (W0) so that the second clipping window conforms to at least one predefined constraint imposed by the coding step (E12, E14, E16 ), the clipping window (W1) of the image being the second clipping window.

Description

METHOD AND DEVICE FOR ENCODING AN IMAGE SEQUENCE

FIELD OF THE INVENTION The invention relates to the general field of coding image sequences. The invention more particularly relates to a method of coding an image sequence in the form of a scalable stream and a coding device implementing the coding method.

2. State of the Art Scalable coding makes it possible to represent the same sequence of images in a single stream in the form of several formats, each of them being adapted to different terminals having different capacities in terms of display and / or or computing power and / or memory and / or network interfaces.

With reference to FIGS. 1 and 2, there is known a coding device 1 capable of coding a sequence SO of initial images I0 in the form of a scalable flow F comprising a base layer and at least one enhancement layer. It is known that such a coding device 1 comprises an input 8 for receiving a sequence SO of initial images I0 and a sub-sampling module 10 able to sub-sample each IO image of said sequence SO to generate a sequence S1 of modified images 11 of lower resolution than that of the initial images 10. The coding device 1 further comprises a first coding module 12 able to encode the images 11 of the modified sequence S1 into a first stream F1 known as a layer base and a second coding module 14 capable of coding the images of the initial sequence SO into a second FO stream known as an enhancement layer from, in particular, coding data (eg motion data, data). residuals, texture data ...) resulting from the coding of the images of the modified sequence S1. The coding device 1 further comprises connected to the outputs of the first and second coding modules 12 and 14, a multiplexing module 16 able to multiplex the base layer F1 and the enhancement layer FO in a single stream F transmitted via the According to a known variant, illustrated in FIG. 3, the sub-sampling module 10 of the initial image sequence images SO 10 is adapted to sub-sample only a part of each image 10 of said sequence SO in order to generate the sequence S1 of modified images 11, which part is delimited in the image IO by a cutting window W0.

It is common for the coding modules 12 and 14 to impose constraints on the subsampling module 10 to define the clipping window W0. Since the IO images are cut into blocks of pixels that do not overlap, a constraint is for example that the top left corner of the clipping window WO is aligned with the block grid in 10. Another example of a constraint imposes that the the top left corner of the cutting window WO is moved between two consecutive images IO of the initial sequence SO by a multiple of 2 pixels along the horizontal axis and the vertical axis. In these cases, the clipping window WO can not be positioned anywhere in the IO images and in particular not optimally with respect to the visual content of these images 10. Moreover, such constraints reduce the visual comfort of the images. a user viewing the modified images 11 generated from the IO images and the window W0.

SUMMARY OF THE INVENTION The object of the invention is to overcome at least one of the drawbacks of the prior art. To this end, the invention relates to a method of encoding an image sequence in the form of a scalable stream comprising a base layer and an enhancement layer. The method comprises the following steps: determining in each image of the sequence, called the first sequence of images, a cutting window; downsampling, for each image of the first sequence, the image portion delimited by the clipping window so as to construct a second sequence of images; and - encoding the second image sequence as the scalable flow base layer and the first image sequence as the scalable flow enhancement layer.

According to a first aspect of the invention, the clipping window is determined in an image of the first sequence according to the following steps: determining in the image a first clipping window; determining a second clipping window surrounding the first clipping window so that the second clipping window conforms to at least one predefined constraint imposed by the coding step, the clipping window being the second clipping window. According to another advantageous aspect of the invention, the second clipping window is the smallest clipping window surrounding the first clipping window and conforms to the at least one predefined constraint imposed by the coding step. According to a third advantageous aspect of the invention, the coding method further comprises a step of encoding, in the base layer, data representative of the distance of each of the edges of the first cutting window with respect to each of the edges. corresponding to the clipping window surrounding the first clipping window. The invention also relates to a device for encoding an image sequence in the form of a scalable stream comprising a base layer and an enhancement layer. The coding device according to the invention comprises: - cropping means for determining in each image of the sequence, called the first sequence of images, a cutting window; subsampling means for downsampling, for each image of the first sequence, the image portion delimited by the clipping window so as to construct a second sequence of images; and coding means for coding the second image sequence as a base layer of the scalable flow and the first image sequence as a scalable flux enhancement layer.

Preferably, the means for reframing the coding device comprise: first means for determining in an image of the first sequence a first clipping window; and second means for determining a second clipping window surrounding the first clipping window so that the second clipping window conforms to at least one predefined constraint imposed by the coding means, the clipping window being the second window cutting.

4. LIST OF FILES The invention will be better understood and illustrated by means of examples of advantageous embodiments and implementations, in no way limiting, with reference to the appended figures in which: FIG. 1 is a diagrammatic representation of FIG. a coding device according to the state of the art; FIG. 2 represents an initial image 10 and an image 11 generated from the image 10 by a subsampling method according to the state of the art; FIG. 3 represents an initial image 10 and an image 11 generated from the image 10 by a subsampling method according to a variant of the sub-sampling method of the state of the art; FIG. 4 is a schematic representation of a coding device according to the invention; Figure 5 is a diagram of a coding method according to the invention; FIG. 6 is a diagram illustrating the coding method according to the invention; FIG. 7 represents an image of the base layer generated according to the invention and a cutting window in said image; FIG. 8 is a schematic representation of a display device 25 according to the invention; and FIG. 9 is a diagram of a display method according to the invention.

5. DETAILED DESCRIPTION OF THE INVENTION With reference to FIG. 4, the invention relates to a coding device 30 capable of coding a sequence SO of initial images I0 in the form of a scalable flow F comprising a base layer. and at least one enhancement layer. An image sequence includes two or more images of two images. The modules of the coding device 2 according to the invention identical to those of the coding device 1 according to the state of the art are identified in FIG. 4 with the same reference numerals as those used in connection with FIG. In Figure 4, the represented modules are functional units, which may or may not correspond to physically distinguishable units. For example, these modules or some of them may be grouped into a single component, or be functionalities of the same software. On the other hand, some modules may be composed of separate physical entities. The coding device 2 according to the invention comprises the input 8 for receiving a sequence SO of initial images I0 and a sub-sampling module 20 able to sub-sample a part of each image 10 of said initial sequence SO to generate a S1 sequence of modified images 11 of lower resolution than that of the initial images 10. The portion of IO image in question is predetermined by the subsampling module 20 before the subsampling itself. The coding device 2 furthermore comprises the first coding module 12 capable of coding the images of the modified sequence S1 into a first stream F1 known as the base layer and the second coding module 14 able to encode the images of the initial sequence SO into a second FO stream known as an enhancement layer from, in particular, coding data derived from the coding of the images of the modified sequence S1. The coding device 2 furthermore comprises connected to the outputs of the first and second coding modules 12 and 14, the multiplexing module 16 able to multiplex the base layer F1 and the enhancement layer FO in a single stream F.

The subsampling module 20 is adapted relative to the corresponding module 10 of the coding device 1 according to the state of the art. The subsampling module 20 is able to implement the steps E10 and E16 of the coding method according to the invention described hereinafter with reference to FIG. 5.

In step E10, each initial image I0 of the sequence SO is subsampled by the module 20 of the coding device 2. For this purpose, the step E10 is itself divided into a series of steps numbered E101 to E103. At step El 01, a first clipping window WO is determined in each image 10. Such a window, represented in FIG. 6, is preferably determined by an automatic reframing method as known in the state of the art. A simple example of such an automatic cropping method consists in calculating, for a given image, a saliency map representative of the perceptual interest of each of the zones of the image in question, it then positions a window of predefined size smaller than the size of the initial image so that it is centered on the point of the most salient image An example of a more complex automatic reframing method is described in the patent application EP 06117768.9 published on 31.01.2007 under the number The invention is however in no way limited to one or another automatic cropping method According to one variant, the first cutting window WO is determined by ns a first frame of the sequence according to an automatic cropping process. According to this variant, the first clipping window WO is determined, in each of the other images of the sequence, as being that which has the same position as that of the first clipping window WO in the first image. According to another variant, the first clipping window WO is determined according to an automatic cropping method in a first image and a second image of the remote sequence of the first image. According to this variant, the first clipping window WO is determined in each intermediate image, i.e. each image of the sequence between the first and the second image, by a method of interpolation of the position of the clipping window. Step E101 makes it possible to define a relevant WO cutting window from a visual content point of view, i.e. comprising the information of the IO image most visually relevant to the user. The window thus defined is visually optimal, but does not necessarily respond to constraints imposed by coding modules 12 and 14. In step E102, the first clipping window WO is extended to form a second clipping window. W1 which surrounds the first clipping window WO and in such a way as to respond to the constraints imposed by the coding modules 12 and 14. Preferably, the second clipping window W1 is the smallest window satisfying the criteria defined above, namely the smallest window which surrounds the first cutting window WO and which responds to the constraints imposed by the coding modules 12 and 14. FIG. 6 illustrates a particular embodiment. In this figure, the second clipping window W1 is the smallest window that surrounds the first clipping window WO and which is aligned with the grid of blocks of pixels of the constrained IO image imposed by the coding modules 12 and 14. In step E103, a modified image 11 is generated by downsampling the image portion I0 included in the second clipping window W1. This step is repeated for all the images of the SO sequence. Any subsampling method of the state of the art can be used for this purpose, for example a method based on bilinear filtering or on filtering by an MPEG4 filter with coefficients {-1,0,9,16 }. In step E12, the modified images 11 are encoded into a stream F1, i.e. the base layer. According to a particular embodiment, the modified images 11 are coded in accordance with the H.264 / MEPG-4 AVC video coding standard as defined in ISO / IEC 14496-10. This encoding step known to those skilled in the art of video encoders is not described further. In step E14, the initial images I0 are encoded from encoding data generated by the encoding step E12 into a stream F0, i.e. the enhancement layer. Indeed, the image data of the pixels of the images I0 located in the clipping window WO are highly correlated with the image data of the pixels of the corresponding images 11. It is therefore particularly advantageous to code the initial images I0 from coding data generated by the coding step E12. According to a particular exemplary embodiment, the initial IO images are coded in accordance with the video coding standard SVC, scalable extension of MPEG-4 AVC, as defined in the document ISO / IEC 14496-10 amendment 3. This scalable extension also was published as JVT-X201 at the joint ISO / IEC MPEG JVT working group meeting and ITU-T VCEG held in Geneva from June 29 to July 5, 2007. coding known to those skilled in the art of video encoders is not described further. In step E16, the streams FO and F1 are multiplexed so as to form a single scalable stream F transmitted via the output 18. Such multiplexing step is known to those skilled in the art of video encoders is not described more. We can refer in particular to the document JVTX201. According to a particularly advantageous variant, the distances d1, d2, d3 and d4 of each of the edges of the subsampled version of the first cutting window in the image 11 at each of the edges of image 11 are also encoded in the base layer F1 at step E12. These distances are shown in Figure 7 using arrows. According to a particular embodiment, the scalable flow F is in accordance with the SVC standard. In this case, the modified images 11 are encoded in step E12 according to the MPEG 4 AVC standard and the initial images I0 are encoded in step E14 according to the SVC standard. In this particular case, the distances d1, d2, d3 and d4 defined previously are coded in the base layer using the cutting tools defined in point 7.4.2.1 of ISO / IEC 14496-10: 2004 or in the document JVT-X201 in section 7.4.2.1.1 entitled Sequence parameter set semantics namely: - frame_cropping_flag; - frame_crop_left_offset, frame_crop_right_offset, frame_crop_top_offset and fram e_crop_bottom_offset.

In connection with FIG. 8, the invention also relates to a display device 3 adapted to a reduced version of the initial sequence SO of images 10. In FIG. 8, the modules represented are functional units, which can or not correspond to physically distinguishable units. For example, these modules or some of them may be grouped into a single component, or be functionalities of the same software. On the other hand, some modules may be composed of separate physical entities. For this purpose, the display device according to the invention comprises an input 30 on which it receives the scalable flow F. It further comprises a module 32 able to extract from the scalable flow F the base layer F1, a decoding module 34 able to decode the base layer F1 to reconstruct images Ir1 and also distances d1, d2, d3 and d4. It further comprises a display module 36 connected to a screen 38 capable of selecting in each of the reconstructed images Ir1 the image portion delimited by the cutting window WO which will have been previously defined using the distances d1, d2 , d3 and d4.

In connection with FIG. 9, the invention also relates to a display method. In step E20 of the display method, the base layer F1 is extracted from the scalable flow F by the module 32 of the display device 3. In step E22, the base layer F1 is decoded and the images Ir1 are reconstructed by the decoding module 34. During this same step the distances d1, d2, d3 and d4 are also decoded. In step E24, in each of the reconstructed images Ir1, a clipping window WO is defined by the display module 36 using the distances d1, d2, d3 and d4 decoded in step E22. In step E26, the portion I'1 of each of the images Ir1 delimited by the clipping window WO is displayed on the screen 38 in place of the reconstructed images Ir1.

Claims (4)

Revendications1. A method of coding an image sequence in the form of a scalable stream comprising a base layer and an enhancement layer comprising the following steps: - determining (El01, E102) in each image of said sequence, said first sequence of images, a slicing window (W1); - subsample (E103), for each image of said first sequence, the image portion delimited by the clipping window (W1) so as to construct a second sequence of images; coding (E12, E14, E16) said second image sequence as a base layer of the scalable flow and said first image sequence as a scalable flux enhancement layer; said method being characterized in that said clipping window (W1) is determined in an image of said first sequence according to the following steps: - determining (E101) in said image a first clipping window (WO); and - determining (E102) a second clipping window surrounding said first clipping window (WO) so that said second clipping window conforms to at least one predefined constraint imposed by the encoding step (E12, E14, E16 ), said clipping window (W1) of said image being said second clipping window. The encoding method according to claim 1, wherein said second clipping window is the smallest clipping window surrounding said first clipping window (WO) and conforming to said at least one predefined constraint imposed by the encoding step (E12, E14, E16) An encoding method according to claim 1 or 2, which further comprises a step of encoding, in the base layer, data representative of the distance from each of the edges of said first splitting window (WO) with respect to each of corresponding edges of said clipping window (W1) surrounding said first clipping window (WO). An encoding device (2) of an image sequence in the form of a scalable stream comprising a base layer and an enhancement layer comprising: - cropping means (20) for determining in each image of said sequence, said first image sequence, a clipping window (W1); subsampling means (20) for downsampling, for each image of said first sequence, the image portion delimited by the clipping window (W1) so as to construct a second sequence of images; coding means (12, 14) for coding said second image sequence as a base layer of the scalable flow and said first image sequence as a scalable flux enhancement layer; said coding device (2) being characterized in that the cropping means (20) comprises: - first means for determining in an image of said first sequence a first clipping window (WO); and second means for determining a second clipping window surrounding said first clipping window (WO) so that said second clipping window conforms to at least one predefined constraint imposed by the coding means (12, 14), said clipping window (W1) of said image being said second clipping window.