JP7217226B2 - Methods, devices and streams for encoding motion-compensated images in global rotation - Google Patents

Description

1. TECHNICAL FIELD This disclosure relates to the field of motion compensated coding in immersive video, for example when such immersive video is processed in a system for virtual reality, augmented reality or augmented virtuality. and when displayed on, for example, a head-mounted display device.

2. Background Recently, there has been an increase in wide-field content (up to 360°) available. Such content is potentially fully viewable by users viewing the content on immersive display devices such as head-mounted displays, smart glasses, PC screens, tablets, smartphones, and the like. not a thing That means that at a given moment the user can only see part of the content. However, users can typically navigate within the content by various means such as head movements, mouse movements, touch screens, voice, and the like. Typically, it is desirable to encode and decode this content.

3. Overview According to an aspect of the present principles, the present disclosure relates to a method of encoding and embedding a first image of an image sequence into a stream. The method is
- obtaining a second image by applying motion compensation in a global rotation to the third image according to the rotation parameter, the third image processing the first image in a step of the compression process; obtained by, obtaining and
- encoding and embedding the first image and associated rotation parameters into the stream by using the second image as input image for a motion compensation module;

According to a particular property, the third image is the first image, so the motion compensation in global rotation is applied directly to the first image at the input of the compression loop.

According to a particular embodiment, the third image is generated by obtaining a residual image in response to the input first image and a motion estimation image in response to the first image during the compression process. and the third image is a reconstructed image obtained by adding the residual image to the motion estimation image.

The present disclosure also relates to a method of decoding a first image of an image sequence from a stream. The method is
- obtaining the associated second image and associated rotation parameters from the stream;
- decoding the first image by applying motion compensation in a global rotation to the third image according to said rotation parameter; The third image is the output image of the motion compensation module obtained from the second image during the decompression process.

According to a particular property, the third is the decoded first image, so a global motion rotation is applied to the decoded image at the output of the decompression loop.

According to a particular embodiment, the third image is obtained by obtaining a residual image in response to the second image and obtaining a motion estimation image in response to the second image; Image 3 is the reconstructed image obtained by adding the residual image to the motion estimation image during the decompression process.

The present disclosure also relates to bitstreams carrying data representing image sequences. Data is,
- a first element of a syntax representing an image sequence;
• a second element of the syntax representing a motion parameter in the parametric global rotation associated with the identifier of at least one image of the image sequence;

The present disclosure also relates to a device that encodes and embeds a first image of an image sequence into a stream. The device includes memory associated with at least one processor, the at least one processor comprising:
- obtaining a second image by applying motion compensation in a global rotation to the third image according to the rotation parameter, the third image processing the first image in a step of the compression process; obtained by, obtaining and
- encoding and embedding the first image and associated rotation parameters in the stream by using the second image as input image for a motion compensation module;

According to a particular embodiment, the third image is said first image, so the motion compensation in global rotation is applied directly to the first image at the input of the compression loop.

According to a particular embodiment, the third image is obtained by obtaining a residual image in response to the first image and obtaining a motion estimation image in response to the first image during the decompression process. and the third image is a reconstructed image obtained by adding the residual image to the motion estimation image.

The present disclosure also relates to a device for decoding a first image of an image sequence from a stream. The device includes memory associated with at least one processor, the at least one processor comprising:
- obtaining a second image and associated rotation parameters from the stream;
- decoding the first image by applying motion compensation in a global rotation to the third image according to said rotation parameter; The third image is the output image of the motion compensation module and is obtained from the second image in a step during the decompression process.

4. List of Figures The present disclosure will be better understood, and other specific features and advantages will become apparent, as the following description refers to the accompanying drawings.

1 depicts a functional overview of an encoding and decoding system according to an exemplary environment of embodiments of the present disclosure; 1 depicts a first embodiment of a system in accordance with certain embodiments of the present principles; 1 depicts a first embodiment of a system in accordance with certain embodiments of the present principles; 1 depicts a first embodiment of a system in accordance with certain embodiments of the present principles; 1 depicts a first embodiment of a system in accordance with certain embodiments of the present principles; 1 depicts a first embodiment of a system in accordance with certain embodiments of the present principles; 1 depicts a first embodiment of a system in accordance with certain embodiments of the present principles; 1 represents a first embodiment of a system according to the present disclosure; 1 represents a first embodiment of a system according to the present disclosure; 1 depicts a first embodiment of an immersive video rendering device according to the present disclosure; 1 depicts a first embodiment of an immersive video rendering device according to the present disclosure; 1 depicts a first embodiment of an immersive video rendering device according to the present disclosure; 4 shows an example of mapping an omni-directional video onto a frame with two different mapping functions, according to a particular embodiment of the present principles; 14 shows an example mapping of the omnidirectional video of FIG. 13 after panning, in accordance with certain embodiments of the present principles; 1 illustrates a method of encoding an image to be encoded by motion compensating an input image in response to the image to be encoded, in accordance with a particular embodiment of the present principles; 1 illustrates a method of decoding images of an image sequence by motion compensating an input image in response to an image to be decoded, in accordance with a particular embodiment of the present principles; 4 illustrates another method of decoding images of an image sequence by motion compensating an input image in response to an image to be decoded, in accordance with a particular embodiment of the present principles; 16 illustrates a particular embodiment of a bitstream data structure carrying data representing an image sequence encoded according to the method of FIG. 15, in accordance with a particular embodiment of the present principles; 16 illustrates a first embodiment of method 150 of FIG. 15, in accordance with certain embodiments of the present principles; 17 illustrates an embodiment of method 160 of FIG. 16, in accordance with certain embodiments of the present principles; 16 illustrates a second embodiment of method 150 of FIG. 15, in accordance with certain embodiments of the present principles; 18 illustrates an embodiment of method 170 of FIG. 17, in accordance with certain embodiments of the present principles; 24 configured to encode images of an image sequence by motion compensating an input image in response to an image to be encoded, according to a particular embodiment of the present principles; 1 shows a block diagram of a first embodiment of a device; FIG. 25 illustrates an embodiment of a device configured to decode a bitstream encoded according to the first embodiment of the method shown in FIG. 24, in accordance with certain embodiments of the present principles; 26 configured to encode images of an image sequence by motion compensating an input image in response to an image to be encoded, according to a particular embodiment of the present principles; Figure 2 shows a block diagram of a second embodiment of the device; 27 illustrates an embodiment of a device configured to decode a bitstream encoded according to the second embodiment of the method shown in FIG. 26, in accordance with certain embodiments of the present principles; 23 illustrates a hardware embodiment of an apparatus configured to implement the method described with respect to FIGS. 15, 16, 17, 19, 20, 21 or 22, in accordance with certain embodiments of the present principles; FIG.

5. DETAILED DESCRIPTION OF EMBODIMENTS Objects are now described with reference to the drawings, and like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the subject matter. It is understood that subject matter embodiments may be practiced without these specific details.

Wide-view content can be, among other things, a three-dimensional computer graphic image scene (3D CGI scene), a point cloud or an immersive video. Many terms can be used to design such immersive images, such as virtual reality (VR), 360, panoramic, 4π, steradian, immersive, omnidirectional or wide field of view.

Immersive video is encoded video on at least one rectangular frame that is a two-dimensional array of "regular" video-like pixels (ie, elements of color information). To render, first, the frame is mapped onto the inner surface (also called mapping surface) of a convex volume (eg, sphere, cube, pyramid), and second, a portion of this volume is captured by a virtual camera. Images captured by the virtual camera are rendered on the screen of the immersive display device. Stereoscopic images are encoded on one or two rectangular frames and projected onto two mapping surfaces, which are combined for capture by two virtual cameras according to device characteristics.

Pixels are encoded according to the mapping function in the frame. The mapping function depends on the mapping surface. For the same mapping surface, several mapping functions are possible. For example, the faces of a cube can be constructed according to different layouts within the frame surface. A sphere can be mapped according to, for example, an equirectangular projection or a recentric projection. The pixel organization resulting from the selected projection function modifies or destroys line continuity, orthogonal local frames, pixel density, and introduces periodicity in time and space. These are typical features used in video encoding and decoding. Encoding and decoding methods lack consideration of the specificity of immersive video. In practice, immersive video is 360° video, so panning, for example, introduces motion and discontinuities that do not change the content of the scene but require the encoding of large amounts of data. Considering the specificity of immersive video while encoding and decoding video frames will provide valuable advantages over state-of-the-art methods.

FIG. 1 shows a general overview of an encoding and decoding system according to an exemplary embodiment. The system of FIG. 1 is a functional system. Pre-processing module 300 may prepare content for encoding by encoding device 400 . The pre-processing module 300 performs multiple image acquisition, merging the acquired multiple images into a common space (typically a 3D sphere when encoding orientation), and, for example, but not limited to: Mapping the 3D sphere to the 2D frame can be performed using equirectangular mapping or cubic mapping. The pre-processing module 300 can also take as input omni-directional video in a particular format (eg, equirectangular) and pre-process the video to change the mapping to a format more suitable for encoding. Depending on the obtained representation of the video data, the pre-processing module 300 can perform mapping space modifications. The encoding device 400 and encoding method are described with respect to other figures herein. After encoding, the data (eg, immersive video data can be encoded or 3D CGI encoded data) is sent to network interface 500, which typically connects to any network. It can also be implemented in an interface (e.g. present in the gateway). The data is then transmitted over a communication network such as the Internet, although any other network can be envisioned. Data is then received via network interface 600 . Network interface 600 can be implemented in a gateway, television, set top box, head mounted display device, immersive wall or any immersive video rendering device. After reception, the data is transmitted to decoding device 700 . The decoding function is one of the processing functions described in FIGS. 2-12 below. The decoded data is then processed by player 800 . The player 800 can prepare data for the rendering device 900 and receive external data from sensors or user input data. More precisely, player 800 prepares a portion of the video content to be displayed by rendering device 900 . Decoding device 700 and player 800 can be integrated into a single device (eg, smart phone, game console, STB, tablet, computer, etc.). In a variant, player 800 is integrated into rendering device 900 .

For example, several types of systems can be envisioned for performing the decoding, playback and rendering functions of an immersive display device when rendering immersive video.

A first system for processing augmented reality, virtual reality or augmented virtuality content is shown in Figures 2-6. Such systems include processing capabilities, immersive video rendering devices (which may be, for example, head mounted displays (HMDs), tablets or smart phones) and may include sensors. Also, the immersive video rendering device may include additional interface modules between the display device and the processing functionality. Processing functions may be performed by one or several devices. The processing functionality can be integrated into the immersive video rendering device, or it can be integrated into one or several processing devices. The processing device includes one or several processors and a communication interface (such as a wireless or wired communication interface) with the immersive video rendering device.

The processing device may also include a second communication interface, either directly or through a network device (such as a home or local gateway), to access content located on wide access networks such as the Internet and clouds. The processing device may also access local storage through a third interface, such as an Ethernet-type local access network interface. In embodiments, a processing device may be a computer system having one or several processing units. In another embodiment, the processing device can be inserted into the housing of the immersive video rendering device, even a smart phone that can be connected to the immersive video rendering device through a wired or wireless link, through a connector or wirelessly to the immersive video rendering device. It can also be a smart phone that can communicate with the video rendering device. The communication interface of the processing device can be a wired interface (eg, a bus interface, a wide area network interface, a local area network interface) or a wireless interface (such as an IEEE 802.11 interface or a Bluetooth® interface).

When the processing functions are performed by the immersive video rendering device, the immersive video rendering device may be provided with an interface to a network, either directly or through a gateway, for receiving and/or transmitting content.

In another embodiment, the system includes an auxiliary device in communication with the immersive video rendering device and processing device. In such embodiments, the ancillary device may house at least one of the processing functions.

An immersive video rendering device may include one or several displays. A device may employ optics, such as lenses, in front of each of its displays. The display can also be part of an immersive display device, as is the case with smartphones or tablets. In another embodiment, the display and optics can be embedded in a helmet, glasses or visor that can be worn by the user. Immersive video rendering devices may also integrate several sensors, as will be explained later. An immersive video rendering device may also include a number of interfaces or connectors. Immersive video rendering devices may also include one or several wireless modules to communicate with sensors, processing capabilities, handhelds or other body part related devices or sensors.

The immersive video rendering device may also include processing functionality, which is executed by one or several processors and configured to decode content or otherwise process content. It is understood herein that by processing the content, all functions prepare the content for display. This may include, for example, decoding the content, merging the content before displaying the content, and modifying the content to fit the display device.

One function of an immersive content-rendering device is to control a virtual camera that captures at least a portion of content structured as a virtual volume. The system may include a pose tracking sensor that fully or partially tracks the user's pose (eg, the user's head pose) to process the pose of the virtual camera. Some positioning sensors are capable of tracking user movement. The system may also include other sensors related to the environment, for example to measure lighting, temperature or sound conditions. Such sensors may also be associated with the user's body, for example to measure perspiration or heart rate. Information obtained through these sensors can be used to process content. The system may also include user input devices (eg, mouse, keyboard, remote control, joystick). Information from the user input device can be used to process content, manage the user interface, or control the pose of the virtual camera. Sensors and user input devices communicate with the processing device and/or the immersive rendering device through wired or wireless communication interfaces.

Using FIGS. 2-6, some of the systems of this first type that display augmented reality, virtual reality, augmented virtual reality, or any content from augmented reality to virtual reality. Embodiments have been described.

FIG. 2 illustrates a particular embodiment of a system configured to decode, process and render immersive video. The system includes an immersive video rendering device 10, a sensor 20, a user input device 30, a computer 40 and a gateway 50 (optional).

The immersive video rendering device 10 shown in FIG. 10 includes a display 101 . The display is for example of the OLED or LCD type. The immersive video rendering device 10 is, for example, an HMD, tablet or smart phone. Device 10 may include a touch surface 102 (eg, a touchpad or tactile screen), a camera 103, memory 105 coupled to at least one processor 104, and at least one communication interface 106. At least one processor 104 processes signals received from sensors 20 . Some of the measurements from the sensors are used to compute the pose of the device and to control the virtual camera. Sensors used for pose estimation are, for example, gyroscopes, accelerometers or compasses. Also, more complex systems (eg, using camera rigs) can be used. In this case, at least one processor performs image processing to estimate the pose of device 10 . Some other measurements are used to process content according to environmental conditions or user reactions. Sensors used to observe the environment and the user are for example microphones, light sensors or touch sensors. Also, more complex systems can be used, such as, for example, a video camera that tracks the user's eyes. In this case, at least one processor performs image processing to manipulate the expected measurements. Data from sensors 20 and user input device 30 may also be transmitted to computer 40, which processes the data in accordance with these sensor inputs.

Memory 105 contains parameters and code program instructions for processor 104 . Memory 105 may also include parameters received from sensors 20 and user input devices 30 . Communication interface 106 allows the immersive video rendering device to communicate with computer 40 . The processing device's communication interface 106 can be a wired interface (eg, a bus interface, a wide area network interface, a local area network interface) or a wireless interface (such as an IEEE 802.11 interface or a Bluetooth® interface). Computer 40 transmits data and optionally controls commands to immersive video rendering device 10 . Computer 40 is responsible for processing data (ie, preparing data for display by immersive video rendering device 10). Processing may be performed exclusively by computer 40 or part of the processing may be performed by computer and another part may be performed by immersive video rendering device 10 . Computer 40 is connected to the Internet either directly or through a gateway or network interface 50 . Computer 40 receives data representing immersive video from the Internet, processes these data (e.g., decodes the data, and optionally renders a portion of the video content to be displayed by immersive video rendering device 10). ) and send the processed data to the immersive video rendering device 10 for display. Alternatively, the system may also include local storage (not displayed) in which data representing the immersive video is stored, said local storage being accessible on computer 40 or, for example, through a local area network. It can be on a local server (not shown).

FIG. 3 represents a second embodiment. In this embodiment, STB 90 is connected to a network, such as the Internet, either directly (ie, STB 90 includes a network interface) or via gateway 50 . STB 90 is connected to a rendering device such as television 100 or immersive video rendering device 200 through a wireless interface or through a wired interface. In addition to the classic functionality of a STB, STB 90 includes processing functionality to process video content for rendering on television 100 or any immersive video rendering device 200 . These processing functions are the same as those described for computer 40 and will not be described again here. Also, the sensors 20 and user input devices 30 are of the same type as previously described with respect to FIG. STB 90 obtains data representing immersive video from the Internet. In a variant, the STB 90 obtains the data representing the immersive video from local storage (not displayed) where the data representing the immersive video is stored.

FIG. 4 represents a third embodiment related to that represented in FIG. Game console 60 processes the content data. Game console 60 transmits data and optionally controls commands to immersive video rendering device 10 . Game console 60 is configured to process data representing immersive video and transmit the processed data to immersive video rendering device 10 for display. Processing may be performed exclusively by game console 60 or part of the processing may be performed by immersive video rendering device 10 .

Game console 60 is connected to the Internet either directly or through a gateway or network interface 50 . Game console 60 obtains data representing the immersive video from the Internet. In a variant, the game console 60 obtains the data representing the immersive video from local storage (not displayed) in which the data representing the immersive video is stored, said local storage being stored on the game console 60 or, for example, , on a local server (not shown) accessible through a local area network.

The game console 60 receives data representing immersive video from the Internet, processes this data (e.g., decodes the data and possibly prepares a portion of the video to be displayed), and renders it for display. send the processed data to the immersive video rendering device 10 for rendering. Game console 60 may receive data from sensors 20 and user input device 30 and use the data to process data representing immersive video obtained from the Internet or from local storage.

FIG. 5 represents a fourth embodiment of a system of the first type above, in which an immersive video rendering device 70 is formed by a smart phone 701 inserted into a housing 705 . The smart phone 701 is capable of connecting to the Internet and can therefore obtain data representing the immersive video from the Internet. In a variant, the smartphone 701 obtains the data representing the immersive video from a local storage (not displayed) where the data representing the immersive video is stored, said local storage being on the smartphone 701 or, for example, local It can be on a local server (not shown) accessible through an area network.

The immersive video rendering device 70 will be described with reference to FIG. 11 which provides a preferred embodiment of the immersive video rendering device 70 . Immersive video rendering device 70 optionally includes at least one network interface 702 and housing 705 for smartphone 701 . A smart phone 701 includes all functions of a smart phone and a display. A smartphone display is used as the immersive video rendering device 70 display. Therefore, displays other than the display of the smart phone 701 are not included. However, an optical system 704 such as a lens is included to view the data on the smartphone display. Smartphone 701 is configured to process (eg, decode and prepare for display) data representing immersive video, optionally according to data received from sensors 20 and from user input device 30 . Some of the measurements from the sensors are used to compute the pose of the device and to control the virtual camera. Sensors used for pose estimation are, for example, gyroscopes, accelerometers or compasses. Also, more complex systems (eg, using camera rigs) can be used. In this case, at least one processor performs image processing to estimate the pose of device 10 . Some other measurements are used to process content according to environmental conditions or user reactions. Sensors used to observe the environment and the user are for example microphones, light sensors or touch sensors. Also, more complex systems can be used, such as, for example, a video camera that tracks the user's eyes. In this case, at least one processor performs image processing to manipulate the expected measurements.

FIG. 6 represents a fifth embodiment of the first type of system described above, in which an immersive video rendering device 80 contains all functions for processing and displaying data content. The system includes an immersive video rendering device 80 , sensors 20 and user input devices 30 . Immersive video rendering device 80 is configured to process (eg, decode and prepare for display) data representing immersive video, optionally in accordance with data received from sensors 20 and from user input device 30. be done. The immersive video rendering device 80 can connect to the Internet and thus obtain data representing the immersive video from the Internet. In a variation, the immersive video rendering device 80 obtains data representing the immersive video from local storage (not displayed) in which the data representing the immersive video is stored, said local storage being stored on the rendering device 80, Or, for example, it may be on a local server (not shown) accessible through a local area network.

An immersive video rendering device 80 is shown in FIG. The immersive video rendering device includes display 801 . The display can be, for example, of the OLED or LCD type, a touchpad (optional) 802, a camera (optional) 803, a memory 805 connected to at least one processor 804 and at least one communication interface 806. . Memory 805 contains parameters and code program instructions for processor 804 . Memory 805 may also include parameters received from sensors 20 and user input devices 30 . The memory can also be large enough to store data representing the immersive video content. Thus, there may be several types of memory, and memory 805 may be a single memory or some type of storage (SD card, hard disk, volatile or non-volatile memory...). Communication interface 806 allows the immersive video rendering device to communicate with an Internet network. Processor 804 processes data representing an image to display the image on display 801 . Camera 803 captures an image of the environment of the image processing step. Data is extracted from this step to control the immersive video rendering device.

A second system for processing augmented reality, virtual reality or augmented virtuality content is shown in FIGS. 7-9. Such systems include immersive walls.

FIG. 7 represents a second type of system. A second type of system includes a display 1000 that is an immersive wall that receives data from a computer 4000 . Computer 4000 can receive immersive video data from the Internet. Computer 4000 is typically connected to the Internet either directly or through a gateway 5000 or network interface. In a variant, the immersive video data is obtained by the computer 4000 from local storage (not shown) in which data representing the immersive video is stored, said local storage being within the computer 4000 or, for example, a local area. It can be in a local server (not shown) accessible through the network.

The system may also include sensors 2000 and user input devices 3000 . The immersive wall 1000 can be of OLED or LCD type. The immersive wall 1000 may be equipped with one or several cameras. The immersive wall 1000 can process data received from the sensor 2000 (or multiple sensors 2000). The data received from sensor 2000 may relate to lighting conditions, temperature, the user's environment (eg, object position).

Immersive wall 1000 may also process data received from user input device 3000 . User input device 3000 transmits data, such as haptic signals, to provide feedback on the user's emotions. Examples of user input devices 3000 are handheld devices such as smart phones, devices with remote control and gyroscope capabilities.

Sensor 2000 and user input device 3000 data may also be transmitted to computer 4000 . Computer 4000 can process video data (eg, decode video data and prepare video data for display) according to data received from these sensors/user input devices. Sensor signals may be received through the communication interface of the immersive wall. This communication interface may be of the Bluetooth type, of the WIFI type, or of any other type of connection, preferably wireless, but may also be a wired connection.

Computer 4000 sends processed data to immersive wall 1000 and optionally controls commands. Computer 4000 is configured to process data to be displayed by immersive wall 1000 (ie, prepare data for display). Processing may be performed exclusively by computer 4000 , or part of the processing may be performed by computer 4000 and another part may be performed by immersive wall 1000 .

FIG. 8 represents another system of the second type. The system includes an immersive (projective) wall 6000, which is configured to process (eg, decode data and prepare data for display) and display video content. be. The system further includes sensors 2000 and user input devices 3000 .

The immersive wall 6000 receives immersive video data from the Internet through the gateway 5000 or directly from the Internet. In variations, the immersive video data is obtained by the immersive wall 6000 from local storage (not shown) in which data representing the immersive video is stored, said local storage being within the immersive wall 6000 or, for example, It can be in a local server (not shown) accessible through a local area network.

The system may also include sensors 2000 and user input devices 3000 . The immersive wall 6000 can be of OLED or LCD type. The immersive wall 6000 may be equipped with one or several cameras. Immersive wall 6000 can process data received from sensor 2000 (or sensors 2000). The data received from sensor 2000 may relate to lighting conditions, temperature, the user's environment (eg, object position).

Immersive wall 6000 may also process data received from user input device 3000 . User input device 3000 transmits data, such as haptic signals, to provide feedback on the user's emotions. Examples of user input devices 3000 are handheld devices such as smart phones, devices with remote control and gyroscope capabilities.

The immersive wall 6000 can process video data (eg, decode video data and prepare video data for display) according to data received from these sensors/user input devices. Sensor signals may be received through the communication interface of the immersive wall. This communication interface may be of the Bluetooth type, of the WIFI type, or of any other type of connection, preferably wireless, but may also be a wired connection. Immersive wall 6000 may include at least one communication interface to communicate with sensors and the Internet.

Figure 9 shows a third embodiment in which an immersive wall is used for the game. One or several game consoles 7000 are connected to the immersive wall 6000, preferably through a wireless interface. The immersive wall 6000 receives immersive video data from the Internet through the gateway 5000 or directly from the Internet. In variations, the immersive video data is obtained by the immersive wall 6000 from local storage (not shown) in which data representing the immersive video is stored, said local storage being within the immersive wall 6000 or, for example, It can be in a local server (not shown) accessible through a local area network.

Game console 7000 sends commands and user input parameters to immersive wall 6000 . Immersive wall 6000 optionally processes immersive video content according to input data received from sensors 2000, user input devices 3000 and game console 7000 to prepare the content for display. The immersive wall 6000 may also include internal memory that stores content to be displayed.

According to non-limiting embodiments of the present disclosure, methods and devices are disclosed for encoding and embedding images of wide-field video into a stream. Also disclosed is a method and device for decoding images of wide-field video from a stream. Also disclosed is a stream syntax for encoding images for wide-field video.

FIG. 13 shows an example of mapping an omnidirectional video to a frame with two different mapping functions. A 3D scene (here a hotel hall) is projected onto a spherical mapping surface 130 . A forward direction is chosen to map the surface to the frame. The forward direction may correspond to the portion of the content displayed in front of the user's eyes when rendered on the immersive video rendering device, as illustrated in FIGS. 2-12. In the example of FIG. 13, the forward direction faces the window on which the "A" is printed. The revolving door with a "B" printed on it is on the left in the forward direction. The preprocessing module of FIG. 1 performs mapping of projections 130 onto frames. Different mapping functions can be used to connect to different frames. In the example of FIG. 13, preprocessing module 300 produces a sequence of frames 131 according to an equirectangular mapping function applied to sphere 130 . In a variant, pre-processing module 300 performs a mapping space modification, converting sphere 130 to cube 132 and then mapping cube 132 to frame 133 according to cube layout 134 . The exemplary cube layout of FIG. 13 divides the frame into two rows with three squares. In the top row are the left, front and right faces of the cubes, and in the bottom row are the top, back and bottom faces of the cubes rotated 90°. Continuity is guaranteed in each row. The numbers in the cube layout 134 representation represent the connections of the edges of the cube.

Mapping a 3D surface onto one or more rectangular pictures inevitably introduces several effects that can affect compression efficiency when encoding the resulting video.

In practice, projection can introduce the following effects.
・ Strong geometry distortion:
- straight lines are no longer straight - orthonormal coordinate systems are no longer orthonormal - non-uniform pixel density: the pixels of the picture to be coded do not always represent the same surface of the surface to be coded ( For example, a sphere's pole may be represented by a line of pixels in a 2D image)
Strong discontinuities: picture layouts can introduce strong discontinuities between two adjacent pixels on the surface Some periodicity can occur in the picture (e.g. from one boundary to the opposite boundary)

Table 1 lists examples of such effects on various mapping functions. Some of these effects appear in video frames 131, 133 of FIG. 13 and frames 141, 143 of FIG.

FIG. 14 shows an example mapping of the omnidirectional video of FIG. 13 after panning. The forward orientation of the 3D scene has been rotated (144). The 3D scene is the same as in Figure 13, only the forward direction is rotated. In the example of FIG. 14, rotation corresponds to panning to the left. The forward direction of the sphere 140 faces the aisle, the window printed with "A" is on the left side, and the revolving door printed with "B" is on the rear side. Mapping sphere 140 with an equirectangular mapping function produces video frame 141 . In a variant, the pre-processing module performs a mapping space modification, transforming the sphere 140 into a cube 142, and then maps the cube 142 onto the frame 143 according to the cube layout 134 of FIG. A 3D rotation 144 of the mapping surface is indicated using Euler angle notation (α, β, γ). Let (α(t), β(t), γ(t)) denote the forward orientation in the 3D scene at time (t). The angles are set according to the reference frame. The forward direction can be expressed according to an absolute frame of reference (eg, the north direction in horizontal planning). In a variant, the forward direction associated with the image (further called motion parameter in global rotation) is set in the reference frame relative to the forward direction of the reference image. In such variations, the angle associated with an image corresponds to the difference (eg, subtraction) between its own forward direction and the forward direction of the reference image.

These angles correspond to the orientation of the acquisition system, which can be real or virtual. Acquisition system sends an angle representing the orientation in the forward direction to encoding module 400 associated with each frame. In a variant, the forward orientation angle is only transmitted when rotation of the acquisition system is detected. In another variation, the forward direction angle is sent to encoding module 400 at regular frequencies (eg, every 10 frames or every 24 frames). In another embodiment, the acquisition system does not rotate. The forward direction is managed by an editing module (not shown in FIG. 1) intermediate the acquisition system and encoding module 400 . The editing module sends to the encoding module 400 the forward orientation angle modified by a human operator or algorithm in association with each frame, or each time a rotation is detected, or at a regular frequency, according to the selected variant. . In another embodiment, preprocessing module 300 processes two mapped surfaces (eg, spheres 130, 140) or two frames (eg, frames 131, 141 or 133, 143) generated according to a selected mapping function. ) to calculate the forward angle. A change in the forward angle corresponds to a movement in the global rotation of the frame. For omnidirectional video, motion in global rotation does not modify the content information of the image, since the entire scene is captured. Frames 131, 141 (reciprocally 133, 143) represent the same 3D scene and contain the same color information (up to grid interpolation) encoded from different forward directions. However, the pixels in pictures 131 and 141 are significantly different. Encoding (according to video compression standard methods) of frame sequences containing motion (e.g., panning) in global rotations implies computation of motion models, which is computationally demanding, and pre-coded 3D scenes. remains the same, but a large amount of data is required.

HEVC, H. Video compression standards such as H.264/AVC exploit temporal redundancy to reduce the information to be encoded. It is based on inter-prediction, which operates motion compensation (MC) of reconstructed blocks of pictures corresponding to previous instants or other instants. Pixel _p2 of picture P2 at _{time t2} corresponds to pixel _p1 of picture _P1 at _time t1.
[ _eq1 ] _{p2 ( x2} , y2, t2) ₌ p1 ₍ x1, _y1 , _t1 )

_The pixel displacement between P1 and P2 can be decomposed as in equation _eq2 .
[ _eq2 ] p2 _{( x2} , y2, t2) _{-p1 (} x1, _y1 , _t1 ) = dp( _t1 , t2) _{+ gmc (} p, _t1 , t2)
where p is the pixel ₍ i.e. identifying p1 and p2 as the same pixel), dp is the apparent motion of the pixel due to the appropriate motion of the physical point in the 3D scene, and _gmc is , is the apparent motion of the pixel due to motion in global rotation. When the forward direction is rotated, the picture pixels change significantly even if the scene does not change (ie dp(t ₁ ,t ₂ )=(0,0)). This often happens in practice because the forward direction is often directed towards objects of interest in the scene and the viewpoint is adapted by rotating the reference frame.

In a non-limiting example embodiment of the present principles, a new image is generated by operating motion compensation on a global rotation relative to the input image. A motion parameter in global rotation (eg, the forward orientation angle associated with the input image) is compared (eg, subtracted) to a motion parameter in global rotation of the first image. The pixels of the input image are reorganized to represent the same content as that of the input image, as if the new image had been captured in the forward direction of the first image. According to this principle, the input image and the reference image have the advantage that they can be easily compared. In a first variant, the input image is the reference image of a compression loop (e.g. a compression loop of a compression standard such as HEVC or H.264/AVC) and the first image is the image to be encoded. . In another variant, the input image is the image to be encoded and the first image is the reference image of the compression loop. This embodiment is non-limiting.

FIG. 15 illustrates a method 150 for encoding image I1 of a sequence of images by motion compensating input image I2 in response to image I1 to be encoded. This method is implemented in encoding module 400 of FIG. In step 151, image I1 is prepared to become input image I2 for motion compensation step 152 in global rotation. Embodiments of step 151 are further described in connection with FIGS. In step 152, motion compensation in global rotation is applied to input image I2 according to the previously obtained motion parameter RP in global rotation (ie, the forward angle). Embodiments of step 152 are further described in connection with FIGS. Step 152 produces a motion compensated image I3 in global rotation. In step 153, image I3 is used as input for the motion compensation module, which in response to the predicted image produces residual data, motion vectors for the block and information data ID1 for the reference image. In a particular embodiment of the present principles, step 154 produces a bitstream S carrying data representing an image to be encoded. The motion parameter RP in global rotation is encoded in the stream in a lossless fashion.

FIG. 16 illustrates a method 160 for decoding images of an image sequence by motion compensating input images in response to images to be decoded. Data source S provides an encoded bitstream according to the first embodiment of method 150 of FIG. 15 (illustrated by FIG. 24). For example, the data source may be local memory (e.g., video memory, random access memory, flash memory, read-only memory, hard disk, etc.), storage interface (e.g., mass storage, interface with optical disk or magnetic support), and a set of sources that includes a communication interface, such as a wired interface (such as a bus interface, a wide area network interface, a local area network interface) or a wireless interface (such as an IEEE 802.11 interface or a Bluetooth® interface). . In step 161, image I4 is obtained from the stream and image I4 corresponds to the input of motion compensation step 162 in global rotation. Motion parameters in global rotation are obtained from the bitstream. An embodiment of step 161 is further described in connection with FIG. In step 162, image I5 is generated by applying motion compensation in global rotation to input image I4 according to motion parameter RP in global rotation. An embodiment of step 162 is further described in connection with FIG. In step 163, image I3 is used as input for the motion compensation module, which produces decoded image I6.

FIG. 17 illustrates a method 170 for decoding images of an image sequence by motion compensating input images in response to images to be decoded. Data source S provides an encoded bitstream according to the second embodiment of method 150 of FIG. 15 (described by FIG. 26). For example, sources include local memory (e.g., video memory, random access memory, flash memory, read-only memory, hard disk, etc.), storage interface (e.g., mass storage, interface with optical disk or magnetic support), and , a communication interface (eg, a wired interface (eg, a bus interface, a wide area network interface, a local area network interface) or a wireless interface (such as an IEEE 802.11 interface or a Bluetooth® interface)). In step 171, residual data, motion vectors and information data ID2 for image references are obtained from the bitstream in response to the image to be decoded, and motion parameters in global rotation are obtained from the bitstream. In step 172, image I7 is decoded by adding the residual image with the motion estimation image obtained according to information data ID2. An embodiment of step 172 is further described in connection with FIG. In step 173, image I8 is generated by applying motion compensation in global rotation to decoded image I7 according to the motion parameters in global rotation. An embodiment of step 173 is further described in connection with FIG.

FIG. 24 illustrates a first embodiment of method 150 of FIG. In this method, HEVC, H. A compression loop is started according to a video compression standard such as H.264/AVC. Some of the images of the sequence are used as reference images and stored in memory. Step 151 of method 150 includes three sub-steps 241, 242, 243 in this embodiment. At step 241, a residual image is calculated by comparing (eg, subtracting) the image to be encoded with a reference image. According to well-known processes, at step 242 a motion estimation process is used to obtain motion-predicted images. The process uses motion in the reference image to minimize the error computed between the pixels of the image to be encoded and the reference image, or to minimize another metric such as the rate-distortion cost. Search for motion data such as vectors. At step 243, a reconstructed image is generated by adding the residual image and the motion estimation image. The output of step 243 corresponds to the output of step 151 . At step 152, motion compensation in global rotation is performed on the reference image according to the GRM parameters of the image to be encoded. In a variant, the image is compensated for angular differences between the RM parameters of the image to be coded and the GRM parameters of the reference image. Step 153 includes motion compensating the image to be encoded according to the motion compensation reference image in global rotation. Since the reference images are motion compensated in global rotation, step 153 is facilitated and the computed motion model requires only a small amount of bit encoding. At step 154, the prepared image to be encoded is entropy encoded. The entropy-encoded images of the sequence form the first element of the bitstream 180 syntax, which is stored in the payload portion 182 of the bitstream 180 . Motion parameters for global rotation are included in the second element of the syntax of the bitstream, said second element of syntax being included in the header portion 181 of the bitstream 180 . The header portion 181 is encoded in a lossless format.

FIG. 25 illustrates an embodiment of method 160 of FIG. In this method, HEVC, H. A decompression loop is started according to a video compression standard such as H.264/AVC. Step 161 of method 160 includes two sub-steps 251, 252 in this embodiment. At step 251 the residual image is entropy decoded and a motion model is obtained from the bitstream. A motion prediction image is calculated according to the motion model. Some of the images extracted from the bitstream are used as reference images and stored in memory. In response to the image to be decoded, the image is reconstructed, for example by adding the residual image and the motion compensated image. At step 252, the GRM parameters associated with the image to be decoded are obtained from the bitstream. Reference images stored in memory are also associated with their GRM parameters as previously decoded images. At step 162, motion compensation in global rotation is performed on the reference image associated with the image to be decoded according to the GRM parameters of the image to be decoded. In a variant, the image is compensated for angular differences between the RM parameters of the image to be coded and the GRM parameters of the reference image. Step 163 includes motion compensating the image to be decoded according to the motion compensation reference image in global rotation. Step 163 is facilitated because the reference image is motion compensated in global rotation.

FIG. 26 illustrates a second embodiment of method 150 of FIG. In this embodiment of method 150, step 151 is to obtain the GRM parameters associated with the frame and image to be encoded. A compression loop has been started and some images of the image sequence have been selected, have reference images and are stored in memory. A reference image is also associated with the GRM parameters. In step 152, the image to be encoded is used as an input image and motion compensated in global rotation according to the associated GRM parameters. The image is for example rotated by an angle, which is contained in the GRM parameters and is defined according to the absolute reference direction (eg the direction north of the horizontal plane). In a variant, the image to be encoded is associated with a reference image stored in memory and rotated by an angle calculated according to the difference between the GRM parameters of the image to be encoded and the GRM parameters of the reference image. In step 153, the image output in step 152 is converted to HEVC, H.264, . A compression loop configured according to a video compression standard such as H.264/AVC is entered. A motion compensation step is performed. According to well-known processes, motion estimation is performed to minimize the error computed between the pixels of the image I1 to be coded and the pixels of the reference image I3, or to minimize another metric such as the rate-distortion cost. For this purpose, the reference image I3 is searched for motion data, such as motion vectors. Since the image I1 to be encoded has been motion compensated in global rotation in step 152, the retrieval of motion data is expedited and the volume of data required for encoding is small. At step 154, the prepared image to be encoded is entropy encoded. The entropy-encoded images of the sequence form the first element of the bitstream 180 syntax, which is stored in the payload portion 182 of the bitstream 180 . Motion parameters for global rotation are included in the second element of the syntax of the bitstream, said second element of syntax being included in the header portion 181 of the bitstream 180 . The header portion 181 is encoded in a lossless format.

FIG. 27 illustrates an embodiment of method 170 of FIG. Data source S provides a bitstream that has been encoded according to the second embodiment of method 150 of FIG. 15 (described by FIG. 26). In this method, HEVC, H. A decompression loop is started according to a video compression standard such as H.264/AVC. Decompression and decoding of at least one reference image has already been performed. At least one reference image is stored in memory and associated with the GRM parameters extracted from the data source S. In step 171 information data for residual data, motion vectors and motion parameters in global rotation are obtained from the bitstream in response to the image to be decoded. In step 172 the image is decoded by adding the residual image with the motion estimation image obtained according to the information data. Images to be decoded are reconstructed according to well-known decompression processes including motion compensation processes relative to reference images stored in memory. In step 173, the reconstructed image is motion compensated in global rotation according to the motion parameters in global rotation associated with the image to be decoded. The reconstructed image is a three-dimensional angle equivalent to the absolute angle values (e.g., in a reference frame using the north direction of the local horizontal plane) encoded by the GRM parameters associated with the image to be decoded. only rotated. In a variant, the reconstructed image is the difference between the angle encoded in its GRM parameters and the angle encoded in the GRM parameters associated with the reference image used to reconstruct the image to be decoded. It is rotated by the corresponding 3D angle.

FIG. 18 shows a particular embodiment of the data structure of a bitstream 180 carrying data representing an image sequence encoded according to method 150 of FIG. The entropy-encoded images of the sequence form the first element of the bitstream 180 syntax, which is stored in the payload portion 182 of the bitstream 180 . Motion parameters for global rotation are included in the second element of the syntax of the bitstream, said second element of syntax being included in the header portion 181 of the bitstream 180 . The header portion 181 is encoded in a lossless format.

Figure 19 shows a block diagram of a first embodiment of a device for encoding images of an image sequence by motion compensating an input image in response to an image to be encoded. The modules of the device of FIG. 19 implement the steps of the method of FIG.

Only functional modules of the device related to temporal prediction based coding (ie, inter-coding) are shown in FIG. Other modules (not shown), well known in the prior art, implement intra-coding, for example, with or without spatial prediction.

An image I1 to be coded is taken as input by a module 191, which extracts a predicted image Ip from the current image I1, eg on a pixel basis, to produce a residual image Ir. Module 192 transforms and quantifies the residual image Ir. Transform T is, for example, a discrete cosine transform (DCT), a block-based transform or a wavelet-based transform. Module 193 implements the inverse operation (ie, inverse quantization IQ followed by inverse transform IT). Module 194 merges the image output from module 193 with the predicted image Ip, for example on a pixel basis, to produce a reconstructed (aliased decoded) image, the reconstructed image being , is stored in memory (module 195), which is the decoded picture buffer (DPB). Module 193 generates and stores image I2 in memory (module 195), which can be used as a reference input image for the temporal prediction module.

式中、関数ｆ及びＰｒｏｊは、画像Ｉ１及びＩ２のマッピングに従って定義された関数であり、Ｒ_ｔは、ＧＲＭパラメータ（α（ｔ），β（ｔ），γ（ｔ））に従って計算された回転行列であり、Ｒ_ｔ’は、ＧＲＭパラメータ（α（ｔ’），β（ｔ’），γ（ｔ’））に従って計算された回転行列である。画素ｎ（ｘ’，ｙ’）は、方程式［ｅｑ．４］に従ってポイントＭｔ’を参照入力画像Ｉ２に投影することによって計算される。
［ｅｑ．４］ ｎ（ｘ’，ｙ’）＝ｆ^－１（Ｐｒｏｊ（Ｍ_ｔ’））

Module 196 takes as input the reference input image I2 stored in memory 195 and a motion parameter in global rotation (angle of forward direction). The apparent motion of each pixel of the current image is calculated by a further module 197 according to the reference input I2 to which the motion in global rotation was first applied. The current image I1 to be coded has motion parameters (GRM parameters) in global rotation (α(t), β(t), γ(t)) and m(x,y) (pixels in this picture) associated with. The reconstructed image I2 from the decoded picture buffer (module 195) GRM parameters (α(t'), β(t'), γ(t')) and n(x',y') is this picture Now for pixels. A global motion-compensated reference picture is reconstructed by applying motion compensation to each pixel of I1 to obtain the corresponding pixel of I2 (as (x,y)→(x',y')). The coordinates (x', y') are computed by applying motion compensation in global rotation. A point Mt' is calculated. Mt' corresponds to the projection of the pixel m(x,y) represented in the reference frame of image I1 at instant t onto the reference frame of image I1 at instant t'. The equation [eq. 3] handles axis system change transformations.

where the functions f and Proj are the functions defined according to the mapping of the images I1 and I2 and R _t is the rotation calculated according to the GRM parameters (α(t), β(t), γ(t)) matrix and R _t' is the rotation matrix calculated according to the GRM parameters (α(t'), β(t'), γ(t')). Pixel n(x', y') is represented by the equation [eq. 4] by projecting the point Mt' onto the reference input image I2.
[eq. 4] n(x′,y′)=f ⁻¹ (Proj(M _t′ ))

The equation [eq. 5] express motion compensation in instantaneous global rotation. In a variation of the embodiment of FIG. 19, images I1 and I2 are encoded according to equirectangular mapping as illustrated by image 131 of FIG. 13 and image 141 of FIG. The equirectangular mapping defines the functions f and Proj as follows.

式中、ｗは、画像幅の３分の１であり、ｈは、画像の高さの半分である。逆関数ｆ^－１は、上記の方程式から明快である。

In a variation of the device embodiment shown in FIG. 19, images I1 and I2 are encoded according to a cubic mapping using layout 134 of FIG. 13 as illustrated by image 133 of FIG. 13 and image 143 of FIG. be. In this case, the functions f and Proj are defined as follows.

where w is one third of the image width and h is half the image height. The inverse function f ⁻¹ is straightforward from the above equation.

The inverse function Proj is straightforward from the above equations.

At module 197 shown in FIG. 19, motion compensation and prediction are performed on image I3 output from module 152 . A module 196 estimates at least one motion vector between the image to be coded I1 and the motion compensated reference image I3 in global rotation. According to well-known processes, motion estimation is performed to minimize the error computed between the pixels of the image I1 to be coded and the pixels of the reference image I3, or to minimize another metric such as the rate-distortion cost. For this purpose, the reference image I3 is searched for motion data, such as motion vectors. Reference image I3 has been motion compensated in global rotation by module 196, thus facilitating the retrieval of motion data. An image I1 to be encoded is encoded as a predicted image Ip.

Module 198 entropy encodes the quantity data and embeds it in a bitstream S of coded data. This module produces a stream as described in connection with FIG. In the first element of the syntax of the bitstream, the elements of the syntax form the payload portion 182 of the bitstream 180 and the modules use conventional CABAC (Context Adaptive Binary Arithmetic Coding) to represent the image sequence. Stores entropy-encoded data. In the second element of the syntax forming the header portion 181 of the bitstream 180, the module losslessly encodes motion parameters in a global rotation associated with the image identifier, among other data. .

FIG. 20 shows an embodiment of a device configured to decode a bitstream S encoded according to the first embodiment of method 150 shown in FIG. The bitstream S represents an image sequence and contains two elements of syntax as described in connection with FIGS. The modules of the device of FIG. 20 implement the steps of the method of FIG.

Module 201 stores the residual data for the encoded image sequence to be decoded and the syntax elements required for reconstruction of the decoded image (e.g., the coding mode, the mapping used to encode the image, the motion data or Implement CABAC to entropy decode the important maps, etc.). The module also extracts motion parameters in global rotation associated with each image.

According to a variant, not shown in FIG. 20, decoding the motion data includes estimating motion. The decoded video data for the image to be decoded is sent to module 202, which applies an inverse transform followed by inverse quantization. Module 202 is similar to module 193 of FIG. Module 202 is linked to module 203, which combines the residual image output from module 202 with the predicted image Ip, for example pixel by pixel, to generate a decoded image I6 representing the image to be decoded. merge with . The reconstructed image I6 is also stored in the memory 204. FIG.

Module 205 takes as input the reference input image I4 stored in memory 204, the motion parameters in global rotation associated with this image I4 and the GRM parameters associated with the current image to be decoded. The apparent motion of each pixel of the current image is calculated by a further module 206 according to the reference input I4 to which the motion in global rotation was first applied. A global motion-compensated reference picture is reconstructed by applying motion compensation to each pixel of I4 to obtain the corresponding pixel of I5 (as (x,y)→(x',y')). The coordinates (x',y') are computed by applying motion compensation in global rotation, as described in connection with module 196 of FIG. The functions f, f ⁻¹ , Proj and Proj ⁻¹ are selected according to the mapping used to encode the image.

Motion compensation and prediction are performed by module 206 on image I5 output from module 205 . Module 206 estimates at least one motion vector between the image to be decoded and the motion compensated reference image I5 in global rotation. According to well-known processes, motion estimation is performed on the reference image I5 to minimize another metric such as the error or rate-distortion cost computed between the pixels of the image to be decoded and the pixels of the reference image I5. At I5, motion data such as motion vectors are retrieved. Reference image I5 has been motion compensated in global rotation by module 205, thus facilitating the retrieval of motion data. The image to be decoded is encoded as a predicted image Ip.

Figure 21 shows a block diagram of a second embodiment of a device configured to encode images of an image sequence by motion compensating input images in response to images to be encoded. The modules of the device of FIG. 21 implement the steps of the method of FIG.

Module 211 is arranged to obtain image I1. Module 211 uses the motion reference parameters in the global rotation (e.g. the GRM parameters of the first picture of the sequence or the GRM parameters of the last picture coded only with intra-prediction and the GRM associated with picture I1 to be coded). parameters), perform motion compensation in global rotation by applying the functions f, f ⁻¹ , Proj and Proj ⁻¹ as described in connection with FIG. Image I1 is mapped according to the forward direction of the reference image. The pixel differences between the reference image and the GRM-compensated image I3 output from module 211 are due to the apparent motion of objects in the scene and no longer due to forward rotation. The GRM-compensated image I3 is motion-compensated by the structure of module 212 according to methods well known in the prior art and entropy-encoded by module 213 . Module 213 entropy-encodes the quantity data and embeds it in a bitstream S of coded data. This module produces a stream as described in connection with FIG. In the first element of the syntax of the bitstream, the elements of the syntax form the payload portion 182 of the bitstream 180 and the modules use conventional CABAC (Context Adaptive Binary Arithmetic Coding) to represent the image sequence. Stores entropy-encoded data. In the second element of the syntax forming the header portion 181 of the bitstream 180, the module losslessly encodes motion parameters in a global rotation associated with the image identifier, among other data. .

FIG. 22 shows an embodiment of a device configured to decode a bitstream S encoded according to the second embodiment of method 150 shown in FIG. The bitstream S represents an image sequence and contains two elements of syntax as described in connection with FIGS. The modules of the device of FIG. 22 implement the steps of the method of FIG.

Module 221 stores the residual data for the encoded image sequence to be decoded and the syntax elements required to reconstruct the decoded image (e.g., the coding mode, the mapping used to encode the image, the motion data or Use CABAC to entropy decode the important maps, etc.). The module also extracts motion parameters in the global rotation associated with each image. According to a variant, not shown in FIG. 20, decoding the motion data includes estimating motion. The decoded video data for the image to be decoded is sent to module 222, which applies inverse quantization followed by inverse transform. Module 222 is similar to module 193 of FIG. Module 222 is linked to module 223, which combines, for example, the residual image output from step 222 with the predicted image Ip, pixel by pixel, to generate a decoded image I7 representing the image to be decoded. merge with . The reconstructed image I7 is also stored in the memory 224. FIG. Motion compensation and prediction are performed on images output from memory 224 by module 225 . A module 226 estimates at least one motion vector between the image to be decoded and the motion compensated reference image in global rotation. According to well-known processes, motion estimation is performed using reference Search for motion data, such as motion vectors, in an image. Retrieval of motion data is facilitated because the reference image has been motion compensated in global rotation by module 225 . An image to be decoded is encoded as a predicted image Ip.

FIG. 23 shows a hardware embodiment of an apparatus 230 configured to implement the method as described in connection with FIGS. In this example, device 230 includes the following elements connected together by an address and data bus 231 which also carries clock signals:
- a microprocessor 232 (or CPU), for example a DSP (or digital signal processor);
- a non-volatile memory 233 of the ROM (Read Only Memory) type,
- random access memory or RAM (234),
- an I/O interface 235 that receives data for transmission from an application;
- a graphics card 236 that can embed random access memory registers;
- Power supply 237
including.

By way of example, the power supply 237 is external to the device. In each of the memories referred to, the term "register" as used herein refers to either a small area (a few bits) or a very large area (e.g. an entire program or a large amount of received or decoded data). can be equivalent to ROM 233 includes at least programs and parameters. ROM 233 may store algorithms and instructions for performing techniques in accordance with the present principles. Upon power up, the CPU 232 uploads the program to RAM and executes the corresponding instructions.

The RAM 234 contains the program (executed by the CPU 232 and uploaded after powering on the device 230) in registers, the input data in registers, the intermediate data of different states of the method in registers, and the Contains other variables in registers that are used for execution.

Implementations described herein may be implemented in, for example, a module or process, apparatus, software program, data stream, or signal of one of the methods 150, 160, 170. Even if discussed only in the context of a single form of implementation (e.g., only as a method or device), the implementation of the discussed features can also be implemented in other forms (e.g., programs). can. The device may be implemented in suitable hardware, software and firmware, which may be, for example, one of the components of the systems illustrated in Figures 2-12. The methods and their modules may be implemented by an apparatus (eg, processor, etc.), which, for example, generally refers to a processing device including, for example, computers, microprocessors, integrated circuits, or programmable logic devices. The processor also provides information between, for example, the components of the systems illustrated in FIGS. 2-12, for example, computers, cell phones, portable/personal digital assistants (“PDAs”), set-top boxes, and end-users. It also includes communication devices such as other devices that facilitate communication.

According to the example of encoding images of an image sequence by motion compensating an input image in response to an image to be encoded as shown in FIGS. obtained from the source. For example, the source is
- local memory (233, 234 or 236) (e.g. video memory or RAM (or random access memory), flash memory, ROM (or read only memory), hard disk),
- a storage device interface (235) (e.g. to interface with a mass storage device, RAM, flash memory, ROM, optical disk or magnetic support),
- a communication interface (235) (e.g. a wired interface (e.g. a bus interface, a wide area network interface, a local area network interface) or a wireless interface (such as an IEEE 802.11 interface or a Bluetooth® interface));
belongs to a set that contains

According to one particular embodiment, an algorithm implementing the steps of method 150 for encoding images of an image sequence by motion-compensating an input image in response to an image to be encoded implements these steps Stored in memory GRAM of graphics card 236 associated with device 230 . According to a variant, a portion of RAM (234) is allocated by CPU (232) for storing algorithms. These steps include local memory (e.g. video memory (234), RAM (234), ROM (233), flash memory (233) or hard disk (233)), storage interface (235) (e.g. mass storage device, RAM, ROM, flash memory, optical disk or magnetic support) and/or communication interface (235) (e.g., point-to-point link, bus, point-to-multiple It provides the generation of the video stream received from the point link or interface with the broadcast network).

According to the example of decoding images of an image sequence by motion compensating input images in response to images to be decoded, a bitstream representing the image sequence is obtained from a source. Illustratively, the bitstream is read from local memory (eg, video memory (234), RAM (234), ROM (233), flash memory (233) or hard disk (233)). In variations, the stream is received from a storage interface (235) (e.g., an interface with mass storage, RAM, ROM, flash memory, optical disk or magnetic support) and/or a communication interface (235). (eg, a point-to-point link, a bus, a point-to-multipoint link, or an interface with a broadcast network).

According to one particular embodiment, the algorithm implementing the steps of the method for decoding images of an image sequence by motion compensating an input image in response to an image to be decoded comprises a device 230 implementing these steps and Stored in the memory GRAM of the associated graphics card 236 . According to a variant, a portion of RAM (234) is allocated by CPU (232) for storing algorithms. These steps are, for example,
- mobile devices,
- communication devices,
- gaming devices;
- set-top boxes - televisions - tablets (or tablet computers),
- Laptop,
- a display;
- decoding chips,
, etc., resulting in the composition of the video sent to the destinations belonging to the set containing the components of the system described in FIGS. 2-12.

Of course, the disclosure is not limited to the previously described embodiments.

Specifically, the present disclosure is not limited to methods of encoding and decoding image sequences, but any method of displaying the decoded video, such as the display devices of FIGS. extends to any device that implements the method of The implementation of the computations required for bitstream encoding and decoding is not limited to shader-type microprogram implementations, but any program-type implementation, e.g., a program that can be executed by a CPU-type microprocessor. Extends to implementation. The use of the method of the present disclosure is not limited to use during filming, but extends to any other use, such as what is known as post-production processing in a recording studio, for example.

Implementations described herein can be implemented as, for example, methods or processes, apparatuses, software programs, data streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., only as a method or device), the implementation of the discussed features can also be implemented in other forms (e.g., programs). can. The device can be implemented with suitable hardware, software and firmware, for example. The methods can be implemented in an apparatus (eg, processor, etc.), which, for example, generally refers to a processing device including, for example, a computer, microprocessor, integrated circuit or programmable logic device. Processors also include communication devices such as, for example, smart phones, tablets, computers, cell phones, portable/personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.

Implementations of the various processes and features described herein may be used in a variety of different devices or applications (e.g., data encoding, data decoding, view generation, texture processing, and, among other things, images, associated texture information and/or or in equipment or applications associated with other processing of depth information). Examples of such equipment are encoders, decoders, post-processors that process the output from the decoders, pre-processors that provide input to the encoders, video coders, video decoders, video codecs, web servers, set-top boxes, laptops, personal Including computers, cell phones, PDAs and other communication devices. As should be clear, the device can be mobile and even installed in a moving vehicle.

Additionally, methods may be implemented by instructions being executed by a processor, such instructions (and/or data values generated by an implementation) being stored in, for example, integrated circuits, software carriers, or other methods. storage devices (e.g., hard disks, compact discs ("CDs"), optical discs (e.g., DVDs, often referred to as digital versatile discs or digital video discs), random access memory ("RAM"), or read-only memory (" ROM")). The instructions may form an application program tangibly embodied on the processor-readable medium. Instructions may be, for example, in hardware, firmware, software, or a combination. Instructions can be found, for example, in the operating system, separate applications, or a combination of the two. Thus, a processor, for example, can be characterized as both a device configured to execute a process and a device that includes a processor-readable medium (such as a storage device) having instructions for executing the process. Further, a processor-readable medium can store data values generated by an implementation in addition to or instead of instructions.

Implementations can generate a variety of signals formatted to convey information that can be stored or transmitted, for example, as will be apparent to those skilled in the art. Information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, signals may be formatted to convey as data rules for writing or reading the syntax of the described embodiments, or to convey as data the actual syntax/values written by the described embodiments. can be done. Such signals can be formatted, for example, as electromagnetic waves (eg, using the high frequency portion of the spectrum) or as baseband signals. Formatting may include, for example, encoding the data stream and modulating a carrier with the encoded data stream. The information that the signal conveys can be, for example, analog or digital information. Signals, as is known, can be transmitted over a variety of different wired or wireless links. A signal can be stored on a processor-readable medium.

A number of implementations have been described. Nevertheless, it will be appreciated that various modifications can be made. For example, elements of different implementations may be combined, supplemented, modified, or removed to produce other implementations. Additionally, one skilled in the art may substitute other structures and processes for those disclosed, and the resulting implementation achieves at least substantially the same results as the disclosed implementation. to perform at least substantially the same function in at least substantially the same manner. Accordingly, these and other implementations are contemplated by this application.
(Appendix 1)
A method for encoding and embedding a first image (I1) of an image sequence into a stream, comprising:
obtaining a second image (I3) by applying motion compensation in a global rotation to the third image (I2) according to a rotation parameter, said third image (I2) being equivalent to said first obtained from an image;
encoding and embedding in said stream said first image (I1) and a rotation parameter associated with said first image by using said second image (I3) as an input image for a motion compensation module; and
A method, including
(Appendix 2)
12. The method of clause 1, wherein the third image is the first image.
(Appendix 3)
The third image is
obtaining a residual image in response to the first image;
obtaining a motion-predicted image in response to the first image;
and wherein the third image is a reconstructed image obtained by adding the residual image to the motion estimation image.
(Appendix 4)
A method for decoding a first image of an image sequence from a stream, comprising:
obtaining a second image from the stream and a rotation parameter associated with the second image;
decoding said first image by applying motion compensation in a global rotation according to said rotation parameter to a third image (I4, I7);
wherein said third image is an output image of a motion compensation module, said output image being obtained from said second image.
(Appendix 5)
5. The method of Clause 4, wherein the third image is the decoded first image.
(Appendix 6)
The third image is
obtaining a residual image in response to the second image;
obtaining a motion-predicted image in response to the second image;
and wherein said third image is a reconstructed image obtained by adding said residual image to said motion estimation image.
(Appendix 7)
A bitstream carrying data representing an image sequence, said data comprising:
a first element of syntax representing the image sequence;
a second element of syntax representing a motion parameter in a parametric global rotation associated with an identifier of at least one image of said image sequence;
A bitstream comprising:
(Appendix 8)
A device for encoding and embedding a first image (I1) of an image sequence into a stream, comprising a memory associated with at least one processor, said at least one processor:
obtaining a second image (I3) by applying motion compensation in a global rotation to the third image (I2) according to a rotation parameter, said third image (I2) being equivalent to said first obtained from an image;
encoding and embedding in said stream said first image (I1) and a rotation parameter associated with said first image by using said second image (I3) as an input image for a motion compensation module; and
A device that is configured to
(Appendix 9)
9. The device of clause 8, wherein the third image is the first image.
(Appendix 10)
The third image is
obtaining a residual image in response to the first image;
obtaining a motion-predicted image in response to the first image;
and wherein the third image is a reconstructed image obtained by adding the residual image to the motion estimation image.
(Appendix 11)
A device for decoding a first image of a sequence of images from a stream, the device comprising memory associated with at least one processor, the at least one processor comprising:
obtaining a second image from the stream and a rotation parameter associated with the second image;
decoding said first image by applying motion compensation in a global rotation according to said rotation parameter to a third image (I4, I7);
wherein said third image is an output image of a motion compensation module, said output image being obtained from said second image.
(Appendix 12)
12. The device of Clause 11, wherein the third is the decoded first image.
(Appendix 13)
The third image is
obtaining a residual image in response to the second image;
obtaining a motion-predicted image in response to the second image;
and wherein the third image is a reconstructed image obtained by adding the residual image to the motion estimation image.
(Appendix 14)
7. A non-transitory processor-readable medium having stored therein instructions that cause a processor to perform at least the steps of the method of any one of Clauses 2, 3, 5 or 6.
(Appendix 15)
A system for immersive rendering of wide-field video encoded and embedded in a bitstream, comprising:
a network interface (600) for receiving said bitstream from a data network;
a device (700) for decoding the bitstream of clause 12 or 13;
an immersive rendering device (900) and
A system that includes at least

Claims (7)

A method for encoding an image according to a reference direction and embedding it in a stream, comprising:
mapping the 3D scene to the first image according to a 3D to 2D mapping operator and a forward direction;
determining a rotation parameter according to the 3D to 2D mapping operator and the angle between the forward direction and the reference direction;
applying motion compensation in global rotation according to said rotation parameter to a third image of reference to obtain a second image, said third image of reference comprising:
is the first image, or
A reconstruction obtained by adding a residual image calculated by comparing the first image with a reference image and a motion - predicted image obtained by applying a motion estimation process to the first image. applying a composed image;
generating the image to be encoded by inputting the second image into a motion compensation module using residual data and motion vectors;
encoding and embedding the image and the rotation parameter into the stream. A method of rendering a third image according to a reference direction , comprising:
obtaining an image and a rotation parameter from the stream , the image representing a 3D scene mapped according to a 3D to 2D mapping operator and a forward direction, and the rotation parameter being the determined according to a mapping operator to 2D and an angle between the forward direction and the reference direction;
obtaining a second image by inputting said image into a motion compensation module using a residual image and motion vectors , said residual image being calculated by comparing said image with a reference image; to get and to
applying motion compensation in global rotation to the second image according to the rotation parameter to obtain the third image, the third image comprising:
is the image , or
A reconstruction obtained by adding a residual image calculated by comparing the second image with a reference image and a motion estimation image obtained by applying a motion estimation process to the second image. applying a composed image;
and rendering the third image . A device for encoding and embedding images in a stream according to a reference direction, the device comprising a memory associated with at least one processor, the at least one processor comprising:
mapping the 3D scene to the first image according to a 3D to 2D mapping operator and a forward direction;
determining a rotation parameter according to the 3D to 2D mapping operator and the angle between the forward direction and the reference direction;
applying motion compensation in global rotation according to said rotation parameter to a third image of reference to obtain a second image, said third image of reference comprising:
there is the first image, or
A reconstruction obtained by adding a residual image calculated by comparing the first image with a reference image and a motion - predicted image obtained by applying a motion estimation process to the first image. applying a composed image;
generating the image to be encoded by inputting the second image into a motion compensation module using residual data and motion vectors;
and encoding and embedding said image and said rotation parameter in said stream. A device for rendering a third image according to a reference direction, the device including memory associated with at least one processor , the at least one processor:
obtaining an image and a rotation parameter from the stream , the image representing a 3D scene mapped according to a 3D to 2D mapping operator and a forward direction, and the rotation parameter being the determined according to a mapping operator to 2D and an angle between the forward direction and the reference direction;
obtaining a second image by inputting said image into a motion compensation module using a residual image and motion vectors , said residual image being calculated by comparing said image with a reference image; to get and to
applying motion compensation in global rotation to the second image according to the rotation parameter to obtain the third image, the third image comprising:
is the image , or
A reconstruction obtained by adding a residual image calculated by comparing the second image with a reference image and a motion estimation image obtained by applying a motion estimation process to the second image. A device that is configured to apply and is a composed image. A non-transitory processor-readable medium having stored therein instructions that cause a processor to perform at least the steps of the method of claim 1. A non-transitory processor-readable medium having stored therein instructions that cause a processor to perform at least the steps of the method of claim 2. A system for immersive rendering of wide-field video encoded and embedded in a bitstream, comprising:
a network interface that receives the bitstream from a data network;
a device for decoding the bitstream of claim 4;
A system including at least an immersive rendering device.

Images