TWI702567B - Method for processing projection-based frame that includes at least one projection face packed in 360-degree virtual reality projection layout - Google Patents

Abstract

A video processing method includes: obtaining a plurality of projection faces from an omnidirectional content of a sphere, wherein the omnidirectional content of the sphere is mapped onto the projection faces via cubemap projection, and the projection faces comprise a first projection face; obtaining, by a re-sampling circuit, a first re-sampled projection face by re-sampling at least a portion of the first projection face through non-uniform mapping; generating a projection-based frame according to a projection layout of the cubemap projection, wherein the projection-based frame comprises the first re-sampled projection face packed in the projection layout; and encoding the projection-based frame to generate a part of a bitstream.

Description

Used to process at least one of the encapsulated in 360-degree virtual reality projection layout Projection-based frame method for a projection surface

Cross-reference of related applications: This application is a partial continuation of the U.S. Patent Application No. 15/917,844 filed on December 3, 2008, and requires the U.S. No. 62/563,787 filed on September 27, 2017 Provisional application, the priority of the U.S. Provisional Application No. 62/583,078 filed on November 8, 2017, and the U.S. Provisional Application No. 62/583,573 filed on November 9, 2017, of which No. 15/917,844 The U.S. Patent Application No. claims priority to the U.S. Provisional Application No. 62/470,425 filed on March 13, 2017.

The entire contents of related applications, including US Patent Application No. 15/917,844, US Provisional Application No. 62/563,787, US Provisional Application No. 62/583,078, US Provisional Application No. 62/583,573 and 62/ The US provisional application No. 470,425 is hereby incorporated with reference to the subject matter of the above-mentioned application.

The present invention relates to processing omnidirectional image/video content, and more specifically, to a method for processing projection-based frame. The projection-based frame includes at least one encapsulated in 360-degree virtual reality (360-degree virtual reality). , 360 VR) projection surface in the projection layout.

Virtual reality (VR) with head-mounted displays (HMDs) is associated with various applications. The ability to display wide-view content to users can be used to provide an immersive visual experience. The real-world environment must be captured in all directions to produce an omnidirectional image/view corresponding to the sphere News content. With the advancement of camera equipment and HMDs, the delivery of VR content may soon become a bottleneck due to the high bit rate required to represent such 360-degree image/video content. When the resolution of omnidirectional video is 4K or higher, data compression/encoding is essential to reduce the bit rate.

Generally, the omnidirectional image/video content corresponding to the sphere is converted into an image sequence, and each image is a projection-based frame with one or more arranged in a 360-degree virtual reality (360 VR) projection layout. The 360-degree image/video content represented by a plurality of projection surfaces is then encoded into a bit stream based on the sequence of the projected frame for transmission. The projection-based frame may have discontinuities in the image content at the layout boundary and/or the surface boundary. Therefore, after compression, the image quality around the layout boundary and/or surface boundary may be poor. In addition, the projection layout conversion of the decoded projection-based frame may introduce artifacts, thereby causing the image quality of the converted projection-based frame to deteriorate.

One of the objectives of the present invention is to provide a method for processing a projection-based frame, which includes at least one projection surface encapsulated in a 360-degree virtual reality (360 VR) projection layout.

According to the first aspect of the present invention, an exemplary video processing method is disclosed. The exemplary video processing method includes: obtaining a plurality of projection surfaces from the omnidirectional content of a sphere, wherein the omnidirectional content of the sphere is mapped onto a projection surface by a cube projection, and the projection surfaces include a first projection surface; by resampling The circuit resamples at least a part of the first projection surface through non-uniform mapping to obtain a first resampled projection surface, wherein the first projection surface has a first source area and a second source area, and the first resampled projection surface has a first A re-sampling area and a second re-sampling area. The first re-sampling area is obtained by re-sampling from the first source area at the first sampling density, and the second re-sampling area is obtained from the second source area at the second re-sampling density After re-sampling, the first sampling density is different from the second sampling density; generating a projection-based frame according to the projection layout of the cube projection, where the projection-based frame includes the first re-sampled projection surface encapsulated in the projection layout; And encode the projection-based frame to generate a bit stream a part of.

According to the second aspect of the present invention, an exemplary video processing method is disclosed. The exemplary video processing method includes: obtaining a plurality of projection surfaces from the omnidirectional content of the sphere according to the cube projection; generating at least one filled area through a filling circuit; and using the projection surfaces and the at least one projection surface packaged in the projection layout of the cube projection Fill the area to generate a projection-based frame, wherein the projection surfaces encapsulated in the projection layout include a first projection surface; the at least one filled area encapsulated in the projection layout includes a first filled area; the first filled area is at least It is connected with the first projection surface and forms at least a part of a boundary of the projection layout; and the projection-based frame is encoded to generate a part of the bit stream.

According to a third aspect of the present invention, an exemplary video processing method is disclosed. The exemplary video processing method includes: receiving a part of the bit stream, and decoding a part of the bit stream to generate a decoded projection-based frame, wherein the projection-based frame is encapsulated in a 360-degree virtual reality (360 VR) projection At least one projection surface and at least one filling area in the layout. The step of decoding a part of the bit stream to generate a decoded projection-based frame includes: the decoded pixel value obtained by mixing the first pixel included in the at least one filled area and the second pixel value included in the at least one projection surface The decoded pixel value obtained by the pixel reconstructs the first pixel included in the at least one filled area.

These and other objects of the present invention will undoubtedly be apparent to those with ordinary knowledge in the field after reading the various drawings and the preferred embodiments shown in the accompanying drawings described in detail below.

100, 1900, 2700, 3200, 3300: 360 VR system

102, 1902, 2702: source electronic equipment

103: Transmission device

104, 3204, 3304, 3704, 3804: target electronic device

112: Video capture equipment

114, 1914, 2714, 3726, 3826: conversion circuit

115, 2716: fill circuit

116: Video Encoder

122, 3222, 3322, 3722, 3822: Decoding circuit

124: Graphic presentation circuit

126: display screen

1915, 2715: Resampling circuit

3224, 3324, 3724, 3824: hybrid circuit

3700, 3800: 360 VR system

202, 2002: Sphere

204: Octahedron

206, 406: Octahedral projection layout

208: Equator

302-308: Right triangle part

310, 310', 510, 510', 1102: compact octahedral projection layout

404: Rotated Octahedron

502-508: Right triangle part

1002: Compact cube projection layout

1202, 1302, 1402: ERP/EAP layout without filling

1202', 1302', 1402': ERP/EAP layout

1502, 1602, 1702, 1802: projection layout

2004: cube

2006: CMP layout

2102, 2104, 2302, 2304, 2402, 2404: square projection surface

2312, 2316: first source area

2314, 2318: second source area

2322, 2326: first resampling area

2324, 2328: second resampling area

2802: 3×2 cube layout with inner boundary padding

2804: 6×1 cube layout with inner boundary padding

2902, 3002: 3×2 cube layout with outer boundary filling and inner boundary filling

2904, 3004: 6×1 cube layout with outer boundary filling and inner boundary filling

Figure 1 is a schematic diagram of the first 360-degree virtual reality (360 VR) system according to an embodiment of the present invention.

Figure 2 is a schematic diagram of a triangular projection surface based on the octahedron projection format obtained from the projection of a sphere to an unrotated octahedron.

Fig. 3 is a schematic diagram of the first compact octahedron projection layout according to an embodiment of the present invention.

Figure 4 is a schematic diagram of a triangular projection surface in the octahedral projection format obtained from the projection of a sphere to a rotated octahedron.

Figure 5 is a schematic diagram of a second compact octahedral projection layout according to an embodiment of the present invention.

Fig. 6 is a schematic diagram of the first compact octahedral projection layout with filling according to an embodiment of the present invention.

Fig. 7 is a schematic diagram of a second compact octahedral projection layout with filling according to an embodiment of the present invention.

Fig. 8 is a schematic diagram of interpolation performed by the filling circuit shown in Fig. 1.

Figure 9 is a schematic diagram of geometric filling performed by the filling circuit shown in Figure 1.

FIG. 10 is a schematic diagram of a projection layout of a filled compact cube (cubemap) according to an embodiment of the present invention.

Figure 11 is a schematic diagram of a third compact octahedral projection layout with filling according to an embodiment of the present invention.

Figure 12 is a schematic diagram of the first ERP/EAP layout with filling according to an embodiment of the present invention.

Figure 13 is a schematic diagram of a second ERP/EAP layout with filling according to an embodiment of the present invention.

Figure 14 is a schematic diagram of a third ERP/EAP layout with filling according to an embodiment of the present invention.

Figure 15 is a schematic diagram of a filled octahedral projection layout according to an embodiment of the present invention.

Figure 16 is a schematic diagram of a filled cube projection layout according to an embodiment of the present invention.

Figure 17 is a schematic diagram of a fourth compact octahedral projection layout with filling according to an embodiment of the present invention.

Figure 18 is a schematic diagram of a filled compact cube projection layout according to an embodiment of the present invention.

Figure 19 is a schematic diagram of a second 360 VR system according to an embodiment of the present invention.

Figure 20 is a schematic diagram of six square projection surfaces in a cube projection layout obtained from a sphere through cubemap projection (CMP).

FIG. 21 is a schematic diagram of re-sampling a square projection surface obtained by cube projection through uniform mapping according to an embodiment of the present invention.

Figure 22 is a schematic diagram of a uniform mapping function curve according to an embodiment of the present invention.

FIG. 23 is a schematic diagram of a first example of re-sampling a square projection surface obtained by cube projection through non-uniform mapping according to an embodiment of the present invention.

FIG. 24 is a schematic diagram of a second example of re-sampling a square projection surface obtained by cube projection through non-uniform mapping according to an embodiment of the present invention.

Figure 25 is a schematic diagram of a first non-uniform mapping function curve according to an embodiment of the present invention.

Figure 26 is a schematic diagram of a second non-uniform mapping function curve according to an embodiment of the present invention.

Figure 27 is a schematic diagram of a third 360 VR system according to an embodiment of the present invention.

FIG. 28 is a schematic diagram of performing edge padding on a cubic projection layout according to an embodiment of the present invention.

FIG. 29 is a schematic diagram of performing boundary padding and inner boundary padding on a cubic projection layout according to an embodiment of the present invention.

FIG. 30 is a schematic diagram of performing outer boundary filling and inner boundary filling on other cubic projection layouts according to an embodiment of the present invention.

FIG. 31 is a schematic diagram of a filling design for generating a filling area of a projection surface by copying a partial area of another projection surface according to an embodiment of the present invention.

Figure 32 is a schematic diagram of a fourth 360 VR system according to an embodiment of the present invention.

Figure 33 is a schematic diagram of a fifth 360 VR system according to an embodiment of the present invention.

Figure 34 is a schematic diagram of a decoder-side mixing operation according to an embodiment of the present invention.

Figure 35 shows the right of pixels involved in updating the pixel values of pixels on the projection surface according to an embodiment of the present invention. A schematic diagram of the relationship between the heavy value and the index value of the pixel.

FIG. 36 is a schematic diagram of another relationship between the weight value of the pixel involved in updating the pixel value of the pixel on the projection surface and the index value of the pixel according to an embodiment of the present invention.

Figure 37 is a schematic diagram of a sixth 360 VR system according to an embodiment of the present invention.

Figure 38 is a schematic diagram of a seventh 360 VR system according to an embodiment of the present invention.

Figure 39 is a schematic diagram of the relationship between the pixel value of the pixel in the projection plane and the pixel value of the filled pixel in the filled area, and the relationship between the weight value of the involved pixel and the index value of the pixel according to an embodiment of the present invention.

Figure 40 is a schematic diagram of another relationship between the pixel value of the pixel in the projection plane and the pixel value of the filled pixel in the filled area, and the weight value of the involved pixel and the index value of the pixel according to an embodiment of the present invention. .

In the specification and subsequent patent applications, certain words are used to refer to specific elements. Those skilled in the art should understand that electronic device manufacturers may use different terms to refer to the same component. The scope of this specification and subsequent patent applications does not use differences in names as a way to distinguish elements, but uses differences in functions of elements as a criterion for distinguishing. The "including" and "including" mentioned in the entire specification and the subsequent request items are open-ended terms and should be interpreted as "including but not limited to...". In addition, the term "coupling" here includes any direct and indirect electrical connection means. Therefore, if it is described that a first device is coupled to a second device, it means that the first device can be directly electrically connected to the second device, or indirectly electrically connected to the second device through other devices or connection means.

Figure 1 is a schematic diagram of the first 360-degree virtual reality (360 VR) system according to an embodiment of the present invention. The 360 VR system 100 includes two video processing devices (for example, a source electronic device 102 and a target electronic device 104). The source electronic device 102 includes a video capture device 112, a conversion circuit 114 and a video encoder 116. For example, the video capture device 112 may be used to provide an omnidirectional image/view corresponding to a sphere. Information content (for example, multiple images covering the entire surrounding environment) S_IN group of cameras. The conversion circuit 114 is coupled between the video capture device 112 and the video encoder 116. The conversion circuit 114 generates a projection-based frame IMG with a 360-degree virtual reality projection layout L_VR according to the omnidirectional image/video content S_IN. For example, the projection-based frame IMG may be one frame included in the sequence of projection-based frames generated from the conversion circuit 114. The video encoder 116 is an encoding circuit for encoding/compressing the projection-based frame IMG to generate a part of the bitstream BS. In addition, the video encoder 116 outputs the bit stream BS to the target electronic device 104 via the transmission device 103. For example, the sequence of projection-based frames can be encoded into the bit stream BS, and the transmission device 103 can be a wired/wireless communication link or a storage medium.

The target electronic device 104 may be a head-mounted display (HMD) device. As shown in FIG. 1, the target electronic device 104 includes a decoding circuit 122, a graphics rendering circuit 124, and a display screen 126. The decoding circuit 122 receives the bit stream BS from the transmission device 103 (for example, a wired/wireless communication link or a memory), and then performs a video decoder function for decoding a part of the received bit stream BS to generate a decoded frame IMG'. For example, the decoding circuit 122 generates a sequence of decoded frames by decoding the received bit stream BS, where the decoded frame IMG' is one frame included in the sequence of decoded frames. In this embodiment, the projection-based frame IMG encoded by the video encoder 116 on the encoder side has a 360 VR projection format and a projection layout. Therefore, after the decoding circuit 122 on the decoder side decodes the bit stream BS, the decoded frame IMG' is a decoded projection-based frame with the same 360 VR projection format and the same projection layout. The graphics rendering circuit 124 is coupled between the decoding circuit 122 and the display screen 126. The graphic presentation circuit 124 presents and displays the output image data on the display screen 126 according to the decoded frame IMG′. For example, the viewport area associated with a part of the 360-degree image/video content carried by the decoded frame IMG′ may be displayed on the display screen 126 via the graphics rendering circuit 124.

As described above, the conversion circuit 114 generates a projection-based frame IMG according to the 360VR projection layout L_VR and the omnidirectional image/video content S_IN. If the 360 VR projection layout L_VR is a compact projection layout without padding, the packaging of the projection surface may cause discontinuities in image content between adjacent projection surfaces boundary.

Figure 2 is a schematic diagram of a triangular projection surface based on the octahedral projection format obtained from a sphere to an unrotated octahedral projection. The omnidirectional image/video content of the sphere 202 is mapped to the eight triangular projection surfaces of the unrotated octahedron 204 (labeled "1", "2", "3", "4", "5", "6", "7", and "8"). As shown in FIG. 2, the triangular projection surfaces "1"-"8" are arranged in an octahedral projection layout 206. The shape of each triangle projection surface "1"-"8" is an equilateral triangle. For the triangular projection surface "K" (K=1-8), the surface has three sides, denoted as SK1, SK2 and SK3. The sphere 202 is composed of a top hemisphere (for example, the northern hemisphere) and a bottom hemisphere (for example, the southern hemisphere). Due to the octahedral projection based on the unrotated octahedron 204, the triangular projection surfaces "1", "2", "3" and "4" are all from the top hemisphere, and the triangular projection surfaces "5", "6"" 7", and "8" are all from the bottom hemisphere, and the equator 208 of the sphere 202 is mapped along the sides S13, S23, S33, S43, S53, S63, S73 and S83 of the triangular projection plane "1"-"8", As shown by the dotted line in Figure 2.

The projection-based frame IMG to be encoded needs to be rectangular. If the octahedral projection layout 206 is directly used to create a projection-based frame IMG, the projection-based frame IMG cannot have a compact frame layout. In the projection-based frame IMG, many vacant areas need to be filled (for example, Fill the area with black, gray or white). Therefore, there is a need for a compact octahedral projection layout that can avoid vacant areas (for example, areas filled with black, gray, or white).

Please refer to Figure 2 and Figure 3 together. Figure 3 is a schematic diagram of the first compact octahedral projection layout according to an embodiment of the present invention. The equator 208 of the sphere 202 is mapped along the sides of the triangular projection plane "1"-"8", as shown by the dotted line in FIG. 3. The compact octahedral projection layout 310 is derived from the octahedral projection layout 206 through triangular projection surface rotation and triangular projection surface division. As shown in the middle part of Figure 3, the triangular projection surface "1" in the octahedral projection layout 206 is rotated 60° clockwise, and the triangular projection surface "3" in the octahedral projection layout 206 is rotated 60° counterclockwise. °, the triangular projection surface "5" in the octahedral projection layout 206 is rotated 60° counterclockwise, and the triangular projection surface "7" in the octahedral projection layout 206 is clockwise The needle rotates 60°. Therefore, the side S21 of the triangular projection surface "2" is connected to the side S12 of the triangular projection surface "1", and the side S22 of the triangular projection surface "2" is connected to the side S31 of the triangular projection surface "3". The side S62 of the triangular projection surface "6" is connected to the side S51 of the triangular projection surface "5", and the side S61 of the triangular projection surface "6" is connected to the side S72 of the triangular projection surface "7".

As shown in the middle part of Figure 3, there is an image content continuity boundary between the side S21 of the triangular projection surface "2" and the side S12 of the triangular projection surface "1" (that is, the content is continuously expressed on the triangular projection surface "1" and "2"), there is an image content continuity boundary between the side S22 of the triangular projection surface "2" and the side S31 of the triangular projection surface "3" (that is, the content is continuously expressed on the triangular projection surface "2" and "3"), there is an image content continuity boundary between the side S23 of the triangular projection surface "2" and the side S63 of the triangular projection surface "6" (that is, the content is continuously expressed on the triangular projection surface "2" and "6"), there is an image content continuity boundary between the side S62 of the triangular projection surface "6" and the side S51 of the triangular projection surface "5" (that is, the content is continuously expressed on the triangular projection surface "5" and "6"), and there is an image content continuity boundary between the side S61 of the triangular projection surface "6" and the side S72 of the triangular projection surface "7" (that is, the content continuously represents the "6" and "7" above).

In addition, the triangular projection surface "8" in the octahedral projection layout 206 is divided into two right-angled triangle parts 302 and 304, and the triangular projection surface "4" in the octahedral projection layout 206 is divided into two right-angled triangle parts 306 And 308. As shown at the bottom of Figure 3, the right-angled triangular portion 304 of the triangular projection surface "8" and the right-angled triangular portion 308 of the triangular projection surface "4" are connected to the triangular projection surfaces "7" and "3", respectively; the triangular projection surface " The right-angled triangle portion 302 of 8" and the right-angled triangle portion 306 of the triangle projection surface "4" are repositioned and connected to the triangle projection surfaces "5" and "1", respectively.

The right-angled triangle portion 302 of the triangular projection surface "8" has three sides S811, S812 and S83_1, where the side S811 is the side S81 of the triangular projection surface "8", and the side S83_1 is the side S83 of the triangular projection surface "8". Part. The right-angled triangle portion 304 of the triangular projection surface "8" has three sides S821, S822 and S83_2, where the side S821 is the side S82 of the triangular projection surface "8", and the side S83_2 is three sides. The second part of side S83 of the angular projection surface "8".

The right-angled triangle portion 306 of the triangular projection surface "4" has three sides S421, S422 and S43_1, where the side S421 is the side S42 of the triangular projection surface "4", and the side S43_1 is the side S43 of the triangular projection surface "4". Part. The right-angled triangle portion 308 of the triangular projection surface "4" has three sides S411, S412 and S43_2, wherein the side S411 is the side S41 of the triangular projection surface "4", and the side S43_2 is the second side of the side S43 of the triangular projection surface "4". Two parts.

According to the compact octahedral projection layout 310, the side S821 of the right-angled triangle portion 304 of the triangular projection surface "8" is connected to the side S73 of the triangular projection surface "7", and the side S83_2 of the right-angled triangle portion 304 of the triangular projection surface "8" is connected with The side S43_2 of the right-angled triangular part 308 of the triangular projection surface "4" is connected, the side S411 of the right-angled triangular part 308 of the triangular projection surface "4" is connected with the side S33 of the triangular projection surface "3", and the right angle of the triangular projection surface "8" The side S811 of the triangle portion 302 is connected with the side S53 of the triangle projection surface "5", the side S83_1 of the right triangle portion 302 of the triangle projection surface "8" is connected with the side S43_1 of the right triangle portion 306 of the triangle projection surface "4", the triangle The side S421 of the triangular portion 306 of the projection surface "4" is connected to the side S13 of the triangular projection surface "1".

There is an image content continuity boundary between the side S83_2 of the right-angled triangle portion 304 of the triangle projection surface "8" and the side S43_2 of the right-angled triangle portion 308 of the triangle projection surface "4". There is an image content continuity boundary between the side S83_1 of the right-angled triangle portion 302 of the triangle projection surface "8" and the side S43_1 of the right-angled triangle portion 306 of the triangle projection surface "4". That is, the content is continuously represented in the triangular projection surfaces "4" and "8". In addition, there is a discontinuous border of image content between the side S821 of the right-angled triangle portion 304 of the triangular projection surface "8" and the side S73 of the triangular projection surface "7". There is a discontinuous boundary of image content between side S411 and side S33 of triangular projection surface "3", and there is a boundary between side S811 of right-angled triangle portion 302 of triangular projection surface "8" and side S53 of triangular projection surface "5" Image content discontinuity boundary and image content discontinuity between side S421 of right-angled triangle portion 306 of triangle projection surface "4" and side S13 of triangle projection surface "1" boundary.

As shown at the bottom of FIG. 3, the 360 VR projection layout L_VR set by the compact octahedral projection layout 310 is a rectangle without any dummy regions (for example, black regions or white regions). In addition, the part of the 360-degree image/video content is continuously represented in the triangular projection surface "1", "2", "3", "5", "6", and "7", and there is no discontinuous image content. However, some discontinuous boundaries of image content still inevitably exist in the compact octahedral projection layout 310. Therefore, if the 360VR projection layout L_VR is set through the compact octahedral projection layout 310, the image quality near the discontinuous boundary of the image content after compression may be poor.

When the triangular projection surfaces "1"-"8" shown in Figure 2 are rearranged and packaged in a compact octahedral projection layout 310, some triangular projection surfaces must be divided and repositioned, resulting in projection-based The image content of the equator 208 in the frame IMG is not continuous. Generally, the top and bottom regions of the sphere 202 usually represent the “sky” and the “ground” respectively, and most moving objects in the surrounding environment are located at the equator 208 of the sphere 202. If the equator 208 in the projection-based frame IMG has discontinuity in image content, the coding efficiency and image quality will be significantly reduced. If the equator 208 of the sphere 202 is mapped along the middle of the triangular projection surface or any position other than the side of the triangular projection surface, coding efficiency and image quality can be improved.

Figure 4 is a schematic diagram of a triangular projection surface based on the octahedral projection format obtained from a sphere to a rotated octahedral projection. The omnidirectional image/video content of the sphere 202 is mapped to the eight triangular projection surfaces of the rotated octahedron 404 (labeled "1", "2", "3", "4", "5", " 6", "7" and "8"). The rotated octahedron 404 shown in FIG. 4 can be obtained by applying a 90-degree rotation to the octahedron 204 shown in FIG. 2. As shown in FIG. 4, the triangular projection surfaces "1"-"8" are encapsulated in an octahedral projection layout 406. The shape of each triangle projection surface "1"-"8" is an equilateral triangle. For the triangular projection surface "K" (K=1-8), the surface has three sides, denoted as SK1, SK2 and SK3. The sphere 202 is composed of a left hemisphere and a right hemisphere. The triangular projection surfaces "1", "2", "3" and "4" all come from the right hemisphere, the triangle The projection surfaces "5", "6", "7" and "8" are all from the left hemisphere. Due to the octahedral projection on the rotating octahedron 404, the equator 208 of the sphere 202 is not mapped along any side of each triangular projection surface. In this embodiment, the equator 208 of the sphere 202 is mapped along the middle of the triangular projection planes "2", "4", "6" and "8", as shown by the dashed line in Figure 4. As mentioned above, the projection-based frame IMG to be encoded needs to be rectangular. Therefore, the projection-based frame IMG should use a compact octahedral projection layout.

Please refer to Figure 4 and Figure 5 together. Figure 5 is a schematic diagram of a second compact octahedral projection layout according to an embodiment of the present invention. The equator 208 of the sphere 202 is mapped along the middle of the triangular projection planes "2", "4", "6" and "8", as shown by the dashed line in Figure 5. The compact octahedral projection layout 510 is derived from the octahedral projection layout 406 through triangular projection surface rotation and triangular projection surface division. As shown in the middle part of Figure 5, the triangular projection surface "7" in the octahedral projection layout 406 is rotated 60° clockwise, and the triangular projection surface "5" in the octahedral projection layout 406 is rotated 60° counterclockwise. °, the triangular projection surface “3” in the octahedral projection layout 406 is rotated 60° counterclockwise, and the triangular projection surface “1” in the octahedral projection layout 406 is rotated 60° clockwise. Therefore, the side S72 of the triangular projection surface "7" is connected to the side S61 of the triangular projection surface "6", and the side S51 of the triangular projection surface "5" is connected to the side S62 of the triangular projection surface "6". The side S31 of the "triangular projection surface "3" is connected to the side S22 of the triangular projection surface "2", and the side S12 of the triangular projection surface "1" is connected to the side S21 of the triangular projection surface "2".

As shown in the middle part of Figure 5, there is an image content continuity boundary between the side S72 of the triangular projection surface "7" and the side S61 of the triangular projection surface "6" (that is, the content is continuously expressed on the triangular projection surface In "6" and "7"), there is an image content continuity boundary between the side S51 of the triangular projection surface "5" and the side S62 of the triangular projection surface "6" (that is, the content is continuously expressed on the triangular projection surface In "5" and "6"), there is an image content continuity boundary between the side S31 of the triangular projection surface "3" and the side S22 of the triangular projection surface "2" (that is, the content is continuously expressed on the triangular projection surface In "2" and "3"), there is an image content continuity boundary between the side S12 of the triangular projection surface "1" and the side S21 of the triangular projection surface "2" (that is, the content is continuously expressed on the triangular projection surface "1" and "2"), as well as the side S23 and triangle on the triangular projection plane "2" There is an image content continuity boundary between the sides S63 of the triangular projection surface "6" (that is, the content is continuously represented in the triangular projection surfaces "2" and "6").

In addition, the triangular projection surface "4" in the octahedral projection layout 406 is divided into two right-angled triangle parts 502 and 504, and the triangular projection surface "8" in the octahedral projection layout 406 is divided into two right-angled triangle parts 506 And 508. As shown in the right part of Figure 5, the right-angled triangle portion 504 of the triangular projection surface "4" and the right-angled triangle portion 508 of the triangular projection surface "8" are respectively connected to the triangular projection surfaces "1" and "5"; the triangular projection surface The right-angled triangle portion 502 of "4" and the right-angled triangle portion 506 of the triangle projection surface "8" are repositioned and connected to the triangle projection surfaces "3" and "7", respectively.

The right-angled triangle portion 502 of the triangular projection surface "4" has three sides S411, S412 and S43_1, where the side S411 is the side S41 of the triangular projection surface "4", and the side S43_1 is the side S43 of the triangular projection surface "4". Part. The right-angled triangle portion 504 of the triangular projection surface "4" has three sides S421, S422 and S43_2, where the side S421 is the side S42 of the triangular projection surface "4", and the side S43_2 is the second side of the side S43 of the triangular projection surface "4". Two parts.

The right-angled triangle portion 506 of the triangular projection surface "8" has three sides S821, S822, and S83_1, where the side S821 is the side S82 of the triangular projection surface "8", and the side S83_1 is the side S83 of the triangular projection surface "8". Part. The right-angled triangle portion 508 of the triangular projection surface "8" has three sides S811, S812 and S83_2, where the side S811 is the side S81 of the triangular projection surface "8", and the side S83_2 is the second side of the side S83 of the triangular projection surface "8". Two parts.

According to the compact octahedral projection layout 510, the side S421 of the right-angled triangle portion 504 of the triangular projection surface "4" is connected to the side S13 of the triangular projection surface "1", and the side S43_2 of the right-angled triangle portion 504 of the triangular projection surface "4" is connected to The side S83_2 of the right-angled triangular part 508 of the triangular projection surface "8" is connected, the side S811 of the right-angled triangular part 508 of the triangular projection surface "8" is connected with the side S53 of the triangular projection surface "5", and the right angle of the triangular projection surface "4" The side S411 of the triangular part 502 is connected to the side S33 of the triangular projection surface "3", and the side S43_1 of the right-angled triangular part 502 of the triangular projection surface "4" is directly connected to the triangular projection surface "8". The side S83_1 of the corner triangular portion 506 is connected, and the side S821 of the triangular portion 506 of the triangular projection surface "8" is connected to the side S73 of the triangular projection surface "7".

There is an image content continuity boundary between the side S43_2 of the right-angled triangle portion 504 of the triangle projection surface "4" and the side S83_2 of the right-angled triangle portion 508 of the triangle projection surface "8". There is an image content continuity boundary between the side S43_1 of the right-angled triangle portion 502 of the triangle projection surface "4" and the side S83_1 of the right-angled triangle portion 506 of the triangle projection surface "8". That is, the content is continuously represented in the triangular projection surfaces "4" and "8". In addition, there is a discontinuous boundary of image content between the side S421 of the right-angled triangle portion 504 of the triangular projection surface "4" and the side S13 of the triangular projection surface "1". There is a discontinuous boundary of image content between side S811 and side S53 of triangular projection surface "5", and there is a boundary between side S411 of right-angled triangle portion 502 of triangular projection surface "4" and side S33 of triangular projection surface "3" The image content is a discontinuous boundary, and there is an image content discontinuity boundary between the side S821 of the right-angled triangle portion 506 of the triangular projection surface "8" and the side S73 of the triangular projection surface "7".

In the right part of FIG. 5, the 360 VR projection layout L_VR arranged in the shape of the compact octahedral projection layout 510 is a rectangle without any vacant areas (for example, areas filled with black, gray, or white). In addition, the part of the 360-degree image/video content is continuously represented in the triangular projection surface "1", "2", "3", "5", "6", and "7", and there is no discontinuous image content. In addition, in the projection-based image IMG (which uses the compact octahedral projection layout 510), the equator 208 represented by the triangular projection surfaces "2", "4", "6" and "8" is not divided by the triangular projection surface. The resulting discontinuity of image content. However, some discontinuous boundaries of image content still inevitably exist in the compact octahedral projection layout 510. Therefore, if the 360 VR projection layout L_VR is set through the compact octahedral projection layout 510, the image quality near the discontinuous boundary of the image content after compression may be poor.

To solve the above-mentioned image quality degradation problem, the present invention proposes an innovative 360 VR projection layout design with filling, which can improve the image quality at the boundary of the projection surface after compression. For example, the 360 VR projection layout L_VR can be configured with a compact cube layout with filling or a compact layout with filling Octahedron layout to set. Specifically, the conversion circuit 114 receives the omnidirectional image/video content of the sphere 202 from the video capture device 112, and obtains a plurality of projection surfaces from the omnidirectional image/video content of the sphere 202, wherein the omnidirectional image of the sphere 202 is /Video content is mapped onto the projection surface through a selected 360VR projection (for example, cube projection or octahedral projection). As shown in Figure 1, the conversion circuit 114 has a filling circuit 115 which is arranged to produce at least one filling area. The conversion circuit 114 creates a projection-based frame IMG by encapsulating a plurality of projection surfaces and at least one filled area in a 360 VR projection layout L_VR (for example, a compact cube layout with filling or a compact octahedron layout with filling).

For example, the projection surface packaged in the 360 VR projection layout L_VR includes a first projection surface and a second projection surface. If the first side of the first projection surface is connected to the first side of the second projection surface, There is a discontinuous border of image content between the first side of the surface and the first side of the second projection surface. The at least one filled area encapsulated in the 360 VR projection layout L_VR includes a first filled area, where the first filled area is connected to the first side of the first projection surface and the first side of the second projection surface for 360 VR projection The first side of the first projection surface in the layout L_VR is separated from the first side of the second projection surface. The first padding area is intentionally inserted to provide more information about the compression process. In this way, the compressed image quality of the first side of the first projection surface and the first side of the second projection surface can be improved.

Fig. 6 is a schematic diagram of the first compact octahedral projection layout with filling according to an embodiment of the present invention. The 360 VR projection layout L_VR adopted by the conversion circuit 114 can be set by the compact octahedral projection layout 310 ′ shown in FIG. 6. The compact octahedral projection layout 310' can be derived from the compact octahedral projection layout 310 shown in FIG. Regarding the compact octahedral projection layout 310 shown in Figure 3, there is an image content discontinuous boundary between the side S821 of the right-angled triangle portion 304 of the triangular projection surface "8" and the side S73 of the triangular projection surface "7" , There is a discontinuous boundary of image content between the side S411 of the right-angled triangle portion 308 of the triangular projection surface "4" and the side S33 of the triangular projection surface "3", and the side of the right-angled triangle portion 302 of the triangular projection surface "8" There is a discontinuous boundary of image content between S811 and the side S53 of the triangular projection surface "5", and the side S421 of the right-angled triangle part 306 of the triangular projection surface "4" and the triangle projection There is a discontinuous border of image content between the sides S13 of the surface "1". As shown in Fig. 6, insert the first filling area PR_1 so as to be aligned with the side S421 of the right-angled triangle portion 306 of the triangular projection surface "4" (which is also the side S42 of the projection surface "4") and the side of the projection surface "1" S13 connection; insert the second filling area PR_2 to connect with the side S411 of the right-angled triangle portion 308 of the triangular projection surface "4" (which is also the side S41 of the projection surface "4") and the side S33 of the projection surface "3"; insert The third filling area PR_3 is connected to the side S811 (which is also the side S81 of the projection surface “8”) of the right-angled triangle portion 302 of the triangular projection surface “8” and the side S53 of the projection surface “5”; and the fourth filling is inserted The area PR_4 is connected to the side S821 of the right-angled triangle portion 304 of the triangular projection "8" (which is also the side S82 of the projection surface "8") and the side S73 of the projection surface "7". Assume that the width of each filled area is D, and the compact octahedral projection layout 310 shown in Figure 3 has a width W and a height H, and the compact octahedral projection layout 310' shown in Figure 6 has a width W+ 2D and height H. For example, the width D of each filled area may be 16 pixels.

Fig. 7 is a schematic diagram of a second compact octahedral projection layout with filling according to an embodiment of the present invention. The 360 VR projection layout L_VR adopted by the conversion circuit 114 can be set by the compact octahedral projection layout 510 ′ shown in FIG. 7. The compact octahedral projection layout 510' can be derived from the compact octahedral projection layout 510 shown in FIG. Regarding the compact octahedral projection layout 510 shown in Figure 5, there is an image content discontinuous boundary between the side S821 of the right-angled triangle portion 506 of the triangular projection surface "8" and the side S73 of the triangular projection surface "7" , There is a discontinuous boundary of image content between the side S411 of the right-angled triangle portion 502 of the triangular projection surface "4" and the side S33 of the triangular projection surface "3", and the side of the right-angled triangle portion 508 of the triangular projection surface "8" There is a discontinuous boundary of image content between S811 and side S53 of the triangular projection surface "5", and there is a graph between the side S421 of the right-angled triangle portion 504 of the triangular projection surface "4" and the side S13 of the triangular projection surface "1". Like content discontinuous boundaries. As shown in Fig. 7, the first filling area PR_1 is inserted to match the side S421 of the right-angled triangle portion 504 of the triangular projection surface "4" (which is also the side S42 of the projection surface "4") and the side of the projection surface "1". S13 connection; insert the second filling area PR_2 to be connected to the side S411 of the right-angled triangle portion 502 of the triangular projection surface "4" (which is also the side S41 of the projection surface "4") and the projection surface "3" The third filling area PR_3 is inserted to connect the side S811 (also the side S81 of the projection surface "8") of the right-angled triangle portion 508 of the triangular projection surface "8" and the side S53 of the projection surface "5"; And the fourth filling area PR_4 is inserted to connect with the side S821 of the right-angled triangle portion 506 of the triangular projection surface "8" (which is also the side S82 of the projection surface "8") and the side S73 of the projection surface "7". Assuming that the height of each filled area is D, and the compact octahedral projection layout 510 shown in Figure 5 has a width W and a height H, then the compact octahedral projection layout 510' shown in Figure 7 has a width W And height H+2D. For example, the height D of each filled area may be 16 pixels.

In an exemplary filling implementation, the filling circuit 115 sets the pixel values of the pixels included in the filling area by performing an interpolation operation based on the pixel values of the pixels included in the adjacent projection plane connected to the filling area. Regarding each of the compact octahedral projection layout 310' shown in FIG. 6 and the compact octahedral projection layout 510' shown in FIG. 7, it is based on the adjacent projection planes "1" and "4 The pixel values of the pixels included in "" are interpolated to obtain the pixel values of the pixels included in the first filling area PR_1; by interpolation based on the pixel values of the pixels included in the adjacent projection planes "3" and "4" Operation to obtain the pixel values of the pixels included in the second filling area PR_2; and obtaining the third filling area PR_3 by performing interpolation operations based on the pixel values of the pixels included in the adjacent projection surfaces "8" and "5" The pixel value of the pixel included in the fourth filling area PR_4 is obtained by performing interpolation calculation based on the pixel value of the pixel included in the adjacent projection surfaces "7" and "8".

The interpolation used can be nearest neighbor interpolation, linear interpolation, bilinear interpolation or other suitable interpolation algorithms. The sampling points used in the interpolation used can be obtained from a single direction or different directions. Fig. 8 is a schematic diagram of interpolation performed by the filling circuit 115 shown in Fig. 1. The filling area PR needs to be inserted between the adjacent projection surfaces A1 and A2, which are obtained from the selected 360VR projection of the sphere, where if the projection surface A1 is connected to the projection surface A2, the projection surface There is a discontinuous border of image content between faces A1 and A2. As shown in the sub-picture (A) of Fig. 8, the sampling points (ie, pixels) P1 and P2 obtained from the adjacent projection surfaces A1 and A2 in the vertical direction are performed Line interpolation. Therefore, according to the sampling values of the sampling points P1 and P2, the distance between the sampling point P1 and the interpolation sampling point S, and the distance between the sampling point P2 and the interpolation sampling point S, the interpolation sampling point (ie, the interpolation pixel) S is determined .

As shown in the sub-picture (B) of Fig. 8, the sampling points (ie pixels) Q1 and Q2 obtained from the adjacent projection planes A1 and A2 in the horizontal direction are interpolated. Therefore, based on the sampling values of the sampling points Q1 and Q2, the distance between the sampling point Q1 and the interpolation sampling point S, and the distance between the sampling point Q2 and the interpolation sampling point S, the interpolation sampling point (ie, the interpolation pixel) is determined S.

As shown in the sub-figure (C) of Fig. 8, the sampling points (ie, pixels) P1 and P2 obtained from adjacent projection surfaces A1 and A2 in the vertical direction and from adjacent projection surfaces A1 and A2 in the horizontal direction The sampling points (ie, pixels) Q1 and Q2 obtained by A2 perform interpolation. Therefore, according to the sampling values of sampling points P1, P2, Q1 and Q2, the distance between sampling point P1 and interpolation sampling point S, the distance between sampling point P2 and interpolation sampling point S, sampling point Q1 and interpolation sampling point S The distance between, and the distance between the sampling point Q2 and the interpolation sampling point S, determine the interpolation sampling point (ie, the interpolation pixel) S.

In another exemplary filling implementation, the filling circuit 115 applies the first geometric filling to an adjacent projection surface to determine the first pixel value of the pixels included in the filled area, and applies the second geometric filling to another phase. The adjacent projection plane determines the second pixel value of the pixels included in the filled area, and sets the filled area by mixing the first pixel value derived from the first geometric filling and the second pixel value derived from the second geometric filling The pixel value of the included pixels. FIG. 9 is a schematic diagram of geometric filling performed by the filling circuit 115 shown in FIG. 1. The filling area PR needs to be inserted between the adjacent projection surfaces A1 and A2. These projection surfaces A1 and A2 are obtained through the selected 360 VR projection of the sphere. If the projection surface A1 is connected to the projection surface A2, the projection surface There is a discontinuous boundary between the projection surfaces A1 and A2 of the image content. The first geometric filling applied to the projection surface A1 determines a geometric mapping area A1_GP, where the geometric mapping area A1_GP is mapped to the content of the area on the sphere (for example, the sphere 202 shown in Figure 2 / Figure 4) To the filled area PR, where the area on the sphere is the same as the area where the projection surface A1 is obtained The domains are adjacent. Therefore, there is an image content continuity boundary between the projection surface A1 and the geometric mapping area A1_GP extending from the projection surface A1 (that is, the content is continuously represented in the projection surface A1 and the geometric mapping area A1_GP).

The second geometric filling applied to the projection surface A2 determines another geometric mapping area A2_GP, where the geometric mapping area A2_GP is obtained by adding the contents of the area on the sphere (for example, the sphere 202 shown in Fig. 2/4) It is obtained by mapping onto the filled area PR, where the area on the sphere is adjacent to the area where the projection surface A2 is obtained. Therefore, there is an image content continuity boundary between the projection surface A2 and the geometric mapping area A2_GP extending from the projection surface A2 (that is, the content is continuously represented in the projection surface A2 and the geometric mapping area A2_GP).

After obtaining the geometric mapping areas A1_GP and A2_GP associated with the same filling area PR, the filling circuit 115 mixes the geometric mapping areas A1_GP and A2_GP to determine the pixel values of the pixels included in the filling area PR. That is, PR=f(A1_GP, A2_GP), where f() is a mixed function. For example, the mixing function f() may be an average function. Regarding each pixel in the filling area PR, the pixel value of the pixel in the filling area PR is set by the average value of the first pixel value of the pixel in the geometric mapping area A1_GP and the second pixel value of the pixel in the geometric mapping area A2_GP .

In yet another exemplary filling implementation, the filling circuit 115 sets the pixel value of the pixel included in the filling area by copying the pixel value of the pixel included in the adjacent projection surface, which is selected from the sphere. Obtained by 360VR projection. For example, the boundary pixels of one side of the projection surface A1 are copied to create filled pixels extending from one side of the projection surface A1, and the boundary pixels of one side of the projection surface A2 are copied to create filled pixels extending from one side of the projection surface A2. In other words, the filling pixels included in the first part of the filling area PR, where each filling pixel is a replica of a boundary pixel of the projection plane A1, and the filling pixels included in the second part of the filling area PR, where each filling pixel is The pixel is a replica of a boundary pixel on the projection surface A2.

If the first side of the first projection surface is connected to the first side of the second projection surface, and the There is a discontinuous border of image content between the first side of the shadow surface and the first side of the second projection surface. By inserting a filling area between the first projection surface and the second projection surface, the compressed first side can be improved. The image quality of the first side of a projection surface and the image quality of the first side of the second projection surface after compression. The projection surface included in the decoding frame IMG′ generated by the decoding circuit 122 may have better image quality. As described above, the graphic presentation circuit 124 presents and displays the output image data on the display screen 126 according to the decoded frame IMG′. Since the filled area in the decoded frame IMG' is additionally added and may not be displayed, the graphics rendering circuit 124 may discard/ignore the filled area in the decoded frame IMG' after generating the decoded frame IMG' from the decoding circuit 122 .

As shown in Figures 6 and 7, padding is added to the compact octahedral projection layout to improve image quality for compression at discontinuous boundaries of image content. However, these are for illustrative purposes only and are not meant to limit the present invention. In fact, padding can be added to other 360 VR projection layouts to improve the compressed image quality at the discontinuous boundaries of the image content. These alternative projection designs with filling belong to the scope of the present invention.

Figure 10 is a schematic diagram of a filled compact cube projection layout according to an embodiment of the present invention. The omnidirectional image/video content of the sphere is mapped to six square projection surfaces through cube projection. The square projection surface includes the left projection surface marked "L", and the orthographic projection surface marked "FR" is marked " The right projection surface of R", the top projection surface labeled "T", the back projection surface labeled "BK" and the bottom projection surface labeled "B". If the bottom edge of the left projection surface "L" and the top edge of the bottom projection surface "B" in the compact cube projection layout without filling are connected, it is between the left projection surface "L" and the bottom projection surface "B" There are discontinuous boundaries of image content. If the bottom edge of the front projection surface FR and the top edge of the back projection surface BK in the compact cube projection layout without filling are connected, there is an image content discontinuous boundary between the front projection surface FR and the back projection surface BK. If the bottom edge of the right projection surface R and the top edge of the top projection surface T in the compact cube projection layout without filling are connected, there is an image content discontinuous boundary between the right projection surface R and the top projection surface T. According to the compact cube projection layout shown in Figure 10 1002. Insert a first filling area PR_1 between the left projection surface L and the bottom projection surface B, insert a second filling area PR_2 between the front projection surface FR and the back projection surface BK, and insert the right projection surface R and the top projection surface A third filling area PR_3 is inserted between T. It is possible to generate each filled region PR_1-PR_3 by using the above interpolation method, selecting one of the geometric filling method and the copying method.

For example, the copy method used by the proposed filling technique can extend the boundary pixels of the projection surface. Therefore, the filling pixels included in the first part of the filling area inserted between the first projection surface and the second projection surface, where each filling pixel is a replica of a boundary pixel of the first projection surface, and is inserted in the first projection surface. The filling pixels included in the second part of the filling area between the projection surface and the second projection surface, wherein each filling pixel is a replica of a boundary pixel of the second projection surface.

For another example, the copy method used by the proposed filling technology can be included in the first projection surface and the second projection surface by copying, but is not connected to the filling area between the first projection surface and the second projection surface. The pixel value of the pixel is used to set the pixel value of the pixel included in the filled area. In the case where the copy method used by the proposed filling technique obtains a copy of a partial area in the projection surface. The first part of the filling area inserted between the first projection surface and the second projection surface is a copy of the partial area of the first projection surface, and the first part of the filling area inserted between the first projection surface and the second projection surface The second part is a copy of the partial area of the second projection surface, wherein the partial area of the first projection surface and the partial area of the second projection surface are not the same as the filling inserted between the first projection surface and the second projection surface. Regional connection.

For another example, the copy method used in the proposed filling technology can be set to be inserted in the first projection surface by copying the pixel values of pixels included in at least one projection surface different from the first projection surface and the second projection surface. The pixel value of the pixel included in the filled area between and the second projection surface. Taking the compact cube projection layout 1002 shown in Figure 10 as an example, at least one projection surface (for example, FR, BK, R, and/or T) can be copied, which is not any of the left projection surface L and the bottom projection surface B A) of the pixels (for example, pixels of a partial area) to set the first filling area PR_1 inserted between the left projection surface L and the bottom projection surface B. You can copy at least one projection plane (for example, L, B, R and/or T, which are not any one of the front projection surface FR and the back projection surface BK) in the pixels (for example, pixels in a partial area), to be inserted between the front projection surface FR and the back projection surface BK The second filling area PR_2, and/or the pixels in at least one projection surface (for example, L, B, FR and/or BK, which is not any one of the right projection surface R and the top projection surface T) can be copied (for example, , Pixels in a partial area) to set a third filling area PR_3 inserted between the right projection surface R and the top projection surface T.

Figure 11 is a schematic diagram of a third compact octahedral projection layout with filling according to an embodiment of the present invention. Through the octahedral projection, the omnidirectional image/video content of the sphere is mapped to eight triangular projection surfaces (marked with reference numbers "1", "2", "3", "4", "5", "6 ", "7" and "8"). The triangular projection surface "8" is divided into two right-angled triangle parts. If one side of a right-angled triangle part of the triangular projection surface "8" in the compact octahedral projection layout without filling is connected with one side of the triangular projection surface "1", then a right-angled triangle on the triangular projection surface "8" There is a discontinuous boundary between the part and the triangular projection surface "1". If the other side of the triangular projection surface "1" in the compact octahedral projection layout without filling is connected to one side of the triangular projection surface "5", then there is a graph between the triangular projection surfaces "1" and "5". Like content discontinuous boundaries. If the other side of the triangular projection surface "5" in the compact octahedral projection layout without filling is connected to one side of the triangular projection surface "2", there is a graph between the triangular projection surfaces "5" and "2". Like content discontinuous boundaries. If the other side of the triangular projection surface "2" in the compact octahedral projection layout without filling is connected to one side of the triangular projection surface "6", there is a graph between the triangular projection surfaces "2" and "6". Like content discontinuous boundaries. If the other side of the projection surface "6" in the compact octahedral projection layout without filling is connected with one side of the triangular projection surface "3", there is an image between the triangular projection surfaces "6" and "3" The content is not continuous. If the other side of the triangular projection surface "3" in the compact octahedral projection layout without filling is connected to one side of the triangular projection surface "7", there is a graph between the triangular projection surfaces "3" and "7". Like content discontinuous boundaries. If the other side of the projection surface "7" in the compact octahedral projection layout without filling is connected to one side of the triangular projection surface "4", there is an image between the triangular projection surfaces "7" and "4" The content is not continuous. If you cast a compact octahedron without filling One side of the other right-angled triangle part of the triangle projection surface "8" in the shadow layout is connected to the other side of the triangle projection "4", then the other right-angled triangle part of the triangle projection surface "8" and the triangle projection surface "4" There are discontinuous boundaries between image content.

According to the compact octahedral projection layout 1102 shown in Figure 11, the first filling area PR_1 is inserted between the triangular projection surface "1" and a right-angled triangle portion of the triangular projection surface "8", and the first filling area PR_1 is inserted on the triangular projection surface "1". Insert the second filling area PR_2 between "and "5", insert the third filling area PR_3 between the triangular projection surfaces "5" and "2", and insert the fourth area between the triangular projection surfaces "2" and "6" The filling area PR_4, the fifth filling area PR_5 is inserted between the triangular projection surfaces "6" and "3", the sixth filling area PR_6 is inserted between the triangular projection surfaces "3" and "7", and the triangular projection surface " The seventh filling area PR_7 is inserted between 7” and “4”, and the eighth filling area PR_8 is inserted between the triangular projection surface “4” and the other right-angled triangular part of the triangular projection surface “8”. Each of the filling areas PR_1-PR_8 can be generated by using the above-mentioned interpolation method, selecting one of the geometric filling method and the copying method.

In addition to the image quality of the discontinuous boundary of the compressed image content, the image quality of the compressed layout boundary can be improved by the proposed filling technique. For example, when the omnidirectional image/video content of a sphere is mapped by equirectangular projection (ERP) or equal-area projection (EAP), only a single projection surface is generated and arranged in ERP/ EAP layout. If the viewing angle of the viewport is 180 degrees, select the decoded partial area on the left border of the ERP/EAP layout and the decoded partial area on the right border of the ERP/EAP layout to combine to form the viewport area to be displayed. Since the blocks at the left border and the blocks at the right border of a typical ERP/EAP layout are independently coded, the decoded part of the area located at the left border of the ERP/EAP layout and the decoded part located at the right border of the ERP/EAP layout The combination of regions creates a viewport area, so artifacts may be generated in the viewport area along the layout boundary. In order to solve this problem, the present invention further proposes to add a padding area to the layout boundary to provide more information for compression processing.

Figure 12 is the first ERP/EAP layout with padding according to an embodiment of the present invention Schematic. A single projection surface A with a top side S_T, a bottom side S_B, a left side S_L, and a right side S_R is arranged in the ERP/EAP layout 1202 without filling. The top side S_T, bottom side S_B, left side S_L, and right side S_R are the four boundaries of the ERP/EAP layout 1202. In addition, the projection surface A in the ERP/EAP layout 1202 has a first partial area P_L and a second partial area P_R. The first partial area P_L includes the boundary pixels at the left side S_L, and the second partial area P_R includes the border at the right side S_R. Pixels. The 360 VR projection layout L_VR adopted by the conversion circuit 114 can be set by the ERP/EAP layout 1202' shown in Figure 12. The ERP/EAP layout 1202' can be derived from the ERP/EAP layout 1202. The projection plane A is obtained by the equidistant rectangular projection/equal area projection of the sphere. The projection surface A, the first filling area PR_L and the second filling area PR_R are encapsulated in the ERP/EAP layout 1202'. As shown in Figure 12, the first filling area PR_L is connected to the left side S_L of the projection surface A and forms the left boundary of the ERP/EAP layout 1202', and the second filling area PR_R is connected to the right side S_R of the projection surface A, and Form the right boundary of the ERP/EAP layout 1202'. For example, the width of the first filling area PR_L may be 8 pixels, and the width of the second filling area PR_R may be 8 pixels. Since the left side S_L and the right side S_R of the projection surface A are opposite sides, the first filling area PR_L is not connected to the right side S_R of the projection surface A, and the second filling area PR_R is not connected to the left side S_L of the projection surface A.

In this embodiment, the first filling area PR_L is a copy of the second partial area P_R of the projection surface A, and the second filling area PR_R is a copy of the first partial area P_L of the projection surface A. Therefore, the pixels in the first filling area PR_L include the boundary pixels at the right side S_R of the projection surface A, but do not include the boundary pixels at the left side S_L of the projection surface A; the pixels in the second filling area PR_R include the left side of the projection surface A The boundary pixels at the side S_L, but the boundary pixels at the right side S_R of the projection plane A are not included. Due to the inherent characteristics of equidistant rectangular projection/equal area projection, there is an image content continuity boundary between the first filled area PR_L encapsulated in the ERP/EAP layout 1202' and the projection surface A (that is, the content is continuously expressed in Projection surface A and the first filling area PR_L), and there is an image content continuity boundary between the projection surface A and the second filling area PR_R encapsulated in the ERP/EAP layout 1202' (that is, the content is continuous on the surface Shown in the projection surface A and the second filling area PR_R).

In addition to adding the filled area to the left and right sides of the projection surface obtained by equidistant rectangular projection/equal area projection, you can also add the filled area to the top and bottom edges of the projection surface to provide more for the compression process information.

Figure 13 is a schematic diagram of a second ERP/EAP layout with filling according to an embodiment of the present invention. A single projection surface A with a top side S_T, a bottom side S_B, a left side S_L, and a right side S_R is arranged in the ERP/EAP layout 1302 without filling. The top side S_T, bottom side S_B, left side S_L and right side S_R are the four boundaries of the ERP/EAP layout 1302. In addition, the projection surface A in the ERP/EAP layout 1302 has a plurality of image areas (labeled "1", "2", "3", "4", "5", "6", "7" and " 8"). The image area "1"-"3" forms a partial area and includes the boundary pixels at the top edge S_T. The image area "3"-"5" forms a partial area and includes the boundary pixels at the right side S_R. The image area "5"-"7" forms a partial area and includes the boundary pixels at the bottom edge S_B. The image areas "1", "8" and "7" form a partial area and include the boundary pixels at the left side S_L.

The 360 VR projection layout L_VR adopted by the conversion circuit 114 can be set by the ERP/EAP layout 1302' shown in FIG. The ERP/EAP layout 1302' can be derived from the ERP/EAP layout 1302. The projection plane A is obtained by the equidistant rectangular projection/equal area projection of the sphere. As shown in Figure 13, by copying the image area "3"-"5" of the projection surface A, the first filling area connected to the left side S_L of the projection surface A is generated; by copying the image area of the projection surface A "1", "8" and "7" to generate the second filling area connected with the right side S_R of the projection surface A; the copied part of the area is obtained by copying the image area "1"-"3" of the projection surface A , And then flip (flipping) the copied partial area to generate a third filling area connected to the top edge S_T of the projection surface A; and obtain the copied part by copying the image area "5"-"7" of the projection surface A Area, and then flip the copied partial area to generate a fourth filled area connected to the bottom edge S_B of the projection surface A.

In order to make the shape of the ERP/EAP layout 1302' become rectangular, copy the image area "3" Obtain the copied filled area, and then flip the copied filled area to generate the upper left filled area; copy the image area "1" to obtain the copied filled area, and then flip the copied filled area to generate the upper right filled area; Obtain the copied filling area by copying the image area "5", and then flip the copied filling area to generate the lower left corner filling area; and by copying the image area "7" to obtain the copied filling area, and then flip the copied filling area Area to generate the filled area in the lower right corner.

Due to the inherent characteristics of equidistant rectangular projection/equal area projection, there is a continuity boundary of image content between the upper left filled area and the first filled area, and there is a continuous image content between the upper left filled area and the third filled area There is an image content continuity boundary between the upper right filled area and the second filled area, and there is an image content continuity border between the upper right filled area and the third filled area, and the lower left filled area and the second An image content continuity boundary exists between a filled area, an image content continuity boundary exists between the lower left filled area and the fourth filled area, and an image content continuity exists between the lower right filled area and the second filled area There is an image content continuity boundary between the filling area in the lower right corner and the fourth filling area.

As shown in Figure 13, the first filling area connected to the left side S_L of the projection surface A forms part of the left boundary of the ERP/EAP layout 1302', and the second filling area connected to the right side S_R of the projection surface A forms the ERP /EAP layout 1302', the third filling area connected to the top edge S_T of the projection surface A forms a part of the top boundary of the ERP/EAP layout 1302', and the second area connected to the bottom edge S_B of the projection surface A The four filled areas form part of the bottom boundary of the ERP/EAP layout 1302'. Due to the inherent characteristics of equidistant rectangular projection/equal area projection, there is an image content continuity boundary between the first filled area encapsulated in the ERP/EAP layout 1302' and the projection surface A (that is, the content is continuously expressed in the first A filled area and projection surface A), there is a border of image content continuity between the second filled area encapsulated in the ERP/EAP layout 1302' and the projection surface A (that is, the content continuously represents the second filling area Area and projection surface A), there is an image content continuity boundary between the third filling area encapsulated in the ERP/EAP layout 1302' and the projection surface A (that is, the content is continuously expressed in the third filling area and Projection surface A), And there is an image content continuity boundary between the fourth filling area encapsulated in the ERP/EAP layout 1302' and the projection surface A (that is, the content is continuously expressed in the fourth filling area and the projection surface A).

Figure 14 is a schematic diagram of a third ERP/EAP layout with filling according to an embodiment of the present invention. A single projection surface A with a top side S_T, a bottom side S_B, a left side S_L, and a right side S_R is arranged in the ERP/EAP layout 1402 without filling. The top side S_T, the bottom side S_B, the left side S_L and the right side S_R are the four boundaries of the ERP/EAP layout 1402. In addition, the projection surface A in the ERP/EAP layout 1402 has a plurality of image areas (labeled "1", "2", "3", "4", "5", "6", "7" and " 8"). The image area "1"-"3" forms a partial area and includes the boundary pixels at the top edge S_T. The image area "3"-"5" forms a partial area and includes the boundary pixels at the right side S_R. The image area "5"-"7" forms a partial area and includes the boundary pixels at the bottom edge S_B. The image areas "7"-"8" and "1" form a partial area and include the boundary pixels at the left side S_L.

The 360 VR projection layout L_VR adopted by the conversion circuit 114 can be set by the ERP/EAP layout 1402' shown in Figure 14. The ERP/EAP layout 1402' can be derived from the typical ERP/EAP layout 1402. The projection surface A is obtained by the equidistant rectangular projection/equal area projection of the sphere. As shown in Figure 14, by copying the image area "3"-"5" of the projection surface A, the first filling area connected to the left side S_L of the projection surface A is generated; by copying the image area of the projection surface A " 1", "8" and "7" to generate the second filling area connected with the right side S_R of the projection surface A; by copying the image area "1"-"3" of the projection surface A to obtain the copied partial area, Then rotate the copied partial area by 180° to generate a third filling area connected to the top edge S_T of the projection surface A; and copy the image area "5"-"7" of the projection surface A to obtain the copy Then, rotate the copied partial area by 180° to generate the fourth filling area connected with the bottom edge S_B of the projection surface A.

In order to make the shape of the ERP/EAP layout 1402' into a rectangle, the copied filled area is obtained by copying the image area "1", and then the copied filled area is rotated by 180° to generate the upper left filled area; by copying the image Area "3" obtains the copied filled area, and then rotates the copied filled area 180°, to generate the upper right corner filling area; by copying the image area "7" to obtain the copied filling area, and then rotating the copied filling area by 180° to generate the lower left corner filling area; and by copying the image area "5" "Obtain the copied filled area, and then rotate the copied filled area 180° to generate the lower right filled area.

Due to the inherent characteristics of equidistant rectangular projection/equal area projection, there is an image content continuity boundary between the upper left corner filling area and the first filling area, and there is image content continuity between the upper left corner filling and the third filling area In the boundary area, there is an image content continuity boundary between the upper right filled area and the second filled area, and an image content continuity boundary exists between the upper right filled area and the third filled area. The lower left filled area and the second An image content continuity boundary exists between a filled area, an image content continuity boundary exists between the lower left filled area and the fourth filled area, and an image content continuity exists between the lower right filled area and the second filled area There is an image content continuity boundary between the filling area in the lower right corner and the fourth filling area.

As shown in Figure 14, the first filling area connected to the left side S_L of the projection surface A forms part of the left boundary of the ERP/EAP layout 1402', and the second filling area connected to the right side S_R of the projection surface A forms the ERP /EAP layout 1402', the third filling area connected to the top edge S_T of the projection surface A forms a part of the top boundary of the ERP/EAP layout 1402', and the second area connected to the bottom edge S_B of the projection surface A The four filled areas form part of the bottom boundary of the ERP/EAP layout 1402'. Due to the inherent characteristics of equidistant rectangular projection/equal area projection, there is an image content continuity boundary between the first filled area encapsulated in the ERP/EAP layout 1402' and the projection surface A (that is, the content is continuously displayed in the first A filled area and projection surface A), there is an image content continuity boundary between the second filled area encapsulated in the ERP/EAP layout 1402' and the projection surface A (that is, the content is continuously expressed in the second filled area And projection surface A), there is an image content continuity boundary between the third filling area encapsulated in the ERP/EAP layout 1402' and the projection surface A (that is, the content is continuously expressed in the third filling area and the projection surface A), and there is an image content connection between the fourth filled area encapsulated in the ERP/EAP layout 1402' and the projection surface A The continuity boundary (that is, the content is continuously represented in the fourth filling area and the projection surface A).

As shown in Figures 12-14, padding is added to the ERP/EAP layout to improve the compressed image quality at the layout boundary. However, these are for illustrative purposes only and are not meant to limit the present invention. In practice, padding can be added to other 360 VR projection layouts to improve the compressed image quality at the layout boundary. These alternative projection designs with filling belong to the scope of the present invention.

Figure 15 is a schematic diagram of a filled octahedral projection layout according to an embodiment of the present invention. Figure 16 is a schematic diagram of a filled cube projection layout according to an embodiment of the present invention. Figure 17 is a schematic diagram of a fourth compact octahedral projection layout with filling according to an embodiment of the present invention. Figure 18 is a schematic diagram of a filled compact cube projection layout according to an embodiment of the present invention. The filled area encapsulated in the projection layout 1502/1602/1702/1802 can be generated by the above geometric filling method, which applies geometric filling to the projection surface to determine the pixel value of the pixels included in the filled area connected to the projection surface; Or it can be generated by the above copying method, which is by copying the pixel value of the boundary pixel of the projection surface, or by copying the pixel value of the pixel included in the projection surface but not connected to the filled area, or by copying the pixel value not included in the projection surface The pixel value of the pixel in is used to set the pixel value of the pixel included in the filling area connected to the projection surface.

It should be noted that the foregoing layout examples are for illustrative purposes only and are not meant to limit the present invention. In other embodiments of the present invention, the filled area may be added to the layout of other projection formats, such as pyramid projection layout, tetrahedron projection layout, and tetragon quartz-based projection layout. layout), icosahedron (icosahedron) projection layout, or hexagonal quartz-based projection layout (hexagon quartz-based projection layout) to obtain a filled 360 VR projection layout.

With the aid of the filling area added to the boundary of the projection layout, the image quality of the boundary after compression can be improved. As mentioned above, the graphics rendering circuit 124 displays the screen 126 according to the decoded frame IMG' The output image data is presented and displayed. Since the filled area in the decoded frame IMG' is additionally added and may not be displayed, the graphics rendering circuit 124 may discard/ignore the filled area in the decoded frame IMG' after generating the decoded frame IMG' from the decoding circuit 122 .

Figure 19 is a schematic diagram of a second 360 VR system according to an embodiment of the present invention. The main difference between 360 VR systems 100 and 1900 is that the conversion circuit 1914 of the source electronic device 1902 has a resampling circuit 1915, which is arranged to perform the proposed encoder side projection surface resampling function to adjust the sampling density (or sampling rate). ). For example, the encoder-side projection surface resampling function can apply down-sampling to the projection surface before encoding. For another example, the encoder-side projection surface resampling function can apply up-sampling to the projection surface before encoding. For yet another example, the encoder-side projection surface resampling function can apply resampling without size changes to the projection surface before encoding.

In some embodiments of the present invention, the 360 VR projection layout L_VR is a cube projection layout, which is a non-viewport based projection layout. Therefore, the conversion circuit 1914 obtains a plurality of square projection surfaces from the omnidirectional image/video content of the sphere, and the omnidirectional image/video content of the sphere is mapped to the plurality of square projection surfaces via cubemap projection (CMP). on. Figure 20 is a schematic diagram of six square projection planes based on the cube projection layout obtained from the cube projection of the sphere. The omnidirectional image/video content of the sphere 2002 is mapped onto the six square projection surfaces of the cube 2004 (labeled "L", "F", "R", "BK", "T" and "B"). The square projection surface "L" represents the left side of the cube 2004. The square projection surface “F” represents the front face of the cube 2004. The square projection surface “R” represents the right side of the cube 2004. The square projection surface “BK” represents the back of the cube 2004. The square projection surface "T" represents the top surface of the cube 2004. The square projection surface "B" represents the bottom surface of the cube 2004. As shown in FIG. 20, the square projection surfaces "L", "F", "R", "BK", "T" and "B" are arranged in the CMP layout 2006 corresponding to the expanded cube. The projection-based frame IMG to be encoded needs to be rectangular. If the CMP layout 2006 is directly used to create a projection-based frame IMG, the projection-based frame IMG must fill the vacant area (for example, the fill is black, gray or white). Colored area) to form a rectangular frame for encoding. Therefore, the square projection surfaces "L", "F", "R", "BK", "T" and "B" can be packaged in another CMP layout, such as 1x6 cube layout, 6x1 cube layout, 3x2 cube layout, Or 2x3 cube layout. In this way, coding efficiency can be improved.

One or more of the square projection surfaces "L", "F", "R", "BK", "T" and "B" can be processed by the resampling circuit 1915 before being packaged into the 360 VR projection layout L_VR , The 360 VR projection layout L_VR is a CMP layout (for example, 1x6 cube layout, 6x1 cube layout, 3x2 cube layout or 2x3 cube layout). For example, the re-sampling circuit 1915 uses the proposed encoder-side projection surface re-sampling function to re-sample at least a part (ie, part or all) of a square projection surface to obtain a re-sampled projection surface. The conversion circuit 1914 generates a projection-based frame IMG according to the adopted CMP layout (for example, 1x6 cube layout, 6x1 cube layout, 3x2 cube layout, or 2x3 cube layout), where the projection-based frame IMG has one or more encapsulated in The resampled projection surface in the CMP layout used. The re-sampling function of the encoder side projection surface may be a re-sampling function with non-uniform mapping or a re-sampling function with uniform mapping, which depends on actual design considerations. Further details of uniform mapping and non-uniform mapping are described below.

Please refer to Figure 21 and Figure 22 together. FIG. 21 is a schematic diagram of re-sampling a square projection surface obtained from a cube projection through uniform mapping according to an embodiment of the present invention. Figure 22 is a schematic diagram of a uniform mapping function curve according to an embodiment of the present invention. The square projection surface 2102 to be resampled may be any one of the square projection surfaces "L", "F", "R", "BK", "T", and "B" shown in Figure 20. In this example, the square projection surface 2104 is obtained by down-sampling the square projection surface 2102 in its height and width directions, where the square projection surface 2102 has a width w and a height h (h=w), and a square projection surface 2104 has a width W and a height H (W=H<h). That is, down-sampling from height h to height H is performed by uniform mapping, and down-sampling from width w to width W is performed by uniform mapping. As an example and not a limitation, the same uniform mapping function is applied to the width direction (i.e., the x-axis direction) and the height direction (i.e., the y-axis direction). For example, you can use the following formula to express Uniform mapping function in different directions.

Therefore, using the integer pixel position at the y-axis coordinate Y in the square projection surface 2104, the corresponding sampling point at the y-axis coordinate y in the square projection surface 2102 can be determined according to the uniform mapping function expressed in formula (1) . Due to the uniform mapping in the height direction, two vertically adjacent sampling points in the square projection surface 2102 are uniformly distributed with a constant distance D. Similarly, by using the integer pixel position at the x-axis coordinate X in the square projection surface 2104, the corresponding sample at the x-axis coordinate x in the square projection surface 2102 can be determined according to the uniform mapping function expressed in formula (2) point. Due to the uniform mapping in the width direction, two horizontally adjacent sampling points in the rectangular projection surface 2102 are uniformly distributed with a constant distance D′. According to formulas (1) and (2), the pixel value of the corresponding sampling position p′ found in the square projection surface 2102 is used to derive the pixel value of the position P in the square projection surface 2104.

The sampling points in the square projection surface 2102 (that is, the obtained pixel position p′) may not be in integer positions. If at least one of the x-axis coordinate x and the y-axis coordinate y of the sampling point in the square projection surface 2102 is a non-integer position, the interpolation filter (not shown) in the conversion circuit 1914 (specifically, the resampling circuit 1915) It can be applied to the integer pixels around the sampling points in the square projection surface 2102 to derive the pixel values of the sampling points.

In order to preserve more details of a specific area in the projection plane, the present invention also proposes to resample the square projection plane obtained from the cube projection through non-uniform mapping. FIG. 23 is a schematic diagram of a first example of re-sampling a square projection surface obtained from a cube projection through non-uniform mapping according to an embodiment of the present invention. The square projection surface 2302 to be resampled may be any one of the square projection surfaces "L", "F", "R", "BK", "T", and "B" shown in Figure 20. In this example, the square projection surface 2304 is from the square projection surface 2302 in its height direction (ie, y-axis direction) and width direction (Ie, the x-axis direction) is obtained by down-sampling, where the square projection surface 2302 has a width w and a height h (h=w), and the square projection surface 2304 has a width W and a height H (W=H<h) .

Using the integer pixel position at the y-axis coordinate in the square projection surface 2304, the corresponding sampling point at the y-axis coordinate in the square projection surface 2302 can be determined according to the non-uniform mapping function. As shown in Figure 23, the interval between two vertically adjacent sampling points is not constant. For example, the interval between two vertically adjacent sampling points may be one of D1, D2, D3, and D4, where D4>D3>D2>D1. Specifically, the sampling points are unevenly distributed in the height direction of the square projection surface 2302. For example, by re-sampling the first source area 2312 of the square projection surface 2302 to obtain pixels in the first re-sampling area 2322 of the square projection surface 2304; and by re-sampling the second source area 2314 of the square projection surface 2302 , To obtain the pixels in the second resampling area 2324 of the square projection surface 2304. Due to the non-uniform mapping in the height direction, the density of sampling points obtained from the first source area 2312 is different from the density of sampling points obtained from the second source area 2314. In other words, different sampling rates are used in the height direction of the square projection surface 2302. The first re-sampling area 2322 is obtained from re-sampling the first source area 2312 at the first sampling rate (or first sampling density) in the height direction, and the second re-sampling area 2324 is obtained from the second source area 2312 in the height direction. The sampling rate (or second sampling density) is obtained by re-sampling the second source region 2314, where the second sampling rate (or second sampling density) is different from the first sampling rate (or first sampling density).

Similarly, by using integer pixel positions at the x-axis coordinates on the square rectangular projection surface 2304, the corresponding sampling points at the x-axis coordinates on the square projection surface 2302 can be determined through a non-uniform mapping function. As shown in Figure 23, the interval between two horizontally adjacent sampling points is not constant. For example, the interval between two horizontally adjacent sampling points may be one of D1', D2', D3' and D4', where D4'>D3'>D2'>D1'. Specifically, the sampling points are unevenly distributed in the width direction of the square projection surface 2302. For example, by re-sampling the first source area 2316, the pixels in the first re-sampling area 2326 of the square projection surface 2304 are obtained; and by re-sampling the second source area of the square projection surface 2302 2318 performs resampling to obtain pixels in the second resampling area 2328 of the square projection surface 2304. Due to the non-uniform mapping in the width direction, the density of sampling points obtained from the first source area 2316 is different from the density of sampling points obtained from the second source area 2318. In other words, different sampling rates are used in the width direction of the square projection surface 2302. The first re-sampling area 2326 is obtained from re-sampling the first source area 2316 at the first sampling rate (or first sampling density) in the width direction, and the second re-sampling area 2326 is obtained from the second source area 2316 in the width direction. The sampling rate (or second sampling density) is obtained by re-sampling the second source region 2318, where the second sampling rate (or second sampling density) is different from the first sampling rate (or first sampling density).

The pixel value of the position P in the square projection surface 2304 is derived based on the non-uniform mapping function used in the x-axis direction and the y-axis direction using the pixel value of the corresponding sampling position p′ found in the square projection surface 2302. The sampling points in the square projection surface 2302 (ie, the obtained pixel position p′) may not be in integer positions. If at least one of the x-axis coordinate x and the y-axis coordinate y of the sampling point in the square projection surface 2302 is a non-integer position, the interpolation filter (not shown) in the conversion circuit 1914 (specifically, the resampling circuit 1915) It can be applied to the integer pixels around the sampling point in the square projection surface 2302 to derive the pixel value of the sampling point.

In this example, the non-uniform mapping applied to the square projection surface 2302 includes a method for mapping at least a part (ie, part or all) of the square projection surface 2302 in the first direction (for example, one of the width direction and the height direction). A first non-uniform mapping function for re-sampling and a function for re-sampling at least a part (ie, part or all) of the square projection surface 2302 in the second direction (for example, the other of the width direction and the height direction) The second non-uniform mapping function. In an exemplary design, the first non-uniform mapping function may be the same as the second non-uniform mapping function. That is, the first direction and the second direction (for example, the width direction and the height direction) may use the same non-uniform mapping curve. In another exemplary design, the first non-uniform mapping function may be different from the second non-uniform mapping function. That is, the first direction and the second direction (for example, the width direction and the height direction) can be used Different non-uniform mapping curves.

In the example shown in Figure 23, downsampling is applied to the original square projection surface through non-uniform mapping to derive the resampled square projection surface. However, this is for illustrative purposes only and is not meant to limit the present invention. Alternatively, the resampled square projection surface can be derived by applying upsampling to the original square projection surface through non-uniform mapping, or the resampled square can be derived by applying resample without size change to the original square projection surface through non-uniform mapping Projection surface. FIG. 24 is a schematic diagram of a second example of re-sampling a square projection surface obtained from a cube projection through non-uniform mapping according to an embodiment of the present invention. The square projection surface 2402 to be resampled may be any one of the square projection surfaces "L", "F", "R", "BK", "T", and "B" shown in Figure 20. In this example, resampling without changing the size is used. Therefore, the square projection surface 2404 is obtained by resampling the square projection surface 2402 in its height direction (ie, y-axis direction) and width direction (ie, x-axis direction), where the square projection surface 2402 has a width w and a height. h (h=w), and the square projection surface 2404 has a width W and a height H (W=H=h). Using the integer pixel position at the y-axis coordinate on the square rectangular projection surface 2404, the corresponding sampling point at the y-axis coordinate on the square projection surface 2402 can be determined according to the non-uniform mapping function in the y-axis direction. Using the integer pixel position at the x-axis coordinate in the square rectangular projection surface 2404, the corresponding sampling point at the x-axis coordinate in the square projection surface 2402 can be determined according to the non-uniform mapping function in the x-axis direction, where The non-uniform mapping function used in the axis direction may be the same as or different from the non-uniform mapping function used in the y-axis direction.

In fact, any non-decreasing function that passes through the coordinate points (0,0) and (1,1) can be used to realize the non-uniform mapping function. That is, the curve of the non-decreasing non-uniform mapping function starts at the coordinate point (0, 0) and ends at the coordinate point (1, 1). For example, the non-uniform mapping function can be piecewise-linear function, exponential function, quadratic function (quadratic equation function) or other functions. Figure 25 is a schematic diagram of a first non-uniform mapping function curve according to an embodiment of the present invention. Figure 26 is a schematic diagram of a second non-uniform mapping function curve according to an embodiment of the present invention. The non-uniform mapping curves shown in Figures 25-26 are for illustrative purposes only, and are not meant to limit the present invention.

Consider the case where the non-uniform mapping function is set by the quadratic equation function. The quadratic equation function can be defined as f(p)=A * p ² + B * p, where A+B=1, and p represents the source square projection in the selected direction (for example, the x-axis direction or the y-axis direction) The pixel position in the plane, and f(p) represents the pixel position in the resampled square projection plane in the selected direction. According to the experimental results, A can be set to -0.385 and B can be set to 1.385, so that the non-uniform mapping function has the best BD-rate (Bjøntegaard-Delta rate).

In addition, the non-uniform mapping functions applied to different square projection surfaces obtained by cube projection are not necessarily the same. For example, the resampling circuit 1915 performs non-uniform mapping on the first square projection surface (for example, the square projection surfaces "L", "F", "R", "BK", "T" and " B" one) at least a part (ie, part or all) is resampled to obtain the first resampled projection surface; the second projection surface (for example, the square shown in Figure 20) is non-uniformly mapped At least a part (ie, part or all) of the other of the projection surface "L", "F", "R", "BK", "T" and "B") is resampled to obtain the second resample Projection surface, where the projection-based frame IMG packaged into a 360 VR projection layout L_VR (which is a CMP layout) contains a first resampled projection surface and a second resampled projection surface, and is used to resample the first projection surface At least one non-uniform mapping function (for example, a non-uniform mapping function in the width direction and/or a non-uniform mapping function in the height direction) and at least one non-uniform mapping function for resampling the second projection surface (for example, the width direction The non-uniform mapping function and/or the non-uniform mapping function in the height direction) are different.

Figure 27 is a schematic diagram of a third 360 VR system according to an embodiment of the present invention. The main difference between 360 VR systems 1900 and 2700 is: the conversion circuit 2714 of the source electronic device 2702 has Resampling circuit 2715 and filling circuit 2716. Similar to the resampling circuit 1915 shown in Fig. 19, the resampling circuit 2715 is arranged to perform the proposed encoder-side projection surface resampling function to adjust the sampling density (or sampling rate). Similar to the filling circuit 115 shown in Figure 1, the filling circuit 2716 is arranged to generate at least one filling area for reducing artifacts. The conversion circuit 2714 creates a projection-based frame IMG by encapsulating the resampled projection surface and at least one filled area in a projection layout with filling. For example, the 360 VR projection layout L_VR is a CMP layout with filling.

Regarding the embodiment shown in Figure 19, the resampled square projection surface is encapsulated in a CMP layout without filling, such as a 1x6 cube layout, a 6x1 cube layout, a 3x2 cube layout or a 2x3 cube layout. However, the projection-based frame IMG after encoding may have artifacts due to the layout boundary of the CMP and/or the discontinuous boundary of the CMP layout. For example, a CMP layout without padding has a top layout boundary, a bottom layout boundary, a left layout boundary, and a right layout boundary. In addition, there is at least one discontinuous border of image content between two adjacent resampled square projection surfaces encapsulated in a CMP layout without filling. Around layout boundaries, discontinuous boundaries and/or sampling rate transitions, additional guard bands created by pixel filling can be inserted to reduce seam artifacts.

In the first exemplary guard band design, pixel padding may be added only at the discontinuous boundary. Figure 28 is a schematic diagram of a cube projection layout with inner boundary filling according to an embodiment of the present invention. The sub-figure (A) of Figure 2 shows the proposed 3×2 cube layout 2802 with inner boundary filling. The resampled square projection surface is marked by "0", "1", "2", "3", "4" and "5". For example, by applying non-uniform mapping to the square projection surface "F" shown in Figure 20 to generate the resampled square projection surface "0", by applying non-uniform mapping to the square projection surface "F" shown in Figure 20 L" is used to generate the resampled square projection surface "1", and the non-uniform mapping is applied to the square projection surface "R" shown in Figure 20 to generate the resampled square projection surface "2". Apply to the square projection surface "BK" shown in Figure 20 to generate a resampled square projection surface "3", by applying non-uniform mapping to The square projection surface "T" shown in Figure 20 is used to generate the resampled square projection surface "4", and the non-uniform mapping is applied to the square projection surface "B" shown in Figure 20 to generate the resampled square projection surface Face "5".

In a typical 3x2 cube projection layout without padding, if the bottom edge of the resampled square projection surface "1" is connected to the top edge of the resampled square projection surface "4", then the resampled square projection surface " There is a discontinuous border of image content between 1" and "4". In a typical 3x2 cube projection layout without padding, if the bottom edge of the resampled square projection surface "0" is connected to the top edge of the resampled square projection surface "3", then the resampled square projection surface " There is a discontinuous border of image content between 0" and "3". In a typical 3x2 cube projection layout without padding, if the bottom edge of the resampled square projection surface "2" is connected to the top edge of the resampled square projection surface "5", then the resampled square projection surface " There is a discontinuous border of image content between 2" and "5". According to the proposed 3×2 cubic projection layout with filling 2802, the filling area PR_DE1 is inserted between the resampled square projection surfaces "1" and "4", and the resampled square projection surfaces "0" and "3" are inserted. A padding area PR_DE2 is inserted between, and a padding area PR_DE3 is inserted between the resampled square projection surfaces "2" and "5".

The first filling area PR_DE1 includes the guard band of the resampled square projection surface "1" and the guard band of the resampled square projection surface "4". Therefore, the bottom of the resampled square projection surface "1" is set in the projection layout 2802. The edge is separated from the top edge of the resampled square projection surface "4". The second filling area PR_DE2 includes the guard band of the resampled square projection plane "0" and the guard band of the resampled square projection plane "3". Therefore, the bottom of the resampled square projection plane "0" is set in the projection layout 2802. The edge is separated from the top edge of the resampled square projection surface "3". The third filling area PR_DE3 includes the guard band of the resampled square projection surface "2" and the guard band of the resampled square projection surface "5". Therefore, the bottom side of the resampled square projection "2" in the projection layout 2802 Isolated from the top edge of the resampled square projection surface "5". Each protective tape has a protective tape size S _GB . Therefore, the width of each of the filling areas PR_DE1/PR_DE2/PR_DE3 is equal to 2*S _GB . For example, the width of the guard band size S _GB may be 8 pixels. It should be noted that the protective tape size S _GB is adjustable. As an option, the width of the guard band size S _GB can be 4 pixels, 16 pixels, or any number of pixels.

The sub-figure (B) of Figure 28 shows the proposed 6×1 cube layout 2804 with inner boundary filling. In a typical 6x1 cube projection layout without padding, if the right side of the resampled square projection surface "2" is connected to the left side of the resampled square projection surface "4", then the resampled square projection surface "2" There is a discontinuous boundary between image content and "4". According to the proposed 6×1 cubic projection layout with filling 2804, a filling area PR_DE is inserted between the resampled square projection surfaces "2" and "4". The filled area PR_DE includes the guard band of the resampled square projection plane "2" and the guard band of the resampled square projection plane "4". Therefore, in the projection layout 2804, the right side of the resampled square projection plane "2" is The resampled square projection surface "4" is isolated from the left. Each protective tape has a protective tape size S _GB . Therefore, the width of the filling area PR_DE is equal to 2 * S _GB . For example, the width of the guard band size S _GB may be 8 pixels. It should be noted that the protective tape size S _GB is adjustable. As an option, the width of the guard band size S _GB can be 4 pixels, 16 pixels, or any number of pixels.

In the second exemplary guard band design, padding may be added at the layout boundary and the discontinuous boundary. FIG. 29 is a schematic diagram of a cube projection layout with outer boundary filling and inner boundary filling according to an embodiment of the present invention. Figure 29, sub-figure (A) shows the proposed 3×2 cube layout 2902 with outer boundary filling and inner boundary filling. If the resampled square projection surface "0", "1", "2", "3", "4" and "5" are encapsulated in a typical 3x2 cube projection layout without padding, then the resampled square projection The top edges of the faces “1”, “0” and “2” form the top layout boundary, and the bottom edges of the resampled square projection faces “4”, “3” and “5” form the bottom layout boundary, and the resampled square projection The left sides of the planes "1" and "4" form the left layout boundary, and the right sides of the resampled square projection planes "2" and "5" form the right layout boundary. The proposed 3x2 cube layout 2902 with external boundary filling and internal boundary filling can be derived by adding external boundary filling to the proposed 3x2 cube layout 2802 with internal boundary filling. Therefore, in addition to the filled areas PR_DE1, PR_DE2, and PR_DE3 at the discontinuous boundaries, the proposed 3x2 cube layout 2902 with outer boundary filling and inner boundary filling also has There are: the top filling area PR_T connected with the top side of the resampled square projection surfaces "1", "0" and "2", and the bottom of the resampled square projection surfaces "4", "3" and "5" The bottom padding area PR_B connected by edges, the left padding area PR_L connected to the left of the resampled square projection surfaces "1" and "4", and the right side of the resampled square projection surfaces "2" and "5" Fill the area PR_R on the right.

The top filling area PR_T includes the guard band of the resampled square projection plane "1", the guard band of the resampled square projection plane "0", and the guard band of the resampled square projection plane "2". The underfill area PR_B includes the guard band of the resampled square projection plane "4", the guard band of the resampled square projection plane "3", and the guard band of the resampled square projection plane "5". The left filling area PR_L includes the guard band of the resampled square projection plane "1" and the guard band of the resampled square projection plane "4". The filling area PR_R on the right includes the guard band of the resampled square projection surface "2" and the guard band of the resampled square projection surface "5". Each protective tape has a protective tape size S _GB . Therefore, the width of each of the outer boundary filling areas PR_T/PR_B/PR_L/PR_R is equal to S _GB . For example, the width of the guard band size S _GB may be 8 pixels. It should be noted that the protective tape size S _GB is adjustable. As an option, the width of the guard band size S _GB can be 4 pixels, 16 pixels, or any number of pixels.

The subfigure (B) of Figure 29 shows the proposed 6×1 cube layout 2904 with outer boundary filling and inner boundary filling. If the resampled square projection planes "0", "1", "2", "3", "4" and "5" are packed in a typical 6x1 cube projection layout without padding, then the resampled square projection plane The top edges of "1", "0", "2", "4", "3" and "5" form the top layout boundary, and the resampled square projection surface "1", "0", "2", " The bottom edges of 4", "3" and "5" form the bottom layout boundary, the left side of the resampled square projection surface "1" forms the left layout boundary, and the right side of the resampled square projection surface "5" forms the right layout boundary . The proposed 6x1 cube layout 2904 with external boundary filling and internal boundary filling can be derived by adding external boundary filling to the proposed 6x1 cube layout 2804 with internal boundary filling. Therefore, in addition to the filled area PR_DE at the discontinuous boundary, the proposed 6x1 cube layout with outer boundary filling and inner boundary filling also has: The top filling area PR_T connected by the top edges of the projection surfaces "1", "0", "2", "4", "3" and "5", and the resampled square projection surface "1", "0", The bottom padding area PR_B connected with the bottom edges of "2", "4", "3" and "5", the left padding area PR_L connected with the left side of the resampled square projection surface "1", and the resampled square The right filling area PR_R connected to the right of the projection surface "5".

The top filling area PR_T includes the guard band of the resampled square projection plane "1", the guard band of the resampled square projection plane "0", the guard band of the resampled square projection plane "2", and the resampled square projection plane The guard band of "4", the guard band of the resampled square projection plane "3", and the guard band of the resampled square projection plane "5". The underfill area PR_B includes the guard band of the resampled square projection plane "1", the guard band of the resampled square projection plane "0", the guard band of the resampled square projection plane "2", and the resampled square projection plane The guard band of "4", the guard band of the resampled square projection plane "3", and the guard band of the resampled square projection plane "5". The left filling area PR_L includes the guard band of the resampled square projection surface "1". The filling area PR_R on the right includes the guard band of the resampled square projection surface "5". Each protective tape has a protective tape size S _GB . Therefore, the width of each of the outer boundary filling areas PR_T/PR_B/PR_L/PR_R is equal to S _GB . For example, the width of the guard band size S _GB may be 8 pixels. It should be noted that the protective tape size S _GB is adjustable. As an option, the width of the guard band size S _GB can be 4 pixels, 16 pixels, or any number of pixels.

In the third exemplary guard band design, padding may be added at the layout boundary, discontinuous boundary, and continuous boundary. FIG. 30 is a schematic diagram of another cube projection layout with outer boundary filling and inner boundary filling according to an embodiment of the present invention. The sub-figure (A) of Fig. 30 shows another proposed 3×2 cube layout 3002 with outer boundary filling and inner boundary filling. In a typical 3x2 cube projection layout without padding, if the right side of the resampled square projection surface "1" is connected to the left side of the resampled square projection surface "0", then the resampled square projection surface "1" and There is an image content continuity boundary between "0". In a typical 3x2 cube projection layout without padding, if the right side of the resampled square projection surface "0" is connected to the left side of the resampled square projection surface "2", then the There is an image content continuity boundary between the square projection planes "0" and "2". In a typical 3x2 cube projection layout without padding, if the right side of the resampled square projection surface "4" is connected to the left side of the resampled square projection surface "3", then the resampled square projection surface "4" and There are continuity boundaries of image content between "3". In a typical 3x2 cube projection layout without padding, if the right side of the resampled square projection surface "3" is connected to the left side of the resampled square projection surface "5", then the resampled square projection surface "3" and There are continuity boundaries of image content between "5". The proposed 3×2 cube layout 3002 with outer boundary filling and inner boundary filling can be derived by adding more padding to the proposed 3×2 cube layout 2902 with outer boundary filling and inner boundary filling. Therefore, in addition to the filled areas PR_DE1, PR_DE2, PR_DE3 at the discontinuous boundary and the filled areas PR_T, PR_B, PR_L, PR_R at the layout boundary, the proposed cube layout 3002 with outer boundary filling and inner boundary filling also Have: the filling area PR_CE1 connected to the right side of the resampled square projection surface "1" and the left side of the resampled square projection surface "0", and the right side of the resampled square projection surface "0" and the resampled square projection The filled area PR_CE2 connected to the left of the surface "2", the filled area PR_CE3 connected to the right of the resampled square projection surface "4" and the left of the resampled square projection surface "3", and the resampled square projection surface The filling area PR_CE4 is connected to the right side of "3" and the left side of the resampled square projection surface "5".

The filling area PR_CE1 includes the guard band of the resampled square projection surface "1" and the guard band of the resampled square projection surface "0". Therefore, in the projection layout 3002, the right side of the resampled square projection surface "1" The sampled square projection plane "0" is isolated from the left. The filled area PR_CE2 includes the guard band of the resampled square projection surface "0" and the guard band of the resampled square projection surface "2". Therefore, in the projection layout 3002, the right side of the resampled square projection surface "0" The sampled square projection plane "1" is isolated from the left. The filled area PR_CE3 includes the guard band of the resampled square projection surface "4" and the guard band of the resampled square projection surface "3". Therefore, in the projection layout 3002, the right side of the resampled square projection surface "4" The sampled square projection surface "3" is isolated from the left. The filling area PR_CE4 includes the guard band of the resampled square projection surface "3" and the guard band of the resampled square projection surface "5". Therefore, in the projection layout 3002, the right side of the resampled square projection surface "3" The sampled square projection surface "5" is isolated from the left. Each protective tape has a protective tape size S _GB . Therefore, the width of each of the padding areas PR_CE1/PR_CE2/PR_CE3/PR_CE4 is equal to 2*S _GB . For example, the width of the guard band size S _GB may be 8 pixels. It should be noted that the protective tape size S _GB is adjustable. As an option, the width of the guard band size S _GB can be 4 pixels, 16 pixels, or any number of pixels.

The sub-figure (B) of Fig. 30 shows another proposed 6×1 cube layout 3004 with outer boundary filling and inner boundary filling. In a typical 6x1 cube projection layout without padding, if the right side of the resampled square projection surface "1" is connected to the left side of the resampled square projection surface "0", then the resampled square projection surface "1" There is a continuity boundary between image content and "0". In a typical 6x1 cube projection layout without padding, if the right side of the resampled square projection surface "0" is connected to the left side of the resampled square projection surface "2", then the resampled square projection surface "0" There is a continuity boundary between image content and "2". In a typical 6x1 cube projection layout without padding, if the right side of the resampled square projection surface "4" is connected to the left side of the resampled square projection surface "3", then the resampled square projection surface "4" There is a continuity boundary between image content and "3". In a typical 6x1 cube projection layout without padding, if the right side of the resampled square projection surface "3" is connected to the left side of the resampled square projection surface "5", then the resampled square projection surface "3" There is a continuity boundary between image content and "5".

The proposed 6×1 cube layout 3004 with outer boundary filling and inner boundary filling can be derived by adding more padding to the proposed 6×1 cube layout 2904 with outer boundary filling and inner boundary filling. Therefore, in addition to the filled area PR_DE at the discontinuous boundary and the filled areas PR_T, PR_B, PR_L, PR_R at the layout boundary, the proposed cube layout 3004 with outer boundary filling and inner boundary filling also has: and resampling To the right of the square projection surface "1" The filling area PR_CE1 connected to the left of the resampled square projection surface "0", the filling area PR_CE2 connected to the right of the resampled square projection surface "0" and the left of the resampled square projection surface "2", and the resampled square projection surface "2". The filling area PR_CE3 connected to the right side of the sampled square projection surface "4" and the left side of the resampled square projection surface "3", and the right side of the resampled square projection surface "3" and the resampled square projection surface "5" "" to the left of the connected padding area PR_CE4.

The filled area PR_CE1 includes the guard band of the resampled square projection plane "1" and the guard band of the resampled square projection plane "0". In the projection layout 3004, the right side of the resampled square projection plane "1" is resampled The square projection surface "0" is isolated from the left. The filled area PR_CE2 includes the guard band of the resampled square projection plane "0" and the guard band of the resampled square projection plane "2". In the projection layout 3004, the right side of the resampled square projection plane "0" is resampled The square projection surface "2" is isolated from the left. The filled area PR_CE3 includes the guard band of the resampled square projection surface "4" and the guard band of the resampled square projection surface "3". In the projection layout 3004, the right side of the resampled square projection surface "4" is resampled The square projection surface "3" is isolated from the left. The filling area PR_CE4 includes the guard band of the resampled square projection surface "3" and the guard band of the resampled square projection surface "5". In the projection layout 3004, the right side of the resampled square projection surface "3" is resampled The square projection surface "5" is isolated from the left. Each protective tape has a protective tape size S _GB . Therefore, the width of each of the padding areas PR_CE1/PR_CE2/PR_CE3/PR_CE4 is equal to 2*S _GB . For example, the width of the guard band size S _GB may be 8 pixels. It should be noted that the protective tape size S _GB is adjustable. As an option, the width of the guard band size S _GB can be 4 pixels, 16 pixels, or any number of pixels.

In the first exemplary filling design, the filling circuit 2716 applies geometric filling to the projection surface to determine the filling area connected to the projection surface (for example, PR_DE, PR_DE1-PR_DE3, PR_T, PR_B, PR_L, PR_R, and PR_CE1-PR_CE4 One) the pixel value of the pixel included in the. Take the filled area PR_T shown in the sub-figure (A) of Figure 29 as an example, which includes the left geometric mapping area, the middle geometric mapping area, and the right geometric mapping area. The left geometric mapping area is used as the resampled area. The guard band of the square projection surface "1", the middle geometric mapping area is used as the guard band of the resampled square projection surface "0", and the right geometric mapping area is used as the guard band of the resampled square projection surface "2". The content of the area on the sphere (for example, the sphere 2002 shown in Figure 20) is mapped to the left geometric mapping area of the filled area PR_T, where the area on the sphere is adjacent to the area where the square projection surface "L" is obtained, And the resampled square projection surface "1" is obtained by applying non-uniform mapping to the square projection surface "L". The content of the area on the sphere (for example, the sphere 2002 shown in Figure 20) is mapped to the middle geometric mapping area of the filled area PR_T, where the area on the sphere is adjacent to the area where the square projection surface "F" is obtained, And the resampled square projection surface "0" is obtained by applying non-uniform mapping to the square projection surface "F". The content of the area on the sphere (for example, the sphere 2002 shown in Figure 20) is mapped to the right geometric mapping area of the filled area PR_T, where the area on the sphere is adjacent to the area where the square projection surface "R" is obtained, And the resampled square projection surface "2" is obtained by applying non-uniform mapping to the square projection surface "R". Therefore, there is image content continuity between the resampled square projection surface "1" and the geometric mapping area on the left side of the filling area PR_T, and there is continuity between the resampled square projection surface "0" and the geometric mapping area in the middle of the filling area PR_T. There is continuity of image content between them, and there is continuity of image content between the resampled square projection surface "2" and the geometric mapping area on the right side of the filling area PR_T. That is to say, the content continuously represents the resampled projection surface "1" and the filled area PR_T in the left few mapping areas, and the content continuously represents the resampled square projection surface "0" and the middle geometry of the filled area PR_T. In the mapping area, the content is continuously represented in the geometric mapping area on the right side of the resampled square projection surface "2" and the filling area PR_T.

In the second exemplary filling design, the filling circuit 2716 sets the filling area (for example, PR_DE, PR_DE1-PR_DE3, PR_T, PR_B, PR_L, PR_R) by duplicating the pixel values of pixels included in the projection plane connected to the filling area. And one of PR_CE1-PR_CE4). For example, copy the boundary pixels of the projection surface to create the filled pixels of the filled area connected to the projection surface. Take the filled area PR_T shown in sub-picture (A) of Fig. 29 as an example, It includes a left copy area, a middle copy area and a right copy area. The left copy area is used as the guard band of the resampled square projection surface "1", and the middle copy area is used as the guard band "0" of the resampled square projection surface. , The copy area on the right is used as the guard band of the resampled square projection surface "2". Since the left copy area is connected to the top side of the resampled square projection surface "1", the boundary pixels on the top side of the resampled square projection surface "1" are directly copied to set the pixels in the left copy area of the filled area PR_T The pixel value. Since the middle copy area is connected to the top side of the resampled square projection surface "0", the boundary pixels on the top side of the resampled square projection surface "0" are directly copied to set the pixels of the middle copy area of the filled area PR_T The pixel value. Since the right copy area is connected to the top side of the resampled square projection surface "2", the boundary pixels located on the top side of the resampled square projection surface "2" are directly copied to set the pixels of the right copy area of the filling area PR_T The pixel value.

In the third exemplary filling design, the filling circuit 2716 sets the filling area of the first projection surface (for example, PR_DE, PR_DE1-PR_DE3) by copying the pixel values of pixels included in the second projection surface that is not connected to the filling area. , PR_T, PR_B, PR_L, PR_R, and PR_CE1-PR_CE4). For example, the first projection surface and the second projection surface correspond to adjacent surfaces of a cube in 3D space (for example, adjacent surfaces of the cube 2004 shown in FIG. 20). FIG. 31 is a schematic diagram of a filling design for generating a filling area of a projection surface by copying a partial area of another projection surface according to an embodiment of the present invention. In this example, the resampled square projection surface "0" is generated by applying the non-uniform mapping to the square projection surface "F" shown in Figure 20, and the non-uniform mapping is applied to the square projection surface "F" shown in Figure 20. The square projection surface "L" is used to generate the resampled square projection surface "1", and the non-uniform mapping is applied to the square projection surface "R" shown in Figure 20 to generate the resampled square projection surface "2". Apply the non-uniform mapping to the square projection surface "BK" shown in Figure 20 to generate the resampled square projection surface "3", and apply the non-uniform mapping to the square projection surface "T" shown in Figure 20. The resampled square projection surface "4" is generated, and the resampled square projection surface "5" is generated by applying non-uniform mapping to the square projection surface "B" shown in FIG. 20. In addition, use the sub-picture (A) shown in Fig. 29 The proposed 3×2 cube layout 2902 with outer boundary filling and inner boundary filling is shown.

Based on the image content continuity feature, the filling area PR_DE1 inserted between the resampled square projection planes "1" and "4" includes a copy of the partial area PK in the resampled square projection plane "5" and A copy of the partial area PA in the sampled square projection surface "1"; the filling area PR_DE2 inserted between the resampled square projection surface "0" and "3" includes the part of the resampled square projection surface "5" A copy of the area PM and a copy of the partial area PD in the resampled square projection surface "1"; the filling area PR_DE3 inserted between the resampled square projection surfaces "2" and "5" includes the resampled square A copy of the partial area PP in the projection surface "5" and a copy of the partial area PE in the resampled square projection surface "1".

In addition, based on the image content continuity feature, the top filling area PR_T connected to the resampled square projection surface "1", "0" and "2" includes a replica of the partial area PI of the resampled square projection surface "4" , The reproduction of the partial area PL in the resampled square projection surface "4", and the reproduction of the partial area PN in the resampled square projection surface "4"; and the resampled square projection surface "4", "3 The underfill area PR_B connected with "5" and "5" includes a copy of the partial area PC in the resampled square projection surface "2", a copy of the partial area PH in the resampled square projection surface "2", and A copy of the partial area PG in the sampled square projection surface "2"; the left fill area PR_L connected with the resampled square projection surfaces "1" and "4" includes the resampled square projection surface "3" A copy of the area PJ and a copy of the partial area PB of the resampled square projection surface "0"; the right filling area PR_R connected with the resampled square projection surfaces "2" and "5" includes the resampled square projection surface The replica of the partial area PO in "3" and the replica of the partial area PF of the resampled square projection plane "0".

In addition, corner filling is performed, that is, corner filling areas are set around some corners of the resampled projection surfaces "1", "2", "4" and "5". Specifically, the pixel value of each filling pixel in the corner filling area is derived by performing interpolation using the boundary pixels of the adjacent filling area. Taking the corner filling area 3102 as an example, the pixel value of the filling pixel C is set _by performing interpolation on the boundary pixel C _y of the copied partial area PB and the boundary pixel C _{x of the} copied partial area PC, where the filling pixel C and the boundary pixel C _y has the same y-axis coordinate, and the filling pixel C and the boundary pixel C _x have the same x-axis coordinate. The horizontal distance between the filling pixel C and the boundary pixel C _x is represented by i. The vertical distance between the filling pixel C and the boundary pixel _Cy is represented by j. The following formula can be used to express interpolation.

Regarding the embodiment shown in Figure 27, the conversion circuit 2714 has a re-sampling circuit 2715 and a filling circuit 2716, according to the 360 VR projection layout L_VR, which is set by the proposed CMP layout with filling, so that the re-sampling circuit 2715 generates The resampled square projection surface and the filled area generated from the filling circuit 2716 are encapsulated in the projection-based frame IMG. However, this is for illustrative purposes only and is not meant to limit the present invention. For example, the proposed CMP layout with filling can be used by the conversion circuit 114 shown in Figure 1. Therefore, the filled area shown in Figures 28-31 can be generated by the filling circuit 115, and the projection planes shown in Figures 28-31 are "0", "1", "2", "3", "4". ", "5" can be the square projection planes "L", "F", "R", "BK", "T", "B" shown in Figure 20. According to the 360 VR projection layout L_VR, the layout is set by the proposed CMP layout with filling, and the square projection surface "L", "F", "R", "BK", and "BK" are obtained directly through cube projection without resampling. "T", "B", and the filled area generated from the filling circuit 114 are encapsulated in a projection-based frame IMG.

The protective belt may be set according to one protective belt design selected from the first exemplary protective belt design, the second exemplary protective belt design, and the third exemplary protective belt design as described above. In addition, the width of the guard band size S _GB may be 4 pixels, 8 pixels, 16 pixels, or any number of pixels. And the guardband information can be sent in the bit stream BS for further reconstruction/presentation on the decoder side. According to the proposed syntax signaling method, the following syntax table can be used.

It should be noted that the descriptors in the above exemplary syntax table specify the parsing process of each syntax element. For example, the descriptor u(n) describes an unsigned integer using n bits.

The syntax element guard_band_width specifies the width of the guard band on the top/left/right/bottom dimensions of each projection surface in units of brightness samples. When the decoded image has 4:2:0 or 4:2:2 chroma format, guard_band_width should be an even number.

The syntax element guard_band_type specifies the pixel filling method on the guard band. When the guard band surrounding the projection surface is not specified, the syntax element guard_band_type can be set to 0 (ie, guard_band_type==0). When deriving the pixels in the guard band from the boundary pixels of the copied projection surface, the syntax element guard_band_type can be set to 1 (ie, guard_band_type==1). When the pixels in the guard band are derived from copying adjacent faces in the 3D space, the syntax element guard_band_type can be set to 2 (ie, guard_band_type == 2). When the pixels in the guard band of the projection surface are derived by applying geometric filling to the projection surface, the syntax element guard_band_type can be set to 3 (ie, guard_band_type==3).

As described above, the decoding circuit 122 of the target electronic device 104 receives the bit stream BS from the transmission device 103 (for example, a wired/wireless communication link or a storage medium), and performs a decoder function to decode the received bit stream BS A part of to generate a decoded frame IMG', which is a projection-based frame with the same 360 VR projection layout L_VR adopted by the conversion circuit 114/2714 of the source electronic device 102/2702. In the case of setting the 360 VR projection layout L_VR through a projection layout with filling (for example, a projection layout with outer boundary filling, a projection layout with inner boundary filling, or a projection layout with outer boundary filling and inner boundary filling), The decoded frame IMG' has the projection The layout boundary and/or the fill area of the surface boundary of the layout. In one embodiment, the decoding circuit 122 may crop the filled area so that only the non-filled area (for example, the omnidirectional image/video content represented in the projection surface obtained from 360 VR projection) is reconstructed, or it is projected from 360 VR and reproduced. The omnidirectional image/video content represented in the resampled projection surface derived from the sampled projection). In an alternative design, the decoding circuit 122 may be modified to perform mixing based on filled pixels in the filled area and pixels in the non-filled area. For example, the pixel value of the pixel in the projection surface can be updated by mixing the original pixel value of the pixel in the projection surface with the pixel value of the corresponding filled pixel in the filling area. For another example, the pixel value of the filled pixel in the filled area can be updated by mixing the original pixel value of the filled pixel in the filled area with the pixel value of the corresponding pixel in the projection surface.

Figure 32 is a schematic diagram of a fourth 360 VR system according to an embodiment of the present invention. The main difference between the 360 VR systems 2700 and 3200 is that the decoding circuit 3222 of the target electronic device 3204 has a mixing circuit 3224, which is configured to perform mixing after encoding. Figure 33 is a schematic diagram of a fifth 360 VR system according to an embodiment of the present invention. The main difference between the 360 VR systems 100 and 3300 is that the decoding circuit 3322 of the target electronic device 3304 has a mixing circuit 3324 configured to perform mixing after encoding.

The decoding circuit 3222/3322 is arranged to decode a part of the bitstream BS to generate a decoded frame (ie, a decoded projection-based frame) IMG', which has a 360 VR projection layout L_VR (for example, with outer boundary padding). In the projection layout, at least one projection surface and at least one filled area in the projection layout with internal boundary filling, or projection layout with external boundary filling and internal boundary filling. During the decoding process for generating the decoded frame IMG', the decoding circuit 3222/3322 uses the mixing circuit 3224/3324 to combine the decoded pixel value obtained for the first pixel included in the projection surface and the decoded pixel value included in the filling area as the first pixel value. The decoded pixel values obtained by the two pixels are mixed to reconstruct the first pixel in the projection surface. For example, by using the third exemplary filling design described above, the filling area is generated by the filling circuit 2716/115 of the encoder side (ie, the source electronic device 2702/102). Encapsulated in projection-based graph The pixel value of the second pixel in the filled area in the frame IMG is generated by copying the pixels of a partial area included in the projection surface. For another example, by using the above-mentioned first exemplary filling design, the filling circuit 2716/115 on the encoder side (ie, the source electronic device 2702/102) generates the filling area. In this way, the pixel value of the second pixel encapsulated in the filled area in the projection-based frame IMG is generated by the geometric mapping of the pixels included in the projection surface. In some embodiments of the present invention, the hybrid circuit 3224/3324 may adopt a distance-based weighting scheme.

Regarding the mixing circuit 3224, it can be used to update the pixel value of the pixel in the projection surface obtained from the resampling process. Regarding the mixing circuit 3324, it can be used to update the pixel value of the pixel in the projection surface that has not undergone the resampling process. Hybrid circuits 3224 and 3324 can use the same distance-based weighting scheme. In the following description of the distance-based weighting scheme, the term "projection surface" may refer to a projection surface obtained from the resampling process or a projection surface that has not undergone the resampling process.

Figure 34 is a schematic diagram of a decoder-side mixing operation according to an embodiment of the present invention. Assume that the fill circuit 2716/115 uses the exemplary fill design shown in Figure 31. Therefore, according to the third exemplary filling design described above, a filling area of one projection surface is obtained by copying a partial area of another projection surface. Based on the characteristics of the continuity of image content, the padding area added to the left of the projection surface "4" in Figure 34 is set by copying the partial area PB in the square projection surface "0". However, the encoding result of the partial area PB in the square projection surface "0" and the encoding result of the filling area added to the left side of the projection surface "4" are not necessarily the same. Therefore, on the decoder side (for example, the target electronic device 3204/3304), the decoded pixels obtained from the decoding of the partial area PB in the square projection surface "0" can be compared with the padding area added to the left side of the projection surface "4". The decoded pixels obtained by the decoding are mixed. If the projection surface has different widths and/or heights due to filling, the filled area needs to be resampled (for example, interpolated filling pixels) according to the ratio of the different widths and/or heights for mixing.

The reconstructed pixel value of the target pixel (ie, the source pixel having the pixel value to be updated) in the projection surface is the updated pixel value, which can be calculated by using the following formula.

In the above formula (4), S _REC represents the reconstructed pixel value (updated pixel value) of the target pixel in the projection surface (for example, pixel A in the square projection surface "0"), and S represents the value obtained for the target pixel The decoded pixel value (original pixel value), T is expressed as the decoded pixel value obtained by the corresponding filling pixel in the filling area (for example, the filling pixel A′ added to the filling area on the left side of the square projection surface "4"). M represents the filling width of the filled area, and N represents the distance between the target pixel and one side of the projection surface. In Figure 34, the distance between pixel A and the top edge of the square projection surface "0" is represented by d (N=d), and the distance between the filling pixel A'and the left side of the square projection surface "4" is represented by d'means. According to the third exemplary filling design described above, since the filling area of one projection surface is obtained by copying a part of the area in another projection surface, the filling pixel A'is located at an integer position in the filling area (ie, (x, y) , Where x and y are integer positions), and the value of d is equal to the value of d'.

However, if according to the aforementioned first exemplary filling design, the filling area of the projection surface is obtained by applying geometric filling to the projection surface, the filling pixel A'may be located at a non-integer position (ie, (x, y), in the filling area Where x is not an integer position, and/or y is not an integer position), and the value of d can be different from the value of d'. Specifically, due to geometric mapping, the 2D coordinates of the filled pixel A′ are converted from the 2D coordinates of the pixel A. That is to say, the pixel A located at an integer position (ie, (X, Y), where X and Y are integer positions) in the square projection surface "0" can be mapped to a non-integer position in the filled area (ie, (x, y), where x is not an integer position, and/or y is not an integer position) filling pixel A'. Since the pixel value of the filled pixel A'located at a non-integer position cannot be directly obtained in the filled area, the mixing circuit 3224/3324 can process the filled pixel located at the integer position in the filled area by using an interpolation filter to determine that it is located The pixel value of the filled pixel A'at a non-integer position in the filled area. After determining the pixel value of the filled pixel A′ located at a non-integer position, the above formula (4) is used to calculate the updated pixel value of the pixel A in the square projection surface “0”.

i<M。第35圖是根據本發明實施例的更新投影面中的像素的像素值所涉及的像素的權重值與像素的索引值之間關係的示意圖。假設M=4且N=i+1。上述公式(4)可以改寫如下。 In the above formula (4), N represents the distance between the target pixel and one side of the projection surface. In the first exemplary design, N is set by a positive integer value. For example, N=i+1, where i is the index (distance) counted from one side of the projection surface, and 0

i<M. FIG. 35 is a schematic diagram of the relationship between the weight value of the pixel and the index value of the pixel involved in updating the pixel value of the pixel on the projection surface according to an embodiment of the present invention. Suppose M=4 and N=i+1. The above formula (4) can be rewritten as follows.

In the above formula (5), A _i represents the decoded pixel value of a target pixel having the index i of the projection plane _obtained, A _{i, updated} value represents a reconstructed pixel (the pixel value updating) of the target pixel in the projection plane , _A'i represents the decoded pixel value obtained by the corresponding filling pixel in the filling area. As shown in FIG. 35, the target pixel right A ₃ (i.e., i = A 3 _i) of weight equal to 8 (i.e., 4 + 3 + 1), and the corresponding padding pixels A _'3 (i.e., i = 3 of a _'i) of weight is equal to 0 (i.e., 4-3-1); a ₂ target pixel (i.e., _i = a i 2) of the weight is equal to 7 (i.e., 2 + 4 + 1), corresponding to the pixels a and the filling _'2 (i.e., i = 2 is a' _i) weight weight equal to 1 (i.e., 4-2-1); target pixel a ₁ (i.e., _i = a i 1) of a weight equal to 6 (i.e., 1 + 4 + 1), and the corresponding padding pixels a _'1 (i.e., i = a 1 a' _i) of weight is equal to 2 (i.e., 4-1-1); and a target pixel a ₀ (i.e., i = a 0 to _i) 5 is equal to the weight (i.e., 0 + 4 + 1), and the filling A'0 of the pixel (i.e., i = a 0 of the _'i) of weight is equal to 3 (i.e., 4-0-1).

i<M。第36圖是根據本發明實施例的更新投影面中的像素的像素值所涉及的像素的權重值與像素的索引值之間的另一關係的示意圖。假設M=4且N=i+0.5。上述公式(4)可以改寫如下。 As shown in Figure 35, since the difference between the adjacent weight values "5" and "3" is equal to 2, the weight does not decrease from 8 to 0 with the index (distance) in a constant step size (constant step size) . To solve this problem, the present invention proposes another setting of the distance between the target pixel and one side of the projection surface. In the second exemplary design, N is set by a positive non-integer value. For example, N=i+0.5, where i is the index (distance) counted from one side of the projection surface, and 0

i<M. FIG. 36 is a schematic diagram of another relationship between the weight value of the pixel and the index value of the pixel involved in updating the pixel value of the pixel in the projection plane according to an embodiment of the present invention. Suppose M=4 and N=i+0.5. The above formula (4) can be rewritten as follows.

In the above formula (6), A _i represents the decoded pixel value of a target pixel having the index i of the projection plane _obtained, A _{i, updated} value represents a reconstructed pixel (the pixel value updating) of the target pixel in the projection plane , _A'i represents the decoded pixel value obtained by the corresponding filling pixel in the filling area. As shown in FIG. 36, the right of the target pixel A ₃ (i.e., _i = A i 3) of weight equal to 7.5 (i.e., 0.5 + 4 + 3), and the corresponding padding pixels A _'3 (i.e. ,, i = 3 the a _'i) of weight is equal to 0.5 (i.e., 4-3-0.5); the target pixel a ₂ (i.e., _i = a i 2) of the weight is equal to 6.5 (i.e., 0.5 + 2 + 4), and the corresponding pixel is filled a _'2 (i.e., i = a 2 a' _i) of weight equal to 1.5 weight (i.e., 4-2-0.5); target pixel a ₁ (i.e., _i = a i 1) of a weight equal to 5.5 (i.e., 1 + 4 +0.5), and the corresponding padding pixels a _'1 (i.e., i = a 1' of weight _i) of weight equal to 2.5 (i.e., 4-1-0.5); and a target pixel a ₀ (i.e., _i = a i 0 of ) of weight equal to 4.5 (i.e., 0.5 + 0 + 4), and filling the pixel a _'0 (i.e., i = a 0 of the' _i) of weight is equal to 3.5 (i.e., 4-0-0.5). The weight is reduced from 7.5 to 0.5 in constant steps.

For some applications, a conversion circuit can be implemented in the target electronic device to convert the decoded frame of the projection layout with the first 360VR projection format into a projection layout with a second 360VR projection format different from the first 360VR projection format The conversion frame. For example, the decoded frame generated from the decoding circuit may be a projection-based frame, which has a projection surface and a filled area encapsulated in a cubic projection layout with filling; and a conversion generated by the conversion circuit and used by the subsequent graphics rendering circuit A converted frame may be a projection-based frame with a projection surface encapsulated in a typical equidistant rectangular projection layout without filling. Pixels located at integer positions in the conversion frame (i.e., (x, y), where x and y are integer positions) can be mapped to pixels located at non-integer positions in the decoding frame (i.e., (x',y' ), where x'is not an integer position and/or y'is not an integer position). That is, when performing projection layout conversion, the conversion circuit may map the pixel values of pixels located at integer positions in the conversion frame to the pixel values of pixels located at non-integer positions in the decoding frame. Because it is in a non-integer The pixel value of the pixel at the position cannot be directly obtained in the decoded frame, so the conversion circuit can determine the pixel at the non-integer position of the decoded frame by using an interpolation filter to process the pixel at the integer position in the decoded frame. Pixel values. In the case where a pixel with a non-integer position is located at or near the layout boundary of the projection layout in the decoded frame, the pixel used by the interpolation filter may include at least one pixel selected from the projection surface and the corresponding padding At least one pixel selected in the area. Then, the pixel values of the pixels in the projection surface are updated by blending (for example, weight based on distance). However, if the pixel value of the filled pixel in the corresponding filled area is not updated by blending (for example, distance-based weight). Since the interpolation is performed using the updated pixel value of the pixel in the projection surface and the original (unupdated) pixel value of the filled pixel in the corresponding filled area, artifacts may be introduced as a result. To solve this problem, the present invention proposes another hybrid solution, that is, the pixels in the projection surface and the filled pixels in the filled area are mixed to update the pixel value.

Figure 37 is a schematic diagram of a sixth 360 VR system according to an embodiment of the present invention. The main difference between the 360 VR system 3200 and 3700 is that the mixing circuit 3724 in the decoding circuit 3722 of the target electronic device 3704 is arranged to update the pixel values of the pixels in the projection surface and the filling pixels in the filling area by mixing. And the target electronic device 3704 also includes a conversion circuit 3726, which is configured to convert a decoded frame with a 360 VR projection layout (ie, a decoded projection-based frame) IMG' into a conversion with a different 360 VR projection layout Frame (ie, converted projection-based frame) IMG". In one embodiment of the present invention, the 360 VR projection layout L_VR may be a CMP layout with filling, and the converted 360 VR projection layout may be without Filled ERP layout. However, this is for illustrative purposes only and is not meant to be a limitation of the present invention.

Figure 38 is a schematic diagram of a seventh 360 VR system according to an embodiment of the present invention. The main difference between the 360 VR system 3300 and 3800 is: the mixing circuit 3824 in the decoding circuit 3822 of the target electronic device 3804 is arranged to update the pixel value by mixing the pixels in the projection surface and the filled pixels in the filling area. And the target electronic device 3804 also includes a conversion circuit 3826, and a conversion circuit 3826 It is arranged to convert a decoded frame IMG' with a 360 VR projection layout into a converted frame IMG with a different 360 VR projection layout. In one embodiment of the present invention, the 360 VR projection layout L_VR may be a CMP layout , And the converted 360 VR projection layout can be an ERP layout. However, this is only for illustrative purposes and is not meant to limit the present invention.

Regarding the embodiment shown in Figure 37 and Figure 37, the 360 VR projection layout L_VR used by the projection-based frame IMG and the decoded frame IMG' can be a projection layout in a 360 VR projection format, which is different from the conversion Another 360 VR projection format associated with the frame IMG. For example, the different projection layouts used by the decoded frame IMG' and the converted frame IMG can be selected from a set of projection layouts, including ERP layouts, and multiple cube-based Projection layouts (for example, CMP layout, pyramid projection layout, truncated square pyramid projection layout, and viewport-based cubic projection layout), multiple triangle-based projection layouts (for example, octahedral projection layout, icosahedral projection layout , Tetrahedral projection layout, projection layout based on quadrilateral quartz, and projection layout based on hexagonal quartz), segmented sphere projection (SSP) layout, equatorial cylindrical projection layout, rotating sphere projection layout, etc.

In some embodiments of the present invention, the hybrid circuit 3724/3824 may adopt a distance-based weighting scheme. Regarding the mixing circuit 3724, it can be used to update the pixel value of the pixel in the projection surface obtained from the resampling process, and can also be used to update the pixel value of the filling pixel in the filling area. Regarding the mixing circuit 3824, it can be used to update the pixel value of the pixel in the projection surface that has not undergone the resampling process, and can also be used to update the pixel value of the filled pixel in the filled area. Hybrid circuits 3724 and 3824 can use the same distance-based weighting scheme. In the following description of the distance-based weighting scheme, the term "projection surface" may refer to a projection surface obtained from the resampling process or a projection surface that has not undergone the resampling process.

Please refer to Figure 34 again. Assume that the fill circuit 2716/115 uses the exemplary fill design shown in Figure 31. Therefore, according to the third exemplary filling design described above, by duplicating another projection surface Part of the area to obtain a fill area of the projection surface. Based on the image content continuity feature, the filled area added to the left of the projection surface "4" is set by a copy of the partial area PB in the square projection surface "0". However, the encoding result of the partial area PB in the square projection surface "0" and the encoding result of the filling area added to the left of the projection surface "4" are not necessarily the same. Therefore, on the decoder side (for example, the target electronic device 3704/3804), the decoded pixels obtained from the decoding of the partial area PB in the square projection surface "0" can be compared with the padding area added to the left of the projection surface "4". The decoded pixels obtained by the decoding are mixed. In this embodiment, the mixing circuit 3724/3824 is used to mix the original pixel values of the pixels in the partial area PB of the square projection surface "0" with the filling pixels added to the filling area on the left of the projection surface "4". The original pixel value is used to update the pixel value of the pixel in the partial area PB in the square projection surface "0"; and the mixing circuit 3724/3824 is also used to add the filling in the filling area on the left of the projection surface "4" by mixing The original pixel value of the pixel and the original pixel value of the pixel in the partial area PB of the square projection surface “0” are used to update the pixel value of the filled pixel added to the filling area on the left of the projection surface “4”. If the projection surface has different widths and/or heights due to filling, the filled area needs to be resampled (for example, interpolation of filling pixels) according to the ratio of different widths and/or heights for mixing.

The reconstructed pixel value of the target pixel (ie, the source pixel having the pixel value to be updated) in any one of the projection surface and the filled area, the updated pixel value can be calculated by using the following formula.

In the above formula (7), _S'REC represents the target pixel (for example, pixel A in the square projection surface "0" shown in Figure 34, or added to the square projection surface "0" shown in Figure 34 The reconstructed pixel value (updated pixel value) of the filled pixel A'in the filled area on the left of 4”, S'represents the decoded pixel value (original pixel value) obtained by the target pixel, and T represents the mixed with the target pixel Corresponding pixels (for example, the filled pixel A'added to the filling area on the left of the square projection surface "4" shown in Figure 34, or the pixel in the square projection surface "0" shown in Figure 34 A) The decoded pixel value obtained; M represents the filling width of the filled area, and N'represents the distance between the target pixel and one side of the projection surface. In the case where the target pixel is the pixel A on the square projection surface "0" shown in FIG. 34, N'is set by a value representing the distance between the pixel A and the top edge of the square projection surface "0". In another case where the target pixel is the filling pixel A'added to the filling area on the left of the square projection surface "4" shown in Fig. 34, N'is represented by the filling pixel A'and the square projection surface "4""To set the value of the distance between the left side.

In Figure 34, the distance between the pixel A and the top edge of the square projection surface "0" is represented by d, and the distance between the filling pixel A'and the left side of the square projection surface "4" is represented by d'. According to the third exemplary filling design described above, since the filling area of one projection surface is obtained by copying a part of the area in another projection surface, the filling pixel A'in the filling area is located at an integer position (ie, (x, y), Where x and y are integer positions), and the value of d is equal to the value of d'.

However, according to the aforementioned first exemplary filling design, if the filling area of the projection surface is obtained by applying geometric filling to the projection surface, if the target pixel (with the pixel value to be updated) is located at an integer position (ie, (x, y) , Where x and y are integer positions), then the non-target pixel (the corresponding pixel that will be mixed with the target pixel) may be located at a non-integer position (ie, (x',y'), where x'is not an integer position, and/ Or y'is not an integer position). In the case where the target pixel is the pixel A in the square projection plane "0", the non-target pixel is the filled pixel A', which may be located at a non-integer position in the filled area, so the value of d may be different from d'. Specifically, due to geometric mapping, the 2D coordinates of the filled pixel A′ are converted from the 2D coordinates of the pixel A. That is to say, the pixel A located at an integer position in the square projection surface "0" (ie, (x, y), where x and y are integer positions) may be mapped to a non-integer position in the filling area (ie, (x', y'), where x'is not an integer position, and/or y'is not an integer position) padding pixel A'. Since the pixel value of the filled pixel A'located at a non-integer position is not directly available in the filled area, the mixing circuit 3724/3824 can determine that the filled pixel located at the integer position in the filled area by using an interpolation filter to process the filled pixel The pixel value of the filled pixel A'at a non-integer position in the area. In the non- After filling the pixel value of the pixel A′ at the integer position, the above formula (7) is used to calculate the updated pixel value of the pixel A.

In another case where the target pixel is the filled pixel A'added to the filling area on the left of the square projection surface "4", the non-target pixel may be a non-integer position located in the square projection surface "0" (ie, ( x', y'), where x'is not an integer position, and/or y'is not an integer position) pixel A, so the value of d may be different from the value of d'. Specifically, due to the geometric mapping, the 2D coordinates of the pixel A are converted from the 2D coordinates of the filled pixel A′. That is to say, the filled pixel A'located at an integer position (ie, (x, y), where x and y are integer positions) in the filled area may be mapped to a non-integer position located in the square projection plane "0" (ie , (X', y'), where x'is not an integer position, and/or y'is not an integer position) pixel A. Since the pixel value of the pixel A located in a non-integer position is not directly available in the square projection surface "0", the mixing circuit 3724/3824 can process the pixel located in the integer position in the square projection surface "0" by using an interpolation filter, To determine the pixel value of the pixel A located at a non-integer position in the square projection plane "0". After determining the pixel value of the pixel A located at the non-integer position, the above formula (7) is used to calculate the updated pixel value of the filled pixel A′.

In the above formula (7), N′ represents the distance between the target pixel and one side of the projection surface, where the target pixel may be a pixel in the projection surface or a filled pixel in a filled area. In the first exemplary design, N'is set by a non-integer value. For example, N'=i+0.5, where i is the index (distance) counted from one side of the projection surface. In this embodiment, when the target pixel is a filled pixel located in a filled area outside the projection surface, i is set to a negative integer value; and when the target pixel is a pixel included in the projection surface, i is set to non-negative Integer value. FIG. 39 is a schematic diagram of the relationship between the weight value of the pixel and the index value of the pixel involved in updating the pixel value of the pixel in the projection plane and the pixel value of the filled pixel in the filled area according to an embodiment of the present invention. Suppose M=4 and N=i+0.5. The above formula (7) can be rewritten as follows.

In the above formula (8), A _i represents the decoded pixel value obtained by the target pixel with index i, A _i,updated represents the reconstructed pixel value (updated pixel value) of the target pixel, and _A'i represents the corresponding The decoded pixel value obtained by the non-target pixel. As shown in FIG. 39, the target pixel right projection plane A ₃ (i.e., _i = A i 3) of weight equal to 7.5 (i.e., 0.5 + 4 + 3), and to fill the area corresponding to the non-target pixel A _'3 (i.e., i = a 3 of the' _i) of weight is equal to 0.5 (i.e., 4-3-0.5); projection target pixel a ₂ (i.e., _i = a i 2) of the weight is equal to 6.5 (i.e. 0.5 + 4 + 2), and the non-filling region corresponding target pixel a _'2 (i.e., i = a 2' of weight _i) of weight equal to 1.5 (i.e., 4-2-0.5); projection target pixel a ₁ (i.e., _i = a i 1) of weight equal to 5.5 (i.e., 4 + 1 + 0.5), and the filling region corresponding to the non-target pixel a _'1 (i.e., i = a 1 a' _i) weight of weight equal to 2.5 (i.e., 4-1-0.5); and a projection plane of the target pixel a ₀ (i.e., _i = a i 0) of a weight equal to 4.5 (i.e., 0.5 + 4 + 0), and the corresponding unfilled region target pixel a _'0 (i.e., i = 0 is a' _i) of weight equal to 3.5 weight (i.e., 4-0-0.5).

Furthermore, as shown in FIG. 39, the right of the target pixel filling region A _-1 (i.e., i = A -1 _i) of weight equal to 3.5 (i.e., 4-1 + 0.5), and the projection plane corresponding to non-target pixels a _'-1 (i.e., with i = a -1 of the' _i) of weight is equal to 4.5 (i.e., 4 + 1-0.5); filling region of a target pixel a _-2 (i.e., i = -2 the a _I) a weight equal to 2.5 (i.e., 4-2 + 0.5), and non-target pixel a corresponding to the projection plane _'2 (i.e., i = -2 to a' _I) of the weight is equal to 5.5 (i.e. 4 + 2-0.5); filling region of a target pixel a _-3 (i.e., i = -3 to a _i) of weight is equal to 1.5 (i.e., 4-3 + 0.5), and the projection plane corresponding to the non-target pixel a _'-3 (i.e., i = -3 is a' _i) of weight equal to 6.5 weight (i.e. 4 + 3-0.5); and a target pixel a in the filling region ₄ (i.e., _i = a i -4 of ) of weight equal to 0.5 (i.e., 4-4 + 0.5), and the projection plane corresponding to the non-target pixel a _'-4 (i.e., i = -4 a' of weight _i) a weight equal to 7.5 (i.e. 4 + 4 -0.5).

In the second exemplary design, N'is set by an integer value. For example, N'=i+1, where i is the index (distance) counted from one side of the projection surface. In this embodiment, when the target pixel is a filled pixel in the filled area, i is set to a negative integer value, and when the target pixel is a pixel in the projection plane, i is set to a non-negative integer value. Figure 40 is an example of updating the pixels in the projection plane according to an embodiment of the present invention. A schematic diagram of another relationship between the weight value of the pixel involved in the pixel value and the pixel value of the filled pixel in the filled area and the index value of the pixel. Assume that M=4 and N'=i+1. The above formula (7) can be rewritten as follows.

In the above formula (9), A _i represents the decoded pixel value obtained by the target pixel with index i, A _i,updated represents the reconstructed pixel value (updated pixel value) of the target pixel, and _A'i represents the corresponding The decoded pixel value obtained by the non-target pixel. As shown in FIG. 40, the target pixel right projection plane A ₃ (i.e., _i = A i 3) is equal to 8 weight (i.e., 1 + 3 + 4), and filling the region corresponding to the non-target pixel A ' ₃ (i.e., A'i with i=3) has a weight equal to 0 (i.e., 4-3-1); the target pixel A ₂ in the projection plane (i.e., A _i with i=2) has a weight equal to 7( i.e. 4 + 2 + 1), and to fill the area corresponding to the non-target pixel a _'2 (i.e., i = a 2 a' _i) of weight is equal to 1 (i.e., 4-2-1); target projection plane pixel a ₁ (i.e., _i = a i 1) of the weight is equal to 6 (i.e. 4 + 1 + 1), and to fill the area corresponding to the non-target pixel a _'1 (i.e., i = 1 is a' _i) the weights are equal to 2 (i.e., 4-1-1); and a projection plane of the target pixel a ₀ (i.e., _i = a i 0) of a weight equal to 5 (i.e., 0 + 4 + 1), and fills the area the non-target pixel a corresponding to the weight _'0 (i.e., i = a 0 of the' _i) a weight equal to 3 (i.e., 4-0-1).

Furthermore, as shown in FIG. 40, the right of the target pixel filling region A _-1 (i.e., i = A -1 _i) of weight equal to 4 (i.e., 4-1 + 1), and the projection plane corresponding to a non-target pixel right _'-1 (i.e., i = a -1 of the' _i) a weight equal to 4 (i.e. 4 + 1-1); filling region of the target pixel a _-2 (i.e., having the i = -2 a _i) of weight is equal to 3 (i.e., 4-2 + 1), and the corresponding non-target pixel a projection plane _'2 (i.e., with i = -2 a' of weight _i) a weight equal to 5 (i.e. , 4 + 2-1); the right of the target pixel a _-3 filled region (i.e., _i = a i -3) of weight equal to 2 (i.e., 4-3 + 1), and the corresponding non-projection plane target pixel a _'-3 (i.e., i = -3 is a' _i) of weight equal to the weight 6 (i.e. 4 + 3-1); and a target pixel a in the filling region ₄ (i.e., i = -4 in a _i) the weight is equal to 1 (i.e., 4-4 + 1), and the projection plane corresponding to the non-target pixel a _'-4 (i.e., i = -4 in a' _i) of weight is equal to 7 (i.e. 4+ 4-1).

Those with ordinary knowledge in the technical field will easily observe that various modifications and changes can be made to the device and method while retaining the teaching of the present invention. Therefore, the above disclosure should be construed as being limited only by the scope and boundaries of the appended claims.

103: Transmission device

112: Video capture equipment

116: Video Encoder

124: Graphic presentation circuit

126: display screen

2702: Source Electronic Equipment

2714: Conversion circuit

2715: Resampling circuit

2716: Fill the circuit

3200: 360VR system

3204: target electronic device

3222: Decoding circuit

3224: hybrid circuit

Claims (20)

A video processing method includes: obtaining a plurality of projection surfaces from the omnidirectional content of a sphere, wherein the omnidirectional content of the sphere is mapped to the projection surfaces by cube projection, and the projection surfaces include a first projection surface; The re-sampling circuit re-samples at least a part of the first projection surface through non-uniform mapping to obtain a first re-sampled projection surface, wherein the first projection surface has a first source area and a second source area, the first The resampled projection surface has a first resampled area and a second resampled area. The first resampled area is obtained by re-sampling from the first source area with a first sampling density, and the second resampled area is obtained from the The second source area is re-sampled at a second sampling density, where the first sampling density is different from the second sampling density; a projection-based frame is generated according to the projection layout of the cube projection, wherein the projection-based frame It includes the first resampled projection surface encapsulated in the projection layout; and encoding the projection-based frame to generate a part of the bit stream. According to the video processing method described in claim 1, wherein the first projection surface and the first resampled projection surface have the same shape type. For the video processing method described in claim 1, wherein the projection surfaces further include a second projection surface, and the video processing method further includes: resampling at least a part of the second projection surface through non-uniform mapping , To obtain a second resampled projection surface, wherein the projection-based frame further includes the second resampled projection surface encapsulated in the projection layout; and at least a portion of the first projection surface for resampling at least One non-uniform mapping function is different from at least one non-uniform mapping function used to resample at least a part of the second projection surface. The video processing method according to claim 1, wherein the non-uniform map includes a first non-uniform map for re-sampling at least a part of the first projection surface in the first direction Function, and a second non-uniform mapping function for re-sampling at least a portion of the first projection surface in a second direction different from the first direction, and the second non-uniform mapping function is the same as the first non-uniform mapping The function is different. According to the video processing method described in claim 1, wherein the non-uniform mapping includes a non-uniform mapping function for resampling at least a part of the first projection surface, and the non-uniform mapping function is represented by: f(p )=A*p ² +B*p, where A+B=1, p represents the pixel position in the first projection plane in the selected direction, and f(p) represents the first projection in the selected direction A resample the pixel position in the projection plane. The video processing method described in item 5 of the scope of patent application, where A=-0.385 and B=1.385. A video processing method includes: obtaining a plurality of projection surfaces from the omnidirectional content of a sphere according to a cube map projection; generating at least one filled area through a filling circuit; and using the projection surfaces and the projection surface packaged in the projection layout of the cube projection At least one filled area to generate a projection-based frame, wherein the projection surfaces encapsulated in the projection layout include a first projection surface; the at least one filled area encapsulated in the projection layout includes the first filled area; the The first filling area is at least connected with the first projection surface, and the first filling area forms at least a part of a boundary of the projection layout; and encoding the projection-based frame to generate a part of the bit stream. The video processing method according to item 7 of the scope of patent application, wherein the at least one filled area encapsulated in the projection layout further includes a second filled area, a third filled area, and a fourth filled area, and the first filled area The area, the second filling area, the third filling area and the fourth filling area respectively form the four boundaries of the projection layout. Such as the video processing method described in item 7 of the scope of patent application, wherein the projection surfaces also include Include a second projection surface; if one side of the first projection surface is connected to one side of the second projection surface, there is an image content difference between the side of the first projection surface and the side of the second projection surface. Continuous boundary; the at least one filled area further includes a second filled area; the second filled area is connected to the one side of the first projection surface and the one side of the second projection surface, and is used to connect the first projection surface in the projection layout The one side of a projection surface is separated from the side of the second projection surface. For example, in the video processing method described in item 7 of the scope of patent application, the projection surfaces also include a second projection surface; if one side of the first projection surface is connected to one side of the second projection surface, There is an image content continuity boundary between the one side of the projection surface and the side of the second projection surface; the at least one filled area further includes a second filled area; the second filled area and the one side of the first projection surface It is connected to the one side of the second projection surface for separating the one side of the first projection surface from the one side of the second projection surface in the projection layout. According to the video processing method described in claim 7, wherein the step of generating the at least one filled area includes: applying geometric filling to the first projection surface to determine the pixels included in the first filled area Pixel values. According to the video processing method described in claim 7, wherein the step of generating the at least one filled area includes: setting the pixel value of a specific pixel included in the first projection plane by copying the pixel value of the specific pixel included in the first projection surface. The pixel value of the pixel in the filled area. According to the video processing method described in claim 7, wherein the projection surfaces encapsulated in the projection layout further include a second projection surface that is not connected to the first filling area, and the at least one filling area is generated The steps include: setting the pixel value of the pixel included in the first filling area by copying the pixel value of the specific pixel included in the second projection surface. According to the video processing method described in claim 7, wherein the filling width of the first filling area is 4 pixels. A video processing method includes: receiving a part of a bit stream; and decoding a part of the bit stream to generate a decoded projection-based frame, which has at least one projection surface and at least one projection surface encapsulated in a projection layout of 360-degree virtual reality projection At least one filled area, the step including: by mixing the decoded pixel value obtained by the first pixel included in the at least one filled area and the decoded pixel value obtained by the second pixel included in the at least one projection surface, reconstructing The first pixel in the at least one filled area. The video processing method according to claim 15, wherein the first pixel is included in the first filling area of the at least one filling area, and the second pixel is included in the first projection surface of the at least one projection surface In; a part of the bit stream is generated by encoding a projection-based frame, wherein the projection-based frame includes the at least one projection surface and the at least one encapsulated in the projection layout of the 360-degree virtual reality projection Filled area; by copying the pixel values of pixels included in the first projection surface, the pixel values of pixels included in the first filled area encapsulated in the projection-based frame are generated. The video processing method according to claim 15, wherein the first pixel is included in the first filling area of the at least one filling area, and the second pixel is included in the first projection surface of the at least one projection surface In; a part of the bit stream is generated by encoding a projection-based frame, wherein the projection-based frame includes the at least one projection surface and the at least one encapsulated in the projection layout of the 360-degree virtual reality projection Filled area; by applying geometric filling to the first projection surface, the pixel values of the pixels included in the first filled area encapsulated in the projection-based frame are generated. The video processing method according to the 15th patent application, wherein the updated pixel value of the first pixel is calculated by using the following formula:

Where _S'REC represents the updated pixel value of the first pixel, S'represents the decoded pixel value obtained by the first pixel, T'represents the decoded pixel value obtained by the second pixel, and M represents the value of the first filled area The filling width, and N represents the distance between the first pixel and one side of the projection plane connected to the first filling area. The video processing method described in item 18 of the scope of patent application, wherein N=i+0.5, i is the index of the first pixel counted from the side of the projection surface connected to the first filling area, and i is not Negative integer less than -M. According to the video processing method described in the scope of patent application, the projection layout is a cubic projection layout with filling.

Images