WO2015156149A1 - 画像処理装置および画像処理方法 - Google Patents
画像処理装置および画像処理方法 Download PDFInfo
- Publication number
- WO2015156149A1 WO2015156149A1 PCT/JP2015/059586 JP2015059586W WO2015156149A1 WO 2015156149 A1 WO2015156149 A1 WO 2015156149A1 JP 2015059586 W JP2015059586 W JP 2015059586W WO 2015156149 A1 WO2015156149 A1 WO 2015156149A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- unit
- pseudo
- depth
- view
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/302—Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/2628—Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N2213/00—Details of stereoscopic systems
- H04N2213/006—Pseudo-stereoscopic systems, i.e. systems wherein a stereoscopic effect is obtained without sending different images to the viewer's eyes
Definitions
- the present disclosure relates to an image processing apparatus and an image processing method, and more particularly, to an image processing apparatus and an image processing method that can change an image capturing method in a pseudo manner using a depth image.
- the sense of presence refers to a sense as if the user is actually in the world represented by the image.
- the real feeling refers to a feeling as if an object represented by an image actually exists.
- the display resolution is 4k resolution
- the display target image resolution is 2k resolution
- the resolution of the image to be displayed is up-converted to the resolution of the display using a linear or non-linear scaling technique mounted on the display, and displayed.
- Nonlinear scaling techniques are described in, for example, Patent Documents 1 to 4 and Non-Patent Documents 1 and 2.
- JP 2007-264456 A JP 2008-242048 A JP 2008-233765 A JP 2009-162899 A
- the present disclosure has been made in view of such a situation, and makes it possible to artificially change an image capturing method using a depth image.
- the image processing device is configured to output a predetermined value from the image based on a value of a parameter determined based on a feature of the image and a depth image representing a position of the subject in the depth direction of the image.
- An image processing apparatus includes a pseudo image generation unit that generates a predicted value of a photographed image obtained by photographing the subject by a photographing method as a pseudo photographed image.
- the image processing method according to the first aspect of the present disclosure corresponds to the image processing apparatus according to the first aspect of the present disclosure.
- the image is obtained by a predetermined shooting method.
- a predicted value of a photographed image obtained by photographing the subject is generated as a pseudo photographed image.
- An image processing apparatus includes a shooting field angle estimation unit that estimates a shooting field angle of the image based on the image and a depth image representing a position of the subject in the depth direction of the image, Based on the shooting angle of view estimated by the shooting angle of view estimation unit and the viewing angle of view of the pseudo-photographed image, an image of the shot image when shooting at the same angle of view as the viewing angle of view is obtained from the image. It is an image processing apparatus provided with the production
- the image processing method according to the second aspect of the present disclosure corresponds to the image processing apparatus according to the second aspect of the present disclosure.
- the shooting angle of view of the image is estimated based on the image and a depth image representing the position of the subject in the depth direction of the image, the estimated shooting angle of view, Based on the viewing angle of the captured image, a predicted value of the captured image when the image is captured at the same angle of view as the viewing angle is generated from the image as a pseudo captured image.
- image processing apparatuses according to the first and second aspects can be realized by causing a computer to execute a program.
- a program to be executed by a computer can be provided by being transmitted via a transmission medium or by being recorded on a recording medium.
- an image can be generated. Further, according to the first aspect of the present disclosure, it is possible to artificially change the image capturing method using the depth image.
- FIG. 3 shows the structural example of the pseudo image generation part of FIG. 3 is a flowchart for explaining processing of the image processing apparatus in FIG. 1. It is a figure explaining the 4th example of the determination method of a parameter. It is a figure explaining the 4th example of the determination method of a parameter. It is a figure explaining the 5th example of the determination method of a parameter.
- FIG. 20 is a block diagram illustrating a configuration example of a third embodiment of an image processing apparatus to which the present disclosure is applied.
- FIG. 33 is a flowchart for describing image processing of the image processing apparatus in FIG. 32. It is a figure explaining the effect by an image processing device. It is a block diagram which shows the structural example of the hardware of a computer.
- First embodiment image processing apparatus (FIGS. 1 to 25) 2.
- Second Embodiment Image Processing Device (FIGS. 26 to 31) 3.
- Third embodiment image processing apparatus (FIGS. 32 to 45) 4). Effects of the first to third embodiments (FIG. 46) 5.
- Fourth Embodiment Computer (FIG. 47)
- FIG. 1 is a block diagram illustrating a configuration example of a first embodiment of an image processing apparatus to which the present disclosure is applied.
- the image processing apparatus 10 generates a pseudo photographed image in which a photographing method is pseudo-changed from an input image that is an image input from the outside.
- the depth image generation unit 11 of the image processing apparatus 10 generates a depth image from the input image and supplies the generated depth image to the peripheral generation unit 12 and the synthesis unit 13.
- a method for generating a depth image for example, a method described in JP2013-172214A can be used.
- Interpolation region information indicating a predetermined region for extrapolation is input to the peripheral generation unit 12 from the outside.
- the peripheral generation unit 12 extrapolates an image of a peripheral region of the input image (hereinafter referred to as a peripheral image) using the input image based on the interpolation region information. Further, the peripheral generation unit 12 extrapolates a depth image (hereinafter referred to as a peripheral depth image) of the peripheral region of the depth image using the depth image supplied from the depth image generation unit 11 based on the interpolation region information. .
- the peripheral generation unit 12 supplies the peripheral image and the peripheral depth image to the synthesis unit 13.
- the synthesizing unit 13 synthesizes the peripheral image supplied from the peripheral generating unit 12 and the input image to generate a synthesized image.
- the synthesizing unit 13 synthesizes the peripheral depth image supplied from the peripheral generating unit 12 and the depth image supplied from the depth image generating unit 11 to generate a depth image of the synthesized image (hereinafter referred to as a synthesized depth image). Generate.
- the synthesis unit 13 supplies the synthesized image and the synthesized depth image to the pseudo image generation unit 16.
- the analysis unit 14 extracts a predetermined area as a feature of the input image from the input image based on the information about the input image. Based on the area, the analysis unit 14 generates an importance map that represents the importance of each pixel constituting the input image with a pixel value, and supplies the importance map to the determination unit 15.
- the determination unit 15 determines a final importance map based on an attribute signal representing an attribute of the input image input from the outside and the importance map supplied from the analysis unit 14. Based on the final importance map, the determination unit 15 determines parameter values so that the important region falls within the central visual field of the viewer viewing the pseudo photographed image, and supplies the parameter value to the pseudo image generation unit 16. To do.
- the central visual field of the viewer is, for example, a region in the screen where the angle is within ⁇ 30 degrees when centered on a recommended viewing position (for example, 1.5 times the height of the screen).
- the viewing position on which the central visual field is based is not the recommended viewing position, but the viewing position set by the viewer, the actual viewing position of the viewer measured by a camera or sensor (not shown), etc. May be.
- the pseudo image generation unit 16 generates a pseudo photographed image from the synthesized image supplied from the synthesizing unit 13 based on the synthesized depth image from the synthesizing unit 13 and the parameter value from the determining unit 15.
- the pseudo photographed image is a predicted value of a photographed image when a subject of a composite image generated from an actually photographed input image is photographed by a photographing method different from the photographing method of the input image.
- the pseudo image generation unit 16 outputs the pseudo photographed image to an external display (not shown).
- the parameters are determined for both the horizontal direction and the vertical direction, but in order to simplify the description, only the parameters for the horizontal direction are determined below.
- the vertical parameter is determined in the same manner as the horizontal parameter.
- FIG. 2 is a block diagram illustrating a configuration example of the peripheral generation unit 12 of FIG.
- 2 includes an extrapolation unit 31, a definition adjustment unit 32, an extrapolation unit 33, and an adjustment unit 34.
- the extrapolation unit 31 of the peripheral generation unit 12 performs the extrapolation performed using the input image by the extrapolation unit 33 based on the extrapolation interpolation information from the extrapolation unit 33 and the interpolation area information input from the outside. Extrapolation is performed using the depth image supplied from the depth image generation unit 11 of FIG. 1 by the same method as the interpolation.
- the extrapolation information is information related to extrapolation of the input image by the extrapolation unit 33, and represents a method of extrapolation by the extrapolation unit 33 and the like.
- a method of extrapolation a method described in Non-Patent Document 2, a hold method, a mirror method, a parallel shift method, or the like can be used.
- the extrapolation unit 31 supplies the peripheral depth image generated as a result of the extrapolation to the definition adjustment unit 32.
- the definition adjusting unit 32 determines the definition of the peripheral depth image supplied from the extrapolation unit 31 based on the extrapolation reliability indicating the accuracy of the extrapolation result of the input image supplied from the extrapolation unit 33. adjust. Specifically, the definition adjustment unit 32 performs a filtering process on the peripheral depth image using a smoothing filter (for example, a Gaussian filter) having the number of taps set based on the extrapolation reliability. The definition adjusting unit 32 supplies the adjusted peripheral depth image to the combining unit 13 in FIG.
- a smoothing filter for example, a Gaussian filter
- the extrapolation unit 33 performs extrapolation using the input image based on the interpolation area information input from the outside, and generates a peripheral image.
- the extrapolation unit 33 supplies the peripheral image to the adjustment unit 34.
- the extrapolation unit 33 generates extrapolation interpolation information and supplies the extrapolation information to the extrapolation unit 31.
- the extrapolation unit 33 generates extrapolation reliability.
- the extrapolation unit 33 represents the accuracy of matching in extrapolation with a value of 0 or more and 1 or less, and sets the extrapolation reliability.
- the extrapolation unit 33 supplies the extrapolation reliability to the definition adjustment unit 32 and the adjustment unit 34.
- the adjustment unit 34 adjusts the peripheral image supplied from the extrapolation unit 33 based on the extrapolation reliability and the interpolation area information supplied from the extrapolation unit 33, and the adjusted peripheral depth image of FIG. This is supplied to the synthesis unit 13.
- FIG. 3 is a diagram for explaining the extrapolation interpolation hold method, mirror method, and parallel shift method.
- the pixel value C of the pixel in the input image adjacent to the innermost pixel in the peripheral region of the input image is The pixel values of the innermost pixel and the pixels outside the pixel in the peripheral area of the input image are used.
- the extrapolation unit 33 when the extrapolation unit 33 performs the extrapolation in the mirror system, the pixel centering on the boundary between the innermost pixel in the peripheral region of the input image and the pixel in the input image The pixel values of the pixels in the peripheral area of the input image are determined so that the values are symmetric.
- the pixel value of the pixel in the first input image from the boundary between the innermost pixel in the peripheral region of the input image and the pixel in the input image is the pixel value C
- 1 The pixel value of the pixel in the peripheral area of the first input image is determined as the pixel value C.
- the pixel value of the pixel in the second input image from the boundary is the pixel value B
- the pixel value of the pixel in the peripheral area of the second input image from the boundary is determined as the pixel value B.
- a predetermined distance is determined from the boundary between the innermost pixel in the peripheral area of the input image and the pixel in the input image.
- the pixel values of the pixels in the range are determined as the pixel values of the pixels from the outermost pixel to the innermost pixel in the peripheral area of the input image.
- the pixel value of the pixel in the first input image from the boundary between the innermost pixel in the peripheral region of the input image and the pixel in the input image is the pixel value C
- the pixel value of the outermost pixel in the region is determined as the pixel value C.
- the pixel value of the pixel in the second input image from the boundary is the pixel value B
- the pixel value of the second pixel from the outermost side in the peripheral area of the input image is determined as the pixel value B.
- FIG. 3 illustrates the extrapolation of each method for the input image, the same applies to the extrapolation of each method for the depth image.
- FIG. 4 is a diagram for explaining extrapolation interpolation information.
- the extrapolation unit 33 As shown in FIG. 4, as a method of extrapolation of the input image 51, there is a matching method for predicting pixels in a peripheral region to be extrapolated with reference to pixels in the input image described in Non-Patent Document 2. When employed, the extrapolation unit 33 generates extrapolation information representing the matching method.
- the extrapolation unit 31 performs extrapolation of the depth image 52 by the matching method represented by the extrapolation information. Specifically, the extrapolation unit 31 refers to the pixel 52a in the depth image 52 and predicts the pixel 52b in the peripheral region to be extrapolated.
- the spatial positional relationship between the reference source pixel 52a and the reference destination pixel 52b in the depth image is the same as the spatial positional relationship between the reference source pixel 51a and the reference destination pixel 51b in the input image.
- a prediction is made so that Thereby, the consistency between the peripheral image and the peripheral depth image can be maintained.
- FIG. 5 is a block diagram illustrating a configuration example of the adjustment unit 34 of FIG.
- 5 includes a contrast adjusting unit 71, a saturation adjusting unit 72, a definition adjusting unit 73, and a brightness adjusting unit 74.
- the contrast adjustment unit 71 of the adjustment unit 34 adjusts the contrast by changing the dynamic range of the peripheral image supplied from the extrapolation unit 33 based on the extrapolation reliability supplied from the extrapolation unit 33 in FIG. . Specifically, the contrast adjustment unit 71 calculates the luminance component of each pixel of the peripheral image after contrast adjustment by performing the following expression (1) on the luminance component of each pixel of the peripheral image.
- LCnst_Y is a luminance component of each pixel of the peripheral image after contrast adjustment
- EY is a luminance component of each pixel of the peripheral image before contrast adjustment
- AVE_Y is an average value of luminance components of peripheral images before contrast adjustment
- CnstGain is a contrast gain set based on the extrapolation reliability.
- the contrast adjustment unit 71 colors an image composed of the luminance component of each pixel of the peripheral image after contrast adjustment and the color difference component of each pixel of the peripheral image supplied from the extrapolation unit 33 as the peripheral image after contrast adjustment. This is supplied to the degree adjustment unit 72.
- the saturation adjustment unit 72 adjusts the saturation of the peripheral image supplied from the contrast adjustment unit 71 based on the extrapolation reliability. Specifically, the saturation adjustment unit 72 multiplies the saturation component of each pixel of the peripheral image by a saturation gain set based on the extrapolation reliability, thereby adjusting the saturation of the peripheral image. adjust. The saturation adjusting unit 72 supplies the peripheral image after the saturation adjustment to the definition adjusting unit 73.
- the fineness adjustment unit 73 adjusts the fineness of the peripheral image supplied from the saturation adjustment unit 72 based on the extrapolation reliability. Specifically, the definition adjusting unit 73 performs a filtering process on the peripheral image using a smoothing filter (for example, a Gaussian filter) having the number of taps set based on the extrapolation reliability. The definition adjusting unit 73 supplies the filtered peripheral image to the brightness adjusting unit 74.
- a smoothing filter for example, a Gaussian filter
- the brightness adjustment unit 74 adjusts the brightness of the peripheral image supplied from the definition adjustment unit 73 based on the extrapolation reliability and the interpolation area information. Specifically, the brightness adjustment unit 74 performs the following equation (2) on the luminance (or lightness) component of each pixel of the peripheral image, whereby the luminance component of each pixel after the brightness adjustment is performed. Is calculated.
- Equation (2) PY is the luminance component of each pixel after brightness adjustment, and SmthY is the luminance component of each pixel before brightness adjustment.
- DarkOffsetPred is an offset value set based on the extrapolation reliability.
- DarkOffsetDist is an offset value set based on the interpolation area information.
- the brightness adjustment unit 74 uses, as a peripheral image after brightness adjustment, an image composed of the luminance component of each pixel after brightness adjustment and the color difference component of each pixel of the peripheral image supplied from the definition adjustment unit 73. , And supplied to the synthesis unit 13 in FIG.
- FIG. 6 is a diagram illustrating an example of contrast gain.
- the contrast gain is a value that is set so as to increase as the extrapolation reliability increases and is greater than 0 and equal to or less than 1. Therefore, when the extrapolation reliability is low, the contrast gain is small, and the contrast of the surrounding image is small. As a result, the peripheral image becomes inconspicuous.
- FIG. 7 is a diagram illustrating an example of the saturation gain.
- the saturation gain is a value that is set so as to increase as the extrapolation reliability increases and is greater than 0 and less than or equal to 1. Therefore, when the extrapolation reliability is low, the saturation gain is reduced and the saturation of the surrounding image is reduced. As a result, the peripheral image becomes inconspicuous.
- FIG. 8 is a diagram illustrating an example of the number of taps of the smoothing filter in the definition adjusting unit 73 in FIG.
- the number of taps of the smoothing filter is a value of 1 or more set so as to increase as the extrapolation reliability decreases. Therefore, when the extrapolation reliability is low, the number of taps of the smoothing filter is increased, and the blurred feeling of the peripheral image after the filter processing is increased. As a result, the peripheral image becomes inconspicuous.
- the number of taps of the smoothing filter in the definition adjustment unit 32 in FIG. 2 is set to be larger as the extrapolation reliability is smaller, as in the case of FIG. Value.
- FIG. 9 is a diagram illustrating an example of the offset value DarkOffsetPred and the offset value DarkOffsetDist in the brightness adjustment by the brightness adjustment unit 74 in FIG.
- the offset value DarkOffsetPred is set so as to decrease as the extrapolation reliability increases.
- the offset value DarkOffsetDist is set so as to increase as the distance of the corresponding pixel from the inside of the peripheral region increases.
- an overlapping area 93 that overlaps the input image 91 exists inside the peripheral area 92 of the input image 91.
- the offset value DarkOffsetDist is set so as to increase as the distance d from the inner side of the peripheral region 92, that is, the inner side of the overlapping region 93, of the corresponding pixel increases.
- the offset value DarkOffsetPred increases and the surrounding image becomes dark. As a result, the peripheral image becomes inconspicuous. Further, since the offset value DarkOffsetDist is larger as the distance d is larger, the outer pixels are darker. As a result, artifacts due to extrapolation can be reduced.
- FIG. 11 is a diagram illustrating the synthesis of the input image by the synthesis unit 13 in FIG.
- the synthesizing unit 13 sets an image of a region other than the overlapping region that overlaps the peripheral region in the input image supplied from the peripheral generating unit 12 as a synthesized image of the region.
- the synthesizing unit 13 sets an image in a region other than the overlapping region that overlaps the input image in the peripheral images supplied from the peripheral generating unit 12 as a synthesized image of the region.
- the synthesis unit 13 synthesizes the overlapping area image in the input image and the overlapping area image in the peripheral images for each pixel according to the following expression (3) to obtain a synthesized image of the overlapping area.
- Blend is the pixel value of each pixel of the composite image in the overlapping area.
- Psig is the pixel value of each pixel in the overlapping area of the peripheral image, and
- Csig is the pixel value of each pixel in the overlapping area of the input image.
- Wp and Wc are weighting factors based on the horizontal or vertical distance from the inside of the peripheral area of the input image to each pixel, and the sum of the weighting factors Wp and Wc is 1. Specifically, when the pixel corresponding to the pixel value Blend is a pixel in the overlapping region existing on the left and right of the input image, weighting factors based on the horizontal distance are used as the weighting factors Wp and Wc. On the other hand, when the pixel corresponding to the pixel value Blend is a pixel in the overlapping region existing above and below the input image, weighting factors based on the distance in the vertical direction are used as the weighting factors Wp and Wc.
- the weighting factor Wp is the horizontal direction (or vertical direction) of the corresponding pixel as shown in FIG. ) Is 1 when the position is the outermost horizontal (or vertical) position -dw of the overlapping region.
- the weight coefficient Wp decreases as the horizontal (or vertical) position of the corresponding pixel approaches 0 which is the innermost horizontal (or vertical) position of the overlapping region, and the position is 0. 0 in some cases.
- the weight coefficient Wc becomes 0 when the position of the corresponding pixel in the horizontal direction (or vertical direction) is the position -dw, and increases as the position approaches 0, and the position becomes If it is 0, it becomes 1.
- FIG. 11 demonstrated the synthesis
- FIG. 12 is a block diagram illustrating a configuration example of the analysis unit 14 of FIG.
- a caption telop detection unit 191 includes a caption telop detection unit 191, a face detection unit 192, a person detection unit 193, a saliency detection unit 194, and an estimation unit 195.
- the subtitle telop detection unit 191 of the analysis unit 14 detects a subtitle telop area that is a subtitle of the input image or a telop area based on OSD (On Screen Display) information of the input image.
- OSD On Screen Display
- a comprehensive method for multilingual video text detection Lyu, MR: Jiqiang Song; Min Cai: localization, and extraction.IEEE Transactions on Circuits and Systems for Video Technology 15 (2 ), 243-255 (2005) can be used.
- the subtitle telop detection unit 191 generates a subtitle telop map indicating the position, size, and likelihood of the detected subtitle telop area.
- a caption telop map is an image that represents the probability that each pixel constituting an input image is a caption telop area by pixel values of 0 to 255.
- the caption telop detection unit 191 supplies the generated caption telop map to the estimation unit 195.
- the face detection unit 192 detects a face area from the input image. For example, when the input image is an image at the time of visual communication, the face detection unit 192 detects a face area based on position information of each window.
- the face detection unit 192 generates a face map indicating the position, size, and likelihood of the detected face area.
- the face map is an image that represents the probability that each pixel constituting the input image is a face area with pixel values of 0 to 255.
- the face detection unit 192 supplies the generated face map to the estimation unit 195.
- the person detection unit 193 detects a person area from the input image. For example, when the input image is an image taken by the monitoring camera, the person detection unit 193 detects the person region based on the tracking target information supplied from the monitoring camera.
- the person detection unit 193 generates a person map indicating the position, size, and likelihood of the detected person area.
- the person map is an image representing the probability that each pixel constituting the input image is a person area with pixel values of 0 to 255.
- the person detection unit 193 supplies the generated person map to the estimation unit 195.
- the saliency (attraction level) detection unit 194 detects, as a main subject area, a subject area that is likely to catch a human eye from an input image based on designation from a viewer.
- a method for detecting a subject area for example, a method described in JP 2010-262506 A can be used.
- the saliency detector 194 generates a subject map indicating the position, size, and likelihood of the detected subject area.
- the subject map is an image that represents the probability that each pixel constituting the input image is a subject area by a pixel value of 0 to 255.
- the saliency detection unit 194 supplies the generated subject map to the estimation unit 195.
- the estimation unit 195 Based on the caption telop map from the caption telop detector 191, the face map from the face detector 192, the person map from the person detector 193, and the subject map from the saliency detector 194, the estimation unit 195 Generate a map.
- the estimation unit 195 supplies the generated importance map to the determination unit 15 in FIG.
- FIG. 13 is a diagram for explaining generation of the importance map by the estimation unit 195 in FIG.
- the estimation unit 195 calculates the pixel value of the caption telop map, the pixel value of the face map, the pixel value of the person map, and the pixel value of the subject map for each pixel by the following formula ( 4) Weighted average is performed, and the weighted average value is used as the pixel value of the importance map.
- Map_sig (x, y) is the pixel value of the pixel at the position (x, y) of the importance map.
- W_text, W_face, W_human, and W_sailency are weighting factors, and the sum of the weighting factors is 1.
- Map_text (x, y) is the pixel value of the pixel at the position (x, y) of the caption telop map
- Map_face (x, y) is the pixel value of the pixel at the position (x, y) of the face map .
- Map_human (x, y) is the pixel value of the pixel at the position (x, y) of the person map
- Map_sailency (x, y) is the pixel value of the pixel at the position (x, y) of the subject map.
- the weighted average value is used as the pixel value of the importance map, but the maximum value among the pixel value of the caption telop map, the pixel value of the face map, the pixel value of the person map, and the pixel value of the subject map is used. It is good also as a pixel value of a map.
- the weighted average value may be normalized so that the dynamic range is 0 to 255 to be the pixel value of the importance map.
- FIG. 14 is a block diagram illustrating a configuration example of the determination unit 15 of FIG.
- the 14 includes an analysis unit 211, an important region determination unit 212, and a parameter determination unit 213.
- the analysis unit 211 of the determination unit 15 generates an importance map based on EPG (Electronic Program Guide) program genre information or the like input as an attribute signal from the outside.
- EPG Electronic Program Guide
- the analysis unit 211 detects an area where subtitles and telops are estimated to be displayed as a highly important area, and An importance map is generated that indicates that the pixels in the region are important.
- the shooting method and shooting angle are determined in advance. For example, in a tennis game, shooting is performed at angles at which the player exists at the top and bottom of the screen. Therefore, when the program genre information represents a tennis game, the analysis unit 211 detects the upper and lower areas of the screen as areas of high importance, and the importance map indicating that the pixels in the areas are important Is generated.
- the importance map generated by the analysis unit 211 is an image that represents the importance of each pixel with a pixel value of 0 to 255. Here, the larger the pixel value, the higher the importance.
- the analysis unit 211 supplies the generated importance map to the important region determination unit 212.
- the important region determination unit 212 uses the following equation (5) to calculate the final importance map. To decide.
- BlendSigMap (x, y) is the pixel value of the pixel at the position (x, y) of the final importance map.
- W is a weighting coefficient and is a value of 0 or more and 1 or less.
- Map_sig (x, y) is the pixel value of the pixel at the position (x, y) of the importance map supplied from the estimation unit 195
- MetaSigMap (x, y) is the importance supplied from the analysis unit 211 This is the pixel value of the pixel at the map position (x, y).
- the important area determination unit 212 binarizes the final importance map and generates a binary map. At this time, the important region determination unit 212 may apply a median filter or a morphological filter as necessary for isolated removal or the like.
- the important area determination unit 212 detects a pixel area having a pixel value of 1 in the binary map, that is, a rectangular area circumscribing a white pixel as an important area, and uses important area information representing the important area as final important area information. It supplies to the parameter determination part 213.
- the parameter determination unit 213 determines parameters based on the final important region information supplied from the important region determination unit 212 so that the important region of the input image is within the central visual field of the viewer.
- the pseudo image generation unit 16 is supplied.
- FIG. 15 is a diagram illustrating an example of a binary map.
- the important area determination unit 212 binarizes each pixel value of the final importance map by setting the pixel value to 1 when the pixel value exceeds the threshold value and to 0 when the pixel value does not exceed the threshold value. Therefore, when the final importance map shown in FIG. 13 is binarized, a binary map shown in FIG. 15 is generated.
- FIG. 16 is a diagram illustrating an example of important regions detected from the binary map of FIG.
- the binary map is the map shown in FIG. 15
- a rectangular area 231 circumscribing a white area having a pixel value of 1 is detected as an important area. Then, as shown in FIG. 16, the parameters are determined so that the rectangular area 231 in the input image 232 is within the central visual field of the viewer.
- FIG. 17 is a diagram illustrating the relationship between the three-dimensional position of the subject and the two-dimensional position on the image.
- FIG. 17 is a top view of the viewer and the display for displaying the image. Also, the alternate long and short dash line represents the physical depth direction position of the subject of the image.
- the positions in the depth direction of the subjects arranged in the horizontal direction are all in front of the display surface of the display 251 as indicated by the alternate long and short dash line, but the positions of the display surface of the display 251 and the position in the depth direction of the subject are The relationship is not limited to this.
- the positions in the depth direction of the subjects arranged in the horizontal direction may be all in the back of the display surface of the display 251 or may be mixed in both the back and the front.
- the position in the depth direction of the subject corresponding to the pixel is expressed by the following equation (6) using the pixel value of the depth image of the pixel at the position xp. ).
- Depth (xp) is the position in the depth direction of the subject corresponding to the pixel at position xp. Further, depth (xp) is a pixel value of the depth image of the pixel at the position xp, and is a value from 0 to 255. Depth_Length is the dynamic range of the physical position of the subject in the depth direction indicated by the one-dot chain line in FIG.
- the image is obtained by converting the three-dimensional position of the subject into a two-dimensional position on the image by perspective transformation. Therefore, in the image, the subject in the three-dimensional space is reproduced in such a manner that the three-dimensional structure is reflected on the two-dimensional image (the near one is large and the far one is small).
- the difference (shift amount by projection) between the horizontal display position on the display 251 and the position xp of the image of the subject at the position Depth (xp) in the depth direction can be obtained by the following equation (7).
- Shiftp (xp) is the difference between the horizontal display position on the display 251 of the image of the subject at the depth Depth (xp) and the position xp.
- xc is the position (coordinates) in the horizontal direction on the display 251 of the virtual viewpoint that is the viewpoint of the image. Note that xp and xc are values that become smaller as the position on the left side, for example.
- Visual_Distance is the distance between the virtual viewpoint and the display 251, that is, the virtual viewing distance.
- the horizontal display position of the image can be changed by changing the virtual viewing distance or the horizontal position of the virtual viewpoint. Therefore, the parameter determination unit 213 determines the virtual viewing distance of the pseudo photographed image and the horizontal position of the virtual viewpoint as parameters so that the important area of the input image is within the central visual field of the viewer.
- FIG. 18 is a diagram for explaining a first example of a parameter determination method.
- FIG. 18 is a top view of the viewer and the display for displaying images.
- the alternate long and short dash line represents the physical depth direction position of the subject of the input image. The same applies to FIGS. 19, 20, and 23 to 25 described later.
- the important regions 271 and 272 exist at the end of the input image 273, for example, when the virtual visual distance is relatively small VD_A, the important regions 271 and 272 are out of the central visual field. However, for example, when the virtual viewing distance is VD_B larger than VD_A, the weight regions 271 and 272 are within the central visual field.
- the parameter determination unit 213 determines the virtual viewing distance as a parameter to VD_B so that the important regions 271 and 272 are within the central visual field.
- visibility refers to the ease of viewing an image and the ease of grasping the contents of an image.
- the pseudo image generation unit 16 may immediately change the virtual viewing distance to VD_B, or may change the virtual viewing distance stepwise from a default value smaller than VD_B to VD_B.
- the pseudo image generation unit 16 dolly-outs the subject of the composite image based on the changed virtual viewing distance and the composite depth image (the camera is moved backward relative to the subject).
- An image having motion parallax at the time of photographing) can be generated as a pseudo photographed image. As a result, the sense of depth of the pseudo photographed image can be emphasized.
- FIG. 19 is a diagram for explaining a second example of the parameter determination method.
- the occupation ratio in the central visual field of the important area 281 becomes very small, which is important.
- the region 281 is difficult to see.
- the occupation ratio in the central visual field of the important area 281 is increased, and thus the visibility of the important area 281 is improved.
- the parameter determination unit 213 determines the virtual viewing distance as a parameter as VD_D so that the important region 281 is within the central visual field and the occupation ratio of the important region 281 in the central visual field is equal to or greater than the threshold value. .
- the visibility of the important area of the pseudo photographed image can be improved.
- the display size of the pixels is small, so that the viewer does not care about the pixel structure even when approaching the display. For example, even if a large display with 4K resolution is close to 1.5 times the height of the screen, and a large display with 8K resolution is close to 0.75 times the height of the screen, the viewer has a pixel structure of the display. It does not bother me. Therefore, the viewer can view the image by approaching the display.
- the pseudo image generation unit 16 may immediately change the virtual viewing distance to VD_D, or may gradually change the virtual viewing distance from a default value larger than VD_D to VD_D.
- the pseudo image generation unit 16 dries-in the subject of the composite image based on the changed virtual viewing distance value and the composite depth image (the camera moves forward relative to the subject). It is possible to generate an image having a motion parallax when shooting) as a pseudo-photographed image. As a result, the sense of depth of the pseudo photographed image can be emphasized.
- FIG. 20 is a diagram for explaining a third example of the parameter determination method.
- the important area 291 exists at the end of the input image 273, for example, if the horizontal position of the virtual viewpoint is the position xc_A near the center, the important area 291 is out of the central field of view. . However, for example, when the horizontal position of the virtual viewpoint is the position xc_B on the relatively important area 291 side, the important area 291 falls within the central visual field.
- the parameter determining unit 213 determines the horizontal position of the virtual viewpoint as a parameter to xc_B so that the important region 291 is within the central visual field. As a result, the visibility of the important area of the pseudo photographed image can be improved.
- the pseudo image generation unit 16 may immediately change the horizontal position of the virtual viewpoint to xc_B, or gradually change from the default value larger than xc_B of the horizontal position of the virtual viewpoint to xc_B. Also good.
- the pseudo image generation unit 16 shoots the subject of the composite image on a track (based on the camera for the subject) based on the changed position value and the composite depth image.
- An image having motion parallax when shooting in parallel can be generated as a pseudo-photographed image. As a result, the sense of depth of the pseudo photographed image can be emphasized.
- FIG. 21 is a block diagram illustrating a configuration example of the pseudo image generation unit 16 of FIG.
- 21 is composed of a conversion unit 311 and a cutout unit 312.
- the conversion unit 311 generates a pseudo photographed image according to the perspective conversion model using the composite image and the composite depth image supplied from the composite unit 13 and the parameters supplied from the determination unit 15.
- the conversion unit 311 calculates the above-described equation (6) using the pixel value of the synthesized depth image as the pixel value depth (xp) to obtain the position Depth (xp) in the depth direction.
- Depth_Length may be a fixed value determined in advance, or may be a variable value that can be changed by an instruction from a viewer or the like.
- Depth_Length can change the feeling of depth (feeling of unevenness) of the pseudo photographed image.
- the conversion unit 311 obtains the difference Shiftp (xp) according to (7) described above using the depth direction position Depth (xp) and the virtual viewing distance and the horizontal position of the virtual viewpoint as parameters.
- the parameter may be either one of the virtual viewing distance and the horizontal position of the virtual viewpoint, or both.
- a fixed value determined in advance as the horizontal position of the virtual viewpoint is used.
- a fixed value determined in advance as the virtual viewing distance is used.
- the conversion unit 311 Based on the difference Shiftp (xp), the conversion unit 311 generates a pseudo photographed image by moving the pixel value of the pixel at the position xp of the composite image by the difference Shiftp (xp) and arranging (rendering) it. . Note that the pixel at the position where the pixel value is not arranged is interpolated using the pixel value of a neighboring pixel or the like.
- the conversion unit 311 supplies the pseudo photographed image to the clipping unit 312.
- the cutout unit 312 performs trimming (deletion) as necessary so that the resolution of the pseudo photographed image supplied from the conversion unit 311 becomes a predetermined resolution, and outputs the result.
- FIG. 22 is a flowchart for explaining processing of the image processing apparatus 10 of FIG.
- step S11 of FIG. 22 the image processing apparatus 10 determines whether an image is input from the outside.
- the image may be input in units of frames, or may be input in units of a plurality of frames.
- step S11 If it is determined in step S11 that no image has been input from the outside, the process waits until an image is input from the outside.
- step S12 the image processing apparatus 10 acquires the image as an input image.
- step S13 the analysis unit 14 detects a caption telop area, a face area, a person area, and a subject area of the input image, and performs an area analysis process for generating an importance map.
- the analysis unit 14 supplies the generated importance map to the determination unit 15.
- step S14 the determination unit 15 determines whether an attribute signal is input from the outside. If it is determined in step S14 that an attribute signal has been input from the outside, the process proceeds to step S15.
- step S15 the analysis unit 211 (FIG. 14) of the determination unit 15 generates an importance map based on program genre information input as an attribute signal from the outside, and advances the processing to step S16.
- step S14 determines whether attribute signal is input from the outside. If it is determined in step S14 that no attribute signal is input from the outside, the process proceeds to step S16.
- step S16 the important region determination unit 212 determines a final importance map based on the importance map from the analysis unit 14 and the importance map generated by the analysis unit 211.
- the important area determination unit 212 generates important area information based on the final importance map, and supplies the important area information to the parameter determination unit 213.
- step S17 the parameter determination unit 213 determines parameters based on the important region information so that the important region of the input image is within the central visual field of the viewer, and supplies the parameter to the pseudo image generation unit 16.
- step S18 the depth image generation unit 11 generates a depth image from the input image and supplies the generated depth image to the periphery generation unit 12 and the synthesis unit 13.
- step S19 the peripheral generation unit 12 performs a peripheral region generation process for generating a peripheral image and a peripheral depth image by performing extrapolation using the input image and the depth image based on the interpolation region information input from the outside. Do.
- the peripheral generation unit 12 supplies the peripheral image and the peripheral depth image to the synthesis unit 13.
- step S20 the synthesizing unit 13 synthesizes the peripheral image supplied from the peripheral generating unit 12 and the input image, and performs a synthesizing process for synthesizing the peripheral depth image and the depth image supplied from the peripheral generating unit 12.
- the synthesis unit 13 supplies the synthesized image and the synthesized depth image obtained as a result of the synthesis to the pseudo image generation unit 16.
- step S21 the pseudo image generation unit 16 generates a pseudo photographed image from the synthesized image supplied from the synthesis unit 13 based on the synthesized depth image from the synthesis unit 13 and the parameter from the determination unit 15.
- step S22 the pseudo image generation unit 16 outputs a pseudo photographed image.
- step S23 the image processing apparatus 10 determines whether an image is newly input. If it is determined in step S23 that an image has been newly input, the process returns to step S12, and the processes in steps S12 to S23 are repeated until no image is newly input.
- step S23 if it is determined in step S23 that no new image has been input, the process ends.
- the image processing apparatus 10 generates a pseudo photographed image from the input image based on the parameter value based on the feature of the input image and the depth image. Therefore, it is possible to artificially change the method for capturing the input image using the depth image.
- model used for generating the pseudo-photographed image may be a model other than the perspective transformation model described above.
- a parameter determination method in this case will be described below.
- (Fourth example of parameter determination method) 23 and 24 are diagrams illustrating a fourth example of the parameter determination method.
- Shifts (xp) is the difference (shift amount by scaling) between the horizontal display position on the display 251 of the subject image at the depth Depth (xp) and the position xp.
- Zpara is the scaling factor of the input image.
- xc is a position (coordinates) in the horizontal direction on the display 251 of the virtual viewpoint.
- the horizontal display position of the input image can be changed by changing the scaling rate. Therefore, when the pseudo photographed image is generated according to the scaling model, the parameter determination unit 213 determines the scaling rate as a parameter so that the important region of the input image is within the central visual field of the viewer.
- the parameter determination unit 213 sets the scaling rate as a parameter to a value larger than 1 so that the important area 331 is within the central visual field and the occupation ratio of the important area 331 in the central visual field is equal to or greater than the threshold value. decide.
- the composite image is enlarged, and the important area 331 in the pseudo photographed image is enlarged to the important area 332, so that the visibility of the important area is improved.
- the important area 332 is illustrated on the front side of the important area 331, but actually, the positions of the important area 331 and the important area 332 in the depth direction are the same. .
- the pseudo image generation unit 16 may immediately change the scaling rate to the parameter value, or may change the scaling rate from 1 to the parameter value step by step.
- the pseudo image generation unit 16 simulates the predicted value of the image when the subject of the composite image is zoomed in (telephoto) based on the changed scaling rate and the composite depth image. It can be generated as a captured image.
- the parameter determination unit 213 sets the scaling rate as a parameter to a value smaller than 1 so that the important area 341 is within the central visual field and the occupation ratio of the important area 341 in the central visual field is equal to or greater than the threshold value. decide. As a result, the composite image is reduced, and the important area 341 in the pseudo photographed image is reduced to the important area 342, so that the visibility of the important area is improved.
- the important area 342 is illustrated on the front side of the important area 341.
- the positions of the important area 341 and the important area 342 in the depth direction are the same. .
- the pseudo image generation unit 16 may immediately change the scaling rate to the parameter value, or may change the scaling rate from 1 to the parameter value step by step.
- the pseudo image generation unit 16 calculates the predicted value of the image when the subject of the composite image is zoomed out (wide angle) based on the changed scaling rate and the composite depth image. It can be generated as a pseudo image.
- the pseudo image generating unit 16 includes an adjusting unit between the converting unit 311 and the clipping unit 312. Then, the adjustment unit sets the depth of field of the pseudo photographed image using the pseudo photographed image supplied from the conversion unit 311, the composite depth image output from the composition unit 13, and the parameter supplied from the determination unit 15. adjust.
- the adjustment unit when the scaling rate as a parameter is greater than 1, the adjustment unit performs a smoothing process on the pixel values in the region on the near side and the region on the far side from the important region of the pseudo photographed image.
- the subject depth becomes shallow when the important area is in focus, and defocusing occurs outside the important area.
- the adjustment unit performs deblurring processing such as super-resolution processing and high-frequency emphasis processing on the out-of-focus area. Thereby, the subject depth of the pseudo photographed image is deepened.
- the pseudo photographed image whose depth of field is adjusted by the adjusting unit is supplied to the clipping unit 312.
- FIG. 25 is a diagram for explaining a fifth example of the parameter determination method.
- a pseudo-photographed image is generated according to a perspective transformation model that considers the line-of-sight direction.
- the depth-direction position Depth (xp) indicated by the one-dot chain line in FIG. 25 is obtained using the above-described equation (6).
- the three-dimensional position of the subject at the depth direction position Depth (xp) is converted into a two-dimensional position by perspective transformation with the line-of-sight direction as the central axis, and the subject image at the depth direction position Depth (xp) is displayed.
- the difference between the horizontal display position on 251 and the position xp is obtained. Specifically, the difference is obtained by using the depth direction position Depth (xp), the position xp, the virtual viewpoint horizontal position, the virtual viewing distance, and the viewing direction angle ⁇ e.
- the line-of-sight angle ⁇ e is an angle formed between the line of sight and the line connecting the virtual viewpoint and the center of the display 251 when the horizontal and vertical positions of the virtual viewpoint are the same as the center of the display 251. .
- the parameter determination unit 213 determines the angle ⁇ e in the line-of-sight direction as a parameter so that the important area of the input image is within the central visual field of the viewer.
- the parameter determination unit 213 determines the angle ⁇ e in the line-of-sight direction as a parameter to a value larger than 0 so that the important region 351 is within the central visual field. As a result, the visibility of the important area of the pseudo photographed image can be improved.
- the pseudo image generation unit 16 may immediately change the angle ⁇ e in the line-of-sight direction to the parameter value, or may change it in steps from 0 to the parameter value.
- the pseudo image generation unit 16 performs panning (tilt) shooting of the subject of the composite image based on the angle ⁇ e of the visual line direction after the change and the composite depth image (with respect to the subject).
- panning tilt
- a predicted value of an image when the camera is photographed while rotating the camera horizontally (vertically) can be generated as a pseudo photographed image.
- FIG. 26 is a block diagram illustrating a configuration example of the second embodiment of an image processing apparatus to which the present disclosure is applied.
- the configuration of the image processing apparatus 400 in FIG. 26 is that the analysis unit 401 is provided instead of the analysis unit 14 and the determination unit 402 is provided instead of the determination unit 15. Different from the configuration.
- the image processing apparatus 400 determines the parameter based on the camera angle at the time of capturing the input image, not the important area information.
- the analysis unit 401 estimates the camera angle at the time of shooting from the input image.
- the analysis unit 401 supplies camera angle image estimation information representing the estimated camera angle to the determination unit 402.
- the determination unit 402 estimates the final camera angle based on the sensor information detected by the built-in sensor input from the camera that captured the input image and the camera angle image estimation information supplied from the analysis unit 401. Camera angle information representing the value is determined. The determination unit 402 determines parameters based on the camera angle information and supplies them to the pseudo image generation unit 16.
- the parameters are determined for both the horizontal direction and the vertical direction, but for the sake of simplicity, only the parameters for the vertical direction are determined below.
- the horizontal parameter is determined in the same manner as the vertical parameter.
- FIG. 27 is a block diagram illustrating a configuration example of the analysis unit 401 in FIG.
- 26 includes a horizontal line detection unit 421, an empty area detection unit 422, a face direction detection unit 423, a depth image generation unit 424, and an angle estimation unit 425.
- the horizontal line detection unit 421 of the analysis unit 401 detects a horizontal line from the input image and supplies the position of the horizontal line to the angle estimation unit 425.
- the empty area detection unit 422 detects an empty area from the input image and supplies the area of the empty area to the angle estimation unit 425.
- the face direction detection unit 423 detects the face direction from the input image and supplies the face direction to the angle estimation unit 425.
- the depth image generation unit 424 generates a depth image from the input image.
- a method for generating the depth image a method using information such as the position of the vanishing point and the vanishing line is employed. This method is described in “Low complexity 3D depth map generation for stereo applications”, “Cheng-An Chien,” ICCE2011, for example.
- the depth image generation unit 424 supplies information such as the position of the vanishing point and the vanishing line used when generating the depth image to the angle estimation unit 425 as erasure information.
- the angle estimation unit 425 includes the position of the horizontal line from the horizontal line detection unit 421, the area of the sky area from the sky area detection unit 422, the face direction from the face direction detection unit 423, and the disappearance information from the depth image generation unit 424. Based on the above, camera angle image estimation information is generated and supplied to the determination unit 402 in FIG.
- FIG. 28 is a block diagram illustrating a configuration example of the angle estimation unit 425 of FIG.
- a horizontal line analysis unit 441 includes a horizontal line analysis unit 441, an empty area analysis unit 442, a face direction analysis unit 443, a vanishing point analysis unit 444, and an angle determination unit 445.
- the horizontal line analysis unit 441 of the angle estimation unit 425 determines that the camera angle of the input image is closer to tilting as the position of the horizontal line from the horizontal line detection unit 421 is closer to the lower side of the screen, and displays the virtual viewpoint.
- the vertical position at is set downward.
- the horizontal line analysis unit 441 determines that the camera angle of the input image is closer to the overhead view shooting as the position of the horizontal line is on the upper side of the screen, and the vertical position on the display of the virtual viewpoint is further upward.
- the horizontal line analysis unit 441 supplies the angle determination unit 445 with horizontal line base virtual viewpoint information indicating the position in the vertical direction on the display of the set virtual viewpoint.
- the sky area analysis unit 442 determines that the larger the area of the sky area supplied from the sky area detection unit 422 is, the closer the camera angle of the input image is to the angle shooting, and the vertical direction of the virtual viewpoint display Set the position further down.
- the sky area analysis unit 442 determines that the camera angle of the input image is closer to the overhead view shooting as the area of the sky region is smaller, and sets the vertical position on the display of the virtual viewpoint to be higher. To do.
- the empty area analysis unit 442 supplies the angle determination unit 445 with empty area-based virtual viewpoint information indicating the position in the vertical direction on the display of the set virtual viewpoint.
- the face direction analysis unit 443 determines that the camera angle of the input image is closer to tilt shooting as the face direction supplied from the face direction detection unit 423 is closer to the upward direction, and the vertical direction in the virtual viewpoint display Set the position of to a lower position. Further, the face direction analysis unit 443 determines that the closer the face direction is to the lower side, the closer the camera angle of the input image is to the overhead view shooting, and the vertical position on the display of the virtual viewpoint is further upward. Set.
- the face direction analysis unit 443 supplies the angle determination unit 445 with face direction base virtual viewpoint information representing the vertical position of the set virtual viewpoint display.
- the vanishing point analyzing unit 444 determines that the camera angle of the input image is an angle closer to tilting shooting so that there is no vanishing point on the lower side, and the virtual viewpoint Set the vertical position on the display to a lower direction. Also, the vanishing point analysis unit 444 determines that the camera angle of the input image is closer to the overhead view shooting so that there is no vanishing point on the upper side, and sets the vertical position on the display of the virtual viewpoint in the upward direction. .
- the face direction analysis unit 443 supplies vanishing point-based virtual viewpoint information representing the position in the vertical direction on the display of the set virtual viewpoint to the angle determination unit 445.
- the angle determination unit 445 uses the horizontal line-based virtual viewpoint information, the empty area-based virtual viewpoint information, the face direction-based virtual viewpoint information, and the vanishing point-based virtual viewpoint information to calculate the final result based on the input image according to the following equation (9).
- the estimated value of the vertical position on the display of a virtual viewpoint is calculated.
- All_xc is an estimated value of the position in the vertical direction on the final virtual viewpoint display based on the input image.
- Wg, Ws, Wh, and Wv are weighting factors, and are determined using the horizontal line, sky region, face direction, vanishing point, and vanishing line probability detected by the analysis unit 401. This certainty is determined by the analysis unit 401 and supplied to the determination unit 402.
- G_xc is a position (coordinates) represented by the horizon-based virtual viewpoint information
- S_xc is a position (coordinates) represented by the empty area-based virtual viewpoint information
- H_xc is a position (or its coordinates) represented by face direction-based virtual viewpoint information
- V_xc is a position (or its coordinates) represented by vanishing point-based virtual viewpoint information.
- the position All_xc is a weighted average of the positions (coordinates) represented by the horizon-based virtual viewpoint information, the empty area-based virtual viewpoint information, the face direction-based virtual viewpoint information, and the vanishing point-based virtual viewpoint information. is there.
- the angle determination unit 445 supplies information representing the estimated value of the position in the vertical direction on the display of the final virtual viewpoint based on the input image to the determination unit 402 of FIG. 26 as camera angle image estimation information.
- FIG. 29 is a diagram for explaining the determination of the position in the vertical direction on the display of the virtual viewpoint based on the vanishing information by the vanishing point analysis unit 444 of FIG.
- V1 to V3 represent the positions of vanishing points in the input image.
- vanishing point positions V1 to V3 do not exist below the input image. Therefore, when the vanishing point positions represented by the vanishing information are the positions V1 to V3 in FIG. 29A, the vanishing point analyzing unit 444 determines that the camera angle of the input image is an angle close to the tilt shooting, and the virtual viewpoint Set the vertical position on the display to the down direction.
- vanishing point positions V1 to V3 do not exist above the input image. Therefore, when the vanishing point positions represented by the vanishing information are the positions V1 to V3 in FIG. 29B, the vanishing point analysis unit 444 determines that the camera angle of the input image is close to the overhead view shooting, and the virtual viewpoint Set the vertical position on the display to the top.
- FIG. 30 is a block diagram illustrating a configuration example of the determination unit 402 in FIG.
- 30 includes an analysis unit 461, an angle determination unit 462, and a parameter determination unit 463.
- Sensor information detected by a GPS (Global Positioning System) or a gyro sensor built in the camera is input to the analysis unit 461 of the determination unit 402 from the camera that captured the input image.
- the analysis unit 461 estimates the vertical position of the virtual viewpoint display as information representing the camera angle, and supplies camera angle sensor estimation information representing the estimated position to the angle determination unit 462. .
- the angle determination unit 462 calculates the camera angle by the following equation (10). Determine information.
- Final_xc is a position represented by the camera angle information.
- W_all is a weighting coefficient and is a value of 0 or more and 1 or less.
- All_xc is a position represented by camera angle image estimation information, and
- Sensor_xc is a position represented by camera angle sensor estimation information.
- the angle determination unit 462 supplies camera angle information to the parameter determination unit 463.
- the parameter determination unit 463 supplies the position represented by the camera angle information as a parameter to the pseudo image generation unit 16 in FIG.
- This parameter is used when the pseudo image generation unit 16 generates a pseudo photographed image.
- the pseudo image generation unit 16 uses the above-described equations (6) and (7) in which the horizontal direction is replaced with the vertical direction to determine the vertical position on the display of each pixel of the input image. A difference between the vertical display position on the display of the image of the subject at the depth Depth (yp) and the position yp when yp is set is generated. Then, the pseudo image generation unit 16 captures the subject of the composite image at a position above or below the capturing position of the input image by moving and arranging the pixel values of each pixel of the composite image based on the difference. The predicted value of the captured image at this time is generated as a pseudo captured image.
- FIG. 31 is a flowchart for explaining processing of the image processing apparatus 400 of FIG.
- step S43 the analysis unit 401 estimates the camera angle at the time of shooting from the input image.
- the analysis unit 401 supplies camera angle image estimation information representing the estimated camera angle to the determination unit 402.
- step S44 the determination unit 402 determines whether sensor information is input from the outside. If it is determined in step S44 that sensor information has been input from the outside, the process proceeds to step S45.
- step S45 the analysis unit 461 (FIG. 30) of the determination unit 402 estimates a camera angle based on sensor information input from the outside.
- the analysis unit 461 supplies camera angle sensor estimation information representing the estimated camera angle to the angle determination unit 462, and the process proceeds to step S46.
- step S44 determines whether sensor information is not input from the outside. If it is determined in step S44 that sensor information is not input from the outside, the process proceeds to step S46.
- step S ⁇ b> 46 the angle determination unit 462 calculates the camera angle according to the above equation (10) based on the camera angle image estimation information supplied from the analysis unit 401 and the camera angle sensor estimation information supplied from the analysis unit 461. Determine information.
- the angle determination unit 462 supplies camera angle information to the parameter determination unit 463.
- step S47 the parameter determination unit 463 determines the position represented by the camera angle information as a parameter based on the camera angle information supplied from the angle determination unit 462.
- the parameter determination unit 463 supplies the parameters to the pseudo image generation unit 16.
- step S48 to S53 Since the processing from step S48 to S53 is the same as the processing from step S18 to S23 in FIG.
- the image processing apparatus 400 generates a pseudo photographed image by shifting the virtual viewpoint of the input image from the center position based on the camera angle information of the input image. Therefore, in the pseudo photographed image, the composition of the camera angle is more emphasized than in the input image, and the photographer's intention is more easily recognized.
- FIG. 32 is a block diagram illustrating a configuration example of a third embodiment of an image processing device to which the present disclosure is applied.
- the image processing apparatus 500 arranges an image having a size based on the characteristics of the input image in a predetermined area of the screen (hereinafter referred to as a screen center area), and removes an area around the screen center area (hereinafter referred to as a screen peripheral area) A wide-field image is generated by interpolation.
- the analysis unit 501 of the image processing apparatus 500 extracts a predetermined region as a feature of the input image from the input image based on information regarding the input image.
- the analysis unit 501 generates an importance map based on the extracted predetermined region, and estimates the shooting angle of view.
- the analysis unit 501 supplies the importance map and the shooting angle of view to the determination unit 502.
- the determination unit 502 determines the final importance map based on the attribute signal for the importance map input from the outside and the importance map supplied from the analysis unit 501. Further, the determination unit 502 determines a final shooting angle of view based on the attribute signal for shooting angle of view and the shooting angle of view supplied from the analysis unit 501.
- the determination unit 502 determines the screen center area based on the final importance map, the shooting angle of view, and viewing environment information that is information regarding the viewing environment input from the outside.
- the viewing environment information is an actual viewing distance that is a distance between an actual viewpoint and a display that displays a wide-field image, which is set by an external sensor or user input, a display size, and the like.
- the determination unit 502 supplies screen center area information specifying the position and size of the screen center area to the center generation unit 503 and the periphery generation unit 504.
- the center generation unit 503 performs a scaling process on the input image so that the size of the input image becomes the size of the screen center region specified by the screen center region information supplied from the determination unit 502, and the image of the screen center region is converted. Generate.
- the center generation unit 503 supplies the generated image of the screen center area to the synthesis unit 505 and the peripheral generation unit 504.
- the periphery generation unit 504 determines a region around the screen center region specified by the screen center region information that overlaps with the screen center region. Determine as.
- the peripheral generation unit 504 extrapolates the screen peripheral region image using the image of the screen central region supplied from the center generation unit 503 and the image input from the outside, and supplies the resultant image to the synthesis unit 505.
- the synthesizing unit 505 synthesizes the image of the screen center area from the center generating unit 503 and the image of the screen peripheral area from the peripheral generating unit 504, and outputs the resultant synthesized image as a wide-field image.
- the shooting angle of view is determined in both the horizontal direction and the vertical direction, but in order to simplify the description, only the shooting angle of view in the horizontal direction is determined below.
- the shooting angle of view in the vertical direction is determined in the same manner as the shooting angle of view in the horizontal direction.
- FIG. 33 is a block diagram illustrating a configuration example of the analysis unit 501 in FIG.
- a face detection unit 192 includes a face detection unit 192, a person detection unit 193, a saliency detection unit 194, a depth image generation unit 521, an object detection unit 522, a perspective detection unit 523, a background measurement unit 524, and an important region estimation unit. 525 and a shooting angle of view estimation unit 526.
- the depth image generation unit 521 generates a depth image from the input image by a method using information such as the position of the vanishing point and the vanishing line, and supplies the depth image to the background measurement unit 524 and the shooting angle of view estimation unit 526.
- the depth image generation unit 521 supplies information such as the vanishing point and the position of the vanishing line used when generating the depth image to the parse detection unit 523 as erasure information.
- the object detection unit 522 extracts various objects (objects) from the input image by object recognition processing.
- the object detection unit 522 determines the likelihood of the object so that the correlation between the sizes of the extracted objects becomes higher as it is closer to the assumed correlation.
- the object detection unit 522 determines the probability of the dog or cat and the person object to a relatively large value.
- the probability of the human object such as a building, forest, mountain, etc. is determined to be a relatively large value.
- the object detection unit 522 generates, for each object, an image having a pixel value of 0 to 255 representing the probability of the object of each pixel as an object map.
- the object detection unit 522 supplies the generated object map to the important region estimation unit 525 and the shooting angle of view estimation unit 526.
- the perspective detection unit 523 generates a perspective intensity from the disappearance information supplied from the depth image generation unit 521. Specifically, the perspective detection unit 523 classifies vanishing points and vanishing lines into types of one-point perspective, two-point perspective, or three-point perspective based on disappearance information. Then, the parse detection unit 523 determines the parse intensity so that the vanishing point becomes larger as it is closer to the center of the screen for each type. Thereby, when the vanishing point is away from the center of the screen, for example, when the vanishing point exists outside the screen, the perspective intensity is lowered. The perspective detection unit 523 supplies the perspective intensity to the shooting angle of view estimation unit 526.
- the background measurement unit 524 determines, based on the depth image supplied from the depth image generation unit 521, a pixel region whose position in the depth direction of the subject is the back side as a background region. The background measurement unit 524 determines whether or not the band distribution of the background region of the input image extends to a high region. For example, the background measurement unit 524 determines whether or not the background region of the input image is an image obtained by relatively up-converting the high frequency signal. Details of this determination method are described in, for example, Japanese Patent No. 5056242.
- the background measurement unit 524 performs frequency conversion such as Fourier transform, and determines whether or not the coefficient is included up to the high range, thereby determining whether or not the band distribution of the background region of the input image extends to the high range. You may make it determine.
- the background measurement unit 524 generates a background part definition signal representing a high-frequency level where the band distribution is extended according to the determination result, and supplies the background part definition signal to the shooting angle-of-view estimation unit 526.
- the important area estimation unit 525 is based on the face map from the face detection unit 192, the person map from the person detection unit 193, the subject map from the saliency detection unit 194, and the object map from the object detection unit 522. Generate a map.
- the importance map generation method is the same as the generation method in the estimation unit 195 in FIG. 12 except that an object map is used instead of the caption telop map.
- the important area estimation unit 525 supplies the generated importance map to the determination unit 502 in FIG.
- the shooting angle of view estimation unit 526 estimates the shooting angle of view using the depth image, the face map, the person map, the object map, the perspective intensity, and the background portion definition signal.
- the shooting angle of view estimation unit 526 supplies the estimated shooting angle of view to the determination unit 502.
- FIG. 34 is a block diagram illustrating a configuration example of the shooting angle-of-view estimation unit 526 in FIG.
- a face determination unit 541 includes a face determination unit 541, a person determination unit 542, an object determination unit 543, and a shooting field angle conversion unit 544.
- the face determination unit 541 of the shooting angle-of-view estimation unit 526 extracts the pixel value of the face area of the depth image supplied from the depth image generation unit 521 based on the face map supplied from the face detection unit 192 in FIG. .
- the face determination unit 541 determines the perspective intensity of the face area by comparing the threshold value based on the pixel value of the face area of the extracted depth image with the size of the face area.
- the face determination unit 541 supplies the determined perspective intensity to the shooting angle-of-view conversion unit 544 as the face base perspective intensity.
- the person determination unit 542 extracts the pixel value of the person area of the depth image supplied from the depth image generation unit 521 based on the person map supplied from the person detection unit 193.
- the person determination unit 542 determines the perspective intensity of the person area by comparing the threshold value based on the pixel value of the person area of the extracted depth image with the size of the person area.
- the person determination unit 542 supplies the determined perspective intensity as the person base perspective intensity to the shooting angle-of-view conversion unit 544.
- the object determination unit 543 extracts the pixel value of the object area of the depth image supplied from the depth image generation unit 521 based on the object map supplied from the saliency detection unit 194 for each object.
- the object determination unit 543 determines the perspective intensity of the object region by comparing the threshold value based on the pixel value of the object region of the extracted depth image with the size of the object region for each object.
- the object determination unit 543 supplies the determined perspective intensity for each object to the shooting angle-of-view conversion unit 544 as the object-based perspective intensity.
- the shooting angle-of-view conversion unit 544 includes a face base perspective intensity from the face determination unit 541, a person base perspective intensity from the person determination unit 542, an object base perspective intensity from the object determination unit 543, and a perspective intensity from the perspective detection unit 523. Based on the above, the total perspective intensity is calculated by the following equation (11).
- All_Pers is the integrated perspective intensity
- F_Pers is the face-based perspective intensity
- H_Pers is the person-based perspective intensity
- O_Pers is the object base parse strength of each object
- V_Pers is the parse strength.
- Wf, Wh, Wo, Wv are weighting factors. Wf, Wh, Wo are determined using the number of corresponding regions (face region, person region, object region) and the probability of the region based on the pixel values of the map (face map, person map, object map). .
- Wv is determined using the vanishing point or the number of vanishing lines represented by the vanishing information.
- the integrated perspective intensity is a weighted average of the face base perspective intensity, the person base perspective intensity, the object base perspective intensity, and the perspective intensity.
- the shooting angle-of-view conversion unit 544 estimates the shooting angle of view based on the integrated perspective intensity, and sets the estimated shooting angle of view as the perspective intensity-based shooting angle of view. Also, the shooting angle-of-view conversion unit 544 estimates the shooting angle of view based on the background part definition signal supplied from the background measurement unit 524 in FIG. 33, and uses the estimated shooting angle of view as the depth-of-field-based shot image. A corner.
- the shooting angle-of-view conversion unit 544 finalizes the shooting angle of view based on the characteristics of the input image according to the following equation (12). Determine the estimate.
- Est_angle is a final estimated value of the shooting angle of view based on the characteristics of the input image
- P_angle is a perspective intensity-based shooting angle of view
- B_angle is a depth-of-field-based shooting angle of view.
- Wp and Wb are weighting factors.
- the final estimated value of the shooting angle of view based on the characteristics of the input image is a weighted average of the perspective intensity-based shooting angle of view and the depth-of-field-based shooting angle of view.
- the shooting angle conversion unit 544 supplies the determined shooting angle of view (final estimated value of the shooting angle of view based on the characteristics of the input image) to the determination unit 502 in FIG.
- FIG. 35 is a diagram for explaining the face base perspective strength.
- the horizontal axis represents the position of the subject in the depth direction represented by the pixel value of the face area of the depth image
- the vertical axis represents the size of the face area
- the face determination unit 541 determines that the position of the face area in the depth direction is based on a threshold value that decreases with a predetermined inclination as the position in the depth direction is on the back side.
- the face base perspective intensity is determined so as to be larger as it is smaller than the threshold and smaller as it is larger.
- the face determination unit 541 determines the face base perspective intensity so that the position in the depth direction of the face area is larger as it is larger than the threshold and smaller as it is smaller. To do.
- the straight line indicating the relationship between the position and size of the face region in the depth direction has a relatively large inclination as shown by the solid line in FIG.
- the straight line indicating the relationship between the position and the size of the face region in the depth direction has a relatively small inclination as shown by a dotted line in FIG.
- the face base perspective intensity increases as the face area of the input image captured at wide angle.
- FIG. 36 is a diagram for explaining the person base perspective intensity.
- the horizontal axis represents the position of the subject in the depth direction represented by the pixel value of the person area of the depth image
- the vertical axis represents the size of the person area
- the person determining unit 542 determines that the position of the person area in the depth direction is based on a threshold value that decreases with a predetermined inclination as the position in the depth direction is on the back side.
- the person base perspective intensity is determined so as to be larger as it is smaller than the threshold and smaller as it is larger.
- the person determination unit 542 determines the person base perspective intensity so that the position of the person area in the depth direction increases as the threshold value increases and decreases as the threshold value decreases. To do.
- the straight line indicating the relationship between the position and the size of the person region in the depth direction has a relatively large inclination as shown by the solid line in FIG.
- the straight line of the relationship between the position and the size of the person region in the depth direction has a relatively small inclination as shown by a dotted line in FIG. 36, for example.
- the person base perspective intensity increases as the person area of the input image captured at wide angle.
- FIG. 37 is a diagram for explaining the object-based perspective intensity.
- the horizontal axis represents the position in the depth direction of the subject represented by the pixel value of the object area of the depth image
- the vertical axis represents the size of the object area
- the object determination unit 543 determines, for each object, the depth direction of the object area when the position of the object area in the depth direction is on the back side based on a threshold value that decreases with a predetermined inclination as the position in the depth direction is on the back side.
- the object-based perspective intensity is determined so that the position of is larger as it is smaller than the threshold and smaller as it is larger.
- the object determination unit 543 determines the object base perspective strength so that the position in the depth direction of the object area is larger as it is larger than the threshold and smaller as it is smaller. To do.
- the straight line indicating the relationship between the position and the size of the object region in the depth direction has a relatively large inclination as shown by a solid line in FIG. 37, for example.
- the straight line indicating the relationship between the position and the size of the object region in the depth direction has a relatively small inclination as shown by a dotted line in FIG. 37, for example.
- the object base perspective intensity increases as the object area of the input image captured at wide angle.
- FIG. 38 is a diagram for explaining a perspective intensity-based shooting field angle.
- the horizontal axis represents the integrated perspective intensity
- the vertical axis represents the perspective intensity-based shooting angle of view determined based on the integrated perspective intensity
- the shooting angle conversion unit 544 estimates that the larger the integrated perspective intensity is, the wider the angle of shooting, that is, the larger the shooting angle of view. Therefore, as shown in FIG. 38, the perspective intensity-based shooting field angle is determined so as to increase as the integrated perspective intensity increases.
- FIG. 39 is a diagram for explaining the depth of field-based shooting angle of view.
- the horizontal axis represents the background portion definition signal
- the vertical axis represents the depth-of-field-based shooting field angle determined based on the background portion definition signal.
- the shooting field angle conversion unit 544 estimates that the shooting field angle is larger as the background definition signal is larger, that is, the background is not blurred. Therefore, as shown in FIG. 39, the depth-of-field-based shooting field angle is set so as to increase as the background definition signal increases.
- FIG. 40 is a block diagram illustrating a configuration example of the determination unit 502 of FIG.
- 40 includes an analysis unit 211, an important region determination unit 212, an analysis unit 561, a shooting angle of view determination unit 562, and an area determination unit 563.
- the analysis unit 561 of the determination unit 502 determines the shooting angle of view based on the focal length at the time of shooting of the input image and the size of the image sensor, which are input from the outside as attribute signals for the shooting angle of view.
- the analysis unit 561 may set the shooting angle of view by obtaining JPEG (JointoPhotographic Experts Group) Exif information from the outside.
- JPEG Joint Photographic Experts Group
- the analysis unit 561 supplies the shooting field angle to the shooting field angle determination unit 562.
- the shooting angle-of-view determination unit 562 finally calculates the following equation (13) based on the shooting angle of view supplied from the shooting angle-of-view conversion unit 544 in FIG. 34 and the shooting angle of view supplied from the analysis unit 561. Determine the correct shooting angle of view.
- Final_angle is the final shooting angle of view
- Est_angle is the shooting angle of view supplied from the shooting angle of view conversion unit 544
- Meta_angle is the shooting angle of view supplied from the analysis unit 561. is there.
- W_est represents a weighting coefficient and is a value of 0 or more and 1 or less.
- the shooting angle of view determination unit 562 supplies the final shooting angle of view to the region determination unit 563.
- the area determination unit 563 obtains the viewing angle based on the actual viewing distance and the display size included in the viewing environment information input from the outside.
- the area determination unit 563 is based on the viewing angle of view, the final shooting angle of view supplied from the shooting angle of view determination unit 562, and the final important area information generated by the important area determination unit 212.
- An area of a predetermined size at a predetermined position in the screen where is identical to the screen is determined as the screen center area.
- the area determination unit 563 determines a screen relative ratio that is a ratio of the screen center area to the screen so that the shooting angle of view of the screen center area is the same as the viewing angle of view. Further, the position of the screen center area is determined so that the important area represented by the final important area information is within the central visual field of the viewer.
- the region determination unit 563 supplies information representing the screen relative ratio and the position of the screen center region to the center generation unit 503 and the periphery generation unit 504 in FIG. 32 as screen center region information.
- a pseudo photographed image which is a predicted value of the photographed image when photographing at a photographing field angle in which the photographing field angle in the screen central region is the same as the viewing field angle, is generated as an image in the screen central region.
- FIG. 41 is a diagram for explaining a method for determining a shooting angle of view in the analysis unit 561 in FIG.
- the relationship between the shooting angle of view ⁇ of the input image, the size x of the image sensor 582, and the focal length f at the time of shooting is It is represented by the following formula (14).
- the analysis unit 561 calculates the shooting angle of view ⁇ by the following equation (15) based on the focal length f and the size x of the image sensor when the input image is input as the attribute signal for the shooting angle of view.
- equation (15) based on the focal length f and the size x of the image sensor when the input image is input as the attribute signal for the shooting angle of view.
- FIG. 42 is a block diagram illustrating a configuration example of the center generation unit 503 in FIG.
- the setting unit 601 of the center generation unit 503 sets the scale rate by the following equation (16) based on the screen relative ratio included in the screen center region information supplied from the region determination unit 563 of FIG. 602 is supplied.
- Scale is a scale rate
- W_disp and H_disp are the size in the horizontal direction (horizontal direction) and the size in the vertical direction (vertical direction), respectively.
- CentralPartRatio is a screen relative ratio
- W_in and H_in are the horizontal size and vertical size of the input image, respectively.
- the scaling unit 602 scales the input image based on the scale rate supplied from the setting unit 601 so that the size of the input image becomes the size of the screen center area.
- the scaling process is an enlargement process.
- this enlargement process for example, a bilinear interpolation technique, a bicubic interpolation technique, a Lanchos interpolation technique, a so-called super-resolution technique, or the like can be used.
- the scaling unit 602 supplies the input image after the scaling process to the peripheral generation unit 504 and the composition unit 505 in FIG. 32 as an image of the screen center region.
- FIG. 43 is a block diagram illustrating a configuration example of the peripheral generation unit 504 in FIG.
- 32 includes a setting unit 621, an extrapolation unit 622, and an adjustment unit 623.
- the setting unit 621 of the peripheral generation unit 504 determines the screen peripheral region based on the screen central region information supplied from the region determination unit 563 of FIG. Then, the setting unit 621 supplies screen peripheral area information specifying the screen peripheral area to the extrapolation unit 622 and the adjustment unit 623.
- the extrapolation unit 622 supplies an image of the screen peripheral region specified by the screen peripheral region information supplied from the setting unit 621, an image of the screen central region supplied from the scaling unit 602 in FIG.
- the image is generated by performing extrapolation using the image.
- a method of extrapolation in extrapolation unit 622 the same method as that in extrapolation unit 31 in FIG. 2 can be used.
- the extrapolation unit 622 refers to an image having a high similarity to the input image with respect to the image, the shooting position, the shooting date, and the like among the images supplied from the outside. To perform extrapolation.
- the extrapolation unit 622 performs extrapolation using not only the image of the screen center region but also the image supplied from the outside. Therefore, the extrapolation unit 622 can perform extrapolation prediction by using an image having a similar image, shooting position, shooting date and time, etc., for an input image of a scene that is difficult to extrapolate using only the input image. it can.
- the image in the screen peripheral area is a typical texture such as forest or lawn, the image quality of the image in the screen peripheral area can be improved by performing extrapolation using the image of the CG database. .
- the extrapolation unit 622 supplies the image of the screen peripheral area generated by the extrapolation to the adjustment unit 623. Further, the extrapolation unit 622 generates an extrapolation reliability indicating the certainty of the extrapolation interpolation result. For example, the extrapolation unit 622 represents the accuracy of matching in extrapolation with a value of 0 or more and 1 or less, and represents the extrapolation reliability. The extrapolation unit 622 supplies the extrapolation reliability to the adjustment unit 623.
- the adjustment unit 623 adjusts the image of the screen peripheral region based on the extrapolation reliability supplied from the extrapolation unit 622, and supplies the adjusted image of the screen peripheral region to the combining unit 505 in FIG.
- the extrapolation unit 622 performs extrapolation using an image with a high degree of similarity between the image of the screen center region and the image supplied from the outside, but the extrapolation unit 622 You may make it perform extrapolation using the picked-up image of the wall behind the display which displays the wide-field image supplied from the outside.
- the display 641 when extrapolation is performed using an image with a high degree of similarity between an image in the screen center area and an image supplied from the outside, the display 641 includes a screen center area 651A and A wide-field image 651 in which the pattern of the screen peripheral area 651B is connected is displayed. Accordingly, the viewer can view the wide-field image 651 having the screen size of the display 641.
- the display 641 when extrapolation is performed using a photographed image of the wall 642 behind the display 641 supplied from the outside, the display 641 includes a screen center region 651A and A wide-field image 652 including a screen peripheral area 652B in which an image of the wall 642 behind the display 641 is arranged is displayed. Accordingly, the image in the screen peripheral area 652B and the wall 642 are integrated, and the viewer feels as if viewing the image in the screen center area 651A from a distance through a small window. Therefore, the real feeling and realistic feeling of the wide-field image are improved.
- extrapolation unit 622 may perform extrapolation using only the image of the screen center region.
- FIG. 45 is a flowchart for explaining image processing of the image processing apparatus 500 of FIG.
- step S74 the analysis unit 501 estimates the shooting angle of view.
- step S75 the determination unit 502 determines whether or not an attribute signal for importance map is input from the outside. If it is determined in step S75 that an importance map attribute signal has been input, the process proceeds to step S76.
- step S76 the analysis unit 211 (FIG. 40) of the determination unit 502 generates an importance map based on EPG program genre information or the like input from the outside as an attribute signal for the importance map.
- the analysis unit 211 supplies the generated importance map to the important region determination unit 212, and the process proceeds to step S77.
- step S75 when it is determined in step S75 that the attribute signal for importance map is not input, the process proceeds to step S77.
- step S77 the important area determination unit 212 determines a final importance map based on the importance map from the analysis unit 501 and the importance map generated by the analysis unit 211.
- the important area determination unit 212 generates important area information based on the importance map and supplies the important area information to the area determination unit 563.
- step S78 the determination unit 502 determines whether an attribute signal for shooting angle of view is input from the outside. If it is determined in step S78 that an attribute signal for shooting angle of view has been input, the process proceeds to step S79.
- step S79 the analysis unit 561 (FIG. 40) of the determination unit 502 determines the shooting field angle from the outside based on the attribute signal for the shooting field angle.
- the analysis unit 561 supplies the shooting field angle to the shooting field angle determination unit 562, and the process proceeds to step S80.
- step S78 determines whether an attribute signal for shooting angle of view has been input. If it is determined in step S78 that an attribute signal for shooting angle of view has not been input, the process proceeds to step S80.
- step S80 the shooting field angle determination unit 562 determines a final shooting field angle based on the shooting field angle supplied from the analysis unit 501 and the shooting field angle supplied from the analysis unit 561.
- the shooting angle of view determination unit 562 supplies the final shooting angle of view to the region determination unit 563.
- step S81 the area determination unit 563 determines the screen center area based on the viewing angle of view, the final important area information, and the final shooting angle of view. Note that the viewing angle of view is obtained based on viewing environment information input from the outside.
- the region determination unit 563 supplies information representing the screen relative ratio and the position of the screen center region to the center generation unit 503 and the periphery generation unit 504 as screen center region information.
- step S82 the center generation unit 503 performs a screen center region generation process for scaling the input image based on the screen center region information from the determination unit 502 and generating an image of the screen center region.
- the center generation unit 503 supplies the generated image of the screen center area to the periphery generation unit 504 and the synthesis unit 505.
- step S83 the peripheral generation unit 504 performs screen peripheral region generation processing based on the screen central region information from the determination unit 502.
- the screen peripheral area generation processing generates an image of the screen peripheral area by performing extrapolation using an image of the screen central area or an image supplied from the outside based on the screen central area information, and extrapolation reliability It is a process to adjust based on.
- the peripheral generation unit 504 supplies the adjusted image of the screen peripheral area to the synthesis unit 505.
- step S84 the synthesis unit 505 performs a synthesis process for synthesizing the image of the screen center area from the center generation unit 503 and the image of the screen periphery area from the periphery generation unit 504.
- step S85 the synthesis unit 505 outputs the synthesized image obtained as a result of the synthesis process as a wide-field image.
- step S86 the image processing apparatus 500 determines whether an image is newly input. If it is determined in step S86 that an image has been newly input, the process returns to step S72, and the processes in steps S72 to S86 are repeated until no new image is input.
- step S86 if it is determined in step S86 that no new image has been input, the process ends.
- the image processing apparatus 500 estimates the shooting angle of view of the input image based on the input image and the depth image. Then, the image processing apparatus 500 predicts a captured image when an image is captured from the input image at the same view angle as the view angle based on the estimated view angle of view and the pseudo view image. The value is generated as a pseudo photographed image. Therefore, it is possible to artificially change the method for capturing the input image using the depth image.
- FIG. 46 is a diagram for explaining the effects of the image processing apparatus 10, the image processing apparatus 400, and the image processing apparatus 500.
- the image processing apparatus 10 generates a pseudo photographed image while moving the position of the virtual viewpoint of the input image in the horizontal direction or the vertical direction so that the important region falls within the central visual field.
- This pseudo photographed image is an image having motion parallax when the subject of the composite image is photographed on the track. Therefore, it can be said that this pseudo photographed image is a pseudo track photographed image obtained by photographing the subject of the composite image in a pseudo manner.
- ⁇ ⁇ The sense of presence, visibility, and depth are improved in the simulated track shot image compared to the input image.
- the pseudo track shot image is an image obtained by shooting the subject of the composite image in a pseudo horizontal track, motion parallax occurs in the horizontal direction as when viewing the scenery outside the train, Improves the sense of depth.
- the image processing apparatus 10 moves the virtual view distance of the input image forward while moving the virtual view distance so that the important area is within the central visual field and the occupation ratio of the important area in the central visual field is equal to or greater than the threshold.
- This pseudo photographed image is an image having motion parallax when the subject of the composite image is dolly-in photographed. Therefore, it can be said that this pseudo photographed image is a pseudo dolly-in photographed image obtained by pseudo dolly-in photographing of the subject of the composite image.
- the sense of reality, visibility, and depth are improved compared to the input image. For example, motion parallax occurs as if a small and difficult-to-see important area is approached and viewed, improving the sense of depth.
- the image processing apparatus 10 generates a pseudo photographed image while moving the virtual viewing distance of the input image backward so that the important region falls within the central visual field.
- This pseudo photographed image is an image having motion parallax when the subject of the composite image is dolly-out photographed. Therefore, it can be said that this pseudo photographed image is a pseudo dolly out photographed image obtained by pseudo dolly out photographing of the subject of the composite image.
- the sense of reality, visibility, and depth are improved compared to the input image. For example, since the motion parallax occurs like when viewing an important area that is too close and difficult to see back, the sense of depth is improved.
- the image processing apparatus 500 reduces the input image so that the viewing angle of view is equal to the shooting angle of view, so that the shooting angle of view and the viewing angle of view are reduced. Generate a wide-field image that is identical. Thereby, for example, the viewer who views the wide-field image can view the scene viewed by the photographer from the photographing position.
- a sense of reality and a real feeling are improved as compared with the input image.
- the viewer looks at the scene image viewed from the shooting position of the photographer far away from the subject as a wide-field image, thereby improving the realism and realism. Can do.
- the image processing apparatus 500 enlarges the input image so that the viewing angle of view is the same as the shooting angle of view, thereby A wide-field image having the same corner is generated.
- the viewer who views the wide-field image can view the scene viewed by the photographer from the photographing position.
- a sense of reality, real feeling, and visibility are improved as compared with the input image.
- the viewer looks at the scene image viewed from the shooting position of the photographer who is close to the subject as a wide-field image, and the presence, realism, and Visibility can be improved.
- the image processing apparatus 400 changes the position of the virtual viewpoint so as to emphasize the camera angle based on the camera angle at the time of shooting the input image, and generates a pseudo shot image.
- This pseudo photographed image is a predicted value of the photographed image when the subject of the composite image is photographed with a more enhanced camera angle than the input image. Therefore, it can be said that this pseudo photographed image is a pseudo camera angle photographed image obtained by photographing the subject of the composite image with the camera angle enhanced in a pseudo manner.
- the impression of the composition intended by the photographer can be strengthened, so the impression of the scene is improved compared to the input image.
- the image processing apparatus 10 also performs pseudo-photographing by smoothing the pixel values of the near side and the far side of the important area while increasing the scaling rate of the input image so that the important area falls within the central visual field. Generate an image.
- This pseudo photographed image is an image having a narrow angle of view and a shallow depth of field when the subject of the composite image is zoomed in. Therefore, it can be said that this pseudo photographed image is a pseudo zoom-in photographed image obtained by pseudo zoom-in photographing of the subject of the composite image.
- the image processing apparatus 10 generates a pseudo photographed image by performing a deblurring process on the blurred area while reducing the scaling rate of the input image so that the important area falls within the central visual field.
- This pseudo photographed image is an image having a wide-angle photographing field angle and a deep depth of field when the subject of the composite image is zoomed out. Therefore, it can be said that this pseudo photographed image is a pseudo zoom-in photographed image obtained by pseudo zoom-in photographing of the subject of the composite image.
- the visibility of the pseudo zoom-out shot image is improved compared to the input image. That is, the depth of field is deepened and the visibility is improved as when the subject of the composite image is taken at a wide angle.
- the image processing apparatus 10 generates a pseudo photographed image while changing the angle of the line-of-sight direction of the input image so that the important region is within the central visual field.
- This pseudo photographed image is a predicted value of the photographed image when the subject of the composite image is panned (tilted). Therefore, it can be said that this pseudo photographed image is a pseudo panning (tilt) photographed image obtained by pseudo panning (tilt) photographing of the subject of the composite image.
- the visibility of the pseudo panning (tilt) image is improved compared to the input image.
- the visibility is improved as when the periphery of the subject of the input image is photographed.
- the depth image may not be used when generating a wide-field image having the same shooting angle of view and viewing angle of view and when generating a pseudo panning (tilt) shot image. Further, the disappearance information may not be used when the pseudo camera angle photographed image is generated.
- ⁇ Fourth embodiment> (Description of computer to which the present disclosure is applied)
- the series of processes described above can be executed by hardware or can be executed by software.
- a program constituting the software is installed in the computer.
- the computer includes, for example, a general-purpose personal computer capable of executing various functions by installing various programs by installing a computer incorporated in dedicated hardware.
- FIG. 47 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above according to a program.
- a CPU Central Processing Unit
- ROM Read Only Memory
- RAM Random Access Memory
- An input / output interface 905 is further connected to the bus 904.
- An input unit 906, an output unit 907, a storage unit 908, a communication unit 909, and a drive 910 are connected to the input / output interface 905.
- the input unit 906 includes a keyboard, a mouse, a microphone, and the like.
- the output unit 907 includes a display, a speaker, and the like.
- the storage unit 908 includes a hard disk, a nonvolatile memory, and the like.
- the communication unit 909 includes a network interface or the like.
- the drive 910 drives a removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
- the CPU 901 loads the program stored in the storage unit 908 to the RAM 903 via the input / output interface 905 and the bus 904 and executes the program. A series of processing is performed.
- the program executed by the computer 900 can be provided by being recorded on a removable medium 911 as a package medium, for example.
- the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.
- the program can be installed in the storage unit 908 via the input / output interface 905 by attaching the removable medium 911 to the drive 910.
- the program can be received by the communication unit 909 via a wired or wireless transmission medium and installed in the storage unit 908.
- the program can be installed in the ROM 902 or the storage unit 908 in advance.
- the program executed by the computer 900 may be a program that is processed in time series in the order described in this specification, or a necessary timing such as when a call is made in parallel. It may be a program in which processing is performed.
- the present disclosure can take a cloud computing configuration in which one function is shared by a plurality of devices via a network and is processed jointly.
- each step described in the above flowchart can be executed by one device or can be shared by a plurality of devices.
- the present disclosure can have the following configurations.
- the image processing apparatus Based on the value of the parameter determined based on the image characteristics and the depth image representing the position of the subject in the depth direction of the image, the predicted value of the photographed image obtained by photographing the subject by a predetermined photographing method from the image
- An image processing apparatus comprising a pseudo image generation unit that generates a pseudo photographed image.
- the image processing apparatus according to (1) wherein the value is determined so that an important region of the image falls within a central visual field of a viewer who views the pseudo photographed image.
- the pseudo image generation unit changes the value of the parameter stepwise from a predetermined value to the determined value, and generates the pseudo photographed image based on the changed value and the depth image.
- the image processing apparatus according to (2) configured.
- the parameter is a position of a virtual viewpoint of the pseudo photographed image, The image processing apparatus according to (3), wherein the predetermined shooting method is configured to be track shooting. (5) The parameter is a virtual visual distance of the pseudo photographed image, The image processing apparatus according to (3) or (4), wherein the predetermined imaging method is configured to be dolly-in imaging or dolly-out imaging. (6) The parameter is a scaling rate of the image; The image processing apparatus according to any one of (3) to (5), wherein the predetermined imaging method is configured to perform zoom-in imaging or zoom-out imaging. (7) The image processing apparatus according to (6), further including: an adjustment unit that adjusts a depth of field of the pseudo photographed image generated by the pseudo image generation unit based on the predetermined photographing method.
- the adjustment unit smoothes the areas of the subject on the near side and the rear side of the position of the subject area in the depth direction of the subject in the pseudo photographed image.
- the image processing device according to (7), wherein the image processing device is configured to adjust the depth of field.
- the adjusting unit is configured to adjust the depth of field by performing a deblurring process on a blurred region of the pseudo photographed image when the predetermined photographing method is zoom-out photographing.
- the image processing apparatus according to 7) or (8).
- the parameter is an angle in a line-of-sight direction of the pseudo photographed image,
- the image processing device according to any one of (3) to (9), wherein the predetermined photographing method is configured to be panning photographing or tilt photographing.
- the parameter is a position of a virtual viewpoint of the pseudo photographed image,
- the image processing apparatus according to (1), wherein the predetermined photographing method is photographing at a position above or below the photographing position of the image.
- the pseudo image generation unit based on the synthesized image obtained by synthesizing the extrapolated peripheral image and the image based on the value, the extrapolated peripheral depth image, and the synthesized depth image obtained by synthesizing the depth image.
- the image processing device configured to generate the pseudo photographed image.
- a peripheral generation unit that extrapolates the peripheral image using the image, and extrapolates the peripheral depth image using the depth image; The peripheral image extrapolated by the peripheral generation unit and the image are combined to generate the combined image, and the peripheral depth image extrapolated by the peripheral generation unit and the depth image are combined to generate the composite image.
- the image processing apparatus according to (12), further including: a combining unit that generates a combined depth image.
- the image processing apparatus further including: a cutout unit that deletes at least a part of the pseudo photographed image generated by the pseudo image generation unit.
- the image processing device Based on the value of the parameter determined based on the image characteristics and the depth image representing the position of the subject in the depth direction of the image, the predicted value of the photographed image obtained by photographing the subject by a predetermined photographing method from the image
- An image processing method including a pseudo image generation step of generating as a pseudo photographed image.
- a shooting angle of view estimating unit that estimates a shooting angle of view of the image based on the image and a depth image representing a position of the subject in the depth direction of the image; Based on the shooting angle of view estimated by the shooting angle of view estimation unit and the viewing angle of view of the pseudo-photographed image, a shot image when shooting from the image at the same angle of view as the viewing angle of view
- An image processing apparatus comprising: a generation unit that generates a predicted value of as a pseudo photographed image.
- the generation unit is configured to generate the pseudo captured image by reducing the image when the viewing angle of view is larger than the shooting angle of view.
- the image processing device A shooting angle of view estimation step for estimating a shooting angle of view of the image based on the image and a depth image representing a position of the subject in the depth direction of the image; Based on the shooting angle of view estimated by the processing of the shooting angle of view estimation and the viewing angle of view of the pseudo-photographed image, when shooting from the image at the same angle of view as the viewing angle of view
- An image processing method comprising: generating a predicted value of a captured image as a pseudo captured image.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Processing (AREA)
- Studio Devices (AREA)
- Processing Or Creating Images (AREA)
Abstract
Description
1.第1実施の形態:画像処理装置(図1乃至図25)
2.第2実施の形態:画像処理装置(図26乃至図31)
3.第3実施の形態:画像処理装置(図32乃至図45)
4.第1乃至第3実施の形態の効果(図46)
5.第4実施の形態:コンピュータ(図47)
(画像処理装置の第1実施の形態の構成例)
図1は、本開示を適用した画像処理装置の第1実施の形態の構成例を示すブロック図である。
図2は、図1の周辺生成部12の構成例を示すブロック図である。
図3は、外挿補間のホールド方式、ミラー方式、および平行シフト方式を説明する図である。
図4は、外挿補間情報を説明する図である。
図5は、図2の調整部34の構成例を示すブロック図である。
図6は、コントラストゲインの例を示す図である。
図7は、彩度ゲインの例を示す図である。
図8は、図5の精細度調整部73における平滑化フィルタのタップ数の例を示す図である。
図9は、図5の明るさ調整部74による明るさ調整におけるオフセット値DarkOffsetPredとオフセット値DarkOffsetDistの例を示す図である。
図11は、図1の合成部13による入力画像の合成を説明する図である。
図12は、図1の解析部14の構成例を示すブロック図である。
図13は、図12の推定部195による重要度マップの生成について説明する図である。
図14は、図1の決定部15の構成例を示すブロック図である。
図15は、バイナリマップの例を示す図である。
図16は、図15のバイナリマップから検出される重要領域の例を示す図である。
図17は、被写体の3次元位置と画像上の2次元位置の関係を示す図である。
図18は、パラメータの決定方法の第1の例を説明する図である。
図19は、パラメータの決定方法の第2の例を説明する図である。
図20は、パラメータの決定方法の第3の例を説明する図である。
図21は、図1の擬似画像生成部16の構成例を示すブロック図である。
図22は、図1の画像処理装置10の処理を説明するフローチャートである。
図23および図24は、パラメータの決定方法の第4の例を説明する図である。
図25は、パラメータの決定方法の第5の例を説明する図である。
(画像処理装置の第2実施の形態の構成例)
図26は、本開示を適用した画像処理装置の第2実施の形態の構成例を示すブロック図である。
図27は、図26の解析部401の構成例を示すブロック図である。
図28は、図27のアングル推定部425の構成例を示すブロック図である。
図29は、図28の消失点解析部444による消失情報に基づく仮想視点のディスプレイにおける垂直方向の位置の決定を説明する図である。
図30は、図26の決定部402の構成例を示すブロック図である。
図31は、図26の画像処理装置400の処理を説明するフローチャートである。
(画像処理装置の第3実施の形態の構成例)
図32は、本開示を適用した画像処理装置の第3実施の形態の構成例を示すブロック図である。
図33は、図32の解析部501の構成例を示すブロック図である。
図34は、図33の撮影画角推定部526の構成例を示すブロック図である。
図35は、顔ベースパース強度を説明する図である。
図36は、人物ベースパース強度を説明する図である。
図37は、オブジェクトベースパース強度を説明する図である。
図38は、パース強度ベース撮影画角を説明する図である。
図39は、被写界深度ベース撮影画角を説明する図である。
図40は、図32の決定部502の構成例を示すブロック図である。
図41は、図40の分析部561における撮影画角の決定方法を説明する図である。
図42は、図32の中心生成部503の構成例を示すブロック図である。
図43は、図32の周辺生成部504の構成例を示すブロック図である。
図45は、図32の画像処理装置500の画像処理を説明するフローチャートである。
図46は、画像処理装置10、画像処理装置400、および画像処理装置500による効果を説明する図である。
(本開示を適用したコンピュータの説明)
上述した一連の処理は、ハードウエアにより実行することもできるし、ソフトウエアにより実行することもできる。一連の処理をソフトウエアにより実行する場合には、そのソフトウエアを構成するプログラムが、コンピュータにインストールされる。ここで、コンピュータには、専用のハードウエアに組み込まれているコンピュータや、各種のプログラムをインストールすることで、各種の機能を実行することが可能な、例えば汎用のパーソナルコンピュータなどが含まれる。
画像の特徴に基づいて決定されたパラメータの値と前記画像の被写体の奥行き方向の位置を表す奥行き画像とに基づいて、前記画像から、所定の撮影方法で前記被写体を撮影した撮影画像の予測値を擬似撮影画像として生成する擬似画像生成部
を備える画像処理装置。
(2)
前記値は、前記画像の重要領域が、前記擬似撮影画像を観視する観視者の中心視野内に収まるように決定される
ように構成された
前記(1)に記載の画像処理装置。
(3)
前記擬似画像生成部は、前記パラメータの値を、所定値から、決定された前記値に段階的に変更し、変更後の前記値と前記奥行き画像に基づいて前記擬似撮影画像を生成する
ように構成された
前記(2)に記載の画像処理装置。
(4)
前記パラメータは、前記擬似撮影画像の仮想視点の位置であり、
前記所定の撮影方法は、トラック撮影である
ように構成された
前記(3)に記載の画像処理装置。
(5)
前記パラメータは、前記擬似撮影画像の仮想視距離であり、
前記所定の撮影方法は、ドリーイン撮影またはドリーアウト撮影である
ように構成された
前記(3)または(4)に記載の画像処理装置。
(6)
前記パラメータは、前記画像のスケーリング率であり、
前記所定の撮影方法は、ズームイン撮影またはズームアウト撮影である
ように構成された
前記(3)乃至(5)のいずれかに記載の画像処理装置。
(7)
前記所定の撮影方法に基づいて、前記擬似画像生成部により生成された前記擬似撮影画像の被写界深度を調整する調整部
をさらに備える
前記(6)に記載の画像処理装置。
(8)
前記調整部は、前記所定の撮影方法がズームイン撮影である場合、前記擬似撮影画像の前記重要領域の前記被写体の奥行き方向の位置よりも手前側と奥側の前記被写体の領域を平滑化することにより、前記被写界深度を調整する
ように構成された
前記(7)に記載の画像処理装置。
(9)
前記調整部は、前記所定の撮影方法がズームアウト撮影である場合、前記擬似撮影画像のボケ領域に対してデブラー処理を行うことにより、前記被写界深度を調整する
ように構成された
前記(7)または(8)に記載の画像処理装置。
(10)
前記パラメータは、前記擬似撮影画像の視線方向の角度であり、
前記所定の撮影方法は、パニング撮影またはチルト撮影である
ように構成された
前記(3)乃至(9)に記載の画像処理装置。
(11)
前記パラメータは、前記擬似撮影画像の仮想視点の位置であり、
前記所定の撮影方法は、前記画像の撮影位置より上または下の位置での撮影である
ように構成された
前記(1)に記載の画像処理装置。
(12)
前記擬似画像生成部は、前記値と、外挿補間された周辺奥行き画像と前記奥行き画像を合成した合成奥行き画像とに基づいて、外挿補間された周辺画像と前記画像を合成した合成画像から前記擬似撮影画像を生成する
ように構成された
前記(1)乃至(11)のいずれかに記載の画像処理装置。
(13)
前記画像を用いて前記周辺画像を外挿補間し、前記奥行き画像を用いて前記周辺奥行き画像を外挿補間する周辺生成部と、
前記周辺生成部により外挿補間された前記周辺画像と前記画像を合成して前記合成画像を生成し、前記周辺生成部により外挿補間された前記周辺奥行き画像と前記奥行き画像を合成して前記合成奥行き画像を生成する合成部と
をさらに備える
前記(12)に記載の画像処理装置。
(14)
前記擬似画像生成部により生成された前記擬似撮影画像の少なくとも一部を削除する切り出し部
をさらに備える
前記(13)に記載の画像処理装置。
(15)
画像処理装置が、
画像の特徴に基づいて決定されたパラメータの値と前記画像の被写体の奥行き方向の位置を表す奥行き画像とに基づいて、前記画像から、所定の撮影方法で前記被写体を撮影した撮影画像の予測値を擬似撮影画像として生成する擬似画像生成ステップ
を含む画像処理方法。
(16)
画像と前記画像の被写体の奥行き方向の位置を表す奥行き画像とに基づいて、前記画像の撮影画角を推定する撮影画角推定部と、
前記撮影画角推定部により推定された前記撮影画角と、擬似撮影画像の観視画角とに基づいて、前記画像から、前記観視画角と同一の画角で撮影した場合の撮影画像の予測値を擬似撮影画像として生成する生成部と
を備える画像処理装置。
(17)
前記生成部は、前記観視画角が前記撮影画角より大きい場合、前記画像を縮小することにより前記擬似撮影画像を生成する
ように構成された
前記(16)に記載の画像処理装置。
(18)
前記生成部は、前記観視画角が前記撮影画角より小さい場合、前記画像を拡大することにより前記擬似撮影画像を生成する
ように構成された
前記(16)または(17)に記載の画像処理装置。
(19)
前記生成部により生成された前記擬似撮影画像または外部から入力される画像を用いて、前記擬似撮影画像の周辺領域の画像を外挿補間する周辺生成部と、
前記周辺生成部により外挿補間された前記周辺領域の画像と前記擬似撮影画像を合成する合成部と
をさらに備える
前記(16)乃至(18)のいずれかに記載の画像処理装置。
(20)
画像処理装置が、
画像と前記画像の被写体の奥行き方向の位置を表す奥行き画像とに基づいて、前記画像の撮影画角を推定する撮影画角推定ステップと、
前記撮影画角推定ステップの処理により推定された前記撮影画角と、擬似撮影画像の観視画角とに基づいて、前記画像から、前記観視画角と同一の画角で撮影した場合の撮影画像の予測値を擬似撮影画像として生成する生成ステップと
を含む画像処理方法。
Claims (20)
- 画像の特徴に基づいて決定されたパラメータの値と前記画像の被写体の奥行き方向の位置を表す奥行き画像とに基づいて、前記画像から、所定の撮影方法で前記被写体を撮影した撮影画像の予測値を擬似撮影画像として生成する擬似画像生成部
を備える画像処理装置。 - 前記値は、前記画像の重要領域が、前記擬似撮影画像を観視する観視者の中心視野内に収まるように決定される
ように構成された
請求項1に記載の画像処理装置。 - 前記擬似画像生成部は、前記パラメータの値を、所定値から、決定された前記値に段階的に変更し、変更後の前記値と前記奥行き画像に基づいて前記擬似撮影画像を生成する
ように構成された
請求項2に記載の画像処理装置。 - 前記パラメータは、前記擬似撮影画像の仮想視点の位置であり、
前記所定の撮影方法は、トラック撮影である
ように構成された
請求項3に記載の画像処理装置。 - 前記パラメータは、前記擬似撮影画像の仮想視距離であり、
前記所定の撮影方法は、ドリーイン撮影またはドリーアウト撮影である
ように構成された
請求項3に記載の画像処理装置。 - 前記パラメータは、前記画像のスケーリング率であり、
前記所定の撮影方法は、ズームイン撮影またはズームアウト撮影である
ように構成された
請求項3に記載の画像処理装置。 - 前記所定の撮影方法に基づいて、前記擬似画像生成部により生成された前記擬似撮影画像の被写界深度を調整する調整部
をさらに備える
請求項6に記載の画像処理装置。 - 前記調整部は、前記所定の撮影方法がズームイン撮影である場合、前記擬似撮影画像の前記重要領域の前記被写体の奥行き方向の位置よりも手前側と奥側の前記被写体の領域を平滑化することにより、前記被写界深度を調整する
ように構成された
請求項7に記載の画像処理装置。 - 前記調整部は、前記所定の撮影方法がズームアウト撮影である場合、前記擬似撮影画像のボケ領域に対してデブラー処理を行うことにより、前記被写界深度を調整する
ように構成された
請求項7に記載の画像処理装置。 - 前記パラメータは、前記擬似撮影画像の視線方向の角度であり、
前記所定の撮影方法は、パニング撮影またはチルト撮影である
ように構成された
請求項3に記載の画像処理装置。 - 前記パラメータは、前記擬似撮影画像の仮想視点の位置であり、
前記所定の撮影方法は、前記画像の撮影位置より上または下の位置での撮影である
ように構成された
請求項1に記載の画像処理装置。 - 前記擬似画像生成部は、前記値と、外挿補間された周辺奥行き画像と前記奥行き画像を合成した合成奥行き画像とに基づいて、外挿補間された周辺画像と前記画像を合成した合成画像から前記擬似撮影画像を生成する
ように構成された
請求項1に記載の画像処理装置。 - 前記画像を用いて前記周辺画像を外挿補間し、前記奥行き画像を用いて前記周辺奥行き画像を外挿補間する周辺生成部と、
前記周辺生成部により外挿補間された前記周辺画像と前記画像を合成して前記合成画像を生成し、前記周辺生成部により外挿補間された前記周辺奥行き画像と前記奥行き画像を合成して前記合成奥行き画像を生成する合成部と
をさらに備える
請求項12に記載の画像処理装置。 - 前記擬似画像生成部により生成された前記擬似撮影画像の少なくとも一部を削除する切り出し部
をさらに備える
請求項13に記載の画像処理装置。 - 画像処理装置が、
画像の特徴に基づいて決定されたパラメータの値と前記画像の被写体の奥行き方向の位置を表す奥行き画像とに基づいて、前記画像から、所定の撮影方法で前記被写体を撮影した撮影画像の予測値を擬似撮影画像として生成する擬似画像生成ステップ
を含む画像処理方法。 - 画像と前記画像の被写体の奥行き方向の位置を表す奥行き画像とに基づいて、前記画像の撮影画角を推定する撮影画角推定部と、
前記撮影画角推定部により推定された前記撮影画角と、擬似撮影画像の観視画角とに基づいて、前記画像から、前記観視画角と同一の画角で撮影した場合の撮影画像の予測値を擬似撮影画像として生成する生成部と
を備える画像処理装置。 - 前記生成部は、前記観視画角が前記撮影画角より大きい場合、前記画像を縮小することにより前記擬似撮影画像を生成する
ように構成された
請求項16に記載の画像処理装置。 - 前記生成部は、前記観視画角が前記撮影画角より小さい場合、前記画像を拡大することにより前記擬似撮影画像を生成する
ように構成された
請求項16に記載の画像処理装置。 - 前記生成部により生成された前記擬似撮影画像または外部から入力される画像を用いて、前記擬似撮影画像の周辺領域の画像を外挿補間する周辺生成部と、
前記周辺生成部により外挿補間された前記周辺領域の画像と前記擬似撮影画像を合成する合成部と
をさらに備える
請求項16に記載の画像処理装置。 - 画像処理装置が、
画像と前記画像の被写体の奥行き方向の位置を表す奥行き画像とに基づいて、前記画像の撮影画角を推定する撮影画角推定ステップと、
前記撮影画角推定ステップの処理により推定された前記撮影画角と、擬似撮影画像の観視画角とに基づいて、前記画像から、前記観視画角と同一の画角で撮影した場合の撮影画像の予測値を擬似撮影画像として生成する生成ステップと
を含む画像処理方法。
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/300,346 US10412374B2 (en) | 2014-04-10 | 2015-03-27 | Image processing apparatus and image processing method for imaging an image by utilization of a pseudo image |
JP2016512665A JP6610535B2 (ja) | 2014-04-10 | 2015-03-27 | 画像処理装置および画像処理方法 |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2014081070 | 2014-04-10 | ||
JP2014-081070 | 2014-04-10 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2015156149A1 true WO2015156149A1 (ja) | 2015-10-15 |
Family
ID=54287726
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2015/059586 WO2015156149A1 (ja) | 2014-04-10 | 2015-03-27 | 画像処理装置および画像処理方法 |
Country Status (3)
Country | Link |
---|---|
US (1) | US10412374B2 (ja) |
JP (1) | JP6610535B2 (ja) |
WO (1) | WO2015156149A1 (ja) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2017102686A (ja) * | 2015-12-01 | 2017-06-08 | 日本電信電話株式会社 | 情報取得装置、情報取得方法及び情報取得プログラム |
JP2022071710A (ja) * | 2020-10-28 | 2022-05-16 | Kddi株式会社 | 歪曲収差補正装置、方法及びプログラム |
WO2022269999A1 (ja) * | 2021-06-25 | 2022-12-29 | ソニーグループ株式会社 | 制御装置、制御方法、およびプログラム |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10196088B2 (en) * | 2011-04-19 | 2019-02-05 | Ford Global Technologies, Llc | Target monitoring system and method |
CN107112038A (zh) * | 2014-12-22 | 2017-08-29 | 汤姆逊许可公司 | 使用递归分级处理生成外推图像的装置和方法 |
TWI636316B (zh) * | 2017-05-05 | 2018-09-21 | 致伸科技股份有限公司 | 通訊裝置及其光學裝置 |
KR102269760B1 (ko) * | 2019-11-26 | 2021-06-29 | 서울과학기술대학교 산학협력단 | 영상 전송 시스템의 다중 영상 시점 보정 장치 |
US20220358619A1 (en) * | 2021-05-10 | 2022-11-10 | Mediatek Inc. | Automatic dolly zoom image processing device |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH08242469A (ja) * | 1995-03-06 | 1996-09-17 | Nippon Telegr & Teleph Corp <Ntt> | 撮像カメラ装置 |
JP2007128009A (ja) * | 2005-11-07 | 2007-05-24 | Research Organization Of Information & Systems | 焦点ぼけ構造を用いたイメージング装置及びイメージング方法 |
JP2010175885A (ja) * | 2009-01-30 | 2010-08-12 | Panasonic Corp | 映像表示装置 |
JP2013254338A (ja) * | 2012-06-06 | 2013-12-19 | Nippon Telegr & Teleph Corp <Ntt> | 映像生成システム、映像生成装置、映像生成方法及びコンピュータプログラム |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP4825561B2 (ja) | 2006-03-29 | 2011-11-30 | 株式会社東芝 | 画像表示装置 |
JP5268271B2 (ja) | 2007-03-23 | 2013-08-21 | 株式会社東芝 | 画像表示装置および画像表示方法 |
JP4455609B2 (ja) | 2007-03-27 | 2010-04-21 | 株式会社東芝 | 画像表示装置および画像表示方法 |
JP2009162899A (ja) | 2007-12-28 | 2009-07-23 | Pioneer Electronic Corp | 画像表示方法 |
JP2014071207A (ja) * | 2012-09-28 | 2014-04-21 | Canon Inc | 画像処理装置、撮像システム、画像処理システム |
US20150379720A1 (en) * | 2013-01-31 | 2015-12-31 | Threevolution Llc | Methods for converting two-dimensional images into three-dimensional images |
JP5662511B2 (ja) * | 2013-04-10 | 2015-01-28 | シャープ株式会社 | 撮像装置 |
-
2015
- 2015-03-27 WO PCT/JP2015/059586 patent/WO2015156149A1/ja active Application Filing
- 2015-03-27 US US15/300,346 patent/US10412374B2/en not_active Expired - Fee Related
- 2015-03-27 JP JP2016512665A patent/JP6610535B2/ja not_active Expired - Fee Related
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH08242469A (ja) * | 1995-03-06 | 1996-09-17 | Nippon Telegr & Teleph Corp <Ntt> | 撮像カメラ装置 |
JP2007128009A (ja) * | 2005-11-07 | 2007-05-24 | Research Organization Of Information & Systems | 焦点ぼけ構造を用いたイメージング装置及びイメージング方法 |
JP2010175885A (ja) * | 2009-01-30 | 2010-08-12 | Panasonic Corp | 映像表示装置 |
JP2013254338A (ja) * | 2012-06-06 | 2013-12-19 | Nippon Telegr & Teleph Corp <Ntt> | 映像生成システム、映像生成装置、映像生成方法及びコンピュータプログラム |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2017102686A (ja) * | 2015-12-01 | 2017-06-08 | 日本電信電話株式会社 | 情報取得装置、情報取得方法及び情報取得プログラム |
JP2022071710A (ja) * | 2020-10-28 | 2022-05-16 | Kddi株式会社 | 歪曲収差補正装置、方法及びプログラム |
JP7473449B2 (ja) | 2020-10-28 | 2024-04-23 | Kddi株式会社 | 歪曲収差補正装置、方法及びプログラム |
WO2022269999A1 (ja) * | 2021-06-25 | 2022-12-29 | ソニーグループ株式会社 | 制御装置、制御方法、およびプログラム |
Also Published As
Publication number | Publication date |
---|---|
JPWO2015156149A1 (ja) | 2017-04-13 |
JP6610535B2 (ja) | 2019-11-27 |
US20170150130A1 (en) | 2017-05-25 |
US10412374B2 (en) | 2019-09-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6610535B2 (ja) | 画像処理装置および画像処理方法 | |
JP6730690B2 (ja) | シーン内に存在する不要なオブジェクトの除去に基づくシーンの画像の動的生成 | |
US10609284B2 (en) | Controlling generation of hyperlapse from wide-angled, panoramic videos | |
JP6746607B2 (ja) | パンニングショットの自動生成 | |
JP5847924B2 (ja) | 拡張現実表現のための二次元画像取込み | |
US9185388B2 (en) | Methods, systems, and computer program products for creating three-dimensional video sequences | |
JP6371553B2 (ja) | 映像表示装置および映像表示システム | |
JP2014215604A (ja) | 画像処理装置および画像処理方法 | |
US20140118482A1 (en) | Method and apparatus for 2d to 3d conversion using panorama image | |
CN104980651A (zh) | 图像处理设备及控制方法 | |
JP2022523478A (ja) | マルチビュー視覚データからの損傷検出 | |
JP7224350B2 (ja) | 画像キャプチャ装置及び画像キャプチャ方法 | |
JP2012222743A (ja) | 撮像装置 | |
US20220358619A1 (en) | Automatic dolly zoom image processing device | |
JP2013172446A (ja) | 情報処理装置、端末装置、撮像装置、情報処理方法、及び撮像装置における情報提供方法 | |
US20240242337A1 (en) | Mobile multi-camera multi-view capture | |
JP2015012481A (ja) | 画像処理装置 | |
CN113805824B (zh) | 电子装置以及在显示设备上显示图像的方法 | |
EP4165874A1 (en) | Producing and adapting video images for presentation on displays with different aspect ratios | |
US20230290061A1 (en) | Efficient texture mapping of a 3-d mesh | |
KR101132976B1 (ko) | 복수 개의 카메라를 구비한 모바일 기기, 이를 이용한 디스플레이 표시방법 | |
JP2023033355A (ja) | 画像処理装置およびその制御方法 | |
CN115589532A (zh) | 防抖处理方法、装置、电子设备和可读存储介质 | |
JP6320165B2 (ja) | 画像処理装置及びその制御方法、並びにプログラム | |
Lee | Novel video stabilization for real-time optical character recognition applications |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 15776025 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2016512665 Country of ref document: JP Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 15300346 Country of ref document: US |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 15776025 Country of ref document: EP Kind code of ref document: A1 |