JP2014134890A - Image data processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image data processing device that performs suitable image luster and shading processing according to an enlargement/reduction rate of image data.SOLUTION: An image data processing device includes: an enlargement/reduction rate calculation part which acquires an enlargement/reduction instruction for image data and calculates an enlargement/reduction rate for the image data from the enlargement/reduction instruction; and an image data generation part which enlarges or reduces partial image data of the image data based upon the enlargement/reduction rate. Further, the image data processing device includes: a subject shape estimation part which acquires a three-dimensional shape of a subject surface from the image data; a light source environment estimation part which acquires incident light information from a light source of the subject surface based upon the image data and the three-dimensional shape; and a luster/shade control part which controls a luminance value of a high-luminance area or low-luminance area of the image data based upon the three-dimensional shape, incident light information, and enlargement/reduction rate.

Description

  The present invention relates to an image data processing apparatus, and more particularly to a technique for performing gloss and shadow control of an image in accordance with the enlargement / reduction ratio of the image data.

  Conventionally, an imaging apparatus such as a digital camera is often equipped with an optical zoom function or a digital zoom function in order to capture image data as if it were close to a subject without changing the imaging position. In the optical zoom, the focal length is changed by moving a lens, and image data is captured by optically enlarging. On the other hand, in the digital zoom, the lens itself is not moved, and the image data is picked up by changing the size of the subject by enlarging a part of the image data. Moreover, not only when capturing image data, but also by displaying the image data enlarged or reduced by digital processing when displaying the image data, as if the positional relationship between the subject and the imaging device was changed. Generating image data is also one of digital zoom functions.

  In recent years, techniques for performing image data processing on enlarged image data to generate suitable image data have been developed. For example, in Japanese Patent Application Laid-Open No. 2004-228688, by giving a commonality to the image quality change obtained by the optical zoom and the image quality change obtained by the digital zoom, the output image is blurred as if it was obtained by using the optical zoom when executing the digital zoom. A method for generating data is disclosed.

JP 2012-160863 A

  However, the method described in Patent Document 1 has a problem in that the texture is deteriorated because the glossy or shaded area representing the texture of the subject is enlarged as it is. Also, just by enlarging or reducing a part of the image, the impression that the distance to the subject has changed cannot be obtained, and the sense of reality has been impaired. For example, an area that is known to be a shadow in the captured image data, but the enlarged image data is entirely dark, or the boundary area between the shadow area and the non-shadow area having a high contrast is enlarged as it is. It was difficult to generate data.

  The present invention has been invented in view of the above problems, and an object of the present invention is to provide an image data processing technique for appropriately controlling gloss and shadow of an image according to the enlargement / reduction ratio of the image data.

In order to solve the above-described problems, the present invention includes the following technical means.
According to an aspect of the present invention, an enlargement / reduction ratio calculation unit that acquires an enlargement / reduction instruction for image data and calculates an enlargement / reduction ratio of the image data from the enlargement / reduction instruction, and the image based on the enlargement / reduction ratio An image data processing apparatus comprising: an image data generation unit that enlarges or reduces partial image data of the data, and acquires a three-dimensional shape of a subject surface from the image data, for example, for each pixel or for each region Based on the subject shape estimation unit, the image data, and the three-dimensional shape, a light source environment estimation unit that acquires incident light information from the light source on the subject surface for each pixel or region, the three-dimensional shape, A gloss / shadow control unit that controls a luminance value of at least one of a high luminance region and a low luminance region of the image data based on the incident light information and the enlargement / reduction ratio. The gloss / shadow control unit performs brightness correction that enhances or suppresses at least one of a high-brightness region and a low-brightness region of the image data according to the enlargement / reduction ratio. An image data processing device is provided.

  According to the above invention, based on the subject shape, the light source information, and the enlargement / reduction ratio, the luminance correction for suppressing the gloss and shadow areas is performed when the image is enlarged, and the gloss and shadow areas are emphasized when the image is reduced. Perform brightness correction.

  According to the image data processing apparatus of the present invention, the shading and gloss of the input image data are adjusted according to the enlargement / reduction ratio of the image data, and the image data appropriately performing the gloss and shading control of the image data is generated. Can do.

It is a functional block diagram which shows the example of 1 structure of the image data display apparatus with an imaging part provided with the image data processing apparatus by the 1st Embodiment of this invention. It is a functional block diagram which shows one structural example of a glossiness / shadow control part. It is a figure explaining input image data and a normal vector. It is a figure explaining a normal vector. It is a figure explaining a light source vector. It is a figure explaining specular reflection. It is a figure explaining the input method of the expansion / contraction instruction | indication. It is a figure explaining the angle | corner which a normal vector and a light source vector make. It is a figure which shows the relationship between an enlarging / reducing rate and the emphasis / suppression degree of gloss and shadow. It is a figure explaining the change result at the time of enlarging image data. It is a figure explaining the change result at the time of reducing image data. It is a flowchart figure which shows the flow of the image processing in the 1st Embodiment of this invention. It is a figure which shows a mode that the enlarged image data and image data or the reduced image data, and the menu screen were displayed simultaneously. It is a functional block diagram which shows one structural example of the image data display apparatus with an imaging part provided with the image data processing apparatus by the 2nd Embodiment of this invention. It is a functional block diagram which shows the structural example of the principal part of FIG. 13A. It is a figure explaining the relationship of the expansion / contraction of the inclination of the image data display apparatus by 2nd Embodiment. It is a figure showing the axis | shaft which shows the inclination angle of the image data display apparatus by 2nd Embodiment. It is the figure which looked at the image data display apparatus by 2nd Embodiment from the side. It is the figure which looked at the image data display apparatus by 2nd Embodiment from the top.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The accompanying drawings show specific embodiments and implementation examples according to the principle of the present invention, but these are for understanding the present invention and are not intended to limit the present invention. Not used. The configuration in each drawing is exaggerated for easy understanding, and is different from the actual interval and size.

  In the following description, the same reference numerals are used in different drawings to omit the description. The image data described below indicates both still image data and moving image data. Content data indicates still image data and moving image data. Further, when the moving image data includes audio data. It also includes audio data. Moreover, the imaging unit and the display unit may be configured separately.

<First Embodiment>
FIG. 1A is a functional block diagram illustrating a configuration example of an image data display device with an imaging unit including the image data processing device according to the first embodiment of the present invention. The subject is imaged by an image data display device with an imaging unit, image data that is appropriately glossy and shaded according to the enlargement / reduction ratio of the subject is generated from the captured image data, and the generated image data is displayed on the display unit. It is a figure which shows the example in the case of displaying.

  Details of the system configuration and operation in the first embodiment of the present invention will be described below with reference to FIGS. 1A and 1B. An image data display device 100 with an imaging unit including the image data processing device according to the present embodiment includes an input / output device 101, an image data processing device 102, and a display unit 103. Includes an imaging unit 104 such as a camera, a transmission / reception unit 105, an image data input / output unit 106, and a storage unit 107, at least one image source, and an input / output unit (such as an operation unit) 108. Furthermore, the image data processing apparatus 102 includes a subject shape estimation unit 109, a light source environment estimation unit 110, an enlargement / reduction rate calculation unit 111, a gloss / shadow control unit 112, and an image data generation unit 113.

  Further, specific examples of the image data processing system according to the first embodiment include, for example, a television receiver, a monitor, a mobile phone, a smartphone, a tablet PC, and a PC that can display image data. Mobile game machines, electronic photo frames, digital still cameras, video cameras, and the like.

  First, the internal components of the input / output device 101 and the display unit 103 will be described. The imaging unit 104 includes an imaging lens and an imaging device such as a CCD (Charge Coupled Device), captures a still image or a moving image of the subject, and outputs the captured image data as input image data.

  The transmission / reception unit 105 is an interface for general contents or the like that receives image data input through a television broadcast wave, the Internet, or other communication lines, and outputs it as input image data. For example, the transmission / reception unit 105 may be a device that inputs and reproduces content data included in a broadcast wave by a television signal receiving device that receives a broadcast wave of a television broadcast. Further, it may be a device that receives and reproduces content data distributed from the Internet or other communication lines.

  The image data input / output unit 106 is a general one that receives image data input from the outside and outputs it as input image data. For example, the input / output unit 108 may be a device that receives and reproduces content data by a receiver or the like that receives an image data signal from an external device such as a disc player. Any device for inputting content data may be used.

  The storage unit 107 is a general memory that stores content data or outputs stored image data. For example, the storage unit 107 may be a device that reads content data from a recording medium and reproduces it. Examples of the recording medium include a storage device in a commercially available video camera and a removable storage device (an electronic medium such as a magneto-optical disk or a semiconductor memory).

  The input / output unit 108 inputs user commands and voices to the image data processing apparatus and outputs voices, such as voice input / output devices such as key buttons, microphones and speakers, and infrared irradiation devices that detect human movement. To do.

  Here, the imaging unit 104, the transmission / reception unit 105, the image data input / output unit 106, and the storage unit 107 constituting the input / output device 101 are means for inputting image data by different methods, respectively. The input / output device 101 only needs to include the imaging unit 104, the transmission / reception unit 105, the image data input / output unit 106, at least one of the storage units 107, and the input / output unit 108. In the following description, an example in which the imaging unit 104, the storage unit 107, and the input / output unit 108 are provided in the input / output device 101 will be described.

  The display unit 103 is configured by a general LCD (Liquid Crystal Display) or the like, and displays image data, character data, and the like input from the outside.

  Moreover, since the input / output device 101 and the display unit 103 described above are general ones and are not directly related to the contents of the present invention, their detailed description is omitted.

Next, a configuration example of the image data processing apparatus 102 will be described.
The image data processing apparatus 102 includes a subject shape estimation unit 109, a light source environment estimation unit 110, an enlargement / reduction ratio calculation unit 111, a gloss / shadow control unit 112, and an image data generation unit 113, and includes an imaging unit 104, The input image data input from the image data input / output unit 106 or the like is subjected to image data processing based on the enlargement / reduction instruction input from the input / output unit 108 to generate output image data, and the generated output image data Are output to the display unit 103, the image data input / output unit 106, or the like. Here, for example, the image data processing apparatus 102 can be configured by a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), or the like.

  The subject shape estimation unit 109 estimates, for example, a normal vector representing the three-dimensional shape of the subject surface in the image data based on the input image data, and outputs it as normal vector information.

  The light source environment estimation unit 110 estimates the direction of light incident on the subject surface based on the input image data and the normal vector estimated by the subject shape estimation unit 109, and estimated incident light information Is output as light source vector information.

  The enlargement / reduction rate calculation unit 111 calculates and outputs the enlargement / reduction rate of the image data generated by the image data generation unit 113 based on an instruction to enlarge or reduce the image data input from the input / output unit 108.

  The gloss / shadow control unit 112 performs appropriate gloss control and / or shadow control processing according to the scaling ratio based on the input normal vector information, light source vector information, and scaling ratio. At this time, as the numerical value of the enlargement / reduction ratio calculated by the enlargement / reduction ratio becomes larger than a predetermined value, it is adjusted and generated so that the gloss and / or shadow effect is suppressed. In addition, as the numerical value of the enlargement / reduction ratio becomes smaller than a predetermined value, it is generated by adjusting so that the gloss and / or shadow effect is enhanced. The gloss / shadow control unit 112 may be able to independently perform at least one of gloss suppression or enhancement control and at least one of shadow suppression or enhancement control, or perform control as a set. You may do it.

  The image data generation unit 113 generates output image data by enlarging or reducing the image data according to the enlargement / reduction ratio.

FIG. 1B is a functional block diagram illustrating a detailed configuration example of the gloss / shadow control unit 112. As shown in FIG. 1B, a normal vector / light source vector inner product calculation unit 112-1 that receives the output from the enlargement / reduction ratio calculation unit 111 and calculates the inner product of the normal vector and the light source vector, and based on the calculation result From the inner product result determination unit 112-2 for obtaining the luminance value of the region, the gloss region / shadow region estimation unit 112-3 for estimating the gloss region and the shadow region based on the determination result, and the enlargement / reduction ratio calculation unit 111 And a glossy shadow area luminance correction unit 112-4 that corrects the luminance of the glossy area / shadow area, and the output is sent to the display unit 103.
The above is the details of the system configuration example according to the first embodiment.

  Next, the operation of the image data display device with an imaging unit according to the first embodiment will be described in detail with reference to FIGS. First, the subject shape estimation operation will be described with reference to FIGS. FIG. 2 is a diagram (bottom diagram) in which input image data (upper diagram) and a normal vector that is a three-dimensional vector representing a subject shape of the input image data are superimposed on the input data. FIG. 3 is a diagram for explaining a normal vector.

  The subject shape estimation unit 109 estimates a normal vector of the subject surface from the image data. The normal vector is a unit vector representing the direction of the subject surface. As shown in FIG. 2, when the subject H is a plane, the normal vector V has the same direction, but when the subject H is a spherical surface, the normal vector V has a different direction. As shown in FIG. 3, the normal vector 301 has an xyz coordinate system when the horizontal direction of the image data is the x axis, the vertical direction is the y axis, and the depth direction, that is, the direction from the imaging device to the subject is the z axis. , The x component (nx) of the normal vector is denoted by reference numeral 302, the y component (ny) is denoted by reference numeral 303, the z component (nz) is denoted by reference numeral 304, the pixel is denoted by reference numeral 305, and the region is denoted by reference numeral 306. , Vectors 302, 303, and 304 can be represented as a three-dimensional vector 301.

As a method for estimating a normal vector from image data, a method of estimating an orthogonal vector of an approximate plane derived from a target coordinate and a plurality of coordinates existing in the vicinity thereof as a normal vector, an object shadow is calculated. Method of estimation using (R. Zhang, P. Tsai, J. Cryer, M. Shah, “Shape from Shading: A
Survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 690-706, 1999), a method in which a database in which a large amount of image data of a subject and its three-dimensional shape are stored is prepared in advance, and the estimation is performed by comparing the image data with the image data in the database (T Hassner, R. Basri, “Example Based 3D Reconstruction Single 2D
Images ", Conference on Computer Vision
and Pattern Recognition Worksshop, pp, 15-23, 2006) and the like, and the method described above may be used.

  As described above, the normal vector is estimated. This normal vector is calculated for each pixel or region. The calculated normal vectors are collected as normal vector information and output to the light source environment estimation unit 110.

  Next, the operation of light source environment estimation in the light source environment estimation unit 110 will be described with reference to FIGS. FIG. 4 is a diagram for explaining a light source vector. FIG. 5 is a diagram for explaining the relationship between the light source vector and the reflection vector in the case of a specular reflection object. Based on the image data and the normal vector information, the direction of light incident on the subject surface, that is, the light source vector is estimated. The light source vector is a unit vector representing the direction of light incident on the subject surface.

As shown in FIG. 4, the light source vector is represented as a three-dimensional vector 401 using the same coordinate axis as the normal vector shown in FIG. The x component (l _x ) of the light source vector is denoted by reference numeral 402, the y component (l _y ) is denoted by reference numeral 403, the z component (l _z ) is denoted by reference numeral 404, the pixel is denoted by reference numeral 405, and the region is denoted by reference numeral 406. The light source vector 401 is represented by a combined vector of vectors 402, 403, and 404.

  As shown in FIG. 5, the method of estimating the light source vector from the image data and the normal vector information assumes that the subject is a specular reflection object (incident angle 501 and reflection angle 502 are the same) 503, and changes the direction of incident light. By comparing the generated image data with the luminance information of the image data, the direction of the incident light with the least error is estimated as the light source vector, the brightness of the shadow observed in the image data, the subject A method of deriving simultaneous equations in which the light source luminance is unknown by considering which direction the light source is shielded from the normal vector of the shadow of, and estimating it by solving it (Imari Sato, Yoichi Sato, Katsushi Ikeuchi , "Estimation of the light source environment based on the shadow of the object", Journal of Information Processing Society of Japan, Vol. 41, pp. 31-40, 2000) is known, and the method described above is used. It may be.

  The light source vector is calculated as described above. This light source vector is calculated for each pixel or region. The calculated light source vectors are collectively output to the gloss / shadow control unit 112 as light source vector information.

  Next, the operation of the enlargement / reduction ratio calculation process will be described with reference to FIG. FIG. 6 is a diagram for explaining an input method of an enlargement / reduction instruction by touch panel or gesture recognition. Based on the enlargement / reduction instruction input from the input / output unit 108, the enlargement / reduction ratio of the image data is calculated. The enlargement / reduction ratio is a value representing a ratio between the size of the original image data and the size of the image data after enlargement or reduction when a part of the image data is enlarged or reduced. For example, when the enlargement / reduction ratio is 1, the size of the image data enlarged or reduced based on the enlargement / reduction ratio is equal to the size of the original image data. When the enlargement / reduction ratio is 0.5, the size of the image data after enlargement / reduction based on the enlargement / reduction ratio is 0.5 times the size of the original image. That is, the image data reduced by a half becomes the image data after enlargement / reduction. When the enlargement / reduction ratio is 3, the size of the image data after enlargement / reduction is three times the size of the original image.

  That is, if the enlargement / reduction ratio is greater than 1, the image is enlarged, and if it is less than 1, the image is reduced. Here, the “image size” is a value represented by the number of vertical and horizontal pixels of the image, the length of the image on the diagonal line, and the like. That is, as the length on the diagonal increases, the number of vertical or horizontal pixels also increases.

  As an input method of the enlargement / reduction instruction, the enlargement / reduction ratio may be input numerically with the remote control device, may be input with a pinch operation of a touch panel provided on the display unit or the remote control device, or both hands of the user. It is also possible to recognize and input a gesture from the movement of

  When inputting with the remote control device, the enlargement / reduction ratio K can be input as a numerical value such as “1.2” or “0.2”. At this time, it is preferable to display the enlargement / reduction center coordinates 601 as a pointer on the numerical value or the image data to be enlarged, or to display the enlargement / reduction range with a frame because the user can easily specify the enlargement / reduction ratio. . Further, it is preferable that the enlargement / reduction center coordinates can also be input because only the enlargement / reduction position can be changed without changing the enlargement / reduction ratio.

  In the case of a pinch operation using a touch panel, the user holds the screen with two fingers, sets the enlargement / reduction ratio K to 1 or less when the fingers are moved close together, and sets the enlargement / reduction ratio K to 1 when the finger is moved away. It can be set above. The value of the enlargement / reduction ratio K is obtained from the ratio between the start points 602 and 603 and the end points 604 and 605 of two fingers. The enlargement / reduction center coordinate 601 is a midpoint 601 of a straight line connecting the start points 602 and 604 of two fingers.

  In the case of a gesture operation, the left hand and the right hand are used for the operation. When the left hand and the right hand are brought close to each other, the enlargement / reduction ratio K is set to 1 or less, and when it is moved away, the enlargement / reduction ratio K is set to 1 or more. The value of K is obtained from the ratio of the start point 603 of the left hand and the start point 602 of the right hand, and the end point 605 of the left hand and the end point 604 of the right hand. The enlargement / reduction center coordinate 601 is a midpoint 601 of a straight line connecting the start point 603 of the left hand and the start point 602 of the right hand. Thus, the enlargement / reduction ratio is calculated.

  Next, the operation of gloss and shadow control will be described with reference to FIG. FIG. 7 is a diagram for explaining the angle formed between the normal vector and the light source vector. For example, the gloss / shadow control unit 112 adjusts parameters so that the gloss and shadow effects are suppressed as the scaling ratio becomes larger than 1, based on the normal vector information, the light source vector information, and the scaling ratio. Then, as the enlargement / reduction ratio becomes smaller than 1, the parameters are adjusted so that the gloss and shadow effects are enhanced. Specifically, the gloss area and shadow area in the image data are estimated, and when the enlargement / reduction ratio is greater than 1, the brightness of the gloss area is lowered, the brightness of the shadow area is increased, and the enlargement / reduction ratio is greater than 1. When it is smaller, the brightness of the glossy area is increased and the brightness of the shadow area is decreased.

First, a method for estimating a glossy area and a shadow area in image data will be described in detail. The inner product I of the normal vector N = (n _x , n _y , n _z ) and the light source vector L = (l _x , l _y , l _z ) is calculated by the following (formula 1). At this time, the direction of the light source vector is reversed, and the light source vector and the start point are matched.

  Since the normal vector N and the light source vector L are unit vectors, the inner product L calculated by (Equation 1) is [−1, 1]. As shown in FIG. 7, (a) the closer the inner product is to 1, the closer the angle 701 between the light source vector and the normal vector is to 0 degrees, and (b) the closer to 0, the light source vector and the normal vector. The angle 702 formed is closer to 90 degrees, and the closer to (c) −1, the closer the angle 703 formed between the light source vector and the normal vector is to 180 degrees. Based on the inner product calculation result, a high luminance region in which the angle between the light source vector and the normal vector is 0 to 90 degrees, that is, a high luminance region in which the inner product result is 0 to 1 is estimated as a glossy region. Further, a low luminance region where the angle between the light source vector and the normal vector is 90 to 180 degrees, that is, a low luminance region where the inner product is −1 to 0 is estimated as a shadow region.

The high luminance region and the low luminance region of the image data may be extracted with a predetermined value set in advance, or based on the luminance distribution in the image data, the region of 10% of the high luminance pixel is the high luminance region and the low luminance region. An area of 15% luminance pixels may be extracted as a low luminance area.
As described above, the glossy area and the shadow area in the image data are estimated.

Next, the gloss / shadow control when the enlargement / reduction ratio is greater than 1 will be described in detail. FIG. 8 is a graph showing an example of the relationship between the enlargement / reduction ratio and the enhancement or suppression control of gloss / shadow control. As shown in FIG. 8, control is performed such that the effect of gloss and shadow is suppressed as the enlargement / reduction ratio becomes larger than 1, and the effect of gloss and shadow is enhanced as the enlargement / reduction ratio becomes smaller than 1. Then, the effect of gloss and shadow is changed within the range of the maximum value K _max 801 and the minimum value K _min 802 of the enlargement / reduction ratio. The change of the gloss and shadow effect may be performed by the linear function 803 or the nonlinear functions 804 and 805 in FIG. Here, a processing example in the case of performing the change by the linear function 803 will be described in detail.

The luminance of the target pixel of the image data is V _i , the luminance of the target pixel of the image data after gloss and shadow control is V _o, and the difference between the inner product L of _the target pixel and the inner product L of the surrounding pixels is equal to or less than a predetermined value _Assuming that the absolute value of the difference from the luminance V _s of the surrounding pixels is V _sub (L) = | V _i −V _s |, (Equation 2) is used for a glossy area, and (Equation 2) is used for a shadow area. 3) is used to correct the brightness of the gloss and shadow areas.

The luminance V _s is an average value of luminance values of peripheral pixels in which the difference between the inner product L of the target pixel and the inner product L of the peripheral pixel is equal to or less than a predetermined value. As described above, gloss and shadow control when the enlargement / reduction ratio is greater than 1 are performed. Next, gloss and shadow control when the enlargement / reduction ratio is smaller than 1 will be described in detail. The luminance of the gloss / shadow area is corrected using (Equation 3) for the glossy area and (Equation 2) for the shadow area. As described above, the gloss / shadow control process when the enlargement / reduction ratio is smaller than 1 is performed.

  Finally, the image data generation operation will be described with reference to FIGS. FIG. 9 is a diagram for explaining the change result of the gloss / shadow effect when the enlargement / reduction ratio is 1 or more. FIG. 10 is a diagram for explaining the change result of the gloss / shadow effect when the enlargement / reduction ratio is 1 or less.

  With reference to FIG. 9, first, the operation of the image data generation unit 113 when the enlargement / reduction ratio is 1 or more will be described in detail. The image data generation unit 113 uses the enlargement / reduction center coordinates 902 and the enlargement / reduction ratio K on the input image data 901 in FIG. 9 to enlarge the image data 903 after the gloss / shadow control. An enlarged range 904 is cut out from the image data, the enlarged range is multiplied by K, and enlarged image data 905 is output.

  Next, the operation of the image data generation unit 113 when the enlargement / reduction ratio is 1 or less will be described in detail with reference to FIG. The image data generation unit 113 reduces the image data 1003 after the gloss / shadow control using the reduction center coordinates 1002 and the enlargement / reduction ratio K on the input image data 1001 in FIG. A reduced range 1003 is cut out from the image data, the reduced range is multiplied by K, and reduced image data 1004 is output. When the non-viewing area 1005 appears by reducing the image data, a method of displaying black in the non-viewing area, a method of extending the image data edge, or a method of displaying the area near the image data by folding back is used. To interpolate the area outside the angle of view.

  The flow of the data processing operation according to the first embodiment will be described below with reference to the flowchart shown in FIG.

  First, processing is started (START), and in step S1101, the image data processing apparatus 102 captures captured image data from the imaging unit 104, the transmission / reception unit 105, and the like. Next, in step S1102, the subject shape estimation unit 109 estimates the normal vector of the subject from the captured image data. For example, a technique for estimating a three-dimensional shape from two-dimensional information obtained from one image using shadow distribution analysis or database collation is known. Next, in step S1103, the light source environment estimation unit 110 estimates a light source vector using image data and normal vector information. In step S <b> 1104, the image data processing apparatus 102 captures enlargement instruction information from the input / output unit 108. In step S1105, the enlargement / reduction ratio calculation unit 111 calculates the enlargement / reduction ratio of the image data using the enlargement instruction information. In step S1106, the luminance correction of the image data is performed based on the normal vector information, the light source vector information, and the enlargement / reduction ratio.

  Specifically, if the enlargement / reduction ratio is larger than a predetermined value, the gloss / shadow control unit 112 performs processing to suppress the gloss / shadow of the image data. If the enlargement / reduction ratio is smaller than a predetermined value, the gloss / shadow control unit performs a process of enhancing the gloss / shadow of the image data. In step S1107, the image data generation unit 113 generates image data obtained by enlarging or reducing the image data after the gloss and shadow control using the enlargement / reduction ratio.

  More specifically, in the gloss / shadow control unit 112, each functional unit in FIG. 1B functions as described above. In step S1108, the image data generated by the image data generation unit 113 is output to the display unit. As described above, the image data display apparatus 100 according to the first embodiment operates.

  In this embodiment, the enlarged image data output from the image data generation unit 113 is displayed on the entire screen of the display device. However, as shown in FIG. 12, the enlarged image data 1201 is superimposed on the input image data. It may be displayed, or the input image data may be reduced and the enlarged image data 1202 may be displayed in an empty space. At this time, when the reduction ratio of the entire input image data is calculated according to the enlargement ratio of the input image data and the gloss / shade of the input image data is emphasized according to the reduction ratio, the enlarged image data and the reduced input image data are Both are suitable because appropriate effects can be imparted. Furthermore, when imaging the state of makeup, when the enlarged image data, the reduced image data, and the menu screen (1203) on which the makeup procedure is described are displayed at the same time, the user confirms the makeup procedure. However, it is possible to proceed with the procedure while confirming the overall atmosphere with the reduced image data with the enlarged image data such as the eyes and lips. . Alternatively, it is also easy to proceed with the procedure while looking at the state of change, including the model production procedure and surgery.

  In the present embodiment, the image data generated by the image data processing device 102 included in the image data display device 100 is displayed on the display unit 103, but is generated by another image data display device including the image data processing device 102. The image data may be received by the transmission / reception unit 105 and displayed on the display unit 103. This configuration is preferable because image data processing such as a video conference and video chat with a remote place can be performed via the transmission / reception unit 105.

  In the present embodiment, the example in which the enlargement / reduction ratio calculation unit 111 designates the upper right corner 602 and the lower left corner 603 of the area to be enlarged of the image data has been described, but the method of designating the upper left corner and the lower right corner focus However, the same effect can be obtained. of course. You may make it extend the whole area | region.

  According to the image data display device including the image data processing device according to the present embodiment described above, it is possible to display enlarged image data appropriately subjected to gloss / shadow control according to the enlargement / reduction ratio of the subject.

  Therefore, even when a digital image is enlarged / reduced by digital zoom, the same texture as when enlarged / reduced by optical zoom can be provided. For example, a delicate image such as when confirming makeup can be confirmed by digital processing. Further, since an optical mechanism such as a zoom lens can be simplified, there is an advantage that the cost can be reduced.

<Second Embodiment>
Next, the configuration of the image data display device according to the second embodiment of the present invention will be described with reference to FIG. 13A. In FIG. 13A, the same reference numerals are given to the same components as those in FIG. 1, and these components perform the same processing as in the embodiment of FIG.

  The difference between the present embodiment and the first embodiment is that the present embodiment includes a tilt angle detector 1301 that detects the tilt of the display. The operation is almost the same as in the first embodiment, and the inclination of the display device including the display unit 103 is converted into the enlargement / reduction rate of the image data, and the image data generation unit 113 does not perform the enlargement / reduction processing, thereby Appropriate gloss / shadow control according to the tilt angle becomes possible. In the first embodiment, a part of the image data is enlarged or reduced. In the second embodiment, enlargement / reduction based on the inclination of the image data according to the position of the x-axis or the y-axis in the image data. Gloss / shadow control is performed by estimating.

  The outline of gloss / shadow control will be described below. As shown in FIG. 14, when the image data display device 1401 is in a tilted (rotated) state 1403 without moving the display device center 1402 greatly, a region 1404 where the position of the image data and the viewer approaches, and the image An area 1405 in which the viewer moves away from the data is generated. A region 1404 in which the image data and the viewer's position are close can be regarded as an enlargement of the image data, and the gloss / shadow control is suppressed. Further, an area 1405 where the image data and the viewer's position move away from each other can be regarded as the image data being reduced, and the gloss / shadow control is emphasized.

  FIG. 13B is a functional block diagram illustrating a configuration example of the gloss / shadow control unit 112. The difference from FIG. 1B of the first embodiment is that, instead of the enlargement / reduction ratio calculation unit 111 of FIG. 1B, in FIG. 13B, an inclination angle detection unit 1301 and an inclination angle as a detection result are input. An enlargement / reduction rate conversion unit 111a is provided for converting each enlargement / reduction rate.

  In other words, in the second embodiment, the image data processing apparatus 102 acquires the inclination angle output from the inclination angle detection unit 1301, and the enlargement / reduction ratio calculation unit 111 converts the inclination angle into an enlargement / reduction ratio. The tilt angle detection unit 1301 includes an Euler angle information detection unit 1301-1 that detects Euler angle information of the image data display device (display unit) 103. The Euler angle information detected by the Euler angle information detection unit 1301-1 includes a roll angle 1501, a pitch angle 1502, and a yaw angle 1503 of the display device as shown in FIG. From the viewpoint of viewing the display screen in the vertical direction from the front, the roll angle is the rotation angle around the vertical axis from the upper part to the lower part of the display device, and the pitch angle is horizontal across the left and right devices of the display device The rotation angle and pitch angle around the axis of the display device is the rotation angle around the horizontal axis passing through the left and right of the display device, and the yaw angle is the rotation around the axis passing from the front to the back of the display device It represents an angle. The basic posture is that the device front of the display device is facing up and the device and the ground are horizontal, that is, the screen is placed on a desk or the like with the roll angle and pitch angle The measured value shows around 0. For the yaw angle, magnetic north and true north are set to zero.

  FIG. 16 is a side view of the image data display device. FIG. 16 shows a roll angle of the display device of 0 degree 1601, a roll angle having a positive value 1602, and a roll angle having a negative value 1603. FIG. 17 is a top view of the image data display device. FIG. 17 shows a pitch angle of 0 degrees 1701, a pitch angle having a positive value 1702, and a pitch angle having a negative value 1703. The tilt angle is continuously detected, and the detected tilt angle is output to the enlargement / reduction ratio calculation unit 111 as an electrical signal.

  The roll angle / pitch angle enlargement / reduction rate conversion unit 111-1 of the enlargement / reduction rate calculation unit 111 converts the roll angle φ and the pitch angle θ into an enlargement / reduction rate. First, the case where the roll angle is larger than the pitch angle (| φ |> | θ |) and the roll angle is a positive value will be described in detail. In this case, the enlargement / reduction ratio K is calculated using (Equation 4). The enlargement / reduction ratio changes according to the position of the y-axis.

Here, the vertical resolution of the image data is H, and the position on the y-axis in the image is y. Further, the maximum value K _max and the minimum value K _min of the enlargement / reduction ratio are obtained using (Equation 5) and (Equation 6), respectively, using the roll angle value.

Here, the maximum value of the roll angle is φ _max . As the roll angle increases, the maximum value of the enlargement / reduction ratio increases, the minimum value decreases, and the effect of gloss / shadow control increases. That is, when the roll angle is a positive value, a pixel whose position on the y-axis is smaller than 2 / H is regarded as approaching the viewer, that is, enlarged, and the enlargement / reduction ratio K is set to 1 or more. Further, a pixel whose position on the y axis is larger than 2 / H is regarded as being away from the viewer, that is, reduced, and the enlargement / reduction ratio K is set to 1 or less. As described above, the enlargement / reduction ratio K when the roll angle is larger than the pitch angle (| φ |> | θ |) and the roll angle is a positive value is calculated.

  Next, the case where the roll angle is larger than the pitch angle (| φ |> | θ |) and the roll angle is a negative value will be described in detail. In this case, the enlargement / reduction ratio K is calculated using (Equation 7).

The maximum value K _max and the minimum value K _min of the enlargement / reduction ratio are calculated by (Expression 5) and (Expression 6). As described above, the enlargement / reduction ratio K when the roll angle is larger than the pitch angle (| φ |> | θ |) and the roll angle is a negative value is calculated.

  Next, the case where the roll angle is smaller than the pitch angle (| φ | <| θ |) and the pitch angle is a positive value will be described in detail. In this case, the enlargement / reduction ratio is calculated using (Equation 8). The enlargement / reduction ratio K changes according to the position of the x-axis.

Here, the horizontal resolution of the image data is W, and the position on the x-axis in the image is x. Further, the maximum value K _max and the minimum value K _min of the enlargement / reduction ratio are obtained using (Equation 9) and (Equation 10), respectively, using the pitch angle value.

Here, the maximum value of the pitch angle is θ _max . As the pitch angle increases, the maximum value of the enlargement / reduction ratio increases, the minimum value decreases, and the effect of gloss and shadow control increases. That is, when the pitch angle is a positive value, a pixel whose position on the x-axis is larger than W / 2 is considered to be close to the viewer, that is, enlarged, and the enlargement / reduction ratio K is set to 1 or more. Further, a pixel whose position on the x-axis is smaller than 2 / H is regarded as being away from the viewer, that is, reduced, and the enlargement / reduction ratio is set to 1 or less. As described above, the enlargement / reduction ratio K when the roll angle is smaller than the pitch angle (| φ | <| θ |) and the pitch angle is a positive value is calculated.

  Next, the case where the roll angle is smaller than the pitch angle (| φ | <| θ |) and the pitch angle is a negative value will be described in detail. In this case, the enlargement / reduction ratio is calculated using (Equation 11).

拡大縮小率の最大値Ｋ_ｍａｘと最小値Ｋ_ｍｉｎは、（式９）と（式１０）により算出する。

The maximum value K _max and the minimum value K _min of the enlargement / reduction ratio are calculated by (Expression 9) and (Expression 10).

  As described above, the enlargement / reduction ratio K when the roll angle is smaller than the pitch angle (| φ | <| θ |) and the pitch angle is a negative value is calculated. As described above, the tilt angle is converted into the enlargement / reduction ratio. By tilting the display device in this way, the area that is considered to be closer to the viewer, i.e., enlarged, is considered to suppress gloss / shade and away from the viewer, i.e., reduced. The area can be subjected to suitable gloss and shadow processing by enhancing the gloss / shade.

  The image data generation unit 113 outputs the image data after the gloss / shadow control to the display unit 103 as it is.

  The tilt angle detection unit 1301 in this embodiment acquires the tilt angle by the tilt sensor, but detects the accelerations in the α, β, and γ directions of the image data display device 100 obtained from the acceleration sensor, and uses the detected accelerations. You may calculate by converting into the Euler angle of the image data display apparatus 100. FIG.

  As described above, according to the image data display device 100 including the image data processing device 102, the inclination angle of the display device is converted into an enlargement / reduction ratio, and appropriate gloss and shadow processing is performed according to the converted enlargement / reduction ratio. It can be carried out.

Note that the present invention is not construed as being limited by the above-described embodiments, and various modifications are possible within the scope of the matters described in the claims, and are included in the technical scope of the present invention. It is.
Each component of the present invention can be arbitrarily selected.

  The program that operates in the image data processing apparatus according to the present invention may be a program that controls a CPU or the like (a program that causes a computer to function) so as to realize the functions of the above-described embodiments relating to the present invention. Information handled by these devices is temporarily accumulated in a RAM (Random Access Memory) during the processing, and then stored in various ROMs and HDDs such as a ROM (Read Only Memory), and as required by the CPU. Reading, correction and writing are performed.

  1 is recorded on a computer-readable recording medium, the program recorded on the recording medium is read into a computer system, and executed by a CPU or the like. You may perform the process of. Here, the “computer system” includes hardware such as an OS (Operating System) and peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included.

  Further, part or all of the image data processing apparatus in the above-described embodiments may be realized as an LSI that is typically an integrated circuit. Each functional block of the image data processing apparatus may be individually chipped, or a part or all of them may be integrated into a chip. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to progress in semiconductor technology, an integrated circuit based on the technology can also be used.

  Further, in the above-described embodiments, the control lines and information lines indicate what is considered necessary for the explanation, and do not necessarily indicate all the control lines and information lines on the product. All the components may be connected to each other.

(Appendix)
The present invention includes the following disclosure.
(1)
An enlargement / reduction ratio calculation unit that acquires an enlargement / reduction instruction of image data and calculates an enlargement / reduction ratio of the image data from the enlargement / reduction instruction, and enlarges or reduces a part of the image data of the image data based on the enlargement / reduction ratio An image data processing device comprising:
A subject shape estimation unit that acquires a three-dimensional shape of the subject surface from the image data;
A light source environment estimation unit that acquires incident light information from a light source on the surface of the subject based on the image data and the three-dimensional shape;
A gloss / shadow control unit for controlling a luminance value of at least one of a high luminance region and a low luminance region of the image data based on the three-dimensional shape, the incident light information, and the enlargement / reduction ratio; With
The gloss / shadow control unit
Image data comprising a gloss / shadow luminance correction unit that performs luminance correction that emphasizes or suppresses at least one of a high luminance region and a low luminance region of the image data in accordance with the enlargement / reduction ratio. Processing equipment.
(2)
The gloss / shadow control unit
When the enlargement / reduction ratio is equal to or greater than 1, brightness correction is performed to emphasize the high brightness area and the low brightness area of the image data,
The image data processing apparatus according to (1), wherein when the enlargement / reduction ratio is 1 or less, luminance correction is performed to suppress a high luminance region and a low luminance region of the image data.
Appropriate gloss and shadow control of the image can be performed according to the enlargement / reduction ratio of the image data.

(3)
The gloss / shadow control unit
A normal vector / light source vector inner product calculation unit that receives an output from the scaling factor calculation unit and calculates an inner product of a normal vector and a light source vector;
An inner product result determination unit for obtaining a luminance value based on a calculation result of the normal vector / light source vector inner product calculation unit;
Based on the determination result of the inner product result determination unit, a gloss region / shadow region estimation unit that estimates a gloss region and a shadow region;
The image data according to (1) or (2), further comprising: a glossy shadow region luminance correction unit that receives the output from the enlargement / reduction ratio calculation unit and corrects the luminance of the glossy region / shadow region Processing equipment.
(4)
The image data generation unit
When the enlargement / reduction ratio is 1 or more, the reduction ratio of the entire image data is calculated,
Performing luminance correction that suppresses at least one of the high luminance region and the low luminance region of the image data based on the enlargement / reduction ratio,
The first image data obtained by enlarging the partial image data of the image data after the luminance correction and the luminance correction for enhancing at least one of the high luminance region and the low luminance region of the image data based on the reduction ratio. The image according to any one of (1) to (3), wherein second image data obtained by reducing partial image data of the image data after performing the luminance correction is generated. Data processing device.
Even when the enlargement / reduction ratio is 1 or more, the entire image can be displayed.

(5)
An inclination angle detection unit for detecting the inclination angle of the display device;
The enlargement / reduction ratio calculation unit includes:
The image data processing apparatus according to any one of (1) to (4), wherein an enlargement / reduction ratio of the image data is calculated based on the tilt angle.
Based on the enlargement / reduction ratio calculated based on the inclination of the display device, appropriate gloss and shadow control of the image can be performed.

(6)
An imaging unit for imaging a subject;
An image data display device comprising: the image data processing device according to any one of (1) to (5) that processes image data of the subject imaged by the imaging unit.
(7)
An enlargement / reduction ratio calculation unit that acquires an enlargement / reduction instruction of image data and calculates an enlargement / reduction ratio of the image data from the enlargement / reduction instruction, and enlarges or reduces a part of the image data of the image data based on the enlargement / reduction ratio An image data processing method using an image data processing device including an image data generation unit to reduce,
A subject shape estimation step for obtaining a three-dimensional shape of the subject surface from the image data;
A light source environment estimation step for acquiring incident light information from a light source on the surface of the subject based on the image data and the three-dimensional shape;
Gloss / shadow control step for controlling the brightness value of at least one of the high brightness area and the low brightness area of the image data based on the three-dimensional shape, the incident light information, and the enlargement / reduction ratio; Have
The gloss / shadow control step includes:
Gloss / shadow luminance correction step for performing luminance correction for emphasizing or suppressing at least one of a high luminance region and a low luminance region of the image data according to the enlargement / reduction ratio. Processing method.
(8)
A program for causing a computer to execute the image data processing method according to (7).

DESCRIPTION OF SYMBOLS A ... Image data processing apparatus, 100 ... Image data display apparatus, 101 ... Input / output device, 102 ... Image data processing apparatus, 103 ... Display part, 104 ... Imaging part, 105 ... Transmission / reception part, 106 ... Image data input / output part DESCRIPTION OF SYMBOLS 107 ... Memory | storage part, 108 ... Input / output part (operation part etc.), 109 ... Subject shape estimation part, 110 ... Light source environment estimation part, 111 ... Enlargement / reduction rate calculation part, 112 ... Gloss / shadow control part, 113 ... Image data Generation unit 104 ... Imaging unit 105 ... Transmission / reception unit 106 ... Image data input / output unit 107 ... Storage unit 108 ... Input / output unit 109 ... Subject shape estimation unit 110 ... Light source environment estimation unit 111 ... Enlargement / reduction Rate calculation unit 112 ... Gloss / shadow control unit 112-1 Normal vector / light source vector inner product calculation unit 112-2 ... Inner product result determination unit 112-3 ... Gloss area / shadow area estimation unit 12-4 ... glossy shadow area luminance correction unit, 113 ... image data generation unit, 1301 ... inclination angle detection unit.

  The present invention can be used for an image processing apparatus.

Claims (5)

An enlargement / reduction ratio calculation unit that acquires an enlargement / reduction instruction of image data and calculates an enlargement / reduction ratio of the image data from the enlargement / reduction instruction, and enlarges or reduces a part of the image data of the image data based on the enlargement / reduction ratio An image data processing device comprising:
A subject shape estimation unit that acquires a three-dimensional shape of the subject surface from the image data;
A light source environment estimation unit that acquires incident light information from a light source on the surface of the subject based on the image data and the three-dimensional shape;
A gloss / shadow control unit for controlling a luminance value of at least one of a high luminance region and a low luminance region of the image data based on the three-dimensional shape, the incident light information, and the enlargement / reduction ratio; With
The gloss / shadow control unit
Image data comprising a gloss / shadow luminance correction unit that performs luminance correction that emphasizes or suppresses at least one of a high luminance region and a low luminance region of the image data in accordance with the enlargement / reduction ratio. Processing equipment. The gloss / shadow control unit
When the enlargement / reduction ratio is equal to or greater than 1, brightness correction is performed to emphasize the high brightness area and the low brightness area of the image data,
The image data processing apparatus according to claim 1, wherein when the enlargement / reduction ratio is 1 or less, luminance correction is performed to suppress a high luminance region and a low luminance region of the image data. The gloss / shadow control unit
A normal vector / light source vector inner product calculation unit that receives an output from the scaling factor calculation unit and calculates an inner product of a normal vector and a light source vector;
An inner product result determination unit for obtaining a luminance value based on a calculation result of the normal vector / light source vector inner product calculation unit;
Based on the determination result of the inner product result determination unit, a gloss region / shadow region estimation unit that estimates a gloss region and a shadow region;
3. The image data processing apparatus according to claim 1, further comprising: a gloss / shadow region luminance correction unit that receives the output from the enlargement / reduction ratio calculation unit and corrects the luminance of the gloss region / shadow region. . The image data generation unit
When the enlargement / reduction ratio is 1 or more, the reduction ratio of the entire image data is calculated,
Performing luminance correction that suppresses at least one of the high luminance region and the low luminance region of the image data based on the enlargement / reduction ratio,
The first image data obtained by enlarging the partial image data of the image data after the luminance correction and the luminance correction for enhancing at least one of the high luminance region and the low luminance region of the image data based on the reduction ratio. 4. The image data according to claim 1, wherein the second image data is generated by reducing a part of the image data of the image data after the brightness correction. 5. Processing equipment. An inclination angle detection unit for detecting the inclination angle of the display device;
The enlargement / reduction ratio calculation unit includes:
5. The image data processing apparatus according to claim 1, wherein an enlargement / reduction ratio of the image data is calculated based on the tilt angle. 6.

Images