WO2016174915A1

WO2016174915A1 - Image processing device, image processing method, and program

Info

Publication number: WO2016174915A1
Application number: PCT/JP2016/056191
Authority: WO
Inventors: 文香中谷; 康孝平澤; 雄飛近藤; 穎陸
Original assignee: ソニー株式会社
Priority date: 2015-04-30
Filing date: 2016-03-01
Publication date: 2016-11-03
Also published as: US20180107108A1; CN107533370A; EP3291052A1; ES2929648T3; CN107533370B; JP6693514B2; US10444617B2; JPWO2016174915A1; EP3291052B1; EP3291052A4

Abstract

A polarized light image acquisition unit 20 acquires a plurality of polarized light images having different polarization directions. A polarized light image is an image in which, for example, a pointing body in a user interface is imaged as a subject to be recognized. A normal computation unit 30 computes a normal for each pixel of the subject to be recognized on the basis of the polarized light images which are acquired by the polarized light image acquisition unit 20. The normal is information according to the three-dimensional shape of the subject to be recognized. Using the normals which are computed with the normal computation unit 30, a recognition unit 40 carries out a recognition of the subject, assesses the type, position, orientation, etc., of the pointing body, and outputs the result of the assessment as input information in the user interface. Thus, it is possible to easily carry out the recognition of the subject with good precision.

Description

Image processing apparatus, image processing method, and program

This technology makes it possible to recognize a subject accurately and easily with respect to an image processing apparatus, an image processing method, and a program.

Conventionally, the normal line of a subject is calculated from polarized images of a plurality of polarization directions. For example, in Non-Patent Document 1 and Non-Patent Document 2, normals are calculated by applying polarization images of a plurality of polarization directions to a model equation.

Also, the subject is recognized using the polarization image. For example, Patent Literature 1 illuminates a subject by arranging illumination means so that illumination light is p-polarized with respect to a predetermined reference plane. Further, Patent Document 1 separates reflected light from a reference surface into s-polarized light and p-polarized light, measures the light intensity for each polarization component, and obtains light obtained while moving the object to be measured along the reference surface. Based on the intensity measurement result, the subject is identified.

Furthermore, the normal is used for the user interface. For example, Patent Document 2 detects movements of an object corresponding to a hand and a forearm and recognizes a movement close to a natural movement of a person when turning a page. The normal of the target surface corresponding to the palm is used. In addition, the object is detected by extracting the distribution of the distance and the contour in the distance image.

JP2011-150689A JP 2012-242901 A

By the way, when the normal is calculated by applying a polarization image of a plurality of polarization directions to the model formula, the relationship between the polarization direction and the luminance of the polarization image has a periodicity of 180 degrees, and the orientation in the normal direction When obtaining the angle, the so-called 180 degree indefiniteness problem remains. Further, in the identification using s-polarized light and p-polarized light, the difference in the surface material of the subject can be identified, but in principle, the three-dimensional shape of the subject cannot be identified from the two polarization directions. In addition, when detecting an object and calculating a normal based on a distance image, an image with a high distance resolution is required to accurately detect the object and calculate the normal. It is not easy to acquire an image with high resolution.

Accordingly, an object of this technique is to provide an image processing apparatus, an image processing method, and a program that can easily and accurately recognize a subject.

The first aspect of this technology is
A polarization image acquisition unit that acquires a plurality of polarization images having different polarization directions in which a subject to be recognized is imaged;
A normal calculation unit that calculates a normal for each pixel based on the polarization image acquired by the polarization image acquisition unit;
The image processing apparatus includes a recognition unit that recognizes the subject using the normal calculated by the normal calculation unit.

In this technique, for example, a plurality of polarized images with different polarization directions captured using an input indicator in a user interface as a subject to be recognized are acquired by the polarization image acquisition unit. The normal calculation unit calculates a normal for each pixel based on the acquired polarization image. For example, the normal calculation unit performs temporary recognition of a subject using a temporary recognition processing image generated from a plurality of polarization images. In addition, normals are calculated from a plurality of polarization images, and the indeterminacy of the calculated normals is resolved based on the provisional recognition result. In the temporary recognition, the model that is closest to the subject using the image for the temporary recognition processing and the model image registered in advance is used as the subject's temporary recognition result, and the indefiniteness of the normal line is eliminated based on the temporarily recognized model. To do. When the subject to be recognized is a hand, the normal calculation unit temporarily recognizes the positions of the fingertip and finger pad using the temporary recognition processing image and the model image registered in advance, and the temporary recognition result Based on the above, the indefiniteness of the normal of the finger region in the hand is eliminated. The recognizing unit determines the pointing direction based on the normal of the finger region in which the ambiguity is eliminated. In addition, the normal line calculation unit temporarily recognizes the position and skeleton structure of the hand region using the temporary recognition processing image and the model image registered in advance, and calculates the hand region based on the temporary recognition result. Line indeterminacy may be eliminated. In this case, the recognizing unit determines the hand shape based on the normal of the hand region in which the ambiguity is eliminated. Further, when the subject to be recognized is a face, the normal calculation unit tentatively recognizes the position of the face region using the temporary recognition processing image and the model image registered in advance, and based on the temporary recognition result. To eliminate the indefiniteness of the normal of the face. The recognizing unit determines a face shape or a facial expression based on the normal of the face area in which the ambiguity is eliminated.

In addition, the recognizing unit determines the distribution of the teacher data corresponding to the learning subject, for example, the normal line of the learning subject from the normals of the learning subject calculated based on a plurality of polarization images having different polarization directions obtained by imaging the learning subject. Are stored in the teacher database as teacher data. The recognizing unit uses the distribution of normals calculated based on a plurality of polarized images with different polarization directions obtained by imaging the subject to be recognized as student data, and student data based on student data and teacher data stored in the teacher database unit. The learning subject corresponding to the teacher data that is most similar to may be used as the recognition result of the subject to be recognized.

According to a second aspect of the present technology, the polarization image acquisition unit acquires a plurality of polarization images having different polarization directions in which the recognition subject is imaged, and
In the normal calculation unit, based on the polarization image acquired by the polarization image acquisition unit, calculating a normal for each pixel;
In the image processing method, the recognition unit may recognize the subject using the normal calculated by the normal calculation unit.

The third aspect of this technique is a procedure for acquiring a plurality of polarization images having different polarization directions in which a subject to be recognized is imaged,
A procedure for calculating a normal for each pixel based on the acquired polarization image;
A program for causing a computer to execute a procedure for recognizing the subject using the calculated normal line.

Note that the program of the present technology is, for example, a storage medium or a communication medium provided in a computer-readable format to a general-purpose computer that can execute various program codes, such as an optical disk, a magnetic disk, or a semiconductor memory. It is a program that can be provided by a medium or a communication medium such as a network. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer.

According to this technique, a plurality of polarization images having different polarization directions in which a subject to be recognized is imaged are acquired, and a normal is calculated for each pixel based on the acquired polarization image. Subject recognition is performed using the normal. For this reason, it is possible to easily and accurately recognize the subject. Note that the effects described in the present specification are merely examples and are not limited, and may have additional effects.

It is a figure which shows the basic composition of an image processing apparatus. 3 is a flowchart showing a basic operation of the image processing apparatus. It is a figure which shows the structure of 1st Embodiment of an image processing apparatus. It is the figure which illustrated the structure in the case of producing | generating a polarization image in a polarization image acquisition part. It is a figure for demonstrating a to-be-photographed object's shape and a polarization image. It is the figure which illustrated the relationship between a brightness | luminance and a polarization angle. It is a figure which shows the relationship between a polarization degree and a zenith angle. It is a figure for demonstrating indefiniteness of 180 degree | times. It is a flowchart which shows operation | movement of 1st Embodiment. It is a flowchart which shows operation | movement of the 1st specific example in 1st Embodiment. It is the figure which illustrated the detection result of the outline of the hand. It is a figure which shows the normal line of a finger pad area | region. It is a figure which shows the pointing direction. It is a figure which shows a gaze direction. It is a flowchart which shows operation | movement of the 2nd specific example in 1st Embodiment. It is a figure for demonstrating discrimination | determination of the three-dimensional shape and face direction of a face. It is a flowchart which shows operation | movement of the 3rd specific example in 1st Embodiment. It is a figure for demonstrating operation | movement in the case of detecting the skeleton of a hand. It is the figure which illustrated the structure of 2nd Embodiment of an image processing apparatus. It is a flowchart which shows learning operation | movement. It is a flowchart which shows the recognition operation | movement using a learning result. The operation | movement of the specific example in 2nd Embodiment is shown. It is the figure which illustrated the user interface used as application object.

Hereinafter, embodiments for carrying out the present technology will be described. The description will be given in the following order.
1. 1. Basic configuration and basic operation of image processing apparatus First embodiment 2-1. Configuration of first embodiment 2-2. Operation of first embodiment 2-3. First specific example in the first embodiment 2-4. Second specific example in the first embodiment 2-5. 3. Third specific example in the first embodiment Second embodiment 3-1. Configuration of second embodiment 3-2. Operation of second embodiment 3-3. Specific example in the second embodiment

<1. Basic Configuration and Basic Operation of Image Processing Device>
FIG. 1 shows a basic configuration of the image processing apparatus. The image processing apparatus 10 includes a polarization image acquisition unit 20, a normal calculation unit 30, and a recognition unit 40.

The polarization image acquisition unit 20 acquires a plurality of polarization images with different polarization directions in which the subject to be recognized is captured, for example, polarization images with three or more polarization directions. The polarization image acquisition unit 20 may include an imaging unit that generates a polarization image with three or more polarization directions, and a configuration that acquires a polarization image with three or more polarization directions from an external device or a recording medium. It may be.

The normal calculation unit 30 calculates the normal of the subject to be recognized based on the polarization image acquired by the polarization image acquisition unit 20. The normal line calculation unit 30 calculates a normal line by applying a plurality of polarization images having different polarization directions acquired by the polarization image acquisition unit 20 to the model formula. In addition, the normal line calculation unit 30 may perform processing for eliminating indefiniteness with respect to the normal line calculated for each pixel, for example.

The recognition unit 40 performs a process of recognizing the subject to be recognized based on the normal calculated by the normal calculation unit 30. For example, when the subject to be recognized is an input indicator in the user interface, the recognition unit 40 recognizes the type, position, orientation, and the like of the subject and outputs the recognition result as input information in the user interface.

FIG. 2 is a flowchart showing the basic operation of the image processing apparatus. In step ST1, the image processing apparatus acquires a polarization image. The polarization image acquisition unit 20 of the image processing apparatus 10 acquires a plurality of polarization images having different polarization directions in which the recognition target subject is imaged, and proceeds to step ST2. In step ST2, the image processing apparatus calculates a normal line. The normal calculation unit 30 of the image processing apparatus 10 calculates a normal for each pixel based on the polarization image acquired in step ST1, and proceeds to step ST3. In step ST3, the image processing apparatus outputs a recognition result. The recognition unit 40 of the image processing apparatus 10 performs recognition processing on the subject to be recognized based on the normal calculated in step ST2, and outputs a recognition result.

<2. First Embodiment>
<2-1. Configuration of First Embodiment>
FIG. 3 shows the configuration of the first embodiment of the image processing apparatus. The image processing apparatus 11 includes a polarization image acquisition unit 20, a normal calculation unit 31, and a user interface (UI) processing unit 41.

The polarization image acquisition unit 20 acquires a plurality of polarization images having different polarization directions. FIG. 4 illustrates a configuration when a polarization image is generated by the polarization image acquisition unit. For example, as illustrated in FIG. 4A, the polarization image acquisition unit 20 generates the image sensor 201 by arranging the polarization filter 202 having a pixel configuration of a plurality of polarization directions to perform imaging. FIG. 4A illustrates a case where a polarizing filter 202 serving as one of four different polarization directions (polarization directions are indicated by arrows) of each pixel is arranged on the front surface of the image sensor 201. ing. In addition, as illustrated in FIG. 4B, the polarization image acquisition unit 20 may generate a plurality of polarization images having different polarization directions using a multi-lens array configuration. For example, a plurality of lenses 203 (four in the figure) are provided on the front surface of the image sensor 201, and an optical image of a subject is formed on the imaging surface of the image sensor 201 by each lens 203. In addition, a polarizing plate 204 is provided on the front surface of each lens 203, and a plurality of polarization images having different polarization directions are generated with the polarization direction of the polarizing plate 204 being different. If the polarization image acquisition unit 20 is configured in this way, a plurality of polarization images can be acquired by one imaging, and thus a recognition target subject can be quickly recognized. Also, as shown in FIG. 4C, a configuration in which polarizing plates 212-1 to 212-4 having different polarization directions are provided in front of the imaging units 210-1 to 210-4, from a plurality of different viewpoints. A plurality of polarization images having different polarization directions may be generated.

In the case where the movement of the recognition target subject is slow or the recognition target subject operates stepwise, a configuration in which a polarizing plate 211 is provided in front of the imaging unit 210 as shown in FIG. It is good. In this case, the polarizing plate 211 is rotated to capture images in a plurality of different polarization directions, and a plurality of polarization images having different polarization directions are acquired.

When the image sensor 201 does not use a color filter, the polarization image acquisition unit 20 can acquire a luminance polarization image. Here, in the case of FIG. 4A, an image equivalent to a non-polarized normal luminance image can be acquired by averaging the luminances of four adjacent pixels in different directions of polarization. . 4B and 4C, the polarization direction is different if the distance between the lenses 203 and the imaging units 210-1 to 210-4 is short enough to be ignored with respect to the distance to the subject. Parallax can be ignored for multiple polarization images. Therefore, by averaging the luminance of polarized images having different polarization directions, an image equivalent to a non-polarized normal luminance image can be obtained. In addition, when parallax cannot be ignored, if polarized images with different polarization directions are aligned according to the amount of parallax and the luminance of the polarized image after alignment is averaged, it is equivalent to a non-polarized normal luminance image Images can be acquired. In the case of FIG. 4D, an image equivalent to a normal luminance image that is non-polarized can be acquired by averaging the luminances of luminance-polarized images having different polarization directions for each pixel.

Furthermore, the polarization image acquisition unit 20 may generate not only the luminance polarization image but also the three primary color images at the same time by providing the image sensor 201 with a color filter, and may simultaneously generate the infrared image and the like. Further, the polarization image acquisition unit 20 may generate a luminance image by calculating the luminance from the three primary color images.

The normal calculation unit 31 calculates a normal from which indefiniteness has been eliminated from a plurality of polarization images acquired by the polarization image acquisition unit 20 and auxiliary information such as various models. The normal line calculation unit 31 includes, for example, a polarization processing unit 301, a provisional recognition processing image generation unit 302, a provisional recognition processing unit 303, a model database unit 304, and an indeterminacy elimination unit 305.

The polarization processing unit 301 calculates a normal line from the polarization image and outputs it to the indeterminacy eliminating unit 305. Here, the shape of the subject and the polarization image will be described with reference to FIG. For example, as shown in FIG. 5, the light source LT is used to illuminate the subject OB, and the imaging unit DC images the subject OB via the polarizing plate PL. In this case, in the captured image, the luminance of the subject OB changes according to the polarization direction of the polarizing plate PL. For ease of explanation, a plurality of polarization images are acquired by rotating the polarizing plate PL, for example, and the highest luminance is Imax and the lowest luminance is Imin. Further, when the x-axis and the y-axis in the two-dimensional coordinates are on the plane of the polarizing plate PL, the angle in the y-axis direction with respect to the x-axis when the polarizing plate PL is rotated is a polarization angle υ.

The polarizing plate PL returns to the original polarization state when rotated 180 degrees and has a period of 180 degrees. The polarization angle υ when the maximum luminance Imax is observed is defined as an azimuth angle φ. When such a definition is performed, the luminance I observed when the polarizing plate PL is rotated can be expressed as in Expression (1). FIG. 6 illustrates the relationship between the luminance and the polarization angle. This example shows a diffuse reflection model. In the case of specular reflection, the azimuth angle is shifted by 90 degrees from the polarization angle.

In Equation (1), the polarization angle υ is clear when the polarization image is generated, and the maximum luminance Imax, the minimum luminance Imin, and the azimuth angle φ are variables. Therefore, by performing the fitting to the model equation shown in the equation (1) using the luminance of the polarization image having three or more polarization directions, the polarization having the maximum luminance based on the model equation indicating the relationship between the luminance and the polarization angle. The azimuth angle φ, which is an angle, can be determined.

Also, the normal of the object surface is expressed in a polar coordinate system, and the normal is defined as an azimuth angle φ and a zenith angle θ. The zenith angle θ is an angle from the z axis toward the normal, and the azimuth angle φ is an angle in the y axis direction with respect to the x axis as described above. Here, when the minimum luminance Imin and the maximum luminance Imax are obtained when the polarizing plate PL is rotated, the degree of polarization ρ can be calculated based on the equation (2).

It is known that the relationship between the degree of polarization ρ and the zenith angle θ has the characteristics shown in FIG. Therefore, the zenith angle θ can be determined from the characteristics shown in FIG. The characteristics shown in FIG. 7 are merely examples, and the characteristics change depending on the refractive index of the subject. For example, the degree of polarization increases as the refractive index increases.

The normal calculated in this way has an indefiniteness of 180 degrees. FIG. 8 is a diagram for explaining the indefiniteness of 180 degrees. When the normal is calculated by imaging the subject OB shown in FIG. 8A by the imaging unit DC, the luminance change according to the rotation of the polarization direction has a cycle of 180 degrees. Therefore, for example, as shown in FIG. 8B, the normal direction (indicated by an arrow) is correct in the upper half area GA of the subject OB, and the normal direction is reverse in the lower half area GB. There is.

The temporary recognition processing image generation unit 302 generates a temporary recognition processing image based on the plurality of polarization images acquired by the polarization image acquisition unit 20. The temporary recognition processing image generation unit 302 calculates the average of a plurality of polarized images, for example, thereby obtaining a temporary recognition process equivalent to a captured image (normal image) obtained by imaging without using a polarizing plate or a polarizing filter. An image is generated. In addition, the temporary recognition processing image generation unit 302 may extract a polarization image of one polarization direction from a plurality of polarization images to obtain a temporary recognition processing image. Further, the temporary recognition processing image generation unit 302 may use the acquired plurality of polarization images as the temporary recognition processing image. Further, the temporary recognition processing image generation unit 302 may use a normal image and a polarization image as the temporary recognition processing image. The temporary recognition processing image generation unit 302 outputs the temporary recognition processing image to the temporary recognition processing unit 303.

The temporary recognition processing unit 303 performs temporary recognition of the subject to be recognized using the temporary recognition processing image generated by the temporary recognition processing image generation unit 302. The temporary recognition processing unit 303 performs subject recognition using the temporary recognition processing image, and determines the type, position, posture, and the like of the subject to be recognized. The temporary recognition processing unit 303 uses, for example, a temporary recognition processing image and images of various object models (a normal image and a polarization image) stored in advance in the model database unit 304, and approximates the subject to be recognized most. Determine the model. In addition, when the temporary recognition processing image generated by the temporary recognition processing image generation unit 302 includes a polarization image, the temporary recognition processing unit 303 determines a model that is closest to the object to be recognized in consideration of polarization characteristics. To do. The temporary recognition processing unit 303 outputs the determined model to the indeterminacy eliminating unit 305 as a temporary recognition result. The temporary recognition processing unit 303 may perform temporary recognition of the subject to be recognized using not only model fitting but also other object recognition methods.

The indeterminacy canceling unit 305 cancels the normal indefiniteness calculated by the polarization processing unit 301 based on the temporary recognition result supplied from the temporary recognition processing unit 303. The temporary recognition result has information indicating the type, position, posture, and the like of the subject to be recognized as described above. Therefore, the ambiguity eliminating unit 305 eliminates the ambiguity of the normal from the normal direction having a phase difference of 180 degrees in the normal based on the model indicated by the provisional recognition result. In other words, the ambiguity elimination unit 305 eliminates the ambiguity of the normal line by specifying the normal direction so as to correspond to the shape of the subject to be recognized, and the normal processing unit 41 eliminates the ambiguity. Output to.

The UI processing unit 41 recognizes a subject using the normal generated by the normal calculation unit 31 from which the ambiguity is eliminated. The UI processing unit 41 uses the subject to be recognized in the polarization image acquired by the polarization image acquisition unit 20 as an input indicator in the user interface. The UI processing unit 41 performs subject recognition based on the normal calculated by the normal calculation unit 31 and generates input information (hereinafter referred to as “UI information”) in the user interface. As described above, the normal is information indicating the three-dimensional shape of the subject to be recognized, and the UI processing unit 41 performs subject recognition, recognizes the type, position, orientation, and the like of the subject to be recognized, and the recognition result. Is output as UI information.

<2-2. Operation of First Embodiment>
FIG. 9 is a flowchart showing the operation of the first embodiment. In step ST11, the polarization image acquisition unit 20 acquires a polarization image. The polarization image acquisition unit 20 performs imaging using a polarizing plate or a polarization filter, acquires a plurality of polarization images having different polarization directions, and proceeds to steps ST12 and ST13.

In step ST12, the normal calculation unit 31 calculates a normal. The normal calculation unit 31 performs fitting to a model equation using pixel values of a plurality of polarization images having different polarization directions for each pixel of the polarization image, and calculates a normal based on the model equation after fitting. Proceed to step ST15.

In step ST13, the normal calculation unit 31 generates a temporary recognition processing image. For example, the normal line calculation unit 31 averages pixel values of a plurality of polarization images having different polarization directions generated in step ST11 for each pixel of the polarization image, and calculates the average value as a temporary recognition processing image (equivalent to a normal image). ), The process proceeds to step ST14.

In step ST14, the normal calculation unit 31 performs provisional recognition processing. The normal line calculation unit 31 performs fitting using, for example, a temporary recognition processing image and a model stored in advance, recognizes the type, position, orientation, and the like of the subject to be recognized, and proceeds to step ST15.

In step ST15, the normal calculation unit 31 eliminates the indefiniteness of the normal. The normal line calculation unit 31 determines the indeterminacy from the normal line calculated in step ST12, that is, the normal line having indefiniteness of 180 degrees, based on the temporary recognition result in step ST14, that is, the type, position, posture, etc. of the subject to be recognized. It cancels and it progresses to step ST16.

In step ST16, the UI processing unit 41 generates UI information. The UI processing unit 41 recognizes the type, position, orientation, and the like of the subject to be recognized based on the normal line in which the indefiniteness is eliminated, and sets the recognition result as UI information.

<2-3. First Specific Example in First Embodiment>
Next, a first specific example in the first embodiment will be described. In the first specific example, the subject to be recognized is a hand, and the temporary recognition processing unit recognizes the positions of the fingertip and finger pad using the temporary recognition processing image and a pre-registered model image. . Further, the indeterminacy eliminating unit eliminates the indefiniteness of the normal of the finger region in the hand based on the positions of the fingertip and the finger pad temporarily recognized by the temporary recognition processing unit, and the incognition unit eliminates the indeterminacy. The pointing direction is determined based on the normal of the finger area.

FIG. 10 is a flowchart showing the operation of the first specific example in the first embodiment. In step ST21, the polarization image acquisition unit 20 acquires a polarization image. The polarization image acquisition unit 20 performs imaging of a pointing hand using a polarizing plate or a polarizing filter. Further, the polarization image acquisition unit 20 acquires a plurality of polarization images having different polarization directions, and proceeds to steps ST22 and ST23.

In step ST22, the normal calculation unit 31 calculates a normal. For each pixel of the polarization image, the normal line calculation unit 31 fits the pixel values of a plurality of polarization images having different polarization directions to the model equation, calculates the normal line based on the model equation after the fitting, and proceeds to step ST25. move on.

In step ST23, the normal calculation unit 31 generates a temporary recognition processing image. For example, the normal line calculation unit 31 averages the pixel values of a plurality of polarization images having different polarization directions generated in step ST21 for each pixel of the polarization image, and calculates the average value as a temporary recognition processing image (equivalent to a normal image). ), The process proceeds to step ST24.

In step ST24, the normal calculation unit 31 detects the fingertip / fingerpad position. The normal line calculation unit 31 detects the fingertip region and the finger pad region from the temporary recognition processing image using an image recognition technique. The normal calculation unit 31 is disclosed in, for example, documents “SKKang, MYNam, and PKRhee,“ Color Based Hand and Finger Detection Technology for User Interaction. ”International Conference on Convergence and Hybrid Information Technology, pp 229-236, 2008. As described, a skin color image region is extracted from the color temporary recognition processing image, and edge detection is performed in the peripheral region of the extracted image region to detect the contour of the hand. FIG. 11 illustrates the detection result of the hand outline, and the image area within the hand outline is the hand area ARh.

Next, the normal calculation unit 31 detects a finger region from the hand region. The normal line calculation unit 31 performs a morphological open process on the hand region. In the morphological open process, small objects in the image are removed first, and large objects retain their shape and size. That is, when the morphological open process is performed on the hand area, the finger area that is thinner than the palm is removed first, so if the difference between the image before the morphological open process and the image after the process is calculated, the finger area is calculated. It becomes possible to detect.

Also, other methods may be used for detecting the finger area. For example, assuming that the external shape of the finger is composed of a plurality of convex hulls, it is possible to detect the convex hull from the hand region and determine the finger region based on the detection result. The convex hull can be easily detected by using, for example, a method for obtaining the convex hull prepared in OpenCV. Therefore, the finger area ARf can be detected as shown in FIG.

The normal calculation unit 31 separates the detected finger area into a fingertip area and a finger pad area. As shown in FIG. 11, the normal line calculation unit 31 sets a difference area between the hand area ARh and the detected finger area ARf as a fist area ARg. Further, in the finger area ARf, the normal line calculation unit 31 sets the area farthest from the center of gravity BG of the fist area ARg as the fingertip area ARfs and the other area as the finger pad area ARft.

In step ST25, the normal calculation unit 31 eliminates the indefiniteness of the normal. The normal calculation unit 31 regards the appearance shape of the finger as a convex shape, and determines the normal of the normal corresponding to the finger region from the normal calculated in step ST22, that is, the normal having an indefiniteness of 180 degrees. Is canceled based on the detection result of step ST24, and the process proceeds to step ST26. 12 illustrates the normal line of the finger pad area, FIG. 12 (a) is the normal line of the finger pad area ARft before the indefiniteness is resolved, and FIG. The normal line of the finger pad area ARft after the erasure is eliminated is shown.

In step ST26, the UI processing unit 41 determines the pointing direction. The UI processing unit 41 determines the direction in which the fingertip is facing, that is, the pointing direction based on the normal of the finger region in which the indefiniteness is eliminated, and sets the determination result as UI information.

The pointing direction is a direction fs substantially perpendicular to the normal of the finger pad area ARft as shown in FIG. Therefore, for example, when the user performs a pointing operation toward the direction of the polarization image acquisition unit 20 and determines the pointing operation, the UI processing unit 41 is in a direction orthogonal to the normal of the finger pad region, A direction FP facing the front side from the back side of the image is taken as a pointing direction.

Assuming that the vector p in the pointing direction is Equation (3) and the vector n in the normal direction in the finger pad region is Equation (4), as shown in Equation (5), the inner product of the vector p and the vector n is “0”. Become. Therefore, the UI processing unit 41 sets a vector p that is expressed by the equation (7) for a plurality of position (for example, k positions) normals in the finger pad region, that is, the normal N shown in the equation (6), by Obtained under constraints.

Here, when the function W is defined as shown in Expression (9), the function that is minimized in Expression (7) can be defined as Function C as shown in Expression (10). That is, a vector p that minimizes the function C may be obtained under the constraint condition of Expression (8). The Lagrangian undetermined multiplier method is used to calculate the vector p that minimizes the function C. When the Lagrange multiplier method is used, Equation (10) can be expressed as Equation (11) using Lagrange multiplier λ. Therefore, if a vector p satisfying the eigen equation shown in Expression (12) is obtained, the vector p satisfying Expression (7) can be calculated. The vector p satisfying the eigen equation is established when the vector p is an eigen vector of the function W, and the value of λ corresponds to the eigen value. At this time, if the eigenvector of the function W is substituted into the vector p, the value of the function to be minimized is “C = λ”. Therefore, the vector p satisfying Expression (7) is the eigenvector corresponding to the minimum eigenvalue of the function W. It becomes.

If such processing is performed, the pointing direction can be determined with higher accuracy than in the case of using a distance image. Further, in the first specific example, the case where the pointing direction is determined is illustrated, but as shown in FIG. 14, the position of the pupil EP is detected, and how much the position of the pupil EP moves in any direction from the center. The line-of-sight direction FP can also be determined depending on whether or not it is.

<2-4. Second Specific Example in First Embodiment>
Next, a second specific example in the first embodiment will be described. In the second specific example, the subject to be recognized is a face, and the temporary recognition processing unit recognizes the position of the face area using the temporary recognition processing image and the model image registered in advance. Further, the indeterminacy eliminating unit eliminates the indeterminacy of the normal of the face based on the position of the face area temporarily recognized by the temporary recognition processing unit, and the recognizing unit corrects the face area in which the indeterminacy is eliminated. The face shape or expression is determined based on the line.

FIG. 15 is a flowchart showing the operation of the second specific example in the first embodiment. In step ST31, the polarization image acquisition unit 20 acquires a polarization image. The polarization image acquisition unit 20 captures a face using a polarizing plate or a polarization filter. Further, the polarization image acquisition unit 20 provides a color filter in the imaging unit, acquires a plurality of color polarization images having different polarization directions, and proceeds to steps ST32 and ST33.

In step ST32, the normal calculation unit 31 calculates a normal. For each pixel of the polarization image, the normal line calculation unit fits the pixel values of a plurality of polarization images having different polarization directions to the model equation, calculates the normal line based on the model equation after the fitting, and proceeds to step ST35. .

In step ST33, the normal calculation unit 31 generates a temporary recognition processing image. For example, the normal line calculation unit 31 averages pixel values of a plurality of polarization images having different polarization directions generated in step ST31 for each pixel of the polarization image, and calculates the average value as a temporary recognition processing image (equivalent to a normal image). ), The process proceeds to step ST34.

In step ST34, the normal line calculation unit 31 detects face recognition and feature points of the recognized face. The normal line calculation unit 31 detects the position of the face from the temporary recognition processing image using a face recognition technique. Further, the normal calculation unit 31 is, for example, a document “TFCootes, CJTaylor, DHCooper, and J.Graham:“ Active Shape Models-Their Training and Application ”, Computer Vision and Image Understanding, Vol.16, No.1. , “January,” pp.38-59, “1995”, and detects the feature points of the face. Active Shape Model can automatically detect feature points that determine the posture of the recognition target in the image. Specifically, first, a plurality of learning images in which feature points are manually arranged are prepared, and an intermediate shape to be recognized is generated from these images. Next, the object to be recognized is searched for by changing the position of the intermediate shape with respect to the image to be recognized. At that time, template matching is performed by looking at the luminance change around the feature point in the intermediate shape. These searches are repeated at each resolution of the image pyramid from coarse to high resolution. The normal calculation unit 31 performs such processing, detects the recognized facial feature points, and proceeds to step ST35.

In step ST35, the normal calculation part 31 eliminates the indefiniteness of the normal. The normal calculation unit 31 determines the three-dimensional shape of the face and the orientation of the face from the positional relationship between the feature points of the face detected in step ST34 and the feature points of the three-dimensional model stored in advance. As shown in FIG. 16, the normal line calculation unit 31 determines the three-dimensional shape of the face and the orientation of the face from the positional relationship between the face OBf and the feature points (for example, eyes, nose, mouth, etc.) of the three-dimensional model ML. Further, based on the determined three-dimensional shape of the face and the orientation of the face, the normal calculation unit 31 calculates the normal corresponding to the face region from the normal calculated in step ST32, that is, the normal having indefiniteness of 180 degrees. Is resolved based on the detection result of step ST34, and the process proceeds to step ST36.

In step ST36, the UI processing unit 41 determines the face shape and expression. The UI processing unit 41 discriminates a detailed face shape and expression based on the normal of the face area in which the indefiniteness is eliminated, and uses the discrimination result as UI information. For example, the UI processing unit 41 integrates the normal of the face area in which the indefiniteness is eliminated, and determines the facial expression from the detailed face shape or the detailed face shape. It should be noted that distance image information, a three-dimensional model, or the like may be used for discrimination of the face shape and expression.

If such processing is performed, the face shape and facial expression can be discriminated more accurately than when the distance image is used. In the second specific example, face recognition is illustrated. However, if a three-dimensional model of a known object is prepared, the known object can be recognized.

<2-5. Third Specific Example in First Embodiment>
Next, a third specific example in the first embodiment will be described. In the third specific example, the subject to be recognized is a hand, and the temporary recognition processing unit recognizes the position of the hand region and the skeleton structure using the temporary recognition processing image and the model image registered in advance. . In addition, the indeterminacy elimination unit eliminates indeterminacy in the normal of the hand region based on the position and skeleton structure of the hand region temporarily recognized by the temporary recognition processing unit, and the recognition unit eliminates indeterminacy. The hand shape is determined based on the normal of the hand region.

FIG. 17 is a flowchart showing the operation of the third specific example in the first embodiment. In step ST41, the polarization image acquisition unit 20 acquires a polarization image. The polarization image acquisition unit 20 performs hand imaging using a polarizing plate or a polarization filter. Further, the polarization image acquisition unit 20 provides a color filter in the imaging unit, acquires a plurality of color polarization images having different polarization directions, and proceeds to steps ST42 and 43.

In step ST42, the normal calculation unit 31 calculates a normal. For each pixel of the polarization image, the normal line calculation unit 31 fits the pixel values of a plurality of polarization images having different polarization directions to the model equation, calculates the normal line based on the model equation after the fitting, and proceeds to step ST45. move on.

In step ST43, the normal calculation unit 31 generates a temporary recognition processing image. For example, the normal calculation unit 31 averages the pixel values of a plurality of polarization images having different polarization directions generated in step ST41 for each pixel of the polarization image, and calculates the average value as a temporary recognition processing image (equivalent to a normal image). ), The process proceeds to step ST44.

In step ST44, the normal calculation unit 31 detects the position / posture of the hand. The normal calculation unit 31 performs the same processing as step ST24, detects the fist or palm region and the fingertip region, and detects the skeleton of the hand by connecting the center of gravity of the fist or palm region and the fingertip. FIG. 18 is a diagram for explaining an operation when detecting a skeleton of a hand. The normal calculation unit 31 detects the palm area ARk and the finger area ARf, and detects the skeleton of the hand as shown by a broken line connecting the center of gravity of the palm area ARk and the tip of the finger area ARf.

The normal calculation unit 31 performs fitting between the detected hand skeleton and a skeleton model stored in advance for each hand posture, and uses the posture of the skeleton model that minimizes the fitting error as the imaged hand posture. . For example, the normal calculation unit 31 matches the centroid of the detected hand skeleton with the centroid of the skeleton model stored in advance, and calculates a sum of absolute differences SAD (sum of absolute difference) of position coordinates such as joints and fingertips. Calculate every time. The normal line calculation unit 31 sets the posture with the smallest calculated sum of absolute differences SAD as the posture of the photographed hand. The normal calculation unit 31 detects the hand position and hand posture in this way, and proceeds to step ST45.

In step ST45, the normal calculation unit 31 eliminates the indefiniteness of the normal. The normal line calculation unit 31 determines the normal line corresponding to the hand region from the normal line calculated in step ST42 based on the hand position and hand posture detected in step ST44, that is, the normal line having an indefiniteness of 180 degrees. The process proceeds to step ST46.

In step ST46, the UI processing unit 41 determines the shape of the hand. The UI processing unit 41 discriminates the shape of the hand based on the normal of the hand region in which the ambiguity is eliminated, and uses the discrimination result as UI information. The UI processing unit 41 determines, for example, the shape of the hand in detail by integrating the normal of the hand region in which the indefiniteness is eliminated. In determining the shape of the hand, distance image information, a three-dimensional model, or the like may be used.

If such processing is performed, the shape of the hand can be discriminated more accurately than when the distance image is used. In the third embodiment, not only the shape of the hand but also the pointing direction may be determined. In this case, in step ST44, the normal calculation unit 31 performs fitting between the finger area detected by the same process as in step ST24 and the finger shape model stored in advance for each pointing direction. In the fitting, the detected finger area (or hand area) and the finger area (or hand area) of the finger shape model are overlapped using the fingertip area detected by the same process as in step ST24 as a fulcrum. The UI processing unit 41 sets the orientation of the finger shape model that minimizes the fitting error, that is, the overlay error, as the pointing direction indicated by the imaged hand.

The operations shown in the flowchart in the first embodiment include a process for generating a normal having indefiniteness and a process for generating a temporary recognition processing image and performing temporary recognition of a subject to be recognized. It is not restricted to performing in parallel. For example, one process may be performed before the other process is performed.

<3. Second Embodiment>
By the way, in the first embodiment described above, the case where UI information is generated based on a normal line in which indefiniteness has been eliminated has been described. However, UI information may be generated based on a normal line having indefiniteness. it can. In the second embodiment, a case where UI information is generated based on a normal having indefiniteness will be described.

<3-1. Configuration of Second Embodiment>
FIG. 19 illustrates the configuration of the second embodiment of the image processing apparatus. The image processing apparatus 12 includes polarization image acquisition units 20-1 and 20-2, normal calculation units 32-1 and 32-2, and a user interface (UI) processing unit 42.

The polarization image acquisition units 20-1 and 20-2 acquire a plurality of polarization images having different polarization directions. The polarization image acquisition units 20-1 and 20-2 are configured in the same manner as the polarization image acquisition unit 20 in the first embodiment. The polarization image acquisition unit 20-1 acquires a plurality of polarization images having different polarization directions in which the recognition subject is imaged, and outputs the plurality of polarization images to the normal calculation unit 32-1. The polarization image acquisition unit 20-2 acquires a plurality of polarization images having different polarization directions in which the learning subject is imaged and outputs the acquired images to the normal calculation unit 32-2. Further, the polarization image acquisition units 20-1 and 20-2 may output the acquired polarization image to the UI processing unit 42.

The normal calculation unit 32-1 (32-2) calculates a normal from a plurality of polarization images acquired by the polarization image acquisition unit 20-1 (20-2). The normal calculation unit 32-1 (32-2) is configured using the polarization processing unit 301 of the normal calculation unit 31 in the first embodiment. The normal calculation unit 32-1 performs processing similar to that of the polarization processing unit 301 described above from the polarization image to be recognized acquired by the polarization image acquisition unit 20-1, calculates a normal, and performs a UI processing unit 42. Output to. Similarly, the normal calculation unit 32-2 calculates a normal from the polarization image of the learning subject acquired by the polarization image acquisition unit 20-2 and outputs the normal to the UI processing unit 42.

The UI processing unit 42 uses the subject to be recognized in the polarization image acquired by the polarization image acquisition unit 20-1 as an input indicator in the user interface. In addition, the UI processing unit 41 performs subject recognition based on the normal that has not been solved by the indeterminacy calculated by the normal calculation unit 32-1, and generates input information (hereinafter referred to as “UI information”) in the user interface. As described above, the normal is information indicating the three-dimensional shape of the subject to be recognized, and the UI processing unit 42 performs subject recognition, recognizes the type, position, orientation, and the like of the subject to be recognized, and the recognition result. Is output as UI information. The UI processing unit 42 stores teacher data corresponding to the learning subject in advance, and is stored as student data calculated based on a plurality of polarization images with different polarization directions obtained by imaging the subject to be recognized. Based on the teacher data, recognition processing of the subject to be recognized is performed.

The UI processing unit 42 includes a teacher data generation unit 421, a teacher database unit 422, and a recognition processing unit 423.

The teacher data generating unit 421 generates teacher data corresponding to the learning subject using the normal calculated by the normal calculating unit 32-2, and stores the teacher data in the teacher database unit 422. Further, the teacher data generation unit 421 generates a non-polarized image (normal image) using the polarized image supplied from the polarized image acquisition unit 20-2, and the feature amount calculated from the non-polarized image and the acquired normal line. Teacher data may be generated using.

The teacher database unit 422 stores the teacher data generated by the teacher data generation unit 421. In addition, the teacher database unit 422 outputs the stored teacher data to the recognition processing unit 423.

The recognition processing unit 423 generates student data based on the normal calculated by the normal calculation unit 32-1 and performs recognition processing using the generated student data and teacher data stored in the teacher database unit 422. And generate UI information. In addition, the recognition processing unit 423 generates a non-polarized image (normal image) using the polarized image supplied from the polarized image acquisition unit 20-1, and calculates the feature amount calculated from the non-polarized image and the acquired normal line. May be used to generate student data.

<3-2. Operation of Second Embodiment>
Next, the operation of the second embodiment will be described. FIG. 20 is a flowchart showing the learning operation. In step ST51, the polarization image acquisition unit 20-2 acquires a polarization image of the learning subject. The polarization image acquisition unit 20-2 captures a learning subject using a polarizing plate or a polarization filter, acquires a plurality of polarization images having different polarization directions, and proceeds to step ST52.

In step ST52, the normal calculation unit 32-2 calculates a normal. The normal calculation unit 32-2 fits pixel values of a plurality of polarization images having different polarization directions to a model equation for each pixel of the polarization image, and calculates a normal based on the model equation after the fitting. Proceed to ST54.

In step ST54, the UI processing unit 42 generates teacher data. The UI processing unit 42 generates teacher data based on the normal calculated based on the polarization image of the learning subject, and proceeds to step ST55.

In step ST55, the UI processing unit 42 stores teacher data. The UI processing unit 42 stores the teacher data generated in step ST54 in the teacher database unit 422.

Further, the processing from step ST51 to step ST55 is performed for each learning subject, and the UI processing unit 42 stores teacher data generated with various objects as learning subjects. When generating UI information based on a normal having indefiniteness and a polarization image, the UI processing unit 42 performs the process of step ST53 to generate a non-polarization image from the polarization image. In step ST54, the UI processing unit 42 generates teacher data by using the normal calculated based on the polarization image of the learning subject and the feature amount calculated from the non-polarized image.

FIG. 21 is a flowchart showing the recognition operation using the learning result. In step ST61, the polarization image acquisition unit 20-1 acquires a polarization image of the recognition target object. The polarization image acquisition unit 20-1 captures a subject to be recognized using a polarizing plate or a polarization filter, acquires a plurality of polarization images having different polarization directions, and proceeds to step ST62.

In step ST62, the normal calculation unit 32-1 calculates a normal. The normal calculation unit 32-1 fits pixel values of a plurality of polarization images having different polarization directions to a model equation for each pixel of the polarization image, and calculates a normal based on the model equation after the fitting. Proceed to ST64.

In step ST64, the UI processing unit 42 generates student data. The UI processing unit 42 generates student data based on the normal calculated based on the polarization image to be recognized, and proceeds to step ST65.

In step ST65, the UI processing unit 42 generates UI information. The UI processing unit 42 determines the type, position, orientation, and the like of the subject to be recognized based on the student data generated in step ST64 and the teacher data stored by performing the processing in steps ST51 to ST55. The result is UI information. In addition, when generating UI information based on a normal having indefiniteness and a polarization image, the UI processing unit 42 performs the process of step ST63 to generate a non-polarization image from the polarization image. In step ST64, the UI processing unit 42 generates student data using the normal calculated based on the polarization image to be recognized and the feature amount calculated from the non-polarized image.

<3-3. Specific Example in Second Embodiment>
Next, a specific example in the second embodiment will be described. FIG. 22 shows an operation of a specific example in the second embodiment. In addition, the specific example has shown the case where UI information is produced | generated based on the normal line which has indefiniteness. In step ST71, the polarization image acquisition unit 20-2 acquires a polarization image (teacher polarization image) of the learning subject. For example, with the hand in the Rock state, the polarization image acquisition unit 20-2 captures the hand using a polarizing plate or a polarization filter, acquires a plurality of polarization images having different polarization directions, and proceeds to Step ST72. move on.

In step ST72, the normal calculation unit 32-2 calculates a normal. For each pixel of the polarization image, the normal line calculation unit 32-2 fits pixel values of a plurality of polarization images having different polarization directions to the model formula, and moves the hand based on the model formula after the fitting. The normal line in the state is calculated, and the process proceeds to step ST73.

In step ST73, the UI processing unit 42 generates teacher data. The UI processing unit 42 generates teacher data based on the normal line of the learning subject. For example, the UI processing unit 42 forms a histogram of normals when the hand is in a rock state, and proceeds to step ST74 using the obtained normal histogram as teacher data.

In step ST74, the UI processing unit 42 stores teacher data. For example, the UI processing unit 42 stores, as teacher data, a normal histogram when the hand is in a rock state in the teacher database unit.

Also, the processing from step ST71 to step ST74 is performed for each learning subject, for example, when the hand is in a par state or a scissors state, and the teacher data in each state is stored in the teacher database unit 422. .

In step ST75, the polarization image acquisition unit 20-1 acquires a polarization image to be recognized. The polarization image acquisition unit 20-1 captures, for example, a hand when performing a soap using a polarizing plate or a polarization filter, acquires a plurality of polarization images having different polarization directions, and proceeds to step ST76.

In step ST76, the normal calculation unit 32-1 calculates a normal. The normal calculation unit 32-1 fits pixel values of a plurality of polarization images having different polarization directions to a model equation for each pixel of the polarization image, and calculates a normal based on the model equation after the fitting. Proceed to ST77.

In step ST77, the UI processing unit 42 generates student data. The UI processing unit 42 generates student data based on the normal line to be recognized. For example, the UI processing unit 42 histograms the normals regarding the state of the hand to be recognized, and proceeds to step ST78 using the obtained normal histograms as student data.

In step ST78, the UI processing unit 42 generates UI information. The UI processing unit 42 determines from the teacher database unit 422 the teacher data most similar to the student data obtained in step ST77. Further, the UI processing unit 42 determines the hand state corresponding to the determined teacher data as the hand state captured in the polarization image acquired in step ST75, and outputs the determination result as UI information.

If such a process is performed, UI information can be generated without performing a process for eliminating the indefiniteness of the normal. In addition, as in the first embodiment, the recognition process can be performed with higher accuracy than when the distance image is used.

In the second embodiment, a configuration for generating a normal from a learning subject and a configuration for generating a normal from a subject to be recognized are provided separately. Therefore, in the configuration in which the normal line is generated from the learning subject, it is possible to generate the teacher data with higher accuracy than in the configuration in which the normal line is generated from the recognition subject. Therefore, by storing the teacher data used as a criterion for discrimination in the teacher database unit 422 as high-precision data, it is possible to obtain an accurate discrimination result. In the second embodiment, the polarization image acquisition unit and the normal line calculation unit may be shared between the case where the normal line is generated from the learning subject and the case where the normal line is generated from the desired recognition target. In this case, the configuration of the image processing apparatus is simplified, and the image processing apparatus can be provided at a low cost. Furthermore, in the second embodiment, a communication unit, a recording medium mounting unit, and the like may be provided in the UI processing unit so that teacher data can be updated or added from the outside. Thus, if updating or addition of teacher data can be performed from the outside via a communication path or a recording medium, more objects can be used as subjects to be recognized, and versatility can be improved.

FIG. 23 exemplifies a user interface to which the present technology is applied. For example, hand recognition is performed using a recognition target as a hand. In hand recognition, the hand shape and pointing direction can be determined. In the hand shape discrimination, since a three-dimensional shape can be obtained with high accuracy, a fine finger shape can also be obtained. Also, face recognition is performed using the recognition target as a face. In face recognition, personal authentication, facial expression and line-of-sight direction can be determined. Furthermore, human recognition is performed with a recognition target as a person. In human recognition, body authentication and pose discrimination can be performed. When object authentication is performed using an object as a recognition target, the posture of a known object can be determined in object authentication.

According to the present technology, since a normal close to the three-dimensional shape of the subject can be calculated from the polarization image as compared with the normal generated from the conventional distance image, for example, even for a subject that is angled toward the camera The normal can be calculated stably. Therefore, the subject to be recognized can be easily and accurately recognized by using the normal calculated from the polarization image. Furthermore, if this technique is applied to the user interface, the pointing direction and the like can be recognized more reliably than when the pointing direction and the like are recognized from the distance image, and thus it is possible to provide a stress-free user interface. For example, if the configuration shown in FIGS. 4A and 4B is used as the polarization image acquisition unit, the normal can be calculated from the polarization image acquired by the monocular camera, so there is no need to use a plurality of cameras. Therefore, application to a user interface is easy.

The series of processes described in the specification can be executed by hardware, software, or a combined configuration of both. When processing by software is executed, a program in which a processing sequence is recorded is installed and executed in a memory in a computer incorporated in dedicated hardware. Alternatively, the program can be installed and executed on a general-purpose computer capable of executing various processes.

For example, the program can be recorded in advance on a hard disk, SSD (Solid State Drive), or ROM (Read Only Memory) as a recording medium. Alternatively, the program is a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical disc), a DVD (Digital Versatile Disc), a BD (Blu-Ray Disc (registered trademark)), a magnetic disk, or a semiconductor memory card. It can be stored (recorded) in a removable recording medium such as temporarily or permanently. Such a removable recording medium can be provided as so-called package software.

In addition to installing the program from the removable recording medium to the computer, the program may be transferred from the download site to the computer wirelessly or by wire via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this way and install it on a recording medium such as a built-in hard disk.

In addition, the effect described in this specification is an illustration to the last, and is not limited, There may be an additional effect which is not described. Further, the present technology should not be construed as being limited to the embodiments of the technology described above. The embodiments of this technology disclose the present technology in the form of examples, and it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present technology. In other words, the scope of the claims should be considered in order to determine the gist of the present technology.

In addition, the image processing apparatus according to the present technology may have the following configuration.
(1) a polarization image acquisition unit that acquires a plurality of polarization images having different polarization directions in which a subject to be recognized is imaged;
A normal calculation unit that calculates a normal for each pixel based on the polarization image acquired by the polarization image acquisition unit;
An image processing apparatus comprising: a recognition unit that recognizes the subject using the normal calculated by the normal calculation unit.
(2) The subject to be recognized is an input indicator in a user interface,
The image processing apparatus according to (1), wherein the recognition unit uses the recognition result of the subject as input information in the user interface.
(3) The normal calculation unit
A provisional recognition processing image generation unit that generates a provisional recognition processing image from the plurality of polarization images;
A temporary recognition recognition processing unit that performs temporary recognition of the subject using the temporary recognition processing image generated by the temporary recognition processing image generation unit;
A polarization processing unit that calculates normals from the plurality of polarization images;
An indeterminacy eliminating unit that eliminates the indeterminacy of the normal calculated by the polarization processing unit based on the temporary recognition result of the temporary recognition processing unit,
The image processing apparatus according to (1) or (2), wherein the recognizing unit recognizes the subject using a normal whose indefiniteness has been eliminated by the normal calculating unit.
(4) The temporary recognition processing unit uses, as a temporary recognition result of the subject, a model closest to the subject using the temporary recognition processing image and a model image registered in advance.
The image processing apparatus according to (3), wherein the indeterminacy canceling unit cancels the indefiniteness of the normal calculated by the polarization processing unit based on the model temporarily recognized by the temporary recognition processing unit.
(5) The subject to be recognized is a hand,
The temporary recognition processing unit recognizes the positions of the fingertips and finger pads using the temporary recognition processing image and a pre-registered model image,
The image processing apparatus according to (4), wherein the indefiniteness eliminating unit eliminates indeterminacy of a normal of a finger region in the hand based on the positions of the fingertip and the finger pad temporarily recognized by the temporary recognition processing unit. .
(6) The image processing device according to (5), wherein the recognizing unit determines a pointing direction based on a normal of the finger region whose indefiniteness has been eliminated by the normal calculation unit.
(7) The subject to be recognized is a face,
The temporary recognition processing unit recognizes the position of the face area using the temporary recognition processing image and a model image registered in advance,
The image processing apparatus according to (4), wherein the indefiniteness eliminating unit eliminates indefiniteness of the normal of the face based on the position of the face area temporarily recognized by the temporary recognition processing unit.
(8) The image processing apparatus according to (7), wherein the recognizing unit determines a face shape or a facial expression based on a normal of the face region in which indefiniteness is eliminated by the normal calculation unit.
(9) The subject to be recognized is a hand,
The temporary recognition processing unit recognizes the position and skeleton structure of the hand region using the temporary recognition processing image and a pre-registered model image,
The image processing according to (4), wherein the indefiniteness eliminating unit eliminates indefiniteness of the normal of the hand region based on a position and a skeleton structure of the hand region temporarily recognized by the temporary recognition processing unit. apparatus.
(10) The image processing apparatus according to (10), wherein the recognizing unit determines a hand shape based on a normal of the hand region in which indefiniteness is eliminated by the normal calculation unit.
(11) The recognition unit
A teacher data generation unit that generates teacher data corresponding to the learning subject from normals calculated based on a plurality of polarized images having different polarization directions obtained by imaging the learning subject;
A teacher database unit for storing the teacher data generated for each learning subject by the teacher data generation unit;
Student data generated according to the recognition target using normals calculated based on a plurality of polarization images having different polarization directions obtained by imaging the subject to be recognized, and teacher data stored in the teacher database unit The image processing apparatus according to (1) or (2), further including a recognition processing unit that recognizes the subject to be recognized.
(12) The polarization image acquisition unit acquires a plurality of polarization images having different polarization directions for each of the recognition target and the learning subject,
The image processing apparatus according to (11), wherein the normal line calculation unit calculates a normal line for each of the recognition object and the learning subject based on the polarization image acquired by the polarization image acquisition unit.
(13) A learning polarization image acquisition unit that acquires a plurality of polarization images having different polarization directions obtained by imaging the learning subject;
The image processing apparatus according to (11) or (12), further including a learning normal calculation unit that calculates a normal based on the polarization image acquired by the learning polarization image acquisition unit.
(14) The teacher data is data indicating a distribution of normals for the learning subject, and the student data is data indicating a distribution of normals calculated for the subject to be recognized. The image processing apparatus according to any one of 13).
(15) The image processing device according to any one of (11) to (14), wherein the recognition processing unit sets the learning subject corresponding to the teacher data closest to the student data as a recognition result.

In the image processing apparatus, the image processing method, and the program according to this technique, a plurality of polarization images having different polarization directions in which a subject to be recognized is captured are acquired, and a normal line is obtained for each pixel based on the acquired polarization image. Is calculated, and the subject is recognized using the calculated normal. For this reason, it is possible to easily and accurately recognize the subject. Therefore, it is suitable for a device having an interface for performing operation control and start / end / change / update of signal processing according to the recognition result of the type, position, orientation, etc. of the object.

DESCRIPTION OF

SYMBOLS

10, 11, 12 ... Image processing device 20, 20-1, 20-2 ... Polarized

image acquisition part

30, 31, 32-1, 32-2 ... Normal calculation part 40 ...

Recognition Unit

41, 42 ... User interface (UI) processing unit 201 ... Image sensor 202 ... Polarizing filter 203 ...

Lens

204, 211, 212-1 to 212-4 ... Polarizing plate 210, 210 -1 to 210-4: Imaging unit 301: Polarization processing unit 302 ... Temporary recognition processing image generation unit 303 ... Temporary recognition processing unit 304 ... Model database unit 305 ... Uncertainty Resolving unit 421 ... Teacher data generation unit 422 ... Teacher database unit 423 ... Recognition processing unit

Claims

A polarization image acquisition unit that acquires a plurality of polarization images having different polarization directions in which a subject to be recognized is imaged;
A normal calculation unit that calculates a normal for each pixel based on the polarization image acquired by the polarization image acquisition unit;
An image processing apparatus comprising: a recognition unit that recognizes the subject using the normal calculated by the normal calculation unit.
The object to be recognized is an input indicator in a user interface,
The image processing apparatus according to claim 1, wherein the recognition unit uses the recognition result of the subject as input information in the user interface.
The normal calculation unit
A provisional recognition processing image generation unit that generates a provisional recognition processing image from the plurality of polarization images;
A temporary recognition recognition processing unit that performs temporary recognition of the subject using the temporary recognition processing image generated by the temporary recognition processing image generation unit;
A polarization processing unit that calculates normals from the plurality of polarization images;
An indeterminacy eliminating unit that eliminates the indeterminacy of the normal calculated by the polarization processing unit based on the temporary recognition result of the temporary recognition processing unit,
The image processing apparatus according to claim 1, wherein the recognizing unit recognizes the subject using a normal whose indefiniteness has been eliminated by the normal calculating unit.
The temporary recognition processing unit uses, as a temporary recognition result of the subject, a model closest to the subject using the image for temporary recognition processing and an image of a model registered in advance.
The image processing apparatus according to claim 3, wherein the indeterminacy canceling unit cancels the indefiniteness of the normal calculated by the polarization processing unit based on the model temporarily recognized by the temporary recognition processing unit.
The object to be recognized is a hand,
The temporary recognition processing unit recognizes the positions of the fingertips and finger pads using the temporary recognition processing image and a pre-registered model image,
The image processing apparatus according to claim 4, wherein the indeterminacy canceling unit cancels the indeterminacy of the normal of the finger region in the hand based on the positions of the fingertip and the finger pad temporarily recognized by the temporary recognition processing unit.
The image processing apparatus according to claim 5, wherein the recognizing unit determines a pointing direction based on a normal of the finger region whose indefiniteness has been eliminated by the normal calculation unit.
The object to be recognized is a face,
The temporary recognition processing unit recognizes the position of the face area using the temporary recognition processing image and a model image registered in advance,
The image processing apparatus according to claim 4, wherein the indeterminacy canceling unit cancels the indefiniteness of the normal of the face based on the position of the face area temporarily recognized by the temporary recognition processing unit.
The image processing apparatus according to claim 7, wherein the recognizing unit determines a face shape or a facial expression based on a normal of the face area whose indefiniteness has been eliminated by the normal calculation unit.
The object to be recognized is a hand,
The temporary recognition processing unit recognizes the position and skeleton structure of the hand region using the temporary recognition processing image and a pre-registered model image,
The image processing apparatus according to claim 4, wherein the ambiguity canceling unit resolves the ambiguity of the normal of the hand region based on a position and a skeleton structure of the hand region temporarily recognized by the temporary recognition processing unit. .
The image processing apparatus according to claim 9, wherein the recognizing unit determines a hand shape based on a normal of the hand region whose indefiniteness has been eliminated by the normal calculation unit.
The recognition unit
A teacher data generation unit that generates teacher data corresponding to the learning subject from normals calculated based on a plurality of polarized images having different polarization directions obtained by imaging the learning subject;
A teacher database unit for storing the teacher data generated for each learning subject by the teacher data generation unit;
Student data generated according to the recognition target using normals calculated based on a plurality of polarization images having different polarization directions obtained by imaging the subject to be recognized, and teacher data stored in the teacher database unit The image processing apparatus according to claim 1, further comprising: a recognition processing unit that recognizes a subject to be recognized.
The polarization image acquisition unit acquires a plurality of polarization images having different polarization directions for each of the recognition target and the learning subject,
The image processing apparatus according to claim 11, wherein the normal line calculation unit calculates a normal line for each of the recognition target and the learning subject based on the polarization image acquired by the polarization image acquisition unit.
A learning polarization image acquisition unit that acquires a plurality of polarization images having different polarization directions obtained by imaging the learning subject;
The image processing apparatus according to claim 11, further comprising a learning normal calculation unit that calculates a normal based on the polarization image acquired by the learning polarization image acquisition unit.
The image processing apparatus according to claim 11, wherein the teacher data is data indicating a distribution of normals of the learning subject, and the student data is data indicating a distribution of normals calculated for the subject to be recognized.
The image processing apparatus according to claim 11, wherein the recognition processing unit recognizes the learning subject corresponding to the teacher data closest to the student data.
Obtaining a plurality of polarization images having different polarization directions in which a subject to be recognized is imaged by the polarization image acquisition unit;
In the normal calculation unit, based on the polarization image acquired by the polarization image acquisition unit, calculating a normal for each pixel;
An image processing method comprising: recognizing the subject using a normal calculated by the normal calculation unit in a recognition unit.
A procedure for acquiring a plurality of polarization images having different polarization directions in which a subject to be recognized is imaged,
A procedure for calculating a normal for each pixel based on the acquired polarization image;
A program for causing a computer to execute a procedure for recognizing the subject using the calculated normal.