JP2018009927A - Image processing device, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to measure a normal line of a measurement object with high accuracy even when a position relation between a light source and the measurement object cannot be measured with high accuracy.SOLUTION: An image processing device has: calculation means that calculates a normal line map holding a normal line of a surface of the measurement object for each pixel by a photometric stereo method on the basis of three or shot images in a state where a measurement object is sequentially applied from each of three or more light sources different in an irradiation direction, and position information on the light source; and determination means that calculates a rough shape of the measurement object on the basis of the normal line map calculated by the calculation means, and compares the calculated rough shape with a preliminarily acquired reference rough shape to thereby determine whether it is necessary to update the position information on the light source; and update means that, when it is determined by the determination means that it is necessary to update the position information on the light source, updates the position information on the light source. The calculation means is configured to recalculate a normal line map, using the position information on the light source updated by the update means.SELECTED DRAWING: Figure 7

Description

  The present invention relates to a technique for measuring a normal map of an object to be measured by an illuminance difference stereo method.

  There is an illuminance difference stereo method as a method for measuring the fine shape of the surface of a measurement object using a camera. The illuminance difference stereo method is based on three or more images captured in a state where the measurement object is sequentially irradiated from each of three or more light sources having different irradiation directions, and the positional relationship between the light source and the measurement object. This is a method for calculating the normal line at each point (fine area) on the surface of the measurement object. When calculating the normal line by this illuminance difference stereo method, there is a problem that the measurement accuracy of the normal line decreases if the positional relationship between each light source and the measurement object cannot be obtained with high accuracy.

  Therefore, in Patent Document 1, the measurement object irradiated sequentially by four or more light sources is imaged, and the normal line of the measurement object is calculated with relatively high accuracy from the obtained four or more images. A method of estimating a combination of three or more images that can be used and calculating a normal is disclosed. Patent Document 2 discloses a method for estimating the positional relationship between a light source and a measurement object by using a measurement pin for measuring the light source position.

JP 2014-38049 A JP 2014-112043 A

  In the technique disclosed in Patent Document 1, it is possible to select a combination of three or more light sources that provide a relatively high measurement accuracy from four or more light sources, but the measurement accuracy decreases unless the positions of the light sources are given correctly. The problem remains.

  In the technique disclosed in Patent Document 2, the position of the light source is estimated based on the position of the tip of the shadow of the measurement pin. However, since the light source generally has a certain size, blurring often occurs at the tip of the shadow. In that case, it is difficult to accurately detect the position of the tip of the shadow, and there is a problem that an error occurs in the position of the obtained light source.

  Therefore, the present invention provides an image processing that makes it possible to measure the normal of a measurement object with high accuracy even when the positional relationship between the light source and the measurement object cannot be measured with high accuracy. For the purpose.

  The image processing apparatus according to the present invention is based on three or more images captured in a state where the measurement object is sequentially irradiated from each of three or more light sources having different irradiation directions, and position information of the light sources. A calculation means for calculating a normal map in which a normal of the surface of the measurement object is held for each pixel by an illuminance difference stereo method, and the measurement object based on the normal map calculated by the calculation means A determination unit that determines whether or not the position information of the light source needs to be updated by calculating a rough shape of the light source and comparing the calculated rough shape with a reference rough shape of the measurement object acquired in advance. And updating means for updating the position information of the light source when it is determined by the determining means that the position information of the light source needs to be updated, and the calculating means is Updated light source Wherein the re-calculating the normal map using location information.

  According to the present invention, in the image processing apparatus that measures the normal line of the measurement object by the illuminance difference stereo method, the normal line of the measurement object is determined even when the positional relationship between the light source and the measurement object cannot be measured with high accuracy. It becomes possible to measure with high accuracy.

It is a figure which shows a detailed shape and a schematic shape typically. It is a figure which shows a mode in case the imaging device in Example 1 images a measuring object. 1 is a diagram illustrating an appearance of an imaging apparatus according to a first embodiment. 1 is a block diagram illustrating an internal configuration of an imaging apparatus according to Embodiment 1. FIG. 1 is a flowchart of overall processing of an imaging apparatus in Embodiment 1. FIG. 2 is a diagram illustrating an internal configuration of an image processing unit in Embodiment 1. FIG. FIG. 3 is a flowchart illustrating a processing flow in an image processing unit according to the first exemplary embodiment. It is a figure which shows typically the update of the light emission part position in Example 1. FIG. FIG. 3 is a diagram illustrating an internal configuration of an evaluation unit in Example 1. It is a flowchart figure which shows the processing flow of the evaluation part in Example 1. FIG. FIG. 10 is a diagram illustrating an internal configuration of an image processing unit according to a second embodiment. FIG. 10 is a flowchart illustrating a processing flow of an image processing unit in Embodiment 2. It is a figure which shows the internal structure of the evaluation part in Example 2. FIG.

  Embodiments of the present invention will be described below with reference to the drawings. However, the following embodiments do not limit the present invention, and all combinations of features described below are not necessarily essential for solving the problems of the present invention. In addition, in each figure referred in the following description, the same code | symbol is provided to the element equivalent to another figure in principle.

[Example 1]
In this embodiment, a normal map (high-precision normal map) including fine irregularities on the surface of the measurement object is measured by the illuminance difference stereo method. Here, the normal line is a perpendicular to the tangent plane at a point on the surface of the measurement object, and represents the direction of the surface. The normal map is obtained by dividing the surface of the measurement object into a plurality of fine regions and storing normals for each region.

  In the following description, the shape of the measurement object is expressed by two words, a detailed shape and a schematic shape. The detailed shape represents a fine shape including fine irregularities on the surface of the measurement object. In this embodiment, an object is to measure the detailed shape of the measurement object with high accuracy by the illuminance difference stereo method. On the other hand, the approximate shape represents a rough shape that does not include fine irregularities on the surface of the measurement object. FIG. 1 is a diagram schematically showing a detailed shape and a schematic shape. The schematic shape of the object whose detailed shape is shown in FIG. 1 (a) is shown in FIG. 1 (b), and the schematic shape of the object whose detailed shape is shown in FIG. Is shown in It should be noted that the size of the detailed shape up to the minute shape is about the size that each pixel of the imaging device used for measurement can expect on the measurement object, and the resolution, imaging distance, and focus of the imaging device. It depends on the distance. In the following description, it is assumed that the normal map and the detailed shape measured by the illuminance difference stereo method have the same resolution as the captured image, and the approximate shape has a resolution lower than the resolution of the captured image.

  In this example, the correct approximate shape of the measurement object is acquired in advance, and the illuminance is increased until the similarity between the approximate shape calculated based on the normal map measured by the illuminance difference stereo method and the correct approximate shape increases. The position information of the light source used for the difference stereo method is repeatedly updated. Thereby, even when the position of the light source cannot be measured with high accuracy, the normal map can be measured with high accuracy. The correct approximate shape acquired in advance is used as a reference approximate shape that is referred to in order to update the position information of the light source, and is also referred to as a reference approximate shape.

  FIG. 2 is a diagram illustrating a state where the imaging apparatus 201 according to the present exemplary embodiment captures an image of the measurement object 204. As shown in FIG. 2, the imaging apparatus 201 in the present embodiment performs imaging of the measurement object 204 installed on the installation table 202 to which the chart 203 is attached. Here, each of the installation table 202 and the measurement object 204 is provided with a reference position (not shown) used for aligning the position during installation. It is assumed that the relative position of the measurement object 204 with respect to the chart 203 can be generally understood by installing the measurement object 204 on the installation table 202 with the reference position being matched.

  Note that the image pickup apparatus 201 can be used by hand, but it is desirable to use it fixed on a tripod (not shown). The chart 203 only needs to be a pattern in which feature points of the chart can be easily detected. In addition to the checker pattern as shown in FIG. 2, a dot pattern in which dots are arranged or a marker used in AR (Augmented Reality) is used. May be.

  FIG. 3 is a diagram illustrating an example of the appearance of the imaging apparatus 201 in the present embodiment. 3A shows the front surface of the imaging device 201, and FIG. 3B shows the back surface of the imaging device 201. As shown in FIG. 3A, on the front surface of the imaging apparatus 201, an imaging unit 301 capable of acquiring a color image, and three light emitting units 302 that irradiate light toward the measurement object in accordance with the imaging. To 304 and an imaging button 305 are provided. In the present embodiment, the number of light emitting units is three, but the number of light emitting units is not limited to this, and a plurality of three or more necessary for obtaining a normal map of the measurement object by the illuminance difference stereo method is used. What is necessary is just to have a light emission part. Further, the arrangement of the light emitting units 302 to 304 is not limited to the illustrated arrangement, and may be an arbitrary arrangement. In the embodiment, an example in which the light emitting units 302 to 304 are incorporated in the imaging device 201 is shown, but each light emitting unit may have a configuration independent of the imaging device 201.

  The imaging button 305 is a two-stage switch that is half-pressed and fully pressed. When the user presses the imaging button 305 halfway, pre-imaging preparation processing such as exposure adjustment and white balance and focus adjustment are performed. When the user further presses the imaging button 305, the light emitting units 302 to 304 sequentially emit light one by one, and each time the light emitting unit emits light, the imaging unit 301 detects the amount of light from the measurement object. Detect with image sensor. The detected light quantity is A / D converted and converted into digital data. In this way, a color image (digital data) corresponding to the light emission of each light emitting unit is acquired. Here, since the number of light emitting units is three, a total of three color images are acquired.

  As shown in FIG. 3B, a display unit 306 including a liquid crystal display and an operation unit 307 for receiving various operations by the user are provided on the back surface of the imaging apparatus 201. The display unit 306 can display an operation state for setting imaging conditions, a GUI for confirming the settings, a preview image or video of a measurement target being imaged, and an image after imaging. The operation unit 307 is used by a user to perform a direction operation or the like for changing or selecting an item, and is arranged in the form of a cross button, for example. The button form is not limited to the cross button. In addition, although an example in which an operation is performed using a button is described in this embodiment, the present invention is not limited to this, and a configuration in which the display unit 306 receives an operation by a touch panel method may be used. For example, the display unit 306 uses various types of touch panels such as a pressure-sensitive type and an electromagnetic induction type, and can be configured to detect the coordinates touched by the user and read the user's instructions based on the detected coordinates.

<Internal configuration of imaging device>
FIG. 4 is a block diagram illustrating an internal configuration of the imaging apparatus 201. The CPU 401 is a central processing unit, and comprehensively controls each component in the imaging device 201 connected via the bus 404. A RAM (Random Access Memory) 402 is a memory that functions as a main memory, a work area, and the like of the CPU 401. A ROM (Read Only Memory) 403 is a memory for storing a control program executed by the CPU 401. The bus 404 functions as a transfer path for various data between the components in the imaging apparatus 201. For example, digital data acquired by the imaging unit 301 is sent to the digital signal processing unit 406 via the bus 404.

  Based on an instruction from the CPU 401, the imaging control unit 405 performs optical system control on the imaging unit 301, such as focusing (focusing process), opening / closing a shutter, and adjusting a diaphragm. Further, the imaging control unit 405 controls the light emission of the light emitting units 302 to 304 based on an instruction from the CPU 401. The digital signal processing unit 406 performs various processes such as white balance processing, gamma processing, and noise reduction processing on the digital data sent from the imaging unit 301. The processing result image is displayed on the display unit 306 or sent to the image processing unit 411 as an image used for the illuminance difference stereo method. It should be noted that for the image used for the illuminance difference stereo method sent to the image processing unit 411, it is desirable not to perform gamma processing and to output a value proportional to the amount of light detected by the imaging unit 301 as a pixel value. .

  The display control unit 407 performs display control of a pre-capture preview image and a captured image displayed on the display unit 306. In addition, display control for displaying characters and icons such as a UI for setting imaging parameters displayed on the display unit 306 and a UI for checking imaging parameters already set is also performed. The encoder unit 408 converts the image resulting from the processing by the digital signal processing unit 406 into a file format such as JPEG or MPEG. The external memory control unit 409 is an interface for connecting the imaging device 201 to a PC or other external memory 410 (for example, a hard disk, a memory card, a CF card, an SD card, a USB memory).

  The image processing unit 411 performs image processing such as calculation of a normal map. In the present embodiment, the image processing unit 411 calculates a high-precision normal map of the measurement object based on the illuminance difference stereo method based on the image sent from the digital signal processing unit 406. The constituent elements of the image pickup apparatus 201 exist in addition to those shown in FIG. 4, but are not the main points of the present invention, and thus description thereof is omitted.

<Processing flow of imaging device>
FIG. 5 is a flowchart of overall processing of the imaging apparatus 201 in the present embodiment. In step S501, the imaging control unit 405 acquires imaging parameters when the imaging unit 301 performs imaging. The imaging parameters include, for example, shutter speed, aperture, ISO sensitivity, and the like, and can be set by the user via the operation unit 307. The imaging control unit 405 also acquires order information indicating the order in which the light emitting units 302 to 304 emit light.

  In step S502, the imaging control unit 405 controls the imaging unit 301 to image the measurement object by causing the light emitting units 302 to 304 to emit light in response to the user pressing the imaging button 305. In this step, according to the order information acquired in step S501, the light emitting units emit light one by one, and the imaging unit 301 captures an image each time one light emitting unit emits light. In this embodiment, three images corresponding to each light emission of the light emitting units 302 to 304 are captured. Here, if the position of the imaging device 201 moves, for example, when the imaging device 201 is hand-held during image capture, alignment is performed on the three captured images. That is, in the three images, pixels having the same coordinates reflect the same position on the measurement object 204. The captured image is stored in the external memory 410.

  In step S503, the image processing unit 411 calculates a high-precision normal map of the measurement object by the illuminance difference stereo method based on the three images captured in step S502. Details of processing performed by the image processing unit 411 in this step will be described later.

  In step S504, the image processing unit 411 outputs the normal map calculated in step S503 to the display control unit 407. The normal map is displayed on the display unit 306 under the control of the display control unit 407. In addition, the image processing unit 411 outputs the normal map calculated in step S503 to the external memory control unit 409. The normal map is recorded in the external memory 410 under the control of the external memory control unit 409.

<Internal configuration of image processing unit>
FIG. 6 is a block diagram illustrating an internal configuration of the image processing unit 411. The image acquisition unit 601 acquires three images corresponding to each of the light emitting units 302 to 304 captured in step S502.

  The correct approximate shape acquisition unit 602 acquires the correct approximate shape of the measurement object. In this embodiment, the correct approximate shape of the measurement object is measured in advance by a 3D scanner or the like and recorded in the external memory 410 or the ROM 403. The correct approximate shape acquisition unit 602 acquires the correct approximate shape by reading data recorded in the external memory 410 or the ROM 403 according to an instruction from the user. As the 3D scanner, for example, there is a TOF (Time-of-Flight) method for measuring the time until the irradiated light is reflected by the measurement object and returned. In addition, there are an active method in which a camera is used to capture an image of a measurement object being irradiated with pattern light, and a passive method in which the shape is restored from images obtained by imaging the measurement object from a plurality of viewpoints.

  The acquired correct answer approximate shape data can be polygon data composed of vertex data having three-dimensional coordinates of XYZ and information of a surface formed by connecting three or more vertices. It can also be modeled by combining geometric basic shapes such as planes, rectangular parallelepipeds, and spheres. The correct answer outline shape data also includes information indicating the reference position of the measurement object 204 used for alignment when the object is installed on the installation table 202.

  The initial light emitting unit position acquisition unit 603 acquires initial information of the light emitting unit position used when calculating the normal by the illuminance difference stereo method. Here, the light emitting unit position represents a relative three-dimensional position of the three light emitting units 302 to 304 with respect to the installation table 202. In the present embodiment, the position of the light emitting unit is calculated from the chart 203 attached to the installation base 202 shown in the captured image. A method for calculating the light emitting unit position will be described later. When the imaging device 201 is fixed to the installation table 202 and the positional relationship with the installation table 202 is known, the initial information of the light emitting unit position is the data recorded in the external memory 410 or the ROM 403. May be used.

  The light source direction calculation unit 604 calculates the light source direction for each region on the measurement object 204 based on the correct approximate shape of the measurement object 204 and the light emitting unit position. Here, the area on the measurement object 204 is an area where the normal is acquired, and represents a fine area on the measurement object 204 corresponding to each pixel of the image captured by the imaging device 201. To do. The light source direction represents a relative position from each region to each light emitting unit for each region on the measurement object 204.

  The normal line calculation unit 605 uses an illuminance difference stereo method based on the light source direction for each region on the measurement object 204 calculated by the light source direction calculation unit 604 and the three images corresponding to the light emitting units 302 to 304. Then, a normal map of the measurement object 204 is calculated.

  The evaluation unit 606 calculates the approximate shape of the measurement object 204 based on the normal map calculated by the normal calculation unit 605, compares the calculated approximate shape with the correct approximate shape, and determines the light emitting unit position (position information). ) Is evaluated (determined).

  When the evaluation unit 606 determines that the light emitting unit position needs to be updated, the light emitting unit position updating unit 607 updates the light emitting unit position used for the illuminance difference stereo method. The output unit 608 outputs the normal map calculated by the normal calculation unit 605 to the display control unit 407 and the external memory control unit 409 when the evaluation unit 606 determines that the light emitting unit position does not need to be updated. . The normal map is displayed on the display unit 306 under the control of the display control unit 407 or recorded in the external memory 410 under the control of the external memory control unit 409.

<Processing flow of image processing unit>
FIG. 7 is a flowchart illustrating a processing flow of the image processing unit 411 in the present embodiment. In step S <b> 701, the image acquisition unit 601 acquires three captured images corresponding to each light emission of the light emitting units 302 to 304 from the external memory 410. As described above, each image has a pixel value that is proportional to the amount of light incident on the imaging unit 301.

  In step S <b> 702, the correct approximate shape acquisition unit 602 reads the correct approximate shape of the measurement object 204 held in the ROM 403 or the external memory 410 in accordance with an instruction input by the user via the operation unit 307.

  In step S703, the initial light emitting unit position acquisition unit 603 acquires initial information of the light emitting unit position used for the illuminance difference stereo method. In the present embodiment, the initial information of the light emitting unit position is acquired by calculating the light emitting unit position from the chart 203 attached to the installation table 202 that is reflected in the captured image. Below, the detail of operation | movement of the initial light emission part position acquisition part 603 in this step is demonstrated.

<< Operation of initial light emitting unit position acquisition unit >>
Since the light emitting units 302 to 304 and the imaging unit 301 are fixed in the imaging device 201, if the positional relationship between the imaging unit 301 and each light emitting unit is known, the relative position of the imaging unit 301 with respect to the chart 203 can be The relative position of each light emitting unit can be calculated. The relative position of the imaging unit 301 with respect to the chart 203 can be calculated by an existing method, for example, the following method.

  The relationship between the two-dimensional coordinates u, v in the captured image obtained by the imaging of the imaging unit 301 and the three-dimensional coordinates X, Y, Z of the imaging target can be expressed by the following equation (1).

Here, A is an internal parameter that represents the focal length and principal point position of the imaging unit 301. R and t are external parameters representing the posture (rotation) and position (translation) of the imaging unit 301, R is a 3 × 3 matrix, and t is a 1 × 3 matrix. s is a scale factor and is known. The three-dimensional coordinates X, Y, and Z consider a three-dimensional space in which any one of the plurality of feature points of the chart 203 (predetermined feature point) is the origin and the XY plane is parallel to the chart 203. 3D coordinates are used. The three-dimensional coordinates of each feature point of the chart 203 in such a three-dimensional space (hereinafter, three-dimensional space) are known.

  In the present embodiment, the two-dimensional coordinates of each feature point of the chart 203 are detected from the photographed image acquired in step S701, and the above equation (from the relationship between the detected two-dimensional coordinates and the three-dimensional coordinates of each feature point is obtained. Estimate A, R, and t in 1).

Then, using the estimated R and t, the relative position of the imaging unit 301 with respect to the chart 203 (that is, the position of the imaging unit 301 in the three-dimensional space) P _{chart, cam} is calculated by the following equation (2). . Here, trans () represents transposition of a matrix.

Further, by adding the relative position P _{cam, light [i]} of the i-th light emitting unit with respect to the imaging unit 301 to the relative position P _{chart, cam} of the imaging unit 301 with respect to the chart 203, the i-th light emission with respect to the chart 203 The relative position P _{chart, light [i] of the part} is calculated. That is, the position of the light emitting unit in the three-dimensional space is calculated by the following equation (3).

  Returning to the flowchart of FIG. Next, in step S704, the light source direction calculation unit 604, for each region on the measurement target 204 based on the correct approximate shape of the measurement target 204 and the position of each light emission unit, from that region to each light emission unit. Calculate the direction. Here, the area on the measurement object 204 represents an area on the measurement object 204 corresponding to each pixel of the captured image. Therefore, in this step, for all pixels in which the measurement object 204 is captured in the captured image, the direction from the region on the measurement object 204 corresponding to the pixel to each light emitting unit is calculated. Below, the detail of operation | movement of the light source direction calculation part 604 in this step is demonstrated.

<< Operation of Light Source Direction Calculation Unit >>
The direction Dir _{obj, light [i]} (u, v) from the measuring object 204 corresponding to the pixel at coordinates (u, v) in the captured image to the i-th light emitting unit is expressed by the following equation (4). As described above, the calculation is performed from the position P _{chart, light [i] of the light} emitting unit and the position P _{chart, obj} (u, v) of the region on the measurement object.

Here, P _{chart, obj} (u, v) indicates that the straight line connecting the position of the imaging unit 301 and the point where the pixel of the coordinate (u, v) in the captured image is projected on the three-dimensional coordinate system is The position of a point that intersects with the correct approximate shape of the measurement object 204 virtually arranged with the reference position aligned with 202 is shown. Specifically, P _{chart, obj} (u, v) can be calculated by the following method.

  First, the position P (u, v) of a point obtained by projecting the pixel of the two-dimensional coordinate (u, v) of the image captured by the imaging unit 301 with the external parameters R and t onto the three-dimensional coordinate system is expressed by the following formula ( 5). Here, f represents the focal length of the imaging unit 301, and cx and cy represent the principal point position of the imaging unit 301.

Then, by finding a point where a straight line connecting the point P (u, v) obtained by projecting the pixel onto the three-dimensional coordinate system and the position of the imaging unit 301 intersects the correct approximate shape of the measurement object 204, P _{chart, obj} (u, v) is calculated. As a method of finding the intersection of the straight line and the approximate shape, a known method such as geometric calculation may be used, or collision determination used at the time of rendering by ray tracing of computer graphics may be used. The light source direction is calculated by performing the above processing for all pixels and all light emitting portions of the captured image.

  Returning to the flowchart of FIG. Subsequently, in step S705, the normal calculation unit 605 calculates a normal for all pixels of the captured image by the illuminance difference stereo method. Below, the detail of operation | movement of the normal calculation part 605 in this step is demonstrated.

<< Operation of Normal Calculation Unit >>
An existing method can be used for the photometric stereo method. For example, the normal N (u, v) of the pixel at the coordinates (u, v) can be calculated by the following equation (6).

Here, I _{light [i]} (u, v) is a pixel value (for example, luminance value) of coordinates (u, v) of a captured image corresponding to light emission of each light emitting unit, and I (u, v) is It is a 1 × 3 matrix that summarizes I _{light [i]} (u, v). S (u, v) is a 3 × 3 obtained by collecting the directions Dir _{obj, light [i]} (u, v) from the region on the measurement object 204 corresponding to the pixel of coordinates (u, v) to each light emitting unit. And can be expressed by the following equation (7).

Inv (S) in Equation (6) represents an inverse matrix of the matrix S. When the number of light emitting units is 4 or more, it is impossible to calculate an inverse matrix of the matrix S (u, v) in which the directions Dir _{obj, light [i]} (u, v) to the respective light emitting units are collected. Therefore, a pseudo inverse matrix like the following formula (8) is used instead of inv (S).

Furthermore, the normal map can be calculated by performing the above calculation for all the pixels of the captured image. Note that the normal line calculation unit 605 may further calculate the detailed shape of the measurement object 204 based on the calculated normal map.

  Returning to the flowchart of FIG. Next, in step S706, the evaluation unit 606 evaluates the necessity of updating the light emitting unit position based on the normal map calculated by the illuminance difference stereo method in step S705 and the correct approximate shape acquired in step S702. To do. That is, the evaluation unit 606 calculates the approximate shape of the measurement object based on the normal map calculated by the illuminance difference stereo method. Then, an evaluation value representing the degree of similarity between the calculated approximate shape and the correct approximate shape is calculated, and the necessity of updating the light emitting portion position is determined based on whether the evaluation value satisfies a preset evaluation criterion. evaluate. For example, when the evaluation value is equal to or less than a predetermined reference value, it is determined that the light emitting unit position needs to be updated, and the process proceeds to step S707. On the other hand, when the evaluation value is larger than the predetermined reference value, it is determined that the light emitting unit position does not need to be updated, and the process proceeds to step S708. Details of the operation of the evaluation unit 606 in this step will be described later.

  In step S707, the light emitting unit position updating unit 607 updates the light emitting unit position used for calculation of the normal map. Here, the update of the light emitting unit position is a process of updating the position information of the light emitting unit used for calculation of the light source direction in step S704, instead of changing the physical position of the light emitting units 302, 303, and 304.

  FIG. 8 is a diagram schematically showing the update of the light emitting unit position in the present embodiment. FIG. 8 shows a cross-sectional view of the measuring object 204 and the light emitting units 302, 303, and 304 installed on the installation table 202 as viewed from above. The positions of the light emitting units 302, 303, and 304 are updated from the dotted line position to the solid line position in the figure.

  FIG. 8 shows an example in which the positions of the light emitting units 302, 303, and 304 are changed by the same amount in the X direction. However, updating the positions of the light emitting units changes the positions of all the light emitting units to a uniform amount and direction. However, the light emitting unit position can be changed in different directions and in different amounts. As a method for determining the change amount and direction of the light emitting unit position, the light emitting unit position may be determined randomly or regularly. For example, after the range that the position of the light emitting unit can take is determined in advance, a random number corresponding to the change amount and direction may be generated every time step S707 is executed, and the position may be changed within the above range. Alternatively, a method of regularly updating the position within the above range at regular intervals may be used. Alternatively, by predicting the change amount and direction in which the evaluation value is improved from the evaluation value for the normal map calculated using the light emission unit position and the change amount and direction of the position of the plurality of light emission units performed in the past. The change amount and direction may be determined. After step S707, the process proceeds to step S704, and the light source direction is calculated using the updated light emitting unit position. And the process of step S704-S707 is repeated until it determines with the update of a light emission part position not being required.

  Next, in step S708, the output unit 608 outputs the normal map (or detailed shape) calculated by the illuminance difference stereo method in step S705 to the display control unit 407 and the external memory control unit 409. The normal map (or detailed shape) is displayed on the display unit 306 under the control of the display control unit 407 or recorded in the external memory 410 under the control of the external memory control unit 409. Above, description of the operation | movement procedure of the image process part 411 is complete | finished.

<< Operation of Evaluation Unit >>
Below, the detail of operation | movement of the evaluation part 606 in step S706 is demonstrated. The evaluation unit 606 calculates the approximate shape of the measurement object based on the normal map calculated by the illuminance difference stereo method in step S705. Then, an evaluation value representing a similarity between the calculated approximate shape of the measurement object and the correct approximate shape of the measurement object acquired in advance in step S702 is calculated. Further, the necessity of updating the light emitting unit position is determined based on whether or not the evaluation value satisfies a preset evaluation criterion.

  Hereinafter, an example will be described in which a depth map is used as shape information when calculating an evaluation value representing the degree of similarity between the approximate shape of a measurement object calculated based on a normal map and the correct approximate shape. Here, the depth map is an image that holds a value proportional to the distance from the imaging unit 301 to the region on the measurement object as the pixel value of each pixel. Note that the evaluation value is not limited to the method of calculating from the depth map, and the evaluation value may be calculated from the form of polygon data including vertices and surfaces to determine the necessity of updating the light emitting unit position.

  FIG. 9 is a block diagram illustrating a detailed internal configuration of the evaluation unit 606. The depth map calculation unit 901 calculates a depth map of the measurement object viewed from the imaging unit 301 based on the normal map calculated by the normal calculation unit 605 by the illuminance difference stereo method.

  The correct answer map calculation unit 902 calculates the correct depth map of the measurement object viewed from the imaging unit 301 based on the correct approximate shape acquired by the correct approximate shape acquisition unit 602.

  The evaluation value calculation unit 903 calculates an evaluation value indicating the degree of similarity between the depth map calculated by the illuminance difference stereo method and the correct answer depth map.

  The determination unit 904 determines the necessity of updating the light emitting unit position based on whether or not the evaluation value calculated by the evaluation value calculation unit 903 satisfies a preset evaluation criterion. When the evaluation value does not satisfy the evaluation standard, it is determined that the light emitting unit position needs to be updated, and when the evaluation value satisfies the evaluation standard, it is determined that the light emitting unit position needs not be updated. When it is determined that the light emitting unit position needs to be updated, the light emitting unit position updating unit 607 performs the light emitting unit position updating process. On the other hand, if it is determined that it is not necessary to update the position of the light emitting unit, the normal map calculation process is terminated, and the normal map is displayed by the output unit 608 or output to an external memory.

  FIG. 10 is a flowchart showing a processing flow of the evaluation unit 606 configured as described above. In step S <b> 1001, the depth map calculation unit 901 integrates the normal map of the measurement target calculated by the illuminance difference stereo method in the normal calculation unit 605, thereby obtaining the depth map of the measurement target viewed from the imaging unit 301. obtain. At this time, since the normal map of the measurement object has a normal for each pixel of the captured image, a high-resolution depth map can be obtained by integrating as it is. Therefore, in this step, the depth map obtained by integrating the normal map is further subjected to a smoothing process using a filter such as a Gaussian filter to calculate a depth map that matches the resolution of the approximate shape. The method for calculating the depth map according to the resolution of the approximate shape is not limited to the above method. For example, in this step, the normal map of the measurement object may be smoothed and then integration may be performed.

In step S1002, the correct depth map calculation unit 902 corrects the depth map when the virtual camera and the correct approximate shape are arranged in the virtual space so as to reproduce the relative positional relationship between the imaging unit 301 and the measurement object 204 at the time of imaging. Calculated as a depth map. Specifically, the relative positional relationship between the imaging unit 301 and the measuring object 204, the relative position P _chart of the imaging unit 301 for chart 203 obtained in step _{S703, cam} and the relative position of the measurement object 204 with respect to the chart 203 P _{chart , obj} (u, v). The correct depth map calculated in this step and the depth map calculated in step S1001 have the same resolution and size, and the depth of pixels having the same coordinates represents the depth of the same region on the measurement object. And

  In step S1003, the evaluation value calculation unit 903 calculates an evaluation value representing the similarity between the depth map calculated based on the normal map obtained by the illuminance difference stereo method and the correct depth map. In this embodiment, a PSNR (Peak Signal to Noise Ratio) value is calculated by the following formula (9) as an evaluation value.

The higher the PSNR value, the higher the similarity between the depth map obtained by the illuminance difference stereo method and the correct depth map. The evaluation value is not limited to the PSNR value, and may be any value that represents the similarity between the depth map obtained by the illuminance difference stereo method and the correct answer depth map. For example, the evaluation value may be lower as the similarity is higher.

  Subsequently, in step S1004, the determination unit 904 determines whether or not the light emitting unit position needs to be updated by comparing the evaluation value calculated in step S1003 with a preset reference value TR. Specifically, when the reference value TR is set in advance and the PSNR value calculated by the equation (9) is equal to or less than the reference value TR, the similarity is low and the light emitting unit position needs to be updated. Determination is made, and the process proceeds to step S707. On the other hand, if the PSNR value is greater than the reference value TR, it is determined that the light emitting unit position does not need to be updated because the degree of similarity is high, and the process moves to step S708.

  As described above, in this embodiment, the correct approximate shape of the measurement object is acquired in advance, and the difference in illuminance is increased until the similarity between the approximate shape calculated from the normal measured by the illuminance difference stereo method and the correct approximate shape increases. The light emitting unit position used for the stereo method is repeatedly updated. Thereby, even when the positional relationship between the light emitting unit and the measurement object cannot be measured with high accuracy, the normal can be measured with high accuracy.

  In this embodiment, the image processing unit 411 is described as being included in the imaging device 201. However, the present invention is not limited to this, and the image processing unit 411 may be included in another device that can communicate with the imaging device 201. Good. Alternatively, the image processing unit 411 may be configured as an image processing device that can communicate with the imaging device 201.

[Example 2]
In the first embodiment, the method of updating the position of the light emitting unit randomly or regularly as the update process of the light emitting unit position has been described. In this embodiment, a method of updating the light emitting unit position according to the distribution of the difference between the depth map calculated from the normal measured by the illuminance difference stereo method and the depth map obtained from the correct approximate shape will be described.

  The operations of the evaluation value calculation unit and the light emitting unit position update unit in the present embodiment are different from the operations of the evaluation value calculation unit 903 and the light emitting unit position update unit 607 in the evaluation unit 606 described in the first embodiment. Hereinafter, operations of the evaluation value calculation unit and the light emitting unit position update unit in the second embodiment will be described in detail. In the description of the present embodiment, the description of the same parts as those of the first embodiment will be omitted or simplified.

  FIG. 11 is a block diagram showing the internal configuration of the image processing unit in this embodiment. In this embodiment, the evaluation unit 1101 obtains an approximate shape from the normal map of the measurement object calculated by the normal calculation unit 605, and obtains an evaluation value representing the similarity to the correct approximate shape acquired by the correct approximate shape acquisition unit 602. calculate. Based on the calculated evaluation value, the necessity of updating the light emitting portion position used for calculating the normal is determined. Further, the evaluation unit 1101 calculates the difference between the approximate shape obtained from the normal map of the measurement object and the correct approximate shape, and passes the difference to the light emitting unit position update unit 1102. When it is determined that the light emitting unit position used for calculating the normal line needs to be updated, the light emitting unit position updating unit 1102 determines the light emitting units 302 to 304 based on the difference in shape calculated by the evaluation unit 1101. Update position.

  FIG. 12 is a flowchart illustrating a processing flow of the image processing unit in the present embodiment. In FIG. 12, step S706 and step S707 are changed to step S1201 and step S1202 from FIG. 7 showing the processing flow of the image processing unit of the first embodiment. Hereinafter, step S1201 and step S1202 will be described.

  In step S1201, the evaluation unit 1101 calculates an evaluation value and a difference between the shapes from the approximate shape obtained from the normal calculated in step S705 and the correct approximate shape (). Then, it is determined whether or not the evaluation value satisfies a preset evaluation criterion. If the evaluation value does not satisfy the criterion, it is determined that the light emitting unit position needs to be updated, and the process proceeds to step S1202. . If the evaluation value satisfies the criterion, it is determined that the light emitting unit position does not need to be updated, and the process proceeds to step S708.

  FIG. 13 is a diagram illustrating a detailed internal configuration of the evaluation unit 1101 in the present embodiment. Of the internal configuration of the evaluation unit 1101, the evaluation value calculation unit 1301 whose operation is different from that of the first embodiment will be described. The evaluation value calculation unit 1301 calculates an evaluation value representing the similarity between the two depth maps from the depth map calculated from the normal map and the correct depth map calculated from the correct approximate shape, and the two depth maps. The difference is calculated. Then, the calculated evaluation value is passed to the determination unit 904, and the depth map difference is passed to the light emitting unit position update unit 1102. Here, since the calculation method of the evaluation value representing the similarity of the depth map is the same as that of the first embodiment, the description thereof is omitted. The difference between the depth maps is calculated by performing a process for calculating the difference between the depths of the pixels having the same coordinates in the two depth maps for all the pixels. Note that the difference may be simply calculated or the absolute value of the difference may be calculated.

  If it is determined in step S1202 that the light emitting unit position needs to be updated, the light emitting unit position updating unit 1102 updates the positions of the light emitting units 302 to 304 used for calculating the normal based on the difference in the depth map. To do.

  Here, the relationship between the light emitting unit position and the distribution of the difference between the depth maps will be described. If an accurate normal map without error can be measured by the illuminance difference stereo method, the depth difference should be zero in all coordinates. However, an error occurs in the measured normal line due to a variation in luminance value due to noise in the captured image and an error in the light source direction due to the fact that the position of the light emitting unit has not been correctly acquired. As a result, a difference between the approximate shape calculated from the normal and the correct approximate shape is generated. Here, since the luminance value of the noise of the captured image generally varies randomly, a normal error due to the influence of the luminance value also randomly occurs. That is, the difference in the depth map due to the noise of the captured image is randomly generated without bias. On the other hand, in the case of an error in the light source direction used for calculating the normal, an error occurs in the normal depending on the direction in which the error in the light source direction occurs. Therefore, a distribution corresponding to the direction of the error in the light source direction also occurs in the difference in the depth map.

  For example, when each light emitting unit position used for calculating the normal has an error in the X direction with respect to the true position, there is a variation in the X direction between the depth map calculated from the normal and the correct depth map. Although large, there is almost no variation in the Y direction. At this time, as the error in the X direction with respect to the true position of each light emitting unit position increases, the amount of variation in the X direction between the depth maps also increases. Similarly, when each light emitting unit position used for normal calculation has an error in the Y direction with respect to the true position, the amount of fluctuation in the Y direction is between the depth map calculated from the normal and the correct depth map. However, the fluctuation in the X direction hardly occurs. In addition, as the error in the Y direction with respect to the true position of each light emitting unit position increases, the amount of variation in the Y direction between the depth maps also increases. When there is an error in the Z direction with respect to the true position of each light emitting unit used for calculating the normal, the distance between the depth map calculated from the normal and the correct depth map is either the X direction or the Y direction. Almost no change in direction. Therefore, it is possible to determine in which direction the position of each light emitting unit used for calculating the normal line should be updated by looking at the direction in which the variation amount between the depth maps is large.

  In the present embodiment, the X direction statistic and the Y direction statistic are calculated as the X direction variation amount Vx and the Y direction variation amount Vy for the distribution of the difference in the depth map. By comparing the calculated statistic in the X direction and the statistic in the Y direction, it is determined in which direction the position of each light emitting unit used for calculating the normal line should be updated. For example, variance can be used as the statistic. Specifically, assuming that the distribution of the difference in the depth map is d (n, m), the variation amount Vx in the X direction can be calculated as in the following equation (10).

  Similarly, the fluctuation amount Vy in the Y direction can be calculated as in the following equation (11).

  Then, by comparing the magnitudes of the fluctuation amount Vx and the fluctuation amount Vy, it is possible to determine which axis of X, Y, and Z is to update each light emitting unit. Specifically, when the variation amount Vx in the X direction and the variation amount Vy in the Y direction are substantially equal (the difference is equal to or less than a predetermined value), each light emitting unit is updated along the Z axis (the position of each light emitting unit). Update the Z component of the information). When the difference exceeds a predetermined value, each light emitting unit is updated along the X axis when the amount of variation Vx in the X direction is larger than the amount of variation Vy in the Y direction (the X component of the position information of each light emitting unit is updated). To do). On the other hand, when the variation amount Vy in the Y direction is larger than the variation amount Vx in the X direction, each light emitting unit is updated along the Y axis (the Y component of the position information of each light emitting unit is updated). Note that, when the magnitudes of the fluctuation amount Vx and the fluctuation amount Vy are compared, the ratio of the fluctuation amounts Vx and Vy may be used. Specifically, when the ratio between the fluctuation amounts Vx and Vy is within a predetermined range, it is determined that both are substantially equal, and each light emitting unit is updated along the Z axis. When the calculated ratio is larger than the maximum value in the above range, it is determined that the variation amount Vx in the X direction is large, and each light emitting unit is updated along the X axis. When the calculated ratio is smaller than the minimum value in the above range, it is determined that the fluctuation amount Vy in the Y direction is large, and each light emitting unit is updated along the Y axis.

  In addition, after determining which X, Y, and Z axis each light emitting unit position is updated by the above method, the following method is used to determine whether to update each axis in a positive direction or a negative direction. can do. That is, the normal map for the light emitting portion position updated by the same amount in both the positive and negative directions of each axis is calculated. The fluctuation amount Vx, Vy or PSNR value is calculated again for each calculated normal map. The direction is determined as a direction in which the recalculated fluctuation amounts Vx and Vy are decreased or a direction in which the recalculated PSNR value is increased.

  In the above example, the dispersion in the X direction and the dispersion in the Y direction are calculated and used as the fluctuation amounts Vx and Vy for the distribution d (n, m) of the depth map difference as in the expressions (10) and (11). However, the index representing the fluctuation is not limited to this, and other indices may be used.

  Moreover, although the method for determining the direction in which each light emitting unit is updated has been described above, the update amount of each light emitting unit position may be determined from the variation amounts Vx and Vy. For example, the update amount C of each light emitting unit can be determined as shown in the following equation (12) using a preset coefficient α. Here, max (Vx, Vy) returns the larger value of Vx and Vy.

  As described above, in the present embodiment, when the light emitting unit position is updated according to the difference distribution between the depth map calculated from the normal measured by the illuminance difference stereo method and the depth map obtained from the correct approximate shape. Determine direction and amount. That is, the direction and amount when the light emitting unit position is updated are determined so that the similarity between the depth map calculated from the normal line and the depth map calculated from the correct approximate shape increases. This makes it possible to reduce the amount of calculation required to obtain a high-precision normal.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

604 Light source direction calculation unit 605 Normal calculation unit 606 Evaluation unit 607 Light emitting unit position update unit

Claims (12)

Based on the illuminance difference stereo method, the measurement target is based on three or more images captured in a state where the measurement target is sequentially irradiated from each of three or more light sources having different irradiation directions and the position information of the light source. A calculation means for calculating a normal map that holds the normal of the surface of the object for each pixel;
By calculating the approximate shape of the measurement object based on the normal map calculated by the calculation means, and comparing the calculated approximate shape with the reference approximate shape of the measurement object acquired in advance, Determination means for determining whether or not the position information of the light source needs to be updated;
An update unit that updates the position information of the light source when the determination unit determines that the position information of the light source needs to be updated, and the calculation unit is updated by the update unit. An image processing apparatus that recalculates the normal map using position information of the light source.   The normal map has the same resolution as the resolution of the image, and the reference approximate shape and the calculated approximate shape have a resolution lower than the resolution of the image. The image processing apparatus described.   The determination unit calculates an evaluation value representing a degree of similarity between the calculated approximate shape and the reference approximate shape. If the calculated evaluation value does not satisfy a predetermined evaluation criterion, the position information of the light source is calculated. The image processing apparatus according to claim 1, wherein the image processing apparatus determines that updating is necessary.   The determination means calculates a first depth map as the approximate shape based on the normal map, calculates a second depth map from the reference approximate shape, and calculates the calculated first depth map and the calculated depth map The image processing apparatus according to claim 3, wherein the evaluation value representing the degree of similarity is calculated with respect to a second depth map.   The determination unit calculates a PSNR value of a difference between the first depth map and the second depth map as the evaluation value, and when the calculated PSNR value is equal to or less than a predetermined reference value, the light source The image processing apparatus according to claim 4, wherein it is determined that the position information of the image needs to be updated.   The image processing apparatus according to claim 4, wherein the update unit updates position information of the light source according to a difference distribution between the first depth map and the second depth map. . The position information of the light source includes an X component, a Y component, and a Z component,
The updating means determines which component in the position information of the light source based on the statistics in the X direction and the statistics in the Y direction calculated for the difference distribution between the first depth map and the second depth map. The image processing apparatus according to claim 6, wherein it is determined whether or not to update. The updating means includes
When the statistics in the X direction and the statistics in the Y direction are substantially equal, the Z component in the position information of the light source is updated; otherwise,
When the statistic in the X direction is larger than the statistic in the Y direction, the X component in the position information of the light source is updated;
The image processing apparatus according to claim 7, wherein the Y component in the position information of the light source is updated when the statistics in the Y direction is larger than the statistics in the X direction.   The updating unit updates the position information of the light source based on a statistic in the X direction and a statistic in the Y direction calculated for the difference distribution between the first depth map and the second depth map. The image processing apparatus according to claim 6, wherein an amount is determined.   The image processing apparatus according to claim 7, wherein the statistic is variance. Based on the illuminance difference stereo method, the measurement target is based on three or more images captured in a state where the measurement target is sequentially irradiated from each of three or more light sources having different irradiation directions and the position information of the light source. A calculation step of calculating a normal map holding the normal of the surface of the object for each pixel;
By calculating the approximate shape of the measurement object based on the normal map calculated in the calculation step, and comparing the calculated approximate shape with the reference approximate shape of the measurement object acquired in advance, A determination step of determining whether or not the position information of the light source needs to be updated;
An update step of updating the position information of the light source when it is determined in the determination step that the position information of the light source needs to be updated, and
In the calculating step, when the position information of the light source is updated in the updating step, the normal map is recalculated using the updated position information of the light source.   The program for functioning a computer as an image processing apparatus of any one of Claims 1 thru | or 10.

Images