JP2022108177A - Image processing apparatus - Google Patents

Abstract

To provide an image processing apparatus using an image data composition method that can compose a plurality of pieces of image data acquired by an imaging unit at different positions in an imaging direction accurately in a short time.SOLUTION: An inspection device 10 being an image processing apparatus has a camera unit 21 and a control unit 30. The control unit 30 changes relative positions in an imaging direction of the camera unit 21 and an inspected body 12 to acquire at least two pieces of image data picked up by the camera unit 21 at different positions in the imaging direction, and executes composite image update processing of determining, for every image data, if pixels of the image data satisfy a condition for a focused pixel, and when determining that the pixels satisfy the condition, selecting values of the pixels as values of corresponding pixels of composite image data.SELECTED DRAWING: Figure 4

Description

The present invention relates to an image processing apparatus using a method for synthesizing image data.

In recent years, electronic circuit boards have come to be mounted on various devices. From this point of view, it is required to increase the mounting density of the electronic circuit board. In such an inspection device for inspecting the state of solder applied to electronic circuit boards and the state of mounting of electronic components, a plurality of light patterns with different periods are used to expand the measurement range and shorten the measurement time. A three-dimensional measuring apparatus based on a phase shift method has been proposed (see, for example, Patent Document 1).

JP 2017-003513 A

In the above method, there is a limit to expansion of the measurement range due to the depth of field of the imaging unit. On the other hand, it is known that a plurality of pieces of measurement data can be pasted together by measuring from a plurality of imaging positions and performing coordinate conversion according to the positional relationship. However, there is a problem that simply synthesizing the measurement results calculated from a plurality of imaging positions causes many measurement errors especially near the joints of the synthesis.

SUMMARY OF THE INVENTION The present invention has been made in view of such problems. It is an object of the present invention to provide an image processing apparatus using an image data synthesizing method capable of

In order to solve the above-described problems, an image processing apparatus according to the present invention includes an imaging unit capable of changing a relative position in an imaging direction with respect to an object to be inspected, and a control unit, wherein the control unit controls the imaging At least two pieces of image data captured by the imaging unit at different positions in the imaging direction are acquired by changing the relative position of the imaging unit and the object to be inspected, and the image is obtained for each of the image data. It is determined whether or not the pixel of the data satisfies the condition of the pixel in focus, and if it is judged that the pixel satisfies the condition, the value of the pixel is changed to the value of the corresponding pixel of the synthesized image data. Executes the composite image update process selected as . Note that the focal length of the imaging unit is fixed.

In such an image processing apparatus according to the present invention, the control unit executes height basic processing for acquiring height information of the object for each pixel of the image data from the image data, and performs the synthesis In the image updating process, it is preferable to determine whether or not the pixel satisfies the condition based on the height information and the relative position between the object to be inspected and the imaging unit.

Further, the image processing apparatus according to the present invention has a projection unit for projecting, onto the object to be inspected, a fringe pattern in which a pattern that changes so as to monotonously increase or decrease in phase is repeatedly formed, and the control unit is configured to obtain the height information from the image data obtained by projecting the stripe pattern by the projection unit in the height basic processing, and the height information includes the imaging unit and Information on the calculated distance from the object to be inspected is included, and in the image synthesis update process, information on the calculated distance, information obtained in the process of calculating the height information from the image data, and the object to be inspected It is preferable to determine whether or not the pixel satisfies the condition based on the relative position between the pixel and the imaging unit.

Further, in the image processing apparatus according to the present invention, the image data obtained when the imaging unit is located at different positions includes image data when the imaging unit is focused on a predetermined reference plane on the object to be inspected. wherein, in the composite image updating process, the control unit receives information about the amplitude of pixels of the image data to be determined, and the image when the imaging unit is focused on the predetermined reference plane. It is preferable to determine whether or not the pixel satisfies the condition by comparing the information about the amplitude of the pixel of the data.

Further, in the image processing apparatus according to the present invention, the control unit, in the height basic processing, for the image data when the imaging unit is focused on the predetermined reference plane, determines the number of measurement points The height information is acquired so as to maximize the measurement error for the image data when the imaging unit is at a position different from the position where the imaging unit is focused on the predetermined reference plane. Preferably, the height information is obtained so as to exclude pixels with high suspicion.

Further, the image processing apparatus according to the present invention further includes an illumination unit for projecting illumination light onto the object to be inspected, and the projection unit projects the stripe pattern onto the image data, and the image data is captured by the imaging unit. and two-dimensional image data acquired by the imaging unit after the illumination light is projected by the illumination unit, the control unit performing the height basic processing in the pattern The height information is acquired from the image data, and when it is determined that the pixel satisfies the condition in the synthetic image update process, the height information of the pixel of the pattern image data is transferred to the synthetic image of the height map. It is preferable to select the value of the corresponding pixel of the data, and select the value of the corresponding pixel of the two-dimensional image data as the value of the corresponding pixel of the synthesized image data of the two-dimensional image data.

Further, in the image processing apparatus according to the present invention, the control unit performs height post-processing such as unmeasured pixel interpolation processing and distortion correction processing on the composite image data of the height map synthesized by the composite image update processing. is preferably performed.

Further, in the image processing apparatus according to the present invention, the control unit, in the composite image updating process, uses two or more pieces of image data of the same pixel when the imaging unit is located at different positions in the imaging direction, and the condition When it is determined that the pixel satisfies Select is preferred.

Further, in the image processing apparatus according to the present invention, the control unit acquires the image data in order from the position of the imaging unit farther from the object to be inspected, and each time the image data is acquired, the synthetic image is obtained. Preferably, an update process is performed.

Further, in the image processing apparatus according to the present invention, the control unit acquires image data when the imaging unit is focused on a predetermined reference plane on the object to be inspected, and then acquires image data of the object to be inspected. It is preferable to acquire the image data from the side where the position of the imaging unit is far from the .

Further, in such an image processing apparatus, the control unit controls the image data captured at another position based on the image data when the imaging unit is focused on the predetermined reference plane. It is preferable to determine whether or not the pixel satisfies the condition of .

Further, in the image processing apparatus according to the present invention, the control unit acquires image data from the side where the imaging unit is positioned farther from the object to be inspected, and finally the imaging unit is aligned with the predetermined reference plane. It is preferable to acquire image data when in focus.

According to the present invention, an image data synthesizing method capable of synthesizing a plurality of image data acquired by imaging units at different positions in the imaging direction (Z direction) with high accuracy and in a short time is used. An image processing device can be provided.

It is an explanatory view for explaining composition of an inspection device containing an image processing device. FIG. 4 is an explanatory diagram for explaining the relationship between the phase of the reference plane and the measured phase; FIG. 4 is an explanatory diagram for explaining an imaging position of a camera unit; 4 is a flow chart showing the flow of a method for synthesizing image data; FIG. 10 is an explanatory diagram for explaining the correspondence relationship between the range of condition (1) at each imaging position and the synthesized image data;

Preferred embodiments of the present invention are described below with reference to the drawings. First, the configuration of an inspection apparatus 10 according to this embodiment will be described using FIG. This inspection apparatus 10 is an apparatus for inspecting an object 12 to be inspected using image data (two-dimensional image data or pattern image data) of the object 12 to be inspected obtained by imaging the object 12 to be inspected. Therefore, this inspection apparatus 10 also has a function as an image processing apparatus. The device under test 12 is, for example, an electronic circuit board on which components are mounted and solder is applied.

The inspection apparatus 10 includes an inspection table 14 for holding an object 12 to be inspected, an imaging unit 20 for illuminating and imaging the object 12 to be inspected, an XY stage 16 for relatively moving the imaging unit 20 with respect to the inspection table 14, and a control unit 30 for controlling the operations of the imaging unit 20 and the XY stage 16 and executing the inspection of the inspection object 12 . For convenience of explanation, as shown in FIG. 1, the inspection object arrangement surface of the inspection table 14 is assumed to be the XY plane, and the direction perpendicular to the arrangement surface (that is, the imaging direction by the camera unit 21 constituting the imaging unit 20 (camera unit The optical axis direction)) of the optical system 21 is defined as the Z direction.

The imaging unit 20 is attached to a moving table (not shown) of the XY stage 16 and is movable in the X and Y directions by the XY stage 16 . The XY stage 16 is, for example, a so-called H-type XY stage. Therefore, the XY stage 16 includes a Y drive unit that moves the moving table in the Y direction along a Y direction guide extending in the Y direction, and supports the Y direction guide at both ends thereof so that the moving table and the Y direction guide move in the X direction. two X-direction guides and an X-drive configured to be movable. The XY stage 16 further includes a Z movement mechanism that moves the imaging unit 20 in the Z direction. Further, a rotation mechanism for rotating the imaging unit 20 may be further provided. The inspection apparatus 10 may further include an XY stage that allows the inspection table 14 to move. In this case, the XY stage 16 that moves the imaging unit 20 may be omitted. A linear motor or a ball screw can be used for the X driving section and the Y driving section. Also, instead of moving the imaging unit 20 in the Z direction, the XY stage 16 may be moved in the Z direction.

The imaging unit 20 includes a camera unit 21 that captures an image from the direction (Z direction) perpendicular to the inspection surface (substrate surface) of the object to be inspected 12 , an illumination unit 22 , and a projection unit 23 . In the inspection apparatus 10 according to this embodiment, the camera unit 21, the illumination unit 22, and the projection unit 23 may be configured as an integrated imaging unit 20. FIG. In this integrated imaging unit 20, the relative positions of the camera unit 21, the illumination unit 22, and the projection unit 23 may be fixed, or each unit may be configured to be relatively movable. Moreover, the camera unit 21, the illumination unit 22, and the projection unit 23 may be separated and configured to be separately movable.

The camera unit 21 includes an imaging device that generates a two-dimensional image of an object, and an optical system (for example, a lens) for forming an image on the imaging device. This camera unit 21 is, for example, a CCD camera. The maximum field of view of the camera unit 21 may be smaller than the inspection object placement area of the inspection table 14 . In this case, the camera unit 21 captures the entire inspection object 12 by dividing it into a plurality of partial images. The control unit 30 controls the XY stage 16 so that the camera unit 21 is moved to the next imaging position each time the camera unit 21 captures a partial image. The control unit 30 synthesizes the partial images to generate an overall image of the inspected object 12 .

Note that the camera unit 21 may include an imaging device that generates a one-dimensional image instead of the two-dimensional imaging device. In this case, by scanning the object 12 to be inspected by the camera unit 21, the entire image of the object 12 to be inspected can be obtained.

The illumination unit 22 is configured to project illumination light for imaging by the camera unit 21 onto the surface of the inspection object 12 . The illumination unit 22 comprises one or more light sources that emit light of a wavelength or wavelength range selected from the wavelength range detectable by the imaging element of the camera unit 21 . The illumination light is not limited to visible light, and ultraviolet light, X-rays, or the like may be used. When multiple light sources are provided, each light source is configured to project light of different wavelengths (eg, red, blue, and green) onto the surface of the inspected object 12 at different projection angles.

The inspection object 12 illuminated by the illumination unit 22 is imaged by the camera unit 21 . The inspection apparatus 10 performs the inspection based on the image data (this image data is referred to as "two-dimensional image data") of the object 12 illuminated and imaged by the illumination unit 22 and a height map to be described later. The presence or absence of defects on the substrate of the body 12 (for example, the presence or absence of parts and whether the arrangement is appropriate, or whether the solder coating state is good or bad) is determined.

In the inspection apparatus 10 according to this embodiment, the illumination unit 22 is a side illumination source that projects illumination light obliquely onto the inspection surface of the object 12 to be inspected. It has a lower light source 22b and a lower light source 22c. In addition, in the inspection apparatus 10 according to the present embodiment, the side illumination sources 22a, 22b, and 22c are ring illumination sources, respectively, which surround the optical axis of the camera unit 21 and are oblique to the inspection surface of the object 12 to be inspected. is configured to project the illumination light onto the Each of these side illumination sources 22a, 22b, 22c may be configured by a plurality of light sources arranged in an annular shape. Also, the upper light source 22a, the intermediate light source 22b, and the lower light source 22c, which are side illumination sources, are configured to project illumination light at different angles to the inspection surface.

The projection unit 23 projects a pattern onto the inspection surface of the object 12 to be inspected. The object 12 onto which the pattern is projected is imaged by the camera unit 21 . The inspection apparatus 10 creates a height map of the inspection surface of the object to be inspected based on the captured image data of the object to be inspected 12 (this image data is called “pattern image data”). Here, the height map is data having height information of an object to be inspected for each pixel of pattern image data. The control unit 30 detects local discrepancies in the pattern image data with respect to the projection pattern, and acquires height information of the site based on the local discrepancies. In other words, the change in the captured pattern with respect to the projected pattern corresponds to the change in height on the inspection surface.

The projected pattern is preferably a one-dimensional striped pattern in which bright lines and dark lines are alternately and periodically repeated. The projection unit 23 is arranged to project a stripe pattern onto the inspection surface of the object 12 to be inspected from an oblique direction. A height discontinuity on the inspection surface of the inspection object 12 appears as a pattern shift in the stripe pattern image. Therefore, the height difference can be obtained from the amount of pattern deviation. For example, the control unit 30 creates a height map by a PMP (Phase Measurement Profilometry) method using a fringe pattern whose brightness changes according to a sine curve. In the PMP method, the shift amount of the fringe pattern corresponds to the phase difference of the sine curve.

The projection unit 23 includes a pattern forming device, a light source device for illuminating the pattern forming device, and an optical system for projecting the pattern onto the inspection surface of the inspection object 12 . The pattern forming device may be, for example, a variable patterning device that can dynamically generate a desired pattern, such as a liquid crystal display, or a fixed patterning device in which a pattern is fixedly formed on a substrate such as a glass plate. It may be a device. When the pattern forming device is a stationary patterning device, the pattern projection position can be made variable by providing a moving mechanism for moving the stationary patterning device or by providing an adjusting mechanism in the optical system for pattern projection. preferable. Also, the projection unit 23 may be configured to be switchable between a plurality of fixed patterning devices having different patterns.

A plurality of projection units 23 may be provided around the camera unit 21 . The plurality of projection units 23 are arranged to project patterns onto the inspection object 12 from different projection directions. By doing so, it is possible to reduce the area where the pattern is not projected due to the shadow due to the height difference on the inspection surface.

The control unit 30 shown in FIG. 1 controls the entire apparatus, and is realized by the CPU, memory, and other LSIs of any computer as hardware, and is loaded into the memory as software. Although it is realized by programs, etc., functional blocks realized by their cooperation are drawn here. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

FIG. 1 shows an example of the configuration of the control unit 30. As shown in FIG. The control unit 30 includes an inspection control section 31 and a memory 35 that is a storage section. The inspection control section 31 includes an image processing section 32 , an inspection data processing section 33 and an inspection section 34 . Further, the image processing section 32 includes an imaging processing section 32a, a height measuring section 32b, and an image synthesizing section 32c. The inspection apparatus 10 also includes an input unit 36 for receiving input from a user or another device, and an output unit 37 for outputting inspection-related information. are each connected to the control unit 30 . The input unit 36 includes, for example, input means such as a mouse and keyboard for receiving input from the user, and communication means for communicating with other devices. The output unit 37 includes known output means such as a display and a printer.

The inspection control unit 31 is configured to execute various control processes for inspection based on input from the input unit 36 and inspection-related information stored in the memory 35 . The inspection-related information includes two-dimensional image data of the object to be inspected 12, a height map of the object to be inspected 12 (calculated from pattern image data), and substrate inspection data. Prior to the inspection, the inspection data processing unit 33 creates substrate inspection data using the two-dimensional image data and height map of the object to be inspected 12 guaranteed to pass all inspection items. The inspection unit 34 performs an inspection based on the created substrate inspection data and the two-dimensional image data and height map of the inspection object 12 to be inspected.

Board inspection data is inspection data created for each type of board. The board inspection data is a collection of inspection data for each component arranged on the board, its position, and solder applied to the board. The inspection data for each component and solder includes the inspection items required for that component and solder, the inspection window that is the inspection area on the image for each inspection item, and the inspection criteria that serve as the criteria for placement and pass/fail judgment for each inspection item. include. One or more inspection windows are set for each inspection item. For example, in an inspection item for judging the quality of the solder coating state, usually, the same number of inspection windows as the number of solder coating regions of the component are set in a layout corresponding to the layout of the solder coating regions. In addition, for an inspection item that uses an image of an object to be inspected that has been subjected to predetermined image processing, the inspection data also includes the details of the image processing.

The inspection data processing unit 33 sets each item of the inspection data according to the board as the board inspection data creation process. For example, the inspection data processing unit 33 automatically sets the position and size of each inspection window for each inspection item so as to match the solder layout of the board. The inspection data processing unit 33 may accept user input for some items of the inspection data. For example, the inspection data processing unit 33 may accept tuning of inspection criteria by the user. Inspection criteria may be set using height information.

The inspection control unit 31 performs imaging processing of the inspected object 12 by the image processing unit 32 as preprocessing for creating substrate inspection data. The object to be inspected 12 that has passed all inspection items is used. As described above, the imaging process is performed by sequentially capturing partial images of the inspection object 12 by controlling the relative movement between the imaging unit 20 and the inspection table 14 while illuminating the inspection object 12 with the illumination unit 22 . . A plurality of partial images are captured so that the entire inspection object 12 is covered. The inspection control unit 31 synthesizes the plurality of partial images to generate board full-surface image data (two-dimensional image data) including the entire inspection surface of the object 12 to be inspected. The inspection control unit 31 stores the board whole surface image data in the memory 35 .

In addition, as preprocessing for creating a height map, the inspection control unit 31 causes the imaging processing unit 32a of the image processing unit 32 to project a pattern onto the object 12 to be inspected by the projection unit 23, and the imaging unit 20 and the inspection table. 14 is controlled, and the pattern image of the object to be inspected 12 is divided and sequentially picked up. The projected pattern is preferably a fringe pattern whose brightness changes according to a sine curve based on the PMP method. The inspection control unit 31 synthesizes the captured divided images to generate pattern image data of the entire inspection surface of the object 12 to be inspected. The inspection control unit 31 stores pattern image data in the memory 35 . Note that the pattern image data may be generated not for the entire inspection surface but for a portion of the inspection surface. Further, as will be described later, the image data captured by the imaging processing unit 32a is combined by the height measuring unit 32b and the image combining unit 32c.

The height measurement unit 32b creates a height map of the entire inspection surface of the inspection object 12 based on the pattern image in the pattern image data. The height measuring unit 32b first obtains a phase difference map of the inspection surface of the inspection object 12 by obtaining the local phase difference between the pattern image data and the reference pattern image data for the entire image data. Here, "reference pattern image data" means image data obtained by projecting a pattern on the reference plane by the projection unit 23 (that is, pattern generated by the pattern forming device incorporated in the projection unit 23 and projected on the reference plane). image data). The height measurement unit 32b creates a height map of the inspection object 12 based on the reference plane serving as a reference for height measurement and the phase difference map. The reference plane is, for example, the substrate surface of the electronic circuit board to be inspected. The reference plane may not necessarily be a flat surface, and may be a curved surface reflecting deformation such as warpage of the substrate.

Specifically, the height measurement unit 32b obtains the phase difference of the stripe pattern between each pixel of the pattern image data and the pixel of the reference pattern image data corresponding to the pixel. The height measurement unit 32b converts the phase difference into height information. Since the distance from the projection unit 23 differs depending on the position on the inspection surface, even if the stripe width of the reference pattern is constant, the stripe width changes from one end to the other end of the pattern projection area on the inspection surface. It is from. The height measurement unit 32b acquires height information from the reference plane based on the converted height information and the reference plane, and creates a height map of the object 12 to be inspected.

The image processing unit 32 of the inspection control unit 31 associates the height information of the height map of the inspection object 12 with each pixel of the two-dimensional image data of the inspection object 12, thereby obtaining an inspection object having a height distribution. You can create an image. Further, the image processing unit 32 may perform three-dimensional modeling display of the inspection object 12 based on the inspection object image data with height distribution. The image processing unit 32 may superimpose the height distribution on the two-dimensional object image data and display it on the output unit 37 . For example, the image data of the object to be inspected may be displayed in different colors depending on the height distribution.

A method for acquiring height information by projecting a pattern using the projection unit 23 in such an inspection apparatus 10 will be described below.

First, a method of obtaining height information by the PMP method using a striped pattern will be described. Focusing on one measurement point (pixel) in the fringe pattern projection area, when the fringe pattern is projected while spatially shifting the phase (in other words, when the fringe pattern is scanned in the direction in which the fringes repeat), the The brightness of the measurement point (pixel) varies periodically. Although the average brightness differs for each measurement point (pixel) depending on the surface characteristics of the measurement point (pixel) such as color and reflectance, the periodic brightness corresponding to the stripe pattern at any measurement point (pixel) Fluctuations occur. Therefore, the phase is calculated from periodic brightness fluctuations, and the phase difference from the initial phase gives height information.

When obtaining height information by the PMP method using such characteristics of the fringe pattern, in principle, it is necessary to shift the phase of the fringe pattern and take images at least three times, typically four times. there is Since the fringe pattern is a sine wave, the brightness variation at each measurement point (pixel) is also a sine wave. Since the pitch of the fringes is known, the sine wave representing the brightness variation can be identified if the mean luminance (brightness), amplitude, and initial phase are known.

By obtaining the measured value of the luminance (brightness) of the measurement point (pixel) from at least three pattern image data captured by shifting the phase start position of the fringe pattern, the average value and amplitude of the luminance (brightness) , and the initial phase can be determined. Let I0, I1, I2, and I3 be the luminances of certain pixels when one stripe pattern is imaged four times with the phase start position shifted by 90 degrees (π/2 radian), respectively. The luminance In in the case of a sine wave is represented by the following formula (a). In this equation (a), A represents the amplitude dependent on the brightness of the object 12 to be inspected, B represents the brightness offset dependent on the camera unit 21 and the environmental brightness, and φ represents the desired phase. , π/2 indicate the amount of phase shift. Note that n=0, 1, 2, 3. However, since errors are generally included due to factors such as sensor noise, when calculating the phase φ, the amplitude A and the offset B, the method of least squares is often obtained. Let E be the error obtained by the method of least squares.

In = A x sin(φ + n x π/2) + B (a)

Calculation of height information is to obtain the phase φ, amplitude A and offset B of each pixel from the pattern image data. For example, when one fringe pattern is imaged four times with the phase start position shifted by 90 degrees (π/2 radians), the phase φ of the measurement point corresponding to each pixel is expressed as the following equation (b). It has been known.

tan(φ) = (I3-I1)/(I2-I0) (b)

Assuming that the pitch of the fringe pattern (the length of one cycle) is P and the initial phase is 0, the shift amount of the fringe pattern is obtained by P×(φ/2π). Height information of the position can be acquired from the amount of pattern deviation using the projection angle of the pattern.

In FIG. 2, as shown in (a), the phase obtained from the pattern image data projected by the projection unit 23 onto the reference plane S and captured by the camera unit 21 is shown in (c), and (b) ), the phase obtained from the pattern image data captured by projecting the same fringe pattern with the object O placed on the reference plane S is shown in (d) as height information of the object O can be calculated from these phase shift amounts (phase difference=measured phase−phase of reference plane). This phase difference can be calculated for each pixel of the image captured by the camera unit 21, and by multiplying the phase difference obtained for each pixel by a coefficient, the height information of the position corresponding to that pixel is calculated. be. This coefficient is simply obtained by, for example, tan (projection angle)×fringe pitch/2π.

Here, the fringe pattern has a sinusoidal brightness, and a pattern that monotonically changes (monotonically increases or monotonously decreases) is repeatedly formed. In one cycle (pitch), when the phase changes from 0 to 2π (radian), the luminance monotonously increases from the darkest state (for example, luminance I = 0) to the brightest state (for example, luminance I = 255). (The relationship between phase and luminance in the pattern may be reversed, ie monotonically decreasing).

As described with reference to FIG. 2, the reference plane S is imaged using the stripe pattern, and the object O (corresponding to the inspected object 12) placed on the reference plane S is imaged using the stripe pattern. In this case, the phase difference Δφ between the phase φ0 of the reference plane S obtained based on the formula (b) and the phase φ obtained with the object O placed is obtained by the following formula (c).

Δφ = φ - φ0 (c)

From the above, it was obtained from the reference plane and at least three pattern image data captured by placing the object O (inspection object 12) on this reference plane and projecting the fringe pattern with the phase starting position shifted. Based on the phase difference Δφ calculated by the formula (c) from the phases φ0 and φ (calculated by the formula (b) when four images are taken with a shift of 90° each), the height information H is obtained by the following formula: (d).

H = P x (Δφ/2π) (d)

As described above, the inspection apparatus 10 according to this embodiment is configured to inspect the object 12 using the height map calculated from the two-dimensional image data and the pattern image data. Since the camera of the unit 21 has a predetermined depth of field (depth of focus), for example, when the camera unit 21 is focused near the surface of the board, which is the inspection object 12, tall parts The top surface may capture out-of-focus image data. Therefore, when inspecting using such image data, in the two-dimensional image data, there may be cases where characters written on the upper surface of a tall component (such as a capacitor) cannot be read. , the image of the striped pattern on the upper surface of the tall component is blurred, and accurate height information cannot be calculated.

Therefore, in the inspection apparatus 10 according to the present embodiment, at least two pieces of image data captured by the camera unit 21 at different positions in the Z direction (imaging direction) are acquired, and relative focus is obtained from each image data. By extracting and synthesizing pixels that are in focus, it is possible to obtain image data that is in focus over the entire field of view even if components with different heights are arranged. Here, the image data acquired when the camera unit 21 is focused on the reference plane (the plane serving as the reference for calculating the height information as described above) and the camera unit 21 in the Z direction. A case will be described in which image data obtained by moving the camera unit 21 at different positions in the Z direction is used for synthesis. A method of synthesizing image data in the inspection apparatus 10, which is an image processing apparatus according to the present embodiment, will be described below. Here, the case of moving the position of the camera unit 21 in the Z direction will be described. You can move it in any direction. As shown in FIG. 3A, a predetermined surface of the device under test 12 (for example, the upper surface of the substrate in the case of an electronic circuit board) is defined as a "reference surface". Here, FIG. 3A shows a case where the reference plane of the object to be inspected 12 and the reference plane match.

In the following description, as shown in FIG. 3A, the position of the camera unit 21 in the Z direction is the position of the camera unit 21 when the camera unit 21 is focused on the reference plane, with Z=0 as the reference. expressed in a coordinate system that Specifically, P(i), which indicates the position of the camera unit 21 in the Z direction, is set to i=0 when the camera unit 21 is in focus on the reference plane, and in the order in which the camera unit 21 moves away from the reference plane. It is expressed as i=1, 2, 3, . . . In the following description, it is assumed that four pieces of image data are acquired at different positions in the Z direction. For example, assuming that the depth of field of the camera unit 21 is 4 mm, image data is acquired at the position "P(0)=0" of the camera unit 21 when the reference plane is in focus, and P( 0)=0, a case where three pieces of image data are acquired at positions shifted upward (in the Z direction) by 4 mm from the position of 0 will be described. The respective positions are expressed as "P(1)=4", "P(2)=8", and "P(3)=12" (the unit mm is omitted). Also, when the camera unit 21 is focused on the reference plane, that is, when the camera unit 21 is at Z=0, the distance from the tip of the camera unit 21 to the reference plane is defined as the focusing distance. Further, a position distant from the tip of the camera unit 21 by a focal distance in the direction of the inspected object 12 is defined as the most focused position FP.

Note that the number of image data in which the position of the camera unit 21 is shifted in the Z direction is not limited to four, and may be two or more. Further, the width by which the camera unit 21 is shifted in the Z direction is not limited to 4 mm. Can be set to any value. Furthermore, instead of changing the position of the camera unit 21 in the Z direction at equal intervals, the intervals at which the respective image data are acquired may be arbitrarily selected.

Further, it is assumed that the position in the Z direction at which the camera unit 21 acquires image data is set in the control unit 30 in advance from the input unit 36 or the like. For example, the position of the reference plane and the height of the object to be inspected (the height of the component mounted on the board) are input, and the positions and intervals in the Z direction for acquiring image data are set. It is also assumed that the camera unit 21 has a fixed focal length.

A method of synthesizing image data in the inspection apparatus 10, which is the image processing apparatus according to the present embodiment, will be described with reference to FIG. In this method of synthesizing image data, among the image data acquired at different positions in the Z direction, the image data acquired at the farthest position from the object to be inspected 12 (reference plane) is subjected to synthesis processing. ing.

As shown in FIG. 4A, when the control unit 30 determines that the process of acquiring the image data of the object 12 to be inspected has started, the imaging processing section 32a moves the camera unit 21 to a predetermined position above the object 12 to be inspected. position (position in the XY direction), and further adjust the position in the Z direction of the camera unit 21 so that the camera unit 21 is positioned at i=0, that is, the camera unit 21 is positioned at P(0). Then, image data of the object to be inspected 12 is obtained (step S100). When the camera unit 21 is moved to the position P(0), the camera unit 21 is focused near the reference plane of the object 12 to be inspected. In step S100, the projection unit 23 projects the striped pattern onto the inspection object 12, and the camera unit 21 captures the pattern image data in order to obtain the height map of the inspection object 12, as described above. At this time, at least three (four if possible) pattern image data are acquired by shifting the phase of the fringe pattern. When a plurality of projection units 23 are provided, the pattern image data is acquired by projecting the fringe pattern onto the inspection object 12 while shifting the phase from each projection unit 23 as described above. Furthermore, when obtaining two-dimensional image data, the side illumination sources 22a, 22b, and 22c of the illumination unit 22 are sequentially turned on to illuminate the inspection object 12, and the camera unit 21 takes an image.

When the control unit 30 finishes imaging the object 12 to be inspected, the pattern image data ("P(0) parameters for removing erroneous measurement values from the height information calculated based on the pattern image data" (step S102). As will be described later, when the camera unit 21 is at the position of i=0 (when P(0)), the image data synthesizing process is executed last. In order to select pixels of the image data at P(0) as pixels of the synthesized image data as much as possible in this final synthesizing process, the above parameters are set to values that maximize the number of measurement points. be done. Then, the height basic processing S200 is executed using this parameter (step S104).

When the control unit 30 starts the height basic processing S200 shown in FIG. Based on the formulas (a) to (d), phase calculation processing is performed for each pixel (within the field of view) in the pattern image data (step 202). Note that when a plurality of projection units 23 are provided, phase calculation processing is executed for each projection unit 23 (for each projection). Next, if the control unit 30 has a plurality of projection units 23, the control unit 30 executes projection synthesis processing for synthesizing the respective phase calculation results, and calculates height information for each pixel (step S204). Finally, noise filter processing using a median filter or the like is performed on the height information for each pixel (step S206), and the execution of the height basic processing S200 for the pattern image of P(0) ends.

In this way, when the camera unit 21 is at the position P(0) first (when the camera unit 21 is focused near the reference plane), the image data is obtained, and the height information is obtained from the pattern image data. is calculated because the amplitude, height information, etc. at P(0) are used in condition (2) used to determine the combining process, as will be described later.

Returning to FIG. 4A, in step S104, when the height basic processing S200 is completed, the control unit 30 causes the imaging processing section 32a to set the position of the camera unit 21 in the Z direction as the next imaging position. It is moved to the farthest position from the object to be inspected 12 among the set imaging positions (step S106). Here, the camera unit 21 is moved to the position of P(3)=12 when i=3.

When the control unit 30 moves the camera unit 21 to the next imaging position in the Z direction, the pattern image data and the two-dimensional image data are captured at that imaging position by the imaging processing unit 32a, as in the case of P(0). Further, parameters are set for removing erroneous measurement values from the height information calculated from the pattern image data (step S108). Further, the control unit 30 causes the height measurement section 32b to perform the height basic processing S200 on the pattern image data captured in step S108 (step S110). Here, since the camera unit 21 is at a position other than i=0 (hollow), the parameter for removing erroneous measurement values from the height information excludes points that are highly suspected to be measurement errors (erroneous measurement in hollow excluding points).

When the height information is calculated from the pattern image data captured at the current imaging position (here, the position of P(3)), the control unit 30 causes the image synthesizing section 32c to generate the image data of the current imaging position. is executed (step S112). Here, among the pixels of the image data at the current imaging position, a focused image (focused pixel) is selected and used as the pixel value of the combined image data. Specifically, for each pixel, based on the height information at the current imaging position of the pixel under the following two equations (conditions (1) and (2)), A determination is made as to whether pixels of the image data are to be selected as composite image data.

P(i)-DL≤H(X,Y,i)+P(i)≤P(i+1)-DH (1)
A(X, Y, i) ≧ A(X, Y, 0)×WA (2)
however,
P(i): Current imaging position P(i+1): Next imaging position farther from the reference plane than the current imaging position H(X, Y, i): Position of pixel to be determined is (X, Y) DL, DH: Constant A(X, Y, i): Amplitude at the current imaging position when the position of the target pixel is (X, Y) A(X, Y, 0): Amplitude at imaging position i=0 when the position of the pixel to be determined is (X, Y) WA: Weighting factor

Condition (1) is to determine whether or not the position (calculated distance) of the inspection object 12 corresponding to the determination target pixel is near the focus of the camera unit 21 at the current imaging position. Here, the calculated distance H(X, Y, i) is the distance in the Z direction from the most focused position FP at P(i) to the inspection object 12 in the determination target pixel. The height information of the pixel at the position (X, Y) obtained in S200 is represented by the coordinate system of the camera unit 21 described above. For example, when P(0)=0, H(X, Y, i)=0 when the inspection object 12 corresponding to the determination target pixel is on the reference plane. As shown in FIG. 3A, when the camera unit 21 moves in the Z direction, the most focused position FP also moves in the Z direction. Also, as shown in FIG. 3B, DL and DH are the current imaging position P(i) of the camera unit 21 and the adjacent imaging position P(i+1) further away from the reference plane than the current imaging position. are constants for determining the range in which focused image data can be obtained by the camera unit 21. For example, when the camera unit 21 is positioned at intervals of 4 mm, DL=1.5 mm and DH=1.5 mm. It can be 0 mm. Of course, the constants DL and DH can be determined according to the specifications of the camera unit 21 (depth of field, etc.) and the interval by which the camera unit 21 is shifted in the Z direction. In general, it is desirable that DL>DH. When the imaging position (here, P(3)=12) farthest from the object 12 (reference plane), the value of P(i+1) (here, P(4)) is set in advance. , this value is used to determine condition (1) (for example, let P(4)=16). From the above, the pixel satisfying the condition (1) is located in the inspection object 12 within the range where the camera unit 21 at the position (P(i)) can obtain focused image data (clear image data). means that

Here, instead of using constants for DL and DH in condition (1), variables as shown below may be used. Note that FUNC indicates a predetermined function, DL(i) is a variable at the imaging position P(i), and the next imaging position P(i−1) and the current imaging position P(i) are parameters. DH(i) is a variable at the imaging position i and is determined by the function FUNC having the current imaging position P(i) and the previous imaging position P(i+1) as parameters. be done.

DL(i)=FUNC(P(i-1), P(i))
DH(i)=FUNC(P(i), P(i+1))

Further, condition (2) is based on the amplitudes A(X, Y, i) and A(X, Y, 0) obtained by the above equation (a) when performing phase calculation using pattern image data. It is determined whether or not the pixel has a large degree. This condition (2) compares the degree of focus, extracts a pixel with a higher degree of focus, and determines that the pixel is in focus when the evaluation value of the pixel is equal to or greater than a threshold. Specifically, the larger the value of the amplitude, the greater the degree of focus. , Y)), when the amplitude A (X, Y, i) at the current imaging position is sufficiently larger than the amplitude A (X, Y, 0) at i=0, the determination target A pixel can be determined to be an in-focus pixel. It should be noted that the value of the weight WA is desirably determined based on the depth of field of the camera unit 21 .

From the above, the condition (1) and the condition (2) are determined for each pixel at the imaging position P(i), and the pixel satisfying the condition (1) and the condition (2) at the same time is determined at the current imaging position P(i). is selected as the pixel of the composite image. Specifically, when the pixel at the position (X, Y) satisfies the conditions (1) and (2), the height information ( H(X, Y, i)+P(i)) is set as the value of the pixel at the same position in the synthesized image data of the height map, and the value of the pixel in the two-dimensional image data captured at the position of P(i) is A value (luminance, etc.) is set as the value of the pixel at the same position in the synthesized image data of the two-dimensional image data. By making determinations using these conditions (1) and (2), it is possible to prevent erroneous measurements, particularly near the joints of the composite.

When the imaging processing section 32a has performed the above synthesis processing for all pixels, the control unit 30 determines whether or not the imaging processing has been completed at all predetermined imaging positions in the Z direction (step S114). ). If it is determined that the imaging process at all imaging positions has not been completed (step S114: No), the control unit 30 causes the imaging processing section 32a to position the camera unit 21 at the next imaging position approaching the reference plane. is set (step S116), and the above-described processing of steps S108 to S114 is repeated at that imaging position. For example, if the current imaging position is P(3)=12, the camera unit 21 is set to the next imaging position P(2)=8, and if the current imaging position is P(2)=8 sets the camera unit 21 to P(1)=4, which is the next imaging position.

On the other hand, if it is determined that the imaging process at all imaging positions has been completed (step S114: Yes), the control unit 30 causes the image synthesizing section 32c to move the camera unit 21 near the reference plane acquired in step S100. Synthetic image update processing of image data (image data of P(0)=0) when is in focus is performed (step S118). Here, since the condition (2) is always satisfied for the image data of P(0), it is determined whether or not the condition (1) is satisfied. That is, in step S118, the pixels for which the value of the composite image data has not yet been selected and which satisfy the condition (1) are the pixel values of the image data at the imaging position of P(0) (height map and the pixel value of the two-dimensional image data) is selected as the pixel value of the combined image data.

In the repeated processing of steps S108 to S114, even if a certain pixel satisfies the conditions (1) and (2), condition (1) is already satisfied at an imaging position farther from the subject 12 than the imaging position. And if the condition (2) is satisfied and the value of the pixel is selected as the value of the composite image data, the value that has already been selected is given priority. That is, for pixels that satisfy the conditions (1) and (2) at a plurality of imaging positions in the Z direction, the value of the pixel at the imaging position farthest from the inspection object 12 is selected. The light emitted by the projection unit 23 is imaged after being reflected by the inspected object 12 . As the camera unit 21 becomes farther from the substrate surface, the amount of reflected light from the inspected object 12 decreases. This leads to a reduction in noise due to unnecessary light, especially in pixels that are not in focus, and as a result, erroneous measurement values included in height information are reduced. Therefore, for pixels that satisfy the conditions (1) and (2) at a plurality of imaging positions in the Z direction, the value of the pixel at the imaging position farthest from the object 12 to be inspected is selected.

FIG. 5 shows the most focused position FP of the camera unit 21 and its position when the object 12 to be inspected is a portion where the components 12a, 12b, and 12c are mounted on the substrate 12d in a certain field of view. It shows the relationship between the captured image data and the synthesized image data. Specifically, the upper part of FIGS. 5A to 5D shows the relationship between the most focused position FP of the camera unit 21 and the components 12a to 12d, and the lower part shows the possibility of being selected as composite image data. A certain portion (pixel) of is indicated by shading. FIG. 5(a) shows when the camera unit 21 is at P(3)=12, and since the upper surface of the component 12a satisfies the condition (1), if the condition (2) is also satisfied, , and as shown in FIG. 5(e), the upper surface of the part 12a of the image data at P(3) is selected as the composite image data. On the other hand, when the camera unit 21 is at P(2)=8 as shown in FIG. )=4, the condition (1) is also satisfied (assuming that the condition (2) is also satisfied). In this case, as shown in FIG. 5E, the upper surface of the component 12b in the image data of P(2) farther from the object 12 is selected as the composite image data. Also, as shown in FIG. 5(c), the upper surface of the component 12c satisfies the condition (1) when the camera unit 21 is at P(1)=4, so the condition (2) is satisfied at the same time. If satisfied, the upper surface of the component 12c in the image data of P(1) is selected as composite image data, as shown in FIG. 5(e). Similarly, the upper surface (reference surface) of the substrate 12d satisfies the condition (1) when the camera unit 21 is at P(0)=0, as shown in FIG. 5(d). If the condition (2) is satisfied, as shown in FIG. 5(e), the upper surface of the substrate 12d in the image data of P(0) is selected as composite image data (see FIG. 5(e)).

Returning to FIG. 4, finally, the control unit 30 executes the height post-processing S300 by the height measuring section 32b (step S120). When the height post-processing S300 shown in FIG. 4C is started, the height measurement unit 32b of the control unit 30 detects unmeasured pixels such as pixels whose values were not selected in the above-described composite image update process in the composite image data. A process of interpolating pixels is executed (step S302), a process of correcting distortion of the synthesized image data is further performed (step S304), the height post-processing S300 is finished, and the synthesis process is finished. If height post-processing is performed on the image data before synthesis (the height map calculated from the pattern image data acquired at each imaging position), the interpolated values and corrected pixels will be lost in the synthesized image data. There is a possibility that it will be selected as a pixel value, and the accuracy of the image data may deteriorate. Therefore, post-height processing such as non-measurement pixel interpolation processing and distortion correction processing is desirably performed on combined image data after combining processing of image data at all imaging positions is completed.

As shown in FIG. 4, in the method of synthesizing image data according to the present embodiment, an image captured when the camera unit 21 is focused near the reference plane (at a position P(0) where i=0) is Except for the data, height basic processing and composite image update processing are executed each time the position of the camera unit 21 is moved in the Z direction and image data is acquired. At this time, in FIG. 4, imaging processing and synthesizing processing of image data acquired in the imaging processing (basic height processing and synthetic image update processing) are described as a series of flows. , the height basic processing and the composite image update processing can be executed in parallel with the image data acquired at the previous imaging position. As described above, in the imaging process, a plurality of pattern image data obtained by shifting the phase of the fringe pattern is acquired for each of the plurality of projection units 23. must be sequentially turned on to acquire a plurality of two-dimensional image data. Therefore, while these imaging processes are being performed at the next imaging position, the height basic processing and composite image update processing of the previous imaging position are performed. By executing in parallel, the overall processing time can be shortened, and as a result, the inspection time can be shortened. At this time, in FIG. 4, after the imaging processing and the basic height processing are executed at the position P(0), the camera unit 21 is moved to P(3), which is the furthest from the object to be inspected 12, and from this imaging position The camera unit 21 is moved in a direction approaching the object 12 to be inspected, and the imaging process, the height basic process, and the composite image update process are executed, and finally the composite image update process of the image data of P(0) is executed. . By executing the imaging process, the basic height process and the composite image update process in this order, the basic height process at the imaging position before the current imaging position where the basic height process and the composite image update process are being executed. And in the composite image update process, when the pixel value of the composite image data has already been selected, the image data of the image pickup position currently undergoing the height basic processing and the composite image update process satisfy the condition (1) and the condition ( 2) is satisfied, the value of the pixel is not selected as the value of the corresponding pixel in the composite image data. When (1) and condition (2) are satisfied, the pixel value of the image data at the imaging position farthest from the object to be inspected 12 is selected as the pixel value of the combined image data. ” can be adapted to the criterion.

Note that the above-described processing is performed for one field of view. When imaging (inspecting) the object 12 to be inspected by dividing it into a plurality of fields of view, the camera unit 21 is moved in the XY directions. Then, the above processing is executed in each field of view.

The inspection apparatus 10 according to the present embodiment performs the inspection as described above based on the synthesized image data (the height map and the two-dimensional image data) acquired by the image processing section 32 of the inspection control section 31 of the control unit 30. to run.

(First Modification of Conditions for Selecting Pixels)
In the composite image updating process described above, the condition (2) is based on the amplitude of the target pixel, but the condition (3) below may be used instead of the condition (2). That is, whether or not to select the pixel of the image data at the current imaging position as the synthesized image data based on the height information of the pixel at the current imaging position according to the conditions (1) and (3) described above. may be configured to determine The contrast C is calculated by C=A/B using the amplitude A and the offset B obtained from the above equation (a).

C(X,Y,i)≧C(X,Y,0)×WC (3)
however,
C(X, Y, i): Contrast at the current imaging position when the position of the pixel to be determined is (X, Y) C(X, Y, 0): The position of the pixel to be determined is (X, Y) contrast at i=0 WC: weighting factor

Condition (3) is for focusing with contrast C(X, Y, i) and C(X, Y, 0) obtained based on the above equation (a) when phase calculation is performed using pattern image data. It is determined whether or not the pixel has a large degree. This condition (3) compares the degree of focus, extracts a pixel with a higher degree of focus, and determines that the pixel is in focus when the evaluation value of the pixel is equal to or greater than a threshold. Specifically, since the greater the contrast value, the greater the degree of focus, the contrast C(X, Y, 0) when i=0 in the determination target pixel (pixel at position (X, Y)) When the contrast C(X, Y, i) at the current imaging position is sufficiently large compared to , it can be determined that the determination target pixel is the in-focus pixel. Note that the value of the weight WC is desirably determined based on the depth of field of the camera unit 21 .

(Modified example of height measurement method)
In the above description, a case has been described in which height information is calculated from three or four pattern image data obtained by projecting one fringe pattern from the projection unit 23 with different phases. In order to improve the accuracy of height information, it is also possible to obtain height information using at least two types of fringe patterns with different periods. For example, in the imaging process of steps S100 and S108 described above, the projection unit 23 projects the long-period first pattern while changing the phase to acquire three or four pattern image data, and also obtains three or four pattern image data. Three or four pieces of pattern image data are acquired by projecting the second pattern while changing the phase. When there are a plurality of projection units 23, each projection unit 23 acquires pattern image data in which the phases of the first pattern and the second pattern are changed. Rough height information (rough height) is obtained from the first pattern with a long period (thick stripes), and precise height information is obtained with a second pattern with a short period (thin stripes). As described above, the PMP method obtains height information based on the phase, so if the height gap (height difference) is large and the fringes are shifted by one period or more, the height information cannot be uniquely specified. By acquiring height information in combination with the first pattern, it is possible to uniquely specify the height information even when the fringes are shifted by one period or more when the second pattern is projected. .

The brightness In obtained from the three or four pattern image data obtained by projecting the first pattern with the phase changed by the projection unit 23 was also obtained by projecting the second pattern with the phase changed. The luminance In obtained from the pattern image data of three or four sheets also has the relationship of the formula (a) described above. Also, the relationship between the luminance In and the phase φ has the relationship of the above-described formula (b).

Here, the phase difference Δφw when the phase of the long-period first pattern is φw and the phase of the reference plane is φw0 has the relationship of the following expression (c1) from the above expression (c). .

Δφw = φw - φw0 (c1)

Similarly, the phase difference Δφf when the phase of the short-period second pattern is φf and the phase of the reference plane is φf0 has the relationship of the following expression (c2) from the above expression (c). there is

Δφf = φf - φf0 (c2)

From these phase differences Δφw and Δφf, detailed height information H can be obtained by the following equation (d'). Here, M is the number of cycles of the second short-cycle pattern with respect to one cycle of the first long-cycle pattern, and Round( ) is a function for rounding to the nearest whole number.

H=Round((M×Δφw−Δφf)/2π)×2π+Δφf(d′)

In the case of a configuration in which height information is calculated using two fringe patterns of a long period and a short period, instead of the determination based on the amplitude shown in the condition (2) described above, as shown in the following condition (2'). It is desirable to make a judgment based on the amplitude ratio. That is, it is desirable to determine whether or not to select the pixels of the image data at the current imaging position as the composite image data based on the conditions (1) and (2') described above.

AR (X, Y, i) ≤ AR (X, Y, 0) x WAR (2')
however,
AR (X, Y, i): Amplitude ratio at the current imaging position when the position of the pixel to be determined is (X, Y) AR (X, Y, 0): The position of the pixel to be determined is (X , Y) at i=0 WAR: weighting factor

Note that the contrast ratio AR (parameters (X, Y, i) are omitted) of a pixel at the position (X, Y) in the imaging position P(i) is calculated based on the following equation (e). be. Here, Aw is the amplitude obtained by the formula (a) from three or four pattern image data obtained by projecting the long-period first pattern, and Af is the short-period second pattern. is the amplitude obtained by the formula (a) from three or four pattern images obtained by projecting .

AR = (Aw-Af)/Af (e)

Similar to the condition (2), the condition (2′) compares the degree of focus and extracts a pixel with a higher degree of focus. It is determined that Specifically, when the camera unit 21 is focused near the reference plane at the determination target pixel (pixel at position (X, Y)) because the smaller the value of the amplitude ratio, the greater the degree of focus. When the amplitude ratio AR (X, Y, i) at the current imaging position is sufficiently smaller than the amplitude ratio AR (X, Y, 0) (when i=0), the pixel to be determined is in focus. It can be determined that it is a pixel. It should be noted that the value of the weight WAR is desirably determined based on the depth of field of the camera unit 21 .

Alternatively, instead of condition (2'), condition (3') shown below may be used for determination. In other words, it may be determined whether or not to select the pixels of the image data at the current imaging position as the composite image data based on the conditions (1) and (3') described above.

CR (X, Y, i) ≤ CR (X, Y, 0) x WCR (3')
however,
CR(X, Y, i): Contrast ratio at the current imaging position when the position of the pixel to be determined is (X, Y) CR(X, Y, 0): The position of the pixel to be determined is (X , Y) at i=0 WCR: weighting factor

Condition (3') determines whether or not the pixel is in focus based on the contrast ratio obtained based on the above equation (a) when the phase is calculated using the pattern image data. This condition (3') compares the degree of focus, extracts a pixel with a higher degree of focus, and determines that the pixel is in focus when the evaluation value of the pixel is equal to or less than a threshold value. Specifically, the smaller the value of the contrast ratio, the greater the degree of focus. When the contrast ratio CR(X, Y, i) at the current imaging position is sufficiently smaller than CR(X, Y, 0), it can be determined that the determination target pixel is in focus.

Note that the contrast ratio CR (parameters (X, Y, i) are omitted) is the contrast Cw obtained from three or four pattern image data obtained by projecting the long-period first pattern. , the contrast Cf obtained from three or four pattern images obtained by projecting the short-period second pattern, and the following equation (f). Aw is the amplitude obtained by the first pattern, Af is the amplitude obtained by the second pattern, Bw is the offset obtained by the first pattern, and Bf is the offset obtained by the second pattern. The value of is obtained from the above equation (a).

CR = (Cw-Cf)/Cf (f)
however,
Cw = Aw/Bw (g)
Cf = Af/Bf (h)

As described above, in the above-described composite image update processing S112, even if pixels that satisfy the conditions (1) and (3′) are selected, the focused pixel is set as the pixel value of the composite image data. be able to.

(Second Modification of Conditions for Selecting Pixels)
In the above-described image data synthesizing method, in the synthetic image updating process S112, if there is only one stripe pattern for obtaining height information, the pixels of the image data captured at the current imaging position are replaced with those of the synthetic image data. In determining whether or not to select a pixel, a pixel that satisfies the conditions (1) and (2) or the condition (1) and the condition (3) at the same time is selected. In the case of two types of short cycles, the configuration was made so as to satisfy the conditions (1) and (2') or the conditions (1) and (3') at the same time, but the conditions (2) and (2') Alternatively, the condition (3) or condition (3') is not determined, and in the image data captured at the current imaging position, when the condition (1) is satisfied, the pixel is selected as the pixel of the composite image data. good too. This is because it is possible to determine whether or not the pixel of the image data at the current imaging position is the in-focus pixel only with condition (1), and the processing time can be further shortened. In the case of such a configuration, the imaging process at the imaging position P(0) when the focus is near the reference plane may be performed last (in other words, the imaging process at the imaging position far from the object 12 to be inspected). may be configured to execute the imaging process, the height basic process, and the composite image update process from (1).

Further, when the imaging process, the height basic process, and the composite image update process are executed from an imaging position far from the object 12 to be inspected, the image when the camera unit 21 is focused near the reference plane (P(0)) Since the data is acquired last, the condition (2a) shown below may be used instead of the condition (2) described above.

A(X, Y, i) ≧ TH (2a)
however,
TH: threshold predetermined from amplitude A

Further, when the above condition (2') is used for determination, the following condition (2a') may be used instead of this condition (2').

AR (X, Y, i) ≤ TH (2a')
however,
TH: threshold value predetermined from amplitude ratio AR

Further, when the condition (3) described above is used for determination, the condition (3a) shown below may be used instead of the condition (3).

C(X, Y, i) ≧ TH (3a)
however,
TH: threshold predetermined from contrast C

Further, when the above condition (3') is used for determination, the following condition (3a') may be used instead of this condition (3').

CR(X,Y,i)≤TH(3a')
however,
TH: threshold value predetermined from contrast ratio CR

Even if conditions (2a), (2a'), (3a), and (3a') are used instead of conditions (2), (2') or conditions (3), (3'), the condition Among the pixels that satisfy (1), it is possible to extract pixels with a higher degree of focus.

(main features and effects)
The main features and effects of the image data synthesizing method according to this embodiment are summarized below.

First, in the image data synthesizing method according to the present embodiment, a plurality of image data captured by moving the camera unit 21 to different positions in the Z direction (imaging direction) with respect to one field of view are synthesized. Image data is acquired, and the composition processing is performed from the image data in which the imaging position of the camera unit 21 is far from the object to be inspected 12 (from the reference plane). With this configuration, as the camera unit 21 becomes farther from the object 12 to be inspected, the amount of light reflected from the object 12 to be inspected decreases. Erroneous measurement values included in height information are reduced.

Secondly, in the method of synthesizing image data according to the present embodiment, in the image data of each imaging position, a focused pixel is selected and used as the pixel value of the synthesized image data. Height information of each pixel measured by the PMP method and amplitude and offset in the PMP method are used as criteria for determining which pixels are in focus. a method of selecting pixels satisfying conditions (1) and conditions (3) at the same time, a method of selecting pixels satisfying only condition (1), conditions (1) and conditions (2′), or A method of selecting pixels that simultaneously satisfy conditions (1) and (3'), and a method of simultaneously satisfying condition (1) and any of conditions (2a), (2a'), (3a), and (3a') There are ways to select satisfying pixels. With this configuration, it is possible to select a pixel that is in focus with high accuracy through simple arithmetic processing. For pixels that satisfy the in-focus condition in the image data of different imaging positions, as described in the first feature, the value of the imaging position farthest from the object 12 (from the reference plane) is selected. be done.

Third, in the method of synthesizing image data according to the present embodiment, when height information is obtained from pattern image data by the PMP method, the above-described synthesizing process is performed on the pattern image data, and the synthesized image is It is configured to perform interpolation processing of non-measurement pixels and correction processing of distortion (height post-processing) on the data. If interpolation processing or correction processing is performed on the height map calculated from the pattern image data captured at each imaging position, the pixel values of the composite image data will not reach the interpolated or corrected pixel values (unreliable data). However, by performing interpolation processing and correction processing on the combined image data (height map), the in-focus image data can be Since interpolation and correction are performed using the height information, highly accurate height information can be obtained.

Fourth, in the method of synthesizing image data according to the present embodiment, when the camera unit 21 is focused near the reference plane (when P(0)), the parameters for removing erroneous measurement values are: When the value is set to maximize the number of measurement points on the reference plane (substrate plane) and the camera unit 21 is not focused near the reference plane (P(1) to P(3)), the measurement error points of high suspicion are set to the value to remove. With this configuration, it is possible to obtain in-focus composite image data and improve the accuracy of height information.

Fifth, in the method of synthesizing image data according to the present embodiment, image data imaging processing is performed in the order in which the position of the camera unit 21 in the Z direction is farthest from the object 12 to be inspected, and basic height processing and synthesis are performed for each imaging. configured to perform an image update process; With such a configuration, when the imaging process is being performed at the next imaging position, the height basic processing and the composite image updating process can be performed in parallel using the image data captured at the previous imaging position. can shorten the processing time.

According to the above configuration, it is possible to acquire focused image data even for the inspected object 12 on which a tall component is mounted on the substrate. The accuracy of the height information of the height map is improved, and clear two-dimensional image data can be obtained.

In addition, the above embodiments do not limit the invention described in the claims, and all combinations of characteristic items described in the embodiments are essential items of the solution. It goes without saying that this is not always the case.

10 inspection device (image processing device)
12 object to be inspected 21 camera unit (imaging unit)
22 Lighting unit (lighting part)
23 projection unit (projection part)
30 control unit (control unit)

Claims (12)

an imaging unit capable of changing a relative position in an imaging direction with respect to an object to be inspected;
a control unit;
The control unit
Acquiring at least two pieces of image data captured by the imaging unit at different positions in the imaging direction by changing the relative positions in the imaging direction of the imaging unit and the object to be inspected;
For each image data, it is determined whether or not a pixel of the image data satisfies a condition of a pixel in focus, and when it is determined that the pixel satisfies the condition, the value of the pixel is synthesized. An image processing device that executes a composite image update process that selects values of corresponding pixels of image data. The control unit
performing height basic processing for acquiring height information of the object to be inspected for each pixel of the image data from the image data;
2. The image according to claim 1, wherein, in said composite image updating process, it is determined whether or not said pixel satisfies said condition based on said height information and the relative position between said object to be inspected and said imaging unit. processing equipment. a projection unit for projecting, onto the object to be inspected, a fringe pattern in which a pattern that changes so as to monotonously increase or decrease in phase is repeatedly formed;
The control unit
In the height basic processing, the height information is obtained from the image data obtained by projecting the stripe pattern by the projection unit,
the height information includes information about a calculated distance between the imaging unit and the object to be inspected;
In the composite image updating process, the condition 3. The image processing apparatus according to claim 2, wherein it is determined whether or not the pixel satisfies The image data obtained when the imaging unit is located at a different position includes image data when the imaging unit is focused on a predetermined reference plane on the object to be inspected,
The control unit
In the synthetic image updating process, information about the amplitude of the pixel of the image data to be determined, and information about the amplitude of the pixel of the image data when the imaging unit is focused on the predetermined reference plane. 4. The image processing apparatus according to claim 3, wherein whether or not the pixel satisfies the condition is determined by comparing with . The control unit
In the height basic processing,
acquiring the height information so as to maximize the number of measurement points for the image data when the imaging unit is focused on the predetermined reference plane;
For the image data when the imaging unit is at a position different from the focused position on the predetermined reference plane, the height information is changed so as to exclude pixels that are highly suspected to be measurement errors. The image processing apparatus according to claim 4, wherein the image is acquired. further comprising an illumination unit for projecting illumination light onto the object to be inspected;
The image data includes pattern image data obtained by projecting the striped pattern by the projection unit and acquired by the imaging unit, and a two-dimensional image obtained by the imaging unit by projecting the illumination light by the illumination unit. including data and
The control unit
obtaining the height information from the pattern image data in the height basic processing;
In the synthetic image update process, when it is determined that the pixel satisfies the condition, the height information of the pixel of the pattern image data is selected as the value of the pixel of the synthetic image data of the height map, The image processing apparatus according to any one of claims 3 to 5, wherein the value of the pixel in the two-dimensional image data is selected as the value of the pixel in the combined image data of the two-dimensional image data. The control unit
The image processing apparatus according to claim 6, wherein height post-processing such as non-measurement pixel interpolation processing and distortion correction processing is executed on the synthetic image data of the height map synthesized by the synthetic image update processing. The control unit
In the composite image updating process, when it is determined that two or more pieces of image data of the same pixel when the imaging unit is located at different positions in the imaging direction and the pixel satisfies the condition, the image data is updated. The image according to any one of claims 1 to 7, wherein the imaging unit selects the value of the pixel of the image data located farthest from the object to be inspected as the value of the corresponding pixel of the combined image data. processing equipment. The control unit
acquiring the image data in order from the position of the imaging unit farther from the object to be inspected;
The image processing apparatus according to any one of claims 1 to 8, wherein the composite image updating process is executed each time the image data is acquired. The control unit acquires image data when the imaging unit is focused on a predetermined reference plane on the object to be inspected, and then acquires image data from the farther position of the imaging unit with respect to the object to be inspected. The image processing apparatus according to any one of claims 1 to 9, which acquires Based on the image data when the imaging unit is focused on the predetermined reference plane, the control unit determines whether image data captured at another position is a pixel satisfying the predetermined condition. The image processing apparatus according to claim 10, wherein the determination is made. The control unit acquires image data from the position of the imaging unit farther from the object to be inspected, and finally acquires image data when the imaging unit is focused on the predetermined reference plane. The image processing device according to any one of claims 1 to 9.

Images