JP7624314B2 - Image Processing Device - Google Patents

Description

The present invention relates to an image processing device that uses a method for synthesizing image data.

In recent years, electronic circuit boards have come to be mounted on a variety of devices, but for devices on which these types of electronic circuit boards are mounted, miniaturization and thinning are always issues, and from this point of view, there is a demand for higher density mounting of electronic circuit boards. In inspection devices that inspect the solder application state and electronic component mounting state of such electronic circuit boards, a three-dimensional measurement device based on the phase shift method has been proposed that uses multiple light patterns with different periods to expand the measurement range and shorten the measurement time (see, for example, Patent Document 1).

JP 2017-003513 A

In the above method, there is a limit to how much the measurement range can be expanded due to the depth of field of the imaging unit. On the other hand, it is known that multiple measurement data can be stitched together by taking measurements from multiple imaging positions and performing coordinate transformation according to the positional relationships. However, there is an issue that simply combining the measurement results calculated from multiple imaging positions will often contain erroneous measurements, especially near the seams of the combination.

The present invention has been made in consideration of these problems, and aims to provide an image processing device that uses an image data synthesis method that can synthesize multiple image data acquired by imaging units located at different positions in the imaging direction (Z direction) with high accuracy and in a short time.

In order to achieve the above object, an image processing device according to the present invention has an imaging section capable of changing a relative position in an imaging direction with respect to an object under inspection, and a control section, wherein the control section performs a first step of performing height basic processing to acquire reference position image data by imaging with the imaging section at a position when focused on a predetermined reference plane on the object under inspection, and acquire height information of the object under inspection for each pixel of the reference position image data, a second step of setting a position in the imaging direction different from the position when focused on the predetermined reference plane and among predetermined positions which are the furthest from the object under inspection as a current position, a third step of performing imaging with the imaging section at the current position to acquire current position image data, and acquire height information of the object under inspection by performing the height basic processing for each pixel of the current position image data, which are included in the height information, and a fourth step of executing a composite image update process for determining whether or not each pixel of the current position image data satisfies the condition of a focused pixel based on the information of the fourth step, and selecting the value of the pixel in the current position image data as the value of the corresponding pixel in the composite image data when it is determined that the pixel satisfies the condition, a fifth step of repeatedly executing the third step and the fourth step with the next position in the direction approaching the specified reference plane from the current position among the predetermined positions as the current position, and a sixth step of setting the position when the pixel is focused on the specified reference plane as the current position when it is determined in the fifth step that the third step and the fourth step have been executed at all of the predetermined positions, and executing the composite image update process for each pixel of the reference position image data and selecting the value of the pixel in the reference position image data as the value of the corresponding pixel in the composite image data when it is determined that the pixel satisfies the condition . Note that the focal length of the imaging unit is fixed.

Furthermore, it is preferable that the image processing device according to the present invention has a projection unit that projects a stripe pattern onto the object to be inspected, the stripe pattern being formed by repeatedly changing a phase so as to monotonically increase or decrease, and the control unit is configured to acquire the reference position image data or the current position image data acquired by projecting the stripe pattern by the projection unit, and acquire the height information by the basic height processing .

In addition, in the image processing device of the present invention, it is preferable that, in the composite image update process, the control unit compares information regarding amplitude contained in the luminance information of the pixel of the current position image data with information regarding amplitude contained in the luminance information of the pixel of the reference position image data to determine whether or not the pixel satisfies the condition.

In addition, in the image processing device of the present invention, it is preferable that, in the basic height processing, the control unit acquires the height information for the reference position image data so as to maximize the number of measurement points, and acquires the height information for the current position image data so as to exclude pixels that are highly suspected of having measurement errors.

Furthermore, the image processing device of the present invention further has an illumination unit that projects illumination light onto the object under inspection, and each of the reference position image data and the current position image data includes pattern image data acquired by the imaging unit when the stripe pattern is projected by the projection unit, and two-dimensional image data acquired by the imaging unit when the illumination light is projected by the illumination unit, and it is preferable that the control unit acquires the height information from the pattern image data in the basic height processing, and when it is determined in the composite image update processing that a pixel satisfies the condition, selects the height information of the pixel in the pattern image data as the value of the corresponding pixel in the composite image data of the height map, and selects the value of the pixel in the two-dimensional image data as the value of the corresponding pixel in the composite image data of the two-dimensional image data.

In addition, in the image processing device according to the present invention, it is preferable that the control unit performs height post-processing, such as unmeasured pixel interpolation processing and distortion correction processing, on the composite image data of the height map synthesized by the composite image update processing.

Furthermore, in the image processing device of the present invention, when the control unit determines in the composite image update process that the same pixel satisfies the condition in two or more of the reference position image data or the current position image data when the imaging unit is in different positions in the imaging direction, it is preferable that the control unit selects the value of the pixel in the image data in which the imaging unit is located farthest from the subject as the value of the corresponding pixel in the composite image data.

According to the present invention, it is possible to provide an image processing device that uses an image data synthesis method that can synthesize multiple image data acquired by imaging units located at different positions in the imaging direction (Z direction) with high accuracy and in a short time.

FIG. 1 is an explanatory diagram for explaining a configuration of an inspection device including an image processing device. FIG. 11 is an explanatory diagram illustrating the relationship between the phase of a reference plane and a measured phase. FIG. 4 is an explanatory diagram for explaining an imaging position of a camera unit. 10 is a flowchart showing the flow of a method for combining image data. FIG. 11 is an explanatory diagram for explaining the correspondence between the range of condition (1) at each imaging position and composite image data.

A preferred embodiment of the present invention will now be described with reference to the drawings. First, the configuration of an inspection device 10 according to this embodiment will be described with reference to FIG. 1. This inspection device 10 is a device that inspects an object 12 using image data (two-dimensional image data or pattern image data) of the object 12 obtained by imaging the object 12. Therefore, this inspection device 10 also functions as an image processing device. The object 12 is, for example, an electronic circuit board on which components are mounted and solder is applied.

The inspection device 10 includes an inspection table 14 for holding the object under inspection 12, an imaging unit 20 for illuminating and imaging the object under inspection 12, an XY stage 16 for moving the imaging unit 20 relative to the inspection table 14, and a control unit 30 for controlling the operation of the imaging unit 20 and the XY stage 16 and inspecting the object under inspection 12. For ease of explanation, as shown in FIG. 1, the surface of the inspection table 14 on which the object under inspection is placed is defined as the XY plane, and the direction perpendicular to the surface (i.e., the imaging direction of the camera unit 21 constituting the imaging unit 20 (the optical axis direction of the optical system of the camera unit 21)) is defined as the Z direction.

The imaging unit 20 is attached to a moving table (not shown) of the XY stage 16, and can be moved in the X and Y directions by the XY stage 16. The XY stage 16 is, for example, a so-called H-shaped XY stage. Therefore, the XY stage 16 includes a Y drive unit that moves the moving table in the Y direction along a Y direction guide extending in the Y direction, and two X direction guides and an X drive unit that support the Y direction guide at both ends and are configured to move the moving table and the Y direction guide in the X direction. The XY stage 16 further includes a Z movement mechanism that moves the imaging unit 20 in the Z direction. It may also include a rotation mechanism that rotates the imaging unit 20. The inspection device 10 may further include an XY stage that can move the inspection table 14, and in this case, the XY stage 16 that moves the imaging unit 20 may be omitted. In addition, a linear motor or a ball screw can be used for the X drive unit and the Y drive unit. Also, instead of moving the imaging unit 20 in the Z direction, the XY stage 16 may be configured to move in the Z direction.

The imaging unit 20 includes a camera unit 21 that captures an image from a direction perpendicular (Z direction) to the inspection surface (substrate surface) of the inspected object 12, an illumination unit 22, and a projection unit 23. In the inspection device 10 according to this embodiment, the camera unit 21, the illumination unit 22, and the projection unit 23 may be configured as an integrated imaging unit 20. In this integrated imaging unit 20, the relative positions of the camera unit 21, the illumination unit 22, and the projection unit 23 may be fixed, or each unit may be configured to be movable relative to the others. In addition, the camera unit 21, the illumination unit 22, and the projection unit 23 may be separate and configured to be movable separately.

The camera unit 21 includes an image sensor that generates a two-dimensional image of the object, and an optical system (e.g., a lens) for focusing the image on the image sensor. The camera unit 21 is, for example, a CCD camera. The maximum field of view of the camera unit 21 may be smaller than the area of the inspection table 14 where the object to be inspected is placed. In this case, the camera unit 21 captures the entire object to be inspected 12 by dividing it into multiple partial images. The control unit 30 controls the XY stage 16 so that the camera unit 21 is moved to the next imaging position each time the camera unit 21 captures a partial image. The control unit 30 synthesizes the partial images to generate an entire image of the object to be inspected 12.

In addition, the camera unit 21 may be equipped with an image sensor that generates a one-dimensional image instead of a two-dimensional image sensor. In this case, the entire image of the test object 12 can be obtained by scanning the test object 12 with the camera unit 21.

The illumination unit 22 is configured to project illumination light for imaging by the camera unit 21 onto the surface of the inspected object 12. The illumination unit 22 includes one or more light sources that emit light of a wavelength or wavelength range selected from the wavelength range detectable by the imaging element of the camera unit 21. The illumination light is not limited to visible light, and ultraviolet light, X-rays, etc. may also be used. When multiple light sources are provided, each light source is configured to project light of a different wavelength (e.g., red, blue, and green) onto the surface of the inspected object 12 at a different projection angle.

The object under inspection 12 illuminated by the lighting unit 22 is imaged by the camera unit 21. The inspection device 10 judges the presence or absence of defects on the board of the object under inspection 12 (for example, whether there are components and whether their arrangement is appropriate, and whether the solder application condition is good or bad) based on image data of the object under inspection 12 illuminated and imaged by the lighting unit 22 (this image data is called "two-dimensional image data") and a height map described below.

In the inspection device 10 according to this embodiment, the illumination unit 22 is a side illumination source that projects illumination light from an oblique direction onto the inspection surface of the inspected object 12, and in this embodiment, includes an upper light source 22a, a middle light source 22b, and a lower light source 22c. In the inspection device 10 according to this embodiment, the side illumination sources 22a, 22b, and 22c are each a ring illumination source that surrounds the optical axis of the camera unit 21 and is configured to project illumination light obliquely onto the inspection surface of the inspected object 12. Each of these side illumination sources 22a, 22b, and 22c may be configured with multiple light sources arranged in a circular ring shape. In addition, the upper light source 22a, the middle light source 22b, and the lower light source 22c, which are side illumination sources, are each configured to project illumination light at different angles onto the inspection surface.

The projection unit 23 projects a pattern onto the inspection surface of the object under inspection 12. The object under inspection 12 onto which the pattern is projected is imaged by the camera unit 21. The inspection device 10 creates a height map of the inspection surface of the object under inspection based on image data of the imaged object under inspection 12 (this image data is called "pattern image data"). Here, the height map is data that has height information of the object under inspection for each pixel of the pattern image data. The control unit 30 detects local mismatches between the projected pattern and the pattern image data, and obtains height information of that portion based on the local mismatches. In other words, changes in the imaged pattern relative to the projected pattern correspond to changes in height on the inspection surface.

The projection pattern is preferably a one-dimensional stripe pattern in which light and dark lines are alternately repeated periodically. The projection unit 23 is arranged to project the stripe pattern from an oblique direction onto the inspection surface of the object 12 under test. Any discontinuity in height on the inspection surface of the object 12 under test appears as a shift in the stripe pattern image. Therefore, the height difference can be obtained from the amount of shift in the pattern. For example, the control unit 30 creates a height map using the PMP (Phase Measurement Profilometry) method, which uses a stripe pattern whose brightness changes according to a sine curve. In the PMP method, the amount of shift in the stripe pattern corresponds to the phase difference of the sine curve.

The projection unit 23 includes a pattern forming device, a light source device for illuminating the pattern forming device, and an optical system for projecting a pattern onto the inspection surface of the inspected object 12. The pattern forming device may be, for example, a variable patterning device capable of dynamically generating a desired pattern, such as a liquid crystal display, or a fixed patterning device in which a pattern is fixedly formed on a substrate, such as a glass plate. When the pattern forming device is a fixed patterning device, it is preferable to provide a moving mechanism for moving the fixed patterning device, or to provide an adjustment mechanism in the optical system for pattern projection, thereby making it possible to vary the projection position of the pattern. The projection unit 23 may also be configured to be able to switch between a plurality of fixed patterning devices having different patterns.

A plurality of projection units 23 may be provided around the camera unit 21. The plurality of projection units 23 are arranged to project the pattern onto the inspection object 12 from different projection directions. In this way, it is possible to reduce the area on the inspection surface that is shaded due to height differences and onto which the pattern is not projected.

The control unit 30 shown in FIG. 1 provides overall control of the entire device, and is realized as hardware using the CPU, memory, and other LSIs of any computer, and as software using programs loaded into memory, but the diagram shows functional blocks realized by the cooperation of these. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways using only hardware, only software, or a combination of both.

FIG. 1 shows an example of the configuration of the control unit 30. The control unit 30 includes an inspection control unit 31 and a memory 35, which is a storage unit. The inspection control unit 31 includes an image processing unit 32, an inspection data processing unit 33, and an inspection unit 34. The image processing unit 32 includes an image capture processing unit 32a, a height measurement unit 32b, and an image synthesis unit 32c. The inspection device 10 also includes an input unit 36 for receiving input from a user or other devices, and an output unit 37 for outputting information related to the inspection, and the input unit 36 and the output unit 37 are each connected to the control unit 30. The input unit 36 includes, for example, input means such as a mouse or keyboard for receiving input from a user, and communication means for communicating with other devices. The output unit 37 includes known output means such as a display or printer.

The inspection control unit 31 is configured to execute various control processes for inspection based on input from the input unit 36 and inspection-related information stored in the memory 35. The inspection-related information includes two-dimensional image data of the inspected object 12, a height map of the inspected object 12 (calculated from the pattern image data), and board inspection data. Prior to inspection, the inspection data processing unit 33 creates board inspection data using the two-dimensional image data and height map of the inspected object 12 that is guaranteed to pass all inspection items. The inspection unit 34 executes the inspection based on the created board inspection data and the two-dimensional image data and height map of the inspected object 12 to be inspected.

Board inspection data is inspection data created for each type of board. Board inspection data is a collection of inspection data for the components placed on the board and their positions, as well as for each solder applied to the board. Inspection data for each component or solder includes the inspection items required for that component or solder, the inspection window which is the inspection area on the image for each inspection item, and the inspection criteria that serve as the basis for judging the placement and pass/fail for each inspection item. One or more inspection windows are set for each inspection item. For example, in an inspection item that judges the pass/fail status of the solder application state, typically inspection windows in the same number as the number of solder application areas for that component are set in a placement that corresponds to the placement of the solder application areas. In addition, for inspection items that use images that have been subjected to a specified image processing on the image of the object to be inspected, the contents of that image processing are also included in the inspection data.

The inspection data processing unit 33 sets each item of the inspection data to suit the board as part of the board inspection data creation process. For example, the inspection data processing unit 33 automatically sets the position and size of each inspection window for each inspection item so as to match the solder layout of the board. The inspection data processing unit 33 may be configured to accept user input for some items of the inspection data. For example, the inspection data processing unit 33 may be configured to accept tuning of the inspection criteria by the user. The inspection criteria may be set using height information.

The inspection control unit 31 performs imaging processing of the inspected object 12 by the image processing unit 32 as a preprocessing for creating the board inspection data. The inspected object 12 used is one that passes all inspection items. As described above, the imaging processing is performed by controlling the relative movement of the imaging unit 20 and the inspection table 14 while illuminating the inspected object 12 by the illumination unit 22, and sequentially capturing partial images of the inspected object 12. Multiple partial images are captured so that the entire inspected object 12 is covered. The inspection control unit 31 combines these multiple partial images to generate full board image data (two-dimensional image data) including the entire inspection surface of the inspected object 12. The inspection control unit 31 stores the full board image data in the memory 35.

As a pre-processing step for creating a height map, the inspection control unit 31 controls the relative movement of the imaging unit 20 and the inspection table 14 while projecting a pattern onto the inspection object 12 by the projection unit 23 using the imaging processing unit 32a of the image processing unit 32, and divides and sequentially captures the pattern image of the inspection object 12. The projected pattern is preferably a stripe pattern whose brightness changes according to a sine curve based on the PMP method. The inspection control unit 31 combines the captured divided images to generate pattern image data for the entire inspection surface of the inspection object 12. The inspection control unit 31 stores the pattern image data in the memory 35. Note that pattern image data may be generated for a part of the inspection surface rather than the entire surface. As described later, the image data captured by the imaging processing unit 32a is subjected to a combination process by the height measurement unit 32b and the image combination unit 32c.

The height measurement unit 32b creates a height map of the entire inspection surface of the inspected object 12 based on the image of the pattern in the pattern image data. The height measurement unit 32b first obtains a phase difference map of the inspection surface of the inspected object 12 by determining the local phase difference between the pattern image data and the reference pattern image data for the entire image data. Here, the "reference pattern image data" is image data in which a pattern is projected onto a reference plane by the projection unit 23 (i.e., image data in which a pattern generated by a pattern forming device built into the projection unit 23 is projected onto the reference plane). The height measurement unit 32b creates a height map of the inspected object 12 based on a reference plane that serves as a reference for height measurement and the phase difference map. The reference plane is, for example, the board surface of the electronic circuit board to be inspected. The reference plane does not necessarily have to be a flat plane, and may be a curved surface that reflects deformation such as warping of the board.

Specifically, the height measurement unit 32b determines the phase difference of the stripe pattern between each pixel of the pattern image data and the pixel of the reference pattern image data that corresponds to that pixel. The height measurement unit 32b converts the phase difference into height information. This is because the distance from the projection unit 23 varies depending on the position on the inspection surface, and therefore the stripe width changes from one end of the pattern projection area on the inspection surface to the other end, even if the stripe width of the reference pattern is constant. The height measurement unit 32b obtains height information from the reference plane based on the converted height information and the reference plane, and creates a height map of the inspected object 12.

The image processing unit 32 of the inspection control unit 31 may create an inspected object image having a height distribution by associating the height information of the height map of the inspected object 12 with each pixel of the two-dimensional image data of the inspected object 12. The image processing unit 32 may also perform a three-dimensional modeling display of the inspected object 12 based on the inspected object image data with height distribution. The image processing unit 32 may also superimpose the height distribution on the two-dimensional inspected object image data and display it on the output unit 37. For example, the inspected object image data may be displayed in a color-coded manner according to the height distribution.

Below, we will explain how such an inspection device 10 projects a pattern using the projection unit 23 to obtain height information.

First, a method for acquiring height information using the PMP method with a stripe pattern will be described. Focusing on one measurement point (pixel) in the stripe pattern projection area, when the stripe pattern is projected while shifting the phase spatially (in other words, when the stripe pattern is scanned in the direction in which the stripes repeat), the brightness of that measurement point (pixel) fluctuates periodically. Although the average brightness differs for each measurement point (pixel) depending on the surface characteristics of the measurement point (pixel), such as color and reflectance, a periodic brightness fluctuation corresponding to the stripe pattern occurs at every measurement point (pixel). Therefore, the phase is calculated from the periodic brightness fluctuation, and the phase difference from the initial phase provides height information.

When obtaining height information using the PMP method using the characteristics of such a stripe pattern, in principle it is necessary to shift the phase of the stripe pattern and capture the image at least three times, typically four times. Since the stripe pattern is a sine wave, the brightness fluctuation of each measurement point (pixel) is also a sine wave. Since the pitch of the stripes is known, the sine wave representing the brightness fluctuation can be identified once the average luminance (brightness), amplitude, and initial phase are known.

By obtaining the measured values of the luminance (brightness) of the measurement points (pixels) from at least three pieces of pattern image data captured by shifting the start position of the phase of the stripe pattern, three variables, the average luminance (brightness), amplitude, and initial phase, can be determined. Here, if the luminance of a pixel when one stripe pattern is captured four times by shifting the start position of the phase by 90 degrees (π/2 radians) each is I0, I1, I2, and I3, respectively, the luminance In when the stripe pattern is a sine wave is expressed by the following formula (a). In this formula (a), A indicates the amplitude depending on the brightness of the inspected object 12, B indicates the brightness offset depending on the camera unit 21 and the environmental brightness, φ indicates the phase to be obtained, and π/2 indicates the phase shift amount (shift amount). Note that n = 0, 1, 2, 3. However, since errors are generally included due to factors such as sensor noise, the least squares method is often required when calculating the phase φ, the amplitude A, and the offset B. Also, let E be the error obtained using the least squares method.

In = A×sin(φ+n×π/2)+B (a)

Height information is calculated by finding the phase φ, amplitude A, and offset B of each pixel from the pattern image data. For example, when one stripe pattern is imaged four times with the starting position of the phase shifted by 90 degrees (π/2 radians), it is known that the phase φ of the measurement point corresponding to each pixel can be expressed as the following equation (b).

tan(φ) = (I3-I1)/(I2-I0) (b)

If the pitch of the stripe pattern (the length of one period) is P and the initial phase is 0, the amount of shift in the stripe pattern can be calculated as P x (φ/2π). Using the projection angle of the pattern, height information at that position can be obtained from the amount of pattern shift.

In FIG. 2, (a) shows the phase obtained from the pattern image data captured by the camera unit 21 after projecting a stripe pattern onto the reference plane S using the projection unit 23, and (c) shows the phase obtained from the pattern image data captured by the camera unit 21 after projecting the same stripe pattern onto the reference plane S with an object O placed on it, as shown in (b), and (d) shows the phase obtained from the pattern image data captured by projecting the same stripe pattern onto the reference plane S. The height information of the object O can be calculated from the shift in phase (phase difference = measured phase - phase of the reference plane). This phase difference can be calculated for each pixel of the image captured by the camera unit 21, and the height information of the position corresponding to that pixel is calculated by multiplying the phase difference calculated for each pixel by a coefficient. This coefficient can be calculated simply as tan (projection angle) x stripe pitch/2π, for example.

The stripe pattern here is formed by repeating a pattern in which the brightness is sinusoidal and changes monotonically (monotonically increases or decreases). Note that in one period (pitch), as the phase changes from 0 to 2π (radians), the brightness increases monotonically from the darkest state (e.g., brightness I = 0) to the brightest state (e.g., brightness I = 255) (the relationship between phase and brightness in the pattern may be reversed, i.e., monotonically decreasing).

As explained with reference to FIG. 2, when a reference plane S is imaged using a stripe pattern and an object O (corresponding to the object under test 12) placed on the reference plane S is also imaged using the stripe pattern, the phase difference Δφ between the phase φ0 of the reference plane S obtained based on equation (b) and the phase φ acquired with the object O placed thereon can be calculated using the following equation (c).

Δφ = φ − φ0 (c)

From the above, height information H can be calculated from the phase difference Δφ calculated using formula (c) from the phases φ0, φ (calculated using formula (b) when four images are captured with a 90° shift) obtained from a reference plane and at least three pieces of pattern image data captured by placing an object O (inspection object 12) on this reference plane and projecting a stripe pattern with shifted phase start positions, based on the phase difference Δφ calculated using formula (c) and the following formula (d).

H = P×(Δφ/2π) (d)

As described above, the inspection device 10 according to this embodiment is configured to inspect the object 12 using a height map calculated from the two-dimensional image data and the pattern image data. However, since the camera of the camera unit 21 has a predetermined depth of field, for example, when the camera unit 21 is focused on the vicinity of the surface of the substrate, which is the object 12, image data of the top surface of a tall component may be captured in out-of-focus image data. Therefore, when such image data is used for inspection, there are cases where the characters written on the top surface of a tall component (such as a capacitor) cannot be read in the two-dimensional image data, and also where the image of the stripe pattern on the top surface of the tall component is blurred in the pattern image data, making it impossible to calculate accurate height information.

Therefore, in the inspection device 10 according to this embodiment, at least two pieces of image data captured by the camera unit 21 at different positions in the Z direction (imaging direction) are acquired, and relatively focused pixels are extracted from each image data and synthesized to obtain image data that is in focus over the entire field of view, even if components at different heights are arranged. Here, a case where image data acquired when the camera unit 21 is focused on the reference plane (a surface that serves as a reference for calculating height information as described above) and image data acquired by the camera unit 21 at different positions in the Z direction by moving the camera unit 21 in the Z direction are used for synthesis is described. Below, a method of synthesizing image data in the inspection device 10, which is an image processing device according to this embodiment, is described. Note that here, a case where the position of the camera unit 21 is moved in the Z direction is described, but it is sufficient that the relative positions of the camera unit 21 and the inspected object 12 in the Z direction change, and the inspected object 12 may be moved in the Z direction. Note that, as shown in FIG. 3(a), a predetermined surface of the inspected object 12 (for example, in the case of an electronic circuit board, the top surface of the board) is the "reference surface". Here, FIG. 3(a) shows a case where the reference surface of the test object 12 coincides with the reference plane.

In the following description, the position of the camera unit 21 in the Z direction is expressed in a coordinate system with Z=0 as the reference, as shown in FIG. 3(a). Specifically, P(i) indicating the position of the camera unit 21 in the Z direction is expressed as i=0 when the camera unit 21 is focused on the reference plane, and i=1, 2, 3, etc. in the order in which the camera unit 21 moves away from the reference plane. In the following description, an example will be taken of a case in which four pieces of image data are acquired at different positions in the Z direction. For example, assuming that the depth of field of the camera unit 21 is 4 mm, image data is acquired at the position "P(0)=0" of the camera unit 21 when it is focused on the reference plane, and three pieces of image data are acquired at positions shifted upward (in the Z direction) by 4 mm from the position of P(0)=0. The respective positions are represented as "P(1) = 4", "P(2) = 8", and "P(3) = 12" (the unit of mm is omitted). In addition, when the camera unit 21 is focused on the reference plane, that is, when the camera unit 21 is located at Z = 0, the distance from the tip of the camera unit 21 to the reference plane is defined as the focal distance. Furthermore, the position away from the tip of the camera unit 21 in the direction of the object under test 12 is defined as the most focused position FP.

The number of image data in which the position of the camera unit 21 is shifted in the Z direction is not limited to four, but may be two or more. The width by which the camera unit 21 is shifted in the Z direction is not limited to 4 mm, but may be set to any value, for example, taking into consideration the height of the components mounted on the inspected object 12 and the depth of field of the camera unit 21. Furthermore, instead of changing the position of the camera unit 21 in the Z direction at equal intervals, the interval at which each image data is acquired may be selected arbitrarily.

The position in the Z direction at which the camera unit 21 acquires image data is set in advance in the control unit 30 via the input unit 36, etc. For example, the position of the reference plane and the height of the inspection target (the height of the components mounted on the board) are input, and the position and interval in the Z direction at which image data is acquired are set. The focal length of the camera unit 21 is also fixed.

A method for synthesizing image data in the inspection device 10, which is an image processing device according to this embodiment, will be described with reference to FIG. 4. This image data synthesis method is configured to perform synthesis processing using image data acquired at a position farthest from the inspected object 12 (reference surface) among image data acquired at different positions in the Z direction.

4(a), when the control unit 30 determines that the process of acquiring image data of the object to be inspected 12 has started, the control unit 30 moves the camera unit 21 to a predetermined position (position in the XY direction) above the object to be inspected 12 by the imaging processing unit 32a, and further adjusts the position of the camera unit 21 in the Z direction so that the camera unit 21 is positioned at position i=0, i.e., P(0), and acquires image data of the object to be inspected 12 (step S100). When the camera unit 21 is moved to the position of P(0), the camera unit 21 focuses on the vicinity of the reference surface of the object to be inspected 12. In this step S100, as described above, in order to acquire a height map of the object to be inspected 12, the projection unit 23 projects a stripe pattern onto the object to be inspected 12 and the camera unit 21 acquires pattern image data, but at this time, the phase of the stripe pattern is shifted to acquire at least three (preferably four) pieces of pattern image data. Furthermore, when there are multiple projection units 23, the stripe patterns are projected onto the inspection object 12 from each projection unit 23 while shifting the phase as described above to obtain pattern image data. Furthermore, when obtaining two-dimensional image data, the side illumination sources 22a, 22b, and 22c of the illumination unit 22 are sequentially turned on to illuminate the inspection object 12, and an image is taken by the camera unit 21.

When the control unit 30 has finished capturing the image of the object under test 12, it sets parameters for removing erroneous measurement values from the height information calculated based on the pattern image data captured when the camera unit 21 is at position i=0, i.e., at position P(0) (referred to as "pattern image data of P(0)") (step S102). As described below, the synthesis process of the image data when the camera unit 21 is at position i=0 (at P(0)) is executed last. In this final synthesis process, the above parameters are set to values that maximize the number of measurement points so that pixels of the image data at P(0) are selected as pixels of the synthesized image data as much as possible. Then, the basic height process S200 is executed using these parameters (step S104).

When the control unit 30 starts the basic height processing S200 shown in FIG. 4(b), the height measurement unit 32b performs phase calculation processing for each pixel (within the field of view) in the pattern image data from three or four pattern image data captured with the phase of the stripe pattern shifted, based on the above-mentioned formulas (a) to (d) (step 202). If there are multiple projection units 23, the phase calculation processing is performed for each projection unit 23 (for each projection). Next, if there are multiple projection units 23, the control unit 30 performs a projection synthesis process to synthesize the respective phase calculation results, and calculates height information for each pixel (step S204). Finally, a noise filter process using a median filter or the like is performed on the height information for each pixel (step S206), and the execution of the basic height processing S200 for the pattern image of P(0) is terminated.

The reason for first acquiring image data when the camera unit 21 is at position P(0) (when the camera unit 21 is focused near the reference plane) and calculating height information from the pattern image data is that, as will be explained below, the amplitude and height information at P(0) are used in condition (2) used to determine the synthesis process.

Returning to FIG. 4(a), when the execution of the basic height process S200 is completed in step S104, the control unit 30 causes the imaging processing unit 32a to move the Z-direction position of the camera unit 21 to the position farthest from the object under test 12 among the set imaging positions as the next imaging position (step S106). Here, the camera unit 21 is moved to the position P(3)=12 when i=3.

When the control unit 30 moves the camera unit 21 to the next imaging position in the Z direction, the imaging processing unit 32a acquires pattern image data and two-dimensional image data at that imaging position, as in the case of P(0), and further sets parameters for removing erroneous measurement values from the height information calculated from the pattern image data (step S108). Furthermore, the control unit 30 executes the basic height processing S200 on the pattern image data captured in step S108 by the height measurement unit 32b (step S110). Here, since the camera unit 21 is at a position other than i=0 (in midair), the parameters for removing erroneous measurement values from the height information are set to exclude points that are highly suspected of being measurement errors (exclude erroneous measurement points in midair).

When the height information is calculated based on the pattern image data captured at the current imaging position (here, position P(3)), the control unit 30 executes a composite image update process using the image data of the current imaging position by the image synthesis unit 32c (step S112). Here, from among the pixels of the image data of the current imaging position, an image that is in focus (in-focus pixel) is selected and set as the pixel value of the composite image data. Specifically, for each pixel, it is determined whether or not to select a pixel of the image data of the current imaging position as composite image data based on the height information of that pixel at the current imaging position, according to the conditions of the two equations shown below (conditions (1) and (2)).

P(i)-DL≦H(X,Y,i)+P(i)≦P(i+1)-DH (1)
A(X, Y, i) ≧ A(X, Y, 0)×WA (2)
however,
P(i): Current imaging position P(i+1): Next imaging position farther away from the reference plane than the current imaging position H(X, Y, i): The position of the pixel to be judged is (X, Y). DL, DH: constants A(X, Y, i): amplitude at the current imaging position when the pixel to be determined is located at (X, Y) A(X, Y, 0): Amplitude at the imaging position i = 0 when the position of the pixel to be judged is (X, Y) WA: Weight coefficient

Condition (1) is to determine whether the position (calculated distance) of the test object 12 corresponding to the pixel to be judged is near the focal point of the camera unit 21 at the current imaging position. Here, the calculated distance H(X, Y, i) is the Z-direction distance from the most focused position FP at P(i) to the test object 12 at the pixel to be judged, and is the height information of the pixel at the position (X, Y) obtained by the basic height processing S200, and is expressed in the coordinate system of the camera unit 21 described above. For example, when P(0)=0, if the test object 12 corresponding to the pixel to be judged is on the reference plane, H(X, Y, i)=0. Note that, as shown in FIG. 3(a), when the camera unit 21 moves in the Z direction, the most focused position FP also moves in the Z direction. Also, as shown in FIG. 3B, DL and DH are constants for determining the range in which focused image data can be obtained by the camera unit 21, based on the current imaging position P(i) of the camera unit 21 and the adjacent imaging position P(i+1) that is farther away from the reference surface than the current imaging position. For example, when the positions of the camera unit 21 are set at 4 mm intervals, DL=1.5 mm and DH=1.0 mm can be set. Of course, these constants DL and DH can be determined based on the specifications of the camera unit 21 (depth of field, etc.) and the interval at which the camera unit 21 is shifted in the Z direction. In addition, it is generally desirable that DL>DH. Also, when the imaging position is the farthest from the inspected object 12 (reference surface) (here, P(3)=12), the value of P(i+1) (here, P(4)) is set in advance, and this value is used to judge the condition (1) (for example, P(4)=16). From the above, a pixel that satisfies condition (1) means that the object 12 is within a range where focused image data (clear image data) can be obtained by the camera unit 21 at that position (P(i)).

Here, instead of making DL and DH in condition (1) constants, they can be variables as shown below. Note that FUNC indicates a predetermined function, DL(i) is a variable at imaging position P(i) and is determined by a function FUNC that has the next imaging position P(i-1) and the current imaging position P(i) as parameters, and DH(i) is a variable at imaging position i and is determined by a function FUNC that has the current imaging position P(i) and the previous imaging position P(i+1) as parameters.

DL(i)=FUNC(P(i-1), P(i))
DH(i)=FUNC(P(i), P(i+1))

Condition (2) is used to determine whether a pixel is in focus or not based on the amplitudes A(X,Y,i) and A(X,Y,0) obtained by the above formula (a) when calculating the phase using pattern image data. This condition (2) is used to compare the focus degrees to extract pixels with higher focus degrees, and determine that the pixel is in focus when the evaluation value of the pixel is equal to or greater than the threshold value. Specifically, since the greater the amplitude value, the greater the focus degree, the threshold value is determined based on the amplitude at P(0) (i=0), and when the amplitude A(X,Y,i) at the current imaging position is sufficiently greater than the amplitude A(X,Y,0) at i=0 for the pixel to be determined (pixel at position (X,Y)), the pixel to be determined can be determined to be in focus. Note that it is preferable to determine the value of the weight WA based on the depth of field of the camera unit 21.

As described above, at the imaging position P(i), conditions (1) and (2) are judged for each pixel, and for pixels that simultaneously satisfy conditions (1) and (2), the value of the pixel at the current imaging position is selected as the pixel of the composite image. Specifically, when a pixel at position (X, Y) satisfies conditions (1) and (2), the height information (H(X, Y, i) + P(i)) calculated from the pattern image data of the pixel P(i) is set as the value of the pixel at the same position in the composite image data of the height map, and the value (brightness, etc.) of the pixel in the two-dimensional image data captured at the position P(i) is set as the value of the pixel at the same position in the composite image data of the two-dimensional image data. By making such judgments using conditions (1) and (2), it is possible to prevent erroneous measurements, especially near the seams of the composite.

When the image capturing processing section 32a has performed the above-mentioned synthesis processing for all pixels, the control unit 30 judges whether the image capturing processing has been completed at all of the predetermined image capturing positions in the Z direction (step S114). When it is judged that the image capturing processing has not been completed at all of the image capturing positions (step S114: No), the control unit 30 sets the position of the camera unit 21 to the next image capturing position approaching the reference plane (step S116) by the image capturing processing section 32a, and repeats the above-mentioned processing of steps S108 to S114 at that image capturing position. For example, if the current image capturing position is P(3)=12, the camera unit 21 is set to the next image capturing position, P(2)=8, and if the current image capturing position is P(2)=8, the camera unit 21 is set to the next image capturing position, P(1)=4.

On the other hand, if it is determined that the imaging process has been completed at all imaging positions (step S114: Yes), the control unit 30 performs a composite image update process (step S118) on the image data (image data of P(0)=0) acquired in step S100 when the camera unit 21 is focused near the reference plane, using the image synthesis unit 32c. In this case, since condition (2) is always satisfied for the image data of P(0), a determination is made based on whether or not condition (1) is satisfied. That is, in step S118, for pixels for which a value for the composite image data has not yet been selected and which satisfy condition (1), the pixel value of the image data at the imaging position of P(0) (height information in the height map and pixel value of the two-dimensional image data) is selected as the pixel value of the composite image data.

In addition, in the repeated processing of steps S108 to S114, even if a certain pixel satisfies conditions (1) and (2), if the pixel satisfies conditions (1) and (2) at an imaging position farther from the object to be inspected 12 than the imaging position and the value of the pixel is selected as the value of the composite image data, the value already selected is prioritized. In other words, for a pixel that satisfies conditions (1) and (2) at multiple imaging positions in the Z direction, the pixel value of the imaging position farthest from the object to be inspected 12 among those imaging positions is selected. The light irradiated by the projection unit 23 is imaged after being reflected by the object to be inspected 12. As the camera unit 21 becomes farther from the substrate surface, the amount of reflected light from the object to be inspected 12 decreases. This leads to a reduction in noise due to unnecessary light, especially at pixels that are not in focus, and as a result, the erroneous measurement value included in the height information is reduced. Therefore, for pixels that satisfy conditions (1) and (2) at multiple imaging positions in the Z direction, the pixel value of the imaging position farthest from the test object 12 among those imaging positions is selected.

Figure 5 shows the relationship between the most focused position FP of the camera unit 21, the image data captured at that position, and the composite image data when the inspected object 12 is a portion in a certain field of view where the components 12a, 12b, and 12c are mounted on the board 12d. Specifically, the upper part of Figures 5(a) to (d) shows the relationship between the most focused position FP of the camera unit 21 and the components 12a to 12d, and the lower part shows the portions (pixels) that may be selected as the composite image data in a shaded manner. Figure 5(a) shows the case where the camera unit 21 is at P(3)=12, and since the top surface of the component 12a satisfies condition (1), if it also satisfies condition (2), then as shown in Figure 5(e), the top surface of the component 12a in the image data at P(3) is selected as the composite image data. On the other hand, the upper surface of the part 12b satisfies the condition (1) when the camera unit 21 is at P(2)=8 as shown in Fig. 5(b) and when the camera unit 21 is at P(1)=4 as shown in Fig. 5(c) (assuming that the condition (2) is also satisfied). In this case, as shown in Fig. 5(e), the upper surface of the part 12b in the image data of P(2) farther from the inspected object 12 is selected as the composite image data. In addition, the upper surface of the part 12c satisfies the condition (1) when the camera unit 21 is at P(1)=4 as shown in Fig. 5(c), and therefore, if the upper surface satisfies the condition (2) at the same time, the upper surface of the part 12c in the image data of P(1) is selected as the composite image data as shown in Fig. 5(e). Similarly, as shown in FIG. 5(d), the top surface (reference surface) of substrate 12d satisfies condition (1) when camera unit 21 is at P(0)=0, and therefore if condition (2) is also satisfied at the same time, the top surface of substrate 12d in the image data of P(0) is selected as the composite image data, as shown in FIG. 5(e) (see FIG. 5(e)).

Returning to FIG. 4, finally, the control unit 30 executes height post-processing S300 by the height measurement unit 32b (step S120). When the height measurement unit 32b of the control unit 30 starts the height post-processing S300 shown in FIG. 4(c), it executes a process of interpolating non-measured pixels such as pixels whose values were not selected in the above-mentioned composite image update process in the composite image data (step S302), and further executes a process of correcting distortion of the composite image data (step S304), terminating the height post-processing S300, and further terminating the synthesis process. If height post-processing is executed on the image data before synthesis (height map calculated from the pattern image data acquired at each imaging position), there is a possibility that even the interpolated value or the corrected pixel will be selected as the pixel value of the composite image data, and the accuracy of the image data may be deteriorated. Therefore, it is desirable to execute height post-processing such as non-measured pixel interpolation processing and distortion correction processing on the composite image data after the synthesis processing of the image data of all imaging positions is completed.

As shown in FIG. 4, in the image data synthesis method according to this embodiment, the height basic processing and the composite image update processing are performed each time the position of the camera unit 21 in the Z direction is moved to acquire image data, except for image data captured when the camera unit 21 is focused near the reference plane (at position P(0) where i=0). At this time, in FIG. 4, the imaging processing and the synthesis processing of the image data acquired in the imaging processing (height basic processing and composite image update processing) are described as a series of flows, but while the imaging processing at the next imaging position is being performed, the height basic processing and the composite image update processing for the image data acquired at the previous imaging position can be performed in parallel. As described above, in the imaging processing, the acquisition of multiple pattern image data with the phase of the stripe pattern shifted is performed for each of the multiple projection units 23, and further, the side illumination sources 22a, 22b, and 22c of the illumination unit 22 must be sequentially turned on to acquire multiple two-dimensional image data. Therefore, while performing these imaging processing at the next imaging position, the height basic processing and the composite image update processing at the previous imaging position are performed in parallel, thereby shortening the overall processing time and, as a result, the inspection time. At this time, in Figure 4, after performing the imaging process and basic height process at position P(0), the camera unit 21 is moved to P(3), which is the farthest from the object under test 12, and the camera unit 21 is moved from this imaging position in a direction approaching the object under test 12 to perform the imaging process, basic height process, and composite image update process, and finally the composite image update process of the image data of P(0) is performed. By performing the imaging process, height basic process, and composite image update process in this order, if a pixel value of the composite image data has already been selected in the height basic process and composite image update process at an imaging position prior to the imaging position at which the height basic process and composite image update process are currently being performed, even if the image data at the imaging position at which the height basic process and composite image update process are currently being performed satisfies conditions (1) and (2), the pixel value is not selected as the value of the corresponding pixel of the composite image data. This makes it possible to comply with the above-mentioned judgment criterion that "if pixels at the same position in image data at two or more imaging positions satisfy conditions (1) and (2), the pixel value of the image data at the imaging position farthest from the subject 12 is selected as the pixel value of the composite image data."

The above-mentioned processing is performed for one field of view. If the inspected object 12 is divided into multiple fields of view and imaged (inspected), the camera unit 21 is moved in the XY direction and the above-mentioned processing is performed for each field of view.

The inspection device 10 according to this embodiment performs the above-described inspection based on the composite image data (height map and two-dimensional image data) acquired by the image processing unit 32 of the inspection control unit 31 of the control unit 30.

(First Modification of the Condition for Selecting Pixels)
In the above-mentioned composite image update process, the determination is made based on the amplitude of the target pixel in condition (2), but the following condition (3) may be used instead of condition (2). That is, the above-mentioned conditions (1) and (3) may be used to determine whether or not to select a pixel of the image data at the current imaging position as composite image data based on the height information of the pixel at the current imaging position. The contrast C is calculated by C=A/B using the amplitude A and offset B obtained from the above-mentioned formula (a).

C(X, Y, i) ≧ C(X, Y, 0)×WC (3)
however,
C(X,Y,i): Contrast at the current imaging position when the position of the pixel to be determined is (X,Y) C(X,Y,0): Contrast at the current imaging position when the position of the pixel to be determined is (X, Y) contrast at i = 0 WC: weight coefficient

Condition (3) is to determine whether a pixel has a high degree of focus based on the contrasts C(X,Y,i) and C(X,Y,0) obtained based on the above formula (a) when calculating the phase using pattern image data. This condition (3) is to compare the degrees of focus to extract pixels with a higher degree of focus, and determine that the pixel is in focus when the evaluation value of the pixel is equal to or greater than a threshold value. Specifically, since the degree of focus is higher when the contrast value is higher, the pixel to be determined (pixel at position (X,Y)) can be determined to be a focused pixel when the contrast C(X,Y,i) at the current imaging position is sufficiently higher than the contrast C(X,Y,0) when i=0. Note that the value of the weight WC is preferably determined based on the depth of field of the camera unit 21.

(Modification of height measurement method)
In the above description, the height information is calculated from three or four pieces of pattern image data obtained by projecting one stripe pattern from the projection unit 23 with different phases. However, in order to improve the accuracy of the height information, the height information can be obtained by using at least two types of stripe patterns with different periods. For example, in the imaging process of the above steps S100 and S108, the projection unit 23 projects a first pattern with a long period while changing the phase to obtain three or four pieces of pattern image data, and projects a second pattern with a short period while changing the phase to obtain three or four pieces of pattern image data. When there are multiple projection units 23, each projection unit 23 obtains pattern image data in which the phases of the first and second patterns are changed. Then, rough height information (rough height) is obtained by the first pattern with a long period (i.e., thick stripes), and precise height information is obtained by the second pattern with a short period (thin stripes). As described above, the PMP method obtains height information based on the phase, so if the height gap (height difference) is large and the stripes are shifted by one period or more, the height information cannot be uniquely identified. By acquiring height information by using the first pattern in combination, it becomes possible to uniquely identify the height information even if the stripes are shifted by one period or more when the second pattern is projected.

The luminance In obtained from the three or four pattern image data obtained by projecting the first pattern with the projection unit 23 while changing the phase, and the luminance In obtained from the three or four pattern image data obtained by projecting the second pattern with the phase changed, have the relationship of the above-mentioned formula (a). Moreover, the relationship between these luminances In and the phase φ has the relationship of the above-mentioned formula (b).

Here, when the phase of the long-period first pattern is φw and the phase of the reference plane is φw0, the phase difference Δφw has the relationship of the following equation (c1) according to the above equation (c).

Δφw = φw - φw0 (c1)

Similarly, when the phase of the short-period second pattern is φf and the phase of the reference plane is φf0, the phase difference Δφf has the relationship of the following equation (c2) based on the above equation (c).

Δφf = φf - φf0 (c2)

From these phase differences Δφw and Δφf, detailed height information H can be obtained by the following formula (d'). Here, M is the number of periods of the short-period second pattern relative to one period of the long-period first pattern, and Round() is a function that rounds off the decimal point.

H = Round ((M×Δφw−Δφf)/2π)×2π+Δφf (d′)

In the case of a configuration in which height information is calculated using two stripe patterns, one with a long period and one with a short period, it is desirable to make a judgment based on the amplitude ratio as shown in the following condition (2') instead of the judgment based on the amplitude as shown in the above-mentioned condition (2). In other words, it is desirable to determine whether or not to select a pixel of the image data at the current imaging position as composite image data based on the above-mentioned conditions (1) and (2').

AR (X, Y, i) ≦ AR (X, Y, 0) × WAR (2')
however,
AR(X,Y,i): Amplitude ratio at the current imaging position when the position of the pixel to be determined is (X,Y) AR(X,Y,0): Amplitude ratio at the current imaging position when the position of the pixel to be determined is (X , Y) at i = 0 WAR: weight coefficient

The contrast ratio AR (parameters (X, Y, i) are omitted) of a pixel at position (X, Y) at imaging position P(i) is calculated based on the following formula (e). Here, Aw is the amplitude obtained by formula (a) from three or four pattern image data obtained by projecting a first pattern with a long period, and Af is the amplitude obtained by formula (a) from three or four pattern images obtained by projecting a second pattern with a short period.

AR = (Aw-Af)/Af (e)

As with condition (2), condition (2') compares the focus degrees to extract pixels with higher focus degrees, and determines that the pixel is in focus when the evaluation value of the pixel is equal to or less than a threshold value. Specifically, since the smaller the amplitude ratio value, the greater the focus degree, the pixel to be determined (pixel at position (X, Y)) can be determined to be in focus when the amplitude ratio AR(X, Y, i) at the current imaging position is sufficiently smaller than the amplitude ratio AR(X, Y, 0) when the camera unit 21 is focused near the reference plane (i=0). Note that the value of the weight WAR is desirably determined based on the depth of field of the camera unit 21.

Alternatively, instead of condition (2'), the following condition (3') may be used for the judgment. In other words, the above-mentioned conditions (1) and (3') may be used to determine whether or not to select a pixel of the image data at the current imaging position as the composite image data.

CR (X, Y, i) ≦ CR (X, Y, 0) × WCR (3')
however,
CR(X,Y,i): contrast ratio at the current imaging position when the position of the pixel to be determined is (X,Y) CR(X,Y,0): contrast ratio at the current imaging position when the position of the pixel to be determined is (X , Y) at i=0 WCR: Weight coefficient

Condition (3') is used to determine whether a pixel is in focus or not based on the contrast ratio obtained based on the above formula (a) when calculating the phase using pattern image data. This condition (3') compares the focus degrees to extract pixels with higher focus degrees, and determines that the pixel is in focus when the evaluation value of the pixel is equal to or less than a threshold value. Specifically, since the smaller the contrast ratio value, the greater the focus degree, the pixel to be determined can be determined to be in focus when the contrast ratio CR(X,Y,i) at the current imaging position is sufficiently smaller than the contrast ratio CR(X,Y,0) when the camera unit 21 is focused near the reference plane (when i=0).

The contrast ratio CR (parameters (X, Y, i) are omitted) is obtained from the contrast Cw obtained from the three or four pattern image data obtained by projecting the first pattern with a long period, and the contrast Cf obtained from the three or four pattern images obtained by projecting the second pattern with a short period, using the following formula (f). Note that Aw is the amplitude obtained from the first pattern, Af is the amplitude obtained from the second pattern, Bw is the offset obtained from the first pattern, and Bf is the offset obtained from the second pattern, and these values are obtained from the above formula (a).

CR = (Cw-Cf)/Cf (f)
however,
Cw = Aw/Bw (g)
Cf = Af/Bf (h)

In this way, in the above-mentioned composite image update process S112, even if a pixel that simultaneously satisfies conditions (1) and (3') is selected, the in-focus pixel can be set as the pixel value of the composite image data.

(Second Modification of the Condition for Selecting Pixels)
In the above-mentioned image data synthesis method, in the synthetic image update process S112, when there is one type of stripe pattern for acquiring height information, a pixel of the image data captured at the current imaging position is selected as a pixel of the synthetic image data by simultaneously selecting a pixel that satisfies the conditions (1) and (2), or the conditions (1) and (3), and when there are two types of stripe patterns, a long period and a short period, a pixel is selected as a pixel of the synthetic image data by simultaneously satisfying the conditions (1) and (2'), or the conditions (1) and (3'). However, it is also possible to select a pixel of the synthetic image data when the condition (1) is satisfied in the image data captured at the current imaging position without judging the conditions (2), (2'), (3), and (3'). This is because it is possible to judge whether the pixel of the image data at the current imaging position is a focused pixel based on only the condition (1), and the processing time can be further shortened. In addition, when configured in this manner, the imaging process at the imaging position P(0) when focused near the reference surface may be performed last (in other words, the imaging process, basic height process, and composite image update process may be performed from an imaging position far from the object under test 12).

In addition, when the imaging process, basic height process, and composite image update process are performed from an imaging position far from the subject 12, the image data when the camera unit 21 is focused near the reference plane (P(0)) is acquired last, so instead of the above-mentioned condition (2), the judgment may be made based on the following condition (2a).

A(X, Y, i) ≧ TH (2a)
however,
TH: A threshold value determined in advance from the amplitude A

When the above-mentioned condition (2') is used for the judgment, the following condition (2a') may be used instead of this condition (2').

AR (X, Y, i) ≦ TH (2a')
however,
TH: A threshold value determined in advance from the amplitude ratio AR

When the above-mentioned condition (3) is used for the judgment, the judgment may be made based on the following condition (3a) instead of this condition (3).

C(X, Y, i) ≧ TH (3a)
however,
TH: A threshold value determined in advance from contrast C

When the above-mentioned condition (3') is used for the judgment, the following condition (3a') may be used instead of this condition (3').

CR (X, Y, i) ≦ TH (3a')
however,
TH: A threshold value determined in advance from the contrast ratio CR

Even if conditions (2a), (2a'), (3a), and (3a') are used in place of conditions (2) and (2') or conditions (3) and (3'), pixels with a higher degree of focus can be extracted from among pixels that satisfy condition (1).

(Main features and effects)
The main features and effects of the image data synthesis method according to this embodiment are summarized below.

First, in the image data synthesis method according to this embodiment, for one field of view, the camera unit 21 is moved to different positions in the Z direction (imaging direction) to capture multiple pieces of image data to obtain synthetic image data, but the imaging position of the camera unit 21 is configured to perform synthesis processing from image data that is far from the inspected object 12 (from the reference surface). With this configuration, as the camera unit 21 becomes farther away from the inspected object 12, the amount of reflected light from the inspected object 12 decreases, and noise due to unnecessary light in unfocused pixels in particular decreases, thereby reducing erroneous measurement values included in the height information.

Secondly, in the image data synthesis method according to this embodiment, in-focus pixels are selected from the image data at each imaging position and used as the pixel value of the synthesized image data. The height information of each pixel measured by the PMP method, and the amplitude and offset in the PMP method are used as criteria for determining in-focus pixels. There are a method of selecting pixels that simultaneously satisfy the above-mentioned conditions (1) and (2), or pixels that simultaneously satisfy conditions (1) and (3), a method of selecting pixels that only satisfy condition (1), a method of selecting pixels that simultaneously satisfy conditions (1) and (2'), or conditions (1) and (3'), and a method of selecting pixels that simultaneously satisfy condition (1) and any of conditions (2a), (2a'), (3a), and (3a'). With this configuration, in-focus pixels can be selected with high accuracy by simple calculation processing. Note that, for pixels that satisfy the in-focus condition in the image data at different imaging positions, as described in the first feature, the value of the imaging position farthest from the test object 12 (from the reference surface) is selected.

Thirdly, in the image data synthesis method according to this embodiment, when height information is obtained from pattern image data by the PMP method, the above-mentioned synthesis process is performed on the pattern image data, and the synthesized image data is subjected to interpolation of unmeasured pixels and distortion correction (height post-processing). If interpolation or correction is performed on a height map calculated from pattern image data captured at each imaging position, even the interpolated or corrected pixel values (unreliable data) may be selected as pixel values of the synthesized image data, which may result in poor accuracy of the image data. However, by performing interpolation and correction on the synthesized image data (height map), interpolation and correction are performed on focused image data, so highly accurate height information can be obtained.

Fourth, in the image data synthesis method according to this embodiment, the parameters for removing erroneous measurement values are set to values that maximize the number of measurement points on the reference surface (substrate surface) when the camera unit 21 is focused near the reference surface (when P(0)), and are set to values that remove points that are highly suspected of being measurement errors when the camera unit 21 is not focused near the reference surface (when P(1) to P(3)). When configured in this manner, in-focus synthetic image data can be obtained, improving the accuracy of the height information.

Fifth, in the image data synthesis method according to this embodiment, the image data is processed in the order of the Z direction position of the camera unit 21 from furthest from the object under test 12, and the basic height processing and composite image update processing are executed for each image capture. With this configuration, while the image capture processing is being executed at the next imaging position, the basic height processing and composite image update processing can be executed in parallel using the image data captured at the previous imaging position, thereby shortening the processing time.

With the above configuration, even for an inspected object 12 that has tall components mounted on a board, focused image data can be acquired, improving the accuracy of the height information of the height map calculated from the pattern image data and providing clear two-dimensional image data.

It goes without saying that the above embodiments do not limit the invention described in the claims, and that not all combinations of the characteristic features described in the embodiments are necessarily essential to the solution.

10 Inspection device (image processing device)
12 Object to be inspected 21 Camera unit (imaging section)
22 Lighting unit (lighting section)
23 Projection unit (projection section)
30 Control unit (control section)

Claims (7)

an imaging unit capable of changing a relative position in an imaging direction with respect to the object under inspection;
A control unit,
The control unit is
a first step of performing a basic height process of acquiring reference position image data by capturing an image with the imaging unit at a position when the imaging unit is focused on a predetermined reference plane on the object to be inspected, and acquiring height information of the object to be inspected for each pixel of the reference position image data;
a second step of setting, as a current position, a position in an imaging direction different from a position when the focus is on the predetermined reference plane and among predetermined positions, the farthest position from the object to be inspected;
a third step of acquiring current position image data by capturing an image with the imaging unit at the current position, and executing the height basic processing for each pixel of the current position image data to acquire height information of the test object;
a fourth step of executing a composite image update process for determining whether or not a condition for a pixel being in focus is satisfied for each pixel of the current position image data based on luminance information for each pixel of the reference position image data and the current position image data, which are included in the height information, and information on the distance between the test object and the most focused position of the imaging unit for each pixel of the current position image data, and selecting a value of the pixel in the current position image data as a value of a corresponding pixel in the composite image data when it is determined that the pixel satisfies the condition ;
a fifth step of repeatedly executing the third step and the fourth step, with a next position in a direction approaching the predetermined reference plane from the current position as a current position, among the predetermined positions;
a sixth step of setting a position when the predetermined position is focused on the predetermined reference plane as a current position when it is determined in the fifth step that the third step and the fourth step have been performed at all of the predetermined positions, and of executing the composite image update process for each pixel of the reference position image data, and selecting the value of the pixel of the reference position image data as the value of the corresponding pixel of the composite image data when it is determined that the pixel satisfies the condition.
Image processing device. a projection unit that projects a stripe pattern, in which a pattern whose phase changes so as to monotonically increase or decrease is repeatedly formed, onto the object under inspection;
The control unit is
The reference position image data or the current position image data acquired by projecting the stripe pattern by the projection unit is acquired, and the height information is acquired by the height basic processing.
The image processing device according to claim 1 . The control unit is
3. The image processing device according to claim 2, wherein in the composite image update process, information regarding amplitude contained in the luminance information of the pixel of the current position image data is compared with information regarding amplitude contained in the luminance information of the pixel of the reference position image data to determine whether or not the pixel satisfies the condition. The control unit is
In the height basic processing,
The height information is acquired for the reference position image data so as to maximize the number of measurement points;
The image processing device according to claim 3 , wherein the height information is acquired from the current position image data so as to exclude pixels that are highly suspected of having a measurement error. The inspection apparatus further includes an illumination unit that projects illumination light onto the inspection object,
each of the reference position image data and the current position image data includes pattern image data acquired by the imaging unit after the stripe pattern is projected by the projection unit, and two-dimensional image data acquired by the imaging unit after the illumination light is projected by the illumination unit;
The control unit is
In the basic height processing, the height information is obtained from the pattern image data;
An image processing device as claimed in any one of claims 2 to 4, wherein, in the composite image update process, when it is determined that a pixel satisfies the condition, height information of the pixel in the pattern image data is selected as the value of the corresponding pixel in the composite image data of the height map, and a value of the pixel in the two-dimensional image data is selected as the value of the corresponding pixel in the composite image data of the two-dimensional image data. The control unit is
The image processing device according to claim 5 , further comprising: performing height post-processing such as non-measured pixel interpolation processing and distortion correction processing on the composite image data of the height map synthesized by the composite image update processing. The control unit is
The image processing device according to any one of claims 1 to 6, wherein, in the composite image update process, when it is determined that the same pixel satisfies the condition in two or more of the reference position image data or the current position image data when the imaging unit is in different positions in the imaging direction, the value of the pixel in the image data in which the imaging unit is located farthest from the subject is selected as the value of the corresponding pixel in the composite image data.