JP2007026423A - Image processor and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To specify the sequence of three-dimensional measurement by a user interface in which available processing items are presented to cause a user to produce the sequence of processing items like the conventional two-dimensional image processor. <P>SOLUTION: A plurality of items of two-dimensional image processing and at least one item of a three-dimensional measurement process are registered in a processing item table 201. The items of two-dimensional image processing includes at least one item of an on-first-image position specification process of specifying a position on a physical object in a first image. A sequence building unit 205 combines the item of the on-first-image position specification process and the item of the three-dimensional measurement process to set the sequence for three-dimensional measurement by selection of the user. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an apparatus that performs three-dimensional measurement based on images photographed by a plurality of cameras arranged to photograph an object from different directions.

  2. Description of the Related Art Conventionally, an apparatus for measuring the shape and surface pattern of an object by two-dimensional image processing has been widely used at various product manufacturing sites. In these two-dimensional image processing apparatuses, a surface representing the object (for example, the bottom surface of the object) or a measurement target surface of the object (including the inspection object surface) is set in a direction perpendicular to the surface, that is, the surface. Measurement (including measurement for the purpose of inspection based on the image taken from the front view direction. In the following, in this specification, in the case of simply “measurement”, taking into account that some measurement is performed in the process of inspection. "Including the case of performing measurement for the purpose of inspection").

  Such a two-dimensional image processing apparatus is provided with a menu of various processing items so that the same apparatus can be applied to various measurement purposes, and the user specifies the processing items to be used and their execution order, and the processing item sequence. In many cases, it can be assembled. For example, FIG. 9 of Patent Document 1 describes an image processing apparatus that presents processing items that can be used to a user and specifies the execution procedure of the processing items.

  On the other hand, a method of three-dimensional measurement based on the principle of stereoscopic vision using a plurality of cameras is known, and such a method is described in Patent Document 2, for example. However, a device that performs three-dimensional measurement is limited to the type and nature of the object, and is equipped with a measurement algorithm for measuring using the characteristics of the given object, and is dedicated to that object. The actual situation is that it has been put to practical use as an apparatus. For example, Patent Document 2 describes an image processing apparatus for three-dimensionally measuring an object having a rotationally symmetric shape.

JP 2005-167678 A JP 2002-99902 A

  Even in the production site of various products, if 3D measurement of an object is possible, measurement that was not possible with conventional 2D image processing devices will be possible, and measurement accuracy and usability will be improved compared to conventional products. There are situations that can be considered. However, a three-dimensional measuring device that can be used for general purposes at various product manufacturing sites has not been known.

  In such a situation, the inventors do not develop a conventional image processing apparatus specialized for three-dimensional measurement, but inherit the function as a conventional two-dimensional image processing apparatus. By developing a device in the direction of adding a three-dimensional measurement function to the above, it was decided to create a new type of image processing device that can perform three-dimensional measurement and has high versatility. Under such a policy, one of the issues set by the inventors is a user who presents available processing items and creates a sequence of processing items as in the case of a conventional two-dimensional image processing apparatus. It is to be able to specify a 3D measurement sequence through an interface.

  Another problem set by the inventors is to utilize a part of the processing used in the conventional two-dimensional image processing apparatus also for three-dimensional measurement.

(1) Similar to the conventional two-dimensional image processing apparatus, processing items suitable for designating a three-dimensional measurement sequence by a user interface that presents available processing items and creates a sequence of processing items. The provided image processing apparatus uses the first image and the second image based on the images photographed by the first camera and the second camera arranged to photograph the object from different directions, respectively. An image processing apparatus that performs processing, comprising: sequence setting means for setting a sequence of process items; and sequence execution means for executing the set sequence. The sequence setting means provides a plurality of items of two-dimensional image processing and at least one item of three-dimensional measurement processing as processing items that can be used as elements of the sequence by user selection. The provided two-dimensional image processing items include at least one first on-image position specifying process item that specifies a position on the object in the first image. The position on the object in the second image corresponding to the position on the object in the first image specified in the first image position specifying process belonging to the same sequence as the sequence to which the process belongs is specified. This includes a position specifying process on two images.

  In this image processing apparatus, processing for specifying a position on the first image for three-dimensional measurement is selected from a plurality of items of two-dimensional image processing. Therefore, according to this image processing apparatus, although it is possible to perform three-dimensional measurement, the uniformity with the conventional two-dimensional image processing apparatus is high, and the learning amount for using the apparatus is small for the user. That's it. Therefore, the user can easily use the device.

Here, the number of cameras used may be three or more. In that case, two of them are the first camera and the second camera.
The first image and the second image may be images captured by the first camera and the second camera, respectively, or may be images that have undergone some processing. For example, an image obtained by converting a perspective image captured by the second camera into a front-view image may be used as the second image.
The first image position specifying process does not prevent the process itself from being a general-purpose process (program) that can also target the second image by setting.

(2) Here, the three-dimensional measurement process further calculates a three-dimensional coordinate using the position specified in the first image position specifying process and the position specified in the second image position specifying process. May be included.
That is, in this case, the processing for specifying the position on the second image and the processing for calculating the three-dimensional coordinates are set as one processing item. For this reason, since the user can complete the entire sequence of the three-dimensional measurement with a small number of instructions, the user can use the apparatus more easily.

(3) This image processing apparatus provides a plurality of types of first image position specifying processing items and a plurality of types of three-dimensional measurement processing items that can be used as elements of a sequence by user selection. Also good.
With this configuration, it is possible to select the processing content of the three-dimensional measurement. Furthermore, when there are a plurality of three-dimensional measurement process items that can be paired with one first image position specifying process item by selection, and when selected, a pair with one three-dimensional measurement process item is selected. In the case where there are a plurality of items on the first image position specifying process that can be performed, it is possible to create more types of three-dimensional measurement sequences than the number of options of the item itself by combining the items. .

(4) The image processing apparatus can use a three-dimensional measurement basic sequence including at least one first image position specifying process item and at least one three-dimensional measurement process item by user selection. It may be provided. According to this configuration, even a user unfamiliar with the 3D measurement sequence can more easily use the 3D measurement function of the apparatus.

(5) The image processing apparatus may be configured as follows. That is, in the at least one first image position specifying process in which the processing item is provided, the first detection area is set for the first image, and the edge detection direction defined for the first detection area is set. The position of the edge on the second image in the at least one three-dimensional measurement process for which the processing item is provided is specified for the position of the edge of the first camera and the second camera. A second detection area having an edge detection direction as a specific direction based on the positional relationship and the edge position specified in the first image position specifying process is set, and an edge in the edge detection direction in the second detection area is set. The position of is specified. According to this configuration, the edge position of the object can be measured three-dimensionally.

(6) Here, the second detection region is set based on a range along the optical axis of the first camera designated by the user as the target range of the three-dimensional measurement in the range along the edge detection direction. It may be a thing. In this way, since the second detection area can be set in a limited range corresponding to the range that the user assumes that there is an edge to be measured, the amount of calculation of the second image position specifying process can be reduced. In addition, it is possible to reduce malfunctions due to an edge that is not a measurement target being included in the second detection region.

(7) In addition, the first camera has a rectangular field of view, and is arranged so as to take an image of the object from the front view direction. The second camera is connected to the first camera. On the other hand, it is located at a position separated in the vertical or horizontal direction in the field of view of the first camera, and is arranged so as to photograph the object from a direction oblique to the direction in which the object is viewed from the front, The first image position specifying means sets the first detection area to a position based on a user instruction with respect to the first image, and the edge detection direction of the first detection area is the first camera and The direction of the second camera may be arranged.

  In this configuration, the edge detection direction in the process of specifying the position of the edge targeted for the first image is the direction in which the cameras are arranged, and the second direction in the vertical or horizontal direction of the field of view of the first camera. Therefore, the vertical or horizontal direction of the camera field of view is the edge detection direction. Further, the first image is an image of the object taken from the front view. Since the detection area in the first image is set at a position based on a user instruction, the setting target image is a front view image, and the vertical or horizontal direction of the camera field of view is the edge detection direction. As a result, the edge detection region can be set with the same ease of understanding as the conventional two-dimensional image processing apparatus, despite the setting for three-dimensional measurement. Therefore, the user can easily use the device.

(8) The image processing apparatus may be configured as follows. That is, at least one first image position specifying process in which a processing item is provided specifies a representative position of an area on the first image that has a high degree of coincidence with a model image registered in advance before measurement execution. The second on-image position specifying process in at least one three-dimensional measurement process provided with the process item specifies the representative position of the area on the second image having a high degree of coincidence with the model image. . According to this configuration, it is possible to three-dimensionally measure the representative position of the region on the object having a high degree of coincidence with a model registered in advance.

(9) Here, the second image position specifying process is performed on the second image based on the positional relationship between the first camera and the second camera and the representative position specified in the first image position specifying process. A search area may be set, and a representative position of an area having a high degree of matching with the model image in the search area may be specified. In this way, the search area can be limited to a range in which the measurement target area may exist in the second image, so that the amount of calculation of the position identification process on the second image can be reduced, and the second It is possible to reduce malfunctions caused by a region that is not a measurement target on the second image being coincident with the model image.

(10) The search area may be set based on a range along the optical axis of the first camera designated by the user as a target range for three-dimensional measurement. In this way, since the search area can be set in a limited range corresponding to the range that the user assumes that there is a measurement target area, the calculation amount of the second image position specifying process can be reduced, and the first It is possible to reduce malfunctions caused by a region that is not a measurement target on the second image being coincident with the model image.

(11) This image processing apparatus may be configured as follows. That is, at least one first on-image position specifying process in which a processing item is provided specifies a representative position of a measurement target area set based on a user instruction on the first image, and an image in the measurement target area Is acquired as a model image, and the second image position specifying process in the at least one three-dimensional measurement process in which the processing item is provided is performed on an area on the second image having a high degree of coincidence with the model image. The representative position is specified.

  According to this configuration, it is possible to three-dimensionally measure the representative position of the region on the object having a high degree of coincidence with the model acquired from the first image. Since the model is actually acquired from the image that is the target of the measurement process, three-dimensional measurement can be performed even when the character or pattern on the surface that is the target of measurement changes for each object. Further, if a plurality of regions of the object are individually measured, it is possible to obtain a three-dimensional shape of the object (a distribution of three-dimensional coordinates on the surface of the object).

(12) Here, the second image position specifying process is performed on the second image based on the positional relationship between the first camera and the second camera and the representative position specified in the first image position specifying process. A search area may be set, and a representative position of an area having a high degree of coincidence with the model image in the search area may be specified. In this way, the search area can be limited to a range in which the measurement target area may exist in the second image, so that the amount of calculation of the position identification process on the second image can be reduced, and the second It is possible to reduce malfunctions caused by a region that is not a measurement target on the second image being coincident with the model image.

(13) The search area may be set based on a range along the optical axis of the first camera designated by the user as a target range for three-dimensional measurement. In this way, since the search area can be set in a limited range corresponding to the range that the user assumes that there is a measurement target area, the calculation amount of the second image position specifying process can be reduced, and the first It is possible to reduce malfunctions caused by a region that is not a measurement target on the second image being coincident with the model image.

(14) The solving means embodied in this image processing apparatus can also be grasped as an image processing method. That is, this image processing method uses the first image and the second image based on the images photographed by the first camera and the second camera arranged so as to photograph the object from different directions. An image processing method, comprising: a sequence setting step for setting a sequence of processing items; and a sequence execution step for executing a set sequence. The sequence setting step can be made an element of a sequence by user selection. A plurality of items of two-dimensional image processing and at least one item of three-dimensional measurement processing are provided as processing items, and among the provided two-dimensional image processing items, a position on the object in the first image Includes at least one item on the first image position specifying process that specifies the three-dimensional measurement process. A position on the second image for specifying a position on the object in the second image corresponding to a position on the object in the first image specified in the position specifying process on the first image belonging to the same sequence as Includes specific processing.

(15) Here, the three-dimensional measurement process further calculates a three-dimensional coordinate using the position specified in the first image position specifying process and the position specified in the second image position specifying process. May be included.

(16) A preferred example of an image processing apparatus capable of utilizing the processing used in the conventional two-dimensional image processing apparatus for three-dimensional measurement has a rectangular field of view and a direction in which the object is viewed from the front A first camera arranged so as to shoot from the first camera, and a position away from the first camera in the vertical or horizontal direction in the field of view of the first camera, with respect to the direction in which the object is viewed from the front An image processing apparatus that performs processing using a first image and a second image based on images respectively captured by a second camera arranged to photograph an object from an oblique direction. The apparatus corresponds to a position on the object in the first image specified by a first image position specifying means for specifying a position on the object in the first image and a position specifying processing means on the first image. A second image position specifying means for specifying a position on the object in the second image, a position specified by the first image position specifying means, and a position specified by the second image position specifying means 3D coordinate calculating means for calculating 3D coordinates using
The first image position specifying means sets the first detection area to a position based on a user instruction with respect to the first image, and sets the edge position in the edge detection direction defined for the first detection area. Identify. The edge detection direction of the first detection area is the direction in which the first camera and the second camera are arranged. The second image position specifying means is set for the second image based on the positional relationship between the first camera and the second camera and the edge position specified by the first image position specifying means. A second detection area having an edge detection direction as one direction is set, and the position of the edge in the edge detection direction in the second detection area is specified.

  According to such an image processing device, the processing for specifying the position of the edge in the detection region used also in the conventional two-dimensional image processing device is performed with the edge detection direction for the first image and the second image being changed. By specifying as described above, it can be used for three-dimensional measurement. In particular, the edge detection direction in the process of specifying the position of the edge for the first image is the direction in which the cameras are arranged, and the second camera is in the vertical or horizontal direction of the field of view of the first camera. Since they are arranged, the vertical direction or the horizontal direction of the field of view of the camera is the edge detection direction. Further, the first image is an image of the object taken from the front view. Since the detection area in the first image is set at a position based on a user instruction, the setting target image is a front view image, and the vertical or horizontal direction of the camera field of view is the edge detection direction. As a result, the edge detection region can be set with the same ease of understanding as the conventional two-dimensional image processing apparatus, despite the setting for three-dimensional measurement. Therefore, the user can easily use the device.

(17) The solving means embodied in such an image processing apparatus can be grasped as an image processing method in which the contents executed by each processing means are processing steps.

  According to the above, since the 3D measurement can be performed by the apparatus or method having high uniformity with the conventional 2D image processing, the user can easily use the 3D measurement function.

FIG. 1 shows a configuration of an imaging unit of an inspection apparatus (image processing apparatus) to which the present invention is applied, along with an installation example thereof.
The inspection apparatus of this embodiment has both three-dimensional and two-dimensional measurement processing functions, and images an inspection object W (hereinafter referred to as “work W”) conveyed on an inspection line L in a factory. Images are sequentially picked up by the unit 1, and measurement processing and discrimination processing according to various inspection purposes are executed.
The imaging unit 1 has a configuration in which two cameras C0 and C1 are incorporated in a housing 15, and is installed above the inspection line L. One camera C0 is installed with its optical axis oriented in the vertical direction (front view with respect to the workpiece W). The other camera C1 is installed with the optical axis tilted so that the field of view overlaps the camera C0. The imaging plane that defines the field of view of the camera C0 and the camera C1 is rectangular, and the camera C1 corresponds to the lateral direction of the field of view of the camera C0 relative to the camera C0 (the x-axis direction of the image A0 in FIG. 2). .) Are arranged in line.

FIG. 2 shows an example of an image of the workpiece W generated by each camera C0, C1. In the figure, A0 (corresponding to the first image) is an image from the camera C0, and A1 (corresponding to the second image) is an image from the camera C1.
In the figure illustrating the image of the workpiece W in FIG. 2 and FIG. 13 to be described later, the workpieces in the images A0 and A1 are also shown by using W as in FIG. The horizontal direction (horizontal direction) of the image A0 is the x-axis direction, and the vertical direction (vertical direction) is the y-axis direction.

  Since the camera C0 is installed with the optical axis oriented in the vertical direction, the image A0 shows a state where the upper surface of the workpiece W is viewed from the front. On the other hand, since the camera C1 is installed with the optical axis directed obliquely, the work W in a state seen from the oblique appears in the image A1. Focusing on such image characteristics, this inspection apparatus first identifies the measurement target position using the image A0 on the camera C0 side where the image distortion is small (close to the plan view of the workpiece W), and then the camera. A position corresponding to the measurement target position on the image A0 is specified on the image A1 on the C1 side.

  FIG. 3 is a block diagram showing the overall configuration of the inspection apparatus. In addition to the imaging unit 1, the inspection apparatus includes a main body unit 2, a monitor 3, a console 4 and the like. The main body 2 is provided with image input units 20 and 21, a camera driving unit 22, an image memory 23, a processing unit 24, a memory 25, a display control unit 26, an output unit 27, and the like for the cameras C0 and C1.

  The camera drive unit 22 receives detection signals from a workpiece detection sensor (not shown) and drives the cameras C0 and C1 simultaneously. The image input units 20 and 21 include an A / D conversion circuit, an image buffer, and the like. Image signals generated by the cameras C0 and C1 are digitally converted by the respective image input units 20 and 21, and each camera is Then, digital images for measurement processing (images A0 and A1) are generated. These images A 0 and A 1 are stored in the image buffers of the image input units 20 and 21, read out in response to a command from the processing unit 24, and stored in the image memory 23.

  The processing unit 24 includes a CPU and its peripheral circuits. The memory 25 is a non-volatile memory such as an EEPROM, and includes a program (a program that controls the operation of the apparatus, a program that executes each image processing item, an image processing sequence, etc.), and a parameter (three-dimensional) for three-dimensional measurement. Various other data files necessary for measurement are stored. The processing unit 24 performs processing according to a sequence described later on the images A0 and A1 stored in the image memory 23, and performs workpiece W measurement and discrimination processing.

  The display control unit 26 receives the images A0 and A1 from the image input units 20 and 21, generates a display screen including these images, and causes the monitor 3 to display the display screen. The processing unit 24 appropriately controls the display control unit 26 to display the measurement result and the inspection result together with the images A0 and A1, and displays a setting screen described later.

  The output unit 27 is an output interface for outputting the results of measurement processing and determination processing to an external device such as a PLC.

  In the inspection apparatus of this embodiment, a plurality of types of processing items are prepared for various types of processing executed for measurement and inspection, and the user can select these processing items and the execution order thereof from image input. A series of sequences up to the process of outputting the processing result can be assembled.

FIG. 4 shows a system configuration relating to sequence setting.
This system includes a processing item table 201, a sequence storage unit 202, a program storage unit 203, a designation operation reception unit 204, a sequence assembly unit 205, a sequence execution unit 206, a processing result output unit 207, and the like. Here, the processing item table 201, the sequence storage unit 202, and the program storage unit 203 are stored as files in the memory 25, and other configurations are programs that control the operation of the device stored in the memory 25. This is a function set in the processing unit 24.

  A plurality of processing items are registered in the processing item table 201 for each purpose of processing. FIG. 5 shows an example of this processing item table, in which specific processing items are set for each type of processing such as “image input related”, “position correction related”, and “general measurement related”. Yes.

  In this table, each processing item belonging to “related to image input” corresponds to the purpose of acquiring an image to be processed. Each processing item belonging to “position correction related” corresponds to the purpose of correcting the positional deviation of the measurement target part on the workpiece in the image so that the measurement target part is at a certain position in the image. .

Further, each processing item belonging to “general measurement related” corresponds to the purpose of performing measurement on the measurement target part, and is further classified into two types of processing, “positioning” and “measurement”. Among these, “positioning” is a process aimed at specifying a measurement target position on the image A0.
The position specified by executing the processing item belonging to “positioning” can be used for three-dimensional measurement. However, for the purpose of two-dimensional position measurement on the image A0, “positioning” is used. The process item to which it belongs may be executed. The measured two-dimensional position can also be used for calculating a dimension between two points on the image A0.

  The purpose of “measurement” is to find a position corresponding to the measurement target position specified on the image A0 side and calculate the three-dimensional coordinates of the measurement target position when performing three-dimensional measurement. It is processing to. In addition, processing for the purpose of two-dimensional measurement, such as processing for obtaining the area of the measurement target region and processing for obtaining the average density for the image A0, is also set as a processing item belonging to “measurement”.

Returning to FIG. 4, for each processing item registered in the processing item table 201, a program for executing processing of the item is stored in the program storage unit 203.
The designation operation accepting unit 204 reads out various process items from the process item table 201 and displays them on the monitor 3 and accepts a designation operation by the console 4. This designation operation selects a processing item to be used and designates its execution order.

The sequence assembling unit 205 receives the designation content received from the designation operation accepting unit 204 and assembles a sequence according to the designation. The assembled sequence is registered in the sequence storage unit 202. Specifically, the sequence has a data configuration in which code information corresponding to a specified processing item is arranged in a specified order.
The processing item table 201, the sequence storage unit 202, the designation operation reception unit 204, and the sequence assembly unit 205 correspond to a sequence setting unit, and the sequence execution unit 206 corresponds to a sequence execution unit.

  The sequence execution unit 206 reads out the sequence stored in the sequence assembly unit 205, and then sequentially reads out and executes programs corresponding to the processing items constituting the sequence from the program storage unit 203. The processing result output unit 207 outputs the processing result finally obtained by executing this sequence.

  In the sequence storage unit 202, a sequence of processing items indicating the flow from image input to two-dimensional measurement or three-dimensional measurement is registered in advance for a plurality of frequently used measurement purposes. Hereinafter, these are referred to as “basic sequences”.

  The designation operation accepting unit 204 can call and display various sequences (including a basic sequence) registered in the sequence storage unit 202 and accept a selection operation. Here, when one sequence is selected and designated to be executed, the process proceeds from the designation operation accepting unit 204 to the sequence executing unit 206, and the selected sequence is executed.

The designation operation accepting unit 204 can accept an operation for adding a processing item, an operation for replacing some processing items, and the like after a sequence is selected. In this case, the sequence selection result and subsequent operation contents are transferred to the sequence assembling unit 205, and editing processing of the selected sequence is performed. With respect to the basic sequence, processing items can be freely added, changed, and deleted.
The edited sequence is registered in place of the previous sequence, but can be registered as a new sequence.

  FIG. 6 shows a menu display example of a sequence registered in the sequence storage unit 202. Here, each sequence in the menu is called a scene, and an individual “scene name” (such as “edge position”) is assigned to each sequence. In the display example of FIG. 6, the basic sequence of 3D measurement is registered in each of the columns 0 to 3, but the columns 4 to 7 are not registered, and an initial name such as “scene 4” is displayed. Yes. When a sequence other than the basic sequence is registered, a new scene name is displayed in the fourth and subsequent columns of this screen.

Here, the basic sequence of the three-dimensional measurement shown in FIG. 6 will be briefly described.
The “edge position” is applied when measuring the three-dimensional coordinates of the edge point of the measurement target part. The position of the edge point to be measured is specified on each of the images A0 and A1, and the edge point is determined. Three-dimensional coordinates are calculated.
The “pattern” is applied when measuring a part including a certain pattern such as a character or a design, and is a model pattern (hereinafter simply referred to as “model”) registered in advance from the images A0 and A1. The representative position of the corresponding area is specified, and the three-dimensional coordinates of this representative position are calculated.

“Height measurement” is applied when the pattern of the measurement target region is not constant (different for each workpiece), and the measurement target region and its representative position in the image A0 based on the position information specified in advance. After the pattern in the area is acquired, the area corresponding to the pattern acquired from the image A0 is searched on the image A1, the representative position is specified, and the representative position of the representative position in both images is determined. Three-dimensional coordinates are calculated. This processing item is often referred to as height measurement because it is often intended to acquire height information (Z coordinate) with the camera C0 placed in a front view (vertical view).
“Multi-point height measurement” is basically the same as “height measurement”, but is used for the purpose of measuring the height of a plurality of points in a specified area.

  When any sequence is selected from the display menu, the monitor screen is switched to a display showing the contents of the selected sequence.

  FIG. 7 (1) shows a display screen when “edge position” in FIG. 6 is selected, and processing items to be executed are displayed together with numbers (0, 1, 2, 3) indicating the execution order. ing. FIG. 7B shows the displayed sequence replaced with a flowchart. Steps 0 to 3 correspond to numbers representing the execution order. Hereinafter, the flow of processing of “edge position” will be described using this flowchart.

  First, in the first step 0 (stereo camera image input), the cameras C0 and C1 are driven to generate images A0 and A1, and the image A0 is stored in the image memory 23. The image A1 is also held in the image buffer of the image input unit 11.

In the next step 1 (light / dark edge position), the x and y coordinates of the edge position are specified by executing binarization and projection processing within the measurement target region in the image A0.
In the next step 2 (camera switching), the image A1 is read from the image buffer of the image input unit 11 corresponding to the camera C1, and the image memory 23 is rewritten with this image.

  In the next step 3 (stereo edge position), a position corresponding to the edge position specified in step 1 is specified based on the relationship of the camera coordinate system between the cameras C0 and C1 with the image A1 as a processing target. Then, the three-dimensional coordinates are calculated using the position specified on the image A0 and the position specified on the image A1.

  FIG. 8A shows a display example of the monitor 3 when “Pattern” in the menu display of FIG. 6 is selected, and FIG. 8B is a flowchart showing the processing contents displayed on this display screen. Replaced and shown. Hereinafter, the flow of the “pattern” process will be described with reference to this flowchart.

  In the first step 0 (stereo camera image input), similarly to step 0 of “edge position”, the cameras C0 and C1 are driven to generate images A0 and A1, and the image A0 is stored in the image memory 23. To do.

  In the next step 1 (high-precision ECM (Edge Code Model) search), the representative position (for example, the region of the region) of the image A0 having a high degree of matching with the model image registered in advance (matching the model image) is targeted. Specify the center position. In this process, the image A0 is differentiated to extract points where the differential value of density is a certain value or more, and the direction of the density gradient at those points (expressed as angle data EC (Edge Code)) is calculated. Then, the method is performed by a method of collating the distribution pattern of the angle data EC with the distribution pattern of the angle data EC in the model image. Refer to the following Patent Document 3 for the detailed processing method.

JP 2002-230549 A

In the next step 2 (camera switching), the image memory 23 is rewritten with the image A1 on the camera C1 side in the same manner as in step 2 of “edge position”.
In the next step 3 (stereo high-precision ECM search), the representative position (for example, the region of the region) having a high degree of coincidence with the model image in the same manner as that performed on the image A0 with the image A1 as a processing target. Specify the center position. Then, the three-dimensional coordinates are calculated using the position specified on the image A0 and the position specified on the image A1.

  Similarly, in other basic three-dimensional measurement sequences, first, the image A1 on the camera C0 side is set as a processing target, and the processing for specifying the position of the point or the representative position of the region that is the target of the three-dimensional measurement is performed. The processing target is switched to obtain the position of the point or the representative position of the region corresponding to the position specified on the image A0 side, and the calculation processing of the three-dimensional coordinate or the Z coordinate is executed.

  According to the processing item table 201 shown in FIG. 5, Steps 0 and 2 in FIGS. 7 and 8 correspond to processing items related to image input. Step 1 corresponds to a processing item related to positioning, and Step 3 corresponds to a processing item related to measurement. In order to perform three-dimensional measurement with the configuration of the inspection apparatus of this embodiment, it is necessary to select processing items corresponding to steps 0 to 3 (provided that both images A0 and A1 can be stored in the image memory). Step 2 is not necessary.

  In the basic sequence of the three-dimensional measurement, since the processing items corresponding to the processes in steps 0 to 3 are already selected, the user can use the conditions and measurement purposes for specifying the position of the part to be inspected of the workpiece W. By selecting a suitable basic sequence, measurement processing according to the purpose can be performed.

  It is also possible to select a plurality of positioning-related processing items and measurement-related processing items in one sequence. For example, “edge position” and “high-precision ECM search” can be selected as positioning relation, and “stereo edge position” and “stereo high-precision ECM search” can be selected as measurement relation. In this case, the position specified in the “edge position” process is given to the “stereo edge position” process, and the corresponding position is specified and the three-dimensional coordinates are calculated in the “stereo edge position” process. The position specified in the “high-precision ECM search” process is given to the “stereo high-precision ECM search” process, and the corresponding position is specified and the three-dimensional coordinates are calculated in the “stereo high-precision ECM search” process. .

  Furthermore, it is possible to set a plurality of processes of the same type in one sequence. For example, a plurality of the same “high-precision ECM search” processes can be set, and the same number of “stereo high-precision ECM search” processes can be set. In this case, three-dimensional coordinates are calculated for each of the paired “high-precision ECM search” (positioning) processing and “stereo high-precision ECM search” (measurement) processing.

  It is necessary to specify which positioning process (first image position specifying process) and which measurement process (second image position specifying process) are to be used for the three-dimensional measurement. Specifically, the setting items for each measurement process include a numerical value indicating the number of the process in the sequence corresponding to the positioning process.

  As another embodiment, when the images A0 and A1 can be stored in the image memory 23 at the same time, there is no need to provide a “camera switching” step. When the dimension measurement process is continuously set in this order, these processes may be handled as a pair.

Only basic processing items essential for three-dimensional measurement are set in the basic sequences of FIGS. 7 and 8, but other processing items can be added as necessary.
FIG. 9 shows an example in which processing items related to position correction are added to the basic sequence of “edge position” in FIG. In the following, a user operation procedure for adding a processing item will be described along the flow of FIG.

  First, the user moves the cursor 31 to a process item to be executed immediately after the newly added process ((A) of FIG. 9) and performs an editing operation (in this example, the shift key and the enter key are operated simultaneously). . When this editing operation is performed, as shown in FIG. 9B, a selection box 32 is set on the display screen, and a word (change, insertion, etc.) indicating the editing method is displayed therein.

  Here, when the user selects “Insert” in the selection box 32, the display of the selection box 32 is switched as shown in FIG. 9C, and the items set in the type column of the processing item table 201 are displayed. The In response to this display, when the user selects a predetermined item (in the illustrated example, “position correction related”), as shown in FIG. 9D, a new selection box 33 is set and selected inside the box. The processing items included in the type are displayed. When the user selects a desired item (“model position correction” in the illustrated example) from among these, as shown in FIG. 9E, the selected processing item is located at the position designated by the cursor 31 in the basic sequence. Is inserted.

  When “Change” is selected in the selection box 32 in FIG. 9B, the currently selected process item (light / dark edge position) can be replaced with another process item. Also, by selecting “Delete”, the selected processing item can be deleted. Also, by selecting “Copy”, it becomes possible to store the processing item being selected and set the same item in other steps. This copy function is used when there are a plurality of measurement target parts (IC leads) and the measurement position must be specified for each measurement target part as in the workpiece W shown in FIGS. Can be used.

  Furthermore, in the above description, an example of editing a basic sequence has been described. However, when a necessary process item is read directly from the process item table 201 and the sequence is completely set manually without depending on the basic sequence, FIG. Since similar selection boxes 32 and 33 are displayed, the user can select a necessary processing item without hesitation.

  Furthermore, in the process item table 201, not only a process for three-dimensional measurement but also a process item for two-dimensional measurement is registered. For example, a pattern matching process is registered as a processing item for inspecting the printing state of characters written on the workpiece W, and a density average value or a density is a processing item for detecting defects such as scratches or chipping. Deviation calculation processing is registered. By incorporating these processing items in the sequence together with the three-dimensional measurement processing items, the two-dimensional and three-dimensional measurement processing can be efficiently executed.

  FIG. 10 shows an example in which processing for two-dimensional measurement is added to the basic sequence of “edge position” in FIG. In this example, after “stereo camera image input” in step 0, “density average / deviation” is set as a processing item related to two-dimensional measurement, and processing items similar to those in the basic sequence are set.

  According to this edited sequence, the processing for calculating the average density and the deviation is executed with the image A0 stored in the image memory 23 as the processing target, and then the three-dimensional measurement target using the same image A0. Since the position is specified and then the process proceeds to the process for the image A1, the same image as the three-dimensional measurement can be used for the two-dimensional measurement. In addition, since the number of rewrites to the image memory 23 can be reduced to one as in the basic sequence, processing delay can be prevented.

  From here, the embodiments described so far will be described in more detail. Hereinafter, the image A0 from the camera C0 is referred to as “front-view image A0”, and the image A1 from the camera C1 is referred to as “perspective image A1”.

  FIG. 11 is a block diagram showing the overall configuration of the inspection apparatus, and shows the details of FIG. 3 in more detail. A description of the same parts as those in FIG. 3 is omitted. This inspection apparatus is provided with an in-device computer 200.

  The in-apparatus computer 200 executes the measurement process using the images for the cameras C0 and C1, and then determines whether the work W is appropriate based on the processing result. In addition to the image memory 23 for storing the images A0 and A1, the in-device computer 200 includes an image processing unit 41, a measurement processing unit 42, a determination unit 43, a display control unit 26, a parameter calculation unit 44, and a parameter storage unit 45. Etc. are provided. Each unit other than the image memory 23 and the parameter storage unit 45 is a function set in the in-device computer 200 by a dedicated program. The image memory 23 and the parameter storage unit 45 are set in the memory (RAM, EEPROM, etc.) of the in-device computer 200.

  Of the above configuration, the image processing unit 41, the measurement processing unit 42, the determination unit 43, the parameter calculation unit 44, and the parameter storage unit 45 correspond to the processing unit 24 in FIG. Although not shown in FIG. 11, the in-device computer 200 corresponds to a memory (memory 25 in FIG. 3) for registering information (inspection area setting conditions, model images, etc.) necessary for the inspection. ) Is also provided. Setting or changing the registration processing in the registration memory and the processing executed by each unit of the in-device computer 200 can be appropriately performed according to the operation of the console 4.

  The image processing unit 41 extracts constituent points of the inspection target part of the workpiece W by binarization, edge extraction, pattern matching, and the like. The measurement processing unit 42 executes processing for measuring the position, size, and the like of the inspection target part extracted by the image processing unit 41. The image processing unit 41 and the measurement processing unit 42 are set so that two-dimensional and three-dimensional processing can be selected and executed.

The determination unit 43 determines the quality of the workpiece W by comparing the measurement result of the measurement processing unit 42 with a predetermined threshold value. The measurement results and determination results are output to the output unit 27 and the display control unit 26.
The display control unit 26 controls the display operation of the monitor 3 and can display the front view image A0 and the perspective image A1 generated by the image input units 20 and 21 in parallel on one screen. Furthermore, the processing results of the image processing unit 41, the measurement processing unit 42, and the determination unit 43 can be appropriately received and displayed together with the image.

The parameter storage unit 45 stores various coefficients included in the arithmetic expression for three-dimensional measurement. The values of these coefficients are the relationship between the stereo coordinate system constituted by the cameras C0 and C1 and the spatial coordinate system representing the position in the actual space (the distance between the origins between the coordinates, the stereo coordinate system for the spatial coordinate system). (Hereinafter, these coefficients are referred to as “parameters”). These parameters are calculated by the image processing unit 41 and the parameter calculation unit 44 and stored in the parameter storage unit 45 prior to the inspection. In the process of calculating this parameter, a calibration work having a plurality of feature points is used.
Further, the parameter storage unit 45 also registers parameters constituting a homography matrix of the arithmetic expression (1) described later.

  The inspection apparatus of this embodiment can assemble an inspection algorithm (sequence) by presenting a plurality of types of inspection menus (sequences of processing items and registered processing items) to the user and accepting a selection operation. is there. In addition, an inspection by a two-dimensional measurement process and an inspection by a three-dimensional measurement process can be selected and executed according to the region to be inspected. In the inspection by the two-dimensional measurement processing, pattern matching processing, binarization processing, edge extraction processing, etc. are executed on the front view image A0 from the camera C0, and the entire workpiece or the inspection target portion in the workpiece is extracted. .

  Further, in this embodiment, the three-dimensional measurement process can be speeded up by effectively using the front-view image A0 from the camera C0 for the inspection by the three-dimensional measurement process. This point will be described in detail later.

Hereinafter, an example of a case where an IC is inspected by the above-described inspection apparatus will be described in detail as to the processing related to the inspection.
FIG. 12 shows an inspection procedure performed when the workpiece W is an IC. This procedure is started upon receiving a detection signal from a workpiece detection sensor. In the first ST1 (ST is an abbreviation of “step”. The same applies to the following), the cameras C0 and C1 are simultaneously driven by the camera driving unit 12 to generate the front view image A0 and the perspective image A1.

  In the next ST2, an inspection for characters printed on the package portion of the IC is executed. In this inspection, two-dimensional image processing using only the front-view image A0 is executed. For example, a character print region is extracted by pattern matching processing, and the suitability of the character print state is determined from the degree of correlation at the time of matching and the matching position.

  In the next ST3, the leads on both sides of the IC are to be inspected. In this inspection, the three-dimensional coordinates of the tip of each lead are obtained by three-dimensional measurement processing using both the front-view image A0 and the perspective image A1, and abnormalities such as floating and bending at the tip of each lead are calculated from the calculated values. Determine if there is any.

  When the inspections of ST2 and 3 are completed, the result of each inspection is output to an external device or the monitor 3 in ST4. Further, when the next IC is detected by the workpiece detection sensor, the process returns to ST1 and the same procedure as described above is executed.

  In this way, by picking up an image of the workpiece W once by the two cameras C0 and C1, the inspection by the two-dimensional measurement and the inspection by the three-dimensional measurement can be executed continuously. Since the front-view image A0 is used in the two-dimensional measurement process, it is possible to perform a highly accurate measurement process using an image with no character distortion.

  When performing the three-dimensional measurement process, the corresponding measurement target point is identified between the front-view image A0 and the perspective image A1, and the three-dimensional coordinates are calculated by applying the calculation formula based on the principle of triangulation. To do.

  FIG. 13 shows an example of setting the detection area in the front view image A0 and the detection area in the perspective image A1. In this example as well, as in FIG. 2, the vertical direction of the image (the direction along the lead arrangement in the front view image A0) is the y direction, and the horizontal direction (the direction along the length of the lead in the front view image A0). Is the x direction. In the front-view image A0, an individual detection area 7 is set for each lead 6, and one edge point corresponding to the tip of the lead is specified for each detection area 7.

That is, here, since the cameras C0 and C1 are arranged in the x direction, parallax mainly occurs in the x direction. Therefore, by binarizing the image in the detection area 7 and projecting the obtained binary image along the y direction, the number of projected “bright” pixels or “dark” pixels is plotted with the x direction as the horizontal axis. Make a histogram with the vertical axis. Then, the x coordinate of the portion where the value of the histogram changes suddenly is set as the x coordinate of the tip of the lead 6. On the other hand, the y coordinate of the midpoint of the detection region 7 in the y direction is assigned to the y coordinate of the tip of the lead 6. Here, the image in the detection area 7 is binarized. However, the present invention is not limited to this. The density of each pixel is integrated in the y direction while the image in the detection area 7 is a grayscale image, and the integration thus obtained is integrated. You may make it obtain | require x coordinate of the location (for example, location which crosses a threshold value) where the value of density distribution changes suddenly along a x direction.
As described above, in the detection region 7, one specific direction is an edge detection target. In the example of FIG. 13, the x direction is the edge detection direction. Here, after the detection area 7 is set in the front-view image A0, the edge detection direction is designated for the detection area 7. However, the present invention is not limited to this, and the detection area 7 having the inherent edge detection direction may be set. Alternatively, the edge detection direction of the detection area 7 is first specified, and then detected in the front-view image A0. The area 7 may be set.

  Even in the perspective image A1, a detection region 8 is set for each lead 6. These detection areas 8 are specified in each detection area 7 of the front-view image A0 based on an arithmetic expression (formula (1) described later) for converting one point on one image into one point on the other image. The coordinates are set using the coordinates of the edge point, the height range specified by the user (the range that the height of the region to be inspected can take), and the like. The height range is a target range for three-dimensional measurement along the optical axis of the camera C0.

  In FIG. 13, only the area setting for the right lead of the workpiece W is shown, but the same setting is performed for the left lead (the same applies to the following drawings).

FIG. 14 shows a teaching processing procedure for IC lead inspection. This procedure is performed before starting the IC inspection of FIG.
In ST11, which is the first step in this procedure, the length of the lead 6, the pitch between the leads 6, and the like are input for the workpiece W to be inspected (IC in this example). The data input here is registered in the working memory and used in ST15 described later.

  In the next ST12, a non-defective workpiece is placed at the position to be imaged, and this is imaged by the cameras C0 and C1. In the teaching process, it is sufficient if the front-view image A0 from the camera C0 is generated. However, in this embodiment, the cameras C0 and C1 are simultaneously driven during the teaching process, and the generated two images are displayed on the monitor 3. They are displayed side by side.

In the next ST13, a positioning region designation operation is accepted in the front view image A1. FIG. 15 shows a display example of the front-view image A1 at the time of this designation operation, and 9 in the figure is a positioning area.
This positioning region 9 is used to extract the end lead (in the illustrated example, the top end lead 6a, hereinafter referred to as “lead lead 6a”) of the leads 6 arranged in a line. It is done. In the example of FIG. 7, a square-shaped region 9 that includes only the top lead 6a is set. The size of the positioning area 9 is adjusted so that the leading lead 6a can be imaged in the positioning area 9 even if the position of the workpiece W is shifted as much as expected. The positioning area 9 is set so that the top lead 6a is imaged in the lower half of the positioning area 9. As a result, it can be confirmed that the top lead 6 a is imaged in the lower half of the positioning area 9 because the lead is not imaged in the upper half of the positioning area 9.

  Returning to FIG. 14, when the positioning area 9 is designated, the head lead 6a is extracted from the positioning area 9 in the next ST14. In this extraction process, for example, the y coordinate of the head lead 6a is obtained by binarizing the image in the positioning area 9 and projecting the binarized image along the x-axis direction. Alternatively, by extracting the edge in the positioning region 9 and its concentration gradient direction, the lead outline may be extracted, and the y coordinate of the lead center line may be obtained.

In ST15, a detection area 7 is set for each lead 6 based on the y coordinate of the lead 6a and the data input in ST11. In this embodiment, the length of the lead 6 and the pitch between the leads 6 on the image are calculated using the data input in ST11, the number of pixels of the camera C0, the magnification, and the like, and each detection region is based on the calculated value. The size of 7 and the interval between regions are determined. Further, the detection area 7 having a size determined for each determined interval is set for each lead 6 including the lead 6a on the basis of the position of the head lead 6a.
The reason why such a method can be taken is that the front-view image A0 that directly reflects the characteristics of the inspection target part of the workpiece W (the length of each lead is equal, the pitch between leads is equal, etc.) is used as it is. It is. Therefore, if the tip lead 6a can be extracted, it is possible to set detection areas for all the leads 6 without extracting other leads 6, and the processing efficiency can be greatly improved.

In ST16, the setting conditions (position and size of the positioning area) specified in ST13 and the setting conditions of the detection area 7 set in ST15 are registered in the registration memory. In ST17, the image in the positioning area 9 is registered as a model in the registration memory. As a result, a series of teaching processes is completed.
Before executing ST16, it is desirable to display an image indicating the setting result of the detection area 7 on the monitor 3 and perform registration in accordance with the user's confirmation operation. At this time, the position and size of the detection region 7 may be finely adjusted.

FIG. 16 shows a detailed procedure for IC lead inspection (ST3 in FIG. 12).
The processing from ST21 to ST24 of this procedure is performed on the front view image A0. First, in ST21, the positioning area 9 is set in the front view image A1 based on the setting conditions registered by teaching. In the next ST22, the image in the positioning region 9 is compared with the model registered in ST17 of the teaching process, and a deviation amount with respect to the model is extracted (for example, a pattern matching technique can be applied to this process). it can.).
ST 21 and ST 22 correspond to the “model position correction” processing item in the processing item table 201.

  In ST23, the setting condition of the detection area registered at the time of teaching is adjusted based on the deviation amount extracted in ST22, and the detection area 7 of each lead is set according to the setting condition after the adjustment. According to the front-view image A0, since it is not necessary to consider the distortion of the workpiece W on the image, the shift amount of the positioning region 9 can be applied to each detection region 7 as it is, and the teaching process is performed on each lead 6. The detection area 7 can be set with the same positional relationship as the time.

  FIG. 17 shows an example of a front-view image A0 at the time of inspection. Since the workpiece W in this example is shifted to the right side from the image A0 at the time of teaching shown in FIG. 7, the tip of the lead 6 is in a state of protruding from the positioning region 9. However, since the adjustment process described above is performed for the detection area 7, the detection area 7 is set to any lead 6 under the same conditions as shown in FIG. 4.

  When the detection area 7 is set for each lead in this way, in the next ST24, the x and y coordinates of the tip of the lead are calculated for each detection area 7. ST23 and ST24 correspond to the “dark edge position” processing item in the processing item table 201. This processing item is set repeatedly for the number of leads. Although not shown in the figure, after this processing item, the “camera switching” processing item is executed.

  In the next ST25, the detection area 8 for the tip of each lead is set on the perspective image A1. In ST26, the same processing as ST24 is executed in the set detection area 8, and the x and y coordinates of the tip of the lead are calculated.

  Thereafter, in ST27, three-dimensional coordinates are calculated for each tip using the coordinates calculated in ST24 and 26, respectively. In ST28, the quality of each lead tip is determined by comparing the calculated three-dimensional coordinates with a reference value registered in advance. For example, if there is a float at a predetermined tip, the Z coordinate representing the height exceeds the reference value, and it is determined that the tip is defective. ST25 to ST27 correspond to the processing item of “stereo edge position”. This processing item is also set repeatedly for the number of leads.

Next, the setting of the detection area 8 in ST25 will be described in detail.
FIG. 18 shows a state in which one point P on the plane D at an arbitrary height position in the space is imaged at points p ₀ and p ₁ on the imaging surfaces F0 and F1 of the cameras C0 and C1, respectively. In FIG. 18, X, Y, and Z are coordinate axes representing a three-dimensional space, and the plane D is parallel to the XY plane. In addition, a two-dimensional coordinate system based on the x ₀ and y ₀ axes is set on the imaging surface F0, and a two-dimensional coordinate system based on the x ₁ and y ₁ axes is set on the imaging surface F1, respectively. In FIG. 10, the point on the plane D that is imaged at the origin of both imaging surfaces is assumed to be P, but this is not limiting, and the position of the point P is arbitrary on the plane D.

Coordinates of the imaging position of the point P on the imaging plane F0 (point _{p 0) (x cam0, y} cam0), the imaging position of the point P on the imaging plane F1 coordinates (point _{p 1) (x cam1, y} cam1 ), The relationship between the points p ₀ and p ₁ is expressed by the following equation (1).

In Expression (1), H _Z is a 3 × 3 homography matrix that represents the relationship between the imaging position on the imaging surface F0 and the imaging position on the imaging surface F1 for a point on the plane D having a height Z. And λ is a constant. The matrix H _Z can be obtained in advance by calibration using known coordinates on the plane D (refer to Non-Patent Document 1 below for details of calibration).
Nobuhiro Miichi, Toshikazu Wada, Takashi Matsuyama “Research on Calibration of Projector / Camera System”, [Search June 1, 2005], Internet <URL: http://vision.kuee.kyoto-u.ac. jp / Research / Thesis / Thesis_PDF / Miichi_2002_P_147.pdf>

Therefore, given the tip of the extracted lead with each detection region 7 of the front view image A0 and the point _{p 0,} when substituted into the coordinates (1) of the _{(x cam0, y cam0),} it was calculated ( x _cam1 , y _cam1 ) can be considered to correspond to the position of the lead tip in the perspective image A1. However, when the height of the lead tip changes, the height Z of the plane D also changes accordingly, the homography matrix H _Z changes accordingly, and the value of (x _cam1 , y _cam1 ) also changes. Become.

In ST25, based on this principle, when the upper limit value of the assumed height range (target range of three-dimensional measurement along the optical axis of the camera C0) is the height Z of the plane D, the lower limit value is the height. By executing the equation (1) using the homography matrix H _Z corresponding to the height Z for each of the Z and Z, two points e and f shown in FIG. 19 as (x _cam1 , y _cam1 ) Get the coordinates. Then, in the perspective image A1, as shown in FIG. 19, the line segment gh obtained by translating the line segment ef to each side in the direction perpendicular to the line segment ef by the half-value width k of the detection region 7 on the front view image A0 side. A line segment g′h ′ is set, and a rectangular area ghh′g ′ connecting these four points is set as a detection area 8.

  FIG. 20 shows the size of the detection area 8 when the height range that the lead 6 can take is 0 to 5 mm and when the height range is −15 to 15 mm for the perspective image A1 similar to FIG. 13. This is shown in contrast. As is clear from this example, the detection region 8 becomes smaller as the fluctuation range of the height range is smaller. In FIG. 20, the detection areas 8 are simplified and drawn in parallel with each other. However, in reality, the perspective image A1 is distorted so that a rectangular object is imaged in a trapezoid shape due to the effect of perspective. Each detection region 8 has a non-parallel arrangement such that the distance between the center lines (line segments ef) increases toward the right in the drawing. In each detection area 8, the binarized image for obtaining the edge tip position is projected in a direction perpendicular to the center line of the detection area 8.

  In the above embodiment, since the detection area 7 corresponding to each lead 6 is set in the reference image A0 in order to specify the tip position of the lead, it is necessary to set processing items as many as the number of leads. Instead of this, one detection area that is long in the y-axis direction may be set so as to include the tip of each lead, and a processing item may be provided so that the tip position of each lead can be obtained individually in this area. .

  Next, a case will be described in which the height of each key is inspected for a work (remote control, telephone, etc.) provided with push button type character keys. In this case, as shown in FIG. 21, prior to the inspection, an area including characters drawn on the key 60 is displayed for each key 60 using a front-view image A0 obtained by imaging a non-defective work W. 70 is designated, and the image in the area 70 is registered as a model.

  FIG. 22 shows an inspection procedure. First, in ST31, the cameras C0 and C1 are simultaneously driven to generate an image. This ST 31 corresponds to the “stereo camera image input” processing item in the processing item table 201. In the next ST32, pattern matching processing using a model registered before the inspection is executed for the front-view image A0, and an area that most closely matches the model is specified as a measurement target area. This process is performed for each model corresponding to each key 60, but here, for the sake of simplicity of explanation, the description is limited to one model.

When the measurement target region is specified, in the next ST33, the coordinates of the representative point of this region (for example, the center point of the region) are specified. A plurality of representative points can be specified (for example, points corresponding to a plurality of feature points on a predetermined model).
ST32 and ST33 correspond to “high-precision ECM search” processing items in the processing item table 201. After execution of this processing item, the “camera switching” processing item (not shown) is executed, and the process proceeds to ST34.

In ST34, a search area is set on the perspective image A1 based on the coordinates of the representative point. Also in this case, as in the case where the detection region 8 is set by the lead inspection described above, the homography matrix H _Z is set with the upper limit value and the lower limit value of the height range specified in advance as the height _Z , and the representative points A region in which a representative point can exist on the image A1 is obtained by executing the calculation twice using the upper limit value and the lower limit value of the coordinates and the height range, and the size of the model is added to the range. Is the search area.

  In ST35, pattern matching processing with the model is executed in the search area, and the measurement target area and the position of the representative point in the area are specified. Furthermore, in ST36, a three-dimensional coordinate is calculated using the coordinates of the representative point of the measurement target region in each of the front view and perspective images A0 and A1. In ST37, the Z coordinate of the calculated three-dimensional coordinates is compared with a predetermined threshold value to determine whether the key height is appropriate. In ST38, the determination result is output and the process is terminated. ST35 to ST38 correspond to the processing items of “stereo high-precision ECM search”.

  When the point corresponding to the feature point determined in the model is specified as the representative point on the reference image A0, the corresponding point on the model can be specified in ST36 as well. In ST35, the model should be imaged by the perspective camera C1 using a homography matrix corresponding to a predetermined height within a specified height range (for example, a standard height when the workpiece is normal). The measurement target region may be specified using the converted model.

  FIG. 23 shows the measurement target area 71 specified for the key 60 on the front-view image A0 in the above inspection, the search area 80 on the perspective image A1 side set based on the position and size of the area 71, and A measurement target area 81 specified in the search area 80 is shown.

  In the procedure shown in FIG. 22, only the inspection by the three-dimensional measurement process is performed, but also in this inspection, the print state of the character of each key is inspected by the two-dimensional measurement process using the front-view image A0. be able to.

In the next embodiment, a work having a circular display area at the center is inspected for appropriateness of the height in the display area.
FIG. 24 shows images A0 and A1 of the workpiece W generated by the cameras C0 and C1. In the figure, S is a display area to be inspected. In the front view image A0, as in the previous embodiment, the front image of the workpiece W appears, so that the outline 72 of the display area S is also circular. On the other hand, in the perspective image A1, since the work W is distorted, the size of the display area S and the shape of the outline 72 are also different from those in the front-view image A0, and the arrangement of characters in the display area S The state is also distorted.

  In this embodiment, prior to the inspection, a non-defective product model of the workpiece W is imaged, and the user designates the radius of the display area S and the measurement target area on the obtained front-view image A0. At this time, the image processing unit 22 executes processing for obtaining the position of the center point of the display area S from the front-view image A0, and registers the relative position of the measurement target area with respect to the center point in the registration memory.

  FIG. 25 shows an enlarged display region S in the front view image A0. In the figure, reference numeral 73 denotes a measurement target area designated by the user, and 74 denotes a center point of the display area S. The position of the center point 74 executes a process of extracting a circular pattern from the front-view image A0, and a contour that best matches a circle (position correction model) having a size designated by the user from the extraction result. The line 72 is specified and obtained.

FIG. 26 shows a height inspection procedure for the workpiece W of FIG.
In ST41, as in the previous embodiment, the cameras C0 and C1 are simultaneously driven to generate the front view image A0 and the perspective image A1. This ST 41 corresponds to the “stereo camera image input” processing item in the processing item table 201. In ST42, the position of the center point 74 of the display area S is obtained from the front view image A0 by the above processing. ST42 corresponds to a processing item of “model position correction”.

In the next ST43, the measurement target region 73 is set based on the previously registered relative position with reference to the coordinates of the center point 74 obtained in ST42. In ST44, the image of the measurement target area 73 is registered as a model image for search, and the representative point position of the measurement target area 73 (for example, the center point position in the area) is also registered.
ST44 corresponds to the “stereo area setting” process item. Although not shown in FIG. 26, the “camera switching” processing item is executed after the processing item is executed.

In ST45, a search area 82 (shown in FIG. 27) is set on the perspective image A1. Also in this embodiment, the coordinates of the representative point registered in ST44 are substituted into the equation (1), and the homography matrix H _Z corresponding to the upper limit value and the lower limit value of the height range specified in advance is used (1). By executing the equation, the position and size of the search area 82 are determined.

In ST46, the correlation matching process is executed in the search area 82 using the image registered in ST44 for the measurement target area 73 of the front-view image A0. Then, an area most similar to the registered image is specified, and this is set as a measurement target area on the perspective image A1 side. In ST47, the coordinates of the representative point are obtained for the measurement target region on the perspective image A1 side, and the three-dimensional coordinates are calculated using the coordinates and the coordinates of the representative point on the front view image A0 side. In subsequent ST48, the suitability of the obtained Z coordinate is determined. In ST49, the determination result is output, and then the process ends.
The processes from ST45 to ST49 correspond to the “stereo height measurement” process item in the process item table 201.

  FIGS. 27A and 27B show the measurement target area 73 in the front view image A0, the search area 82 in the perspective image A1, and the measurement target area 83, respectively, for one set of the front view image A0 and the perspective image A1. Further, in the front view image A0, the outline 72 of the display area S is shown with a bold line, and the position where the center point 74 of the display area S is obtained is shown. The position of the workpiece W is different between the image of (1) and the image of (2) in FIG. 27, but in both cases, the measurement target area 73 is set correctly so that the target character is contained therein. Yes.

  According to the inspection procedure shown in FIG. 26, a model image for search is acquired from each work W to be inspected, so that a surface pattern such as characters to be subjected to three-dimensional measurement is provided for each work W. Even in different cases, the same sequence can be applied to perform three-dimensional measurement. Here, although the surface pattern may be different for each workpiece W, the shape of the workpiece W and the position of the measurement target area based on the shape of the workpiece W are common to all the workpieces W. The target area is set.

Also in the procedure of FIG. 26, the registered model image for search is converted into a shape that should be captured by the perspective camera C1 using the homography matrix Hz, and the measurement target region 83 is specified using the converted image. You may do it.
Furthermore, the process of inspecting the print state of the characters in the display area S using the front-view image A0 can also be incorporated into the procedure of FIG.

It is a perspective view which shows the structure of the imaging part of the test | inspection apparatus with which this invention was applied with the example of installation. It is explanatory drawing which shows the example of the image produced | generated with each camera. It is a block diagram of an inspection device. It is a functional block diagram which shows the system configuration for sequence setting. It is explanatory drawing which shows the data structural example of a process item table. It is explanatory drawing which shows the example of a menu display of a sequence. FIG. 5 is an explanatory diagram showing a sequence of “edge positions” and a flowchart showing the flow thereof in association with each other. FIG. 5 is an explanatory diagram showing a “pattern” sequence in association with a flowchart showing the flow of the sequence. It is explanatory drawing which shows the operation procedure in the case of adding another process item to a basic sequence. It is explanatory drawing which shows the example which added the process item regarding 2D measurement to the basic sequence. It is a block diagram of an inspection device. It is a flowchart which shows the procedure concerning a test | inspection of IC. It is explanatory drawing which shows the example of a setting of a detection area. It is a flowchart which shows the procedure of a teaching process. It is explanatory drawing which shows the example of a setting of a positioning area | region. It is a flowchart which shows the detailed procedure of a lead test | inspection. It is explanatory drawing which shows the relationship between the positioning area | region with respect to the workpiece | work in the front view image at the time of a test | inspection, and a detection area. It is explanatory drawing which shows the relationship of the corresponding point between each image. It is explanatory drawing which shows the setting method of a detection area. It is explanatory drawing which shows the relationship between the range of height and a detection area. It is explanatory drawing which shows the example which carries out the model registration of the test | inspection site | part on the workpiece | work which is a quality item. It is a flowchart which shows the procedure concerning inspection of a character key. It is explanatory drawing which shows the example which set the measurement object area | region and the search area | region with respect to the image of a test object. It is explanatory drawing which shows the front view image and perspective view concerning a workpiece | work which have a circular display area. It is explanatory drawing which shows the designation | designated result of the measurement object area | region with respect to a workpiece | work. It is a flowchart which shows the procedure in the case of performing the workpiece | work height inspection of FIG. It is explanatory drawing which shows the example in which a measurement object area | region and a search area follow the position change of a workpiece | work.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image pick-up part 2 Main body part 3 Monitor 4 Console 7, 8 Detection area 24 Processing part 25 Memory 41 Image processing part 42 Measurement processing part 80, 82 Search area 200 In-machine computer 201 Process item table 202 Sequence storage part 203 Program storage part 204 Designated operation accepting unit 205 Sequence assembly unit 206 Sequence execution unit A0 Front view image A1 Perspective image C0, C1 Camera W Workpiece

Claims (17)

An image processing apparatus that performs processing using a first image and a second image based on images captured by a first camera and a second camera that are arranged so as to photograph an object from different directions. There,
Sequence setting means for setting a sequence of processing items;
A sequence execution means for executing the set sequence,
The sequence setting means provides a plurality of items of two-dimensional image processing and at least one item of three-dimensional measurement processing as processing items that can be used as elements of the sequence by user selection,
Among the items of the provided two-dimensional image processing, at least one item on the first image position specifying process for specifying the position on the object in the first image is included,
The three-dimensional measurement process is performed on the object in the second image corresponding to the position on the object in the first image specified in the first image position specifying process belonging to the same sequence as the sequence to which the process belongs. An image processing apparatus including a second image position specifying process for specifying the position of the image.   The three-dimensional measurement process further includes a process of calculating three-dimensional coordinates using the position specified in the first image position specifying process and the position specified in the second image position specifying process. The image processing apparatus according to claim 1.   The image processing according to claim 1 or 2, wherein a plurality of types of first image position specifying processing items and a plurality of types of three-dimensional measurement processing items that can be used as elements of the sequence by user selection are provided. apparatus.   The basic sequence of three-dimensional measurement including at least one item of position identification processing on the first image and at least one item of three-dimensional measurement processing is provided to be available according to user selection. The image processing apparatus described. At least one first on-image position specifying process in which a processing item is provided sets a first detection area for the first image, and detects an edge in the edge detection direction defined for the first detection area. To identify the location,
The second image position specifying process in the at least one three-dimensional measurement process in which the process item is provided includes the positional relationship between the first camera and the second camera and the first image position specifying process with respect to the second image. A second detection area having an edge detection direction as a specific direction based on the edge position specified in step 1 is set, and the edge position in the edge detection direction in the second detection area is specified. Item 3. The image processing apparatus according to Item 1 or 2.   The second detection region is set based on a range along the optical axis of the first camera designated by the user as the target range of the three-dimensional measurement in the range along the edge detection direction. Image processing apparatus. The first camera has a rectangular field of view, and is arranged to shoot from the direction in which the object is viewed from the front,
The second camera is located at a position away from the first camera in the vertical or horizontal direction in the field of view of the first camera, and is inclined from the direction in which the object is viewed from the front. Are arranged to shoot
The first image position specifying means sets the first detection area to a position based on a user instruction with respect to the first image,
The image processing apparatus according to claim 5, wherein the edge detection direction of the first detection region is a direction in which the first camera and the second camera are arranged. The at least one first image position specifying process in which the processing item is provided is for specifying a representative position of an area on the first image having a high degree of coincidence with a model image registered in advance before the execution of measurement. ,
The second on-image position specifying process in at least one three-dimensional measurement process provided with a processing item specifies a representative position of an area on the second image having a high degree of coincidence with the model image. Item 3. The image processing apparatus according to Item 1 or 2.   The second image position specifying process sets a search area on the second image based on the positional relationship between the first camera and the second camera and the representative position specified in the first image position specifying process, The image processing apparatus according to claim 8, wherein a representative position of an area having a high degree of coincidence with the model image is specified in a search area.   The image processing apparatus according to claim 9, wherein the search region is further set based on a range along the optical axis of the first camera designated by the user as a target range for three-dimensional measurement. The at least one first on-image position specifying process in which the processing item is provided specifies a representative position of the measurement target area set based on a user instruction on the first image, and models the image in the measurement target area. Which is acquired as an image,
The second on-image position specifying process in at least one three-dimensional measurement process provided with a processing item specifies a representative position of an area on the second image having a high degree of coincidence with the model image. Item 3. The image processing apparatus according to Item 1 or 2.   The second image position specifying process sets a search area on the second image based on the positional relationship between the first camera and the second camera and the representative position specified in the first image position specifying process, The image processing apparatus according to claim 11, wherein a representative position of an area having a high degree of coincidence with the model image is specified in a search area.   The image processing apparatus according to claim 11, wherein the search area is further set based on a range along the optical axis of the first camera designated by the user as a target range for three-dimensional measurement. An image processing method using a first image and a second image based on images respectively captured by a first camera and a second camera arranged to photograph an object from different directions,
A sequence setting step for setting a sequence of processing items;
A sequence execution step for executing a set sequence;
The sequence setting step provides a plurality of items of two-dimensional image processing and at least one item of three-dimensional measurement processing as processing items that can be elements of the sequence by user selection,
Among the items of the provided two-dimensional image processing, at least one item on the first image position specifying process for specifying the position on the object in the first image is included,
The three-dimensional measurement process is performed on the object in the second image corresponding to the position on the object in the first image specified in the first image position specifying process belonging to the same sequence as the sequence to which the process belongs. An image processing method including a second on-image position specifying process for specifying the position of the image.   The three-dimensional measurement process further includes a process of calculating three-dimensional coordinates using the position specified in the first image position specifying process and the position specified in the second image position specifying process. The image processing method according to claim 14. A first camera having a rectangular field of view and arranged to take an image of the object from a front view; and a vertical or horizontal direction in the field of view of the first camera relative to the first camera First images based on images respectively taken by a second camera located at a position apart from each other and arranged so as to photograph the object from a direction oblique to the direction in which the object is viewed from the front And an image processing apparatus that performs processing using the second image,
First on-image position specifying means for specifying a position on the object in the first image;
Second on-image position specifying means for specifying a position on the object in the second image corresponding to a position on the object in the first image specified by the on-image position specifying processing means;
Three-dimensional coordinate calculating means for calculating three-dimensional coordinates using the position specified by the first image position specifying means and the position specified by the second image position specifying means;
The first image position specifying means sets the first detection area to a position based on a user instruction with respect to the first image, and sets the edge position in the edge detection direction defined for the first detection area. The edge detection direction of the first detection area is the direction of the arrangement of the first camera and the second camera,
The second image position specifying means is set for the second image based on the positional relationship between the first camera and the second camera and the edge position specified by the first image position specifying means. An image processing apparatus for setting a second detection area having one direction as an edge detection direction and specifying the position of an edge in the edge detection direction in the second detection area. A first camera having a rectangular field of view and arranged to take an image of the object from a front view; and a vertical or horizontal direction in the field of view of the first camera relative to the first camera First images based on images respectively taken by a second camera located at a position apart from each other and arranged so as to photograph the object from a direction oblique to the direction in which the object is viewed from the front And an image processing method for performing processing using the second image,
A first image position specifying step for specifying a position on the object in the first image;
A second on-image position specifying step for specifying a position on the object in the second image corresponding to a position on the object in the first image specified by the on-image position specifying processing means;
A three-dimensional coordinate calculating step for calculating a three-dimensional coordinate using the position specified in the first image position specifying step and the position specified in the second image position specifying step;
In the first image position specifying step, the first detection area is set to a position based on a user instruction with respect to the first image, and the edge position in the edge detection direction defined for the first detection area is set. The edge detection direction of the first detection area is the direction of the arrangement of the first camera and the second camera,
The second image position specifying step is a setting that is set for the second image based on the positional relationship between the first camera and the second camera and the edge position specified by the first image position specifying means. An image processing method for setting a second detection area having one direction as an edge detection direction and specifying the position of the edge in the edge detection direction in the second detection area.

Images