JP2018144166A - Image processing device, image processing method, image processing program and recording medium readable by computer as well as equipment with the same recorded - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow interference of an end effector to be easily determined.SOLUTION: The image processing device comprises: an end effector model registration part 8u for registering an end effector model, three-dimensional CAD data, which virtually expresses a three-dimensional shape of an end effector; an additional model preparation part 8v for preparing an additional model in which an additional region expressed by one or more prescribed three-dimensional graphics is added to a surface of the end effector model; and an interference determination part 8m for performing interference determination by which presence and absence of interference of the effector with other objects that may interfere operation of the end effector is determined when operating the effector by the additional model prepared by the additional model preparation part 8v.SELECTED DRAWING: Figure 31

Description

  The present invention relates to an image processing apparatus, an image processing method, an image processing program, a computer-readable recording medium, and a recorded device. For example, the present invention relates to a sensor for detecting a three-dimensional shape of a plurality of workpieces stacked in a work space. Image processing device, image processing method, image processing program, and computer readable for controlling a robot that performs a bulk picking operation that is measured by a head and gripped by an end effector provided at the tip of a robot arm and sequentially taken out The present invention relates to a recording medium and recorded equipment.

  The robot vision is combined with the manipulator, the target work is imaged with the robot vision, height information is acquired, the appropriate position is gripped (pick), and placed at the desired position (place or placement) ) Possible robotic devices have been developed. Using such a robotic device, a large number of workpieces placed in a returnable box, called bulk picking, are imaged with the cameras and sensors that make up the robot vision to determine the posture and the appropriate gripping position. After that, the robot arm is moved to this gripping position, gripped by an end effector such as a hand provided at the tip of the arm, and placed at a predetermined position outside the return box. It has been broken.

  In such picking using robot vision, the positional relationship between the workpiece and the robot when the workpiece is gripped is registered as a gripping position at the time of setting, and the robot grips the workpiece detected by the robot vision during actual operation The position is calculated, and the robot is moved to the calculated position and picked.

  In such bulk picking, the end effector may interfere with other obstacles such as other works and boxes when gripping the work. In order to avoid this, it is conceivable to calculate the position of the end effector at the gripping position, determine interference with surrounding objects, and output a solution that does not interfere as an effective solution. Here, the surrounding objects include a three-dimensional point group obtained by measurement with a sensor unit, a box or a floor registered in advance.

  When creating such an end effector interference determination model, it is considered that the method of reading the end effector's three-dimensional CAD data is the least laborious. For example, a method has been proposed in which three-dimensional CAD data is read to perform interference determination (Patent Document 1). In this method, three-dimensional CAD data of an end effector is read, and interference determination is performed using the end effector model. However, when it is desired to add an interference area, it is necessary to edit the 3D CAD data using a 3D CAD program or the like to create and add the shape of the interference area, which requires a complicated operation. It was.

  In addition, a method of creating and substituting an end effector model with a rectangular parallelepiped has been proposed (Patent Document 2). In this method, in order to imitate the end effector, an end effector substitute model is created by combining rectangular parallelepipeds. However, it is not easy to create a model close to a real end effector only by combining cuboids, and there is a problem in terms of reproducibility.

Japanese Patent Laying-Open No. 2015-9314 Japanese Patent No. 3786679

  The present invention has been made in view of such circumstances, and one object of the present invention is to read an image processing apparatus, an image processing method, an image processing program, and a computer that can easily determine the interference of an end effector. An object is to provide a possible recording medium and a recorded device.

Means for Solving the Problems and Effects of the Invention

  According to the image processing apparatus of the first embodiment of the present invention, the three-dimensional shape of a plurality of workpieces stacked in the work space is measured by the sensor unit and held by the end effector provided at the tip of the robot arm unit. An image processing apparatus for controlling a robot that performs a picking operation, and virtually represents a three-dimensional shape of a work model registration unit for registering a work model indicating a three-dimensional shape of a work, and an end effector. When the work model registered by the work model registration unit is gripped by an end effector model registration unit for registering an end effector model which is three-dimensional CAD data and the end effector registered by the end effector model registration unit Set the position and orientation of the end effector and the gripping position of the work model. A gripping position specifying unit, an additional model creating unit for creating an additional model in which an additional region expressed by one or more predefined three-dimensional figures is added to the surface of the end effector model, and the gripping The end effector model registered by the end effector model registration unit when the gripping position of the work model set by the position specifying unit is held at the position and posture of the end effector set by the holding position specifying unit. And an interference determination unit for performing interference determination to determine whether the additional model created by the additional model creation unit interferes with other objects that may interfere with the gripping operation. . With the above configuration, it is easy to add an interference determination area with a simple shape such as a rectangular parallelepiped or a cylinder to the end effector model without editing the 3D CAD data, and perform interference determination according to each area. Interference determination can be performed.

  Moreover, according to the image processing apparatus which concerns on a 2nd form, in addition to the said structure, the said additional model creation part can be comprised so that a rectangular parallelepiped or a cylinder can be selected as a figure prescribed | regulated previously. With the above configuration, an additional region can be created with a simple shape and added to the work model, so that the user can easily create an additional model for performing interference determination.

  Furthermore, according to the image processing apparatus according to the third embodiment, in addition to any of the above-described configurations, the additional model creation unit can configure an additional region by combining a plurality of predefined figures. With the above configuration, it is possible to increase variations of shapes that can be expressed by combinations of figures.

  Furthermore, according to the image processing device according to the fourth aspect, in addition to any of the above-described configurations, the additional model creation unit may include the additional region between the flange surface at the tip of the arm unit and the end effector model. It can be configured to be insertable between them.

  Furthermore, according to the image processing apparatus according to the fifth aspect, in addition to any of the above-described configurations, the additional model creation unit can be configured to have the additional region formed in a columnar shape or a prismatic shape extending in one direction. And can. With the above configuration, an advantage that an additional region imitating a cable or a harness attached to the end effector model or the arm portion can be easily configured is obtained.

  Furthermore, according to the image processing device according to the sixth aspect, in addition to any of the above-described configurations, the image processing device further has a three-dimensional shape from an image including a plurality of workpieces measured by the sensor unit and surrounding objects. An input image acquisition unit that acquires an input image is provided, and the interference determination unit moves the end effector to the position so as to grip any of a plurality of workpieces included in the input image acquired by the input image acquisition unit. In order to determine whether or not an object around the workpiece interferes with the end effector, an interference determination is performed on an image constituting the object around the workpiece in the input image. it can. With the above configuration, it is possible to determine the interference by comparing the end effector model with the input image.

  Furthermore, according to the image processing device according to the seventh aspect, in addition to any of the above configurations, each of the plurality of work groups included in the input image acquired by the input image acquisition unit A search model registration unit for registering a search model that virtually represents the three-dimensional shape of the workpiece, which is used when performing a three-dimensional search for specifying the posture and position of the workpiece, and acquired by the input image acquisition unit A three-dimensional search unit for performing a three-dimensional search for specifying an image region corresponding to the position and orientation of each workpiece using the search model registered in the search model registration unit from the input image, The interference determination unit specifies the gripping position specifying unit to grip the workpiece with respect to the search result corresponding to the position and posture of each workpiece searched from the input image by the three-dimensional search unit. When placing the end effector model the gripping position, it can be configured to determine the presence or absence of interference between an object existing around the workpiece.

  Furthermore, according to the image processing device of the eighth aspect, in addition to any of the above-described configurations, the end effector model can be configured with STL data.

  Furthermore, according to the image processing device of the ninth aspect, in addition to any of the above-described configurations, the input image can be configured with a set of three-dimensional point cloud data.

  Furthermore, according to the image processing method according to the tenth embodiment, the three-dimensional shape of a plurality of workpieces stacked in the work space is measured by the sensor unit, and is gripped by the end effector provided at the tip of the robot arm unit. An image processing method for controlling a robot that performs a picking operation, the step of registering an end effector model, which is three-dimensional CAD data that virtually represents a three-dimensional shape of an end effector, and the registered end effector In addition to setting the position and orientation of the end effector and the gripping position of the work model when gripping a work model that shows the three-dimensional shape of the work, an additional area expressed by one or more predefined three-dimensional figures Creating an additional model added to the surface of the end effector model, and the set work module. When the gripping position of the robot is gripped at the set end effector position and posture, the registered end effector model and the created additional model may interfere with the gripping operation. Determining whether to interfere with the object. This makes it easy to add an interference determination area with a simple shape such as a rectangular parallelepiped or a cylinder to the end effector model without editing the 3D CAD data, and perform interference determination according to each area. Interference determination can be performed.

  Furthermore, according to the image processing program according to the eleventh embodiment, the three-dimensional shape of a plurality of workpieces stacked in the work space is measured by the sensor unit, and is gripped by the end effector provided at the tip of the arm unit of the robot. An image processing program for controlling a robot that performs a picking operation, including a function for registering a work model indicating a three-dimensional shape of a work, and three-dimensional CAD data that virtually represents the three-dimensional shape of an end effector. A function for registering an end effector model, and a function for setting the position and orientation of the end effector and the gripping position of the work model when gripping a work model showing the three-dimensional shape of the work by the registered end effector. And an additional area expressed by one or more predefined three-dimensional figures, the end effector A function of creating an additional model added to the surface of the Dell, and when the gripping position of the set work model is gripped by the set position and posture of the end effector, the registered end effector model The computer can realize a function of determining whether or not the created additional model interferes with another object that may interfere with the gripping operation. With the above configuration, it is easy to add an interference determination area with a simple shape such as a rectangular parallelepiped or a cylinder to the end effector model without editing the 3D CAD data, and perform interference determination according to each area. Interference determination can be performed.

  Furthermore, a computer-readable recording medium or a stored device according to the twelfth embodiment stores the image processing program. Recording media include CD-ROM, CD-R, CD-RW, flexible disk, magnetic tape, MO, DVD-ROM, DVD-RAM, DVD-R, DVD + R, DVD-RW, DVD + RW, Blu-ray, HD A medium that can store a program such as a magnetic disk such as a DVD (AOD), an optical disk, a magneto-optical disk, a semiconductor memory, or the like is included. The program includes a program distributed in a download manner through a network line such as the Internet, in addition to a program stored and distributed in the recording medium. Furthermore, the stored devices include general-purpose or dedicated devices in which the program is implemented in a state where it can be executed in the form of software, firmware, or the like. Furthermore, each process and function included in the program may be executed by computer-executable program software, or each part of the process or hardware may be executed by hardware such as a predetermined gate array (FPGA, ASIC), or program software. And a partial hardware module that realizes a part of hardware elements may be mixed.

It is a schematic diagram which shows a mode that the bulk picking operation | movement is performed using a robot system. It is a block diagram of a robot system. It is a perspective view which shows an example of a sensor part. 4A is a schematic cross-sectional view showing an example in which workpieces are randomly placed in a storage container and stacked, FIG. 4B is a schematic cross-sectional view showing an example in which workpieces are stacked on a floor surface, and FIG. It is a perspective view which shows the state arranged on the top. 5A is a schematic diagram showing an example of gripping a workpiece with an end effector, FIG. 5B is a schematic diagram showing an example of gripping a workpiece having a cavity from the inner surface, and FIG. 5C shows an example of sucking and gripping a plate-like workpiece. It is a schematic diagram. 1 is a block diagram illustrating a robot system according to a first embodiment. It is a perspective view which shows the work model constructed | assembled by three-dimensional CAD data. It is an image figure which shows the state which sets the origin of a search model with respect to the work model of FIG. 9A is a basic direction image when the work model of FIG. 7 is viewed from the positive direction of the X axis, FIG. 9B is a basic direction image when viewed from the positive direction of the Y axis, and FIG. 9C is a basic direction image when viewed from the positive direction of the Z axis. FIG. 9D is a basic direction image viewed from the negative direction of the Z-axis. It is an image figure which shows the search model registration screen which registers a basic direction image as a search model of a three-dimensional search. It is an image figure which shows the measurement data which imaged the workpiece | work corresponding to the workpiece | work model of FIG. 7 from the positive direction of the X-axis. 12A is an image diagram showing a state where feature points are extracted from the image viewed from the X-axis direction of FIG. 9A, and FIG. 12B is an image diagram showing a state where the image of FIG. 12A is displayed in three dimensions. 13A is an input image in which a work group is displayed in two dimensions, FIG. 13B is an input image in which FIG. 13A is displayed in three dimensions, FIG. 13C is a result of performing a three-dimensional search on FIG. 13A, and FIG. It is an image figure which each shows the state made to display three-dimensionally. It is an image figure of the user interface screen which shows a grip registration screen. It is an image figure which shows a holding | grip posture addition screen. 16A is a state in which a plurality of cubic workpieces are stacked, FIG. 16B is a state in which this work model is registered in the search model, and FIG. 16C is a conceptual diagram showing a state in which a gripping posture is registered in the work model in FIG. It is. It is a functional block diagram which shows the robot system which concerns on Embodiment 2. It is an image figure which shows the state before rotating an end effector model according to a ZYX system Euler angle. It is an image figure which shows the state rotated 90 degrees around the Z-axis from FIG. It is an image figure which shows the state rotated 90 degrees around the Y-axis from FIG. It is an image figure which shows the state rotated 90 degrees around the X-axis from FIG. It is a schematic diagram showing a rotation shaft of R _Y and R _Z in the state of FIG. 21. In the state of FIG. 22 is an image diagram showing a state of displaying the corrected rotational shaft of R _Z. It is an image figure which shows the state which displays the correction | amendment rotating shaft of _RY in the state of FIG. In the state of FIG. 22 is an image diagram showing a state of displaying the corrected rotational shaft of R _X. It is a block diagram which shows the robot system which concerns on Embodiment 3. FIG. It is a flowchart which shows the procedure of the teaching work performed before actual operation. It is a flowchart which shows the procedure which registers 3D CAD data as a search model. It is a flowchart which shows the procedure which registers an end effector model. It is a flowchart which shows the procedure of registration of the end effector model which added the additional area | region. It is a block diagram which shows the robot system which concerns on Embodiment 4. It is an image figure which shows an additional area | region setting screen. It is an image figure which shows a basic figure edit screen. It is a flowchart which shows the procedure which registers the holding position and attitude | position which hold | grip a workpiece | work with an end effector. FIG. 35A is an image diagram of a work model composed of three-dimensional CAD data, and FIG. 35B is an image diagram showing an XY designation screen. It is an image figure which shows a _ZRZ designation _| designated screen. It is an image figure which shows a _RY designation _| designated screen. It is a schematic diagram showing the R _X designation screen. It is an image figure which shows a position parameter designation | designated screen. It is an image figure which shows a XYZ designation | designated screen. It is an image figure which shows a mode that a gripping position is designated in the state which projected the work model of FIG. 35B from the diagonal direction. It is an image figure which shows a mode that the holding position on a three-dimensional image viewer is designated. It is an image figure which shows a flat workpiece. 44A is a plan view of a work model representing the work of FIG. 43, FIG. 44B is a bottom view, FIG. 44C is a front view, FIG. 44D is a rear view, FIG. 44E is a right side view, and FIG. FIG. 44 is an image diagram showing a result of performing a three-dimensional search on point cloud data obtained by imaging a workpiece group in which a plurality of workpieces in FIG. 43 are stacked. It is a schematic diagram which shows the example which represents the attitude | position of a workpiece | work using the Euler angle of a ZYZ type | system | group. It is an image figure which shows a search model registration screen. It is a flowchart which shows the registration procedure of the search model which provided attitude | position restriction | limiting. It is an image figure which shows the inclination angle and rotation angle setting screen. 50A is a posture of a workpiece at the time of registration, FIG. 50B is a posture of an input image, and FIG. 50C is an image diagram showing an inclination angle. 51A is a workpiece posture at the time of registration, FIG. 51B is a posture of an input image, and FIG. 51C is an image diagram showing a rotation angle. It is a flowchart which shows the procedure which calculates | requires an inclination angle and a rotation angle from a three-dimensional attitude | position. 53A is a workpiece posture at the time of registration of a search model, FIG. 53B is a posture of an input image, and FIG. 53C is an image diagram showing a state of obtaining an inclination angle. 54A is a view showing a state in which a rotation axis is shown in the search model of FIG. 53C, and FIG. 54B is an image view showing a state in which the Z′-axis in FIG. 54A is three-dimensionally rotated so as to coincide with the Z-axis. 55A is a posture of the workpiece at the time of registration, FIG. 55B is a state in which the inclination of FIG. 54B is removed, and FIG. 55C is a conceptual diagram showing how the rotation angle when viewed from the Z vertical direction of FIG. It is an image figure which shows an end effector attachment position setting screen. It is an image figure which shows a holding | grip position designation | designated screen. It is an image figure which shows a multiple holding position selection screen. It is an image figure which shows an end effector attachment position calibration screen. It is a flowchart which shows the procedure which calibrates an end effector attachment position automatically. It is an image figure which shows an end effector imaging screen. It is a flowchart which shows the procedure which registers actual measurement data as a search model. It is a flowchart which shows the procedure which determines the presence or absence of the grasping solution of each workpiece | work at the time of actual operation. 49 is a flowchart showing a procedure in actual operation in which a picking operation is performed in a state where a search model is registered in the procedure of FIG. 48. It is a flowchart which shows the procedure of the interference determination using a cross-sectional model. It is an image figure which shows an end effector model, its basic axis, and a cross-sectional position. 67A is a cross-sectional SS1 of FIG. 66, FIG. 67B is a cross-sectional SS2, FIG. 67B is a cross-sectional SS3, FIG. 67D is a cross-sectional SS4, and FIG. 68A is an image diagram showing the three-dimensional point and the basic axis of the end effector model of FIG. 66, FIG. 68B is a state in which the projection point of the three-dimensional point of FIG. 68A does not interfere with the section SS3, and FIG. It is an image figure which shows each state. It is a flowchart which shows the procedure of the interference determination using an additional model. FIG. 10 is a block diagram illustrating a robot system according to a seventh embodiment. 18 is a flowchart illustrating a procedure for determining whether or not each workpiece has a gripping solution during actual operation according to the seventh embodiment. It is a perspective view which shows an example of a workpiece | work. 73A is a height image A when the workpiece of FIG. 72 is viewed from the positive direction side of the Z axis, FIG. 73B is a height image B when viewed from the negative direction side of the Z axis, and FIG. 73D is a height image D as viewed from the negative direction side of the X axis, FIG. 73E is a height image E as viewed from the positive direction side of the Y axis, and FIG. 73F is from a negative direction side of the Y axis. It is an image figure which shows the seen height image F, respectively. It is an image figure which shows a workpiece | work selection screen. It is an image figure which shows the grip solution candidate display screen which selected the grip OK. It is an image figure which shows the grip solution candidate display screen which selected grip NG. It is an image figure which shows the further another example of the grip solution candidate display screen which selected the grip NG. It is an image figure which shows the example of the grip solution candidate display screen with respect to the search model C. FIG. 79 is an image diagram showing an example of a gripping solution candidate display screen in which another gripping position candidate is selected in FIG. 78. It is an image figure which shows the example of the grip solution candidate display screen which added the grip position and became grip OK. FIG. 10 is a block diagram illustrating a robot system according to an eighth embodiment. 10 is a flowchart illustrating a search model registration procedure according to an eighth embodiment. 83A is a state in which the gripping posture is registered in the basic direction image in FIG. 73A, FIG. 83B is a state in which the gripping posture is registered in the basic direction image in FIG. 73B, and FIG. 83C is a state in which the gripping posture is registered in the basic direction image in FIG. 83D is a state in which the gripping posture is registered in the basic direction image of FIG. 73D, FIG. 83E is a state in which the gripping posture is registered in the basic direction image of FIG. 73E, and FIG. 83F is a state in which the gripping posture is registered in the basic direction image of FIG. It is an image figure which shows a state, respectively. It is an image figure which shows a workpiece | work selection screen. It is an image figure which shows the holding | grip solution candidate display screen which selected the holding | grip attitude | position C-000. It is an image figure which shows the grip solution candidate display screen which selected grip posture A-000. 16 is a flowchart illustrating a procedure for performing a three-dimensional search and a gripping possibility determination during actual operation according to the eighth embodiment. FIG. 10 is a block diagram illustrating a robot system according to a ninth embodiment. It is an image figure which shows a function selection screen. It is an image figure which shows a search model registration method selection screen. It is an image figure which shows the measurement data imaging screen. It is an image figure which shows a search model exclusion area | region setting screen. It is an image figure which shows a rotational symmetry setting screen. It is an image figure which shows a search model registration screen. It is an image figure which shows the search model registration screen which added the search model. It is an image figure which shows a search area | region setting screen. It is an image figure which shows the search area | region setting screen which displayed the floor surface designation | designated dialog. It is an image figure which shows the search area | region setting screen which displayed the floor surface information. It is an image figure which shows a search parameter setting screen. It is an image figure which shows the search parameter setting screen which displayed the detailed detection condition setting dialog. It is an image figure which shows the search parameter setting screen designated for every search model. It is an image figure which shows a search model registration method selection screen. It is an image figure which shows a CAD data reading screen. It is an image figure which shows the search model registration screen based on three-dimensional CAD data. It is an image figure which shows the search model registration screen which displayed the rotational symmetry designation | designated dialog. It is an image figure which shows a model list display screen. It is an image figure which shows a search result display screen. It is an image figure which shows a search result display screen. It is an image figure which shows a search result display screen. It is an image figure which shows a search result display screen. It is an image figure which shows a search result display screen. It is an image figure which shows a search result display screen. It is an image figure which shows a search result display screen. It is an image figure which shows a model edit screen. It is an image figure which shows a model area detailed setting screen. It is an image figure which shows the feature detailed setting screen. It is an image figure which shows a three-dimensional pick initial setting screen. It is an image figure which shows an end effector model setting screen. It is an image figure which shows an end effector model edit screen. It is an image figure which shows a parts addition screen. It is an image figure which shows a CAD position and orientation setting screen. It is an image figure which shows a CAD position and orientation setting screen. It is an image figure which shows a CAD position and orientation setting screen. It is an image figure which shows an end effector model edit screen. It is an image figure which shows an additional area position and orientation setting screen. It is an image figure which shows an end effector model edit screen. It is an image figure which shows an end effector model edit screen. It is an image figure which shows an end effector model edit screen. It is an image figure which shows an end effector model edit screen. It is an image figure which shows an end effector model edit screen. It is an image figure which shows a grip registration screen. It is an image figure which shows a holding | grip setting dialog. It is an image figure which shows a setting method selection dialog. It is an image figure which shows an XY designation | designated screen. It is an image figure which shows a _ZRZ designation _| designated screen. It is an image figure which shows a _ZRZ designation _| designated screen. It is an image figure which shows a _ZRZ designation _| designated screen. It is an image figure which shows a _RY designation _| designated screen. It is an image figure which shows a _RY designation _| designated screen. It is a schematic diagram showing the R _X designation screen. It is a schematic diagram showing the R _X designation screen. It is an image figure which shows a grip registration screen. It is an image figure which shows a grip registration screen. It is an image figure which shows a condition verification screen. It is an image figure which shows a verification dialog. It is an image figure which shows a detection result display dialog. It is an image figure which shows a detection result display dialog. It is an image figure which shows a detection condition detailed setting screen. It is an image figure which shows a place setting screen. It is a flowchart which shows the procedure which performs deviation | shift correction | amendment manually. It is an image figure showing an example of a shift amendment screen when there is a shift which cannot be ignored. It is an image figure which shows the example of the deviation correction screen in case a deviation can be disregarded. It is an image figure which shows the example of the deviation correction screen provided with the manual correction and automatic correction function of deviation.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiment described below is an example for embodying the technical idea of the present invention, and the present invention is not limited to the following. Moreover, this specification does not specify the member shown by the claim as the member of embodiment. In particular, the dimensions, materials, shapes, relative arrangements, and the like of the component parts described in the embodiments are not intended to limit the scope of the present invention only to specific examples unless otherwise specifically described. Only. Note that the size, positional relationship, and the like of the members shown in each drawing may be exaggerated for clarity of explanation. Furthermore, in the following description, the same name and reference sign indicate the same or the same members, and detailed description will be omitted as appropriate. Furthermore, each element constituting the present invention may be configured such that a plurality of elements are configured by the same member and the plurality of elements are shared by one member. It can also be realized by sharing.
(Embodiment 1)

  FIG. 1 shows a configuration example of a robot system 1000 for picking a workpiece to be picked as a first embodiment. This example shows an example in which a plurality of workpieces WK stacked in the work space are sequentially picked up using a robot RBT and placed in a predetermined position for bulk stacking picking. The robot RBT is also called a manipulator or the like, and includes an arm part ARM and an end effector HND provided at the tip of the arm part ARM. The arm part ARM includes a plurality of movable parts, and moves the end effector EET to a desired position by the angle formed by the two arms and the rotation of the arm fulcrum. The end effector EET can grip or release the workpiece WK.

  The operation of the robot RBT is controlled by the robot controller 6. The robot controller 6 controls the movement of the arm part ARM and the opening / closing operation of the end effector EET. Also, the robot controller 6 acquires information necessary for controlling the robot RBT from the image processing apparatus 100. For example, a workpiece WK, which is a large number of parts randomly placed in the storage container BX, is acquired by a sensor unit 2 such as a three-dimensional camera or illumination, and the calculation unit 10 of the image processing apparatus 100 calculates the workpiece. The position and orientation are detected and information is sent to the robot controller 6. The robot controller 6 grips the workpieces WK one by one with the end effector EET provided at the tip of the arm part ARM of the robot RBT, and arranges them on a predetermined position on the stage STG, for example, on the conveyor belt.

  A functional block diagram of the robot system 1000 is shown in FIG. A robot system 1000 shown in this figure includes an image processing apparatus 100, a sensor unit 2, a display unit 3, an operation unit 4, a robot body 5, a robot controller 6, and a robot operation tool 7.

  The operation unit 4 performs settings related to image processing. The sensor unit 2 acquires a three-dimensional shape obtained by imaging the workpiece. Furthermore, the setting and the operation state are confirmed on the display unit 3. Further, the calculation unit 10 performs a three-dimensional search, interference determination, grip solution calculation, and the like. On the other hand, the robot controller 6 controls the robot according to the result of the calculation unit 10. The robot operation tool 7 performs operation settings for the robot. In the example of FIG. 2, the operation unit 4 and the robot operation tool 7 are separate members, but they may be common members.

  The sensor unit 2 is a member called a robot vision or the like that images a work space or a workpiece. Three-dimensional shape data indicating the three-dimensional shape of the stacked workpieces is acquired from the image captured by the sensor unit 2. Note that methods for obtaining a three-dimensional shape include a pattern projection method, a stereo method, a lens focus method, a light cutting method, an optical radar method, an interference method, and a TOF method. In this embodiment, the phase shift method is used in the pattern projection method.

  The configuration of the sensor unit 2 is determined according to a three-dimensional shape measurement technique. The sensor unit 2 includes a camera, an illumination, a projector, or the like. For example, when measuring the three-dimensional shape of the workpiece by the phase shift method, the sensor unit 2 includes a projector PRJ and a plurality of cameras CME1, CME2, CME3, and CME4 as shown in FIG. In addition, the sensor unit may be configured by a plurality of members such as a camera and a projector, or may be configured integrally. For example, a 3D imaging head that is formed into a head shape by integrating a camera and a projector can be used as the sensor unit.

  Also, the generation itself of the three-dimensional shape data can be performed on the sensor unit side. In this case, an image processing IC or the like that realizes a three-dimensional shape data generation function is provided on the sensor unit side. Alternatively, the generation of 3D shape data is not performed on the image processing device side, and the raw image captured by the sensor unit is processed on the image processing device side to generate 3D shape data such as a 3D image. Also good.

  Furthermore, based on the image captured by the sensor unit 2, calibration described later is executed, and the actual position coordinates of the workpiece WK (coordinates of the movement position of the end effector EET) and the display unit 3 are displayed. Can be linked to the position coordinates on the image.

  The image processing apparatus 100 performs a three-dimensional search, interference determination, gripping solution calculation, and the like based on the three-dimensional shape data of the workpiece obtained in this way. The image processing apparatus 100 can use a general-purpose computer in which a dedicated image processing program is installed, a dedicated image processing controller, a robot vision apparatus, or the like. 2 illustrates an example in which the sensor unit 2, the robot controller 6, and the like are configured by members separate from the image processing apparatus 100. However, the present invention is not limited to this configuration. For example, the sensor unit and the image processing unit The apparatus can be integrated, or a robot controller can be incorporated in the image processing apparatus. Thus, the division of the members shown in FIG. 2 is an example, and a plurality of members can be integrated. For example, the operation unit 4 that operates the image processing apparatus 100 and the robot operation tool 7 that operates the robot controller 6 may be a common member.

  However, the sensor unit 2 is separated from the robot body 5. That is, the present invention is directed to a form called an off-hand type in which the sensor unit 2 is not provided in the arm unit ARM of the robot body 5. In other words, an aspect called an on-hand type in which the sensor unit is provided in the end effector is not included in the present invention.

  The display unit 3 is a member for displaying the three-dimensional shape of the workpiece acquired by the image processing apparatus 100 and for confirming various settings and operation states. A liquid crystal monitor, an organic EL display, a CRT, or the like can be used. . The operation unit 4 is a member for performing various settings such as image processing, and an input device such as a keyboard or a mouse can be used. Further, by using the display unit 3 as a touch panel, the operation unit and the display unit can be integrated.

  For example, when the image processing apparatus 100 is configured by a computer in which an image processing program is installed, a graphical user interface (GUI) screen of the image processing program is displayed on the display unit 3. Various settings can be performed from the GUI displayed on the display unit 3, and processing results such as simulation results can be displayed. In this case, the display unit 3 can be used as a setting unit for performing various settings.

  The robot controller 6 controls the operation of the robot based on information captured by the sensor unit 2. The robot operation tool 7 is a member for setting the operation of the robot body 5, and a pendant or the like can be used.

  The robot body 5 includes a movable arm part ARM and an end effector EET fixed to the tip of the arm part ARM. The robot body 5 is driven by the robot controller 6 to operate the arm part ARM, pick one workpiece WK, move it to a desired position, place it, and release it. Therefore, an end effector EET for gripping the workpiece WK is provided at the tip of the arm part ARM. Examples of the placement position for placing the workpiece WK include a tray and a conveyor.

  As shown in FIG. 1, a plurality of workpieces WK are randomly stored in a storage container BX such as a return box. The sensor unit 2 is disposed above the work space. The sensor unit 2 includes a camera and illumination, and the sensor unit 2 can measure the three-dimensional shape of the workpiece WK. Based on the three-dimensional shape of the workpiece WK measured by the sensor unit 2, the robot controller 6 identifies the workpiece WK to be gripped from a plurality of workpieces, and controls the robot to grip the workpiece WK. To do. Then, while holding the workpiece WK, the arm unit ARM is operated to move to a predetermined placement position, for example, on the stage STG, and the workpiece WK is placed in a predetermined posture. In other words, the robot controller 6 identifies the workpiece WK to be picked by the sensor unit 2, grips this workpiece WK with the end effector EET, and places the gripped workpiece WK in a predetermined reference posture. The operation of the robot is controlled so that the end effector EET is opened at a certain mounting position.

  Here, in the present specification, bulk picking means that the work WK randomly placed in a storage container BX as shown in FIG. 4A is gripped by a robot and placed at a predetermined position. An example of gripping and placing the workpiece WK stacked in a predetermined area without using a storage container as shown in FIG. 4B, or a workpiece stacked and arranged in a predetermined posture as shown in FIG. 4C It is also used to include an example of sequentially holding and placing WK. Further, it is not always necessary that the workpieces are stacked, and workpieces placed randomly on a plane without overlapping workpieces are also referred to as loose stacking in this specification (sequentially). This is the same reason that picking is done and there is no overlap between the workpieces at the end of picking. Note that the present invention is not necessarily limited to bulk picking, and can be applied to picking up workpieces that are not bulk stacked.

  In the example of FIG. 1, the sensor unit 2 is fixed above the work space. However, any position where the work space can be imaged is sufficient. However, the aspect which arrange | positions a sensor part in the indefinite position which moves like the arm part ARM is excluded. Further, the number of cameras and illuminations included in the sensor unit 2 is not limited to one and may be plural. Furthermore, the connection with the sensor unit 2, the robot, and the robot controller 6 is not limited to a wired connection, and may be a wireless connection.

  In addition to gripping the workpiece, the outside of the workpiece WK is sandwiched as shown in FIG. 5A, and the claw portion of the end effector EET2 is inserted and expanded inside the workpiece WK2 having a cavity as shown in FIG. 5B. It is used in the meaning including an example of holding by a step and an example of an end effector EET3 for sucking and holding a plate-like workpiece WK3 as shown in FIG. 5C. Below, the aspect which grips the outer surface of a workpiece | work from both sides as an example of a workpiece | work holding is demonstrated. In addition, as shown in FIG. 1, a plurality of workpieces are stored in a storage container BX and are randomly stacked, and are gripped by the end effector EET one by one with respect to the plurality of workpieces WK. The setting of the gripping position (teaching work) in the bulk picking operation that repeats the work of placing on the placement position will be described below.

When performing the bulk picking operation in the robot system 1000, teaching including setting for performing the bulk picking operation is performed in advance. Specifically, registration is made such as which part of the workpiece is gripped by the end effector and in what posture. Such setting is performed by the robot operation tool 7 such as a pendant.
(Teaching work)

  FIG. 6 shows a functional block diagram of a robot system including an image processing apparatus that implements the function of teaching the gripping position described above. A robot system 1000 shown in this figure includes an image processing apparatus 100, a display unit 3, an operation unit 4, a sensor unit 2, and a robot RBT.

The sensor unit 2 measures the three-dimensional shape of the workpiece placed at the work position in three dimensions. The sensor unit 2 is controlled by a sensor control unit 2b. In this example, the sensor control unit 2b is integrated with the sensor unit 2, but these may be provided separately. The robot part includes an arm part ARM and an end effector EET. This robot is controlled by the image processing apparatus 100 to grip the workpiece at the gripping position and grip it at the gripping position. Here, the state in which the workpiece is gripped in the reference posture and placed in the reference posture is imaged and registered by the sensor unit 2. Here, the reference posture includes the position and posture of the workpiece.
(Display unit 3)

  The display unit 3 displays a virtual model of a work model that virtually represents the three-dimensional shape of the work and an end effector model that is configured of three-dimensional CAD data that virtually represents the three-dimensional shape of the end effector. Each is displayed in three dimensions on the original space. The height image is an image having height information and is also called a distance image. Furthermore, the display unit 3 has a six-view drawing display area 3a for displaying a basic direction image of the work model as a six-view drawing. As a result, each posture of the work model can be displayed as a six-sided view, and the gripping position setting work can be performed for the basic direction image in which the gripping position can be easily set. Can be easily performed.

The operation unit 4 is a member for performing various settings such as image processing, and an input device such as a keyboard or a mouse can be used. Further, by using the display unit 3 as a touch panel, the operation unit and the display unit can be integrated.
(Image processing apparatus 100)

  The image processing apparatus 100 of FIG. 6 includes an input image acquisition unit 2c, a storage unit 9, a calculation unit 10, an input / output interface 4b, a display interface 3f, and a robot interface 6b.

  The input image acquisition unit 2c acquires an input image having a three-dimensional shape from an image including a plurality of workpieces measured by the sensor unit 2 and surrounding objects. The input image having a three-dimensional shape may be constructed on the sensor unit or sensor control unit side, or may be constructed on the image processing apparatus side (for example, the input image acquisition unit). In the example of FIG. 6, an input interface for the image processing apparatus 100 to acquire an input image indicating a three-dimensional shape acquired by the sensor unit 2 is configured. However, the present invention is not limited to this configuration, and an input image captured in advance and held in a storage unit such as a recording medium may be read and acquired.

  The storage unit 9 is a member for holding various settings, and a nonvolatile memory, a hard disk, a storage medium, or the like can be used. The storage unit 9 functions as a gripping position storage unit for storing the gripping positions of the work model and the end effector model.

  The input / output interface 4b is connected to an input device such as a keyboard and a mouse, and accepts data input.

  The display interface 3 f constitutes an output interface with the display unit, and controls to display image data to be displayed on the display unit constructed by the calculation unit 10.

The robot interface 6b constitutes a communication interface with the robot controller 6.
(Calculation unit 10)

  The calculation unit 10 includes a positioning unit 8c, a basic direction image generation unit 8e ′, a basic direction image selection unit 8e, a search model selection unit 8i, a work model registration unit 8t, a search model registration unit 8g, an end effector. A model registration unit 8u, a gripping position specifying unit 8d, an end effector mounting surface setting unit 8f, a rotation angle limiting unit 8h, a three-dimensional search unit 8k, and a three-dimensional pick determination unit 8l are provided.

  The positioning unit 8c is a member for adjusting the position and orientation of the work model displayed on the display unit 3 in the virtual three-dimensional space.

  The basic direction image generation unit 8e ′ has at least the workpiece model positioned in the virtual three-dimensional space by the positioning unit 8c, as viewed from each of the three axial directions orthogonal to each other in the virtual three-dimensional space. This is a member for generating three height images as basic direction images. By automatically generating the basic direction image in this way, the user does not need to manually change the direction of the work and individually acquire the basic direction image, and there is an advantage that the work of registering the gripping position can be saved. .

  The basic direction image selection unit 8e is for selecting any of a plurality of basic direction images that are different in appearance from other basic direction images with respect to at least three basic direction images displayed on the display unit 3. It is a member. In this way, by deleting the surface with the common view, it is possible to eliminate unnecessary display of the basic direction image and the like, thereby further simplifying the setting operation. For example, one of the basic direction images that look the same, such as the top and bottom surfaces of a cylindrical workpiece, is deleted. The basic direction image selection unit 8e can be configured to allow the user to manually select in a state where at least three basic direction images are displayed on the display unit 3. Alternatively, the basic direction image selection unit may be configured to automatically extract and select a basic direction image having a common appearance from at least three basic direction images.

  The search model selection unit 8i is a member for selecting a basic direction image to be registered as a search model. As will be described later, the search model used for the three-dimensional search and the model for specifying the gripping position are made common so that the search model selection unit 8i, the basic direction image selection unit 8e, and the common image selection unit 8j can be configured. .

The work model registration unit 8t is a member for registering a work model that virtually represents a three-dimensional shape of a work. Here, for example, the work model registration unit 8t uses, as a work model, 3D point cloud data obtained by imaging an actual work. In this case, the three-dimensional point cloud data acquired by the sensor unit 2 or the input image acquisition unit 2c is registered as a work model by the work model registration unit 8t. Alternatively, three-dimensional CAD data representing the shape of a work created separately is read and registered as a work model. In this case, the 3D CAD data input via the input / output interface 4b is registered as a work model by the work model registration unit 8t. Alternatively, three-dimensional CAD data imitating a workpiece may be created and registered. In this case, the work model registration unit 8t realizes a simple three-dimensional CAD function.
(Search model registration unit 8g)

  The search model registration unit 8g uses a third order of workpieces used when performing a three-dimensional search for specifying the posture and position of each workpiece for a plurality of workpiece groups included in the input image acquired by the input image acquisition unit 2c. This is a member for registering a search model that virtually represents the original shape. By providing the search model registration unit 8g in this way, the user can save the setting work by registering the search model for performing the three-dimensional search as common with the basic direction image for designating the grip position of the work model. . Also, even in actual operation, by registering the gripping position of the workpiece for each basic direction image that searches for a workpiece that can be gripped, searching for a basic direction image without a gripping position is eliminated, and the search is performed in reverse. Since it is possible to examine whether or not the basic direction image can be gripped at the gripping position set in the basic direction image, the processing can be performed efficiently.

  The search model registration unit 8g is preferably configured to select whether or not to use the basic direction image selected by the basic direction image selection unit 8e as a search model for a three-dimensional search. As a result, it is possible to select whether or not to use the basic direction image as a search model for a three-dimensional search. In other words, unnecessary basic direction images can be excluded from the target of the three-dimensional search. By excluding the basic direction image that may be erroneously detected from the image seen from the side, it is possible to set so that the plate-like workpiece is upright and not to perform a three-dimensional search, effectively limiting the posture of the workpiece model It can be set easily.

  Note that the search model registration unit 8g and the work model registration unit 8t can be provided separately or integrated. For example, in the robot system 1000 of FIG. 6, the search model registration unit 8g and the work model registration unit 8t are integrated as a common model registration unit 8g '. Thus, by registering a model related to one workpiece, it is possible to share it for registering a gripping position and registering a search model for a three-dimensional search, thereby obtaining an advantage that the setting can be simplified.

  The end effector model registration unit 8u is a member for registering an end effector model in which the three-dimensional shape of the end effector is virtually represented by three-dimensional CAD data.

  The end effector mounting surface setting unit 8f causes the display unit 3 to display the end effector model and the mounting surface to which the end effector model is mounted on the tip of the robot arm, and also displays the mounting surface on the display unit 3. Thus, it is a member for defining the posture of the end effector model with respect to the mounting surface.

The rotation angle limiting unit 8h is for registering any one of the basic direction images as a search model for performing a three-dimensional search for specifying the posture and position of each workpiece with respect to a plurality of stacked workpieces. The search model registration unit 8g is a member for setting an allowable rotation angle range for each selected search model individually with respect to the rotation of the work model.
(Gripping position specifying part 8d)

  The gripping position specifying unit 8d has at least three workpiece models positioned in the virtual three-dimensional space by the positioning unit 8c as viewed from the three axial directions orthogonal to each other in the virtual three-dimensional space. This is a member for specifying a gripping position at which the work model is gripped by the end effector with respect to at least one of the height images in a state where the basic direction image is displayed on the display unit 3. The gripping position specifying unit 8d includes a workpiece-side gripping location specifying unit 8d1, an end effector-side gripping setting unit 8d2, and a relative position setting unit 8d5.

The workpiece-side gripping location specifying unit 8d1 displays a plurality of basic direction images selected by the basic direction image selecting unit 8e on the display unit 3, and the basic direction image is displayed for any of the basic direction images. It is a member for designating the gripping position when gripping the workpiece model shown by the end effector model. The workpiece-side gripping location specifying unit 8d1 is configured to be able to register a plurality of gripping positions for each of a plurality of basic direction images.
(End effector side gripping setting section 8d2)

The end effector side gripping setting unit 8d2 is a member that performs settings related to gripping of the work model with respect to the end effector model. The end effector side gripping setting unit 8d2 can include a gripping reference point setting unit 8d3 and a gripping direction setting unit 8d4. The gripping reference point setting unit 8d3 defines a gripping reference point corresponding to the position where the workpiece model is gripped with respect to the end effector model. On the other hand, the gripping direction setting unit 8d4 defines a gripping direction in which the work model is gripped by the end effector model. The gripping reference point setting unit 8d3 and the gripping direction setting unit 8d4 may be a common member or may be configured separately. The gripping reference point setting unit 8d3 and the gripping direction setting unit 8d4 can set the gripping reference point and the gripping direction to predetermined values that are set in advance. For example, the gripping reference point is set as the center between the claw portions that are provided at the end of the end effector model and sandwich the workpiece. The gripping direction is any coordinate axis that constitutes the tool coordinate axis that defines the end effector model. For example, by setting the Z-axis direction, it is possible to move the end effector model along the Z-axis direction, that is, the height direction, to approach the workpiece model so as to grasp it, and the user can easily grasp it sensuously. Alternatively, the gripping reference point setting unit 8d3 and the gripping direction setting unit 8d4 may allow the user to adjust the gripping reference point and the gripping direction.
(Relative position setting unit 8d5)

The relative position setting unit 8d5 moves the end effector model displayed on the display unit 3 until it interferes with the work model, and automatically holds the gripping state in a posture separated from the position reaching the interference position by a predetermined amount. It is a member for prescribing. As a result, the user can move the end effector model automatically to contact the workpiece without manually moving the end effector model to bring it into contact with the workpiece. The advantage that work can be saved greatly
(Register grip position)

  In teaching work, the positional relationship between the workpiece and the end effector when the workpiece is gripped is registered as a gripping position. Below, the example which grips a workpiece | work with a hand part as a typical example of a grip using a hand part as an end effector is demonstrated. During actual operation of robot picking using robot vision with the gripping position registered, each workpiece is detected by robot vision from a group of workpieces stacked in multiple pieces, and the position of the detected workpiece or The gripping position on the end effector side is calculated with respect to the posture, and the robot is operated so that the end effector is positioned at the calculated position to perform picking. Here, as a method of registering the gripping position, there are a method of actually registering by operating a robot, and a method of registering by operating an end effector model in a virtual three-dimensional space using three-dimensional CAD. However, the method of registering the gripping position by actually operating the robot requires troublesome registration work while actually moving the robot, which requires a large verification environment and takes time for trial and error. there were. On the other hand, the method of registering by virtually operating the robot in the space of 3D CAD has the advantage of being able to register without the need for a robot, but it is a virtual end for the 3D posture of the virtual work model. It is necessary to accurately match the three-dimensional posture of the effector model, and it is difficult to set three-dimensional coordinates. In addition, since this method requires 3D CAD data of the workpiece and the end effector, it cannot be set without 3D CAD data at hand.

Therefore, in the present embodiment, a plurality of height images viewed from the respective axial directions of the three-dimensional CAD model are displayed as basic direction images, and a basic direction image desired by the user is selected from among them, and the selected basic image is selected. By setting the grip position for the direction image, grip registration in the virtual three-dimensional space can be easily performed. As a result, when the end effector model is operated and registered in the virtual three-dimensional space in which the work model of the three-dimensional CAD data imitating the work is arranged, the basic direction viewed from each axis direction defining the virtual three-dimensional space By selecting and registering an image for gripping, it is possible to register with reference to the posture of the end effector model viewed from a substantially vertical direction with respect to each basic direction image. . Further, according to this method, even when there is no 3D CAD data, data obtained by 3D measurement of an actual workpiece can be used as the basic direction image. For this reason, even when there is no 3D CAD data, grip registration can be easily performed in the virtual 3D space by the same procedure.
(Three-dimensional pick determination unit 8l)

  The three-dimensional pick determination unit 8l grips the gripping direction specified by the gripping direction setting unit 8d4 so that the gripping direction is orthogonal to the work plane representing the posture of the work model displayed in the image display area and along the gripping direction. It is a member for adjusting the relative position of the end effector model and the work model so as to position the reference point and the gripping position. Thereby, when simulating the gripping state in which the workpiece is gripped by the end effector, there is an advantage that the position setting operation for gripping the workpiece model by the end effector model can be easily performed. In particular, the gripping direction is perpendicular to the workpiece plane, and the gripping reference point and the gripping position are positioned on the gripping direction axis, so that the end effector model can be simply moved closer to the workpiece model along the gripping direction. Since the adjustment becomes possible, there is an advantage that the work burden on the user side is greatly reduced. Conventionally, the user manually moves the end effector and manually adjusts it to the posture for gripping the workpiece. There are many parameters and how to adjust the position and posture of the end effector with a large degree of freedom. Therefore, it was an extremely complicated task. In contrast, the gripping reference position and gripping direction are defined in advance on the end effector model side, the gripping direction is orthogonal to the workpiece plane, and the gripping reference point of the end effector model and the gripping position of the workpiece are on the gripping direction axis. Is set so that the movement direction of the end effector model is defined, and the user can obtain a gripping state by adjusting only the distance between the end effector model and the work model. As a result, significant labor savings can be expected in the gripping operation of the gripping position between the end effector model and the work model, which has been extremely complicated in the past.

  The operation of adjusting the relative position between the end effector model and the work model so that the gripping direction is orthogonal to the work plane and the gripping reference point and the gripping position are positioned on the gripping direction axis is a three-dimensional pick determination unit. It is preferable to carry out automatically by 8 l. As a result, the gripping reference position and gripping direction are defined in advance on the end effector model side, the gripping direction is orthogonal to the workpiece plane, and the gripping reference point of the end effector model and the gripping position of the workpiece are on the gripping direction axis. By automatically adjusting the position to be positioned, the moving direction of the end effector model is defined, and the user can obtain a gripping state by adjusting only the distance between the end effector model and the work model. As a result, significant labor savings can be expected in the gripping operation of the gripping position between the end effector model and the work model, which has been extremely complicated in the past.

  However, it is not necessarily limited to automatic adjustment by the three-dimensional pick determination unit. For example, with the positioning unit 8c, the user manually makes the gripping direction orthogonal to the workpiece plane, and the gripping reference point of the end effector model and the workpiece on the axis in the gripping direction. The three-dimensional pick determination unit may be configured to support such adjustment work when performing the adjustment to position the holding position. For example, in the state where the work plane and the gripping direction are displayed in the image display area as the first step, an auxiliary line is shown to the user so that they are orthogonal to each other, or “the gripping direction is orthogonal to the work plane” with characters or images. Please adjust the end effector model as described above. ”May be displayed to prompt the user to perform adjustment work. Further, as a second step, an extension line in the gripping direction is displayed in the image display area so that the gripping reference point of the end effector model and the gripping position of the workpiece are positioned on the gripping direction axis. The 3D pick determination unit can also guide the user to display a message such as “Adjust so that the gripping reference point of the end effector model and the gripping position of the workpiece are positioned.”

The fit function can also be realized by the three-dimensional pick determination unit 8l. For example, the three-dimensional pick determination unit 8l sets the relative position between the end effector model and the work model in a state where the gripping direction is orthogonal to the work plane and the gripping reference point and the gripping position are positioned on the gripping direction axis. The effector model can be moved along the gripping direction until it interferes with the work model, and the gripping state can be automatically defined in a posture separated from the position reaching the interference position by a predetermined amount. As a result, it is possible to automatically adjust the position and posture at which the work model is gripped by the end effector model, and it is possible to obtain the advantage of further reducing the work burden on the user side.
(3D search part)

The three-dimensional search unit is a member for performing a three-dimensional search for specifying the posture and position of each workpiece using the search model registered in the search model registration unit from the input image. Prior to the three-dimensional search, the search model registration unit performs a three-dimensional search for identifying the posture and position of each workpiece included in the input image from the input image indicating a state where a plurality of workpiece groups are stacked. A search model that virtually represents the three-dimensional shape of the workpiece is registered in advance. In this state, the three-dimensional pick determination unit 8l determines the end effector at the gripping position designated for the workpiece by the workpiece-side gripping location designation unit with respect to the search result in the input image searched by the three-dimensional search unit. To determine whether or not gripping is possible. For example, an input image acquisition unit acquires an input image having a three-dimensional shape from images of a plurality of workpieces measured by a sensor unit, and is registered in a search model registration unit by a three-dimensional search unit from the input image. A three-dimensional search that identifies the posture and position of each workpiece is performed using the search model. As a result, a three-dimensional search can be performed on an input image obtained by actually capturing a workpiece, and gripping determination can be performed in accordance with the actual situation.
(Search model)

In bulk picking, in order to determine a work that can be gripped from a plurality of work pieces stacked in bulk, it is necessary to first extract each work piece. Here, with respect to the shape of the workpiece group having the height information obtained by the sensor unit, the shape of the workpiece to be searched is registered in advance as a workpiece model, and a three-dimensional search is performed using this workpiece model. The position and orientation are detected.
(Basic direction image)

A search model for three-dimensional search of a workpiece is created using a height image when the workpiece is viewed from a specific direction. As the height image used as the search model, three-dimensional CAD data that is a work model expressing the work three-dimensionally, or actual measurement data obtained by actually imaging the work with the sensor unit can be used. Here, an example in which three-dimensional CAD data is registered as a search model will be described. For example, consider a case where a work model CWM constructed with three-dimensional CAD data as shown in FIG. 7 is used as a search model. From this work model CWM, six height images viewed from the positive and negative directions of the three axes orthogonal to each other in the virtual three-dimensional space (for example, the X axis, the Y axis, and the Z axis), that is, A hexahedral view is acquired as a basic direction image. For example, the basic direction image generation unit 8e ′ shown in FIG. 6 generates six basic direction images and displays them in the six-view drawing display area 3a of the display unit 3. The six basic direction images are, for example, “up”, “down”, “left”, “right”, “front”, “back” of the work model CWM, that is, a plan view, a bottom view, a left side view, and a right side view. The basic direction image generation unit 8e ′ automatically calculates from the origin (details will be described later) of the work model CWM and the surfaces constituting the work model CWM so as to be a figure, a front view, and a rear view. Here, “upper” is a height image viewed from the positive direction (+ side) of the Z axis, “lower” is a height image viewed from the negative direction (−side) of the Z axis, and “left” is a negative image of the X axis. Height image viewed from the direction, “Right” is the height image viewed from the positive direction of the X axis, “Front” is a height image viewed from the negative direction of the Y axis, and “Back” is from the positive direction of the Y axis Each viewed height image is shown. However, these are only examples, and different coordinate systems may be used, and the positive and negative directions of each axis are based on a coordinate system orthogonal to this axis with the X = Y line in the XY plane as the axis. You may use the height image seen from. Also, when generating a height image from three-dimensional CAD data, it is not necessarily from the direction orthogonal to the CAD axis ("up", "down", "left", "right", "front", "back"). It is not necessary to be the viewed height image. For example, the posture (viewpoint) of the work model may be arbitrarily changed, and the height image may be generated from the changed viewpoint.
(Work model origin)

Here, the origin of the work model is automatically determined by the image processing apparatus from the coordinate information of the three-dimensional CAD data. For example, for the three-dimensional CAD data of the work model CWM in FIG. 7, a virtual cuboid IBX circumscribing the work model CWM is defined as shown by a broken line in FIG. 8, and the center of gravity of the virtual cuboid IBX is defined as the origin O of the work model. Set as.
(Basic direction image selection unit 8e)

  Note that the number of basic direction images is not necessarily six, and at least a plurality of basic direction images are sufficient. For example, if the opposing surfaces have the same shape, such as a rectangular parallelepiped, a basic direction image viewed from one of the surfaces is sufficient. In other words, it is possible to reduce the processing load of the three-dimensional search by eliminating the basic direction image having the same shape. Such a function of deleting a basic direction image that is common to any of the basic direction images is realized by the basic direction image selection unit 8e. As an example, the basic direction images acquired from the work model CWM of FIG. 7 are shown in FIGS. 9A to 9D. In these drawings, FIG. 9A is a height image when the work model CWM of FIG. 7 is viewed from the positive direction of the X axis, FIG. 9B is a height image when viewed from the positive direction of the Y axis, and FIG. 9D is a height image viewed from the positive direction of the axis, and FIG. 9D is a height image viewed from the negative direction of the Z axis. Here, as the height image, a height image in which height information is expressed as a luminance value is used so that a higher point is brighter and a lower point is darker.

Here, the matching / non-coincidence of the appearance is the height image of a total of six faces as seen from the top and bottom (Z axis ± direction), front and back (Y axis ± direction), and left and right (X axis ± direction). And confirm by matching. Here, the rotation coincidence is confirmed at a step angle of 90 °, and the surface that appears to match the other surface is excluded from the registration target of the search model. In the work model CWM of FIG. 7, the height image viewed from the negative direction of the X axis and the height image viewed from the negative direction of the Y axis are in a state where the appearance matches that of FIGS. 9A and 9B, respectively. Therefore, it is excluded from the search model generation target. In this way, search models are generated for the number of faces that look different. In this way, the basic direction image excluding the unnecessary height image is displayed in the six-view drawing display area 3a. Note that the six-face drawing display area is not limited to its name, and it is not always necessary to display all six faces of the work model. In the present specification, including the mode of display, is referred to as a hexahedral display area.
(Search model registration screen 130)

  Further, such exclusion can be performed manually by the user, automatically by the image processing apparatus, or a combination thereof.

  For example, in the search model registration screen 130 for registering the basic direction image BDI as the search model of the three-dimensional search shown in FIG. 10, the basic direction image generation unit 8e ′ automatically generates a basic direction image BDI corresponding to a hexahedral view. Then, it is displayed on the six-sided view display area 3a of the display unit 3. At this time, if a common basic direction image exists, the user is prompted to exclude it from the search model registration target. Here, as a search model selection unit 8i that sets whether or not a search model can be registered, a “registration target” check box 131 is provided for each basic direction image BDI that is a candidate for the search model. When the “registration target” check box 131 is turned ON, the basic direction image BDI is registered as a search model, and when the “registration target” check box 131 is OFF, the basic direction image is excluded from the search model. it can.

At this time, for the basic direction image having a common appearance, the basic model image selection unit 8e causes the user to display the search model registration screen 130 with the “registration target” check box 131 turned off in advance. The user confirms that the basic direction image BDI to be registered in the search model as an initial state and the basic direction image to be excluded from the search model are correctly selected, and approves this selection, or if necessary. Can be corrected and replaced. In this way, by selecting a basic direction image that should be registered as a search model for a 3D search by default and by presenting candidates for basic direction images that should be excluded from registration, the user can save search model registration work. it can.
(Actual measurement data MWM)

  The above describes an example in which 3D CAD data is used as a search model for 3D search. However, as described above, the present invention does not necessarily limit the search model to three-dimensional CAD data. For example, a plurality of two-dimensional CAD data is analyzed and converted into three-dimensional data, or the workpiece is actually processed by the sensor unit. The actually measured data captured can also be used as a search model. As an example, FIG. 11 shows actual measurement data MWM obtained by imaging a workpiece corresponding to the workpiece model CWM in FIG. 7 from the positive direction of the X axis. Thus, when there is no CAD data of a workpiece, it is also possible to use data obtained by three-dimensional measurement of an actual workpiece. As shown in FIG. 10, the number of actual measurement data MWM necessary for the three-dimensional search is imaged and registered.

When registering an actual workpiece, information on the bottom surface on which the workpiece is placed (for example, the shape of the floor around the workpiece) is also measured three-dimensionally. Therefore, it is preferable to exclude unnecessary bottom surface information, for example, by cutting out only a portion having a certain height from the bottom surface by threshold processing. This makes it possible to register only the shape part necessary for the three-dimensional search.
(Extraction of feature points)

  Next, a search model of the registered surface is generated in a state where height images corresponding to each surface of the workpiece to be the search model are registered in this way. Here, feature points necessary for the three-dimensional search are extracted for each registered surface. Here, an example will be described in which two types of feature points are used as feature points: a feature point representing a contour of a shape (a feature point on the contour) and a feature point representing a surface shape (a feature point on the surface). FIG. 12A shows a state where two types of feature points are extracted from the search model SM of the height image (corresponding to FIG. 9A) viewed from the X-axis direction. FIG. 12B shows a state where the search model SM is displayed in a perspective view. Here, it is preferable that the feature point SCP on the surface and the feature point OCP on the contour are displayed in different display modes. For example, the feature point SCP on the surface is displayed in white, and the feature point OCP on the contour is displayed in light blue. Alternatively, other color schemes may be used, such as displaying the feature point on the surface white and purple so that it can be more easily distinguished. By displaying the feature points by changing the color in this way, it becomes easy for the user to visually distinguish the meaning of each feature point.

As the feature points SCP on the surface, for example, the surface of the work model is extracted at regular intervals. On the other hand, the feature points OCP on the contour are extracted as feature points at regular intervals, for example, by performing edge extraction such as locations where the height changes, and further thinning processing. Thus, each feature point is a feature representing the three-dimensional shape of the surface.
(Three-dimensional search method)

A three-dimensional search is performed using the search model from which feature points are extracted in this way. Here, as an input image, as shown in FIG. 13A and FIG. 13B, a method of performing a three-dimensional search for extracting a work group from a state in which a plurality of works are stacked and a three-dimensional shape is acquired. Will be described. Each characteristic point of first search models, the best match state, the position and orientation (X, Y, Z, R X, R Y, R Z) , and searches from the input image. Here, R _X , R _Y , and R _Z represent a rotation angle with respect to the X axis, a rotation angle with respect to the Y axis, and a rotation angle with respect to the Z axis, respectively. Various methods for expressing such a rotation angle have been proposed. Here, a ZYX Euler angle is used (details will be described later). Further, the matching positions and orientations do not have to be one for each search model, and a plurality of matching positions and orientations may be detected.

  Here, as an input image, a two-dimensional display as shown in FIG. 13A or a work group as shown in FIG. As a result, a two-dimensional display as shown in FIG. 13C or a searched image as shown in FIG. 13D in which this is displayed in three dimensions is obtained as a three-dimensional search result. As shown in FIGS. 13C and 13D, it can be seen that the feature points of the search model are searched from the input image, and the workpiece corresponding to the search model is detected. FIG. 13D shows a state where search results of search model A and search model B are obtained. These search model A and search model B correspond to search models A and B displayed in a search model selection column in FIG. For the work WK on the right side of FIG. 13D, two search results of search models A and B are obtained with the same work. Therefore, if gripping at the gripping position registered in each search model is possible, a plurality of gripping positions are obtained in this work.

As a search model used for 3D search in this way, the 3D search calculation process is simplified compared to the perspective view, etc., by using an image of the work as seen on each side, such as a hexagonal view. It is possible to obtain an advantage that the processing can be lightly loaded and speeded up. In addition, the displayed state is easy to see even when the search model is registered, and the user can easily understand it visually.
(Evaluation index of 3D search results)

  Furthermore, an evaluation index for a three-dimensional search result can be set. For example, in the examples shown in FIGS. 13C and 13D, how much the corresponding feature points exist in the input image (for example, the ratio of the number of feature points corresponding to the search result with an error of a certain distance or less, , Scoring the 3D search result by a value obtained by deducting the feature point error amount as a penalty according to the prescribed calculation formula, etc. With this method, there are many invalid data (invalid pixels) that could not be measured in 3D, In this way, the score can be used as an index representing the reliability of the three-dimensional search result, for example, the workpiece can be set to be gripped preferentially in descending order of the score. The three-dimensional search result below the score may be determined to be erroneously detected and may be set so as to be excluded from the workpiece gripping target, for example, the image processing in FIG. In the apparatus 100, an evaluation index calculation unit 8q is provided in the calculation unit 10, and the evaluation index is calculated with respect to the search result based on a predetermined reference, whereby priority is set in order from the search result having the highest evaluation index. For example, it can be set so that the workpiece is gripped preferentially from the highest position in the Z direction among the results of a certain score or higher. The higher the position in the direction, the more difficult it is to interfere with other workpieces. Therefore, setting the gripping with priority from the one with the highest position in the Z direction has the effect of reducing the processing load for interference determination. can get.

By detecting each workpiece from the workpiece group stacked in this manner, the workpiece to be grasped can be recognized on the image processing apparatus side. Next, in order to grip the workpiece with the end effector, it is necessary to recognize for each workpiece which position of the workpiece is to be gripped and in which posture. Therefore, the workpiece gripping position is registered. In the present specification, the registration of “grip position” and “grip posture” is used to include the position where the workpiece is gripped and the posture at that time. One or more workpiece gripping positions can be registered for the workpiece. Further, the registration of the gripping position is preferably performed in units of workpiece surfaces, from the viewpoint of ease of gripping registration work and recognition of the gripping position. That is, the gripping position is registered after the posture of the work is defined as a posture with a specific surface as the upper surface.
(Grip registration screen 140)

  Here, FIG. 14 and FIG. 15 show examples of user interface screens for performing grip registration for registering grip positions where the end effector model grips the work model. On the grip registration screen 140, a surface to be registered in the work model is designated, and a grip posture is registered for each surface. The example of FIG. 14 shows a user interface screen for specifying four types of surfaces (each surface of the search model) and registering a gripping posture. Here, the search model “C” is selected from the search models A to D, and three gripping postures are displayed in the search model C.

  In the grip registration screen 140 of FIG. 14, an image display column 141 for displaying an image on the left side and an operation column 142 for performing various operations on the right side are provided. In the image display column 141, the work model CWM and the end effector model EEM are displayed. The viewpoint can be changed by dragging the screen of the image display column 141. In this way, it functions as a positioning unit 8c that adjusts the position and orientation of the work model displayed in the display area in the virtual three-dimensional space. Further, the display mode of the image display column 141 can be switched to two-dimensional display or three-dimensional display. In order to show the current display mode in an easy-to-understand manner, a three-dimensional reference coordinate axis BAX is displayed in an overlapping manner in the image display field 141. In the example of FIG. 14, the image display column 141 displays the gripping posture registered in the gripping posture 001 selected in the operation column 142, that is, how a part of the work model CWM is gripped by the end effector model EEM. ing. In this example, the entire end effector model EEM is displayed, but it is not always necessary to display the entire end effector model at the time of grip registration, and at least a part for gripping the work model CWM, for example, a claw portion is displayed. If it is enough.

Further, the operation column 142 includes a search model selection column 143 for selecting a search model, and a gripping posture display column 144 indicating a gripping posture registered for the search model selected in the search model selection column 143. It has been. When a gripping posture displayed as a list in the search model selection column 143 is selected, a registered gripping posture corresponding to this is displayed in the image display column 141. When the edit button 145 is pressed, the registered gripping posture can be corrected. Further, the operation field 142 is provided with an add button 146 for adding a gripping posture and a delete button 147 for deleting a registered gripping posture. When the delete button 147 is pressed, the selected gripping posture is deleted.
(Grip posture addition screen 150)

To add a new gripping posture, the add button 146 is pressed. As a result, a gripping posture addition screen 150 is displayed as shown in FIG. 15, and a gripping posture can be added. In the image display column 141, an end effector model EEM and a work model CWM are displayed. In the operation column 142, gripping posture coordinate information 153 that defines the gripping posture is displayed. Here, the position parameters X, Y, Z, R _X , R _Y , and R _Z displayed as the gripping posture coordinate information 153 represent the position and posture data of the end effector model EEM with respect to the origin of the search model. ing. As described above, the origin of the search model can be the center of gravity of the work model CWM, the center coordinates of CAD data, or the like.

When the X, Y, Z, R _X , R _Y , and R _Z values of the gripping posture coordinate information 153 are changed, the position and posture of the end effector model EEM displayed three-dimensionally in the image display field 141 are changed to this. Will be updated accordingly. Conversely, by dragging and moving the end effector model EEM in the image display column 141, the display content of the gripping posture coordinate information 153 in the operation column 142 is updated to the gripping posture coordinates after movement. Thus, the user can register the grip position and posture while confirming the end effector model EEM in the image display column 141 and the grip posture coordinate information 153 in the operation column 142. In the image display column 141, a three-dimensional reference coordinate axis BAX can be displayed in an overlapping manner.

  When registering the gripping position, with respect to the end effector model EEM displayed in the image display field 141, the gripping reference point corresponding to the position where the work model CWM is gripped, and the gripping direction in which the end effector model EEM grips the work model CWM Is specified. These gripping reference points and gripping directions are preferably set as default values on the image processing apparatus 100 side. Then, on the gripping posture addition screen 150 in FIG. 15, the position of the end effector model EEM according to the position and posture of the work model CWM so that the gripping direction is orthogonal to the work plane representing the posture of the work model CWM. And adjust the posture automatically. This adjustment can be performed, for example, by the three-dimensional pick determination unit 8l. Alternatively, the configuration is such that the user manually adjusts the position and orientation of the end effector model EEM so that the gripping direction of the end effector model EEM is orthogonal to the workpiece plane of the workpiece model CWM using the positioning unit 8c. Also good.

  Further, when the gripping position is registered, in the initial state where the gripping posture addition screen 150 of FIG. 15 is opened, the position and posture of the end effector model EEM are set so that the end effector model EEM is a downward posture and the work model CWM is the end effector model EEM. The state located below is the initial value. In this way, if the end effector model EEM is lowered and brought into contact with the work model CWM, the gripping posture is determined, so that the user side operation can be performed intuitively, and conventionally, the position in three dimensions Problems that are difficult to align can be reduced. That is, the gripping posture can be simply registered simply by setting the alignment of the end effector model EEM in the X, Y, and Z directions and the rotation with respect to the Z axis.

  Furthermore, by setting the gripping position to be gripped by the end effector model EEM for the work model CWM, the position of the end effector model EEM is automatically set so that the gripping position is positioned on the axis extending the gripping direction. It can also be configured to adjust to.

The gripping posture is registered based on the workpiece image. For this reason, it is not always necessary to use CAD data, and as described above, the gripping posture can be registered for the actual measurement data obtained by actually imaging the workpiece.
(Fit function)

Furthermore, when the gripping position is designated, not only can the user manually perform the operation of moving the end effector model to the gripping position of the work model, but a fitting function for automatically performing this operation can be provided. Conventionally, when specifying the gripping positions X, Y, Z and gripping postures R _X , R _Y , R _Z for gripping the workpiece model with the end effector model, for example, the end effector model is dragged in the image display field 141 of FIG. In the operation field 142, a numerical value is input in the height direction Z, and the work model is set. However, it is troublesome for the user to visually move the end effector model and adjust it to the posture of gripping the work model, and the numerical values of the position parameters can be specified by how to adjust the six position parameters. It ’s hard to tell if it ’s good. Therefore, a fitting function is provided that automatically performs an operation of positioning the end effector model at the gripping position of the work model.

  If the work model or end effector model has height information such as 3D CAD data, the position to be gripped on the work model is specified by clicking the mouse, and the height of this position is specified. A direction, that is, a Z coordinate is acquired, and a position obtained by adding an offset to the Z coordinate is set as the Z coordinate after the end effector model is moved. As a result, the user can manually move the end effector model to the gripping position or manually input position parameters such as the Z coordinate, and the gripping position can be specified accurately. As an example of such a fit function, a “fit” button 154, which is an aspect of the relative position setting unit 8d5, is arranged in the operation column 142 shown in the gripping posture addition screen 150 in FIG. When the “fit” button 154 is pressed, the end effector model EEM is virtually moved in the Z direction and stopped at a position where it comes into contact with the work model CWM, and a position returned by an offset amount in the opposite direction from this interference position. The gripping position is assumed. The offset amount is a numerical value for slightly separating the end effector model EEM so that the tip of the end effector model EEM contacts and interferes with the workpiece model CWM, and can be defined according to the workpiece and application, for example, 1 mm.

  In the above example, the method for performing grip registration using the height image used when generating the three-dimensional search model for performing the three-dimensional search has been described. By sharing the search model for extracting the workpiece and the model for registering the grip position in this way, the user can perform a three-dimensional search and set the grip position for the common workpiece. A sense of unity is obtained and it becomes easy to understand sensuously. However, the present invention does not necessarily require that the search model for the three-dimensional search and the model to be grasped and registered be matched. If the correspondence between the three-dimensional search model and the model used for grip registration is known, it is not always necessary to use the model used as the search model for grip registration.

Further, it is not necessary to confirm the matching of the appearance after generating the height image from the three-dimensional CAD data only to confirm the positive and negative directions of each axis. For example, a work such as a cube has the same appearance when viewed from any of the ± direction of the X axis, the ± direction of the Y axis, and the ± direction of the Z axis. It ’s fine. Similarly, it is only necessary to register gripping positions for one surface by specifying gripping. With this alone, all surfaces (X axis ± direction, Y axis ± direction, Z axis ± direction) can be gripped. For example, for loose stacking of cube-shaped workpieces WKC as shown in FIG. 16A, one model of the cube is registered as a model as shown in FIG. As a result, as shown in FIG. 16C, the three-dimensional search and the end effector model EEM can be performed for all six surfaces.
(Calibration)

In the grip registration screen 140 described above, the relative position and orientation of the end effector model EEM when gripping the workpiece are registered with respect to the origin of the search model. On the other hand, when picking a workpiece with an actual end effector, the robot controller 6 is used when the robot controller 6 actually drives the robot from the vision coordinates which are the coordinates of the three-dimensional space (vision space) obtained by imaging the workpiece with the sensor unit. Need to convert to robot coordinates. Specifically, the position and posture of the workpiece obtained as a result of the three-dimensional search are obtained by the position (X, Y, Z) and posture (R _X , R _Y , R _Z ) in the vision space (note that posture ( R _X , R _Y , and R _Z ) indicate postures expressed by ZYX Euler angles described later). Similarly, the posture of the end effector that holds this is also obtained as the position (X, Y, Z) and posture (R _X , R _Y , R _Z ) in the virtual three-dimensional space of the image processing apparatus. In order for the robot controller 6 to drive the robot on the basis of the position and orientation in such a vision space, these are determined based on the position (X ′, Y ′, Z ′) and orientation (R _X ′, R _Y ′) in the robot space. , R _Z ′). The process of calculating a conversion formula for converting the position and orientation calculated in the coordinate system displayed by such an image processing apparatus into the position and orientation of the coordinate system in which the robot controller operates the end effector is performed by calibration. Called
(Embodiment 2)

An example of a robot system having a calibration function between such an image processing apparatus (machine vision apparatus) and a robot is shown in a functional block diagram of FIG. 17 as a second embodiment. A robot system 2000 shown in this figure includes an image processing device 200, a display unit 3, an operation unit 4, a sensor unit 2, a robot controller 6, and a robot RBT. In the robot system shown in this figure, members common to those in FIG. 6 and the like described above are denoted by the same reference numerals, and detailed description thereof is omitted as appropriate.
(Image processing apparatus 200)

  The image processing apparatus 200 includes an input image acquisition unit 2c, a storage unit 9, a calculation unit 10, an input / output interface 4b, a display interface 3f, and a robot interface 6b.

  The input image acquisition unit 2c acquires an input image having a three-dimensional shape from an image including an end effector measured by the sensor unit. When the input image of the end effector is acquired by the input image acquisition unit 2c, it is preferable to take an image so as to include the attachment position attached to the flange surface at the tip of the arm portion of the robot. In addition, the input image is captured in a posture that increases the area of the end effector. For example, the user operates the robot so as to capture an input image from a horizontal posture with the end effector as a horizontal posture.

  The calculation unit 10 includes an end effector model registration unit 8u, a work model registration unit 8t, a calibration unit 8w, a search model registration unit 8g, a three-dimensional search unit 8k, a conversion unit 8x, and an end effector attachment position calibration. A unit 8y and a gripping position specifying unit 8d.

  The end effector model registration unit 8u is a member for registering an end effector model that is three-dimensional CAD data that virtually represents the three-dimensional shape of the end effector. For example, the end effector model registration unit 8u reads three-dimensional CAD data representing the shape of the end effector created separately and registers it as an end effector model. In this case, the three-dimensional CAD data input via the input / output interface 4b is registered as an end effector model by the end effector model registration unit 8u. Alternatively, three-dimensional point cloud data obtained by imaging an actual end effector can be used as an end effector model. In this case, the three-dimensional point cloud data acquired by the sensor unit 2 or the input image acquisition unit 2c is registered as an end effector model by the end effector model registration unit 8u. Alternatively, three-dimensional CAD data imitating an end effector may be created and registered. In this case, the end effector model registration unit 8u realizes a simple three-dimensional CAD function.

  The calibration unit 8w uses the position and orientation calculated in the coordinate system of the vision space, which is a virtual three-dimensional space displayed on the display unit, as the position of the coordinate system in the robot space where the robot controller operates the end effector. And a member for acquiring calibration information for coordinate conversion into a posture.

  The calibration unit 8w calculates, for a plurality of position coordinates, a conversion formula between the actual position coordinates of the end effector EET of the robot and the position coordinates on the image displayed on the image processing apparatus. The coordinate conversion method is not particularly limited, and a known method such as three-dimensional affine transformation can be used as appropriate.

  The storage unit 9 stores calibration information from the calibration unit 8w.

  The three-dimensional search unit 8k uses the end effector model as a search model from the input image acquired by the input image acquisition unit 2c to perform a three-dimensional search for specifying an image region corresponding to the position and orientation of the end effector. It is a member.

  The conversion unit 8x converts the vision coordinates into robot coordinates based on the calibration information obtained by the calibration unit 8w. The conversion unit 8x reads the calibration information stored in the storage unit 9.

  The end effector mounting position calibration unit 8y uses the information obtained by converting the position and posture of the end effector in the vision space searched by the three-dimensional search unit into the position and posture in the robot space by the conversion unit 8x. The error between the position and posture of the end effector in the vision space and the position and posture in the robot space is calibrated. As a result, by performing a three-dimensional search using the end effector model, it becomes possible to correct the error in the virtual end effector model according to the actual end effector mounting state, thereby enabling more accurate setting work. It becomes like this.

The gripping position specifying unit 8d specifies one or more gripping positions at which the work model is gripped by the end effector model with respect to the work model registered by the work model registration unit 8t.
(ZYX Euler angle)

Here, the ZYX system Euler angle will be described. Conventionally, in order to define a gripping position for gripping a workpiece with an end effector, a ZY-X system Euler angle is used to position the posture of the workpiece model or the end effector model using three-dimensional CAD data. In this case, the position and orientation of the end effector model with respect to the work model are expressed by six position parameters (X, Y, Z, R _X , R _Y , R _Z ). Here, X, Y, and Z are orthogonal coordinate axes that define a three-dimensional space, and R _X , R _Y , and R _Z indicate rotation angles for rotation about the X, Y, and Z axes, respectively.

  Here, an example in which the end effector model is rotated in accordance with the ZYX system Euler angle will be described with reference to FIGS. First, FIG. 18 shows a state in which the end effector model EEM and the work model WM11 are displayed in the image display area of the display unit 3. In this example, for easy understanding, a reference coordinate axis BAX (XYZ) that defines a virtual three-dimensional space and a three-dimensional space coordinate axis after a rotating object to be rotated (here, an end effector model EEM) is rotated. The rotated coordinate axes RAX (XYZ) are respectively displayed.

  In the example of FIG. 18, since the work model WM11 side is stationary and the end effector model EEM side is rotated, the reference coordinate axis BAX is the position and orientation of the work model WM11, and the rotated coordinate axis RAX is the position of the end effector model EEM. And the posture are shown respectively. The rotated coordinate axis RAX on the end effector model EEM side has its origin coincided with a gripping reference point HBP that is a gripping position of the end effector model EEM (details will be described later).

Further, in the state before rotation, the rotation angles R _X , R _Y , R _Z of the XYZ axes are R _X = 0 °, R _Y = 0 °, R _Z = 0 °, respectively, and the reference coordinate axis BAX and the rotated coordinate axis RAX Are consistent. In this state, when the end effector model EEM is rotated 90 ° counterclockwise about the Z axis, the result is as shown in FIG. In the state of FIG. 19, the rotation angles R _X , R _Y , and R _Z are R _X = 0 °, R _Y = 0 °, and R _Z = 90 °, respectively, and have been rotated and displayed in the image display area accordingly. The coordinate axis RAX is updated to the post-rotation posture. Further, from the state of FIG. 19, when it is rotated 90 ° clockwise around the Y axis, it becomes as shown in FIG. In this state, the rotation angles R _X , R _Y , and R _Z are R _X = 0 °, R _Y = 90 °, and R _Z = 90 °, respectively, and the rotated coordinate axis RAX is similarly rotated after that. Updated to Further, from the state of FIG. 20, when rotated 90 ° clockwise around the X axis, the state of FIG. 21 is obtained, and the rotation angles R _X , R _Y , R _Z are R _X = 90 ° R _Y = 90 ° R, respectively. _Z = 90 °.

Conventionally, the generation and position of a work model and an end effector model have been expressed using such a ZY-X system Euler angle. However, since there are as many as six position parameters that can be operated, it is not easy to adjust these position parameters to the position and posture assumed by the user. In addition, since the XYZ axes used for moving the position while maintaining the posture are different from the R _X , R _Y , and R _Z axes used for rotation, it is intuitive how the position parameters change. There was a problem that it was difficult to understand. This will be described with reference to FIG.

Now, in the state shown in FIG. 21, the rotated coordinate axis RAX, which is the orthogonal coordinate axis after the rotation angles R _X = 90, R _Y = 90, and R _Z = 90, is displayed in the image display area. If _RY and _RZ are changed from this state, the image is rotated around an axis different from the displayed rotated coordinate axis RAX. For example, the rotation of R _Y is not the Y axis indicated by the rotated coordinate axis RAX in FIG. 22, but the Y axis before the R _X rotation (Y axis indicated by the rotated coordinate axis RAX in FIG. 19), as indicated by a broken line in FIG. Will rotate around. Also, R _Z does not rotate around the Z axis indicated by the rotated coordinate axis RAX in FIG. 22, but the Z axis before R _X and R _Y rotation indicated by the broken line (Z axis indicated by the rotated coordinate axis RAX in FIG. 18). Will rotate around.

This is because the Z-Y-X system Euler angle is defined on the assumption that the Z-axis, the Y-axis, and the X-axis are rotated in this order. According to the ZY-X system Euler angle, the Z axis is a reference, and the rotation axes of R _X and R _Y are rotated by the rotation of R _Z around the Z axis. Here, while the rotation axis around the Z axis does not move, R _X and R _Y rotate by the rotation R _Z around the Z axis. When the R _{Y is} rotated around the Y axis determined according to this, the X axis is also rotated accordingly. Here, since the rotation axis around the Y axis is determined by the rotation R _Z around the Z axis, both the rotation axis around the Y axis and the rotation axis around the Z axis change even if the rotation around the X axis is R _X. do not do. In other words, in the ZYX Euler angle, R _Z has an independent rotation axis, R _Y has a rotation axis dependent on R _Z , and R _X has a rotation axis dependent on R _Y. It can be said.

In this way, in the ZY-X system Euler angle that is generally used in the control of robots in the past, the three rotation axes are related to each other with the axes after rotation around the other rotation axes. Since it rotates, it is very difficult for the user to know which axis is the rotation axis, and it is not easy to rotate it in the intended direction.
(Display of correction rotation axis)

On the other hand, in the method according to the present embodiment, the rotation axes of R _X , R _Y , and R _Z are displayed with reference to the actual rotation axis. This axis is different from the conventional display in the three-dimensional space using the ZY-X system Euler angles (FIGS. 18 to 21), and the state after rotation around the other rotation axis The actual rotation axis corrected in consideration of the above is displayed as the corrected rotation axis. For example in the state of FIG. 22, consider the example of displaying the corrected rotational shaft of R _Z. In this case, as shown in FIG. 23, the Z axis when R _X and R _Y are calculated as 0 ° is the corrected rotation axis. The corrected rotation Z axis AXZ displayed here is the Z axis when R _X = 0 °, R _Y = 0 °, and R _Z = 90 °.

On the other hand, in order to display the corrected rotation axis of R _Y in the state of FIG. 22, as shown in FIG. 24, the Y axis when R _X is calculated as 0 ° becomes the actual rotation axis. The corrected rotation Y axis AXY displayed here is the Y axis when R _X = 0 °, R _Y = 90 °, and R _Z = 90.

Furthermore, in order to display the corrected rotation axis of R _X in the state of FIG. 22, the X axis may be displayed as it is. FIG. 25 shows an example in which the corrected rotation X axis is displayed as the corrected rotation axis. The corrected rotation X axis AXX displayed here is the X axis when R _X = 90 °, R _Y = 90 °, and R _Z = 90 °.

Note that the starting point of these correction rotation axes is preferably the center of rotation of the end effector model. For example, the intermediate position between the pair of claws, which is the gripping position of the end effector model EEM.
(XYZ Euler angle)

Even when the rotation order of the correction rotation axis is changed, the rotation axis can be displayed in the same way. For example, in the example of FIGS. 23 to 25 described above, since the ZYX system Euler angles are employed, the rotation is performed in the order of R _Z , R _Y , and R _X. On the other hand, when the X-Y-Z Euler angle is adopted, rotation is performed in the order of R _X , R _Y , R _Z , and therefore the axis calculated as 0 ° is different from the ZY-X system Euler angle. For example, assuming that the rotation axis is rotated in the order of R _X , R _Y and R _Z in order to obtain the correction rotation axis at the XYZ Euler angle, the Z axis is displayed as the correction rotation axis. To do this, the Z axis after rotation is displayed as it is. In order to display the Y axis as the correction rotation axis, the Y axis when R _Z is calculated as 0 ° is displayed. Further, in order to display the X-axis as the correction rotation axis, the X-axis when calculated with R _Z and R _Y being 0 ° is displayed.

In this way, when the user sets the grip position and posture using the Euler angle, the rotation axis can be displayed in an easy-to-understand manner, and it is easy to operate intuitively without understanding the complicated Euler angle concept. it can.
(Embodiment 3)

It is not easy for the user to set the position and posture of the end effector and workpiece defined by a plurality of position parameters such as Euler angles. Therefore, it is also possible to guide the position parameter setting procedure so that the user can specify the position parameters that can be specified from among the plurality of position parameters, and sequentially set the necessary position parameters according to the guidance. Such an example is shown in FIG. 26 as a robot system 3000 according to the third embodiment. A robot system 3000 shown in this figure includes an image processing device 300, a display unit 3B, an operation unit 4, a sensor unit 2, a robot controller 6, and a robot RBT. In the robot system shown in this figure, members that are the same as in FIG.
(Image processing apparatus 300)

  The image processing apparatus 300 includes an input image acquisition unit 2c, a calculation unit 10, a storage unit 9, an input / output interface 4b, a display interface 3f, and a robot interface 6b. The storage unit 9 includes a gripping position storage unit 9b. The gripping position storage unit 9b is a member for storing the gripping position of the work model or the end effector model specified by the gripping position specifying unit 8d.

  The calculation unit 10 includes a gripping position specifying unit 8d, a gripping position copying unit 8d8, a relative position setting unit 8d5, a search model registration unit 8g, a three-dimensional search unit 8k, a three-dimensional pick determination unit 8l, a cross-sectional model. A generator 8s.

The gripping position specifying unit 8d specifies the position and orientation of either or both of the work model and the end effector model displayed on the virtual three-dimensional space on the display unit, on the X axis, the Y axis, and the Z axis. X-coordinate is a coordinate position, Y-coordinate, Z coordinate, and the X-axis, Y-axis, R _X rotation angle is a rotation angle about the Z axis, R _Y rotation angle, the six positional parameters of R _Z rotation angle Are members for designating each.

  The gripping position copy unit 8d8 is a member for reading the gripping position of the work model or the end effector model stored in the gripping position storage unit 9b, changing the gripping position, and registering it as a new gripping position. As a result, when registering a plurality of gripping positions, the gripping positions that have already been registered are read out, the gripping position is changed based on this, and the gripping position can be registered as a new gripping position. Compared with the case of registration, a gripping position can be added more easily, and the labor of registration work can be saved.

  The three-dimensional pick determination unit 8l determines whether or not the end effector can grip each search result searched by the three-dimensional search unit 8k at the gripping position specified for the work model by the gripping position specifying unit 8d. It is a member for determining.

The cross-section model generation unit 8s includes a cross section obtained by cutting the end effector model along a plane that is orthogonal to the basic axis along a linear basic axis set in one direction with respect to the end effector model, and an orthogonal including each cross section. A cross-section model composed of a set of a plurality of cross-sections and cross-section positions is generated with a point where the plane intersects the basic axis as a cross-section position. Polygon data or the like is used as an end effector model for creating this cross-sectional model.
(Gripping position specifying part 8d)

26 includes a first designation unit 8d9, a second designation unit 8d6, and a third designation unit 8d7. The first designation unit 8d9 is a member that can designate at least one of the X coordinate, the Y coordinate, and the Z coordinate, and cannot designate other positional parameters. The second designation unit 8d6 has at least one of the R _X rotation angle, the R _Y rotation angle, and the R _Z rotation angle with respect to the work model or the end effector model in which some position parameters are designated by the first designation unit 8d9. Any member can be specified, and other position parameters cannot be specified. The third designation unit 8d7 is configured to perform an R _X rotation angle, an R _Y rotation angle, a work model or an end effector model in which some position parameters are designated by the first designation unit 8d9 and the second designation unit 8d6. This is a member that can designate a position parameter that is not designated by the second designation unit 8d6 and that cannot designate other position parameters, out of at least one of the R _Z rotation angles.

As a result, instead of having all six position parameters specified on a single screen, the gripping position can be specified by dividing the specified parameters into a plurality of specified parts and specifying the position parameters individually. By doing so, it is possible to avoid the situation where multiple position parameters are entangled with each other and it is difficult to grasp the position and posture, and by sequentially wearing the position parameters, information necessary for specifying the position and posture is set. Is possible. In particular, the first designation unit 8d9 has a configuration in which the initial position of the gripping position is designated by clicking a mouse or the like on any part of the work model that is displayed on the display unit in a planar shape. It is possible to let the user specify in a visually easy way to specify directly on the work model, which is much simpler than the complicated and cumbersome work of traditional posture determination and numerical specification it can. Note that the first designation unit 8d9 is not necessarily limited to the mode in which the work model is displayed in a two-dimensional form, and may be displayed in a three-dimensional form.
(Procedure for creating settings)

  Here, the procedure of teaching work for setting work for operating the robot system before actual operation will be described based on the flowchart of FIG.

  First, in step S2701, a search model for performing a three-dimensional search for a workpiece is registered. Here, as described above, a work model, here, three-dimensional CAD data can be registered as the search model. Alternatively, actual measurement data obtained by actually imaging the workpiece by the sensor unit may be registered as a search model.

In step S2702, the robot end effector model is registered. Again, three-dimensional CAD data can be registered as an end effector model. Next, in step S2703, the surface of the work model to be grasped is selected from the height image. Next, in step S2704, the position and posture of the robot when gripping the selected surface are registered. In step S2705, it is determined whether the necessary number of positions and postures have been registered. If not, the process returns to step S2703 to repeat the above processing. If the necessary number of registrations are completed, the process is terminated.
(Procedure for registering 3D CAD data in the search model)

  Here, an example of a procedure for registering a workpiece search model when three-dimensional CAD data is used as a search model in step S2701 will be described based on the flowchart of FIG. First, in step S2801, a three-dimensional CAD data model of a work is read. In step S2802, the center of the circumscribed cuboid of the three-dimensional CAD data model is corrected to the origin of the three-dimensional CAD data. Further, in step S2803, a height image viewed from each direction of “up”, “down”, “left”, “right”, “front”, and “rear” is generated. Here, when the height image is generated from the three-dimensional CAD data, the height image is generated so that the origin of the CAD becomes the center of the height image.

  Next, in step S2804, the height image that looks the same is deleted from the generated height image. Finally, in step S2805, a search model is registered using the generated height image.

In this way, the user can register the search model used for the three-dimensional search according to the guidance.
(Register end effector model)

  Next, details of the procedure for registering the end effector model in step S2702 of FIG. 27 will be described based on the flowchart of FIG. Here, it is assumed that the three-dimensional CAD data constituting the end effector model is composed of polygon data.

First, in step S2901, polygon data of the end effector model is read. Next, in step S2902, the direction in which the cross section is created is determined. In step S2903, a cross-sectional model is created. The cross-section model is created by the cross-section model generation unit 8s in FIG. Details of creation of the cross-sectional model will be described later. In this way, the end effector model is registered in the storage unit 9.
(Additional area)

  In addition, when registering the end effector model, an additional region can be added to the original end effector model. Such a procedure will be described based on the flowchart of FIG. First, in step S3001, an end effector model is registered. For example, an end effector model composed of three-dimensional CAD data such as STL data is read.

In step S3002, an additional area is set. For example, in the case of interference determination (details will be described later), the additional area is for adding an actual end effector shape or a shape simulating the attached cover or harness to the surface of the end effector model. The accuracy of interference determination can be improved.
(Embodiment 4)

An additional model creation function is used to set such an additional area. An example of a robot system to which an additional model creation function is added is shown in the block diagram of FIG. A robot system 4000 shown in this figure includes an image processing device 400, a display unit 3, an operation unit 4, a sensor unit 2, a robot controller 6, and a robot RBT. In the robot system shown in this figure, members that are the same as in FIG.
(Image processing apparatus 400)

  The image processing apparatus 400 includes an input image acquisition unit 2c, a calculation unit 10, a storage unit 9, an input / output interface 4b, a display interface 3f, and a robot interface 6b.

  The calculation unit 10 includes a work model registration unit 8t, an end effector model registration unit 8u, an additional model creation unit 8v, a gripping position specifying unit 8d, a search model registration unit 8g, a three-dimensional search unit 8k, and interference determination. 8m and a cross-sectional model generation unit 8s.

  The additional model creation unit 8v is a member for creating an additional model in which an additional region expressed by one or more three-dimensional figures defined in advance is added to the surface of the end effector model. This makes it easy to add an interference determination area with a simple shape such as a rectangular parallelepiped or a cylinder to the end effector model without editing the 3D CAD data, and perform interference determination according to each area. Interference determination can be performed. The three-dimensional figure includes a figure that can be freely designed by the user in addition to the basic figure prepared in advance on the image processing apparatus side.

  The gripping position specifying unit 8d is a member for specifying one or more gripping positions at which the work model is gripped by the end effector model with respect to the work model registered by the work model registration unit 8t.

  The three-dimensional search unit 8k specifies an image region corresponding to the position and orientation of each workpiece using the search model registered by the search model registration unit 8g from the input image acquired by the input image acquisition unit 2c. This is a member for performing a three-dimensional search.

The interference determination unit 8m is an additional model created by the additional model creation unit 8v. When the end effector is operated, the interference determination unit determines whether there is interference with another object that may interfere with the operation. It is a member for performing. When the interference determination unit 8m moves the end effector to the position in an attempt to grip any of the plurality of workpieces included in the input image acquired by the input image acquisition unit 2c, In order to determine whether or not to interfere with the effector, interference determination is performed on an image constituting an object around the work in the input image. Thereby, interference can be determined by comparing the end effector model with the input image. For example, the interference determination unit 8m is specified by the gripping position specifying unit 8d for gripping the workpiece with respect to the search result corresponding to the position and posture of each workpiece searched from the input image by the three-dimensional search unit 8k. When the end effector model is placed at the gripping position, it is determined whether there is interference with an object existing around the workpiece.
(Additional area setting screen 310)

  The additional area is added to the end effector model composed of three-dimensional CAD data. Here, as one aspect of the additional model creation unit 8v, an additional region setting screen 310 for adding an additional region to the end effector model is shown in FIG. The additional area setting screen 310 can read an end effector model EEM, which is three-dimensional CAD data of an end effector, and add a figure to the end effector model EEM. The additional area setting screen 310 includes an image display field 141 and an operation field 142. In the image display column 141, the three-dimensional CAD data of the read end effector model EEM is displayed. In the operation field 142, a basic graphic display field 311 is provided. The basic figure display field 311 displays a list of selectable basic figures. Basic figures include a rectangular parallelepiped, a cylinder, a cone, a triangular pyramid, a hexagonal column, and the like. The user can add a basic figure to a specified position on the end effector model EEM by selecting a desired basic figure from the basic figure display field 311 and pressing an “add” button 312. Each line displayed as a list in the basic graphic display field 311 is provided with an “edit” button 313. When this button is pressed, the setting of the selected basic graphic can be changed. For example, basic graphic parameters such as the length, radius and height of one side of the bottom surface can be adjusted.

  In the example of FIG. 32, a rectangular parallelepiped is selected as a basic figure to be added, and is highlighted in the basic figure display field 311. When the “edit” button 313 is pressed in this state, a basic graphic editing screen 320 shown in FIG. 33 is displayed. The operation field 142 of the basic figure editing screen 320 is provided with a basic image parameter adjustment field 321 from which the size of the basic figure and the position and orientation of the end effector with respect to the reference position can be changed. When the basic graphic parameter is adjusted from the basic image parameter adjustment field 321, the display content of the image display field 141 is also updated accordingly.

  In the example of FIG. 32, the additional area ADA is inserted between the flange surface FLS at the tip of the arm portion and the end effector model EEM. Thereby, the state where the attachment jig etc. were interposed between the arm part and the end effector can be reproduced. Further, when the flange surface at the tip of the arm portion is displaced from the surface set for calculation on the image processing apparatus side, it can also be used to offset this displacement amount.

  In addition, multiple figures can be added. In the example of FIG. 32, various basic figures can be added by selecting a desired basic figure from the basic figure display field 311 and pressing the “Add” button 312. Thus, various shapes can be expressed by combining a plurality of basic figures.

  In this way, by providing a function for setting an additional area for the end effector model, the basics prepared in advance can be used without changing the shape by directly editing the shape of the three-dimensional CAD data as in the past. A shape can be easily added only by adding a figure. As a result, when the end effector is placed at the gripping position for gripping the workpiece, it is not included in the end effector model in the interference judgment to confirm in advance whether or not it interferes with objects around the workpiece, but it actually exists When there is a member to be performed, the accuracy of the determination result can be improved as a shape imitating these. For example, when the end effector is connected to the tip of the arm part via a joint, the entire end effector may be offset and the tip may protrude slightly. Further, there may be an additional element such as a cover for preventing contact on the outside of the end effector or a cable extending from the end effector. Such additional elements are often not included in the three-dimensional CAD data of the end effector model, and CAD data is rarely prepared separately. However, it is troublesome to process the three-dimensional CAD data of the end effector model and edit it so as to change the shape according to these additional elements. Even if a complicated shape can be expressed by CAD data, if such a complicated shape of three-dimensional CAD data is used for interference determination, the arithmetic processing becomes complicated and the processing time becomes long.

Therefore, by expressing such additional elements as graphics, it is possible to obtain an end effector model that is close to the actual situation in a simple additional area without complicated editing work. It becomes possible to easily express the shape. In addition, with this method, since the interference determination area is added without changing the shape of the CAD data, an increase in processing time can be suppressed.
(Grip position setting procedure)

Next, the procedure for registering the gripping position and posture for gripping the workpiece with the end effector in step S2704 in FIG. 27 will be described based on the flowchart in FIG. 34 and FIGS. 35A to 38. Here, when the work model WM11 is held with the work model WM11 being fixed using the work model WM11 and the end effector model EEM composed of the three-dimensional CAD data, the end effector model EEM of the work model WM11 is grasped. The six position parameters that define the position and posture (hereinafter collectively referred to as “grip position”), that is, the grip positions X, Y, Z and the grip postures R _X , R _Y , R _Z are shown in FIG. Setting is performed in the order shown in the flowchart.
(XY designation part)

First, in step S3401, the X and Y coordinates of the grip position are designated. Here, a two-dimensional planar image obtained by projecting the workpiece onto a certain plane is displayed by the XY designation unit, and the gripping position is designated on the planar image. FIG. 35B shows an XY designation screen 230 as an example of such an XY designation unit. The XY designation screen 230 shown in this figure displays the three-dimensional CAD data of the workpiece shown in FIG. 35A in a plan view in the image display column 141. In this state, the user is allowed to specify which position of the work model WM11F displayed in a plan view is to be gripped by the end effector model. As shown in FIG. 35B, by selecting a gripping position on a plan view with a pointing device such as a mouse, a desired position can be designated in an easily understandable manner without being conscious of the posture and the rotation angle. In particular, by clicking and specifying a position to be gripped on the work model WM11F displayed in a plan view, an advantage that it can be easily specified is obtained as compared to inputting a numerical value and adjusting the position. The X and Y coordinates of the grip position can be determined from the position manually designated by the user and the scale of the image. Note that an X and Y coordinate display field for displaying X and Y coordinates as numerical values may be provided on the XY designation screen.
(Z-R _Z designation part)

In step S3402, it specifies the gripping position Z and the grasping posture R _Z from Z-R _Z designation unit. FIG. 36 shows a Z-R _Z designation screen 240 as one form of the Z-R _Z designation unit. The Z- _RZ designation screen 240 shown in this figure includes an image display field 141 and an operation field 142. The operation field 142 includes a Z coordinate designation field 241 for designating the Z coordinate of the gripping position and an R _Z rotation angle designation field 242 for designating the R _Z rotation angle of the gripping posture. At this stage, since the X and Y coordinates have already been determined in step S3401, it is sufficient to display only the Z axis. Therefore, with the end effector model EEM and the work model WM11 displayed in the image display field, only the corrected rotation Z axis AXZ is displayed as a correction rotation axis in an overlapping manner. This makes it easier for the user to visualize how the end effector model EEM displayed in the image display field 141 is rotated when the rotation angle is changed, and the position adjustment work of the end effector model EEM can be easily performed. Benefits that can be obtained. In particular, by not displaying other rotation axes, the user can visually grasp the rotation direction without being confused. Furthermore, since other position parameters other than the Z coordinate and R _Z rotation angle cannot be specified, it is also possible to change other position parameters by mistake, and the user concentrates on only the position parameters that can be specified. It is possible to concentrate on only the necessary settings while eliminating confusion such as misunderstandings and misconfigurations.

Rotation of the end effector model EEM, upon inputting a value R _Z rotation angle designation column 242, end effector model EEM in the image display field 141 is displayed automatically rotated accordingly. When the end effector model EEM displayed in the image display column 141 is dragged and rotated, the value of the R _Z rotation angle displayed in the R _Z rotation angle designation column 242 is changed accordingly.
(R _Y designation part)

In step S3403, a gripping posture _RY is designated. Here, with R _Y designation screen 250 shown in FIG. 37 designates the R _Y rotation angle from R _Y rotation angle designation column 251. In this example, the postures of the end effector model EEM and the work model WM11 are expressed by ZYX system Euler angles. Therefore, the direction of the rotation axis of R _Y can be corrected by using the value of R _Z specified in step S3402. The corrected rotation Y axis AXY that is the calculated correction rotation axis is displayed in the image display field 141 in the same manner as described above. Displaying the rotation axis in this way makes it easy for the user to imagine how the end effector model EEM changes when the _RY rotation angle is adjusted, and makes it easier to set.
(R _X designation part)

Finally, in step S3404, it specifies the gripping posture R _X. Here, with R _X designation screen 260 shown in FIG. 38 designates the R _X rotation angle from R _X rotation angle designation column 261. Similarly, the corrected rotation X axis AXX indicating the direction of the rotation axis of R _{X is} also displayed in the image display column 141.

In this way, the user can designate six position parameters X, Y, Z, R _X , R _Y , and R _Z as gripping positions for gripping the workpiece with the end effector. In particular, the position parameters that can be adjusted are limited for each screen, and the rotation axis to be displayed is limited to the display of only the rotation axis related to the specified position parameter, so that the user can set according to such guidance. Thus, the necessary position parameters can be defined sequentially. As a result, it is possible to smoothly perform the grip position designation work in which the three-dimensional position and posture are designated, which has been difficult in the past. Also, when setting the Euler angles R _X , R _Y , R _Z rotation angles, the rotation axis is calculated and displayed as a correction rotation axis in the image display column 141, so that the position parameter being set by the user is displayed. With this change, you can easily imagine how it works.

Furthermore, by dividing the registration of the gripping position into a plurality of steps and limiting the position parameters that can be adjusted in each step, it becomes possible to easily imagine how the position moves when the position parameter is moved. In particular, by providing a designation screen for each step and registering different position parameters on different designation screens, the position parameters to be set can be presented to the user without confusion, and setting work can be guided. In the example of FIGS. 36 to 38, an example in which different designation screens are provided has been described. However, the present invention is not limited to this example. For example, as shown in the position parameter specification screen 270 of FIG. 39, the position parameters that can be specified can be limited for each step while using a common specification screen for specifying a plurality of position parameters. In the example of FIG. 39, only the X and Y coordinates can be designated from the X and Y coordinate designation column 271, and other position parameters are grayed out so that they cannot be designated and cannot be selected. And X, after finishing specifies the Y coordinate, by pressing the "Next" button 272, a different position parameter as a next step, for example, Z coordinates and R _Z only the rotation angle Z coordinate, R _Z rotation angle specified It is possible to specify from the column 273, and other position parameters including the specified X and Y coordinates are all grayed out or not displayed, for example, not displayed. Then, by pressing the “Next” button 272 again after designating the Z coordinate and the R _Z rotation angle, another position parameter, for example, only the R _X and R _Y rotation angles are rotated by the R _X and R _Y rotations as the next step. Designation is possible from the corner designation column 274, and other position parameters cannot be selected. Even with such a configuration, it is possible to realize a guidance function that restricts and presents position parameters that can be specified to the user and sequentially sets them.

Further, the number and division of steps for sequentially defining the position parameters are not limited to the above example. For example, the Z coordinate and the R _Z rotation angle may be separate steps, or the R _X rotation angle and the R _Y rotation angle may be set in the same step. For example, as an example in which the position step is divided into a plurality of steps, the Z-R _Z designation unit is designated as the Z designation unit for designating the Z coordinate, and the R _Z rotation angle around the Z axis is designated as R. _{Consists of the Z} designation part. Alternatively, the R _X -R _Y designation part is designated with an R _X designation part for designating an R _X rotation angle around the X axis and an R _Y designation for designating an R _Y rotation angle around the Y axis. It can also consist of parts. Alternatively, FIG. 37 and FIG. 38 may be integrated into an R _X -R _Y designating unit that can designate the R _X rotation angle and the R _Y rotation angle on one screen.

  In the above example, the example in which the position and orientation of the end effector model are adjusted with the work model side fixed is described. However, the present invention is not limited to this, and the position on the work model side with the end effector model fixed. And the posture may be adjusted. Or you may comprise so that the position and attitude | position of both an end effector model and a workpiece | work model may be adjusted.

In the above example, the six positional parameters X, Y, Z, R _X , R _Y , and R _Z are converted into the X and Y coordinates in the XY designation section, the Z coordinates and R _{Z in the} Z-R _Z designation section. Although an example in which R _X and R _Y rotation angles are defined in order in the rotation angle and R _X -R _Y designating part has been described, the order and combinations for defining the position parameters are not limited to the above configuration. For example, the X-Y-Z designation unit, and R _Z designation unit, R _X -R _Y specify section was prepared, X-Y-Z at the specified portion, X of the end effector model or the work model, Y, Z-coordinate Next, in the R _Z designation part, specify the R _Z rotation angle around the Z axis. Finally, in the R _X -R _Y designation part, the R _X rotation angle around the X axis and the Y axis The R _Y rotation angle with respect to the center may be designated. In this case, the XYZ designating unit can use, for example, an XYZ designating screen 280 as shown in FIG. 40, and with respect to the end effector model EEM or work model displayed three-dimensionally, , Y, Z coordinates can be designated from the X, Y, Z coordinate designation column 281. As described above, the image for first specifying the gripping position is not limited to the planar image projected on the XY plane as shown in FIG. 35B, and different projection directions can be used. For example, as shown in FIG. It can also be specified using the perspective view WM11P projected from the direction.

Furthermore, the gripping position may be specified without being projected as shown in FIG. 35A, not limited to the method of specifying the gripping position from the image once projected onto the plane as shown in FIG. 35B. For example, in a state where the work model WM11 is displayed three-dimensionally on a three-dimensional image viewer 290 as shown in FIG. 42, a gripping position is designated from this screen using an operation unit configured with a pointing device such as a mouse. To do.
(Modification)

Further, in the above example, the procedure for specifying the six positional parameters of X, Y, Z, R _X , R _Y , and R _Z according to the ZY-X system Euler angles has been described. Without limitation, other modes that define the position and orientation of the end effector and the workpiece, such as XYZ Euler angle, XZY Euler angle, YXZ Euler angle, YZ -X Euler angle, ZXY Euler angle, XYX Euler angle, XZX Euler angle, YXY Euler angle, YZY Euler angle The position and orientation of the end effector can also be specified according to the Z-X-Z Euler angle, Z-Y-Z Euler angle, roll / pitch / yaw angle expression, rotation axis / rotation angle expression, etc. .

In the above example, an example in which only the correction rotation axis related to the position parameter to be specified is displayed as the correction rotation axis has been described. However, the other correction rotation axes are limited to a configuration in which they are completely hidden. Not. For example, the same effect can be obtained by displaying the three orthogonal axes including the other correction rotation axes while displaying the rotation axis related to the rotation angle to be specified with emphasis over the other correction rotation axes. Can be obtained. As an example, the rotation axis for which the rotation angle is specified is displayed in bold, highlighting such as coloring or blinking, conversely the rotation axis that can not be specified is grayed out, or displayed with thin lines , To distinguish on the appearance. Especially in the 3D display of images, if there are orthogonal coordinate axes, the user can intuitively understand that the 3D display is in progress, so the target rotation axis can be set differently while maintaining the 3 axis display. By distinguishing, it is difficult to confuse the user. Furthermore, it is preferable that the origin when displaying such three axes is the center of rotation. For example, the intermediate position between the pair of claws, which is the gripping position of the above-described end effector model EEM.
(Grip position copy function)

Furthermore, when registering a plurality of gripping positions, a gripping position copy function may be provided in which the gripping positions that have already been registered are read out, the gripping positions are changed on the basis thereof, and registration is possible as new gripping positions. . In general, a plurality of gripping positions are often registered for one workpiece. This is because, if a plurality of gripping positions are registered, the optimum solution can be selected from a plurality of gripping solutions. Therefore, even if gripping solution candidates obtained interfere with each other, This is because, if there is a candidate, the possibility that it is determined that it can be gripped increases. When registering a plurality of such gripping solutions, if the grip registration is performed from the beginning each time, the number of steps is increased when registering the same gripping position, and the work becomes troublesome. Therefore, it is possible to save a plurality of gripping positions by copying a gripping position that has already been registered, changing a part of the position parameters set at this gripping position, and saving it as a new gripping position. Can be easily registered. Similarly, it is also possible to read out an existing gripping position, modify the position parameters as appropriate, and overwrite and store them.
(Holding position copy unit 8d8)

  Such a gripping position copy function and a gripping position editing function are realized by the gripping position copy unit 8d8 shown in FIG. The gripping position copying unit 8d8 reads the gripping positions of the work model and the end effector model stored in the gripping position storage unit 9b, and changes and registers the position parameters constituting the gripping position. In addition, when the gripping position is changed by the gripping position copy unit 8d8, as described above, if the order is registered according to the guidance, it is easy to understand which position parameter should be changed in which step. Registration of grip becomes easy.

Furthermore, as described above, when specifying the gripping position, a fitting function for automatically moving the end effector model to the gripping position of the work model may be added. For example, by providing a “fit” button 154 in the operation column 142 on the Z- _RZ designation screen 240 of FIG. 36, the fit function can be executed to automatically set the Z coordinate.
(Embodiment 5)

  Furthermore, in order to improve the accuracy of the three-dimensional search, it is possible to limit the postures and angles that the search model can take. For example, consider that the position and orientation of a workpiece are detected by a three-dimensional search from a workpiece group in which a plurality of flat workpieces WK9 as shown in FIG. 43 are stacked. In this case, each surface of the work model WM9 in which the work WK9 in FIG. 43 is represented by three-dimensional CAD data is as shown in FIGS. 44A to 44F. 44A is a plan view of the work model WM9, FIG. 44B is a bottom view, FIG. 44C is a front view, FIG. 44D is a rear view, FIG. 44E is a right side view, and FIG. From these basic direction images, an overlapping surface is removed and registered as a search model. Here, it is assumed that the four surfaces of FIGS. 44A, 44B, 44C, and 44E are registered. In this state, a group of workpieces is picked up to acquire point cloud data having 3D information, and a 3D search is performed. As a result, the workpiece may be erroneously detected as if the workpiece is present in the vertical posture as indicated by X in FIG. Actually, it is difficult to think that a thin flat work exists in an upright posture, but the front view of the flat work is almost linear or rectangular as shown in FIG. 44C. May be common to other parts constituting the structure, for example, the shape of a part of FIGS. 44A and 44B, and as a result, erroneous detection occurs.

  On the other hand, it is conceivable to perform a three-dimensional search by limiting the postures that can be taken by the workpiece, but setting conditions for posture limitation is extremely difficult. For example, it is known that all postures of a workpiece can be expressed by defining the rotation angle range of the Z axis, the Y axis, and the Z axis by using the Euler angles of the ZYZ system as shown in FIG. However, it is not easy to define the rotation angle of each axis so as to be in the posture range desired by the user by this method.

Therefore, in the fifth embodiment, the posture that the user can take can be easily set without using such a troublesome concept. Specifically, when a search model is registered, such a plane is prevented from being detected by eliminating postures that are unlikely to occur when the search models are stacked. The search model registration unit 8g shown in FIG. 6 is used for such setting. The search model registration unit 8g registers, as a search model for performing a three-dimensional search for specifying the position and orientation of each workpiece, for a plurality of stacked workpieces from any of the basic direction images. Select the basic orientation image. The difference between the search model registration unit 8g and the basic direction image selection unit 8e is that the basic direction image selection unit 8e selects one of the basic direction images with the same appearance. On the other hand, the search model registration unit 8g is for excluding, from the search model registration target, basic direction images that are not to be searched for the three-dimensional search. Preferably, the basic direction image selecting unit 8e excludes the basic direction images whose appearances overlap, and the search model registration unit 8g further excludes unnecessary basic direction images from the search model registration target. Alternatively, the basic direction image to be searched is selected. This allows the user to select the basic direction images that are candidates for search model registration in advance, thereby reducing the number of candidates and facilitating search model registration.
(Search model registration screen 130B)

  Here, FIG. 47 shows an example of a search model registration screen 130B for registering a search model for the work shown in FIG. In the search model registration screen 130B shown in this figure, six basic direction images corresponding to the six-sided view generated by the basic direction image generation unit 8e ′ from the work model WM9 corresponding to the work WK9 of FIG. The basic direction image selection unit 8e preliminarily removes the basic direction images whose appearances are overlapped. In the example of FIG. 44, the remaining four basic direction images are displayed on the display unit in the state where two images of FIGS. 44D and 44F are excluded. It is displayed in the surface view display area 3a. In this state, as one form of the search model registration unit 8g for setting whether or not the search model can be registered, a selection check box 131B for “register” displayed on each basic direction image in the search model registration screen 130B is provided. Yes. If the user checks the selection check box 131B, the basic direction image is registered as a search model. Conversely, if the user turns off the selection check box 131B, the basic direction image is excluded from the search model registration target. The user can eliminate postures that are unlikely to occur in an actual stacking state while viewing each basic direction image displayed in the hexahedral view display area 3a. In the example of FIG. 47, the selection check box 131B for the model C and the model D is turned off, and the posture in which such a side can be seen, that is, the posture in which the plate-like workpiece is upright is excluded from the target of the three-dimensional search. Can do. As a result, the user can eliminate the images that do not need to be searched or select only the necessary images while looking at the basic direction image that shows the posture of the workpiece without going through the troublesome task angle calculation and range specification. As a result, it is possible to easily limit the posture of the workpiece to be subjected to the three-dimensional search.

  Note that the search model registration unit 8g is not limited to a mode in which the user manually selects a search model registration target or exclusion target, but calculates a shape, a center of gravity, and the like of the work model, and a basic direction image with a posture that is unlikely to occur. May be automatically extracted and excluded.

  Alternatively, a combination of automatic calculation and manual selection, for example, as shown in FIG. 47, on the search model registration screen 130B, all basic direction images that have been selected by the basic direction image selection unit 8e are displayed, and it is determined that it is difficult to occur by calculation. The basic direction image may be displayed with the selection check box 131B off as an initial state. By referring to the result of automatic determination by the search model registration unit 8g, the user manually corrects the selection as necessary and presses OK, so that the search model can be properly registered more reliably.

  In FIG. 47, an example in which selection by the basic direction image selection unit 8e is automatically calculated has been described. However, automatic determination by the basic direction image selection unit or manual selection can be combined in the same manner. For example, as a result of the determination by the basic direction image selection unit, one image determined to be a basic direction image having a common appearance is grayed out and displayed in the six-view drawing display area, so that the basic direction image is displayed to the user. The result of automatic determination by the selection unit can be confirmed. If the determination result is incorrect, the user can manually select the basic direction image and leave it as the basic direction image or conversely eliminate it. Thus, the accuracy of image selection can be further improved by displaying the result of the automatic calculation and allowing the user to confirm the result. In addition, by setting a state in which selection / non-selection is set in advance based on the automatic determination result as an initial state, if the selection result by automatic determination is correct, the user can confirm this by pressing OK. Time and effort can be minimized.

The purpose of excluding unnecessary surfaces from the target of the three-dimensional search can be applied to other than the upright state of the workpiece. For example, in a case where a work group is given as an input image in a mode in which only a specific surface is exposed in addition to a mode in which workpieces are loaded completely randomly, false detection can be avoided by eliminating the surface that is not exposed. . In particular, it is effective for a workpiece whose front and back have similar shapes and whose back surface is easily misdetected. In this way, the same effect as that in which the posture of the workpiece is limited so that the back side of the workpiece is not detected can be obtained only by removing the search model of the back surface that is not actually seen from the target of the three-dimensional search. .
(Procedure for registering a search model)

  The above-described search model registration procedure with posture restriction will be described with reference to the flowchart of FIG. Here, a procedure for registering the three-dimensional CAD data of the workpiece as a search model will be described.

  First, in step S4801, the three-dimensional CAD data of the workpiece is read. In step S4802, the center of the circumscribed cuboid of the three-dimensional CAD data model is corrected to the origin of the three-dimensional CAD data. In step S4803, a height image viewed from each direction of “up”, “down”, “left”, “right”, “front”, and “rear” is generated. Here, when the height image is generated from the three-dimensional CAD data, the height image is generated so that the origin of the CAD becomes the center of the height image. In step S4804, a height image that looks the same is deleted from the generated height images.

  In step S4805, a search model used for the three-dimensional search is selected from the remaining height images. Here, the search model selection unit 8i eliminates unnecessary basic direction images and selects necessary basic direction images.

Finally, in step S4806, the selected height image is registered as a search model. Here, the search model registration unit 8g registers the search model. In this way, by eliminating the basic direction images of unnecessary postures, it becomes possible to register the search model in a state in which posture restrictions are substantially provided.
(Embodiment 6)

In the above, the example which restrict | limits the object of a three-dimensional search using an actual image instead of a numerical value was demonstrated. However, the present invention is not limited to this mode, and instead of this, or in addition to this, the posture can be limited by the inclination angle or rotation angle of the search model. This makes it possible to appropriately set the three-dimensional search conditions according to how the workpieces are actually stacked, and to reduce erroneous detection of the search. An example of an inclination angle / rotation angle setting screen 160 for performing posture restriction based on such an inclination angle and a rotation angle is shown in FIG. An inclination angle / rotation angle setting screen 160 shown in this figure includes an attitude restriction search model selection field 161 for selecting a search model for attitude restriction, and an inclination angle upper limit setting field for designating an upper limit of an angle allowing inclination. 162, a rotation angle range setting field 163 that defines an angle range in which rotation is allowed is provided.
(Inclination angle / rotation angle setting screen 160)

The tilt angle / rotation angle setting screen 160 is not a difficult method of specifying a three-dimensional posture with respect to the workpiece, for example, R _X , R _Y , R _Z indicating the rotation angle of the coordinate axes, but has specialized knowledge. As parameters that are easy to understand even if they are not specified, they are designated using “inclination angle” and “rotation angle” from the attitude at the time of registration of the search model. Here, the basic direction image registered as the search model is set as the registered posture, and the tilt angle and the rotation angle from this state are designated (details will be described later).

  Specifically, in the tilt angle / rotation angle setting screen 160 of FIG. 49, the search model to be subjected to posture restriction is selected in the posture restriction search model selection column 161. In the example of FIG. 49, the search model of model A registered on the search model registration screen 130B of FIG. 47 is selected.

  In addition, the tilt angle upper limit setting column 162 sets how many tilt angles are output as a result of the three-dimensional search for the posture at the time of registration.

Further, in the rotation angle range setting field 163, a reference angle and an angle range are set. First, in the reference angle setting field, the rotation angle with respect to the posture at the time of registration is input. The rotation angle set here becomes a reference angle for designating the rotation angle range. Further, in the range setting column, a rotation angle range output as a result of the three-dimensional search is set with respect to the reference angle set in the reference angle setting column.
(Attitude limit method by tilt angle and rotation angle)

  Here, a method for obtaining the tilt angle and the rotation angle from the three-dimensional posture of the search model will be described. In the above-described method for limiting the surfaces registered as search models in the three-dimensional search, posture restriction is automatically added. On the other hand, in order to apply a posture restriction to a surface registered as a search model, the fact that the surface is registered is used.

Here, posture restriction is performed at two angles: an “inclination angle” with respect to the registered posture and a “rotation angle” viewed from directly above the registered posture. With these two angles, it is easy to understand as a concept and the angle can be limited relatively easily.
(Inclination angle)

First, as shown in FIGS. 50A to 50C, the inclination angle is defined by the angle of the posture of the work model at the time of registration as a search model with respect to the Z vertical direction. FIG. 50A shows the posture of the work model WMR when registering the search model and the Z axis defined as the vertical direction of the work model. In contrast, it is assumed that the posture of the workpiece WKI obtained as an input image and the Z ′ axis defined as the vertical direction of the workpiece WKI are in a state as shown in FIG. 50B. In such a state, as shown in FIG. 50C, the “tilt angle” is defined as the tilt angle between the Z axis and the Z ′ axis, that is, the tilt in the vertical direction of the workpiece. As a result, regardless of the direction of the inclination, it can be expressed by one angle, that is, the inclination from the registered state of the search model. Moreover, the advantage which is easy to grasp | ascertain also as a concept for a user is acquired.
(rotation angle)

  On the other hand, the rotation angle is defined by a vertical angle at the time of registration, that is, a rotation angle when viewed from the Z axis. Here, the rotation angle will be described with reference to FIGS. 51A to 51C. In these drawings, FIG. 51A shows a work model WMR at the time of registering a search model and a Y axis that defines an XY plane. On the other hand, it is assumed that the posture of the workpiece WKI obtained as an input image and the Y ′ axis that defines the XY plane are in a state as shown in FIG. 51B. In this state, as shown in FIG. 51C, the “rotation angle” is defined as the rotation angle when viewed from the vertical direction, that is, the Z axis. In this example, the rotation angle is defined by the angle of the Y axis viewed from the Z axis direction with reference to the coordinate axis in the Y axis direction that defines the workpiece model and the workpiece. By defining in this way, the same concept as the angle in the conventional two-dimensional pattern search has the advantage of being easily grasped by the user.

A known algorithm can be used as a method for defining the origin and the XYZ axes with respect to the workpiece. For example, as shown in FIG. 8, a figure circumscribing the workpiece, such as a rectangular parallelepiped or a sphere, is calculated from the workpiece shape information of the workpiece model or the input image, the center of gravity is calculated, and this is used as the workpiece origin, XYZ Define the axis. In the method of defining the XYZ axes, when CAD data is used, the direction of the coordinate axes of the original CAD data may be used as it is as the direction of each XYZ axis. When actually measured data obtained by 3D imaging of the workpiece is used, the vertical direction of the measured data is the Z axis, the Y axis is the upward direction of the measured data in plan view, and the X axis is the right direction of the measured data in plan view. What should I do? As for the origin, the center of gravity of the work model or the center coordinates of the CAD data may be used as the origin.
(Procedure for obtaining the tilt angle and rotation angle from the 3D posture)

  Here, a procedure for obtaining the tilt angle and the rotation angle from the three-dimensional posture will be described based on the flowchart of FIG. First, in step S5201, an inclination angle is obtained. For example, if the posture of the work WKI obtained as an input image is as shown in FIG. 53B with respect to the work model WMR at the time of registration in the state shown in FIG. 53A, the Z-axis vertical as shown in FIG. 53C The inclination angle is obtained as the inclination of the direction.

  Next, in step S5202, the input image is three-dimensionally rotated so that the Z ′ axis coincides with the Z axis. In other words, it eliminates the slope. For example, the input image WKI in FIG. 53C is rotated so that the Z′-axis overlaps the Z-axis with the outer product vector VP of the Z′-axis vector and the Z-axis vector as the rotation axis as shown in FIG. 54A. As a result, the rotated input image WKI ′ is obtained as shown in FIG. 54B, and the inclination is removed.

Finally, in step S5203, the rotation angle is obtained. For example, the Y axis of the work model WMR at the time of registration shown in FIG. 55A and the Y ′ axis of the input image WKI ′ with the inclination removed as shown in FIG. 55B are obtained. In this state, as shown in FIG. 55C, the angle between the Y axis and the Y ′ axis when viewed from the vertical direction, that is, the Z axis is obtained as the rotation angle. In this way, the tilt angle and the rotation angle can be obtained from the three-dimensional posture of the input image.
(Setting the upper limit of tilt angle)

As an example of setting the upper limit of the tilt angle on the tilt angle / rotation angle setting screen 160 of FIG. 49, for example, in the case of a highly glossy metal workpiece, the range in which three-dimensional measurement cannot be performed increases as the surface tilts. If a three-dimensional search is performed in such a state, the risk that the three-dimensional search is erroneously detected increases. Therefore, by excluding a posture that is tilted more than a certain level with a risk of erroneous detection from the target of the three-dimensional search, it becomes possible to leave only a highly reliable search result. For example, for a workpiece that is easily misdetected when tilted by 40 degrees or more, the search result tilted by 40 degrees or more can be excluded by setting the tilt angle upper limit to 40 degrees.
(Setting the upper limit of the rotation angle range)

On the other hand, as an example of setting the rotation angle range on the tilt angle / rotation angle setting screen 160 of FIG. 49, for example, the above-described model A and model B of FIG. Since there are many portions, if the number of places where three-dimensional measurement cannot be performed due to the influence of multiple reflection or the like increases, there may occur a case where the rotational posture is erroneous by 180 degrees. Moreover, when only a certain direction occurs as the actual work stacking method, the result of the reverse direction can be eliminated by using this posture restriction. For example, when the rotation angle range is set to ± 30 degrees with respect to the reference angle, only search results within this angle range are detected.
(End effector mounting position setting screen 170)

  The above-described method for defining the position and orientation using the Euler angles can be used not only for defining the position and orientation for gripping the work model with the end effector model, but also for defining the position for attaching the end effector to the tip of the robot arm. . Here, a procedure for setting the position where the end effector is attached to the tip of the arm portion of the robot will be described. Such an end effector mounting position is set on an end effector mounting position setting screen 170 as shown in FIG. The end effector attachment position setting screen 170 shown in this figure includes an image display field 141 and an operation field 142. In the image display column 141, an end effector model is displayed. The end effector model reads three-dimensional CAD data created in advance. Alternatively, it may be configured by displaying a basic figure such as a rectangular parallelepiped or a cylinder by imitating an end effector and displayed.

The operation field 142 is provided with an end effector attachment position setting field 171 that defines the attachment position of the end effector attached to the tip of the arm part. The end effector attachment position parameters set in the end effector attachment position setting field 171 are the attachment position (X, Y, Z) and the attachment posture (R _X , R _Y , R _Z ) of the end effector model. For example, the Euler angle is defined with respect to the center of the flange surface FLS at the tip of the arm portion.

In the work model gripping position registration screen shown in FIG. 14 and the like described above, the position of the end effector model part gripping the work model (for example, a pair of claws provided at the tip) and the work gripping position. The relationship is registered. On the other hand, on the end effector attachment position setting screen 170, the attachment position at which the end effector model is attached to the tip of the arm part is registered. Therefore, the display of the work model or the like is not necessary, and the end effector model and the flange surface FLS are displayed in the image display column 141.
(Gripping position designation screen 180)

  On the other hand, on the gripping position designation screen 180 for designating the gripping position of the work model, the end effector model and the work model are displayed as shown in FIG. The gripping position designation screen 180 shown in this figure is provided with an image display field 141 and an operation field 142. In the image display field 141, a workpiece to be gripped is selected and the surface for gripping the workpiece is directed upward. The image is drawn and displayed in a three-dimensional shape with the selected posture. Further, the end effector model is displayed and adjusted to the position and posture for gripping the work model. The operation field 142 is provided with a grip position designation field 181 for designating a grip position. The user adjusts the position parameter from the gripping position designation column 181 so that the relative relationship between the work model and the end effector model is correct. Alternatively, the position and orientation are adjusted on the screen by dragging the end effector model or work model displayed in the image display field 141. The position parameter adjusted on the screen is reflected in the gripping position designation field 181. The Euler angles described above can also be used for the position parameter notation in the gripping position designation field 181. (Multiple gripping position selection screen 190)

Furthermore, the registration of the gripping position is not limited to one place with respect to one surface (for example, the basic direction image) of one work, and a plurality of places can be designated as described above. Also, the gripping positions registered at a plurality of locations can be confirmed with images. For example, FIG. 58 shows an example of the multiple gripping position selection screen 190. The multiple gripping position selection screen 190 also includes an image display field 141 and an operation field 142. The operation field 142 is provided with a surface selection field 191 for selecting a surface of the work model defining the gripping position, and a gripping position list display 192 for displaying a list of gripping positions set on the surface selected in the surface selection field 191. It is done. In this example, the surface C is selected in the surface selection field 191, and grips 1 to 3 are displayed as a list in the gripping position list display 192 as gripping positions set for the surface C. When grip 2 is selected, a state in which the end effector model grips the work model at the grip position and posture registered by grip 2 is displayed in the image display field 141 accordingly. In this way, the user can select and confirm a plurality of registered gripping positions. Further, the registered gripping position can be corrected and updated as necessary. Again, the gripping posture can be displayed using the Euler angles described above. In this way, while performing registration using the Euler angle, the user side can visually display and correct the rotation axis without having to understand the concept of the difficult Euler angle in detail. By limiting the parameters that can be displayed and adjusted by the guidance function, the setting operation can be performed with less confusion.
(End effector mounting position calibration function)

  As described above, the end effector is attached to the tip of the arm portion of the robot. On the other hand, the above-described registration (teaching) of the gripping position, gripping determination (simulation) of whether or not the end effector model can grip the workpiece model at the registered position and orientation, and the like are not actual end effectors but image processing apparatuses. On the virtual three-dimensional space on the side (robot vision side). Then, the conversion from the coordinate position of the vision space, which is a virtual three-dimensional space, to the coordinate position of the robot space, which is a real space, is performed by the conversion unit 8x based on the calibration information obtained by the calibration unit 8w of FIG. Has been done.

  However, if the attachment state of the end effector model virtually set on the image processing apparatus side is different from the attachment state of the actual end effector EET of the robot RBT, the vision space and the robot space defined by the calibration information Is not maintained, and a shift occurs when an actual robot RBT is used to grip a workpiece.

Here, FIG. 59 shows the positional relationship between the end effector model and the flange surface FLS. In this figure, the origin O _F flange face FLS of the tip of the arm of the robot, as the position of the end effector model EEM, the center O _E of the end effector model EEM, respectively show. The position of the end effector model EEM is set as a position (X, Y, Z) and posture (R _X , R _Y , R _Z ) with respect to the flange surface FLS at the tip of the robot arm on the image processing apparatus side. Yes. That is, in the example of FIG. 59 as viewed from the origin O _F flange face FLS, the center O _E of the end effector model EEM, position (X, Y, Z) and attitude _{(R X, R Y, R} Z) is set Has been. Here, if an error occurs in the actual mounting state of the end effector, the position of the end effector model EEM with respect to the origin of the flange surface FLS may be displaced. As a result, due to the attachment state of the end effector, the image processing apparatus side and the robot side are gripped at the shifted positions.

Therefore, in the present embodiment, an end effector attachment position calibration function for reflecting the actual attachment state of the end effector on the image processing apparatus side is provided so that such deviation and error do not occur. The end effector mounting position calibration function is performed by the end effector mounting position calibration unit 8y shown in FIG. Specifically, using 3D CAD data that virtually represents the 3D shape of the end effector, which is the end effector model used for registering the gripping position of the workpiece and determining the interference when the end effector is placed. The image processing apparatus detects such an error by performing a three-dimensional search on actual measurement data obtained by imaging an actual end effector and performing three-dimensional measurement, and acquiring the position and orientation of the actual mounting position. Calibrate the side end effector mounting position setting. As described above, the deviation between the point group obtained by measuring the actual end effector three-dimensionally and the attachment state of the end effector model defined in the virtual three-dimensional space on the image processing apparatus side is displayed on the image display area of the display unit. You can adjust manually while checking.
(Automatic correction procedure)

Here, a calibration function for automatically correcting an error between an actual end effector and an end effector model constituted by three-dimensional CAD data or the like will be described based on the flowchart of FIG. Note that calibration for converting the coordinate position of the vision space, which is the virtual space, into the coordinate position of the robot space, which is the real space, is executed in advance by the calibration unit 8w shown in FIG. 17, and the calibration information is stored in the storage unit 9 It is assumed that Furthermore, it is assumed that an end effector model is also prepared as three-dimensional CAD data.
(End effector imaging screen 330)

  First, in step S6001, preparation for imaging an actual end effector by the sensor unit is performed. Specifically, the end effector is moved by operating the robot so that the end effector is reflected in a three-dimensional camera that is a form of the sensor unit. Next, in step S6002, three-dimensional measurement is performed on the end effector.

  For example, an end effector imaging screen 330 illustrated in FIG. 61 is used as one aspect of the end effector imaging unit for imaging an actual end effector with the sensor unit. The end effector imaging screen 330 is provided with an image display field 141 and an operation field 142. In the image display field 141, the end effector EET attached to the flange part at the tip of the arm part ARM imaged by the sensor part 2 is displayed in real time. The operation field 142 is provided with an end effector position designation field 331 indicating the position and posture of the end effector EET. From this end effector imaging screen 330, the user manually operates the robot so that the end effector EET of the robot is reflected on the camera constituting the sensor unit. At this time, what is to be imaged is not the claw portion of the end effector EET that holds the workpiece but the attachment state with the arm portion. Therefore, it is desirable to image so that the area of the end effector EET is large. Preferably, the end effector EET is changed from a downward posture in which a normal workpiece is easily gripped to a horizontal posture so that the whole image is reflected on the camera. In this way, after positioning the posture of the end effector EET, the user presses the “imaging” button 332. Thereby, the three-dimensional captured image of the end effector is acquired. In other words, three-dimensional measurement is performed on the end effector.

  Next, in step S6003, the position / orientation A of the flange portion of the robot coordinate system is acquired. Here, the position and orientation A in the robot coordinate system is, for example, the position and orientation in the robot coordinate system of the flange portion FLS to which the captured end effector is attached. Note that the order of steps S6003 and S6002 may be interchanged.

  In step S6004, the position and orientation A of the flange portion of the robot coordinate system is converted into a position and orientation B in the vision space. Here, based on the calibration information for performing coordinate conversion between the robot space and the vision space obtained by executing calibration in advance, the conversion unit 8x in FIG. 17 performs conversion between the real space and the virtual space.

  In step S6005, a three-dimensional search is performed to detect the position and orientation C of the end effector in the vision space. Here, the CAD data used for the end effector model is used as a search model, and the three-dimensional search is performed on the three-dimensional measurement data obtained in step S6002. As the three-dimensional search method, a known algorithm can be used as appropriate. Thereby, the position and posture of the end effector in the vision space are detected.

  In step S6006, the relative position and orientation of the end effector position and orientation C with respect to the position and orientation B of the flange portion FLS in the vision space are calculated. The coordinate position obtained here becomes the position and posture of the end effector model with respect to the flange portion FLS, in other words, taking into account the positional deviation and the like.

Finally, in step S6007, the obtained position and orientation are reflected in the end effector settings on the vision side. That is, it is reflected in the setting of the position and orientation of the vision-side end effector model. In this way, the actual end effector mounting state can be automatically reflected on the vision side.
(Procedure for manual correction of displacement)

  The procedure for automatically correcting the deviation between the robot space and the vision space has been described above. However, the present invention is not limited to the configuration in which the shift correction is automatically performed, and can be manually performed. Next, the procedure for manually performing the correction of deviation will be described based on the flowchart of FIG. Here, steps S1501 to S1504 are the same as the steps S6001 to S6004 of FIG. 60 relating to the automatic correction described above, and detailed description thereof is omitted. That is, in step S1501, the robot is operated so that the end effector of the robot is reflected on the three-dimensional camera. In step S1502, three-dimensional measurement is performed on the end effector. In step S1503, the position and orientation of the flange portion of the robot coordinate system are measured. A is acquired, and in step S1504, the position and orientation A of the flange portion of the robot coordinate system is converted into a position and orientation B in the vision space.

  In step S1505, the position and orientation D of the end effector are calculated from the position and orientation B of the flange portion in the vision space based on the end effector settings. Here, the position / posture D of the end effector is obtained from the position / posture B of the flange that is the tip of the robot in the vision space and the position / posture of the end effector with respect to the flange set in the end effector setting.

  In step S1506, the CAD display of the point group measured in three dimensions and the position and orientation D of the end effector are displayed in a superimposed manner. Here, the end effector model EEM, which is the three-dimensional CAD data of the end effector, is displayed at the position of the position and orientation D of the end effector obtained in step S1505, and the points obtained by three-dimensional measurement of the actual end effector The group PC is displayed in a superimposed manner. 151 and 152 show examples of deviation correction screens that are displayed in such a superimposed manner.

In step S1507, it is determined whether there is a non-negligible shift between the point cloud and the CAD display. If there is a deviation that cannot be ignored, the process proceeds to step S1508, the end effector setting is changed so as to correct the deviation, and the process returns to step S1505 to repeat the above-described processing. This end effector setting includes the position and posture of the end effector with respect to the flange portion. On the other hand, if there is no deviation, the process is terminated.
(Deviation correction part)

  As an example of a deviation correction unit that performs such deviation correction, examples of a deviation correction screen are shown in FIGS. 151 and c. In these figures, FIG. 151 shows a case where there is a shift that cannot be ignored. In the deviation correction screen 840 shown in this figure, the point cloud PC is superimposed and displayed on the end effector model EEM in the image display field 141. As shown in this figure, it can be confirmed that the point group PC, which is indicated by a white dot and actually measured in three dimensions with the end effector, is shifted in the diagonally lower left direction from the end effector model EEM.

The operation field 142 is provided with an end effector setting field 841 that defines the position (X, Y, Z) and posture (R _X , R _Y , R _Z ) of the end effector with respect to the flange portion. Here, the coordinate axis of the longer XYZ shown on the right in figure shows the origin O _F of the flange portion FLS is robot tip. The coordinate axes of the shorter XYZ shown in the left side, represents with respect to the robot distal (flange portion), the origin O _E of the three-dimensional CAD data of the end effector model EEM. The user adjusts the position and orientation of the end effector model EEM while confirming the end effector model EEM and the point cloud PC displayed in the image display field 141. For example, the end effector model EEM is designated by numerically specifying the position and orientation of the end effector model EEM in the operation field 142 or by dragging and moving the end effector model EEM on the image display field 141. Is adjusted to overlap with the point cloud PC.

  FIG. 152 shows an example in which the end effector model EEM is matched with the point group PC in this way. In this example, in the end effector setting field 841, the position in the Z direction is set to 90 mm, and the end effector model EEM is offset in the Z direction to match. When the “OK” button 842 is pressed after the position and orientation of the end effector model are adjusted, the end effector settings are updated. Thereby, the position and orientation on the vision side can be adjusted according to the actual attachment state of the end effector.

  Furthermore, the above-described manual correction of deviation can be performed in combination with automatic correction. Such an example will be described based on the deviation correction screen 850 of FIG. In the deviation correction screen 850 shown in this figure, an end effector setting field 841 for manually adjusting the position and orientation of the end effector model EEM is provided in the operation field 142, and an “automatic correction” button 843 is provided in the lower stage. . When this “automatic correction” button 843 is pressed, a three-dimensional search is executed internally, and the position and orientation of the end effector model EEM are automatically corrected to coincide with the point group PC. For example, while confirming the state of the end effector model EEM and the point cloud PC superimposed and displayed in the image display column 141, automatic deviation correction can be executed, or the automatic correction result can be finely adjusted manually. In particular, when the end effector is made of a highly glossy metal and the point cloud PC cannot be accurately detected by reflected light, etc., and automatic correction cannot be performed well, for example, when a three-dimensional search fails, manual Can be accommodated by adjustment. In this way, the user can manually correct the deviation between the robot space and the vision space.

  Conventionally, when a mounting member such as a connector is actually attached to the mounting part of the flange and end effector, for example, you have forgotten to set the offset on the vision side. There has been a problem that the position and posture are shifted. In such a case, if there is no function for confirming the deviation or the function for correcting the deviation, there is a case where the robot cannot be gripped well or a problem such as a collision occurs when the robot is gripped in actual operation. There are various factors that cause such errors, and in the past, investigating the cause and debugging have been extremely time consuming and labor intensive. On the other hand, according to the present embodiment, such a shift can be corrected manually or automatically, and flexible adjustment according to the actual attachment state of the end effector can be easily performed.

In the example of FIG. 59, the position of the end effector model EEM is the center position of the three-dimensional CAD data. However, the position is not limited to this example, and other positions can be defined. For example, in FIG. 59, the center of the rectangular parallelepiped circumscribing the three-dimensional CAD data is used, but the end of the circumscribed rectangular parallelepiped or the origin of the input three-dimensional CAD data may be used as a reference.
(Procedure for registering actual measurement data in the search model)

  The procedure for registering three-dimensional CAD data as a search model has been described above. However, in the present invention, as described above, the search model is not limited to the three-dimensional CAD data, and for example, actual measurement data obtained by actually imaging the workpiece by the sensor unit can be registered as the search model. Here, the procedure for registering such measured data as a search model in step S2701 of FIG. 27 will be described based on the flowchart of FIG.

  First, in step S6201, the workpiece is placed on a plane with the surface to be registered of facing upward, and the sensor unit performs three-dimensional measurement.

  In step S6202, the obtained actual measurement data is registered as a search model.

Finally, in step S6203, it is determined whether or not the search models necessary for the three-dimensional search have been registered. Ends the process. Details of these procedures will be described later with reference to FIGS.
(Procedure 1 during actual operation)

  In the state where the necessary setting work is completed as described above, the picking operation is performed on the work group actually stacked. Here, in the state where the search model is registered in the procedure of FIG. 28, a procedure for determining whether or not a workpiece can be gripped during actual operation, that is, whether or not there is a gripping solution for each detected workpiece, This will be described based on the flowchart of FIG. Here, the calculation unit 10 in FIG. 6 determines whether there is a gripping solution.

  First, in step S6301, three-dimensional measurement is started for a bulk workpiece. Here, a group of workpieces stacked in bulk by the sensor unit is imaged and three-dimensional measurement is performed to obtain a three-dimensional shape having height information.

  Next, in step S6302, a three-dimensional search is performed on the obtained three-dimensional shape of the workpiece group using a workpiece model to detect the position and orientation of each workpiece.

  Next, in step S6303, the position and posture where the end effector is to be arranged are calculated based on the position of the detected workpiece and the gripping posture of the workpiece registered at the time of setting.

  Next, in step S6304, whether or not the end effector interferes with surrounding objects at the calculated position is determined using the end effector model.

  In step S6305, it is determined whether or not the end effector interferes. If there is no interference, the processing is terminated with a gripping solution for the workpiece.

  On the other hand, if it is determined that there is interference, the process advances to step S6306 to determine whether there is another gripping position registered for the work. If another gripping position is registered, the process returns to step S6303 and the above processing is repeated for this gripping position.

  On the other hand, if another gripping position is not registered, the process advances to step S6307 to determine whether there is another detected workpiece. If there is another work, the process returns to step S6303 and the above process is repeated with the work replaced. If there is no other workpiece, the process ends with no gripping solution.

  In this way, the calculation unit 10 in FIG. 6 determines whether there is a gripping solution that can grip the workpiece. If a gripping solution is obtained, an instruction is sent to the robot controller 6 to pick the workpiece at the gripping position determined by the end effector. Accordingly, the robot controller 6 controls the end effector so as to pick the instructed work.

In the above procedure, when a gripping solution is obtained for any workpiece, the processing for examining the gripping position is terminated at that point, and control is performed so that the workpiece is gripped at the gripping position corresponding to the obtained gripping solution. ing. However, the present invention is not limited to this method, and for example, it may be configured to determine which gripping position candidate is selected after obtaining all gripping positions that can be gripped as gripping position candidates. For example, the evaluation index calculation unit 8q calculates a score as an evaluation index for each gripping position candidate, and the one having the highest score is selected as the gripping position. In addition, paying attention to the height information of the work, select the one with the high height where the work is placed, in other words, the one located in the upper part of the group of works stacked in bulk. You can also. Preferably, the calculation unit 10 selects from a plurality of gripping solutions in consideration of both the score and the height information. In this way, more appropriate picking can be performed.
(Procedure 2 during actual operation)

  In the above, the procedure at the time of actual operation was explained with the search model registered in the procedure of FIG. When registering the search model, as shown in FIG. 48, the search model can be registered in a state in which the posture of the search model is limited. Here, the procedure at the time of actual operation for performing the picking operation on the work group actually stacked in a state where the search model is registered by the procedure of FIG. 48 will be described based on the flowchart of FIG. First, in step S6401, three-dimensional measurement is started for a bulk workpiece. Here, a group of workpieces stacked in bulk by the sensor unit is imaged and three-dimensional measurement is performed to obtain a three-dimensional shape having height information.

  Next, in step S6402, a three-dimensional search is performed on the obtained three-dimensional shape of the workpiece group using a workpiece model to detect the position and orientation of each workpiece.

  In step S6403, out-of-range postures are excluded from the tilt angle and rotation angle range settings.

  Next, in step S6404, the position and posture where the end effector should be placed are calculated based on the position of the detected workpiece and the gripping posture of the workpiece registered at the time of setting.

  Next, in step S6405, whether or not the end effector interferes with surrounding objects at the calculated position is determined using the end effector model.

  In step S 6406, it is determined whether or not the end effector interferes. If there is no interference, the processing is terminated with the gripping solution being present for the workpiece.

  On the other hand, if it is determined that there is interference, the process proceeds to step S6407, where it is determined whether there is another gripping position registered for the workpiece. If another gripping position is registered, the process returns to step S6404 and the above processing is repeated for this gripping position.

On the other hand, if another gripping position is not registered, the process advances to step S6408 to determine whether there is another detected workpiece. If there is another work, the process returns to step S6404 and the above process is repeated with the work replaced. If there is no other workpiece, the process ends with no gripping solution.
(Interference judgment)

  Here, the interference determination method using the end effector model in step S6304 of FIG. 63 and step S6405 of FIG. 64 will be described. When gripping a workpiece with an end effector, if the end effector interferes with surrounding obstacles such as other workpieces or storage containers, gripping cannot be performed correctly. Accordingly, the position and orientation of the end effector when the end effector model grips the workpiece model with gripping position candidates in advance are calculated by the interference determination unit 8m in FIG. 31, thereby performing interference determination with surrounding members. At this time, as surrounding members, a group of workpieces and a three-dimensional point group of storage containers obtained by actually measuring with the sensor unit 2 are used. It is also possible to register the position of the storage container or the like in advance and perform interference determination with the end effector model. On the other hand, the three-dimensional CAD data of the end effector is generally composed of polygon data. For example, STL data frequently used as three-dimensional CAD data is expressed by a collection of minute triangles called polygons.

In order to perform interference determination between such polygon data and 3D point cloud data, conventionally, it is examined whether each 3D point constituting the 3D point cloud data is inside or outside the end effector model, It was determined that there was interference when inside and no interference when outside. However, in this method, it is necessary to perform calculation and comparison for each point, and the amount of calculation becomes enormous as the data becomes large.
(Interference judgment procedure using a cross-sectional model)

  Therefore, in each embodiment of the present invention, a cross-section model is created from polygon data of the end effector model, each point of the three-dimensional point cloud data is projected onto the cross-section model, and the inside / outside is determined to determine interference. ing. Such interference determination is performed by the interference determination unit 8m in FIG. Here, the procedure for performing the interference determination will be described based on the flowchart of FIG.

  First, in step S6501, the end effector polygon data is read. In step S6502, a cross-sectional model is created from polygon data of the end effector. The cross-section model is generated by the cross-section model generation unit 8s shown in FIG. Here, a method by which the cross-section model generation unit 8s creates a cross-section model will be described with reference to FIGS. 66 to 67E. First, the basic axis BSL is set for the polygon data of the end effector model EEM shown in FIG. Preferably, the basic axis BSL is set along the longitudinal direction of the end effector model EEM. Further, the end effector model is cut along an orthogonal plane orthogonal to the basic axis BSL to create a plurality of cross sections. Here, the position for creating the cross section is, for example, the vertex position of the polygon data. Alternatively, a plurality of cross sections may be created along a basic axis at a certain distance, and the obtained cross sections may be arranged according to the shape. For example, cross sections having the same shape are excluded. In creating the cross section, it is preferable to set the basic axis BSL so that the total number of necessary cross sections is reduced. For example, the cross section model generation unit may be configured to automatically calculate the direction of the basic axis BSL so that the number of cross sections is reduced. In addition, the designation of the cross-sectional position at which a cross-section is acquired on the set basic axis BSL can be designated so as to automatically extract a cross-section having a large change, for example.

  In this way, a cross-sectional model having the cross-sectional shape along the basic axis BSL of the end effector model EEM and the cross-sectional position on the basic axis BSL corresponding to each cross-section can be created. For example, in the example of FIG. 66, the cross sections SS1 to SS5 having the shapes shown in FIGS. 67A to 67E are obtained at five positions of the cross sectional positions SP1 to SP5 along the basic axis BSL. Using the cross-sectional model obtained in this manner, it is determined whether or not the end effector model interferes with the three-dimensional point group.

  Specifically, in step S6503, among the three-dimensional point group, a three-dimensional point to be subjected to interference determination is selected, and the position of the point in the basic axis BSL direction is selected from a plurality of cross-sections of the cross-sectional model. Select which cross section to determine interference. For this selection, the cross-sectional position set for each cross-section is used. For example, let us consider a case where the interference with the end effector model EEM is determined for the three-dimensional point TDP shown in FIG. 68A among the three-dimensional points. The three-dimensional point TDP is located between the cross-sectional positions SP3 and SP4 in the direction along the basic axis BSL. Therefore, the interference with the three-dimensional point TDP is determined with the cross section SS3 indicating the cross-sectional shape between them.

  Specifically, in step S6504, a projection point PP3 projected from the three-dimensional point TDP onto the orthogonal plane including the cross section SS3 is calculated. In step S6505, interference determination is performed. Here, if the position of the projection point PP3 of the three-dimensional point is outside the cross section SS3 as shown in FIG. 68B, it is determined that there is no interference. On the other hand, if it is inside the cross section SS3 as shown in FIG. 68C, it is determined that there is interference. Finally, in step S6506, it is determined whether or not there is another three-dimensional point. If an unprocessed three-dimensional point remains, the process returns to step S6503 and the above processing is repeated. When the interference determination is completed at all three-dimensional points, the process is terminated.

  In this way, it is possible to determine the interference between the measured three-dimensional point group and the end effector model. Further, the interference determination with the three-dimensional point cloud data has been described above. However, the present invention is not limited to the target of the interference determination, and the interference determination is performed not only for the three-dimensional point but also for other objects such as lines and surfaces in the same procedure. Can be implemented.

Furthermore, in the above example, the CAD data of the end effector has been described as an example of polygon data. However, as long as the cross-sectional shape of the end effector can be calculated, not only polygon data but also other CAD data formats are similarly used for interference determination. Can be done. Furthermore, in the above example, the case where the cross-sectional shape is expressed by a two-dimensional plan view has been described. You may have it.
(Interference judgment with an additional model with additional areas added)

  Further, in the interference determination of the end effector model, the interference determination can be performed using an additional model obtained by adding an additional region expressed by a three-dimensional basic figure to the end effector model. The procedure of interference determination using such an additional model will be described based on the flowchart of FIG. Here, an example will be described in which interference determination is performed on an additional model obtained by adding an additional region obtained by combining a rectangular parallelepiped and a cylinder as a basic figure to a three-dimensional CAD model.

  First, in step S6901, interference determination is performed for a rectangular parallelepiped region among the basic figures constituting the additional region. Depending on this result, if it is determined in step S6902 that there is interference, the interference determination process is stopped, and it is output that there is interference, and the process ends.

  On the other hand, if it is determined that there is no interference, the process advances to step S6903 to perform interference determination with a cylinder that is another graphic among the basic graphics forming the additional region. As a result, if it is determined in step S6904 that there is interference, the interference determination process is stopped, the presence of interference is output, and the process ends.

  On the other hand, if it is determined that there is no interference, the process proceeds to step S6905, where interference with the three-dimensional CAD data is determined. As a result, if it is determined in step S6906 that there is interference, the interference determination processing is stopped, the presence of interference is output, and the processing ends.

  On the other hand, if it is determined that there is no interference, the process is terminated as no interference.

As described above, the interference determination is sequentially performed for each basic figure or for each three-dimensional CAD data unit, and when it is determined that there is interference, the interference determination process is stopped at that time and an interference determination is output. In addition, although the above demonstrated the example in which the additional area | region was comprised with two basic figures, a rectangular parallelepiped and a cylinder, even if a basic figure increases, a procedure is the same, performs interference judgment for every basic figure one by one, If it is determined that there is interference, the process is stopped.
(Embodiment 7)
(Holding availability verification function)

The procedure for determining the presence / absence of a gripping solution as a result of interference determination during actual operation has been described above. However, the present invention not only determines the presence / absence of a gripping solution, but also grasps whether or not the gripping solution is obtained for a gripping position candidate for which a gripping solution is not obtained. May be provided. For example, a list of gripping solution candidates is displayed, and a gripping position determined as having a gripping solution as a result of interference determination is displayed as OK, and a gripping position candidate determined as having no gripping solution is displayed as NG. In this state, by selecting the gripping position determined to be NG and displaying the reason why it is determined that there is no gripping solution, the user can refer to this information to determine what gripping position. Thus, it becomes possible to correct the gripping position and add a new gripping position. Such an example is shown in the block diagram of FIG. 70 as a robot system 7000 according to the seventh embodiment. A robot system 7000 shown in this figure includes an image processing device 700, a display unit 3B, an operation unit 4, a sensor unit 2, a robot controller 6, and a robot RBT. In addition, about the same member as FIG. 6, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted suitably.
(Display unit 3B)

  The display unit 3B includes an image display area 3b and a gripping solution candidate display area 3c. The gripping solution candidate display area 3c includes a work gripping possibility display area 3d and a work gripping inability factor display area 3e.

  The image display area 3b is a member for displaying an end effector model composed of three-dimensional CAD data that virtually represents the three-dimensional shape of the end effector in a three-dimensional shape in a virtual three-dimensional space. is there.

  The gripping solution candidate display area 3c is a member for enumerating and displaying all set gripping positions for any one of the search results of one or more workpieces searched by the three-dimensional search unit 8k. .

  The workpiece gripping availability display area 3d is a member for displaying a judgment result of whether gripping is possible or not at a gripping position designated for each workpiece by the three-dimensional pick determination unit 8l.

The work grip impossibility factor display area 3e displays the cause of the gripping failure for the gripping position determined by the three-dimensional pick determination unit 8l that the gripping position specified for each work is not possible. It is a member for making it.
(Image processing apparatus 700)

  The image processing apparatus 700 includes an input image acquisition unit 2c, a calculation unit 10, a storage unit 9, an input / output interface 4b, a display interface 3f, and a robot interface 6b. The calculation unit 10 includes an end effector model registration unit 8u, a work model registration unit 8t, a gripping position specification unit 8d, a search model registration unit 8g, a three-dimensional search unit 8k, a three-dimensional pick determination unit 8l, An angle setting unit 8n.

  The grip position specifying unit 8d includes a workpiece side grip location specifying unit 8d1 and an end effector side grip setting unit 8d2. The workpiece-side gripping location specifying unit 8d1 displays a workpiece model composed of three-dimensional CAD data that virtually represents the three-dimensional shape of the workpiece in a state where the end effector model is displayed in the image display area 3b. This is a member for designating a gripping position when gripping with. The end effector side gripping setting unit 8d2 is a member for designating a gripping position for gripping a workpiece with respect to the end effector model displayed in the image display area 3b.

  The search model registration unit 8g performs a three-dimensional search for specifying the posture and position of each workpiece with respect to a plurality of workpiece groups included in the input image as a second workpiece model that virtually represents the three-dimensional shape of the workpiece. It is a member for registering as a search model for performing. The second work model registered as the search model is preferably the same as the work model for which the grip position is designated by the work-side grip location specifying unit 8d1. As a result, the user can save the setting work by sharing the search model for performing the three-dimensional search with the work model for designating the gripping position. In actual operation, by matching the work model that searches for work that can be gripped with the work model that performs gripping judgment, the searched work model can be gripped at the gripping position set in this work model. Therefore, it is possible to perform processing efficiently.

  The inclination angle setting unit 8n is a member for setting an allowable inclination angle range for the posture of the workpiece.

  The three-dimensional pick determination unit 8l uses the end effector at the gripping position designated for the workpiece by the workpiece-side gripping location designation unit 8d1 with respect to the search result of each workpiece searched by the three-dimensional search unit 8k. It is a member for determining whether or not it can be gripped. The three-dimensional pick determination unit 8l includes an interference determination unit 8m and an angle determination unit 8o.

  The interference determination unit 8m exists around the workpiece with respect to the gripping position designated for the workpiece by the workpiece-side gripping location designation unit 8d1 with respect to the search result of each workpiece searched by the three-dimensional search unit 8k. It is a member for determining the presence or absence of interference with the object to be performed. In addition, the three-dimensional pick determination unit 8l determines that the workpiece that is determined to have interference by the interference determination unit 8m cannot be gripped. As a result, the cause in the case where the workpiece cannot be gripped is indicated, which can contribute to resetting the gripping position, for example, by making it easier for the user to consider what gripping position should be added.

  The angle determination unit 8o is a member for determining whether the posture of the search result of the workpiece searched by the three-dimensional search unit 8k is within the tilt angle range set by the tilt angle setting unit 8n.

When the angle determination unit 8o determines that the posture of the workpiece search result searched by the three-dimensional search unit 8k is not within the inclination angle range set by the inclination angle setting unit 8n by the angle determination unit 8o. In addition, it is configured to determine that the workpiece cannot be gripped. As a result, when the workpiece posture is too steep and cannot be gripped, or accuracy of 3D measurement cannot be expected, by eliminating this, selection of the gripping position and misjudgment of judgment can be avoided and reliability can be improved. Can do.
(Handling solution candidate list display function)

  Next, the procedure for displaying the reason for the gripping solution NG will be described based on the flowchart of FIG. The procedure at the time of setting and the procedure at the time of actual operation are basically the same as those described above with reference to FIGS. 27, 28, 63, etc., and detailed description thereof will be omitted as appropriate.

  First, in step S7101, three-dimensional measurement is performed on a target work group. Here, actual measurement data having shape information is acquired using a sensor unit for a group of workpieces stacked in bulk, and this is used as an input image.

  In step S7102, a three-dimensional search is performed on the input image to detect the position and orientation of each workpiece in the input image.

  In step S7103, one workpiece to be verified is selected from the detected workpieces.

  In step S7104, the position and orientation of the end effector model when the end effector is intended to be gripped are calculated from the detected position of the selected workpiece and the gripping posture registered in advance for the workpiece.

  Next, in step S7105, it is determined whether the calculated inclination angle in the position and orientation of the end effector model is outside the set range. If it is determined that it is out of the setting range, the process proceeds to step S7106, where it is determined as gripping NG, and the factor of NG is set to “inclination angle”, and then the process jumps to step S7111.

  On the other hand, if it is determined in step S7105 that the tilt angle of the end effector model is within the set range, the process proceeds to step S7107, and interference between the end effector model and surrounding objects is determined at the calculated position. Here, the surrounding objects are storage containers, other works, etc. existing around the end effector model. When surrounding objects are modeled in advance using 3D CAD data, etc., and the end effector model is moved to the position and orientation where the workpiece model is gripped, it is calculated whether these and the end effector model interfere with each other. judge.

  If it is determined in step S7108 that the end effector model is interfering with a surrounding object as a result of the interference determination, the gripping NG determination is made in step S7109, and the NG factor is set to “point cloud interference”. Above, the process jumps to step S7111.

  On the other hand, if it is determined in step S7108 that the end effector model does not interfere with surrounding objects as a result of the interference determination, the process proceeds to step S7111 after the gripping OK determination is made in step S7110.

  In step S7111, it is determined whether another gripping posture is set for the selected workpiece. If there is another gripping posture, the process returns to step S7104 to repeat the above processing. On the other hand, if it is determined that there is no gripping posture, in step S7112, the gripping OK / gripping NG determination result and the factor in the case of gripping NG are displayed in the workpiece gripping disability factor display area 3e for all gripping solution candidates. Display.

In this way, it is possible to verify whether or not a given workpiece group can be gripped, and to list the factors when gripping is not possible.
(Gripping simulation)

Next, details of a gripping simulation for performing specific gripping determination will be described. Here, an example will be described in which it is determined whether gripping is possible or not when performing bulk picking using the workpiece WK10 as shown in FIG. For this work WK10, a basic direction image corresponding to a hexahedral view is generated by the basic direction image generation unit 8e ′, and each basic direction image is registered as a search model by the search model registration unit 8g. Examples of basic direction images are shown in FIGS. 73A to 73E. In these drawings, FIG. 73A is a height image A when the workpiece of FIG. 72 is viewed from the positive direction side of the Z axis, and FIG. 73B is a height image B when viewed from the negative direction side of the Z axis. Is a height image C viewed from the positive direction side of the X axis, FIG. 73D is a height image D viewed from the negative direction side of the X axis, and FIG. 73E is a height viewed from the positive direction side of the Y axis. FIG. 73F shows the height image F viewed from the negative direction side of the Y axis.
(Work selection screen 210)

In order to determine whether or not the workpiece can be gripped by the end effector, a three-dimensional search is performed on an input image obtained by capturing a group of stacked workpieces, and the target workpiece is selected when the workpiece is detected. (Step S7103 of FIG. 71). The workpiece is selected from, for example, a workpiece selection screen 210 shown in FIG. The work selection screen 210 shown in this figure includes an image display field 141 and an operation field 142. In the image display column 141, an input image which is actual measurement data obtained by imaging a group of workpieces stacked is displayed. In addition, search results are displayed as point clouds over the input image at the position where the workpiece is detected. The viewpoint can be changed by dragging the screen of the image display column 141.
(Label number, model number)

The operation column 142 includes a target workpiece selection column 211 for selecting a target workpiece, a detection search model display column 212 indicating a search model used for three-dimensional search of the target workpiece, and a selected workpiece. A “check grip” button 213 for displaying a list of all gripping position candidates is provided. In the example of FIG. 74, as a result of performing a three-dimensional search on the input image, there are 18 workpieces, and the third workpiece is selected among them. In order to distinguish the search results detected here, identification information is set in the search results. Here, a serial number label number is set as the identification information. In the target work selection column 211 of FIG. 74, “3” is displayed as the label number of the target work being selected. In the image display column 141, the search result of the selected target work is displayed. Specifically, as the state of the search result, the feature points of the corresponding search model are displayed in a form superimposed on the point group of the input image. When the label number of the target work is changed in this state, the work selected in the image display field 141 is also changed accordingly. Further, in the detection search model display column 212, a search model E (a height image in FIG. 73E) is displayed as a search model used for the three-dimensional search of the target workpiece. Note that individual identification information is also assigned to the search model, and is referred to as a model number here. In FIG. 74, “E” of the work model with the serial number label number 3 is displayed as the model number in the detection search model display field 212.
(Gripping solution candidate display screen 220)

  Furthermore, the gripping position set for each workpiece, that is, gripping solution candidates can be displayed in a list in the gripping solution candidate display area 3c. In the example of the workpiece selection screen 210 in FIG. 74, when the “check grip” button 213 is pressed, a gripping solution candidate display screen 220 in FIG. 75 is displayed on the display unit. In the gripping solution candidate display screen 220, which is one form of the gripping solution candidate display area 3c, all gripping positions set for the target work selected in the target work selection field 211 of FIG. 74 are gripping solutions as gripping solution candidates. They are listed in the candidate display column 221. In the gripping solution candidate display column 221, for each gripping solution, a workpiece gripping position display column 223 indicating the label number of the gripping position, a workpiece gripping availability display column 224 indicating a result of whether or not a workpiece is gripped, and a case where gripping is not possible are displayed. A work grip impossibility factor display field 225 indicating the reason is provided. In the example of FIG. 75, five gripping candidates are listed for the target workpieces of the third search result selected in the target workpiece selection field 211, and NG along with the gripping OK / NG determination results. The cause of the error is displayed. In addition, the workpiece selected in the image display column 141 and the end effector model that grips the workpiece are displayed in a posture for gripping the gripping position corresponding to the gripping position of the workpiece selected in the gripping solution candidate display column 221. The When the selection of the gripping position is changed in the gripping solution candidate display field 221, the work in the image display field 141 is changed accordingly, and the posture of the end effector model that grips the work is also updated.

  The example of FIG. 75 shows a state in which the workpiece of FIG. 73E is gripped by the gripping position candidate of label number 3. In this case, since the determination result is the gripping OK, nothing is displayed in the work gripping inability factor display field 225. On the other hand, when the gripping position candidate with the label number 4 is selected in the work gripping impossibility factor display field 225, the display in the image display field 141 is also changed as shown in FIG. Since the gripping position candidate of label number 4 is a determination result of gripping NG, the reason why it is determined that gripping is not possible is displayed in the work gripping inability factor display field 225. Further, in the image display column 141, an end effector model corresponding to this is displayed, and it is possible to visually confirm what state the gripping posture is actually in. Here, “point cloud interference” is displayed, and it can be confirmed that the end effector interferes with the plane of the storage container. As a result, the user can consider adding a gripping position at a position or posture that does not interfere with the workpiece in such a posture.

  Similarly, when a gripping position candidate with label number 2 is selected in the work gripping disability factor display field 225, a gripping solution candidate display screen 220 shown in FIG. 77 is displayed, and the gripping position candidate with label number 2 becomes the determination result of gripping NG. As the reason, “Inclination angle” is displayed in the work grip impossibility factor display field 225. Further, in the image display column 141, an end effector model corresponding to this is displayed, and the end effector is in a state of attempting to grip at a steep inclination angle, and it can be seen that it is excluded from the interference determination target in the first place.

  Further, the display mode of the end effector displayed in the image display field 141 may be changed according to the determination result. In the example of FIG. 75, the end effector model is displayed in white in the case of gripping OK, and the end effector model is displayed in red in the case of gripping NG in the examples of FIGS. As a result, the user can easily distinguish visually the determination result of gripping. Further, when adding the grip position, the display of the image display column 141 may be updated so that the display mode is changed when the posture of the end effector model is adjusted and no interference occurs. . By configuring in this way, the user can change the posture of the end effector model from red to white, for example, and the advantage that the posture adjustment work can be easily understood is obtained.

  By providing the gripping possibility verification function in this way, how to set gripping when it is determined that the workpiece to be gripped is not grippable, for example, adding a new gripping position or existing gripping This is a guideline when considering work such as changing the position setting. For example, in the examples shown in FIGS. 78 and 79, although there are two gripping position candidates for the workpiece of the search model C (FIG. 73C), both cannot be gripped because of point cloud interference. Yes. In either case, as indicated by a broken-line circle in the image display field 141, it can be seen that the cause is that the tip of the nail of the end effector model interferes with another workpiece. From this, it can be seen that if a new gripping position is added so as to grip a portion on the opposite side of the current gripping position, an OK gripping solution can be obtained. Therefore, if the gripping position specifying unit 8d is set to add a new gripping position (griping posture C-002) also on the opposite side of the current gripping position, a gripping OK determination result can be obtained as shown in FIG. It becomes like this. In this way, when the user performs adjustment work of the gripping position and posture, an environment in which countermeasures can be easily considered is given, and a robot system that is easy to set is realized.

  It is also possible to change the threshold value of the tilt angle of the end effector by setting by the user. For example, when using a box with a deep bottom as a work storage container, if the inclination of the end effector becomes large, the arm part of the robot will easily collide with the wall of the storage container in addition to the end effector. To do. On the contrary, when a box with a shallow bottom is used as the storage container, a case where the arm part of the robot does not collide with the box is unlikely to occur unless even the end effector interferes, so that the angle range can be set wide. As described above, by adjusting the setting of the angle range, it is possible to flexibly adjust the determination of whether or not the gripping can be performed according to the actual situation.

As described above, all gripping solutions for the selected workpiece are displayed in a list. For those that are NG, the NG factor is also displayed, so a workpiece that is not a candidate for a gripping solution until now is likely to be taken. If there is a problem, it becomes easy to identify the cause of why the problem is not grasped. Further, since the cause can be identified, it becomes easy to understand what kind of gripping posture should be newly added.
(Embodiment 8)

  In the above example, an example in which search results are individually acquired for each search model in the three-dimensional search has been described. That is, a three-dimensional search is performed with a plurality of basic direction images showing different surfaces of the same workpiece, and the obtained results are recognized as separate workpieces. In other words, different surfaces of the same workpiece may be detected as different workpieces as a result of individual search. On the other hand, in the conventional three-dimensional search, since a search is performed with a three-dimensional shape representing one work model, such a situation does not occur, and different surfaces of the same work are also detected as one work. However, in a work having many surfaces, the search becomes complicated and the possibility of erroneous detection increases. On the other hand, since the method according to the above-described embodiment performs a simple search, the search process can be simplified, which is advantageous in terms of reduction in load and speed. On the other hand, as a result of searching for each surface, the search result obtained as described above is regarded as a separate work, and there is a problem that different surfaces are detected individually even for the same work.

Therefore, by integrating the obtained search results for each work and estimating the search result of another surface from the search result of one surface, the surface that could not be detected by the 3D search and the detection accuracy is low The surface can also be used as a gripping solution candidate. FIG. 81 is a block diagram showing such an example as a robot system according to the eighth embodiment. A robot system 8000 shown in this figure includes an image processing apparatus 800, a display unit 3B, an operation unit 4, a sensor unit 2, and a robot RBT. In addition, about the same member as FIG. 6, FIG. 70 etc., the same number is attached | subjected and detailed description is abbreviate | omitted suitably.
(Image processing apparatus 800)

  The image processing apparatus 800 includes an input image acquisition unit 2c, a calculation unit 10, a storage unit 9, an input / output interface 4b, a display interface 3f, and a robot interface 6b. The calculation unit 10 includes a basic direction image generation unit 8e ′, a gripping position specifying unit 8d, a search model registration unit 8g, a three-dimensional search unit 8k, an image estimation unit 8z, a search result integration unit 8p, a three-dimensional A pick determination unit 8l and an inclination angle setting unit 8n are provided.

  The basic direction image generation unit 8e ′ generates, as a basic direction image, a plurality of height images viewed from any one of the three axial directions orthogonal to each other in the virtual three-dimensional space with respect to the work model. It is a member.

  The gripping position specifying unit 8d is for specifying a plurality of gripping positions where the end effector grips the work model indicated by the basic direction image with respect to one of the basic direction images generated by the basic direction image generating unit 8e ′. It is a member. The grip position specifying unit 8d includes a workpiece side grip location specifying unit 8d1 and an end effector side grip setting unit 8d2.

  The workpiece-side gripping location specifying unit 8d1 displays a workpiece model composed of three-dimensional CAD data that virtually represents the three-dimensional shape of the workpiece in a state where the end effector model is displayed in the image display area 3b. This is a member for designating a gripping position when gripping with.

  The end effector side gripping setting unit 8d2 is a member for designating a gripping position for gripping a workpiece with respect to the end effector model displayed in the image display area 3b.

  The search model registration unit 8g applies the plurality of basic direction images generated by the basic direction image generation unit 8e ′ to the plurality of work groups included in the input image acquired by the input image acquisition unit 2c. It is a member for registering each as a search model used when performing a three-dimensional search for specifying the posture and position. The search model registration unit 8g can also register the relative positions of a plurality of registered search models as relationship information (details will be described later).

  The image estimation unit 8z is searched by the three-dimensional search unit 8k and extracted for each search model from the input image. For each search result, each search model used for this search uses the original work model. This is the basic direction image seen from which direction, and based on the relative positional relationship with the basic direction image of other search models registered for this work model, This is a member for estimating the position and orientation of an unsearched basic direction image not included in the search result of the three-dimensional search unit as an estimated image. For example, when the posture of the workpiece search result searched by the three-dimensional search unit 8k by the angle determination unit 8o described later is within the tilt angle range set by the tilt angle setting unit 8n, the image estimation unit 8z The estimation image having a relative positional relationship with the search result can be estimated.

  The search result integration unit 8p is searched by the three-dimensional search unit 8k and extracted for each search model from the input image. For each search result, the search model used for the search uses the original work model. This is a member for integrating adjacent search results among a plurality of search results as an integration result related to a common work based on the relative positional relationship of the basic direction image viewed from which direction.

  The three-dimensional pick determining unit 8l performs the gripping position specifying unit 8d on the work model with respect to the integrated results integrated by the search result integrating unit 8p and the unintegrated search results searched by the three-dimensional search unit 8k. This is a member for determining whether or not the end effector can grip at the specified gripping position. The three-dimensional pick determination unit 8l includes an interference determination unit 8m and an angle determination unit 8o.

As a result, it is possible to more accurately estimate the position and orientation of the workpiece using the relationship information between the faces, instead of performing individual holding determinations on the search results that are three-dimensionally searched. In addition, it becomes possible to detect even a surface with a low accuracy of the search result, and it is possible to consider it as a holding position candidate including a surface that is difficult to search in a normal state, thereby increasing the possibility of obtaining a holding solution. It becomes possible.
(Procedure for registering a search model including relationship information between faces)

  For example, the procedure shown in the flowchart of FIG. 27 can be used as the procedure for setting the registration of the work model and end effector model, the registration of the gripping position, and the like for the above image processing apparatus. Here, in step S2701 of FIG. 27, the procedure for using the three-dimensional CAD data as the search model and registering the search model including the relationship information between the surfaces of the workpiece will be described based on the flowchart of FIG.

  First, in step S8201, a three-dimensional CAD data model of a work is read.

  In step S8202, the center of the circumscribed cuboid of the three-dimensional CAD data model is corrected to the origin of the three-dimensional CAD data.

  In step S8203, a height image viewed from each of the “up”, “down”, “left”, “right”, “front”, and “rear” directions is generated as a basic direction image. The basic direction image is performed by the basic direction image generation unit 8e 'of FIG. Here, when the height image is generated from the three-dimensional CAD data, the height image is generated so that the origin of the CAD becomes the center of the height image.

  In step S8204, a height image that looks the same is deleted from the generated height images.

  In step S8205, the relative positions of a plurality of registered search models are registered as relationship information. Here, the search model registration unit 8g stores the remaining height image and the relationship information between the front, back, front, back, left, and back surfaces.

Finally, in step S8205, a search model is registered using the generated height image. In this way, the search model including the relationship information between the work surfaces is registered.
(Relationship information of faces)

  Here, the relationship information indicating the relative positional relationship between the surfaces when the workpiece is viewed from a specific direction will be described. For example, in the case of the workpiece as shown in FIG. 7, the surfaces with different appearances are the four surfaces shown in FIGS. 9A to 9D. In these drawings, model A shown in FIG. 9A is a height image viewed from the positive direction of the X axis. However, it also matches the image viewed from the negative direction of the X axis. Further, the model B shown in FIG. 9B is a height image viewed from the positive direction of the Y axis, and matches the image viewed from the negative direction of the Y axis. On the other hand, the model C shown in FIG. 9C matches only the image viewed from the positive direction of the Z axis. The model D shown in FIG. 9D matches only the image viewed from the negative direction of the Z axis.

  Here, models A, B, C, and D, which are height images generated in advance, match the three-dimensional CAD data of the workpiece shown in FIG. Is the information representing the relationship information of the surface. With such surface relationship information, the orientation of the original three-dimensional CAD model can be obtained from the search result of each model. For example, the model B in FIG. 9B is adjacent to the left and right surfaces of the model A in FIG. 9A. Further, the model C in FIG. 9C is adjacent to the upper surface of the model A, and the model D in FIG. 9D is adjacent to the lower surface. As a result, it can be seen that, in the searched images obtained as a result of the three-dimensional search, when the detection results of the model A and the model B are adjacent to each other, the search for the same work is performed. Therefore, these search results can be integrated as one search result. Here, a set of integrated search results is referred to as an integrated result.

Also, the evaluation index can be updated by integrating the search results. In other words, even if the search result is not highly accurate, if a highly accurate search result is obtained in the integrated result, the evaluation index of the integrated result is increased so that a correct gripping solution can be obtained when selecting a gripping position. Can be prioritized.
(Unsearched basic direction image)

Furthermore, even if there is a surface that is not searched, such as when the three-dimensional search fails or when the search itself is not performed, it is possible to estimate using the integration result. That is, when a plurality of search results are integrated to obtain a work integration result, even if there are faces that are not detected by the three-dimensional search among the faces constituting this work, If the search result is obtained, the posture and position of the workpiece itself can be calculated. As a result, when a work model in which the gripping position is registered in advance with respect to the work or a search model for three-dimensional search is registered, information on a surface that has not been searched can also be estimated from the posture of the work. Therefore, a surface (referred to as an unsearched basic direction image) obtained by estimation that is not obtained as a result of such a search was also not actually searched by using it as a gripping position candidate. The gripping position of the surface can also be considered as a gripping solution candidate, and there is an advantage that an appropriate gripping solution can be easily obtained. For example, when the model B shown in FIG. 9B is obtained as a result of the three-dimensional search, it can be estimated that the surface of the model A shown in FIG. As a result, even if model A is not actually detected as a search result, the surface of model A can be estimated from the search result of model B, so that it is registered as a gripping solution candidate in the surface of model A. As a result, the number of gripping solution candidates is increased and picking is easily performed.
(Calculation of gripping solution using surface relationship information)

  Next, the effect of gripping solution calculation using the surface relationship information will be described using the workpiece WK10 shown in FIG. 73A to 73F are generated from the three-dimensional CAD data of the workpiece WK10 by the basic direction image generation unit 8e ', and these are registered as search models by the search model registration unit 8g. In these figures, FIG. 73A is a height image viewed from the positive direction of the Z axis, and this is model A. Similarly, FIG. 73B is a height image viewed from the negative direction of the Z axis. FIG. 73C is a height image viewed from the positive direction of the X axis, and is model C. Further, FIG. 73D is a height image viewed from the negative direction of the X axis and is model D, FIG. 73E is a height image viewed from the positive direction of the Y axis and is model E, and FIG. The model F is shown as a height image as seen.

  Here, for each of these basic direction images, as shown in FIGS. 83A to 83F, one gripping posture for gripping the work model WM10 with the end effector model EM10 is registered in the gripping position specifying unit 8d. Shall. Here, FIG. 83A shows a state in which the gripping posture A-000 is registered with respect to the basic direction image of FIG. 73A. FIG. 83B shows a state where a gripping posture B-000 is registered for the basic direction image of FIG. 73B. Further, FIG. 83C shows a state in which the gripping posture C-000 is registered with respect to the basic direction image in FIG. 73C, and FIG. 83D shows a state in which the gripping posture D-000 is registered with respect to the basic direction image in FIG. 83E is a state where the gripping posture E-000 is registered for the basic direction image of FIG. 73E, and FIG. 83F is a state where the gripping posture F-000 is registered for the basic direction image of FIG. 73F. Each state is shown. In this case, calculation of the gripping solution using the surface relationship information will be described with reference to FIGS. 84, 85, and 86.

  The workpiece selection screen 210 in FIG. 84 shows a state where 12 workpieces are detected as a result of performing a three-dimensional search on an input image obtained by imaging a group of workpieces stacked in bulk. In the image display field 141, an input image is displayed. The operation column 142 includes a target workpiece selection column 211 for selecting a target workpiece, a detection search model display column 212, and a “grip confirmation” button for displaying a list of gripping position candidates for the selected workpiece. 213 is provided. In the example of FIG. 84, the second workpiece is selected in the target workpiece selection column 211 among the twelve workpieces subjected to the three-dimensional search, and the selected second target workpiece is displayed in the image display column 141. Search results for are displayed. Specifically, as the state of the search result, the feature points of the corresponding search model are displayed in a form superimposed on the point group of the input image. Further, in the detected search model display column 212, search models C, F, and A are displayed as search models for detecting the second target workpiece. In this state, three types of search models C, F, and A (FIGS. 73C, 73F, and 73A) are integrated.

  Here, assuming that the relationship information on the surface of the workpiece is not used, the model F and the model A are in a state where only a part of the side surface is visible on the input image as shown in the image display column 141. As a result, it seems that it was difficult to detect. Even if it can be detected, the evaluation index is low, and it seems that the result has a low priority. On the other hand, by using the relationship information, not only the search model C but also the search model F and the search model A can be detected or estimated and used as gripping position candidates as shown in FIG. Become.

  When the “grip confirmation” button 213 is pressed on the work selection screen 210 in FIG. 84, a gripping solution candidate display screen 220 in FIG. 85 is displayed. In the gripping solution candidate display screen 220, the gripping position set for the second target work selected in the target work selection column 211 in FIG. 84 is listed in the gripping solution candidate display column 221 as gripping solution candidates. In the gripping solution candidate display field 221, whether or not the workpiece can be gripped at the gripping positions (FIGS. 83C, 83F, and 83A) set for the search models C, F, and A displayed in the detection search model display field 212, respectively. The determination result is displayed in the work gripping availability display field 224, and the reason for the inability to grip is displayed in the work gripping disability factor display field 225. First, for the search model C, it is shown that the determination result is gripping NG, and the reason is interference of point cloud data. Further, in the image display area, the end effector model EM10 that is to grip the workpiece in the posture corresponding to the gripping position C-000 is displayed in red, indicating that it cannot be gripped by interference. Here, if the relationship information of the surface is not used, only the search model C is three-dimensionally searched. As a result, the gripping solution candidate is only C-000 of this search model, and the gripping solution can be obtained as a result. It will not be.

On the other hand, by using the relationship information, as shown in FIGS. 85 and 86, the search models F and A are also targeted in addition to the search model C, and F-000 and A-000 are added as gripping position candidates. You can see that Among the added gripping position candidates, the gripping position candidate F-000 of the search model F has a gripping NG determination result, but the gripping position candidate A-000 of the search model A has a gripping OK result. The determination result is obtained, and the end effector is displayed in white as shown in FIG. 86, and the gripping solution is obtained safely without interfering with the point cloud data. As a result, even when the gripping solution cannot be obtained without using the relationship information, the gripping solution can be obtained by using the relationship information. In other words, without changing the setting of the 3D search, for example, adjusting the lighting or camera arrangement, changing the search algorithm, etc., while maintaining the same search conditions, in other words, without changing the accuracy of the 3D search The possibility of obtaining a gripping solution can be increased. In this way, by using the surface relationship information, it is possible to increase the number of gripping solution candidates and obtain a result that is easily picked.
(Procedure 3 for actual operation)

  Here, a procedure for performing a three-dimensional search and a gripping possibility determination in actual operation with the search model registered in the procedure of FIG. 82 will be described based on the flowchart of FIG.

  First, in step S8701, three-dimensional measurement is started for a bulk workpiece. Here, a group of workpieces stacked by the sensor unit 2 shown in FIG. 81 is imaged to perform three-dimensional measurement, and a three-dimensional shape having height information is acquired as an input image by the input image acquisition unit 2c.

In step S8702, a three-dimensional search is performed on the obtained three-dimensional shape of the workpiece group using a workpiece model, and the position and orientation of each workpiece are detected. The three-dimensional search is performed by the three-dimensional search unit 8k in FIG.
(Generation of integration results)

Further, in step S8703, results indicating the same work are integrated from the search result of the three-dimensional search and the relationship information of each surface. Here, the search result integration unit 8p uses the relationship information registered in the search model used for the search to associate search results obtained by imaging the same workpiece. In other words, based on the relative positional relationship of the basic direction image from which the common work model from which the search results are searched is viewed, the adjacent search results are searched for among the search results. The result is integrated by the result integration unit 8p. The integration results obtained in this way are commonly handled as indicating the same work.
(Generation of unsearched basic direction images)

  In step S8704, a non-detected surface is estimated from the three-dimensional search result and the relationship information of each surface. Here, based on the posture and position of the workpiece indicated by the integration result, the position and posture of the unsearched basic direction image not included in the result of the three-dimensional search for the workpiece are estimated. Then, the gripping position set by the gripping position specifying unit 8d with respect to the unsearched basic direction image is set as a gripping possibility determination target. As a result, the surface of the workpiece that was not originally searched as a result of the three-dimensional search can be estimated from the posture and position of the workpiece indicated by the integrated search result, and can be determined as a gripping possibility determination target. The possibility of obtaining a solution can be further increased.

  Next, in step S8705, the position and posture where the end effector should be arranged are calculated for the detected one workpiece based on the workpiece position and the gripping posture of the workpiece registered at the time of setting.

  Next, in step S8706, whether or not the end effector interferes with surrounding objects at the calculated position is determined using the end effector model.

  In step S8707, it is determined whether or not the end effector interferes. If there is no interference, the processing is terminated with a gripping solution for the workpiece.

  On the other hand, if it is determined that there is interference, the process advances to step S8708 to determine whether there is another gripping position registered for the workpiece. If another gripping position is registered, the process returns to step S8705 and the above processing is repeated for this gripping position.

  On the other hand, if another gripping position is not registered, the process advances to step S8709 to determine whether there is another detected workpiece. If there is another work, the process returns to step S8705, and the above process is repeated by replacing the work. If there is no other workpiece, the process ends with no gripping solution.

  In this way, the presence / absence of a gripping solution capable of gripping the workpiece is determined using the integration result and the unsearched basic direction image. When the gripping solution is obtained, the robot controller 6 is controlled so that the workpiece is picked at the gripping position determined by the end effector.

In this specification, an image is not limited to strictly continuous data, but is used to include a set of discrete data such as a set of point cloud data.
(Embodiment 9)
(Integration of evaluation indicators)

  Further, in addition to the above procedure, evaluation indexes that define the priority order can be integrated with the integration result. FIG. 88 shows such an example as a robot system 9000 according to the ninth embodiment. The image processing apparatus 900 shown in this figure includes an evaluation index calculation unit 8q and a grip priority order determination unit 8r. In addition, about the member common to the member shown in FIG. 81 etc., the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.

  The evaluation index calculation unit 8q is a member for calculating an evaluation index for each gripping position candidate using the gripping position determined to be grippable by the three-dimensional pick determination unit 8l as a gripping position candidate.

  The grip priority order determination unit 8r is a member for determining a priority order for gripping the workpiece with the end effector based on the evaluation index calculated by the evaluation index calculation unit 8q.

  This image processing apparatus calculates an evaluation index for each search result by the evaluation index calculation unit 8q. As the evaluation index, as described above, for example, a score ratio of feature points corresponding to errors within a certain distance included in the search result can be used.

  Here, the search result integration unit 8p gives the highest evaluation index among the evaluation indexes calculated by the evaluation index calculation unit 8q for each search result constituting the integration result to the integration result. In general, in the evaluation index of the three-dimensional search, search conditions and workpiece posture (for example, when the workpiece tilt angle is large and sufficient reflected light cannot be obtained, or when the reflected light is too strong due to a glossy workpiece), etc. Depends on. For this reason, the evaluation index of the search result may be calculated to be low, and may be excluded from the target of the three-dimensional search due to a low priority or the fact that reflected light cannot be obtained in the first place. These are lower positions as candidates for the gripping position, and will not be listed as candidates. However, in such a workpiece, it is simply not placed in an appropriate posture for 3D search. May be arranged in a suitable posture. Even in such a case, since it was conventionally excluded at the stage of 3D search or had a low priority, it was unlikely to be selected as a gripping solution and could not be fully utilized. . On the other hand, according to the present embodiment, even if the result of the three-dimensional search is a search result with a low evaluation index, a high evaluation value is obtained in other adjacent search results. By using, it becomes possible to use it as a gripping position candidate with high priority. As a result, even gripping position candidates that have not been used in the past can be used, and the possibility of obtaining a gripping solution can be increased. In other words, an appropriate workpiece can be gripped regardless of the accuracy of the three-dimensional search.

In the above method, the entire integrated result is evaluated with the highest evaluation index among the search results constituting the integrated result, or the evaluation index of each search result is changed from the original evaluation value to a higher evaluation value. The aspect of rewriting has been described. This is based on the reliability that the search results obtained with a high evaluation index are highly reliable. However, the present invention is not limited to this mode, and other modes, for example, the evaluation index of each search result constituting the integration result may be averaged and handled as the evaluation index of the integration result. In either case, search results that were originally low evaluation indexes are evaluated with a higher evaluation index than the original evaluation index so that they can be candidates for gripping positions, etc., using search results with other high evaluation indexes. This is the purpose of this embodiment.
(Image processing program GUI)

Here, an example of the GUI of the image processing program will be described based on FIGS. 89 to 149. FIG. 89 shows a function selection screen 340 for selecting various functions, and available functions are listed in a button form in the function list display field 341. When the “3D search” button 342 is pressed from this state, the three-dimensional search function is executed.
(3D search screen)

  Necessary settings must be made to execute the 3D search function. Specifically, registration of a search model, registration of a floor surface or a storage container on which a workpiece is placed, setting of conditions for a three-dimensional search, and the like can be given. In addition to making it possible for the user to arbitrarily set all items, these settings can be set for each item while sequentially showing necessary procedures to the user. The guidance function for such setting will be described based on the three-dimensional search screens shown in FIGS.

In these three-dimensional search screens, a flow display unit 351 for displaying a flow of a procedure to be set is provided in the upper part of the image display field 141 at the lower left of the screen. In the flow display section 351, an outline of each procedure is shown by text, symbols, etc., and the currently set procedure is highlighted. In the example of FIG. 90, the “model registration” icon 352 is displayed in a bright color, and the other “floor / box setting” icon 353 and “search setting” icon 354 are displayed in a dark color. You can announce the subject you are doing. Further, in the operation column 142 provided on the right side of the three-dimensional search screen, the description of the item currently being set is displayed in text or the like, and the setting to be made to the user can be described in characters. The description may include not only text information but also a combination of symbols and moving images as needed. In addition, a setting field for parameters to be set is also provided.
(Registering a search model based on measured data)

  The three-dimensional search screens shown in FIGS. 90 to 101 show an example in which actual measurement data obtained by imaging the actual work is registered as a search model. This registration procedure can be performed according to the procedure described in the flowchart of FIG. In the three-dimensional search screens of FIGS. 90 to 101, as described above, the “model registration” icon 352 is highlighted on the flow display unit 351 as the first procedure.

FIG. 90 shows a search model registration method selection screen 350 for selecting a method for registration as a search model. The operation column 142 of the search model registration method selection screen 350 is provided with a model registration method selection column 355 for selecting a method for registering a model. In the model registration method selection field 355, as a model registration method, it is possible to select with radio buttons whether to capture and register an actual workpiece or register from CAD data. In the example of the search model registration method selection screen 350 in FIG. 90, it is selected that an actual work is imaged and registered, and the operation column 142 correspondingly describes the procedure for imaging and registering an actual work. Is displayed with letters and designs. In this example, “Model the 3D information of the workpiece surface as seen from the camera. Register as many as the number of surfaces you want to search. This tool matches the registered surface models to workpieces that have various postures. By recognizing it, the position and posture of the workpiece are recognized. ”The work to be performed is shown to the user with the text information and the design. As a result, the user can easily grasp the work to be performed on this screen. When selecting the registration of an actual work on the search model registration method selection screen 350 in FIG.
(Measurement data imaging unit)

The procedure for actually registering the actual work is as shown in FIGS. In this example, the model registration work is divided into three screens and explained to the user sequentially. First, on the actual measurement data imaging screen 360 of FIG. 91, a workpiece is imaged as the first work of model registration. In this example, in the operation column 142, as a work to be performed by the user, “1. Place the work on a plane so that the surface of the work to be registered can be seen by the camera, and press the“ Measure ”button at the lower right. 2. Place the model area on the left camera screen to surround the workpiece. Is described. Further, a detailed description of the optimal work placement can be displayed on another screen by pressing the “?” Button 364. As a result, it is possible to avoid a situation in which a lot of information is displayed on the screen and the screen becomes difficult to see or the user is confused. In this way, the user places the workpiece at a position where the sensor unit can capture an image, and performs positioning while checking the state of the workpiece in the image display field 141. When the “Measurement execution” button 361 is pressed, the imaged work is displayed as shown in FIG. Here, the image is displayed as a three-dimensional image in which the color is changed according to the height of the obtained workpiece. In addition, it is good also as a height image which expresses height not only with a color display but with the luminance value of a pixel. In the example of FIG. 91, six workpieces having the same shape are arranged and arranged in different postures. In this state, the user sets the model area 362 so as to surround the image of the workpiece to be registered as a search model. When the setting of the model area 362 is completed, the “Next” button 363 is pressed. Thereby, the screen is switched to the search model exclusion region setting screen 370 shown in FIG.
(Background removal setting part)

In the search model exclusion area setting screen 370 of FIG. 92, as the second work of model registration, an exclusion area to be excluded when registering as a search model is set. As an example of the search model exclusion region, there is a function of removing a background such as a floor surface on which a workpiece is placed when registering an actual workpiece. On the search model exclusion area setting screen 370 of FIG. 92, “2. Set background removal so that only the workpiece surface to be registered remains” is described in the operation column 142 as the second work of model registration. The In addition, a “?” Button 371 can be pressed to display a detailed description of the background removal setting method on a separate screen. Furthermore, a background removal setting field 372 is provided in the middle of the operation field 142, and designates a height to be removed as a background with a numerical value. In this example, 1.5 mm is designated, and a range of 0 to 1.5 mm in height within the range designated as the model region 362 is excluded from the search model registration targets. In accordance with the setting in the background removal setting field 372, the model area 362 displayed in the image display field 141 is updated, and the area excluded from the registration target of the search model is displayed in black. While confirming this state, the user can determine whether the setting of the exclusion area is appropriate, and can perform resetting or the like as necessary.
(Mask area setting part)

  Furthermore, not only in the height direction but also in the plane direction, a range that is desired to be excluded from the search model registration target can be designated as a mask region. Under the background removal setting field 372 of the search model exclusion area setting screen 370 in FIG. 92, “2. If there is something other than the workpiece surface to be registered, place a mask area” is explained. . Similarly, the user can press the “?” Button 373 to display a detailed description of the mask area setting method on a separate screen. When the “mask area” button 374 is pressed, a mask area can be set as an area to be excluded from search model registration within the range specified as the model area 362. The mask area can be set by automatically detecting a free curve or a boundary portion in addition to a prescribed figure such as a rectangle or a circle. Further, as described above, the model area 362 displayed in the image display field 141 is updated according to the setting of the mask area.

In the example of FIG. 92, the mask region setting is performed after the background removal setting. However, the order may be changed. When the setting of the area to be excluded from the search model registration is completed in this way, the “Next” button 375 is pressed. Thereby, the screen is switched to the rotational symmetry setting screen 380 shown in FIG.
(Rotation symmetry setting part)

  In the rotational symmetry setting screen 380 of FIG. 93, rotational symmetry is set as the third work of model registration. In this example, the operation field 142 is explained as “1. If the workpiece surface to be registered has rotational symmetry, specify the corresponding one”. Similarly, the “?” Button 381 can be pressed to display a detailed description of the rotational symmetry setting method on a separate screen. A rotational symmetry setting field 382 is provided in the middle of the operation field 142, and a candidate for rotational symmetry can be selected from options such as circular symmetry, N-fold symmetry, and none. Further, when N-fold symmetry is selected, a numerical value input box for the symmetry number is displayed, and the rotational symmetry number corresponding to the shape of the workpiece surface is set. For example, when the work surface is rectangular and the appearances of 180 degrees rotated with respect to the axis when the work surface is viewed directly above are the same, 2 is an appropriate value for the rotational symmetry number. Alternatively, when the work surface is a square and all the appearances rotated every 90 degrees with respect to the axis when the work surface is viewed directly above are the same, 4 is an appropriate value for the rotational symmetry number. Alternatively, when the work surface has a hexagonal shape and the appearance of rotation every 30 degrees with respect to the axis viewed from directly above the work surface is the same, 6 is an appropriate value for the rotational symmetry number. On the other hand, circular symmetry is set when the rotated view is all the same for an arbitrary rotation angle, such as when the work surface looks circular when looking at a sphere or columnar workpiece from directly above. Is recommended. When the rotational symmetry is set, even when the workpiece is in the posture designated as the rotational symmetry when the search model is registered, the workpiece is recognized as the same workpiece. For example, in the case of a two-fold symmetrical workpiece with the rotational symmetry number set to 2, the posture rotated 180 degrees and the posture before rotation are recognized as the same workpiece.

When the setting of the rotational symmetry is completed in this way, the “Register” button 383 is pressed. As a result, the search model is registered according to the set conditions. Of the three tasks required for model registration, the second and third may be interchanged.
(Register multiple search models)

  When the search model is registered, a search model registration screen 390 shown in FIG. 94 is displayed. The operation column 142 of the search model registration screen 390 is provided with a model list display column 391 for displaying a list of registered search models. Here, the search model with model number 0 is displayed together with the reduced image as the registered search model. It is also displayed that the rotational symmetry setting for the search model is “none”.

  In order to register additional search models, an “add” button 392 is pressed. Thereby, the search model can be registered in the same procedure as described above. As a method for registering the search model, it is possible to select in the model registration method designation field 393 whether to capture and register an actual workpiece or register from three-dimensional CAD data. Here, “Register with actual work” is selected in the model registration method designation field 393 to perform additional registration.

FIG. 95 shows a search model registration screen 410 to which a search model has been added in this way. In the search model registration screen 410 shown in this figure, among the six works arranged in the image display column 141, a work arranged in the lower left is additionally registered as a new search model. The additionally registered search model is added as model number 1 to the model list display field 391.
(Display of simple 3D search results)

Furthermore, on the search model registration screen of FIGS. 94 and 95, the result of a simple three-dimensional search using a registered search model can be displayed. For example, in the example of FIG. 94, as a result of performing a three-dimensional search on the six workpieces displayed in the image display column 141 using the search model with model number 0, the upper left workpiece used for registration is searched. Yes. A point cloud that matches the search model is displayed on the searched workpiece. In this example, of the surface shape of the search model, a planar point group is displayed in white, and a point group indicating an outline is displayed in purple. Furthermore, the score calculated as the evaluation index of the simple three-dimensional search result can be displayed numerically. In the example of FIG. 94, 96.4 is displayed as the score value. The score does not reach the highest value (99.9 in this example) with respect to the original work registered as a search model because it varies depending on the imaging state when the model is registered and the input image when the three-dimensional search is executed. This is because the score was slightly lowered. From the result of the score value, it can be understood that the state is approximately 96% or more and the state at the time of model registration.
(Search area setting part)

  When the search model is registered in this manner, the search area is set as the setting of the three-dimensional search function. Specifically, when the necessary search model registration is completed on the search model registration screen 410 in FIG. 95 and the “complete” button 411 in the lower right is pressed, the process proceeds to the search area setting screen 420 in FIG. On this search area setting screen 420, a search area for performing a three-dimensional search is designated. In the flow display section 351 at the top of the screen, the highlight display moves from the “model registration” icon 352 to the “floor / box setting” icon 353, and after the model registration is completed, the floor surface or storage container on which the workpiece is placed. It is shown to the user that it is a stage where the registration is performed.

A search area setting field 421 is provided in the operation field 142 of the search area setting screen 420. The search area setting field 421 is provided with a “designation method selection navigation” button 422 for explaining to a user of a search area designation method for performing a three-dimensional search, and a designation method selection field 423 for selecting a search area designation method. It is done. When the “designation method selection navigation” button 422 is pressed, a navigation screen is displayed that guides the user how to designate a search area for performing a three-dimensional search. On the other hand, in the designation method selection field 423, one of floor designation, box designation, box search, and no designation can be selected from the drop box as an option. Of these, when floor designation is selected, the search area setting screen 430 of FIG. 97 is displayed, and a floor designation dialog 431 is displayed, explaining that “Please click on three points to calculate the floor.” Is shown to the user. Accordingly, when the user designates an arbitrary point indicating the floor surface from the image display field 141 with a pointing device such as a mouse, the spatial coordinates (X, Y, Z) of the designated point are displayed. A gauge is displayed as the extraction area. This extraction area represents a range to be obtained with reference to the clicked position when calculating the Z coordinate value at the clicked point. Since the data of each point measured three-dimensionally includes an error, in order to suppress the influence of the error, the average value of the data within a certain range or the value obtained by the median is used. preferable. In this case, the Z coordinate value is obtained from the average value of the range specified by the extraction area. When the floor surface is designated in this way, the floor surface information is calculated, and the floor surface information calculated in the search region setting field 421 is displayed as the floor surface information display as in the search region setting screen 440 of FIG. It is displayed in the column 441. Here, in the floor surface information display field 421, the X-tilt, Y-tilt, and Z-intercept of the floor surface are respectively displayed as floor surface information as numerical values. Further, an offset amount can be set as necessary. For example, considering a variation in the thickness of the floor surface of the storage container, a predetermined height from the floor surface is removed from the search area. In the example of FIG. 98, a noise removal column 442 is provided to remove noise from the floor surface. Here, the range from the designated floor surface height to the height of 1.5 mm designated in the noise removal column 442 is excluded from the target region of the three-dimensional search. When the setting of the floor surface is completed as described above, the “complete” button 443 is pressed. This switches to the search parameter setting screen 450 of FIG.
(Search parameter setting section)

  In the search parameter setting screen 450 of FIG. 99, search parameters are set as conditions for executing a three-dimensional search. In the flow display section 351 at the top of the screen, the highlight display moves from the “floor / box setting” icon 353 to the “search setting” icon 354, the search area setting is completed, and the process proceeds to the stage of setting the search conditions. Is shown to the user. The operation column 142 of the search parameter setting screen 450 is provided with columns for setting search parameters necessary for executing the three-dimensional search. Specifically, a detection condition setting field 451, a feature extraction condition setting field 452, and a determination condition setting field 453 are provided. Among these, the feature extraction condition setting field 452 sets an edge extraction threshold value that is a threshold value for extracting an edge when detecting a workpiece from the input image. In the determination condition setting field 453, an upper limit and a lower limit are set for the measured value of the score, or an upper limit and a lower limit are set for the measured value of the X coordinate of the position, and a determination condition for whether or not to select the search result is set. To do.

  In the detection condition setting field 451, a condition for detecting a workpiece is set. In this example, a detection number designation field 454 for designating an upper limit of the number of workpieces to be detected, an inclination angle upper limit designation field 455 for designating an upper limit of the tilt angle of the workpiece to be detected, and a score for designating a lower limit of a score as an evaluation index A lower limit designation field 456 is provided. The designation can be performed numerically or can be continuously adjusted by a slider or the like. Further, when the detailed setting button 457 is pressed, the screen is switched to the search parameter setting screen 460 in FIG. 100, and a detection detailed condition setting dialog 461 for setting more detailed detection conditions is displayed.

  In the detailed detection condition setting dialog 461, a basic setting field 462, a detailed detection condition setting field 463, and an option setting field 464 are provided. In the basic setting column 462, in addition to the detection number designation column 454, a search sensitivity setting column 465 for setting the sensitivity of the three-dimensional search and a search accuracy setting column 466 for setting the accuracy of the three-dimensional search are provided. In addition, in the detection condition detail setting field 463, in addition to the inclination angle upper limit designation field 455 and the score lower limit designation field 456, a reference angle setting field 163a for setting a reference angle, an angle range are set as a rotation angle range setting field 163. A range setting field 163b for setting is provided.

  In addition, the detection condition can be set for each search model as well as for the search model. In the example of FIG. 100, a search model individual designation field 467 is provided. By turning this check box ON, it is possible to individually set a detection condition for a three-dimensional search for each search model. Such an example is shown in the search parameter setting screen 470 of FIG. Here, when the check box of the search model individual designation column 467 is turned ON, a search model selection column 467b is displayed, and a search model for setting detection conditions can be selected. This example shows a state in which search model A is selected. As detection conditions for search model A shown in model list display field 391 in FIG. 94 and the like, inclination angle upper limit designation field 455, reference angle setting field 163a, The range setting field 163b and the score lower limit designation field 456 can be individually designated.

  On the other hand, in the option setting field 464 shown in FIG. 100 and the like, for the label number assigned as a serial number for identifying the search result, the label order designation field 468 that defines the order of the labels, and the judgment label designation that designates the judgment label A column 469 is provided. In the label order designation field 468, a descending order of correlation values, an ascending order, a descending order of height in the Z direction, an ascending order, or the like can be selected. The correlation value here is a synonym for the above-mentioned “score”. In the determination label specification field 469, it is possible to specify a target for determining whether or not the position or posture is different from the expected for a specific search result. Here, it is designated by a serial number for identifying the search result. For example, the setting is performed when it is necessary to determine whether or not the search result is unreliable data, such as whether the search result is an unexpected position, rotation angle, or inclination angle.

When the search settings for the three-dimensional search are completed in this way, the “OK” button 471 provided at the lower right of the operation column 142 is pressed to end the three-dimensional search condition setting.
(Registering a search model based on 3D CAD data)

  In the above example, the procedure for capturing and registering an actual workpiece as a search model has been described (corresponding to FIG. 62 and the like). However, according to the present invention, as described with reference to FIG. 28 and the like, three-dimensional CAD data indicating the shape of a work created in advance can be registered as a search model. Hereinafter, a procedure for registering such 3D CAD data as a work model will be described with reference to FIGS.

First, in the search model registration method selection screen 480 in FIG. 102, registration from CAD data (STL format) is selected from the model registration method selection field 355 with a radio button. In response to this, the operation column 142 displays a description of the procedure for registering the three-dimensional CAD data as a search model in characters and symbols. In this example, “Model the 3D information of the workpiece surface that can be seen from the positive and negative directions (total 6 directions) of the XYZ axes in the 3D CAD data. Select only the surface you want to search from the generated model. This tool recognizes the position and posture of the workpiece by matching the registered surface model to the workpiece in various postures. ” It shows. In this manner, when registration from CAD data is selected on the search model registration method selection screen 480 of 102 and the “OK” button 481 is pressed, the screen shifts to the CAD data reading screen of FIG. Specifically, the screen shifts to the search model registration screen 490, and a CAD data reading dialog 491 is displayed.
(CAD data reading section)

In the CAD data reading dialog 491 shown in FIG. 103, the storage destination of the CAD data to be read is designated, and further the CAD data is selected. As a reading destination, a reading medium such as a semiconductor memory can be selected in addition to the storage unit 9. When a file is selected from the file list display field 492 and the “execute” button 493 is pressed, reading of the three-dimensional CAD data is executed.
(Registered model selection part)

When the 3D CAD data is read, a surface to be registered as a search model is selected. Here, a search model registration screen 510 shown in FIG. 104 is displayed, and a hexahedral view corresponding to the basic direction image of the work model is automatically displayed on the registration model selection screen 511. In this state, the user can select a face to be registered. Here, a check box is turned on by default, and a surface that does not need to be registered as a search model can be excluded from the search model by turning off the check box. When the registration target is selected in this way, the “next” button 512 is pressed to proceed to the search model registration screen 520 of FIG.
(Rotation symmetry designation part)

  In the search model registration screen 520 of FIG. 105, a rotational symmetry designation dialog 521 is displayed, and among the basic direction images selected as registration targets by the user, those having rotational symmetry, such as up and down, front and rear, left and right, Even when rotated, a plane having a symmetrical shape can be set. When the rotational symmetry is set in this way, the “registration” button is pressed to register the search model.

When the search model is registered, a list of search models registered in the model list display field 391 is displayed as in the search model registration screen 530 of FIG. As described above, a plurality of search models can be registered for one work. For example, in the example of FIG. 106, six search models A, B, C, D, E, and F shown in FIG. 105 and the like are registered for the three-dimensional CAD work model of model number 0. Therefore, the model list display field 391 displays a list of search models A to F with model number 0. Search models that cannot be displayed on one screen in the model list display field 391 can be scrolled and switched.
(Search model editor)

In addition, registered search models can be edited. In the example of FIG. 106, when the “edit” button 531 in the operation column 142 is pressed, a model edit screen 620 shown in FIG. 114 is displayed. In the model editing screen 620, registered search models are displayed in the image display field 141, and set items are displayed in the operation field 142. Here, a model region setting field 621, a feature setting field 622, a background removal setting field 327, a rotational symmetry setting field 382, and a model registration field 623 are provided with a model number display field. The user can select an item to be changed and change it appropriately.
(Model area detailed setting section)

  When a detailed setting button 624 provided in the model area setting field 621 is pressed, a model area detailed setting screen 630 shown in FIG. 115 is displayed. The model area detail setting screen 630 can set a mask area as an area to be excluded from search model registration targets within the range of the model area 362 displayed in the image display field 141. In the operation column 142, a model region designation column 621, a mask region designation column 631, and a weighting region designation column 632 are provided. In the mask area designation column 631, a plurality of mask areas can be set. Each mask region can be set by automatically detecting a free curve or a boundary portion in addition to a prescribed figure such as a rectangle or a circle. In addition to the mask areas, weights can be set in the weight area designation column 632.

When scoring a search result, if there is a point of a feature point to be emphasized over other feature points included in the search model, the score result can be adjusted by surrounding it with a weighted region. For example, if there is a shape that leads to the identification of the search model depending on the presence or absence of some features in the overall shape or the posture in the rotation direction is determined, a weighting area is set for the important feature As a result, the search detection capability can be improved. Moreover, the weighting degree with respect to another feature point can also be designated by a ratio. For example, when the weighting degree is set to “3”, the score value can be adjusted to be calculated with an influence degree three times that of other feature points.
(Feature details setting section)

  Further, when a detail setting button 625 provided in the feature setting field 622 is pressed, a feature detail setting screen 640 shown in FIG. 116 is displayed. On the feature detail setting screen 640, feature settings and presence / absence of rotational symmetry are set for the model region 362 displayed in the image display field 141. In the operation column 142, a feature detail setting column 641 and a rotation target setting column 382 are provided. The feature detail setting column 641 is provided with a feature extraction interval specifying column 642 for specifying a feature extraction interval, a background removal setting column 327, and an edge extraction threshold setting column 643. The feature extraction interval is a parameter that affects the feature point extraction interval when the search model is generated, and “precision priority”, “standard”, and “speed priority” can be selected. When “accuracy priority” is selected, the feature point extraction interval in the search model is narrowed, and even finer features are captured to increase detection accuracy, but the data amount increases and processing time is delayed. On the other hand, when “speed priority” is selected, the feature point extraction interval in the search model is widened, and fine features are lost to reduce the detection accuracy, but the data amount is reduced and the processing time is increased. Adjustment can be made according to the shape and size of the workpiece to be used and the allowable processing time and detection accuracy.

  In this way, the user can adjust necessary items so that a more appropriate search result can be obtained while referring to the result of the simple three-dimensional search and the obtained score. When registration of the search model is completed in this way, the subsequent steps (search area setting and search parameter setting) are followed by imaging the actual work described in FIGS. 96 to 101 and registering the search model. The procedure is the same as described above, and the description thereof is omitted.

A search model is registered in advance as described above, and then a three-dimensional search is performed. 3D search is used for actual operation, that is, when picking up a group of separated workpieces with the sensor unit and performing picking to specify the gripping position of the workpiece that can be gripped. It is also used when performing. In the 3D pick simulation, can each end result be searched by the 3D search unit 8k be gripped by the end effector at the gripping position previously specified for the work model by the gripping position specifying unit 8d? This is an operation for determining whether or not.
(Search result display area)

Next, an example in which a three-dimensional search is performed on workpieces stacked in bulk will be described with reference to FIGS. FIG. 107 shows an example of a search result display screen 540 showing a result of executing a three-dimensional search for a group of workpieces stacked in bulk. The search result display screen 540 is provided with an image display column 141 and an operation column 142, and a list of search models used for the three-dimensional search is displayed in the model list display column 391 in the operation column 142 in the vertical direction. On the other hand, the image display column 141 displays a point cloud that is a search result and a score that is an evaluation index of the work in a stacked state obtained as an input image, superimposed on the work obtained as a search result. I am letting.
(Search results list)

  In addition, a search result list display field 541 is provided in the upper part of the image display field 141, and a list of used search models and scores is displayed for the searched search results. Here, the number of searched items, presence / absence of workpiece, total volume / volume of workpiece, model number and score of search result of specific label number, position (X, Y, Z), etc. are displayed. The total volume / workpiece volume is the value obtained by dividing the total volume of the point cloud located higher than the floor or bottom of the return box by the average volume assumed from the registered search model. The data indicates how many workpieces are present in the image. When this numerical value is larger than a certain value, it is determined that there is a result of presence / absence of a workpiece (1). If the number of searched items is 0 and no gripping solution is obtained, check whether the result of presence / absence of this workpiece is present, and whether the picking is finished with the workpiece actually gone, It can be confirmed whether or not the process ended in a state in which it remained.

The display of the search result list display field 541 can be switched. For example, a switching button 542 for switching display contents is provided at the lower right of the search result list display field 541 of the search result display screen 540 shown in FIG. The switching button 542 is composed of left and right arrows, and a screen number 543 is further displayed on the top thereof. In this example, “1/11” indicating the first of 11 screens is displayed as the number of screens 543. By pressing the switch button 542 to increase the number, the search result list is displayed accordingly. The display contents of the display field 541 can be switched. For example, when the switching button 542 is operated to change the screen number 543 to “2/11”, the screen is switched to the search result display screen 550 shown in FIG. 108, and an overview of the three-dimensional search results is displayed in the search result list display column 541. The Here, in addition to the searched number, the presence / absence of a workpiece, the total volume / the volume of the workpiece, an inclination angle, a rotation angle, a posture (R _X , R _Y , R _Z ) and the like are displayed. When the screen number 543 is changed to “3/11”, the screen is switched to the search result display screen 560 shown in FIG. 109, and the search result list display field 541 displays the number of searched items and the presence / absence of workpieces as an overview of the three-dimensional search results. In addition to the total volume / workpiece volume, XYZ inclination and the like are displayed as floor surface information.

Further, when the screen number 543 is changed to “4/11”, the screen is switched to the search result display screen 570 shown in FIG. 110, and the search result list display column 541 displays, for each label number identifying the search result, as an outline of the three-dimensional search result. Further, the model number and the score of the search model used for detecting the search result are displayed in the horizontal direction. Furthermore, when the screen number 543 is changed to “5/11”, the screen is switched to the search result display screen 580 shown in FIG. 111, and the search result list display field 541 has a label number for identifying the search result as an outline of the three-dimensional search result. Each time the position (X, Y, Z) is displayed. Furthermore, when the screen number 543 is changed to “6/11”, the screen is switched to the search result display screen 590 shown in FIG. 112, and the search result list display field 541 has a label number for identifying the search result as an outline of the three-dimensional search result. The tilt angle and the rotation angle are displayed for each time. Furthermore, when the screen number 543 is changed to “7/11”, the screen is switched to the search result display screen 610 shown in FIG. 113, and the search result list display field 541 has a label number for identifying the search result as an outline of the three-dimensional search result. The posture (R _X , R _Y , R _Z ) is displayed every time.
(3D pick)

Next, a procedure for performing a three-dimensional pick simulation for determining whether or not each end result can be grasped by an end effector will be described with reference to FIGS. 117 to 149. When the “3D pick” button 343 is pressed on the function selection screen 340 in FIG. 89, the three-dimensional pick initial setting screen 650 in FIG. 117 is displayed, and settings necessary for performing the three-dimensional pick are sequentially performed for the user. The guidance function is executed. Here, a 3D pick guidance field 655 is displayed in the upper part of the operation field 142, and operations to be sequentially performed are displayed as icons. In this example, four steps of “Initial setting”, “Grip registration”, “Condition / Prize”, and “Place setting” are displayed.
(3D pick initial setting screen)

In the three-dimensional pick initial setting screen 650 of FIG. 117, first, initial setting for performing a three-dimensional pick is performed. In the 3D pick guidance field 655 at the top of the operation field 142, an “initial setting” icon is highlighted, and the work to be performed at this stage is shown to the user. The operation column 142 includes a calibration data selection column 651, a detection tool setting column 652, and a hand registration column 653. In the calibration data selection field 651, calibration data calculated for coordinate conversion of the coordinate system displayed in advance by the image processing apparatus into the coordinate system for operating the end effector is called and registered. In the detection tool setting field 652, a means for detecting a workpiece is designated. In this example, the three-dimensional search for workpiece detection set to the tool number 100 is selected. The search model settings registered through the above-described three-dimensional search settings are read. When search models for different workpieces are set in advance for different tool numbers, the setting target of the three-dimensional pick can be switched by switching the selection of this detection tool.
(End effector model setting part)

In the hand registration field 653, an end effector model is registered. When the “Create hand model” button 654 is pressed here, an end effector model setting screen 660 shown in FIG. 118 is displayed. On the end effector model setting screen 660, a list of registered end effector models is displayed in the hand model list display field 661. Since the example in FIG. 118 is not registered, no end effector model is displayed in the image display field 141 or the end effector model list display field 661 in the operation field 142. Therefore, an “add” button 662 is pressed to add an end effector model. In the image display field 141 of the end effector model setting screen 660, the flange surface FLS is displayed as a reference surface at the tip of the arm portion of the robot on which the end effector model is arranged. Further, a tool coordinate axis TAX indicating a coordinate axis on the end effector model EEM side is also displayed (details will be described later).
(End effector model editor)

When the “add” button 662 is pressed in FIG. 118, an end effector model editing screen 670 shown in FIG. 119 is displayed. In the end effector model editing screen 670, the parts constituting the end effector model are set. Here, the registered parts are displayed in the parts list display field 671. In the example of FIG. 119, since no parts are registered yet, nothing is displayed in the parts list display field 671. When the “Add” button 672 is pressed in this state, the part addition dialog shown in FIG. 120 is displayed, and it is determined whether to add a part to be added by reading CAD data, or to create a new part using a rectangular parallelepiped model or a cylindrical model. Select with the radio button. In addition, an additional part is one form of the additional area | region mentioned above, Arbitrary shapes, such as not only a rectangular parallelepiped and a cylinder but prismatic shape, triangular shape, spherical shape, can be utilized. In the example of FIG. 120, CAD data is selected. When the created 3D CAD data is selected by pressing the “Add” button 681, the CAD position / orientation setting screen 690 shown in FIG. 121 is displayed, and the 3D CAD data is read and created in the image display column 141 by CAD. A CAD model CDM is displayed, and a position / posture column 691 for designating the position and posture of the CAD model CDM is displayed in the operation column 142. The position and orientation of the CAD model CDM are specified based on the flange surface FLS. The user designates the attachment position and angle of the end effector model from the position / posture column 691. For example, when 20 ° is designated as the rotation angle R _X from the state of FIG. 121, the CAD model CDM displayed in the image display column 141 is displayed in a posture rotated by 20 ° about the X axis as shown in FIG. . Alternatively, if 50 mm is designated in the Z direction from the state of FIG. 121, the CAD model CDM is displayed with an offset of 50 mm in the Z direction as shown in FIG. When the position and orientation of the CAD model CDM are adjusted in this way and the “OK” button 692 in FIG. 121 is pressed, CAD data is registered and displayed in the parts list display field 671.

Multiple parts can be added. For example, when the “add” button 682 is pressed from the end effector model editing screen of FIG. 124, a cylinder model is selected in the part addition dialog 680 of FIG. 120 and the “add” button 681 is pressed, the additional region position of FIG. A posture setting screen is displayed, and a columnar model is newly displayed in the image display field 141 as an additional area ADA. Note that the part being edited is highlighted. In the example of FIG. 125, the additional area ADA is displayed in red, and the CAD model CDM is displayed in gray or a wire frame. This makes it easy for the user to visually grasp the part currently being edited. Similarly, the position and orientation of the additional area ADA to be added are designated from the position / orientation column 691. In this example, the Z position is specified as −50 mm in the position / posture column 691, and the additional area ADA is arranged in the direction opposite to the CAD model CDM from the flange surface FLS. Further, as the size of the additional area ADA, the radius and height of the cylindrical model can be designated in the size designation column 693. After the conditions of the additional area ADA are set in this way, when the “OK” button 692 is pressed, the additional area ADA is registered, and a cylinder is displayed in the parts list display field 671 of the end effector model editing screen as shown in FIG. Is added and displayed.
(Tool coordinate specification part)

It is also possible to adjust the position and orientation of the registered parts. For example, when the selected part is changed from the additional area ADA to the CAD model CDM from the image display column 141 or the parts list display column 671 on the end effector model editing screen of FIG. 126, the result is as shown in FIG. In the tool coordinate designation field 673, the position (X, Y, Z) and the posture (R _X , R _Y , R _Z ) can be adjusted. For example, as shown in FIG. 128, when the Z position is designated as 145 mm, the tool coordinate axis TAX is moved and displayed in the Z direction from the flange surface FLS by the designated distance in the image display field 141. Further, in this state, as shown in FIG. 129, when the rotation R _X is designated as 20 °, the tool coordinate axis TAX is rotated 20 ° clockwise around the X axis and displayed.

  In this way, a plurality of parts are registered, and an end effector model EEM is constructed by combining these parts. When the setting of each part is completed, when an “OK” button 674 is pressed on the end effector model editing screen of FIG. 128 or the like, a plurality of parts are synthesized and registered as the end effector model EEM, and the end effector model setting screen of FIG. As indicated by 660, “hand 0” is displayed in the hand model list display field 661. In the image display column 141, the parts that have been displayed separately are displayed as an integrated end effector model EEM.

In this example, the example in which the end effector model is configured by combining parts has been described. However, the parts are not necessarily limited to the end effector model or a part thereof, and for example, cables and the like can be expressed by parts. In this way, the parts form one form of an additional model that represents a part of the end effector model, an object added to the surface of the end effector model, or a peripheral member.
(Grip reference point HBP)

  Further, the gripping reference point HBP of the end effector model EEM is displayed so as to overlap the image display field 141. The gripping reference point HBP indicates a position where the workpiece is gripped by the end effector model EEM. For example, in the end effector model EEM, the middle between the claws that grip the workpiece is set as the gripping reference point HBP. Note that the gripping reference point HBP may be configured so that the user can specify an arbitrary position in addition to calculation on the image processing apparatus side so as to be determined according to a predetermined rule.

Furthermore, it is preferable that the tool coordinate axes that define the position and orientation of the end effector model EEM are also displayed in an overlapping manner on the image display field 141. The tool coordinate axis is preferably a rotated coordinate axis RAX that is changed according to the rotation of the end effector model EEM. Furthermore, it is more preferable to make the origin of the rotated coordinate axis RAX coincide with the gripping reference point HBP. This makes it easier for the user to visually grasp the state of the end effector model EEM when setting the grip position.
(Grip registration part)

When the end effector model EEM is registered as an initial setting in this way, the gripping position is registered next. In the grip registration screen 710 of FIG. 131, the highlight display in the 3D pick guidance field 655 moves from the “initial setting” icon to the “grip registration” icon. The operation column 142 includes a search model switching column 711 for selecting a registered search model, and a grip registration list display column indicating a list of registered gripping positions for the search model selected in the search model switching column 712. 712 is displayed. As described above, in this example, the work model for registering the gripping position is shared with the search model used for the three-dimensional search. Here, it is assumed that six search models A to F of model number 0 registered in FIG. 104 are registered in the search model switching field 711 as registered search models. In the example of FIG. 131, since the grip position has not been registered yet, nothing is displayed in the grip registration list display field 712. Search model A is selected in search model switching field 711, and search model SMA corresponding to A is displayed in image display field 141. Further, a reference coordinate axis BAX, which is a coordinate axis indicating the position and orientation of the search model SMA, is displayed in an overlapping manner. From this state, the user registers the gripping position for the search model A. Specifically, when an “add” button 713 is pressed, the screen is switched to the grip registration screen shown in FIG. 132, and a grip setting dialog 720 is displayed.
(Grip setting part)

  In the grip setting dialog 720, a basic setting field 721, a grip position coordinate designation field 724, and a movement amount setting field 727 are provided in the operation field 142. The basic setting field 721 includes a grip label setting field 722 and a hand selection field 723. In the grip label setting field 722, a grip label that is identification information for specifying a grip position is set. Here, “A” indicating the search model and the serial number “000” of the grip position are set as the grip labels. In the hand selection field 723, the end effector model EEM that grips the gripping position set in the grip label setting field 722 is selected. In this example, “hand 0” is selected in the hand selection field 723, and the end effector model EEM is displayed in the image display field 141 accordingly. Thus, the grip position is set in association with the end effector model EEM.

In the grip position coordinate designation field 724, a coordinate position designation field 725 in which the user can directly designate the coordinate position (X, Y, Z) and posture (R _X , R _Y , R _Z ) indicating the grip position numerically; An “easy setting navigation” button 726 is provided for executing a guidance function for guiding the user to set the grip position.

The movement amount setting field 727 defines the direction in which the end effector model EEM is moved. Here, the approach movement amount that brings the end effector model EEM closer to the search model SMA is defined as the distance in the X direction, the Y direction, and the Z direction so that the end effector model EEM holds the search model SMA. In the example of FIG. 132, an example of moving 100 mm in the Z direction is shown.
(Grip position setting guidance function)

When a “simple setting navigation” button 726 is pressed from the screen of FIG. 132, a gripping position setting guidance function is executed. Specifically, as shown in FIG. 133, a setting method selection dialog 730 is displayed, and for the user, whether to set the gripping position using a three-dimensional viewer or to select using a robot, Select with the radio button. In this example, setting on the three-dimensional viewer is selected. When an “OK” button 731 is pressed in this state, a registration guidance function on the 3D viewer is executed.
(Registration guidance function on 3D viewer)

In the registration guidance function on the three-dimensional viewer, the grip position is registered through four steps. Here, the user is caused to designate a gripping position via the three-dimensional viewer registration screen shown in FIGS. 134 to 141. Specifically, when the “OK” button 731 is pressed in FIG. 133, the screen is switched to the XY designation screen 740 in FIG.
(First step: XY designation)

In FIG. 134, an XY designation screen 740 is displayed as one form of the XY designation unit. Here, as a first step for registering the gripping position, the gripping position is designated from the plan view in a state where the search model SMA is displayed in a plan view. Specifically, the search model SMA is displayed two-dimensionally on the XY plane in the image display column 141, and the X and Y coordinates to be gripped are designated on the search model SMA. In the operation column 142, as STEP1, the work to be performed for the user is described in text or the like. Here, “Please select the XY position to be gripped on the camera image” is displayed. In the example of FIG. 134, the search model A is enlarged and displayed as a height image (search model SMA) in the image display field 141. In this state, the user holds a part to be gripped using a pointing device such as a mouse. Select P1. The XY coordinates are input to the XY coordinate designation column 741 in accordance with the grip designation position P1. When the X and Y coordinates in the XY coordinate designation column 741 are adjusted, the grip designation position P1 in the image display column 141 is also updated accordingly. In the XY designation screen 740, only the X and Y coordinates of the position parameters can be designated. In other words, the designation of other position parameters is impossible. In this way, it is possible to set a large number of position parameters at the same time as in the past, avoiding confusion for the user, and by limiting the position parameters that can be set, the user can accurately grasp the position parameters that can be set. Then, the user is guided to sequentially set the position parameters. When the designation of the XY coordinates is completed in this way, the “Next” button 742 is pressed. Thereby, the process proceeds to the second step, and the screen is switched to the Z- _RZ designation screen 750 in FIG.
(Second step: Z-R _Z designation)

A Z-R _Z designation screen 750 in FIG. 135 is a form of the Z-R _Z designation unit, and is a screen for allowing the user to designate the Z coordinate and the posture R _Z. In the operation column 142, the work to be performed by the user in the second step is displayed and explained in text. Here, “STEP2 Please specify the Z position to grip and the posture angle R _{Z of the} hand” is displayed. The operation field 142 includes a Z designation field 752 for designating the _Z position and an R _Z designation field 753 for designating the R _Z posture as the Z-R _Z designation field 751. In the image display column 141, the end effector model EEM and the search model SMA are displayed in a three-dimensional manner. That is, the image display column 141 functions as a three-dimensional viewer that displays the end effector model and the search model in a three-dimensional manner. When the numerical values in the Z designation column 752 and the R _Z designation column 753 are changed, the three-dimensional display contents of the end effector model EEM and the search model SMA in the image display column 141 are updated accordingly. Also, since the user can change the viewpoint of the three-dimensional viewer by dragging the image display column 141, the gripping setting state can be confirmed in detail.

  In addition, in the image display field 141 which is a three-dimensional viewer, only the Z axis is displayed as the coordinate axis related to the currently set position parameter among the coordinate axes. This Z axis is the corrected rotation Z axis AXZ after correction displayed with the ZYX Euler angle described above. In this way, by displaying only the coordinate axes that are currently adjustable, and making it impossible to adjust other position parameters on this screen, the user can properly adjust according to a predetermined order without causing unnecessary confusion. Proper guidance is realized so that only correct position parameters are sequentially set.

  In the example of FIG. 135 and the like, the end effector model EEM is brought closer to the work model by displaying the axis in the gripping direction (here, the corrected rotation Z axis AXZ) through the gripping reference point HBP. Therefore, it is possible to show the direction of movement for easy understanding. Moreover, you may comprise so that the movement direction of an end effector model may be shown with the arrow. At this time, the moving direction of the end effector model may be displayed on the axis in the gripping direction or may be displayed at another position.

Further, the Z- _RZ designation screen 750 is provided with a “fit” button 154 as one form of the relative position setting unit. As described above, the “fit” button 154 realizes a fitting function for automatically moving the end effector model to the gripping position of the work model. When the “fit” button 154 is pressed, the end effector model EEM is moved as shown in FIG. 136 and changes to a state in which the end effector model EEM is in contact with the search model SMA. That is, the grip reference point HBP of the end effector model EEM is moved toward the grip designation position P1 of the search model SMA. It is preferable to adjust the initial position when displaying the end effector model EEM in the state shown in FIG. 135 so that the moving direction coincides with the Z axis. By moving the end effector model EEM only in the Z direction, the search model SMA can be brought into contact with the end effector model EEM, and the user can move the end effector model EEM. It can be done easily.

  Further, according to the movement of the end effector model EEM in the image display column 141, the Z position of the Z designation column 752 in the operation column 142 is changed. In this example, it is changed from 200 mm in FIG. 135 to 50 mm. Here, with the fit function, when the end effector model EEM is moved in the height direction, that is, the Z-axis direction, toward the grip specified position P1 specified in FIG. 134, the position that contacts the search model SMA is used as a reference. From here, it is moved to a position returned to the front by a predetermined offset amount. Further, only the currently set Z-axis among the coordinate axes is displayed in the image display field 141 with the gripping designated position P1 as a reference.

In this state, the user further designates the posture R _Z from the R _Z designation column 753. In the posture of the end effector model EEM shown in FIG. 136, since the side surface of the search model SMA tilted in a C shape is gripped, the end effector model EEM is rotated in the Z-axis direction to be parallel. Adjust to grip a smooth surface. At this time, the rotation of the end effector model EEM can be specified more easily by providing the angle selection button 754 for inputting a predetermined angle in addition to the user directly inputting the rotation angle in the R _Z designation field 753. In this example, as the angle selection button 754, four angle buttons of 0 °, 90 °, 180 °, and −90 ° are arranged. When the user presses the 90 ° button, 90 ° is entered in the R _Z designation field 753, and the end effector model EEM is rotated 90 ° in the Z-axis direction as shown in FIG. 137, and the search model SMA is correctly grasped. It can be corrected to a ready state.
(Third process: R _Y designation)

When Z-R _Z designation for the second process is completed in this way, the “Next” button 755 is pressed to proceed to R _Y designation for the third process. Specifically, when the “Next” button 755 is pressed, the screen shifts to the _RY designation screen 760 in FIG. 138. The R _Y designation screen 760 is a form of the R _Y designation unit, and designates the posture R _Y. In the operation field 142 of the _RY designation screen 760, an _RY designation field 761 is displayed, and “STEP3 Please specify the hand posture angle _RY ” as the work to be performed in the third step. Is displayed. Further, the image display column 141 displays only the corrected rotation Y axis AXY which is the currently set Y axis among the coordinate axes. From this state, the user designates the orientation _RY from the _RY designation screen 760 according to the orientation of the end effector model EEM that holds the search model SMA. Alternatively, the end effector model EEM is dragged and rotated on the image display field 141 which is a three-dimensional viewer. For example, in the state of FIG. 138, if 90 ° is designated as the orientation R _Y from the R _Y designation screen 760, the screen switches to the R _Y designation screen of FIG. 139, and the end effector model EEM is rotated to 90 ° around the Y axis. Is displayed. When the _RY designation of the third process is finished in this way, the “next” button 762 is pressed to shift to the _RY designation of the fourth process.
(Fourth process: R _Y designation)

Figure 140 shows the R _X designation screen 770 of the fourth step. The R _X designation screen 770 is one form of the R _X designation unit, and is provided with an R _X designation column 771 for designating the posture R _X. In the image display field 141, the corrected rotation X axis AXX that is the X axis related to the position parameter R _{X that} can be set on the R _X designation screen 770 is displayed on the end effector model EEM and the search model SMA. The In this state, the user designates the posture R _X from the R _X designation column 771. Alternatively, the end effector model EEM is dragged and rotated in the image display field 141. For example, as posture R _X , 180 ° is input in the R _X designation field 771 in the example of FIG. 140 in correspondence with the current gripping position. When 150 ° is designated, the posture is designated as shown in FIG. orientation of the end effector model EEM and search model SMA is changed in the image display field 141 is a three-dimensional viewer in accordance with the attitude R _X.

In this manner, when the finish the designation of attitude R _X and, by pressing the "Finish" button 772, to terminate the registration of the gripping position. When the grip position is registered, grip registration A-000 is added to the grip registration list display field 712, as shown in FIG. Also, the search model A corresponding to the grip registration A-000 is highlighted in the search model switching field 711. Further, in the image display column 141, a state in which the search model SMA is gripped by the position and orientation of the end effector model EEM registered for gripping is displayed in a three-dimensional manner. Further, a reference coordinate axis BAX on the search model SMA side is displayed as a coordinate axis.

Similarly, another gripping position is registered in the search model A, or gripping positions are added to the other search models BF. The grip position can be added by pressing an “add” button 713 provided in the grip registration list display field 712 as described above. The example in FIG. 143 shows an example in which grip registration B-000 for the search model B is registered.
(Copy function)

Further, when additionally registering the gripping position, a copy function for copying the already registered gripping position can be used. When a “copy” button 714 provided in the grip registration list display field 712 is pressed on the screen of FIG. 142 or the like, the registered grip position can be duplicated. Furthermore, by correcting the copied gripping position as a base, the gripping position registration work can be performed efficiently. When the “edit” button 715 is pressed in the state of FIG. 142 or the like, the grip setting dialog is displayed, and the grip position can be set as described above. Here, as described above, the “simple setting navigation” button 726 is pressed to execute the gripping position setting guidance function, and necessary portions can be corrected.
(Condition verification screen 780)

  When the gripping positions are registered in the search models A to F as described above, the “complete” button 716 is pressed on the screen of FIG. 143 or the like to end the gripping registration process and shift to the condition verification process. In the condition verification step, a three-dimensional search is performed on a group of stacked workpieces to detect the workpiece, and it is verified whether the workpiece can be gripped by the end effector at a specified gripping position. Here, the simulation is performed using the end effector model EEM and the search model. An example of the condition verification screen 780 is shown in FIG. In the condition verification screen 780, the highlight display in the 3D pick guidance field 655 moves from the “gripping registration” icon to the “condition / verification” icon. The operation column 142 is provided with a detection condition setting column 781 for setting a condition for detecting a gripping position based on the three-dimensional search result, and a verification column 785 for verifying whether or not the detected workpiece can be gripped. In the detection condition setting field 781, a detection number designation field 782 for designating an upper limit of the number of gripping solutions to be detected, a hand tilt angle upper limit designation field 783 for designating an upper limit of the tilt angle of the end effector model, and an end effector model interfering with each other. In order to extend the range, a margin setting field 784 for setting a margin from the wall surface of the storage container is provided. If the end effector model EEM enters the range of the distance specified in the margin setting field 784 (5 mm in FIG. 144), it is determined that interference occurs.

On the other hand, in the verification column 785, the detection number display column 786 indicating the number of search results actually detected as a result of the simulation of the three-dimensional search, and the label number of the search model to be verified in the detected search results are displayed. A display label designation field 787 to be designated and a “verification for each work” button 788 for executing verification for each work are provided. The search result of the label number designated in the display label designation column 787 is displayed in the image display column 141.
(Verification for each work)

  When a “verification for each work” button 788 is pressed, a verification dialog 790 shown in FIG. 145 is displayed. The verification dialog 790 is a form of the work selection screen 210 described with reference to FIG. The verification dialog 790 in FIG. 145 verifies whether or not each detected workpiece can be gripped. In this example, the verification dialog 790 is provided with a target work selection field 791, a detection search model display field 792, and a “grip confirmation” button 793. When a “check grasp” button 783 is pressed, a detection result display dialog 810 shown in FIG. 146 is displayed. This detection result display dialog 810 is one form of the gripping solution candidate display screen 220 shown in FIG. Here, for each surface (search image and estimated image) detected for the workpiece selected in the target workpiece selection field 791 (second workpiece in FIG. 145), the determination result of whether or not the gripping is possible at the set gripping position. Are displayed in a list in the gripping solution candidate display field 811. In the example of FIG. 146, the grip label C-000 is selected, and in accordance with this, the end effector model EEM is arranged and displayed at the corresponding grip position in the image display column 141, and the end is displayed according to the determination result. The effector model EEM is displayed in color. In this example, since the determination result is NG, the end effector model EEM is displayed in red. In the gripping solution candidate display field 811, a point cloud interference error is displayed as the cause of the gripping failure. Thereby, the user is provided with a guideline for performing necessary countermeasures such as correction or addition of the gripping position.

  In addition, when the grip label A-000 that is OK is selected in the detection result display dialog of FIG. 147, the end effector model EEM gripped at the corresponding gripping position is displayed in the image display field 141 accordingly. Is displayed. In this way, it is possible to determine whether or not gripping is possible and to take appropriate measures as necessary.

  Furthermore, on the condition verification screen 780 of FIG. 144, when a detection condition detail setting button 789 provided in the detection condition setting field 781 is pressed, the screen is switched to the detection condition detail setting screen 820 of FIG. The detection condition detail setting screen 820 includes a gripping position detection condition setting field 821, an end effector / box / floor interference determination setting field 822, and an end effector / three-dimensional point group interference determination setting field 823. It has been.

  In this way, when the condition verification process is completed, the process finally proceeds to a place setting process for defining the placement position of the workpiece. An example of the place setting screen 830 is shown in FIG. In the 3D pick guidance field 655 of the place setting screen 830, the highlight display shifts from the “condition / verification” icon to the “place setting” icon. In the operation column 142, a tool coordinate setting column 831 and a place position registration column 832 are provided. From this screen, the user makes settings related to the work place.

  The image processing apparatus, the image processing method, the image processing program, the computer-readable recording medium, and the recorded device according to the present invention can be suitably used for the purpose of verifying the operation of robot bulk picking.

1000, 2000, 3000, 4000, 7000, 8000, 9000 ... Robot systems 100, 200, 300, 400, 700, 800, 900 ... Image processing device 2 ... Sensor unit: 2b ... Sensor control unit; 2c ... Input image acquisition unit 3, 3B ... display unit; 3a ... hexagonal display area; 3b ... image display area; 3c ... gripping solution candidate display area; 3d ... work gripping availability display area; 3e ... work gripping inability factor display area; 3f ... display interface DESCRIPTION OF SYMBOLS 4 ... Operation part; 4b ... Input / output interface 5 ... Robot main body 6 ... Robot controller; 6b ... Robot interface 7 ... Robot operation tool 8c ... Positioning part; 8d ... Gripping position specification part; ... End effector side grip setting part; 8d3 ... Grasping reference point setting part; 8d4 ... Grasping direction 8d5 ... Relative position setting unit; 8d6 ... Second designation unit; 8d7 ... Third designation unit; 8d8 ... Gripping position copy unit; 8d9 ... First designation unit; 8e ... Basic direction image selection unit; 8f ... End effector Mounting surface setting unit; 8g ... search model registration unit; 8g '... model registration unit; 8h ... rotation angle limiting unit; 8i ... search model selection unit; 8j ... image selection unit; 8k ... three-dimensional search unit; Pick determination unit; 8m ... interference determination unit; 8n ... inclination angle setting unit; 8o ... angle determination unit; 8p ... search result integration unit; 8q ... evaluation index calculation unit; 8r ... grip priority order determination unit; Unit: 8t ... work model registration unit; 8u ... end effector model registration unit; 8v ... additional model creation unit; 8w ... calibration unit; 8x ... conversion unit; 8y ... end effector mounting position calibration unit; Image estimation unit 9 ... storage unit; 9b ... gripping position storage unit 10 ... calculation unit 8e '... basic direction image generation unit 130, 130B ... search model registration screen 131 ... "registration target" check box; 131B ... selection check box 140 ... Grasping registration screen 141 ... Image display field 142 ... Operation field 143 ... Search model selection field 144 ... Grasping posture display field 145 ... Edit button 146 ... Add button 147 ... Delete button 150 ... Grasping posture addition screen 153 ... Grasping posture coordinate information 154 ... "Fit" button 160 ... Inclination angle / rotation angle setting screen 161 ... Attitude restriction search model selection column 162 ... Inclination angle upper limit setting column 163 ... Rotation angle range setting column; 163a ... Reference angle setting column; 163b ... Range setting column 170 ... End effector mounting position setting screen 171 ... End effector mounting position setting field 180 ... Holding position designation screen 181 ... Gripping position designation field 190 ... Multiple gripping position selection screen 191 ... Surface selection field 192 ... Grasping position list display 210 ... Work selection screen 211 ... Target work selection field 212 ... Detection search model display field 213 ... "Gripping "Confirm" button 220 ... gripping solution candidate display screen 221 ... gripping solution candidate display column 223 ... workpiece gripping position display column 224 ... workpiece gripping availability display column 225 ... workpiece gripping impossibility factor display column 230 ... XY designation screen 240 ... Z- _RZ designation screen 241 ... Z coordinate designation column 242 ... _RZ rotation angle designation column 250 ... _RY designation screen 251 ... _RY rotation angle designation column 260 ... _RX designation screen 261 ... _RX rotation angle designation column 270 ... Position parameters specifying screen 271 ... X, Y coordinate designation column 272 ... "next" button 273 ... Z coordinate, R _Z rotation angle designation column 274 ... R _X, R _Y rotation angle designation column 280 ... X Y-Z designation screen 281 ... X, Y, Z coordinate designation column 290 ... 3D image viewer 310 ... Additional area setting screen 311 ... Basic figure display field 312 ... "Add" button 313 ... "Edit" button 320 ... Basic figure editing Screen 321 ... Basic image parameter adjustment field 327 ... Background removal setting field 330 ... End effector imaging screen 331 ... End effector position designation field 332 ... "Imaging" button 340 ... Function selection screen 341 ... Function list display field 342 ... "3D search" Button 343 ... "3D pick" button 350 ... Search model registration method selection screen 351 ... Flow display section 352 ... "Model registration" icon 353 ... "Floor / box setting" icon 354 ... "Search setting" icon 355 ... Model registration method selection Column 356... "OK" button 360 ... Actual measurement data imaging screen 361 ... "Measurement execution" Tan 362 ... model area 363 ... "Next" button 364 ... "? "Button 370 ... search model exclusion area setting screen 371 ..."? "Button 372 ... background removal setting field 373 ..."? "Button 374 ..." mask area "button 375 ..." next "button 380 ... rotational symmetry setting screen 381 ... "?" Button 382 ... rotational symmetry setting field 383 ... "registration" button 390 ... search model registration screen 391 ... model list display field 392 ... "add" button 393 ... model registration method designation field 410 ... search model registration screen 411 ... "Done" button 420 ... Search area setting screen 421 ... Search area setting field 422 ... "Designation method selection navigation" button 423 ... Designation method selection field 430 ... Search area setting screen 431 ... Floor surface designation dialog 440 ... Search area setting screen 441 ... Floor surface information display field 442 ... Noise removal field 443 ... "Done" button 450 ... Search parameters Setting screen 451 ... detection condition setting field 452 ... feature extraction condition setting field 453 ... determination condition setting field 454 ... detection number designation field 455 ... inclination angle upper limit designation field 456 ... score lower limit designation field 457 ... detail setting button 460 ... search parameter Setting screen 461 ... Detailed detection condition setting dialog 462 ... Basic setting field 463 ... Detailed detection condition setting field 464 ... Option setting field 465 ... Search sensitivity setting field 466 ... Search accuracy setting field 467 ... Search model individual specification field; 467b ... Search Model selection field 468 ... Label order designation field 469 ... Determination label designation field 470 ... Search parameter setting screen 471 ... "OK" button 480 ... Search model registration method selection screen 481 ... "OK" button 490 ... Search model registration screen 491 ... CAD Data reading dialog 492 ... File list Column 493 ... "Execute" button 510 ... Search model registration screen 511 ... Registered model selection screen 512 ... "Next" button 520 ... Search model registration screen 521 ... Rotation symmetry designation dialog 522 ... "Register" button 530 ... Search model registration Screen 531 ... "Edit" button 540 ... Search result display screen 541 ... Search result list display field 542 ... Switch button 543 ... Number of screens 550 ... Search result display screens 560, 570, 580, 590, 610 ... Search result display screen 620 ... Model edit screen 621 ... Model area setting field 622 ... Feature setting field 623 ... Model registration field 624 ... Detailed setting button 625 ... Detailed setting button 630 ... Model area detailed setting screen 631 ... Mask area specifying field 632 ... Weighting area specifying field 640 ... Detailed feature setting screen 641... Detailed feature setting field 642. Feature extraction interval designation field 643 ... Edge extraction threshold value setting field 650 ... Three-dimensional pick initial setting screen 651 ... Calibration data selection field 652 ... Detection tool setting field 653 ... Hand registration field 654 ... "Create hand model" button 655 ... 3D Pick guidance field 660 ... End effector model setting screen 661 ... Hand model list display field 662 ... "Add" button 670 ... End effector model edit screen 671 ... Parts list display field 672 ... "Add" button 673 ... Tool coordinate specification field 674 ... "OK" button 680 ... Add part dialog 681 ... "Add" button 690 ... CAD position / orientation setting screen 691 ... Position / orientation field 692 ... "OK" button 693 ... Size designation field 710 ... Grasping registration screen 711 ... Search model switching field 712 ... Gripping registration list display field 713 ... "Add" button 714 ... "Copy" button 715 ... "Edit" button 716 ... "Done" button 720 ... Grip setting dialog 721 ... Basic setting field 722 ... Grip label setting field 723 ... Hand selection field 724 ... Gripping position coordinate designation field 725 ... Coordinate orientation designation field 726 ... "Easy setting navigation" button 727 ... Movement amount setting field 730 ... Setting method selection dialog 731 ... "OK" button 740 ... XY designation screen 741 ... XY coordinate designation field 742 ... "Next" to "button 750 ... Z-R _Z designation screen 751 ... Z-R _Z designation column 752 ... Z designation column 753 ... R _Z designation column 754 ... angle selection button 755 ..." next "button 760 ... R _Y specifying screen 761 ... R _Y designation field 762 ... "Next" button 770 ... R _X designation screen 771 ... R _X designation field 772 ... "Done" button 780 ... Condition verification screen 781 ... Detection condition setting field 782 ... Detection number designation field 783 ... Hand tilt angle upper limit designation field 784 ... Margin setting field 785 ... Verification field 786 ... Detection number display field 787 ... Display label designation field 788 ... "Verification for each workpiece" button 789 ... Detailed detection condition setting Button 790 ... Verification dialog 791 ... Target work selection field 792 ... Detection search model display field 793 ... "Grasping confirmation" button 810 ... Detection result display dialog 811 ... Grasping solution candidate display field 820 ... Detection condition detail setting screen 821 ... Grasping Position detection condition setting field 822 ... End effector / box / floor interference determination setting field 823 ... End effector / three-dimensional point group interference determination setting field 830 ... Place setting screen 831 ... Tool coordinate setting field 832 ... Place position Registration fields 840, 850 ... Deviation correction screen 841 ... End effector setting field 842 ... "OK" Button 843 ... "Automatic correction" button TAX ... Tool coordinate axis BAX ... Reference coordinate axis RAX ... Rotated coordinate axis HBP ... Grip reference point EET, EET2, EET3 ... End effector ARM ... Arm WK, WK6, WK7, WK10 ... Work WK2 ... Cavity Workpiece WK3 ... Plate-like workpiece WK9 ... Flat plate-like workpiece WKC ... Cubic-like workpiece CWM ... Three-dimensional CAD data work model CDM ... CAD model WM9 ... Flat plate-like workpiece model WM10, WM11 ... Work model WM11F ... Plan view Work model WM11P displayed in a shape ... Work model WMR displayed in a perspective view ... Work model WKI when registering a search model ... Work ADA obtained as an input image ... Additional area FLS ... Flange surface VP ... Cross product vector WKI ' ... Rotated input image IBX Virtual rectangular parallelepiped MWM ... Measured data BDI ... Basic direction image SM ... Search model SMA ... Search model P1 ... Grip specified position SCP ... Feature point OCP ... Feature point EEM, EM10 ... End effector model PRJ ... Projector CME1, CME2, CME3, CME4 ... Camera PC ... Point group BSL ... Basic axes SP1-SP5 ... Cross-section positions SS1-SS5 ... Cross-section TDP ... Three-dimensional point PP3 ... Projection point RBT ... Robot BX ... Storage container STG ... Stage AXX ... Correction rotation X Axis AXY ... correction rotation Y-axis AXZ ... correction rotation Z-axis

Claims (12)

An image processing apparatus for controlling a robot that measures a three-dimensional shape of a plurality of workpieces stacked in a work space by a sensor unit and performs a picking operation held by an end effector provided at a tip of a robot arm unit. ,
A work model registration unit for registering a work model indicating a three-dimensional shape of the work;
An end effector model registration unit for registering an end effector model, which is three-dimensional CAD data, which virtually represents the three-dimensional shape of the end effector;
For setting the position and posture of the end effector and the gripping position of the work model when the work model registered by the work model registration unit is gripped by the end effector registered by the end effector model registration unit. A gripping position specifying part;
An additional model creating unit for creating an additional model in which an additional region expressed by one or more predefined three-dimensional figures is added to the surface of the end effector model;
When the gripping position of the work model set by the gripping position specifying unit is gripped by the position and posture of the end effector set by the gripping position specifying unit, the end registered by the end effector model registration unit An interference determination unit for performing an interference determination to determine whether the effector model and the additional model created by the additional model creation unit interfere with other objects that may interfere with the gripping operation;
An image processing apparatus comprising: The image processing apparatus according to claim 1,
The additional model creation unit is an image processing apparatus configured so that a rectangular parallelepiped or a cylinder can be selected as a predefined figure. The image processing apparatus according to claim 1, wherein:
The additional model creation unit is an image processing apparatus that can configure an additional region by combining a plurality of predefined figures. The image processing apparatus according to any one of claims 1 to 3,
The additional model creation unit is an image processing apparatus configured such that the additional region can be inserted between a flange surface at a tip of an arm unit and an end effector model. The image processing apparatus according to any one of claims 1 to 4,
The additional model creation unit is an image processing apparatus in which the additional region can be configured in a columnar shape or a prismatic shape extending in one direction. The image processing apparatus according to claim 1, further comprising:
An input image acquisition unit that acquires an input image having a three-dimensional shape from an image including a plurality of workpieces measured by the sensor unit and objects around the workpiece;
When the interference determination unit moves the end effector to the position in an attempt to grip any of the plurality of workpieces included in the input image acquired by the input image acquisition unit, an object around the workpiece is moved to the end effector. In order to determine whether or not to interfere with each other, an image processing apparatus configured to perform interference determination on an image constituting an object around the workpiece in the input image. The image processing apparatus according to claim 6, further comprising:
A virtual representation of the three-dimensional shape of the workpiece used when performing a three-dimensional search for specifying the posture and position of each workpiece for a plurality of workpiece groups included in the input image acquired by the input image acquisition unit. A search model registration unit for registering a search model,
For performing a three-dimensional search for specifying an image region corresponding to the position and orientation of each workpiece using the search model registered in the search model registration unit from the input images acquired in the input image acquisition unit. A three-dimensional search unit;
With
The interference determination unit is specified by the grip position specifying unit to grip the workpiece with respect to a search result corresponding to the position and posture of each workpiece searched from the input image by the three-dimensional search unit. An image processing apparatus configured to determine whether or not there is interference with an object existing around a workpiece when an end effector model is disposed at a gripping position. The image processing apparatus according to any one of claims 1 to 7,
An image processing apparatus in which the end effector model is composed of STL data. An image processing apparatus according to any one of claims 1 to 8,
An image processing apparatus in which the input image is composed of a set of three-dimensional point cloud data. An image processing method for controlling a robot that measures a three-dimensional shape of a plurality of workpieces stacked in a work space by a sensor unit and performs a picking operation that is gripped by an end effector provided at a tip of a robot arm unit,
Registering an end effector model that is three-dimensional CAD data that virtually represents the three-dimensional shape of the end effector;
The registered end effector sets the position and orientation of the end effector and the gripping position of the workpiece model when gripping the workpiece model indicating the three-dimensional shape of the workpiece, and at least one predetermined three-dimensional shape. Creating an additional model in which an additional area expressed by a simple figure is added to the surface of the end effector model;
When the gripping position of the set work model is gripped by the set position and posture of the end effector, the registered end effector model and the created additional model may interfere with the gripping operation. Determining whether to interfere with other sexual objects;
An image processing method including: An image processing program for controlling a robot that performs a picking operation by measuring a three-dimensional shape of a plurality of workpieces stacked in a work space with an end effector provided at a tip of a robot arm. ,
A function for registering a workpiece model indicating the three-dimensional shape of the workpiece;
A function of registering an end effector model which is three-dimensional CAD data for virtually expressing the three-dimensional shape of the end effector;
A function of setting the position and orientation of the end effector and the gripping position of the work model when gripping the work model indicating the three-dimensional shape of the work by the registered end effector;
A function of creating an additional model in which an additional region expressed by one or more three-dimensional figures defined in advance is added to the surface of the end effector model;
When the gripping position of the set work model is gripped by the set position and posture of the end effector, the registered end effector model and the created additional model may interfere with the gripping operation. A function to determine whether or not to interfere with other objects
Processing method for realizing the above in a computer.   The computer-readable recording medium which recorded the program of Claim 11, or the apparatus memorize | stored.