JP6877191B2 - Image processing equipment, image processing methods, image processing programs and computer-readable recording media - Google Patents

Description

The present invention is measured, the image processing apparatus, image processing method, a readable recording medium body by the image processing program and a computer, a three-dimensional shape of a plurality of workpieces are picked for which e.g. piled up in the working space by the sensor unit and, an image processing unit for controlling the robot to perform the picking operation for gripping the end effector provided at the tip of the arm of the robot, image processing method, and a readable recording medium body by the image processing program and a computer.

By combining robot vision with a manipulator, the target work is imaged with robot vision to acquire height information, then an appropriate position is grasped (picked) and placed at a desired position (placement or placement). ) Possible robot devices have been developed. Using such a robot device, a large number of workpieces placed in a returnable box, which is called bulk picking, are imaged by the cameras and sensors that make up the robot vision to grasp the posture and grasp the appropriate gripping position. After that, the robot arm is moved to this gripping position, gripped by an end effector such as a hand part provided at the tip of the arm, and placed in a fixed position outside the return box. It has been.

In bulk picking using such robot vision, the positional relationship between the work and the robot when gripping the work is registered as the gripping position at the time of setting, and the robot grips the work detected by the robot vision during actual operation. The position is calculated, and the robot is moved to the calculated position for picking.

Here, as a method of registering the gripping position of the work by the end effector, there is a method of moving and registering the end effector model in the virtual space represented by the three-dimensional CAD data. At this time, since the gripping position is registered in the CAD space, if the mounting state of the end effector set on the robot vision side and the mounting state of the actual end effector are different, when gripping using an actual robot, , There was a problem that a deviation would occur.

Japanese Patent No. 3782679 Japanese Patent No. 4962123

The present invention has been made in view of such circumstances, and one of the purposes thereof is an image processing device, an image processing method, and an image processing device so that the mounting state of an end effector can be easily reflected on the robot vision side. and to provide a readable recording medium body by the image processing program and a computer.

Means for Solving Problems and Effects of Invention

According to the image processing apparatus according to the first aspect of the present invention, the three-dimensional shape of a plurality of works stacked in the work space is measured by a sensor unit arranged at a fixed position capable of imaging the work space. , An image processing device for controlling a robot that performs a picking operation that is gripped by an end effector provided at the tip of the arm of the robot with a robot controller, and is a tertiary that virtually expresses the three-dimensional shape of the end effector. An end effector model registration unit for registering an end effector model that is original CAD data, a display unit for displaying the end effector model in a three-dimensional manner in a virtual three-dimensional space, and a display unit on the display unit. Ri virtual three-dimensional space der displayed in the position and orientation of the end effector calculated in the coordinate system of the vision space of the captured end effector by the sensor unit, the robot controller of a robot space for operating the end effector a calibration unit you get the calibration information for coordinate transformation and the position and orientation of the coordinate system, based on the calibration information obtained by the calibration unit, coordinates of the vision space and coordinates of the robot space A conversion unit that converts coordinates between the system and an input image acquisition unit that acquires an input image having a three-dimensional shape from an image including an end effector measured by the sensor unit, and an input image acquisition unit that acquires the input image. From the input image, the end effector model is used as a search model, and the three-dimensional search unit for specifying the image area corresponding to the position and orientation of the end effector, and the end effector in the coordinate system of the robot space By reflecting the difference between the position and orientation of the attached flange portion and the position and orientation of the end effector in the position and orientation of the end effector model registered by the end effector model registration unit, the end effector model can be used. It is provided with an end effector mounting position calibration unit for calibrating the position and orientation , and the conversion unit acquires the position and orientation of the flange portion in the coordinate system of the robot space, converts it into the coordinate system of the vision space, and converts the end The effector mounting position calibration unit includes the position and orientation of the flange portion in the vision space coordinate system converted by the conversion unit and the end effector in the vision space coordinate system detected by searching with the three-dimensional search unit. The difference between the position and the posture is calculated, and the calculated difference is used as described above. By reflecting the position and orientation of the end effector model, you to calibrate the end effector model registered by the end effector model registration unit in the state mounted in position and orientation offset said difference from the flange portion it can. With the above configuration, by performing a three-dimensional search using the end effector model, it is possible to correct the error in the virtual end effector model according to the actual mounting state of the end effector, and more accurate setting work can be performed. You will be able to do it.

Further, according to the image processing apparatus according to the second side surface, the three-dimensional shape of a plurality of works stacked in the work space is measured by a sensor unit arranged at a fixed position capable of imaging the work space. An image processing device for controlling a robot that performs a picking operation that is gripped by an end effector provided at the tip of the arm of the robot with a robot controller, and is a three-dimensional image processing device that virtually expresses the three-dimensional shape of the end effector. An end effector model registration unit for registering an end effector model that is CAD data, a display unit for displaying the end effector model in a three-dimensional manner in a virtual three-dimensional space, and a display unit on the display unit. It is a virtual three-dimensional space that is displayed, and the position and orientation of the end effector calculated by the coordinate system of the vision space where the end effector is imaged by the sensor unit, and the coordinate system of the robot space in which the robot controller operates the end effector. A calibration unit that acquires calibration information for coordinating the position and orientation of the robot, and a coordinate system in the vision space and a coordinate system in the robot space based on the calibration information obtained by the calibration unit. Among the input image acquired by the input image acquisition unit, the conversion unit that converts the coordinates between the two, the input image acquisition unit that acquires the input image having a three-dimensional shape from the image including the end effector measured by the sensor unit, and the input image acquisition unit. Therefore, using the end effector model as a search model, a three-dimensional search unit for performing a three-dimensional search for specifying an image area corresponding to the position and orientation of the end effector, and a flange to which the end effector is attached in the coordinate system of the robot space. By reflecting the difference between the position and orientation of the unit and the position and orientation of the end effector in the position and orientation of the end effector model registered by the end effector model registration unit, the position and orientation of the end effector model can be reflected. The conversion unit is provided with an end effector mounting position calibration unit for calibrating the robot, and the conversion unit acquires the position and orientation of the end effector in the coordinate system of the vision space, converts the position and orientation of the end effector into the coordinate system of the robot space, and converts the end effector mounting position into the coordinate system of the robot space. The calibration unit calculates the difference between the position and orientation of the end effector in the coordinate system of the robot space converted by the conversion unit and the position and orientation of the flange portion in the coordinate system of the robot space, and the calculated difference. Is reflected in the position and orientation of the end effector model. By doing so, the end effector model registered by the end effector model registration unit can be calibrated so that it is attached at a position and posture slightly deviated from the flange portion.
Further, according to the image processing apparatus according to the third side surface, in addition to any of the above configurations, the input image acquisition unit determines the mounting position where the end effector is attached to the flange portion at the tip of the arm portion of the robot. It can be configured to acquire the including input image.

Further, according to the image processing apparatus according to the other aspect, in addition to any of the above configurations, the input image acquisition unit can be configured to acquire the input image in a posture in which the area of the end effector becomes large.

Furthermore , according to the image processing apparatus according to the other side surface, in addition to any of the above configurations, the input image acquisition unit can be configured to acquire the input image from the side surface with the end effector in a horizontal posture.

Furthermore , according to the image processing apparatus according to the fourth aspect, in addition to any of the above configurations, a storage unit for storing calibration information by the calibration unit is further provided, and the conversion unit has a storage unit. It can be configured to read the calibration information stored in the storage unit and convert the coordinate position.

Furthermore , according to the image processing apparatus according to the fifth aspect, in addition to any of the above configurations, the calibration information is the position of the coordinate system in the vision space calculated by the calibration unit and the position of the coordinate system. A conversion formula for converting the posture into the position and posture of the coordinate system in the robot space can be included.

Furthermore , according to the image processing apparatus according to the sixth aspect, in addition to any of the above configurations, a work model registration unit for registering a work model that virtually represents the three-dimensional shape of the work is provided. With respect to the work model registered in the work model registration unit, the grip position specifying unit for specifying one or more gripping positions for gripping the work model in the end effector model, and the input image acquisition unit acquire the work model. To register a search model that virtually represents the three-dimensional shape of the work, which is used when performing a three-dimensional search that specifies the posture and position of each work for a plurality of work groups included in the input image. It is provided with a search model registration unit, and the three-dimensional search unit performs a three-dimensional search from the input images acquired by the input image acquisition unit using the search model registered by the search model registration unit. , It can be configured to specify the image area corresponding to the position and orientation of each work. With the above configuration, when teaching the gripping position using the end effector model, it is possible to correct the error between the actual end effector and the virtual end effector model, so that more accurate setting work can be performed. become.

Furthermore, according to the image processing apparatus according to the seventh aspect, in addition to any of the above configurations, the end effector mounting position calibration unit calibrates the grip position specified by the grip position specifying unit. A three-dimensional pick determination unit for determining whether or not the work can be gripped by the end effector model is provided.
Furthermore, according to the image processing apparatus according to the eighth side surface, in addition to any of the above configurations, when the work model is gripped at the gripping position specified by the gripping position specifying portion, the work model is further described. An interference determination unit for determining whether or not the end effector model calibrated by the end effector mounting position calibration unit interferes with another work can be provided.
Furthermore, according to the image processing apparatus according to the ninth aspect, in addition to any of the above configurations, the basic axis of the end effector can be set, and the interference determination unit is the end effector mounting position calibration unit. A plurality of cross-section models can be generated by cutting the end effector model configured by the above in a plane intersecting the basic axis of the end effector, and interference determination can be made using each of the plurality of cross-section models.
Furthermore , according to the image processing device related to the other side surface, in addition to any of the above configurations, the three-dimensional shape of a plurality of workpieces stacked in the work space is measured by the sensor unit, and the robot controller is used to measure the three-dimensional shape of the robot. An image processing device for controlling a robot that performs a picking operation that is gripped by an end effector provided at the tip of the arm, and is an end effector that is three-dimensional CAD data that virtually represents the three-dimensional shape of the end effector. An image including an end effector model registration unit for registering a model, a display unit for displaying the end effector model in a three-dimensional shape in a virtual three-dimensional space, and an end effector measured by the sensor unit. An input image acquisition unit that acquires an input image having a three-dimensional shape from the robot, a mounting position of the end effector on the tip of the robot arm, and a position of the end effector model on the virtual three-dimensional space on the display unit. It may be provided with an end effector mounting position calibration unit for calibrating the error.

Furthermore , according to the image processing method according to the tenth aspect, the three-dimensional shapes of a plurality of works stacked in the work space are measured by a sensor unit arranged at a fixed position capable of imaging the work space. A three-dimensional CAD that virtually expresses the three-dimensional shape of an end effector, which is an image processing method that controls a robot that performs a picking operation that is gripped by an end effector provided on the flange at the tip of the robot with a robot controller. a step of registering the end effector model is data, the end effector model, Ri virtual three-dimensional space der on the display unit, the three-dimensional shape in the coordinate system of the vision space of the captured end effector sensor unit Includes a step of acquiring calibration information for coordinate conversion between the position and orientation to be displayed and the position and orientation of the coordinate system in the robot space in which the robot controller operates the end effector, and the end effector measured by the sensor unit. A three-dimensional search that specifies an image area corresponding to the position and orientation of the end effector using the end effector model as a search model from the process of acquiring an input image having a three-dimensional shape from the image and the acquired input image. The step of detecting the position and orientation of the end effector in the coordinate system of the vision space by the three-dimensional search , and the step of acquiring the position and orientation of the flange portion to which the end effector is attached in the coordinate system of the robot space. The step of converting the position and orientation of the flange portion in the acquired coordinate system of the robot space into the coordinate system of the vision space, the position and orientation of the flange portion in the coordinate system of the converted vision space, and the above. By calculating the difference from the position and orientation of the end effector in the detected coordinate system of the vision space and reflecting the calculated difference in the position and orientation of the end effector model, the end effector model can be transferred from the flange portion. It can include a step of calibrating the mounted state in the position and the posture deviated by the difference. As a result, by performing a three-dimensional search using the end effector model, it is possible to correct the error in the virtual end effector model according to the actual mounting state of the end effector, and more accurate setting work can be performed. Will be.

Furthermore , according to the image processing program related to the eleventh side surface, the three-dimensional shapes of a plurality of works stacked in the work space are measured by a sensor unit arranged at a fixed position capable of imaging the work space. , An image processing program for controlling a robot that performs a picking operation that is gripped by an end effector provided on the flange at the tip of the robot with a robot controller, and is a tertiary that virtually expresses the three-dimensional shape of the end effector. a function of registering the end effector model is based on CAD data, the end effector model, Ri virtual three-dimensional space der on the display unit, three-dimensional coordinate system of the vision space of the captured end effector sensor unit the position and orientation are displayed on Jo, the function of the robot controller to acquire calibration information for the position and the coordinate transformation the orientation of the coordinate system of the robot space for operating the end effector, the end effector which is measured by the sensor unit A function of acquiring an input image having a three-dimensional shape from an image including the above, and a tertiary that specifies an image area corresponding to the position and orientation of the end effector from the acquired input image using the end effector model as a search model. The function of performing the original search, the function of detecting the position and orientation of the end effector in the coordinate system of the vision space, and the position and orientation of the flange to which the end effector is attached are acquired in the coordinate system of the robot space by the three-dimensional search. a function of the position and orientation of the flange portion in the coordinate system of the obtained robot space, and a function of converting the coordinate system of the vision space, the position and orientation of the flange portion in the coordinate system of the converted vision space By calculating the difference from the position and orientation of the end effector in the detected coordinate system of the vision space and reflecting the calculated difference in the position and orientation of the end effector model, the end effector model is flanged. It is possible to realize a function of calibrating the mounted state in a position and posture deviated from the unit by the computer. With the above configuration, by performing a three-dimensional search using the end effector model, it is possible to correct the error in the virtual end effector model according to the actual mounting state of the end effector, and more accurate setting work can be performed. You will be able to do it.

Furthermore, a computer-readable recording medium according to the twelfth embodiment is intended for storing the image processing program. Recording media include CD-ROM, CD-R, CD-RW, flexible disc, magnetic tape, MO, DVD-ROM, DVD-RAM, DVD-R, DVD + R, DVD-RW, DVD + RW, Blu-ray, HD. A medium capable of storing a magnetic disk such as a DVD (AOD), an optical disk, a magneto-optical disk, a semiconductor memory, or other programs is included. Further, the program includes a program stored in the above-mentioned recording medium and distributed, and a program distributed by download through a network line such as the Internet. Further, the stored device includes a general-purpose or dedicated device in which the above program is implemented in a state in which it can be executed in the form of software, firmware, or the like. Furthermore, each process and function included in the program may be executed by program software that can be executed by a computer, and each part of the process may be executed by hardware such as a predetermined gate array (FPGA, ASIC), or program software. It may be realized in a form in which and a partial hardware module that realizes a part of the hardware are mixed.

It is a schematic diagram which shows the state of performing a bulk picking operation using a robot system. It is a block diagram of a robot system. It is a perspective view which shows an example of a sensor part. FIG. 4A is a schematic cross-sectional view showing an example in which workpieces are randomly placed in a storage container and stacked, FIG. 4B is a schematic sectional view showing an example in which workpieces are stacked on a floor surface, and FIG. 4C is a tray in which the workpieces are placed in a constant posture. It is a perspective view which shows the state arranged above. FIG. 5A is a schematic view showing an example of gripping a work with an end effector, FIG. 5B is a schematic view showing an example of gripping a work having a cavity from the inner surface, and FIG. 5C shows an example of sucking and gripping a plate-shaped work. It is a schematic diagram. It is a block diagram which shows the robot system which concerns on Embodiment 1. FIG. It is a perspective view which shows the work model constructed by 3D CAD data. It is an image diagram which shows the state which sets the origin of the search model with respect to the work model of FIG. 9A is a basic direction image of the work model of FIG. 7 viewed from the positive direction of the X axis, FIG. 9B is a basic direction image of the work model viewed from the positive direction of the Y axis, and FIG. 9C is a basic direction image of the work model viewed from the positive direction of the Z axis. , FIG. 9D is a basic direction image viewed from the negative direction of the Z axis. It is an image diagram which shows the search model registration screen which registers a basic direction image as a search model of 3D search. It is an image diagram which shows the actual measurement data which image | imaged the work corresponding to the work model of FIG. 7 from the positive direction of the X axis. FIG. 12A is an image diagram showing a state in which feature points are extracted from an image viewed from the X-axis direction of FIG. 9A, and FIG. 12B is an image diagram showing a state in which the image of FIG. 12A is three-dimensionally displayed. 13A is an input image in which the work group is displayed in two dimensions, FIG. 13B is an input image in which FIG. 13A is displayed in three dimensions, FIG. 13C is a result of performing a three-dimensional search on FIG. 13A, and FIG. 13D is FIG. 13C. It is an image diagram which shows each 3D display state. It is an image diagram of the user interface screen which shows the grip registration screen. It is an image figure which shows the gripping posture addition screen. 16A is an image diagram showing a state in which a plurality of cubic workpieces are stacked separately, FIG. 16B is a state in which this work model is registered in a search model, and FIG. 16C is an image diagram showing a state in which a gripping posture is registered with respect to the work model of FIG. 16B. Is. It is a functional block diagram which shows the robot system which concerns on Embodiment 2. It is an image diagram which shows the state before rotating the end effector model according to the ZZ Euler angle. It is an image diagram which shows the state rotated by 90 ° around the Z axis from FIG. It is an image diagram which shows the state rotated by 90 ° around the Y axis from FIG. It is an image diagram which shows the state rotated by 90 ° around the X axis from FIG. It is an image diagram which shows the rotation axis of _RY and R _{Z in} the state of FIG. 21. It is an image diagram which shows the state which displays the correction rotation axis of R _{Z in the state of FIG.} FIG. 22 is an image diagram showing a state in which the correction rotation axis of _{RY is displayed in the state of FIG. 22.} In the state of FIG. 22 is an image diagram showing a state of displaying the corrected rotational shaft of R _X. It is a block diagram which shows the robot system which concerns on Embodiment 3. It is a flowchart which shows the procedure of the teaching work performed before the actual operation. It is a flowchart which shows the procedure of registering 3D CAD data as a search model. It is a flowchart which shows the procedure of registering an end effector model. It is a flowchart which shows the procedure of registration of the end effector model which added the additional area. It is a block diagram which shows the robot system which concerns on Embodiment 4. It is an image diagram which shows the additional area setting screen. It is an image diagram which shows the basic figure edit screen. It is a flowchart which shows the procedure of registering the gripping position and the posture which grips a work by an end effector. FIG. 35A is an image diagram of a work model composed of three-dimensional CAD data, and FIG. 35B is an image diagram showing an XY designation screen. It is an image diagram which shows the Z-R _{Z designation screen.} It is an image diagram which shows the _{RY designation screen.} It is an image diagram which shows the R _{X designation screen.} It is an image diagram which shows the position parameter specification screen. It is an image diagram which shows the XYZ designation screen. FIG. 3 is an image diagram showing a state in which a gripping position is specified in a state where the work model of FIG. 35B is projected from an oblique direction. It is an image diagram which shows the state of designating a gripping position on a three-dimensional image viewer. It is an image diagram which shows a flat plate-shaped work. 44A is a plan view of a work model representing the work of FIG. 43, FIG. 44B is a bottom view, FIG. 44C is a front view, FIG. 44D is a rear view, FIG. 44E is a right side view, and FIG. 44F is a left side view. It is an image diagram which shows the result of having performed 3D search on the point cloud data which imaged the work group which piled up a plurality of works of FIG. 43 separately. It is a schematic diagram which shows the example which expresses the posture of a work using Euler angles of ZZ system. It is an image diagram which shows the search model registration screen. It is a flowchart which shows the registration procedure of the search model which provided the posture restriction. It is an image diagram which shows the tilt angle / rotation angle setting screen. FIG. 50A is an image diagram showing the posture of the work at the time of registration, FIG. 50B is an image diagram showing the posture of the input image, and FIG. 50C is an image diagram showing an inclination angle. 51A is an image diagram showing the posture of the work at the time of registration, FIG. 51B is an image diagram showing the posture of the input image, and FIG. 51C is an image diagram showing a rotation angle. It is a flowchart which shows the procedure which obtains a tilt angle and a rotation angle from a three-dimensional posture. FIG. 53A is an image diagram showing a posture of the work when the search model is registered, FIG. 53B is an image diagram showing a posture of an input image, and FIG. 53C is an image diagram showing a state of obtaining an inclination angle. FIG. 54A is an image diagram showing a state in which the rotation axis is shown in the search model of FIG. 53C, and FIG. 54B is an image diagram showing a state in which the Z'axis of FIG. 54A is three-dimensionally rotated so as to coincide with the Z axis. FIG. 55A is an image diagram showing the posture of the work at the time of registration, FIG. 55B is a state excluding the inclination of FIG. 54B, and FIG. 55C is an image diagram showing a state in which the rotation angle when viewed from the Z vertical direction of FIG. 55B is obtained as the rotation angle. It is an image diagram which shows the end effector mounting position setting screen. It is an image diagram which shows the gripping position designation screen. It is an image diagram which shows the multiple gripping position selection screen. It is an image figure which shows the end effector mounting position calibration screen. It is a flowchart which shows the procedure which shows the procedure of automatically calibrating the end effector mounting position. It is an image diagram which shows the end effector imaging screen. It is a flowchart which shows the procedure of registering the measured data as a search model. It is a flowchart which shows the procedure which determines the presence or absence of the gripping solution of each work in the actual operation. It is a flowchart which shows the procedure at the time of the actual operation which performs the picking operation in the state which registered the search model by the procedure of FIG. 48. It is a flowchart which shows the procedure of the interference determination using a cross-section model. It is an image diagram which shows the end effector model and its basic axis and cross-sectional position. 67A is an image diagram showing a cross section SS1 of FIG. 66, FIG. 67B is a cross section SS2, FIG. 67B is a cross section SS3, FIG. 67D is a cross section SS4, and FIG. 67E is an image diagram showing a cross section SS5. FIG. 68A is an image diagram showing a three-dimensional point and the basic axis of the end effector model of FIG. 66, FIG. 68B is a state in which the projection point of the three-dimensional point of FIG. 68A does not interfere with the cross section SS3, and FIG. 68C interferes with the cross section SS3. It is an image diagram which shows each state. It is a flowchart which shows the procedure of the interference determination using an additional model. It is a block diagram which shows the robot system which concerns on Embodiment 7. FIG. 5 is a flowchart showing a procedure for determining the presence or absence of a gripping solution for each work during actual operation according to the seventh embodiment. It is a perspective view which shows an example of a work. 73A is a height image A of the work of FIG. 72 viewed from the positive direction side of the Z axis, FIG. 73B is a height image B viewed from the negative direction side of the Z axis, and FIG. 73C is a height image B viewed from the positive direction side of the X axis. Height image C, FIG. 73D is the height image D seen from the negative direction side of the X axis, FIG. 73E is the height image E seen from the positive direction side of the Y axis, and FIG. 73F is the height image E seen from the negative direction side of the Y axis. It is an image figure which shows each of the seen height image F. It is an image diagram which shows the work selection screen. It is an image diagram which shows the gripping solution candidate display screen which selected gripping OK. It is an image diagram which shows the gripping solution candidate display screen which selected the gripping NG. It is an image diagram which shows still another example of the gripping solution candidate display screen which selected gripping NG. It is an image diagram which shows the example of the gripping solution candidate display screen for a search model C. FIG. 78 is an image diagram showing an example of a gripping solution candidate display screen in which another gripping position candidate is selected in FIG. 78. It is an image diagram which shows the example of the gripping solution candidate display screen which added the gripping position and became gripping OK. It is a block diagram which shows the robot system which concerns on Embodiment 8. It is a flowchart which shows the registration procedure of the search model which concerns on Embodiment 8. 83A is a state in which the gripping posture is registered in the basic direction image of FIG. 73A, FIG. 83B is a state in which the gripping posture is registered in the basic direction image of FIG. 73B, and FIG. 83C is a state in which the gripping posture is registered in the basic direction image of FIG. 73C. 83D is a state in which the gripping posture is registered in the basic direction image of FIG. 73D, FIG. 83E is a state in which the gripping posture is registered in the basic direction image of FIG. 73E, and FIG. 83F is a state in which the gripping posture is registered in the basic direction image of FIG. 73F. It is an image diagram which shows each state. It is an image diagram which shows the work selection screen. It is an image figure which shows the gripping solution candidate display screen which selected the gripping posture C-000. It is an image figure which shows the gripping solution candidate display screen which selected the gripping posture A-000. It is a flowchart which shows the procedure which performs 3D search and graspability determination at the time of the actual operation which concerns on Embodiment 8. It is a block diagram which shows the robot system which concerns on Embodiment 9. It is an image diagram which shows the function selection screen. It is an image diagram which shows the search model registration method selection screen. It is an image figure which shows the actual measurement data imaging screen. It is an image diagram which shows the search model exclusion area setting screen. It is an image diagram which shows the rotation symmetry setting screen. It is an image diagram which shows the search model registration screen. It is an image diagram which shows the search model registration screen which added the search model. It is an image diagram which shows the search area setting screen. It is an image diagram which shows the search area setting screen which displayed the floor surface designation dialog. It is an image diagram which shows the search area setting screen which displayed the floor surface information. It is an image diagram which shows the search parameter setting screen. It is an image diagram which shows the search parameter setting screen which displayed the detection detailed condition setting dialog. It is an image diagram which shows the search parameter setting screen specified for each search model. It is an image diagram which shows the search model registration method selection screen. It is an image diagram which shows the CAD data reading screen. It is an image diagram which shows the search model registration screen based on 3D CAD data. It is an image diagram which shows the search model registration screen which displayed the rotation symmetry designation dialog. It is an image diagram which shows the model list display screen. It is an image diagram which shows the search result display screen. It is an image diagram which shows the search result display screen. It is an image diagram which shows the search result display screen. It is an image diagram which shows the search result display screen. It is an image diagram which shows the search result display screen. It is an image diagram which shows the search result display screen. It is an image diagram which shows the search result display screen. It is an image diagram which shows the model edit screen. It is an image diagram which shows the model area detailed setting screen. It is an image diagram which shows the feature detailed setting screen. It is an image diagram which shows the 3D pick initial setting screen. It is an image diagram which shows the end effector model setting screen. It is an image diagram which shows the end effector model edit screen. It is an image diagram which shows the part addition screen. It is an image figure which shows the CAD position posture setting screen. It is an image figure which shows the CAD position posture setting screen. It is an image figure which shows the CAD position posture setting screen. It is an image diagram which shows the end effector model edit screen. It is an image figure which shows the additional area position position setting screen. It is an image diagram which shows the end effector model edit screen. It is an image diagram which shows the end effector model edit screen. It is an image diagram which shows the end effector model edit screen. It is an image diagram which shows the end effector model edit screen. It is an image diagram which shows the end effector model edit screen. It is an image diagram which shows the grip registration screen. It is an image figure which shows the gripping setting dialog. It is an image figure which shows the setting method selection dialog. It is an image diagram which shows the XY designation screen. It is an image diagram which shows the Z-R _{Z designation screen.} It is an image diagram which shows the Z-R _{Z designation screen.} It is an image diagram which shows the Z-R _{Z designation screen.} It is an image diagram which shows the _{RY designation screen.} It is an image diagram which shows the _{RY designation screen.} It is an image diagram which shows the R _{X designation screen.} It is an image diagram which shows the R _{X designation screen.} It is an image diagram which shows the grip registration screen. It is an image diagram which shows the grip registration screen. It is an image diagram which shows the condition verification screen. It is an image diagram which shows the verification dialog. It is an image figure which shows the detection result display dialog. It is an image figure which shows the detection result display dialog. It is an image diagram which shows the detection condition detailed setting screen. It is an image diagram which shows the place setting screen. It is a flowchart which shows the procedure which performs the deviation correction manually. It is an image diagram which shows the example of the deviation correction screen when the deviation which cannot be ignored exists. It is an image diagram which shows the example of the deviation correction screen when the deviation can be ignored. It is an image diagram which shows the example of the deviation correction screen which has the manual correction of deviation and the automatic correction function.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiments shown below are examples for embodying the technical idea of the present invention, and the present invention is not specified as the following. Further, the present specification does not specify the members shown in the claims as the members of the embodiment. In particular, the dimensions, materials, shapes, relative arrangements, and the like of the components described in the embodiments are not intended to limit the scope of the present invention to the specific description unless otherwise specified, and are merely explanatory examples. It's just that. The size and positional relationship of the members shown in each drawing may be exaggerated to clarify the explanation. Further, in the following description, members of the same or the same quality are shown with the same name and reference numeral, and detailed description thereof will be omitted as appropriate. Further, each element constituting the present invention may be configured such that a plurality of elements are composed of the same member and the plurality of elements are combined with one member, or conversely, the function of one member is performed by the plurality of members. It can also be shared and realized.
(Embodiment 1)

As the first embodiment, FIG. 1 shows a configuration example of a robot system 1000 for picking a work to be picked. In this example, a plurality of work WKs stacked in a work space are sequentially taken out by using a robot RBT, and an example of performing loose-stack picking in which they are arranged at a predetermined position is shown. The robot RBT is also called a manipulator or the like, and includes an arm portion ARM and an end effector HND provided at the tip of the arm portion ARM. The arm portion ARM includes a plurality of movable portions, and moves the end effector EET to a desired position by the angle formed by the two arms and the rotation of the arm fulcrum. The end effector EET can grip or release the work WK.

The operation of this robot RBT is controlled by the robot controller 6. The robot controller 6 controls the movement of the arm portion ARM and the opening / closing operation of the end effector EET. Further, the robot controller 6 acquires information necessary for controlling the robot RBT from the image processing device 100. For example, a work WK, which is a large number of parts randomly placed in a storage container BX, acquires a three-dimensional shape by a sensor unit 2 such as a three-dimensional camera or lighting, and a calculation unit 10 of the image processing device 100 acquires the three-dimensional shape of the work. The position and posture are detected, and the information is sent to the robot controller 6. The robot controller 6 grips the work WKs one by one with the end effector EET provided at the tip of the arm portion ARM of the robot RBT, and arranges them at a predetermined position on the stage STG, for example, on a conveyor belt.

A functional block diagram of the robot system 1000 is shown in FIG. The robot system 1000 shown in this figure includes an image processing device 100, a sensor unit 2, a display unit 3, an operation unit 4, a robot main body 5, a robot controller 6, and a robot operation tool 7.

The operation unit 4 makes settings related to image processing. Further, the sensor unit 2 captures an image of the work and acquires a three-dimensional shape. Further, the display unit 3 confirms the setting and the operating state. Furthermore, the calculation unit 10 performs three-dimensional search, interference determination, calculation of gripping solution, and the like. On the other hand, the robot controller 6 controls the robot according to the result of the calculation unit 10. Further, the robot operating tool 7 sets the operation of the robot. In the example of FIG. 2, the operation unit 4 and the robot operation tool 7 are separate members, but these may be common members.

The sensor unit 2 is a member called a robot vision or the like that images a work space or a work. From the image captured by the sensor unit 2, three-dimensional shape data showing the three-dimensional shape of the workpieces stacked separately is acquired. The methods for acquiring the three-dimensional shape include a pattern projection method, a stereo method, a lens focusing method, an optical cutting method, an optical radar method, an interferometry method, and a TOF method. In this embodiment, the phase shift method is used among the pattern projection methods.

The configuration of the sensor unit 2 is determined according to the three-dimensional shape measurement technique. The sensor unit 2 includes a camera, lighting, a projector, and the like. For example, when measuring the three-dimensional shape of a work by the phase shift method, the sensor unit 2 includes a projector PRJ and a plurality of cameras CME1, CME2, CME3, and CME4 as shown in FIG. The sensor unit may be composed of a plurality of members such as a camera and a projector, or may be integrally configured. For example, a 3D image pickup head in which a camera and a projector are integrated into a head shape can be used as a sensor unit.

Further, the generation of the three-dimensional shape data itself can be performed on the sensor unit side. In this case, an image processing IC or the like that realizes a function of generating three-dimensional shape data is provided on the sensor unit side. Alternatively, the three-dimensional shape data is not generated on the image processing device side, but the raw image captured by the sensor unit is image-processed on the image processing device side to generate three-dimensional shape data such as a three-dimensional image. May be good.

Further, by executing the calibration described later based on the image captured by the sensor unit 2, the actual position coordinates of the work WK (coordinates of the moving position of the end effector EET) are displayed on the display unit 3. It is possible to link with the position coordinates on the image.

The image processing device 100 performs three-dimensional search, interference determination, gripping solution calculation, and the like based on the three-dimensional shape data of the work thus obtained. As the image processing device 100, a general-purpose computer in which a dedicated image processing program is installed, an image processing controller designed exclusively, a robot vision device, and the like can be used. In the example of FIG. 2, an example in which the sensor unit 2, the robot controller 6, and the like are configured by a member separate from the image processing device 100 is shown, but the present invention is not limited to this configuration, for example, the sensor unit and image processing. The device can be integrated, or the robot controller can be incorporated into the image processing device. As described above, the division of the members shown in FIG. 2 is an example, and a plurality of members can be integrated. For example, the operation unit 4 that operates the image processing device 100 and the robot operating tool 7 that operates the robot controller 6 may be common members.

However, the sensor unit 2 is separate from the robot body 5. That is, the present invention targets a form called an off-hand type in which the sensor unit 2 is not provided on the arm unit ARM of the robot body 5. In other words, an aspect called an on-hand type or the like in which the sensor unit is provided on the end effector is not included in the present invention.

The display unit 3 is a member for displaying the three-dimensional shape of the work acquired by the image processing device 100 and confirming various settings and operating states, and a liquid crystal monitor, an organic EL display, a CRT, or the like can be used. .. The operation unit 4 is a member for performing various settings such as image processing, and an input device such as a keyboard or a mouse can be used. Further, by using the display unit 3 as a touch panel, the operation unit and the display unit can be integrated.

For example, when the image processing device 100 is configured by a computer on which an image processing program is installed, a graphical user interface (GUI) screen of the image processing program is displayed on the display unit 3. Various settings can be made from the GUI displayed on the display unit 3, and processing results such as simulation results can be displayed. In this case, the display unit 3 can be used as a setting unit for making various settings.

The robot controller 6 controls the operation of the robot based on the information captured by the sensor unit 2. Further, the robot operating tool 7 is a member for setting the operation of the robot main body 5, and a pendant or the like can be used.

The robot body 5 includes a movable arm portion ARM and an end effector EET fixed to the tip of the arm portion ARM. The robot body 5 is driven by the robot controller 6 to operate the arm portion ARM, picks one work WK, moves it to a desired position, places it, and then releases it. Therefore, the tip of the arm portion ARM is provided with an end effector EET for gripping the work WK. Further, the mounting position on which the work WK is mounted may be, for example, on a tray or a conveyor.

As shown in FIG. 1, a plurality of work WKs are randomly stored in a storage container BX such as a returnable box. The sensor unit 2 is arranged above such a work space. The sensor unit 2 includes a camera and lighting, and the sensor unit 2 can measure the three-dimensional shape of the work WK. The robot controller 6 identifies the work WK to be gripped from a plurality of works based on the three-dimensional shape of the work WK measured by the sensor unit 2, and controls the robot so as to grip the work WK. To do. Then, while holding the work WK, the arm portion ARM is operated to move it to a predetermined mounting position, for example, on the stage STG, and the work WK is mounted in a predetermined posture. In other words, the robot controller 6 identifies the work WK to be picked by the sensor unit 2, grips the work WK with the end effector EET, and places the gripped work WK in a predetermined reference posture. The operation of the robot is controlled so as to be mounted at a certain mounting position and the end effector EET is released.

Here, in the present specification, bulk picking refers to work WK that is randomly stacked in a storage container BX as shown in FIG. 4A by a robot and placed in a predetermined position. An example in which the work WKs stacked in a predetermined area are gripped and placed without using the storage container as shown in FIG. 4B, or the works arranged and stacked in a predetermined posture as shown in FIG. 4C. It is used in the sense of including an example of sequentially grasping and placing WK. Further, it is not always necessary that the workpieces are stacked, and the workpieces randomly placed on a plane without overlapping the workpieces are also referred to as loose stacking in this specification (sequentially). Even if the workpieces are picked and the workpieces do not overlap at the end of the picking, it is still called loose picking). The present invention is not necessarily limited to picking in bulk, and can also be applied to picking workpieces that are not stacked in bulk.

Further, in the example of FIG. 1, the sensor unit 2 is fixed above the work space, but a position where the work space can be imaged is sufficient, and the sensor unit 2 can be arranged at an arbitrary fixed position such as diagonally, laterally, or downward. However, the mode of arranging the sensor unit at a movable indefinite position such as on the arm unit ARM is excluded. Further, the number of cameras and lights included in the sensor unit 2 is not limited to one, and may be plural. Furthermore, the connection with the sensor unit 2, the robot, and the robot controller 6 is not limited to the wired connection, but may be a wireless connection.

Further, gripping the work means sandwiching the outside of the work WK as shown in FIG. 5A, and inserting the claw portion of the end effector EET2 into the inside of the work WK2 having a cavity as shown in FIG. 5B to expand the work. It is used in the sense of including an example of holding by the end effector EET3 which sucks and holds a plate-shaped work WK3 as shown in FIG. 5C. Hereinafter, as an example of gripping the work, a mode of gripping the outer surface of the work from both sides will be described. Further, as shown in FIG. 1, a large number of workpieces are stored in the storage container BX and randomly stacked, and one workpiece is gripped by the end effector EET for each of the plurality of workpieces WK. The setting of the gripping position (teaching work) in the bulk picking operation in which the work of placing the container in the mounting position is repeated will be described below.

When performing the bulk picking operation in the robot system 1000, teaching including the setting for performing the bulk picking operation is performed in advance. Specifically, the part of the work to be gripped by the end effector in what posture, the gripping position, and the like are registered. Such a setting is performed by a robot operating tool 7 such as a pendant.
(Teaching work)

FIG. 6 shows a functional block diagram of a robot system including an image processing device that realizes the function of teaching the gripping position described above. The robot system 1000 shown in this figure includes an image processing device 100, a display unit 3, an operation unit 4, a sensor unit 2, and a robot RBT.

The sensor unit 2 three-dimensionally measures the three-dimensional shape of the work arranged at the working position. The sensor unit 2 is controlled by the sensor control unit 2b. In this example, the sensor control unit 2b is integrally configured with the sensor unit 2, but these may be provided individually. The robot unit includes an arm unit ARM and an end effector EET. This robot is controlled by the image processing device 100 to grip the work at the gripping position and grip it at the gripping position. Here, the state in which the work is gripped in the reference posture and placed in the reference posture is imaged by the sensor unit 2 and registered. Here, the reference posture includes the position and posture of the work.
(Display unit 3)

The display unit 3 virtually represents a work model that virtually represents the three-dimensional shape of the work and an end effector model that is composed of three-dimensional CAD data that virtually represents the three-dimensional shape of the end effector. Each is displayed three-dimensionally in the original space. The height image is an image having height information, and is also called a distance image or the like. Further, the display unit 3 has a six-view display area 3a for displaying the basic direction image of the work model as a six-view view. As a result, each posture of the work model can be displayed in a hexagonal view, and the gripping position can be set for the basic direction image in which the gripping position can be easily set. Will be easy to do.

The operation unit 4 is a member for performing various settings such as image processing, and an input device such as a keyboard or a mouse can be used. Further, by using the display unit 3 as a touch panel, the operation unit and the display unit can be integrated.
(Image processing device 100)

The image processing device 100 of FIG. 6 includes an input image acquisition unit 2c, a storage unit 9, a calculation unit 10, an input / output interface 4b, a display interface 3f, and a robot interface 6b.

The input image acquisition unit 2c acquires an input image having a three-dimensional shape from an image including a plurality of workpieces and objects around them measured by the sensor unit 2. The input image having a three-dimensional shape may be constructed on the sensor unit or the sensor control unit side, or may be constructed on the image processing device side (for example, the input image acquisition unit). Further, in the example of FIG. 6, an input interface is configured for the image processing device 100 to acquire an input image showing the three-dimensional shape acquired by the sensor unit 2. However, the present invention is not limited to this configuration, and an input image captured in advance and held in a storage unit such as a recording medium may be read out and acquired.

The storage unit 9 is a member for grasping various settings, and a non-volatile memory, a hard disk, a storage medium, or the like can be used. The storage unit 9 functions as a gripping position storage unit for storing the gripping position of the work model or the end effector model.

The input / output interface 4b is connected to an input device such as a keyboard or mouse and accepts data input.

The display interface 3f constitutes an output interface with the display unit, and controls to display the image data to be displayed on the display unit constructed by the calculation unit 10.

The robot interface 6b constitutes a communication interface with the robot controller 6.
(Calculation unit 10)

The calculation unit 10 includes a positioning unit 8c, a basic direction image generation unit 8e', a basic direction image selection unit 8e, a search model selection unit 8i, a work model registration unit 8t, a search model registration unit 8g, and an end effector. It includes a model registration unit 8u, a grip position specifying unit 8d, an end effector mounting surface setting unit 8f, a rotation angle limiting unit 8h, a three-dimensional search unit 8k, and a three-dimensional pick determination unit 8l.

The positioning unit 8c is a member for adjusting the position and orientation of the work model displayed on the display unit 3 in the virtual three-dimensional space.

The basic direction image generation unit 8e'sees at least the work model positioned in the virtual three-dimensional space by the positioning unit 8c from the respective axial directions of the three axes orthogonal to each other in the virtual three-dimensional space. It is a member for generating three height images as basic direction images. By automatically generating the basic direction image in this way, the user does not have to manually change the direction of the work and acquire the basic direction image individually, which has an advantage that the labor for registering the gripping position can be saved. ..

The basic direction image selection unit 8e is for selecting one of a plurality of basic direction images whose appearance is different from other basic direction images with respect to at least three basic direction images displayed on the display unit 3. It is a member. In this way, by deleting the surfaces having a common appearance, it is possible to eliminate unnecessary display of the basic direction image and the like, and to further simplify the setting work. For example, for a basic direction image having the same appearance, such as the top surface and the bottom surface of a columnar work, one of them is deleted. The basic direction image selection unit 8e can be configured to allow the user to manually select at least three basic direction images on the display unit 3. Alternatively, the basic direction image selection unit may be configured to automatically extract and select a basic direction image having a common appearance from at least three basic direction images.

The search model selection unit 8i is a member for selecting a basic direction image to be registered as a search model. As will be described later, by sharing the search model used for the three-dimensional search and the model for specifying the gripping position, the search model selection unit 8i, the basic direction image selection unit 8e, and the common image selection unit 8j can be configured. ..

The work model registration unit 8t is a member for registering a work model that virtually represents the three-dimensional shape of the work. Here, for example, the work model registration unit 8t uses three-dimensional point cloud data obtained by imaging an actual work as a work model. In this case, the three-dimensional point cloud data acquired by the sensor unit 2 and the input image acquisition unit 2c is registered as a work model by the work model registration unit 8t. Alternatively, the three-dimensional CAD data representing the shape of the work created separately is read and registered as a work model. In this case, the three-dimensional CAD data input via the input / output interface 4b is registered as a work model by the work model registration unit 8t. Alternatively, three-dimensional CAD data imitating a work may be created and registered. In this case, the work model registration unit 8t realizes a simple three-dimensional CAD function.
(Search model registration unit 8g)

The search model registration unit 8g is used when performing a three-dimensional search for specifying the posture and position of each work for a plurality of work groups included in the input image acquired by the input image acquisition unit 2c. It is a member for registering a search model that virtually represents the original shape. By providing the search model registration unit 8g in this way, the user can save labor in the setting work by registering the search model for performing the three-dimensional search in common with the basic direction image that specifies the gripping position of the work model. .. In addition, even in actual operation, by registering the gripping position of the work for each basic direction image for searching the work that can be gripped, the waste of searching for the basic direction image without the gripping position is eliminated, and the search is performed on the contrary. Since it is possible to examine whether or not the basic direction image can be gripped at the gripping position set in the basic direction image, it is possible to efficiently perform processing.

Further, it is preferable that the search model registration unit 8g is configured to select whether or not to use the basic direction image selected by the basic direction image selection unit 8e as a search model for a three-dimensional search. As a result, it is possible to select whether or not to use the basic direction image as a search model for 3D search, and in other words, unnecessary basic direction images can be excluded from the target of 3D search. Therefore, for example, a plate-shaped workpiece can be used. By eliminating the basic direction image that may be erroneously detected as the image seen from the side, it is possible to set the plate-shaped work to not perform a three-dimensional search in the upright state, which effectively limits the posture of the work model. It can be set easily.

The search model registration unit 8g and the work model registration unit 8t may be provided separately or integrated. For example, in the robot system 1000 of FIG. 6, the search model registration unit 8g and the work model registration unit 8t are integrated as a common model registration unit 8g'. As a result, by registering the model related to one work, it becomes possible to share it for the registration of the gripping position and the registration of the search model for the three-dimensional search, and there is an advantage that the setting can be simplified.

The end effector model registration unit 8u is a member for registering an end effector model in which the three-dimensional shape of the end effector is virtually represented by three-dimensional CAD data.

The end effector mounting surface setting unit 8f displays the end effector model and the mounting surface for mounting the end effector model on the tip of the robot arm on the display unit 3, and displays the mounting surface on the display unit 3. It is a member for defining the attitude of the end effector model with respect to this mounting surface.

The rotation angle limiting unit 8h is for registering any of the basic direction images as a search model for performing a three-dimensional search for specifying the posture and position of each work for a plurality of work groups stacked separately. The search model registration unit 8g and a member for individually setting a range of allowable rotation angles with respect to the rotation of the work model for each selected search model.
(Gripping position specifying part 8d)

The gripping position specifying unit 8d has at least three views from each of the three axes orthogonal to each other in the virtual three-dimensional space with respect to the work model positioned in the virtual three-dimensional space by the positioning unit 8c. This is a member for specifying a gripping position for gripping this work model with an end effector with respect to at least one height image in a state where the basic direction image is displayed on the display unit 3. The gripping position specifying unit 8d includes a work-side gripping location designating unit 8d1, an end effector-side gripping setting unit 8d2, and a relative position setting unit 8d5.

The work-side gripping location designation unit 8d1 displays a plurality of basic direction images selected by the basic direction image selection unit 8e on the display unit 3, and the basic direction image is displayed on the display unit 3 with respect to any of the basic direction images. This is a member for designating a gripping position when gripping the work model shown by the end effector model. The work-side gripping location designation unit 8d1 is configured to be able to register a plurality of gripping positions for each of the plurality of basic direction images.
(End effector side grip setting unit 8d2)

The end effector side grip setting unit 8d2 is a member that sets the end effector model for gripping the work model. The end effector side grip setting unit 8d2 can include a grip reference point setting unit 8d3 and a grip direction setting unit 8d4. The gripping reference point setting unit 8d3 defines a gripping reference point corresponding to the position where the work model is gripped with respect to the end effector model. On the other hand, the gripping direction setting unit 8d4 defines the gripping direction in which the work model is gripped by the end effector model. The gripping reference point setting unit 8d3 and the gripping direction setting unit 8d4 may be common members or may be individually configured. Further, the gripping reference point setting unit 8d3 and the gripping direction setting unit 8d4 can set the gripping reference point and the gripping direction to predetermined values set in advance, respectively. For example, the gripping reference point is set as the center between the claws that sandwich the work, which is provided at the tip of the end effector model. The gripping direction is any coordinate axis that constitutes the tool coordinate axis that defines the end effector model. For example, by setting the Z-axis direction, the end effector model can be moved along the Z-axis direction, that is, the height direction to approach the operation of approaching the work model in order to grip the work model, so that the user can intuitively grasp the model. Alternatively, the gripping reference point setting unit 8d3 and the gripping direction setting unit 8d4 may allow the gripping reference point and the gripping direction to be adjusted by the user.
(Relative position setting unit 8d5)

The relative position setting unit 8d5 moves the end effector model displayed on the display unit 3 until it interferes with the work model, and automatically holds the gripped state in a posture separated from the position where the interference position is reached by a predetermined amount. It is a member for defining. This makes it possible for the user to automatically move the end effector model and bring it into contact with the work to indicate the gripping position without having to manually move the end effector model and bring it into contact with the work. The advantage is that the work can be greatly reduced.
(Registration of gripping position)

In the teaching work, the positional relationship between the work and the end effector when gripping the work is registered as the gripping position. In the following, as a typical example of gripping, an example in which a hand portion is used as an end effector and the work is gripped by the hand portion will be described. In the actual operation of robot picking using robot vision with the gripping position registered, each work is detected by robot vision from the work group in which multiple works are stacked separately, and the position of the detected work and the position of the detected work are displayed. The gripping position on the end effector side is calculated with respect to the posture, and the robot is operated so that the end effector is located at the calculated position for picking. Here, as a method of registering the gripping position, there are a method of actually operating the robot and registering it, and a method of operating and registering the end effector model in a virtual three-dimensional space using three-dimensional CAD. However, the method of actually operating the robot and registering the gripping position requires time-consuming registration work while actually moving the robot, which requires a large-scale verification environment and takes time for trial and error. there were. On the other hand, the method of virtually operating and registering a robot in the space of 3D CAD has the advantage that it can be registered without a robot, but it has a virtual end for the 3D posture of a virtual work model. It was necessary to accurately match the three-dimensional orientation of the effector model, and it was difficult to set the three-dimensional coordinate alignment. In addition, since this method requires three-dimensional CAD data of the work and the end effector, it cannot be set without the three-dimensional CAD data at hand.

Therefore, in the present embodiment, a plurality of height images viewed from each axial direction of the three-dimensional CAD model are displayed as basic direction images, and the basic direction image desired by the user is selected from these, and the selected basic is selected. By setting the gripping position with respect to the direction image, gripping registration in the virtual three-dimensional space can be easily performed. As a result, when the end effector model is operated and registered in the virtual three-dimensional space in which the work model of the three-dimensional CAD data imitating the work is arranged, the basic direction seen from each axial direction that defines the virtual three-dimensional space. By selecting an image and registering the grip, it is possible to register the posture of the end effector model viewed from a substantially vertical direction with respect to each basic direction image, so that the grip can be easily set. .. Further, according to this method, even when there is no three-dimensional CAD data, the data obtained by three-dimensionally measuring the actual work can be used as the basic direction image. Therefore, even if there is no three-dimensional CAD data, grip registration can be easily performed in the virtual three-dimensional space by the same procedure.
(3D pick determination unit 8l)

The three-dimensional pick determination unit 8l grips the gripping direction defined by the gripping direction setting unit 8d4 so that the gripping direction is orthogonal to the work plane representing the posture of the work model displayed in the image display area and is along the gripping direction. It is a member for adjusting the relative position of the end effector model and the work model so as to position the reference point and the gripping position. As a result, when simulating the gripping state in which the work is gripped by the end effector, there is an advantage that the setting work of the position where the work model is gripped by the end effector model can be easily performed. In particular, by positioning the gripping reference point and the gripping position on the axis of the gripping direction while the gripping direction is orthogonal to the work plane, the gripping position can be set by simply bringing the end effector model closer to the work model along the gripping direction. Since the adjustment can be performed, there is an advantage that the work load on the user side is greatly reduced. In the past, the user manually moved the end effector while visually adjusting it to the posture of gripping the work, and there are many parameters and a large degree of freedom in how to adjust the position and posture of the end effector. Therefore, it was an extremely complicated task. On the other hand, the gripping reference position and the gripping direction are defined in advance on the end effector model side, the gripping direction is orthogonal to the work plane, and the gripping reference point of the end effector model and the gripping position of the work are on the axis of the gripping direction. By setting the position of and, the moving direction of the end effector model is defined, and the user can obtain the gripping state by adjusting only the distance between the end effector model and the work model. As a result, it can be expected that the work of sliding the gripping positions between the end effector model and the work model, which has been extremely complicated in the past, can be significantly reduced.

The work of adjusting the relative positions of the end effector model and the work model so that the gripping direction is orthogonal to the work plane and the gripping reference point and the gripping position are positioned on the axis of the gripping direction is a three-dimensional pick determination unit. It is preferable that the operation is automatically performed by 8 liters. As a result, the gripping reference position and the gripping direction are defined in advance on the end effector model side, the gripping direction is orthogonal to the work plane, and the gripping reference point of the end effector model and the gripping position of the work are aligned on the axis of the gripping direction. By automatically adjusting the position, the moving direction of the end effector model is defined, and the user can obtain the gripping state by adjusting only the distance between the end effector model and the work model. As a result, it can be expected that the work of sliding the gripping positions between the end effector model and the work model, which has been extremely complicated in the past, can be significantly reduced.

However, it is not necessarily limited to automatic adjustment by the three-dimensional pick determination unit. For example, the user manually makes the gripping direction orthogonal to the work plane by using the positioning unit 8c, and the gripping reference point and the work of the end effector model are on the axis of the gripping direction. The three-dimensional pick determination unit may be configured to support such an adjustment work when adjusting the position of the gripping position of the above. For example, in the state where the work plane and the gripping direction are displayed in the image display area as the first step, an auxiliary line is shown to the user so that they are orthogonal to each other, or "the gripping direction is orthogonal to the work plane" by characters or images. Please adjust the end effector model as such. ”, Etc. may be displayed to encourage the user to perform the adjustment work. Further, as the second step, an extension line of the gripping direction is displayed in the image display area so that the gripping reference point of the end effector model and the gripping position of the work are positioned on the axis of the gripping direction, and "on the axis of the gripping direction". It is also possible to display a message such as "Please adjust so that the gripping reference point of the end effector model and the gripping position of the work are positioned" and guide the three-dimensional pick judgment unit to prompt the setting.

Further, the fit function can be realized by the three-dimensional pick determination unit 8l. For example, the three-dimensional pick determination unit 8l sets the relative positions of the end effector model and the work model in a state where the gripping direction is orthogonal to the work plane and the gripping reference point and the gripping position are positioned on the axis of the gripping direction. The effector model can be moved along the gripping direction until it interferes with the work model, and the gripping state can be automatically defined in a posture separated from the position where the interference position is reached by a predetermined amount. As a result, it becomes possible to automatically adjust the position and posture in which the work model is gripped by the end effector model, and there is an advantage that the work load on the user side can be further reduced.
(3D search section)

The three-dimensional search unit is a member for performing a three-dimensional search that specifies the posture and position of each work from the input image using the search model registered by the search model registration unit. Prior to the 3D search, when the search model registration unit performs a 3D search to specify the posture and position of each work included in the input image from the input image showing the state in which a plurality of work groups are stacked separately. A search model that virtually represents the three-dimensional shape of the work to be used for is registered in advance. In this state, the 3D pick determination unit 8l makes an end effector at the gripping position designated for this work by the work side gripping point designation unit for the search result in the input image searched by the 3D search unit. Determines whether or not it can be gripped with. For example, the input image acquisition unit acquires an input image having a three-dimensional shape from the images of a plurality of work groups measured by the sensor unit, and is registered in the three-dimensional search unit and the search model registration unit from the input images. A three-dimensional search is performed to specify the posture and position of each work using the search model. As a result, it becomes possible to perform a three-dimensional search on the input image acquired by actually imaging the work, and it becomes possible to perform a gripping determination more realistically.
(Search model)

In bulk picking, in order to determine a work that can be gripped from a plurality of work groups stacked in bulk, it is first necessary to extract each work unit. Here, for the shape of the work group having the height information obtained by the sensor unit, the shape of the work to be searched is registered in advance as a work model, and a three-dimensional search is performed with this work model for each work. Detects position and orientation.
(Basic direction image)

A search model for three-dimensionally searching a work is created by using a height image when the work is viewed from a specific direction. As the height image used as the search model, three-dimensional CAD data, which is a work model expressing the work in three dimensions, and actual measurement data obtained by actually imaging the work by the sensor unit can be used. Here, an example of registering three-dimensional CAD data as a search model will be described. For example, consider a case where a work model CWM constructed with three-dimensional CAD data as shown in FIG. 7 is used as a search model. From this work model CWM, six height images viewed from the positive and negative directions of the three axes (for example, the X-axis, Y-axis, and Z-axis) that are orthogonal to each other in the virtual three-dimensional space, that is, The hexagonal view is acquired as a basic direction image. For example, the basic direction image generation unit 8e'of FIG. 6 generates six basic direction images and displays them in the hexagonal view display area 3a of the display unit 3. The six basic direction images are, for example, "top", "bottom", "left", "right", "front", and "rear" of the work model CWM, that is, plan view, bottom view, left side view, and right side view. The basic direction image generation unit 8e'automatically calculates from the origin of the work model CWM (details will be described later) and the planes constituting the work model CWM so as to be a view, a front view, and a rear view. Here, "top" is the height image seen from the positive direction (+ side) of the Z axis, "bottom" is the height image seen from the negative direction (-side) of the Z axis, and "left" is the negative of the X axis. Height image seen from the direction, "right" is the height image seen from the positive direction of the X axis, "front" is the height image seen from the negative direction of the Y axis, "rear" is from the positive direction of the Y axis The height images seen are shown respectively. However, these are examples, and different coordinate systems may be used, and the positive and negative directions of each axis are based on the coordinate systems orthogonal to each other with the straight line of XY = Y in the XY plane as the axis. The height image seen from the front may be used. Also, when generating a height image from 3D CAD data, it is not always from the direction orthogonal to the CAD axis ("top", "bottom", "left", "right", "front", "rear"). The height image does not have to be the viewed height image. For example, the posture (viewpoint) of the work model may be arbitrarily changed to generate the height image from the changed viewpoint.
(Origin of work model)

Here, the origin of the work model is automatically determined by the image processing apparatus from the coordinate information possessed by the three-dimensional CAD data. For example, for the three-dimensional CAD data of the work model CWM of FIG. 7, a virtual rectangular parallelepiped IBX circumscribing the work model CWM is defined as shown by a broken line in FIG. Set as.
(Basic direction image selection unit 8e)

The number of basic direction images does not necessarily have to be six, and at least a plurality of images is sufficient. For example, if the opposing surfaces have the same shape as in a rectangular parallelepiped, the basic direction image viewed from either surface is sufficient. In other words, it is possible to eliminate the basic direction image having the same shape and reduce the processing load of the three-dimensional search. Such a function of deleting a basic direction image having a common appearance with any of the basic direction images is realized by the basic direction image selection unit 8e. As an example, the basic direction images acquired from the work model CWM of FIG. 7 are shown in FIGS. 9A to 9D. In these figures, FIG. 9A is a height image of the work model CWM of FIG. 7 viewed from the positive direction of the X axis, FIG. 9B is a height image of the work model CWM of FIG. 7 viewed from the positive direction of the Y axis, and FIG. 9C is Z. It is a height image seen from the positive direction of the axis, and FIG. 9D is a height image seen from the negative direction of the Z axis. Here, as the height image, a height image expressing height information as a luminance value is used so that the higher the height, the brighter the point, and the lower the height, the darker the point.

Here, the matching / disagreement of the appearance is a total of six height images viewed from the top and bottom (± direction of the Z axis), front and back (± direction of the Y axis), and left and right (± direction of the X axis) of the work. Is generated and confirmed by performing a match confirmation. Here, the rotation match is confirmed at a step angle of 90 °, and the face that looks like matching with other faces is excluded from the registration target of the search model. In the work model CWM of FIG. 7, the height image seen from the negative direction of the X-axis and the height image seen from the negative direction of the Y-axis are in a state of matching with FIGS. 9A and 9B, respectively. Therefore, it is excluded from the target of search model generation. In this way, search models are generated for the number of faces that look different. The basic direction image excluding the unnecessary height image in this way is displayed in the hexagonal view display area 3a. The six-sided view display area is not limited to its name, and it is not always necessary to display all six sides of the work model. As described above, the height image having a common appearance is excluded as an unnecessary image, and five or less sides are excluded. In this specification, it is referred to as a six-view display area including the mode of displaying with.
(Search model registration screen 130)

Further, such exclusion may be performed manually by the user, may be performed automatically on the image processing apparatus side, or may be combined.

For example, on the search model registration screen 130 for registering the basic direction image BDI as the search model for the three-dimensional search shown in FIG. 10, the basic direction image generation unit 8e'automatically generates the basic direction image BDI corresponding to the hexagonal view. Then, it is displayed in the hexagonal view display area 3a of the display unit 3. At this time, if there is a common basic direction image, the user is urged to exclude it from the registration target of the search model. Here, as the search model selection unit 8i for setting whether or not the search model can be registered, a "registration target" check box 131 is provided for each of the basic direction image BDIs that are candidates for the search model. When the user turns on the "registration target" check box 131, this basic direction image BDI is registered as a search model, and when the "registration target" check box 131 is turned off, this basic direction image can be excluded from the search model. it can.

At this time, for the basic direction images having a common appearance, the basic direction image selection unit 8e causes the user to display the search model registration screen 130 with the "registration target" check box 131 turned off in advance. The user confirms that the basic direction image BDI to be registered in the search model as an initial state and the basic direction image to be excluded from the search model are correctly selected, and then approves this selection or if necessary. Can be modified or replaced. In this way, by selecting the basic direction image to be registered as the search model for the three-dimensional search by default and presenting the candidates for the basic direction image to be excluded from the registration, the labor for registering the search model by the user is saved. it can.
(Actual measurement data MWM)

The above has described an example of using 3D CAD data as a search model for 3D search. However, as described above, the present invention does not necessarily limit the search model to three-dimensional CAD data. For example, a plurality of two-dimensional CAD data are analyzed and converted into three-dimensional data, or a work is actually performed by the sensor unit. The captured actual measurement data can also be used as a search model. As an example, FIG. 11 shows actual measurement data MWM obtained by imaging the work corresponding to the work model CWM of FIG. 7 from the positive direction of the X axis. As described above, when there is no CAD data of the work, it is also possible to use the data obtained by three-dimensionally measuring the actual work. As shown in FIG. 10, the actual measurement data MWM of the number required for the three-dimensional search is imaged and registered.

When registering an actual work, information on the bottom surface on which the work is placed (for example, the shape of the floor around the work) is also measured three-dimensionally. Therefore, for example, it is preferable to exclude unnecessary bottom surface information by cutting out only a portion having a height above a certain level from the bottom surface by threshold processing. This makes it possible to register only the shape portion required for the three-dimensional search.
(Extraction of feature points)

Next, the search model of the registered surface is generated in the state where the height images corresponding to each surface of the work to be the target of the search model are registered in this way. Here, the feature points required for the three-dimensional search are extracted for each registered surface. Here, an example will be described in which two types of feature points are used, that is, a feature point that represents the contour of the shape (feature point on the contour) and a feature point that represents the surface shape (feature point on the surface). FIG. 12A shows a state in which two types of feature points are extracted from the search model SM of the height image (corresponding to FIG. 9A) viewed from the X-axis direction. Further, FIG. 12B shows a state in which the search model SM is displayed in a perspective view. Here, it is preferable that the feature point SCP on the surface and the feature point OCP on the contour are displayed in different display modes. For example, the feature point SCP on the surface is displayed in white, and the feature point OCP on the contour is displayed in light blue. Alternatively, another color scheme may be used, such as displaying white as a feature point on the surface and purple so that it can be more easily distinguished. By displaying the feature points by changing the color in this way, the user can easily visually distinguish the meaning of each feature point.

Feature points on the surface SCP extracts, for example, the surface of the work model at regular intervals. On the other hand, the feature point OCP on the contour performs edge extraction, for example, a portion where the height changes, and extracts the portion further thinned as a feature point at regular intervals. In this way, each feature point is a feature that represents the three-dimensional shape of the surface.
(Three-dimensional search method)

A three-dimensional search is performed using the search model from which the feature points are extracted in this way. Here, as an input image, as shown in FIGS. 13A and 13B, a method of performing a three-dimensional search in order to extract each work from a state in which a group of works in which a plurality of works are stacked separately is imaged and a three-dimensional shape is acquired. Will be described. First, the position and orientation (X, Y, Z, _RX , _RY , R _Z ) in which each feature point of the search model best matches is searched from the input image. Here, R _X , _RY , and R _Z represent the rotation angle with respect to the X axis, the rotation angle with respect to the Y axis, and the rotation angle with respect to the Z axis, respectively. Various methods for expressing such rotation angles have been proposed, but here, ZZ Euler angles are used (details will be described later). Further, the matching positions and postures do not have to be one for each search model, and a plurality of matching positions and postures may be detected for a certain number or more.

Here, as an input image, a two-dimensional display as shown in FIG. 13A or a work group as shown in FIG. 13B which is displayed three-dimensionally is subjected to a three-dimensional search to search the entire input image. As a result, a two-dimensional display as shown in FIG. 13C, or a searched image as shown in FIG. 13D, which is a three-dimensional display of the two-dimensional display, is obtained as a three-dimensional search result. As shown in FIGS. 13C and 13D, it can be seen that the feature points of the search model are searched from the input image and the work corresponding to the search model is detected. FIG. 13D shows how the search results of the search model A and the search model B are obtained. These search models A and B correspond to the search models A and B displayed in the search model selection column of FIG. 14, which will be described later. Regarding the work WK on the right side of FIG. 13D, two search results of search models A and B are obtained with the same work. Therefore, if gripping at the gripping positions registered in each search model is possible, a plurality of gripping positions are obtained in this work.

As a search model used for the three-dimensional search in this way, by using an image of the work viewed for each surface as in the six-view view, the arithmetic processing of the three-dimensional search is simplified as compared with the case of using a perspective view or the like. It is possible to obtain the advantage that the processing can be lightly loaded and speeded up. In addition, the displayed state can be easily seen even when the search model is registered, so that the user can easily understand it visually.
(Evaluation index of 3D search results)

Further, an evaluation index of the three-dimensional search result can be set. For example, in the examples shown in FIGS. 13C and 13D, how much the corresponding feature points existed in the input image (for example, the score ratio of the corresponding feature points with an error of a certain distance or less with respect to the search result). , The three-dimensional search result is scored by the value obtained by deducting the error amount of the feature point as a penalty according to the specified calculation formula. With this method, when there are many invalid data (invalid pixels) that could not be measured three-dimensionally, The score becomes low. In this way, the score can be used as an index showing the reliability of the three-dimensional search result. For example, it can be set to preferentially grip the work in descending order of the score. The three-dimensional search result below the score may be set to be excluded from the gripping target of the work, judging that there is a high possibility of erroneous detection. For example, in the image processing device 100 of FIG. 6, the calculation unit 10 may be set. The evaluation index calculation unit 8q is provided, and the evaluation index is calculated for the search result based on a predetermined standard. As a result, the priority is set in order from the search result having the highest evaluation index, and the work is gripped according to this priority. For example, among the results of a certain score or more, it can be set to preferentially grip the one having the highest position in the Z direction. The higher the position in the Z direction, the higher the grip. Since it has a characteristic that it does not easily interfere with other workpieces, it is possible to obtain an effect of reducing the processing load of interference determination by setting the grip from the one having the highest position in the Z direction.

By detecting each work from the work group stacked in this way, the work to be gripped can be recognized on the image processing apparatus side. Next, in order to grip the work with the end effector, it is necessary for each work to recognize which position of the work is to be gripped and in what posture. Therefore, the gripping position of the work is registered. In this specification, when the "grip position" and "grip posture" are registered, they are used to include the position where the work is gripped and the posture at that time. One or more gripping positions of the work can be registered with respect to the work. Further, it is preferable to register the gripping position for each surface of the work in terms of ease of grip registration work and recognition of the gripping position. That is, the posture of the work is defined as the posture with a specific surface as the upper surface, and then the gripping position is registered.
(Grip registration screen 140)

Here, FIGS. 14 and 15 show examples of a user interface screen for performing grip registration for registering a grip position in which the end effector model grips the work model. On the grip registration screen 140, the surfaces to be registered in the work model are specified, and the grip posture is registered for each surface. In the example of FIG. 14, a user interface screen for registering the gripping posture by designating four types of faces (each face of the search model) is shown. Here, the search model "C" is selected from the search models A to D, and the search model C displays three gripping postures.

In the grip registration screen 140 of FIG. 14, an image display field 141 for displaying an image is provided on the left side, and an operation field 142 for performing various operations is provided on the right side. The work model CWM and the end effector model EEM are displayed in the image display field 141. The viewpoint can be changed by dragging the screen of the image display field 141. In this way, it functions as a positioning unit 8c that adjusts the position and orientation of the work model displayed in the display area in the virtual three-dimensional space. Further, the display mode of the image display field 141 can be switched to a two-dimensional display or a three-dimensional display. In order to show the current display mode in an easy-to-understand manner, the three-dimensional reference coordinate axis BAX is superimposed and displayed in the image display field 141. In the example of FIG. 14, the image display field 141 displays a gripping posture registered in the gripping posture 001 selected in the operation field 142, that is, a state in which a part of the work model CWM is gripped by the end effector model EEM. ing. In this example, the entire end effector model EEM is displayed, but it is not always necessary to display the entire end effector model when registering the grip, and at least the part that grips the work model CWM, for example, the claw portion is displayed. All you need is.

Further, the operation field 142 is provided with a search model selection field 143 for selecting a search model and a grip posture display field 144 indicating a grip posture registered for the search model selected in the search model selection field 143. Has been done. When the gripping postures listed in the search model selection field 143 are selected, the corresponding registered gripping postures are displayed in the image display field 141. Further, by pressing the edit button 145, it is possible to correct the registered gripping posture. Further, the operation column 142 is provided with an additional button 146 for adding a gripping posture and a delete button 147 for deleting the registered gripping posture. When the delete button 147 is pressed, the selected gripping posture is deleted.
(Gripping posture addition screen 150)

If you want to add a new gripping posture, press the add button 146. As a result, the gripping posture addition screen 150 is displayed as shown in FIG. 15, and the gripping posture can be added. The end effector model EEM and the work model CWM are displayed in the image display field 141. Further, in the operation column 142, the gripping posture coordinate information 153 that defines the gripping posture is displayed. Here, the position parameters X, Y, Z, _RX , _RY , and R _Z displayed as the gripping posture coordinate information 153 represent the position and posture data of the end effector model EEM with respect to the origin of the search model. ing. The origin of the search model can be the center of gravity of the work model CWM, the center coordinates of the CAD data, or the like as described above.

When the values of X, Y, Z, R _X , _RY , and R _Z of the gripping posture coordinate information 153 are changed, the position and posture of the end effector model EEM displayed three-dimensionally in the image display field 141 becomes this. It will be updated accordingly. On the contrary, by dragging and moving the end effector model EEM in the image display field 141, the display content of the gripping posture coordinate information 153 in the operation field 142 is updated to the gripping posture coordinates after the movement. As a result, the user can register the gripping position and the posture while checking the end effector model EEM of the image display field 141 and the gripping posture coordinate information 153 of the operation field 142. Further, the three-dimensional reference coordinate axis BAX can be superimposed and displayed in the image display field 141.

When registering the gripping position, the gripping reference point corresponding to the position where the work model CWM is gripped with respect to the end effector model EEM displayed in the image display field 141, and the gripping direction in which the work model CWM is gripped by the end effector model EEM. To specify. It is preferable that the gripping reference point and the gripping direction are set as default values on the image processing apparatus 100 side. Then, on the gripping posture addition screen 150 of FIG. 15, the position of the end effector model EEM is determined according to the position and posture of the work model CWM so that the gripping direction is orthogonal to the work plane representing the posture of the work model CWM. And automatically adjust the posture. This adjustment can be performed, for example, by the three-dimensional pick determination unit 8l. Alternatively, the user can manually adjust the position and orientation of the end effector model EEM by using the positioning unit 8c so that the gripping direction of the end effector model EEM is orthogonal to the work plane of the work model CWM. May be good.

Further, when registering the gripping position, in the initial state in which the gripping posture addition screen 150 of FIG. The initial value is the state located below. In this way, if the end effector model EEM is lowered and brought into contact with the work model CWM, the gripping posture is determined, so that the user can intuitively perform the operation. Conventionally, the position in three dimensions. It is possible to reduce the problem that the alignment was troublesome. That is, basically, the gripping posture can be easily registered only by setting the alignment of the end effector model EEM in the X, Y, and Z directions and the rotation with respect to the Z axis.

Furthermore, by setting the gripping position to be gripped by the end effector model EEM for the work model CWM, the position of the end effector model EEM is automatically set so that this gripping position is located on the axis extending the gripping direction. It can also be configured to adjust to.

The gripping posture is registered based on the image of the work. Therefore, it is not always necessary to use CAD data, and as described above, the gripping posture can be registered with respect to the actually measured data obtained by actually imaging the work.
(Fit function)

Further, when the gripping position is specified, not only the user manually performs the work of moving the end effector model to the gripping position of the work model, but also a fit function that automatically performs this can be provided. Conventionally, the gripping position X for holding the workpiece model end effector model, Y, Z and the grasping posture R _X, R _Y, when specifying the R _Z, for example, in the image display field 141 in FIG. 15, by dragging the end effector model It was set by moving it until it came into contact with the work model, or by inputting a numerical value in Z in the height direction in the operation field 142. However, it is troublesome for the user to visually move the end effector model to match the posture of gripping the work model, and it is also possible to specify the numerical values of the position parameters by adjusting the six position parameters. It's hard to tell if it's good. Therefore, a fit function is provided that automatically performs the work of positioning the end effector model at the gripping position of the work model.

Here, when the work model or the end effector model has height information such as three-dimensional CAD data, the position to be gripped on the work model is specified by clicking the mouse or the like, and the height of this position is specified. The direction, that is, the Z coordinate is acquired, and the position where the offset is added to the Z coordinate is set as the Z coordinate after the movement of the end effector model. This saves the user the trouble of manually moving the end effector model to the gripping position and manually inputting the position parameters such as the Z coordinate, and can specify the gripping position accurately. As an example of such a fit function, the "fit" button 154, which is one aspect of the relative position setting unit 8d5, is arranged in the operation column 142 shown on the gripping posture addition screen 150 of FIG. When the "fit" button 154 is pressed, the end effector model EEM is virtually moved in the Z direction, stopped at the position where it contacts the work model CWM, and the position returned by the offset amount in the opposite direction from this interference position is set. , The gripping position. The offset amount is a numerical value for slightly separating the tip of the end effector model EEM in order to prevent the tip of the end effector model EEM from coming into contact with the work model CWM and interfering with or damaging it, and can be specified according to the work and application, for example, 1 mm.

In the above example, a method of performing grip registration has been described using the height image used when generating the three-dimensional search model for performing the three-dimensional search. By sharing the search model for extracting the work and the model for registering the gripping position in this way, the user can perform a three-dimensional search and set the gripping position for the common work, and operate the work. A sense of unity is obtained and it becomes easier to understand sensuously. However, the present invention does not necessarily have to match the search model of the three-dimensional search with the model to be grasped and registered. If the correspondence between the three-dimensional search model and the model used for grip registration is known, it is not always necessary to use the model used as the search model for grip registration.

Further, after the height image is generated from the three-dimensional CAD data, the confirmation of the matching of the appearances does not have to be limited to the confirmation of the positive and negative directions of each axis. For example, a work such as a cube looks the same in any direction, such as the ± direction of the X-axis, the ± direction of the Y-axis, and the ± direction of the Z-axis. Just do it. Similarly, for registration of the gripping position, it is sufficient to specify gripping for one surface. With this alone, it is possible to grip all the surfaces (± direction of the X axis, ± direction of the Y axis, ± direction of the Z axis). For example, for the bulk stacking of cubic work WKCs as shown in FIG. 16A, one surface of the cube is registered as a model as shown in FIG. 16B, and one gripping posture for grasping this surface from directly above is registered. As shown in FIG. 16C, it is possible to perform a three-dimensional search on all six surfaces and grip by the end effector model EEM.
(Calibration)

On the grip registration screen 140 described above, the relative position and posture of the end effector model EEM when gripping the work is registered with respect to the origin of the search model. On the other hand, when picking a work with an actual end effector, the robot controller 6 actually drives the robot from the vision coordinates, which are the coordinates of the three-dimensional space (vision space) in which the work is imaged by the sensor unit. Need to convert to robot coordinates. Specifically, the position and orientation of the work obtained as a result of the three-dimensional search can be obtained by the position (X, Y, Z) and the attitude ( _RX , _RY , R _Z ) in the vision space (note that the attitude (Note). R _X , _RY , and R _Z ) indicate postures expressed by ZZ Euler angles, which will be described later). Further, the posture of the end effector that grips the end effector is also obtained as the position (X, Y, Z) and the posture ( _RX , _RY , R _Z ) of the image processing device in the virtual three-dimensional space. In order for the robot controller 6 to drive the robot based on the position and posture in such a vision space, the robot controller 6 must have the position (X', Y', Z') and the posture ( _RX ', _RY ') in the robot space. , R _Z ') need to be converted. The process of calculating the conversion formula for converting the position and orientation calculated in the coordinate system displayed by such an image processing device into the position and orientation of the coordinate system in which the robot controller operates the end effector is calibrated. It is called a posture.
(Embodiment 2)

An example of a robot system having a calibration function between such an image processing device (machine vision device) and a robot is shown in the functional block diagram of FIG. 17 as the second embodiment. The robot system 2000 shown in this figure includes an image processing device 200, a display unit 3, an operation unit 4, a sensor unit 2, a robot controller 6, and a robot RBT. In the robot system shown in this figure, the members common to those in FIG. 6 and the like described above are designated by the same reference numerals, and detailed description thereof will be omitted as appropriate.
(Image processing device 200)

The image processing device 200 includes an input image acquisition unit 2c, a storage unit 9, a calculation unit 10, an input / output interface 4b, a display interface 3f, and a robot interface 6b.

The input image acquisition unit 2c acquires an input image having a three-dimensional shape from the image including the end effector measured by the sensor unit. When the input image acquisition unit 2c acquires the input image of the end effector, it is preferable to take an image so as to include the attachment position attached to the flange surface at the tip of the arm portion of the robot. In addition, the input image is captured in a posture in which the area of the end effector becomes large. For example, the user operates the robot so that the input image is captured from the sideways posture with the end effector in the horizontal posture.

The calculation unit 10 includes an end effector model registration unit 8u, a work model registration unit 8t, a calibration unit 8w, a search model registration unit 8g, a three-dimensional search unit 8k, a conversion unit 8x, and an end effector mounting position calibration. A portion 8y and a gripping position specifying portion 8d are provided.

The end effector model registration unit 8u is a member for registering an end effector model, which is three-dimensional CAD data that virtually represents the three-dimensional shape of the end effector. For example, the end effector model registration unit 8u reads a separately created three-dimensional CAD data representing the shape of the end effector and registers it as an end effector model. In this case, the three-dimensional CAD data input via the input / output interface 4b is registered as an end effector model by the end effector model registration unit 8u. Alternatively, the three-dimensional point cloud data obtained by imaging the actual end effector can be used as the end effector model. In this case, the three-dimensional point cloud data acquired by the sensor unit 2 and the input image acquisition unit 2c is registered as an end effector model by the end effector model registration unit 8u. Alternatively, three-dimensional CAD data imitating an end effector may be created and registered. In this case, the end effector model registration unit 8u realizes a simple three-dimensional CAD function.

The calibration unit 8w sets the position and orientation calculated in the coordinate system of the vision space, which is a virtual three-dimensional space displayed on the display unit, to the position of the coordinate system in the robot space in which the robot controller operates the end effector. It is a member for acquiring calibration information for converting coordinates into a posture.

The calibration unit 8w calculates a conversion formula between the actual position coordinates of the robot end effector EET and the position coordinates on the image displayed on the image processing device for the plurality of position coordinates. The method of coordinate transformation is not particularly limited, and a known method such as three-dimensional affine transformation can be appropriately used.

The storage unit 9 stores the calibration information by the calibration unit 8w.

The three-dimensional search unit 8k uses the end effector model as a search model to perform a three-dimensional search for specifying an image area corresponding to the position and orientation of the end effector from the input image acquired by the input image acquisition unit 2c. It is a member.

The conversion unit 8x converts the vision coordinates into robot coordinates based on the calibration information obtained by the calibration unit 8w. The conversion unit 8x reads out the calibration information stored in the storage unit 9.

The end effector mounting position calibration unit 8y uses the information obtained by converting the position and orientation of the end effector in the vision space searched by the three-dimensional search unit into the position and orientation in the robot space by the conversion unit 8x. , Calibrate the error between the position and orientation of the end effector in the vision space and the position and orientation in the robot space. As a result, by performing a three-dimensional search using the end effector model, it is possible to correct the error in the virtual end effector model according to the actual mounting state of the end effector, and more accurate setting work can be performed. Will be.

The gripping position specifying unit 8d specifies one or more gripping positions for gripping the work model with the end effector model with respect to the work model registered by the work model registration unit 8t.
(ZYX Euler angles)

Here, ZZ-based Euler angles will be described. Conventionally, ZZ Euler angles have been used to position the posture of a work model or an end effector model using three-dimensional CAD data in order to define a gripping position for gripping a work with an end effector. In this case, the position and orientation of the end effector model with respect to the work model are represented by six position parameters (X, Y, Z, _RX , _RY , R _Z ). Here X, Y, Z are orthogonal coordinate axes defining a three-dimensional space, R _X, R _Y, each R _Z X-axis, Y-axis shows the rotation angle to rotate about the Z axis.

Here, an example of rotating the end effector model according to the ZZ Euler angles will be described with reference to FIGS. 18 to 21. First, FIG. 18 shows a state in which the end effector model EEM and the work model WM11 are displayed in the image display area of the display unit 3. In this example, in order to make the explanation easy to understand, the reference coordinate axis BAX (XYZ) that defines the virtual three-dimensional space and the three-dimensional space coordinate axis after the rotating object to be rotated (here, the end effector model EEM) are rotated. The rotated coordinate axes RAX (XYZ) are displayed respectively.

In the example of FIG. 18, since the work model WM11 side is stationary and the end effector model EEM side is rotated, the reference coordinate axis BAX is the position and orientation of the work model WM11, and the rotated coordinate axis RAX is the position of the end effector model EEM. And the posture are shown respectively. The rotated coordinate axis RAX on the end effector model EEM side has its origin coincided with the gripping reference point HBP which is the gripping position of the end effector model EEM (details will be described later).

In yet a pre-rotation state, the rotation angle R _X of the XYZ-axis, R _Y, R _Z each _{R X = 0 °, R Y} = 0 °, an R _Z = 0 °, the reference coordinate axes BAX and rotational already coordinate RAX Are in agreement. In this state, when the end effector model EEM is rotated 90 ° counterclockwise about the Z axis, it becomes as shown in FIG. In the state of FIG. 19, the rotation angles R _X , _RY , and R _Z are R _X = 0 °, _RY = 0 °, and R _Z = 90 °, respectively, and have been rotated and displayed in the image display area accordingly. The coordinate axis RAX is updated to the posture after rotation. Further, when the state of FIG. 19 is rotated 90 ° clockwise around the Y axis, the result is as shown in FIG. In this state, the rotation angles R _X , _RY , and R _Z are R _X = 0 °, _RY = 90 °, and R _Z = 90 °, respectively, and the rotated coordinate axis RAX is similarly rotated. Will be updated to. Further, when the rotation is 90 ° clockwise around the X axis from the state of FIG. 20, the state of FIG. 21 is obtained, and the rotation angles R _X , _RY , and R _Z are R _X = 90 ° _RY = 90 ° R, respectively. _Z = 90 °.

Conventionally, such ZYX Euler angles have been used to express the generation and position of work models and end effector models. However, since the number of operable position parameters is as large as 6, it is not easy to adjust these position parameters to the position and posture assumed by the user. Also the XYZ axes used when moving the position while maintaining the posture, R _X to be used in rotation, R _Y, because where the R _Z axis different, varying the position parameter, how changes to the intuition There was a problem that it was difficult to understand. This situation will be described with reference to FIG.

Now, in the state of FIG. 21, the rotated coordinate axis RAX, which is the orthogonal coordinate axis after _{rotation of the rotation angles R X} = 90, _RY = 90, and R _{Z = 90, is displayed in the image display area. If RY} or R _Z is changed from this state, it will rotate around an axis different from the displayed rotated coordinate axis RAX. For example the rotation of R _Y is not a Y axis indicated by rotation already axes RAX in Figure 22, as indicated by broken lines in FIG. 22, R _X pre-rotation Y-axis (Y-axis indicated by rotation already axes RAX in Figure 19) Will rotate around. Further, R _Z does not rotate around the Z axis shown by the rotated coordinate axis RAX in FIG. 22, but also _{the Z axis before the rotation of R X} and _RY shown by the broken line (Z axis shown by the rotated coordinate axis RAX in FIG. 18). Will rotate around.

This is because the ZYX Euler angles are defined on the premise that they are rotated in the order of Z-axis, Y-axis, and X-axis. According to the ZYX Euler angles, the Z axis is the reference, and the rotation axes of _{R X} and _RY are rotated by the rotation _{of R Z around the Z axis, respectively.} Here, the rotation axis around the Z axis does not move, while the rotation axes themselves of _{R X} and _{RY are rotated by the rotation R Z} around the Z axis. _{When the rotation is RY} around the Y axis determined in accordance with this, the X axis is also rotated accordingly. Here, since the rotation axis around the Y axis _{is determined by the rotation R Z around the Z axis, even if only R X is} rotated around the X axis, the rotation axis around the Y axis and the rotation axis around the Z axis change. do not. In other words, in ZZ Euler angles, R _Z has an independent axis of rotation, _RY has an axis of rotation dependent on _{R Z , and R X} has an axis of rotation dependent on _RY. It can be said.

In this way, in the ZZ Euler angles that have been generally used for robot control, the three rotation axes are related to each other, and the rotation axes around the other rotation axes are used. Since it rotates, it is extremely difficult for the user to grasp which axis is the axis of rotation, and it is not easy to rotate the axis in the intended direction.
(Display of correction rotation axis)

In the method according to the present embodiment the contrary, R _X, is R _Y, the axis of rotation of R _Z, to display, based on the actual rotation axis. This axis is different from the conventional display of three-dimensional space using ZZ Euler angles (FIGS. 18 to 21), and is in a state after rotation around another axis of rotation. The actual rotation axis corrected in consideration of the above is displayed as the correction rotation axis. For example, consider an example of displaying the correction rotation axis of _{R Z in} the state of FIG. 22. In this case, as shown in FIG. 23, _{the Z axis when R X} and _RY are calculated as 0 ° is the correction rotation axis. The correction rotation Z-axis AXZ displayed here is the Z-axis when R _X = 0 °, _RY = 0 °, and R _Z = 90 °.

On the other hand, to display the correct rotation shaft of R _Y is in the state of FIG. 22, as shown in FIG. 24, Y axis when calculating the R _X is 0 ° becomes the actual rotation axis. The correction rotation Y-axis AXY displayed here is the Y-axis when R _X = 0 °, _RY = 90 °, and R _Z = 90.

To display the corrected rotational axis R _X is still the state of FIG. 22 may be displayed in the X-axis as it is. FIG. 25 shows an example in which the correction rotation X axis is displayed as the correction rotation axis. The correction rotation X-axis AXX displayed here is the X-axis when R _X = 90 °, _RY = 90 °, and R _Z = 90 °.

The starting point of these correction rotation axes is preferably the center of rotation of the end effector model. For example, it is set to an intermediate position between a pair of claws, which is a gripping position of the end effector model EEM.
(XYZ Euler angles)

This correction rotation axis can display the rotation axis in the same way even when the order of rotation is changed. For example, in the above-mentioned examples of FIGS. 23 to 25, since the ZZ Euler angles are adopted, they are rotated in the order of _{R Z} , _RY , and R _X. Meanwhile, different Employing X-Y-Z Euler angles, R _X, R _Y, from rotating in the order of R _Z, axes calculated as 0 ° is the Z-Y-X based Euler angles. For example, to obtain a corrected rotary axis X-Y-Z Euler angles, similarly to FIGS. 18 to 21 R _X, R _Y, assuming that is rotated in the order of R _Z, displays the Z-axis as a correction rotation axis To do so, display the rotated Z-axis as it is. To display the Y-axis as the correction rotation axis, display the Y-axis when R _Z is calculated as 0 °. Further, in order to display the X-axis as the correction rotation axis, the X-axis when R _Z and _RY are calculated as 0 ° is displayed.

In this way, when the user sets the gripping position and posture using Euler angles, the rotation axis can be displayed in an easy-to-understand manner, and it is easy to operate intuitively without understanding the complicated concept of Euler angles. it can.
(Embodiment 3)

It is not easy for the user to set the position and orientation of end effectors and workpieces defined by multiple position parameters such as Euler angles. Therefore, it is also possible to guide the setting procedure of the position parameter so that the user can specify the position parameter that can be specified among the plurality of position parameters, and sequentially set the necessary position parameter according to the guidance. Such an example is shown in FIG. 26 as the robot system 3000 according to the third embodiment. The robot system 3000 shown in this figure includes an image processing device 300, a display unit 3B, an operation unit 4, a sensor unit 2, a robot controller 6, and a robot RBT. In the robot system shown in this figure, the members common to those in FIG. 6 and the like are designated by the same reference numerals and detailed description thereof will be omitted as appropriate.
(Image processing device 300)

The image processing device 300 includes an input image acquisition unit 2c, a calculation unit 10, a storage unit 9, an input / output interface 4b, a display interface 3f, and a robot interface 6b. The storage unit 9 has a gripping position storage unit 9b. The gripping position saving unit 9b is a member for storing the gripping position of the work model or the end effector model designated by the gripping position specifying unit 8d.

The calculation unit 10 includes a grip position specifying unit 8d, a grip position copy unit 8d8, a relative position setting unit 8d5, a search model registration unit 8g, a three-dimensional search unit 8k, a three-dimensional pick determination unit 8l, and a cross-section model. It is provided with a generation unit 8s.

The grip position specifying unit 8d is on the X-axis, Y-axis, and Z-axis that specifies the position and orientation of either or both of the work model and the end effector model displayed on the virtual three-dimensional space on the display unit. Six position parameters of _{X coordinate, Y coordinate, Z coordinate, which are coordinate positions, and RX} rotation angle, _RY rotation angle, and R _Z rotation angle, which are rotation angles centered on the X axis, Y axis, and Z axis. , Each is a member for designating.

The gripping position copy unit 8d8 is a member for reading out the gripping position of the work model or the end effector model stored in the gripping position storage unit 9b, changing the gripping position, and registering it as a new gripping position. As a result, when registering a plurality of gripping positions, the already registered gripping position is read out, the gripping position is changed based on this, and the gripping position can be registered as a new gripping position, so that the gripping position can be changed from scratch. Compared to the case of registration, the gripping position can be added more easily, and labor saving in the registration work can be achieved.

Whether or not the three-dimensional pick determination unit 8l can grip each search result searched by the three-dimensional search unit 8k with the end effector at the gripping position specified for the work model by the gripping position specifying unit 8d. It is a member for determining whether or not.

The cross-section model generation unit 8s includes a cross-section obtained by cutting the end-effector model on a plane orthogonal to the basic axis along a linear basic axis set in one direction with respect to the end-effector model, and an orthogonal cross section including each cross-section. A cross-section model composed of a set of a plurality of cross-sections and cross-section positions is generated with the point where the plane intersects the basic axis as the cross-section position. Polygon data or the like is used for the end effector model for creating this cross-section model.
(Gripping position specifying part 8d)

The gripping position specifying portion 8d shown in FIG. 26 includes a first designated portion 8d9, a second designated portion 8d6, and a third designated portion 8d7. The first designation unit 8d9 is a member capable of designating at least one of the X coordinate, the Y coordinate, and the Z coordinate, and cannot specify other position parameters. The second designated unit 8d6, to the part of the state where the position parameter is specified workpiece model or the end effector model first designated portion 8d9, R _X rotation angle, R _Y rotation angle, at least R _Z rotation angle One of the members can be specified, and the other position parameters cannot be specified. Third specifying unit 8d7, to the first part of the position parameter specifying unit 8d9 and the second specifying unit 8d6 is designated states work model or the end effector model, R _X rotation angle, R _Y rotation angle, It is a member in which at least one of the R _Z rotation angles, a position parameter not specified by the second designated portion 8d6 can be specified, and the other position parameters cannot be specified.

As a result, the gripping position is not specified by all six position parameters on one screen, but the parameters that can be specified are divided into a plurality of designated parts, and the position parameters are individually specified for each. By doing so, it is possible to avoid the situation where multiple position parameters are entangled with each other and it becomes difficult to grasp the position and posture, and by wearing the position parameters in sequence, the information necessary for specifying the position and posture can be set. Is possible. In particular, the initial position of the gripping position on the first designated part 8d9 is displayed on the display part in a flat shape, and any part on the work model is designated by clicking the mouse or the like, so that the posture is easy to see. It is possible to let the user specify by directly specifying on the work model that has been done, which is much easier than the complicated and troublesome work such as determining the posture and defining the numerical value as in the past. it can. The first designated unit 8d9 is not necessarily limited to displaying the work model in a two-dimensional manner, and may be displayed in a three-dimensional shape.
(Procedure when creating settings)

Here, the procedure of the teaching work for operating the robot system before the actual operation will be described with reference to the flowchart of FIG. 27.

First, in step S2701, a search model for three-dimensionally searching the work is registered. Here, as the search model, as described above, a work model, here, three-dimensional CAD data can be registered. Alternatively, the actual measurement data obtained by actually imaging the work by the sensor unit may be registered as a search model.

Next, in step S2702, the robot end effector model is registered. Here, too, three-dimensional CAD data can be registered as an end effector model. Next, in step S2703, the surface of the work model to be gripped is selected from the height images. Next, in step S2704, the position and posture of the robot when gripping with respect to the selected surface are registered. Next, in step S2705, it is determined whether or not the required number of positions and postures have been registered, and if not, the process returns to step S2703 and the above process is repeated. When the required number of registrations have been completed, the process ends.
(Procedure to register 3D CAD data in the search model)

Here, an example of a procedure for registering a work search model when three-dimensional CAD data is used as the search model in step S2701 will be described with reference to the flowchart of FIG. 28. First, in step S2801, the three-dimensional CAD data model of the work is read. Next, in step S2802, the center of the circumscribing rectangular parallelepiped of the 3D CAD data model is corrected to the origin of the 3D CAD data. Further, in step S2803, a height image viewed from each of the "upper", "lower", "left", "right", "front", and "rear" directions is generated. Here, when the height image is generated from the three-dimensional CAD data, the height image is generated so that the origin of the CAD is the center of the height image.

Next, in step S2804, the height image having the same appearance is deleted from the generated height images. Finally, in step S2805, the search model is registered using the generated height image.

In this way, the user can register the search model used for the three-dimensional search according to the guidance.
(Registration of end effector model)

Next, the details of the procedure for registering the end effector model in step S2702 of FIG. 27 will be described with reference to the flowchart of FIG. 29. Here, it is assumed that the three-dimensional CAD data constituting the end effector model is composed of polygon data.

First, in step S2901, the polygon data of the end effector model is read. Next, in step S2902, the direction in which the cross section is created is determined. Further, in step S2903, a cross-section model is created. The cross-section model is created by the cross-section model generation unit 8s of FIG. Details of creating the cross-section model will be described later. In this way, the end effector model is registered in the storage unit 9.
(Additional area)

Further, when registering the end effector model, an additional area can be added to the original end effector model. Such a procedure will be described with reference to the flowchart of FIG. First, in step S3001, the end effector model is registered. For example, an end effector model composed of three-dimensional CAD data such as STL data is read.

Next, in step S3002, an additional area is set. The additional area adds, for example, the shape of the actual end effector and the shape imitating the added cover, harness, etc. to the surface of the end effector model at the time of interference determination (details will be described later). The accuracy of interference determination can be improved.
(Embodiment 4)

An additional model creation function is used to set such an additional area. An example of a robot system to which an additional model creation function is added is shown in the block diagram of FIG. 31 as the fourth embodiment. The robot system 4000 shown in this figure includes an image processing device 400, a display unit 3, an operation unit 4, a sensor unit 2, a robot controller 6, and a robot RBT. In the robot system shown in this figure, the members common to those in FIG. 6 and the like are designated by the same reference numerals and detailed description thereof will be omitted as appropriate.
(Image processing device 400)

The image processing device 400 includes an input image acquisition unit 2c, a calculation unit 10, a storage unit 9, an input / output interface 4b, a display interface 3f, and a robot interface 6b.

The calculation unit 10 includes a work model registration unit 8t, an end effector model registration unit 8u, an additional model creation unit 8v, a grip position identification unit 8d, a search model registration unit 8g, a three-dimensional search unit 8k, and an interference determination. A section 8m and a cross-section model generation section 8s are provided.

The additional model creation unit 8v is a member for creating an additional model in which an additional region represented by one or more predetermined three-dimensional figures is added to the surface of the end effector model. This makes it easy to add interference judgment areas with simple shapes such as rectangular parallelepipeds and cylinders to the end effector model without editing the 3D CAD data, and perform interference judgment according to each area. Interference judgment can be performed. The three-dimensional figure includes not only basic figures prepared in advance on the image processing apparatus side but also figures that can be freely designed by the user.

The gripping position specifying unit 8d is a member for specifying one or more gripping positions for gripping this work model with the end effector model with respect to the work model registered by the work model registration unit 8t.

The three-dimensional search unit 8k specifies an image area corresponding to the position and orientation of each work from the input images acquired by the input image acquisition unit 2c, using the search model registered by the search model registration unit 8g. It is a member for performing a three-dimensional search.

The interference determination unit 8m is an additional model created by the additional model creation unit 8v, and when the end effector is operated, the interference determination unit 8m determines the presence or absence of interference with other objects that may interfere with the operation. It is a member for performing. When the interference determination unit 8m moves the end effector to the position in an attempt to grip any of the plurality of works included in the input image acquired by the input image acquisition unit 2c, the object around the work ends. In order to determine whether or not to interfere with the effector, the interference determination is executed for the images constituting the objects around the work in the input image. As a result, the end effector model can be compared with the input image to determine the interference. The interference determination unit 8m is specified by the gripping position specifying unit 8d in order to grip the work, for example, with respect to the search result corresponding to the position and orientation of each work searched from the input image by the three-dimensional search unit 8k. When the end effector model is placed at the gripping position, it is determined whether or not there is interference with an object existing around the work.
(Additional area setting screen 310)

The additional area is added to the end effector model composed of 3D CAD data. Here, as one aspect of the additional model creation unit 8v, an additional area setting screen 310 for adding an additional area to the end effector model is shown in FIG. 32. The additional area setting screen 310 can read the end effector model EEM, which is the three-dimensional CAD data of the end effector, and add a figure to the end effector model EEM. The additional area setting screen 310 includes an image display field 141 and an operation field 142. In the image display field 141, the three-dimensional CAD data of the read end effector model EEM is displayed. Further, the operation column 142 is provided with a basic graphic display column 311. A list of selectable basic figures is displayed in the basic figure display field 311. Examples of basic figures include a rectangular parallelepiped, a cylinder, a cone, a triangular pyramid, and a hexagonal column. The user can select a desired basic figure from the basic figure display field 311 and press the "Add" button 312 to add the basic figure to a designated position on the end effector model EEM. Further, each line listed in the basic figure display field 311 is provided with an "edit" button 313, and by pressing this button, the setting of the selected basic figure can be changed. For example, basic graphic parameters such as the length, radius, and height of one side of the bottom surface can be adjusted.

In the example of FIG. 32, a rectangular parallelepiped is selected as the basic figure to be added, and is highlighted in the basic figure display field 311. When the "edit" button 313 is pressed in this state, the basic figure editing screen 320 of FIG. 33 is displayed. The operation field 142 of the basic figure editing screen 320 is provided with a basic image parameter adjustment field 321 from which the size of the basic figure and the position and orientation of the end effector with respect to the reference position can be changed. When the basic graphic parameter is adjusted from the basic image parameter adjustment field 321, the display content of the image display field 141 is also updated accordingly.

In the example of FIG. 32, the additional region ADA is inserted between the flange surface FLS at the tip of the arm portion and the end effector model EEM. As a result, it is possible to reproduce a state in which a mounting jig or the like is interposed between the arm portion and the end effector. Further, when the flange surface at the tip of the arm portion has a deviation from the surface set for calculation on the image processing apparatus side, it can be used to offset the deviation amount.

Furthermore, multiple figures can be added. In the example of FIG. 32, various basic figures can be added by selecting a desired basic figure from the basic figure display field 311 and pressing the "Add" button 312. This makes it possible to express various shapes by combining a plurality of basic figures.

In this way, by providing a function to set an additional area for the end effector model, the basics prepared in advance without changing the shape by directly editing the shape of the three-dimensional CAD data as in the conventional case. It is possible to easily add a shape simply by adding a figure. As a result, when the end effector is placed at the gripping position where the work is gripped, it is not included in the end effector model in the interference determination for confirming in advance whether or not it interferes with the objects around the work, but it actually exists. When there are members to be used, the accuracy of the determination result can be improved by imitating these members. For example, when the end effector is connected to the tip of the arm portion via a joint, the entire end effector may be offset and the tip may be slightly projected. There may also be additional elements on the outside of the end effector, such as a contact protection cover or a cable extending from the end effector. Such additional elements are often not included in the three-dimensional CAD data of the end effector model, and the CAD data is rarely prepared separately. However, it is troublesome to process the three-dimensional CAD data of the end effector model and edit it so as to change the shape according to these additional elements. Further, even if a complicated shape can be expressed by CAD data, if the three-dimensional CAD data having such a complicated shape is used for the interference determination, the arithmetic processing becomes complicated and the processing time becomes long.

Therefore, by expressing such additional elements with figures, it is possible to obtain an end effector model in a form that is close to the actual situation in a simple additional area without going through troublesome editing work, and it is immediately in reality. It becomes possible to easily express the shaped shape. Further, in this method, since the interference determination area is added without changing the shape of the CAD data, the amount of increase in the processing time can be suppressed.
(Procedure for setting the grip position)

Next, in step S2704 of FIG. 27, a procedure for registering a gripping position and a posture for gripping the work by the end effector will be described with reference to the flowchart of FIG. 34 and FIGS. 35A to 38. Here, the end effector model EEM with respect to the work model WM11 when the work model WM11 is held while the work model WM11 is fixed by using the work model WM11 and the end effector model EEM composed of three-dimensional CAD data. Six position parameters that define the position and posture (hereinafter collectively referred to as "grasping position"), that is, the gripping positions X, Y, Z and the gripping postures _RX , _RY , _RZ are shown in FIG. 34. Set in order according to the procedure shown in the flowchart.
(XY designation part)

First, in step S3401, the X coordinate and the Y coordinate of the gripping position are designated. Here, a two-dimensional plane image in which the work is projected onto a certain plane is displayed by the XY designation unit, and the gripping position is designated on this plane image. As one form of such an XY designation unit, the XY designation screen 230 is shown in FIG. 35B. On the XY designation screen 230 shown in this figure, the three-dimensional CAD data of the work shown in FIG. 35A is displayed in a plan view in the image display field 141. In this state, the user is made to specify which position of the work model WM11F displayed in the plan view is desired to be gripped by the end effector model. By selecting the gripping position on the plan view with a pointing device such as a mouse as shown in FIG. 35B, it is possible to specify the desired position in an easy-to-understand manner without being conscious of the posture and the rotation angle. In particular, by specifying the gripping position on the work model WM11F displayed in the plan view by clicking, an advantage that the position can be easily specified can be obtained as compared with inputting a numerical value to adjust the position. Then, the X and Y coordinates of the gripping position can be determined from the position manually specified by the user and the scale of the image. An X-Y coordinate display field for displaying the X-Y coordinates numerically may be provided on the XY designation screen.
(Z-R _Z designation part)

Next, in step S3402, the gripping position Z and the gripping posture R _Z are designated from the Z-R _{Z designated portion.} As a form of Z-R _Z designation section, showing a Z-R _Z designation screen 240 in FIG. 36. _{The Z-RZ} designation screen 240 shown in this figure includes an image display field 141 and an operation field 142. The operation column 142 is provided with a Z coordinate designation column 241 for designating the Z coordinate of the gripping position and _{an R Z} rotation angle designation column 242 _{for designating the R Z} rotation angle of the gripping posture. At this stage, since the X and Y coordinates have already been determined in step S3401, it is sufficient to display only the Z axis. Therefore, in the state where the end effector model EEM and the work model WM11 are displayed in the image display column, only the correction rotation Z axis AXZ is displayed in an overlapping manner as the correction rotation axis. This makes it easier for the user to imagine how the end effector model EEM displayed in the image display field 141 is rotated when the rotation angle is changed, and facilitates the position adjustment work of the end effector model EEM. You get the benefits you can. In particular, by not displaying other rotation axes, the user can visually grasp the rotation direction without confusion. Furthermore, by making it impossible to specify other position parameters other than the Z coordinate and R _Z rotation angle, it is eliminated that other position parameters are accidentally changed, and the user concentrates on the position parameters that can be specified. You can concentrate on the necessary settings without any confusion such as misunderstandings and misconfigurations.

When a _{numerical value is input in the R Z} rotation angle designation field 242, the end effector model EEM in the image display field 141 is automatically rotated and displayed in accordance with the rotation of the end effector model EEM. Further, when the end effector model EEM displayed in the image display field 141 is dragged and rotated, the value of the _{R Z rotation angle displayed in the R Z} rotation angle designation field 242 is also changed accordingly.
( _RY designation part)

Further, in step S3403, the gripping posture _RY is specified. Here, with R _Y designation screen 250 shown in FIG. 37 designates the R _Y rotation angle from R _Y rotation angle designation column 251. In this example, the postures of the end effector model EEM and the work model WM11 are expressed by ZZ Euler angles. Therefore, the direction of the rotation axis of _RY can be corrected by using the value of _{R Z} specified in step S3402. The calculated correction rotation Y-axis AXY, which is the calculated correction rotation axis, is displayed in the image display field 141 in the same manner as described above. By displaying the rotation axis in this way, it is easy for the user to imagine how the end effector model EEM changes when the _{RY rotation angle is adjusted, and it is easy to set.}
( _RX designation part)

Finally, in step S3404, it specifies the gripping posture R _X. Here, with R _X designation screen 260 shown in FIG. 38 designates the R _X rotation angle from R _X rotation angle designation column 261. Again, the correction rotation X axis AXX indicating the direction of the axis of rotation of the R _X, is displayed in the image display field 141.

Thus, as the gripping position for gripping the workpiece with an end effector, X, Y, Z, R X, R Y, six positional parameters R _Z can be specified by the user. In particular, the adjustable position parameter is limited for each screen, and the rotation axis to be displayed is also limited to the display of only the rotation axis related to the specified position parameter, so that the user can set according to such guidance. Therefore, the required position parameters can be specified in sequence. As a result, it is possible to smoothly perform the work of designating the gripping position by designating the three-dimensional position and posture, which has been difficult in the past. Further, _{when setting the R X} , _RY , and R _Z rotation angles of Euler angles, the rotation axis is calculated and displayed in the image display field 141 as the correction rotation axis, so that the position parameter set by the user is set. With this change, you can easily imagine how it works.

Further, by dividing the registration of the gripping position into a plurality of steps and limiting the position parameters that can be adjusted in each step, it becomes possible to easily imagine how the position parameters move. In particular, by providing a designation screen for each step and registering different position parameters on different designation screens, the position parameters to be set can be presented to the user without confusion, and the setting work can be guided. In the examples of FIGS. 36 to 38, an example in which different designation screens are provided has been described, but the present invention is not limited to this example. For example, as shown in the position parameter designation screen 270 of FIG. 39, it is possible to limit the position parameters that can be specified for each step while using a common designation screen for designating a plurality of position parameters. In the example of FIG. 39, only the X and Y coordinates can be specified from the X and Y coordinate designation fields 271, and other position parameters are grayed out so that they cannot be specified and cannot be selected. After specifying the X and Y coordinates, by pressing the "Next" button 272, another position parameter, for example, only the Z coordinate and the R _Z rotation angle, are specified as the _{Z coordinate and the R Z rotation angle as the next step.} It is possible to specify from column 273, and all other position parameters, including the specified X and Y coordinates, are grayed out or hidden so that they cannot be selected. Then, by pressing the "Next" button 272 again after specifying the Z coordinate and the R _Z rotation angle, as the next step, another position parameter, for example, R _X , R _Y rotation angle only, is R _X , R _Y rotation. It can be specified from the angle specification field 274, and other position parameters cannot be selected. Even with such a configuration, it is possible to realize a guidance function in which the position parameters that can be specified are limited and presented to the user and are sequentially set.

Furthermore, the number and division of steps that sequentially define the position parameters are not limited to the above examples. For example, the Z coordinate and the R _Z rotation angle may be set in separate steps, or the _RX rotation angle and the _RY rotation angle may be set in the same step. For example, as an example of dividing a position step into a plurality of steps, the Z-R _Z designation part is divided into a Z designation part for specifying the Z coordinate and an R _Z rotation angle for designating the R Z rotation angle centered on the Z axis. It consists of _{Z designation part.} Alternatively, the R _X -R _Y designating unit, R _Y designation for designating a R _X designating section for designating a R _X rotation angle around the R _Y rotation angle around the Y axis X axis It can also be composed of parts. Alternatively, integrated FIGS. 37 and 38, the R _X rotation angle and R _Y rotation angle may be R _X -R _Y specify unit which can be specified on a single screen.

In the above example, the position and orientation of the end effector model are adjusted while the work model side is fixed. However, the present invention is not limited to this, and the position on the work model side is adjusted with the end effector model fixed. And the posture may be adjusted. Alternatively, it may be configured to adjust the position and orientation of both the end effector model and the work model.

Further, in the above example, _{the six position parameters of X, Y, Z, R X} , _RY , and R _Z are set to the X and Y coordinates in the XY designation part, and the Z coordinate and R _{Z in the Z-R Z designation part.} An example in which the R _X and R _Y rotation angles are specified in order in the rotation angle and R _{X- RY} designation section has been described, but the order and combination in which each position parameter is specified is not limited to the above configuration. For example, the X-Y-Z designation unit, and R _Z designation unit, R _X -R _Y specify section was prepared, X-Y-Z at the specified portion, X of the end effector model or the work model, Y, Z-coordinate Then, in the R _{Z designation part, specify the R Z} rotation angle centered on the Z axis, and finally in the R _{X −} R _Y designation part, specify the R _X rotation angle centered on the X axis and the Y axis. _{The RY} rotation angle centered on may be specified. In this case, as the XYZ designation unit, for example, the XYZ designation screen 280 as shown in FIG. 40 can be used, and X is applied to the end effector model EEM and the work model displayed three-dimensionally. , Y, Z coordinates can be specified from the X, Y, Z coordinate designation field 281. As described above, the image for first designating the gripping position is not limited to the plane image projected on the XY plane as shown in FIG. 35B, and different projection directions can be used. It can also be specified using the perspective view WM11P projected from the direction.

Further, the method is not limited to the method of designating the gripping position from the image once projected on the plane as shown in FIG. 35B described above, and the gripping position may be designated in the non-projected state as shown in FIG. 35A. For example, in a state where the work model WM11 is displayed three-dimensionally on the three-dimensional image viewer 290 as shown in FIG. 42, the gripping position is specified from this screen using an operation unit configured by a pointing device such as a mouse. To do.
(Modification example)

Furthermore, in accordance with Z-Y-X based Euler angles in the above example, X, Y, Z, R X, R Y, has been described how to specify the six positional parameters of R _Z, the present invention in this aspect Other aspects that define the position and orientation of end effectors and workpieces, such as XYZ Euler angles, XZZ Euler angles, YXX Euler angles, YZ -X-based Euler angles, Z-XY-based Euler angles, XY-X-based Euler angles, X-Z-X-based Euler angles, Y-XY-based Euler angles, Y-Z-Y-based Euler angles , ZXXZ Euler angles, ZZ Euler angles, roll pitch / yaw angles, rotation axes / rotation angles, etc., the positions and orientations of end effectors, etc. can be specified. ..

Further, in the above example, an example in which only the correction rotation axis related to the position parameter to be specified is displayed as the correction rotation axis has been described, but the other correction rotation axes are limited to the configuration in which they are completely hidden. Not. For example, the same effect can be obtained by displaying three orthogonal axes including other correction rotation axes and displaying the rotation axis related to the rotation angle to be specified more emphasized than the other correction rotation axes. Obtainable. As an example, the rotation axis for which the rotation angle is specified is displayed in bold, highlights such as coloring and blinking are displayed, and conversely, the rotation axis that cannot be specified is grayed out or displayed with thin lines. , Make a distinction in appearance. Especially in the three-dimensional display of an image, if there are orthogonal coordinate axes, the user can intuitively understand that the three-dimensional display is in progress, so while maintaining the three-axis display, the target rotation axis can be set to another. By making a distinction, it is possible to prevent users from being confused. Further, the origin when displaying such three axes is preferably the center of rotation. For example, it is set to an intermediate position between the pair of claws, which is the gripping position of the end effector model EEM described above.
(Grip position copy function)

Further, when registering a plurality of gripping positions, it may be provided with a gripping position copy function that reads out the already registered gripping positions, changes the gripping position based on the registered gripping positions, and enables registration as a new gripping position. .. In general, a plurality of gripping positions are often registered for one work. This is because if a plurality of gripping positions are registered, the optimum solution can be selected from a plurality of gripping solutions, so even if the obtained gripping solution candidates interfere with each other, other gripping solutions can be selected. This is because if there is a candidate, the possibility of being determined to be grippable increases. When registering a plurality of such gripping solutions, if the gripping registration is performed from the beginning each time, the number of steps is increased and the work becomes troublesome when registering the same gripping position. Therefore, by copying the already registered gripping position, changing a part of the position parameters set at this gripping position, and making it possible to save as a new gripping position, a plurality of gripping positions can be saved. Can be easily registered. Similarly, it is also possible to read out the existing gripping position, modify the position parameter as appropriate, and save by overwriting.
(Gripping position copy unit 8d8)

Such a gripping position copying function and a gripping position editing function are realized by the gripping position copying unit 8d8 shown in FIG. The gripping position copying unit 8d8 reads out the gripping positions of the work model and the end effector model stored in the gripping position saving unit 9b, changes the position parameters constituting the gripping position, and registers them. Further, when changing the gripping position in the gripping position copying unit 8d8, if the order is registered according to the guidance as described above, it becomes easier to understand which position parameter should be changed in which step. Registration of grip is easy.

Further, as described above, when the gripping position is specified, a fitting function for automatically moving the end effector model to the gripping position of the work model may be added. For example, by providing the "fit" button 154 in the operation field 142 on the Z-R _Z designation screen 240 of FIG. 36, the fit function can be executed and the Z coordinate can be automatically set.
(Embodiment 5)

Further, in order to improve the accuracy of the three-dimensional search, it is possible to limit the postures and angles that the search model can take. For example, consider detecting the position and orientation of a work by a three-dimensional search from a work group in which a plurality of flat work WK9s as shown in FIG. 43 are stacked separately. In this case, each surface of the work model WM9 in which the work WK9 of FIG. 43 is shown by three-dimensional CAD data is as shown in FIGS. 44A to 44F. That is, FIG. 44A is a plan view of the work model WM9, FIG. 44B is a bottom view, FIG. 44C is a front view, FIG. 44D is a rear view, FIG. 44E is a right side view, and FIG. 44F is a left side view. From these basic direction images, the overlapping surfaces are excluded and registered as a search model. Here, it is assumed that the four surfaces of FIGS. 44A, 44B, 44C, and 44E are registered. In this state, a group of loosely stacked works is imaged, point cloud data having three-dimensional information is acquired, and a three-dimensional search is performed. As a result, it may be erroneously detected as if the work exists in the vertical posture as shown by X in FIG. 45. In reality, it is unlikely that a thin flat work is present in an upright posture, but the front view of the flat work is almost linear or rectangular as shown in FIG. 44C, so that the surface of the work is It may be common to other parts constituting the above, for example, a part of the shapes of FIGS. 44A and 44B, and as a result, erroneous detection occurs.

On the other hand, it is conceivable to perform a three-dimensional search by limiting the postures that the work can take, but it is extremely difficult to set the conditions for limiting the postures. For example, it is known that all postures of a work can be expressed by defining the rotation angle ranges of the Z-axis, the Y-axis, and the Z-axis by using the Euler angles of the ZYZ system as shown in FIG. However, it is not easy to specify the rotation angle of each axis so as to be within the posture range desired by the user by this method.

Therefore, in the fifth embodiment, the posture that the user can take the work can be easily set without using such a troublesome concept. Specifically, when registering a search model, such a surface is prevented from being detected by eliminating a posture that is unlikely to occur when the search model is piled up in bulk. For such a setting, the search model registration unit 8g shown in FIG. 6 is used. The search model registration unit 8g registers as a search model for performing a three-dimensional search for specifying the position and orientation of each work for a plurality of work groups stacked separately from any of the basic direction images. Select a basic orientation image. The difference between the search model registration unit 8g and the basic direction image selection unit 8e is that the basic direction image selection unit 8e selects one of the basic direction images having the same appearance. On the other hand, the search model registration unit 8g is for excluding the basic direction image to be excluded from the search in the three-dimensional search from the registration target of the search model. Preferably, in a state where the basic direction image selection unit 8e excludes the basic direction images whose appearances overlap, the search model registration unit 8g further excludes unnecessary basic direction images from the registration target of the search model. Alternatively, select the basic direction image you want to search. As a result, the number of candidates can be reduced and the search model registration work can be performed more easily by allowing the user to select the basic direction image that is a candidate for registration of the search model in advance.
(Search model registration screen 130B)

Here, FIG. 47 shows an example of the search model registration screen 130B for registering the search model for the work of FIG. 43 described above. In the search model registration screen 130B shown in this figure, from the work model WM9 corresponding to the work WK9 in FIG. 43, six basic direction images corresponding to the six views generated by the basic direction image generation unit 8e'are displayed. The basic direction image selection unit 8e removes the basic direction images whose appearances overlap in advance, that is, the two images of FIGS. 44D and 44F in the example of FIG. 44, and displays the remaining four basic direction images on the display unit 6. It is displayed in the view display area 3a. In this state, as a form of the search model registration unit 8g for setting whether or not the search model can be registered, a selection check box 131B for "register" displayed on each basic direction image on the search model registration screen 130B is provided. There is. If the user checks the selection check box 131B, this basic direction image is registered as a search model, and conversely, if the selection check box 131B is cleared, this basic direction image is excluded from the registration target of the search model. The user can eliminate the posture that is unlikely to occur in the actual loose stacking state while looking at each basic direction image displayed in the hexagonal view display area 3a. In the example of FIG. 47, the selection check box 131B of the model C and the model D is turned off, and the posture in which the side surface can be seen, that is, the posture in which the plate-shaped work is upright is excluded from the target of the three-dimensional search. Can be done. As a result, the user eliminates images that do not need to be searched or selects only the necessary images while looking at the basic direction image showing the posture of the work, without going through the troublesome work of calculating the angle of the work and specifying the range. Therefore, it is possible to easily limit the posture of the work to be searched in three dimensions.

The search model registration unit 8g is not limited to the mode in which the user manually selects the search model registration target or exclusion target, but also calculates the shape and center of gravity of the work model to obtain a basic direction image of a posture that is unlikely to occur. May be configured to be automatically extracted and eliminated.

Alternatively, a combination of automatic calculation and manual selection, for example, as shown in FIG. 47, on the search model registration screen 130B, while displaying all the basic direction images selected by the basic direction image selection unit 8e, it is determined that the calculation is unlikely to occur. The selected basic direction image may be configured to be displayed by clearing the selection check box 131B as an initial state. The user can manually correct the selection as necessary while referring to the result of the automatic determination by the search model registration unit 8g, and press OK to more reliably register the search model.

Further, in FIG. 47, an example in which selection by the basic direction image selection unit 8e is an automatic calculation has been described, but automatic determination by the basic direction image selection unit or manual selection can be combined in the same manner. For example, as a result of the judgment by the basic direction image selection unit, one of the images judged to be the basic direction image having a common appearance is grayed out and displayed in the hexagonal view display area so that the user can see the basic direction image. The result of the automatic determination by the selection unit can be confirmed. If the determination result is incorrect, the user can manually select the basic direction image and leave it as the basic direction image, or vice versa. In this way, by displaying the result of the automatic calculation and letting the user confirm it, the accuracy of image selection can be further improved. In addition, by setting the selection / non-selection in advance based on the automatic judgment result as the initial state, if the selection result by the automatic judgment is correct, the user can press OK to approve it, and the user can approve it. The effort can be minimized.

Further, the purpose of excluding unnecessary surfaces from the target of the three-dimensional search can be applied to other than the upright state of the work. For example, in a situation where a work group is given as an input image in a mode in which only a specific surface is displayed in addition to the mode in which the works are stacked completely randomly, erroneous detection can be avoided by eliminating the surface that is not displayed. .. In particular, it is effective for workpieces whose front and back surfaces are similar in shape and whose back surface is likely to be erroneously detected. In this way, by simply excluding the search model on the back surface, which is not actually visible, from the target of the three-dimensional search, the same effect as limiting the posture of the work surface so that the back surface side of the work is not detected can be obtained. ..
(Procedure to register search model)

The procedure for registering the search model with the posture restriction described above will be described with reference to the flowchart of FIG. 48. Here, the procedure for registering the three-dimensional CAD data of the work as a search model will be described.

First, in step S4801, the three-dimensional CAD data of the work is read. Next, in step S4802, the center of the circumscribing rectangular parallelepiped of the 3D CAD data model is corrected to the origin of the 3D CAD data. Further, in step S4803, a height image viewed from each of the "upper", "lower", "left", "right", "front", and "rear" directions is generated. Here, when the height image is generated from the three-dimensional CAD data, the height image is generated so that the origin of the CAD is the center of the height image. Furthermore, in step S4804, the height image having the same appearance is deleted from the generated height images.

Next, in step S4805, a search model to be used for the three-dimensional search is selected from the remaining height images. Here, the search model selection unit 8i eliminates unnecessary basic direction images and selects necessary basic direction images.

Finally, in step S4806, the selected height image is registered as a search model. Here, the search model is registered by the search model registration unit 8g. By eliminating the basic direction image of the unnecessary posture in this way, it is possible to register the search model in a state where the posture is substantially restricted.
(Embodiment 6)

In the above, an example of limiting the target of the three-dimensional search by using an actual image instead of a numerical value has been described. However, the present invention is not limited to this aspect, and instead of or in addition to this, the posture can be restricted by the tilt angle or rotation angle of the search model. As a result, it is possible to appropriately set the conditions for the three-dimensional search according to the actual stacking method of the workpieces and reduce the false detection of the search. An example of the tilt angle / rotation angle setting screen 160 that limits the posture according to the tilt angle and the rotation angle is shown in FIG. 49 as the sixth embodiment. The tilt angle / rotation angle setting screen 160 shown in this figure has a posture restriction search model selection field 161 for selecting a search model for posture restriction, and an inclination angle upper limit setting field for designating an upper limit of an angle that allows tilt. 162, a rotation angle range setting field 163 that defines an angle range that allows rotation is provided.
(Inclination angle / rotation angle setting screen 160)

In the inclination angle and rotation angle setting screen 160, three-dimensional orientation relative to the workpiece, for example R _X indicating the rotation angle of the coordinate axes, R _Y, rather than the esoteric methods such specified by R _Z, are familiar As parameters that are easy to understand without it, the "tilt angle" and "rotation angle" from the posture at the time of registering the search model are specified. Here, the basic direction image registered as the search model is used as the registered posture, and the tilt angle and the rotation angle from this state are specified (details will be described later).

Specifically, on the tilt angle / rotation angle setting screen 160 of FIG. 49, the posture restriction search model selection field 161 selects a search model to which the attitude restriction is applied. In the example of FIG. 49, the search model of the model A registered on the search model registration screen 130B of FIG. 47 is selected.

Further, in the tilt angle upper limit setting field 162, it is set how many times the tilt angle is output as the result of the three-dimensional search with respect to the posture at the time of registration.

Further, in the rotation angle range setting field 163, a reference angle and an angle range are set. First, in the reference angle setting field, the rotation angle with respect to the posture at the time of registration is input. The rotation angle set here serves as a reference angle when specifying the rotation angle range. Further, in the range setting field, the rotation angle range to be output as a result of the three-dimensional search is set with respect to the reference angle set in the reference angle setting field.
(Posture restriction method by tilt angle / rotation angle)

Here, a method of obtaining the tilt angle and the rotation angle from the three-dimensional posture of the search model will be described. In the above-mentioned method of limiting the surface to be registered as a search model in the three-dimensional search, the posture is automatically restricted. On the other hand, in order to impose posture restrictions on the surface registered as a search model, the fact that the surface is registered is used.

Here, the posture is restricted by two angles, the "tilt angle" with respect to the registered posture and the "rotation angle" viewed from directly above with respect to the registered posture. With these two angles, it is easy to understand as a concept, and the angle can be limited relatively easily.
(Inclination angle)

First, as shown in FIGS. 50A to 50C, the inclination angle is defined by the angle of the posture of the work model at the time of registration as a search model with respect to the Z vertical direction. FIG. 50A shows the posture of the work model WMR at the time of registering the search model and the Z axis defined as the vertical direction of the work model. On the other hand, it is assumed that the posture of the work WKI obtained as the input image and the Z'axis defined as the vertical direction of the work WKI are in the state shown in FIG. 50B. In such a state, the "tilt angle" is defined as the tilt angle between the Z axis and the Z'axis, that is, the vertical tilt of the work, as shown in FIG. 50C. Thereby, regardless of the direction of inclination, it can be expressed by one angle, that is, the inclination from the registered state of the search model. In addition, there is an advantage that it is easy for the user and the concept to understand.
(rotation angle)

On the other hand, the rotation angle is defined by the vertical direction at the time of registration, that is, the rotation angle when viewed from the Z axis. Here, the rotation angle will be described with reference to FIGS. 51A to 51C. In these figures, FIG. 51A shows the work model WMR at the time of registering the search model and the Y axis defining the XY plane. On the other hand, it is assumed that the posture of the work WKI obtained as the input image and the Y'axis defining the XY plane are in the state shown in FIG. 51B. In such a state, as shown in FIG. 51C, the "rotation angle" is defined as the rotation angle when viewed from the vertical direction, that is, the Z axis. In this example, the rotation angle is defined by the angle of the Y-axis viewed from the Z-axis direction with reference to the coordinate axis in the Y-axis direction that defines the work model and the work. By defining in this way, it is possible to obtain an advantage that the user can easily grasp the angle as the same concept as the angle in the conventional two-dimensional pattern search.

A known algorithm can be used as a method for defining the origin and the XYZ axes for the work. For example, as shown in FIG. 8, a figure circumscribing the work, for example, a rectangular parallelepiped or a sphere is calculated from the shape information of the work held by the work model or the input image, the center of gravity is calculated, and this is set as the origin of the work, and XYZ Define the axis. In the method of defining the XYZ axes, when CAD data is used, the direction of the coordinate axes of the original CAD data may be used as it is as the direction of each XYZ axes. When using the measured data obtained by actually capturing the work in 3D, the vertical direction of the measured data is the Z axis, the Y axis is the upward direction of the measured data in the plan view, and the X axis is the right direction of the measured data in the plan view. It should be done. As for the origin, the center of gravity of the work model or the center coordinates of the CAD data may be used as the origin.
(Procedure to obtain the tilt angle and rotation angle from the three-dimensional posture)

Here, the procedure for obtaining the tilt angle and the rotation angle from the three-dimensional posture will be described with reference to the flowchart of FIG. First, in step S5201, the inclination angle is obtained. For example, when the posture of the work WKI obtained as an input image is as shown in FIG. 53B with respect to the work model WMR at the time of registration as shown in FIG. 53A, the Z-axis is vertical as shown in FIG. 53C. Find the tilt angle as the tilt in the direction.

Next, in step S5202, the input image is three-dimensionally rotated so that the Z'axis coincides with the Z axis. In other words, eliminate the slope. For example, with respect to the input image WKI of FIG. 53C, as shown in FIG. 54A, the outer product vector VP of the Z'axis vector and the Z axis vector is used as the rotation axis, and the Z'axis is rotated so as to overlap the Z axis. As a result, the rotated input image WKI'as shown in FIG. 54B is obtained, and the inclination is removed.

Finally, in step S5203, the rotation angle is obtained. For example, the Y-axis of the work model WMR at the time of registration shown in FIG. 55A and the Y-axis of the input image WKI'with the inclination removed as shown in FIG. 55B are obtained. In this state, as shown in FIG. 55C, the angle between the Y-axis and the Y'axis when viewed from the vertical direction, that is, the Z-axis is obtained as the rotation angle. In this way, the tilt angle and the rotation angle can be obtained from the three-dimensional posture of the input image.
(Setting the upper limit of the tilt angle)

As an example of setting the upper limit of the tilt angle on the tilt angle / rotation angle setting screen 160 of FIG. 49, for example, in the case of a metal work having a strong gloss, the range in which three-dimensional measurement cannot be performed increases as the surface is tilted. If the three-dimensional search is performed in such a state, the risk of the three-dimensional search being erroneously detected increases. Therefore, it is possible to leave only highly reliable search results by excluding the posture that is tilted too much by a certain amount or more, which has a risk of false detection, from the target of the three-dimensional search. For example, for a work that is easily erroneously detected when tilted by 40 degrees or more, the search result tilted by 40 degrees or more can be excluded by setting the tilt angle upper limit to 40 degrees.
(Setting the upper limit of the rotation angle range)

On the other hand, as an example of setting the rotation angle range on the tilt angle / rotation angle setting screen 160 of FIG. 49, for example, the model A and the model B of FIG. 47 described above have similar overall shape features even when rotated 180 degrees. Since there are many parts that are present, if the number of parts that cannot be measured three-dimensionally due to the influence of multiple reflections or the like increases, there may be a case where the rotational posture is mistaken by 180 degrees. In addition, when only a certain direction is generated as the actual way of stacking the workpieces, the result of the opposite direction can be excluded by using this posture restriction. For example, when the rotation angle range is set to ± 30 degrees with respect to the reference angle, only the search results within this angle range are detected.
(End effector mounting position setting screen 170)

The above method of defining the position and posture using Euler angles can be used not only for defining the position and posture for gripping the work model with the end effector model, but also for defining the position for attaching the end effector to the tip of the arm of the robot. .. Here, a procedure for setting a position where the end effector is attached to the tip of the arm portion of the robot will be described. Such setting of the end effector mounting position is performed on the end effector mounting position setting screen 170 as shown in FIG. 56. The end effector mounting position setting screen 170 shown in this figure includes an image display field 141 and an operation field 142. The end effector model is displayed in the image display field 141. The end effector model reads the three-dimensional CAD data created in advance. Alternatively, the end effector may be imitated and displayed by being composed of basic figures such as a rectangular parallelepiped and a cylinder.

The operation column 142 is provided with an end effector mounting position setting column 171 that defines a mounting position of the end effector mounted on the tip of the arm portion. The end effector mounting position parameters set in the end effector mounting position setting field 171 are the mounting position (X, Y, Z) and the mounting posture ( _RX , _RY , R _Z ) of the end effector model. For example, Euler angles are used to define the center of the flange surface FLS at the tip of the arm.

In the registration screen of the gripping position of the work model shown in FIG. 14 and the like described above, the positions of the end effector model portion (for example, a pair of claws provided at the tip) and the gripping work are held. You have registered a relationship. On the other hand, on the end effector mounting position setting screen 170, the mounting position for mounting the end effector model on the tip of the arm portion is registered. Therefore, it is not necessary to display the work model or the like, and the end effector model and the flange surface FLS are displayed in the image display field 141.
(Grip position designation screen 180)

On the other hand, on the gripping position designation screen 180 for designating the gripping position of the work model, as shown in FIG. 57, the end effector model and the work model are displayed, and the position where the work model is gripped is designated. The gripping position designation screen 180 shown in this figure is provided with an image display field 141 and an operation field 142. In the image display field 141, a work to be gripped is selected and the surface for gripping the work is directed to the upper surface. The image is drawn and displayed in three dimensions in a vertical posture. Furthermore, the end effector model is displayed and adjusted to the position and posture in which the work model is gripped. The operation column 142 is provided with a grip position designation column 181 for designating the grip position. The user adjusts the position parameter from the gripping position designation field 181 so that the relative relationship between the work model and the end effector model is correct. Alternatively, the position and posture can be adjusted on the screen by dragging the end effector model or work model displayed in the image display field 141. The position parameter adjusted on the screen is reflected in the gripping position designation field 181. The Euler angles described above can also be used for the notation of the position parameters in the gripping position designation field 181. (Multiple gripping position selection screen 190)

Further, the registration of the gripping position is not limited to one place with respect to one surface (for example, the basic direction image) of one work, and a plurality of places can be specified as described above. It is also possible to confirm the gripping positions registered at a plurality of locations with an image. For example, FIG. 58 shows an example of a plurality of gripping position selection screens 190. The multiple gripping position selection screen 190 also includes an image display field 141 and an operation field 142. The operation column 142 is provided with a surface selection field 191 for selecting a surface of the work model whose grip position is defined, and a grip position list display 192 for displaying a list of grip positions set for the surface selected in the surface selection field 191. Be done. In this example, the surface C is selected in the surface selection field 191 and the gripping positions 1 to 3 are listed in the gripping position list display 192 as the gripping positions set on the surface C. When gripping 2 is selected, the image display field 141 displays how the end effector model grips the work model at the gripping position and posture registered in gripping 2. In this way, the user can select and confirm a plurality of registered gripping positions. In addition, the registered gripping position can be modified and updated as needed. Here, too, the gripping posture can be displayed using the Euler angles described above. In this way, while registering using Euler angles, the user side can display them in an easy-to-understand manner without having to understand the esoteric concept of Euler angles in detail, and also correct the rotation axis. By displaying and limiting the parameters that can be adjusted by the guidance function, it is possible to reduce confusion and proceed with the setting work.
(Calibration function for end effector mounting position)

As described above, the end effector is attached to the tip of the arm portion of the robot. On the other hand, the above-mentioned registration of the gripping position (teaching) and the gripping determination (simulation) of whether or not the end effector model can grip the work model at the registered position and posture are performed by the image processing device, not the actual end effector. It is performed on the virtual three-dimensional space on the side (robot vision side). Then, the conversion from the coordinate position of the vision space, which is a virtual three-dimensional space, to the coordinate position of the robot space, which is a real space, is performed by the conversion unit 8x based on the calibration information obtained by the calibration unit 8w of FIG. It is done.

However, if the mounting state of the end effector model virtually set on the image processing device side and the mounting state of the end effector EET of the actual robot RBT are different, the vision space and robot space specified in the calibration information will be displayed. The correspondence between the above is not maintained, and a deviation occurs when trying to grip the work using the actual robot RBT.

Here, FIG. 59 shows the positional relationship between the end effector model and the flange surface FLS. In this figure, the origin O _F flange face FLS of the tip of the arm of the robot, as the position of the end effector model EEM, the center O _E of the end effector model EEM, respectively show. The position of this end effector model EEM is set as the position (X, Y, Z) and the posture ( _RX , _RY , R _Z ) with respect to the flange surface FLS at the tip of the arm portion of the robot on the image processing device side. There is. That is, in the example of FIG. 59 as viewed from the origin O _F flange face FLS, the center O _E of the end effector model EEM, position (X, Y, Z) and attitude _{(R X, R Y, R} Z) is set Has been done. Here, if an error occurs in the actual mounting state of the end effector, the position of the end effector model EEM with respect to the origin of the flange surface FLS may shift. As a result, due to the mounting state of the end effector, the image processing device side and the robot side will grip at different positions.

Therefore, the present embodiment is provided with an end effector mounting position calibration function that reflects the actual mounting state of the end effector on the image processing apparatus side so as not to cause such a deviation or error. The calibration function of the end effector mounting position is performed by the end effector mounting position calibration unit 8y shown in FIG. Specifically, using 3D CAD data that virtually represents the 3D shape of the end effector, which is an end effector model used for registering the gripping position of the work and determining interference when the end effector is placed. , The image processing device detects such an error by performing a three-dimensional search on the measured data obtained by imaging the actual end effector and measuring it three-dimensionally and acquiring the position and orientation of the actual mounting position. Calibrate the mounting position setting for the side end effector. In this way, the user can display the difference between the point cloud obtained by measuring the actual end effector in three dimensions and the mounting state of the end effector model defined in the virtual three-dimensional space on the image processing device side on the image display area of the display unit. You can adjust it manually while checking with.
(Procedure for automatic deviation correction)

Here, a calibration function for automatically correcting an error between an actual end effector and an end effector model composed of three-dimensional CAD data or the like will be described with reference to the flowchart of FIG. 60. The calibration unit 8w shown in FIG. 17 executes calibration for converting the coordinate position of the vision space, which is a virtual space, into the coordinate position of the robot space, which is a real space, and stores the calibration information in the storage unit 9. It is assumed that it is gripped by. Further, it is assumed that the end effector model is also prepared as three-dimensional CAD data.
(End effector imaging screen 330)

First, in step S6001, the sensor unit prepares for imaging an actual end effector. Specifically, the robot is operated to move the end effector so that the end effector is captured by a three-dimensional camera, which is a form of the sensor unit. Next, in step S6002, three-dimensional measurement is performed on the end effector.

As one aspect of the end effector imaging unit for imaging the actual end effector with the sensor unit, for example, the end effector imaging screen 330 shown in FIG. 61 is used. The end effector imaging screen 330 is provided with an image display field 141 and an operation field 142. In the image display field 141, the end effector EET attached to the flange portion at the tip of the arm portion ARM imaged by the sensor portion 2 is displayed in real time. Further, the operation column 142 is provided with an end effector position designation column 331 indicating the position and orientation of the end effector EET. From the end effector imaging screen 330, the user manually operates the robot so that the end effector EET of the robot is projected on the cameras constituting the sensor unit. At this time, the object of imaging is not the claw portion of the end effector EET that grips the work, but the mounting state with the arm portion. Therefore, it is desirable to take an image so that the area of the end effector EET is large. Preferably, the end effector EET is changed from a downward posture in which it is easy to grip a normal work to a sideways posture so that the entire image is captured by the camera. In this way, the user presses the "imaging" button 332 after positioning the posture of the end effector EET. As a result, a three-dimensional captured image of the end effector is acquired. In other words, three-dimensional measurement is performed on the end effector.

Next, in step S6003, the position / orientation A of the flange portion of the robot coordinate system is acquired. Here, the position / orientation A of the robot coordinate system is, for example, the position and orientation of the robot coordinate system of the flange portion FLS to which the imaged end effector is attached. The order of the steps of step S6003 and step S6002 may be interchanged.

Further, in step S6004, the position / orientation A of the flange portion of the robot coordinate system is converted into the position / orientation B in the vision space. Here, the conversion unit 8x of FIG. 17 converts the real space and the virtual space based on the calibration information for performing the coordinate conversion between the robot space and the vision space obtained by executing the calibration in advance.

Further, in step S6005, a three-dimensional search is performed to detect the position / orientation C of the end effector in the vision space. Here, using the CAD data used for the end effector model as a search model, a three-dimensional search is performed on the three-dimensional measurement data obtained in step S6002. As the three-dimensional search method, a known algorithm can be appropriately used. As a result, the position and orientation of the end effector in the vision space are detected.

Further, in step S6006, the relative position / orientation of the end effector position / orientation C with respect to the position / orientation B of the flange portion FLS on the vision space is calculated. The coordinate position obtained here is the position and orientation of the end effector model that is accurate with respect to the flange portion FLS, in other words, in consideration of misalignment and the like.

Finally, in step S6007, the obtained position and orientation are reflected in the end effector setting on the vision side. That is, it is reflected in the position and orientation setting of the end effector model on the vision side. In this way, the actual mounting state of the end effector can be automatically reflected on the vision side.
(Procedure for manual correction of deviation)

The procedure for automatically correcting the deviation between the robot space and the vision space has been described above. However, the present invention is not limited to the configuration in which the deviation correction is automatically performed, and the deviation can be manually performed. Next, the procedure for manually performing the deviation correction will be described with reference to the flowchart of FIG. 150. Here, in steps S1501 to S1504, the procedure is the same as in steps S6001 to S6004 of FIG. 60 relating to the above-mentioned automatic correction, and detailed description thereof will be omitted. That is, in step S1501, the robot is operated so that the end effector of the robot is reflected on the three-dimensional camera, three-dimensional measurement is performed on the end effector in step S1502, and the position and orientation of the flange portion of the robot coordinate system are performed in step S1503. A is acquired, and in step S1504, the position / orientation A of the flange portion of the robot coordinate system is converted into the position / orientation B in the vision space.

Next, in step S1505, the position / orientation D of the end effector is calculated from the position / orientation B of the flange portion on the vision space based on the end effector setting. Here, the position and orientation D of the end effector is obtained from the position and orientation B of the flange portion, which is the tip of the robot in the vision space, and the position and orientation information of the end effector with respect to the flange portion set in the end effector setting.

Next, in step S1506, the three-dimensionally measured point cloud and the CAD display of the position / orientation D of the end effector are superimposed and displayed. Here, the end effector model EEM, which is the three-dimensional CAD data of the end effector, is displayed at the position of the position / orientation D of the end effector obtained in step S1505, and the points obtained by three-dimensional measurement of the actual end effector. The group PCs are superimposed and displayed. Examples of the deviation correction screen with such a superimposed display are shown in FIGS. 151 and 152.

Then, in step S1507, it is determined whether or not there is a non-negligible deviation between the point cloud and the CAD display. If there is a deviation that cannot be ignored, the process proceeds to step S1508, the end effector setting is changed so as to correct the deviation, and the process returns to step S1505 to repeat the above-described processing. This end effector setting includes the position and orientation of the end effector with respect to the flange portion. On the other hand, if there is no deviation, the process ends.
(Displacement correction unit)

An example of a deviation correction screen is shown in FIGS. 151 and c as one aspect of the deviation correction unit that performs such deviation correction. In these figures, FIG. 151 shows the case where there is a non-negligible deviation. In the deviation correction screen 840 shown in this figure, the point cloud PC is superimposed and displayed on the end effector model EEM in the image display field 141. As shown in this figure, it can be confirmed that the point cloud PC in which the end effector is actually measured three-dimensionally, which is indicated by the white dot, is displaced diagonally downward to the left from the end effector model EEM.

Further, the operation column 142 is provided with an end effector setting column 841 which defines the position (X, Y, Z) and the posture ( _RX , _RY , R _Z ) of the end effector with respect to the flange portion. Here, the coordinate axis of the longer XYZ shown on the right in figure shows the origin O _F of the flange portion FLS is robot tip. The coordinate axes of the shorter XYZ shown in the left side, represents with respect to the robot distal (flange portion), the origin O _E of the three-dimensional CAD data of the end effector model EEM. The user adjusts the position and orientation of the end effector model EEM while checking the end effector model EEM and the point cloud PC displayed in the image display field 141. For example, in the end effector setting field 841 of the operation field 142, the position and orientation of the end effector model EEM can be specified numerically, or the end effector model EEM can be dragged and moved on the image display field 141 to move the end effector model EEM. Is adjusted so that it overlaps with the point cloud PC.

An example of matching the end effector model EEM with the point cloud PC in this way is shown in FIG. 152. In this example, in the end effector setting field 841, the position in the Z direction is set to 90 mm, and the end effector model EEM is offset in the Z direction to match. After adjusting the position and orientation of the end effector model, pressing the "OK" button 842 updates the end effector settings. As a result, the position and orientation on the vision side can be adjusted according to the actual mounting state of the end effector.

Further, the manual correction of the deviation described above can be performed in combination with the automatic correction. Such an example will be described based on the deviation correction screen 850 of FIG. 153. The deviation correction screen 850 shown in this figure is provided with an end effector setting field 841 for manually adjusting the position and orientation of the end effector model EEM in the operation field 142, and an "automatic correction" button 843 at the bottom. .. When the "automatic correction" button 843 is pressed, a three-dimensional search is internally executed, and the position and orientation of the end effector model EEM are automatically corrected so as to match the point cloud PC. For example, while checking the states of the end effector model EEM and the point cloud PC superimposed and displayed in the image display field 141, it is possible to execute automatic correction of deviation and manually fine-tune the automatic correction result. In particular, if the end effector is made of a highly glossy metal and the point cloud PC is not detected accurately by the reflected light, and the automatic correction does not work well, for example, if the 3D search fails, it is done manually. It can be adjusted. In this way, the user can manually correct the deviation between the robot space and the vision space.

Conventionally, when a mounting member such as a connector is actually attached to the mounting part of the flange and the end effector, but the offset is forgotten on the vision side. , There was a problem that the position and posture were shifted. In such a case, if there is no function of confirming the deviation or a function of correcting the deviation, when the robot grips the robot during actual operation, it may not be gripped well or a problem such as a collision may occur. There are various factors that cause such an error, and in the past, it took a lot of time and effort to investigate the cause and debug. On the other hand, according to the present embodiment, it is possible to manually or automatically correct such a deviation, and it is possible to easily perform flexible adjustment according to the actual mounting state of the end effector.

In the example of FIG. 59, the position of the end effector model EEM is set as the center position of the three-dimensional CAD data, but the present invention is not limited to this example, and other positions can be specified. For example, in FIG. 59, the center of the rectangular parallelepiped circumscribing the three-dimensional CAD data is used, but the end portion of the tangent rectangular parallelepiped or the origin of the input source three-dimensional CAD data may be used as a reference.
(Procedure to register actual measurement data in search model)

The procedure for registering the three-dimensional CAD data as a search model has been described above. However, as described above, the present invention is not limited to the three-dimensional CAD data as the search model, and for example, the actual measurement data obtained by actually imaging the work by the sensor unit can be registered as the search model. Here, in step S2701 of FIG. 27, a procedure for registering such measured data as a search model will be described based on the flowchart of FIG. 62.

First, in step S6201, the work is placed on a flat surface with the surface to be registered of the work facing upward, and the sensor unit performs three-dimensional measurement.

Next, in step S6202, the obtained measured data is registered as a search model.

Finally, in step S6203, it is determined whether or not the search models required for the three-dimensional search have been registered. If not, the process returns to step S6201 and the above process is repeated to complete the registration of the required number. Ends the process. Details of these procedures will be described later with reference to FIGS. 102 to 106.
(Procedure 1 during actual operation)

With the necessary setting work completed as described above, the picking operation is performed on the work groups actually stacked separately. Here, in the state where the search model is registered in the procedure of FIG. 28, the procedure for determining whether or not the work can be gripped, that is, whether or not there is a gripping solution for each detected work, is performed in actual operation. This will be described based on the flowchart of FIG. 63. Here, the calculation unit 10 of FIG. 6 determines the presence or absence of a gripping solution.

First, in step S6301, three-dimensional measurement is started for the loosely stacked workpiece. Here, the sensor unit captures a group of workpieces stacked separately, performs three-dimensional measurement, and acquires a three-dimensional shape having height information.

Next, in step S6302, a three-dimensional search is performed on the three-dimensional shape of the obtained work group using the work model, and the position and orientation of each work are detected.

Next, in step S6303, for one detected work, the position and posture in which the end effector should be arranged are calculated based on the position of this work and the gripping posture of the work registered at the time of setting.

Next, in step S6304, an interference determination is performed using the end effector model to determine whether or not the end effector interferes with surrounding objects at the calculated position.

Then, in step S6305, it is determined whether or not the end effector interferes, and if it does not interfere, the process ends with a gripping solution for this work.

On the other hand, if it is determined that they are interfering with each other, the process proceeds to step S6306, and it is determined whether or not there is another gripping position registered for this work. If another gripping position is registered, the process returns to step S6303 and the above process is repeated for this gripping position.

On the other hand, if another gripping position is not registered, the process proceeds to step S6307 to determine whether or not there is another detected work. If there is another work, the process returns to step S6303, the work is replaced, and the above process is repeated. If there is no other work, the process ends with no gripping solution.

In this way, the calculation unit 10 of FIG. 6 determines whether or not there is a gripping solution capable of gripping the work. Then, when a gripping solution is obtained, an instruction is sent to the robot controller 6 to pick the work at the gripping position determined by the end effector. According to this, the robot controller 6 controls the end effector so as to pick the instructed work.

In the above procedure, when a gripping solution is obtained from any of the workpieces, the process of examining the gripping position is completed at that time, and the work is controlled to be gripped at the gripping position corresponding to the obtained gripping solution. ing. However, the method is not limited to this method, and for example, all gripping positions that can be gripped may be obtained as gripping position candidates, and then which gripping position candidate should be selected may be determined. For example, the evaluation index calculation unit 8q calculates a score as an evaluation index for each gripping position candidate, and selects the one with the highest obtained score as the gripping position. Also, paying attention to the height information on which the work is placed, the one with a high height on which the work is placed, in other words, the one located higher than the group of works stacked separately is selected as the gripping position. You can also do it. Preferably, the calculation unit 10 selects from a plurality of gripping solutions in consideration of both the score and the height information. By doing so, more appropriate picking can be performed.
(Procedure 2 during actual operation)

The procedure at the time of actual operation has been described above with the search model registered in the procedure of FIG. 28. When registering the search model, as shown in FIG. 48, it is also possible to register the search model in a state in which the posture of the search model is restricted. Here, a procedure at the time of actual operation in which a picking operation is actually performed on a group of works actually stacked in a state where the search model is registered in the procedure of FIG. 48 will be described with reference to the flowchart of FIG. First, in step S6401, three-dimensional measurement is started for the loosely stacked workpiece. Here, the sensor unit captures a group of workpieces stacked separately, performs three-dimensional measurement, and acquires a three-dimensional shape having height information.

Next, in step S6402, a three-dimensional search is performed on the three-dimensional shape of the obtained work group using a work model, and the position and orientation of each work are detected.

Next, in step S6403, the posture outside the range is excluded from the range setting of the tilt angle and the rotation angle.

Next, in step S6404, for one of the detected works, the position and posture in which the end effector should be arranged are calculated based on the position of this work and the gripping posture of the work registered at the time of setting.

Next, in step S6405, interference determination is performed using the end effector model to determine whether or not the end effector interferes with surrounding objects at the calculated position.

Then, in step S6406, it is determined whether or not the end effector interferes, and if it does not interfere, the process ends with a gripping solution for this work.

On the other hand, if it is determined that the work is interfering, the process proceeds to step S6407, and it is determined whether or not there is another gripping position registered for this work. If another gripping position is registered, the process returns to step S6404 and the above process is repeated for this gripping position.

On the other hand, if another gripping position is not registered, the process proceeds to step S6408 to determine whether or not there is another detected work. If there is another work, the process returns to step S6404, the work is replaced, and the above process is repeated. If there is no other work, the process ends with no gripping solution.
(Interference judgment)

Here, the method of interference determination using the end effector model in step S6304 of FIG. 63 and step S6405 of FIG. 64 described above will be described. When gripping the work with the end effector, if the end effector interferes with surrounding obstacles such as other workpieces and storage containers, gripping cannot be performed correctly. Therefore, when the end effector model grips the work model with the gripping position candidates in advance, the position and posture of the end effector are calculated by the interference determination unit 8m in FIG. 31 to determine the interference with the surrounding members. At this time, as the surrounding members, a group of loosely stacked works and a group of three-dimensional points of the storage container actually measured and obtained by the sensor unit 2 are used. It is also possible to register the position of the storage container or the like in advance and perform interference determination with the end effector model. On the other hand, the three-dimensional CAD data of the end effector is generally composed of polygon data. For example, STL data, which is often used as three-dimensional CAD data, is represented by a collection of minute triangles called polygons.

In order to make an interference judgment between such polygon data and 3D point cloud data, conventionally, it is examined whether each 3D point constituting the 3D point cloud data is inside or outside the end effector model. It was judged that if it was inside, it would interfere, and if it was outside, it would not interfere. However, in this method, it is necessary to perform calculation and contrast for each point, and the amount of calculation becomes enormous as the data becomes large.
(Procedure for interference judgment using cross-section model)

Therefore, in each embodiment of the present invention, a cross-section model is created from the polygon data of the end effector model, each point of the three-dimensional point cloud data is projected on the cross-section model, and the inside and outside are judged to determine the interference. ing. Such interference determination is performed by the interference determination unit 8m in FIG. 31. Here, the procedure for performing the interference determination will be described with reference to the flowchart of FIG. 65.

First, in step S6501, the polygon data of the end effector is read. Next, in step S6502, a cross-section model is created from the polygon data of the end effector. The cross-section model is generated by the cross-section model generation unit 8s of FIG. 26. Here, a method in which the cross-section model generation unit 8s creates the cross-section model will be described with reference to FIGS. 66 to 67E. First, the basic axis BSL is set for the polygon data of the end effector model EEM shown in FIG. Preferably, the basic axis BSL is set along the longitudinal direction of the end effector model EEM. Further, the end effector model is cut in an orthogonal plane orthogonal to the basic axis BSL to create a plurality of cross sections. Here, the position where the cross section is created is, for example, the vertex position of the polygon data. Alternatively, after creating a plurality of cross sections at regular intervals along the basic axis, the obtained cross sections may be arranged according to the shape. For example, cross sections of the same shape are excluded. When creating a cross section, it is preferable to set the basic axis BSL so that the total number of required cross sections is reduced. For example, the cross-section model generation unit may be configured to automatically calculate the direction of the basic axis BSL so that the number of cross-sections is reduced. Further, the designation of the cross-section position for acquiring the cross-section on the set basic axis BSL can be specified, for example, to automatically extract the cross-section having a large change.

In this way, a cross-sectional model having the shape of each cross section along the basic axis BSL of the end effector model EEM and the cross-sectional position on the basic axis BSL corresponding to each cross section can be created. For example, in the example of FIG. 66, the cross sections SS1 to SS5 having the shapes shown in FIGS. 67A to 67E can be obtained at the five positions of the cross sections SP1 to SP5 along the basic axis BSL, respectively. Using the cross-section model thus obtained, it is determined whether or not the end effector model interferes with the three-dimensional point cloud.

Specifically, in step S6503, a three-dimensional point to be determined for interference is selected from the three-dimensional point cloud, and from the position of this point in the basic axis BSL direction, among the plurality of cross sections of the cross-section model. , Select which cross section to judge interference with. For this selection, the cross-section position set for each cross-section is used. For example, consider a case where interference with the end effector model EEM is determined for the three-dimensional point TDP shown in FIG. 68A among the three-dimensional points. The three-dimensional point TDP is located between the cross-sectional positions SP3 and SP4 in the direction along the basic axis BSL. Therefore, the interference with the three-dimensional point TDP is determined by the cross section SS3 showing the cross-sectional shape during this period.

Specifically, in step S6504, the projection point PP3 projected from the three-dimensional point TDP onto the orthogonal plane including the cross section SS3 is calculated. Then, in step S6505, the interference determination is performed. Here, if the position of the projection point PP3 of the three-dimensional point is outside the cross section SS3 as shown in FIG. 68B, it is determined that there is no interference. On the other hand, if it is inside the cross section SS3 as shown in FIG. 68C, it is determined that the interference occurs. Finally, in step S6506, it is determined whether or not there are other three-dimensional points, and if unprocessed three-dimensional points remain, the process returns to step S6503 and the above processing is repeated. Then, when the interference determination is completed at all the three-dimensional points, the process ends.

In this way, it is possible to determine the interference between the measured three-dimensional point cloud and the end effector model. Further, although the interference determination with the three-dimensional point cloud data has been described above, the present invention does not limit the object of the interference determination to the three-dimensional point, but also determines the interference of other objects such as lines and surfaces by the same procedure. Can be carried out.

Further, in the above example, the example in which the CAD data of the end effector is polygon data has been described. However, as long as the cross-sectional shape of the end effector can be calculated, the interference determination is not limited to the polygon data but also in other CAD data formats. Can be done. Furthermore, in the above example, the case where the cross-sectional shape is represented by a two-dimensional plan view has been described, but the way of holding the cross-sectional shape data is not limited to this form, and the data in the form of, for example, a set of contour lines is used. You may have it.
(Interference judgment in an additional model with an additional area added)

Further, in the interference determination of the end effector model, the interference determination can be performed by using an additional model in which an additional area represented by a three-dimensional basic figure is added to the end effector model. The procedure for determining interference using such an additional model will be described with reference to the flowchart of FIG. 69. Here, an example will be described in which an additional region in which a rectangular parallelepiped and a cylinder are combined as a basic figure is subjected to interference determination with respect to the additional model added to the three-dimensional CAD model.

First, in step S6901, interference determination is executed for a rectangular parallelepiped region among the basic figures constituting the additional region. If it is determined in step S6902 that there is interference according to this result, the process of determining interference is stopped, the process is output as having interference, and the process is terminated.

On the other hand, if it is determined that there is no interference, the process proceeds to step S6903, and interference determination with a cylinder, which is another figure among the basic figures constituting the additional area, is performed. As a result, if it is determined in step S6904 that there is interference, the process of determining interference is stopped, the process is output as having interference, and the process is terminated.

On the other hand, if it is determined that there is no interference, the process proceeds to step S6905 to determine interference with the three-dimensional CAD data. As a result, if it is determined in step S6906 that there is interference, the process of determining interference is stopped, the process is output as having interference, and the process is terminated.

On the other hand, if it is determined that there is no interference, the process is terminated with no interference.

In this way, the interference determination is sequentially performed for each basic figure or for each three-dimensional CAD data, and when it is determined that there is interference, the interference determination process is stopped at that point and the interference determination is output. In the above, an example in which the additional area is composed of two basic figures, a rectangular parallelepiped and a cylinder, has been described, but the procedure is the same even if the number of basic figures increases, and interference determination is sequentially performed for each basic figure, and each time. If it is determined that there is interference, the process is stopped.
(Embodiment 7)
(Gripability judgment verification function)

The procedure for determining the presence or absence of a gripping solution as a result of the interference determination during actual operation has been described above. However, the present invention not only determines the presence or absence of a gripping solution, but also verifies the reason why a gripping solution could not be obtained for a candidate for a gripping position for which a gripping solution could not be obtained. May be provided. For example, the gripping solution candidates are displayed in a list, and as a result of the interference determination, the gripping position determined to have a gripping solution is displayed as OK, and the gripping position candidate determined to have no gripping solution is displayed as NG. In this state, by selecting the gripping position determined to be NG and displaying the reason why it was determined that there is no gripping solution, the user can refer to this information and whatever the gripping position is. For example, it becomes possible to examine whether it can be selected as a gripping solution, modify the gripping position, or add a new gripping position. Such an example is shown in the block diagram of FIG. 70 as the robot system 7000 according to the seventh embodiment. The robot system 7000 shown in this figure includes an image processing device 700, a display unit 3B, an operation unit 4, a sensor unit 2, a robot controller 6, and a robot RBT. The same members as those in FIG. 6 are designated by the same reference numerals, and detailed description thereof will be omitted as appropriate.
(Display unit 3B)

The display unit 3B includes an image display area 3b and a gripping solution candidate display area 3c. The gripping solution candidate display area 3c includes a work gripping possibility display area 3d and a work gripping impossible factor display area 3e.

The image display area 3b is a member for displaying an end effector model composed of three-dimensional CAD data, which virtually represents the three-dimensional shape of the end effector, in a three-dimensional shape in a virtual three-dimensional space. is there.

The gripping solution candidate display area 3c is a member for listing and displaying all the set gripping positions for any one of the search results of one or more workpieces searched by the three-dimensional search unit 8k. ..

The work gripping possibility display area 3d is a member for displaying the determination result of gripping possibility at the gripping position designated for each work by the three-dimensional pick determination unit 8l.

The work non-gripable factor display area 3e displays the factor that the work cannot be gripped at the gripping position determined by the three-dimensional pick determination unit 8l at the gripping position designated for each work. It is a member for making it.
(Image processing device 700)

The image processing device 700 includes an input image acquisition unit 2c, a calculation unit 10, a storage unit 9, an input / output interface 4b, a display interface 3f, and a robot interface 6b. The calculation unit 10 includes an end effector model registration unit 8u, a work model registration unit 8t, a grip position identification unit 8d, a search model registration unit 8g, a three-dimensional search unit 8k, a three-dimensional pick determination unit 8l, and an inclination. It is provided with an angle setting unit 8n.

The gripping position specifying portion 8d includes a work-side gripping location designating portion 8d1 and an end effector-side gripping setting portion 8d2. The work-side gripping location designation unit 8d1 creates a work model composed of three-dimensional CAD data that virtually represents the three-dimensional shape of the work in a state where the end effector model is displayed in the image display area 3b. It is a member for designating a gripping position when gripping with. The end effector side grip setting unit 8d2 is a member for designating a grip position for gripping the work with respect to the end effector model displayed in the image display area 3b.

The search model registration unit 8g performs a second work model that virtually represents the three-dimensional shape of the work, and performs a three-dimensional search that specifies the posture and position of each work for a plurality of work groups included in the input image. It is a member for registering as a search model for performing. It is preferable that the second work model registered as the search model is the same as the work model in which the gripping position is designated by the work side gripping point designation unit 8d1. As a result, the user can save labor in the setting work by sharing the search model for performing the three-dimensional search with the work model for designating the gripping position. In addition, even during actual operation, by matching the work model that searches for a work that can be gripped with the work model that determines gripping, the searched work model can be gripped at the gripping position set in this work model. Since it can be examined whether or not it can be processed efficiently.

The tilt angle setting unit 8n is a member for setting a range of allowable tilt angles with respect to the posture of the work.

The three-dimensional pick determination unit 8l is an end effector at a gripping position designated for the work by the work-side gripping point designating unit 8d1 for the search result of each work searched by the three-dimensional search unit 8k. It is a member for determining whether or not it can be gripped. The three-dimensional pick determination unit 8l includes an interference determination unit 8m and an angle determination unit 8o.

The interference determination unit 8m exists around the work with respect to the grip position designated for this work by the work side grip location designation unit 8d1 with respect to the search result of each work searched by the three-dimensional search unit 8k. It is a member for determining the presence or absence of interference with an object. Further, the three-dimensional pick determination unit 8l determines that the work that is determined to have interference by the interference determination unit 8m cannot be gripped. As a result, by showing the cause when the work cannot be gripped, the user can contribute to resetting the gripping position, for example, it becomes easier to consider what kind of gripping position should be added.

The angle determination unit 8o is a member for determining whether or not the posture of the search result of the work searched by the three-dimensional search unit 8k is within the inclination angle range set by the inclination angle setting unit 8n.

When the three-dimensional pick determination unit 8l determines by the angle determination unit 8o that the posture of the search result of the work searched by the three-dimensional search unit 8k is not within the inclination angle range set by the inclination angle setting unit 8n. It is configured to determine that the work cannot be gripped. As a result, when the posture of the work is too steep to grip, or when the accuracy of the three-dimensional measurement cannot be expected, by eliminating this, it is possible to avoid erroneous judgment of gripping position selection and judgment and improve reliability. Can be done.
(Gripping solution candidate list display function)

Next, a procedure for displaying the reason for the gripping solution NG will be described with reference to the flowchart of FIG. 71. The procedure at the time of setting and the procedure at the time of actual operation are basically the same as those of FIGS. 27, 28, 63 and the like described above, and detailed description thereof will be omitted as appropriate.

First, in step S7101, three-dimensional measurement is performed on the target work group. Here, for the work group stacked separately, the sensor unit is used to acquire actual measurement data having shape information, and this is used as an input image.

Next, in step S7102, a three-dimensional search is performed on the input image to detect the position and orientation of each work in the input image.

Then, in step S7103, one work to be verified is selected from the detected works.

Further, in step S7104, the position and posture of the end effector model when the work is to be gripped by the end effector are calculated from the detection position of the selected work and the gripping posture registered in advance for the work.

Next, in step S7105, it is determined whether or not the calculated tilt angle in the position and orientation of the end effector model is out of the set range. If it is determined that it is out of the set range, the process proceeds to step S7106, and it is determined that the grip is NG. At the same time, the factor of NG is set to "tilt angle", and then the process jumps to step S7111.

On the other hand, if it is determined in step S7105 that the tilt angle of the end effector model is within the set range, the process proceeds to step S7107, and interference determination between the end effector model and surrounding objects is performed at the calculated position. Here, the surrounding object is a storage container, other work, or the like existing around the end effector model. When the surrounding objects are modeled in advance with 3D CAD data or the like and the end effector model is moved to the position and orientation in which the work model is gripped, whether or not these and the end effector model interfere with each other is calculated by calculation. judge.

Then, in step S7108, when it is determined that the end effector model is interfering with a surrounding object as a result of the interference determination, the gripping NG determination is made and the NG factor is set as "point cloud interference" in step S7109. Above, jump to step S7111.

On the other hand, if it is determined in step S7108 that the end effector model does not interfere with surrounding objects as a result of the interference determination, the gripping OK determination is made in step S7101, and then the process proceeds to step S7111.

Then, in step S7111, it is determined whether or not another gripping posture is set for the selected work, and if there is another gripping posture, the process returns to step S7104 and the above process is repeated. On the other hand, when it is determined that there is no gripping posture, in step S7112, for all the gripping solution candidates, the determination result of gripping OK / gripping NG and the factor in the case of gripping NG are displayed in the work gripping impossible factor display area 3e. Display it.

In this way, it is possible to verify whether or not gripping is possible for a given work group, and to list the factors when gripping is not possible.
(Gripping simulation)

Next, the details of the gripping simulation for determining the specific gripping will be described. Here, an example of determining whether or not gripping is possible when bulk picking is performed using the work WK10 as shown in FIG. 72 will be described. For this work WK10, a basic direction image corresponding to a hexagonal view is generated by the basic direction image generation unit 8e', and each basic direction image is registered as a search model by the search model registration unit 8g. Examples of basic direction images are shown in FIGS. 73A to 73E. In these figures, FIG. 73A is a height image A of the work of FIG. 72 viewed from the positive direction side of the Z axis, FIG. 73B is a height image B of the work of FIG. 72 viewed from the negative direction side of the Z axis, and FIG. 73C. Is the height image C seen from the positive direction side of the X axis, FIG. 73D is the height image D seen from the negative direction side of the X axis, and FIG. 73E is the height seen from the positive direction side of the Y axis. Image E, FIG. 73F shows the height image F seen from the negative direction side of the Y axis.
(Work selection screen 210)

In order to determine whether or not the work can be gripped by the end effector, a three-dimensional search is performed on the input image obtained by capturing the group of works stacked separately, and the target work is selected in the state where the work is detected. (Step S7103 in FIG. 71). The work is selected, for example, from the work selection screen 210 shown in FIG. 74. The work selection screen 210 shown in this figure includes an image display field 141 and an operation field 142. In the image display field 141, an input image which is actual measurement data obtained by imaging a group of workpieces stacked separately is displayed. In addition, the search result is displayed as a point cloud on the input image at the position where the work is detected. The viewpoint can be changed by dragging the screen of the image display field 141.
(Label number, model number)

The operation field 142 includes a target work selection field 211 for selecting a target work, a detection search model display field 212 indicating a search model used for three-dimensional search of the target work, and a selected work. A "grasping confirmation" button 213 is provided for displaying a list of all gripping position candidates. In the example of FIG. 74, as a result of the three-dimensional search for the input image, 18 works exist, and the state in which the third work is selected among them is shown. In order to distinguish the search results detected here, identification information is set in the search results. Here, the label number of the serial number is set as the identification information. In the target work selection field 211 of FIG. 74, “3” is displayed as the label number of the selected target work. Further, in the image display field 141, the search result of the selected target work is displayed. Specifically, as the state of the search result, the feature points of the corresponding search model are displayed in a form of being superimposed on the point cloud of the input image. If the label number of the target work is changed in this state, the work selected in the image display field 141 is also changed accordingly. Further, in the detection search model display field 212, a search model E (height image of FIG. 73E) is displayed as a search model used for three-dimensionally searching the target work. In addition, individual identification information is also given to the search model, which is referred to as a model number here. In FIG. 74, “E” of the work model having the serial number label number 3 is displayed as the model number in the detection search model display field 212.
(Gripping solution candidate display screen 220)

Further, the gripping position set for each work, that is, the gripping solution candidates can be displayed in a list in the gripping solution candidate display area 3c. In the example of the work selection screen 210 of FIG. 74, when the “grasping confirmation” button 213 is pressed, the gripping solution candidate display screen 220 of FIG. 75 is displayed on the display unit. In the gripping solution candidate display screen 220, which is a form of the gripping solution candidate display area 3c, all the gripping positions set for the target work selected in the target work selection field 211 of FIG. 74 are gripping solutions as gripping solution candidates. It is listed in the candidate display field 221. In the gripping solution candidate display field 221, for each gripping solution, a work gripping position display field 223 indicating the label number of the gripping position, a work gripping possibility display field 224 indicating the gripping possibility determination result of the work, and a case where gripping is not possible A work gripping inability factor display column 225 is provided to indicate the reason. In the example of FIG. 75, for the target work of the third search result selected in the target work selection field 211, all five gripping candidates are listed, and together with the judgment results of gripping OK and NG, NG. The cause of becoming is displayed. Further, the work selected in the image display field 141 and the end effector model for gripping the work are displayed in a posture of gripping the gripping position corresponding to the gripping position of the work selected in the gripping solution candidate display field 221. To. When the selection of the gripping position is changed in the gripping solution candidate display field 221, the work in the image display field 141 is also changed accordingly, and the posture of the end effector model that grips this work is also updated.

In the example of FIG. 75, a state in which the work of FIG. 73E is gripped by the gripping position candidate of label number 3 is shown. Here, since it is the determination result of gripping OK, nothing is displayed in the work gripping impossible factor display column 225. On the other hand, when the gripping position candidate of label number 4 is selected in the work gripping impossible factor display field 225, the display in the image display field 141 is also changed as shown in FIG. 76. Since the gripping position candidate of the label number 4 is the determination result of gripping NG, the reason why it is determined that gripping is not possible is displayed in the work gripping impossible factor display column 225. Further, in the image display field 141, the end effector model corresponding to the end effector model is displayed, and it is possible to visually confirm what kind of state the gripping posture is actually in. Here, "point cloud interference" is displayed, and it can be confirmed that the end effector is interfering with the flat surface of the storage container. As a result, the user can consider adding a gripping position in a position or posture that does not interfere with the work in such a posture.

Similarly, when the gripping position candidate of label number 2 is selected in the work gripping impossible factor display field 225, the gripping solution candidate display screen 220 shown in FIG. 77 is displayed, and the gripping position candidate of label number 2 is the determination result of gripping NG. As a reason, the "tilt angle" and the work gripping impossible factor display column 225 are displayed. Further, in the image display field 141, the end effector model corresponding to this is displayed, and the end effector is in a state of trying to grip at a steep inclination angle, and it can be seen that the end effector is excluded from the interference determination target in the first place.

Further, the display mode of the end effector displayed in the image display field 141 may be changed according to the determination result. In the example of FIG. 75, the end effector model is displayed in white when the grip is OK, and in the example of FIGS. 76 and 77, the end effector model is displayed in red when the grip is NG. As a result, the user can easily visually distinguish the gripping determination result. Further, when the gripping position is added, the posture of the end effector model may be adjusted to update the display in the image display field 141 so that the display mode is changed when the state does not interfere. .. With such a configuration, the user can change the posture of the end effector model from red to white, for example, and has an advantage that the posture adjustment work can be easily understood.

By providing the gripability determination verification function in this way, when it is determined that the workpiece to be gripped cannot be gripped, how to set the grip, for example, adding a new gripping position or existing gripping It will be a guide when considering work such as changing the position setting. For example, in the examples shown in FIGS. 78 and 79, although there are two gripping position candidates for the work of the search model C (FIG. 73C), they cannot be gripped due to point cloud interference. There is. In each case, as shown by the broken line circle in the image display field 141, it can be seen that the cause is that the tip of the claw of the end effector model interferes with another work. From this, it can be seen that if a new gripping position is added so as to grip the portion opposite to the current gripping position, an OK gripping solution can be obtained. Therefore, if the gripping position specifying unit 8d is set so as to add a new gripping position (grasping posture C-002) to the opposite side of the current gripping position, a determination result of gripping OK can be obtained as shown in FIG. Will be. In this way, when the user performs the gripping position and posture adjustment work, an environment is provided in which it is easy to consider countermeasures, and a robot system that is easy to set is realized.

It is also possible to change the threshold value of the tilt angle of the end effector by setting by the user. For example, when a box with a deep bottom is used as the storage container for the work, if the inclination of the end effector becomes large, the arm part of the robot other than the end effector tends to collide with the wall of the storage container, so the angle range is set narrow. To do. On the contrary, when a box with a shallow bottom is used as the storage container, it is unlikely that the arm portion of the robot collides with the box unless the end effector interferes, so that the angle range can be set wide. By adjusting the setting of the angle range in this way, it is possible to flexibly adjust the gripping possibility determination according to the actual situation.

As described above, all the gripping solutions for the selected work are displayed in a list, and for those that were NG, the NG factors are also displayed. If there is, it becomes easy to identify the cause of why it is not grasped. In addition, since the cause can be identified, it becomes easy to understand what kind of gripping posture should be newly added.
(Embodiment 8)

In the above example, an example of acquiring search results individually for each search model in a three-dimensional search has been described. That is, a three-dimensional search is performed with a plurality of basic direction images showing different surfaces of the same work, and the obtained results are also recognized as separate works. In other words, different faces of the same work may be detected as different works as a result of being individually searched. On the other hand, in the conventional three-dimensional search, since the search is performed with a three-dimensional shape indicating one work model, such a situation does not occur, and different surfaces of the same work are detected as one work. However, in a work having many aspects, the search becomes complicated and the possibility of false detection increases. On the other hand, in the method according to the above-described embodiment, since the search is performed in a simple aspect, the search process can be simplified, which is advantageous in terms of low load, high speed, and the like. On the other hand, as a result of searching for each surface, the search results obtained as described above are regarded as separate works, and there is a problem that different surfaces are individually detected even for the same work.

Therefore, by integrating the obtained search results and summarizing them for each work, or by estimating the search results of another surface from the search results of one surface, the surfaces that could not be detected by the three-dimensional search and the detection accuracy are low. The surface can also be used as a candidate for a gripping solution. Such an example is shown in the block diagram of FIG. 81 as a robot system according to the eighth embodiment. The robot system 8000 shown in this figure includes an image processing device 800, a display unit 3B, an operation unit 4, a sensor unit 2, and a robot RBT. The same members as those in FIGS. 6 and 70 are assigned the same numbers and detailed description thereof will be omitted as appropriate.
(Image processing device 800)

The image processing device 800 includes an input image acquisition unit 2c, a calculation unit 10, a storage unit 9, an input / output interface 4b, a display interface 3f, and a robot interface 6b. The calculation unit 10 includes a basic direction image generation unit 8e', a grip position identification unit 8d, a search model registration unit 8g, a three-dimensional search unit 8k, an image estimation unit 8z, a search result integration unit 8p, and three dimensions. A pick determination unit 8l and an inclination angle setting unit 8n are provided.

The basic direction image generation unit 8e'generates, as a basic direction image, a plurality of height images viewed from any of the three axial directions orthogonal to each other in the virtual three-dimensional space with respect to the work model. It is a member of.

The gripping position specifying unit 8d is for specifying a plurality of gripping positions for gripping the work model shown by the basic direction image by the end effector with respect to one of the basic direction images generated by the basic direction image generation unit 8e'. It is a member. The gripping position specifying portion 8d includes a work-side gripping location designating portion 8d1 and an end effector-side gripping setting portion 8d2.

The work-side gripping location designation unit 8d1 creates a work model composed of three-dimensional CAD data that virtually represents the three-dimensional shape of the work in a state where the end effector model is displayed in the image display area 3b. It is a member for designating a gripping position when gripping with.

The end effector side grip setting unit 8d2 is a member for designating a grip position for gripping the work with respect to the end effector model displayed in the image display area 3b.

The search model registration unit 8g applies a plurality of basic direction images generated by the basic direction image generation unit 8e'to a plurality of work groups included in the input image acquired by the input image acquisition unit 2c. It is a member for registering each as a search model used when performing a three-dimensional search for specifying a posture and a position. Further, the search model registration unit 8g can also register the relative positions of a plurality of registered search models as relationship information (details will be described later).

The image estimation unit 8z is searched by the three-dimensional search unit 8k, and for each search result extracted from the input image in search model units, each search model used for this search uses the original work model. It is a basic direction image viewed from which direction, and based on the relative positional relationship with the basic direction image of other search models registered for this work model, with respect to the work model indicated by this search result. , A member for estimating the position and orientation of an unsearched basic direction image not included in the search result of the three-dimensional search unit as an estimated image. In the image estimation unit 8z, for example, when the posture of the search result of the work searched by the three-dimensional search unit 8k by the angle determination unit 8o described later is within the inclination angle range set by the inclination angle setting unit 8n. , Can be configured to estimate an estimated image that has a relative positional relationship with this search result.

The search result integration unit 8p is searched by the three-dimensional search unit 8k, and for each search result extracted in search model units from the input image, each search model used for the search uses the original work model. It is a member for integrating adjacent ones among a plurality of search results as an integration result for a common work based on the relative positional relationship of the basic direction image viewed from which direction.

The three-dimensional pick determination unit 8l sets the grip position specifying unit 8d for the work model for the integrated result integrated by the search result integration unit 8p and each unintegrated search result searched by the three-dimensional search unit 8k. It is a member for determining whether or not it can be gripped by the end effector at the gripping position designated by the above. The three-dimensional pick determination unit 8l includes an interference determination unit 8m and an angle determination unit 8o.

As a result, the position and orientation of the work can be estimated more accurately by using the relationship information between the surfaces, instead of individually determining the holding of the search results that have been searched three-dimensionally. Or, even if the accuracy of the search result is low, it will be possible to detect it, and it will be possible to consider it as a holding position candidate including the side that is normally difficult to search, increasing the possibility of obtaining a holding solution. It becomes possible.
(Procedure for registering a search model including relationship information between faces)

For the procedure for setting the work model, the end effector model, the gripping position, and the like for the above image processing apparatus, for example, the procedure shown in the flowchart of FIG. 27 can be used. Here, in step S2701 of FIG. 27, a procedure of using the three-dimensional CAD data as the search model and registering the search model including the relationship information between the surfaces of the work will be described based on the flowchart of FIG. 82.

First, in step S8201, the three-dimensional CAD data model of the work is read.

Next, in step S8202, the center of the circumscribing rectangular parallelepiped of the 3D CAD data model is corrected to the origin of the 3D CAD data.

Further, in step S8203, a height image viewed from each of the "upper", "lower", "left", "right", "front", and "rear" directions is generated as a basic direction image. The basic direction image is performed by the basic direction image generation unit 8e'in FIG. 81. Here, when the height image is generated from the three-dimensional CAD data, the height image is generated so that the origin of the CAD is the center of the height image.

Next, in step S8204, among the generated height images, the height images having the same appearance are deleted.

Further, in step S8205, the relative positions of the plurality of registered search models are registered as the relationship information. Here, the search model registration unit 8g stores the remaining height image and the relationship information of each surface of the top, bottom, left, right, front, and back.

Finally, in step S8205, the search model is registered using the generated height image. In this way, the search model including the relationship information between the work surfaces is registered.
(Surface relationship information)

Here, the relationship information indicating the relative positional relationship between the surfaces when the work is viewed from a specific direction will be described. For example, in the case of the work shown in FIG. 7, the surfaces having different appearances are the four surfaces shown in FIGS. 9A to 9D. In these figures, the model A shown in FIG. 9A is a height image seen from the positive direction of the X-axis. However, it also matches the image seen from the negative direction of the X-axis. Further, the model B shown in FIG. 9B is a height image viewed from the positive direction of the Y axis, and coincides with the image viewed from the negative direction of the Y axis. On the other hand, the model C shown in FIG. 9C matches only the image viewed from the positive direction of the Z axis. Further, the model D shown in FIG. 9D matches only the image viewed from the negative direction of the Z axis.

Here, which models A, B, C, and D, which are height images generated in advance, match the three-dimensional CAD data of the work shown in FIG. 7 with the image viewed from which coordinate axis direction and in which rotation state. Is the information that represents the relationship information of the surface. With such surface relationship information, the posture of the original 3D CAD model can be obtained from the search results of each model. For example, the model B of FIG. 9B exists adjacent to each other on the left and right surfaces of the model A of FIG. 9A. Further, a model C of FIG. 9C is present on the upper surface of the model A, and a model D of FIG. 9D is adjacent to each other on the lower surface. As a result, it can be seen that in the searched image obtained as a result of the three-dimensional search, when the detection results of the model A and the model B are adjacent to each other, the search for the same work is performed. Therefore, these search results can be integrated as one search result. Here, the set of integrated search results is called an integrated result.

In addition, the evaluation index can be updated by integrating the search results. That is, even if the search result is not highly accurate, if a highly accurate search result is obtained in the integrated result, the evaluation index of the integrated result is increased to obtain a correct gripping solution when selecting the gripping position. Can be prioritized.
(Unsearched basic direction image)

Further, even if there is a surface that is not searched, such as when the three-dimensional search fails or the search itself is not performed, it is possible to estimate using the integrated result. That is, when a plurality of search results are integrated to obtain an integrated work result, even if there is a surface that is not detected by the three-dimensional search among the surfaces constituting this work, the other surface is present. If the search result is obtained in, the posture and position of the work itself can be calculated. As a result, when a work model in which the gripping position is registered for the work in advance or a search model for three-dimensional search is registered, information on the surface that has not been searched can also be estimated from the posture of the work. Therefore, the surface obtained by estimation (referred to as the unsearched basic direction image), which was not obtained as a result of such a search, was not actually searched by using it as a candidate for the gripping position. The gripping position of the surface can also be examined as a candidate for the gripping solution, and there is an advantage that an appropriate gripping solution can be easily obtained. For example, when the model B shown in FIG. 9B is obtained as a result of the three-dimensional search, it can be estimated that the surface of the model A shown in FIG. 9A exists next to the model B. As a result, even if model A is not actually detected as a search result, the surface of model A can be estimated from the search result of model B, so that the surface of model A is registered in the surface of model A as a candidate for a gripping solution. It becomes possible to use the gripping posture, and as a result, the number of candidates for gripping solutions increases and picking becomes easier.
(Calculation of gripping solution using surface relationship information)

Next, the effect of calculating the gripping solution using the surface relationship information will be described using the work WK10 shown in FIG. 72. From the three-dimensional CAD data of the work WK10, the basic direction images shown in FIGS. 73A to 73F are generated by the basic direction image generation unit 8e', and these are registered as the search model by the search model registration unit 8g. In these figures, FIG. 73A is a height image seen from the positive direction of the Z axis, and this is referred to as model A. Similarly, FIG. 73B is a height image viewed from the negative direction of the Z axis, and this is referred to as model B. Further, FIG. 73C is a height image viewed from the positive direction of the X-axis, and is referred to as model C. Further, FIG. 73D is a height image viewed from the negative direction of the X axis and is model D, FIG. 73E is a height image viewed from the positive direction of the Y axis and is model E, and FIG. 73F is a height image viewed from the negative direction of the Y axis. It is a height image seen, and the model F is shown respectively.

Here, for each of these basic direction images, as shown in FIGS. 83A to 83F, one gripping posture for gripping the work model WM10 by the end effector model EM10 is registered in the gripping position specifying unit 8d. It shall be. Here, FIG. 83A shows a state in which the gripping posture A-000 is registered with respect to the basic direction image of FIG. 73A. Further, FIG. 83B shows a state in which the gripping posture B-000 is registered with respect to the basic direction image of FIG. 73B. Further, FIG. 83C shows a state in which the gripping posture C-000 is registered with respect to the basic direction image of FIG. 73C, and FIG. 83D shows a state in which the gripping posture D-000 is registered with respect to the basic direction image of FIG. 73D. FIG. 83E shows a state in which the gripping posture E-000 is registered with respect to the basic direction image of FIG. 73E, and FIG. 83F shows a holding posture F-000 registered with respect to the basic direction image of FIG. 73F. The state of doing is shown respectively. In this case, the calculation of the gripping solution using the surface relationship information will be described with reference to FIGS. 84, 85, and 86.

The work selection screen 210 of FIG. 84 shows a state in which 12 works are detected as a result of performing a three-dimensional search on the input image obtained by capturing the group of works stacked separately. An input image is displayed in the image display field 141. Further, in the operation field 142, a target work selection field 211 for selecting a target work, a detection search model display field 212, and a "grasping confirmation" button for displaying a list of gripping position candidates in the selected work are displayed. 213 is provided. In the example of FIG. 84, the second work among the 12 works searched three-dimensionally is selected in the target work selection field 211, and the second target work selected in the image display field 141. Search results are displayed. Specifically, as the state of the search result, the feature points of the corresponding search model are displayed in a form of being superimposed on the point cloud of the input image. Further, in the detection search model display column 212, search models C, F, and A are displayed as search models for detecting the second target work. In this state, three types of search models C, F, and A (FIGS. 73C, 73F, and 73A) are integrated.

Here, assuming that the relationship information of the surface of the work is not used, the model F and the model A are in a state where only a part of the side surface is visible on the input image as shown in the image display field 141, so that the search is performed. As a result, it seems that it was difficult to detect. Even if it can be detected, the evaluation index is low and the result is considered to be low priority. On the other hand, by using the relationship information, as shown in FIG. 84, not only the search model C but also the search model F and the search model A can be detected or estimated and used as gripping position candidates. Become.

When the "grasping confirmation" button 213 is pressed on the work selection screen 210 of FIG. 84, the gripping solution candidate display screen 220 of FIG. 85 is displayed. On the gripping solution candidate display screen 220, the gripping positions set for the second target work selected in the target work selection field 211 of FIG. 84 are listed in the gripping solution candidate display field 221 as gripping solution candidates. In the gripping solution candidate display field 221, whether or not the work can be gripped with respect to the gripping positions (FIGS. 83C, 83F, 83A) set for the search models C, F, and A displayed in the detection search model display column 212, respectively. The determination result is displayed in the work gripping possibility display column 224, and the reason why the work cannot be gripped is displayed in the work gripping impossible factor display column 225, respectively. First, regarding the search model C, it is shown that the determination result is gripping NG, and the reason is the interference of the point cloud data. Further, in the image display area, the end effector model EM10 that tries to grip the work in the posture corresponding to the gripping position C-000 is displayed in red, indicating that the work cannot be gripped due to interference. Here, assuming that the surface relationship information is not used, only the search model C is searched three-dimensionally, and as a result, the candidate for the gripping solution is only C-000 of this search model, and as a result, the gripping solution can be obtained. It will not be.

On the other hand, by using the relationship information, as shown in FIGS. 85 and 86, the search models F and A are also targeted in addition to the search model C, and F-000 and A-000 are added as gripping position candidates. You can see that. Then, among the added gripping position candidates, the gripping position candidate F-000 of the search model F has a determination result of gripping NG, but the gripping position candidate A-000 of the search model A has a gripping OK. The determination result is obtained, the end effector is displayed in white as shown in FIG. 86, and the gripping solution is safely obtained without interfering with the point cloud data. As a result, even if a gripping solution cannot be obtained without using the relationship information, it is possible to obtain a gripping solution by using the relationship information. That is, without changing the 3D search settings, for example, adjusting the lighting or camera arrangement, changing the search algorithm, etc., and keeping the same search conditions, in other words, without changing the accuracy of the 3D search. , It is possible to increase the possibility of obtaining a gripping solution. In this way, by using the surface relationship information, the number of candidates for gripping solutions can be increased, and a result that is easily picked can be obtained.
(Procedure 3 during actual operation)

Here, with the search model registered in the procedure of FIG. 82, a procedure for performing a three-dimensional search and grasping availability determination in actual operation will be described with reference to the flowchart of FIG. 87.

First, in step S8701, three-dimensional measurement is started for the loosely stacked workpiece. Here, the work group stacked separately by the sensor unit 2 shown in FIG. 81 is imaged to perform three-dimensional measurement, and the three-dimensional shape having height information is acquired by the input image acquisition unit 2c as an input image.

Next, in step S8702, a three-dimensional search is performed on the three-dimensional shape of the obtained work group using a work model, and the position and orientation of each work are detected. The three-dimensional search is performed by the three-dimensional search unit 8k shown in FIG.
(Generation of integration result)

Further, in step S8703, the search result of the three-dimensional search and the result indicating the same work are integrated from the relationship information of each surface. Here, the search result integration unit 8p associates the search results obtained by imaging the same work with each other by using the relationship information registered in the search model used for the search. That is, among a plurality of search results, adjacent ones are searched based on the relative positional relationship of the basic direction image viewed from which direction the common work model from which the search result is searched is searched. The result integration unit 8p integrates. The integration results obtained in this way are commonly treated as indicating the same work.
(Generation of unsearched basic direction image)

Further, in step S8704, the undetected surface is estimated from the three-dimensional search result and the relationship information of each surface. Here, based on the posture and position of the work indicated by the integration result, the position and posture of the unsearched basic direction image that is not included in the result of the three-dimensional search for this work is estimated. Then, the gripping position set by the gripping position specifying unit 8d with respect to the unsearched basic direction image is set as the target of the gripping possibility determination. As a result, even the surface of the work that was not originally searched as a result of the three-dimensional search can be subject to the gripping possibility judgment by estimating from the posture and position of the work indicated by the integrated search result. The possibility of obtaining a solution can be increased.

Next, in step S8705, for one of the detected works, the position and posture in which the end effector should be arranged are calculated based on the position of this work and the gripping posture of the work registered at the time of setting.

Next, in step S8706, an interference determination is performed using the end effector model to determine whether or not the end effector interferes with surrounding objects at the calculated position.

Then, in step S8707, it is determined whether or not the end effector interferes, and if it does not interfere, the process ends with a gripping solution for this work.

On the other hand, if it is determined that the work is interfering, the process proceeds to step S8708, and it is determined whether or not there is another gripping position registered for this work. If another gripping position is registered, the process returns to step S8705 and the above process is repeated for this gripping position.

On the other hand, if another gripping position is not registered, the process proceeds to step S8709 to determine whether or not there is another detected work. If there is another work, the process returns to step S8705, the work is replaced, and the above process is repeated. If there is no other work, the process ends with no gripping solution.

In this way, the presence or absence of a gripping solution capable of gripping the work is determined using the integrated result and the unsearched basic direction image. Then, when a gripping solution is obtained, the robot controller 6 is controlled so that the work is picked at the gripping position determined by the end effector.

In the present specification, the image is not limited to strictly continuous data, but is used in the sense of including a set of discrete data such as a set of point cloud data.
(Embodiment 9)
(Integration of evaluation indicators)

In addition to the above procedure, it is also possible to integrate the evaluation index that defines the priority with the integration result. Such an example is shown in FIG. 88 as a robot system 9000 according to the ninth embodiment. The image processing device 900 shown in this figure includes an evaluation index calculation unit 8q and a gripping priority determination unit 8r. The same members as those shown in FIG. 81 and the like are designated by the same reference numerals, and detailed description thereof will be omitted.

The evaluation index calculation unit 8q is a member for calculating an evaluation index for each grip position candidate, using the grip position determined to be grippable by the three-dimensional pick determination unit 8l as a grip position candidate.

Further, the gripping priority order determination unit 8r is a member for determining the priority order for gripping the work by the end effector based on the evaluation index calculated by the evaluation index calculation unit 8q.

In this image processing apparatus, the evaluation index calculation unit 8q calculates the evaluation index for each search result. As the evaluation index, as described above, for example, the score ratio of the feature points included in the search result and corresponding with an error of a certain distance or less can be used.

Here, the search result integration unit 8p assigns the highest evaluation index among the evaluation indexes calculated by the evaluation index calculation unit 8q to each search result constituting the integration result to the integration result. Generally, in the evaluation index of 3D search, the search conditions and the posture of the work (for example, when the inclination angle of the work is large and sufficient reflected light cannot be obtained, or conversely, the reflected light is too strong due to the glossy work), etc. Depends on. Therefore, the evaluation index of the search result may be calculated low, and the search result may be excluded from the target of the three-dimensional search because it has a low priority or the reflected light cannot be obtained in the first place. These will be inferior as candidates for gripping position or will not be listed as candidates, but in such a work, simply because they are not arranged in an appropriate posture for 3D search, they can be gripped. It is also possible that they are arranged in a suitable posture. Even in such a case, in the past, it was excluded at the stage of 3D search or had a low priority, so it was unlikely to be selected as a grasping solution and could not be fully utilized. .. On the other hand, according to the present embodiment, even if the result of the three-dimensional search is a search result having a low evaluation index, if a high evaluation value is obtained in other adjacent search results, this is the case. By using, it becomes possible to use it as a candidate for a gripping position with a high priority. As a result, even a candidate for a gripping position that has not been utilized in the past can be used, and the possibility of obtaining a gripping solution can be increased. In other words, it is possible to grip an appropriate workpiece regardless of the accuracy of the three-dimensional search.

In the above method, the entire integrated result is evaluated with the highest evaluation index among the search results constituting the integrated result, or the evaluation index of each search result is changed from the original evaluation value to a higher evaluation value. The mode of rewriting has been described. This is based on the reliability that the three-dimensional search result is highly reliable for the search result obtained with a high evaluation index. However, the present invention is not limited to this aspect, and other aspects, for example, the evaluation indexes of each search result constituting the integrated result may be averaged and treated as the evaluation index of the integrated result. In either case, the search result, which was originally a low evaluation index, is evaluated with a higher evaluation index than the original evaluation index so that it can be a candidate for the gripping position, etc., using the search results of other high evaluation indexes. It is the purpose of this embodiment.
(GUI of image processing program)

Here, an example of the GUI of the image processing program will be described with reference to FIGS. 89 to 149. FIG. 89 shows a function selection screen 340 for selecting various functions, and available functions are listed in a button shape in the function list display field 341. When the "3D search" button 342 is pressed from this state, the three-dimensional search function is executed.
(3D search screen)

It is necessary to make necessary settings when executing the 3D search function. Specific examples include registration of a search model, registration of a floor surface on which a work is placed and a storage container, and setting of conditions for a three-dimensional search. In addition to allowing the user to arbitrarily set all the items, these settings can also be set for each item while sequentially indicating the necessary procedures to the user. The guidance function of such a setting will be described based on the three-dimensional search screens of FIGS. 90 to 101.

In these three-dimensional search screens, a flow display unit 351 for displaying the flow of the procedure to be set is provided in the upper part of the image display field 141 at the lower left of the screen. In the flow display unit 351, the outline of each procedure is shown by text, a pattern, or the like, and the procedure currently being set is highlighted. In the example of FIG. 90, the "model registration" icon 352 is displayed in a bright color, and the other "floor / box setting" icon 353 and the "search setting" icon 354 are displayed in a dark color, which are currently set for the user. It is possible to announce the target who is doing. Further, in the operation field 142 provided on the right side of the three-dimensional search screen, the explanation of the item currently being set is displayed in text or the like, and the setting to be made to the user can be explained in characters. The explanation is not limited to text information, but may be appropriately combined with symbols and moving images as needed. In addition, a setting field for parameters to be set is also provided.
(Registration of search model based on actual measurement data)

The three-dimensional search screens of FIGS. 90 to 101 show an example in which the actual measurement data obtained by imaging the actual work piece by the sensor unit is registered as a search model. This registration procedure can be performed by the procedure described in the flowchart of FIG. 27. In the three-dimensional search screens of FIGS. 90 to 101, the “model registration” icon 352 is highlighted as the first procedure on the flow display unit 351 as described above.

FIG. 90 shows a search model registration method selection screen 350 for selecting a method for registering as a search model. The operation field 142 of the search model registration method selection screen 350 is provided with a model registration method selection field 355 for selecting a method for registering a model. In the model registration method selection field 355, as a model registration method, it is possible to select whether to image and register the actual work or to register from the CAD data with a radio button. In the example of the search model registration method selection screen 350 of FIG. 90, it is selected to image and register the actual work, and accordingly, the operation column 142 describes the procedure for imaging and registering the actual work. Is displayed with letters and patterns. In this example, "Model the 3D information of the work surface seen from the camera. Register as many faces as you want to search. With this tool, the registered surface model is matched for the work that takes various postures. By letting them recognize the position and posture of the work, "the user is shown the work to be done with text information and a pattern. This allows the user to easily understand the work to be done on this screen. When the registration of the actual work is selected on the search model registration method selection screen 350 of FIG. 90 and the "OK" button 356 is pressed, the imaging of the work is started.
(Actual measurement data imaging unit)

The procedure for actually registering the actual work is as shown in FIGS. 91 to 93. In this example, the model registration work is divided into three screens and explained to the user in sequence. First, on the actual measurement data acquisition screen 360 of FIG. 91, the work is imaged as the first operation of model registration. In this example, in the operation field 142, as the work to be performed by the user, "1. Place the work on a flat surface so that the surface of the work to be registered can be seen by the camera, and press the" Measurement execution "button at the lower right. 2. Place the model area so that it surrounds the work on the left camera screen. Is explained. Further, a detailed explanation about the optimum work arrangement can be displayed on a separate screen by pressing the "?" Button 364. As a result, it is possible to avoid a situation in which a large amount of information is displayed on the screen, making the screen difficult to see or confusing the user. In this way, the user arranges the work at a position where the sensor unit can take an image, and positions the work while checking the state of the work in the image display field 141. Then, when the "measurement execution" button 361 is pressed, the imaged work is displayed as shown in FIG. 91. Here, as a three-dimensional image, it is displayed as an image in which the color is changed according to the height of the obtained work. The height image is not limited to the color display, and the height may be expressed by the brightness value of the pixel. In the example of FIG. 91, six workpieces having the same shape are arranged side by side and arranged in different postures. In this state, the user sets the model area 362 so as to surround the image of the work to be registered as the search model. When the setting of the model area 362 is completed, the "Next" button 363 is pressed. As a result, the screen switches to the search model exclusion area setting screen 370 shown in FIG. 92.
(Background removal setting section)

On the search model exclusion area setting screen 370 of FIG. 92, as the second operation of model registration, an exclusion area to be excluded when registering as a search model is set. As an example of the search model exclusion area, there is a function of removing a background such as a floor on which a work is placed when registering an actual work. On the search model exclusion area setting screen 370 of FIG. 92, as the second operation of model registration, it is explained in the operation field 142 that "1. Set the background removal so that only the work surface to be registered remains." To. It is also possible to press the "?" Button 371 to display a detailed explanation of the background removal setting method on a separate screen. Further, a background removal setting field 372 is provided in the middle of the operation field 142, and the height to be removed as the background is numerically specified. In this example, 1.5 mm is specified, and within the range specified as the model area 362, the range from 0 to 1.5 mm in height is excluded from the registration target of the search model. The model area 362 displayed in the image display field 141 is updated according to the setting in the background removal setting field 372, and the area excluded from the registration target of the search model is displayed in black. While checking this situation, the user can determine whether or not the exclusion area setting is appropriate, and can reset the setting as necessary.
(Mask area setting unit)

Further, it is possible to specify as a mask area a range to be excluded from the registration target of the search model not only in the height direction but also in the plane direction. Under the background removal setting field 372 of the search model exclusion area setting screen 370 of FIG. 92, it is explained as "2. If something other than the work surface to be registered remains, place a mask area." .. Similarly, the "?" Button 373 can be pressed to display a detailed explanation of the mask area setting method on a separate screen. When the "mask area" button 374 is pressed, the mask area can be set as an area to be excluded from the registration target of the search model within the range designated as the model area 362. The mask area can be set by a specified figure such as a rectangle or a circle, as well as an automatic detection of a free curve or a boundary portion. Further, as described above, the model area 362 displayed in the image display field 141 is updated according to the setting of the mask area.

In the example of FIG. 92, the mask area is set after the background removal setting, but the order may be changed. In this way, when the setting of the area to be excluded from the registration of the search model is completed, the "Next" button 375 is pressed. As a result, the screen switches to the rotational symmetry setting screen 380 shown in FIG. 93.
(Rotational symmetry setting unit)

On the rotational symmetry setting screen 380 of FIG. 93, rotational symmetry is set as the third operation of model registration. In this example, the operation column 142 explains, "1. If the workpiece surface to be registered has rotational symmetry, specify the corresponding one." Similarly, the "?" Button 381 can be pressed to display a detailed explanation of the rotational symmetry setting method on a separate screen. A rotational symmetry setting column 382 is provided in the middle of the operation column 142, and a candidate for rotational symmetry can be selected from options such as circular symmetry, N-fold symmetry, and none. Further, when N-fold symmetry is selected, a numerical input box for the number of symmetry is displayed, and the number of rotational symmetry according to the shape of the work surface is set. For example, when the work surface is rectangular and the appearance of the work surface rotated 180 degrees with respect to the axis viewed directly above is the same, the rotational symmetry number of 2 is an appropriate value. Alternatively, when the work surface is on a square and the appearances of the work surface rotated every 90 degrees with respect to the axis viewed directly above are all the same, the rotational symmetry number of 4 is an appropriate value. Alternatively, when the work surface has a hexagonal shape and the appearance of the work surface rotated every 30 degrees with respect to the axis directly above or viewed is the same, the rotational symmetry number of 6 is an appropriate value. On the other hand, circular symmetry is set when the surface of the work looks circular when the spherical or columnar work is viewed from directly above, and when the rotation looks the same for any rotation angle. Is recommended. When rotational symmetry is set, even if there is a work in the posture specified as rotational symmetry when registering the search model, it will be recognized as the same work. For example, in the case of a double-symmetrical work in which the number of rotational symmetry is set to 2, the posture rotated by 180 degrees and the posture before rotation are recognized as the same work.

When the setting of the rotational symmetry is completed in this way, the "register" button 383 is pressed. As a result, the search model is registered according to the set conditions. Of the three operations required for model registration, the second and third may be interchanged.
(Multiple registration of search models)

When the search model is registered, the search model registration screen 390 shown in FIG. 94 is displayed. The operation field 142 of the search model registration screen 390 is provided with a model list display field 391 for displaying a list of registered search models. Here, the search model of model number 0 is displayed together with the reduced image as the registered search model. It is also displayed that the rotational symmetry setting is "None" for the search model.

To register additional search models, press the "Add" button 392. As a result, the search model can be registered in the same procedure as described above. As a method of registering the search model, it is possible to select whether to image and register the actual work or to register from the three-dimensional CAD data in the model registration method designation field 393. Here, "Register with actual work" is selected in the model registration method specification field 393 for additional registration.

FIG. 95 shows a search model registration screen 410 to which the search model is added in this way. On the search model registration screen 410 shown in this figure, the work arranged at the lower left of the six works arranged in the image display field 141 is additionally registered as a new search model. The additionally registered search model is added as the model number 1 to the model list display field 391.
(Display of simple 3D search results)

Further, on the search model registration screens of FIGS. 94 and 95, the result of a simple three-dimensional search using the registered search model can be displayed. For example, in the example of FIG. 94, as a result of performing a three-dimensional search on the six works displayed in the image display field 141 using the search model of model number 0, the upper left work used for registration is searched. There is. A point cloud matching the search model is displayed on the searched work. In this example, among the surface shapes of the search model, the planar point cloud is displayed in white, and the outline point cloud is displayed in purple. Furthermore, the score calculated as an evaluation index of the simple three-dimensional search result can be displayed numerically. In the example of FIG. 94, 96.4 is displayed as the score value. The score does not reach the highest value (99.9 in this example) for the original work registered as a search model, depending on the imaging state at the time of model registration and the input image at the time of executing the three-dimensional search. This is because the score has dropped slightly. From the result of the score value, it can be considered that the state is in agreement with the state at the time of model registration by about 96% or more.
(Search area setting section)

When the search model is registered in this way, the search area is then set as the setting of the three-dimensional search function. Specifically, when the "Done" button 411 at the lower right is pressed after completing the registration of the necessary search model on the search model registration screen 410 of FIG. 95, the process proceeds to the search area setting screen 420 of FIG. 96. On the search area setting screen 420, a search area for performing a three-dimensional search is specified. On the flow display section 351 at the top of the screen, the highlight display has moved from the "model registration" icon 352 to the "floor / box setting" icon 353, and after model registration has been completed, the floor surface and storage container on which the work is placed are now placed. The user is informed that it is in the stage of registering.

A search area setting field 421 is provided in the operation field 142 of the search area setting screen 420. The search area setting field 421 is provided with a "designation method selection navigation" button 422 for explaining to the user how to specify a search area for performing a three-dimensional search, and a designation method selection field 423 for selecting a search area designation method. Be done. When the "designation method selection navigation" button 422 is pressed, a navigation screen is displayed that guides the user to the method of designating the search area for performing the three-dimensional search. On the other hand, in the designation method selection field 423, one of floor designation, box designation, box search, and no designation can be selected from the drop box. When floor designation is selected, the search area setting screen 430 of FIG. 97 is displayed, the floor surface designation dialog 431 is displayed, and the explanation "Please click three points for calculating the floor surface." Is shown to the user. According to this, when the user specifies an arbitrary point indicating the floor surface from the image display field 141 with a pointing device such as a mouse, the spatial coordinates (X, Y, Z) of the specified point are displayed. In addition, a gauge is displayed as an extraction area. This extraction area represents to what extent the clicked position is referred to when calculating the Z coordinate value at the clicked point. Since the data of each point measured three-dimensionally includes an error, it is recommended to use the average value of the data within a certain range or the value obtained by the median in order to suppress the influence of the error. preferable. In this case, the Z coordinate value is obtained from the average value of the range specified by the extraction area. When the floor surface is designated in this way, the floor surface information is calculated, and the floor surface information calculated in the search area setting field 421 is displayed as the floor surface information as shown in the search area setting screen 440 of FIG. 98. It is displayed in column 441. Here, in the floor surface information display column 421, the X inclination, the Y inclination, and the Z section of the floor surface are numerically displayed as floor surface information. The offset amount can also be set as needed. For example, in consideration of fluctuations in the thickness of the floor surface of the storage container, a predetermined height from the floor surface is removed from the search area. In the example of FIG. 98, a noise removal column 442 is provided to remove noise from the floor surface. Here, the range from the specified floor height to the height of 1.5 mm specified in the noise removal column 442 is excluded from the target area of the three-dimensional search. When the floor surface setting is completed as described above, the "Done" button 443 is pressed. As a result, the screen switches to the search parameter setting screen 450 of FIG. 99.
(Search parameter setting section)

On the search parameter setting screen 450 of FIG. 99, the search parameter is set as a condition for executing the three-dimensional search. On the flow display section 351 at the top of the screen, the highlight display moves from the "floor / box setting" icon 353 to the "search setting" icon 354, and the search area setting is completed and the search condition is currently set. Is shown to the user. The operation field 142 of the search parameter setting screen 450 is provided with a field for setting each search parameter required for executing the three-dimensional search. Specifically, a detection condition setting field 451, a feature extraction condition setting field 452, and a determination condition setting field 453 are provided. Among these, the feature extraction condition setting field 452 sets an edge extraction threshold value, which is a threshold value for extracting edges when detecting a work from the input image. Further, in the judgment condition setting field 453, an upper limit or a lower limit is set for the measured value of the score, or an upper limit or a lower limit is set for the measured value of the X coordinate of the position, and a judgment condition for whether or not to select as a search result is set. To do.

In the detection condition setting field 451, a condition for detecting a work is set. In this example, the detection number specification column 454 that specifies the upper limit of the number of workpieces to be detected, the inclination angle upper limit specification column 455 that specifies the upper limit of the inclination angle of the detected workpieces, and the score that specifies the lower limit of the score that is an evaluation index. A lower limit designation column 456 is provided. In addition to being specified numerically, it can also be continuously adjusted with a slider or the like. Further, when the detailed setting button 457 is pressed, the screen is switched to the search parameter setting screen 460 of FIG. 100, and the detection detailed condition setting dialog 461 for setting more detailed detection conditions is displayed.

In the detection detailed condition setting dialog 461, a basic setting field 462, a detection condition detailed setting field 463, and an option setting field 464 are provided. In the basic setting field 462, in addition to the detection number designation field 454, a search sensitivity setting field 465 for setting the sensitivity of the three-dimensional search and a search accuracy setting field 466 for setting the accuracy of the three-dimensional search are provided. Further, in the detection condition detailed setting column 463, in addition to the tilt angle upper limit designation column 455 and the score lower limit designation column 456, the rotation angle range setting column 163 is set as the reference angle setting column 163a for setting the reference angle and the angle range. A range setting field 163b for setting is provided.

In addition, the detection conditions can be set in common with the search model, or can be set individually for each search model. In the example of FIG. 100, a search model individual designation field 467 is provided, and by turning on this check box, the detection conditions for the three-dimensional search can be set individually for each search model. An example of this is shown on the search parameter setting screen 470 of FIG. Here, when the check box of the search model individual designation field 467 is turned ON, the search model selection field 467b is displayed, and the search model for which the detection condition is set can be selected. In this example, the search model A is selected, and as the detection conditions for the search model A shown in the model list display column 391 of FIG. 94, the tilt angle upper limit designation column 455, the reference angle setting column 163a, The range setting field 163b and the score lower limit designation field 456 can be individually specified.

On the other hand, in the option setting field 464 shown in FIG. 100 and the like, the label order specification field 468 that defines the label order and the judgment label designation that specifies the judgment label are specified with respect to the label number assigned as the serial number for identifying the search result. Column 469 is provided. In the label order designation field 468, the descending order of the correlation value, the ascending order, the descending order of the height in the Z direction, the ascending order, and the like can be selected. The correlation value here is synonymous with the above-mentioned "score". In the determination label designation field 469, it is possible to specify a target for determining whether or not the position or posture is different from the assumption for a specific search result. Here, it is specified by a serial number for identifying the search result. For example, it is set when it is necessary to determine whether the search result is unreliable data such as an unexpected position or a rotation angle / tilt angle.

When the search settings for the three-dimensional search are completed in order in this way, the "OK" button 471 provided at the lower right of the operation field 142 is pressed to complete the condition setting for the three-dimensional search.
(Registration of search model based on 3D CAD data)

In the above example, the procedure of imaging and registering an actual work as a search model has been described (corresponding to FIG. 62 and the like). However, as described with reference to FIG. 28 and the like described above, the present invention can also register three-dimensional CAD data indicating the shape of the work created in advance as a search model. Hereinafter, the procedure for registering such three-dimensional CAD data as a work model will be described with reference to FIGS. 102 to 106.

First, on the search model registration method selection screen 480 of FIG. 102, from the model registration method selection field 355, select registration from CAD data (STL format) with a radio button. Correspondingly, in the operation column 142, the explanation of the procedure for registering the three-dimensional CAD data as the search model is displayed in characters and symbols. In this example, "Model the 3D information of the work surface that can be seen from each positive and negative direction (6 directions in total) of the XYZ axes in the 3D CAD data. Select only the surface you want to search from the generated model. This tool recognizes the position and orientation of the work by matching the registered surface model to the work that takes various postures. " It is shown in. In this way, when the "OK" button 481 is pressed after selecting the registration from the CAD data on the search model registration method selection screen 480 of 102, the screen shifts to the CAD data reading screen of FIG. 103. Specifically, the screen shifts to the search model registration screen 490, and the CAD data reading dialog 491 is displayed.
(CAD data reading unit)

In the CAD data reading dialog 491 shown in FIG. 103, the storage destination of the CAD data to be read is specified, and the CAD data is further selected. As the reading destination, a reading medium such as a semiconductor memory can be selected in addition to the storage unit 9. When a file is selected from the file list display field 492 and the "execute" button 493 is pressed, the reading of the three-dimensional CAD data is executed.
(Registered model selection section)

When the 3D CAD data is read, the surface to be registered as a search model is selected. Here, the search model registration screen 510 shown in FIG. 104 is displayed, and the hexagonal view corresponding to the basic direction image of the work model is automatically displayed on the registration model selection screen 511. In this state, the user can select the surface to be registered. Here, the check box is turned on by default, and the side that does not need to be registered as a search model can be excluded from the search model by turning off the check box. When the registration target is selected in this way, the "Next" button 512 is pressed to proceed to the search model registration screen 520 of FIG. 105.
(Rotational symmetry designation part)

On the search model registration screen 520 of FIG. 105, the rotation symmetry designation dialog 521 is displayed, and among the basic direction images selected by the user as the registration target, those having rotation symmetry are vertically, front-back, left-right, and the like. A surface having a symmetrical shape can be set even if it is rotated. After setting the rotational symmetry in this way, the "register" button is pressed to register the search model.

When the search model is registered, the search models registered in the model list display field 391 are displayed in a list as shown in the search model registration screen 530 of FIG. 106. As described above, a plurality of search models can be registered for one work. For example, in the example of FIG. 106, six search models A, B, C, D, E, and F shown in FIG. 105 and the like are registered for the three-dimensional CAD work model of model number 0. Therefore, in the model list display field 391, the search models A to F having the model number 0 are displayed in a list. The search model that cannot be displayed on one screen in the model list display field 391 can be scrolled to switch the display.
(Search model editorial department)

Furthermore, the registered search model can be edited. In the example of FIG. 106, when the “edit” button 531 of the operation field 142 is pressed, the model edit screen 620 shown in FIG. 114 is displayed. On the model edit screen 620, the registered search model is displayed in the image display field 141, and the set items are displayed in the operation field 142. Here, a model area setting field 621, a background removal setting field 327 as a feature setting field 622, a rotation symmetry setting field 382, and a model number display field as a model registration field 623 are provided. The user can select the item to be changed and change it as appropriate.
(Model area detailed setting section)

When the detailed setting button 624 provided in the model area setting field 621 is pressed, the model area detailed setting screen 630 shown in FIG. 115 is displayed. On the model area detailed setting screen 630, it is possible to set a mask area as an area to be excluded from the registration target of the search model within the range of the model area 362 displayed in the image display field 141. The operation column 142 is provided with a model area designation column 621, a mask area designation column 631, and a weighting area designation column 632. In the mask area designation field 631, a plurality of mask areas can be set. Each mask area can be set by a defined figure such as a rectangle or a circle, as well as an automatic detection of a free curve or a boundary portion. In addition to each mask area, weighting can be set in the weighting area designation field 632.

When scoring the search result, if there is a feature point that is more important than other feature points included in the search model, the score result can be adjusted by enclosing it in a weighted area. For example, if there is a shape in the overall shape that leads to the identification of the search model depending on the presence or absence of some features, or the posture in the rotation direction is determined, a weighting area is set for the important features. Therefore, the detection ability of the search can be improved. In addition, the degree of weighting for other feature points can be specified by a ratio. For example, when the weighting degree is set to "3", the score value can be adjusted so as to calculate the score value with the influence degree three times as much as the other feature points.
(Detailed feature setting section)

Further, when the detailed setting button 625 provided in the feature setting field 622 is pressed, the feature detailed setting screen 640 shown in FIG. 116 is displayed. On the feature detail setting screen 640, the feature setting and the presence / absence of rotational symmetry are set for the model area 362 displayed in the image display field 141. The operation column 142 is provided with a feature detail setting column 641 and a rotation target setting column 382. The feature detail setting field 641 is provided with a feature extraction interval designation field 642 for designating a feature extraction interval, a background removal setting field 327, and an edge extraction threshold setting field 643. The feature extraction interval is a parameter that affects the feature point extraction interval when a search model is generated, and "accuracy priority", "standard", and "speed priority" can be selected. When "accuracy priority" is selected, the extraction interval of feature points in the search model becomes narrower, and even finer features can be captured to improve the detection accuracy, but the amount of data increases and the processing time becomes slower. On the other hand, when "speed priority" is selected, the extraction interval of feature points in the search model is widened, fine features are omitted and the detection accuracy is lowered, but the amount of data is reduced and the processing time is shortened. It can be adjusted according to the shape and size of the workpiece to be used, the allowable processing time, and the detection accuracy.

In this way, the user can adjust necessary items so that more appropriate search results can be obtained while referring to the results of the simple three-dimensional search and the obtained scores. After the registration of the search model is completed in this way, the subsequent procedures (search area setting, search parameter setting) are performed by imaging the actual work and registering the search model as described in FIGS. 96 to 101 described above. The procedure is the same as the procedure for the above, and the description thereof will be omitted.

After registering the search model in advance as described above, the three-dimensional search is performed. The 3D search is used in actual operation, that is, when the sensor unit captures a group of workpieces stacked in bulk and performs bulk picking to specify the gripping position of the workpieces that can be gripped, and also simulates a 3D pick in advance. It is also used when performing. In the 3D pick simulation, is it possible for the end effector to grip each search result searched by the 3D search unit 8k at the gripping position specified in advance for the work model by the gripping position specifying unit 8d? This is an operation for determining whether or not.
(Search result display section)

Next, an example of performing a three-dimensional search on the workpieces stacked separately will be described with reference to FIGS. 107 to 113. FIG. 107 shows an example of a search result display screen 540 showing the result of executing a three-dimensional search on a group of workpieces stacked separately. The search result display screen 540 is provided with an image display field 141 and an operation field 142, and a list of search models used for the three-dimensional search is vertically displayed in the model list display field 391 in the operation field 142. On the other hand, in the image display field 141, the point cloud which is the search result and the score which is the evaluation index are displayed on the work obtained as the search result for the work in the loosely stacked state obtained as the input image. I'm letting you.
(Search result list)

Further, a search result list display field 541 is provided in the upper part of the image display field 141, and a list of search models and scores used for the searched search results is displayed. Here, the number of searches, the presence / absence of works, the total volume / volume of works, the model number and score of the search result of a specific label number, the position (X, Y, Z), and the like are displayed. The total volume / work volume indicates the value obtained by dividing the total volume of the point cloud located higher than the floor surface or the bottom surface of the returnable box by the average volume assumed from the registered search model, and is input. It is data showing how many works are present in the image. When this value is larger than a certain value, it is determined that there is a result of the presence or absence of the work (1). When the number of searched pieces is 0 and a gripping solution cannot be obtained, by checking whether or not there is a result of the presence or absence of this work, it is possible to check whether the picking is completed when the work is actually lost or not. It is possible to confirm whether or not the process ended with the remaining state.

The display of the search result list display field 541 can be switched. For example, a switching button 542 for switching the display contents is provided at the lower right of the search result list display field 541 of the search result display screen 540 shown in FIG. 107. The switching button 542 is composed of left and right arrows, and the number of screens 543 is displayed above the switching button 542. In this example, "1/11" indicating that the number of screens is 543, which is the first among 11 screens, is displayed. By pressing the switching button 542 to limit the number, a list of search results is obtained accordingly. The display content of the display field 541 can be switched. For example, when the switching button 542 is operated to change the number of screens 543 to "2/11", the screen is switched to the search result display screen 550 shown in FIG. 108, and the outline of the three-dimensional search result is displayed in the search result list display field 541. To. Here, in addition to the searched number, the presence / absence of the work, the total volume / the volume of the work, the inclination angle, the rotation angle, the posture ( _RX , _RY , R _Z ) and the like are displayed. When the number of screens 543 is changed to "3/11", the screen is switched to the search result display screen 560 shown in FIG. 109, and the search result list display field 541 shows the number of searches and the presence / absence of workpieces as an outline of the three-dimensional search results. , In addition to the total volume / volume of the work, the inclination of XYZ etc. is displayed as the information of the floor surface.

Further, when the number of screens 543 is changed to "4/11", the screen is switched to the search result display screen 570 shown in FIG. 110, and the search result list display field 541 shows each label number that identifies the search result as an outline of the three-dimensional search result. The model number and score of the search model used to detect the search result are displayed in the horizontal direction. Furthermore, when the number of screens 543 is changed to "5/11", the screen is switched to the search result display screen 580 shown in FIG. 111, and the search result list display field 541 shows a label number for identifying the search result as an outline of the three-dimensional search result. The position (X, Y, Z) is displayed for each position. Furthermore, when the number of screens 543 is changed to "6/11", the screen is switched to the search result display screen 590 shown in FIG. 112, and the search result list display field 541 shows a label number for identifying the search result as an outline of the three-dimensional search result. The tilt angle and rotation angle are displayed for each. Furthermore, when the number of screens 543 is changed to "7/11", the screen is switched to the search result display screen 610 shown in FIG. 113, and the search result list display field 541 shows a label number for identifying the search result as an outline of the three-dimensional search result. The posture ( _RX , _RY , R _Z ) is displayed for each.
(3D pick)

Next, a procedure for simulating a three-dimensional pick for determining whether or not the end effector can grip each of the three-dimensionally searched search results will be described with reference to FIGS. 117 to 149. When the "3D pick" button 343 is pressed on the function selection screen 340 of FIG. 89, the three-dimensional pick initial setting screen 650 of FIG. 117 is displayed, and the user is sequentially made the necessary settings for performing the three-dimensional pick. The guidance function is executed. Here, the 3D pick guidance column 655 is displayed in the upper part of the operation column 142, and the work to be performed sequentially is displayed by an icon. In this example, four steps of "initial setting", "grasping registration", "condition / sweepstakes", and "place setting" are displayed.
(3D pick initial setting screen)

On the three-dimensional pick initial setting screen 650 of FIG. 117, first, initial settings for performing a three-dimensional pick are performed. In the 3D pick guidance column 655 at the top of the operation column 142, the "initial setting" icon is highlighted to indicate to the user what work to do at this stage. The operation field 142 is provided with a calibration data selection field 651, a detection tool setting field 652, and a hand registration field 653. In the calibration data selection field 651, the calibration data calculated in advance for coordinate conversion of the coordinate system displayed by the image processing device into the coordinate system in which the robot controller operates the end effector is called and registered. In the detection tool setting field 652, a means for detecting a work is specified. In this example, the work detection three-dimensional search set to tool number 100 is selected. The registered search model settings are read through the above-mentioned three-dimensional search settings. When a search model for a different work is set in advance for a different tool number, the setting target of the three-dimensional pick can be switched by switching the selection of this detection tool.
(End effector model setting section)

In the hand registration field 653, the end effector model is registered. When the "Create hand model" button 654 is pressed here, the end effector model setting screen 660 shown in FIG. 118 is displayed. On the end effector model setting screen 660, a list of registered end effector models is displayed in the hand model list display field 661. Since the example of FIG. 118 is not registered, the end effector model is not displayed in the image display field 141 or the end effector model list display field 661 of the operation field 142. Therefore, the "add" button 662 is pressed to add the end effector model. In the image display field 141 of the end effector model setting screen 660, the flange surface FLS is displayed as a reference surface at the tip of the arm portion of the robot on which the end effector model is arranged. Further, the tool coordinate axis TAX indicating the coordinate axes on the end effector model EEM side is also displayed (details will be described later).
(End effector model editorial department)

When the "Add" button 662 is pressed in FIG. 118, the end effector model editing screen 670 shown in FIG. 119 is displayed. On the end effector model edit screen 670, the parts constituting the end effector model are set. Here, the registered parts are displayed in the parts list display field 671. In the example of FIG. 119, since the parts have not been registered yet, nothing is displayed in the parts list display column 671. When the "Add" button 672 is pressed in this state, the part addition dialog shown in FIG. 120 is displayed, and whether to read the CAD data and register the parts to be added, or to newly create a rectangular parallelepiped model or a cylinder model. Select with the radio button. The additional part is a form of the above-mentioned additional region, and is not limited to a rectangular parallelepiped or a cylinder, and any shape such as a rectangular parallelepiped, a triangular shape, or a spherical shape can be used. In the example of FIG. 120, CAD data is selected. When the "Add" button 681 is pressed and the created 3D CAD data is selected, the CAD position / orientation setting screen 690 shown in FIG. 121 is displayed, the 3D CAD data is read, and the CAD is created in the image display field 141. Along with displaying the CAD model CDM, the operation column 142 displays a position / attitude column 691 for designating the position and orientation of the CAD model CDM. The position and orientation of the CAD model CDM are specified with reference to the flange surface FLS. The user specifies the mounting position and angle of the end effector model from the position / orientation column 691. For example, specifying a 20 ° as rotation angle R _X from the state of FIG. 121, CAD model CDM displayed on the image display section 141 as shown in FIG. 122 is displayed on the 20 ° rotated orientation about the X axis .. Alternatively, if 50 mm is specified in the Z direction from the state of FIG. 121, the CAD model CDM is displayed with an offset of 50 mm in the Z direction as shown in FIG. 123. When the position and orientation of the CAD model CDM are adjusted in this way and the "OK" button 692 of FIG. 121 is pressed, the CAD data is registered and displayed in the parts list display field 671.

You can add multiple parts. For example, when the "Add" button 682 is pressed from the end effector model edit screen of FIG. 124, the cylindrical model is selected in the part addition dialog 680 of FIG. 120, and the "Add" button 681 is pressed, the additional area position of FIG. The posture setting screen is displayed, and a columnar model is displayed in the image display field 141 as a new additional area ADA. The part being edited is highlighted. In the example of FIG. 125, the additional region ADA is displayed in red and the CAD model CDM is displayed in gray or wireframe. This makes it easier for the user to visually grasp the part currently being edited. Similarly, the position and posture of the additional area ADA to be added are specified from the position / posture column 691. In this example, the Z position is specified as −50 mm in the position / orientation column 691, and the additional region ADA is arranged from the flange surface FLS in the direction opposite to the CAD model CDM. Further, as the size of the additional area ADA, the radius and height of the columnar model can be specified in the size specification field 693. After setting the conditions for the additional area ADA in this way, when the "OK" button 692 is pressed, the additional area ADA is registered, and as shown in FIG. 126, a cylinder is displayed in the parts list display field 671 of the end effector model edit screen. Is added and displayed.
(Tool coordinate specification part)

It is also possible to adjust the position and posture of the registered parts. For example, when the selected part is changed from the additional area ADA to the CAD model CDM from the end effector model edit screen of FIG. 126 and from the image display field 141 or the parts list display field 671, the result is as shown in FIG. 127. , The position (X, Y, Z) and the posture ( _RX , _RY , R _Z ) can be adjusted in the tool coordinate specification field 673. For example, as shown in FIG. 128, when the Z position is specified to 145 mm, the tool coordinate axis TAX is displayed in the image display field 141 by being moved from the flange surface FLS in the Z direction by a specified distance. _{Further, when the rotation R X} is specified to 20 ° as shown in FIG. 129 in this state, the tool coordinate axis TAX is displayed rotated by 20 ° clockwise around the X axis.

In this way, a plurality of parts are registered, and these are combined to build an end effector model EEM. When the setting of each part is completed, when the "OK" button 674 is pressed on the end effector model editing screen shown in FIG. 128, a plurality of parts are combined and registered as the end effector model EEM, and the end effector model setting screen shown in FIG. As shown in 660, it is displayed as "hand 0" in the hand model list display field 661. Further, in the image display field 141, the separately displayed parts are displayed as an integrated end effector model EEM.

In this example, an example in which parts are combined to form an end effector model has been described, but the parts are not necessarily limited to the end effector model or a part thereof, and for example, cables and the like can be represented by parts. In this way, the parts form a form of an additional model that expresses a part of the end effector model, an object added to the surface of the end effector model, a peripheral member, or the like.
(Gripping reference point HBP)

Further, the gripping reference point HBP of the end effector model EEM is displayed superimposed on the image display field 141. The gripping reference point HBP indicates a position where the work is gripped by the end effector model EEM. For example, in the end effector model EEM, the middle between the claws for gripping the work is set as the gripping reference point HBP. The gripping reference point HBP may be calculated on the image processing apparatus side so as to be determined according to a predetermined rule, or may be configured so that the user can specify an arbitrary position.

Further, it is preferable that the tool coordinate axes that define the position and orientation of the end effector model EEM are also displayed on the image display field 141. The tool coordinate axes are preferably rotated coordinate axes RAX that are changed according to the rotation of the end effector model EEM. Further, it is more preferable that the origin of the rotated coordinate axis RAX coincides with the gripping reference point HBP. This makes it easier for the user to visually grasp the state of the end effector model EEM when setting the gripping position.
(Grip registration unit)

When the end effector model EEM is registered as the initial setting in this way, the gripping position is next registered. On the grip registration screen 710 of FIG. 131, the highlight display shifts from the “initial setting” icon to the “grasping registration” icon in the 3D pick guidance field 655. In the operation column 142, a search model switching column 711 for selecting a registered search model and a grip registration list display column showing a list of registered grip positions for the search model selected in the search model switching column 712. 712 is displayed. As described above, in this example, the work model for registering the gripping position is shared with the search model used for the three-dimensional search. Here, it is assumed that six search models A to F with model number 0 registered in FIG. 104 are registered as registered search models in the search model switching column 711. Further, in the example of FIG. 131, since the gripping position has not been registered yet, nothing is displayed in the gripping registration list display column 712. The search model A is selected in the search model switching field 711, and the search model SMA corresponding to A is displayed in the image display field 141. Further, the reference coordinate axis BAX, which is a coordinate axis indicating the position and orientation of the search model SMA, is superimposed and displayed. From this state, the user registers the gripping position with respect to the search model A. Specifically, when the "add" button 713 is pressed, the screen is switched to the grip registration screen shown in FIG. 132, and the grip setting dialog 720 is displayed.
(Grip setting unit)

In the grip setting dialog 720, the operation field 142 is provided with a basic setting field 721, a grip position coordinate designation field 724, and a movement amount setting field 727. The basic setting field 721 is provided with a grip label setting field 722 and a hand selection field 723. In the gripping label setting field 722, a gripping label which is identification information for specifying the gripping position is set. Here, as the gripping label, "A" indicating the search model and the serial number "000" of the gripping position are set. Further, in the hand selection field 723, the end effector model EEM that grips the gripping position set in the gripping label setting field 722 is selected. In this example, "hand 0" is selected in the hand selection field 723, and the end effector model EEM is displayed in the image display field 141 accordingly. In this way, the gripping position is set in association with the end effector model EEM.

In the grip position coordinate specification field 724, the coordinate position (X, Y, Z) and the posture ( _RX , _RY , R _Z ) indicating the grip position can be directly specified numerically by the user. A "simple setting navigation" button 726 is provided to execute a guidance function for guiding the user to set the gripping position.

Further, the movement amount setting field 727 defines the direction in which the end effector model EEM is moved. Here, the amount of approach movement that brings the end effector model EEM closer to the search model SMA is defined as the distances in the X, Y, and Z directions so that the end effector model EEM grips the search model SMA. In the example of FIG. 132, an example of moving 100 mm in the Z direction is shown.
(Grip position setting guidance function)

When the "simple setting navigation" button 726 is pressed from the screen of FIG. 132, the gripping position setting guidance function is executed. Specifically, as shown in FIG. 133, the setting method selection dialog 730 is displayed, and the user is asked to either set the gripping position using the three-dimensional viewer or select using the robot. Select with the radio button. In this example, it is selected to be set on the 3D viewer. When the "OK" button 731 is pressed in this state, the registration guidance function on the three-dimensional viewer is executed.
(Registration guidance function on 3D viewer)

In the registration guidance function on the three-dimensional viewer, the gripping position is registered through four steps. Here, the user is asked to specify the gripping position via the three-dimensional viewer registration screen shown in FIGS. 134 to 141. Specifically, when the "OK" button 731 is pressed in FIG. 133, the screen switches to the XY designation screen 740 in FIG. 134.
(First step: XY designation)

In FIG. 134, the XY designation screen 740 is displayed as a form of the XY designation unit. Here, as the first step for registering the gripping position, the gripping position is specified from the plan view while the search model SMA is displayed in the plan view. Specifically, the search model SMA is displayed in the image display field 141 in a two-dimensional manner on the XY plane, and the X and Y coordinates to be gripped are specified on the search model SMA. In the operation column 142, the work to be performed by the user is described in text or the like as STEP1. Here, "Please select the XY position to be gripped on the camera image." Is displayed. Further, in the example of FIG. 134, the search model A is enlarged and displayed as a height image (search model SMA) in the image display field 141, and in this state, the user uses a pointing device such as a mouse to grip the portion. Select P1. The XY coordinates are input to the XY coordinate designation field 741 according to the gripping designated position P1. Further, when the X coordinate and the Y coordinate of the XY coordinate designation field 741 are adjusted, the gripping designated position P1 of the image display field 141 is also updated accordingly. In the XY designation screen 740, only the X coordinate and the Y coordinate can be specified among the position parameters, in other words, other position parameters cannot be specified. In this way, by avoiding the situation where the user is confused by making it possible to set a large number of position parameters at the same time as in the past, and by limiting the position parameters that can be set, the user can accurately grasp the position parameters that can be set. Then, you will be guided to set the position parameters in sequence. When the specification of the XY coordinates is completed in this way, the "Next" button 742 is pressed. As a result, the process proceeds to the second step, and the screen is switched to the _{ZZ designation screen 750 of FIG. 135.}
(Second process: Z-R _Z designation)

The Z-R _Z designation screen 750 of _{FIG. 135 is a form of the Z-R Z} designation unit, and is a screen for allowing the user to specify the Z coordinate and the posture R _Z. In the operation column 142, the work to be performed by the user in the second step is displayed and described in text. Here, "STEP2 Please specify the Z position to be gripped and the posture angle R _{Z of the} hand" is displayed. Further, the operation column 142 is provided with a Z designation column 752 _{for designating the Z} position and an R _Z designation column 753 _{for designating the R Z posture as the Z-R Z designation column 751.} Further, in the image display field 141, the end effector model EEM and the search model SMA are displayed three-dimensionally. That is, the image display field 141 functions as a three-dimensional viewer that displays the end effector model and the search model in a three-dimensional manner. When the numerical values in the Z designation field 752 and the R _Z designation field 753 are changed, the three-dimensional display contents of the end effector model EEM and the search model SMA in the image display field 141 are updated accordingly. Further, by dragging the image display field 141, the user can change the viewpoint of the three-dimensional viewer, so that the gripping setting state can be confirmed in detail.

In addition, in the image display field 141, which is a three-dimensional viewer, only the Z axis is displayed as the coordinate axis related to the position parameter currently being set among the coordinate axes. This Z-axis is the corrected rotation Z-axis AXZ after correction displayed by the above-mentioned ZZ Euler angles. In this way, by displaying only the currently adjustable coordinate axes and making it impossible to adjust other positional parameters on this screen, it does not cause unnecessary confusion to the user and is appropriate according to a predetermined order. Appropriate guidance is realized so that only various positional parameters are sequentially set.

Further, in the example of FIG. 135 and the like, the end effector model EEM is brought closer to the work model by displaying the axis in the gripping direction (here, the correction rotation Z axis AXZ) extending through the gripping reference point HBP. It is possible to clearly indicate the moving direction for the vehicle. Further, the moving direction of the end effector model may be indicated by an arrow. At this time, the moving direction of the end effector model may be displayed overlapping on the axis of the gripping direction, or may be displayed at another position.

Further, the Z-R _Z designation screen 750 is provided with a "fit" button 154 as a form of the relative position setting unit. As described above, the "fit" button 154 realizes a fit function that automatically moves the end effector model to the gripping position of the work model. When the "fit" button 154 is pressed, the end effector model EEM is moved as shown in FIG. 136, and the end effector model EEM is changed to the state of being in contact with the search model SMA. That is, the gripping reference point HBP of the end effector model EEM is moved toward the gripping designated position P1 of the search model SMA. It is preferable to adjust the initial position when the end effector model EEM is displayed in the state of FIG. 135 so that the moving direction coincides with the Z axis. As a result, by moving the end effector model EEM only in the Z direction, the search model SMA can be brought into contact with the search model SMA, which can be displayed in an easy-to-understand manner, and the user can also move the end effector model EEM. It can be done easily.

Further, the Z position of the Z designation field 752 in the operation field 142 is changed according to the movement of the end effector model EEM in the image display field 141. In this example, it is changed from 200 mm in FIG. 135 to 50 mm. Here, when the end effector model EEM is moved in the height direction, that is, in the Z-axis direction toward the gripping designated position P1 designated in FIG. 134 by the fit function, the position in contact with the search model SMA is used as a reference. From here, it is moved to the position returned to the front by a predetermined offset amount. Further, with reference to the gripping designated position P1, only the Z axis currently being set is displayed in the image display field 141 among the coordinate axes.

In this state, the user further _{specifies the posture R Z} from the R _Z designation field 753. In the posture of the end effector model EEM shown in FIG. 136, since the side surface of the search model SMA inclined in a V shape is gripped, the end effector model EEM is rotated in the Z-axis direction and parallel to each other. Adjust to grip the surface. At this time, _{in addition to the user directly inputting the rotation angle in the R Z} designation field 753, the rotation of the end effector model EEM can be more easily specified by providing the angle selection button 754 for inputting a predetermined angle. In this example, as the angle selection button 754, four angle buttons of 0 °, 90 °, 180 °, and −90 ° are arranged. When the user presses the 90 ° button _{, 90 ° is input to the R Z} designation field 753, the end effector model EEM is rotated 90 ° in the Z-axis direction as shown in FIG. 137, and the search model SMA is correctly grasped. It can be modified to the state where it can be done.
(Third process: _RY designation)

When the Z-R _Z designation of the second step is completed in this way, the "Next" button 755 is pressed to proceed to the _{RY designation of the third step.} Specifically, when the "Next" button 755 is pressed, the screen shifts to the _{RY designation screen 760 of FIG. 138.} The _RY designation screen 760 is _{a form of the RY} designation unit, and the posture _RY is designated. The _{RY designation field 761 is displayed in the operation field 142 of the RY} designation screen 760, and the user is told "Please specify _{the posture angle RY} of the STEP3 hand" as the work to be performed in the third step. Is displayed. Further, in the image display field 141, only the correction rotation Y-axis AXY, which is the currently set Y-axis, is displayed among the coordinate axes. _{From this state, the user specifies the posture RY} from the _RY designation screen 760 according to the posture of the end effector model EEM holding the search model SMA. Alternatively, the end effector model EEM is dragged and rotated on the image display field 141, which is a three-dimensional viewer. For example, in the state of FIG. 138, when 90 ° is specified as the _{attitude RY from the RY} designation screen 760 _{, the screen is switched to the RY} designation screen of FIG. 139, and the end effector model EEM is rotated by 90 ° about the Y axis. Is displayed. _{When the RY} designation of the third step is completed in this way, the "Next" button 762 is pressed to shift to the _{RY designation of the fourth step.}
(Fourth process: _RY designation)

Figure 140 shows the R _X designation screen 770 of the fourth step. R _X specifying screen 770 is a form of R _X designating unit, is provided with the R _X designation field 771 for designating a posture R _X. Also in the image display field 141, superimposed on the end effector model EEM and search model SMA, a X-axis regarding R _X settable position specifying screen 770 Parameter R _X among the coordinate axes correction rotation X axis AXX appears To. In this state the user designates the position R _X from R _X specification field 771. Alternatively, the end effector model EEM is dragged and rotated in the image display field 141. For example as the posture R _X, the R _X designation column 771 in the example of FIG. 140, where 180 ° corresponds to the current of the gripping position is input, specifying 0.99 °, as shown in FIG. 141, the specified orientation of the end effector model EEM and search model SMA is changed in the image display field 141 is a three-dimensional viewer in accordance with the attitude R _X.

In this manner, when the finish the designation of attitude R _X and, by pressing the "Finish" button 772, to terminate the registration of the gripping position. When the gripping position is registered, the gripping registration A-000 is added to the gripping registration list display field 712 as shown in FIG. 142. Further, the search model A corresponding to the grip registration A-000 is highlighted in the search model switching column 711. Further, in the image display field 141, the state of gripping the search model SMA at the position and posture of the end effector model EEM registered for gripping is displayed three-dimensionally. Further, as the coordinate axes, the reference coordinate axis BAX on the search model SMA side is superimposed and displayed.

Hereinafter, in the same manner, another gripping position is registered in the search model A, or a gripping position is added to the other search models B to F. The gripping position can be added by pressing the "addition" button 713 provided in the gripping registration list display field 712 as described above. In the example of FIG. 143, an example in which the grip registration B-000 for the search model B is registered is shown.
(Copy function)

Further, when additionally registering the gripping position, a copy function for copying the already registered gripping position can be used. By pressing the "copy" button 714 provided in the grip registration list display field 712 on the screen shown in FIG. 142 or the like, the registered grip position can be duplicated. Further, by modifying the copied gripping position as a base, it is possible to efficiently register the gripping position. When the "edit" button 715 is pressed from the state shown in FIG. 142 or the like, the grip setting dialog is displayed and the grip position can be set as described above. Here, as described above, the "easy setting navigation" button 726 can be pressed to execute the gripping position setting guidance function, and necessary parts can be corrected.
(Condition verification screen 780)

After registering the gripping positions in the search models A to F as described above, the “done” button 716 is pressed on the screen shown in FIG. In the condition verification step, a three-dimensional search is performed on the work group stacked separately to detect the work, and it is verified whether or not the work can be gripped by the end effector at the designated gripping position. Here, the simulation is performed using the end effector model EEM and the search model. An example of the condition verification screen 780 is shown in FIG. 144. On this condition verification screen 780, the highlight display shifts from the "grasping registration" icon to the "condition / verification" icon in the 3D pick guidance column 655. The operation column 142 is provided with a detection condition setting column 781 for setting a condition for detecting the gripping position based on the three-dimensional search result, and a verification column 785 for verifying whether or not the detected workpiece can be gripped. In the detection condition setting field 781, the end effector model interferes with the detection number designation field 782 that specifies the upper limit of the number of gripping solutions to be detected, the hand tilt angle upper limit designation field 783 that specifies the upper limit of the tilt angle of the end effector model, and the end effector model. In order to expand the range to be used, a margin setting field 784 for setting a margin from the wall surface of the storage container is provided. If the end effector model EEM penetrates into the range of the distance (5 mm in FIG. 144) specified in the margin setting field 784, it is determined that the end effector model EEM interferes.

On the other hand, in the verification column 785, a detection number display column 786 indicating the number of search results actually detected as a result of the three-dimensional search simulation and a label number of the search model to be verified among the detected search results are displayed. A label designation field 787 to be designated and a "verification for each work" button 788 for executing verification for each work are provided. The search result of the label number specified in the display label designation field 787 is displayed in the image display field 141.
(Verification for each work)

When the "verification for each work" button 788 is pressed, the verification dialog 790 shown in FIG. 145 is displayed. The verification dialog 790 is a form of the work selection screen 210 described with reference to FIG. 74. In the verification dialog 790 of FIG. 145, it is verified whether or not each of the detected workpieces can be gripped. In this example, the verification dialog 790 is provided with a target work selection field 791, a detection search model display field 792, and a "grasping confirmation" button 793. When the "grasping confirmation" button 783 is pressed, the detection result display dialog 810 shown in FIG. 146 is displayed. The detection result display dialog 810 is a form of the gripping solution candidate display screen 220 shown in FIG. 75 and the like. Here, for each surface (search image and estimated image) detected for the work selected in the target work selection field 791 (second work in FIG. 145), the determination result of whether or not gripping is possible at the set gripping position Is listed in the gripping solution candidate display column 811. In the example of FIG. 146, the gripping label C-000 is selected, and accordingly, the end effector model EEM is arranged and displayed at the corresponding gripping position in the image display field 141, and the end is displayed according to the determination result. The effector model EEM is colored and displayed. In this example, since the determination result is NG, the end effector model EEM is displayed in red. Further, in the gripping solution candidate display column 811, a point cloud interference error is displayed as the cause of the gripping failure. As a result, the user is given a guideline for taking necessary measures such as correction or addition of the gripping position.

Further, when the gripping label A-000, which is a determination OK, is selected in the detection result display dialog of FIG. 147, the end effector model EEM gripped at the corresponding gripping position is displayed in white in the image display field 141 accordingly. It is displayed in. In this way, it is possible to determine whether or not gripping is possible and take appropriate measures as necessary.

Further, on the condition verification screen 780 of FIG. 144, when the detection condition detailed setting button 789 provided in the detection condition setting field 781 is pressed, the screen switches to the detection condition detailed setting screen 820 of FIG. 148. The detection condition detailed setting screen 820 is provided with a detection condition setting field 821 for the gripping position, an interference judgment setting field 822 for the end effector and the box, and the floor, and an interference judgment setting field 823 for the end effector and the three-dimensional point cloud. Has been done.

In this way, when the condition verification process is completed, the process finally shifts to the place setting process that defines the placement position of the work. An example of the place setting screen 830 is shown in FIG. 149. In the 3D pick guidance field 655 of the place setting screen 830, the highlight display shifts from the "condition / verification" icon to the "place setting" icon. The operation field 142 is provided with a tool coordinate setting field 831 and a place position registration field 832. From this screen, the user makes settings related to the place of work.

The image processing apparatus of the present invention, an image processing method, image processing program, and readable recording medium body in a computer can be suitably used for applications to verify the operation of the bulk picking robot.

1000, 2000, 3000, 4000, 7000, 8000, 9000 ... Robot system 100, 200, 300, 400, 700, 800, 900 ... Image processing device 2 ... Sensor unit: 2b ... Sensor control unit; 2c ... Input image acquisition unit 3, 3B ... Display unit; 3a ... Six-view display area; 3b ... Image display area; 3c ... Gripping solution candidate display area; 3d ... Work gripping possibility display area; 3e ... Work gripping impossible factor display area; 3f ... Display interface 4 ... Operation unit; 4b ... Input / output interface 5 ... Robot body 6 ... Robot controller; 6b ... Robot interface 7 ... Robot operating tool 8c ... Positioning unit; 8d ... Gripping position specifying unit; 8d1 ... Work side gripping location designation unit; 8d2 ... end effector side grip setting unit; 8d3 ... grip reference point setting unit; 8d4 ... grip direction setting unit; 8d5 ... relative position setting unit; 8d6 ... second designated unit; 8d7 ... third designated unit; 8d8 ... grip position copy unit 8d9 ... First designated unit; 8e ... Basic direction image selection unit; 8f ... End effector mounting surface setting unit; 8g ... Search model registration unit; 8g'... Model registration unit; 8h ... Rotation angle limiting unit; 8i ... Search model Selection unit; 8j ... Image selection unit; 8k ... Three-dimensional search unit; 8l ... Three-dimensional pick determination unit; 8m ... Interference determination unit; 8n ... Tilt angle setting unit; 8o ... Angle determination unit; 8p ... Search result integration unit; 8q ... Evaluation index calculation unit; 8r ... Gripping priority determination unit; 8s ... Cross section model generation unit; 8t ... Work model registration unit; 8u ... End effector model registration unit; 8v ... Additional model creation unit; 8w ... Calibration unit; 8x ... Conversion unit; 8y ... End effector mounting position calibration unit; 8z ... Image estimation unit 9 ... Storage unit; 9b ... Gripping position storage unit 10 ... Calculation unit 8e'... Basic direction image generation unit 130, 130B ... Search model registration screen 131 ... "Registration target" check box; 131B ... Selection check box 140 ... Grip registration screen 141 ... Image display field 142 ... Operation field 143 ... Search model selection field 144 ... Gripping posture display field 145 ... Edit button 146 ... Add button 147 ... Delete button 150 ... Gripping posture addition screen 153 ... Gripping posture coordinate information 154 ... "Fit" button 160 ... Tilt angle / rotation angle setting screen 161 ... Attitude restriction search model selection field 162 ... Tilt angle upper limit setting field 163 ... Rotation angle range setting Column; 163a ... Reference angle setting column; 163b ... Range setting column 170 ... End effector mounting position setting screen 171 ... End effector mounting position setting column 180 ... Gripping position designation screen 181 ... Gripping position designation field 190 ... Multiple gripping position selection screen 191 ... Surface selection field 192 ... Gripping position list display 210 ... Work selection screen 211 ... Target work selection field 212 ... Detection search model display field 213 ... " Gripping confirmation button 220 ... Gripping solution candidate display screen 221 ... Gripping solution candidate display field 223 ... Work gripping position display field 224 ... Work gripping possibility display field 225 ... Work gripping impossible factor display field 230 ... XY designation screen 240 ... Z -R _Z designation screen 241 ... Z coordinate specification field 242 ... R _Z rotation angle designation field 250 ... R _Y designation screen 251 ... R _Y rotation angle designation field 260 ... R _X designation screen 261 ... R _X rotation angle designation field 270 ... position Parameter specification screen 271 ... X, Y coordinate specification field 272 ... "Next" button 273 ... Z coordinate, R _Z rotation angle specification field 274 ... R _X , _RY rotation angle specification field 280 ... X-Y-Z specification screen 281 ... X, Y, Z coordinate specification field 290 ... Three-dimensional image viewer 310 ... Additional area setting screen 311 ... Basic figure display field 312 ... "Add" button 313 ... "Edit" button 320 ... Basic figure edit screen 321 ... Basic image parameters Adjustment column 327 ... Background removal setting column 330 ... End effector imaging screen 331 ... End effector position specification column 332 ... "Image capture" button 340 ... Function selection screen 341 ... Function list display column 342 ... "3D search" button 343 ... "3D pick" Button 350 ... Search model registration method selection screen 351 ... Flow display unit 352 ... "Model registration" icon 353 ... "Floor / box setting" icon 354 ... "Search setting" icon 355 ... Model registration method selection field 356 ... "OK" Button 360 ... Actual measurement data capture screen 361 ... "Measurement execution" button 362 ... Model area 363 ... "Next" button 364 ... "? Button 370 ... Search model exclusion area setting screen 371 ... "?" Button 372 ... Background removal setting field 373 ... "?" Button 374 ... "Mask area" button 375 ... "Next" button 380 ... Rotational symmetry setting screen 381 ... "?" Button 382 ... Rotational symmetry setting field 383 ... "Registration" button 390 ... Search model registration screen 391 ... Model list display field 392 ... "Add" button 393 ... Model registration method specification field 410 ... Search model registration screen 411 ... "Done" button 420 ... Search area setting screen 421 ... Search area setting field 422 ... "Designation method selection navigation" button 423 ... Designation method selection field 430 ... Search area setting screen 431 ... Floor surface designation dialog 440 ... Search area setting screen 441 ... Floor information display field 442 ... Noise removal field 443 ... "Done" button 450 ... Search parameter setting screen 451 ... Detection condition setting field 452 ... Feature extraction condition setting field 453 ... Judgment condition setting field 454 ... Detection number designation field 455 ... ... Inclination angle upper limit specification field 456 ... Score lower limit specification field 457 ... Detailed setting button 460 ... Search parameter setting screen 461 ... Detection detailed condition setting dialog 462 ... Basic setting field 463 ... Detection condition detailed setting field 464 ... Option setting field 465 ... Search sensitivity setting field 466 ... Search accuracy setting field 467 ... Search model individual specification field; 467b ... Search model selection field 468 ... Label order specification field 469 ... Judgment label specification field 470 ... Search parameter setting screen 471 ... "OK" button 480 ... Search model registration method selection screen 481 ... "OK" button 490 ... Search model registration screen 491 ... CAD data reading dialog 492 ... File list display field 493 ... "Execute" button 510 ... Search model registration screen 511 ... Registration model selection screen 512 ... "Next" button 520 ... Search model registration screen 521 ... Rotational symmetry designation dialog 522 ... "Registration" button 530 ... Search model registration screen 531 ... "Edit" button 540 ... Search result display screen 541 ... Search result list display field 542 ... Switching button 543 ... Number of screens 550 ... Search result display screen 560, 570, 580, 590, 610 ... Search result display screen 620 ... Model edit screen 621 ... Model area setting field 622 ... Feature setting field 623 ... Model registration field 624 ... Detailed setting button 625 ... Detailed setting button 630 ... Model area detailed setting screen 631 ... Mask area specification field 632 ... Weighted area designation field 640 ... Feature detail setting screen 641 ... Feature detail setting field 642 ... Feature extraction interval specification field 643 ... Edge extraction threshold setting field 650 ... Three-dimensional pick initial setting screen 651 ... Calibration data selection field 652 ... Detection tool setting field 653 ... Hand registration field 654 ... "Create hand model" button 655 ... 3D Pick guidance field 660 ... End effector model setting screen 661 ... Hand model list display field 662 ... "Add" button 670 ... End effector model edit screen 671 ... Parts list display field 672 ... "Add" button 673 ... Tool coordinate specification field 674 ... "OK" button 680 ... Parts addition dialog 681 ... "Add" button 690 ... CAD position / orientation setting screen 691 ... Position / orientation column 692 ... "OK" button 693 ... Size specification column 710 ... Grip registration screen 711 ... Search model switching column 712 ... Gripping registration list display field 713 ... "Add" button 714 ... "Copy" button 715 ... "Edit" button 716 ... "Done" button 720 ... Gripping setting dialog 721 ... Basic setting field 722 ... Gripping label setting field 723 ... Hand selection field 724 ... Grip position coordinate specification field 725 ... Coordinate orientation specification field 726 ... "Easy setting navigation" button 727 ... Movement amount setting field 730 ... Setting method selection dialog 731 ... "OK" button 740 ... XY designation screen 741 ... XY coordinate designation field 742 ... "Next" button 750 ... Z-R _Z designation screen 751 ... Z-R _Z designation field 752 ... Z designation field 753 ... R _Z designation field 754 ... Angle selection button 755 ... "Next" Button 760 ... R _Y designation screen 761 ... R _Y designation field 762 ... "Next" button 770 ... R _X designation screen 771 ... R _X designation field 772 ... "Done" button 780 ... Condition verification screen 781 ... Detection condition setting field 782 ... Detection quantity designation column 783 ... Hand tilt angle upper limit designation column 784 ... Margin setting column 785 ... Verification column 786 ... Detection quantity display column 787 ... Display label specification column 788 ... "Verification for each work" button 789 ... Detection condition detailed setting Button 790 ... Verification dialog 791 ... Target work selection field 792 ... Detection search model display field 793 ... "Gripping confirmation" button 810 ... Detection result display dialog 811 ... Gripping solution candidate display field 820 ... Detection condition detailed setting screen 821 ... Gripping Position detection condition setting field 822 ... Interference judgment setting field between end effector and box, floor 823 ... Interference judgment setting field between end effector and 3D point group 830 ... Place setting screen 831 ... Tool coordinate setting field 832 ... Place position Registration fields 840, 850 ... Misalignment correction screen 841 ... End effector setting Column 842 ... "OK" button 843 ... "Automatic correction" button TAX ... Tool coordinate axis BAX ... Reference coordinate axis RAX ... Rotated coordinate axis HBP ... Gripping reference point EET, EET2, EET3 ... End effector ARM ... Arm WK, WK6, WK7, WK10 ... Work WK2 ... Work WK3 with cavities ... Plate-shaped work WK9 ... Flat plate-shaped work WKC ... Cubic work CWM ... Three-dimensional CAD data work model CDM ... CAD model WM9 ... Flat plate-shaped work models WM10, WM11 ... Work model WM11F ... Work model displayed in a plan view WM11P ... Work model displayed in a perspective view ... Work model WKI at the time of search model registration ... Work ADA obtained as an input image ... Additional area FLS ... Flange surface VP ... Outer product vector WKI'... Rotated input image IBX ... Virtual rectangular parallelepiped MWM ... Measured data BDI ... Basic direction image SM ... Search model SMA ... Search model P1 ... Grip designated position SCP ... Surface feature point OCP ... On contour Feature points EEM, EM10 ... End effector model PRJ ... Projectors CME1, CME2, CME3, CME4 ... Camera PC ... Point group BSL ... Basic axes SP1 to SP5 ... Cross section positions SS1 to SS5 ... Cross section TDP ... Three-dimensional point PP3 ... Projection point RBT ... Robot BX ... Storage container STG ... Stage AXX ... Correction rotation X-axis AXY ... Correction rotation Y-axis AXZ ... Correction rotation Z-axis

Claims (12)

The three-dimensional shape of a plurality of workpieces stacked in the work space is measured by a sensor unit arranged at a fixed position capable of imaging the work space, and an end effector provided at the tip of the arm portion of the robot by a robot controller. It is an image processing device for controlling a robot that performs a picking operation.
An end effector model registration unit for registering an end effector model, which is three-dimensional CAD data that virtually represents the three-dimensional shape of the end effector,
A display unit for displaying the end effector model in a three-dimensional manner in a virtual three-dimensional space,
Ri virtual three-dimensional space der displayed on the display unit, operating the position and orientation of the calculated end-effector, the robot controller an end effector coordinate system of vision space of the captured end effector sensor unit a calibration unit get the calibration information for the position and the coordinate transformation the orientation of the coordinate system of the robot space to be,
A conversion unit that converts coordinates between the coordinate system in the vision space and the coordinate system in the robot space based on the calibration information obtained by the calibration unit.
An input image acquisition unit that acquires an input image having a three-dimensional shape from an image including an end effector measured by the sensor unit, and an input image acquisition unit.
A three-dimensional search unit for performing a three-dimensional search for specifying an image area corresponding to the position and orientation of the end effector using the end effector model as a search model from the input images acquired by the input image acquisition unit.
The difference between the position and orientation of the flange where the end effector is attached in the coordinate system of the robot space and the position and orientation of the end effector is reflected in the position and orientation of the end effector model registered by the end effector model registration unit. It is provided with an end effector mounting position calibration unit for calibrating the position and orientation of the end effector model.
The conversion unit acquires the position and orientation of the flange portion in the coordinate system of the robot space and converts it into the coordinate system of the vision space.
The end effector mounting position calibration unit
The position and orientation of the flange portion in the coordinate system of the vision space converted by the conversion unit, and
The position and orientation of the end effector in the coordinate system of the vision space detected by searching with the three-dimensional search unit.
By calculating the difference between the above and reflecting the calculated difference in the position and orientation of the end effector model, the end effector model registered by the end effector model registration unit is displaced from the flange portion by the difference. Ru image processing apparatus name calibrated state mounted in position. The three-dimensional shape of a plurality of workpieces stacked in the work space is measured by a sensor unit arranged at a fixed position capable of imaging the work space, and an end effector provided at the tip of the arm portion of the robot by a robot controller. It is an image processing device for controlling a robot that performs a picking operation.
An end effector model registration unit for registering an end effector model, which is three-dimensional CAD data that virtually represents the three-dimensional shape of the end effector,
A display unit for displaying the end effector model in a three-dimensional manner in a virtual three-dimensional space,
It is a virtual three-dimensional space displayed on the display unit, and the position and orientation of the end effector calculated in the coordinate system of the vision space in which the end effector is imaged by the sensor unit and the robot controller operate the end effector. A calibration unit that acquires calibration information for coordinate conversion between the position and orientation of the coordinate system in robot space, and
A conversion unit that converts coordinates between the coordinate system in the vision space and the coordinate system in the robot space based on the calibration information obtained by the calibration unit.
An input image acquisition unit that acquires an input image having a three-dimensional shape from an image including an end effector measured by the sensor unit, and an input image acquisition unit.
A three-dimensional search unit for performing a three-dimensional search for specifying an image area corresponding to the position and orientation of the end effector using the end effector model as a search model from the input images acquired by the input image acquisition unit.
The difference between the position and orientation of the flange where the end effector is attached in the coordinate system of the robot space and the position and orientation of the end effector is reflected in the position and orientation of the end effector model registered by the end effector model registration unit. By making it, the end effector mounting position calibrating unit for calibrating the position and orientation of the end effector model
With
The conversion unit acquires the position and orientation of the end effector in the coordinate system of the vision space and converts it into the coordinate system of the robot space.
The end effector mounting position calibration unit
The position and orientation of the end effector in the coordinate system of the robot space converted by the conversion unit, and
The position and orientation of the flange in the coordinate system of the robot space
By calculating the difference between the above and reflecting the calculated difference in the position and orientation of the end effector model, the end effector model registered by the end effector model registration unit is displaced from the flange portion by the difference. An image processing device that is calibrated so that it is mounted in a posture. The image processing apparatus according to claim 1 or 2.
An image processing device configured such that the input image acquisition unit acquires an input image including an attachment position in which an end effector is attached to a flange portion at the tip of an arm portion of the robot. The image processing apparatus according to any one of claims 1 to 3, further comprising:
It is provided with a storage unit for storing calibration information by the calibration unit.
An image processing device configured such that the conversion unit reads out the calibration information stored in the storage unit and converts the coordinate position. The image processing apparatus according to any one of claims 1 to 4, further comprising:
An image processing device including a conversion formula in which the calibration information is calculated by the calibration unit to convert the position and orientation of the coordinate system in the vision space into the position and orientation of the coordinate system in the robot space. The image processing apparatus according to any one of claims 1 to 5, further comprising:
A work model registration unit for registering a work model that virtually represents the three-dimensional shape of the work,
With respect to the work model registered in the work model registration unit, a gripping position specifying unit for specifying one or more gripping positions for gripping the work model with the end effector model,
A virtual representation of the three-dimensional shape of the work used when performing a three-dimensional search to specify the posture and position of each work for a plurality of work groups included in the input image acquired by the input image acquisition unit. The search model registration unit for registering the searched search model,
Is equipped with
The three-dimensional search unit performs a three-dimensional search from the input images acquired by the input image acquisition unit using the search model registered by the search model registration unit, and corresponds to the position and orientation of each work. An image processing device configured to specify an image area. The image processing apparatus according to claim 6, further
An image including a three-dimensional pick determination unit that determines whether or not the work can be gripped by the end effector model calibrated by the end effector mounting position calibration unit at the grip position specified by the grip position specifying unit. Processing equipment The image processing apparatus according to claim 6 or 7, further comprising.
When gripping the work model at the gripping position specified by the gripping position specifying unit, it is determined whether or not the end effector model calibrated by the end effector mounting position calibration unit interferes with other work. An image processing device including an interference determination unit. The image processing apparatus according to claim 8.
The basic axis of the end effector can be set,
The interference determination unit generates a plurality of cross-section models by cutting the end effector model configured by the end effector mounting position calibration unit at a plane intersecting the basic axis of the end effector, and generates a plurality of cross-section models. An image processing device configured to determine interference using each of the models. The three-dimensional shape of a plurality of workpieces stacked in the work space is measured by a sensor unit arranged at a fixed position capable of imaging the work space, and an end effector provided on the flange portion at the tip of the robot by a robot controller. It is an image processing method that controls a robot that performs a picking operation.
The process of registering the end effector model, which is 3D CAD data that virtually represents the 3D shape of the end effector,
Said end effector model, Ri virtual three-dimensional space der on the display unit, the position and orientation to be displayed on the three-dimensional shape in the coordinate system of the vision space of the captured end effector by the sensor unit, the robot controller end effectors The process of acquiring calibration information for coordinate conversion between the position and orientation of the coordinate system in the robot space that operates
The process of acquiring an input image having a three-dimensional shape from the image including the end effector measured by the sensor unit, and
A step of performing a three-dimensional search for specifying an image area corresponding to the position and orientation of the end effector using the end effector model as a search model from the acquired input images.
The process of detecting the position and orientation of the end effector in the coordinate system of the vision space by the three-dimensional search, and
The process of acquiring the position and orientation of the flange where the end effector is attached in the coordinate system of the robot space, and
The process of converting the position and orientation of the flange portion in the acquired coordinate system of the robot space into the coordinate system of the vision space, and
The difference between the position and orientation of the flange portion in the coordinate system of the converted vision space and the position and orientation of the end effector in the detected coordinate system of the vision space is calculated, and the calculated difference is used as the end effector. A step of calibrating the end effector model so that it is attached in a position and orientation slightly deviated from the flange portion by reflecting it in the position and orientation of the model.
Image processing method including. The three-dimensional shape of a plurality of workpieces stacked in the work space is measured by a sensor unit arranged at a fixed position capable of imaging the work space, and an end effector provided on the flange portion at the tip of the robot by a robot controller. It is an image processing program for controlling a robot that performs a picking operation.
A function to register an end effector model, which is 3D CAD data that virtually represents the 3D shape of the end effector,
Said end effector model, Ri virtual three-dimensional space der on the display unit, the position and orientation to be displayed on the three-dimensional shape in the coordinate system of the vision space of the captured end effector by the sensor unit, the robot controller end effectors A function to acquire calibration information for coordinate conversion between the position and orientation of the coordinate system in the robot space that operates
A function to acquire an input image having a three-dimensional shape from an image including an end effector measured by the sensor unit, and
A function of performing a three-dimensional search for specifying an image area corresponding to the position and orientation of the end effector using the end effector model as a search model from the acquired input images.
A function to detect the position and orientation of the end effector in the coordinate system of the vision space by the three-dimensional search, and
A function to acquire the position and orientation of the flange where the end effector is attached in the coordinate system of the robot space, and
The function of converting the position and orientation of the flange portion in the acquired coordinate system of the robot space into the coordinate system of the vision space, and
The difference between the position and orientation of the flange portion in the coordinate system of the converted vision space and the position and orientation of the end effector in the detected coordinate system of the vision space is calculated, and the calculated difference is used as the end effector. A function to calibrate the end effector model so that it is mounted in a position and posture slightly deviated from the flange portion by reflecting it in the position and posture of the model.
An image processing program to realize the above on a computer. Computer readable recording medium having recorded thereon a program according to claim 1 1.

Images