JP2834597B2 - Control method and control device for automatic gripping device using visual sense - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a control method and a control device for recognizing a gripping position of a workpiece to be gripped from an image of the workpiece to be gripped, and controlling an automatic gripping device for gripping the workpiece.

[0002]

2. Description of the Related Art A troublesome operation in controlling an automatic assembling apparatus is teaching of position coordinates of a work or the like. Since the robot can move the hand or finger only to the position where it was taught, if the teaching position is incorrect or incorrect, the assembly operation will fail or the hand or finger will be destroyed I do.

Therefore, recently, visual sensors (for example, CC
(D camera) has been proposed. As an example of such an automatic assembling apparatus, see, for example,
JP-A-56884, JP-A-60-123974 and JP-A-62-140724. Further, as a technique proposed by the present applicant, Japanese Patent Application No. 1-71895, 2-
No. 55032. In these methods, an image of a work is captured by a visual sensor, the work position is converted from the image into a robot coordinate system by image processing, and an assembling operation is performed based on the work position in the robot coordinate system.

An automatic assembling apparatus having such a conventional visual sensor has a robot control device for controlling the operation of a robot hand and the like, and an image processing device for inputting and processing an image of a CCD camera. , For example, RS2
It is common to prepare an interface and a communication program that are electrically connected by a cable such as 32C and enable communication to both control devices.

[0005] A series of robot operation programs are input in advance to the robot controller. This program uses instructions for sending and receiving information to and from the image processing apparatus through the interface and the cable. Similarly, in the image processing apparatus, a series of programs for performing recognition processing from an image of a workpiece from an image input of the work, and information of information through the communication means between the robot control apparatus and the series of programs are included in the series of programs. It is configured to prepare an instruction for giving and receiving.

[0006]

However, in the above conventional example, as described above, an image processing device and a robot control device, which can operate independently, are prepared.
The following disadvantages have been encountered in connecting each of them by means of communication means so as to perform a series of assembling operations. : It is necessary to separately prepare an interpreting means such as a computer for interpreting a command for performing a series of operations or processes in the image processing device and the robot control device, and separately prepare those programs. Become. : When the CPU is separately prepared as in the above, a series of assembling operations are performed in about 2 seconds, but the time required for the image processing is about 0.3 seconds. CPU will not do any work most of the time.

[0007] The above two control devices are used in a very common R
If the connection is made in S232C, communication takes time, and the assembling time is extended by communication that does not generate any value. The fact that a CPU for interpreting two instructions is required means that two operation instruction interpreting programs must be prepared, and it takes time to maintain the programs. : It is necessary to prepare input / output means such as a keyboard and a CRT for each of the two control devices, and it is necessary to prepare a program and an interface for supporting each of them, resulting in an increase in cost. The CPU on the image processing apparatus side must always perform an operation of waiting for a processing start command from the robot control apparatus.

[0008] All of these problems are due to the separate execution of control programs having different properties of image processing and gripping.

The present invention has been proposed in view of the above-mentioned problems of the prior art, and an object of the present invention is to interpret a processing instruction to an image processing unit and a gripping operation instruction in a robot, a hand, and the like, and to execute these image processing. The present invention provides a control method and a control device for controlling an automatic gripping device that unifies the interpretation of an image processing program and a gripping operation program by using a single command interpreting unit that issues a command to a unit and a gripping unit. is there.

[0010]

Means for Solving the Problems According to the present invention for achieving the above object, an image of a workpiece is taken, and the image is taken from the image.
An image processing unit for calculating position information of the work;
A gripper for gripping the workpiece based on the position information;
According to the position information of the workpiece,
Position, and attach the gripper holding the workpiece to the assembly
The moving means for moving to the stage
Controlling an automatic gripping device having assembly means for assembling the body.
The control device captures the work and determines whether the image of the work is
A series of image processing instructions for calculating position information from
A series of gripping operation commands for gripping the robot based on the position information
And operation control describing operations of the moving means and the assembling means.
A storage unit for storing a program including the instruction and the professional
The image processing command, the gripping command and the motion control in the program
And identifying and interpreting the image processing instruction.
Intermediate commands corresponding to the gripping command and the motion control command, respectively
Code, and convert the converted instruction codes to the respective images.
A processing unit and one interpreting unit for passing to the gripping unit.
It is characterized by having. Further, according to the second aspect of the present invention,
An image of the work is taken, and the position of the work is
An image processing unit for calculating information;
Automatic gripping device consisting of a gripper that grips based on information
The control device controls the imaging of the work, and
A series of image processing instructions for calculating position information from images and
A series of gripping movements for gripping the workpiece based on the position information
A storage unit for storing a program that includes a work instruction, this preference
Identify the image processing command and the gripping command in the program
The image processing instruction and the gripping operation instruction
Are converted to the corresponding intermediate instruction codes, and the converted
To pass command codes to the image processing unit and the gripping unit, respectively
And one interpretation unit, wherein the interpretation unit is one
Interpret image processing instructions or gripping operation instructions and issue instruction codes
After the conversion to the following image processing command or grip operation
It is characterized by interpreting instructions. Further, according to claim 3
Image of the work according to the present invention, and
An image processing unit for calculating position information of the work;
And a gripper for gripping the robot based on the position information.
A control device for controlling the moving gripping device captures an image of the work,
A series of image processing for calculating position information from the image of the work
And grasp the workpiece based on the position information
Storage for storing a program including a series of gripping operation instructions
Unit, the image processing instruction in this program and the gripping operation
And identifying and interpreting the image processing instruction.
Converts gripping operation instructions to corresponding intermediate instruction codes
And converts the converted instruction codes into the image processing unit and the
One interpretation unit for passing the image data to the holding unit.
The processing unit interprets the instruction code converted from the image processing instruction.
First executing means for executing the release, wherein
Interprets and executes the instruction code converted from the holding operation instruction.
2 is provided.

According to a seventh aspect of the present invention, there is provided the above-mentioned object.
According to the present invention, an image of a work is taken, and
An image processing unit for calculating position information of the workpiece;
Automatic comprising a gripper for gripping based on the position information
The control method for controlling the gripping device includes capturing an image of the workpiece,
Series of image processing to calculate position information from images of different workpieces
An instruction and a method for gripping the workpiece based on the position information
A program including a series of gripping operation instructions,
An interpretation step of identifying and interpreting the command and the gripping operation command;
The image processing instruction and the gripping operation instruction are
A conversion step of converting between instruction code converted instruction code
Passing the code to the image processing unit and the gripping unit, respectively.
And Bei, of one image processing instructions or gripping operation instruction interpreting
After being converted into an instruction code, in the interpretation step,
The next image processing instruction or gripping operation instruction is interpreted
It is characterized by. Further, according to claim 8 of the present invention,
Image of the workpiece, and from this image the position information of the workpiece is obtained.
An image processing unit for calculating, and the work based on the position information.
Controlling the automatic gripping device consisting of
The control method is to image the work and determine whether the image of the work is
A series of image processing instructions for calculating position information from
A series of gripping operation commands for gripping the robot based on the position information
And a program including the image processing instruction and the gripping operation command.
An interpreting step of identifying and interpreting the image processing instruction , and the image processing instruction
And the gripping operation command into corresponding intermediate command codes.
And converting the converted instruction code to the image
Comprising the step of passing the image processing unit and the holding unit, the image processing
In the control unit, the instruction converted from the image processing instruction
Interpret and execute the code, and in the gripper,
Interpret and execute instruction codes converted from operation instructions
It is characterized by.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In this embodiment, the present invention is applied to a control device of a so-called shuttle type automatic assembly device. <Overall Configuration> FIG. 1 is a perspective view of a system including a robot assembling apparatus and a control apparatus according to this embodiment. This system is roughly composed of component supply devices 11 and 12, a shuttle type robot 6 for performing assembly, and an automatic assembly control device 1
7

Reference numeral 9 denotes a shuttle on which the robot 6 is mounted. The shuttle 9 can be moved in one direction on the guides 16a and 16b by a shuttle moving motor 6e. Reference numeral 15 denotes a shuttle base, which is fixed to the floor in this embodiment, and has guide rails 16a and 16b mounted on the upper surface thereof in parallel with bolts or the like.

The shuttle 9 is controlled by the control device 17 so as to stop moving to the position taught in advance.
Further, the shuttle base 15 has a component supply device 11 for supplying articles to be assembled and stored in a pallet or the like.
12 are attached. In the system of the present embodiment,
Two first component supply devices 11 and second component supply devices 12 capable of supplying two different components are provided. Each supply device is capable of storing a plurality of pallets 13a and 13b, and each of the supply control units 14 and 21 replaces the empty pallet with a new pallet containing the components when the components in the pallet become empty. Supply device 1
1 and 12 are controlled.

Although the mounting robot 6 is fixed on the shuttle 9, the robot 6 is moved along the guide rails 16a and 16b to the positions of the pallets 13a and 13b so that the components can be removed from the component supply devices 11 and 12. Moving.

The robot according to the present embodiment has a so-called horizontal articulated four-degree-of-freedom, and two horizontal arms are driven by motors 6a and 6b, and the tip of the arm is moved up and down by a motor 6c. The arm tip is rotated by 6d. A hand 7 is attached to the wrist of the robot 6, and three motors 22a, 22a are provided.
The finger is driven by b and 22c, and it is possible to grip the component. Note that 6a, 6b, 6c, 6
The motors d, 6e, 22a, 22b, and 22c operate according to the operation stored in advance by the control device 17.

The position of the robot 6 in FIG. 1 is at a position where a component on the pallet 13a on the component supply device 11 can be grasped. The detachable part 7a of the hand is
Is released by releasing a clamper (not shown) in response to the command (1). The hand holding base 10 is for holding a plurality of hands. The robot 6 can automatically attach an arbitrary hand on the holding table 10 according to a command from the control device 17, and can also remove the hand currently attached to the wrist from the hand holding table 10 and put it on the hand holding table 10. It is. The assembling stage 8 is a place for assembling parts held by the robot 6, and can be provided with a guide and an auxiliary function as needed. When the operation of the auxiliary function is required, a drive source is placed on the assembly stage 8 and can be controlled by connecting the drive source to the controller 17 with a cable. Also, since the mounting stage 8 rides on the shuttle 9 similarly to the robot 6, after the robot 6 grabs a part from the pallet and rises slightly, even if the shuttle 9 starts moving, It is possible to continue the assembling operation.

Reference numeral 5 denotes a visual pillar, the root of which is shuttle 9
Fixed on top. An XY table 3 is provided at the tip. The lens 1 is attached to a fixed portion of the XY table 3. The CCD camera 2 can be moved by stepping motors 4a and 4b in the X and Y directions. That is, although the lens 1 has a view of the entire area of one pallet, by moving the CCD camera 2,
An arbitrary partial area of the pallet can be captured with high precision as image information. In other words, the entire surface of the pallet 13b formed through the lens 1 is moved to a CCD sufficiently smaller than the imaging area to capture image information only at the place where the pallet 13b is to be input.

Since the visual column 5 is mounted on the shuttle 9, the visual column 5 is moved together with the shuttle to a supply device containing components to be assembled. An image of a pallet containing components to be picked can be obtained by moving the CCD camera 2 (that is, the shuttle 9) to a position where the image can be input, and the image is processed. Accordingly, the image-processed component can be gripped by the robot.

An automatic assembling control device 17 for controlling the entire assembling device is connected to a robot 6, a robot 6, a shuttle 9,
It is possible to control actuators such as the Y table 3 and the hand 7 and to input sensors (not shown). Reference numeral 19 denotes a CRT, and reference numeral 20 denotes a keyboard which enables input / output.

<Features of the embodiment apparatus> The functions specific to the embodiment system are mainly realized by a control unit 17 described later. Such functions are as follows: In order to identify and recognize a component to be assembled (a gripping target) in the pallet, an image of the component in the pallet is taken in and pattern matching is performed. With this recognition, no matter how the parts are stored in the pallet,
Recognition of parts becomes possible. When the presence of a target component can be recognized in the pallet by pattern matching, the position of the component is recognized in a visual coordinate system and further converted to a robot coordinate system. The robot can grip a target component at a gripping position in a robot-based coordinate system.

An image processing program that describes processing for recognizing a target component and an assembly program that describes control for grasping and moving the recognized component are stored in the RAM 52 in the instruction interpreting unit 87 described below as intermediate codes. It is converted and stored. These two programs are executed by multitask control. : The image processing program (task) is activated by a special task activation instruction included in the imposition program. The activated image processing program is processed in parallel with the imposition program.

The image processing program and the assembling program, which have been converted into intermediate codes in advance, are interpreted by the same instruction interpreting unit, and various instructions contained therein are converted into instruction codes. The conversion into the instruction code absorbs a difference in the property between the instruction in the image processing program and that of the assembling program, and facilitates unified management of the image processing program and the assembling program. : The instruction code of the image processing program is stored in a dual port RAM described later. The instruction code of the installation program is stored in a shared memory 60 described later.
By using the dual port RAM, it is possible to select a microcomputer of the image processing unit and a microcomputer of the operation control unit from those suitable for each purpose.

Hereinafter, the configuration of the control device 17 and the operation thereof will be described. <Structure of Control Device 17> FIG. The control device 17 drives and controls the four-axis motors (6 a to 6 d) of the robot 6 and the motor 6 e of the shuttle 9, and the three-axis motors (22 a to 22 c) of the hand 7. A hand operation control unit 63 for controlling the driving motors 4a and 4b of the XY table and an image processing unit 64 for controlling the camera 2;
An interface 54 for controlling an interface with the control units 14 and 21 of the two supply devices 11 and 12, and a command interpretation unit 87 for interpreting and executing a command of the robot program.
And a shared memory 60 shared among the robot operation control unit 62, the hand operation control unit 63, the image processing unit 64, and the command interpretation unit 87. These components are connected to a system bus 59. In this embodiment, the bus 59 is a multi-bus.

Command Interpreting Unit 87 The command interpreting unit 87 will be described with reference to FIG. 50 is C
A PU 51 is a ROM. In this ROM 51,
As shown in FIG. 5, a robot language command interpreting program, and a multitasking operating system (hereinafter, referred to as a "task") that interprets visual processing commands and robot and hand operation commands in the command interpreting program as necessary.
“OS”) and various programs supporting them are stored in advance. Details of these programs will be described later.

In the RAM 52 of the command interpreting section 87, an image processing program and an operation program for robot / hand, which are input by the keyboard 20, are stored.

Each element in the instruction interpreting section 87 is connected by a local bus 86. That is, the local bus 86 includes the ROM 51, the RAM 52, and the interface IF 53.
Are combined. The data in the RAM 52 is CRT1
9 and can be input by the keyboard 20 or deleted. In addition, a command for an actual operation of the present system is issued from the keyboard 20.

Reference numerals 62 and 63 denote operation controllers for the robot and the hand. When the instruction codes corresponding to the robot operation and the hand operation are written in the shared memory 60, these control units 62 and 63 perform the operation processing according to the instruction codes, and perform each device, that is, each motor 6a to 6e.
Rotate and stop. As described above, the instruction code is written into the shared memory 60 by the instruction interpreting unit 87.

The image processing section 64 performs image processing for recognizing components, and performs processing in accordance with the instruction code written in the IF 61. Reference numeral 69 denotes a known magnetic external storage means, for example, a floppy disk drive or a hard disk drive, which can store a robot hand operation command image processing command through the IF 53 by operating the keyboard 20. It is possible.

Further, by bringing the floppy disk stored by an external computer or the like or the floppy disk already stored, the processing command and the operation command are stored in the RAM 52 in the specified areas. It is also possible.

The control units 14 and 21 of the component supply device have optical communication circuits 56a and 56b, and can exchange information with the optical communication circuits 55a and 55b on the shuttle 9. Each of these optical communication circuits has a light emitting unit and a light receiving unit. Light emission and light reception are opposed to each other, and communication becomes possible when the robot 6 stops at a predetermined supply device position.

The interface IF 57 is an interface for communicating with an external computer or the like.
The information in the assembling apparatus is sent to the outside or the information is input from the outside. For example, by communicating with a computer that manages an unmanned warehouse, it is possible to calculate the timing of automatic delivery of pallets. Reference numeral 59 denotes a robot operation unit 63 of the CPUs 50 and 62, and an IF 61 of the hand operation unit 61.
And a common bus connecting the shared memory 60 and a multi-bus in this embodiment.

The instruction code interpreted and executed by the CPU 50 of the interpreting section 87 is set in the shared memory 60.
These instruction codes are received by the robot operation control unit 62 or the hand operation control unit 63. When the control units 62 and 63 perform the actual operation and end the operation, the shared memory 60
The control unit writes an operation end signal upward.

When the instruction interpreted and executed by the CPU 50 is an instruction related to image processing, the instruction code is stored in a two-port RAM (2-PORT-RA) provided in the IF 61.
M), when the image processing is completed and the communication of the processing result is necessary, similarly to the above-described assembling operation instruction, when the processing result and the processing end signal are sent to the two-port RAM in the IF 61, Write to.

Image Processing Unit 64 FIG. 3 is a detailed block diagram of the image processing unit 64. A CPU 65 controls the entire image processing unit. In the two-port RAM 61, the bus 59 of FIG. 2 and the bus 72 of FIG. 3 are connected from both directions so that data can be exchanged with other elements of the control device 17, and reading and writing can be performed from both directions. It is.

Two buses are provided between the buses of the CPU 50 and the CPU 65.
By providing the port RAM 61, the CPU 50 and the C
It is possible to make the type of CPU of the PU 65 different. It should be noted that there is no problem even if the formats of the CPU 65 and the CPU 50 are made equal, as was possible with the shared memory 60 of FIG. The transfer of data between the robot operation control unit 62, the hand operation control unit 63 and the command interpretation unit 87 in FIG.
There is no problem even if the communication is performed by providing as an interface.

In the image processing section 64, reference numeral 2 denotes a camera for taking in an image. In this embodiment, the lens is fixed to an XY table. XY table 3 is C
By moving the CD within the imaging plane, it is possible to capture image information of a place where a pallet is required. The image signal taken by the camera 2 is digitized at the brightness level by the A / D converter 73 and stored in the frame memory 75.

FIG. 4 shows the appearance of a pallet used for storing components in the component supply device. As shown in the figure, components are stored in divided cells in the pallet. The part is specified by the cell position information of the pallet. Since the present assembling apparatus recognizes the exact position of the part by image processing, it is not necessary to fix the part accurately in the pallet in advance. Also, the cells of the pallet need not have the exact dimensions. It suffices to know the approximate position of the cell containing the required component. That is, if the cell position of the required component is specified in the robot program, image processing is performed on an image around the cell position, and the exact position of the required component is found in the cell by pattern recognition.

If the field of view of the CCD camera 2 is set to be large enough to cover the entire pallet, the CCD element may be made larger by accepting an increase in cost, or the CCD element may be made smaller by accepting deterioration in resolution. , So that the entire pallet can be seen. Therefore, in the present embodiment, the camera 2 is made to be movable, and only a necessary portion of the pallet image is taken out. Therefore,
The camera 2 is provided with an XY table driving unit 76. The driving unit 76 drives the two stepping motors 4a and 4b of the XY table to rotate, and moves the camera to a predetermined position. The area setting unit 74 has a function of limiting an area when it is not necessary to capture image information of the entire area of the CCD when capturing an image with the camera 2. In the assembling machine, the shape and size of the assembled parts are known in advance, so when it is desired to capture only the image information of one part, a processing instruction for limiting the area is programmed in the image processing instruction. deep.

The binarizing section 66 converts the image information stored in the frame memory 75 into, for example, "1" for a bright one and "0" for a dark one based on a brightness level specified in advance by a processing command. "To binarize. The binary image is stored in the frame memory 75 again.

The labeling section 67 performs labeling.
In this labeling process, for example, a group of connected dots, that is, a group of “1” clusters is found in the binarized image information, and a different label is attached to each group.

The feature parameter calculator 68 calculates, for each of the labeled connection regions, for example, the center-of-gravity position area and the principal axis of inertia. By comparing the feature parameter of the image of a certain work calculated by the calculation unit 68 with the reference feature parameter stored in the two-port RAM 61, it is determined whether or not the image is an image of a specified component. Is determined. If it is determined that the part of the image is the specified part, the position and inclination of the part in the visual coordinate system are obtained, and the two-port RA
Using the conversion parameters for the robot coordinate system stored in advance in M61, the robot is converted to the robot coordinate system,
The position data is stored on the two-port RAM 61 as position data. If the position data is required during the robot operation instruction, the position data is read, and the operation instruction that operates according to the result is executed, thereby grasping the components in the pallet with the position information calculated by the image processing unit, and assembling. Can be performed. The feature parameters and coordinate conversion parameters stored in the two-port RAM 61 are stored in the battery backing up area of the RAM 52 in FIG. 2 and automatically turned on when the control device is turned on. Is also stored in the two-port RAM 61.

In FIG. 3, a ROM 71 previously stores a program for performing a series of processing of an image. <Program Structure> FIG. 5 shows a storage state of a program stored in the local memory ROM 51 of the instruction interpreting section 87 described with reference to FIG.

The ROM 51 includes a boot program 100 that is executed before power-on, a multitask OS 101 that manages and executes a plurality of tasks, a text editor 102 for creating a program, and a created text. A compiler 103 for converting a program into an intermediate code that can be interpreted, and an intermediate code converted by the compiler 103 are interpreted, and lower-level control units (robot operation control unit 62, hand operation control unit 63, image processing unit 64) Interpretation program 104 for generating instruction codes (FIGS. 13 to 15) for operation, an IF driver program 105 for controlling input / output of input / output devices such as a keyboard and a CRT, an operation state of a robot, and a prepared assembly Robot task display for displaying the status of robot tasks such as operation programs A program 106, the image processing unit 64
And a vision task display program 107 for displaying the status of the vision task such as the created vision operation program.

FIG. 6 is a diagram showing a data storage state in the local memory RAM 52 of the instruction interpreting section 87.
The RAM 52 includes an assembling operation program 108 (FIGS. 9 and 10) composed of a series of operation instructions for the robot assembling operation and a series of processing operation instructions for the processing operation of the image processing unit. Visual Operation Program 109
(FIG. 11) and target position data 110 of the XY stage
And target position data 111 of the shuttle, target position data 112 for positioning the robot, reference characteristic parameters 113 indicating the characteristics of the work to be assembled, and coordinate conversion parameters 114. The parameter 114 is an operation parameter used for a coordinate conversion operation from the robot coordinate system to the XY stage coordinate system, and a similar operation parameter from the XY stage coordinate system to the robot coordinate system. The RAM 52 is volatile, but the data stored therein is important, so that it is backed up as described later.

<Robot Program> FIGS. 7 and 8 are diagrams showing the steps of an assembling operation for assembling a work by receiving two components from the component supply units 11 and 12 in the assembly system of FIG. That is, FIG. 7 shows that the robot 6
A stored in the pallet 13a of the component supply unit 11
Shows a state in which the shuttle 9 is stopped at a position at which the shuttle 9 can be accessed. FIG. 8 shows a part B (part B is a part supply unit 12) which is held by the robot 6 following the assembly of the part A.
The shuttle 9 is stopped at a position where the shuttle 9 can be accessed.

As can be seen from FIG. 7, especially FIG. 8, while the robot 6 is performing the operation of assembling the part A, the camera 2 is mounted on the pallet 13 in which the work B is stored.
b, the assembling operation of the part A and the part B
Can be performed in parallel with the imaging operation. Due to the parallelism of the assembling operation processing and the imaging operation (image processing), FIG.
As shown in the above, it is significant to make the assembly operation program and the vision operation program into tasks.

FIG. 9 shows a series of robot operation instructions (program) for assembling the part A, and FIG. 10 shows a series of robot operation instructions (program) for assembling the part B.
FIG. 11 shows a series of image processing instructions (programs) for calculating position information by performing image processing on the image of the component B.

9 to 11 do not show a program for performing image processing on the image of the component A to calculate position information. In addition, to complete the work,
For example, if the part C is necessary, a series of image processing instructions (programs) for image-processing the image of the part C to calculate position information and a series of robot operation instructions (parts) for assembling the part C ( 9 to 1
It becomes necessary in addition to one program. Also, for convenience,
The program in FIG. 9 is referred to as program 1, the program in FIG. 10 is referred to as program 2, and the program in FIG.

The programs shown in FIGS. 9 to 11 are shown in FIGS.
Is described. In order to assemble the part A and then assemble the part B, first, the program 1
(FIG. 9) must be activated beforehand. The program 1 is being executed, and the program 1 itself starts the program 2. The activation of the program 2 by the program 1 is performed by calling CALL 2 at the line number 390 and calling the program 2. To start the assembling operation of the part C after the completion of the assembly of the part B, the CALL 3 of the line number 250 is executed, and the program 3 (not shown) is called. During the assembling operation of the part A, when the acquisition of the image of the part B and the image processing thereof become possible, the line number 3
At 60, START 3,2 is executed to call program 3 (FIG. 11) under multitasking control. In the START command, the first operand is a task number for an image, and the second operand is a program number of the image. That is, the program 1 and the program 3 operate in parallel. During the assembling operation of the part B, when the acquisition of the image of the part C and the image processing thereof become possible, at line 220, START 3,3 is executed, and the robot program 2 and the image program 3 are connected. Operate in parallel. In the START instruction which is a parallel processing start instruction, the first operand is a task number (3 is given to an image), and the next operand is a program number. Although the third procedure (program) of the program 3 is not shown, since the part C does not have a special shape,
That is, since special image processing according to the shape is not required, the same one as the program in FIG. 11 can be used.

The meaning of the instruction of each statement in the programs of FIGS. 9 to 11 will be described later. <Parallel Processing> As shown in FIGS. 7 and 8, in the robot system of this embodiment, the assembling operation and the image processing can be performed in parallel. Also, as shown in FIGS. 9 to 11, the programs describing these operations are also conscious of parallel processing. Therefore, in order to realize this parallel processing, the characteristic control of the control device 17 will be described below. As shown in FIG. 12, these characteristics are as follows: The so-called robot program (the program in FIGS. 9 to 11) is interpreted by the instruction interpreting program 104 (FIG. 5) of the instruction interpreting section 87. The program 104 determines whether an instruction (statement) in the program relates to an assembling operation (the robot operation control unit 62 or the hand operation control unit 63) or an image processing (to the image processing unit 64). Judgment is made, and corresponding instruction codes are respectively generated.

The instruction code generated from the instruction relating to the assembling operation is written in the shared memory 60, and the instruction code generated from the instruction relating to the image processing is a two-port instruction code.
It is written into the RAM 61. : Robot operation control unit 62 or hand operation control unit 6
3 executes the instruction code written in the shared memory 60. The CPU 65 of the image processing unit 64 executes the instruction code written in the two-port RAM 61.
FIG. 13 to FIG. 15 are diagrams illustrating an example of a memory storage format of the instruction code.

Instruction Code FIG. 13 is a diagram showing a general storage format of instruction codes. The storage format in FIG. 13 includes a “next instruction storage address pointer” 200 indicating a memory address where the next instruction is stored;
A “P code” 201 indicating the type of instruction and a parameter type (for example, whether it is a direct number, a variable, a real number, or an integer) for supplementing the instruction are shown. A data type 202A, a data portion 203A of the parameter, a delimiter 204A indicating the end of the data, and a checksum 205 indicating the end of the instruction code and detecting an error in the contents are stored in this order.

Some instructions do not require parameters in the instruction code. As shown in FIG. 14, the instruction code of such an instruction has a P-code 201 followed by a checksum 205 indicating the end of the instruction.

In the case of an instruction requiring a plurality of sets of parameters in the instruction code, as shown in FIG. 15, a data type 202, data 203,
The checksum 205 is stored after the required number of sets of the delimiter 204 are stored.

As described above, the instruction statement is restructured into an instruction code (intermediate code) including the instruction type (P code 201) and the parameters, and is stored in the memory, so that the assembly operation program 108 and the vision The operation program 109 can be stored in the same format.
In addition, the fact that both instructions of the assembling operation program 108 and the vision operation program 109 are converted into instruction codes of the same format means that a memory for storing them can be shared. That is, specifically, the instruction code for the image processing unit can be stored in the shared memory 60 instead of the two-port RAM 61 which is a memory for storing the instruction code for the image processing unit 64. It means there is. In this embodiment, the reason why the two-port RAM 61 is used in addition to the shared memory 60 is to avoid contention for access to the memory due to the unification of the memory, and to use two different directions in the two-port RAM 61. , High-speed processing is possible by simultaneous access from, and the assembling operation and image processing can be easily parallelized.
4 is that a CPU having an architecture different from that of the CPU of the instruction interpretation unit 87 can be used.

<Control for Parallel Processing> FIG. 16 is a diagram for explaining the structure between programs for controlling parallel processing in the present system. Assembly operation program 108
Is called a "robot task".
Starting interpretation of the assembly operation program 108 is referred to as "starting a robot task". Similarly, a series of procedures for interpreting the vision operation program 109 is called a “vision task”, and starting interpretation of the vision operation program 109 is called “starting a vision task”.
Activating the vision task is performed by the aforementioned START command. In the present assembling system, since the image processing is assisted during the assembling process, a vision task is started during the assembling program.
A TART statement will be described. This facilitates conflict arbitration between the robot task and the vision task. And, furthermore, the activation of the robot task
This is performed when a start key is input from the keyboard 20 of the control device 17.

The robot command interpreted by the robot task is converted into a command code (FIGS. 13 to 15) and stored in the shared memory 60 as described above. Similarly, the image processing instruction interpreted in the vision task is converted into an instruction code in the same manner, and a two-port RA
It is stored in M61. The instruction code of the shared memory 60 is
The robot operation control unit 62 and the hand operation control unit 63 perform the operation after being fetched. Robot operation control unit 62,
Although not shown, the hand operation control unit 63 includes a dedicated microprocessor, a memory for storing the programs shown in FIGS. 18 and 19, a servo control circuit, a servo amplifier, and the like.

As shown in FIG. 16, three flags (MWF, TWF, PWF) are prepared in the instruction interpretation program. That is, the “operation completion waiting flag” (MWF)
Indicates that among the instructions of the robot task, an instruction that needs to wait for the execution end of the instruction is currently being executed by the robot task. The operation completion waiting flag MWF is set by executing an instruction such as, for example, MOVP (4). These instructions are
This is a command that requires an operation such as actual movement of the robot, and the execution of the next command requires waiting for the completion of the command. As for the control commands such as IF... THEN..., The operation completion wait flag MWF is not set because the actual movement of the robot is not accompanied.

"Process end wait flag" (PWF)
Indicates that among the instructions of the vision task, an instruction that needs to wait for the execution end of the instruction is currently being executed by the vision task. These MWFs and PWFs are provided to prevent two or more instructions from being executed in an overlapping manner in a robot task or a vision task.

The "timing wait flag" TWF was introduced into the present system for the following reasons. That is, under the multitask control, the instructions in the robot task and the instructions in the vision task are executed in parallel. In order to start one instruction of the robot task, it may be necessary to confirm the end of one instruction of the vision task, that is, to arbitrate between the two tasks. For example, when capturing an image of an object with the CCD camera 2,
This is because it is necessary to prevent a situation in which an image cannot be captured due to the arm of the robot 6 entering the field of view of the camera 2 or the movement of the shuttle 9. The above MWF and PWF are:
To the last, arbitration of instruction execution between the same tasks is performed. Therefore, a "timing wait flag" T is used as a flag for one task to temporarily suspend the other task.
There is WF.

In the system of this embodiment, a WAIT instruction is prepared as an instruction for one task to temporarily suspend the other task. This WAIT instruction is mainly used in the vision task, and when it is executed, the stop flag S
F (suspended flag) is set. The flag SF is provided in the two-port RAM 61, and the WAIT instruction is automatically added by the system (the compiler 103 in FIG. 5) without explicitly describing those instructions in the program. . That is, in the example of FIG.
The compiler 103 automatically adds a WAIT instruction after an imaging instruction (GET) for imaging with the D camera 2 or after a pattern recognition instruction (MATCH). In other words, when the GET instruction or the MATCH instruction is executed, the WAIT instruction is automatically executed, and the flag SF is set. The reset of the flag SF is performed by the special instruction (G
ET command or MATCH command). Therefore, on the robot task side, a command for checking the set state of the flag SF, that is, a timing confirmation command (for example,
CHECKF instruction) is inserted into the robot task to coordinate the robot task with the vision task.

As shown in FIG. 16, the instruction interpretation program has two counters (RPC and IPC). The counter RPC stores the line number of the next execution instruction in the assembly operation program 108 of the robot task. For example, if the instruction to be executed next by the robot task is SSHUTL at line 300 (FIG. 9), the counter R
300 is stored in the PC. Also, the counter IP
C stores the line number of the next execution instruction in the vision operation program 109 of the vision task. For example, if the instruction to be executed next by the vision task is the BIN of line number 20 (FIG. 11), 20 is stored in the counter RPC.

The parallel execution control of the robot task and the vision task will be described with reference to FIGS. In the control step S200 of the robot task, when the start of the assembling operation is instructed from the keyboard 20, the command interpreting section 87 executes the step S201.
By starting the robot task, the interpretation of the assembling operation program 108 in the RAM 52 is started (step S201).

In step 202, it is checked whether the robot task has been started. The reason for checking whether or not the robot task is activated in step S202 is that a statement for stopping the robot task can be described in the robot program itself or in the vision task program itself. . Therefore, when the robot task is stopped, only the vision task is executed, and in order to restart the robot task programmatically, it is necessary to execute an instruction to start the robot task in the vision task. 9 to 11
Since the robot task is not stopped programmatically when the program of the example of FIG.
The process proceeds to step S03, where the determination of the operation completion waiting state is performed based on the operation completion waiting flag MWF. As described above, M
Since the WF is set for a move instruction or the like, if such an instruction has not been executed in the previous cycle, or if an unnecessary instruction for setting the MWF has been executed, the MWF is set. = 0, so step S
Proceed to 204.

If MWF = 0, the flow advances to step S204 to read one instruction from the assembly operation program 108 and interpret the instruction. When one instruction is interpreted, the counter RPC is changed to indicate the next line number.

In step S205, the content of the interpreted command is stored in the vision task start command ("START").
3 ") is determined. If the instruction is a vision task activation instruction, the vision task activation processing is performed in step S220, and then the process proceeds to step S221.
In the case other than the vision task start instruction, step S20
6 and the timing confirmation instruction (the above-described CHECKF
Etc.) is determined.

If it is determined in step S206 that the content of the instruction is not a timing confirmation instruction, the robot operation unit 6
2. An instruction that needs to issue an instruction code to the hand operation unit 63 and the image processing unit 64 (for example, a command to move the shuttle 9 or an operation instruction to the hand 7) or an instruction that does not need to issue an instruction code (for example, IF .. (The control command such as THEN) is determined in step S207.

If it is necessary to issue an instruction code, the instruction code is determined in step S208 according to whether the instruction is an instruction relating to a robot operation or an instruction relating to a hand operation. Are stored in the shared memory 60 so that they can be executed separately by the robot operation control unit or the hand operation control unit. The robot operation control unit or the hand operation control unit is configured as shown in FIG.
According to the control procedure, the corresponding instruction code is retrieved from the shared memory 60 and executed. Then, step S2
At 09, the operation completion wait flag MWF is set. Then, the process proceeds to step S221.

An instruction that needs to issue an instruction code to the operation control units 62 and 63 takes time until the operation corresponding to the instruction is completed, such as the movement of the shuttle 9 (MOV) or the movement of the robot 6. Things. The flag MWF is set in step S209 in order to prevent a movement command or the like from being issued continuously. However, instructions that do not need to issue an instruction code (for example, control instructions such as IF... THEN...) Should be executed continuously. If it is determined in step S207 that such an issuance unnecessary instruction is to be executed, the process proceeds from step S207 to step S208, and the instruction is directly executed without setting the MWF. Then, the process proceeds to step S221.

If it is determined in step S206 that the content of the command is a timing confirmation command, step S21
0, the flag SF in the two-port RAM 61
Check the set status of. As described above, this flag SF is set by a special task (GET or MATCH) which is another task (a vision task in this embodiment), and is set by the end of the command. Resets. Further, since the flag SF is placed in the two-port RAM 61, not only the image processing unit 64 but also the command interpreting unit 87 can be accessed. Therefore, if it is determined in step S210 that the flag SF is set, the interpretation / execution of the command in the robot task must be interrupted, and the flag TWF is set in step S212 to store the fact. Set with Then, in step S209, the flag MWF is set. That is, the flag MWF is set both when the interpreting unit 87 interprets an instruction that needs to issue an instruction code and when the TWF is set by interpreting the timing confirmation instruction.

In step S221, it is determined whether or not the vision task has been activated. If the vision task has not been activated, the process returns to step S202. When a certain robot program is being executed, the flag MWF is not set as long as an instruction that does not need to issue an instruction code is executed. Therefore, the process proceeds from step S202 to step S203 to step S204, and the next instruction is executed. Interpret and execute. That is, the execution of the instructions of the robot program is performed continuously without waiting.

On the other hand, the robot operation command in the line numbers 300 to 350 of the program in FIG. 9 sets the flag MWF. In such a case, control is passed to step S202
→ The process proceeds to step S203, and since the flag MWF has already been set, the process proceeds to step S214. In step S214, the setting state of the flag TWF is checked in order to check whether the setting of the flag MWF is due to execution of an instruction requiring an instruction code or execution of a timing confirmation instruction.

If the TWF flag has been reset in step S214, the flag MWF was set by execution of an instruction that requires the issuance of an instruction code, and the flow advances to step S215 to check the completion of the robot operation. . This operation completion check is determined by whether or not the instruction code remains in the shared RAM 60. Shared memory 60
The instruction code is stored in the instruction interpretation program in step S208. The instruction code in the shared memory 60 is cleared by the robot operation control unit 62 and the hand operation control unit 63.

FIG. 18 is a flowchart showing a control procedure in the robot operation control unit 62 and the hand operation control unit 63. Robot operation control unit 62, hand operation control unit 6
3 is the shared memory 6 in step S300.
While scanning inside 0, it is checked whether the instruction code for itself has been stored in the memory 60. The determination as to whether or not the instruction code is directed to the user is made based on the instruction type indicated by the P code (FIGS. 13 to 15). If an instruction code for the user is found, the instruction code is executed in step S301, and in step S302, the end (for example, the end of the arm raising or lowering operation) is waited. When the operation is completed, the instruction code is cleared in step S303.

In the control procedure of the processing of the instruction code in the image processing section 64 in FIG. 19, the scan / execution / clear of the instruction code is the same as that in FIG. Returning to the flowchart of FIG. 17, in step S215, the shared R
If it is confirmed that the instruction code stored in the AM has been cleared, the process proceeds to step S216, where the flag MWF is set.
Reset. Then, the process proceeds to step S221. If the vision task has not been activated, the process proceeds to step S22.
1 → Step S202 → Step S203 → Step S
Proceeding to 204, the next command in the robot program is executed.

If it is determined that TWF = 1 in the process of determining the reason for waiting in step S214, to wait for the flag SF to be reset by the vision task,
In step S217, the flag SF is checked. This flag S
The fact that F is set (SF = 1) means that the user still has to wait, so step S
The process returns to step S202 via 221.

If it is confirmed in step S217 that the flag SF has been reset, the flow advances to step S218.
Resets the flag TWF to indicate that the waiting state has been released. Further, step S212 → step S20
The flag MWF set in the process of step 9 is set in step S21.
Reset with 6.

Parallel execution of robot task and vision task
As described above, the execution of the instruction requiring the issuance of the instruction code, the execution of the unnecessary instruction, and the execution of the timing confirmation instruction are performed. Next, the case where the vision task is activated will be described in more detail.

When the START 3 instruction of the line number 360 is executed in step S205, the vision task is activated in step S220, and then the process proceeds to step S221. In step S221, it is determined based on the flag PWF whether the current state is a waiting state waiting for the end of the processing of any instruction of the previously executed vision task. This flag PWF has the same function as the flag MWF in the robot task. If no image processing command has been issued before, since PWF = 0, step S223 is executed.
And interprets one image processing command according to the counter IPC. Step S223, step S224,
The type of instruction interpreted in step S223 is determined. That is, if the instruction is an END instruction (the statement at line number 50 in FIG. 11), the vision task is stopped in step S225. If the instruction is not the END instruction, it is checked in step S226 whether the instruction needs to issue an instruction code. The instruction code needs to be issued is an instruction requiring some processing in the image processing unit 64, for example, an imaging instruction (GET), a binarization instruction (BIN), a matching instruction (MATCH), or the like. is there. Further, control instructions such as IF... THEN and the above-described WAIT instruction need only access the two-port RAM 61, and processing by the image processing unit 64 is unnecessary, so that instruction code issuance is unnecessary.

When executing an instruction which requires the issuance of an instruction code, in step S227, the two-port RAM is executed.
The instruction code is written in 61. Then, step S2
At 28, the process completion waiting flag PWF is set, and the process returns to step S202.

Prior to interpreting the instructions of the vision task,
If the previously executed instruction has not been processed yet, the process proceeds from step S222 to step S230. In step S230, it is checked whether the previously issued instruction code in the two-port RAM 61 has been cleared by the image processing unit 64. This instruction code is cleared as shown in FIG.
Is performed by the image processing unit 64 in accordance with the control procedure of

If it is determined in step S222 that the instruction code has already been cleared, the flow proceeds to step S231.
Resets the flag PWF. As described above, the parallel processing of the robot task and the vision task is performed.

As described above, when the interpreting section 87 interprets and executes a special instruction (GET, WATCH) of the vision task and stores the instruction code in the RAM 61, the interpretation section 87 follows the flowchart of FIG. Image processing unit 64
Fetches the instruction code from the RAM 61 and executes it in step S312. Then, step S31
At 1, the flag SF is set in the RAM 61. When the execution of such a special instruction is completed, the operation of clearing the instruction code and the operation of resetting the flag SF are performed in step S313.

FIG. 20 shows the specific assembling program shown in FIGS. 9 and 10 and the vision operation program shown in FIG. 11 interpreted by the instruction interpreting program shown in FIG. The timing chart shows how the robot unit and the image processing unit 64 operate when the control units 62 and 63 and the image processing unit 64 execute according to the control procedures of FIGS. 18 and 19. That is, FIG. 20 shows a time chart of a part of the assembling operation of the part A, the assembling operation of the part B, and the part of the assembling operation of the part C. In FIG. 20, for the sake of convenience, at the start of this time chart, the robot 6 is connected to the pallet 1 of the component supply unit 11.
It is assumed that the part A is gripped by the hand 7 from 3a, and the robot 6 is retracted above the pallet 13a.

The order of the assembling operation of this embodiment will be described below with reference to FIG. In the periods T ₁ , T ₂ , T ₅ and T ₆ , only the control of the robot operation is performed. That is, in T ₅ period,
The shuttle 9 starts moving to the position for assembling the part B (line 3
00 SSHUTL instruction), and the period T _6, CSET command X-Y table 3 is then assembled movement start the visual object position of the pallet 13b of the part B (row number 310) to.
Then, the robot 6, T in _one period, while holding the part A in the hand 7 moves in the sky point of the assembly stage 8, further descends stop the assembling stage 8 in the vertical direction (row number 3
20, 330 MOV instructions).

In T ₃ , T ₄ , T ₇ , T ₈ , T ₉ T ₁₀ ,
Robot control and image processing control are performed in parallel.
Confirm that the movement of shuttle 9 and the movement of XY stage 3 have been completed (SCHECK, C of line 340, 350)
CHECK command), and the image processing control is started (36).
(The 0th line number START instruction). Thus, the robot task and the vision task are executed in parallel.

In the control of the robot task in the parallel processing, the hand 7 is moved and the part A is released during the period T ₃ (OUT command at line number 370). Then, in the period T _4,
Robot 6 raises and stops hand 7 (MO at line 380)
V command), and the assembly of the part A is completed.

[0089] The in parallel to the image processing side of the control performed with the control in the robot task, the T ₇ period, image the image processing unit 64 of the CCD camera 2 of the visual target component B
Uptake (GET instruction on line No. 10), the T ₈ period, the image by the image processing unit 64 the binarization processing (instruction BIN row No. 20), further being the binarized at T ₉ period Image feature extraction processing (instruction SOP at line number 30),
In T ₁₀ period is performed the position detection processing of the visual target component B (instruction MATCH line number 40).

When the period T ₄ has expired and the period T ₁₀ has expired, the hand 7 can hold the component B located above the assembly stage 8 in the component supply unit, and the image processing unit 64. Detects the position of the part B on the pallet 13b.

When the hand 7 retreats to the sky above the stage 8 by the MOV instruction at the line number 380, the program 2 (FIG. 10) is called at the line number 390. This program 2 is a part B
Is described from the supply unit 12 for assembling.

At the line number 100 of the program 2, the instruction CHECKF 3,2 is executed. This instruction is an instruction for confirming the end of task 3 (that is, program 2 of the vision task, that is, image processing of part B). When it is confirmed that the position detection processing of the component B has been completed by this instruction, the processing of the line number 110 and thereafter is performed.

[0093] In _{T 11 ~T 16, T 19,} T 20 in FIG. 20, only the robot control. In the robot control, the position detection result is read as the movement target position of the robot 6 at line number 110.

[0094] P (20) = VREAD (2 ) Then, in order to grip the component B, and T _11, robot 6 moves the hand 7 high over point of palette 13b (line No. 12
0), in T _12, descend and stop (line No. 130). Next, the hand 7 T ₁₃ is (OUT instruction on line No. 140) grips the target component B and, at T ₁₄ robot 6 moves stops the hand 7 grips the component B high over point of palette 13b (line number 150).

[0095] Operation of the robot control for component B of the _{T 15, T 16, T 19} , T 20 is, T ₁ for the component A,
Same as T ₂ , T ₅ , T ₆ . T ₁₇ , T ₁₈ , T
Also with respect to the operation of the image processing control for the robot control and the component C are performed in parallel for part B of ₂₁ through T _24, it is similar to the operation of the _{T 3, T 4, T 7} ~T 10.

FIG. 21 shows an instruction interpreting unit 87 and a robot operation control unit 62 in the process of sequentially executing the instructions of line numbers 330 to 390 in FIG. 9 and the instructions of line numbers 100 to 120 in FIG. 4 is a detailed time chart of instruction interpretation execution in the hand operation control unit 63 and the image processing unit 64.

[0097] First, at T _100, interprets the instruction interpreting unit the robot operation instruction line number 330 in FIG. 9, to generate an instruction code of the movement of the robot, for transferring the instruction code to the robot operation unit 62 An instruction code is set in the shared memory 60.

[0098] In T _120, robot operation control unit 62, through the shared memory 60, receive ivy by the instruction code to drive the robot 6 is stopped in the position indicated arm. The robot operation control unit 62 determines that the execution of the instruction code has been completed by clearing the instruction code.
Contact 7 At _T120 , the instruction interpreting unit 87
Are in a waiting state until the execution of the instruction code from the robot operation control unit 62 is completed.

[0099] command interpretation unit 87 in T ₁₀₁ interprets the robot operation instruction at the line number 340 of FIG. 9, to generate an instruction code of the shuttle stop confirmation (Scheck), the robot operation control unit 62, the shared Send instruction code through memory. The robot operation control unit 62 confirms the shuttle stop state based on the instruction code, and notifies the instruction interpretation unit 87 of the end of execution of the instruction code via the shared memory 60. Subsequently, the instruction interpreting unit 87 that has been in the waiting state for resetting the MWF restarts interpretation of the next operation instruction.

T₁₀₂ Indicates that the instruction interpreter 87
Interpret the bot operation command and confirm stop of XY table 3
And generates a command code of
T ₁₀₁ Delivery of instruction code and instruction code
Delivery of the end of the code execution is performed.

[0101] In the T _103, the instruction interpreting unit 87 is line number 360
The robot operation instruction (START) is interpreted, and the interpretation and execution of the image processing instruction group (vision task) shown in FIG. 11 is started. After T ₁₀₃ includes a robot operation instructions, it will interpret the image processing instructions in parallel.

[0102] In T _110, command interpretation unit 87 interprets the image processing instruction line number 10 in FIG. 11, generates an instruction code for capturing an image of the CCD camera to an image processing unit, to-the image processing section 64 • Port • Send the instruction code via the RAM 61.

At T ₁₃₀ , the image processing unit 64 receives the image capture instruction code from the two-port RAM 61.
Captures the image of the CCD camera 2 into the image processing unit, and after completion of the capture, notifies the instruction interpreting unit 87 of the end of the execution of the instruction code via the two-port RAM 61.

[0104] In T _104, command interpretation unit 87, after the excisional completion of interpretation and instruction code of the image processing instruction T _110,
The robot operation command at line number 370 in FIG. 12 is interpreted. T
_{At 104} , an instruction code for moving the hand 7 is generated, and the instruction code is set in the shared memory 60. As the next state, the instruction interpreting unit 87, a state awaiting completion of execution of the instruction code issued to the image processing unit in T _110, and a wait state of completion of execution of the instruction code issued to the robot operation control unit 62 I'm sorry.

At T ₁₁₁ , the end of execution of the instruction code is notified from the image processing unit 64 via the two-port RAM 61. Further, in a state where the execution end is not notified from the robot operation control unit 62, the instruction interpreting unit 87 interprets the image processing instruction of the line number 20 in FIG. 11 and generates an instruction code of the binarization process of the captured image. , Two Port RA
Set on M61.

Similarly, the instruction interpreting section 87 interprets and sets the completed instruction group at the time when the robot operation instruction and the image processing instruction have been executed (T ₁₀₅ , T ₁₁₂ , T ₁₁₂₎ . _T113 , _T106 ).

Further, at _T114 , the command interpreting section 87 interprets the image processing command of the line number 50. Since the instruction in line number 50 is the interpretation execution end instruction (END), the instruction interpretation unit 87 stops the vision task.

[0108] T ₁₀₇ and later, interpreting section 87 interprets the instruction excisional only robot operating instructions.

<Image Processing> The image processing performed by the image processing section 64 will be described below. The purpose of the image processing in the present system is to recognize a part, to detect the posture of the part in the pallet, and to detect the barycentric coordinates of the part.

[0110] Features Parameters Figure 22 is a plan view of a component B which is an example used in this embodiment. Part B is a flat plate and has holes of different sizes at three locations. FIG. 23 shows the reference values of the characteristic parameters of the part B. s ₁ is the area of the hole H ₁ , s ₂ is the area of the hole H ₂
And s ₃ is the area of the hole H ₃ .

In this embodiment, the coordinates (X ₁ , Y ₁ ) of the center of the hole H ₁ are set as the origin of the component, and the image processing unit 64 uses the coordinate so that the robot 6 can grip the component. It is used as a reference point when calculating the position. Also, as the center of the hole _{H 2 (X 2, Y 2} ) and the holes H ₁ center (X _1, Y _1), the vector α directed from hole H ₂ in the direction of the hole H _1, the inclination of the component B It is considered to represent This reference axis alpha (vector alpha), the center of the hole H ₁ of the center (X _1, Y ₁₎ and the hole H ₃ (X
₃ , Y ₃ ) is defined as β ₃ . Also,
The center of the hole _{H 1 (X 1, Y 1} ) and the center of the hole _{H 2 (X 2, Y 2} )
The distance between the l _2, represented by _{l 2 = {(X 1 -X} 2) 2 + (Y 1 -Y 2) 2} 1/2. The center of the hole _{H 1 (X 1, Y 1} ) and the hole H ₃
The distance between the centers (X ₃ , Y ₃ ) is represented by l _3, and represented by l ₃ = {(X ₂ −X ₃ ) ² + (Y ₂ −Y ₃ ) ² } ^1/2 . S ₁ , s ₂ , s ₃ , β ₃ ,
Let l ₂ , l ₃ and the area S of the entire part be reference amounts representing the features of the part B. In other words, whether or not a certain part is the target part B is determined from the image of the part by s ₁ ′, s
₂ ′, s ₃ ′, β ₃ ′, l ₂ ′, l ₃ ′, S ′ are calculated, and these are calculated as reference parameters s ₁ , s ₂ , s ₃ , β ₃ , l ₂ , l
_3. Judge by comparing with S. These reference parameters are calculated in advance and stored in the RAM 52.
The reference parameters are used in the image processing unit 64, and the reference parameters in the RAM 52 are stored in the two-port mode of the image processing unit 64 when the control device 17 is turned on.
The data is transferred to the RAM 61.

[0112] Recognition view 24 of the part takes an image by the camera 2 is a diagram for explaining the principle of calculating the centroid of the formed component hole (center). As described above, the parts are stored in cells in the pallet. The area setting unit 74 in FIG. 3 has a function of cutting out an image of an entire cell of a pallet in which a target component is stored.

FIG. 24 (a) shows a result obtained by capturing an image of a part by the camera 2 and binarizing the image into "1" and "0" according to the brightness level in the frame memory 75. FIG. Each cell in (a) corresponds to one CCD element and one memory in the frame memory 75. Although it is delicate whether the hole portion is quantized to “1” or “0”, in the present embodiment, cells having 50% or more of the maximum luminance are set to “0” and less than 50%. The cell is "1".

Next, a description will be given of a labeling processing method for determining that H ₁ , H ₂ , and H _{3 in} FIG. 22 are holes. In order from the beginning of the cell, 0/1 of the density is determined for each cell,
When "1" is found, it is determined whether or not there is a cell already labeled in a block portion composed of 3 × 3 cells centering on the cell where "1" is found. "1" I found
Is a cell that is the first "1" in the block portion composed of 3 × 3 cells, a label that is not yet used is assigned to the cell of the “1”. If there is a cell already labeled, the same label as the cell is given. Similar processing is performed for all cells to be processed. In this way, each closed area is labeled differently. Taking the part B of FIG. 22 as an example, it is possible to label the _three holes as “H ₁ ”, “H ₂ ”, and “H ₃ ”, respectively.

Next, a method of obtaining the areas s ₁ ′ to s ₃ ′ and the center position (centroid position) of the labeled holes will be described with reference to FIG. First, the area of the hole is counted by counting the number of "1" cells in the labeled and bound area, and the total is defined as the area. Next, when the center of the hole is obtained, a histogram of the number of cells “1” is created in each of the X and Y directions of the image of FIG. Is the center of gravity (X ₁ ′, Y ₁ ′) of the hole H ₁ .

If the above processing is performed on the three hole images, the area s ₁ ′, s ₂ ′, s ₃ ′ of the three holes of the part to be grasped and the CCD coordinates of the three holes Coordinates of the center in the system (X ₁ ′, Y ₁ ′), (X ₂ ′, Y ₂ ′), (X ₃ ′,
Y ₃ ′). Figure 2
By comparing with the reference value of the feature amount stored in advance as in 3, it can be determined whether or not the component is correct.

That is, as a comparison of the feature values, s₁ ≒ s₁',
s_Two ≒ s_Two', S_Three ≒ s_Three', L_Two≒ l _Two', L_Three ≒ l_Three', Β_Three
 = Β_Three'

Coordinate Conversion Next, the conversion of the coordinate system of the image processing and the coordinates of the robot will be described. FIG. 25 shows the relationship between the robot coordinate system X _R Y _R and the visual coordinate system X _V Y _V. The target position in the robot coordinate system is (P _xR , P _YR , 1), and the target position in the visual coordinate system is (P _xV , P _YV , 1). Also,
And the origin O _R of the coordinate system X _R Y _R of the robot, the deviation of the origin O _V of the coordinate system X _V Y _V vision is the (δ _x, δ _Y), a robot coordinate system X _R Y _r is θ degrees by Assuming that the image is rotating and the resolution of the visual system is km / pixel,

[0119]

(Equation 1) There is a relationship.

The above-mentioned conversion parameters are stored in the RAM 52 in FIG. 2 in advance, and are automatically written into the two-port RAM 61 when the power of the control device 17 is turned on. Using this parameter, the position information of the part is converted from the image coordinate system to the robot coordinate system or from the robot coordinate system to the image coordinate system.

FIG. 26 shows a preparatory operation routine (the starting program 10 in FIG. 5) until the controller 17 is turned on and the keyboard 20 is activated.
It is a flow chart of 0).

First, the power is supplied to the control device 17 in step S.
When inserted in step 400, step S401
The reference value of the characteristic parameter is set for each part by the instruction interpreting unit 8.
7 through the buses 86 and 59 from the inside of the RAM 52
The data is transferred to an area of the two-port RAM 61 designated in advance. At this time, needless to say, the data in the RAM 52 remains as it is.

Next, in step S402, the above-mentioned conversion parameters used for the coordinate conversion calculation are also transferred from the RAM 52 to the two-port RAM 61. Next, in step S403, the origin of the XY table 3 is determined. The table 3 is for moving the visual camera 2 and the lens in a plane. Next, step S4
In step 04 and step S405, the origin of the robot 6 and shuttle 9 is determined, and in step S406, the apparatus is ready to start.

After the power is turned on, it goes without saying that keyboard input is not accepted until the process goes to step S406.

<Effects of Embodiment> According to the embodiment described above, the following effects can be obtained. That is, in the control device 17, one command interpreting unit 87 interprets two different types of commands, ie, a robot operation command for performing assembly and an image processing command for performing image processing. On the other hand, the actual execution of the robot operation is controlled by the assembly operation control (that is, the robot operation control unit 62 and the hand operation control unit 63), and the image processing is actually performed by the image processing unit 64. I have to.
That is, two different processes, robot operation and image processing, are described by two modularized programs,
One program interpreter executes these program modules, while execution of the robot operation and execution of the image processing are performed by the assembly operation controller (that is, the robot operation controller 6).
2, the hand operation control unit 63) performs image processing, and the image processing unit 64 performs image processing.

For this reason, there is an effect that the development load of the control device 17 can be greatly reduced. This is because arbitration control and synchronization control of two program modules that operate in parallel can be easily performed by unifying instruction interpretation. In addition, since the instruction interpreting units have the same specifications, there is an effect that the apparatus can be easily used by the user after the factory introduction. Further, since the command interpreting means is the same, the communication interface and the communication program between the respective devices are also simplified as the respective commands, and the devices are inexpensive and can be used more easily by the user. . In addition, since the execution of the robot operation and the execution of the image processing are performed in two steps, the development of the control procedure in the assembling operation control unit can be performed independently of the development of the control procedure in the image processing unit 64, which is efficient. Because there is.

More specifically, in the control device 17, one command interpreting section 87 comprises an input / output means comprising a CPU 50, memories (51, 52), a keyboard 20, and a CRT 19. These are connected by a local bus 86 that can communicate only within the command interpreter 87. The CPU 50, the robot operation controller 62, the hand operation controller 63, and the image processor 64 are separate from the local bus 86. Are connected by a common bus 59. The common bus 59 is a general standard and is a multi-bus. The ROM 51, which is a part of the memory of the instruction interpreting section 87, stores a command interpreting program and a multitasking OS or a program for appropriately using the command interpreting program by a robot, a hand operation instruction, and an image processing instruction. A program that supports output is stored in advance.
These programs are stored in the RAM 52, which is a random access memory of the instruction interpreting unit 87, by dividing the area in advance. Further, a robot and hand operation command and an image processing command can be separately input by input means such as a keyboard. Then, when the assembling apparatus is activated by input means such as a keyboard, the control apparatus first interprets and executes the robot and hand operation commands, and starts image processing during the robot operation commands. When the CPU of the command interpreting unit finds a command to perform the image processing command, the CPU of the command interpreting unit starts to sequentially interpret and execute the image processing command. The operation is stopped and only the robot command interpretation is performed.

For this reason, for example, it is possible to easily describe an operation such as taking in an image at a timing when the robot arm escapes from the camera area. Further, since the timing can be controlled via the shared memory 2-PORT-RAM, arbitration can be performed at a very high access time level of the computer. When the image processing is unnecessary, the CPU of the instruction interpreting unit can concentrate on only the instruction interpretation of the operation of the robot, and one CPU performs a plurality of processes. Natsuta

Further, since the instruction interpreting means has the same specification, there is an effect that the apparatus can be easily used by the user after the factory introduction. Further, since the command interpreting means is the same, the communication interface and the communication program between the respective devices are simplified as the respective commands, and there is an effect that the devices can be inexpensive and the user can use the device more easily.

The instructions for the operation control units 62 and 63, the instructions for the image processing unit 64, and the like are composed of instruction codes in a unified format (FIG. 1).
3 to 15) and stored in the memory 60 or the RAM 61. That is, a medium called an instruction code and a memory exists between the upper control unit (interpretation unit 87) and the lower control unit (operation control units 62 and 63), and the robot in the upper control unit (interpretation unit 87). It absorbs the difference between instructions for tasks and vision tasks.

In the robot task which is the main task,
Instruction to start the vision task which is a child task (STA
RT), the arbitration of activation between the two tasks becomes easy. Further, since a timing confirmation command or the like is prepared, the start and stop between tasks becomes easy.

Various parameters such as coordinate conversion parameters and reference feature parameters are stored in a battery-backed RAM, and are transferred to the required RAM 61 when the power is turned on. As a result, centralized management of these parameters is facilitated. Specifically: -1: All operations can be performed by providing only one input / output means such as a keyboard, a CRT floppy disk drive, or the like, and a portion required for input / output of operation commands such as a robot. -2: There is no need to back up the RAM of the image processing unit. -3: There is no need to input parameters again even if the image processing unit is replaced due to a failure or the like. -4: There is no need to store parameters twice, so there is no malfunction due to deviation or the like.

<Modifications> The present invention can be variously modified without departing from the gist thereof. For example, in the above embodiment, as shown in FIG. 4, the work arranged in a matrix in the pallet has been described. However, the arrangement of the work in the pallet is not limited to this. You can arrange them,
They may be randomly stored in a pallet. In short,
It is sufficient that the position of the work in the pallet is roughly determined, and it is important that the robot can be roughly informed of the position of each work.

In the above embodiment, the size of the cell is constant, and one type of work is used. The present invention can be applied to a case where different sizes of cells are different or different works exist in one pallet. In this case, the control device 17 determines the correspondence between the position of the cell, the size of the cell, and the type of work in the cell by the image processing unit 64.
Send it to Also, the present invention can be applied to a case where a plurality of works exist in one cell. In the above embodiment, the coordinate conversion is performed in the visual system, but may be performed in the robot controller.

In the above embodiment, the position of the center of gravity of the holes A and B is determined as the position of the maximum value of the histogram as the center of gravity, as described with reference to FIG. 24. That is, in FIG. 24, when the number of all dots is n,
The point at which the cumulative frequency is n / 2 is defined as the center of gravity. According to this method, the center of gravity can be calculated even for a hole having a shape that is not the target.

In the above-described embodiment, the characteristic parameters and the coordinate conversion parameters are always stored in the memory 52 of the command interpreting section 87, and are transferred to a 2-PORT-RAM accessible by the image processing section after power-on. I am trying to do it.
The present invention is not limited to this. For example, parameters used independently by the image, for example, the absolute start position of image capture, the gain of the input of the camera 2 and the like are stored in the memory of the command interpretation unit, and May be forwarded to Also, parameters required for the robot operation unit and the hand operation unit may be transferred from the memory of the instruction interpretation unit to the shared memory.

In the above embodiment, the operation of the robot is started first, and the processing of the image is started within the operation command of the robot. However, the processing time of the image is long, and the assembling apparatus is intermittently operated. In the case of application to a device that moves the robot, the robot operation command interpretation is activated from the image processing command, and a stop command is inserted in the robot operation command, so that the CPU processing is not performed. No problem.

In the above embodiment, if a sequence control command (jump command, repetition command, timer command) other than the command code to the robot operation unit and the image processing unit exists in the series of operation and processing commands, 9 that the instruction interpreting unit itself understands and executes by the instruction interpreting means.
Can be easily considered from the description of the vision task start instruction at line 360 in FIG.

In the above-described embodiment, the means for the image processing unit to transmit and receive the instruction code from the instruction interpreting unit is a 2PO
Although the RT-RAM 61 is used, the shared memory 60 may be used similarly to the robot operation unit of the above-described embodiment.

Similarly, in the above-described embodiment, the means for the robot operation unit and the hand operation unit to transmit and receive the instruction code from the instruction interpretation unit is the shared memory 60. However, as in the image processing unit of the above-described embodiment, the shared memory 60 is used. It is also possible to use a RAM. It is stored in the storage unit and, if necessary, transferred from this storage unit to the gripping unit and / or the image processing unit where it is used. For this reason, only one parameter needs to be stored, and the management thereof can be unified.

[0141]

As described above, according to the present invention,
Grip operation command, image operation processing command, and movement
The control device includes a single interpreting means for interpreting an operation control command describing an assembling operation, and performs an operation or a process in the gripping operation unit, the image processing unit, or the moving unit according to the corresponding instruction code. This has the effect of significantly reducing the development load. That is, each instruction is
Since the instruction code is broken down, the development of the program in the image processing unit, the gripping unit, and the moving means can be easily performed by unifying the instruction code into a uniform format.

According to the control device of the present invention having another configuration,
For example , this automatic gripping device has a function of not only gripping operation but also moving a gripping part to a position where a work is placed, moving a gripping part gripping a work to an assembling stage, and assembling the gripped work into an assembly. The above-mentioned system
It has the effect of greatly reducing the development load of control equipment.
According to the control device of the present invention having another configuration , the interpretation unit includes:
After interpreting one image processing instruction or gripping operation instruction and converting it into an instruction code, the next image processing instruction or gripping operation instruction is interpreted to develop the control device.
It has the effect of greatly reducing the load . According to a preferred aspect of the present invention, each of the image processing unit and the holding unit includes an execution unit that interprets and executes the instruction code.

According to a preferred aspect of the present invention, the memory storing the instruction code converted from the image processing instruction is a dual port RAM. Generally, a micro computer suitable for image processing exists in image processing, and a micro computer suitable for grip control exists in a gripping unit. Dual Port RAM
Enables the use of microcomputers with these different bus architectures.

[Brief description of the drawings]

FIG. 1 is a perspective view of a robot assembly system according to a preferred embodiment to which the present invention is applied.

FIG. 2 is a block diagram showing a configuration of a control device 17 of the FIG. 1 system.

FIG. 3 is a block diagram of an image processing unit 64 of the control device 17;

FIG. 4 is a plan view showing the structure of a pallet used in the system shown in FIG. 1;

FIG. 5 is a diagram showing a data configuration of a ROM 51 of an interpretation unit 87.

FIG. 6 is a diagram showing a data configuration of a ROM 52 of an interpretation unit 87.

FIG.

FIG. 8 is a diagram specifically showing the operation of the shuttle and the robot of the system shown in FIG. 1;

FIG.

FIG. 10 is a flow chart of a program for describing a robot task as an example.

FIG. 11 is a flowchart of a program for describing a vision task as an example.

FIG. 12 is a diagram schematically showing a state where a robot program is converted into an instruction code by an interpretation unit 87 and the instruction code is executed.

FIG.

FIG.

FIG. 15 is a diagram showing a format of an instruction code.

FIG. 16 is a view showing a relationship between an interpretation program and each task, and showing flags necessary for controlling interpretation in the task.

FIG. 17A,

17B is a flowchart showing the interpretation procedure of the interpretation unit 87. FIG.

FIG. 18 is a flowchart showing a control procedure in a robot / hand operation control unit.

FIG. 19 is a flowchart showing a control procedure of the image processing unit 64.

FIG.

FIG. 21 is a timing chart showing the operation of each unit when the program shown in FIGS. 9 to 11 is executed.

FIG.

FIG. 23 is a view for explaining the definition of characteristic parameters in component recognition.

FIG. 24 is a view for explaining the principle of detecting the position of the center of gravity necessary for component recognition.

FIG. 25 is a diagram showing a relationship between a visual coordinate system and a robot coordinate system.

FIG. 26 is a flowchart showing a control procedure of the activation program 100.

[Explanation of symbols]

 2 CCD camera 3 XY table 6 Robot 7 Hand 8 Assembly stage 9 Shuttle 11, 12 Parts supply unit 13a, 13b Pallet 17 Control unit 18 Cable 19 CRT 50 CPU 51 ROM 52 RAM 60 Shared memory 61 Two-port RAM 62 Robot operation Control unit 63 Hand operation control unit 64 Image processing unit 87 Command interpretation unit

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁶ Identification symbol FI G05B 19/414 G05D 3/12 K G05D 3/12 G05B 19/18 N (72) Inventor Yu Shibata 3-chome Shimomaruko, Ota-ku, Tokyo No. 30-2 Canon Inc. (56) References JP-A-2-56004 (JP, A) (58) Fields investigated (Int. Cl. ⁶ , DB name) G05B 19/4155 B25J 9/10 B25J 9/16 B25J 13/08 G05B 19/19 G05B 19/414 G05D 3/12

Claims (8)

(57) [Claims] 1. A captures an image of the workpiece, an image processing unit for calculating the position information of the workpiece from the image, and a grip portion for gripping the basis of the work on the position information, the word
This gripper can be placed on the workpiece according to the workpiece position information.
To the right position and assemble the gripper holding the work
The moving means to move to the stage, and this gripped work
A control device for controlling an automatic gripping device having assembling means for assembling into an assembly , comprising: a series of image processing instructions for imaging the work and calculating position information from an image of the work; a series of gripping operation instruction for gripping Te, the mobile hands
A storage unit for storing a program including an operation control instructions describing the operation of the stage and the assembly means, and gripping operation instruction and the image processing instruction in the program
In addition to identifying and interpreting the operation control instruction , the image processing instruction, the gripping operation instruction, and the operation control instruction are converted into corresponding intermediate instruction codes, and the converted instruction codes are respectively converted into the image processing unit and the gripping unit. A control device for an automatic grasping device using a sight, comprising: one interpreting unit for passing the information to the automatic grasping device. 2. An image of a work is taken, and a preceding image is taken from this image.
An image processing unit for calculating position information of the workpiece;
And a gripper for gripping the hook based on the position information.
A control device for controlling an automatic gripping device, which images the work and calculates position information from an image of the work.
A series of image processing commands to be issued and the work
And a series of gripping operation commands for gripping based on the
A storage unit for storing a ram, the image processing instruction and the gripping operation instruction in the program,
And interprets the image processing instruction and the gripping motion.
Operation instructions are converted to the corresponding intermediate instruction codes and converted.
The obtained instruction code is passed to the image processing unit and the gripping unit, respectively.
For interpreting the one image processing instruction or gripping operation instruction and converting it into a command code, and then interpreting the next image processing instruction or gripping operation instruction. A control device for an automatic gripping device using vision. 3. An image of a workpiece is taken, and a preceding image is taken from this image.
An image processing unit for calculating position information of the workpiece;
And a gripper for gripping the hook based on the position information.
A control device for controlling an automatic gripping device, which images the work and calculates position information from an image of the work.
A series of image processing commands to be issued and the work
And a series of gripping operation commands for gripping based on the
A storage unit for storing a ram, the image processing instruction and the gripping operation instruction in the program,
And interprets the image processing instruction and the gripping motion.
Operation instructions are converted to the corresponding intermediate instruction codes and converted.
The obtained instruction code is passed to the image processing unit and the gripping unit, respectively.
The image processing unit has first execution means for interpreting and executing an instruction code converted from the image processing instruction, and the gripping unit converts the instruction code from the gripping operation instruction. A control device for an automatic grasping device using a visual sense, comprising a second execution means for interpreting and executing the instruction code. 4. The interpreting unit has one CPU, and an instruction code converted from the image processing instruction and an instruction code converted from the gripping operation instruction are two different first codes accessible by the CPU. 4. The method according to claim 1, wherein the data is stored in each of the first and second memory circuits.
A control device for an automatic gripping device using a visual sense according to any one of the claims. 5. The automatic visual gripping device according to claim 4 , wherein the second memory storing the instruction code converted from the image processing instruction is a dual port RAM. Control device. 6. An automatic gripping device, comprising: moving means for moving the gripping portion to a position where a workpiece is placed according to the position information of the workpiece, and moving the gripping portion gripping the workpiece to an assembling stage; Further comprising assembling means for assembling the gripped work into an assembly, wherein the program includes an instruction for describing an operation of the moving means and the assembling means, and the instruction is also interpreted by the interpreting unit; The visual code according to claim 4, wherein both an instruction code corresponding to an instruction describing a movement operation of the gripper and an instruction code corresponding to the gripping operation instruction are stored in the second memory circuit. Control device of automatic gripping device using 7. An automatic gripping device, comprising: an image processing unit that captures an image of a work and calculates position information of the work from the image; and a gripping unit that grips the work based on the position information. A control method, comprising: a program including a series of image processing instructions for imaging the work and calculating position information from an image of the work, and a series of gripping operation instructions for gripping the work based on the position information. An interpretation step of identifying and interpreting the image processing instruction and the gripping operation instruction; a conversion step of converting the image processing instruction and the gripping operation instruction into corresponding intermediate instruction codes, respectively; comprising the step of passing the image processing unit and the holding unit, one image processing instructions or grasping operation instruction is interpreted life
After being converted to the instruction code,
A method for controlling an automatic gripping device using vision, wherein an image processing instruction or a gripping operation instruction is interpreted . 8. An image of a workpiece is taken, and a preceding image is taken from this image.
An image processing unit for calculating position information of the workpiece;
And a gripper for gripping the hook based on the position information.
A control method for controlling an automatic gripping device, wherein an image of the work is taken and position information is calculated from an image of the work.
A series of image processing commands to be issued and the work
And a series of gripping operation commands for gripping based on the
Ram, by identifying the image processing command and the gripping operation command
The interpreting step and the image processing instruction and the gripping operation instruction
A conversion step of converting the converted instruction code into the image processing unit and the gripping unit, respectively.
The image processing unit interprets and executes the instruction code converted from the image processing instruction, and the gripping unit interprets and executes the instruction code converted from the gripping operation instruction. A method for controlling an automatic gripping device using vision.