JP2018144228A - Robot control apparatus, robot, robot system, teaching method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To teach a robot various kinds of robot's motion more easily.SOLUTION: A robot control apparatus includes: a display part for displaying an image of a robot; a receiving section for receiving an operation to the image of the robot displayed by the display part; and a teaching section for teaching the robot robot's motion. The display part displays an image indicating movement of a grip part of the robot, rotation of the grip part, or opening/closing of the grip part in response to the operation. The teaching section teaches the robot the movement of the grip part of the robot, the rotation of the grip part, or the opening/closing of the grip part in response to the operation.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a robot control device, a robot, a robot system, a teaching method, and a program.

  Patent Document 1 discloses a robot system that displays a CG image of a robot on a touch screen, virtually operates the CG image of the robot on the touch screen, and operates the robot so as to follow the virtual operation. Have been described. Patent Document 1 describes, for example, that the joint angle of the arm of the CG image is changed by a drag operation.

JP 2012-171024 A

  However, in Patent Document 1, it is not possible to sufficiently teach the operation actually required for the robot operation. For example, when teaching an operation of gripping an object with a hand, it is necessary to provide various teachings such as changing the angle of the arm, opening and closing the hand, positioning the hand to the object, and the operation is complicated. In addition, it is difficult to teach in detail on the touch screen how to change the angle of the arm, open and close the hand, align the hand with the target, and the like so that the target can actually be gripped by the hand.

  Accordingly, an object of the present invention is to teach a variety of operations of a robot more easily.

  A first aspect of the present invention that solves the above problem is a robot control device, a display unit that displays an image of a robot, a reception unit that receives an operation on the image of the robot displayed by the display unit, A teaching unit that teaches the operation of the robot, and the display unit displays an image representing movement of the gripping part of the robot, rotation of the gripping part, or opening / closing of the gripping part according to the operation. The teaching unit teaches the robot to move the gripping part of the robot, rotate the gripping part, or open / close the gripping part in accordance with the operation. As a result, a robot image representing operations corresponding to various operations on the image is displayed, and teaching corresponding to these operations is performed. Therefore, teaching of various operations of the robot can be performed more easily.

  In the robot control apparatus, the receiving unit may receive a movement of a contact position on a screen as the operation. As a result, the operation of the robot can be taught more intuitively because the operation can be performed by a touch operation or the like on the screen on which the image is displayed.

  In the robot control device, when the receiving unit receives an operation of moving the contact position from the image of the gripping unit to the image of the target, the display unit moves the gripping unit to the target Then, an image representing a gripping operation may be displayed, and the teaching unit may teach the robot to move and grip the gripping unit to the object. Thereby, when teaching the gripping operation, the gripping operation of the robot can be taught more easily by a moving operation such as dragging without inputting the detailed operation of the robot.

  In the robot control apparatus, when the reception unit receives an operation of selecting an object image in a state where the image of the grip unit is selected, the display unit moves the grip unit to the target object. An image representing a gripping operation may be displayed, and the teaching unit may teach the robot to perform a gripping operation by moving the gripping unit to the object. Thereby, when teaching the gripping operation, the gripping operation of the robot can be taught more easily by the selection operation such as the tap operation without inputting the detailed operation of the robot.

  In the robot control apparatus, the accepting unit accepts the contact position on the image of the target object as a gripping position, and the display unit represents an operation of moving and gripping the gripping part to the gripping position. May be displayed, and the teaching unit may teach the robot to move and hold the gripping unit to the gripping position. Thus, the grip position can be easily taught while performing a selection operation such as a drag operation or a tap operation.

  The robot control device may further include an image acquisition unit that acquires a captured image obtained by capturing the target object, and the display unit may generate and display an image of the target object based on the captured image. Good. Thereby, the operation | movement with respect to an actual target object can be taught easily.

  In the robot control device, when the receiving unit receives an operation of moving the contact position so as to draw an arc, the display unit displays an image representing an operation of rotating the gripping unit, and the teaching The unit may teach the robot to perform an operation of rotating the grip unit. Thereby, the rotation operation of the grip portion can be easily taught.

  In the robot control apparatus, the contact position includes a first contact position and a second contact position, and the reception unit has received an operation of moving the first contact position and the second contact position closer to or away from each other. In this case, the display unit may display an image representing an operation of closing or opening the gripping unit, and the teaching unit may teach the robot to close or open the gripping unit. Thereby, the opening / closing operation | movement of a holding part can be taught easily.

  In the robot control device, when the reception unit receives an operation of moving the contact position in a predetermined area on the screen, the display unit displays an image of the grip unit by changing a viewpoint. You may do it. Thereby, a viewpoint can be changed easily.

  In the robot control apparatus, the accepting unit accepts, as the operation, a movement of an instruction position indicated by an instruction image on the screen operated by a pointing device, instead of the movement of the contact position on the screen. Also good. Thereby, teaching can be easily performed even using a mouse or the like.

  A second aspect of the present invention that solves the above problem is a robot, wherein the display unit displays an image of the robot, a reception unit that receives an operation on the image of the robot displayed by the display unit, A teaching unit that teaches the operation of the robot, and the display unit displays an image representing movement of the gripping part of the robot, rotation of the gripping part, or opening / closing of the gripping part according to the operation. The teaching unit teaches the robot to move the gripping unit of the robot, rotate the gripping unit, or open / close the gripping unit according to the operation. As a result, a robot image representing operations corresponding to various operations on the image is displayed, and teaching corresponding to these operations is performed. Therefore, teaching of various operations of the robot can be performed more easily.

  A third aspect of the present invention that solves the above problem is a robot system including a robot and a robot control device that controls the robot, the display unit displaying an image of the robot, and the display by the display unit A receiving unit that receives an operation on the image of the robot, and a teaching unit that teaches the operation of the robot, and the display unit moves the gripping unit of the robot according to the operation, and the gripping unit Image indicating rotation or opening / closing of the gripping unit, and the teaching unit moves the gripping unit of the robot, rotates the gripping unit, or opens / closes the gripping unit according to the operation. Teach the robot. As a result, a robot image representing operations corresponding to various operations on the image is displayed, and teaching corresponding to these operations is performed. Therefore, teaching of various operations of the robot can be performed more easily.

  A fourth aspect of the present invention that solves the above problem is a robot motion teaching method, comprising: a display step for displaying an image of the robot; and an operation for the image of the robot displayed in the display step. A receiving step and a teaching unit step for teaching the operation of the robot, and the display step moves the gripping unit of the robot, rotates the gripping unit, or opens and closes the gripping unit according to the operation. The teaching step teaches the robot to move the gripping part of the robot, rotate the gripping part, or open / close the gripping part according to the operation. As a result, a robot image representing operations corresponding to various operations on the image is displayed, and teaching corresponding to these operations is performed. Therefore, teaching of various operations of the robot can be performed more easily.

  A fifth aspect of the present invention that solves the above problem is a program for a robot control apparatus, which includes a display unit that displays an image of the robot, and a reception unit that receives an operation on the image of the robot displayed by the display unit And the robot control device functions as a teaching unit for teaching the operation of the robot, and the display unit moves the gripping unit of the robot, rotates the gripping unit, or the gripping unit according to the operation. The teaching unit teaches the robot to move the gripping part of the robot, rotate the gripping part, or open / close the gripping part in accordance with the operation. As a result, a robot image representing operations corresponding to various operations on the image is displayed, and teaching corresponding to these operations is performed. Therefore, teaching of various operations of the robot can be performed more easily.

It is a figure which shows an example of schematic structure of the robot system which concerns on one Embodiment of this invention. It is a figure which shows an example of a function structure of a robot system. It is a figure which shows an example of the hardware constitutions which implement | achieve the function of a robot control apparatus. It is a flowchart (the 1) which shows an example of a teaching process. It is a flowchart (the 2) which shows an example of a teaching process. It is FIG. (1) which shows an example of operation and a display on a touch panel. It is FIG. (2) which shows an example of operation and a display on a touch panel. It is FIG. (3) which shows an example of operation and a display on a touch panel.

  An embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating an example of a schematic configuration of a robot system according to an embodiment of the present invention.

  The robot system 1 includes a robot 2 and a robot control device 3. The robot 2 and the robot control device 3 are connected to be communicable. In addition, the operation terminal 4 is connected to the robot control device 3 so as to be communicable.

  The robot 2 performs work according to instructions from the robot control device 3. The robot 2 is disposed near the work table T, for example, and performs work in a predetermined work area on the work table T. Although the work content of the robot 2 is not particularly limited, for example, there is a work of gripping an object (hereinafter also referred to as “work”) W, moving the gripped work W, or combining a plurality of works W. .

  The robot 2 includes a body part 20, an arm 21, a hand 22, a leg part 24, and a head part 25. The head 25 is provided with two imaging devices 26. The hand 22 corresponds to the gripping part of the present invention.

  The arm 21 is provided on the trunk portion 20. The arm 21 includes one or more joints (also referred to as “rotary shafts”) 21 a and one or more arm members (also referred to as “links”) 21 b.

  The joint 21a is provided with an actuator (not shown) for operating them. The actuator includes, for example, a servo motor and an encoder. The encoder value output from the encoder is used for feedback control of the robot 2 and the like.

  Further, a force sensor 21 c (not shown) is provided at the tip of the arm 21. The force sensor 21c is a sensor that detects a force and a moment received as a reaction force with respect to the force generated by the robot 2. As the force sensor 21c, for example, a 6-axis force sensor capable of simultaneously detecting 6 components of a force component in the translational 3 axis direction and a moment component around the 3 rotation axes can be used. The force sensor is not limited to six axes, and may be, for example, three axes.

  By linking each rotation axis, the posture of the arm member 21b is changed, and the position of interest (also referred to as “end point”) set at the tip of the arm 21 is freely moved within a predetermined movable range. Can be directed in any direction. Note that the position of the end point is not limited to the tip of the arm, and may be set at the tip of the hand 22, for example.

  The arm 21 can be said to be a kind of manipulator. The manipulator is a mechanism that moves the position of the end point, and is not limited to an arm and can take various forms. Further, the number of manipulators is not limited to two as shown in the figure, and one or three or more manipulators may be provided.

  The hand 22 is provided at the tip of the arm 21. The hand 22 includes, for example, a plurality of fingers, and can hold or release the workpiece with at least two fingers. In the following, it is assumed that the hand 22 has four fingers. The hand 22 may be detachable from the tip of the arm 21. Each finger is provided with an actuator (not shown) for operating them. The actuator includes, for example, a servo motor and an encoder.

  The hand 22 can be said to be a kind of end effector. The end effector is a member for gripping, lifting, lifting, adsorbing, or processing a workpiece. The end effector can take various forms such as a hand, a hook, and a suction cup. Further, a plurality of end effectors may be provided for one arm.

  The imaging device 26 is disposed on the head 25 so that the work area on the work table T, the arm 21, the hand 22, and the like can be imaged. The imaging device 26 outputs the captured image data to a control unit (not shown) in the robot 2 and the robot control device 3. The imaging device 26 is a camera having, for example, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like. In the present embodiment, a stereo image is captured by the two imaging devices 26, and three-dimensional stereoscopic vision, object recognition, and the like can be performed by image processing.

  Of course, the configuration of the robot 2 is not limited to the illustrated example. For example, FIG. 1 shows an example in which the number of joints is seven (seven axes), but the number of joints (also referred to as “number of axes”) and the number of links may be increased or decreased. In addition, the shape, size, arrangement, structure, and the like of various members such as joints, links, and hands may be changed as appropriate.

  The robot control device 3 receives an operation input related to the operation of the robot 2 from the user. For example, the robot control device 3 outputs a model image or the like of the robot 2 to the operation terminal 4 or the display device 96 (see FIG. 3) and displays it. Further, for example, the robot control device 3 receives an operation on the model image of the robot 2 from the user via the operation terminal 4 or the input device 95 (see FIG. 3). Further, for example, the robot control device 3 generates teaching information indicating the operation of the robot 2 based on an operation on the received model image.

  Further, the robot control device 3 generates a control command for operating the robot 2 by position control based on the trajectory plan, visual servo control, or the like based on the teaching information, and outputs the generated control command to the robot 2. The robot 2 operates by driving an actuator in accordance with a control command output from the robot control device 3. In addition, the robot control device 3 controls the imaging device 26 to perform imaging and acquire a captured image.

  The operation terminal 4 is controlled by the robot control device 3 and functions as a user interface for inputting an operation from the user, outputting information to the user, and the like. In the present embodiment, the operation terminal 4 includes a touch screen, a communication interface, and the like. The touch screen includes, for example, a capacitive touch panel, a display, and the like. For example, the operation terminal 4 generates an operation signal representing a user operation on the touch screen, and transmits the operation signal to the robot control device 3 via the communication interface. For example, the operation terminal 4 receives screen information from the robot control device 3 via a communication interface, and displays a screen corresponding to the screen information on the touch screen.

  The configuration of the robot system 1 described above is the main configuration in describing the features of the present embodiment, and is not limited to the above configuration example. Further, the configuration of a general robot system is not excluded.

  For example, the imaging device 26 may be provided on the tip portion of the arm 21 of the robot 2, the trunk portion 20, or the like. For example, the imaging device 26 can be installed on a ceiling, a wall, or the like. Further, for example, in addition to the imaging device 26, another imaging device may be provided. For example, the imaging device 26 may be connected to the robot control device 3. For example, the robot control apparatus 3 may be incorporated in the robot 2 as a control unit. Further, for example, the function related to teaching of the robot control device 3 is provided as a teaching device different from the robot control device 3, and a teaching or control command is sent from the teaching device to the control unit of the robot control device 3 or the robot 2. You may make it send.

  FIG. 2 is a diagram illustrating an example of a functional configuration of the robot system.

  The robot 2 includes an operation control unit 200 and the like. The robot control device 3 includes a storage unit 300, a reception unit 310, a display unit 320, a teaching unit 330, an image acquisition unit 340, a control unit 350, and the like.

  The motion control unit 200 indicates the current position of the end point by the movement command value based on, for example, the control command including the movement command value output from the robot control device 3 and the encoder value of the actuator of each joint 21a. The actuator of each joint 21a is driven so that it may become a target position. Note that the current position of the end point can be obtained from, for example, an encoder value in the actuator of each joint 21a.

  Further, the operation control unit 200 closes or opens the finger by a designated movement amount in accordance with, for example, a control command for each finger of the hand 22 output from the robot control device 3. In addition, for example, the operation control unit 200 rotates the predetermined joint by a specified rotation amount in accordance with a control command for rotating the predetermined joint output from the robot control device 3.

  Further, for example, the operation control unit 200 controls the imaging device 26 according to an instruction from the robot control device 3 to perform imaging, acquires a captured image, and outputs the captured image to the robot control device 3.

  The storage unit 300 stores model images of the workpiece W, the robot 2, and the like as a model image storage unit. The model image is, for example, three-dimensional CG (computer graphics). A model image of the workpiece W can be prepared for each workpiece W so as to correspond to a plurality of different work contents of the robot 2. The model image of the robot 2 can also be prepared for each part (joint 21a, arm member 21b, hand 22, finger of hand 22, head 25, trunk 20, leg 24, etc.). The three-dimensional CG may be a deformed part of the actual robot 2 or the workpiece W.

  The accepting unit 310 accepts a user operation via the operation terminal 4 or the input device 95 (see FIG. 3). The accepting unit 310 accepts an operation signal representing an operation on the screen of the operation terminal 4 or the display device 96 (see FIG. 3) from the operation terminal 4 or the input device 95, for example. The receiving unit 310 will be described in detail later. In the following, the case where the touch screen of the operation terminal 4 is used as the input means will be mainly described.

  The display unit 320 displays information to the user via the operation terminal 4 or the display device 96 (see FIG. 3). The display unit 320 outputs and displays a model image of the robot 2 on the operation terminal 4 or the display device 96, for example. For example, the display unit 320 generates screen information corresponding to the operation signal received by the reception unit 310 and causes the operation terminal 4 or the display device 96 to display the screen information. The display unit 320 will be described in detail later. In the following, the case where the operation terminal 4 is used as the display means will be mainly described.

  The teaching unit 330 generates teaching information indicating the operation of the robot 2 based on the operation signal received by the receiving unit 310. The teaching information includes, for example, an instruction to move the hand 22 (including a target position), an instruction to open / close the hand 22 (including a finger movement amount, a moving direction (either opening / closing), etc.), and rotating the hand 22 Instructions (including the amount of rotation of the joint, the direction of rotation, etc.). The teaching unit 330 will be described in detail later.

  The image acquisition unit 340 sends an imaging instruction to the robot 2 and acquires a captured image captured by the imaging device 26. Further, the image acquisition unit 340 recognizes the position and orientation of the workpiece W or the like from the captured image. For example, the image acquisition unit 340 extracts a shape that matches the model image of the workpiece W stored in the storage unit 300 from the captured image by pattern matching or the like. Since general techniques can be adopted for object recognition, detailed description is omitted.

  Based on the teaching information generated by the teaching unit 330, the control unit 350 generates a control command for operating the robot 2 by position control based on the trajectory plan, visual servo control, and the like, and outputs the generated control command to the robot 2. To do.

  For example, when an instruction to move the hand 22 is included in the teaching information, the control unit 350 performs a trajectory plan based on the current position of the end point acquired from the robot 2 and the target position included in the teaching information. Then, the control unit 350 generates a movement command value for moving the end point of the robot 2 next based on the planned trajectory, and outputs the movement command value to the robot 2.

  In the robot 2 having a joint, when the angle of each joint is determined, the position of the end point is uniquely determined by forward kinematics processing. In other words, an N-joint robot can express one position by N joint angles, and if the set of N joint angles is one target joint angle, the trajectory of the end point is a set of target joint angles. Can be considered. Therefore, the movement command value may be a value related to the position (target position) or a value related to the joint angle (target angle).

  Further, the control unit 350 opens / closes the hand 22 or rotates the hand 22 based on the teaching information generated by the teaching unit 330.

  For example, when an instruction to open and close the hand 22 is included in the teaching information, the control unit 350 generates a movement command value for each finger based on the movement amount and movement direction of the finger included in the teaching information, and Output. In addition, for example, when the instruction to rotate the hand 22 is included in the teaching information, the control unit 350 rotates the predetermined joint closest to the hand 22 based on the rotation amount and the rotation direction of the joint included in the teaching information. A value is generated and output to the robot 2.

  FIG. 3 is a diagram illustrating an example of a hardware configuration for realizing the functions of the robot control device.

  The robot controller 3 includes, for example, an arithmetic device 91 such as a CPU (Central Processing Unit), a main storage device 92 such as a RAM (Random Access Memory), an HDD (Hard Disk Drive), and the like as shown in FIG. An auxiliary storage device 93, a communication interface (I / F) 94 for connecting to a communication network by wire or wireless, an input device 95 such as a mouse, keyboard, touch sensor, and touch panel, and a display device 96 such as a liquid crystal display It can be realized by a computer 90 including a read / write device 97 that reads and writes information from and on a portable storage medium such as a DVD (Digital Versatile Disk).

  In the reception unit 310, the display unit 320, the teaching unit 330, the image acquisition unit 340, the control unit 350, and the like, for example, the arithmetic device 91 executes a predetermined program loaded from the auxiliary storage device 93 or the like to the main storage device 92. It is realized with. The storage unit 300 may be realized by the arithmetic device 91 using the main storage device 92 or the auxiliary storage device 93, for example. Communication with the robot 2 is realized, for example, when the arithmetic device 91 uses the communication I / F 94. The predetermined program may be installed from, for example, a storage medium read by the read / write device 97 or may be installed from the network via the communication I / F 94.

  The operation terminal 4 can also be realized by a computer 90 as shown in FIG. Further, the operation control unit 200 can be realized by, for example, a controller board including an arithmetic device, a storage device, a processing circuit, a driving circuit, and the like.

  The functional configuration of the robot system 1 described above is classified according to main processing contents in order to facilitate understanding of the configuration of the robot system 1. The present invention is not limited by the way of classification and names of the constituent elements. The configuration of the robot system 1 can be classified into more components depending on the processing content. Moreover, it can also classify | categorize so that one component may perform more processes. Further, the processing of each component may be executed by one hardware or may be executed by a plurality of hardware.

  Further, the sharing of functions and processes of the robot system 1 is not limited to the example described. For example, at least a part of the functions of the robot control device 3 may be included in the robot 2 and realized by the robot 2. Further, for example, at least a part of the functions of the robot 2 may be included in the robot control device 3 and realized by the robot control device 3.

  FIG. 4 is a flowchart (part 1) illustrating an example of the teaching process. FIG. 5 is a flowchart (part 2) illustrating an example of the teaching process. The flow in FIG. 4 is started when, for example, the reception unit 310 receives a teaching start instruction from the user via the operation terminal 4.

  FIG. 6 (a diagram showing an example of operation and display on the touch panel (part 1)), FIG. 7 (a diagram showing an example of operation and display on the touch panel (part 2)), and FIG. And a diagram showing an example of display (No. 3)).

  First, the control unit 350 acquires the current posture of the robot (step S1). Specifically, the control unit 350 determines the current posture of each part (the joint 21a, the arm member 21b, the hand 22, the finger of the hand 22, the head 25, the torso 20, the leg 24, etc.) from the robot 2. Get the information shown. The information indicating the current posture can be, for example, the position and orientation of each part (represented by values such as x, y, z, u, v, and w). In addition, the robot 2 shall hold | maintain the position and direction of the said part for every own part.

  Then, the display unit 320 displays a model image of the robot 2 (step S2). Specifically, the display unit 320 arranges the model image of each part stored in the storage unit 300 based on the information indicating the position and orientation of each part acquired in step S1, thereby allowing the robot 2 The three-dimensional model image is generated. Further, the display unit 320 generates a predetermined three-dimensional model image of the robot 2 from a set position and orientation of the user's viewpoint (represented by values such as x, y, z, u, v, and w). A three-dimensional model image when viewed at a viewing angle is generated. Then, the display unit 320 converts the generated three-dimensional model image of the robot 2 into a two-dimensional model image and displays it on the operation terminal 4.

  Then, the image acquisition unit 340 acquires an image (step S3). Specifically, the image acquisition unit 340 sends an imaging instruction to the robot 2 and acquires a captured image captured by the imaging device 26. Further, the image acquisition unit 340 recognizes the workpiece W from the acquired captured image by pattern matching or the like using the model image of the workpiece W stored in the storage unit 300. At this time, the image acquisition unit 340 specifies the position and orientation (represented by values such as x, y, z, u, v, and w) of the recognized workpiece W.

  Then, the display unit 320 displays a model image of the workpiece W (Step S4). Specifically, the display unit 320 arranges the three-dimensional model image of the workpiece W stored in the storage unit 300 based on the position and orientation of the workpiece W recognized in step S3, and A three-dimensional model image is generated when the three-dimensional model image is viewed at a predetermined viewing angle from the set position and orientation of the user's viewpoint. Then, the display unit 320 converts the generated three-dimensional model image of the workpiece W into a two-dimensional model image and displays it on the operation terminal 4 together with the two-dimensional model image of the robot 2 displayed in step S2. In addition, when the workpiece | work W is not recognized by step S3, you may skip the process of step S4.

  In this way, for example, as shown in FIGS. 6A, 7A, and 8A, the model image RG of the robot 2 (the model image of the arm 21, the model image HG of the hand 22 and the like are included. ) And the model image WG of the workpiece W are displayed as initial images. 6 to 8 show model images of part of the workpiece W, the hand 22 and the arm 21 for easy understanding, but the display unit 320 displays the model image of the entire robot 2. May be displayed.

  Then, the reception unit 310 receives a user operation (step S11). Specifically, the accepting unit 310 accepts an operation signal representing a user operation on the screen on which the model image or the like of the robot 2 is displayed in step S4 via the operation terminal 4. For example, the operation unit 310 receives, as operation signals, operation types such as tap, drag, rotation, pinch-in, and pinch-out, tap position, touch position, drag operation start and end positions, rotation operation rotation direction and rotation amount. , Receiving operation parameters such as the amount of movement for pinch-in and pinch-out operations. The accepting unit 310 identifies the operation content of the user based on the content of the screen currently displayed by the display unit 320 and the operation signal.

  The drag operation is, for example, an operation for continuously moving contact on the screen with a finger or the like from the start position to the end position. The rotation operation is an operation for moving the contact position on the screen with a finger or the like so as to draw an arc. One finger or the like may be brought into contact with the screen as the center of the arc (first contact position), and the contact position (second contact position) may be moved so as to draw an arc with the other finger or the like. The direction in which the arc is drawn corresponds to the rotation direction, and the rotation angle of the arc corresponds to the rotation amount. In addition, the pinch-in operation is an operation in which, for example, two fingers or the like are brought into contact with the screen (first contact position and second contact position), and the contact positions of these fingers are brought closer. In addition, the pinch-out operation is an operation in which, for example, two fingers or the like are brought into contact with the screen (first contact position and second contact position) and the contact positions of these fingers are moved away. A change in the distance between the first contact position and the second contact position corresponds to the amount of movement.

  When accepting the user's operation in step S11, the accepting unit 310 determines whether or not the accepted operation is a teaching end instruction (for example, a tap of an “end” button on the screen) (step S12). If the instruction is an instruction to end teaching (Y in step S12), the receiving unit 310 ends the flow shown in FIGS.

  If it is not an instruction to end teaching (N in Step S12), the receiving unit 310 determines whether or not the operation in Step S11 is a selection operation (for example, a tap on the screen) (Step S21).

  If it is a selection operation (Y in step S21), the accepting unit 310 determines whether or not the operation in step S11 is to select the model image HG of the hand 22 on the screen (step S22).

  If the model image HG of the hand 22 is selected (Y in Step S22), the display unit 320 highlights the model image HG of the hand 22 on the screen (Step S23), and the process returns to Step S11. The highlighting may be performed by making the color of the model image HG of the hand 22 different from the color of the model image of other parts or the workpiece W, for example.

  In this way, for example, as shown in FIG. 6 (B1), FIG. 8 (B1), and FIG. 8 (C1), the model image HG of the hand 22 of the robot 2 is selected and highlighted. 6 to 8, the tap position, the touch position, and the like are shown as the contact position P.

  When the model image HG of the hand 22 is not selected (N in Step S22), the receiving unit 310 determines whether or not the operation in Step S11 is to select the model image WG of the work W on the screen ( Step S24). If the model image WG of the workpiece W is not selected (N in Step S24), the reception unit 310 returns the process to Step S11.

  When the model image WG of the work W is selected (Y in Step S24), the display unit 320 highlights the model image WG of the work W on the screen (Step S25). The highlighting may be performed by making the color of the model image WG of the work W different from the color of the model image of other parts or the work W, for example.

  In this way, for example, as shown in FIG. 6 (B2), the model image WG of the workpiece W is selected and highlighted.

  Then, the reception unit 310 determines whether the model image HG of the hand 22 on the screen has been selected (step S26). When the model image HG of the hand 22 has not been selected (N in step S26), the reception unit 310 returns the process to step S11.

  When the model image HG of the hand 22 has been selected (Y in step S26), the display unit 320 displays the gripping operation of the hand 22 on the screen (step S27). Specifically, the display unit 320 acquires the tap position when the model image WG of the workpiece W is selected from the reception unit 310 as the gripping position of the workpiece W by the finger of the hand 22. In addition, the display unit 320 acquires the current position of the model image HG of the hand 22 displayed on the screen. Further, the display unit 320 moves from the current position to a position near the gripping position (a predetermined position that is a predetermined distance away from the gripping position), generates a model image HG of the hand 22 representing the operation of gripping the workpiece W, and sequentially displays it on the screen. To display.

  The teaching unit 330 generates teaching information corresponding to the operation displayed in step S27 (step S28), and the process returns to step S11. Specifically, the teaching unit 330 acquires, from the display unit 320, a predetermined position that is a predetermined distance away from the gripping position of the workpiece W by the finger of the hand 22 as a target position, and teaches to move the hand 22 (including the target position). ) Is generated. At this time, the teaching unit 330 converts the two-dimensional target position on the screen into a three-dimensional target position in the robot 2 and includes it in the teaching. Further, the teaching unit 330 generates a teaching (including a finger movement amount and a moving direction) for opening and closing the hand 22. At this time, the teaching unit 330 determines the amount and direction of movement of the finger based on the shape, position, orientation, and the like of the workpiece W recognized in step S3, and includes them in the teaching. The teaching unit 330 stores the generated teaching in the storage unit 300 as teaching information. Note that the content of the teaching to be generated is not limited to the above example. For example, a teaching that moves the hand 22 to a predetermined position that is a predetermined distance away from the gripping position, a teaching that opens the hand 22, a teaching that moves the hand 22 from the predetermined position to the gripping position, a teaching that closes the hand 22, etc. Good.

  In this way, when an operation for selecting the model image HG of the hand 22 and the model image WG of the work W is performed, for example, as shown in FIGS. 6B1 and 6B2, it is shown in FIG. 6B3. As described above, the model image HG representing the operation of moving the hand 22 from the current position to the gripping position and gripping the workpiece W is sequentially displayed. Further, teaching information corresponding to the display content is generated.

  On the other hand, when it is not a selection operation (N in step S21), the reception unit 310 determines whether or not the operation in step S11 is a drag operation (step S31 in FIG. 5).

  When it is a drag operation (Y in step S31), the reception unit 310 determines whether or not the start point of the drag operation in step S11 is on the model image HG of the hand 22 on the screen (step S32).

  When the starting point of the drag operation is on the model image HG of the hand 22 (Y in step S32), the accepting unit 310 determines whether the end point of the drag operation in step S11 is on the model image WG of the work W on the screen. Is determined (step S33).

  When the end point of the drag operation is on the model image WG of the workpiece W (Y in step S33), the display unit 320 displays the gripping operation of the hand 22 on the screen (step S34). Specifically, the display unit 320 acquires the end point position when the drag operation is performed as the gripping position of the workpiece W by the finger of the hand 22 from the reception unit 310. In addition, the display unit 320 acquires the current position of the model image HG of the hand 22 displayed on the screen. Further, the display unit 320 moves from the current position to a position near the gripping position (a predetermined position that is a predetermined distance away from the gripping position), generates a model image HG of the hand 22 representing the operation of gripping the workpiece W, and sequentially displays it on the screen. To display. Note that the display unit 320 may highlight the model image HG at the start position of the drag operation and the model image WG at the end position.

  The teaching unit 330 generates teaching information corresponding to the operation displayed in step S34 (step S35), and returns the process to step S11 (FIG. 4). Specifically, the teaching unit 330 acquires, from the display unit 320, a predetermined position that is a predetermined distance away from the gripping position of the workpiece W by the finger of the hand 22 as a target position, and teaches to move the hand 22 (including the target position). ) Is generated. At this time, the teaching unit 330 converts the two-dimensional target position on the screen into a three-dimensional target position in the robot 2 and includes it in the teaching. Further, the teaching unit 330 generates a teaching (including a finger movement amount and a moving direction) for opening and closing the hand 22. At this time, the teaching unit 330 determines the amount and direction of movement of the finger based on the shape, position, orientation, and the like of the workpiece W recognized in step S3, and includes them in the teaching. The teaching unit 330 stores the generated teaching in the storage unit 300 as teaching information. Note that the content of the teaching to be generated is not limited to the above example. For example, an instruction to move the hand 22 to a predetermined position away from the gripping position, an instruction to open the hand, an instruction to move the hand from the predetermined position to the gripping position, an instruction to close the hand, and the like may be generated.

  In this way, for example, as shown in FIG. 7B1, when the operation of dragging the model image HG of the hand 22 to the model image WG of the work W is performed, the hand 22 is moved as shown in FIG. 7B2. The model image HG representing the operation of moving from the current position to the gripping position and gripping the workpiece W is sequentially displayed. Further, teaching information corresponding to the display content is generated.

  On the other hand, when the end point of the drag operation is not on the model image WG of the workpiece W (N in step S33), the display unit 320 displays the movement operation of the hand 22 on the screen (step S36). Specifically, the display unit 320 acquires the end point position when the drag operation is performed from the reception unit 310 as the target position of the movement of the hand 22. In addition, the display unit 320 acquires the current position of the model image HG of the hand 22 displayed on the screen. In addition, the display unit 320 generates a model image HG of the hand 22 representing an operation of moving from the current position to the target position, and sequentially displays it on the screen. Note that the display unit 320 may highlight the model image HG at the starting point position of the drag operation.

  The teaching unit 330 generates teaching information corresponding to the operation displayed in step S36 (step S37), and the process returns to step S11 (FIG. 4). Specifically, the teaching unit 330 acquires the target position of movement of the model image HG of the hand 22 from the display unit 320 and generates teaching (including the target position) for moving the hand 22. At this time, the teaching unit 330 converts the two-dimensional target position on the screen into a three-dimensional target position in the robot 2 and includes it in the teaching. The teaching unit 330 stores the generated teaching in the storage unit 300 as teaching information.

  Thus, for example, as shown in FIG. 7 (C1), when an operation of dragging the model image HG of the hand 22 to an area other than the model image WG of the work W is performed, as shown in FIG. 7 (C2), Model images HG representing the movement of moving the hand 22 from the current position to the target position are sequentially displayed. Further, teaching information corresponding to the display content is generated.

  On the other hand, when the starting point of the drag operation is not on the model image HG of the hand 22 (N in step S32), the accepting unit 310 sets the start point of the drag operation on an area (other area) other than the model image WG of the workpiece W. It is determined whether or not there is (step S38). If the starting point of the drag operation is not on the other area (N in step S38), the reception unit 310 returns the process to step S11 (FIG. 4).

  If the starting point of the drag operation is on the other area (Y in step S38), the display unit 320 changes the viewpoint (step S39), and returns the process to step S11 (FIG. 4). Specifically, the display unit 320 acquires the start point position and the end point position when the drag operation is performed from the reception unit 310. The display unit 320 obtains the direction and distance from the start point position to the end point position. Further, the display unit 320 determines the position of the user's viewpoint based on a table or a mathematical expression in which a predetermined direction and distance of the drag operation are associated with a change direction and a change amount of the position and direction of the user's viewpoint. And change the orientation. In addition, the display unit 320 generates a three-dimensional model image when the current three-dimensional model image of the robot 2 and the workpiece W is viewed at a predetermined viewing angle from the changed position and orientation of the user's viewpoint. Then, the display unit 320 converts the generated three-dimensional model image of the robot 2 and the workpiece W into a two-dimensional model image and displays it on the operation terminal 4.

  Thus, for example, as shown in FIG. 7 (D1), when a drag operation is performed in an area other than the model image HG of the hand 22 or the model image WG of the work W, the viewpoint as shown in FIG. 7 (D2). Is changed to display the model image HG and the model image WG.

  On the other hand, when it is not a drag operation (N in step S31), the reception unit 310 determines whether or not the operation in step S11 is a rotation operation (step S41).

  When it is a rotation operation (Y in step S41), the reception unit 310 determines whether the model image HG of the hand 22 on the screen has been selected (step S42). When the model image HG of the hand 22 has not been selected (N in Step S42), the reception unit 310 returns the process to Step S11 (FIG. 4).

  When the model image HG of the hand 22 has been selected (Y in step S42), the display unit 320 displays the rotation operation of the hand 22 on the screen (step S43). Specifically, the display unit 320 acquires the rotation direction and the rotation amount when the rotation operation is performed from the reception unit 310. Further, the display unit 320 displays the joint based on a table or a mathematical expression in which a predetermined rotation direction and rotation amount of the rotation operation and a rotation direction and rotation amount of a predetermined joint closest to the hand 22 are associated with each other. Determine the rotation direction and amount of rotation. Further, the display unit 320 generates a model image HG of the hand 22 representing the operation of rotating the hand 22 based on the determined rotation direction and rotation amount, and sequentially displays them on the screen.

  The teaching unit 330 generates teaching information corresponding to the operation displayed in step S43 (step S44), and returns the process to step S11 (FIG. 4). Specifically, the teaching unit 330 acquires the rotation direction and rotation amount of a predetermined joint from the display unit 320, and generates a teaching (including the rotation amount and rotation direction of the joint) for rotating the hand 22. At this time, the teaching unit 330 converts the rotation amount and rotation direction of the joint on the screen into the rotation amount and rotation direction of the joint in the robot 2 and includes them in the teaching. The teaching unit 330 stores the generated teaching in the storage unit 300 as teaching information.

  Thus, for example, as shown in FIG. 8 (B2), when an operation of rotating the hand 22 is performed, model images HG representing the operation of rotating the hand 22 are sequentially displayed as shown in FIG. 8 (B3). Is displayed. Further, teaching information corresponding to the display content is generated.

  On the other hand, when it is not a rotation operation (N in step S41), the reception unit 310 determines whether the operation in step S11 is a pinch-in operation or a pinch-out operation (step S51). If it is not a pinch-in operation or a pinch-out operation (N in step S51), the accepting unit 310 returns the process to step S11 (FIG. 4).

  If the operation is a pinch-in operation or a pinch-out operation (Y in step S51), the reception unit 310 determines whether the model image HG of the hand 22 on the screen has been selected (step S52). When the model image HG of the hand 22 has not been selected (N in Step S52), the reception unit 310 returns the process to Step S11 (FIG. 4).

  When the model image HG of the hand 22 has been selected (Y in step S52), the display unit 320 displays an operation for closing or opening the hand 22 on the screen (step S53). Specifically, the display unit 320 acquires, from the reception unit 310, the amount of pinch-in movement or the amount of pinch-out movement when a pinch-in operation or a pinch-out operation is performed. Further, the display unit 320 is based on a predetermined table or formula that associates the amount of movement of pinch-in or pinch-out with the amount of movement of closing the finger of the hand 22 or the amount of movement of opening the finger. Determine the direction of finger movement (closed or opened) and the amount of movement. Further, the display unit 320 generates a model image HG image of the hand 22 representing an operation of closing or opening the hand 22 based on the determined moving direction and moving amount, and sequentially displays them on the screen.

  The teaching unit 330 generates teaching information corresponding to the operation displayed in step S53 (step S54), and returns the process to step S11 (FIG. 4). Specifically, the teaching unit 330 acquires the moving direction and moving amount of the finger from the display unit 320 and generates teaching (including the moving direction and moving amount) for moving the finger of the hand 22. At this time, the teaching unit 330 converts the finger movement direction and movement amount on the screen into the finger movement direction and movement amount in the robot 2 and includes them in the teaching. The teaching unit 330 stores the generated teaching in the storage unit 300 as teaching information.

  Thus, for example, as shown in FIG. 8 (C2), when a pinch-out operation of the hand 22 is performed, as shown in FIG. 8 (C3), a model image HG representing an operation of opening the finger of the hand 22 is displayed. Displayed sequentially. Further, teaching information corresponding to the display content is generated.

  When the flowcharts of FIGS. 4 and 5 are finished (Y in step S12), the control unit 350 generates a control command for operating the robot 2 based on the teaching information stored in the storage unit 300, Output to the robot 2. Of course, the control unit 350 may generate a control command and output it to the robot 2 when receiving an instruction from the user via the receiving unit 310.

  The embodiment of the present invention has been described above. According to the present embodiment, it is possible to more easily teach various operations of the robot through an input unit such as a touch screen.

  For example, in the present embodiment, an image of a robot representing an operation corresponding to various operations on the image is displayed and teaching corresponding to this operation is performed. Therefore, teaching of various operations of the robot can be performed more easily. It can be carried out. Further, for example, in the present embodiment, since an operation can be performed by a touch operation or the like on a screen on which an image is displayed, the operation of the robot can be taught more intuitively.

  Further, for example, in this embodiment, when teaching a gripping operation, the robot's gripping operation can be taught more easily by a moving operation such as dragging without inputting a detailed operation of the robot. Further, for example, in this embodiment, when teaching a gripping operation, it is possible to more easily teach the robot's gripping operation by a selection operation such as a tap operation without inputting a detailed operation of the robot. .

  Further, for example, in this embodiment, when teaching a gripping operation, the gripping position is determined while performing a selection operation such as a drag operation or a tap operation in order to set the contact position on the image of the object as the gripping position. Can be taught easily. Further, for example, in the present embodiment, an image of the target object is generated based on the captured image, so that an operation on the actual target object can be easily taught.

  Of course, the above-described embodiments of the present invention are intended to illustrate the spirit and scope of the present invention and are not intended to be limiting. For example, the following modifications may be added to the above embodiment.

  For example, in the above embodiment, the operation of the hand 22 can be taught, but the operation of the arm member 21b may be taught. Further, for example, the operation of the torso 20, the operation of the head 25, the operation of the imaging device 26, and the like may be taught. Further, the same can be applied to end effectors other than the hand 22.

  Further, for example, in the above embodiment, the work W is recognized from the captured image and the model image WG is displayed before accepting the user's operation (steps S3 and S4 in FIG. 4). It may be executed at an arbitrary timing by a user operation. Specifically, when the receiving unit 310 receives an image acquisition instruction (for example, a tap on an “image acquisition” button on the screen), the same processing as in steps S3 and S4 may be performed. In this way, it is possible to flexibly deal with changes in work content such as adding a work W to the work area, removing the work W from the work area, or changing the work W posture.

  Further, for example, when the reception unit 310 receives an instruction to display a camera image (for example, a tap of a “camera image” button on the screen), the display unit 320 acquires a current captured image by the image acquisition unit 340. Then, the acquired captured image may be displayed on the screen. In this way, the user can check the status of the work area.

  Further, for example, before the generation of teaching information (step S28 in FIG. 4, steps S35, S37, S44, S54, etc. in FIG. 5) is started, the display unit 320 determines whether or not to generate teaching information. A message to be confirmed may be displayed on the screen. In this case, when the reception unit 310 receives an instruction to generate teaching information, the teaching unit 330 generates teaching information. When the reception unit 310 receives an instruction not to generate teaching information, the reception unit 310 cancels the user operation related to the operation of the robot 2 performed last, and the display unit 320 displays the model image. May be returned to the state before the operation is performed and displayed on the screen.

  For example, in the above-described embodiment, the case where the touch screen of the operation terminal 4 is used as the input unit has been mainly described. However, various pointing devices such as a mouse connected to the operation terminal 4 may be used as the input unit. . In this case, the accepting unit 310 accepts an instruction position indicated by an instruction image such as a pointer on the screen operated by the pointing device, instead of the contact position P on the screen. Note that the input device 95 of the robot control device 3 may be used as the input means, and the display device 96 of the robot control device 3 may be used as the display means.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiment. In addition, it is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention. Various aspects of the provided aspects of the invention can also be employed. For example, the present invention may be provided as a robot including the function of the robot control device. Further, for example, a function related to teaching of the robot control device may be included in the teaching device, and provided as a robot system having the teaching device, the robot control device, and the robot separately. Further, for example, a robot system including a teaching device and a robot including the function of the robot control device may be provided. The present invention can also be provided as a robot control method, a program, a storage medium storing the program, and the like.

1: Robot system, 2: Robot, 3: Robot control device, 4: Operation terminal, 20: Body, 21: Arm, 21a: Joint, 21b: Arm member, 21c: Force sensor, 22: Hand, 24: Leg part, 25: head, 26: imaging device, 90: computer, 91: arithmetic device, 92: main storage device, 93: auxiliary storage device, 94: communication I / F, 95: input device, 96: display device 97: read / write device, 200: operation control unit, 300: storage unit, 310: reception unit, 320: display unit, 330: teaching unit, 340: image acquisition unit, 350: control unit, HG: model image of a hand, P: contact position, RG: model image of robot, T: work table, W: work, WG: model image of work

Claims (14)

A display for displaying an image of the robot;
A receiving unit that receives an operation on the image of the robot displayed by the display unit;
A teaching unit that teaches the operation of the robot,
The display unit displays an image indicating movement of the gripping unit of the robot, rotation of the gripping unit, or opening / closing of the gripping unit according to the operation,
The teaching unit teaches the robot to move the gripping part of the robot, rotate the gripping part, or open / close the gripping part according to the operation. The robot control device according to claim 1,
The said reception part is a robot control apparatus which receives the motion of the contact position on a screen as said operation. The robot control device according to claim 2,
When the reception unit receives an operation of moving the contact position from the image of the gripping unit to the image of the object,
The display unit displays an image representing an operation of moving and gripping the gripping unit to the object;
The teaching unit is a robot control device that teaches the robot an operation of moving and gripping the gripping unit to the object. The robot control device according to claim 2,
When the receiving unit receives an operation of selecting an image of an object in a state where the image of the gripping unit is selected,
The display unit displays an image representing an operation of moving and gripping the gripping unit to the object;
The teaching unit is a robot control device that teaches the robot an operation of moving and gripping the gripping unit to the object. The robot control device according to claim 3 or 4,
The receiving unit receives the contact position on the image of the object as a gripping position;
The display unit displays an image representing an operation of moving the gripping unit to the gripping position and gripping;
The robot teaching apparatus, wherein the teaching unit teaches the robot to move and grip the gripping unit to the gripping position. The robot control device according to any one of claims 3 to 5,
An image acquisition unit that acquires a captured image obtained by imaging the object;
The said control part is a robot control apparatus which produces | generates and displays the image of the said target object based on the said captured image. The robot control device according to any one of claims 2 to 6,
When the accepting unit accepts an operation for moving the contact position so as to draw an arc,
The display unit displays an image representing an operation of rotating the gripping unit;
The teaching unit is a robot control device that teaches the robot to perform an operation of rotating the gripping unit. The robot control device according to any one of claims 2 to 6,
The contact position includes a first contact position and a second contact position,
When the accepting unit accepts an operation of bringing the first contact position and the second contact position closer or away from each other,
The display unit displays an image representing an operation of closing or opening the gripping unit;
The teaching unit is a robot control device that teaches the robot to close or open the gripping unit. The robot control device according to any one of claims 2 to 6,
When the accepting unit accepts an operation for moving the contact position in a predetermined area on the screen,
The robot controller according to claim 1, wherein the display unit displays an image of the grip unit by changing a viewpoint. The robot control device according to any one of claims 1 to 9,
The robot control apparatus, wherein the reception unit receives, as the operation, a movement of an instruction position indicated by an instruction image on the screen operated by a pointing device instead of the movement of the contact position on the screen. A robot,
A display unit for displaying an image of the robot;
A receiving unit that receives an operation on the image of the robot displayed by the display unit;
A teaching unit that teaches the operation of the robot,
The display unit displays an image indicating movement of the gripping unit of the robot, rotation of the gripping unit, or opening / closing of the gripping unit according to the operation,
The teaching unit teaches the robot to move the gripping part of the robot, rotate the gripping part, or open / close the gripping part according to the operation. A robot system comprising a robot and a robot control device that controls the robot,
A display unit for displaying an image of the robot;
A receiving unit that receives an operation on the image of the robot displayed by the display unit;
A teaching unit that teaches the operation of the robot,
The display unit displays an image indicating movement of the gripping unit of the robot, rotation of the gripping unit, or opening / closing of the gripping unit according to the operation,
The teaching unit teaches the robot to move the gripping part of the robot, rotate the gripping part, or open / close the gripping part in accordance with the operation. A robot motion teaching method comprising:
A display step for displaying an image of the robot;
An accepting step of accepting an operation on the image of the robot displayed in the displaying step;
A teaching unit step for teaching the operation of the robot,
The display step displays an image representing movement of the gripping part of the robot, rotation of the gripping part, or opening / closing of the gripping part according to the operation,
In the teaching method, the teaching step teaches the robot to move the gripping part of the robot, rotate the gripping part, or open / close the gripping part in accordance with the operation. A program for a robot controller,
A display for displaying an image of the robot;
A receiving unit that receives an operation on the image of the robot displayed by the display unit;
As the teaching unit for teaching the operation of the robot, the robot control device is caused to function,
The display unit displays an image indicating movement of the gripping unit of the robot, rotation of the gripping unit, or opening / closing of the gripping unit according to the operation,
The teaching unit teaches the robot to move the gripping unit of the robot, rotate the gripping unit, or open / close the gripping unit according to the operation.

Images