WO2022014133A1 - Mobile manipulator, method for controlling same, program - Google Patents

Abstract

Provided is a mobile manipulator equipped with: a dolly part equipped with a movement means that moves towards a dolly target stop attitude and a dolly target stop position in the vicinity of a placement body; a manipulator equipped with a hand-end camera at the tip or in the vicinity of the tip; a relative relationship recognition unit for recognizing the relative distance and relative attitude of the hand-end camera with respect to the placement body after a stop; an object recognition unit for recognizing an object located in the field-of-view of the hand-end camera after a stop; and a reaching control unit. If the recognized object includes a target object, the reaching control unit performs reaching operation control on a reaching operation of the manipulator with respect to the target object on the basis of the relative distance and the relative attitude. If the recognized object does not include the target object, the reaching control unit acquires position information for the recognized object, moves the hand-end position of the manipulator on the basis of the position information, and then performs reaching operation control on a reaching operation of the manipulator with respect to the target object on the basis of the relative distance and the relative attitude.

Description

Mobile manipulators and their control methods and programs

The present invention relates to a mobile body equipped with a manipulator and autonomously moving, for example, a mobile manipulator or the like.

In recent years, mobile manipulators that move autonomously for the purpose of transporting goods in factories, warehouses, etc. are attracting attention. As an example of a mobile robot provided with a manipulator, Patent Document 1 discloses a mobile robot that transports an article acquired from an article storage device.

Japanese Patent No. 4007204

By the way, in order to transport an object to a predetermined destination, the mobile manipulator moves to the vicinity of a shelf or the like on which the target object is placed while performing self-position estimation or the like, and grips the target object. It is necessary to acquire it by doing so, move to the next destination again while holding it, and place the target object.

However, in a mobile manipulator that performs autonomous movement while performing self-position estimation in a relatively large space such as a factory or warehouse, an error in self-position estimation and a control error in the dolly used for movement may occur. There is. Due to these errors, an error occurs between the target stop position / posture of the mobile manipulator and the actual stop position / posture in the vicinity of the shelf, etc., and as a result, subsequent operations such as leaching and gripping fail. There was a risk that it would end up.

The present invention has been made to solve the above-mentioned technical problems, and an object of the present invention is to ensure subsequent operations such as leaching even if an error occurs in the stop position / posture after movement. The purpose is to provide mobile manipulators and the like that can be achieved.

The above technical problem can be solved by a mobile manipulator or the like having the following configuration.

That is, the mobile manipulator according to the present invention is a mobile manipulator that moves to the vicinity of the mounted body and reaches the target object on the previously described stationary body, and is a trolley target stop position and a trolley target stop position near the previously described stationary body. The relative distance and the relative posture of the trolley part equipped with a moving means for moving toward the trolley target stop posture, the manipulator equipped with the hand camera at the tip or near the tip, and the hand camera after stopping with respect to the previously described object. Recognizing, relative relationship recognition unit, object recognition unit that recognizes an object placed in the field of view of the hand camera after stopping, and when the recognized object includes the target object, the relative distance and the relative The reach operation control of the manipulator with respect to the target object is performed based on the posture, and when the recognized object does not include the target object, the position information of the recognized object is acquired, and the position information is obtained based on the position information. It is provided with a reaching control unit that controls the leaching operation of the manipulator with respect to the target object based on the relative distance and the relative posture after moving the hand position of the manipulator.

According to such a configuration, it is possible to provide a mobile manipulator or the like that can surely achieve a subsequent operation such as leaching even if an error occurs in the stop position / posture after movement.

It further comprises a reach preparation operation control unit that controls the manipulator to be in the reach preparation distance and the reach preparation posture based on the relative distance and the relative posture between the hand camera and the above-mentioned relative body. There may be.

According to such a configuration, various subsequent operations can always be performed starting from the reach preparation distance and the reach preparation posture.

The bogie target stop position and the bogie target stop posture may be generated based on the reach preparation distance and the reach preparation posture.

According to such a configuration, since the stop position and posture of the bogie are generated backward from the position and posture suitable for leaching, stable leaching, gripping and the like can be realized after movement.

The leaching preparation distance and the leaching preparation posture may be generated based on the position of the target object and the relative relationship between the preset pre-described body and the hand.

According to such a configuration, the optimum preparation distance and preparation posture for leaching are generated from the position of the target object, so that stable leaching, gripping, and the like can be realized.

The recognition of the target object by the hand camera may be performed at the reach preparation distance and the reach preparation posture.

According to such a configuration, it is possible to recognize the target object under a constant positional relationship, so that the recognition accuracy of the object can be improved.

The moving means may be controlled based on the result of the self-position estimation process, which is provided with a self-position estimation unit that performs the self-position estimation process by the environment recognition means.

According to such a configuration, it is possible to provide a mobile manipulator or the like that can surely achieve the subsequent operation such as leaching even if some error occurs in the stop position or the posture due to the error of self-position estimation or the like. can.

The environmental recognition means may be a rider.

With such a configuration, self-position estimation can be performed more easily and with high accuracy than incorporating a camera system or the like.

The hand camera may be a monocular camera.

With such a configuration, a hand camera can be realized at low cost.

It may be further provided with a gripping control unit that controls gripping of the target object after the leaching operation control.

According to such a configuration, the target object can be gripped, so that it can be transported or the like.

The gripping control unit may further include a gripping position determining unit that determines the gripping position of the target object based on the image acquired by the hand camera.

According to such a configuration, the gripping position of the target object can be appropriately determined, so that the possibility of gripping failure can be reduced.

The gripping position determining unit further comprises a gripping position estimation unit including a trained model machine-learned to estimate the gripping position of the target object based on an image acquired by the hand camera. You may.

According to such a configuration, the gripping position of the target object can be adaptively determined by the action of machine learning, so that the possibility of gripping failure can be reduced.

The movement of the hand position of the manipulator may be performed in parallel with the previously described body.

According to such a configuration, since the relative relationship between the hand and the mounted body is not changed, the hand can be moved in a manner suitable for the subsequent leaching operation.

It may be provided with a bogie movement control unit that moves the bogie unit for moving the hand position of the manipulator.

According to such a configuration, reaching and the like can be realized even when the target object is outside the reach of the manipulator.

The relative relationship recognition unit further determines the relative distance and the relative posture between the hand camera and the previously described body based on the image information including the first marker provided on the target object or the previously described body. It may be one that recognizes and is provided with a marker recognition unit.

According to such a configuration, it is possible to recognize the relative distance and the relative posture between the hand camera and the mounted body with high accuracy by using the marker.

A plurality of the first markers may be arranged on the target object or the above-mentioned predicate.

According to such a configuration, it is possible to recognize the relative relationship with higher accuracy by using a plurality of markers.

The relative distance and the relative posture may be recognized based on the information regarding the plurality of the first markers and the constraint conditions based on the shape of the above-mentioned figurine.

According to such a configuration, for example, when the marker is provided on the front surface of the shelf, the marker is present on a plane perpendicular to the floor surface by adding a constraint condition to the hand camera and the above-mentioned object. The recognition of the relative distance and the relative posture between them becomes more accurate.

The image information including the first marker may be an image captured by the hand camera.

According to such a configuration, it is not necessary to provide another camera or the like for capturing the first marker, so that the cost can be reduced.

The image information including the plurality of the first markers is imaged by pointing the hand camera obliquely with respect to the plurality of the first markers so as to keep the plurality of the first markers in the field of view. It may be obtained by.

According to such a configuration, by aiming the hand camera at an angle with respect to the plurality of first markers, it is possible to efficiently take an image even when a hand camera having a narrow field of view is used. ..

The mobile manipulator further comprises a camera arranged on the manipulator so as to have a wider field of view than the hand camera, and the image information including the first marker is captured by the camera. May be good.

According to such a configuration, a plurality of first markers can be easily imaged in a wide field of view, so that rapid imaging can be performed.

The mobile manipulator may further include a head, and the camera may be provided on the head.

According to such a configuration, a plurality of first markers can be easily imaged by using a camera that can image a wide field of view from the head.

The trolley unit may be an omnidirectional moving trolley.

With such a configuration, it is easy to move in a narrow place or move in parallel to a shelf.

The manipulator is an articulated arm, the articulated arm includes one or a plurality of force sensors at its tip, its joint or its root, and the leaching motion control unit further has a certain or more force sensors. It may be provided with a leaching stop control unit that controls to stop the leaching operation when a value is detected.

According to such a configuration, since the leaching operation is performed until it comes into contact with the target object, it is possible to surely realize the subsequent gripping of the target object.

The manipulator includes an articulated arm and an end effector provided at the tip of the articulated arm, and the end effector includes a force sensor and / or a contact sensor on a contact surface with the object to control the leaching motion. The unit may include a leaching stop control unit for an effector that controls the leaching operation to be stopped when a value equal to or higher than a certain value is detected by the force sensor and / or the contact sensor.

According to such a configuration, since the gripping operation is performed until it comes into contact with the target object, it is possible to reliably grip the target object and the like.

The manipulator may include an arm portion and a vertical movement mechanism provided at the base of the arm portion.

According to such a configuration, the manipulator can be moved including the vertical movement, so that the manipulator can be easily moved in parallel, and as a result, the leaching operation to the target object becomes easier.

The object recognition in the object recognition unit may be performed by capturing the two-dimensional code information provided in the object.

With such a configuration, it is possible to reliably recognize an object based on a two-dimensional code.

The reaching operation control unit further receives the transmission unit that transmits the recognized identification information of the object to a predetermined data server via a network, and the position information corresponding to the identification information from the data server. , A receiver, and may be provided.

According to such a configuration, it is possible to accurately generate control information up to the target object by using a data server that centrally manages the position information of the object.

The present invention can also be considered as a method. That is, the method according to the present invention is a method for controlling a mobile manipulator that moves to the vicinity of a mounted body and reaches a target object on the previously described body, and the manipulator is near the previously described body. The control method comprises a trolley portion provided with a moving means for moving toward a target stop position of the trolley and a target stop posture of the trolley, and a manipulator equipped with a hand camera at the tip or near the tip. The relative relationship recognition step of recognizing the relative distance and the relative posture of the hand camera with respect to the previously described object, the object recognition step of recognizing the object placed in the field of view of the hand camera after stopping, and the recognized object. When the object includes the target object, the reach operation control of the manipulator with respect to the target object is performed based on the relative distance and the relative posture, and when the recognized object does not include the target object, the recognized object is used. Reaching control step that acquires the position information of the manipulator, moves the hand position of the manipulator based on the position information, and then controls the leaching operation of the manipulator with respect to the target object based on the relative distance and the relative posture. And have.

Furthermore, the present invention can also be thought of as a program. That is, the program according to the present invention is a control program of a mobile manipulator that moves to the vicinity of the mounted body and reaches the target object on the previously described body, and the manipulator is near the previously described body. The control program comprises a trolley unit equipped with a moving means for moving toward a trolley target stop position and a trolley target stop posture, and a manipulator equipped with a hand camera at or near the tip, and the control program is described after the stop. The relative relationship recognition step of recognizing the relative distance and the relative posture of the hand camera with respect to the previously described object, the object recognition step of recognizing the object placed in the field of view of the hand camera after stopping, and the recognized object. When the object includes the target object, the reach operation control of the manipulator with respect to the target object is performed based on the relative distance and the relative posture, and when the recognized object does not include the target object, the recognized object is used. Reaching control step that acquires the position information of the manipulator, moves the hand position of the manipulator based on the position information, and then controls the leaching operation of the manipulator with respect to the target object based on the relative distance and the relative posture. And have.

According to the present invention, it is possible to provide a mobile manipulator or the like that can surely achieve a subsequent operation such as leaching even if an error occurs in a stop position / posture after movement.

FIG. 1 is an overall configuration diagram of the system. FIG. 2 is an external perspective view of the mobile manipulator. FIG. 3 is a functional block diagram of a mobile manipulator that moves autonomously. FIG. 4 is a functional block diagram of a mobile manipulator that controls the arm portion. FIG. 5 is an explanatory diagram regarding the layout in the factory. FIG. 6 is a general flowchart regarding the operation of the mobile manipulator. FIG. 7 is a detailed flowchart of the target setting process. FIG. 8 is an explanatory diagram regarding the target position and the target posture of the hand camera. FIG. 9 is a detailed flowchart of the movement process. FIG. 10 is an explanatory diagram showing a state of the mobile manipulator when it is determined that the target position and posture of the bogie have been reached. FIG. 11 is a detailed flowchart of the picking process. FIG. 12 is a perspective view of the mobile manipulator and the shelf after the movement process. FIG. 13 is a conceptual diagram relating to the process of calculating the relative distance and posture of the hand camera with respect to the shelf. FIG. 14 is an explanatory diagram showing a state of the mobile manipulator after the control process of the hand camera. FIG. 15 is an explanatory diagram showing a state after moving the hand camera to the target position and posture. FIG. 16 is a perspective view of a mobile manipulator that performs a translation operation of the hand camera.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(1. First Embodiment)
As a first embodiment, an example in which the present invention is applied to a mobile manipulator that autonomously moves in a factory to perform work will be described. The present invention is not limited to such a device, and can be applied to various environments other than factories.

(1.1 Configuration)
The configuration of the system 500 according to the present embodiment will be described with reference to FIGS. 1 to 4.

FIG. 1 is an overall configuration diagram of the system 500. As is clear from the figure, the system 500 is configured by connecting the mobile manipulator 100, the data server 200, and the client device 300 to each other via a LAN in the factory. Although one is shown in the figure, the present invention is not limited to such an example, and a plurality of each device may be provided.

The data server 200 is composed of an information processing device, stores various information described later, and provides the data server 200 in response to a request from the mobile manipulator 100 or the client device 300. For example, it manages the object transportation command and setting information, the book ID, the book position information, etc., which will be described later, and provides the corresponding information in response to a request from the mobile manipulator 100 or the like.

The data server 200 communicates with a control unit including a CPU or the like for executing various programs, a storage unit including a ROM, RAM or flash memory for storing the executed program and various data, and an external device. It is provided with a communication unit for performing, an input unit for processing input from a keyboard, a mouse, and the like, a display unit for displaying various images, and the like, and they are connected to each other via a bus.

The client device 300 is composed of an information processing device, and in cooperation with the data server 200, provides management information about the system 500 to the system administrator and the like, and as will be described later, various types of the mobile manipulator 100 related to the user. It is possible to input settings.

The client device 300 communicates with a control unit including a CPU or the like for executing various programs, a storage unit including a ROM, RAM or flash memory for storing the executed program and various data, and an external device. It is provided with a communication unit for performing, an input unit for processing input from a keyboard, a mouse, and the like, a display unit for displaying various images, and the like, and they are connected to each other via a bus.

FIG. 2 is an external perspective view of the mobile manipulator 100. As is clear from the figure, the mobile manipulator 100 of the present embodiment has a substantially human-shaped shape, and includes a robot main body 10, a mobile trolley 20 that supports the robot main body 10, and a robot main body 10. It is composed of a head portion 30 provided at the upper end and an arm portion 40 extending from the front surface of the robot main body portion 10.

The mobile trolley 20 that supports the robot main body 10 is an omnidirectional mobile trolley having a function of moving on the surface (floor surface) on which the mobile manipulator 100 is placed. The omnidirectional moving trolley is a trolley provided with, for example, a plurality of omni wheels as driving wheels and configured to be able to move in all directions. The omnidirectional mobile trolley may be referred to as an omnidirectional trolley, an omnidirectional mobile trolley, or an omnidirectional trolley. In addition, the omnidirectional moving trolley can move 360 degrees in all directions and can move freely even in a narrow passage or the like. As shown in the figure, the mobile carriage 20 of the present embodiment has three drive wheels 22 and a moving mechanism driving means 2 for driving the drive wheels 22 via a belt or the like inside the skirt 21 represented on the outside. It may be composed of and. The moving mechanism driving means 2 of the present embodiment is composed of one or a plurality of motors, and hereinafter, the moving mechanism driving means 2 is also referred to as a motor 2.

The head 30 may be provided at the upper end of the robot main body 10 as a configuration corresponding to a human head (a portion above the neck) in the approximately humanoid mobile manipulator 100. The head 30 is configured so that the front surface of the head 30 can be recognized as a face when viewed by a person. The head 30 is configured to be able to control the direction in which the front faces. Specifically, the mobile manipulator 100 of the present embodiment includes one or a plurality of head driving means 3 (actuators 3), and controls the direction of the front surface thereof by moving the head via the head driving means 3. It is configured to be able to. In the following, the front surface of the head 30 will be referred to as a face portion 31.

A head camera 35 is provided on the upper part of the face portion 31. According to the head camera 35, it is possible to take an image in a wider field of view than the hand camera 42 described later due to the arrangement thereof.

The mobile manipulator 100 of the present embodiment has a vertical rotation axis that rotates the head 30 in the horizontal direction (horizontal direction) with respect to the floor surface, and enables the head 30 to be rotationally driven in the left-right direction. It has a unit driving means 3 (servomotor 3a). As a result, the mobile manipulator 100 can rotate and drive only the head 30 to the left and right with respect to the vertical direction. However, the arrangement of the servomotor 3a is not limited to the arrangement shown in the figure, and may be appropriately changed as long as the direction of the face portion 31 can be moved in the left-right direction. For example, the servomotor 3a may be provided so as to rotationally drive the robot main body 10 to the left and right with respect to the moving carriage 20. In this case, the servomotor 3a can change the direction of the face portion 31 by rotationally driving the head portion 30 together with the robot main body portion 10 to the left and right in the vertical direction.

Further, the mobile manipulator 100 has a horizontal rotation axis that drives the head 30 in the vertical direction with respect to the floor surface, and the head driving means 3 (servo motor) that enables an operation of looking up and down of the head 30. 3b) may be provided. As a result, the mobile manipulator 100 can also move the direction of the face portion 31 in the vertical direction.

An environment recognition means 32 is provided in the protrusion of the front upper edge of the mobile trolley 20. In the present embodiment, the environment recognition means 32 is a lidar (LiDAR: Light Detection And Ranking) unit that measures a distance and a direction to an object by using a laser beam. However, the environment recognition means 32 is not limited to the rider unit, and various other sensors and the like can be adopted. For example, an image sensor may be provided to recognize the environment by image processing, or a radar, a microphone array, or the like may be used.

The mobile manipulator 100 is provided with an arm portion 40 on the front surface of the robot body portion 10. The arm portion 40 is provided with a manipulator mechanism, a gripping mechanism 41 for gripping an article to be transported, and in the present embodiment, a gripper is provided at a tip corresponding to a free end. Further, a monocular hand camera 42 is provided near the base of the gripping mechanism 41. With this hand camera 42, it is possible to take an image of a gripping object or the like facing the gripping mechanism 41 from the front.

However, the shape, number, and arrangement of the arm portions 40 are not limited to the illustrated embodiments and may be appropriately changed according to the purpose. For example, the mobile manipulator 100 may be configured as a dual-arm robot having arms 40 on both side surfaces of the robot body 10.

The above is a configuration example of the mobile manipulator 100 of this embodiment. Although omitted in FIG. 2, the mobile manipulator 100 includes other configurations required for controlling the operation of the mobile manipulator 100, for example, other actuators for driving the arm portion 40 and the like, and a power source. A battery or the like may be provided.

FIG. 3 is a functional block diagram of the mobile manipulator 100 that moves autonomously. As is clear from the figure, the environmental information obtained from the environmental recognition unit 32 composed of riders is acquired by the environmental information acquisition unit 101 and provided to the mobile vehicle control unit 102. Further, the mobile trolley 20 is provided with an encoder that detects the rotation of each drive wheel 22 (not shown), and the value from this encoder is read out by the encoder information acquisition unit 106 to the mobile trolley control unit 102. Provided.

The mobile trolley control unit 102 performs various processes described later based on the environmental information and the encoder information to generate control information for the mobile trolley 20, and controls the mobile mechanism driving means 2.

FIG. 4 is a functional block diagram of the mobile manipulator 100 when controlling the arm portion 40. As is clear from the figure, the manipulator control unit 113 controls one or a plurality of actuators 118 including a servomotor or the like that controls the operation of the arm unit 40. Further, the arm portion 40 is provided with a sensor 116 including an encoder (not shown), a torque sensor, or the like that senses the movement of each arm, and information from the sensor 116 is acquired by the sensor information acquisition unit 117 to control the manipulator. Provided to unit 113.

The hand camera information acquisition unit 110 acquires image information from the hand camera 42 and provides it to the manipulator control unit 113. Further, the head camera information acquisition unit 111 acquires image information from the head camera 35 and provides it to the manipulator control unit 113. That is, the manipulator control unit 113 can control the arm unit 40 based on the image information acquired from each camera (35, 42).

(1.2 operation)
Next, the operation of the system 500 according to the present embodiment will be described.

FIG. 5 is an explanatory diagram regarding the layout in the factory 800 in which the mobile manipulator 100 operates. First, an outline of the operation according to the present embodiment will be described with reference to the figure.

In the initial state, the mobile manipulator 100 exists at the lower left position in the figure. In this state, when the mobile manipulator 100 receives the transport command of the target object 601a in the present embodiment, the mobile manipulator 100 starts moving and stops in front of the shelf 600 arranged in the upper right of the figure. The transport command of the present 601a is set by the client device 300 and stored in the data server 200.

After the mobile manipulator 100 is stopped, the picking operation described later is performed to acquire the book 601a. The mobile manipulator 100 holding the book 601a then moves to the front of the work table 700 arranged at the lower right of the figure, stops, and releases the book 601a onto the work table. This ends the series of operations. Although the book is exemplified as a target object in the present embodiment, the present invention is not limited to such a configuration. Therefore, for example, in the case of a factory, the target object may be a material, a work tool, or the like used for manufacturing.

FIG. 6 is a general flowchart regarding the operation of the mobile manipulator 100. As is clear from the figure, when the process starts, the mobile manipulator 100 performs a process of requesting setting information to the data server 200 via the LAN in the factory 800 and a process of acquiring the setting information (S1). Here, in the present embodiment, the setting information includes the relative relationship information between the hand camera 42 and the shelf 600 and the target object information.

The relative relationship information is information on the relationship between the hand camera 42 and the shelf 600 after moving to the front of the shelf 600 and before the leaching operation, that is, the relative distance and the relative posture between the hand camera 42 and the shelf 600. Contains information. For example, the information is such that the relative distance is 200 mm and the relative posture is 180 °. The relative posture of 180 ° means a state in which the hand faces or faces the shelf 600.

The target object information includes information about an object held or carried by the mobile manipulator 100, that is, a book ID which is identification information of a book in the present embodiment and a book position information indicating a position of the book in the factory 800. There is.

In the present embodiment, the setting information is acquired from the data server 200, but the present invention is not limited to such a configuration, and for example, the mobile manipulator 100 may store the setting information in advance. ..

When the mobile manipulator 100, which has been in the standby state (S2NO) until receiving the setting information, receives the setting information from the data server 200 (S2YES), performs the target setting process (S3).

FIG. 7 is a detailed flowchart of the target setting process. As is clear from the figure, when the process starts, the book ID of the book which is the target object and the book position information are read from the setting information (S31).

After that, a process of calculating the target position and the target posture of the hand camera 42 before the leaching operation is performed based on the book position information (S32). That is, the position and posture of the hand camera 42 when the relative distance from the book, which is the target object, is 200 mm and the relative posture is 180 ° are calculated.

When the target position and target posture of the hand camera 42 are calculated, the target position and target posture of the moving trolley 20 are calculated so as to calculate back from the target position and target posture (S34). At this time, a predetermined reference posture is preset in the mobile manipulator 100. In this embodiment, this reference posture is set to a predetermined posture in which the arm 40 is folded to a certain extent in consideration of safety of movement.

After that, the process of storing the calculation result in the storage unit is performed, and the target setting process ends (S36).

FIG. 8 is an explanatory diagram regarding the target position and the target posture of the hand camera 42. The mobile manipulator 100 in the initial state is drawn in the lower left of the figure, and the shelf 600 and the book 601a of the target object arranged on the shelf 600 are drawn in the upper right of the figure. Further, in the figure, the initial position and the initial posture of the hand camera 42 are indicated by two orthogonal arrows C1, and the initial position and the initial posture of the mobile carriage 20 are also indicated by the two orthogonal arrows C2. Has been done.

Further, the calculated target position and target posture of the hand camera 42 are indicated by two orthogonal arrows C3 in front of the shelf 600, and further, the target position and target posture of the mobile trolley 20 are two orthogonal. It is indicated by the arrow C4. As will be described later, the mobile manipulator 100 autonomously moves toward the target position and target posture of the mobile vehicle 20.

Returning to FIG. 6, when the target setting process (S3) is completed, the mobile manipulator 100 is then moved (S4).

FIG. 9 is a detailed flowchart of the movement process (S4). As is clear from the figure, when the process starts, the environment information acquisition unit 101 acquires the environment information by using the environment recognition means 32 and performs the process of providing the environment information to the mobile trolley control unit 102 (S41).

Upon acquiring the environmental information, the mobile trolley control unit 102 then performs recognition processing of the local environment around itself based on the environmental information, and estimates the self-position on the global environment map generated in advance for the factory 800. Processing is performed (S43). As this self-position estimation process, various methods known to those skilled in the art can be applied.

After this self-position estimation process is performed, the mobile trolley control unit 102 performs a process (pass planning) for planning a movement route (S44). In the present embodiment, a global map and a local map around the mobile manipulator 100 are generated from the environmental information, and a route from the estimated self-position to the end point is generated. More specifically, a route is planned in which the cost defined by the distance from the wall or the like is equal to or less than a predetermined threshold value and the arrival time to the end point is minimized. However, the route planning method is not limited to such a method, and any method known to those skilled in the art may be used.

After the route planning process is completed, the mobile trolley control unit 102 controls the movement control process (SS46), that is, the process of controlling the moving mechanism driving means 2 so as to move along the route planned by the route planning process (S44). I do.

After that, when the movement control process is completed, a process of determining whether or not the moving trolley 20 has reached the target position and posture is performed (S47). That is, the self-position estimation process is performed, and the determination process of whether or not the target position and the posture of the moving trolley 20 are matched or sufficiently approached is performed. (S41 to S47) is repeated. On the other hand, when it is determined that the target position and the posture of the moving carriage 20 match or are sufficiently approached (S47YES), the route movement processing ends and stops.

FIG. 10 is an explanatory diagram showing the state of the mobile manipulator 100 when it is determined that the mobile trolley 20 has reached the target position and posture. As is clear from the figure, although the mobile manipulator 100 determines that the target position and posture of the mobile trolley 20 have been reached based on the self-position estimation, the position and posture of the mobile trolley 20 and the hand camera 42 are actually obtained. The position and posture of are deviated from the target position and posture, respectively.

That is, the actual position and posture of the hand camera 42 shown by C1 in the figure deviates from the target position and posture of the hand camera 42 shown by C3. Further, the actual position and posture of the mobile trolley 20 shown by C2 in the figure deviates from the target position and posture of the mobile trolley 20 shown by C4. For example, if the picking process or the like is performed on the 601a in this state, there is a possibility that the picking process will fail.

Returning to FIG. 6, when the movement process is completed, the picking process (S6) is performed next.

FIG. 11 is a detailed flowchart of the picking process. As is clear from the figure, when the process starts, first, the process of recognizing the relative relationship between the mobile manipulator 100 and the shelf 600 is performed (S61).

FIG. 12 is a perspective view of the mobile manipulator 100 and the shelf 600 after the movement process. As is clear from the figure, in the present embodiment, AR markers 603 are arranged at predetermined intervals on the surface of the shelf 600 facing the mobile manipulator 100. As is clear from the enlarged view in the figure, the AR marker 603 is two-dimensional code information. Further, a QR code (registered trademark) 604 composed of a two-dimensional dot pattern is arranged at the lower part of the spine cover 601x of the book arranged on the shelf.

In order to recognize the relative relationship, the mobile manipulator 100 uses the head camera 35 to capture a plurality of AR markers 603 in the field of view and take an image. Thereby, the relative distance and posture between the head camera 35 and the shelf 600 can be acquired based on the sizes and angles of the plurality of AR markers 603 in the field of view.

FIG. 13 is a conceptual diagram relating to the process of calculating the relative distance and posture of the hand camera 42 with respect to the shelf 600 based on the plurality of AR markers 603. By using a plurality of AR markers 603, the calculation accuracy can be improved as compared with the use of one AR marker.

First, as shown in the figure on the left side of the figure, the manipulator control unit 113 sets a plane equation perpendicular to the floor surface based on the positions and postures of the six AR markers 603 acquired by the head camera 35. The equation of a plane is set in this way based on the assumption that the front surface of the shelf 600 is a plane perpendicular to the floor surface. After that, as shown in the figure on the right side, the relative distance and the relative posture of the hand camera 42 with respect to the set plane are geometrically calculated. In this way, the actual positional relationship of the hand camera 42 with respect to the shelf 600 can be grasped. The calculation of the relative relationship is not limited to the method of the present embodiment, and various known methods can be adopted.

Returning to FIG. 11, when the recognition process of the relative relationship with the shelf 600 is completed, the manipulator control unit 113 controls the arm portion 40 without moving the moving carriage 20 and designates the hand camera 42 by the setting information. The relative distance and the relative posture, that is, the process of facing the hand camera 42 at a position 200 mm from the shelf 600 is performed (S62).

After the control of the hand camera 42, the process of recognizing a book existing in the front or near the front of the hand camera 42 in the field of view, that is, arranged on the spine of the book placed on the shelf board of the shelf 600. A process of recognizing the QR code 604 and detecting the book ID is performed (S63).

The manipulator control unit 113 receives the position information of the book currently in the field of view from the data server 200 by inquiring the result of this recognition process to the data server 200 (S65).

After that, the manipulator control unit 113 performs a process of determining whether or not the book existing in front of or near the hand camera 42 is the target book (S66). As a result of this determination process, when the book existing in front of or near the hand camera 42 is the target book 601a (S66YES), the gripping position specifying process (S74) described later is performed.

As a result of the determination process, if the book existing in front of or near the hand camera 42 is not the target book 601a (S66NO), the hand is targeted along the front of the shelf 600 based on the position information acquired from the data server 200. A process of translating toward the book 601a is performed (S68). After this predetermined translation, the process of recognizing the book in the field of view of the hand camera 42 by the QR code 604 is performed again (S69), and the process of inquiring the recognition result to the data server 200 is performed (S70).

After that, a process of determining whether or not the book existing in front of or near the hand camera 42 is the target book 601a is performed again (S72). As a result of this determination process, when the book existing in front of or in the vicinity of the hand camera 42 is the target book 601a (S72YES), the gripping position specifying process (S74) described later is performed. On the other hand, as a result of the determination process, if the book existing in front of or near the hand camera 42 is not the target book 601a (S72NO), a series of processes (S68 to S70) are performed again.

FIG. 14 is an explanatory diagram showing an example of the state of the mobile manipulator 100 after the control process (S62) of the hand camera. In the example of the figure, the book 601b, which is not the target book 601a, is arranged in front of or near the hand camera 42.

In this case, the mobile manipulator 100 determines by the object recognition process (S63) in the field of view that the book existing in front of or near the hand camera 42 is not the target book 601a (S66NO), so the data server 200 is reached. Based on the relative positional relationship from the object in the visual field obtained by the inquiry process (S65) to the target book 601a, the hand is moved in parallel toward the target book 601a (S68). This translation process is performed until the target object 601a is placed in front of or in the vicinity thereof (S69 to S72YES). That is, the process of moving the position and posture of the hand camera 42 indicated by C1 toward the target position and posture of the hand camera 42 indicated by C3 is performed.

FIG. 15 is an explanatory diagram showing a state after the hand camera 42 is translated to the target position and posture. As is clear from the figure, by translating the hand camera 42 by the distance d, the position and posture of the hand camera 42 indicated by C1 coincide with the target position and posture of the hand camera 42 shown by C3. There is.

According to such a configuration, even if the position and posture of the moving carriage 20 deviate from the target position and posture due to a self-position estimation error or the like, the position of the hand camera 42 surely matches the target position and posture. Therefore, the possibility of failure of the leaching operation or the like (S74, S75) described later can be significantly reduced.

FIG. 16 is a perspective view of an example of a mobile manipulator 100 that performs a translation operation of the hand camera 42. In the figure, the figure on the left side shows the state before the translation operation (S68) of the hand camera 42, and the figure on the right side shows the state after the translation operation (S68) of the hand camera 42.

As is clear from the figure on the left side, in the state before the translation operation (S68), the book 601b, which is not the target book 601a, is arranged in front of or near the hand camera 42. When the mobile manipulator 100 determines that the book existing in front of or near the hand camera 42 is not the target book 601a, the mobile manipulator 100 acquires the position information of the object in the field of view from the data server 200 and targets the book. Translation is started toward 601a (S68 to S72).

As is clear from the figure on the right side after the translation operation (S72YES), the target book 601a is arranged in front of or near the hand camera 42 in the state after the translation operation. In this state, by performing the leaching operation or the like (S74, S75) described later, it is possible to surely grip the target book 601a or the like.

The above-mentioned translation process of the hand or hand camera 42 was performed in a plane along the front surface of the shelf 600, but the present invention is not limited to such a configuration. Therefore, for example, it may be moved while maintaining a constant distance and posture along a surface including a curved surface or a step corresponding to the shape of a mounted body such as a shelf or a storage body.

Returning to FIG. 11, after arranging the hand camera 42 at a relative distance of 200 mm in front of the target object 601a (180 °), a process of specifying the gripping position of the target object, that is, the book 601a is performed. It is (S73).

The process of specifying the gripping position is performed using the trained model by detecting the contour line of the gripping object from the image obtained from the hand camera 42 and inputting the information on the detected contour line. This trained model is generated by performing machine learning based on training data including the correct gripping position annotated by a human on the contour line of the gripping object.

By the gripping position specifying process using such a learned model, it is possible to accurately identify the gripping position with a high possibility of successful gripping.

Although the machine learning method is obtained by using a neural network, particularly deep learning, in the present embodiment, it is not limited to such a machine learning method, and other known methods may be used. good.

After this gripping position specifying process, the manipulator control unit 113 controls the leaching motion of extending or flexing each joint of the arm portion 40 toward the gripping position (S74). This leaching operation is performed until a constant reaction force is detected by the sensor 116 including the joint torque sensor of the arm portion 40, that is, until the arm portion 40 comes into contact with the main 601a.

After this leaching operation, the manipulator control unit 113 controls the operation of gripping the target object 601a by using the gripping mechanism 41 which is a gripper (S75). This gripping operation is performed until a constant reaction force is detected by measuring the current value of the gripper or the like.

After the gripping process, the mobile manipulator 100 performs a process of folding the arm portion 40 by bending each joint of the arm portion 40 while gripping the book 601a (S77). This ends the picking process (S6).

Returning to FIG. 6, when the picking process is completed, the mobile manipulator 100 performs the move process again (S8). That is, the mobile manipulator 100 reads out the target mounting position 701 on the workbench 700 from the data server 200, and calculates the target position and posture of the hand camera 42. After that, the target stop position and posture of the moving trolley 20 are calculated from the target position and posture of the hand camera 42, and the process of moving while holding the book 601a while estimating the self-position toward the target position and posture is performed. conduct. Since the details of the movement process based on the self-position estimation are substantially the same as those shown in FIG. 9, the description thereof will be omitted.

When the mobile manipulator 100 reaches the target stop position and posture of the mobile trolley 20 near the workbench 700, the mobile manipulator 100 performs a process of releasing the book 601a by mounting the book 601a to a predetermined mounting position 701 on the workbench 700 (S9). .. This completes a series of transportation processes.

According to such a configuration, even if an error occurs in the stop position / posture after movement due to a self-position estimation error or the like, it is possible to provide a mobile manipulator 100 or the like that can surely achieve a subsequent operation such as leaching. Can be done.

Although the embodiments of the present invention have been described above, the above embodiments are only a part of the application examples of the present invention, and the technical scope of the present invention is limited to the specific configuration of the above embodiments. No. Further, the above embodiments can be appropriately combined as long as there is no contradiction.

The present invention can be used at least in an industry that manufactures autonomous mobile objects such as mobile manipulators.

10 Robot body 2 Mobile mechanism drive means 20 Mobile cart 21 Skirt 22 Drive wheel 3 Head drive means 30 Head 31 Face 32 Environmental recognition means 35 Head camera 40 Arm 41 Grip mechanism 42 Hand camera 100 Mobile manipulator 200 Data Server 300 Client equipment 500 System 800 Factory

Claims (28)

1. A mobile manipulator that moves to the vicinity of the mounting body and reaches the target object on the mounting body described above.
   The bogie unit equipped with a moving means for moving toward the bogie target stop position and the bogie target stop posture in the vicinity of the above-mentioned body, and
   With a manipulator equipped with a hand camera at or near the tip,
   A relative relationship recognition unit that recognizes the relative distance and the relative posture of the hand camera with respect to the previously described body after stopping.
   An object recognition unit that recognizes an object placed in the field of view of the hand camera after stopping,
   When the recognized object includes the target object, the reach operation control of the manipulator with respect to the target object is performed based on the relative distance and the relative posture, and when the recognized object does not include the target object. After acquiring the recognized position information of the object and moving the hand position of the manipulator based on the position information, the reaching operation control of the manipulator with respect to the target object is performed based on the relative distance and the relative posture. Reaching control unit and
   Equipped with a mobile manipulator.
2. A claim further comprising a reach preparation operation control unit that controls the manipulator to be in the reach preparation distance and the reach preparation posture based on the relative distance and the relative posture between the hand camera and the above-mentioned relative body. The mobile manipulator according to 1.
3. The mobile manipulator according to claim 2, wherein the trolley target stop position and the trolley target stop posture are generated based on the leaching preparation distance and the leaching preparation posture.
4. The mobile manipulator according to claim 3, wherein the reach preparation distance and the reach preparation posture are generated based on the position of the target object and the relative relationship between the preset predicated body and the hand. ..
5. The mobile manipulator according to claim 2, wherein the recognition of the target object by the hand camera is performed at the reach preparation distance and the reach preparation posture.
6. Equipped with a self-position estimation unit that performs self-position estimation processing by environment recognition means
   The mobile manipulator according to claim 1, wherein the moving means is controlled based on the result of the self-position estimation process.
7. The mobile manipulator according to claim 6, wherein the environment recognition means is a rider.
8. The mobile manipulator according to claim 1, wherein the hand camera is a monocular camera.
9. The mobile manipulator according to claim 1, further comprising a gripping control unit that controls gripping of the target object after the leaching motion control.
10. The grip control unit further
    The mobile manipulator according to claim 9, further comprising a gripping position determining unit that determines a gripping position of the target object based on an image acquired by the hand camera.
11. The gripping position determining unit further
    The mobile manipulator according to claim 10, comprising a gripping position estimation unit including a trained model machine-learned to estimate the gripping position of the target object based on an image acquired by the hand camera.
12. The mobile manipulator according to claim 1, wherein the movement of the hand position of the manipulator is performed in parallel with the previously described body.
13. The mobile manipulator according to claim 1, further comprising a trolley movement control unit that moves the trolley unit for moving the hand position of the manipulator.
14. The relative relationship recognition unit further
    A marker recognition unit that recognizes a relative distance and a relative posture between the hand camera and the previously described body based on image information including a first marker provided on the target object or the previously described body. The mobile manipulator according to claim 1.
15. The mobile manipulator according to claim 14, wherein a plurality of the first markers are arranged on the target object or the above-mentioned device.
16. The mobile manipulator according to claim 15, which recognizes the relative distance and the relative posture based on the information regarding the plurality of the first markers and the constraint condition based on the shape of the above-mentioned body.
17. The mobile manipulator according to claim 14, wherein the image information including the first marker is captured by the hand camera.
18. The image information including the plurality of the first markers is imaged by pointing the hand camera obliquely with respect to the plurality of the first markers so as to keep the plurality of the first markers in the field of view. The manipulator according to claim 15, which is obtained by the above.
19. The mobile manipulator further comprises a camera arranged on the manipulator so as to have a wider field of view than the hand camera.
    The mobile manipulator according to claim 14, wherein the image information including the first marker is captured by the camera.
20. The mobile manipulator also comprises a head.
    The mobile manipulator according to claim 19, wherein the camera is provided on the head.
21. The mobile manipulator according to claim 1, wherein the trolley unit is an omnidirectional mobile trolley.
22. The manipulator is an articulated arm and is
    The articulated arm comprises one or more force sensors at its tip, its joint or its root.
    The leaching motion control unit further
    The mobile manipulator according to claim 1, further comprising a leaching stop control unit that controls the leaching operation to be stopped when a value equal to or higher than a certain value is detected by the force sensor.
23. The manipulator includes an articulated arm and an end effector provided at the tip of the articulated arm.
    The end effector includes a force sensor and / or a contact sensor on the contact surface with the object.
    The leaching operation control unit is
    The mobile manipulator according to claim 1, further comprising a leaching stop control unit for an effector that controls the leaching operation to be stopped when a value equal to or higher than a certain value is detected by the force sensor and / or the contact sensor.
24. The mobile manipulator according to claim 1, wherein the manipulator includes an arm portion and a vertical movement mechanism provided at the base of the arm portion.
25. The mobile manipulator according to claim 1, wherein the recognition of an object in the object recognition unit is performed by capturing a two-dimensional code information provided on the object.
26. The leaching motion control unit further
    A transmission unit that transmits the recognized identification information of the object to a predetermined data server via a network.
    The mobile manipulator according to claim 1, further comprising a receiving unit that receives location information corresponding to the identification information from the data server.
27. It is a control method of a mobile manipulator that moves to the vicinity of the mounting body and reaches the target object on the mounting body described above.
    The manipulator is
    The bogie unit equipped with a moving means for moving toward the bogie target stop position and the bogie target stop posture in the vicinity of the above-mentioned body, and
    With a manipulator, equipped with a hand camera at or near the tip,
    The control method is
    A relative relationship recognition step that recognizes the relative distance and the relative posture of the hand camera with respect to the previously described object after stopping.
    An object recognition step that recognizes an object placed in the field of view of the hand camera after stopping,
    When the recognized object includes the target object, the reach operation control of the manipulator with respect to the target object is performed based on the relative distance and the relative posture, and when the recognized object does not include the target object. After acquiring the recognized position information of the object and moving the hand position of the manipulator based on the position information, the reaching operation control of the manipulator with respect to the target object is performed based on the relative distance and the relative posture. Reaching control steps to perform and
    How to control a mobile manipulator with.
28. A mobile manipulator control program that moves to the vicinity of the mounting body and reaches the target object on the mounting body described above.
    The manipulator is
    The bogie unit equipped with a moving means for moving toward the bogie target stop position and the bogie target stop posture in the vicinity of the above-mentioned body, and
    With a manipulator, equipped with a hand camera at or near the tip,
    The control program is
    A relative relationship recognition step that recognizes the relative distance and the relative posture of the hand camera with respect to the previously described object after stopping.
    An object recognition step that recognizes an object placed in the field of view of the hand camera after stopping,
    When the recognized object includes the target object, the reach operation control of the manipulator with respect to the target object is performed based on the relative distance and the relative posture, and when the recognized object does not include the target object. After acquiring the recognized position information of the object and moving the hand position of the manipulator based on the position information, the reaching operation control of the manipulator with respect to the target object is performed based on the relative distance and the relative posture. Reaching control steps to perform and
    Mobile manipulator control program with.

Images