WO2024224751A1

WO2024224751A1 - Image processing method and image processing device

Info

Publication number: WO2024224751A1
Application number: PCT/JP2024/004653
Authority: WO
Inventors: テイテイ虎; 康治井村
Original assignee: パナソニックＩｐマネジメント株式会社
Priority date: 2023-04-27
Filing date: 2024-02-09
Publication date: 2024-10-31

Abstract

This image processing method: executes a first process for acquiring a captured image of a target object, detecting the position or posture of the target object from the captured image, and generating a template of the target object; and executes, during the execution of the first process, a second process a plurality of times for performing template matching on the basis of a captured image and the template of the target object to obtain information regarding the movement of the target object. In the first process, a template of the target object corresponding to a predicted posture obtained by predicting the postureof the target object is generated on the basis of the detected position or posture of the target object and the information regarding the movement of the target object as obtained a plurality of times by the second process.

Description

Image processing method and image processing device

This disclosure relates to an image processing method and an image processing device.

In production processes in factories, it is sometimes necessary to determine whether a part being picked by an end effector such as a robot hand is the correct part (for example, a part to be used in the production of an industrial product). When making such a determination, it is necessary to perform the determination process as quickly as possible to avoid reducing the takt time of the production process. A known example of a conventional determination process is the template matching method, which compares a prepared template (for example, an image) of the part with an image of the part captured by a camera installed in the factory and performs a matching process.

Patent Document 1 discloses a template creation device that creates a set of templates used in an object recognition device that recognizes objects by template matching. The template creation device acquires multiple templates from multiple images of an object in different poses, calculates the similarity of image features between two templates selected from the multiple templates, and performs clustering to divide the multiple templates into multiple groups based on the similarity. For each of the multiple groups, the template creation device integrates all the templates in the group into a single integrated template, and generates a template set with an integrated template for each group.

Japanese Patent Application Publication No. 2016-207147

In Patent Document 1, the object recognition device performs a hierarchical search by creating a hierarchical template set, performing rough recognition using a low-resolution template set, and then using the results to perform detailed recognition using a high-resolution template set. However, matching processing must be performed in at least two stages, such as recognition processing using a low-resolution template set and recognition processing using a high-resolution template set, which inevitably increases the processing load on the object recognition device.

Furthermore, when attempting to apply the technology of Patent Document 1 to the above-mentioned production process in a factory, in which the end effector and the camera are moved to capture an image of the part to be picked by the camera in order to determine whether the part to be picked by the end effector is the correct part, the following problem arises. Specifically, if the camera moves as the end effector moves, the way the part appears to the camera (in other words, the posture of the part) changes as the position of the end effector changes. For this reason, unless the position of the end effector (the position of the camera, so to speak) is taken into consideration during template matching, efficient template matching cannot be performed even if a pre-generated template set is used, and the reliability of template matching does not improve.

The present disclosure has been devised in consideration of the conventional circumstances, and aims to provide an image processing method and image processing device that achieves highly accurate template matching of an object even when the orientation of the object relative to an imaging device changes as the imaging device moves.

The present disclosure provides an image processing method performed by an image processing device that is mobile and capable of communicating with a camera capable of capturing an image of an object, which performs a first process of acquiring an image of the object, detecting a position or orientation of the object from the captured image, and generating a template of the object, and while performing the first process, performs template matching based on the captured image and the template of the object, and executes a second process multiple times to acquire information related to the movement of the object, and the first process predicts the orientation of the object based on the detected position or orientation of the object and the information related to the movement of the object acquired multiple times in the second process, and generates a template of the object corresponding to the predicted orientation of the object.

The present disclosure also provides an image processing device that includes an acquisition unit that acquires an image of an object captured by a movable camera capable of capturing an image of the object, a first processing unit that detects a position or posture of the object from the captured image and generates a template of the object, and a second processing unit that performs template matching based on the captured image and the object template multiple times while the first processing unit is generating the template of the object, and acquires information related to the movement of the object, and the first processing unit predicts the posture of the object based on the detected position or posture of the object and the information related to the movement of the object acquired multiple times by the second processing unit, and generates a template of the object corresponding to the predicted posture of the object.

According to the present disclosure, highly accurate template matching of an object can be achieved even in a situation where the orientation of the object from the imaging device changes as the imaging device moves.

A diagram explaining an example of the configuration of a picking system Block diagram showing an example of the internal configuration of a picking system FIG. 1 is a functional block diagram illustrating functions of a first processing unit and a second processing unit in a first embodiment. FIG. 1 is a diagram for explaining an example of an overall operation procedure of a picking system according to the first embodiment. 11 is a flowchart showing a second example of a processing procedure of the image processing device according to the first embodiment. 1 is a flowchart showing a first example of a processing procedure of the image processing device according to the first embodiment. FIG. 13 is a diagram showing an example of a template display screen. FIG. 1 is a diagram for explaining an example of an overall operation procedure of a picking system according to a first modified example of the first embodiment. A flowchart showing an example of an operation procedure of a first processing unit of an image processing device according to a second modification of the first embodiment.

(Background to this disclosure)
Conventionally, an object recognition device has been disclosed that includes a template creation device for objects on a belt conveyor using images captured by a camera installed at a fixed location that cannot be moved relative to a production line, etc. In addition, when actually performing matching processing, the object recognition device uses a template set generated based on an integrated template for each group as is.

However, when moving the end effector and camera to capture an image of the part to be picked by the camera in order to determine whether the part to be picked by the end effector in the above-mentioned production process in the factory is the correct part, the following problem occurs when attempting to apply the technology of Patent Document 1 (JP Patent Publication JP 2016-207147 A). Specifically, if the camera moves as the end effector moves, the way the part appears to the camera (in other words, the posture of the part) changes as the position of the end effector changes. For this reason, if the position of the end effector (the position of the camera, so to speak) is not taken into consideration during template matching, efficient template matching cannot be performed even if a pre-generated template set is used, and the reliability of template matching does not improve.

In order to solve this problem, one method is to stop or slow down the belt conveyor in accordance with the matching process speed, but when this method is adopted, the part picking efficiency (in other words, production efficiency) decreases. Therefore, there has been a demand for an object recognition device that can realize part matching and part picking without stopping the operation of the belt conveyor and end effector.

In the following embodiments, we therefore describe examples of image processing methods and image processing devices that register highly accurate templates of objects that can be used for template matching even in situations where the orientation of the object from the imaging device changes as the imaging device moves.

Below, with appropriate reference to the attached drawings, each embodiment that specifically discloses the image processing method and image processing device according to the present disclosure will be described in detail. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters and duplicate explanation of substantially identical configurations may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to make it easier for those skilled in the art to understand. Note that the attached drawings and the following explanation are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

(Embodiment 1)
In the first embodiment, for example, in a production process in a factory, when determining whether a part to be picked by an end effector such as a robot hand (for example, a part used in the production of an industrial product) is correctly recognized by template matching, an object recognition process is performed on an image captured by a camera, and a template required for template matching is predicted. A template registration device (for example, an image processing device) according to the present disclosure images an object, and registers information based on an input image of an object captured by an imaging device that can change the imaging position relative to the object by moving (see below) and information based on the input image of the object (see below) in a storage unit in association with position information of the imaging device and information based on the input image of the object (see below) as templates to be used for template matching.

FIG. 1 is a diagram for explaining an example of the configuration of a picking system. FIG. 2 is a block diagram showing an example of the internal configuration of a picking system. As shown in FIG. 2, the picking system 100 includes an actuator AC, a camera CM, an image processing device P1, a display 13, and an operation device 14. The actuator AC and the image processing device P1, the camera CM and the image processing device P1, and the image processing device P1 and the operation device 14 are connected so as to enable input/output (transmission/reception) of data signals.

The positional relationship between the camera CM and the target object Tg will be described with reference to FIG. 1. Note that the description of FIG. 1 is applicable not only to the first embodiment but also to a modified version of the first embodiment described below.

In the following description, the object Tg is an object that is picked by the end effector EF of the picking system 100 deployed in a factory, and is, for example, an industrial part, an industrial product, etc. If it is an industrial part, for example, after being picked, it is moved to another lane (production line) for assembling a finished product. If it is an industrial product, for example, after being picked, it is stored in a box such as a cardboard box. It goes without saying that the type of object Tg is not limited to the industrial parts and industrial products described above.

As shown in Figure 1, the actuator AC controls the camera CM to be movable three-dimensionally, thereby changing the positional relationship between the object Tg moving on the belt conveyor, the end effector EF that picks the object Tg, and the camera CM fixed to the end effector EF.

The actuator AC controls the end effector EF and the camera CM attached to the end effector EF so that they can be moved in three dimensions using multiple axes. In other words, the actuator AC can control the recognition, maintenance, or change of the three-dimensional position (coordinates) of the camera CM.

The end effector EF is, for example, a robot hand provided at the tip of a robot arm deployed in correspondence with the picking system 100, and approaches the target object Tg under the control of the actuator AC, and picks up the target object Tg.

The camera CM is placed near the end effector EF and moves together with the end effector EF under the control of the actuator AC to capture an image of the object Tg. The camera CM captures the object Tg at a predetermined frame rate (e.g., 1000 frames per rate (hereinafter referred to as "fps")) and transmits the captured image of the object Tg (an example of an input image) obtained each time it is captured to the image processing device P1.

The image processing device P1 acquires the captured image of the object Tg transmitted from the camera CM. The image processing device P1 is configured by a computer capable of executing a first process (see Figs. 4 and 6) in which the image processing device detects the posture of the object Tg using the captured image of the object Tg transmitted from the camera CM, observes the movement of the object Tg based on information on the movement of the object Tg obtained by the second process while detecting the posture of the object Tg, predicts the posture of the object Tg at the timing when the posture detection process of the object Tg ends based on the detected posture of the object Tg and the movement information of the object Tg, and generates a detection template corresponding to the predicted posture of the object Tg (an example of a detected posture), and a second process (see Figs. 4 and 5) in which the feature amount of the object Tg is extracted, feature matching based on the extracted feature amount (in other words, template matching) is performed to acquire position information of the object Tg, and the acquired position information of the object Tg is constantly or periodically transmitted (feedback) to the actuator AC.

The image processing device P1 may be, for example, a personal computer (hereinafter referred to as "PC"), or may be a dedicated hardware device specialized for executing each of the above-mentioned first and second processes. The image processing device P1 realizes recognition processing of the position and orientation of the target object Tg picked by the end effector EF by executing each of the above-mentioned first and second processes. The image processing device P1 includes a communication unit 10, a processor 11, a memory 12, and a 3D model database DB.

The image processing device P1 accepts a user operation, and based on the user operation, generates a template display screen SC (see FIG. 7) including a template TP1 (see FIG. 7) of the object Tg viewed from the position and posture of the icon PP1 described below, a prediction template TP2 (see FIG. 7) of the object Tg corresponding to the predicted posture of the object Tg predicted by the first process, and a detection template TP3 (see FIG. 7) of the object Tg corresponding to the posture of the object Tg detected by the first process, and displays it on the display 13.

The communication unit 10 (an example of an acquisition unit) is connected to the actuator AC, the camera CM, the display 13, and the operation device 14 so that data can be communicated between them, and transmits and receives data. The communication unit 10 outputs the captured image transmitted from the camera CM and the control command transmitted from the operation device 14 to the processor 11. The communication unit 10 transmits the template display screen SC (see FIG. 7) output from the processor 11 to the display 13.

The processor 11 is configured using, for example, a Central Processing Unit (CPU) or a Field Programmable Gate Array (FPGA), and performs various processes and controls in cooperation with the memory 12. Specifically, the processor 11 references the programs and data stored in the memory 12 and executes the programs to realize the respective functions of the first processing unit 110 and the second processing unit 120.

The first processing unit 110 executes a detection process for the object Tg and executes a first process (see FIG. 4 and FIG. 6) for generating a prediction template TP2 for the object Tg. The first process is an advanced image process using deep learning, and takes a longer time (e.g., 17 ms) than the time required for executing the second process. Here, the position and posture of the object Tg continue to change while the first process is being executed. Therefore, in order to maintain the validity of the detection template TP3 output at the timing when the first process ends, the first processing unit 110 predicts template candidates for the object Tg using the results of the second process executed by the second processing unit 120, which is executed multiple times while the first process is being executed. As a result, the first processing unit 110 can track the changes in posture of the object Tg that change during the execution of the first process in real time, and predict template candidates that are more suitable for the feature matching executed by the second processing unit 120.

The second processing unit 120 executes a second process (see Figures 4 and 5) in which feature matching is performed using the prediction template TP2 obtained by the first process and the captured image captured by the camera CM, the position information of the object Tg is estimated, and the estimated position information of the object Tg is transmitted to the actuator AC. The second process is a simple image process using feature matching, and requires a shorter time (e.g., 1 ms) than the time required to execute the first process. The second processing unit 120 executes the second process multiple times while the first processing unit 110 executes the first process once.

Memory 12 has, for example, Random Access Memory (RAM) as a working memory used when executing each process of processor 11, and Read Only Memory (ROM) that stores programs and data that define the operation of processor 11. Data or information generated or acquired by processor 11 is temporarily stored in RAM. Programs that define the operation of processor 11 are written in ROM.

The 3D model database DB (an example of a database) is, for example, a flash memory, a hard disk drive (HDD), or a solid state drive (SSD). The 3D model database DB stores (registers) 3D model data of at least one object Tg to be picked, and information about the object Tg (for example, the name and identification number of each object) for each object Tg.

The display 13 is a device that outputs (displays) the template display screen SC (see FIG. 7) generated by the image processing device P1, and is configured, for example, by a Liquid Crystal Display (LCD) or an organic electroluminescence (EL) device.

The operation device 14 is an interface that detects user operation input, and is composed of, for example, a mouse, a keyboard, or a touch panel. When the operation device 14 receives a user operation, it generates an electrical signal based on the user operation and transmits it to the image processing device P1.

Next, the functions realized by the first processing unit 110 and the second processing unit 120 will be described with reference to FIG. 3. FIG. 3 is a functional block diagram illustrating the functions of the first processing unit 110 and the second processing unit 120 in the first embodiment.

The first processing unit 110 includes an object detection unit 111, a 3D model selection unit 112, a first time prediction unit 113, and a 3D model synthesis unit 117.

The object detection unit 111 uses Deep Learning to perform image recognition processing on the captured image transmitted from the camera CM and detects an object (target object Tg) from the captured image. The object detection unit 111 outputs information on the detected object (target object Tg) to the 3D model selection unit 112 and the first time prediction unit 113. Note that the Deep Learning used by the object detection unit 111 may be any learning method suitable for detecting the target object Tg, such as a Convolutional Neural Network (CNN).

The 3D model selection unit 112 selects a 3D model MD that corresponds to information about an object (object Tg) specified by a user operation from among the 3D models of at least one object Tg registered in the 3D model database DB. The 3D model selection unit 112 outputs the selected 3D model MD of the object Tg to each of the 3D matching unit 114 and the 3D model synthesis unit 117.

The first time prediction unit 113 tracks (observes) changes in the position and posture of the object Tg based on the movement information of the object Tg obtained by the second process while the detection process of the object Tg is being performed by the first process, and generates a prediction template TP2 corresponding to the posture of the object Tg at the time when the first process ends. The first time prediction unit 113 includes a 3D matching unit 114, a template prediction unit 115, and a prediction model update unit 116.

The 3D matching unit 114 executes 3D matching in which the information about the object (target object Tg) output from the object detection unit 111 and the 3D model MD of the target object Tg output from the 3D model selection unit 112 are matched in a three-dimensional space, and recognizes the posture of the target object Tg that appears in the captured image captured by the camera CM (hereinafter referred to as the "detected posture"). The 3D matching unit 114 associates information about the detected posture of the target object Tg with the 3D model MD of the target object Tg used in the 3D matching, and outputs the information to the template prediction unit 115 and the 3D model synthesis unit 117.

The template prediction unit 115 generates a prediction template TP2 corresponding to the posture of the object Tg at the time when the first process ends, based on information on the detected posture of the object Tg and the 3D model MD of the object Tg sent from the 3D matching unit 114, and the prediction model output from the prediction model update unit 116. Note that the prediction model here is a prediction model in which the posture of the object Tg is predicted, and is a mathematical model based on information on the movement of the object Tg obtained by the second process that has been executed multiple times.

The template prediction unit 115 generates a 2D image (hereinafter referred to as a "prediction template") that is the posture of the camera CM when the next captured image is captured, and that is obtained when the 3D model MD is viewed from the predicted posture (angle) of the object Tg, based on the prediction result. The template prediction unit 115 associates information about the posture of the object Tg with the generated prediction template TP2 of the object Tg (see Figure 7), and outputs the information to the 3D model synthesis unit 117 and the template update unit 121, respectively.

The prediction model update unit 116 updates the prediction model for predicting changes in the posture of the object Tg that occur while the detection process for the object Tg is being performed by the first process, based on the feature matching result output from the feature matching unit 123 and the movement of the object Tg output from the position fitting unit 124. The prediction model update unit 116 outputs the updated prediction model to the template prediction unit 115.

The 3D model synthesis unit 117 acquires the detected orientation of the object Tg output from the 3D matching unit 114 and the detection template TP3 (see FIG. 7), and generates an icon PP3 indicating the angle at which the object Tg in the captured image was captured based on the detected orientation of the object Tg. The 3D model synthesis unit 117 acquires the predicted orientation of the object Tg output from the template prediction unit 115 and the prediction template TP2, and generates an icon PP2 indicating the angle of the object Tg to be captured in the next captured image based on the predicted orientation of the object Tg.

The 3D model synthesis unit 117 generates a template display screen SC based on the 3D model MD of the object Tg, the prediction template TP2, the detection template TP3, and each of the icons PP1, PP2, and PP3, and transmits it to the display 13.

The second processing unit 120 includes a template update unit 121, a feature extraction unit 122, a feature matching unit 123, a position fitting unit 124, a second time prediction unit 125, and a control unit 126.

The template update unit 121 acquires the prediction template TP2 output from the template prediction unit 115 of the first processing unit 110, and updates the template (2D data) of the object Tg used for feature matching to the acquired prediction template TP2.

The feature extraction unit 122 extracts the feature amount of the object Tg from the captured image transmitted from the camera CM. The feature extraction unit 122 outputs the extracted feature amount of the object Tg to the feature matching unit 123.

The feature matching unit 123 matches the feature amount of the object Tg included in the prediction template TP2 output from the template update unit 121 with the feature amount of the object Tg output from the feature extraction unit 122. The feature matching unit 123 outputs the matching result to the prediction model update unit and the position fitting unit 124.

The position fitting unit 124 acquires the matching results output from the feature matching unit 123. The position fitting unit 124 fits the position information of the object Tg appearing in the captured image based on the result of the feature matching.

The position fitting unit 124 outputs the position information of the object Tg after position fitting to the prediction model update unit 116 and the second time prediction unit 125.

The second time prediction unit 125 predicts the movement of the object Tg while the second process is being performed based on the position information of the object Tg output from the position fitting unit 124, and predicts the position of the object Tg at the time when the second process ends. The second time prediction unit 125 outputs information on the predicted position of the object Tg to the control unit 126.

The control unit 126 outputs the information on the predicted position of the object Tg output from the second time prediction unit 125 to the actuator AC.

Next, the overall operation procedure of the picking system 100 will be described with reference to FIG. 4. FIG. 4 is a diagram illustrating an example of the overall operation procedure of the picking system 100 according to the first embodiment.

Note that the example of the overall operation procedure of the picking system 100 shown in FIG. 4 is just one example, and is not limited to this. In FIG. 4, in order to make it easier to understand the relationship between the first process and the second process, an example is shown in which the first process is executed once and the second process is executed N times (N: an integer of 3 or more) when picking one target object Tg, but the number of times that the first process and the second process are executed is not limited to this. It goes without saying that the picking system 100 may execute the first process multiple times when picking one target object Tg.

The actuator AC picks up the object Tg being transported on the belt conveyor by the camera CM while capturing images of the object Tg at a predetermined frame rate (e.g., 1000 fps). The actuator AC shown in FIG. 4 shows a part of the picking process of the object Tg (time t11 to time t1N), and is executed repeatedly, for example, until the object Tg is picked.

The image processing device P1 acquires an image of the object Tg captured by the camera CM at a predetermined frame rate, and performs a second process (step St100) on the acquired image.

In addition, in parallel with the second process, the image processing device P1 executes the first process (step St200) based on the captured image captured by the camera CM, the feature matching results (in other words, matching tendency) obtained by the multiple second processes executed during the first process, and the amount of movement of the object Tg (in other words, movement information). The image processing device P1 feeds back to the second process the prediction template TP2 (see FIG. 7), which is a template candidate for the object Tg obtained by the first process.

In the example shown in FIG. 4, the camera CM captures an image of the object Tg at time t11 and transmits the captured image Img11 to the image processing device P1. The image processing device P1 acquires the first captured image Img11 (image data) transmitted from the camera CM and executes the first process and the second process using the first captured image Img11. The image processing device P1 outputs the matching tendency of the object Tg obtained by the second process and the movement information of the object Tg to the first processing unit 110, and transmits information on the predicted position (x1, y1, z1) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired three-dimensional predicted position (x1, y1, z1) of the object Tg.

At time t12, the camera CM captures an image of the object Tg. The camera CM transmits the captured image Img12 to the image processing device P1. The image processing device P1 acquires the second captured image Img12 (image data) transmitted from the camera CM and executes the second process using the second captured image Img12. The image processing device P1 outputs the matching tendency of the object Tg obtained by the second process and the movement information of the object Tg to the first processing unit 110, and transmits information on the predicted position (x2, y2, z2) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired predicted position (x2, y2, z2) of the object Tg.

At time t13, the camera CM captures an image of the object Tg. The camera CM transmits the captured image Img13 to the image processing device P1. The image processing device P1 acquires the third captured image Img13 (image data) transmitted from the camera CM and executes the second process using the third captured image Img13. The image processing device P1 outputs the matching tendency of the object Tg obtained by the second process and the movement information of the object Tg to the first processing unit 110, and transmits information on the predicted position (not shown) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired predicted position of the object Tg.

At time t1 (N-2), the camera CM captures an image of the object Tg. The camera CM transmits the captured image Img1 (N-2) to the image processing device P1. The image processing device P1 acquires the (N-2)th captured image Img1 (N-2) (image data) transmitted from the camera CM and executes the second process using the (N-2)th captured image Img1 (N-2). The image processing device P1 outputs the matching tendency of the object Tg and the movement information of the object Tg obtained by the second process to the first processing unit 110, and transmits information on the predicted position (not shown) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired predicted position of the object Tg.

At time t1 (N-1), the camera CM captures an image of the object Tg. The camera CM transmits the captured image Img1 (N-1) to the image processing device P1. The image processing device P1 acquires the (N-1)th captured image Img1 (N-1) (image data) transmitted from the camera CM and executes the second process using the (N-1)th captured image Img1 (N-1). The image processing device P1 outputs the matching tendency of the object Tg and the movement information of the object Tg obtained by the second process to the first processing unit 110, and transmits information on the predicted position (not shown) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired predicted position of the object Tg.

At time t1N, the camera CM captures an image of the object Tg. The camera CM transmits the captured image Img1N to the image processing device P1. The image processing device P1 acquires the Nth captured image Img1N (image data) transmitted from the camera CM and executes the second processing using the Nth captured image Img1N. The image processing device P1 outputs the matching tendency of the object Tg obtained by the second processing and the movement information of the object Tg to the first processing unit 110, and transmits information on the predicted position (xN, yN, zN) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired predicted position (xN, yN, zN) of the object Tg to pick up the object Tg. The image processing device P1 feeds back the template candidate (prediction template TP2) of the object Tg obtained by the first process at time t1(N+1) to the second processing unit 120, and updates the template candidate (prediction template TP2).

Note that, although not shown in the example shown in FIG. 4, in the second process executed subsequently, the image processing device P1 performs feature matching using the latest obtained prediction template TP2 until a new prediction template TP2 is fed back again by the first process.

Next, the second process of the image processing device P1 will be described with reference to FIG. 5. FIG. 5 is a flowchart showing an example of the second process procedure (step St100) of the image processing device P1 in embodiment 1.

The second processing unit 120 sets the update flag for updating the template of the target object Tg to "1" based on the control command notifying the start of the picking process acquired via the operation device 14 (St11). The second processing unit 120 acquires the captured image transmitted from the camera CM (St12).

The second processing unit 120 determines whether the currently set update flag is "1" (St13).

If the second processing unit 120 determines in the processing of step St13 that the currently set update flag is "1" (St13, YES), it updates the template (template TP1 or prediction template TP2) of the object Tg used for feature matching (St14). The second processing unit 120 outputs the captured image of the object Tg for generating a template candidate (prediction template TP2) to the first processing unit 110 (St15), and sets the update flag to "0" (St16).

On the other hand, if the second processing unit 120 determines in the processing of step St13 that the currently set update flag is not "1" (St13, NO), it determines whether or not there is a template (prediction template TP2) of the object Tg to be used for feature matching (St17).

If the second processing unit 120 determines in step St17 that there is a template (prediction template TP2) of the object Tg to be used for feature matching (St17, YES), it extracts features from the captured image (St18).

On the other hand, if the second processing unit 120 determines in the processing of step St17 that there is no template (prediction template TP2) of the object Tg to be used for feature matching (St17, NO), it returns to the processing of step St13 and waits until a template (template TP1 or prediction template TP2) of the object Tg is fed back from the first processing unit 110.

The second processing unit 120 performs feature matching between the features extracted from the captured image (i.e., the features of the captured image) and the features of the object Tg based on the template (i.e., the features of the template). Based on the matching result, the second processing unit 120 fits the position of the object Tg appearing in the captured image captured by the camera CM (St19).

The second processing unit 120 calculates the amount of movement of the object Tg between the captured image used for feature matching (i.e., the most recent captured image) and the captured image captured immediately before this captured image. Based on the calculated amount of movement, the second processing unit 120 predicts the amount of movement of the object Tg while the second process is being performed, predicts the position of the object Tg at the time when the second process ends, and obtains the predicted position of the object Tg (St20). The second processing unit 120 transmits the predicted position of the object Tg to the actuator AC (St21).

The second processing unit 120 outputs the matching tendency, which is the result of feature matching, and the movement information (movement information) of the object Tg to the first processing unit 110 (St22).

The second processing unit 120 determines whether the picking process of the object Tg has been completed based on a control command notifying the completion of the picking process of the object Tg by the actuator AC (St23).

If the second processing unit 120 determines in step St23 that the picking process of the target object Tg has been completed (St23, YES), it terminates the second process (step St100) shown in FIG. 5.

On the other hand, if the second processing unit 120 determines in the processing of step St23 that the picking process of the target object Tg has not been completed (St23, NO), the processing returns to the processing of step St11.

As described above, the image processing device P1 in the first embodiment can predict the amount of movement of the object Tg while the second process is being performed based on the captured images captured at a high frame rate by the second process, and can predict the position of the object Tg at the timing when the second process ends. This allows the image processing device P1 to support the real-time tracking of the object Tg by the actuator AC.

Next, the first process of the image processing device P1 will be described with reference to FIG. 6. FIG. 6 is a flowchart showing an example of the first process procedure (step St200) of the image processing device P1 in the first embodiment.

The first processing unit 110 acquires the captured image output from the second processing unit 120 (St31). The first processing unit 110 detects the object Tg from the acquired captured image using advanced image processing techniques such as Deep Learning (St32).

Based on the detected object Tg, the first processing unit 110 selects a 3D model MD corresponding to the object Tg from among the 3D models stored in the 3D model database DB. The first processing unit 110 performs 3D matching between the selected 3D model MD and the detected object Tg to obtain the detected orientation of the object Tg in the captured image (in other words, the captured orientation of the object Tg when it is captured) (St33). This allows the first processing unit 110 to obtain the detected orientation for generating the detection template TP3 (see FIG. 7). The first processing unit 110 updates a prediction model for predicting the orientation of the object Tg (in other words, the captured orientation) based on the feature matching result and the movement information of the object Tg obtained from the second processing unit 120 (St34).

The first processing unit 110 predicts a template candidate corresponding to the posture of the object Tg at the timing (time) when the first processing ends based on a prediction model for predicting the updated posture of the object Tg (in other words, the imaging posture) and the detected posture of the object Tg (St35), and feeds back (outputs) the predicted template candidate (prediction template TP2) to the second processing unit 120 (St36).

The first processing unit 110 sets the update flag for updating the template of the object Tg to "1" (St37).

The first processing unit 110 determines whether the picking process of the object Tg has been completed based on a control command notifying the completion of the picking process of the object Tg by the actuator AC (St38).

If the first processing unit 110 determines in step St38 that the picking process of the target object Tg has been completed (St38, YES), it ends the first process (step St200) shown in FIG. 6.

On the other hand, if the first processing unit 110 determines in the processing of step St38 that the picking process of the target object Tg has not been completed (St38, NO), the processing returns to the processing of step St31.

As described above, the image processing device P1 in embodiment 1 can perform more advanced image recognition processing through the first processing, and thereby select with higher accuracy a 3D model MD of the object Tg for generating a prediction template TP2 to be used in the second processing. Furthermore, the image processing device P1 can track changes in the posture of the object Tg that change during the first processing, based on the feature matching results and movement information of the object Tg output from the second processing unit 120, which has a short processing time, and generate a template (prediction template TP2) of the object Tg that is closer to the actual posture of the object Tg. This allows the image processing device P1 to improve the feature matching accuracy of the second processing unit 120, as well as improve the tracking accuracy of tracking the object Tg.

Next, the template display screen will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of the template display screen SC. Note that the template display screen SC shown in FIG. 7 is an example and is not limited to this.

The first processing unit 110 generates a template display screen SC based on the result of the first processing, and transmits the generated template display screen SC to the display 13 for display. The template display screen SC includes a first display area AR1, a second display area AR2, a third display area AR3, a fourth display area AR4, a fifth display area AR5, and a registration button BT.

The first display area AR1 includes a 3D model MD of the object Tg obtained by image processing in the first process or selected by user operation in a modified example of the first embodiment described below, and an XYZ coordinate system set in the data of the 3D model MD of the object Tg.

The first display area AR1 also includes an icon PP1 indicating the imaging angle of the camera CM corresponding to the template TP1, an icon PP2 indicating the imaging angle of the camera CM corresponding to the prediction template TP2 (an example of the first imaging position), and an icon PP3 indicating the imaging angle of the camera CM corresponding to the detection template TP3 (an example of the second imaging position).

The icon PP1 can receive user operations via the operation device 14. When the position of the icon PP1 is changed by a user operation, the image processing device P1 generates a template (2D) of the 3D model MD of the object Tg viewed from the position (angle) of the icon PP1. The image processing device P1 generates a template display screen SC in which the generated template (2D) of the 3D model MD is displayed as the template TP1 in the third display area AR3, and transmits it to the display 13 for display.

Icon PP2 is the position (angle) of the camera CM capable of capturing an image of the template (2D) of the 3D model MD of the prediction template TP2, and indicates the imaging position of the object Tg to be imaged next. When the image processing device P1 predicts the predicted attitude of the object Tg by the first process based on the predicted position of the object Tg obtained by the second process, the image processing device P1 updates the position of the icon PP2 and the prediction template TP2 displayed in the fourth display area AR4 based on the predicted attitude of the object Tg. The image processing device P1 generates a template display screen SC in which the position of the icon PP2 and the prediction template TP2 have been updated, and transmits it to the display 13 for display.

Icon PP3 is the position (angle) of the camera CM capable of capturing an image of the template (2D) of the 3D model MD of the detection template TP3, and indicates the imaging position of the object Tg detected from the captured image on which image processing has been performed. The image processing device P1 updates the position of the icon PP3 and the detection template TP3 displayed in the fifth display area AR5 based on the posture of the object Tg detected by the first processing. The image processing device P1 generates a template display screen SC in which the position of the icon PP3 and the detection template TP3 have been updated, and transmits it to the display 13 for display.

The second display area AR2 includes a 3D model MD of at least one object stored in the 3D model database DB. The second display area AR2 shown in FIG. 7 includes a 3D model MD (3D) of object "A12", a 3D model MD (3D) of object "A13", a 3D model MD (3D) of object "A14", and a 3D model MD (3D) of object "A15".

The second display area AR2 can receive a user operation to select one of the objects via the operation device 14. The image processing device P1 receives a user operation to select one of the selection areas SL1, SL2, SL3, and SL4 corresponding to each object, and displays a 3D model corresponding to any one of the selection areas SL1 to SL4 specified by the user operation in the first display area AR1.

In addition, the image processing device P1 in the second variation of the first embodiment accepts a user operation on the icon PP1 displayed in the first display area AR1, and displays a 3D model template (2D) corresponding to the position of the icon PP1 in the third display area AR3 based on the position of the icon PP1 moved by the user operation. When the registration button BT is selected (pressed) by the user operation, the image processing device P1 updates (registers) the template (2D) displayed in the third display area AR3 as a template to be used for feature matching in the second process.

The third display area AR3 includes a template TP1 (2D) when the 3D model MD of the object Tg is imaged from the position (angle) of the icon PP1.

The fourth display area AR4 includes the position (angle) of the icon PP2, i.e., the predicted template TP2 (2D) when the 3D model MD of the object Tg is imaged from the predicted posture.

The fifth display area AR5 includes the position (angle) of the icon PP3, i.e., the detection template TP3 (2D) when the 3D model MD of the object Tg is imaged from the posture in which the object Tg is detected.

The registration button BT is a button that can accept the creation of a template (2D) that corresponds to the icon PP1 that has been moved based on a user operation.

(First Modification of First Embodiment)
The picking system 100 according to the first embodiment has shown an example of performing template prediction by 3D matching using a 3D model. The picking system 100 according to the first modification of the first embodiment will be described as performing template prediction based on the position of the target object Tg detected in the first process, information on the movement of the target object Tg obtained in multiple second processes performed while the first process is being performed, and captured images used in the multiple second processes.

Note that the internal configuration example of the picking system 100 according to the first variation of the first embodiment has almost the same configuration as the internal configuration example of the picking system 100 according to the first embodiment, so a description thereof will be omitted.

Next, the overall operation procedure of the picking system 100 will be described with reference to FIG. 8. FIG. 8 is a diagram illustrating an example of the overall operation procedure of the picking system 100 according to the first modification of the first embodiment.

Note that the example of the overall operation procedure of the picking system 100 shown in FIG. 8 is just one example, and is not limited to this. In FIG. 8, in order to make it easier to understand the relationship between the first process and the second process, an example is shown in which the first process is executed once and the second process is executed N times when picking one target object Tg, but the number of times that the first process and the second process are executed is not limited to this. It goes without saying that the picking system 100 may execute the first process multiple times when picking one target object Tg.

The actuator AC picks up the object Tg being transported on the belt conveyor by the camera CM while capturing images of the object Tg at a predetermined frame rate (e.g., 1000 fps). The actuator AC shown in FIG. 8 shows a part of the picking process of the object Tg (time t11 to time t1N), and is executed repeatedly, for example, until the object Tg is picked.

In addition, in parallel with the second process, the image processing device P1 executes the first process (step St200A) based on the captured image captured by the camera CM, information related to the movement of the object Tg, such as the movement information, obtained by the multiple second processes executed during the first process, and the captured image used in the multiple second processes. The image processing device P1 feeds back to the second process the prediction template TP2 (see FIG. 7), which is a template candidate for the object Tg obtained by the first process.

In the example shown in FIG. 8, the camera CM captures an image of the object Tg at time t11 and transmits the captured image Img11 to the image processing device P1. The image processing device P1 acquires the first captured image Img11 (image data) transmitted from the camera CM and executes the first process and the second process using the first captured image Img11. The image processing device P1 outputs the movement information (information related to movement) of the object Tg obtained by the second process and the first captured image Img11 used in the second process to the first processing unit 110, and transmits information on the predicted position (x1, y1, z1) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired three-dimensional predicted position (x1, y1, z1) of the object Tg.

At time t12, the camera CM captures an image of the object Tg. The camera CM transmits the captured image Img12 to the image processing device P1. The image processing device P1 acquires the second captured image Img12 (image data) transmitted from the camera CM and executes a second process using the second captured image Img12. The image processing device P1 outputs the movement information (information related to movement) of the object Tg obtained by the second process and the second captured image Img12 used in the second process to the first processing unit 110, and transmits information on the predicted position (x2, y2, z2) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired predicted position (x2, y2, z2) of the object Tg.

Then, the picking system 100 repeatedly executes the same process from time t13 to time t1(N-1).

At time t1N, the camera CM captures an image of the object Tg. The camera CM transmits the captured image Img1N to the image processing device P1. The image processing device P1 acquires the Nth captured image Img1N (image data) transmitted from the camera CM and executes the second process using the Nth captured image Img1N. The image processing device P1 outputs the movement information (information related to movement) of the object Tg obtained by the second process and the Nth captured image Img1N used in the second process to the first processing device 110, and transmits information on the predicted position (xN, yN, zN) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired predicted position (xN, yN, zN) of the object Tg to pick up the object Tg. The image processing device P1 feeds back the template Img31 of the object Tg selected by the user's operation to the second processing device 120.

As a result, the image processing device P1 in variant 1 of embodiment 1 can perform template prediction without using the 3D model MD of the object Tg.

(Modification 2 of the First Embodiment)
The picking system 100 according to the first embodiment has shown an example in which a template is predicted using the matching tendency obtained by the second process and information on the movement of the target object Tg. The picking system 100 according to the second modification of the first embodiment will be described below with reference to an example in which a template is generated using a template TP1 obtained based on a user operation.

Note that the internal configuration example of the picking system 100 according to the second variation of the first embodiment has almost the same configuration as the internal configuration example of the picking system 100 according to the first embodiment, so a description thereof will be omitted.

Next, the first process of the image processing device P1 will be described with reference to FIG. 9. FIG. 9 is a flowchart showing an example of the first process procedure (step St200B) of the image processing device P1 in the second modification of the first embodiment.

Note that the example of the operation procedure of the first process shown in FIG. 9 is similar to the example of the operation procedure of the first process shown in FIG. 6 in steps St31 to St32 and steps St34 to St38, and therefore a description thereof will be omitted.

The first processing unit 110 determines whether or not a template TP1 of the target object Tg based on a user operation of operating the icon PP1 and pressing the registration button BT has been registered (St30A).

If the first processing unit 110 determines in the processing of step St30A that a template TP1 of the target object Tg has been registered (St30A, YES), it feeds back the registered template TP1 to the second processing unit 120 instead of the prediction template TP2 (St30B).

On the other hand, if the first processing unit 110 determines in the processing of step St30A that the template TP1 of the object Tg has not been registered (St30A, NO), it acquires the captured image output from the second processing unit 120 (St31).

As described above, the image processing device P1 in the second modification of the first embodiment performs feature matching using the template TP1 specified by the user instead of the prediction template TP2, thereby improving the feature matching accuracy of the second processing unit 120 and also improving the tracking accuracy of tracking the target object Tg.

As described above, the image processing device P1 according to the first embodiment and the first modification of the first embodiment is movable and capable of communicating with a camera CM capable of capturing an image of an object Tg, and executes a first process (steps St200, St200A) in which the image of the object Tg is captured, the position or posture of the object Tg is detected from the captured image, and a prediction template TP2 (an example of a template) of the object Tg is generated. During the first process, the image processing device P1 executes feature matching (an example of template matching) based on the captured image and the prediction template TP2 of the object Tg, and executes a second process multiple times to acquire information regarding the movement of the object Tg. The first process predicts the posture of the object Tg based on the detected position or posture of the object Tg and the information regarding the movement of the object Tg acquired multiple times in the second process, and generates a prediction template TP2 of the object Tg corresponding to the predicted posture of the object Tg.

As a result, the image processing device P1 according to embodiment 1 and variant 1 of embodiment 1 generates a prediction template TP2 by the first process, and tracks the change in posture of the object Tg that changes during the first process based on the feature matching result and the position information of the object Tg output from the second processing unit 120, and can generate a template (prediction template TP2) of the object Tg that is closest to the posture of the actual object Tg.

In addition, the image processing device P1 according to the first embodiment and the first modification of the first embodiment extracts features of the object Tg from the captured image, and performs feature matching based on the extracted features of the object Tg and the features of the object Tg depicted in the prediction template TP2. This allows the image processing device P1 according to the first embodiment and the first modification of the first embodiment to obtain the position of the object Tg depicted in the captured image by position fitting based on feature matching.

The first process in the image processing device P1 according to embodiment 1 performs 3D matching based on the object Tg detected from the captured image and the 3D model MD recorded in a 3D model database DB (an example of a database) to identify the posture of the object Tg, predicts the posture of the object Tg based on information related to the movement of the object Tg obtained by the second process executed multiple times, and generates a prediction template TP2 for the object Tg. This allows the image processing device P1 according to embodiment 1 to obtain the posture of the object Tg appearing in the captured image.

The image processing device P1 according to the first embodiment acquires a first imaging position of the camera CM relative to the 3D model MD based on the predicted orientation of the object Tg, generates a prediction template TP2 corresponding to the predicted orientation based on the 3D model MD and the predicted orientation of the object Tg, and outputs the prediction template TP2 to the display 13 by associating the 3D model MD, the first imaging position relative to the 3D model MD, and the prediction template TP2. This allows the image processing device P1 according to the first embodiment to visualize to the user that the prediction template TP2 is a template (2D image) when the 3D model MD of the object Tg is imaged from the first imaging position. The user can visually confirm whether the 3D model MD of the object Tg recognized by the image processing device P1 is the correct 3D model MD based on the prediction template TP2 and the first imaging position.

The image processing device P1 according to the first embodiment detects the object Tg from the captured image, identifies the detected orientation of the detected object Tg, acquires the second imaging position of the camera CM relative to the 3D model MD based on the detected orientation, generates a detection template TP3 corresponding to the detected orientation based on the 3D model MD and the detected orientation, associates the 3D model MD, the second imaging position relative to the 3D model MD, and the detection template TP3, and outputs them to the display 13. This allows the image processing device P1 according to the first embodiment to visualize to the user that the detection template TP3 is a template (2D image) when the 3D model MD of the object Tg is imaged from the second imaging position. Based on the detection template TP3 and the second imaging position, the user can visually confirm whether the 3D model MD of the object Tg recognized by the image processing device P1 is the correct 3D model MD.

The image processing device P1 according to the first embodiment also acquires specification information that specifies the 3D model MD, and generates a prediction template TP2 of the object Tg based on the 3D model MD that corresponds to the specification information. As a result, the image processing device P1 according to the first embodiment generates a template with a clean background other than the object Tg, i.e., with little noise, enabling more accurate position identification in the feature matching process of the second process. The image processing device P1 also generates a prediction template TP2 in which the position or posture of the object Tg that changes during the first process is corrected, thereby enabling real-time tracking of changes in posture or position of the object Tg captured by the camera CM.

The image processing device P1 according to the second modification of the first embodiment also acquires designation information that designates the template TP1, extracts features of the object Tg from the captured image, and performs feature matching based on the extracted features of the object Tg and the features of the object Tg appearing in the template TP1 that corresponds to the designation information. This allows the image processing device P1 according to the second modification of the first embodiment to acquire the position of the object Tg appearing in the captured image by position fitting based on feature matching.

Furthermore, in the image processing device P1 according to the first embodiment and the first and second variations of the first embodiment, the second processing is executed at a frame rate different from that of the first processing for the image captured by the camera CM. As a result, the image processing device P1 according to the first embodiment and the first and second variations of the first embodiment can track changes in the posture or position of the object Tg captured by the camera CM in real time by combining advanced low-speed image processing (first processing) with a low-level high-speed image processing technique (second processing).

The image processing device P1 according to the first embodiment and the first modification of the first embodiment includes a communication unit 10 (an example of an acquisition unit) that acquires an image of the object Tg captured by a movable camera CM capable of capturing an image of the object Tg, a first processing unit 110 that detects the position or posture of the object Tg from the captured image and generates a template of the object Tg, and a second processing unit 120 that performs template matching based on the captured image and the template of the object Tg multiple times while the first processing unit 110 is generating the template of the object Tg, and acquires information regarding the movement of the object Tg. The first processing unit 110 predicts the posture of the object Tg based on the detected position or posture of the object Tg and the information regarding the movement of the object Tg acquired multiple times by the second processing unit 120, and generates a template of the object Tg corresponding to the predicted posture of the object Tg.

As a result, the image processing device P1 according to embodiment 1 and variant 1 of embodiment 1 generates the prediction template TP2 by the first process, and is able to track the change in posture of the object Tg that changes during the first process based on the feature matching result and the position information of the object Tg output from the second processing unit 120, and generate a template (prediction template TP2) of the object Tg that is closest to the posture of the actual object Tg.

Although various embodiments have been described above with reference to the attached drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, amendments, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components in the various embodiments described above may be combined in any manner as long as they do not deviate from the spirit of the invention.

This application is based on a Japanese patent application (Patent Application No. 2023-073618) filed on April 27, 2023, the contents of which are incorporated by reference into this application.

The present disclosure is useful as an image processing method and image processing device that registers highly accurate templates of an object that can be used for template matching even in a situation where the orientation of the object from the imaging device changes as the imaging device moves.

REFERENCE SIGNS LIST 10 Communication unit 11 Processor 12 Memory 13 Display 14 Operation device 110 First processing unit 120 Second processing unit AC Actuator CM Camera DB 3D model database MD 3D model P1 Image processing device Tg Object

Claims

An image processing method performed by an image processing device that is movable and capable of communicating with a camera that can capture an image of an object, comprising:
A first process is executed to acquire a captured image of the object, detect a position or a posture of the object from the captured image, and generate a template of the object;
executing a second process a plurality of times to execute template matching based on the captured image and a template of the object while executing the first process, and acquiring information regarding a movement of the object;
The first process predicts a posture of the object based on the detected position or posture of the object and information on the movement of the object acquired multiple times in the second process, and generates a template of the object corresponding to the predicted posture of the object.
Image processing methods.
extracting a feature amount of the object from the captured image, and performing template matching based on the extracted feature amount of the object and the feature amount of the object captured in the template;
The image processing method according to claim 1 .
The first process performs 3D matching between the object detected from the captured image and a 3D model recorded in a database to identify a posture of the object, predicts the posture of the object based on information on the movement of the object acquired in the second process executed multiple times, and generates a template of the object.
The image processing method according to claim 1 .
obtaining a first imaging position of the camera relative to the 3D model based on the predicted pose of the object;
generating a predictive template based on the 3D model and the predicted pose of the object, the predictive template corresponding to the predicted pose;
outputting the 3D model, the first imaging position with respect to the 3D model, and the prediction template to a display in association with each other;
The image processing method according to claim 3.
Detecting the object from the captured image and identifying a detected attitude of the detected object;
obtaining a second imaging position of the camera relative to the 3D model based on the detected orientation;
generating a detection template corresponding to the detected pose based on the 3D model and the detected pose;
outputting the 3D model, the second imaging position with respect to the 3D model, and the detection template to a display in association with each other;
The image processing method according to claim 3.
Acquire designation information for designating the 3D model;
generating a template of the object based on a 3D model corresponding to the specified information;
The image processing method according to claim 3.
acquiring designation information designating the template;
Extracting a feature amount of the object from the captured image;
performing template matching based on the extracted feature amount of the object and the feature amount of the object depicted in a template corresponding to the designation information;
The image processing method according to claim 1 .
The second processing is executed at a frame rate different from that of the first processing for the captured image captured by the camera.
The image processing method according to claim 1 .
an acquisition unit that acquires an image of an object captured by a movable camera that can capture an image of the object;
a first processing unit that detects a position or an orientation of the object from the captured image and generates a template of the object;
a second processing unit that performs template matching based on the captured image and the template of the object a plurality of times while the template of the object is being generated by the first processing unit, and acquires information regarding a movement of the object;
The first processing unit predicts a posture of the object based on the detected position or posture of the object and information on the movement of the object acquired multiple times by the second processing unit, and generates a template of the object corresponding to the predicted posture of the object.
Image processing device.