KR20220043847A - Method, apparatus, electronic device and storage medium for estimating object pose - Google Patents

Abstract

The present disclosure relates to an object pose estimation method, a device thereof, an electronic apparatus, and a storage medium. The object pose estimation method determines the reliability of a depth image according to a color image and a depth image of an object, estimates the pose of the object based on three-dimensional key points when the depth image is reliable, and estimates the pose of the object based on two-dimensional key points when the depth image is not reliable.

Description

Object pose estimation method, apparatus, electronic device and storage medium {METHOD, APPARATUS, ELECTRONIC DEVICE AND STORAGE MEDIUM FOR ESTIMATING OBJECT POSE

The following embodiments relate to the field of artificial intelligence and augmented reality technology, and more particularly, to an object pose estimation method, an apparatus, an electronic device, and a storage medium.

Due to the development of information technology and artificial intelligence, the society has increasing requirements for automation and smartization, and technologies for virtual reality, autonomous driving and robots are also receiving more and more attention. Among them, the object pose estimation technology can solve the pose information of the camera for the object and can configure the spatial shape around the object according to the pose information, so it plays an important role in technologies such as virtual reality, autonomous driving, and augmented reality. .

In the prior art, object pose estimation is mainly performed based on a color image. However, since the method is limited in realizing the accuracy of the result in a complex application, the accuracy required for the actual application cannot be achieved.

The conventional object pose estimation method can estimate the pose of an object using a depth image and a color image, but it has been found that this method has an inaccurate pose of the object according to the quality of the depth image. That is, the accuracy of the object pose acquired in a situation where depth data is missing or there are errors and noises is not high.

In addition, when estimating an object 6D pose based on a color image and a depth image, the color image and the depth image are processed using heterogeneous reception, respectively, and features are extracted using a fusion network, and the extracted features are based on the estimation of the pose. However, this method uses only the characteristics of one object, and there is a problem in that it is difficult to accurately estimate the object pose when images overlap or shadows are seen in the object.

In addition, when estimating an object pose by fusing color image features and depth features, a large amount of memory and resources must be consumed, so the efficiency of accurate pose estimation is low, so the real-time requirements of object pose estimation cannot be met. Have.

An object of the present invention is to provide a method, an apparatus, an electronic device, and a storage medium for estimating an object pose.

An object pose estimation method according to an embodiment of the present invention includes: determining reliability of the depth image according to a color image and a depth image of the object; estimating the pose of the object based on a three-dimensional key point when the depth image is reliable; and if the depth image is not reliable, estimating the pose of the object based on a two-dimensional key point.

In this case, the determining of the reliability of the depth image according to the color image and the depth image includes extracting image features based on the color image or based on the color image and the depth image, and the depth image extracting a point cloud feature according to the method, obtaining a fusion feature by fusing the image feature and the point cloud feature, and determining the reliability of the depth image based on the fusion feature.

In this case, the determining of the reliability of the depth image based on the fusion feature may include: acquiring an object instance segmentation image and a depth reliability image based on the fusion feature; The method may include determining the reliability of the depth image corresponding to each target object among the color images according to the object instance segmentation image and the depth reliability image.

In this case, the determining of the reliability of the depth image according to the color image and the depth image includes extracting image features based on the color image and the depth image, and determining the reliability of the depth image according to the image features. may include steps.

In this case, the determining of the reliability of the depth image according to the image feature may include: acquiring an object instance segmentation image and a depth reliability image according to the image feature; and determining the reliability of the depth image corresponding to each target object in the color image according to the object instance segmentation image and the depth reliability image.

In this case, according to the image characteristic, the obtaining of the object instance segmentation image and the depth reliability image may include, according to the image characteristic, obtaining a regional image characteristic of an image region corresponding to each target object; and, for each target object, based on a region image feature corresponding to the target object, determining a depth reliability image of the corresponding target object, and dividing the object instance based on a region image feature corresponding to each target object It may include acquiring an image.

In this case, the method for estimating an object pose includes: acquiring a first appearance feature of each target object and a geometrical relationship feature between each target object based on the color image and the depth image; and for each target object, based on a first appearance characteristic of the target object, a first appearance characteristic of a target object other than the target object, and a geometrical relationship characteristic between the target object and another target object other than the target object, The method may further include determining a second appearance characteristic of the corresponding target object.

In this case, the step of estimating the pose of the object based on the three-dimensional key point may include estimating the pose of each target object according to a fusion feature and a second appearance feature of each target object.

In this case, the step of estimating the pose of the object based on the three-dimensional key point may include estimating the pose of each target object according to an image feature and a second appearance feature of each target object.

At this time, based on the color image and the depth image, the step of obtaining the first appearance feature of each target object and the geometrical relationship feature between the respective target objects may include: based on the color image or the color image and the depth extracting an image feature based on an image, extracting a point cloud feature according to the depth image, fusing the image feature and a point cloud feature to obtain a fusion feature, and based on the fusion feature, the first of each target object The method may include obtaining an appearance feature and an object instance segmentation image, and obtaining a geometrical relationship feature between the respective target objects based on the object instance segmentation image.

At this time, based on the color image and the depth image, the step of obtaining the first appearance feature of each target object and the geometrical relationship feature between each target object includes: image features based on the color image and the depth image extracting, obtaining a region image feature corresponding to the image region of each target object according to the image feature, and a first appearance feature of each target object based on the region image feature corresponding to the image region of each target object and obtaining a corresponding object detection result, and obtaining a geometrical relationship characteristic of each target object based on the object detection result of each target object.

In this case, the method for estimating the object pose may further include determining whether the video frame is an initial frame by detecting whether the target object or the target pose first appears in the video frame.

In this case, the step of determining whether the video frame is an initial frame by detecting whether the target object or the target pose first appears in the video frame may include: obtaining an image bounding box of each target object in the video frame; matching the image bounding box of each target object with the image bounding box corresponding to each target object in each pose result list; If there is a matching target object in the pose result list, the first point cloud data of the image bounding box corresponding to each target object in the corresponding video frame and the corresponding target object in the previous video frame of the corresponding video frame By comparing second point cloud data frames, it is determined whether there is a difference between the first point cloud data and the second point cloud data, and if there is a difference, it is determined that the pose of the object corresponding to the target object appears first. to do; and when there is no matching target object in the object pose result list, determining that the target object appears first.

In this case, the method for estimating an object pose is, if the video frame is not an initial frame, obtains a motion parameter corresponding to the video frame, and based on the motion parameter and an object pose result of an initial frame corresponding to the video frame, determining a pose result corresponding to the video frame; and updating an object pose result of the initial frame corresponding to the video frame in a pose result list according to the pause result corresponding to the video frame.

An apparatus for estimating an object pose according to an embodiment of the present invention includes: an image reliability determining unit configured to determine the reliability of the depth image according to a color image and a depth image of the object; and a pose configured to estimate the pose of the object based on a three-dimensional key point when the depth image is reliable, and to estimate the pose of the object based on a two-dimensional key point when the depth image is unreliable. Includes estimates.

1 is a flowchart illustrating a process of estimating an object pose according to an exemplary embodiment.
2 is a diagram illustrating a process of estimating an object pose based on a single feature extraction according to an embodiment.
3 is a diagram illustrating a process of estimating an object pose based on two times of feature extraction according to an embodiment.
4 is a flowchart illustrating a process of estimating an object pose based on a depth image according to an exemplary embodiment.
5 is a flowchart illustrating a process of acquiring a fused feature by fusing an image feature and a point cloud feature, according to an embodiment.
6 is a diagram illustrating a process of estimating an object pose based on one-time feature extraction and using a second appearance feature of a target object, according to an embodiment.
7 is a diagram illustrating a process of estimating an object pose based on two-time feature extraction and using a second appearance feature of a target object, according to an embodiment.
8 is a flowchart illustrating a process of estimating an object pose according to another exemplary embodiment.
9 is a diagram illustrating a process of estimating an object pose based on a color image and a depth image, according to an embodiment.
10 is a flowchart illustrating a process of estimating an object pose based on an initial frame of a video according to an embodiment.
11 is a diagram illustrating an example of a method for selecting three initial frames according to an embodiment.
12 is a flowchart illustrating a process of determining whether a video frame is an initial frame based on object detection, according to an embodiment.
13 is a flowchart illustrating a process of confirming the existence of a new object or a new pose, according to an exemplary embodiment.
14 is a diagram illustrating a schematic configuration of an apparatus for estimating an object pose according to an embodiment.
15 is a diagram illustrating a schematic configuration of an electronic device according to an exemplary embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all modifications, equivalents and substitutes for the embodiments are included in the scope of the rights.

The terms used in the examples are used for the purpose of description only, and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that a feature, number, step, operation, component, part, or a combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are given the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

In addition, in describing the components of the embodiment, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms. When it is described that a component is "connected", "coupled" or "connected" to another component, the component may be directly connected or connected to the other component, but another component is between each component. It will be understood that may also be "connected", "coupled" or "connected".

Components included in one embodiment and components having a common function will be described using the same names in other embodiments. Unless otherwise stated, descriptions described in one embodiment may be applied to other embodiments as well, and detailed descriptions within the overlapping range will be omitted.

Related technologies will be described below for better understanding of the present disclosure.

An object is an object with six degrees of freedom in space, also called a six-degrees of freedom (DOF) object, such as a cup, monitor, or book.

Augmented reality technology adds virtual content to the display scene in front of the user, providing the user with a real information experience. In order to complete the high-quality virtual reality fusion effect in front of the user, the augmented reality system in the three-dimensional space requires high-precision real-time processing and understanding of the three-dimensional state of the surrounding objects.

Pose estimation expresses the structure and shape of an object using a geometric model or structure, establishes a correspondence relationship between a model and an image by extracting object features, and estimates the spatial pose of an object by geometric or other methods. The model used here may be a simple geometric body such as a plane, cylinder, or some kind of geometry, or it may be a 3D model obtained by laser scanning or other methods.

6D pose estimation of an image, for a given single image containing color and depth information, a 6D pose of a target object in the image is estimated. A 6D pose includes a three-dimensional position and a three-dimensional spatial orientation, also referred to as a 6-DOF pose.

A multi-layer perceptron (MLP) network is a multi-layer neural network, which is a kind of forward-structured artificial neural network.

Semantic segmentation is the separation of the target object from the background.

Object detection is the determination of the location of an object in an image.

Instance segmentation determines which pixels belong to each target object.

The present disclosure relates to a method for estimating an object pose for solving at least one of the above problems, and when estimating an object pose based on a color image and a depth image, the accuracy of the object pose estimation can be effectively improved, and the reliability of the depth image May not depend on possible quality.

Hereinafter, according to an embodiment of the present disclosure An object pose estimation method, an apparatus, an electronic device, and a storage medium will be described in detail with reference to the accompanying FIGS. 1 to 15 .

1 is a flowchart illustrating a process of estimating an object pose according to an exemplary embodiment.

Referring to FIG. 1 , the apparatus for estimating an object pose determines reliability of a depth image according to a color image and a depth image of the object ( 110 ).

Then, when the depth image is reliable, the object pose estimation apparatus estimates the pose of the object based on the 3D key point ( 120 ).

Then, when the object pose estimation apparatus does not trust the depth image, it estimates the pose of the object based on the two-dimensional key point ( 130 ).

In this case, the color image and the depth image are images including the same object in a situation corresponding to the same scene, and the type and number of objects are not limited thereto. Embodiments of the present disclosure do not limit a method of acquiring a color image and a corresponding depth image. For example, the color image and the depth image may be the color image and the depth image simultaneously acquired through the general image acquisition device and the depth image acquisition device, and also the image acquisition device or video acquisition having both the depth image and color image acquisition function may be obtained by the device. For example, an RGBD (Red+Green+Blue+depth) image (an image including three primary colors and depth information of red/green/blue) is acquired through an image acquisition device, and the depth image and color image are It may be obtained based on

In this case, in the case of a color image, the object pose estimation apparatus may configure and select whether to use a gray image or a color image according to actual application requirements. For example, the object pose estimation apparatus may use a color image to obtain a better pose estimation effect, and may use a gray image to increase the efficiency of pose estimation. For convenience of explanation, a color image will be used as an example in the following description.

In practical applications, continuous and real-time object pose estimation is required in many application scenes. At this time, optionally, the RGBD video of the object may be collected through video acquisition equipment, and each frame image of the video is an RGBD image, and may be obtained by extracting the color image and depth image of the object from the same video frame, For each video frame, all corresponding color images and depth images can be obtained. Objects of the color image and the depth image corresponding to the same video frame are identical, and there may be more than one object. For a color image and a depth image corresponding to each obtained video frame, real-time estimation of an object pose may be realized based on the method of the present disclosure.

The object pose estimation apparatus may extract and process image features according to the color image and the depth image, and determine reliability of the depth image using the processed image features. Specifically, the reliability determination of the depth image may be the reliability determination of the depth data of the depth image. In general, a depth image includes various data such as depth data and contour data, and determining the reliability of the depth image may specifically determine whether the depth data of the depth image is reliable.

Determination of the reliability of the depth image may be obtained by setting a reliability threshold or by setting a threshold for the ratio of reliability pixels. When the reliability corresponding to the depth image exceeds the set reliability threshold or the ratio of statistically reliable pixels exceeds the set ratio threshold, this indicates that the depth image can be trusted.

The object pose estimation apparatus estimates the pose of the object based on the three-dimensional key point when the depth image is reliable. And, when the object pose estimation apparatus cannot trust the depth image, the object pose estimation apparatus estimates the pose of the object based on the two-dimensional key point.

At this time, the two-dimensional key point can be obtained using only a color image, and in this situation, among the network structures described in the present disclosure, the MLP network in which the input is a point cloud feature format for two-dimensional key point extraction, the input is the image feature It is replaced by a CNN network in the form In addition, both two-dimensional key points and three-dimensional key points can be obtained based on depth images and color images.

When the depth image is reliable, the object pose estimation apparatus estimates the pose of the object based on the three-dimensional key point. Optionally, the object pose estimation apparatus may obtain a three-dimensional key point based on the depth image and the color image, and estimate the pose of the object using the obtained three-dimensional key point. The data of the corresponding three-dimensional key point includes depth data provided by the depth image. Compared to a method using only a color image for object pose estimation, the object pose estimation apparatus of the present disclosure estimates an object pose based on a color image and a depth image. This increases the size of the underlying data, which helps to improve the accuracy and precision of object pose estimation. In addition, accurate object poses can be obtained even under conditions such as object occlusion, sensor noise, and insufficient lighting.

When the depth image is unreliable, the object pose estimation apparatus estimates the pose of the object based on the two-dimensional key point, and optionally, obtains the two-dimensional key point of the image according to the contour data of the color image and the depth image, , the pose is estimated using the obtained two-dimensional key points. Since the depth data of the depth image is unreliable, in this case, when the object pose is estimated based on the depth data, an error may occur in the finally obtained object pose. Therefore, when the depth image is unreliable, the object pose estimation apparatus may estimate a pose based on the object contour data of the color image and the depth image without relying on the integrity of the depth data of the depth image. Therefore, in a situation where depth data is missing or there are errors and noise, pose estimation based on color images and object contour data improves the accuracy of the estimated object poses, compared to object pose estimation methods based on color images only. can be improved

In the present disclosure, an apparatus for estimating an object pose estimates an object pose using a depth image and a color image. First, the object pose estimation apparatus determines the reliability of the depth image according to the color image and the depth image. When the depth image is reliable, the object pose estimation device estimates the pose of the object based on the three-dimensional key point, and when the depth image is not reliable, that is, in the situation where the depth data is missing or there are errors and noise, The apparatus for estimating an object pose may reduce dependence on a complete depth image and increase accuracy of object pose estimation by estimating a pose of an object based on a two-dimensional key point. In addition, the robustness of object pose estimation may be improved by adaptively selecting a depth image or a color image to estimate the object pose according to the reliability determination result of the depth image.

Hereinafter, each optional embodiment provided in the present disclosure will be described in detail.

In an optional embodiment, determining the reliability of the depth image according to the color image and the depth image of the object may be obtained through any one of the following methods A and B.

Option A may include the following steps.

A1: Extract image features based on color images, or based on color images and depth images.

A2: Extract point cloud features according to the depth image.

A3: By fusing the image feature and the point cloud feature, the fusion feature is obtained.

A4: Determine the reliability of the depth image based on the fusion feature.

It should be understood that there may be no back-and-forth order between some of the above steps, for example, there may be no order between steps A1 and A2.

Method A provided in the embodiment may extract image features based on a color image, and may extract image features based on a color image and a depth image. In this case, the image feature extraction may be performed through an image feature extraction network, an image feature extraction algorithm, or the like. Since the subsequent step of method A will fuse image features and point cloud features, in a situation where image features are extracted based on only color images, the fused features include depth data and contour data of the depth image.

A solution provided by an alternative embodiment is illustrated in FIG. 2 . Detailed description of each step of FIG. 2 will be described later. In method A, features are extracted based on color images and depth images, and image features are extracted through an image feature extraction network. That is, method A inputs a color image and a depth image to the image feature extraction network. If the color image is color, the input to the image feature extraction network is a four-channel image of H*W obtained by pixel-by-pixel splicing of the color image and the depth image. In this case, H is the height of the image, W is the width of the image, and the 4 channels are 3 channels corresponding to RGB data of the color image and a channel corresponding to the depth data of the depth image, respectively, the output of the image feature extraction network contains the image feature vector of each pixel.

Method A may optionally extract point cloud features according to the depth image. First, method A converts the point cloud for the depth image to obtain the point cloud data corresponding to the depth image, and then extracts the point cloud feature based on the point cloud data again to obtain the point cloud feature corresponding to the depth image can do. In this case, the point cloud feature extraction may be performed through the point cloud feature extraction network. The input of the point cloud feature extraction network is point cloud data, and the output of the point cloud feature extraction network includes a point cloud feature vector of each three-dimensional point, thereby obtaining a point cloud feature vector of each pixel. In this case, the point cloud feature vector may be characterized as a geometric feature vector.

Method A fuses the acquired image features and point cloud features to obtain a fusion feature. Specifically, for each pixel included in the image, method A fuses the image feature vector of each pixel obtained by image feature extraction and the point cloud feature vector of each pixel obtained by point cloud feature extraction for each pixel, Acquire fusion characteristics.

Optionally, the fusion operation may be high-density fusion, and the fusion operation may be splicing an image feature vector and a point cloud feature vector, or projecting an image feature vector and/or a point cloud feature vector to another feature space, The feature vector of the corresponding feature space is used as the fusion feature of the corresponding pixel. In this case, the fusion operation may filter out interference information of the image and reduce information duplication of the fusion feature through a fusion method of projecting an image feature vector and/or a point cloud feature vector to another feature space.

Method A for judging the reliability of the depth image based on the fusion feature, compared to the method for judging the reliability using a color image alone, the judgment result is more accurate and helps to increase the judgment efficiency of the depth image reliability becomes this

Optionally, the process of determining the reliability of the depth image based on the corresponding fusion feature is as follows. Acquire a depth confidence image composed of the depth data of each pixel based on the fusion feature, and determine whether the depth image is reliable according to the depth data of each pixel of the depth confidence image and a preset confidence threshold or proportionality threshold do.

Option B may include the following steps.

B1: Extract image features based on color images and depth images.

B2: Determine the reliability of the depth image according to the image features.

Method B provided in the optional embodiment is shown in FIG. 3 , and a detailed description of each step of FIG. 2 will be described later. Method B uses an image feature extraction network to extract features of color images and depth images to obtain image features. In the method B, when there are at least two target objects in the image, the region is divided for the image region of the suspected target object, and an image region (ROI, Region of Interest) of each target object is obtained. Method B performs a pooling process on an image region to obtain image features of the image region, and according to the pooled image features, acquires a depth reliability image corresponding to a target object. Method B compares the depth reliability image corresponding to the target object with a preset reliability threshold or proportional threshold to determine the reliability of image data.

In an optional embodiment, determining the reliability of the depth image based on the fusion feature, which may proceed in a manner comprising the following steps.

C1: Acquire an object instance segmentation image and a depth confidence image based on the fusion feature.

C2: According to the object instance segmentation image and the depth reliability image, the reliability of the depth image corresponding to each target object among the color images is determined.

As shown in FIG. 2 , the object pose estimation apparatus obtains the fusion feature of the image feature and the point cloud feature, and then can predict the reliability of the depth image based on the fusion feature, and determine the depth reliability image corresponding to the image. can Also, the apparatus for estimating an object pose may perform semantic segmentation and center offset estimation based on the fusion feature to segment an object instance, and obtain an object instance segmentation image. Then, according to the object instance segmentation image and the depth reliability image, the reliability of the depth image corresponding to each target object in the color image is determined.

Specifically, the apparatus for estimating an object pose may obtain a pixel of an image region corresponding to each target object according to the object instance segmentation image, and may obtain depth data of each pixel of the corresponding image region according to the depth reliability image. That is, the object pose estimation apparatus may obtain a depth image corresponding to each target object. The object pose estimation apparatus may determine the reliability of the depth image corresponding to each target object according to a preset reliability threshold or a proportional threshold. Specifically, when the depth data of a depth image of a certain target object exceeds a preset threshold or the statistically reliable ratio of pixels is greater than a preset threshold, the object pose estimation apparatus determines that the depth image of the target object is If the image is a reliable image, it may be determined that the depth image of the corresponding target object is not reliable.

Since there may be more than one target object in the image, when some target objects are reliable and some are unreliable, the reliability result of each target object cannot be obtained according to the reliability determination result of the depth image corresponding to the entire image. For the problem, in the method provided by the present disclosure, an object instance segmentation image corresponding to each target object is obtained using a fusion feature, then a depth image corresponding to each target object is obtained, and a preset reliability threshold is combined. to determine the reliability of the depth image corresponding to each target object to determine the reliability of each target object in the image.

Alternatively, the determination of the reliability of the depth image may be an overall determination rather than a separate determination of the object. That is, even if there is one or more target objects of the image, the entire image may be determined and a determination result may be obtained. If the determination result is reliable, the object pose estimation apparatus may estimate the pose of the object using a pose estimation method based on a three-dimensional key point for each target object. When the determination result is not reliable, the object pose estimation apparatus may estimate the pose of the object using a pose estimation method based on a two-dimensional key point for each target object. The method can reduce the data processing amount in the reliability determination process and increase the determination efficiency.

In an optional embodiment provided by the present disclosure, determining the reliability of the depth image according to the image feature may include the following steps.

D1: According to the image feature, obtain an object instance segmentation image and a depth reliability image.

D2: According to the object instance segmentation image and the depth reliability image, the reliability of the depth image corresponding to each target object among the color images is determined.

As shown in FIG. 3 , image features are acquired based on a color image and a depth image, and region processing is performed on the image features. The zoning process may be performed through the region suggestion network, and an image region that may belong to the target object may be extracted. The object pose estimation apparatus obtains an object instance segmentation image and a depth reliability image corresponding to the target object based on the image region of each target object, and selects a depth image corresponding to the target object among them according to the object instance segmentation image and the depth reliability image is determined, and the reliability of the depth image corresponding to the target object is determined according to a result of comparing the depth data of each pixel in the depth image with a preset threshold value.

The method provided in this embodiment is to determine the reliability of the depth image corresponding to the target object based on the image feature, segment the target object based on the image feature, and obtain a depth image corresponding to each target object, It reduces the data throughput for obtaining the reliability judgment of the depth image.

According to an optional embodiment, the apparatus for estimating an object pose may acquire an object instance segmentation image and a depth reliability image according to image characteristics. This can be implemented in a way comprising the following steps.

E1: According to the image feature, acquire a regional image feature of the image region corresponding to each target object.

E2: For each target object, based on the area image feature corresponding to the target object, determine the depth reliability image of the target object, and based on the area image feature corresponding to each target object, the object instance segmentation image acquire

As illustrated in FIG. 3 , the apparatus for estimating an object pose may determine an image feature based on a color image and a depth image, and may perform regionalization processing on the image feature.

The apparatus for estimating an object pose may segment a target object through a region proposal network or a region segmentation algorithm, and may obtain an image region corresponding to each target object. The object pose estimation apparatus may again extract features based on an image region corresponding to each target object to obtain a region image feature, and through this, the image may be segmented in the target object layer. In this case, the region image feature may be obtained by pooling the object image region (ROI pooling).

The apparatus for estimating an object pose may process a corresponding region image feature for each target object through the entire connection layer of the neural network, based on the region image feature corresponding to the target object. And, after determining the depth reliability image corresponding to the target object, the apparatus for estimating the object pose may determine whether the depth reliability image is reliable by combining it with a preset threshold value of depth data corresponding to the target object.

For each target object, the object pose estimation apparatus performs object instance segmentation on the image using a neural network (eg, convolutional neural network) based on the region image feature corresponding to the target object, and generates the object instance segmentation image. It is possible to obtain the outline information of the target object and detailed information of the object within the outline through the object instance segmentation image.

Based on this, the object pose estimation apparatus determines a depth image corresponding to each target object according to the object instance segmentation image and the corresponding depth reliability image, and the depth of the target object according to a preset threshold value corresponding to the depth image The reliability of the image can be determined.

The method provided in the present disclosure is to segment an area of a target object layer based on image features, and obtain an accurate segmentation of the target object by acquiring the outline and detailed information of the target object through neural network processing, which is It helps to improve the judgment accuracy of the reliability result of obtaining the corresponding depth image.

In the corresponding embodiment, the object pose estimation apparatus additionally estimates the pose of the object based on the depth image, or the pose of the object based on the color image, based on the object instance segmentation image and the region image feature corresponding to each target object. It can be implemented to estimate . The specific steps are as follows.

E3: According to the object instance segmentation image and the corresponding region image feature of each target object, obtain an image feature of each target object.

E4: For each target object, estimate the pose of the object based on the image feature of the target object.

As shown in FIG. 3 , the apparatus for estimating an object pose may process, for example, multiply a segmented image of an object instance and a corresponding region image feature to obtain an image feature corresponding to the target object.

For each target object, the image characteristics of that target object are combined with the basic data of object pose estimation (e.g., basic data of object pose estimation based on two-dimensional key points, or object pose based on three-dimensional key points). (basic data of estimation) to obtain a two-dimensional key point offset image or a three-dimensional key point offset image of an object according to the image characteristics of the target object, the accuracy of the two-dimensional key point offset image and the three-dimensional key point offset image helps to improve

In step 120 of FIG. 1 , the object pose estimation apparatus estimates the pose of the object based on the 3D key point. Step 120 may proceed in the same manner as in FIG. 4 below.

4 is a flowchart illustrating a process of estimating an object pose based on a depth image according to an exemplary embodiment.

Referring to FIG. 4 , the object pose estimation apparatus extracts image features based on a color image or a color image and a depth image ( S410 ).

Then, the object pose estimation apparatus extracts a point cloud feature based on the depth image ( 420 ).

Then, the object pose estimation apparatus acquires a fusion feature by fusing the image feature and the point cloud feature ( 430 ).

Then, the object pose estimation apparatus estimates the pose of the object according to the fusion feature ( 440 ).

The method provided in the present disclosure may extract image features based on a color image, and may extract image features based on a color image and a depth image. In this case, the image feature extraction may be performed through an image feature extraction network, an image feature extraction algorithm, or the like. Thereafter, since the fusion of the image feature and the point cloud feature proceeds, even when the image feature is extracted based on only the color image, the fused feature includes data such as depth data and contour data of the depth image.

The method provided in the optional embodiment may be as shown in FIG. 2 . In this method, image features are extracted through an image feature extraction network based on a color image and a depth image. That is, the object pose estimation apparatus may extract image features by inputting the color image and the depth image to the image feature network. In this case, the output of the feature extraction network includes the image feature vector of each pixel.

Optionally, the object pose estimation apparatus extracts a point cloud feature according to the depth image. First, the object pose estimation apparatus transforms the point cloud for the depth image, obtains point cloud data corresponding to the depth image, extracts point cloud features based on the point cloud data, and points cloud features corresponding to the depth image to acquire The object pose estimation apparatus may extract a point cloud feature through a point cloud feature extraction network. In this case, the input of the cloud feature extraction network is point cloud data, and the output includes the point cloud feature vector of each three-dimensional point.

The object pose estimation apparatus obtains a fusion feature by fusing the acquired image feature and the point cloud feature. Specifically, for each pixel included in the image, for each pixel, the image feature vector and the point cloud feature vector of the pixel are fused to obtain a fusion feature. The fusion operation may be a high density fusion.

The object pose estimation apparatus may determine the reliability of the depth image based on the fusion feature. Judging the reliability of the depth image based on the fusion feature compared to the method of judging the reliability using a color image alone, the judgment result is much more accurate, and it helps to increase the efficiency of obtaining the reliability of the depth image. do.

Acquiring a two-dimensional key point offset image and a three-dimensional key point offset image based on the fusion feature can obtain a two-dimensional key point offset image and a three-dimensional key point offset image through deep learning. Then, the object pose estimation apparatus estimates the pose of the object according to the reliability determination result of the depth image, and if the depth image is reliable, estimates the pose of the object according to the three-dimensional key point offset image, and the depth image can be trusted. If there is none, the pose of the object is estimated according to the two-dimensional key point offset image.

As shown in FIG. 3 , the object pose estimation apparatus performs segmentation processing on the target object using image features to obtain instance segmentation image features of the target object, and extracts point cloud features from the depth image to instance After acquiring the segmented point cloud feature, fusing the segmented image feature and the point cloud feature, estimating a three-dimensional key point offset image based on the fusion feature, and then estimating the pose of the object based on the three-dimensional key point offset image do.

In this case, the three channels of the three-dimensional key point offset image represent a deviation vector from the 3D coordinates of the object point corresponding to the pixel to the 3D coordinates of the reference key point preset on the object.

Specifically, estimating the pose of the object according to the 3D key point offset image may be performed in the following way. The object pose estimation apparatus determines a straight line passing through a preset reference key point of the object in an arbitrary 3D pixel of the target object. The apparatus for estimating an object pose may obtain N straight lines passing through a preset reference key point according to a three-dimensional key point offset image corresponding to all N pixels of a pixel region of a corresponding target object. Then, the object pose estimation apparatus may determine the three-dimensional coordinates of the preset reference key point of the target object through a voting method. After obtaining the three-dimensional coordinates of the M key points on the target object in this way, the M corresponding points on the three-dimensional model of the object are estimated by least squares, and the three-dimensional rotation and three-dimensionality between the two groups of three-dimensional points By obtaining the transform, we can get the 6D pose of the object.

As shown in FIG. 3 , the object pose estimation apparatus extracts image features based on the color image and the depth image, and determines the reliability of the depth image based on the image features to obtain the reliability determination result of the depth image. there is. When the depth image is reliable, the object pose estimation apparatus fuses the image feature and the point cloud feature extracted based on the depth image to obtain a three-dimensional key point based on the fused feature.

When there are a plurality of target objects, the apparatus for estimating an object pose fuses an image feature and a point cloud feature corresponding to each target object, and acquires a three-dimensional key point based on the fused feature.

Specifically, the object pose estimation apparatus acquires the fusion feature by fusion according to the image feature and the point cloud feature.

The process of object pose estimation based on the fusion image is as follows. The object pose estimation apparatus estimates a three-dimensional key point offset image according to a fusion characteristic of a target object by using a 3D image drawing tool such as an MPL tool, and estimates an object pose based on the three-dimensional key point offset image.

Meanwhile, the step of estimating the pose of the object based on the two-dimensional key point provided in step 130 of FIG. 1 may be performed in the following manner.

F1: Extract image features based on color image and depth image.

F2: Estimate the pose of the object according to the image features.

This method is a method of estimating a pose of an object based on a color image when the depth image is unreliable. The object pose estimation apparatus acquires contour data of an object from among the depth images based on the depth image, and includes a color image and a depth The pose of the object is estimated according to the contour data of the object in the image. Since depth data of an object is not included in the object pose estimation standard, object pose estimation according to image features is also called two-dimensional key point-based object pose estimation.

Optionally, the pose estimation of the object based on the image features may be performed in a method comprising: The object pose estimation apparatus estimates a two-dimensional key point offset image of a corresponding target object according to image characteristics of the target object using a convolutional neural network, and estimates a pose of the object based on the two-dimensional key point offset image.

The apparatus for estimating an object pose optionally obtains a region image feature by regionalizing the target object according to the image feature, and obtains an instance segmentation image feature based on the region image feature. Then, the object pose estimation apparatus estimates a two-dimensional key point offset image by using the instance segmentation image feature as an input of the convolutional neural network, and estimates the pose of the object based on the two-dimensional key point offset image.

In this case, two channels of each pixel in the two-dimensional key point offset image represent a deviation vector from the two-dimensional coordinates of the object point corresponding to the pixel to the two-dimensional coordinates of the reference point preset in the target object.

Specifically, estimating the pose of the object according to the two-dimensional key point offset image may be performed in the following way. The object pose estimation apparatus determines a straight line passing through the preset reference point in an arbitrary pixel of the target object. The object pose estimation apparatus may obtain N straight lines passing through the preset reference point according to the two-dimensional key point offset image corresponding to all N pixels of the pixel region of the corresponding target object. Then, the object pose estimation apparatus may obtain the two-dimensional coordinates of the corresponding preset reference point in the target object through a voting method. In this way, the object pose estimation apparatus obtains the two-dimensional coordinates of M preset reference points on the target object, and then uses the corresponding points of the M key points on the three-dimensional model according to the PnP (Perspective-n-Point) algorithm. By calculating the 3D rotation and 3D movement between the camera coordinate system and the object coordinate system, a 6D pose of the object can be obtained.

The step of acquiring the fusion feature by fusing the image feature and the point cloud feature provided in step 430 of FIG. 4 may be implemented in the manner of FIG. 5 below.

5 is a flowchart illustrating a process of acquiring a fused feature by fusing an image feature and a point cloud feature, according to an embodiment.

Referring to FIG. 5 , the apparatus for estimating an object pose obtains an object instance segmentation image according to an image feature ( 510 ).

Then, the apparatus for estimating the object pose acquires image features and point cloud features of each target object according to the segmented image of the object instance ( 520 ).

Then, the apparatus for estimating an object pose acquires a fusion characteristic of each target object by fusing an image characteristic and a point cloud characteristic of the same target object ( 530 ).

As shown in FIG. 3 , the apparatus for estimating an object pose obtains a region image feature by performing regionalization processing based on the image feature, and obtains an object instance segmentation image by performing object instance segmentation based on the region image feature. The apparatus for estimating the object pose acquires an instance segmentation image feature, that is, an image feature of the target object, according to the region image feature and the object instance segmentation image. Then, the object pose estimation apparatus acquires the instance segmentation geometrical feature, that is, the point cloud feature of the target object, according to the instance segmentation image and the point cloud feature. Then, the object pose estimation apparatus acquires the fusion feature of each target object by fusing the image feature and the point cloud feature corresponding to the same target object.

The embodiment of FIG. 3 provides another method for acquiring the fusion feature of a target object, and the method can be applied when one or more target objects exist in a color image, extracting features from a color image and a depth image, and Compared to the fusion method, it is helpful to subdivide the image features and point cloud features of each target object through processing such as regionalization processing and instance segmentation, and to obtain more accurate fusion features through two feature extractions.

Then, the object pose estimation apparatus estimates the pose of the object according to the fusion feature, which includes the following steps. The object pose estimation apparatus estimates a pose of each target object according to a fusion characteristic of each target object.

In this case, the object pose estimation apparatus estimates the pose of the corresponding target object according to the fusion characteristic of each target object, and the following operations are performed for each target object.

The object pose estimation apparatus estimates a three-dimensional key point offset image based on the fusion characteristic of the target object, then estimates the pose of the corresponding target object based on the three-dimensional key point offset image, and in turn acquires the pose of each target object do.

When there are multiple target objects, the apparatus for estimating the target object acquires accurate fusion characteristics of each target object by segmentation processing and instance segmentation of each target object, and then estimating the pose of each target object based on the fusion characteristics. . According to the fusion characteristic of the target object, the method of estimating the pose of each target object is effective to obtain accurate poses of all target objects in the color image.

For a situation in which there are a plurality of target objects in an image, the present disclosure provides an embodiment of estimating a pose based on an object relationship of the target objects.

H1: The object pose estimation apparatus acquires, based on the color image and the depth image, a first appearance feature of each target object and a geometrical relationship feature between each target object.

H2: for each target object, the apparatus for estimating the object pose, for each target object, includes a first appearance feature of the corresponding target object, first appearance features of other target objects other than the corresponding target object, and a geometrical relationship between the target object and other target objects other than the corresponding target object. Based on the relational feature, a second appearance feature of the corresponding target object is acquired.

As shown in FIG. 6 , the object pose estimation apparatus performs feature extraction and fusion according to the color image and the depth image, and based on the fusion feature, the first appearance feature of each target object and the geometrical relationship feature between each target object can be obtained. there is. Step-by-step detailed description of FIG. 6 will be described later. In this case, the apparatus for estimating an object pose obtains an object instance segmentation image based on the fusion feature, and obtains a geometrical relationship feature between each target object based on the object instance segmentation image.

In this case, a method of acquiring the first appearance feature of each target object based on the fusion feature may be implemented in the following way. The object pose estimation apparatus predicts the center point coordinates of the target object to which each pixel belongs and the appearance characteristics of the target object from the input data through a voting method based on the fusion characteristic, and the predicted center point coordinates through a clustering method A plurality of target objects and pixels of each target object are acquired, and for each target object, pixel features belonging to the respective target objects are fused to obtain a first appearance feature of each target object.

As shown in FIG. 7 , the object pose estimation apparatus extracts image features based on a color image and a depth image, obtains regional image features through segmentation processing based on the image features, and based on the regional image features. A first appearance characteristic of each target object and a geometrical relationship characteristic between each target object may be obtained. Step-by-step detailed description of FIG. 7 will be described later. In this case, the object pose estimation apparatus detects the object based on the area image feature, and acquires the geometrical relationship feature between each target object based on the object detection result.

For each target object, the apparatus for estimating the object pose determines, for each target object, a first appearance characteristic of the target object, a first appearance characteristic of another target object other than the corresponding target object, and a geometrical relationship characteristic between the corresponding target object and another target object other than the target object. Based on the determination, a second appearance characteristic of the target object is determined, and a pose of the object is estimated based on the second appearance characteristic.

Meanwhile, optionally, the apparatus for estimating the object pose may estimate the pose of the object according to the following method according to the fusion characteristic. The object pose estimation apparatus estimates the pose of each target object according to the fusion characteristic and the second appearance characteristic of each target object.

As shown in FIG. 6 , the object pose estimation apparatus fuses the relational features according to the fusion feature of the target object and the second appearance feature corresponding to the target object, and a two-dimensional key point offset image and a three-dimensional key point offset image to estimate The object pose estimation apparatus estimates the pose of the object based on the two-dimensional key point offset image when the depth image is not reliable, and estimates the pose of the object based on the three-dimensional key point offset image when the depth image is reliable. .

Embodiments of the present disclosure further provide another method, and may estimate a pose of an object according to a fusion feature. 7 , the apparatus for estimating an object pose performs regionalization based on the image feature to obtain a region image feature, and divides the object instance based on the region image feature to obtain an object instance segmentation image, and a region image Obtain a segmented image feature based on the feature and object instance segmentation image, fuse the feature based on the segmented image feature, the segmented point cloud feature, and the second appearance feature of the target object, and three-dimensionally based on the fusion feature Estimate the key point offset image, and estimate the pose of the object based on the estimated three-dimensional key point offset image.

The method of estimating the pose of the object according to the image feature estimates the pose of each target object according to the image feature and the second appearance feature of each target object.

Specifically, the object pose estimation apparatus extracts image features based on a color image and a depth image, acquires regional image features based on image features, acquires segmented image features based on regional image features, and selects a target object Estimate a two-dimensional key point offset image according to an image feature of and a corresponding second appearance feature, and estimate a pose of the target object based on the estimated two-dimensional key point offset image.

The method of the present disclosure introduces a relationship feature of a target object, extracts and fuses geometric relationship information between several objects, and adds first appearance features of different target objects to determine a second appearance feature of the target object, This improves the accuracy and robustness of object occlusion, field cutting, and object pose estimation methods in small object situations.

In an embodiment of the present disclosure, the obtaining of the first appearance feature of each target object and the geometrical relationship feature between each target object based on the color image and the depth image may include at least one of the following methods (Scheme 1 (P1-P5) ), method 2 (Q1-Q4)).

Method 1 (P1-P5) is as follows.

P1: Image feature extraction based on a color image, or a color image and a depth image.

P2: Extract point cloud data according to the depth image.

P3: Acquire a fusion feature by fusing the image feature and the point cloud feature.

P4: Acquire a first appearance feature and an object instance segmentation image of each target object based on the fusion feature.

P5: Based on the object instance segmentation image, obtain geometrical relationship features between each target object.

In Scheme 1, the apparatus for estimating an object pose determines an appearance feature and an object instance segmentation image of each target object based on the fusion feature, and acquires a geometrical relationship feature between each target object based on the object instance segmentation image. Method 1 determines the appearance feature of the target object and the geometrical relationship feature between each target object based on one-time feature extraction, and it is possible to quickly grasp the appearance feature of the target object and the geometrical relationship feature between the target object.

Option 2 (Q1-Q4) is as follows

Q1: Extract image features based on color images and depth images.

Q2: Obtain an image feature corresponding to an image region of each target object according to the image feature.

Q3: Acquire a first appearance feature of each target object and a corresponding object detection result based on the area image feature corresponding to the area image of each target object.

Q4: Based on the object detection result of each target object, obtain geometrical relationship characteristics of each target object.

Method 2 is based on two feature extractions, one is image feature extraction and the other is region image feature extraction. Through this, the object pose estimation apparatus calculates the first appearance feature of the target object and the geometrical relational feature of each target object. acquire The two-time feature extraction helps to improve the accuracy of the first appearance feature and the geometrical relationship feature of each target object.

On the other hand, the color image and the depth image are obtained based on the one-frame color depth image of the video. can be decided

A color image and a depth image are extracted from a video frame including color information and depth information. When the video frame corresponding to the color image and the depth image is an initial frame of the video, that is, when there is no reference information about the pose of the video frame before the corresponding frame, the object pose estimation apparatus uses the color image and the depth image to obtain the depth image. determine the reliability of The reliability of the depth image may be determined using any one of the above-described embodiments.

8 is a flowchart illustrating a process of estimating an object pose according to another exemplary embodiment.

Referring to FIG. 8 , after obtaining the pose estimation result of the object in steps 120 and 130 of FIG. 1 , the method further includes the following steps.

The object pose estimation apparatus stores the pose estimation result of the object in the pose result list ( 140 ). In this case, the pose result includes each target object and an object pose corresponding to each target object. In order to proceed with the subsequent pose estimation based on the acquired object pose, the pose estimation result of the acquired object is stored in a pre-configured pose result list. That is, the object pose in the pose result list is used as a reference to simplify subsequent object pose acquisition, improve real-time performance of object pose acquisition, and reduce data throughput.

Based on this, if the video frame corresponding to the color image and the depth image is not the initial frame, the object pose estimation apparatus obtains a motion parameter of the video acquisition device corresponding to the video frame, and corresponds to the motion parameter and the video frame Based on the object pose result of the initial frame, a pose result corresponding to the video frame is determined ( 150 ).

Then, the object pose estimation apparatus updates the pose result of the initial frame corresponding to the video frame in the pose result list according to the pose result corresponding to the video frame ( 160 ).

One video may have a plurality of video frame sequences, each video frame sequence has only one initial frame, and the remaining video frames are not initial frames. When the video frame corresponding to the color image and the depth image is not an initial frame, that is, when an object pose corresponding to at least one target object is stored in the pose result list (a pose result corresponding to at least an initial frame in the pose result list) list is stored), and acquires motion parameters of the video acquisition device for non-initial frames corresponding to color images and depth images. The motion parameter of the corresponding video acquisition device may be one relative motion parameter. The corresponding relative motion parameter is a motion parameter of the acquiring device of the video frame of the non-initial frame with respect to the initial frame. Based on the relative motion parameter and the object pose result of the initial frame of the corresponding video frame sequence, the object pose result of the corresponding video frame is decided

The object pose estimation apparatus obtains pose results corresponding to all target objects in the video frame, stores the target object of the video frame and pose results constituting the corresponding object poses in the pose result list, and stores the pose results in the corresponding video frame. Considering the corresponding pause result as the pause result of the initial frame corresponding to the video frame, the pause result list is updated.

More specifically, the object pose estimation apparatus determines the object pose estimation of a non-initial frame in the video frame sequence in which the corresponding initial frame is located by using the pose result corresponding to the initial frame and the motion parameters of the video acquisition apparatus, and Estimate the pose of the object using the difference. The method of estimating the pose of a non-initial frame by using the estimated pose of the initial frame helps to improve the efficiency of object pose acquisition and improve the real-time performance of the object pose acquisition. In addition, since the above method has a small data difference between adjacent video frames, the pause result list is updated using the pause result corresponding to the video frame of the non-initial frame, which helps to reduce data processing of the next video frame. .

Several more methods are provided below for determining whether a video frame is an initial frame in a video.

In the case of any one video frame, if the corresponding video frame is the first frame of a video, the corresponding video frame can be viewed as an initial frame of the corresponding video.

For a video frame, object detection is performed on the video frame to determine the target object and object pose in the video frame, and it is detected whether the target object or object pose first appears in the video frame. It may be determined whether this is an initial frame. When a target object or pose of an object first appears in a video frame, that video frame is the initial frame of the video.

10 is a flowchart illustrating a process of estimating an object pose based on an initial frame of a video according to an embodiment.

Referring to FIG. 10 , the object pose estimation apparatus reads a video frame of an RGBD image ( 1010 ).

The object pose estimation apparatus may obtain a color image and a depth image of an object based on the RGBD image, and determines whether a corresponding video frame is an initial frame ( 1012 ).

If it is an initial frame as a result of confirmation in step 1012 , the object pose estimation apparatus performs object instance segmentation based on the convolutional neural network and estimates a 6-DOF pose of the object based on the object instance segmentation image ( 1014 ).

Then, the object pose estimation apparatus obtains an object pose corresponding to the target object, and updates the pose result list based on the target object and the corresponding object pose (1016).

If it is determined in 1012 that it is not the initial frame, the object pose estimation apparatus calculates the motion parameters of the camera in the current video frame for each object instance in the pose result list based on the acquired 6-DOF camera motion parameters of the current video frame. is used to transform the pose of the object instance of the initial frame into the current video frame, and obtains the object pose of the non-initial frame ( 1018 ).

Meanwhile, FIG. 10 is only an optional method, and does not limit the method of the embodiment of the present application.

In the selection of the initial frame, it can be represented as shown in FIG. 11 .

11 is a diagram illustrating an example of a method for selecting three initial frames according to an embodiment.

In the case of any video frame, if the video frame is the first frame of the video, the video frame is the initial frame of the video. When there is only one initial frame in one video, the corresponding initial frame becomes the first frame of the video, which corresponds to the initial frame selection method (1) of FIG. 11 .

In the case of video, the video may be divided into a plurality of video frame sequences according to a preset period. The first frame of each video frame sequence is an initial frame of the corresponding video frame sequence, and corresponds to the initial frame selection method (2).

The present disclosure provides an initial frame selection method (3). And, the initial frame selection method 3 may divide the video into at least one video frame sequence. For example, it is the same as the initial frame selection controller disclosed in the initial frame selection method (3) of FIG. By monitoring the new object and new pose, the initial frame selection controller determines that the video frame of the new object and the new pose is the initial frame of the corresponding video frame sequence.

Specifically, the object pose estimation apparatus performs object detection on each video frame of the video, determines a target object and pose in the video frame according to the detection result, and detects the target object or object in the video frame through detection. Determines whether the pose appears first. For example, the object pose estimation apparatus compares the target object and the object pose corresponding to the current video frame with the target object and the object pose in the pose result list, and determines whether the target object and the object pose corresponding to the current video frame appear for the first time. may be determined, and when at least one new target object or new object pose appears among the video frames, the corresponding video frame is determined as an initial frame of the corresponding video or video clip.

The present disclosure may determine whether a video frame is an initial frame through the above-described method, and may perform target processing according to the determination result. When the video frame is not the initial frame, the object pose estimation apparatus estimates the pose of the object using the object pose of the initial frame and a motion parameter corresponding to the video, thereby improving real-time performance of object pose estimation.

A method of determining whether a video frame is an initial frame according to whether a new object or a new pose appears in the video frame is suitable for the object pose estimation method.

Compared to the method of setting only the first frame as the initial frame in one video, the method of advancing the initial frame depending on whether a new object or a new pose appears requires object pose estimation of a non-initial video frame based on the initial frame. It helps to reduce the data throughput for

12 is a flowchart illustrating a process of determining whether a video frame is an initial frame based on object detection, according to an embodiment.

Referring to FIG. 12 , the object pose estimation apparatus acquires a new RGBD video frame ( 1210 ).

Then, the object pose estimation apparatus acquires the target object and the object pose in the video frame through the object detection 1212, and compares the pose result list (the current object instance and its 6-DOF pose list) (1214).

Then, the object pose estimation apparatus determines whether there is a new object instance or a new pose according to the comparison result ( 1216 ).

If it is determined in step 1216 that there is no new object instance or new pose, the apparatus for estimating the object pose returns to step 1210 and acquires a new frame.

If it is determined in step 1216 that there is a new object instance or a new pose, the object pose estimation apparatus sets the current video frame as an initial frame ( 1218 ).

After determining whether a video frame is an initial frame by any of the methods described above, the object pose estimation apparatus determines the reliability of depth data for a depth image when it is an initial frame, and uses a method of estimating the pose of an object according to the determination result. The pose of the object is estimated, and if it is not the initial frame, the pose of the object is estimated using the method provided in steps 150 to 160 of FIG. 8 .

The step of detecting whether the target object or object pose first appears in the video frame may also be realized through the following method, and the method may include the following step.

K1: Acquire the image bounding box of each target object in the corresponding video frame.

K2: Match the image bounding box of each target object with the image bounding box corresponding to each target object in the pose result list.

K3: If there is a matching target object in the pose result list, the first point cloud data of the corresponding image bounding box corresponding to each target object in the corresponding video frame and the corresponding target object in the previous video frame of the corresponding video frame By comparing the second point cloud data frames, it is determined whether there is a difference between the first point cloud data and the second point cloud data, and if there is a difference, it is determined that the pose of the object corresponding to the target object appears first.

K4: If there is no matching target object in the object pose result list, it is determined that the target object appears first.

The detection method of K1 - K4 compares the image bounding box of the target object in the video frame with the image bounding box of the target object in the pose result list, and the image bounding box matching the image bounding box of the target object of the video frame in the pose result list If there is no, it indicates that the target object of the video frame appears for the first time. .

In the case of a video frame in which the target object does not appear for the first time, the object pose estimation apparatus further determines the object pose of the target object, and converts the first point cloud data of the image bounding box corresponding to the target object in the video frame to the video frame. It compares with the second point cloud data corresponding to the target object in the previous video frame of the frame. When the first point cloud data and the second point cloud data are the same, the object pose estimation apparatus indicates that the pose of the target object does not appear first in the video frame, and when the first point cloud data and the second point cloud data are different , that is, if there is a data difference, it indicates that the object pose of the target object appears first.

13 is a flowchart illustrating a process of confirming the existence of a new object or a new pose, according to an exemplary embodiment.

Referring to FIG. 13 , the apparatus for estimating an object pose checks an image bounding box of each object detected by the object ( 1310 ).

Then, the object pose estimation apparatus compares whether two image bounding boxes match and overlap for each object in the current object instance list ( 1312 ).

Then, the apparatus for estimating the object pose checks whether an object matching the two image bounding boxes can be found ( 1314 ).

If it is determined in step 1314 that there is an object whose image bounding boxes do not match, the object pose estimation apparatus determines that a new target object is detected ( 1316 ).

As a result of the check in step 1314, if there is an object with which the two image bounding boxes match, the object pose estimation apparatus determines whether the point cloud of the bounding box is changed in comparison with the previous frame of the current video frame (1318).

If the point cloud of the bounding box is changed as a result of the check in step 1318, the object pose estimation apparatus determines that a new object pose is detected (1320).

If it is determined in step 1318 that the point cloud of the bounding box is not changed, the object pose estimation apparatus determines that a new object and a new pose are not detected ( 1322 ).

The present disclosure provides a method of determining whether a target object or an object pose first appears in a video frame. When it is determined that any one of the target object or the pose of the object appears for the first time, the corresponding video frame is determined as an initial frame, and the pose of the object is estimated according to the method provided in steps 110 to 140 of FIG. 8 . In this case, the determination of the target object based on the image bounding box may quickly and accurately determine whether the target object is a new object without more detailed data. By comparing the point cloud data of the image bounding box, it is possible to accurately determine whether an object pose first appears.

In order to better understand and explain the methods provided by the embodiments of the present disclosure, a few specific examples will be combined and described below in detail. On the other hand, in FIGS. 2, 3, 6, and 7, square rectangles indicate operation steps, and round rectangles indicate processing results.

Example 1

9 is a diagram illustrating a process of estimating an object pose based on a color image and a depth image, according to an embodiment.

9, the object pose estimation apparatus obtains a color image 901 and a depth image 902 of the same object, and uses an image convolution network 911 to obtain a color image 901 and a depth image 902. After acquiring image features by performing image feature extraction for , by fusing (920) the image feature and the point cloud feature to obtain the fusion feature.

The object pose estimation apparatus performs the following two branching operations based on the fusion feature.

As a first branch, the object pose estimation apparatus performs depth reliability prediction 931 based on the fusion feature to obtain a depth reliability image, and performs semantic segmentation 932 on the object based on the fusion feature, Acquire a semantic segmentation image, perform object center offset estimation 933 to obtain an object center image, and in the corresponding branch, perform instance segmentation 934 on the image based on the object semantic segmentation image and the object center image Thus, an object instance segmentation image is obtained.

Then, the object pose estimation apparatus determines ( 940 ) the depth data reliability based on the object instance segmentation image and the depth reliability image.

In the second branch, the object pose estimation apparatus acquires the fused feature by fusing 950 the relational feature with respect to the object instance segmentation image and the fused feature.

The object pose estimation apparatus determines whether to use a two-dimensional key point or a three-dimensional key point for object pose estimation according to a result of the determination of the reliability of the depth data. Specifically, when the depth data is unreliable, the object pose estimation apparatus performs two-dimensional key point offset estimation 961 based on the fused feature to obtain a two-dimensional key point offset image. According to the two-dimensional key point offset image, a two-dimensional key point voting 962 is performed to obtain the 2D coordinates of a preset reference key point, and the 3D coordinates of the reference key point are preset according to the ePnP algorithm 963 to pose a 6D pose is estimated (964).

When the object pose estimation apparatus trusts the depth data, the three-dimensional key point offset estimation 971 is performed based on the fused feature, and the three-dimensional key point voting 972 is performed according to the three-dimensional key point offset image. and estimate (974) the 6D pose through least squares fitting (973).

Example 2

2 is a diagram illustrating a process of estimating an object pose based on a single feature extraction according to an embodiment.

2 , the object pose estimation apparatus obtains a color image 201 and a depth image 202 of the same object, and uses an image feature extraction network 211 to obtain a color image 201 and a depth image 202 to extract the image features, obtain image features, and extract the point cloud features from the point cloud data of the depth image 202 using the point cloud feature extraction network 212 to obtain the point cloud features .

The object pose estimation apparatus fuses (220) the acquired image feature and the point cloud feature. Here, the fusion operation may be high-density fusion, and the object pose estimation apparatus performs the following three branches based on the fusion feature.

First, the apparatus for estimating an object pose predicts depth reliability for an image based on a fusion feature, and obtains a depth reliability image 231 .

Second, the object pose estimation apparatus semantically divides the object based on the fusion feature to obtain the object semantic segmentation image 232 , and obtains the object center image 233 through object center offset estimation.

Third, the object pose estimation apparatus obtains a two-dimensional key point offset image 234 and a three-dimensional key point offset image 235 .

Then, the object pose estimation apparatus performs object instance segmentation according to the object semantic segmentation image 232 and the object center image 233 to obtain an object instance segmentation image 241, and an object instance segmentation image 241 and Determining the reliability of the depth data according to the depth reliability image 231 (250), and determining whether to estimate the object pose using the two-dimensional key point or the three-dimensional key point according to the determination result .

Specifically, when the depth data is not reliable, the object pose estimation apparatus performs a two-dimensional key point voting 261 according to the two-dimensional key point offset image to obtain the 2D coordinates of a preset reference key point, and a PnP algorithm ( 262), the 3D coordinates of the reference key point are preset to estimate (263) a 6D pose.

If the depth data is reliable, the object pose estimation apparatus performs three-dimensional key point voting 271 according to the three-dimensional key point offset image to obtain the 3D coordinates of the reference key point, and 6D through least squares fitting 272 A pose is estimated (273).

The object pose estimation apparatus may use an object CAD model 280 to estimate a 6D pose, wherein the object CAD model 280 may be a simple geometric body such as a plane, a cylinder, or some kind of geometric structure, It may also be a 3D model obtained by laser scanning or other methods.

Example 3

3 is a diagram illustrating a process of estimating an object pose based on two times of feature extraction according to an embodiment.

Referring to FIG. 3 , the object pose estimation apparatus obtains a color image 301 and a depth image 302 of the same object, and uses an image feature extraction network 311 to obtain a color image 301 and a depth image 302 . By performing image feature extraction on , image features are obtained. The object pose estimation apparatus inputs image features to a Region Proposal Network 321 for regionalization processing, and again acquires regional image features through ROI pooling 322 .

The object pose estimation apparatus performs feature processing of the entire connected layer 331 based on the region image feature to obtain a depth reliability image 332, and uses a convolutional (CNN) segmentation network 341 to obtain an object instance is divided to obtain an object instance segmentation image 342 .

The apparatus for estimating an object pose obtains an instance segmentation image feature 343 based on the object instance segmentation image 342 and the region image feature.

The object pose estimation apparatus determines (350) reliability of the depth data according to the instance segmentation image feature (343) and the depth reliability image (332).

When the object pose estimation apparatus does not trust the depth data, the object pose estimation is input using the instance segmented image feature 343 as a two-dimensional key point. Specifically, the object pose estimation apparatus outputs a two-dimensional key point offset image of an object using the instance segmentation image feature 343 as an input of the convolutional neural network 361, and according to the two-dimensional key point offset image, the two-dimensional The key point voting 362 is performed to obtain the 2D coordinates of the preset reference key points, and the 3D coordinates of the reference key points are preset according to the EPnP algorithm 363 to estimate 364 a 6D pose.

When the depth data corresponding to the depth image is reliable, the object pose estimation apparatus fuses 372 the geometrical features 371 of the obtained instance segmentation using an instance segmentation image feature and a power feature extraction network, and the fused Estimate the pose of the object based on the characteristic 3D key point.

Specifically, the apparatus for estimating an object pose uses a three-dimensional drawing tool (MPL) 373 to estimate a three-dimensional key point offset image according to the fusion characteristic of the target object, and a three-dimensional key point according to the three-dimensional key point offset image A vote 374 is performed to obtain the 3D coordinates of the reference key point, and a 6D pose is estimated 376 via least squares fitting 375 .

The object pose estimation apparatus may use an object CAD model 380 to estimate a 6D pose, where the object CAD model 380 may be a simple geometric body such as a plane, a cylinder, or some kind of geometric structure, It may also be a 3D model obtained by laser scanning or other methods.

Compared with FIG. 2, the method of FIG. 3 adopts a two-step feature extraction network for feature extraction, and further refines the extracted features to help obtain accurate object pose estimation results.

Example 4

6 is a diagram illustrating a process of estimating an object pose based on one-time feature extraction and using a second appearance feature of a target object, according to an embodiment.

Referring to FIG. 6 , the apparatus for estimating an object pose uses an image feature extraction network 611 to extract image features for a color image 601 and a depth image 602 to obtain image features, and point cloud feature extraction The network 612 is used to extract the point cloud features from the transformed point cloud data of the depth image to obtain the point cloud features.

The object pose estimation apparatus fuses 620 the image feature and the point cloud feature to obtain the fusion feature, and performs object semantic segmentation 641 and object instance segmentation 632 based on the fusion feature to perform object instance segmentation image 643 ) is obtained.

Compared to the scheme shown in FIG. 2 , the scheme shown in FIG. 6 adds an object relationship branch 630 . The weighted appearance feature 637 obtained from the object relationship branch 630 affects object pose estimation based on three-dimensional key points, but does not affect object pose estimation based on two-dimensional key points. The object pose estimation based on the two-dimensional key point is the same as the scheme shown in FIG. 2 . Therefore, next, object pose estimation based on three-dimensional key points is used as an example for explanation. In the object relationship branch 630 , the object pose estimation apparatus determines an appearance feature 631 of a single target object and an appearance feature 632 of a plurality of target objects according to the fusion feature. That is, the object pose estimation apparatus obtains a first appearance feature of each target object, obtains a geometric relationship feature 633 between a plurality of target objects based on the object instance segmentation image 643 , and a first appearance feature of each target object and adding ( 634 ) attention to the geometrical relationship feature between each target object to obtain a relationship feature ( 635 ) of the target object.

The apparatus for estimating the object pose adds (636) per pixel to the obtained relational feature (635) of the target object and the first appearance feature corresponding to itself (636), and the weighted appearance feature of the target object (637) , that is, the second appearance feature of the corresponding target object is obtained.

The object pose estimation apparatus performs pose regression 638 processing based on the second appearance feature. For example, using a regression network, the second appearance feature is processed to obtain the object pose predicted from the object relationship branch.

The apparatus for estimating the object pose estimates (652) a three-dimensional key point offset by fusing (651) the relational feature based on the second appearance feature and the fusion feature, and obtains a three-dimensional key point offset image. If the object pose estimation apparatus can trust the depth image, it performs three-dimensional key point voting (653) using the three-dimensional key point offset image to obtain the 3D coordinates of the reference key point, and through least squares fitting (654) Estimate the object pose, and obtain the object pose based on three-dimensional key points.

The object pose estimation apparatus obtains the object pose predicted in the object relationship branch 630 and the object pose based on the three-dimensional key point, selects the object pose 655, and adjusts the selection criteria according to the actual situation. By realizing the 6-DOF pose estimation 656 of the object, a 6D pose of the object is obtained.

The object pose estimation apparatus may use the object CAD model 660 to estimate the 6D pose.

In this method, object pose was estimated by adding object relationship analysis and combining the relationship characteristics between different objects. This helps to increase the accuracy of the image region corresponding to each target object, and accurate image region segmentation helps to improve the accuracy of object pose estimation.

Example 5

7 is a diagram illustrating a process of estimating an object pose based on two-time feature extraction and using a second appearance feature of a target object, according to an embodiment.

Referring to FIG. 7 , the object pose estimation apparatus uses an image feature extraction network 711 to extract image features for a color image 701 and a depth image 702 , and obtain image features. The object pose estimation apparatus inputs the image features to the region suggestion network 721 for region processing, and again obtains region image features through ROI pooling 722 .

Compared to the scheme shown in FIG. 3 , the scheme shown in FIG. 7 adds an object relationship branch 730 , i.e., based on the area image feature, first appearance features 731 and 732 of each target object and each Perform geometrical relationship feature 733 between target objects. Here, the first appearance feature of each target object corresponds to the image appearance feature 731 of the single target object and the image appearance feature 732 of the plurality of target objects. In the object relationship branch 730 , the geometric relationship feature 733 between each target object is obtained based on the object detection result 741 , and the object detection result 741 divides the acquired regional image feature into a convolutional network 740 . It is entered into and obtained through processing.

The object pose estimation apparatus adds (734) attention to the image appearance feature (731) of a single target object, the image appearance feature (732) of the plurality of target objects, and the geometrical relationship feature (733) between the plurality of target objects, so that the target Get the relational feature 735 of the object. The apparatus for estimating the object pose obtains a weighted appearance feature 737, that is, a second appearance feature of the target object, by adding 736 to each pixel for the relational feature and the appearance feature of the target object. The object pose estimation apparatus performs pose regression 738 processing based on the second appearance feature. For example, using a regression network, the second appearance feature is processed to obtain the object pose predicted from the object relationship branch.

Compared with the scheme shown in Fig. 3, the scheme shown in Fig. 7 adds an object relationship branch 730, and the weighted appearance feature 737 obtained from the object relationship branch 730 is based on a three-dimensional key point. affects the object pose estimation based on two-dimensional key points, but does not affect the object pose estimation based on two-dimensional key points. The object pose estimation based on the two-dimensional key point is the same as the method shown in FIG. 3 . Therefore, next, object pose estimation based on three-dimensional key points will be described as an example.

After the ROI pooling 722, the apparatus for estimating the object pose acquires a region image feature, inputs the region image feature to the segmentation convolution network 740 to obtain an object instance segmentation image 742, and then obtains the region image feature and the object. A segmented image feature 743 , that is, an instance segmented image feature, is obtained according to the instance segmented image 742 .

The object pose estimation apparatus performs point cloud data transformation and point cloud feature extraction on the depth image to obtain a segmented geometric feature 751 , that is, an instance segmented geometric feature 751 .

The object pose estimation apparatus fuses (752) the acquired segmented image feature (743), segmented geometric feature (751), and weighted appearance feature (737), and uses a three-dimensional drawing tool (MPL) (753). to estimate the three-dimensional key point offset 754 according to the fusion feature, and perform a three-dimensional key point voting 755 according to the three-dimensional key point offset image to obtain the 3D coordinates of the reference key point 753, least squares An object pose is estimated through fitting 757, and an object pose is obtained based on a three-dimensional key point.

The object pose estimation apparatus selects a pose according to the object pose predicted in the object relationship branch 730 and the object pose based on the three-dimensional key point (760), and estimates the 6-DOF pose of the object (761) according to the set selection condition. ), that is, to determine the 6D pose.

This method adds object relationship analysis based on two-step feature extraction. Based on more accurate features, it helps to improve the accurate segmentation of each object region, and again combines the relational features between different objects to estimate the object pose, which increases the accuracy of the image region corresponding to each target object and It helps to increase the accuracy of object pose estimation.

14 is a diagram illustrating a schematic configuration of an apparatus for estimating an object pose according to an embodiment.

Referring to FIG. 14 , the object pose estimation apparatus 1400 includes an image reliability determination unit 1410 and a pose estimation unit 1420 .

The image reliability determining unit 1410 determines the reliability of the depth image according to the color image and the depth image of the object.

The pose estimator 1420 estimates the pose of the object based on the 3D key point when the depth image is reliable.

The pose estimator 1420 also estimates the pose of the object based on the two-dimensional key point when the depth image is unreliable.

The image reliability determiner 1410 may include at least one of a fusion feature reliability determiner and an image feature reliability determiner.

The fusion feature reliability determining unit extracts image features based on a color image or based on a color image and a depth image, extracts a point cloud feature according to the depth image, and fuses the image feature and the point cloud feature to determine the fusion feature. and determine the reliability of the depth image based on the fusion feature.

The image feature reliability determining unit extracts image features based on the color image and the depth image, and determines the reliability of the depth image according to the image features.

More specifically, the fusion feature reliability determining unit obtains an object instance segmentation image and a depth reliability image based on the fusion feature, and determines the reliability of the depth image corresponding to each target object among the color images according to the object instance segmentation image and the depth reliability image. decide

The image feature reliability determining unit obtains an object instance segmentation image and a depth reliability image according to the image characteristics, and determines the reliability of a depth image corresponding to each target object among the color images according to the object instance segmentation image and the depth reliability image.

The image feature reliability determining unit acquires a regional image feature of an image region corresponding to each target object according to an image characteristic, and for each target object, based on the region image feature corresponding to the target object, the depth of the target object A reliability image is determined, and an object instance segmentation image is obtained based on a region image feature corresponding to each target object.

The object pose estimating apparatus 1400 may further include a geometrical relationship feature acquirer and a second appearance feature determiner.

The geometrical relationship feature acquiring unit acquires a first appearance feature of each target object and a geometrical relationship feature between each target object based on the color image and the depth image.

For each target object, the second appearance characteristic determining unit includes a first appearance characteristic of the corresponding target object, first appearance characteristics of other target objects other than the corresponding target object, and a geometric relationship between the corresponding target object and other target objects other than the corresponding target object Based on the characteristic, a second appearance characteristic of the corresponding target object is determined.

The pose estimator 1420 may estimate the pose of each target object according to the fusion feature and the second appearance feature of each target object. Alternatively, the pose estimator 1420 may estimate the pose of each target object according to the image feature and the second appearance feature of each target object.

The geometrical relationship feature acquisition unit may acquire the geometrical relationship feature through at least one of the following two methods.

As a first method, the geometric relation feature obtaining unit extracts image features based on a color image, or based on a color image and a depth image; Extracting a point cloud feature according to the depth image, fusing the image feature and the point cloud feature to obtain a fusion feature, and obtaining a first appearance feature and object instance segmentation image of each target object based on the fusion feature, and object instance Based on the segmented image, a geometrical relationship characteristic between each target object may be obtained.

As a second method, the geometric relation feature acquisition unit extracts image features based on the color image and the depth image, and according to the image features, acquires region image features corresponding to the image regions of each target object, and image regions of each target object Based on the area image feature corresponding to , obtain a first appearance feature of each target object and a corresponding object detection result, and based on the object detection result of each target object, obtain a geometrical relationship feature of each target object there is.

Meanwhile, the object pose estimation apparatus 1400 may further include an initial frame unit.

The initial frame part is used to determine whether the video frame is an initial frame by detecting whether the target object or target pose first appears in the video frame.

The initial frame unit acquires the image bounding box of each target object in the corresponding video frame, matches the image bounding box of each target object with the image bounding box corresponding to each target object in each pose result list, and matches the pose result list If there is a target object, by comparing the first point cloud data of the image bounding box corresponding to each target object in the video frame with the second point cloud data frame corresponding to each target object in the previous video frame of the video frame, Determine whether there is a difference between the first point cloud data and the second point cloud data, if there is a difference, it is determined that the pose of the object corresponding to the target object appears first, and if there is no matching target object in the object pose result list It can be determined that the target object appears first.

The present disclosure may further provide an electronic device including a memory and a processor. In this case, the memory stores the computer program, and the processor may be used to execute the method provided in the embodiment of the present disclosure when the computer program is executed.

The present disclosure further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method provided in the embodiments of the present disclosure can be executed.

15 is a diagram illustrating a schematic configuration of an electronic device according to an exemplary embodiment.

Referring to FIG. 15 , the electronic device 1500 includes a processor 1510 and a memory 1530 . The processor 1510 and the memory 1530 are connected to each other through, for example, a bus 1520 . Optionally, the electronic device 1500 may further include a transceiver 1540 . In actual application, the transceiver 1540 is not limited to one, and the structure of the electronic device 1500 is not limited to the embodiment of the present disclosure.

The processor 1510 includes a central processing unit (CPU), a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), field programmable gate arrays) or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may be implemented or executed in combination with the various illustrative logical blocks, modules, and circuits described in this disclosure. Processor 1510 may also be a combination that realizes computing functions, such as a combination of one or more microprocessors, a combination of DSP and microprocessors, and the like.

Bus 1520 may include a path for transferring information between configurations. The bus 1520 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus 4002 may be divided into an address bus, a data bus, a control bus, and the like. For convenience of expression, only one thick line is used in FIG. 15, but this does not indicate that there is only one bus or that there is only one bus type.

Memory 1530 may include read only memory (ROM) or other type of static storage that may store static information and instructions, random access memory (RAM), or other type of dynamic storage that may store information and instructions. may be a storage device, or may be an Electrically Erasable Programmable Read Only Memory (EEPROM), Compact Disc Read Only Memory (CD-ROM) or other optical disc storage device, an optical disc storage device, a disc storage medium or other magnetic storage device, or instructions or any other medium that can be accessed by a computer and used to carry or store the desired program code in the form of a data structure, but is not limited thereto.

The memory 1530 is used to store application program code for executing an embodiment of the present disclosure, and execution is controlled by the processor 1510 . The processor 1510 is used to execute the application program code (computer program) stored in the memory 1530 to implement the contents of the above method embodiment.

In the embodiment provided in the present disclosure, the method for estimating the object pose executed in the electronic device may be performed using an artificial intelligence model.

The method executed in the electronic device 1500 of the present disclosure may use image data or video data as input data of an artificial intelligence model to obtain output data of a recognized image or an image characteristic of an image. Artificial intelligence models can be acquired through training. Here, 'acquired through training' means acquiring a predefined operating rule or artificial intelligence model to perform a desired feature (or purpose) by training a basic artificial intelligence model with a plurality of training data through a training algorithm. do. An artificial intelligence model may include a plurality of neural network layers. Each layer of the plurality of neural network layers includes a plurality of weight values, and a neural network calculation is performed through calculations between the calculation results of the previous layer and the plurality of weight values.

Visual comprehension is the art of recognizing things like human vision and processing objects, and includes, for example, object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction/positioning or image enhancement.

In the apparatus for estimating an object pose of the present disclosure, at least one configuration among a plurality of configurations may be implemented through an AI model. Functions related to AI can be performed in non-volatile memory, volatile memory, and processors.

The processor may include one or more processors. In this case, the one or more processors are general-purpose processors (eg, central processing unit (CPU), application processor (AP), etc.) or pure graphics processing unit (eg, graphics processing unit (GPU), visual processing unit (VPU) and/or It may be an AI-specific processor (eg, a neural processing unit (NPU)).

The one or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. It provides predefined operating rules or artificial intelligence models through training or learning.

Here, providing through learning means acquiring an AI model with predefined operating rules or desired characteristics by applying a learning algorithm to multiple learning data. The corresponding learning may be performed in the device itself in which the AI according to the embodiment is performed, and/or may be implemented as a separate server/system.

The AI model may be composed of a plurality of neural network layers. Each layer has a plurality of weight values, and the calculation of one layer is performed based on the calculation result of the previous layer and the plurality of weights of the current layer. Examples of neural networks include convolutional neural networks (CNN), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep trust neural networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks ( GAN) and deep Q networks.

A learning algorithm is a method of making, allowing, or controlling a predetermined target device (eg, a robot) to determine or predict a target device by training a predetermined target device (eg, a robot) using multiple training data. Examples of the corresponding learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may store program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (17)

determining the reliability of the depth image according to the color image and the depth image of the object;
estimating the pose of the object based on a three-dimensional key point when the depth image is reliable; and
estimating the pose of the object based on a two-dimensional key point if the depth image is not reliable;
An object pose estimation method comprising a.
According to claim 1,
Determining the reliability of the depth image according to the color image and the depth image,
Based on the color image, or based on the color image and the depth image, image features are extracted, point cloud features are extracted according to the depth image, and fusion features are obtained by fusing the image features and point cloud features. obtaining, and determining the reliability of the depth image based on the fusion feature
An object pose estimation method comprising a.
3. The method of claim 2,
Determining the reliability of the depth image based on the fusion feature comprises:
acquiring an object instance segmentation image and a depth reliability image based on the fusion feature; and
Determining the reliability of the depth image corresponding to each target object among the color images according to the object instance segmentation image and the depth reliability image
An object pose estimation method comprising a.
According to claim 1,
Determining the reliability of the depth image according to the color image and the depth image,
extracting image features based on the color image and the depth image, and determining the reliability of the depth image according to the image features
An object pose estimation method comprising a.
5. The method of claim 4,
Determining the reliability of the depth image according to the image feature comprises:
obtaining, according to the image feature, an object instance segmentation image and a depth reliability image; and
determining the reliability of the depth image corresponding to each target object in the color image according to the object instance segmentation image and the depth reliability image;
An object pose estimation method comprising a.
6. The method of claim 5,
According to the image feature, obtaining the object instance segmentation image and the depth reliability image comprises:
obtaining, according to the image feature, a region image feature of an image region corresponding to each target object; and
For each target object, based on a region image feature corresponding to the target object, a depth reliability image of the corresponding target object is determined, and the object instance segmentation image is based on a region image feature corresponding to each target object steps to obtain
An object pose estimation method comprising a.
According to claim 1,
acquiring a first appearance feature of each target object and a geometrical relationship feature between each target object based on the color image and the depth image; and
For each target object, based on the first appearance characteristic of the corresponding target object, the first appearance characteristic of the target object other than the corresponding target object, and the geometrical relationship characteristic between the target object and the target object other than the corresponding target object, the corresponding determining a second appearance characteristic of the target object;
An object pose estimation method further comprising a.
8. The method of claim 7,
The step of estimating the pose of the object based on the three-dimensional key point,
estimating a pose of each target object according to the fusion characteristic and the second appearance characteristic of each target object;
An object pose estimation method comprising a.
8. The method of claim 7,
The step of estimating the pose of the object based on the three-dimensional key point,
estimating a pose of each target object according to the image characteristic and the second appearance characteristic of each target object;
An object pose estimation method comprising a.
8. The method of claim 7,
Based on the color image and the depth image, obtaining a first appearance feature of each target object and a geometrical relationship feature between each target object includes:
extracting an image feature based on the color image or based on the color image and the depth image, extracting a point cloud feature according to the depth image, fusing the image feature and the point cloud feature to obtain a fusion feature, obtaining a first appearance feature and an object instance segmentation image of each target object based on the fusion feature, and obtaining a geometrical relationship feature between each target object based on the object instance segmentation image
An object pose estimation method comprising a.
8. The method of claim 7,
Based on the color image and the depth image, obtaining a first appearance feature of each target object and a geometrical relationship feature between each target object includes:
extracting an image feature based on the color image and the depth image, obtaining a region image feature corresponding to the image region of each target object according to the image feature, and a region image corresponding to the image region of each target object obtaining a first appearance feature of each target object and a corresponding object detection result based on the characteristic, and obtaining a geometric relational feature of each target object based on the object detection result of each target object;
An object pose estimation method comprising a.
According to claim 1,
determining whether the video frame is an initial frame by detecting whether a target object or target pose first appears in the video frame;
An object pose estimation method further comprising a.
13. The method of claim 12,
determining whether the video frame is an initial frame by detecting whether the target object or the target pose first appears in the video frame,
obtaining an image bounding box of each target object in the corresponding video frame;
matching the image bounding box of each target object with the image bounding box corresponding to each target object in each pose result list;
If there is a matching target object in the pose result list, the first point cloud data of the image bounding box corresponding to each target object in the corresponding video frame and the corresponding target object in the previous video frame of the corresponding video frame By comparing the second point cloud data frames, it is determined whether there is a difference between the first point cloud data and the second point cloud data, and if there is a difference, it is determined that the pose of the object corresponding to the target object appears first. to do; and
If there is no matching target object in the object pose result list, determining that the target object appears first
An object pose estimation method comprising a.
13. The method of claim 12,
If the video frame is not an initial frame, a motion parameter corresponding to the video frame is obtained, and a pose result corresponding to the video frame is obtained based on the motion parameter and an object pose result of an initial frame corresponding to the video frame. determining; and
updating an object pose result of the initial frame corresponding to the video frame in a pose result list according to the pose result corresponding to the video frame;
An object pose estimation method further comprising a.
an image reliability determining unit configured to determine the reliability of the depth image according to the color image and the depth image of the object; and
a pose weight configured to estimate the pose of the object based on a three-dimensional key point when the depth image is reliable, and to estimate the pose of the object based on a two-dimensional key point when the depth image is not reliable government
An object pose estimation device comprising a.
including memory and a processor;
The memory stores a computer program,
The processor is used to execute the method of any one of claims 1 to 14 when executing the computer program.
electronic device.
A computer-readable recording medium in which a program for executing the method of any one of claims 1 to 14 is recorded.

Images