CN115143960A

CN115143960A - SLAM system, method, device, apparatus, medium, and program product

Info

Publication number: CN115143960A
Application number: CN202210735658.1A
Authority: CN
Inventors: 李思旭; 高洋; 章国锋; 刘浩敏; 杭蒙; 焦明俊
Original assignee: Shanghai Sensetime Technology Development Co Ltd
Current assignee: Shanghai Sensetime Technology Development Co Ltd
Priority date: 2022-06-27
Filing date: 2022-06-27
Publication date: 2022-10-04

Abstract

The present disclosure relates to a SLAM system, method, apparatus, device, medium, and program product. The SLAM system includes: the hardware accelerator is used for processing a parallel task set in the SLAM task to obtain a task processing result set corresponding to the parallel task set, wherein the parallel task set comprises at least one parallel task; the processor is used for processing a first serial task set in the SLAM task to obtain a task processing result set corresponding to the first serial task set, wherein the first serial task set comprises at least one serial task; the first memory is used for storing at least part of task processing results in the task processing result set corresponding to the parallel task set and/or at least part of task processing results in the task processing result set corresponding to the first serial task set.

Description

SLAM system, method, device, apparatus, medium, and program product

Technical Field

The present disclosure relates to the field of computer vision technologies, and in particular, to a SLAM system, a method, an apparatus, an electronic device, a storage medium, and a program product.

Background

The SLAM (Simultaneous Localization And Mapping) algorithm may use images collected by a camera And Measurement data collected by an IMU (Inertial Measurement Unit) sensor to sense And model the surrounding environment, and output a map of the unknown environment And the current location.

The SLAM algorithm is usually operated in a universal serial computing platform, such as a Central Processing Unit (CPU), because the system is large, the difference between each sub-module is large, and the computation is complex. However, since the CPU has high versatility and low efficiency in executing a specific operation, a large power consumption overhead is incurred.

Disclosure of Invention

The present disclosure provides a synchronous positioning and mapping SLAM technical scheme.

According to an aspect of the present disclosure, there is provided a synchronous positioning and mapping SLAM system, including:

the hardware accelerator is used for processing a parallel task set in the SLAM task to obtain a task processing result set corresponding to the parallel task set, wherein the parallel task set comprises at least one parallel task;

the processor is used for processing a first serial task set in the SLAM task to obtain a task processing result set corresponding to the first serial task set, wherein the first serial task set comprises at least one serial task;

the first memory is used for storing at least part of task processing results in the task processing result set corresponding to the parallel task set and/or at least part of task processing results in the task processing result set corresponding to the first serial task set.

According to the embodiment of the disclosure, the hardware accelerator is added in the SLAM system, and the hardware accelerator is used for processing the parallel tasks in the SLAM tasks, so that the operation efficiency of the SLAM tasks can be improved, and the operation speed of the SLAM tasks can be improved.

In one possible implementation, the hardware accelerator is further configured to:

and processing a second serial task set in the SLAM tasks to obtain a task processing result set corresponding to the second serial task set, wherein the second serial task set comprises at least one serial task.

In this implementation, the hardware accelerator processes the second serial task set in the SLAM task to obtain a task processing result set corresponding to the second serial task set, so that a core with a simple structure can be designed for the serial tasks in the second serial task set in the hardware accelerator, and the computation speed of the serial tasks in the second serial task set can be increased.

In one possible implementation, the second serial task set includes a feature point coordinate prediction task;

the hardware accelerator includes: the first kernel is used for acquiring image coordinates of the feature points in the previous frame of image and current pose data of an Inertial Measurement Unit (IMU) sensor, and determining predicted values of the image coordinates of the feature points in the current frame of image according to the image coordinates of the feature points in the previous frame of image and the current pose data.

In the implementation mode, the first kernel in the hardware accelerator acquires the image coordinates of the feature points in the previous frame of image and the current pose data of the IMU sensor, and determines the predicted values of the image coordinates of the feature points in the current frame of image according to the image coordinates of the feature points in the previous frame of image and the current pose data, so that the first kernel with a simpler structure in the hardware accelerator is used for predicting the coordinates of the feature points, and the speed of predicting the image coordinates of the feature points in the current frame of image can be improved.

In one possible implementation, the second serial task set further includes a three-dimensional coordinate correction task;

the hardware accelerator includes: the second core is used for acquiring corrected values of image coordinates of feature points in at least three frames of images and three-dimensional coordinates of a camera corresponding to the at least three frames of images, determining three-dimensional coordinates corresponding to the feature points in the at least three frames of images according to the corrected values of the image coordinates of the feature points in the at least three frames of images, and correcting the three-dimensional coordinates of the camera and/or the three-dimensional coordinates corresponding to the feature points according to the three-dimensional coordinates of the camera and the three-dimensional coordinates corresponding to the feature points in the at least three frames of images.

In this implementation manner, the second kernel of the hardware accelerator obtains the corrected value of the image coordinate of the feature point in at least three frames of images and the three-dimensional coordinate of the camera corresponding to the at least three frames of images, determines the three-dimensional coordinate corresponding to the feature point in the at least three frames of images according to the corrected value of the image coordinate of the feature point in the at least three frames of images, and corrects the three-dimensional coordinate of the camera and/or the three-dimensional coordinate corresponding to the feature point according to the three-dimensional coordinate of the camera and the three-dimensional coordinate corresponding to the feature point in the at least three frames of images, so that the second kernel with a simpler structure in the hardware accelerator is used to correct the three-dimensional coordinate of the camera and/or the three-dimensional coordinate corresponding to the feature point, thereby improving the speed of correcting the three-dimensional coordinate and/or the three-dimensional coordinate corresponding to the feature point.

In one possible implementation, the second core and the first core are the same core.

In the implementation mode, the feature point coordinate prediction task and the three-dimensional coordinate correction task are executed by the same core in the hardware accelerator, so that the space of the hardware accelerator can be saved, and the reduction of the chip area of the hardware accelerator is facilitated.

In one possible implementation, the parallel task set includes IMU integration tasks;

the hardware accelerator includes: and the third kernel is used for acquiring the measurement data acquired by the IMU sensor, and integrating the measurement data to obtain the current pose data of the IMU sensor.

In the implementation mode, the measurement data acquired by the IMU sensor is acquired through the third core in the hardware accelerator, and the measurement data is integrated to obtain the current pose data of the IMU sensor, so that the parallel processing capability of the third core in the hardware accelerator can be utilized to improve the IMU integration speed.

and storing the current pose data.

In this implementation, the hardware accelerator stores the current pose data of the IMU sensor, so that the speed at which the kernel in the hardware accelerator acquires the current pose data of the IMU sensor can be increased.

In one possible implementation, the parallel task set further includes a feature point tracking task;

the hardware accelerator includes: and the fourth kernel is used for acquiring a current frame image, a predicted value of an image coordinate of a feature point in the current frame image and a target image block corresponding to the feature point, and tracking the feature point in the current frame image according to the predicted value and the target image block to obtain a corrected value of the image coordinate of the feature point in the current frame image.

In this implementation manner, a current frame image, a predicted value of an image coordinate of a feature point in the current frame image, and a target image block corresponding to the feature point are obtained by a fourth core in a hardware accelerator, and the feature point is tracked in the current frame image according to the predicted value and the target image block to obtain a corrected value of the image coordinate of the feature point in the current frame image, so that the parallel processing capability of the fourth core in the hardware accelerator can be utilized to improve the speed of tracking the feature point.

In one possible implementation, the hardware accelerator is further configured to store: the image processing method comprises the steps of obtaining a target image block corresponding to the feature point, the latest value of the image coordinate of the feature point and the latest value of the three-dimensional coordinate corresponding to the feature point.

In this implementation, the hardware accelerator stores the target image block corresponding to the feature point, the latest value of the image coordinate of the feature point, and the latest value of the three-dimensional coordinate corresponding to the feature point, thereby increasing the transmission speed of the target image block corresponding to the feature point, the latest value of the image coordinate of the feature point, and the latest value of the three-dimensional coordinate corresponding to the feature point in the hardware accelerator.

obtaining feature point screening results from the processor;

and screening the stored feature points according to the feature point screening result.

In the implementation mode, a feature point screening result is obtained from the processor through a hardware accelerator; and screening the stored feature points according to the feature point screening result, so that the storage space of the hardware accelerator can be saved.

In one possible implementation, the parallel task set further includes a feature extraction task;

the hardware accelerator includes: and the fifth kernel is used for extracting the characteristic points in the key frame image acquired by the camera.

In this implementation, the fifth core of the hardware accelerator extracts the feature points in the key frame image captured by the camera, so that the parallel processing capability of the fifth core of the hardware accelerator can be utilized to improve the speed of feature extraction.

In one possible implementation, the parallel task set includes a matrix modification task;

the hardware accelerator includes: and the sixth core is used for acquiring the feature vector of the feature point of the key frame image and the corrected track information of the feature point, and correcting the matrix corresponding to the currently generated map and/or equipment track according to the feature vector and the corrected track information.

In this implementation, the feature vector of the feature point of the keyframe image and the corrected trajectory information of the feature point are obtained by the sixth core in the hardware accelerator, and the matrix corresponding to the currently generated map and/or device trajectory is corrected according to the feature vector and the corrected trajectory information, so that the speed of correcting the matrix corresponding to the map and/or device trajectory can be increased by using the parallel processing capability of the sixth core in the hardware accelerator.

and storing the images of the preset number and/or the preset duration collected by the camera.

In this implementation, the hardware accelerator stores the images acquired by the camera in the preset number and/or the preset duration, so that the speed of acquiring the images by the core in the hardware accelerator can be increased.

In one possible implementation, the first set of serial tasks includes at least one of: the method comprises a characteristic point screening task, a key frame identification task, a light beam adjustment optimization task, a track estimation task and a track optimization task.

Because the processor is a serial execution unit, the implementation mode can obtain higher operation efficiency by processing at least one of a feature point screening task, a key frame identification task, a beam adjustment optimization task, a track estimation task and a track optimization task through the processor.

According to an aspect of the present disclosure, there is provided a synchronous positioning and mapping SLAM method, including:

processing a parallel task set in an SLAM task through a hardware accelerator to obtain a task processing result set corresponding to the parallel task set, wherein the parallel task set comprises at least one parallel task;

processing a first serial task set in the SLAM task through a processor to obtain a task processing result set corresponding to the first serial task set, wherein the first serial task set comprises at least one serial task;

and storing at least part of task processing results in the task processing result set corresponding to the parallel task set and/or at least part of task processing results in the task processing result set corresponding to the first serial task set through a first memory.

In one possible implementation, the method further includes:

and processing a second serial task set in the SLAM task through the hardware accelerator to obtain a task processing result set corresponding to the second serial task set, wherein the second serial task set comprises at least one serial task.

the hardware accelerator includes: the second kernel is used for obtaining correction values of image coordinates of the feature points in at least three frames of images and three-dimensional coordinates of cameras corresponding to the at least three frames of images, determining three-dimensional coordinates corresponding to the feature points in the at least three frames of images according to the correction values of the image coordinates of the feature points in the at least three frames of images, and correcting the three-dimensional coordinates of the cameras and/or the three-dimensional coordinates corresponding to the feature points according to the three-dimensional coordinates of the cameras and the three-dimensional coordinates corresponding to the feature points in the at least three frames of images.

and storing the current pose data.

the hardware accelerator includes: and the fourth core is used for acquiring a current frame image, a predicted value of an image coordinate of a feature point in the current frame image and a target image block corresponding to the feature point, tracking the feature point in the current frame image according to the predicted value and the target image block, and obtaining a corrected value of the image coordinate of the feature point in the current frame image.

obtaining feature point screening results from the processor;

According to an aspect of the present disclosure, there is provided a synchronous positioning and mapping SLAM apparatus, including:

the system comprises a first processing module, a second processing module and a task execution module, wherein the first processing module is used for processing a parallel task set in an SLAM task through a hardware accelerator to obtain a task processing result set corresponding to the parallel task set, and the parallel task set comprises at least one parallel task;

the second processing module is used for processing a first serial task set in the SLAM task through a processor to obtain a task processing result set corresponding to the first serial task set, wherein the first serial task set comprises at least one serial task;

and the storage module is used for storing at least part of task processing results in the task processing result set corresponding to the parallel task set and/or at least part of task processing results in the task processing result set corresponding to the first serial task set through a first storage.

In one possible implementation, the apparatus further includes:

and the third processing module is used for processing a second serial task set in the SLAM tasks to obtain a task processing result set corresponding to the second serial task set, wherein the second serial task set comprises at least one serial task.

In one possible implementation, the second set of serial tasks further includes a three-dimensional coordinate correction task;

In one possible implementation, the second core is the same core as the first core.

and storing the current pose data.

obtaining feature point screening results from the processor;

According to an aspect of the present disclosure, there is provided an electronic device including: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to invoke the memory-stored executable instructions to perform the above-described methods.

According to an aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the above-described method.

According to an aspect of the disclosure, there is provided a computer program product comprising computer readable code, or a non-transitory computer readable storage medium carrying computer readable code, which when run in an electronic device, a processor in the electronic device performs the above method.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

Fig. 1 shows a system architecture diagram of a SLAM system provided by an embodiment of the present disclosure.

Fig. 2 shows a schematic diagram of a feature point track in the SLAM system provided by the embodiment of the present disclosure.

Fig. 3 illustrates a schematic diagram of three-dimensional coordinate correction in a SLAM system provided by an embodiment of the present disclosure.

Fig. 4 shows a schematic diagram of a feature point tracking process in a SLAM system provided by an embodiment of the present disclosure.

Fig. 5 shows a schematic diagram of a SLAM system provided in an embodiment of the present disclosure.

Fig. 6 shows another schematic diagram of a SLAM system provided by an embodiment of the present disclosure.

Fig. 7 shows a flowchart of a synchronous positioning and mapping SLAM method provided by the embodiment of the present disclosure.

Fig. 8 shows a block diagram of a synchronous positioning and mapping SLAM device provided by an embodiment of the present disclosure.

Fig. 9 shows a block diagram of an electronic device 1900 provided by an embodiment of the disclosure.

Detailed Description

Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers can indicate functionally identical or similar elements. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of a, B, and C, and may mean including any one or more elements selected from the group consisting of a, B, and C.

Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements and circuits that are well known to those skilled in the art have not been described in detail so as not to obscure the present disclosure.

The embodiment of the disclosure provides a SLAM (Simultaneous Localization And Mapping) system, which includes a hardware accelerator, a processor, and a first memory, where the hardware accelerator is configured to process a parallel task set in a SLAM task to obtain a task processing result set corresponding to the parallel task set, where the parallel task set includes at least one parallel task, and the processor is configured to process a first serial task set in the SLAM task to obtain a task processing result set corresponding to the first serial task set, where the first serial task set includes at least one serial task, and the first memory is configured to store at least a part of task processing results in the task processing result set corresponding to the parallel task set And/or at least a part of task processing results in the task processing result set corresponding to the first serial task set, so that the hardware accelerator is added to the SLAM system, and the parallel task in the SLAM task is processed by the hardware accelerator, thereby improving the operation efficiency of the SLAM task And increasing the operation speed of the SLAM task.

In addition, since the SLAM system has noise during the three-dimensional reconstruction, it is necessary to correct errors. In the related art, since the speed of the SLAM operation is slow, the error cannot be corrected in time. As errors accumulate, it may cause the SLAM system to crash. In the embodiment of the disclosure, the hardware accelerator accelerates the SLAM, so that the error can be corrected in time, and the SLAM system can have the capability of running for a long time.

Fig. 1 shows a system architecture diagram of a SLAM system provided by an embodiment of the present disclosure. As shown in fig. 1, the SLAM system includes: the hardware accelerator 100 is configured to process a parallel task set in the SLAM task to obtain a task processing result set corresponding to the parallel task set, where the parallel task set includes at least one parallel task; the processor 200 is configured to process a first serial task set in the SLAM task to obtain a task processing result set corresponding to the first serial task set, where the first serial task set includes at least one serial task; a first memory 300, configured to store at least a part of task processing results in a task processing result set corresponding to the parallel task set and/or at least a part of task processing results in a task processing result set corresponding to the first serial task set.

In the present disclosed embodiment, the SLAM system may be provided in the target device, that is, the target device may include the SLAM system. The target device may be any device that needs to perform SLAM operations. For example, the target device may be a mobile robot, AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, a drone, a cell phone, an automobile, or the like. In addition, embodiments of the present disclosure may be applied in a variety of environments, such as swimming pools, sports stadiums, museums, field environments, and so forth.

In the embodiment of the present disclosure, the SLAM system includes a processor 200, a first memory 300, and a hardware accelerator 100, wherein the processor 200 may be connected with the first memory 300, and the first memory 300 may be connected with the hardware accelerator 100. The processor 200 and the hardware accelerator 100 may interact with data through the first memory 300. In one possible implementation, the hardware accelerator 100 may also be connected to a camera and/or an IMU (Inertial Measurement Unit) sensor, so that an image acquired by the camera and Measurement data acquired by the IMU sensor can be acquired.

In the embodiment of the present disclosure, the processor 200 may be a Central Processing Unit (CPU), a Graphic Processing Unit (GPU), or other types of processors, and is not limited herein. The first storage 300 may include a memory, and may also include other types of storage, which is not limited herein.

In embodiments of the present disclosure, the set of parallel tasks may include one or more parallel tasks, and the first set of serial tasks may include one or more serial tasks. The parallel tasks may have a higher degree of parallelism, i.e. the number of sub-tasks that can be processed simultaneously in the parallel tasks may be larger. The serial task may have a lower degree of parallelism, i.e., the number of sub-tasks that can be processed simultaneously in the serial task may be smaller, or there may be no sub-tasks that can be processed simultaneously. The number of sub-tasks that can be processed simultaneously in any parallel task may be greater than the number of sub-tasks that can be processed simultaneously in any serial task.

In one possible implementation, the parallel tasks may represent tasks that satisfy a preset parallel condition. The serial tasks may represent tasks that do not satisfy a preset parallel condition. The preset parallelism condition may include a preset subtask number threshold. If the number of the subtasks which can be processed simultaneously in any task is greater than or equal to a preset subtask number threshold, determining that the task meets a preset parallel condition, and determining the task as a parallel task; if the number of the subtasks which can be processed simultaneously in any task is smaller than a preset subtask number threshold, it can be determined that the task does not meet a preset parallel condition, and the task can be determined as a serial task.

In this embodiment of the present disclosure, for any parallel task in the parallel task set, the hardware accelerator 100 may process the parallel task to obtain a task processing result corresponding to the parallel task. The task processing results corresponding to each parallel task in the parallel task set can form a task processing result set corresponding to the parallel task set. For any serial task in the first serial task set, the processor 200 may process the serial task to obtain a task processing result corresponding to the serial task. The task processing results corresponding to the respective serial tasks in the first serial task set may form a task processing result set corresponding to the first serial task set. The first memory 300 may be used to store: and/or, the task processing result set corresponding to the first serial task set is a part or all of the task processing results in the task processing result set.

In one possible implementation, the hardware accelerator 100 is further configured to: and processing a second serial task set in the SLAM tasks to obtain a task processing result set corresponding to the second serial task set, wherein the second serial task set comprises at least one serial task.

In this implementation, the second set of serial tasks may include one or more than two serial tasks. For any serial task in the second serial task set, the hardware accelerator 100 may process the serial task to obtain a task processing result corresponding to the serial task. The task processing results corresponding to the respective serial tasks in the second serial task set may form a task processing result set corresponding to the second serial task set.

In the related art, the processing speed for some serial tasks is slow due to the complex function of the mainstream processor 200. In this implementation, the hardware accelerator 100 processes the second serial task set in the SLAM task to obtain a task processing result set corresponding to the second serial task set, so that a kernel having a simple structure can be designed for the serial tasks in the second serial task set in the hardware accelerator 100, and the computation speed of the serial tasks in the second serial task set can be increased.

As one example of this implementation, the second set of serial tasks includes a feature point coordinate prediction task; the hardware accelerator 100 includes: the first kernel is used for acquiring image coordinates of the feature points in the previous frame of image and current pose data of the IMU sensor, and determining predicted values of the image coordinates of the feature points in the current frame of image according to the image coordinates of the feature points in the previous frame of image and the current pose data.

In this example, the feature point coordinate prediction task may also be referred to as a point location prediction task. The input of the feature point coordinate prediction task can comprise the image coordinates of the feature points in the previous frame of image and the current pose data of the IMU sensor; the output of the feature point coordinate prediction task may include a prediction value of image coordinates of the feature point in the current frame image.

In one example, the first kernel may obtain image coordinates of the feature point in the previous frame of image from a visual feature point state library. Wherein the visual feature point state library may be disposed in a second memory, which may be a cache in the hardware accelerator 100.

In the visual feature point state library, the information stored for any feature point may include: the coordinates of the characteristic points and characteristic information corresponding to the characteristic points. The coordinates of the feature point may include the latest value of the image coordinates of the feature point and/or the latest value of the three-dimensional coordinates corresponding to the feature point. And the three-dimensional coordinates corresponding to the characteristic points represent the three-dimensional coordinates of the three-dimensional points corresponding to the characteristic points in a world coordinate system. The feature information corresponding to the feature point may be any information that can indicate the feature of the feature point. In one example, the feature information corresponding to the feature point may include the target image block corresponding to the feature point. The target image block corresponding to the feature point may be an image block corresponding to the feature point in the previous frame image, or may be an image block corresponding to the feature point in the latest key frame image. The image block corresponding to the feature point may represent a small block of image around the feature point. For any frame of image collected by the camera, if the similarity between the image and the previous frame of image of the image is less than or equal to a first preset similarity threshold, the image may be determined as a key frame image, and if the similarity between the image and the previous frame of image of the image is greater than the first preset similarity threshold, the image may be determined as a non-key frame image. In another example, the feature information corresponding to the feature point may include a feature vector corresponding to the feature point. In one example, in the visual feature point state library, the information stored for any feature point may include: the latest value of the image coordinate of the feature point, the latest value of the three-dimensional coordinate corresponding to the feature point, and the image block corresponding to the feature point in the previous frame of image.

In one example, the first core may acquire current pose data of the IMU sensor from the third core. The current pose data of the IMU sensor can represent pose data obtained by integrating acceleration and three-axis attitude angles acquired by the IMU sensor.

In this example, the predicted value of the image coordinate of the feature point in the current frame image may also be referred to as an initial value of the image coordinate of the feature point in the current frame image. In one example, the predicted value of the image coordinates of the feature point in the current frame image obtained by the first kernel may be transmitted to the fourth kernel.

In this example, the first kernel in the hardware accelerator 100 acquires the image coordinates of the feature point in the previous frame image and the current pose data of the IMU sensor, and determines the predicted value of the image coordinates of the feature point in the current frame image according to the image coordinates of the feature point in the previous frame image and the current pose data, so that the first kernel with a simpler structure in the hardware accelerator 100 is used for feature point coordinate prediction, and the speed of predicting the image coordinates of the feature point in the current frame image can be increased.

As an example of this implementation, the second set of serial tasks further includes a three-dimensional coordinate correction task; the hardware accelerator 100 includes: the second kernel is used for obtaining correction values of image coordinates of the feature points in at least three frames of images and three-dimensional coordinates of cameras corresponding to the at least three frames of images, determining three-dimensional coordinates corresponding to the feature points in the at least three frames of images according to the correction values of the image coordinates of the feature points in the at least three frames of images, and correcting the three-dimensional coordinates of the cameras and/or the three-dimensional coordinates corresponding to the feature points according to the three-dimensional coordinates of the cameras and the three-dimensional coordinates corresponding to the feature points in the at least three frames of images.

In this example, the three-dimensional coordinate correction task may also be referred to as a triangularization task. The input of the three-dimensional coordinate correction task can comprise correction values of image coordinates of the feature points in at least three frames of images and three-dimensional coordinates of cameras corresponding to the at least three frames of images; the output of the three-dimensional coordinate correction task may include corrected three-dimensional coordinates of the camera and/or corrected three-dimensional coordinates corresponding to the feature points.

In one example, the first memory 300 may include a visual feature trajectory state library, which may stitch together trajectories of feature points from feature points in each frame image. According to the tracks of the feature points, the connection between the feature points in different frames can be established. Fig. 2 shows a schematic diagram of a feature point track in the SLAM system provided by the embodiment of the present disclosure. In the example shown in fig. 2, the tracks of the feature points may be obtained by stitching according to the image coordinates of the feature points in the first frame image, the image coordinates of the feature points in the second frame image, and the image coordinates of the feature points in the third frame image.

In one example, the second kernel may obtain correction values of image coordinates of the feature points in at least three frames of images from the visual feature trajectory state library.

Fig. 3 illustrates a schematic diagram of three-dimensional coordinate correction in a SLAM system provided by an embodiment of the present disclosure. As shown in fig. 3, three-dimensional coordinates corresponding to the feature points in the three frames of images can be mapped according to the correction values of the image coordinates of the feature points in the three frames of images. And obtaining three rays according to the three-dimensional coordinates corresponding to the feature points in the three frames of images and the three-dimensional coordinates of the camera corresponding to the three frames of images. According to the intersection condition of the three rays, the three-dimensional coordinate of the camera and/or the three-dimensional coordinate corresponding to the characteristic point can be corrected, so that the corrected three rays intersect at the same point.

In one example, the corrected three-dimensional coordinates of feature points may be stored in a filtered & optimized visual feature trajectory state library. Wherein the filtered & optimized visual feature trajectory state library may be disposed in the first memory 300.

In this example, the correction value of the image coordinates of the feature point in at least three frames of images and the three-dimensional coordinates of the camera corresponding to the at least three frames of images are obtained by the second core of the hardware accelerator 100, the three-dimensional coordinates corresponding to the feature point in the at least three frames of images are determined from the correction value of the image coordinates of the feature point in the at least three frames of images, and the three-dimensional coordinates of the camera and/or the three-dimensional coordinates corresponding to the feature point in the at least three frames of images are corrected according to the three-dimensional coordinates of the camera and the three-dimensional coordinates corresponding to the feature point in the at least three frames of images, so that the three-dimensional coordinates of the camera and/or the three-dimensional coordinates corresponding to the feature point are corrected by the second core of the hardware accelerator 100, which has a simpler structure, and the speed of correcting the three-dimensional coordinates and/or the three-dimensional coordinates corresponding to the feature point can be increased.

In one example, the second core is the same core as the first core. Since the first core and the second core may accelerate the serial task, the first core and the second core may also be referred to as high-speed serial acceleration cores. In this example, the feature point coordinate prediction task and the three-dimensional coordinate correction task are executed by the same core in the hardware accelerator 100, whereby the space of the hardware accelerator 100 can be saved, thereby contributing to reduction of the chip area of the hardware accelerator 100.

In one possible implementation, the parallel task set includes IMU integration tasks; the hardware accelerator 100 includes: and the third kernel is used for acquiring the measurement data acquired by the IMU sensor, and integrating the measurement data to obtain the current pose data of the IMU sensor.

In this implementation, the input of the IMU integration task may be measurement data acquired by the IMU sensor, and the output of the IMU integration task may be current pose data of the IMU sensor. The current pose data of the IMU sensor can also be referred to as the current state of the IMU for short.

In this implementation, the hardware accelerator 100 includes a third core. The third core may also be referred to as a sensor processing core, since it may be used to process measurement data acquired by the IMU sensor. The third core may acquire measurement data acquired by the IMU sensor, where the measurement data acquired by the IMU sensor may include three-axis attitude angles and accelerations. The third kernel may obtain a change Value of the pose at the current time with respect to the previous time through a linear algebraic operation (e.g., SVD (Singular Value Decomposition, etc.) with a small scale, so as to obtain current pose data. The pose change value may include a position change value and an angle change value, and the position change value may include displacement in an x-axis direction, a y-axis direction, and a z-axis direction. The current pose data may include current position data and current pose data, the current position data may be represented in x, y, and z coordinates, and the current pose data may be represented in angle.

In this implementation manner, the third core in the hardware accelerator 100 acquires measurement data acquired by the IMU sensor, and integrates the measurement data to obtain the current pose data of the IMU sensor, so that the parallel processing capability of the third core in the hardware accelerator 100 can be utilized to improve the IMU integration speed.

As an example of this implementation, the hardware accelerator 100 is further configured to: and storing the current pose data. For example, the current pose data of the IMU sensors may be stored in an IMU current state library. In this example, the current pose data of the IMU sensor is stored by the hardware accelerator 100, thereby enabling an increase in the speed at which the cores in the hardware accelerator 100 acquire the current pose data of the IMU sensor.

In one possible implementation, the parallel task set further includes a feature point tracking task; the hardware accelerator 100 includes: and the fourth core is used for acquiring a current frame image, a predicted value of an image coordinate of a feature point in the current frame image and a target image block corresponding to the feature point, tracking the feature point in the current frame image according to the predicted value and the target image block, and obtaining a corrected value of the image coordinate of the feature point in the current frame image.

In this implementation, the feature point tracking task may also be referred to as a visual feature tracking task. Since the fourth kernel is used for tracking feature points, the fourth kernel may also be referred to as a visual tracking kernel.

In this implementation, the input of the feature point tracking task may include a current frame image, a predicted value of image coordinates of a feature point in the current frame image, and a target image block corresponding to the feature point, and the output of the feature point tracking task may be a corrected value of image coordinates of the feature point in the current frame image.

As one example of this implementation, the fourth core may acquire a current frame image from the camera.

As an example of this implementation, the fourth core may acquire, from the first core, a predicted value of image coordinates of the feature point in the current frame image.

As an example of this implementation, the fourth core may obtain a target image block corresponding to the feature point from the visual feature point state library.

In this implementation, the fourth core may find an image block matched with the target image block corresponding to the feature point in the current frame image, so as to determine the position of the feature point in the current frame image. For example, the predicted value of the image coordinates of the feature point in the current frame image may be used as an initial value, and the target position may be searched for all around and iterated step by step.

Fig. 4 is a schematic diagram illustrating a feature point tracking process in the SLAM system according to an embodiment of the present disclosure. In fig. 4, the upper left corner image may represent the previous frame image, and the upper right corner image, the lower right corner image, and the lower left corner image may represent the current frame image. The fourth core may acquire a current frame image from the camera, and may acquire a target image block corresponding to the feature point (i.e., an image block corresponding to the feature point in the previous frame image) from the visual feature point state library. The positions of the small dots in the upper left corner image and the upper right corner image can represent the image coordinates of the feature points in the previous frame image. As the device moves, the image observed by the camera is displaced, i.e., the image coordinates of the feature point in the current frame image are different from the image coordinates of the feature point in the previous frame image. The position of the small circle point in the lower right corner image can represent the predicted value of the image coordinate of the feature point in the current frame image. The fourth core may acquire a predicted value of image coordinates of the feature point in the current frame image from the first core. The fourth kernel may search around using the predicted value of the image coordinate of the feature point in the current frame image as an initial value, for example, the search may be performed upward, downward, leftward, and rightward. The similarity between the image block obtained by the rightward search and the target image block is the highest as known from the search results of the upward, downward, leftward and rightward searches, so that the correction value of the image coordinate of the feature point in the current frame image can be obtained according to the search result of the rightward search.

As an example of this implementation, after the fourth core obtains the corrected value of the image coordinate of any feature point in the current frame image, the corrected value of the image coordinate of the feature point in the current frame image may be stored in the visual feature point state library as the latest value of the image coordinate of the feature point.

In this implementation, a current frame image, a predicted value of an image coordinate of a feature point in the current frame image, and a target image block corresponding to the feature point are obtained by a fourth core in the hardware accelerator 100, and the feature point is tracked in the current frame image according to the predicted value and the target image block to obtain a corrected value of the image coordinate of the feature point in the current frame image, so that the parallel processing capability of the fourth core in the hardware accelerator 100 can be utilized to improve the speed of tracking the feature point.

As an example of this implementation, the hardware accelerator 100 is further configured to store: the image processing method comprises the steps of obtaining a target image block corresponding to the feature point, the latest value of the image coordinate of the feature point and the latest value of the three-dimensional coordinate corresponding to the feature point.

The latest value of the image coordinates of the feature point may be a corrected value of the image coordinates of the feature point in the latest image output by the fourth core. For example, after the fourth core obtains the corrected value of the image coordinate of the feature point in the ith frame image and before the fourth core obtains the corrected value of the image coordinate of the feature point in the (i + 1) th frame image, the latest value of the image coordinate of the feature point may be the corrected value of the image coordinate of the feature point in the ith frame image; the latest value of the image coordinate of the feature point may be the corrected value of the image coordinate of the feature point in the i +1 th frame image after the fourth core obtains the corrected value of the image coordinate of the feature point in the i +1 th frame image and before the fourth core obtains the corrected value of the image coordinate of the feature point in the i +2 th frame image; and so on.

In this example, the hardware accelerator 100 stores the target image block corresponding to the feature point, the latest value of the image coordinate of the feature point, and the latest value of the three-dimensional coordinate corresponding to the feature point, thereby making it possible to increase the transmission speed of the target image block corresponding to the feature point, the latest value of the image coordinate of the feature point, and the latest value of the three-dimensional coordinate corresponding to the feature point in the hardware accelerator 100.

In one example, the hardware accelerator 100 is further configured to: obtaining feature point screening results from the processor 200; and screening the stored feature points according to the feature point screening result.

In this example, the processor 200 may filter the feature points, and discard the feature points whose tracking effect does not satisfy the preset tracking condition, to obtain a feature point filtering result. The similarity between the image block corresponding to the feature point in the current frame image and the target image block corresponding to the feature point can be determined; if the similarity is greater than or equal to a second preset similarity threshold, it may be determined that the tracking effect of the feature point satisfies a preset tracking condition, and if the similarity is less than the second preset similarity threshold, it may be determined that the tracking effect of the feature point does not satisfy the preset tracking condition. The second preset similarity threshold may be flexibly set according to an actual application scenario, and is not limited herein.

In one example, the visual feature trajectory state library in the hardware accelerator 100 may obtain a feature point screening result from the processor 200, and discard feature points whose tracking effects do not satisfy a preset tracking condition according to the feature point screening result.

In this example, feature point screening results are obtained from the processor 200 by the hardware accelerator 100; the stored feature points are filtered according to the feature point filtering result, so that the storage space of the hardware accelerator 100 can be saved.

In one possible implementation, the parallel task set further includes a feature extraction task; the hardware accelerator 100 includes: and the fifth kernel is used for extracting the characteristic points in the key frame image acquired by the camera.

In this implementation, the input of the feature extraction task may include key frame images, and the output may include information of feature points in the key frame images. Since the fifth core is used to process the key frame image, the fifth core may also be referred to as an image processing core.

As an example of this implementation, the fifth core may obtain a key frame image from a key frame identification module of processor 200. The fifth core may perform feature extraction on the key frame image to obtain image coordinates of feature points in the key frame image and image blocks corresponding to the feature points, and may transmit the image coordinates of the feature points and the image blocks corresponding to the feature points to the visual feature point state library. In one example, the visual feature point state library may discard information of a part of previous feature points in response to acquiring information of feature points of a new key frame image, for example, a preset number or a preset proportion of feature points with the worst tracking effect may be discarded.

As another example of this implementation, the fifth core may retrieve the key frame image from a key frame image library. The fifth kernel can obtain the feature vectors of the feature points in the key frame image by performing feature extraction on the key frame image, and can transmit the feature vectors of the feature points to the re-extraction visual feature point information base.

In this implementation, feature points in the keyframe image captured by the camera are extracted by the fifth kernel of the hardware accelerator 100, this makes it possible to increase the speed of feature extraction by utilizing the parallel processing capability of the fifth core in the hardware accelerator 100.

In one possible implementation, the parallel task set includes a matrix modification task; the hardware accelerator 100 includes: and the sixth core is used for acquiring the feature vector of the feature point of the key frame image and the corrected track information of the feature point, and correcting the matrix corresponding to the currently generated map and/or equipment track according to the feature vector and the corrected track information.

In this implementation, the matrix modification task may also be referred to as a linear equation system solving task, and the sixth kernel may also be referred to as a linear equation solving kernel. The input of the matrix correction task can comprise feature vectors of feature points of the key frame images and corrected track information of the feature points; the output of the matrix modification task may include a modified map and/or a modified device trajectory. Wherein the device trajectory may represent a trajectory of the target device.

As one example of this implementation, the sixth core may obtain feature vectors of feature points of the key frame image from the re-extraction visual feature point information base.

As an example of this implementation, the sixth core may obtain the revised trajectory information of the feature points from the filtered & optimized visual feature trajectory state library.

As one example of this implementation, the sixth core may obtain mathematical description information of the map/trajectory from a state matrix repository. For example, the mathematical description information of the map/track in the state matrix library may be in the form of a matrix or the like.

In this implementation, the feature vector of the feature point of the key frame image and the corrected trajectory information of the feature point are obtained by the sixth core in the hardware accelerator 100, and the matrix corresponding to the currently generated map and/or device trajectory is corrected according to the feature vector and the corrected trajectory information, so that the speed of correcting the matrix corresponding to the map and/or device trajectory can be increased by using the parallel processing capability of the sixth core in the hardware accelerator 100.

In one possible implementation, the hardware accelerator 100 is further configured to: and storing the images of the preset number and/or the preset duration collected by the camera. As an example of this implementation, the latest captured images of the camera in a preset number and/or a preset duration may be stored in the camera image repository of the hardware accelerator 100. In this implementation, the hardware accelerator 100 stores a preset number and/or a preset duration of images captured by the camera, so that the speed of acquiring images by the core in the hardware accelerator 100 can be increased.

In one example, the processor 200 may include a trace result filtering module that may be used to process the feature point filtering task. The input of the feature point screening task may include a correction value of image coordinates of the feature point in the current frame image output by the fourth kernel, and the output of the feature point screening task may include a feature point screening result. The tracking result screening module can screen the feature points according to the corrected value of the image coordinates of the feature points output by the fourth kernel in the current frame image, and discard the feature points with the tracking effect not meeting the preset tracking condition to obtain the feature point screening result.

In one example, the processor 200 may include a key frame identification module that may be used to process key frame identification tasks. The input of the key frame identification task can be a feature point screening result, and the output can be a key frame identification result. The key frame identification module may determine the number of currently tracked feature points according to the feature point screening result, and may reacquire the key frame image in response to that the number of currently tracked feature points is less than or equal to a preset number threshold. The preset number threshold may be 20, which is not limited herein. In one example, the key frame identification module may acquire a latest key frame image from the key frame image library in response to the number of currently tracked feature points being less than or equal to a preset number threshold, and may output the latest key frame image to the fifth kernel for feature point extraction by the fifth kernel.

In one example, processor 200 may include a beam adjustment optimizer that may be used to perform beam adjustment optimization tasks. The light beam adjustment optimizer can acquire the corrected track information of the feature points from the screened and optimized visual feature track state library, can acquire the feature vectors of the feature points of the key frame image from the re-extracted visual feature point information library, and can perform collaborative optimization on the corrected track information of all the feature points according to the feature vectors of the feature points to obtain a high-precision map/track in an unknown environment.

In one example, the processor 200 may include a visual feature estimation module that may be used to perform a trajectory estimation task. The visual feature estimation module can acquire the information of the feature points from the visual feature point state library, so that the offset of the feature points between the current frame image and the previous frame image can be determined. The visual characteristic estimation module can correct the measurement data acquired by the IMU sensor according to the offset to obtain the rough track of the target equipment.

In one example, the processor 200 may include a filter optimization module that may be used to perform trajectory optimization tasks. The filter optimization module can perform collaborative optimization according to the track of the feature points in the latest N frames of images in the screened and optimized visual feature track state library, so as to obtain the fine track of the target device. Where N is greater than 1, for example, N may be equal to 10. Since N is greater than 1, a more accurate trajectory can be obtained compared to the visual feature estimation module.

Because the processor 200 is a serial execution unit, the implementation method can obtain higher operation efficiency by processing at least one of the feature point screening task, the key frame recognition task, the beam adjustment optimization task, the trajectory estimation task, and the trajectory optimization task by the processor 200.

In another possible implementation, at least one of the feature point screening task, the key frame identification task, the beam adjustment optimization task, the trajectory estimation task, and the trajectory optimization task may also be performed by the hardware accelerator 100.

In one possible implementation, the first memory 300 may include at least one of: a key frame image library, a visual characteristic track state library, a screened and optimized visual characteristic track state library, a re-extracted visual characteristic point information library, a state matrix library and a map/track library. The key frame image library can be used for storing key frame images, the visual feature track state library can be used for storing tracks of feature points, the screened and optimized visual feature track state library can be used for storing corrected tracks of the feature points, the re-extracted visual feature point information library can be used for storing feature vectors of the feature points, the state matrix library can be used for storing a currently generated map and/or a matrix corresponding to an equipment track, and the map/track library can be used for storing the currently generated map and/or the equipment track.

The SLAM system provided by the embodiment of the present disclosure may be applied to the technical fields of computer vision, architecture, digital integrated circuit, hardware accelerator, etc., and may be applied to an edge computing platform, for example, may be applied to products such as AR glasses, VR glasses, etc., and is not limited herein.

The SLAM system provided by the embodiment of the present disclosure is described below by a specific application scenario. Fig. 5 shows a schematic diagram of a SLAM system provided in an embodiment of the present disclosure. In the application scenario, the processor employs a CPU, and the first storage employs a memory, that is, the SLAM system includes an accelerator, a memory, and a CPU, where the accelerator is a hardware accelerator. The accelerator may be connected to the camera, the IMU sensor, and the memory, respectively, and the CPU may be connected to the memory, and may also be connected to the accelerator. The accelerator may include 5 cores, respectively: a high speed serial acceleration core (i.e., first/second core above), a visual tracking core (i.e., fourth core above), an image processing core (i.e., fifth core above), a linear equation solving core (i.e., sixth core above), and a sensor processing core (i.e., third core above). Also, the accelerator may include an accelerator cache.

Fig. 6 shows another schematic diagram of a SLAM system provided by an embodiment of the present disclosure. With reference to fig. 5 and 6, the high-speed serial acceleration core may be configured to perform a point location prediction task (i.e., the above feature point coordinate prediction task) and a triangularization task (i.e., the above three-dimensional coordinate correction task), the visual tracking core may be configured to perform a visual feature tracking task (i.e., the above feature point tracking task), the image processing core may be configured to perform a feature point extraction task and a feature point extraction matching task, the linear equation system solving core may be configured to perform a linear equation system solving task (i.e., the above matrix correction task), and the sensor processing core may be configured to perform an IMU integration task. The feature point extraction task and the feature point extraction matching task in fig. 6 both belong to the above feature extraction task.

As shown in fig. 6, the accelerator may include a visual feature point state library, an IMU current state library, and a camera image repository. In the visual feature point state library, the information stored for any feature point may include: the latest value of the image coordinate of the feature point, the latest value of the three-dimensional coordinate corresponding to the feature point, and the image block corresponding to the feature point in the previous frame of image. The IMU current state library may be used to store IMU sensor current pose data. The camera image repository may be used to store a preset number and/or a preset duration of images captured by the camera.

As shown in fig. 6, the memory may include a key frame image library, a visual feature trajectory state library, a filtered & optimized visual feature trajectory state library, a re-extracted visual feature point information library, a state matrix library, and a map/trajectory library. The key frame image library can be used for storing key frame images, the visual feature track state library can be used for storing tracks of feature points, the screened and optimized visual feature track state library can be used for storing corrected tracks of the feature points, the re-extracted visual feature point information library can be used for storing feature vectors of the feature points, the state matrix library can be used for storing a currently generated map and/or a matrix corresponding to an equipment track, and the map/track library can be used for storing the currently generated map and/or the equipment track.

As shown in fig. 6, the processor may include a tracking result screening module, a key frame identification module, a beam adjustment optimizer, a visual feature estimation module, and a filter optimization module.

Wherein, the visual characteristic estimation module can output a rough track, the filter optimization module can output a fine track, and the map/track library can comprise a high-precision map/track. The delay of the coarse track may be in the order of 1ms, the delay of the fine track may be in the order of 10ms, and the delay of the high-precision map/track may be in the order of 1 s. In delay-sensitive scenes such as AR and VR, the fastest visual feedback can be provided for a user through a rough track, and the correction can be performed at the back end to obtain a fine track and a high-precision map/track.

In the application scenario, a beam adjustment optimizer in the related technology can be disassembled into 5 parts, namely a feature point extraction matching task, a linear equation system solving task, a beam adjustment optimizer, a re-extraction visual feature point information base and a state matrix base. Wherein, the feature point extraction matching task can perform feature point extraction on the key frame image, and the feature point extraction matching task can extract more feature points than the feature extraction task. The feature point extracting and matching task can describe and match the feature points by adopting high-dimensional feature vectors, so as to obtain re-extracted visual feature point information. The state matrix library may include maps/tracks a matrix of map/track correspondences in the library. The linear equation system solving task can acquire information from the re-extraction visual feature point information base and the screened and optimized visual feature track state base so as to correct the matrix corresponding to the map/track. The beam adjustment optimizer may update the map/track library according to the modified matrix corresponding to the map/track. By disassembling the beam adjustment optimizer in the related art into 5 parts and processing the parts respectively through the accelerator and the CPU, the operation efficiency can be improved.

The application scene provides a complete SLAM system, and the SLAM calculation of the edge platform with low power consumption and high performance can be realized.

Fig. 7 shows a flowchart of a method for synchronously positioning and mapping SLAM according to an embodiment of the present disclosure. In one possible implementation manner, the execution main body of the synchronization positioning and mapping SLAM method may be a synchronization positioning and mapping SLAM device, for example, the synchronization positioning and mapping SLAM method may be executed by a terminal device or a server or other electronic devices. The terminal device may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, or a wearable device. In some possible implementations, the synchronized positioning and mapping SLAM method may be implemented by a processor calling computer readable instructions stored in a memory. As shown in fig. 1, the synchronous positioning and mapping SLAM method includes S21 to S23.

In step S21, a parallel task set in the SLAM task is processed by a hardware accelerator to obtain a task processing result set corresponding to the parallel task set, where the parallel task set includes at least one parallel task;

in step S22, a processor processes a first serial task set in the SLAM task to obtain a task processing result set corresponding to the first serial task set, where the first serial task set includes at least one serial task;

in step S23, at least a part of the task processing results in the task processing result set corresponding to the parallel task set and/or at least a part of the task processing results in the task processing result set corresponding to the first serial task set are stored in the first memory.

In one possible implementation, the method further includes:

and storing the current pose data.

obtaining feature point screening results from the processor;

It is understood that the above-mentioned method embodiments of the present disclosure can be combined with each other to form a combined embodiment without departing from the logic of the principle, which is limited by the space, and the detailed description of the present disclosure is omitted. Those skilled in the art will appreciate that in the above methods of the specific embodiments, the specific order of execution of the steps should be determined by their function and possibly their inherent logic.

In addition, the present disclosure also provides a synchronous positioning and mapping SLAM device, an electronic device, a computer-readable storage medium, and a computer program product, which can all be used to implement any synchronous positioning and mapping SLAM method provided by the present disclosure, and corresponding technical solutions and technical effects can be referred to in corresponding descriptions of the method sections and are not described again.

Fig. 8 shows a block diagram of a synchronous positioning and mapping SLAM device provided in an embodiment of the present disclosure. As shown in fig. 8, the synchronous positioning and mapping SLAM device includes:

the first processing module 31 is configured to process a parallel task set in an SLAM task through a hardware accelerator to obtain a task processing result set corresponding to the parallel task set, where the parallel task set includes at least one parallel task;

a second processing module 32, configured to process, by a processor, a first serial task set in the SLAM tasks to obtain a task processing result set corresponding to the first serial task set, where the first serial task set includes at least one serial task;

the storage module 33 is configured to store, through the first memory, at least a part of task processing results in the task processing result set corresponding to the parallel task set and/or at least a part of task processing results in the task processing result set corresponding to the first serial task set.

In one possible implementation, the apparatus further includes:

the hardware accelerator includes: and the third kernel is used for acquiring measurement data acquired by the IMU sensor, and integrating the measurement data to obtain the current pose data of the IMU sensor.

and storing the current pose data.

obtaining feature point screening results from the processor;

the hardware accelerator includes: and the fifth kernel is used for extracting the characteristic points in the key frame images acquired by the camera.

In some embodiments, functions or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiments, and specific implementations and technical effects thereof may refer to the description of the above method embodiments, which are not described herein again for brevity.

Embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the above-described method. The computer readable storage medium may be a non-volatile computer readable storage medium, or may be a volatile computer readable storage medium.

Embodiments of the present disclosure also provide a computer program, which includes computer readable code, and when the computer readable code runs in an electronic device, a processor in the electronic device executes the above method.

The disclosed embodiments also provide a computer program product comprising computer readable code or a non-volatile computer readable storage medium carrying computer readable code, which when run in an electronic device, a processor in the electronic device performs the above method.

An embodiment of the present disclosure further provides an electronic device, including: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to invoke the memory-stored executable instructions to perform the above-described method.

The electronic device may be provided as a terminal, server, or other form of device.

Fig. 9 shows a block diagram of an electronic device 1900 provided by an embodiment of the disclosure. For example, the electronic device 1900 may be provided as a server. Referring to fig. 9, electronic device 1900 includes a processing component 1922 further including one or more processors and memory resources, represented by memory 1932, for storing instructions, e.g., applications, that are executable by processing component 1922. The application programs stored in memory 1932 may include one or more modules that each correspond to a set of instructions. Further, the processing component 1922 is configured to execute instructions to perform the above-described method.

The electronic device 1900 may also include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input/output (I/O) interface 1958. The electronic device 1900 may operate based on an operating system, such as the Microsoft Server operating system (Windows Server), stored in the memory 1932 ^TM ) Apple Inc. of a graphical user interface based operating system (Mac OS X) ^TM ) Multi-user, multi-process computer operating system (Unix) ^TM ) Free and open native code Unix-like operating System (Linux) ^TM ) Open native code Unix-like operating System (FreeBSD) ^TM ) Or the like.

In an exemplary embodiment, a non-transitory computer readable storage medium, such as a memory 1932, is also provided that includes computer program instructions executable by a processing component 1922 of an electronic device 1900 to perform the above-described methods.

The disclosure relates to the field of augmented reality, and aims to detect or identify relevant features, states and attributes of a target object by means of various visual correlation algorithms by acquiring image information of the target object in a real environment, so as to obtain an AR effect combining virtual and reality matched with specific applications. For example, the target object may relate to a face, a limb, a gesture, an action, etc. associated with a human body, or a marker, a marker associated with an object, or a sand table, a display area, a display item, etc. associated with a venue or a place. The vision-related algorithms may involve visual localization, SLAM, three-dimensional reconstruction, image registration, background segmentation, keypoint extraction and tracking of objects, pose or depth detection of objects, and the like. The specific application can not only relate to interactive scenes such as navigation, explanation, reconstruction, virtual effect superposition display and the like related to real scenes or articles, but also relate to special effect treatment related to people, such as interactive scenes such as makeup beautification, limb beautification, special effect display, virtual model display and the like. The detection or identification processing of the relevant characteristics, states and attributes of the target object can be realized through the convolutional neural network. The convolutional neural network is a network model obtained by performing model training based on a deep learning framework.

The present disclosure may be systems, methods, and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied thereon for causing a processor to implement various aspects of the present disclosure.

The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.

Computer program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the disclosure are implemented by personalizing an electronic circuit, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA), with state information of computer-readable program instructions, which can execute the computer-readable program instructions.

Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The computer program product may be embodied in hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK), or the like.

The foregoing description of the various embodiments is intended to highlight different aspects of the various embodiments that are the same or similar, which can be referenced with one another and therefore are not repeated herein for brevity.

If the technical scheme of the embodiment of the disclosure relates to personal information, a product applying the technical scheme of the embodiment of the disclosure clearly informs personal information processing rules before processing the personal information, and obtains personal autonomous consent. If the technical scheme of the embodiment of the disclosure relates to sensitive personal information, before the sensitive personal information is processed, a product applying the technical scheme of the embodiment of the disclosure obtains individual consent, and simultaneously meets the requirement of 'explicit consent'. For example, at a personal information collection device such as a camera, a clear and significant identifier is set to inform that the personal information collection range is entered, the personal information is collected, and if the person voluntarily enters the collection range, the person is regarded as agreeing to collect the personal information; or on the device for processing the personal information, under the condition of informing the personal information processing rule by using obvious identification/information, obtaining personal authorization by modes of popping window information or asking a person to upload personal information of the person by himself, and the like; the personal information processing rule may include information such as a personal information processor, a personal information processing purpose, a processing method, and a type of personal information to be processed.

The foregoing description of the embodiments of the present disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A synchronous positioning and mapping SLAM system, comprising:

2. The system of claim 1, wherein the hardware accelerator is further configured to:

3. The system of claim 2, wherein the second set of serial tasks comprises a feature point coordinate prediction task;

the hardware accelerator includes: the first kernel is used for acquiring image coordinates of the feature points in a previous frame of image and current pose data of an Inertial Measurement Unit (IMU) sensor, and determining predicted values of the image coordinates of the feature points in a current frame of image according to the image coordinates of the feature points in the previous frame of image and the current pose data.

4. The system of claim 2 or 3, wherein the second set of serial tasks further comprises a three-dimensional coordinate correction task;

5. The system of claim 4, wherein the second core is the same core as the first core.

6. The system of any of claims 1 to 5, wherein the set of parallel tasks comprises IMU integration tasks;

7. The system of claim 6, wherein the hardware accelerator is further configured to:

and storing the current pose data.

8. The system according to any one of claims 1 to 7, wherein the parallel task set further comprises a feature point tracking task;

9. The system of claim 8, wherein the hardware accelerator is further configured to store: the image processing method comprises the steps of obtaining a target image block corresponding to the feature point, the latest value of the image coordinate of the feature point and the latest value of the three-dimensional coordinate corresponding to the feature point.

10. The system of claim 9, wherein the hardware accelerator is further configured to:

obtaining feature point screening results from the processor;

11. The system according to any one of claims 1 to 10, wherein the parallel task set further comprises a feature extraction task;

12. The system according to any one of claims 1 to 11, wherein the set of parallel tasks comprises matrix modification tasks;

13. The system of claims 1 to 12, wherein the hardware accelerator is further configured to:

14. The system of any one of claims 1 to 13, wherein the first set of serial tasks comprises at least one of: the method comprises a characteristic point screening task, a key frame identification task, a light beam adjustment optimization task, a track estimation task and a track optimization task.

15. A synchronous positioning and mapping SLAM method is characterized by comprising the following steps:

16. A synchronous positioning and mapping SLAM device, comprising:

17. An electronic device, comprising:

one or more processors;

a memory for storing executable instructions;

wherein the one or more processors are configured to invoke the memory-stored executable instructions to perform the method of claim 15.

18. A computer readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the method of claim 15.

19. A computer program product comprising computer readable code or a non-transitory computer readable storage medium carrying computer readable code which, when run in an electronic device, a processor in the electronic device performs the method of claim 15.