Nothing Special   »   [go: up one dir, main page]

CN113094545B - Redundant key frame eliminating method, SLAM method, system and electronic equipment - Google Patents

Redundant key frame eliminating method, SLAM method, system and electronic equipment Download PDF

Info

Publication number
CN113094545B
CN113094545B CN202010022951.4A CN202010022951A CN113094545B CN 113094545 B CN113094545 B CN 113094545B CN 202010022951 A CN202010022951 A CN 202010022951A CN 113094545 B CN113094545 B CN 113094545B
Authority
CN
China
Prior art keywords
key frame
key
frame
frames
redundant
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010022951.4A
Other languages
Chinese (zh)
Other versions
CN113094545A (en
Inventor
周刊
黄菊
周俊
王川川
兰国清
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sunny Optical Zhejiang Research Institute Co Ltd
Original Assignee
Sunny Optical Zhejiang Research Institute Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sunny Optical Zhejiang Research Institute Co Ltd filed Critical Sunny Optical Zhejiang Research Institute Co Ltd
Priority to CN202010022951.4A priority Critical patent/CN113094545B/en
Publication of CN113094545A publication Critical patent/CN113094545A/en
Application granted granted Critical
Publication of CN113094545B publication Critical patent/CN113094545B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/71—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Library & Information Science (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)
  • Auxiliary Devices For Music (AREA)

Abstract

A redundant key frame eliminating method, a SLAM method, a system and an electronic device thereof. The redundant key frame eliminating method comprises the following steps: responding to the running of global optimization, and screening candidate redundant key frames from a key frame database by checking the condition that observation points and source points on key frames in the key frame database are tracked by other key frames which are in common view relationship in the key frame database; removing effective key frames from the candidate redundant key frames to take the rest key frames in the candidate redundant key frames as determined redundant key frames, wherein the effective key frames are key frames with effective visual constraints with any image frame in a sliding window in the candidate redundant key frames; and deleting the determined redundant key frame from the key frame database to update the data in the key frame database.

Description

Redundant key frame eliminating method, SLAM method, system and electronic equipment
Technical Field
The invention relates to the technical field of SLAM (selective level shifting), in particular to a redundant key frame eliminating method, a SLAM method, a system and electronic equipment thereof.
Background
As a core technology of positioning and navigation of mobile robots, virtual reality, augmented reality and other devices, an algorithm framework of the positioning and three-dimensional mapping (English: simultaneous Localization and Mapping, SLAM for short) technology is perfect day by day, and the application of the algorithm framework is gradually changed from theoretical research to market application. Currently, a SLAM algorithm framework based on visual sensors or visual sensors coupled with Inertial Measurement Units (IMUs) generally includes: the system comprises a front-end module for extracting and tracking image features, a rear-end optimization module for estimating spatial positions and self poses, a loop detection module for correcting offset, a module for establishing a three-dimensional map and the like, wherein the rear-end optimization module mainly adopts two types of means of filtering optimization and nonlinear optimization. Representative cluster optimization (Bundle Adjustment) in nonlinear optimization has obvious advantages, and the pose graph and factor graph algorithm become a main application module of back-end optimization in SLAM systems.
Currently, since in SLAM problem solution based on nonlinear optimization, the state estimation problem is modeled as a maximum posterior probability problem, that is, a least squares calculation problem that minimizes the loss function assuming gaussian distribution, and is converted into an incremental iterative solution problem by taylor expansion. Therefore, to limit the state variables (i.e., parameters to be optimized) from increasing with the operation of the system, the existing SLAM method generally uses a sliding window to continuously read a limited volume of image frames for local optimization to reduce the calculation amount. Meanwhile, the conventional SLAM method also adopts a key frame construction mode (namely, common image frames are selectively lifted to be stored as key frames) so as to execute quick loop detection and global optimization, and therefore the quick loop detection and the global optimization are fed back into pose optimization of the common frames in a sliding window to realize compromise lifting of precision and speed and global consistent state information.
Although a large amount of information of common frames is skipped by constructing key frames in the rear end of the SLAM system to accelerate the execution of nonlinear optimization, as the system operates, the common image frames acquired by the sensor are continuously promoted to the key frames to be accumulated and stored in a key frame database, so that the number of parameters to be optimized is continuously increased, and further the speed and the precision of nonlinear optimization are reduced, and even the positioning precision and the memory occupation of the whole SLAM system are affected. Therefore, in the back-end optimization process of the SLAM system, a strategy of redundant keyframe elimination is needed to maintain a keyframe database with a reasonable volume for global optimization, so that the real-time performance of the system is maintained for a longer time in an actual application scene (such as a robot navigation or an AR wearing device), and the robustness is improved. Existing redundant key frame rejection strategies: one is to decide whether to raise the current image frame as key frame by setting the inserting frequency and judging the tracking stability, then when constructing the local map, the key frame with at least 90% of the characteristic points in the frame being repeatedly observed by the rest common view key frames is eliminated; the other is to refer to the average parallax between the current image frame and the previous frame and the number of the tracked feature points to determine whether to extract the current image frame in the sliding window as a key frame, and meanwhile, the downsampling process is carried out according to the distribution density, so that a pose image optimized key database is kept at a limited size, centralized key frames are discarded, and the time consumption of closed loop detection and global optimization is prevented from increasing.
However, although the above two existing redundant key frame rejection strategies reduce the volume of the key frame container (i.e., the key frame database), the more random key frame rejection method also discards considerable information and constraints contained in the key frames, resulting in a decrease in positioning accuracy of the SLAM system, which cannot meet the increasingly higher positioning accuracy requirements in the current augmented reality and positioning navigation fields.
Disclosure of Invention
The invention provides a redundant key frame eliminating method, an SLAM method, a system and electronic equipment thereof, which can selectively eliminate redundant key frames containing more repeated observation information, retain key frames with effective visual constraints, and facilitate the improvement of the positioning accuracy of an SLAM system while ensuring the instantaneity of the SLAM system.
Another advantage of the present invention is to provide a redundant key frame eliminating method, a SLAM method, a system and an electronic device thereof, wherein in an embodiment of the present invention, the redundant key frame eliminating method can select a candidate redundant key frame according to the tracking conditions of an observation point and a source point on the key frame, and then select a key frame affecting sliding window optimization from the candidate redundant key frames, so as to eliminate the remaining redundant key frames, thereby maintaining the key frame database volume in the SLAM system to be kept at a proper level.
Another advantage of the present invention is to provide a redundant key frame rejection method, a SLAM method, a system thereof, and an electronic device, wherein in an embodiment of the present invention, the redundant key frame rejection method can be applied to a currently mainstream SLAM technical scheme for implementing back-end global optimization based on a constructed key frame class.
Another advantage of the present invention is to provide a redundant key frame eliminating method, a SLAM method, a system and an electronic device thereof, where in an embodiment of the present invention, the redundant key frame eliminating method can maintain a key frame database having the most abundant observation information in the minimum coverage sense while eliminating as much as possible the key frames containing redundant information in the key database of the back-end global optimization, so that the global optimization time is greatly shortened, and the positioning accuracy and the real-time performance of the SLAM system are effectively improved.
Another advantage of the present invention is to provide a redundant key frame rejection method, a SLAM method, a system thereof, and an electronic device, wherein in an embodiment of the present invention, the redundant key frame rejection method can process co-vision conditions of an observation point and a source point included on a key frame by viewing tracking conditions of three-dimensional space points observed on the key frame, respectively.
Another advantage of the present invention is to provide a redundant key frame rejection method, a SLAM method, a system thereof, and an electronic device, wherein in an embodiment of the present invention, the redundant key frame rejection method can reject key frames containing more repeated observation information and less effective constraint according to co-view conditions of feature points on the key frames, thereby avoiding randomly discarding key frames containing effective information.
Another advantage of the present invention is to provide a redundant keyframe culling method and SLAM method, and a system and electronic device thereof, wherein in an embodiment of the present invention, the redundant keyframe culling method can simultaneously inspect the keyframes having visual constraints with the local image frames in the sliding window optimization, so as to preserve the keyframes to avoid the loss of the local optimization effect.
Another advantage of the present invention is to provide a redundant key frame rejection method and SLAM method, and a system and an electronic device thereof, wherein in order to achieve the above advantages, a complex structure and a huge calculation amount are not required in the present invention, and the requirements on software and hardware are low. Therefore, the present invention successfully and effectively provides a solution, not only provides a redundant key frame rejection method and a SLAM method and a system thereof and an electronic device, but also increases the practicability and reliability of the redundant key frame rejection method and the SLAM method and the system thereof and the electronic device.
To achieve at least one of the above or other advantages and objects, the present invention provides a redundant key frame culling method, including the steps of:
responding to the running of global optimization, and screening candidate redundant key frames from a key frame database by checking the condition that observation points and source points on key frames in the key frame database are tracked by other key frames which are in common view relationship in the key frame database;
removing effective key frames from the candidate redundant key frames to take the rest key frames in the candidate redundant key frames as determined redundant key frames, wherein the effective key frames are key frames with effective visual constraints with any image frame in a sliding window in the candidate redundant key frames; and
deleting the determined redundant key frame from the key frame database to update the data in the key frame database.
In one embodiment of the present invention, the step of screening candidate redundant key frames from the key frame database by checking that the observation point and the source point on the key frame in the key frame database are tracked by other key frames in the key frame database that are in a co-view relationship in response to the operation of global optimization includes the steps of:
Traversing all the key frames in the key frame database to take the currently traversed key frame as a current key frame;
orderly selecting a viewing key frame and a reference key frame which have a common view relation with the current key frame from the key frame database;
the three-dimensional space points observed by the reference key frame are respectively retrieved from the observation points and the source points in the inspection key frame so as to obtain the tracking times of each observation point and each source point in the inspection key frame;
counting the number of the observation points with the tracking times larger than a first preset time threshold and the number of the source points with the tracking times larger than a second preset time threshold on the detection key frame respectively to obtain a repeated visual observation number proportion and an effective visual observation number proportion; and
and in response to the number of repeated visual observations being greater than a predetermined repetition duty cycle threshold and the number of valid visual observations being less than a predetermined valid duty cycle threshold, determining the inspection keyframe as the candidate redundant keyframe.
In an embodiment of the present invention, the step of sequentially selecting the viewing key frame and the reference key frame having a common view relationship with the current key frame from the key frame database includes the steps of:
Sequentially taking out a certain key frame with a common view relation with the current key frame from the key frame database to sequentially serve as the view key frame; and
and circularly extracting another key frame with a common view relation with the current key frame from the key frame database to serve as the reference key frame.
In an embodiment of the present invention, the step of screening candidate redundant key frames from the key frame database by checking that the observation point and the source point on the key frame in the key frame database are tracked by other key frames in the key frame database that are in a co-view relationship in response to the running of the global optimization further includes the steps of:
and adaptively adjusting the predetermined repeated duty ratio threshold and the predetermined effective duty ratio threshold according to the total number of the observation points in the inspection key frame and the total number of the source points in the inspection key frame, so as to improve the judgment precision of invalid information in the inspection key frame.
In an embodiment of the present invention, the step of screening candidate redundant key frames from the key frame database by checking that the observation point and the source point on the key frame in the key frame database are tracked by other key frames in the key frame database that are in a co-view relationship in response to the running of the global optimization further includes the steps of:
And ending the step of screening the candidate redundant key frames from the key frame database in response to the number of the candidate redundant key frames reaching a predetermined ending threshold or traversing all the key frames in the key frame database.
In an embodiment of the present invention, the redundant key frame rejection method further includes the steps of:
and responding to the result that the oldest image frame in the sliding window is marginalized or the global optimization is fed back to the sliding window, and screening the key frame with effective visual constraint with any image frame in the sliding window from the key frame database to serve as the effective key frame.
In an embodiment of the present invention, the step of filtering out a key frame having an effective visual constraint with any image frame in the sliding window from the key frame database as the effective key frame in response to the oldest image frame in the sliding window being marginalized or the result of the global optimization being fed back to the sliding window includes the steps of:
screening the key frame nearest to the oldest image frame in the sliding window from the key frame database to be used as an associated key frame and the effective key frame;
screening out the key frames with the common view relation with the associated key frames from the key frame database to be used as the effective key frames; and
And screening the key frames with common view relation with other image frames in the sliding window from the key frame database to serve as the effective key frames.
According to another aspect of the present invention, there is also provided a SLAM method comprising the steps of:
acquiring successive image frames acquired via an image sensor;
the method comprises the steps of carrying out feature point tracking and extraction on a current image frame in the continuous image frames at the front end to obtain observation point data and source point data on the current image frame;
sliding window optimization is carried out on the current image frame and the observation point data and the source point data on the current image frame at the rear end so as to obtain a local optimization result;
in response to detecting that a loop exists or a new key frame is stored in a key frame database, performing global optimization on all key frames in the key frame database, and feeding back a global optimization result to a sliding window at the back end to correct the local optimization result;
responding to the running of global optimization, and screening candidate redundant key frames from the key frame database by checking the condition that observation points and source points on key frames in the key frame database are tracked by other key frames which are in common view relationship in the key frame database;
Removing effective key frames from the candidate redundant key frames to take the rest key frames in the candidate redundant key frames as determined redundant key frames, wherein the effective key frames are key frames with effective visual constraints with any image frame in a sliding window in the candidate redundant key frames; and
deleting the determined redundant key frame from the key frame database to update the data in the key frame database.
In an embodiment of the present invention, the SLAM method further includes the steps of:
and responding to the sliding window to execute the operation of sliding the image frames, and screening the key frames with effective visual constraints with any image frame in the sliding window from the key frame database to serve as the effective key frames.
According to another aspect of the present invention, there is also provided a redundant keyframe rejection system comprising:
the candidate frame screening module is used for responding to the running of global optimization, and screening candidate redundant key frames from the key frame database by checking the condition that observation points and source points on key frames in the key frame database are tracked by other key frames which are in common view relationship in the key frame database;
A redundant frame determining module, configured to reject a valid key frame from the candidate redundant key frames, where the valid key frame is a key frame having a valid visual constraint with any image frame in the sliding window in the candidate redundant key frames, and use remaining key frames in the candidate redundant key frames as determined redundant key frames; and
and a redundant frame deleting module for deleting the determined redundant key frame from the key frame database to update the data in the key frame database.
In an embodiment of the present invention, the candidate frame screening module includes a traversing module, a selecting module, a retrieving module, a statistics module and a determining module that are communicatively connected to each other, where the traversing module is configured to traverse all the key frames in the key frame database, so as to take the currently traversed key frame as a current key frame; the selecting module is used for orderly selecting a viewing key frame and a reference key frame which have a common view relation with the current key frame from the key frame database; the retrieval module is used for retrieving three-dimensional space points observed by the reference key frame from the observation points and the source points in the inspection key frame respectively so as to obtain the tracking times of each observation point and each source point in the inspection key frame; the statistics module is used for respectively counting the number of the observation points with the tracking times larger than a first preset time threshold and the number of the source points with the tracking times larger than a second preset time threshold on the inspection key frame so as to calculate the repeated visual observation number duty ratio and the effective visual observation number duty ratio; wherein the determination module is to determine the inspection keyframe to be the candidate redundant keyframe in response to the number of repeated visual observations being greater than a predetermined repetition duty cycle threshold and the number of valid visual observations being less than a predetermined valid duty cycle threshold.
In an embodiment of the present invention, the candidate frame filtering module further includes an adaptive adjustment module, configured to adaptively adjust the predetermined repetition duty threshold and the predetermined effective duty threshold according to a size between the total number of the observation points in the inspection keyframe and the total number of the source points in the inspection keyframe, so as to improve the accuracy of determining the invalid information in the inspection keyframe.
In an embodiment of the present invention, the candidate frame filtering module further includes an ending module, configured to end the step of filtering the candidate redundant key frames from the key frame database in response to the number of the candidate redundant key frames reaching a predetermined ending threshold or traversing all the key frames in the key frame database.
In an embodiment of the present invention, the redundant keyframe rejection system further includes an active frame screening module, configured to screen a keyframe having an active visual constraint with any image frame in the sliding window from the keyframe database as the active keyframe in response to the oldest image frame in the sliding window being marginalized or the result of the global optimization being fed back to the sliding window.
According to another aspect of the present invention, there is also provided a SLAM system including:
an acquisition module for acquiring successive image frames acquired via the image sensor;
the front-end module is used for tracking and extracting characteristic points of a current image frame in the continuous image frames at the front end so as to obtain observation point data and source point data on the current image frame;
the back-end module is used for carrying out sliding window optimization processing on the current image frame and the observation point data and the source point data on the current image frame at the back end so as to obtain a local optimization result;
the global optimization module is used for carrying out global optimization on all key frames in the key frame database in response to the detection that a loop exists or a new key frame is stored in the key frame database, and feeding back the result of the global optimization to the sliding window at the rear end to correct the result of the local optimization; and
a redundant keyframe culling system, wherein the redundant keyframe culling system comprises:
the candidate frame screening module is used for responding to the running of global optimization, and screening candidate redundant key frames from the key frame database by checking the condition that observation points and source points on key frames in the key frame database are tracked by other key frames which are in common view relationship in the key frame database;
A redundant frame determining module, configured to reject a valid key frame from the candidate redundant key frames, where the valid key frame is a key frame having a valid visual constraint with any image frame in the sliding window in the candidate redundant key frames, and use remaining key frames in the candidate redundant key frames as determined redundant key frames; and
and a redundant frame deleting module for deleting the determined redundant key frame from the key frame database to update the data in the key frame database.
According to another aspect of the present invention, there is also provided an electronic apparatus including:
at least one processor for executing instructions; and
a memory communicatively coupled to the at least one processor, wherein the memory has at least one instruction, wherein the instruction is executed by the at least one processor to cause the at least one processor to perform some or all of the steps in a SLAM method, wherein the SLAM method comprises the steps of:
acquiring successive image frames acquired via an image sensor;
the method comprises the steps of carrying out feature point tracking and extraction on a current image frame in the continuous image frames at the front end to obtain observation point data and source point data on the current image frame;
Sliding window optimization is carried out on the current image frame and the observation point data and the source point data on the current image frame at the rear end so as to obtain a local optimization result;
in response to detecting that a loop exists or a new key frame is stored in a key frame database, performing global optimization on all key frames in the key frame database, and feeding back a global optimization result to a sliding window at the back end to correct the local optimization result;
responding to the running of global optimization, and screening candidate redundant key frames from the key frame database by checking the condition that observation points and source points on key frames in the key frame database are tracked by other key frames which are in common view relationship in the key frame database;
removing effective key frames from the candidate redundant key frames to take the rest key frames in the candidate redundant key frames as determined redundant key frames, wherein the effective key frames are key frames with effective visual constraints with any image frame in a sliding window in the candidate redundant key frames; and
deleting the determined redundant key frame from the key frame database to update the data in the key frame database.
Further objects and advantages of the present invention will become fully apparent from the following description and the accompanying drawings.
These and other objects, features and advantages of the present invention will become more fully apparent from the following detailed description, the accompanying drawings and the appended claims.
Drawings
Fig. 1 is a flow chart of a redundant keyframe rejection method according to an embodiment of the present invention.
Fig. 2 shows a flow diagram of one of the steps of the redundant key frame culling method according to the above embodiment of the present invention.
Fig. 3 is a schematic flow chart of a second step of the redundant keyframe rejection method according to the above embodiment of the present invention.
Fig. 4 shows an example of the redundant key frame culling method according to the above-described embodiment of the present invention.
Fig. 5 shows a flow diagram of a SLAM method according to an embodiment of the invention.
FIG. 6 shows a block diagram schematic of a redundant keyframe culling system according to one embodiment of the present invention.
FIG. 7 shows a block diagram schematic of a SLAM system according to an embodiment of the present invention.
Fig. 8 shows a block diagram schematic of an electronic device according to an embodiment of the invention.
Detailed Description
The following description is presented to enable one of ordinary skill in the art to make and use the invention. The preferred embodiments in the following description are by way of example only and other obvious variations will occur to those skilled in the art. The basic principles of the invention defined in the following description may be applied to other embodiments, variations, modifications, equivalents, and other technical solutions without departing from the spirit and scope of the invention.
In the present invention, the terms "a" and "an" in the claims and specification should be understood as "one or more", i.e. in one embodiment the number of one element may be one, while in another embodiment the number of the element may be plural. The terms "a" and "an" are not to be construed as unique or singular, and the term "the" and "the" are not to be construed as limiting the amount of the element unless the amount of the element is specifically indicated as being only one in the disclosure of the present invention.
In the description of the present invention, it should be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. In the description of the present invention, unless explicitly stated or limited otherwise, the terms "connected," "connected," and "connected" should be interpreted broadly, and may be, for example, fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; may be directly connected or indirectly connected through a medium. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art according to the specific circumstances.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
In a back-end module of an existing SLAM system based on a constructed key frame, although the back-end module can perform nonlinear optimization on all the pose of the key frame and the position of a three-dimensional space point on the pose of the key frame in a manner of constructing the key frame so as to obtain the pose and the space position of the camera with global consistency, the common image frame acquired by a sensor is continuously lifted into the key frame along with the operation of the system, so that the number of parameters to be optimized is continuously increased due to the accumulation of the key frame stored in a key frame database, the speed and the precision of nonlinear optimization are reduced, and the positioning precision and the instantaneity of the whole SLAM system are further affected. Therefore, the judgment and elimination strategy of redundant key frames is necessary to be coupled in the global optimization process of the rear end of the SLAM system, so that the volume of a key frame database is reduced, and the instantaneity of the SLAM system is ensured.
However, although the existing redundant key frame judgment and rejection strategy can reduce the volume of a key frame database in the SLAM system, the random or rough rejection mode discards a considerable amount of information and constraints contained in the key frame, which results in the reduced positioning accuracy of the SLAM system. Therefore, in order to solve the above-mentioned problems, the present invention provides a redundant key frame eliminating method, a SLAM method, a system thereof and an electronic device, which can selectively eliminate key frames with more repeated observation information in the key frame database, and reserve key frames with effective visual constraints in the key frame database, so that the SLAM system of the present invention can still effectively maintain the number of key frames at a reasonable level in the later operation period, thereby improving the positioning accuracy of the system while reducing the time consumption of nonlinear optimization.
Schematic method
Referring to fig. 1 to 4 of the drawings of the specification, a redundant key frame culling method according to an embodiment of the present invention is illustrated. Specifically, as shown in fig. 1, the redundant key frame rejection method includes the steps of:
s100: responding to the running of global optimization, and screening candidate redundant key frames from a key frame database by checking the condition that observation points and source points on key frames in the key frame database are tracked by other key frames which are in common view relationship in the key frame database;
S200: removing effective key frames from the candidate redundant key frames to take the rest key frames in the candidate redundant key frames as determined redundant key frames, wherein the effective key frames are key frames with effective visual constraints with any image frame in a sliding window in the candidate redundant key frames; and
s300: deleting the determined redundant key frames from the key frame database to update the data in the key frame database.
Notably, in the process of eliminating redundant key frames, the redundant key frame eliminating method not only considers the situation that the observation point and the source point on the key frame in the key frame database are tracked by other key frames which are in common view relationship in the key frame database, so as to delete the observation frames containing excessive repeated observation information in the key frame database; but also consider the case of a key frame in the key frame database that has a valid visual constraint with an image frame in the sliding window, so as to preserve the key frame in the key frame database that has a valid visual constraint. Therefore, the redundant key frame eliminating method can maintain the volume of the key frame database in the SLAM system at a proper level, so that the SLAM system has higher real-time performance, and simultaneously, the key frames containing effective information can be prevented from being randomly discarded, so that the positioning accuracy of the SLAM system is improved.
It will be appreciated that in the present invention: the observation point refers to that a certain feature point on the current key frame is extracted from a historical key frame (namely a previous key frame) instead of the current key frame being observed for the first time; the source point is a feature point on a current key frame, which is observed by the current key frame for the first time, namely, a newly extracted feature point on the current key frame; the co-view relationship refers to a relationship between two image frames (and/or key frames) in which the same three-dimensional spatial point can be observed.
It should be noted that the front-end visual odometer of the SLAM system generally tracks the feature points (i.e., observation points) of the previous image frame through the pyramid LK optical flow method on the continuous image frames input by the image sensor, and when the number of tracked feature points is difficult to maintain tracking stability, newly extracts a proper number of feature points (i.e., source points) on the current image frame, and then transmits the image frames and the feature point data to the back end of the SLAM system. The rear end of the SLAM system adopts a sliding window (namely a sliding window) local optimization mode to estimate and optimize the space positions of a fixed number of continuous image frames and characteristic points on the continuous image frames, and when new incoming image frames enter the sliding window, the oldest image frames in the sliding window are subjected to marginalization processing. At the same time, when the back-end data is input, a proper image frame is also extracted as a key frame to be stored in the key frame database.
When detecting that a loop exists or a new key frame is stored in the key frame database, performing global optimization to nonlinear optimize state variables including all key frames and characteristic points on the key frame database, and then feeding back a globally consistent optimization result to sliding window optimization for correction. In other words, the process of global optimization needs to perform nonlinear optimization on objective functions constructed by all key frames and feature points on the key frames in the key frame database, so that the number of the key frames subjected to global optimization and three-dimensional space points observed on the key frames directly determines the solving speed of the global optimization, and therefore, with the operation of the SLAM system, the solving speed of the global optimization is slower, and the real-time performance and the positioning accuracy of the whole system are further affected.
Therefore, in the step S200 of the redundant keyframe culling method of the present invention, in response to the running of global optimization (i.e. performing global optimization), the situation that the observation point and the source point on the keyframe in the keyframe database are tracked by other keyframes in the keyframe database that are in a co-view relationship together is inspected, so as to screen the candidate redundant keyframes from the keyframe database.
Specifically, as shown in fig. 2, in the redundant key frame rejection method of the present invention, the step S100 may include the steps of:
s110: traversing all the key frames in the key frame database to take the currently traversed key frame as a current key frame;
s120: orderly selecting a viewing key frame and a reference key frame which have a common view relation with the current key frame from the key frame database;
s130: the three-dimensional space points observed by the reference key frame are respectively retrieved from the observation points and the source points on the inspection key frame so as to obtain the tracking times of each observation point and each source point on the inspection key frame;
s140: counting the number of the observation points with the tracking times greater than a first preset time threshold and the number of the source points with the tracking times greater than a second preset time threshold in the inspection key frame respectively to obtain a repeated visual observation number duty ratio and an effective visual observation number duty ratio respectively; and
s150: and in response to the number of repeated visual observations being greater than a predetermined repetition duty cycle threshold and the number of valid visual observations being less than a predetermined valid duty cycle threshold, determining the inspection keyframe as the candidate redundant keyframe.
Still further, in an example of the present invention, the step S120 may include the steps of:
sequentially taking out a certain key frame with a common view relation with the current key frame from the key frame database to sequentially serve as the viewing key frame; and
and circularly fetching another key frame with a common view relation with the current key frame from the key frame database as the reference key frame.
Illustratively, as shown in fig. 4, in the step S100 of the present invention, after the global optimization operation, the detection link of the redundant keyframes will be started to be executed: traversing all key frames in the key frame database (such as key frame container) and taking the currently traversed key frame as a current relationKey frame L k The method comprises the steps of carrying out a first treatment on the surface of the And then sequentially fetching the current key frame L k A key frame with a common view relationship as a viewing key frame X i The method comprises the following steps: x is X i ∈{X 1 ，X 2 ，X 3 …X n -wherein n is the current key frame L k Is a number of co-view key frames. Further, from the inspection keyframe X i The number of observation points (feature points of the frame before tracking) and information thereof, and the number of source points (i.e., newly extracted feature points) and information thereof are obtained. At this point, the inspection keyframe X is initialized for i Upper observation point and source point tracking condition counting container C m And C s The sizes of which respectively correspond to the inspection key frames X i Total number N of upper observation points m And total number of source points N s And sets the initial count of each bit to zero.
Subsequently, cyclically retrieving the current key frame L from the key frame database k Another keyframe also having a co-view relationship is taken as reference keyframe X j The method comprises the following steps: x is X j ∈{X 1 ，X 2 ，X 3 …X n I +.j }, where n is the current key frame L k Is a number of co-view key frames. At this point, the inspection keyframe X is retrieved i And the reference key frame X j Wherein the three-dimensional space point is simultaneously inspected by the keyframe X when the three-dimensional space point originates from a feature point of a previous frame i And the reference key frame X j Tracking and recording as observation points to obtain the three-dimensional space points in the inspection keyframe X i Corresponding to the position in the observation point container and in the observation key frame X i Counting container C for tracking observation point of (1) m The corresponding serial number position count is incremented by one; and when the three-dimensional space point originates from the inspection keyframe X i When the newly extracted feature points are up, the three-dimensional space points are inspected by the key frame X i Observed for the first time and is observed by the reference key frame X j Tracking to obtain the three-dimensional space point in the inspection key frame X by taking record as a source point i Bits in the up-corresponding source point containerPut and view key frame X i Source point tracking condition counting container C s The corresponding sequence number position count is incremented by one.
Then, after the inspection key frame X is circularly traversed i And all of the reference key frames X j A co-view relationship between (i.e. the current key frame L k Is related to the current common view key frame and the current key frame L k The remaining common view key frames of (2) calculate the view key frame X i Tracking of all three-dimensional space points contained above. In other words, the inspection keyframes X are processed separately i Tracking the upper observation point and the source point by counting the inspection key frame X i Counting container C for tracking observation point of (1) m And source point tracking situation counting container C s . For example, when the keyframe X is viewed i One observation point on the current key frame L is also the same as the other observation point k Keyframes having a co-view relationship (i.e. the reference keyframe X j ) The inspection key frame X when the observed total times meet the first preset times threshold i Observation point counter M of (1) m To pass through the observation point counter M m Representing the viewing keyframe X by the number of repeated visual observations i The degree of repeated visual observation information is included; while viewing key frame X i A source point on the current key frame L is also replaced with another key frame k Keyframes having a co-view relationship (i.e. the reference keyframe X j ) The inspection key frame X when the observed total times meet the second preset times threshold value i Source point counter M of (2) s To be incremented by one to pass through the source counter M s Is representative of the viewing keyframe X (i.e., the amount of active visual inspection information) i Including the extent of effective visual observation information.
It should be noted that, in the above embodiment of the present invention, the first predetermined number of times threshold value and the second predetermined number of times threshold value may be the same or different. For example, the first predetermined number of times threshold may be implemented as, but is not limited to, 3; and the second predetermined number of times threshold may be implemented as, but is not limited to, 2 or 3.
Thus, the inspection keyframe X is counted each time i After the number of repeated visual observation information (i.e. the number of observation points with the tracking number greater than the first predetermined number of times threshold) and the number of effective visual observation information (i.e. the number of source points with the tracking number greater than the second predetermined number of times threshold), a duty ratio threshold judgment link is performed: the inspection key frame X i Number M of the repeated visual observation information m And the number M of the effective visual observation information s Respectively correspond to the inspection key frame X i Total number N of upper observation points m And total number of source points N s Comparing to find the ratio M of the number of repeated visual observations m /N m And the effective visual observation quantity ratio M s /N s . At this time, if the number of repeated visual observations is the ratio M m /N m Greater than the predetermined repetition duty threshold a), and the effective number of visual observations duty cycle M s /N s If the predetermined effective duty ratio threshold b% is smaller, then the inspection key frame X is considered i Contains excessive invalid information, the inspection key frame X can be inspected i As candidate redundant key frames, storing the candidate redundant key frame eliminating sequences; otherwise, the inspection key frame X is not inspected i As candidate redundant key frames.
It should be noted that, since the front end of the SLAM system generally maintains a fixed number of feature points per frame of image in order to prevent the failure of tracking the front and rear images during the process of extracting and tracking the feature points of the image, the sum of the total number of observation points and the total number of source points tends to be constant (i.e., N m +N s =constant), the redundant key frame rejection method of the present invention may also adaptively adjust the magnitudes of the predetermined repetition duty cycle threshold and the predetermined effective duty cycle threshold.
Specifically, as shown in fig. 2, the step S100 of the redundant key frame rejection method may further include, before the step S140, the steps of:
s160: and adaptively adjusting the magnitudes of the preset repeated duty ratio threshold and the preset effective duty ratio threshold according to the magnitude between the total number of the observation points and the total number of the source points on the inspection key frame so as to improve the judgment accuracy of invalid information in the inspection key frame.
Illustratively, in adaptively adjusting the predetermined repetition duty cycle threshold and the predetermined effective duty cycle threshold, when the keyframe X is viewed i Correspondingly increasing the preset repeated duty ratio threshold value a% and/or the preset effective duty ratio threshold value b% when the total number of the observation points is more; when the inspection key frame X i And correspondingly reducing the preset repeated duty ratio threshold value a% and/or the preset effective duty ratio threshold value b% when the total number of the source points is more.
In addition, in order to reduce the calculation amount of screening the candidate redundant key frames, so as to further improve the real-time performance of the SLAM system, the redundant key frame eliminating method of the present invention preferably decides whether to skip the screening cycle according to the number of the candidate redundant key frames, that is, when the number of the candidate redundant key frames reaches a sufficient number, the screening process of the candidate redundant key frames is ended (i.e. the i cycle, the j cycle or the k cycle in step S100 is ended), and it is not necessary to traverse all the key frames in the key frame database, so as to reduce the overall calculation amount and improve the real-time performance of the SLAM system.
Specifically, as shown in fig. 2, the step S100 of the redundant keyframe rejection method of the present invention may further include the steps of:
s170: and ending the step of screening the candidate redundant key frames from the key frame data in response to the number of candidate redundant key frames reaching a predetermined ending threshold or traversing all the key frames in the key frame database.
Preferably, the predetermined end threshold may be 1, that is, when one of the candidate redundant key frames is screened out from the key frame database, the screening process of the candidate redundant key frame is ended. Of course, in other examples of the present invention, the predetermined ending threshold may be more than one, and may be set according to requirements of a specific application scenario.
According to the above embodiment of the present invention, after obtaining the candidate redundant key frames, in order to keep key frames with valid visual constraints while deleting key frames containing redundant information in the key frame database as much as possible, the redundant key frame culling method of the present invention further avoids loss of local optimization effects of the SLAM system by viewing key frames with visual constraints with any image frame in the sliding window before deleting corresponding key frames from the key frame database. In other words, when a key frame with visual constraint with any image frame in the sliding window exists in the candidate redundant key frames, the key frame with visual constraint is removed from the candidate redundant key frames, and the rest of the candidate redundant key frames are used as the determined redundant key frames so as to delete the determined redundant key frames from the key frame database.
Specifically, as shown in fig. 1, the redundant key frame eliminating method of the present invention further includes the steps of:
s400: and in response to the oldest image frame in the sliding window being marginalized or the result of the global optimization being fed back to the sliding window, screening out key frames from the key frame database, which have effective visual constraints with any one of the image frames in the sliding window, as the effective key frames.
More specifically, as shown in fig. 3, the step S400 includes the steps of:
s410: screening the key frame nearest to the oldest image frame in the sliding window from the key frame database to be used as an associated key frame and the effective key frame;
s420: screening out the key frames with the common view relation with the associated key frames from the key frame database to be used as the effective key frames; and
s430: and screening the key frames with common view relation with other image frames in the sliding window from the key frame database to serve as the effective key frames.
It will be appreciated that the associated keyframe of the present invention may be the keyframe closest to the oldest image frame in the sliding window in the keyframe database, since the image frame in the sliding window, when marginalized, will shift the constraint it carries to the keyframe closest thereto. That is, in the key frame database, the time interval between the associated key frame and the oldest image frame within the sliding window is minimal.
According to another aspect of the present invention, as shown in fig. 5, the present invention further provides a SLAM method, including the steps of:
a: acquiring successive image frames acquired via an image sensor;
b: the method comprises the steps of carrying out feature point tracking and extraction on a current image frame in the continuous image frames at the front end to obtain observation point data and source point data on the current image frame;
c: performing sliding window optimization processing on the current image frame and the observation point data and the source point data on the current image frame at the rear end to obtain a local optimization result;
d: in response to detecting that a loop exists or a new key frame is stored in a key frame database, performing global optimization on all key frames in the key frame database, and feeding back a global optimization result to a sliding window at the rear end to correct the local optimization result;
e: responding to the running of global optimization, and screening candidate redundant key frames from the key frame database by checking the condition that observation points and source points on the key frames in the key frame database are tracked by other key frames which are in common view relationship in the key frame database;
f: removing effective key frames from the candidate redundant key frames to take the rest key frames in the candidate redundant key frames as determined redundant key frames, wherein the effective key frames are key frames with effective visual constraints with any image frame in the sliding window in the candidate redundant key frames; and
G: deleting the determined redundant key frames from the key frame database to update the data in the key frame database.
It is noted that the sensor data utilized by the SLAM method may include not only continuous image frames acquired via an image sensor, but also IMU data such as acquired via an Inertial Measurement Unit (IMU), etc., which is not described in detail herein.
Furthermore, in the process of sliding window optimization by the back end, every time a new image frame is input from the front end to the sliding window of the back end, the oldest one of the sliding windows needs to be marginalized to maintain the volume of the sliding window, while the information and constraints (i.e., marginalized constraints) carried on the oldest one will be preserved by the Schur complement matrix. Since the marginalized constraint generated by this process optimizes the objective function of the nonlinear optimization placed in the sliding window, both the reference key frame associated with the oldest frame in the sliding window and the key frame having a co-view relationship with the reference key frame need to be preserved to prevent loss of relevant constraint information. Meanwhile, the common view key frame associated with other image frames in the sliding window also needs to be reserved, so, as shown in fig. 5, the SLAM method of the present invention further includes the steps of:
H: and responding to the sliding window to execute the operation of sliding the image frames, and screening the key frames with effective visual constraints with any image frame in the sliding window from the key frame database to serve as the effective key frames.
Of course, in other examples of the invention, the SLAM method may also include the steps of: and responding to the result of the global optimization and feeding back to the sliding window, and screening key frames with effective visual constraints on any image frame in the sliding window from the key frame database to serve as the effective key frames.
Schematic System
Referring to fig. 6 of the drawings, a redundant keyframe rejection system according to an embodiment of the present invention is illustrated, wherein the redundant keyframe rejection system is capable of determining and rejecting keyframes that contain more repetitive observation information based on observation points and source point tracking on the keyframes, while preserving keyframes with valid visual constraints. Specifically, as shown in fig. 6, the redundant key frame removal system 10 includes a candidate frame screening module 11, a redundant frame determination module 12, and a redundant frame deletion module 13 that are communicatively connected to each other. The candidate frame screening module 11 is configured to screen candidate redundant key frames from the key frame database by checking that an observation point and a source point on a key frame in the key frame database are tracked by other key frames in the key frame database that are in a common-view relationship in response to the global optimization. The redundant frame determination module 12 is configured to reject a valid key frame from the candidate redundant key frames, where the valid key frame is a key frame that has a valid visual constraint with any image frame in the sliding window in the candidate redundant key frames, and use the remaining key frames in the candidate redundant key frames as determined redundant key frames. The redundant frame deletion module 13 is configured to delete the determined redundant key frame from the key frame database to update the data in the key frame database.
More specifically, as shown in fig. 6, in an example of the present invention, the candidate frame screening module 11 includes a traversing module 111, a selecting module 112, a retrieving module 113, a statistics module 114, and a determining module 115 that are communicatively connected to each other, where the traversing module 111 is configured to traverse all the key frames in the key frame database, so as to take the currently traversed key frame as a current key frame; wherein the selection module 112 is configured to sequentially select a viewing keyframe and a reference keyframe having a common view relationship with the current keyframe from the keyframe database; the retrieving module 113 is configured to retrieve three-dimensional space points observed by the reference key frame from the observation points and the source points in the inspection key frame, respectively, so as to obtain tracking times of each observation point and each source point in the inspection key frame; wherein the statistics module 114 is configured to count the number of observation points with the tracking number greater than the first predetermined number threshold and the number of source points with the tracking number greater than the second predetermined number threshold on the inspection keyframe, respectively, so as to calculate a repeated visual observation number duty ratio and an effective visual observation number duty ratio; wherein the determining module 115 is configured to determine the inspection keyframe to be the candidate redundant keyframe in response to the number of repeated visual observations being greater than a predetermined repetition duty cycle threshold and the number of valid visual observations being less than a predetermined valid duty cycle threshold.
It should be noted that, as shown in fig. 6, in an example of the present invention, the candidate frame filtering module 11 further includes an adaptive adjustment module 116, configured to adaptively adjust the predetermined repetition duty threshold and the predetermined effective duty threshold according to the magnitude between the total number of the observation points in the inspection keyframe and the total number of the source points in the inspection keyframe, so as to improve the accuracy of determining the invalid information in the inspection keyframe.
In addition, as shown in fig. 6, the candidate frame filtering module 11 may further include an ending module 117, configured to end the step of filtering the candidate redundant key frames from the key frame database in response to the number of the candidate redundant key frames reaching a predetermined ending threshold or traversing all the key frames in the key frame database.
In accordance with the above embodiment of the present invention, as shown in fig. 6, the redundant keyframe rejection system 10 further includes an active frame screening module 14 for screening keyframes from the keyframe database that have an active visual constraint with any image frame in the sliding window as the active keyframes in response to the oldest image frame in the sliding window being marginalized or the result of the global optimization being fed back to the sliding window.
According to another aspect of the present invention, there is further provided a SLAM system, as shown in fig. 7, wherein the SLAM system 1 includes an acquisition module 20, a front-end module 30, a back-end module 40, a global optimization module 50, and any of the redundant keyframe rejection systems 10 described above, which are communicatively connected to each other. The acquisition module 20 is used for acquiring successive image frames acquired via the image sensor. The front-end module 30 is configured to track and extract feature points of a current image frame in the continuous image frames at the front end, so as to obtain observation point data and source point data on the current image frame. The back-end module 40 is configured to perform sliding window optimization on the current image frame and the observation point data and the source point data on the current image frame at the back end to obtain a locally optimized result. The global optimization module 50 is configured to perform global optimization on all key frames in the key frame database in response to detecting that a loop exists or a new key frame is stored in the key frame database, and feed back a result of the global optimization to the sliding window of the back end to correct the result of the local optimization. The redundant key frame rejection system 10 includes a candidate frame screening module 11, a redundant frame determination module 12, and a redundant frame deletion module 13 communicatively coupled to one another.
In particular, as shown in fig. 7, the candidate frame screening module 11 is configured to screen candidate redundant key frames from the key frame database by checking that the observation point and the source point on the key frame in the key frame database are tracked by other key frames in the key frame database that are in a co-view relationship in response to the global optimization. The redundant frame determination module 12 is configured to reject a valid key frame from the candidate redundant key frames, where the valid key frame is a key frame that has a valid visual constraint with any image frame in the sliding window in the candidate redundant key frames, and use the remaining key frames in the candidate redundant key frames as determined redundant key frames. The redundant frame deletion module 13 is configured to delete the determined redundant key frame from the key frame database to update the data in the key frame database.
Schematic electronic device
Next, an electronic device according to an embodiment of the present invention is described with reference to fig. 8. As shown in fig. 8, the electronic device 90 includes one or more processors 91 and memory 92.
The processor 91 may be a Central Processing Unit (CPU) or other form of processing unit having data processing and/or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions. In other words, the processor 91 comprises one or more physical devices configured to execute instructions. For example, the processor 91 may be configured to execute instructions that are part of: one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, implement a technical effect, or otherwise achieve a desired result.
The processor 91 may include one or more processors configured to execute software instructions. Additionally or alternatively, the processor 91 may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processors of the processor 91 may be single-core or multi-core, and the instructions executed thereon may be configured for serial, parallel, and/or distributed processing. The various components of the processor 91 may optionally be distributed across two or more separate devices, which may be remotely located and/or configured for coordinated processing. Aspects of the processor 91 may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration.
The memory 92 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random Access Memory (RAM) and/or cache memory (cache), and the like. The non-volatile memory may include, for example, read Only Memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer readable storage medium and executed by the processor 11 to perform some or all of the steps in the above-described exemplary methods of the present invention, and/or other desired functions.
In other words, the memory 92 includes one or more physical devices configured to hold machine readable instructions executable by the processor 91 to implement the methods and processes described herein. In implementing these methods and processes, the state of the memory 92 may be transformed (e.g., different data is saved). The memory 92 may include removable and/or built-in devices. The memory 92 may include optical memory (e.g., CD, DVD, HD-DVD, blu-ray disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and/or magnetic memory (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), among others. The memory 92 may include volatile, nonvolatile, dynamic, static, read/write, read-only, random access, sequential access, location-addressable, file-addressable, and/or content-addressable devices.
It is to be appreciated that the memory 92 includes one or more physical devices. However, aspects of the instructions described herein may alternatively be propagated through a communication medium (e.g., an electromagnetic signal, an optical signal, etc.) that is not held by a physical device for a limited period of time. Aspects of the processor 91 and the memory 92 may be integrated together into one or more hardware logic components. These hardware logic components may include, for example, field Programmable Gate Arrays (FPGAs), program and application specific integrated circuits (PASICs/ASICs), program and application specific standard products (PSSPs/ASSPs), system on a chip (SOCs), and Complex Programmable Logic Devices (CPLDs).
In one example, as shown in FIG. 8, the electronic device 90 may also include an input device 93 and an output device 94, which are interconnected by a bus system and/or other form of connection mechanism (not shown). For example, the input device 93 may be, for example, a camera module or the like for capturing image data or video data. As another example, the input device 93 may include or interface with one or more user input devices such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input device 93 may include or interface with selected Natural User Input (NUI) components. Such component parts may be integrated or peripheral and the transduction and/or processing of the input actions may be processed on-board or off-board. Example NUI components may include microphones for speech and/or speech recognition; infrared, color, stereoscopic display, and/or depth cameras for machine vision and/or gesture recognition; head trackers, eye trackers, accelerometers and/or gyroscopes for motion detection and/or intent recognition; and an electric field sensing component for assessing brain activity and/or body movement; and/or any other suitable sensor.
The output device 94 may output various information including the classification result and the like to the outside. The output device 94 may include, for example, a display, speakers, a printer, and a communication network and remote output devices connected thereto, etc.
Of course, the electronic device 90 may further comprise the communication means, wherein the communication means may be configured to communicatively couple the electronic device 90 with one or more other computer devices. The communication means may comprise wired and/or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem may be configured for communication via a wireless telephone network or a wired or wireless local area network or wide area network. In some embodiments, the communications apparatus may allow the electronic device 90 to send and/or receive messages to and/or from other devices via a network such as the Internet.
It will be appreciated that the configurations and/or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various acts illustrated and/or described may be performed in the sequence illustrated and/or described, in other sequences, in parallel, or omitted. Also, the order of the above-described processes may be changed.
Of course, only some of the components of the electronic device 90 that are relevant to the present invention are shown in fig. 8 for simplicity, components such as buses, input/output interfaces, etc. are omitted. In addition, the electronic device 90 may include any other suitable components depending on the particular application.
Illustrative computing program product
In addition to the methods and apparatus described above, embodiments of the invention may also be a computer program product comprising computer program instructions which, when executed by a processor, cause the processor to perform the steps in a method according to various embodiments of the invention described in the "exemplary methods" section of this specification.
The computer program product may write program code for performing the operations of embodiments of the present invention in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the C programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present invention may also be a computer readable storage medium, having stored thereon computer program instructions, which when executed by a processor, cause the processor to perform the steps of the method described above in the present specification.
The computer readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The basic principles of the present invention have been described above in connection with specific embodiments, however, it should be noted that the advantages, benefits, effects, etc. mentioned in the present invention are merely examples and not intended to be limiting, and these advantages, benefits, effects, etc. are not to be considered as essential to the various embodiments of the present invention. Furthermore, the specific details disclosed herein are for purposes of illustration and understanding only, and are not intended to be limiting, as the invention is not necessarily limited to practice with the above described specific details.
The block diagrams of the devices, apparatuses, devices, systems referred to in the present invention are only illustrative examples and are not intended to require or imply that the connections, arrangements, configurations must be made in the manner shown in the block diagrams. As will be appreciated by one of skill in the art, the devices, apparatuses, devices, systems may be connected, arranged, configured in any manner. Words such as "including," "comprising," "having," and the like are words of openness and mean "including but not limited to," and are used interchangeably therewith. The terms "or" and "as used herein refer to and are used interchangeably with the term" and/or "unless the context clearly indicates otherwise. The term "such as" as used herein refers to, and is used interchangeably with, the phrase "such as, but not limited to.
It is also noted that in the apparatus, devices and methods of the present invention, the components or steps may be disassembled and/or assembled. Such decomposition and/or recombination should be considered as equivalent aspects of the present invention.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the invention. Thus, the present invention is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
It will be appreciated by persons skilled in the art that the embodiments of the invention described above and shown in the drawings are by way of example only and are not limiting. The objects of the present invention have been fully and effectively achieved. The functional and structural principles of the present invention have been shown and described in the examples and embodiments of the invention may be modified or practiced without departing from the principles described.

Claims (14)

1. The redundant key frame eliminating method is characterized by comprising the following steps:
responding to the running of global optimization, and screening candidate redundant key frames from a key frame database by checking the condition that observation points and source points on key frames in the key frame database are tracked by other key frames which are in common view relationship in the key frame database;
when a key frame with visual constraint with any image frame in the sliding window exists in the candidate redundant key frames, eliminating effective key frames from the candidate redundant key frames, and taking the rest key frames in the candidate redundant key frames as determined redundant key frames, wherein the effective key frames are key frames with effective visual constraint with any image frame in the sliding window in the candidate redundant key frames; the effective key frames comprise the key frame nearest to the oldest image frame in the sliding window, the key frame with a common view relation with the associated key frame and the key frame with a common view relation with other image frames in the sliding window; and
Deleting the determined redundant key frame from the key frame database to update the data in the key frame database;
wherein, in response to the running of the global optimization, the step of screening candidate redundant key frames from the key frame database by checking the condition that the observation point and the source point on the key frame in the key frame database are tracked by other key frames which are in the same common view relationship in the key frame database comprises the steps of:
traversing all the key frames in the key frame database to take the currently traversed key frame as a current key frame;
orderly selecting a viewing key frame and a reference key frame which have a common view relation with the current key frame from the key frame database;
the three-dimensional space points observed by the reference key frame are respectively retrieved from the observation points and the source points in the inspection key frame so as to obtain the tracking times of each observation point and each source point in the inspection key frame;
counting the number of the observation points with the tracking times larger than a first preset time threshold and the number of the source points with the tracking times larger than a second preset time threshold on the detection key frame respectively to obtain a repeated visual observation number proportion and an effective visual observation number proportion; and
And in response to the number of repeated visual observations being greater than a predetermined repetition duty cycle threshold and the number of valid visual observations being less than a predetermined valid duty cycle threshold, determining the inspection keyframe as the candidate redundant keyframe.
2. The redundant key frame elimination method of claim 1, wherein the step of sequentially selecting the viewing key frame and the reference key frame having a co-view relationship with the current key frame from the key frame database comprises the steps of:
sequentially taking out a certain key frame with a common view relation with the current key frame from the key frame database to sequentially serve as the view key frame; and
and circularly extracting another key frame with a common view relation with the current key frame from the key frame database to serve as the reference key frame.
3. The redundant key frame culling method as claimed in claim 2, wherein said step of screening candidate redundant key frames from the key frame database by checking that an observation point and a source point on a key frame in the key frame database are tracked by other key frames in the key frame database that are co-located in a co-view relationship in response to the operation of global optimization, further comprises the steps of:
And adaptively adjusting the predetermined repeated duty ratio threshold and the predetermined effective duty ratio threshold according to the total number of the observation points in the inspection key frame and the total number of the source points in the inspection key frame, so as to improve the judgment precision of invalid information in the inspection key frame.
4. A redundant key frame culling method as defined in claim 3, wherein said step of screening candidate redundant key frames from a key frame database by examining the condition that an observation point and a source point on a key frame in the key frame database are tracked by other key frames in the key frame database that are co-located in a co-view relationship in response to the operation of global optimization, further comprises the steps of:
and ending the step of screening the candidate redundant key frames from the key frame database in response to the number of the candidate redundant key frames reaching a predetermined ending threshold or traversing all the key frames in the key frame database.
5. The redundant key frame rejection method according to any one of claims 1 to 4, further comprising the steps of:
and responding to the result that the oldest image frame in the sliding window is marginalized or the global optimization is fed back to the sliding window, and screening the key frame with effective visual constraint with any image frame in the sliding window from the key frame database to serve as the effective key frame.
6. The redundant keyframe culling method of claim 5, wherein said step of screening out a keyframe having a valid visual constraint with any image frame in the sliding window from the keyframe database as the valid keyframe in response to the oldest image frame in the sliding window being marginalized or the result of the global optimization being fed back to the sliding window, comprises the steps of:
screening the key frame nearest to the oldest image frame in the sliding window from the key frame database to be used as an associated key frame and the effective key frame;
screening out the key frames with the common view relation with the associated key frames from the key frame database to be used as the effective key frames; and
and screening the key frames with common view relation with other image frames in the sliding window from the key frame database to serve as the effective key frames.
A slam method, comprising the steps of:
acquiring successive image frames acquired via an image sensor;
the method comprises the steps of carrying out feature point tracking and extraction on a current image frame in the continuous image frames at the front end to obtain observation point data and source point data on the current image frame;
sliding window optimization is carried out on the current image frame and the observation point data and the source point data on the current image frame at the rear end so as to obtain a local optimization result;
In response to detecting that a loop exists or a new key frame is stored in a key frame database, performing global optimization on all key frames in the key frame database, and feeding back a global optimization result to a sliding window at the back end to correct the local optimization result;
responding to the running of global optimization, and screening candidate redundant key frames from the key frame database by checking the condition that observation points and source points on key frames in the key frame database are tracked by other key frames which are in common view relationship in the key frame database;
when a key frame with visual constraint with any image frame in the sliding window exists in the candidate redundant key frames, eliminating effective key frames from the candidate redundant key frames, and taking the rest key frames in the candidate redundant key frames as determined redundant key frames, wherein the effective key frames are key frames with effective visual constraint with any image frame in the sliding window in the candidate redundant key frames; the effective key frames comprise the key frame nearest to the oldest image frame in the sliding window, the key frame with a common view relation with the associated key frame and the key frame with a common view relation with other image frames in the sliding window; and
Deleting the determined redundant key frame from the key frame database to update the data in the key frame database;
wherein, in response to the running of the global optimization, the step of screening candidate redundant key frames from the key frame database by checking the condition that the observation point and the source point on the key frame in the key frame database are tracked by other key frames which are in the same common view relationship in the key frame database comprises the steps of:
traversing all the key frames in the key frame database to take the currently traversed key frame as a current key frame;
orderly selecting a viewing key frame and a reference key frame which have a common view relation with the current key frame from the key frame database;
the three-dimensional space points observed by the reference key frame are respectively retrieved from the observation points and the source points in the inspection key frame so as to obtain the tracking times of each observation point and each source point in the inspection key frame;
counting the number of the observation points with the tracking times larger than a first preset time threshold and the number of the source points with the tracking times larger than a second preset time threshold on the detection key frame respectively to obtain a repeated visual observation number proportion and an effective visual observation number proportion; and
And in response to the number of repeated visual observations being greater than a predetermined repetition duty cycle threshold and the number of valid visual observations being less than a predetermined valid duty cycle threshold, determining the inspection keyframe as the candidate redundant keyframe.
8. The SLAM method of claim 7, further comprising the step of:
and responding to the sliding window to execute the operation of sliding the image frames, and screening the key frames with effective visual constraints with any image frame in the sliding window from the key frame database to serve as the effective key frames.
9. A redundant keyframe culling system comprising, communicatively coupled to each other:
the candidate frame screening module is used for responding to the running of global optimization, and screening candidate redundant key frames from the key frame database by checking the condition that observation points and source points on key frames in the key frame database are tracked by other key frames which are in common view relationship in the key frame database;
a redundant frame determining module, configured to, when a key frame having a visual constraint with any image frame in the sliding window exists in the candidate redundant key frames, reject a valid key frame from the candidate redundant key frames, so as to use remaining key frames in the candidate redundant key frames as determined redundant key frames, where the valid key frame is a key frame having a valid visual constraint with any image frame in the sliding window in the candidate redundant key frames; the effective key frames comprise the key frame nearest to the oldest image frame in the sliding window, the key frame with a common view relation with the associated key frame and the key frame with a common view relation with other image frames in the sliding window; and
A redundant frame deleting module for deleting the determined redundant key frame from the key frame database to update the data in the key frame database;
the candidate frame screening module comprises a traversing module, a selecting module, a retrieving module, a statistics module and a judging module which are mutually connected in a communication mode, wherein the traversing module is used for traversing all key frames in the key frame database so as to take the currently traversed key frame as a current key frame; the selecting module is used for orderly selecting a viewing key frame and a reference key frame which have a common view relation with the current key frame from the key frame database;
the retrieval module is used for retrieving three-dimensional space points observed by the reference key frame from the observation points and the source points in the inspection key frame respectively so as to obtain the tracking times of each observation point and each source point in the inspection key frame;
the statistics module is used for respectively counting the number of the observation points with the tracking times larger than a first preset time threshold and the number of the source points with the tracking times larger than a second preset time threshold on the inspection key frame so as to calculate the repeated visual observation number duty ratio and the effective visual observation number duty ratio;
Wherein the determination module is to determine the inspection keyframe to be the candidate redundant keyframe in response to the number of repeated visual observations being greater than a predetermined repetition duty cycle threshold and the number of valid visual observations being less than a predetermined valid duty cycle threshold.
10. The redundant key frame rejection system of claim 9, wherein the candidate frame screening module further comprises an adaptive adjustment module for adaptively adjusting the predetermined repetition duty threshold and the predetermined effective duty threshold according to a size between a total number of the observation points in the inspection key frame and a total number of the source points in the inspection key frame to improve a determination accuracy of invalid information in the inspection key frame.
11. The redundant key frame elimination system of claim 10 wherein the candidate frame screening module further comprises an ending module for ending the step of screening candidate redundant key frames from the key frame database in response to the number of candidate redundant key frames reaching a predetermined ending threshold or traversing all of the key frames in the key frame database.
12. A redundant keyframe rejection system according to any one of claims 9 to 11, further comprising an active frame screening module for screening keyframes from the keyframe database that have an active visual constraint with any one of the image frames in the sliding window as the active keyframes in response to the oldest image frame in the sliding window being marginalized or the result of the global optimization being fed back to the sliding window.
Slam system, characterized by comprising, communicatively connected to each other:
an acquisition module for acquiring successive image frames acquired via the image sensor;
the front-end module is used for tracking and extracting characteristic points of a current image frame in the continuous image frames at the front end so as to obtain observation point data and source point data on the current image frame;
the back-end module is used for carrying out sliding window optimization processing on the current image frame and the observation point data and the source point data on the current image frame at the back end so as to obtain a local optimization result;
the global optimization module is used for carrying out global optimization on all key frames in the key frame database in response to the detection that a loop exists or a new key frame is stored in the key frame database, and feeding back the result of the global optimization to the sliding window at the rear end to correct the result of the local optimization; and
a redundant keyframe culling system, wherein the redundant keyframe culling system comprises:
the candidate frame screening module is used for responding to the running of global optimization, and screening candidate redundant key frames from the key frame database by checking the condition that observation points and source points on key frames in the key frame database are tracked by other key frames which are in common view relationship in the key frame database;
A redundant frame determining module, configured to, when a key frame having a visual constraint with any image frame in the sliding window exists in the candidate redundant key frames, reject a valid key frame from the candidate redundant key frames, so as to use remaining key frames in the candidate redundant key frames as determined redundant key frames, where the valid key frame is a key frame having a valid visual constraint with any image frame in the sliding window in the candidate redundant key frames; the effective key frames comprise the key frame nearest to the oldest image frame in the sliding window, the key frame with a common view relation with the associated key frame and the key frame with a common view relation with other image frames in the sliding window; and
a redundant frame deleting module for deleting the determined redundant key frame from the key frame database to update the data in the key frame database;
the candidate frame screening module comprises a traversing module, a selecting module, a retrieving module, a statistics module and a judging module which are mutually connected in a communication mode, wherein the traversing module is used for traversing all key frames in the key frame database so as to take the currently traversed key frame as a current key frame; the selecting module is used for orderly selecting a viewing key frame and a reference key frame which have a common view relation with the current key frame from the key frame database;
The retrieval module is used for retrieving three-dimensional space points observed by the reference key frame from the observation points and the source points in the inspection key frame respectively so as to obtain the tracking times of each observation point and each source point in the inspection key frame;
the statistics module is used for respectively counting the number of the observation points with the tracking times larger than a first preset time threshold and the number of the source points with the tracking times larger than a second preset time threshold on the inspection key frame so as to calculate the repeated visual observation number duty ratio and the effective visual observation number duty ratio;
wherein the determination module is to determine the inspection keyframe to be the candidate redundant keyframe in response to the number of repeated visual observations being greater than a predetermined repetition duty cycle threshold and the number of valid visual observations being less than a predetermined valid duty cycle threshold.
14. An electronic device, comprising:
at least one processor for executing instructions; and
a memory communicatively coupled to the at least one processor, wherein the memory has at least one instruction, wherein the instruction is executed by the at least one processor to cause the at least one processor to perform all of the steps in a SLAM method, wherein the SLAM method comprises the steps of:
Acquiring successive image frames acquired via an image sensor;
the method comprises the steps of carrying out feature point tracking and extraction on a current image frame in the continuous image frames at the front end to obtain observation point data and source point data on the current image frame;
sliding window optimization is carried out on the current image frame and the observation point data and the source point data on the current image frame at the rear end so as to obtain a local optimization result;
in response to detecting that a loop exists or a new key frame is stored in a key frame database, performing global optimization on all key frames in the key frame database, and feeding back a global optimization result to a sliding window at the back end to correct the local optimization result;
responding to the running of global optimization, and screening candidate redundant key frames from the key frame database by checking the condition that observation points and source points on key frames in the key frame database are tracked by other key frames which are in common view relationship in the key frame database;
when a key frame with visual constraint with any image frame in the sliding window exists in the candidate redundant key frames, eliminating effective key frames from the candidate redundant key frames, and taking the rest key frames in the candidate redundant key frames as determined redundant key frames, wherein the effective key frames are key frames with effective visual constraint with any image frame in the sliding window in the candidate redundant key frames; the effective key frames comprise the key frame nearest to the oldest image frame in the sliding window, the key frame with a common view relation with the associated key frame and the key frame with a common view relation with other image frames in the sliding window; and
Deleting the determined redundant key frame from the key frame database to update the data in the key frame database;
wherein, in response to the running of the global optimization, the step of screening candidate redundant key frames from the key frame database by checking the condition that the observation point and the source point on the key frame in the key frame database are tracked by other key frames which are in the same common view relationship in the key frame database comprises the steps of:
traversing all the key frames in the key frame database to take the currently traversed key frame as a current key frame;
orderly selecting a viewing key frame and a reference key frame which have a common view relation with the current key frame from the key frame database;
the three-dimensional space points observed by the reference key frame are respectively retrieved from the observation points and the source points in the inspection key frame so as to obtain the tracking times of each observation point and each source point in the inspection key frame;
counting the number of the observation points with the tracking times larger than a first preset time threshold and the number of the source points with the tracking times larger than a second preset time threshold on the detection key frame respectively to obtain a repeated visual observation number proportion and an effective visual observation number proportion; and
And in response to the number of repeated visual observations being greater than a predetermined repetition duty cycle threshold and the number of valid visual observations being less than a predetermined valid duty cycle threshold, determining the inspection keyframe as the candidate redundant keyframe.
CN202010022951.4A 2020-01-09 2020-01-09 Redundant key frame eliminating method, SLAM method, system and electronic equipment Active CN113094545B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010022951.4A CN113094545B (en) 2020-01-09 2020-01-09 Redundant key frame eliminating method, SLAM method, system and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010022951.4A CN113094545B (en) 2020-01-09 2020-01-09 Redundant key frame eliminating method, SLAM method, system and electronic equipment

Publications (2)

Publication Number Publication Date
CN113094545A CN113094545A (en) 2021-07-09
CN113094545B true CN113094545B (en) 2023-05-16

Family

ID=76664066

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010022951.4A Active CN113094545B (en) 2020-01-09 2020-01-09 Redundant key frame eliminating method, SLAM method, system and electronic equipment

Country Status (1)

Country Link
CN (1) CN113094545B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113447014A (en) * 2021-08-30 2021-09-28 深圳市大道智创科技有限公司 Indoor mobile robot, mapping method, positioning method, and mapping positioning device
CN115376051B (en) * 2022-10-25 2023-03-24 杭州华橙软件技术有限公司 Key frame management method and device, SLAM method and electronic equipment

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107610175A (en) * 2017-08-04 2018-01-19 华南理工大学 The monocular vision SLAM algorithms optimized based on semi-direct method and sliding window
CN108592919A (en) * 2018-04-27 2018-09-28 百度在线网络技术（北京）有限公司 The drawing of opposite edges and localization method, device, storage medium and terminal device
WO2019169540A1 (en) * 2018-03-06 2019-09-12 斯坦德机器人（深圳）有限公司 Method for tightly-coupling visual slam, terminal and computer readable storage medium
CN110322511A (en) * 2019-06-28 2019-10-11 华中科技大学 A kind of semantic SLAM method and system based on object and plane characteristic

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9709404B2 (en) * 2015-04-17 2017-07-18 Regents Of The University Of Minnesota Iterative Kalman Smoother for robust 3D localization for vision-aided inertial navigation
US10636198B2 (en) * 2017-12-28 2020-04-28 Beijing Jingdong Shangke Information Technology Co., Ltd. System and method for monocular simultaneous localization and mapping

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107610175A (en) * 2017-08-04 2018-01-19 华南理工大学 The monocular vision SLAM algorithms optimized based on semi-direct method and sliding window
WO2019169540A1 (en) * 2018-03-06 2019-09-12 斯坦德机器人（深圳）有限公司 Method for tightly-coupling visual slam, terminal and computer readable storage medium
CN108592919A (en) * 2018-04-27 2018-09-28 百度在线网络技术（北京）有限公司 The drawing of opposite edges and localization method, device, storage medium and terminal device
CN110322511A (en) * 2019-06-28 2019-10-11 华中科技大学 A kind of semantic SLAM method and system based on object and plane characteristic

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
基于视觉电力巡检机器人导航系统设计与实现;李彬彬;《中国优秀博硕士学位论文全文数据库(硕士) 工程科技II辑》;20180815;文章第20、32-34页 *

Also Published As

Publication number Publication date
CN113094545A (en) 2021-07-09

Similar Documents

Publication Publication Date Title
CN110858394B (en) Image quality evaluation method and device, electronic equipment and computer readable storage medium
US20010031005A1 (en) Frame decimation for structure from motion
JP6474854B2 (en) Method and apparatus for updating a background model
CN110659658B (en) Target detection method and device
CN113094545B (en) Redundant key frame eliminating method, SLAM method, system and electronic equipment
US11871125B2 (en) Method of processing a series of events received asynchronously from an array of pixels of an event-based light sensor
CN107451156B (en) Image re-identification method and identification device
CN111627079B (en) Instantaneous single event reconstruction using convolutional neural networks
CN105243395A (en) Human body image comparison method and device
US20210056307A1 (en) Object tracking across multiple images
CN112508803A (en) Denoising method and device for three-dimensional point cloud data and storage medium
CN113011231B (en) Classification sliding window method, SLAM positioning method, system and electronic equipment
CN115049731A (en) Visual mapping and positioning method based on binocular camera
CN108289177B (en) Information interaction method, device and system
CN109389624A (en) Model drift rejection method and its device based on measuring similarity
US20220122341A1 (en) Target detection method and apparatus, electronic device, and computer storage medium
CN115588150A (en) Pet dog video target detection method and system based on improved YOLOv5-L
CN113096024B (en) Flying spot removing method for depth data, system and electronic equipment thereof
KR102589150B1 (en) Long-distance object detection system using cumulative difference image
CN113012216B (en) Feature classification optimization method, SLAM positioning method, system and electronic equipment
CN110349183B (en) Tracking method and device based on KCF, electronic equipment and storage medium
JP7584425B2 (en) Method for processing a sequence of events received asynchronously from a pixel array of an event-based light sensor - Patent 7732637
CN110909730A (en) Imaging system for movement detection
CN116664372A (en) Open field preference area determining method, device, equipment and storage medium
CN112115285A (en) Picture cleaning method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
EE01 Entry into force of recordation of patent licensing contract
EE01 Entry into force of recordation of patent licensing contract

Application publication date: 20210709

Assignee: Zhejiang Shunwei Technology Co.,Ltd.

Assignor: SUNNY OPTICAL (ZHEJIANG) RESEARCH INSTITUTE Co.,Ltd.

Contract record no.: X2024330000055

Denomination of invention: Redundant keyframe removal method and SLAM method, as well as their systems and electronic devices

Granted publication date: 20230516

License type: Common License

Record date: 20240515