Nothing Special   »   [go: up one dir, main page]

WO2021088497A1 - Virtual object display method, global map update method, and device - Google Patents

Virtual object display method, global map update method, and device Download PDF

Info

Publication number
WO2021088497A1
WO2021088497A1 PCT/CN2020/113321 CN2020113321W WO2021088497A1 WO 2021088497 A1 WO2021088497 A1 WO 2021088497A1 CN 2020113321 W CN2020113321 W CN 2020113321W WO 2021088497 A1 WO2021088497 A1 WO 2021088497A1
Authority
WO
WIPO (PCT)
Prior art keywords
map
coordinate system
global
electronic device
slam
Prior art date
Application number
PCT/CN2020/113321
Other languages
French (fr)
Chinese (zh)
Inventor
姜军
林涛
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2021088497A1 publication Critical patent/WO2021088497A1/en

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/05Geographic models
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00Manipulating 3D models or images for computer graphics
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00Manipulating 3D models or images for computer graphics
    • G06T19/006Mixed reality
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00Indexing scheme for image data processing or generation, in general
    • G06T2200/08Indexing scheme for image data processing or generation, in general involving all processing steps from image acquisition to 3D model generation

Definitions

  • This application relates to the technical field of virtual scenes, in particular to virtual object display methods, global map updating methods, electronic equipment and servers.
  • VR Virtual Reality
  • AR Augmented Reality
  • MR Mixed Reality
  • VR technology is a simulation technology that can create and experience a virtual world
  • AR technology is a technology that can superimpose and interact between virtual reality and the real world
  • MR technology is a new visualization environment generated by merging the real world and virtual world, and a comprehensive technology that builds an interactive feedback information loop between the real world, virtual world and users.
  • SLAM Simultaneous Localization and Mapping
  • SLAM technology can specifically realize that when electronic devices (such as mobile phones, VR glasses and other mobile electronic devices) start to move from an unknown location in the environment, they can locate themselves according to location estimates and maps during the movement, and at the same time build on the basis of their own positioning. Incremental map to facilitate subsequent positioning.
  • a system or module using SLAM technology can also be called a spatial positioning engine.
  • the global map is the carrier of virtual objects and the bridge between users. After the local coordinate system of the electronic device is synchronized with the global coordinate system of the global map, the virtual objects in the global coordinate system can be displayed on the electronic device.
  • the global map created by the global map generation company often has limited coverage, and the urban architectural landscape often changes in the real environment, the environmental landscape changes with the seasons, and the environmental visibility changes with the illumination, and so on. These factors will cause the map content in the global map to be different from the real environment, which will lead to inaccurate or even inaccurate global positioning of electronic devices. Therefore, how to effectively update the global map is a technical problem that needs to be solved urgently by those skilled in the art.
  • the embodiments of the present application provide a virtual object display method, a global map update method, and a device, which can implement efficient global map update and improve user experience.
  • the embodiments of the present application provide a virtual object display method, which is applied to a first electronic device having a display component (such as a display screen (for example, a touch screen, a flexible screen, a curved screen, etc., or an optical component)) and a camera.
  • a display component such as a display screen (for example, a touch screen, a flexible screen, a curved screen, etc., or an optical component)
  • the first electronic device can be a handheld terminal (such as a mobile phone), VR or AR glasses, a drone, an unmanned vehicle, etc.
  • the method includes: detecting a user's operation to open the application; in response to the operation, downloading the first
  • the global sub-map is stored in the real-time positioning and mapping (SLAM) system of the first electronic device; the first global sub-map is a sub-map associated with the location of the first electronic device in the global map;
  • the display component displays a virtual object based on the pose data of the first electronic device (or displays the position and posture of the virtual object on the display component), and the pose data of the first electronic device It is obtained by the SLAM system performing pose calculation at least according to the video image collected by the camera and the first global sub-map, wherein the global map on the server is based on the map sent by one or more second electronic devices
  • Update information is obtained by updating, wherein the one or more second electronic devices are electronic devices other than the first electronic device, and the second electronic device may also be a handheld terminal (such as a mobile phone), VR or AR, for example. Glasses, drones, unmanned vehicles, etc.
  • the so-called “operation to open the application” can be to open the application by clicking, touching, sliding, or shaking, or by voice control or other means to open the application, which is not limited in this application.
  • the navigation function in the application is activated, the camera is activated, and so on.
  • the server can be used as a platform for providing content and information support to the VR or AR or MR applications of the first electronic device and the second electronic device.
  • a global map is stored in the server.
  • the global map is a high-precision map with a large geographic area.
  • the so-called "large geographic area” is relative to the geographic area represented by the SLAM map in the electronic device.
  • the global map may be obtained by fusing map update information of multiple SLAM maps generated by one or more second electronic devices according to certain rules.
  • the global sub-map refers to the sub-map associated with the location of the first electronic device in the global map (also referred to as the sub-map associated with the location of the first electronic device), that is, the actual location of the first electronic device can be
  • the location point of the location in the global map is the starting point, and the map content in the preset area around the starting point is acquired as the global sub-map.
  • this application is not limited to user triggering. It can also be that the first electronic device automatically detects whether it meets the conditions for downloading the global sub-map (or starting the download of the global sub-map), that is, in another possible implementation of this application, the first An electronic device may also initiate the step of downloading the global sub-map in other ways, for example, by detecting changes in ambient light or detecting changes in the environment to initiate the step of downloading the global sub-map.
  • the first electronic device can be installed with virtual scene applications such as VR or AR or MR applications, and can run the VR or AR or MR applications based on user operations (such as clicking, touching, sliding, shaking, voice control, etc.) .
  • the first electronic device can collect video images in the environment through a local camera, combine the collected video images and the downloaded global submap to determine the current pose of the electronic device, and then display it on the display component based on the current pose of the first electronic device The position and posture of the virtual object.
  • the virtual object may correspondingly be a virtual object in a VR scene, an AR scene, or an MR scene (that is, an object in a virtual environment).
  • the map update information is used to indicate the difference between the SLAM map in the second coordinate system or the first coordinate system with respect to the global sub-map in the second coordinate system.
  • the first coordinate system is the coordinate system of the SLAM map of the SLAM system
  • the second coordinate system is the coordinate system of the global sub-map
  • the first or second here is only for the convenience of distinction.
  • the computing resources of each second electronic device are fully utilized to calculate and obtain map update information, thereby reducing the computing load of the server.
  • the map update information has less data volume, which saves transmission bandwidth resources.
  • the server can quickly implement global map update based on the map update information of one or more second electronic devices, and update efficiency high. Since the global map in the server can be updated efficiently and accurately, it also promotes the global positioning of the first electronic device to have higher accuracy and better robustness.
  • the first electronic device detects the user's operation to open the application, it requests the server to download the global sub-map, and uses the global sub-map with higher accuracy than the SLAM map as the visual observation input to the SLAM system.
  • the SLAM system uses the global sub-map and the collected video
  • the image is used to estimate the pose of the electronic device to ensure that the display position and direction of the virtual object on the electronic device will not shift during the long-term (for example, more than 1 minute) operation of the VR or AR or MR application, and the virtual object will be displayed correctly for a long time (For example, under the environment and duration represented by the video image, the virtual objects are displayed in a relatively accurate manner), thereby enhancing the user experience.
  • the pose data of the first electronic device may be the pose data of the first electronic device in a first coordinate system, or the pose data of the first electronic device in a second coordinate system;
  • the first coordinate system is the coordinate system of the SLAM map of the SLAM system
  • the second coordinate system is the coordinate system of the global sub-map, and the first or second here is only for the convenience of distinction.
  • the coordinate system used to construct the SLAM map can be called the first coordinate system.
  • the first coordinate system in this article may also be called the local coordinate system, the SLAM coordinate system, the camera coordinate system or others in some application scenarios. Some suitable term.
  • the pose (pose data) embodied by the electronic device in the local coordinate system can be called a local pose.
  • the coordinate system used to construct the global map may be called the second coordinate system.
  • the second coordinate system may also be called the global coordinate system, the world coordinate system, or some other appropriate terminology.
  • the pose (pose data) embodied by the electronic device in the global coordinate system can be referred to as a global pose.
  • the map update information is a comparison between the SLAM system of the second electronic device according to the SLAM map in the second coordinate system and the global sub-map in the second coordinate system
  • the result is obtained; wherein the SLAM map in the second coordinate system is based on the SLAM map in the first coordinate system constructed by the SLAM system of the electronic device, and the first coordinate system and the second coordinate system
  • the comparison result reflects the difference between the SLAM map in the second coordinate system and the global sub-map in the second coordinate system, and the SLAM system can further Obtain the map update information.
  • the map update information is a comparison result of the SLAM system of the second electronic device according to the SLAM map in the first coordinate system and the global sub-map in the second coordinate system, and Obtained from the coordinate system transformation information, the comparison result reflects the difference between the SLAM map in the first coordinate system and the global sub-map in the second coordinate system, and the SLAM system can convert the difference to The second coordinate system continues to indicate that the map update information can be further obtained according to the difference in the second coordinate system.
  • the SLAM system can use key frames to construct a SLAM map.
  • the SLAM map can be understood as a map drawn by the SLAM system according to the environmental information collected by the collection device.
  • the collection device can include, for example, an image collection device in an electronic device (such as a camera or Camera) and Inertial Measurement Unit (IMU).
  • the IMU may include sensors such as gyroscopes and accelerometers.
  • the SLAM map can contain the following map content: multiple key frames, triangulated feature points, and the association between key frames and feature points.
  • the key frame may be formed based on the image collected by the camera and the camera parameters used to generate the image (for example, the pose of the electronic device in the SLAM coordinate system).
  • a feature point ie, a feature as used herein is an interesting or significant part of an image.
  • the feature points may represent different 3D map points along the three-dimensional space in the SLAM map and the feature descriptions on the 3D map points.
  • Each feature point can have an associated feature location.
  • Each feature point can represent a 3D coordinate position and is associated with one or more descriptors.
  • Feature points can also be called 3D features, feature points, 3D feature points or other suitable names.
  • the image quality of the new key frame itself should be good, for example, it cannot be a very blurry image, the number of feature points should be sufficient, and the distribution of feature points should be as uniform as possible; on the other hand, the new key frame needs to be
  • the existing key frames in the SLAM map have a small number of common-view relationships, and most of the feature points are new feature points to achieve the effect of both existing constraints and as little information redundancy as possible, such as a neighboring key frame in the SLAM map In the new key frame, 10% of the feature points can be observed, and the remaining 90% cannot be observed, and so on.
  • 3D map points represent the coordinates on the three-dimensional space axes X, Y, and Z.
  • the 3D map points are expressed in local coordinates It is the coordinates on the three-dimensional space axis X, Y, and Z.
  • the 3D map points represent the coordinates on the three-dimensional axis X, Y, and Z of the global coordinate system.
  • the concept of 3D map points may include the vision in the two-dimensional image. Characteristic information.
  • the map update information includes: pose data corresponding to the target key frame of the SLAM map constructed by the SLAM system of the second electronic device, and three-dimensional (3D) map points
  • the common-view data is used to indicate the common-view relationship between the target key frame and the key frame of the second global sub-map in the second electronic device.
  • the second global sub-map is a sub-map associated with the location of the second electronic device in the global map.
  • the target key frame is a non-redundant key frame
  • the non-redundant key frame represents a key frame that exists in the SLAM map of the second electronic device but does not exist in the second global sub-map .
  • the map update information may be related information after the target key frame is converted to the global coordinate system and then merged into the global sub-map (ie, the pose, common view relationship, 3D map point, etc. information corresponding to the target key frame). That is, the map update information is the updated part of the global sub-map compared to the original global sub-map.
  • the map update information only needs to include the changed key frame pose, common view relationship, and 3D map points. Compared with the previously downloaded global sub-map, the amount of data will be much smaller, so the transmission bandwidth is further saved. Resources.
  • the server can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the update efficiency and robustness of the global map are improved.
  • the map update information includes: key frames existing in the SLAM map of the second electronic device but not in the second global submap.
  • the second global sub-map is a sub-map associated with the location of the second electronic device in the global map.
  • the map update information can be directly non-redundant key frames.
  • the second electronic device recognizes the non-redundant key frames of the SLAM map, it sends the non-redundant key frames to the server, and the server reuses the non-redundant key frames.
  • the redundant key frames are merged into the global map, so that the global map in the server can be updated efficiently.
  • the downloading the first global submap from the server in response to the operation includes: sending to the server a first position indicating the initial position of the first electronic device Fingerprint information; receiving the first global submap from the server, the first global submap corresponding to second location fingerprint information, and the first location fingerprint information matches the second location fingerprint information.
  • Performing the map matching operation through the server improves the matching efficiency and accuracy, which in turn helps to reduce the delay in downloading the map.
  • the embodiments of the present application provide a global map update method, which is applied to electronic devices, which can be handheld terminals (such as mobile phones), VR or AR glasses, drones, unmanned vehicles, etc.
  • the method includes: obtaining a global sub-map corresponding to the location of the electronic device from a server, the global sub-map being a sub-map associated with the location of the electronic device in a global map in a second coordinate system;
  • the comparison result of the SLAM map and the global sub-map in the second coordinate system and the coordinate system transformation information obtain map update information, wherein the SLAM map in the second coordinate system is based on the first coordinate system
  • the SLAM map and the coordinate system transformation information between the first coordinate system and the second coordinate system are obtained, and the map update information is used to indicate the
  • map update information is used to indicate the second coordinate system.
  • the comparison result reflects the difference between the SLAM map in the second coordinate system and the global sub-map in the second coordinate system, and the SLAM system can further obtain the map update information according to the difference.
  • the map update information is used To indicate the difference between the SLAM map in the first coordinate system and the global sub-map in the second coordinate system.
  • the comparison result reflects the difference between the SLAM map in the first coordinate system and the global sub-map in the second coordinate system.
  • the “difference” here is the difference expressed in the first coordinate system, and the SLAM system can further convert the difference to the information based on the coordinate system transformation information between the first coordinate system and the second coordinate system.
  • the second coordinate system is further represented, and the map update information can be further obtained according to the difference in the second coordinate system. Therefore, the map update information can be used to "indicate the difference”.
  • the server can be used as a platform for providing content and information support to VR or AR or MR applications of electronic devices.
  • a global map is stored in the server.
  • the global map is a high-precision map with a large geographic area.
  • the so-called "large geographic area” is relative to the geographic area represented by the SLAM map in the electronic device.
  • the global map may be obtained by integrating multiple SLAM maps generated by one or more electronic devices according to certain rules.
  • the global sub-map refers to the sub-map associated with the location of the electronic device in the global map, that is to say, the actual location of the electronic device can be taken as the starting point to obtain the map in the preset area around the starting point.
  • the content serves as the global sub-map.
  • the electronic device downloads the global sub-map in the global map from the server, compares the global sub-map with the SLAM map in the electronic device, and obtains map update information by using the computing resources and computing capabilities of the electronic device.
  • the information can be used to update the map content in the global submap. Therefore, after the electronic device sends the map update information to the server, the server can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the efficiency of the server to update the global map is improved.
  • the map update information has a smaller amount of data, so the transmission bandwidth resources are also saved.
  • the SLAM map in the second coordinate system is specifically obtained by transforming the SLAM map in the first coordinate system based on the coordinate system transformation information;
  • Obtaining map update information according to the comparison result of the SLAM map in the second coordinate system constructed by the SLAM system of the electronic device and the global sub-map in the second coordinate system includes: comparing the second coordinate system Under the SLAM map and the global sub-map under the second coordinate system to obtain non-redundant key frames, the non-redundant key frames indicate that they exist in the SLAM map under the second coordinate system but do not exist in the Key frames in the global sub-map in the second coordinate system; fusing the non-redundant key frames to the global sub-map in the second coordinate system to obtain the fused global sub-map; from the The map update information is obtained from the global sub-map after fusion.
  • the SLAM system can use key frames to construct a SLAM map.
  • the SLAM map can be understood as a map drawn by the SLAM system according to the environmental information collected by the collection device.
  • the SLAM map can contain the following map content: multiple key frames, triangulation Feature points and the association between key frames and feature points.
  • the key frame may be formed based on the image collected by the camera and the camera parameters used to generate the image (for example, the pose of the electronic device in the SLAM coordinate system).
  • a feature point ie, a feature
  • the feature points may represent different 3D map points along the three-dimensional space in the SLAM map and the feature descriptions on the 3D map points.
  • Each feature point can have an associated feature location.
  • Each feature point can represent a 3D coordinate position and is associated with one or more descriptors.
  • 3D map points represent the coordinates on the three-dimensional space axes X, Y, and Z.
  • the 3D map points represent the coordinates on the three-dimensional space axes X, Y, and Z of the local coordinate system. Coordinates on Y and Z.
  • the 3D map points represent the coordinates on the three-dimensional axis X, Y, and Z of the global coordinate system.
  • the concept of 3D map points may include the two-dimensional image Visual feature information.
  • the key frames and map points in the SLAM map can be transformed into a global coordinate system for representation, and the SLAM map can also be transformed into a global coordinate system for representation.
  • the SLAM map will be continuously updated.
  • the so-called "update of the SLAM map” is embodied in the continuous insertion of key frames into the SLAM map in the global coordinate system.
  • the so-called redundancy of key frames means that the 3D map point information associated with the key frames already exists in the global sub-map, that is, the global sub-map has saved the visual information in the current environment, and there is no need to add the key frame.
  • the key frame is not redundant, that is, the 3D map point information associated with the key frame does not exist in the global sub-map, and the global sub-map needs to add the key frame to realize the update, so that the map content can be synchronized with the real environment.
  • the key frames in the SLAM map in the global coordinate system that are not redundant with the key frames in the global sub-map may be referred to as non-redundant key frames. Subsequently, the non-redundant key frames can be merged into the global sub-map in the global coordinate system.
  • the obtained map update information may include the pose, common view relationship, and 3D map point corresponding to the non-redundant key frame.
  • Equal information that is, the map update information is the part of the updated global sub-map compared to the original global sub-map in the fused global sub-map. Therefore, the electronic device can obtain the map update information from the merged global sub-map and send it to the server.
  • the electronic device determines whether the key frame inserted by the SLAM system is the same as the download
  • the key frames in the global sub-map are redundant, that is, whether there are non-redundant key frames.
  • the non-redundant key frames indicate that the SLAM map in the local coordinate system does not have redundancy with the key frames in the global sub-map Keyframe. If it exists, the non-redundant key frame can be transformed into the global coordinate system for representation, and the non-redundant key frame in the global coordinate system can be merged into the global sub-map in the global coordinate system.
  • the obtained map update information may include the corresponding non-redundant key frames in the global coordinate system
  • the position and pose, common view relationship, 3D map points and other information, that is, the map update information is the part of the information that has been updated in the fused global sub-map compared to the original global sub-map. Therefore, the electronic device can obtain the map update information from the merged global sub-map and send it to the server.
  • the map update information can be obtained from the global sub-map after fusion, so the data volume will be much smaller than the previously downloaded global sub-map, so the transmission bandwidth resources are further saved.
  • the server can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the update efficiency and robustness of the global map are improved.
  • the fusing the non-redundant key frame into the global sub-map in the second coordinate system includes: using the non-redundant key frame as a new key The frame node is added to the global sub-map in the second coordinate system; the common view relationship between the non-redundant key frame and the key frame in the global sub-map in the second coordinate system is updated; the non-redundant key frame is calculated; The pose data and the three-dimensional map points corresponding to the redundant key frames; correspondingly, the obtaining the map update information from the fused global sub-map includes: from the fused global sub-map Obtain the pose data, three-dimensional map points, and common view data corresponding to the non-redundant key frame as the map update information (that is, the map update information is the fused global sub-map compared to the original global sub-map , The updated part of the information), the common view data is used to indicate the common view relationship between the non-redundant key frame and the key frame of the global submap
  • the electronic device can calculate which 3D map points in the global sub-map can be simultaneously observed by at least one key frame and non-redundant key frame in the global sub-map, thereby updating the key frames and non-redundant key frames in the global sub-map The common-view relationship between the keys.
  • the electronic device can use the triangulation algorithm to calculate new 3D map points, and use the BA algorithm to optimize the pose and 3D map point coordinates corresponding to the non-redundant key frames to complete
  • the global sub-map is updated, and the pose data, three-dimensional map points and common view data corresponding to the non-redundant key frames are used as the Map update information.
  • the map update information only needs to include the changed key frame pose, common view relationship, and 3D map points. Compared with the previously downloaded global sub-map, the amount of data will be much smaller, so the transmission bandwidth is further saved. Resources.
  • the server can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the update efficiency and robustness of the global map are improved.
  • the map update information is obtained according to the comparison result of the SLAM map in the first coordinate system and the global sub-map in the second coordinate system and the coordinate system transformation information ,include:
  • a non-redundant key frame is obtained, where the non-redundant key frame exists in the first coordinate system
  • the key frame of the SLAM map, and the non-redundant key frame is transformed by the coordinate system transformation information and does not exist in the global sub-map in the second coordinate system; the coordinate system transformation information is transformed
  • Non-redundant key frames are fused to the global sub-map in the second coordinate system to obtain the map update information.
  • the non-redundant key frame transformed by the coordinate system transformation information is the key frame in the second coordinate system.
  • the map update information is obtained
  • the map update information is used to indicate the difference between the SLAM map in the second coordinate system and the global sub-map in the second coordinate system.
  • the comparison result reflects the difference between the SLAM map in the second coordinate system and the global sub-map in the second coordinate system, and this difference may be non-redundant in the SLAM map in the second coordinate system Key frame, that is, the non-redundant key frame does not exist in the global submap.
  • the SLAM system can subsequently fuse the non-redundant key frames into the global sub-map, thereby obtaining map update information (including pose data, three-dimensional map points, and common view data corresponding to the non-redundant key frames) and sending it to the server.
  • map update information including pose data, three-dimensional map points, and common view data corresponding to the non-redundant key frames
  • the SLAM system can also directly send the non-redundant key frame as map update information to the server.
  • the map update information is used To indicate the difference between the SLAM map in the first coordinate system and the global sub-map in the second coordinate system.
  • the comparison result reflects the difference between the SLAM map in the first coordinate system and the global sub-map in the second coordinate system. This difference may be non-redundant key frames in the SLAM map in the first coordinate system.
  • the SLAM system can convert the non-redundant key frame to the second coordinate system to obtain the non-redundant key frame in the second coordinate system.
  • the SLAM system can merge the non-redundant key frames in the second coordinate system into the global sub-map to obtain map update information (including the pose data corresponding to the non-redundant key frames in the second coordinate system, and the three-dimensional map points And common view data) to the server.
  • map update information including the pose data corresponding to the non-redundant key frames in the second coordinate system, and the three-dimensional map points And common view data
  • the SLAM system may also directly send the non-redundant key frames in the second coordinate system as map update information to the server.
  • the obtaining the global submap corresponding to the location of the electronic device from the server includes: sending indication information of the initial location of the electronic device to the server; Receiving the global sub-map, the global sub-map being determined according to the initial position of the electronic device.
  • the indication information of the initial position of the electronic device may include first position fingerprint information for indicating the initial position of the electronic device; the global sub-map corresponds to the second position fingerprint information, and the first position The fingerprint information matches the second location fingerprint information.
  • the method further includes: determining that the electronic device is in the first coordinate system according to the collected video image and the SLAM map in the first coordinate system The first pose data in the SLAM map; according to the video image and the global sub-map in the second coordinate system, determine the second pose of the electronic device in the global sub-map in the second coordinate system Data; according to the first pose data and the second pose data, obtain the coordinate system transformation information between the first coordinate system of the SLAM map and the second coordinate system of the global map.
  • the video image in an embodiment of the present application may also be directly an image.
  • the video image in another embodiment of the present application may also represent the K-th frame image in the video image sequence collected by the camera of the electronic device, and K is an integer greater than or equal to 1.
  • the video image collected by the camera may be a video sequence (video stream) including multiple frames of images
  • the K-th frame image may be a certain frame in the video stream.
  • the embodiment of the application obtains the pose of the terminal in the local coordinate system and the pose in the global coordinate system respectively according to the same frame. Based on these two poses, the coordinate system transformation information between the two coordinate systems can be obtained (for example, Coordinate system transformation matrix). According to this coordinate system transformation matrix, the two coordinate systems can be synchronized. In this way, the information originally represented by the local coordinate system (such as local pose, image feature points, SLAM map 3D map points Etc.) Then it can be transformed to the global coordinate system based on the coordinate system transformation matrix. In this way, the SLAM map in the local coordinate system can be converted to the global coordinate system, or the non-redundant key frames in the local coordinate system can be converted to the global coordinate system.
  • the coordinate system transformation information between the two coordinate systems can be obtained (for example, Coordinate system transformation matrix).
  • the two coordinate systems can be synchronized.
  • the information originally represented by the local coordinate system such as local pose, image feature points, SLAM map 3D map points Etc.
  • the first position of the electronic device in the SLAM map in the first coordinate system is determined according to the collected video image and the SLAM map in the first coordinate system.
  • Pose data including:
  • the movement data includes movement speed data and movement direction data.
  • the electronic device is provided with an IMU
  • the input signal of the SLAM system includes the video image collected by the camera, the motion data collected by the IMU, and the SLAM map in the local coordinate system.
  • the IMU detects the angular velocity and linear acceleration of the electronic device at a high frequency, and integrates the angular acceleration and linear acceleration separately, and then can calculate the posture of the electronic device.
  • the video image collected by the camera is matched in the SLAM map in the local coordinate system, so that the position and posture of the electronic device can also be calculated. Then, based on the calculation of these two poses with a certain algorithm, the first pose data can be obtained.
  • the determining the first position of the electronic device in the global sub-map in the second coordinate system according to the video image and the global sub-map in the second coordinate system Two pose data, including:
  • the image feature and the map feature are calculated to obtain the second pose data in the global sub-map of the electronic device in the second coordinate system.
  • the electronic device performs feature detection on the K-th frame image, and extracts the image position of the feature in the K-th frame image.
  • the feature detection algorithm is not limited to feature detection methods such as FAST, ORB, SIFT, SURF, D2Net, and SuperPoint. Then perform feature description for each detected feature.
  • the feature description algorithm is not limited to feature description methods such as ORB, SIFT, SURF, BRIEF, BRISK, FREAK, D2Net, SuperPoint, etc., to form a one-dimensional vector for subsequent feature matching .
  • the electronic device can match the most similar map content (such as one or more key frames) to the K-th frame image from the global submap.
  • Specific methods include, for example, traditional image retrieval methods such as BOW and VLAD, and New image retrieval methods of NetVLAD and AI.
  • BOW and VLAD traditional image retrieval methods
  • NetVLAD and AI New image retrieval methods of NetVLAD and AI.
  • PnP, EPnP, 3D-3D and other registration algorithms can be used to calculate Output the second pose data.
  • the embodiments of the present application can be implemented on the side of the electronic device, making full use of the computing power of the electronic device to calculate the first pose data and the second pose data, which improves the processing efficiency and can also reduce the calculation burden of the server.
  • the determining the first position of the electronic device in the global sub-map in the second coordinate system according to the video image and the global sub-map in the second coordinate system Two pose data, including: sending the video image to the server; receiving the second pose data from the server, the second pose data being the server according to the video image and the first
  • the global sub-map associated with the location of the electronic device in the global map in the two-coordinate system is determined by feature extraction and feature matching.
  • the K-th frame image when the video image is the K-th frame image in the video image sequence collected by the camera of the electronic device, the K-th frame image may be the first frame image in the video image sequence captured by the camera.
  • the electronic device since the electronic device needs to download the global submap of the corresponding area first.
  • the process of downloading the map takes a certain amount of time.
  • the first global pose estimation can be done on the server side. In other words, after the application is started, the first global pose estimation is performed on the server side. While starting the global pose estimation, the server obtains the global sub-map and transmits it to the electronic device accordingly, which improves the user's access speed to the application. The user does not perceive the delay of the map downloading process. In this way, the user waiting caused by the download delay can be avoided, and the user experience is further improved.
  • an embodiment of the present application provides a method for updating a global map, which is applied to a server, and the method includes:
  • the global sub-map corresponding to the location of the electronic device is sent to the electronic device, and the global sub-map is the sub-map associated with the location of the electronic device in the global map in the second coordinate system.
  • Map receiving map update information from the electronic device; the map update information is used to indicate that the SLAM map in the first coordinate system or in the second coordinate system constructed by the SLAM system of the electronic device is relative to the The difference of the global sub-map in the second coordinate system updates the global map according to the map update information.
  • the map update information is used to indicate the difference between the SLAM map in the second coordinate system constructed by the SLAM system of the electronic device and the global sub-map in the second coordinate system .
  • the map update information is obtained by the SLAM system of the electronic device according to the comparison result of the SLAM map in the second coordinate system and the global sub-map in the second coordinate system; wherein, the second coordinate The SLAM map under the system is obtained based on the SLAM map in the first coordinate system constructed by the SLAM system and the coordinate system transformation information between the first coordinate system and the second coordinate system; the comparison result reflects The difference between the SLAM map in the second coordinate system and the global sub-map in the second coordinate system, the SLAM system can further obtain the map update information according to the difference, and send the map update information To the server.
  • the map update information is used to indicate how the SLAM map in the first coordinate system constructed by the SLAM system of the electronic device is relative to the global sub-map in the second coordinate system difference.
  • the map update information is the comparison result of the SLAM system of the second electronic device according to the comparison result of the SLAM map in the first coordinate system and the global sub-map in the second coordinate system and the coordinate system transformation Information obtained, the comparison result reflects the difference between the SLAM map in the first coordinate system and the global sub-map in the second coordinate system, and the SLAM system can convert the difference to the second coordinate system It means that the map update information can be further obtained according to the difference in the second coordinate system, and the map update information can be sent to the server.
  • the server sends the global sub-map in the global map to the electronic device, and the electronic device obtains map update information by comparing the global sub-map with the SLAM map in the electronic device and using the computing resources and computing capabilities of the electronic device.
  • the map update information can be used to update the map content in the global submap. Therefore, after the server receives the map update information sent by the electronic device, it can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the efficiency of the server to update the global map is improved. Compared with the image data captured by the camera, the map update information has a smaller amount of data, so the transmission bandwidth resources are also saved.
  • the map update information represents information associated with a non-redundant key frame;
  • the non-redundant key frame may represent a SLAM map existing in the second coordinate system Keyframes in the global submap that do not exist in the second coordinate system.
  • the updating the global map according to the map update information includes: adding the map update information to the global map.
  • the map update information specifically includes: pose data, three-dimensional map points, and common view data corresponding to the non-redundant key frame, and the common view data is used to indicate the non-redundant key frame and the second coordinate The common view relationship between the key frames of the global sub-map under the system.
  • the map update information represents related information obtained after the non-redundant key is fused into the global sub-map (for example, information such as the pose, common view relationship, 3D map point, etc.) corresponding to the non-redundant key. That is, the map update information is the updated part of the global sub-map compared to the original global sub-map.
  • the server After the server receives the map update information, it can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the update efficiency and robustness of the global map are improved.
  • the updating the global map according to the map update information includes: adding the map update information to the global map. Since the map update information is obtained from the integrated global sub-map, and the global sub-map is obtained from the server's global map, the server can also directly add the map update information to the server's global map. In, the update efficiency of the global map is greatly improved.
  • the map update information includes non-redundant key frames
  • the update of the global map according to the map update information includes: A key frame is added to the global map as a new key frame node; the common view relationship between the non-redundant key frame and the key frame in the global map is updated; the pose corresponding to the non-redundant key frame is calculated Data and 3D map points.
  • the server can calculate which 3D map points in the global map can be simultaneously observed by at least one key frame in the global map and the non-redundant key frame, thereby updating one of the key frames and the non-redundant key in the global map.
  • the server can use the triangulation algorithm to calculate new 3D map points, and optimize the pose and 3D map point coordinates corresponding to the non-redundant key frames through the BA algorithm, thus completing the transformation
  • the non-redundant key frames are merged (or inserted) into the global map to realize the update of the global map.
  • the electronic device identifies the non-redundant key frame by comparing the SLAM map and the global sub-map, and sends the non-redundant key frame to the server, and the server uses the non-redundant key frame to merge into the global map, thereby It can also realize the efficient update of the global map in the server.
  • Such a solution is beneficial to reduce the computational load of electronic equipment and ensure the accuracy of map fusion.
  • the sending a global sub-map corresponding to the location of the electronic device to the electronic device according to the request of the electronic device includes: receiving the initial location of the electronic device Indication information; acquiring the global sub-map corresponding to the location of the electronic device in the global map according to the initial position of the electronic device; sending the global sub-map to the electronic device.
  • the indication information of the initial location of the electronic device may include first location fingerprint information for indicating the initial location of the electronic device; the global sub-map in the global map corresponds to the second location fingerprint information; According to the initial position of the electronic device, acquiring the global sub-map corresponding to the position of the electronic device in the global map includes: according to the first location fingerprint information, in the global map Finding the global sub-map with second location fingerprint information matching the first location fingerprint information. Performing the map matching operation through the server improves the matching efficiency and accuracy, which in turn helps to reduce the delay in downloading the map.
  • an embodiment of the present application provides an electronic device for displaying virtual objects, including: one or more processors, memories, one or more application programs, and one or more computer programs; the one or more A plurality of computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to execute the first aspect or any one of the first aspects above
  • an embodiment of the present application provides an electronic device for updating a global map, including: one or more processors, a memory, and one or more computer programs; the one or more computer programs are stored In the memory, the one or more computer programs include instructions, and when the instructions are executed by the electronic device, the electronic device executes the second aspect or any one of the possible implementations described in the second aspect.
  • the virtual object display method including: one or more processors, a memory, and one or more computer programs; the one or more computer programs are stored In the memory, the one or more computer programs include instructions, and when the instructions are executed by the electronic device, the electronic device executes the second aspect or any one of the possible implementations described in the second aspect.
  • an embodiment of the present application provides a server for global map update, including: one or more processors, memories, and one or more computer programs; the one or more computer programs are stored in In the memory, the one or more computer programs include instructions, and when the instructions are executed by the electronic device, the electronic device executes the foregoing third aspect or any one of the possible implementation manners of the third aspect.
  • Virtual object display method including: one or more processors, memories, and one or more computer programs; the one or more computer programs are stored in In the memory, the one or more computer programs include instructions, and when the instructions are executed by the electronic device, the electronic device executes the foregoing third aspect or any one of the possible implementation manners of the third aspect.
  • an embodiment of the present application provides a chip.
  • the chip includes a processor and a data interface.
  • the processor reads instructions stored in a memory through the data interface, and executes the first aspect or any of the first aspects.
  • a virtual object display method in a possible implementation manner.
  • the chip may further include a memory in which instructions are stored, and the processor is configured to execute instructions stored on the memory.
  • the processor is configured to execute the virtual object display method in the first aspect or any possible implementation of the first aspect.
  • an embodiment of the present application provides a chip.
  • the chip includes a processor and a data interface.
  • the processor reads instructions stored in a memory through the data interface, and executes the second aspect or any of the second aspect.
  • a global map update method in a possible implementation.
  • the chip may further include a memory in which instructions are stored, and the processor is configured to execute instructions stored on the memory.
  • the processor is configured to execute the global map update method in the second aspect or any possible implementation of the second aspect.
  • an embodiment of the present application provides a chip.
  • the chip includes a processor and a data interface.
  • the processor reads instructions stored in a memory through the data interface, and executes any of the third aspect or the third aspect.
  • a global map update method in a possible implementation.
  • the chip may further include a memory in which instructions are stored, and the processor is configured to execute instructions stored on the memory.
  • the processor is configured to execute the global map update method in the third aspect or any possible implementation manner of the third aspect.
  • an embodiment of the present application provides an electronic device for updating a global map, including:
  • a communication module configured to obtain a global sub-map corresponding to the location of the electronic device from a server, the global sub-map being a sub-map associated with the location of the electronic device in the global map in the second coordinate system;
  • the SLAM module is used to compare the results of the SLAM map in the second coordinate system constructed by the SLAM module with the global sub-map in the second coordinate system, or according to the first coordinate system constructed by the SLAM module
  • the map update information is used to indicate the second coordinate system or the first coordinate
  • the communication module is further configured to send the map update information to the server, and the map update information is used to update the global map in the second coordinate system in the server.
  • the SLAM module may be the SLAM system described in the embodiment of the present application, for example, it may be the SLAM system 12 described in the following embodiments of this document.
  • the SLAM map in the second coordinate system is specifically obtained by transforming the SLAM map in the first coordinate system based on the coordinate system transformation information;
  • the SLAM module is specifically configured to: compare the SLAM map in the second coordinate system with the global sub-map in the second coordinate system to obtain a non-redundant key frame, where the non-redundant key frame indicates that there is Key frames in the SLAM map in the second coordinate system but not in the global sub-map in the second coordinate system; fusing the non-redundant key frames to the global sub-map in the second coordinate system Map to obtain a fused global sub-map; obtain the map update information from the fused global sub-map.
  • the SLAM module is specifically configured to: add the non-redundant key frame as a new key frame node to the global submap in the second coordinate system; update The common view relationship between the non-redundant key frame and the key frame in the global sub-map in the second coordinate system; calculate the pose data and three-dimensional map points corresponding to the non-redundant key frame; Obtain the pose data, three-dimensional map points, and common view data corresponding to the non-redundant key frame from the fused global sub-map as the map update information, and the common view data is used to indicate the non-redundant key frame The common view relationship with the key frames of the global sub-map in the second coordinate system.
  • the SLAM module is specifically configured to: compare the SLAM map in the first coordinate system with the global sub-map in the second coordinate system to obtain non-redundant A key frame, the non-redundant key frame is a key frame of the SLAM map existing in the first coordinate system, and the non-redundant key frame does not exist in the first coordinate system after being transformed by the coordinate system transformation information.
  • the global sub-map in the two-coordinate system; the non-redundant key frames transformed by the coordinate system transformation information are fused to the global sub-map in the second coordinate system to obtain the map update information.
  • the non-redundant key frame transformed by the coordinate system transformation information is a key frame in the second coordinate system.
  • the communication module is specifically configured to: send indication information of the initial position of the electronic device to the server; receive the global sub-map from the server, and the global sub-map It is determined according to the initial position of the electronic device.
  • the indication information of the initial location of the electronic device includes first location fingerprint information for indicating the initial location of the electronic device; the global sub-map corresponds to the second location Fingerprint information, and the first location fingerprint information matches the second location fingerprint information.
  • the electronic device further includes a global positioning module and a coordinate system change matrix calculation module;
  • the SLAM module is specifically used to, according to the collected video image and the SLAM map in the first coordinate system , Determine the first pose data of the electronic device in the SLAM map in the first coordinate system;
  • the global positioning module is specifically configured to, according to the video image and the global sub-map in the second coordinate system, Determine the second pose data of the electronic device in the global submap in the second coordinate system;
  • the coordinate system change matrix calculation module is specifically configured to, according to the first pose data and the second The pose data is used to obtain the coordinate system transformation information between the first coordinate system of the SLAM map and the second coordinate system of the global map.
  • the SLAM module is specifically configured to: obtain the electronic device according to the video image, the SLAM map in the first coordinate system, and the motion data collected by the electronic device.
  • the global positioning module is specifically configured to perform feature extraction on the video image to obtain image features; Feature matching in the sub-map to obtain map features that match the image features; according to the image features and the map features, the electronic device in the global sub-map in the second coordinate system is calculated The second pose data.
  • the communication module is specifically configured to: send the video image to the server; receive the second pose data from the server, and the second pose data
  • the server performs feature extraction and feature matching determination based on the video image and the global submap associated with the location of the electronic device in the global map in the second coordinate system.
  • an embodiment of the present application provides a server for a global map, including:
  • the communication module is configured to send a global sub-map corresponding to the location of the electronic device to the electronic device according to the request of the electronic device, where the global sub-map is the global map in the second coordinate system and the electronic device The sub-map associated with the location;
  • the communication module is also used to receive map update information from the electronic device;
  • the map update information is used to indicate the first coordinate system constructed by the SLAM system of the electronic device or the second coordinate system The difference between the SLAM map and the global sub-map in the second coordinate system;
  • the processing module is configured to update the global map according to the map update information.
  • the difference information indicates information associated with a non-redundant key frame; the non-redundant key frame indicates that it exists in the SLAM map in the second coordinate system But there is no key frame in the global submap in the second coordinate system.
  • the map update information specifically includes: pose data, three-dimensional map points, and common view data corresponding to the non-redundant key frame, and the common view data is used to indicate The common view relationship between the non-redundant key frame and the key frame of the global submap in the second coordinate system.
  • the processing module is specifically configured to: add the map update information to the global map.
  • the communication module is specifically configured to: receive indication information of the initial position of the electronic device; and obtain all information from the global map according to the initial position of the electronic device.
  • the global sub-map corresponding to the location of the electronic device; sending the global sub-map to the electronic device.
  • the indication information of the initial position of the electronic device includes first position fingerprint information for indicating the initial position of the electronic device; the global sub-map in the global map Corresponding to the second location fingerprint information; the processing module is specifically configured to: according to the first location fingerprint information, find the second location fingerprint information matching the first location fingerprint information in the global map The global submap.
  • an embodiment of the present application provides a computer-readable storage medium that stores program code for device execution, and the program code includes the program code for executing the first aspect or any of the first aspect.
  • An instruction of the method in a possible implementation manner, or the program code includes an instruction for executing the method in any possible implementation manner of the second aspect or the third aspect.
  • an embodiment of the present invention provides a computer program product
  • the computer program product may be a software installation package
  • the computer program product includes program instructions, when the computer program product is executed by an electronic device, the electronic device
  • the processor executes the method in any possible implementation manner of the foregoing first aspect, second aspect, or third aspect.
  • the electronic device downloads the global sub-map in the global map from the server, and obtains map update information by comparing the global sub-map with the SLAM map in the electronic device and using the computing resources and computing capabilities of the electronic device.
  • the map update information can be used to update the map content in the global sub-map. Therefore, after the electronic device sends the map update information to the server, the server can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the efficiency of the server to update the global map is improved. Compared with the image data taken by the camera, the map update information has a smaller amount of data, so the transmission bandwidth resources are also saved.
  • FIG. 1 is a schematic diagram of an application architecture provided by an embodiment of the present application
  • FIG. 2 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • FIG. 3 is a schematic structural diagram of a server provided by an embodiment of the present application.
  • FIG. 4 is a system provided by an embodiment of the present application, and a schematic diagram of the structure of electronic devices and servers in the system;
  • FIG. 5 is another system provided by an embodiment of the present application, and a schematic diagram of the structure of electronic devices and servers in the system;
  • FIG. 6 is a schematic diagram of functional realization of a map fusion module provided by an embodiment of the present application.
  • FIG. 7 is a schematic flowchart of a method for displaying a virtual object provided by an embodiment of the present application.
  • FIG. 8 is a schematic diagram of a scenario implemented by using the method of the present application provided by an embodiment of the present application.
  • FIG. 9 is a schematic flowchart of a method for updating a global map provided by an embodiment of the present application.
  • FIG. 10 is a schematic flowchart of yet another global map update method provided by an embodiment of the present application.
  • FIG. 11 is a schematic diagram of a scene related to a coordinate system transformation matrix provided by an embodiment of the present application.
  • FIG. 12 is a schematic diagram of a scene describing the common view relationship between key frames according to an embodiment of the present application.
  • FIG. 13 is a schematic diagram of yet another scenario implemented by the method of this application provided by an embodiment of this application;
  • FIG. 14 is a schematic diagram of yet another scenario implemented by the method of the present application provided by an embodiment of the present application.
  • 15 is a schematic flowchart of another global map update method provided by an embodiment of the present application.
  • FIG. 16 is another system provided by an embodiment of the present application, and a schematic diagram of the structure of electronic devices and servers in the system;
  • FIG. 17 is a schematic flowchart of another method for updating a global map provided by an embodiment of the present application.
  • words such as “exemplary” or “for example” are used as examples, illustrations, or illustrations. Any embodiment or design solution described as “exemplary” or “for example” in the embodiments of the present application should not be construed as being more preferable or advantageous than other embodiments or design solutions. To be precise, words such as “exemplary” or “for example” are used to present related concepts in a specific manner.
  • the multiple electronic devices include a first electronic device, one or more second electronic devices (in Figure 1 there are two second electronic devices).
  • the one or more second electronic devices are several electronic devices other than the first electronic device, and each electronic device and the server can communicate with each other.
  • Wi wireless-fidelity
  • Bluetooth communication Bluetooth communication
  • cellular 2/3/4/5 generation (2/3/4/5generation, 2G/3G/4G/5G) communication cellular 2/3/4/5 generation
  • the “multiple second electronic devices” used herein are only used to indicate multiple other electronic devices except the first electronic device, and it does not limit whether the types of the respective second electronic devices are the same.
  • each electronic device of the multiple electronic devices shown in Figure 1 it can be various types of devices equipped with cameras and display components.
  • the electronic devices can be terminal devices such as mobile phones, tablets, laptops, video recorders, etc. (as shown in the figure).
  • each electronic device is a mobile phone as an example), it can also be a device used for virtual scene interaction, such as VR glasses, AR devices, MR interactive devices, etc., can be wearable electronic devices such as smart watches, smart bracelets, and It can be the on-board equipment in vehicles such as unmanned vehicles and drones.
  • the embodiments of this application do not impose special restrictions on the specific form of the electronic device.
  • any electronic device among the plurality of electronic devices may also be referred to as user equipment (UE), subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device , Mobile subscriber station, terminal equipment, access terminal, mobile terminal, wireless terminal, smart terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term.
  • UE user equipment
  • subscriber station mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device
  • Mobile subscriber station terminal equipment, access terminal, mobile terminal, wireless terminal, smart terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term.
  • the server may specifically be one or more physical servers (for example, one physical server is exemplarily shown in FIG. 1), a computer cluster, or a virtual machine in a cloud computing scenario, and so on.
  • each of the multiple electronic devices can be installed with virtual scene applications such as VR or AR or MR applications, and can be based on user operations (such as clicking, touching, sliding, shaking, voice control, etc.) Run the VR or AR or MR application.
  • the electronic device can collect a video image of any object in the environment through a local camera and/or sensor, and display the virtual object on the display component according to the collected video image.
  • the virtual object may correspondingly be a virtual object in a VR scene, an AR scene, or an MR scene (that is, an object in a virtual environment).
  • the virtual scene application in the electronic device may be an application built in the electronic device itself, or may be an application provided by a third-party service provider installed by the user, which is not limited in the embodiment of this application. .
  • each of the multiple electronic devices is also equipped with a simultaneous localization and mapping (SLAM) system.
  • the SLAM system can create a map in a completely unknown environment and use the map Carry out autonomous positioning, pose (position and posture) determination, navigation, etc.
  • the map constructed by the SLAM system can be referred to as the SLAM map.
  • the SLAM map can be understood as a map drawn by the SLAM system according to the environmental information collected by the collection device.
  • the collection device can include, for example, an image collection device in an electronic device (such as a camera). (Or camera) and an inertial measurement unit (Inertial Measurement Unit, IMU).
  • the IMU may include sensors such as gyroscopes and accelerometers.
  • the SLAM map can contain the following map content: multiple key frames, triangulated feature points, and the association between key frames and feature points.
  • the key frame may be formed based on the image collected by the camera and the camera parameters used to generate the image (for example, the pose of the electronic device in the SLAM coordinate system).
  • a feature point ie, a feature as used herein is an interesting or significant part of an image.
  • the feature points may represent different 3D map points along the three-dimensional space in the SLAM map (for example, the coordinates on the three-dimensional space axes X, Y, and Z) and the feature descriptions on the 3D map points.
  • Each feature point can have an associated feature location.
  • Each feature point can represent a 3D coordinate position and is associated with one or more descriptors.
  • the image quality of the new key frame itself should be good, for example, it cannot be a very blurry image, the number of feature points should be sufficient, and the feature point distribution should be as uniform as possible; on the other hand, the new key frame needs to be
  • the existing key frames in the SLAM map have a small amount of common view relations, but most of the feature points are new feature points to achieve the effect of both existing constraints and as little information redundancy as possible, such as a neighboring key frame in the SLAM map In the new key frame, 10% of the feature points can be observed, and the remaining 90% cannot be observed, and so on.
  • the coordinate system used to construct the SLAM map can be called the first coordinate system.
  • the first coordinate system in this article may also be called the local coordinate system, the SLAM coordinate system, the camera coordinate system or some other in some application scenarios.
  • Appropriate terminology For the convenience of understanding, the following text will mainly introduce the scheme under the name of "local coordinate system".
  • the pose embodied by the electronic device in the local coordinate system can be called a local pose.
  • the server can be used as a platform for providing content and information support to VR or AR or MR applications of any electronic device.
  • a global map is stored in the server.
  • the global map contains a larger area than the SLAM map in a single electronic device, and the map content is more accurate, and the global map is maintained and updated by the server.
  • the global map may be obtained by integrating multiple map update information provided by one or more of the multiple electronic devices according to certain rules.
  • the coordinate system used to construct the global map may be called the second coordinate system.
  • the second coordinate system may also be called the global coordinate system, the world coordinate system, or some other appropriate terminology.
  • the following text will mainly use the name "global coordinate system" to introduce the scheme.
  • the posture embodied by the electronic device in the global coordinate system can be referred to as the global posture.
  • FIG. 2 exemplarily shows a schematic structural diagram of the electronic device 10.
  • the electronic device 10 may be at least one of the first electronic device and the second electronic device. It should be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown, or combine certain components, or split certain components, or arrange different components.
  • the various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and/or application specific integrated circuits.
  • the electronic device 10 may include: a chip 310, a memory 315 (one or more computer-readable storage media), a user interface 322, a display component 323, a camera 324, a positioning module 331 for device positioning, and a user interface For communication transceiver 332. These components may communicate on one or more communication buses 314.
  • the chip 310 may integrate: one or more processors 311, a clock module 312, and a power management module 313.
  • the clock module 312 integrated in the chip 310 is mainly used to provide the processor 311 with a timer required for data transmission and timing control, and the timer can realize the clock function of data transmission and timing control.
  • the processor 311 can perform operations according to the instruction operation code and timing signals, generate operation control signals, and complete the control of fetching instructions and executing instructions.
  • the power management module 313 integrated in the chip 310 is mainly used to provide a stable and high-precision voltage for the chip 310 and other components of the electronic device 10.
  • the processor 110 may also be referred to as a central processing unit (CPU, central processing unit).
  • the processor 110 may specifically include one or more processing units.
  • the processor 110 may include an application processor (AP). Tuning processor, graphics processing unit (GPU), image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor , And/or neural-network processing unit (NPU), etc.
  • AP application processor
  • Tuning processor graphics processing unit
  • GPU image signal processor
  • ISP image signal processor
  • controller video codec
  • digital signal processor digital signal processor
  • DSP digital signal processor
  • NPU And/or neural-network processing unit
  • the different processing units may be independent devices or integrated in one or more processors.
  • the processor 110 may include one or more interfaces.
  • the interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, and a universal asynchronous transmitter/receiver (universal asynchronous) interface.
  • I2C integrated circuit
  • I2S integrated circuit built-in audio
  • PCM pulse code modulation
  • PCM pulse code modulation
  • UART universal asynchronous transmitter/receiver
  • MIPI mobile industry processor interface
  • GPIO general-purpose input/output
  • SIM subscriber identity module
  • USB Universal Serial Bus
  • the memory 315 may be connected with the processor 311 through a bus, or may be coupled with the processor 311, and used to store various software programs and/or multiple sets of instructions.
  • the memory 315 may include a high-speed random access memory (such as a cache memory), and may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage. equipment.
  • the memory 315 may store an operating system, such as an embedded operating system such as ANDROID, IOS, WINDOWS, or LINUX.
  • the memory 315 is also used to store related programs of the SLAM system.
  • the memory 315 is used to store data (for example, image data, point cloud data, map data, key frame data, pose data, coordinate system conversion information, map update information, etc.).
  • the memory 315 may also store a communication program, which may be used to communicate with one or more servers or other devices.
  • the memory 315 may also store one or more application programs. As shown in the figure, these applications may include: virtual scene applications such as AR/VR/MR, map applications, image management applications, and so on.
  • the memory 115 can also store a user interface program.
  • the user interface program can vividly display the content of the application program (such as virtual objects in virtual scenes such as AR/VR/MR) through a graphical operation interface and display it through the display component 323 Present, and realize the control operation of the application program received by the user through input controls such as menus, dialog boxes, and buttons.
  • the memory 315 may be used to store computer executable program code, and the executable program code includes instructions.
  • the user interface 322 may be, for example, a touch panel, through which the user's operation instructions on the touch panel can be detected, and the user interface 322 may also be a small keyboard, a physical button, or a mouse.
  • the electronic device 10 may include one or more display components 323.
  • the electronic device 10 may jointly implement a display function through the display component 323, a graphics processing unit (GPU) and an application processor (AP) in the chip 310, and so on.
  • the GPU is a microprocessor used for image processing, and is connected to the display component 323 and the application processor.
  • the GPU is used to perform mathematical and geometric calculations for graphics rendering.
  • the display component 323 is used to display the interface content currently output by the system, such as displaying images and videos in virtual scenes such as AR/VR/MR.
  • the interface content can include the interface of the running application and the system-level menu, etc., which can be specified by the following The composition of the interface elements: input interface elements, such as buttons (Button), text input boxes (Text), scroll bars (Scroll Bar), menus (Menu), etc.; and output interface elements, such as windows, labels (Label), images, videos, animations, etc.
  • input interface elements such as buttons (Button), text input boxes (Text), scroll bars (Scroll Bar), menus (Menu), etc.
  • output interface elements such as windows, labels (Label), images, videos, animations, etc.
  • the display component 323 may be a display panel, lenses (for example, VR glasses), a projection screen, and the like.
  • the display panel may also be called a display screen, for example, it may be a touch screen, a flexible screen, a curved screen, etc., or other optical components. That is to say, when the electronic device has a display screen in this application, the display screen can be a touch screen, a flexible screen, a curved screen or other forms of screen.
  • the display screen of the electronic device has the function of displaying images. As for the specific material of the display screen and The shape is not limited in this application.
  • the display panel may use a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode or an active matrix.
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • AMOLED active-matrix organic light emitting diodes
  • FLED flexible light-emitting diodes
  • Miniled MicroLed, Micro-oLed, quantum dot light emitting diodes, QLED
  • the touch panel in the user interface 322 and the display panel in the display assembly 323 can be coupled together to set up.
  • the touch panel can be set under the display panel, and the touch panel can be used to detect that the user passes through the display panel.
  • the touch pressure that acts on the display panel when a touch operation (such as click, slide, touch, etc.) is input from the panel, and the display panel is used for content display.
  • the camera 324 may be a monocular camera, a binocular camera, or a depth camera, and is used to photograph/video the environment to obtain an image/video image.
  • the image/video image collected by the camera 324 may be used as a kind of input data of the SLAM system, or the image/video image may be displayed through the display component 323, for example.
  • the camera 324 can also be regarded as a sensor.
  • the image collected by the camera 324 may be in IMG format or other format types, which is not limited here.
  • the sensor 325 can be used to collect data related to the state change (such as rotation, swing, movement, jitter, etc.) of the electronic device 10, and the data collected by the sensor 325 can be used as a kind of input data of the SLAM system, for example.
  • the sensor 325 may include one or more sensors, such as an Inertial Measurement Unit (IMU), a Time of Flight (TOF) sensor, and so on.
  • the IMU may further include sensors such as gyroscopes and accelerometers.
  • the gyroscope can be used to measure the angular velocity of the electronic device when it moves, and the accelerometer is used to measure the acceleration of the electronic device when it moves.
  • the TOF sensor can further include a light transmitter and a light receiver.
  • the light transmitter can be used to emit light, such as laser, infrared, and radar waves
  • the light receiver can be used to detect reflected light, such as reflected laser, infrared, and radar waves. Wait.
  • the sensor 325 may also include more other sensors, such as inertial sensors, barometers, magnetometers, wheel speedometers, and so on.
  • the positioning module 331 is used to implement physical positioning of the electronic device 10, for example, to obtain the initial position of the electronic device 10.
  • the positioning module 331 may include, for example, one or more of a WIFI positioning module, a Bluetooth positioning module, a base station positioning module, and a satellite positioning module.
  • the satellite positioning module can be equipped with a Global Navigation Satellite System (GNSS) to assist positioning.
  • GNSS is not limited to the Beidou system, GPS system, GLONASS system, and Galileo system.
  • the transceiver 332 is used to implement communication between the electronic device 10 and a server or other electronic devices.
  • the transceiver 332 integrates a transmitter and a receiver, which are used to send and receive radio frequency signals, respectively.
  • the transceiver 332 may include, but is not limited to: an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chip, a SIM card, a storage medium, etc.
  • the transceiver 332 may also be implemented on a separate chip.
  • the transceiver 332 may, for example, support data network communication through at least one of 2G/3G/4G/5G, etc., and/or support at least one of the following short-range wireless communication methods: Bluetooth (BT) communication, Wireless Fidelity (WiFi) communication, Near Field Communication (NFC), Infrared (IR) wireless communication, Ultra Wide Band (UWB) communication, ZigBee communication.
  • Bluetooth Bluetooth
  • WiFi Wireless Fidelity
  • NFC Near Field Communication
  • IR Infrared
  • UWB Ultra Wide Band
  • the processor 311 executes various functional applications and data processing of the electronic device 10 by running instructions stored in the memory 315. Specifically, the processor 311 may execute the method steps shown in the embodiment of FIG. 7 or may execute The functions of the electronic device side in the embodiment shown in Figure 9, Figure 10, Figure 15 or Figure 17.
  • FIG. 3 is a structural block diagram of an implementation manner of a server 20 according to an embodiment of the present application.
  • the server 20 may be the server described in the embodiment of FIG. 1.
  • the server 20 includes a processor 403, a memory 401 (one or more computer-readable storage media), and a transceiver 402. These components may communicate on one or more communication buses 404. among them:
  • the processor 403 may be one or more central processing units (CPU). In the case where the processor 403 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.
  • the memory 401 may be connected with the processor 403 via a bus, or may be coupled with the processor 403 to store various software programs and/or multiple sets of instructions and data (for example, map data, pose data, etc.).
  • the memory 401 includes, but is not limited to, random access memory (RAM), read-only memory (Read-Only Memory, ROM), and erasable programmable read-only memory (Erasable Programmable Read Only Memory, EPROM), or portable read-only memory (Compact Disc Read-Only Memory, CD-ROM).
  • the transceiver 402 mainly integrates a receiver and a transmitter, where the receiver is used to receive data (such as requests, images, etc.) sent by the electronic device, and the transmitter is used to send data (such as map data, pose data, etc.) to the electronic device.
  • data such as requests, images, etc.
  • the transmitter is used to send data (such as map data, pose data, etc.) to the electronic device.
  • server 20 is only an example provided by the embodiment of the present application. In a specific implementation, the server 20 may have more components than shown in the figure.
  • the processor 403 may be used to call program instructions in the memory 401, and execute the server-side functions in the embodiment shown in FIG. 9, FIG. 10, FIG. 15 or FIG.
  • Coupled means directly connected to or connected through one or more intervening components or circuits. Any signal provided on the various buses described herein can be time-multiplexed with other signals and provided on one or more shared buses.
  • interconnection between various circuit elements or software blocks may be shown as a bus or a single signal line. Each bus may alternatively be a single signal line, and each single signal line may alternatively be a bus, and a single wire or bus may represent any one or more of a large number of physical or logical mechanisms for communication between components .
  • an electronic device when an electronic device sends a global positioning request to the server, the electronic device uploads an image to the server, the server performs global positioning based on the image, and at the same time triggers the update of the global map on the server. Specifically, the server extracts feature points from the image uploaded by the electronic device, inserts the uploaded image as a node into the global map, and optimizes the global map through a global nonlinear optimization algorithm, thereby realizing the update of the global map.
  • the global positioning request of the electronic device is usually low-frequency, and the frequency of uploading images is low, so it will be slower to update the global map with this solution.
  • the electronic device can frequently upload images to the server to update the global map in the server, not only the electronic device consumes more communication resources, but also when multiple users upload images at the same time, the server load will be particularly heavy.
  • the electronic device directly uploads the image to the server, there is a risk of privacy leakage.
  • FIG. 5 is a structural block diagram of a system provided by an embodiment of the present application.
  • the system includes an electronic device 10 and a server 20, and the electronic device 10 and the server 20 can communicate with each other through respective transceivers.
  • the SLAM system 12, the data acquisition module 13, the interaction module 14, and the communication module 11 are configured in the electronic device 10.
  • the SLAM system 12, the data acquisition module 13, the interaction module 14 and the communication module 11 may exist in the form of software code, in a specific
  • the data/programs of these functional modules can be stored in the memory 315 shown in FIG. 2 and can be run on the processor 311 shown in FIG. 2. among them:
  • the communication module 11 can use the transceiver 332 shown in FIG. 2 to communicate with the server 20. Specifically, the communication module 11 is configured to obtain a global sub-map from the server 20, and the global sub-map is stored by the server 20. A sub-map associated with the location information of the electronic device 10 in the global map. And the global sub-map is stored in the database 123 of the global sub-map in the SLAM system 12 of the electronic device 10.
  • the data acquisition module 13 is configured to use the sensor 325 shown in FIG. 2 to obtain state data of the electronic device 10, the camera 324 shown in FIG. 2 to obtain video images, and the positioning module 331 shown in FIG. 2 To get the location of the electronic device.
  • the interaction module 14 is configured to use the user interface 322 shown in FIG. 2 to realize the detection and acquisition of user operations, and to use the display component 323 shown in FIG. 2 to realize the display of images/videos/virtual objects, such as AR/VR/ Display of application content such as MR.
  • the calculation module 121 in the SLAM system 12 is configured to perform pose calculation according to the video image collected by the camera 324 and the downloaded global submap to obtain the pose data of the electronic device 10; the interaction module 14 may be based on the position of the electronic device 10. Pose data to display virtual objects on the display component.
  • the SLAM map constructed by the SLAM system 12 itself is stored in the SLAM map database 122, and the SLAM system is also configured to update the SLAM map in the database 122 based on the pose data of the electronic device 10.
  • the functional modules in the electronic device 10 can cooperate with each other to execute the steps in the method shown in the embodiment shown in FIG. 7, or execute the electronic device in the embodiment shown in FIG. 9, FIG. 10, FIG. 15 or FIG. Side function.
  • the server 20 is configured with a communication module 21, a map update module 22, and a database 23 of the global map.
  • the communication module 21, the processing module 22, and the database 23 of the global map may exist in the form of software code.
  • the data/programs of these functional modules can be stored in the memory 401 shown in FIG. 3 and can be run on the processor 403 shown in FIG. 3. among them:
  • the database 23 of the global map is used to store, maintain and update the global map.
  • the map update module 22 may be configured to update the global map in the database 23 based on the map update information uploaded by the electronic device.
  • the communication module 21 can use the transceiver 402 as shown in FIG. 3 to realize communication with the electronic device 10. Specifically, the communication module 21 may send the global sub-map to the electronic device 10 and receive map update information from the electronic device 10.
  • the functional modules in the server 20 can cooperate with each other to perform the server-side functions in the embodiment shown in FIG. 9, FIG. 10, FIG. 15 or FIG. 17.
  • FIG. 5 shows the components (or sub-modules) that may be further included in each functional module in the electronic device 10 shown in FIG. 4 in a specific implementation, and the functional modules in the server 20 are further The components (or sub-modules) that may be included.
  • the functional modules in the electronic device 10 for example, the SLAM system, the data acquisition module 13, the interactive module 14
  • the function modules in the server 20 for example, the processing module 22
  • the components (or sub-modules) are only examples of the embodiments of the present application. In other embodiments, the above-mentioned functional modules may also include more or fewer components (or sub-modules).
  • the calculation module 121 in the SLAM system 12 further includes a mapping module 1211, a pose estimation module 1212, a pose estimation module 1212, a feature processing module 1213, a feature matching module 1214, and a map fusion module 1215.
  • the electronic device 10 also includes a global positioning module 16 and a software development kit (Software Development Kit, SDK).
  • the global positioning module 16 further includes an image retrieval module 161, a feature extraction module 162, a feature matching module 163, and a pose estimation module 164.
  • the SDK may include a database for the global pose and the local pose respectively, and the coordinate system change matrix calculation module 15.
  • the SDK may also call the interaction module 14 to realize display through display components.
  • the feature processing module 1213 can be used for operations related to visual feature processing.
  • the feature processing module 1213 can further include a feature detection function and a feature description function.
  • the feature detection function is to extract the image position of the feature in the image
  • the feature description function is to describe each detected feature to form a one-dimensional vector.
  • the feature matching module 1214 can be used to implement feature matching between image features and map features.
  • the pose estimation module 1212 may further perform pose estimation according to the result of feature matching.
  • the data acquisition module 13 can output high-frequency angular velocity and linear acceleration.
  • the pose estimation module 1212 integrates the angular acceleration and linear acceleration separately, and combines the video images taken by the camera to estimate the pose and calculate the electronic equipment Position and posture.
  • the result of pose estimation can be output as SLAM system.
  • the result of the pose estimation can also be used as the input of the mapping module 1211.
  • the mapping module 1211 creates an environment map perceivable by the SLAM system under the local coordinate system, that is, the SLAM map. As the SLAM system continues to operate in the space, it is constantly creating/updating SLAM maps.
  • the SLAM map can also be used as the input of the pose estimation module 1212 to estimate the pose, so as to improve the accuracy of the pose estimation.
  • the database 123 stores a global sub-map downloaded from the server 10, and the pose estimation module 1212 is configured to perform execution based on the global sub-map and the video images collected by the camera 324 and the motion data collected by the sensor 325.
  • the pose calculation is used to obtain the pose data of the electronic device 10, that is, to obtain the global pose of the electronic device 10, so as to realize the tracking and positioning of the pose of the electronic device 10.
  • the movement data includes movement speed data and movement direction data of the electronic device 10, such as acceleration and angular velocity.
  • the feature processing module 1213 can be used to extract 2D features of the video image, and the pose estimation module 1212 can be based on the 2D features of the video image, the 3D map points of the global submap, and the movement collected by the data collection module 13 (for example, IMU). Data to obtain the global pose of the electronic device 10 in the global sub-map.
  • the data collection module 13 for example, IMU
  • the pose estimation module 1212 of the SLAM system can also be used to determine the local pose of the electronic device 10 in the SLAM map in the local coordinate system based on the video image collected by the camera and the SLAM map in the local coordinate system.
  • the feature processing module 1213 can be used to extract 2D features in the video image, and the pose estimation module 1212 can be based on the 2D features in the video image, the 3D map points of the SLAM map in the local coordinate system, and the data collection module. 13 (such as IMU) collected motion data to obtain the local pose of the electronic device 10 in the SLAM map in the local coordinate system; the motion data includes the motion speed data and motion direction data of the electronic device 10, such as acceleration, Angular velocity etc.
  • the global pose module 16 is used to determine the global pose of the electronic device 10 in the global sub-map according to the collected video image and the global sub-map.
  • the video image can be obtained by the image retrieval module 161, the feature extraction module 162 can perform feature extraction based on the video image to obtain the image feature, and the feature matching module 163 can perform feature matching on the image feature in the global submap.
  • a map feature matching the image feature is obtained; the pose estimation module 164 calculates and obtains the global pose of the electronic device 10 in the global sub-map according to the image feature and the map feature.
  • the above-mentioned local pose and global pose can be stored in the database in the SDK respectively.
  • the coordinate system change matrix calculation module 15 may be configured to calculate the coordinate system transformation between the local coordinate system of the SLAM map and the global coordinate system of the global map according to the local pose and the global pose of the same video image. Information (for example, coordinate system transformation matrix). And the coordinate system transformation information is fed back to the SLAM system.
  • the mapping module 1211 in the SLAM system may be configured to update the SLAM map in the database 122 of the SLAM system according to the pose data of the electronic device 10. Specifically, the mapping module 1211 may transform the SLAM map to the global coordinate system according to the coordinate system transformation information, and use the global pose of the electronic device 10 as the pose data in the SLAM map in the global coordinate system to update the SLAM map in the global coordinate system.
  • the mapping module 1211 will continue to create SLAM maps, which is reflected in the continuous insertion of key frames.
  • the key frames and map points in the SLAM map can be transformed to the global coordinate system.
  • the map fusion module 1215 can be used to compare the SLAM map and the global sub-map, thereby identifying whether there are non-redundant key frames in the current SLAM map. If there is a non-redundant key frame, then the non-redundant key frames are merged into the global sub-map. Map, get map update information.
  • the map update information may indicate the difference between the SLAM map in the global coordinate system and the global sub-map in the global coordinate system.
  • the map update information includes: pose data, three-dimensional map points, and common view data corresponding to the non-redundant key frame, and the common view data is used to indicate the non-redundant key frame and the common view data. The common view relationship between the key frames of the global sub-map.
  • the process of map fusion performed by the map fusion module 1215 can be described as: the map fusion module 1215 determines whether the key frame newly inserted by the SLAM system is the same as the key in the downloaded global submap
  • the frame is redundant.
  • Keyframe redundancy means that the 3D map point information associated with the keyframe already exists in the global submap, that is, the global submap has saved the visual information in the current environment, and there is no need to add keyframes.
  • the key frame non-redundancy means that the 3D map point information associated with the key frame does not exist in the global sub-map (such key frames may be referred to as non-redundant key frames).
  • keyframe nodes undirected graph structure
  • update the common view relationship between the keyframes in the global submap After updating the key frame common view relationship of the global sub-map, use the triangulation algorithm to calculate new 3D map points, and optimize the pose and 3D map point coordinates of the key frames through the BA algorithm, so as to realize the update of the global sub-map, namely
  • the obtained updated global sub-map (or called the fused global sub-map) is stored in the database 123.
  • the updated global sub-map can be applied to the global positioning module 16 to calculate the global pose.
  • the updated global sub-map can be used to provide the server 20 with map update information (for example, the poses corresponding to non-redundant key frames, 3D map points, common view relationships, etc.), that is, the electronic device 10 can be updated from The global sub-map of obtains the map update information and sends it to the server 20, so that the server 20 uses the map update information to update the global map.
  • map update information for example, the poses corresponding to non-redundant key frames, 3D map points, common view relationships, etc.
  • the electronic device 10 also directly sends the non-redundant key frame to the server 20; the server 20 further uses the non-redundant key frame to update the global map.
  • the updated global sub-map can be applied to the global positioning module 16 to calculate the global pose.
  • the functional modules in the electronic device 10 can cooperate with each other to execute the steps in the method shown in the embodiment shown in FIG. 7, or execute the electronic device in the embodiment shown in FIG. 9, FIG. 10, FIG. 15 or FIG. Side function.
  • the server 20 also includes a sub-map update module 221 and a location fingerprint matching module 222.
  • the map update module 22 may be configured to add the map update information to the global map. , Realize the update of the global map; or be configured to merge non-redundant key frames into the global map to realize the update of the global map.
  • the location fingerprint matching module 222 is configured to search the global map for the location fingerprint information of the initial location sent by the electronic device 10 (herein may be referred to as the first location fingerprint information). Matched location fingerprint information (here can be referred to as second location fingerprint information).
  • the sub-map processing module 221 is configured to retrieve the global sub-map with the second location fingerprint information from the global map in the database 23.
  • the global sub-map can also be additionally stored in the database of the server 20 for quick access next time.
  • the server 20 may send the global sub-map with the second location fingerprint information to the electronic device 10.
  • the functional modules in the server 20 can cooperate with each other to perform the server-side functions in the embodiment shown in FIG. 9, FIG. 10, FIG. 15 or FIG. 17.
  • FIG. 7 is a schematic flowchart of a virtual object display method provided by an embodiment of the present application.
  • the method may be applied to a first electronic device having a display component and a camera.
  • the method includes but is not limited to the following steps:
  • the user inputs an operation for opening an application (APP) on the electronic device, such as clicking, touching, sliding, shaking, voice control, etc., in response to the operation, on the one hand, the display of the electronic device
  • the application interface is displayed on the component (such as the display panel or the lens), and on the other hand, the process of downloading the global submap from the server or other devices (such as other terminal devices or hard disks, USB and other storage media) is started.
  • the application may be an application such as AR/VR/MR installed in an electronic device.
  • the server receives map update information from each second electronic device, and can continuously update the global map in the server based on the map update information.
  • the map update information is based on the comparison result of the second electronic device's SLAM map and the global sub-map downloaded by the second electronic device, and the second electronic device's own local coordinate system and the global sub-map.
  • the coordinate system transformation information between the global coordinate system is obtained.
  • the map update information may specifically include: pose data corresponding to the target key frame constructed by the SLAM system of the second electronic device, three-dimensional map points (or 3D map points), and common view data
  • the common view data is used to indicate the common view relationship between the target key frame and the key frame of the global sub-map in the second electronic device.
  • the target key frame is a non-redundant key frame
  • the non-redundant key frame represents a key frame that exists in the SLAM map of the second electronic device but does not exist in the global sub-map of the second electronic device.
  • the computing resources of each second electronic device are fully utilized to calculate and obtain map update information, thereby reducing the calculation load of the server.
  • the map update information has a smaller amount of data, which saves transmission bandwidth resources.
  • the server can quickly implement global map update based on the map update information of the second electronic device, and the update efficiency is high. Since the global map in the server can be updated efficiently and accurately, it also promotes the global positioning of the first electronic device to have higher accuracy and better robustness.
  • the electronic device 10 in the figure is a mobile phone, and a certain virtual scene application is installed in the electronic device 10 (for example, it may be a navigation map application with AR function),
  • a) in FIG. 8 shows a graphical user interface (GUI) on the display panel of the electronic device 10, and the GUI is the electronic device Desktop 101.
  • GUI graphical user interface
  • the electronic device detects that the user has clicked the icon 102 of the virtual scene application on the desktop 101, on the one hand, it starts the process of downloading the global sub-map in the background; on the other hand, it displays on the display panel after starting the virtual scene application.
  • GUI graphical user interface
  • the user interface 103 of the GUI is an AR navigation interface as an example.
  • the user interface 103 may include a viewing frame 104.
  • the viewing frame 104 can display the preview video stream of the real environment where the electronic device 10 is located in real time.
  • the preview video stream is captured by the camera of the electronic device 10.
  • virtual objects of the AR application are also superimposed.
  • the number of virtual objects can be one or more.
  • the virtual objects exemplarily take the navigation indicator 105 and the interactive robot 107 as examples.
  • the navigation indicator 105 can indicate the current position in real time through the azimuth arrow.
  • the navigation route of the location to a certain destination, the interactive robot 107 can be used to implement voice conversations, voice introductions, or just as interesting displays on the street, and so on.
  • the pose calculation is performed according to the obtained global sub-map and the video image captured by the camera in real time, so as to obtain the current pose of the electronic device 10 in the global coordinate system in real time.
  • Data ie global pose.
  • the position and posture of the virtual object in the AR scene can be determined by the global posture of the electronic device 10, that is, the global posture of the electronic device represents the position and posture of the virtual object. Therefore, the position and posture of the virtual object in the AR scene can be displayed in the viewing frame 104 based on the global pose of the electronic device 10 in real time.
  • AR applications can use computer graphics and visualization techniques to generate virtual objects that do not exist in the real environment, and based on the current global pose of the electronic device 10, superimpose the virtual objects into the viewing frame 104, thereby placing the virtual objects in the viewing frame 104 It is superimposed on the video stream of the view frame 104.
  • the AR application sends the video image to the AR cloud server to request the AR cloud server to obtain the object to be rendered corresponding to the video image.
  • the object to be rendered may include the object to be rendered. Identification and/or information such as the name and metadata of the object to be rendered.
  • the AR cloud server sends the object to be rendered corresponding to the video image to the electronic device 10.
  • the electronic device 10 determines the business rule corresponding to the object to be rendered, and uses the business rule of each object to be rendered to render the corresponding object to be rendered, thereby Multiple one or more AR objects (ie virtual objects) are generated, and based on the global pose of the electronic device 10, the virtual objects are superimposed on the video stream of the view frame 104.
  • the electronic device 10 can continue to accurately superimpose virtual objects on the video stream of the viewfinder 104 to improve the user Use experience.
  • the electronic device 10 when the electronic device 10 is about to move out of the geographic range of the global sub-map, it can request to download a new global sub-map in advance based on the location of the electronic device 10, and the subsequent electronic device 10 can use the new global sub-map to download Performing the global pose estimation can further avoid the occurrence of sudden changes in the pose when the geographic ranges corresponding to the two global submaps are switched, and further improve the user experience.
  • FIG. 9 is a schematic flowchart of a global map update method provided by an embodiment of the present application.
  • the method is described separately from the electronic device and the server side.
  • the electronic device may be the first described above.
  • An electronic device may also be the second electronic device described above.
  • the method includes but is not limited to the following steps:
  • the server issues a global sub-map to the electronic device based on the request of the electronic device.
  • the electronic device receives the global sub-map.
  • the global sub-map is a sub-map associated with the location of the electronic device in the global map.
  • the electronic device stores the global sub-map in the SLAM system of the electronic device.
  • the electronic device obtains map update information according to the SLAM map and global sub-map constructed by itself.
  • the map update information can be used to indicate the difference between the SLAM map in the global coordinate system and the global sub-map in the global coordinate system.
  • the electronic device can use the coordinate system transformation information between the local coordinate system and the global coordinate system to convert the SLAM map built by itself to the global coordinate system, and then compare the SLAM map in the global coordinate system The comparison result is obtained with the global sub-map (the comparison result at this time is the comparison result in the global coordinate system), and then the map update information is obtained according to the comparison result.
  • the comparison result may be at least one key frame in the SLAM map in the global coordinate system, that is, the at least one key frame does not exist in the global sub-map, or is different from each key frame in the global sub-map. If there are significant differences, the map update information can be obtained according to the at least one key frame.
  • the map update information may be related information after the at least one key frame is merged into the global sub-map (such as pose, common view relationship, 3D map Point and other information), the map update information may also directly be the at least one key frame.
  • the electronic device first obtains the comparison result directly according to the SLAM map in the local coordinate system and the global sub-map in the global coordinate system (the comparison result at this time is the comparison result in the local coordinate system), and then uses the local The coordinate system transformation information between the coordinate system and the global coordinate system, and the comparison result is converted to the global coordinate system, thereby obtaining map update information.
  • the comparison result may be at least one key frame in the SLAM map in the local coordinate system, and the at least one key frame does not exist in the global sub-map after being converted to the global coordinate system, or is different from the global sub-map.
  • Each key frame in has a significant difference, then the map update information can be obtained according to the at least one key frame.
  • the map update information may be that the at least one key frame is converted to the global coordinate system and then merged into the global sub-map Related information (such as pose, common view relationship, 3D map point, etc.), map update information may also be directly the key frame formed by the conversion of the at least one key frame to the global coordinate system.
  • the aforementioned at least one key frame may also be referred to as a non-redundant key frame.
  • the electronic device sends the map update information to the server, and correspondingly, the server receives the map update information.
  • the server updates the global map in the server according to the map update information. Since the map update information can represent the missing map content in the sub-maps of the global map, the map update information can be used to update the global map in the server.
  • the electronic device downloads the global sub-map in the global map from the server, and obtains map update information by comparing the global sub-map with the SLAM map in the electronic device and using the computing resources and computing capabilities of the electronic device.
  • the map update information can be used to update the map content in the global sub-map. Therefore, after the electronic device sends the map update information to the server, the server can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the efficiency of the server to update the global map is improved. Compared with the image data captured by the camera, the map update information has a smaller amount of data, so the transmission bandwidth resources are also saved.
  • FIG. 10 is a schematic flowchart of another global map update method provided by an embodiment of the present application, which is described separately from the electronic device and the server side.
  • the method includes but is not limited to the following steps:
  • the electronic device sends first location fingerprint information used to indicate the initial location of the electronic device to the server.
  • the electronic device uploads location fingerprint information to the server.
  • the initial location may be the location information of the electronic device when the electronic device requests to download the map.
  • the source of the location fingerprint information includes GNSS/WiFi/Bluetooth/ The initial position information, signal strength information, signal characteristic information, etc. measured by the base station and other positioning methods; it may also be the position information input by the user.
  • the server obtains a global submap matching the fingerprint information of the first location from the global map.
  • the global sub-map is a sub-map associated with the location of the electronic device in the global map.
  • the server delivers the global sub-map to the electronic device.
  • the electronic device receives the global sub-map.
  • the server matches the location fingerprint information of the submap stored in the database in advance by the server according to the above-mentioned first location fingerprint information, and the submaps in the database are submaps belonging to the global map. If there is a matching sub-map, the matching sub-map is transmitted to the electronic device.
  • the server traverses the global map saved by the server according to the above-mentioned first location fingerprint information until it finds an area matching the location fingerprint information, and the server takes the area from the global map as a global sub-map, and uses the global The sub-map is transmitted to the electronic device.
  • the electronic device stores the global sub-map in the SLAM system of the electronic device.
  • the electronic device calculates coordinate system transformation information, and transforms the SLAM map based on the coordinate system transformation information.
  • the process of calculating the coordinate system transformation information of the electronic device can be described as follows: the electronic device obtains a frame of video image collected by the camera; then, on the one hand, according to the video image and the SLAM map, determine the local coordinates originally constructed by the electronic device
  • the local pose in the SLAM map under the system here can be referred to as the first pose data.
  • the electronic device is provided with an IMU
  • the input signal of the SLAM system includes the video image collected by the camera, the motion data collected by the IMU, and the SLAM map in the local coordinate system.
  • the IMU detects the angular velocity and linear acceleration of the electronic device at a high frequency, and integrates the angular acceleration and linear acceleration separately, and then can calculate the posture of the electronic device.
  • the video image collected by the camera is matched in the SLAM map in the local coordinate system, so that the position and posture of the electronic device can also be calculated. Then, based on the calculation of these two poses with a certain algorithm, the first pose data can be obtained.
  • the electronic device is also provided with a positioning module related to pose or movement (GPS positioning, Beidou positioning, WIFI positioning, or base station positioning, etc.), then the SLAM system can also refer to the camera collection
  • the first pose data is calculated by using the video image of the video image, the motion data collected by the IMU, the SLAM map in the local coordinate system, and the data collected by the positioning module.
  • feature extraction can be performed on the video image based on the video image and the global sub-map, and feature matching can be performed on the global sub-map, so as to determine the global pose of the electronic device in the global sub-map (herein may be referred to as the first Two pose data).
  • Feature detection algorithms are not limited to feature detection methods such as FAST, ORB, SIFT, SURF, D2Net, and SuperPoint. Then perform feature description for each detected feature.
  • the feature description algorithm is not limited to feature description methods such as ORB, SIFT, SURF, BRIEF, BRISK, FREAK, D2Net, SuperPoint, etc., to form a one-dimensional vector for subsequent feature matching .
  • feature matching the electronic device can match the most similar map content (such as one or more key frames) to the video image from the global submap.
  • Specific methods include traditional image retrieval methods such as BOW and VLAD, and NetVLAD, AI's new image retrieval method.
  • Feature matching specifically calculates the degree of similarity between two feature descriptions. For Float-type vectors, it can be calculated by Euclidean distance; for binary vectors, it can be calculated by XOR. After finding the most similar map content to the video image, the pose estimation can be performed based on the video image and the most similar map content. For example, registration algorithms such as PnP, EPnP, 3D-3D, etc. can be used to calculate the second Pose data.
  • the electronic device can obtain coordinate system transformation information between the first coordinate system of the SLAM map and the second coordinate system of the global map according to the first pose data and the second pose data,
  • the coordinate system transformation information may be, for example, a coordinate system transformation matrix.
  • G T L can represent the coordinate system transformation matrix between the local coordinate system and the global coordinate system.
  • the two coordinate systems can be synchronized.
  • the information originally represented by the local coordinate system For example, key frames, local poses, image feature points, SLAM map 3D map points, etc.
  • the original SLAM map represented by the local coordinate system can be transformed to the global coordinate system based on the coordinate system transformation matrix.
  • the electronic device judges whether there is a non-redundant key frame.
  • the key frames and map points in the SLAM map can be transformed into the global coordinate system for representation, and the SLAM map can also be transformed into the global coordinate system for representation.
  • the SLAM map will be continuously updated.
  • the so-called "update of the SLAM map" is embodied in the continuous insertion of key frames into the SLAM map.
  • the electronic device determines whether the key frame inserted by the SLAM system is redundant with the key frame in the downloaded global sub-map.
  • the so-called redundancy of key frames means that the 3D map point information associated with the key frames already exists in the global sub-map, that is, the global sub-map has saved the visual information in the current environment, and there is no need to add the key frame.
  • the key frame is not redundant, that is, the 3D map point information associated with the key frame does not exist in the global sub-map, and the global sub-map needs to add the key frame to realize the update, so that the map content can be synchronized with the real environment.
  • the key frames that are not redundant with the key frames in the global submap can be referred to as non-redundant key frames for short.
  • the method for the electronic device to identify the non-redundant key frame may be, for example, detecting the association between the current key frame and the map point of the key frame in the global sub-map of the current location of the electronic device, if the map on the global sub-map The point is projected on the current key frame, and no associated 2D feature points can be found, or there are few 2D feature points on the association (for example, less than 50%, it should be noted that this is only used for explanation), then judge the key The frame is a non-redundant key frame.
  • map point on the global submap is projected on the current key frame, many 2D feature points can be associated (for example, greater than or equal to 50%, it should be noted that this is only used for explanation), then it is judged that the key frame is not non- Redundant key frames.
  • the electronic device when there is a non-redundant key frame, the electronic device adds a key frame node (undirected graph structure) in the global sub-map, and updates the difference between the non-redundant key frame and the key frame in the global sub-map.
  • the common view relationship is shown in FIG. 12, which is a schematic diagram for explaining the common view relationship between non-redundant key frames and key frames in the global sub-map. For a certain 3D map point in the global submap, if the 3D map point can be observed by several key frames at the same time, then the several key frames have a common view relationship with the 3D map point.
  • the electronic device can calculate which 3D map points in the global sub-map can be simultaneously observed by at least one key frame and non-redundant key frame in the global sub-map, thereby updating the key frames and non-redundant key frames in the global sub-map
  • the common view relationship between the keys (as shown by the dotted line in the figure).
  • the electronic device can use the triangulation algorithm to calculate new 3D map points, and use the BA algorithm to optimize the pose and 3D map point coordinates corresponding to the non-redundant key frames to complete In order to integrate (or insert) non-redundant key frames into the global sub-map, the update of the global sub-map is realized.
  • the electronic device sends the map update information to the server, and correspondingly, the server receives the map update information.
  • the server updates the global map in the server according to the map update information.
  • the map update information uploaded by the electronic device includes the pose, common view data, and 3D map points corresponding to the updated non-redundant key frames in the global submap.
  • the common view data is used to indicate the common view relationship between the non-redundant key frame and the key frame of the global submap in the second coordinate system.
  • the map update information includes the key frame pose, common view relationship, and 3D map points that change after the global sub-map is downloaded to the electronic device.
  • the electronic device when the electronic device determines that the electronic device is about to leave the actual geographic range corresponding to the global sub-map, it will request the server to download another global sub-map based on the current location information. Before the electronic device requests the another sub-map, or while the electronic device requests the other sub-map, the electronic device uploads the map update information to the server. Compared with the previously downloaded global sub-map, the uploaded map update information only needs to include the changed key frame pose, common view relationship, and 3D map points, so the amount of data is much smaller.
  • the server After the server receives the map update information, it can directly update the key frame common view relationship on the server's global map based on the map update information, update the map point coordinate value, and update the key frame pose. After such an update, it is equivalent to It is equivalent to fusing non-redundant key frames to the global map, thus realizing the high-efficiency update of the global map.
  • the electronic device in this embodiment has completed triangulation calculations, BA optimization calculations, and other computationally intensive calculation steps in advance to obtain map update information, so there is no need for the server to use map update information to update the global map. Perform triangulation calculations, BA optimization calculations and other calculation steps, thereby greatly reducing the computing burden of the server.
  • the process from the user clicking the virtual scene application to the electronic device uploading map update information to the server can be implemented in the following ways:
  • a certain virtual scene application (for example, a navigation map application with AR function) is installed in the electronic device 10, and (a) in FIG. 13 shows The desktop 101 on the display panel of the electronic device 10 is displayed.
  • the electronic device detects that the user has clicked the icon 102 of the virtual scene application on the desktop 101, on the one hand, the electronic device enters the application, for example, the display panel of the electronic device 10 displays as shown in (b) in FIG. 13
  • the user interface 106 where the user interface 106 includes, for example, an electronic map interface and multiple controls.
  • the illustration includes an electronic map control, a satellite map control, and an AR control.
  • the electronic map control is a control that controls the application to enter the electronic map interface
  • the satellite map control is a control that controls the application to enter the satellite map interface
  • the AR control is a control that controls the application to enter the AR interface.
  • the electronic device 10 starts the process of downloading the global sub-map in the background. After the download of the global sub-map is completed, the electronic device 10 further performs the identification of non-redundant key frames in the background. If there is a non-redundant key frame, it can further Fusion of non-redundant key frames into the global sub-map to obtain map update information.
  • a control for user operation can be popped up in the user interface 106 of the electronic device 10, such as the control 109 shown in (b) in FIG. 13.
  • 109 indicates to the user whether it is necessary to upload map update information in order to update the global map on the server side.
  • the control 109 may include two options of "reject” and "agree".
  • the electronic device 10 detects that the user clicks "agree”
  • the map update information upload process is started, so that the map update information is uploaded to the server to update the global map.
  • the electronic device 10 detects that the user clicks "reject”, it does not start the uploading process of the map update information.
  • both the timeliness of uploading the map update information is taken into consideration, and the user's current willingness to share the map update information (for example, the user may not upload it when the user does not want to consume additional data), thereby improving the user experience.
  • the process from the user clicking on the virtual scene application to the electronic device uploading map update information to the server can also be implemented in the following ways:
  • a certain virtual scene application (for example, a navigation map application with AR function) is installed in the electronic device 10, and (a) in FIG. 14 shows The desktop 101 on the display panel of the electronic device 10 is displayed.
  • the electronic device detects that the user has clicked the icon 102 of the virtual scene application on the desktop 101, it displays a user interface 108 as shown in (b) of FIG. 14 on the display panel of the electronic device 10, and the user interface 108 includes A text box for entering the account and password to prompt the user to log in to the application by verifying their identity.
  • the user interface 108 also includes a control 110 to be selected.
  • the to-be-selected control 110 is used to indicate whether the electronic device is allowed to automatically upload map update information.
  • the electronic device 10 detects that the user clicks to trigger the check of the to-be-selected control 110, the automatic upload of map update information is allowed in the background.
  • the user does not click to trigger the check of the to-be-selected control 110, the automatic upload of map update information is not allowed in the background.
  • the electronic device 10 After the user of the electronic device 10 clicks the trigger to check the control to be selected 11, and detects that the user enters the account and password to log in (the electronic device 10 verifies that the account and password are correct), on the one hand, the electronic device enters the application, for example, in the electronic device 10
  • the user interface 106 shown in (c) in Figure 14 is displayed on the display panel of FIG. 14, where the user interface 106 includes, for example, an electronic map interface and multiple controls.
  • the illustration includes an electronic map control, a satellite map control, and AR controls.
  • the electronic map control is a control that controls the application to enter the electronic map interface
  • the satellite map control is a control that controls the application to enter the satellite map interface
  • the AR control is a control that controls the application to enter the AR interface.
  • the electronic device 10 starts the process of downloading the global sub-map in the background. After the download of the global sub-map is completed, the electronic device 10 further performs the identification of non-redundant key frames in the background. If there is a non-redundant key frame, it can further Fusion of non-redundant key frames into the global sub-map to obtain map update information, and upload the map update information to the server in the background to update the global map.
  • the electronic device downloads the global sub-map in the global map from the server, and obtains the terminal's pose in the local coordinate system and the pose in the global coordinate system according to the same frame, based on these two
  • the pose can obtain the coordinate system transformation information between the two coordinate systems (such as the coordinate system transformation matrix).
  • the two coordinate systems can be synchronized.
  • the original local coordinate system is used to express Information (such as local pose, image feature points, 3D map points of SLAM map, etc.) can be transformed to the global coordinate system based on the coordinate system transformation matrix.
  • the server can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the update efficiency and robustness of the global map are improved.
  • FIG. 15 is a schematic flowchart of another global map update method provided by an embodiment of the present application, which is described separately from the electronic device and the server side.
  • the main difference between FIG. 15 and the aforementioned embodiment in FIG. 10 is that the map update information in the embodiment in FIG. 10 is the changed key frame pose, common view relationship, and 3D map points in the global sub-map, while in the embodiment in FIG.
  • the map update information is directly non-redundant key frames.
  • S501 The electronic device sends first location fingerprint information used to indicate the initial location of the electronic device to the server.
  • the server obtains a global submap matching the fingerprint information of the first location from the global map.
  • the global sub-map is a sub-map associated with the location of the electronic device in the global map.
  • S503 The server delivers the global sub-map to the electronic device. Correspondingly, the electronic device receives the global sub-map.
  • the electronic device stores the global sub-map in the SLAM system of the electronic device.
  • the electronic device calculates coordinate system transformation information, and transforms the SLAM map based on the coordinate system transformation information.
  • S506 The electronic device judges whether there is a non-redundant key frame.
  • the server updates the global map in the server according to the map update information.
  • the electronic device judges that the new key frame is a non-redundant key frame, in order to save the computing power of the electronic device, it can also be used directly.
  • the non-redundant key frames are sent to the server, and the server executes the process of fusing (or inserting) the non-redundant key frames into the global map in the server.
  • the server can add key frame nodes (undirected graph structure) to the global map, and update the common view relationship between the non-redundant key frame and the key frames in the global map, that is, the server can calculate the global map Which 3D map points can be simultaneously observed by at least one key frame in the global map and the non-redundant key frame, thereby updating the common view relationship between the key frame and the non-redundant key in the global map.
  • key frame nodes undirected graph structure
  • the server can use the triangulation algorithm to calculate new 3D map points, and optimize the pose and 3D map point coordinates corresponding to the non-redundant key frames through the BA algorithm, thereby completing the non-redundant
  • the process of fusion (or insertion) of redundant key frames into the global map realizes the update of the global map.
  • the electronic device downloads the global sub-map in the global map from the server, and obtains the terminal's pose in the local coordinate system and the pose in the global coordinate system according to the same frame, based on these two
  • the pose can obtain the coordinate system transformation information between the two coordinate systems (such as the coordinate system transformation matrix).
  • the two coordinate systems can be synchronized.
  • the original local coordinate system is used to express Information (such as local pose, image feature points, 3D map points of SLAM map, etc.) can be transformed to the global coordinate system based on the coordinate system transformation matrix.
  • the non-redundant key frames of the SLAM map are identified, and the non-redundant key frames are sent to the server, and the server uses the non-redundant key frames to merge into the global map, thereby It can also realize the efficient update of the global map in the server.
  • Such a solution is beneficial to reduce the computational load of electronic equipment and ensure the accuracy of map fusion.
  • FIG. 16 shows the components that may be further included in the functional modules in the electronic device 10 shown in FIG. 4 and the components that may be further included in the functional modules in the server 20 in another specific implementation.
  • the main difference between the embodiment in FIG. 16 and the embodiment in FIG. 5 is that in the functional module architecture shown in the embodiment in FIG. 16, the functional configuration of the global positioning module 16 is implemented on the server 20 side, that is, the server 20 also includes an image retrieval module 161, The feature extraction module 162, the feature matching module 163, and the pose estimation module 164.
  • the global pose module 16 in the server 20 is used to obtain at least one frame of video image uploaded by the electronic device at the initial moment or any time thereafter, and calculate the global pose of the electronic device 10 in the global submap based on the video image (For example, the second pose data), and send the global pose to the electronic device 10.
  • the video image in the video image sequence uploaded by the electronic device 10 can be obtained through the image retrieval module 161, and the feature extraction module 162 can perform feature extraction according to the video image to obtain the image feature, and the feature matching module 163 can use the feature matching module 163 to perform feature extraction on the video image.
  • the feature is matched in the global sub-map to obtain the map feature that matches the image feature; the pose estimation module 164 calculates that the electronic device 10 is in the global sub-map based on the image feature and the map feature.
  • the global pose of (for example, the second pose data), and the global pose is sent to the electronic device 10.
  • each functional module in the electronic device 10 shown in FIG. 16 may be similar to the related description of the electronic device 10 in the embodiment of FIG. 5. For the sake of brevity of the description, the details are not repeated here.
  • the functional modules in the electronic device 10 can cooperate with each other to perform the functions on the electronic device side in the embodiment of FIG. 17.
  • the functional modules in the server 20 can cooperate with each other to perform the server-side functions in the embodiment of FIG. 17.
  • FIG. 17 is a schematic flowchart of yet another virtual object display method provided by an embodiment of the present application, which is described separately from the electronic device and the server side.
  • the method includes but is not limited to the following steps:
  • the electronic device sends first location fingerprint information and at least one frame of video image used to indicate the initial location of the electronic device to the server.
  • the electronic device in order to realize the first global positioning of the electronic device, the electronic device needs to upload location fingerprint information and at least one or more currently collected video images to the server.
  • the at least one frame of video image may be a video image in a video image sequence shot by an electronic device through a camera.
  • it may be the first frame image in a video image sequence taken by a camera.
  • the initial location indicated by the location fingerprint information may be the geographic location information of the electronic device when the electronic device requests to download the map.
  • the source of the location fingerprint information includes the initial location measured by GNSS/WiFi/Bluetooth/base station. Location information, signal strength information, signal feature information, etc.; it may also be the location information entered by the user.
  • the electronic device may package and send the first location fingerprint information and a frame of video image to the server.
  • the electronic device may also independently send the first location fingerprint information and a frame of video image to the server.
  • the server obtains a global submap matching the fingerprint information of the first location from the global map.
  • the global sub-map is a sub-map associated with the location of the electronic device in the global map.
  • the server matches the location fingerprint information of the submap stored in the database in advance by the server according to the above-mentioned first location fingerprint information, and the submaps in the database are submaps belonging to the global map. If there is a matching sub-map, the sub-map is a global sub-map that needs to be subsequently issued to the electronic device.
  • the server traverses the global map saved by the server according to the above-mentioned first location fingerprint information until it finds an area matching the location fingerprint information, and the server takes the area from the global map as a global submap.
  • S603 The server performs pose calculation according to the video image and the global sub-map, and obtains the global pose of the electronic device in the global sub-map (herein may also be referred to as second pose data).
  • the first calculation of the global pose of the electronic device can be done on the server side.
  • the process of performing the global pose calculation by the server also includes image retrieval, feature extraction, feature matching, pose estimation, etc. ,
  • the server performs feature detection on the video image, and extracts the image location of the feature from the video image.
  • the feature detection algorithm is not limited to feature detection methods such as FAST, ORB, SIFT, SURF, D2Net, SuperPoint, etc. Then perform feature description for each detected feature.
  • the feature description algorithm is not limited to feature description methods such as ORB, SIFT, SURF, BRIEF, BRISK, FREAK, D2Net, SuperPoint, etc., to form a one-dimensional vector for subsequent feature matching .
  • the server can match the most similar map content (such as one or more key frames) to the video image from the global submap.
  • Feature matching specifically calculates the degree of similarity between two feature descriptions. For Float-type vectors, it can be calculated by Euclidean distance; for binary vectors, it can be calculated by XOR. After finding the map content that is most similar to the video image, the pose estimation can be performed based on the video image and the most similar map content. For example, registration algorithms such as PnP, EPnP, 3D-3D, etc. can be used to calculate the first image. Two pose data.
  • the server issues the global pose (second pose data) of the electronic device in the global submap to the electronic device.
  • the electronic device receives the second pose data.
  • the electronic device can subsequently use the second pose data to calculate coordinate system transformation information (for example, a coordinate system transformation matrix).
  • S605 The server delivers the global sub-map to the electronic device. Correspondingly, the electronic device receives the global sub-map.
  • the electronic device stores the global sub-map in the SLAM system of the electronic device.
  • the electronic device calculates coordinate system transformation information.
  • the process of the electronic device calculating coordinate system transformation information can be described as follows: the electronic device obtains a video image, and the video image is the same image as the video image sent to the server in S601; then, on the one hand, according to the video image and the SLAM map, Determine the local pose of the electronic device in the SLAM map under the originally constructed local coordinate system (here it may be referred to as the first pose data).
  • the specific implementation process may be similar to the description of the first pose data in S305 of the embodiment in FIG. 10. For the sake of brevity of the description, details are not repeated here.
  • the electronic device can further obtain the coordinates between the first coordinate system of the SLAM map and the second coordinate system of the global map according to the first pose data and the second pose data obtained through S604
  • the system transformation information, and the coordinate system transformation information may be, for example, a coordinate system transformation matrix.
  • the related content of the coordinate system transformation matrix can be similarly referred to the related description of the embodiment in FIG. 11. For the sake of brevity of the description, the details are not repeated here.
  • the electronic device judges whether there is a non-redundant key frame.
  • the electronic device fuses the non-redundant key frame into the global sub-map to obtain map update information.
  • the electronic device sends the map update information to the server, and correspondingly, the server receives the map update information.
  • the server updates the global map in the server according to the map update information.
  • the electronic device needs to download the global sub-map of the corresponding area first, and the process of downloading the map takes a certain amount of time.
  • the first global pose estimation can be done on the server side. In other words, after the application is started, the first global pose estimation is performed on the server side. While starting the global pose estimation, the server obtains the global sub-map and transmits it to the electronic device accordingly, which improves the user's access speed to the application. The user does not perceive the delay of the map downloading process. In this way, the user waiting caused by the download delay can be avoided, and the user experience is further improved.
  • the electronic device obtains map update information by comparing the global sub-map with the SLAM map in the electronic device, and uses the computing resources and computing capabilities of the electronic device to obtain map update information, which can be used to update the map content in the global sub-map. Therefore, after the electronic device sends the map update information to the server, the server can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the efficiency of the server to update the global map is improved. Compared with the image data captured by the camera, the map update information has a smaller amount of data, so the transmission bandwidth resources are also saved.
  • the computer may be implemented in whole or in part by software, hardware, firmware or any combination.
  • software it can be implemented in the form of a computer program product in whole or in part.
  • the computer program product includes one or more computer instructions, and when the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part.
  • the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
  • the computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a network site, computer, server, or data center.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer, and may also be a data storage device such as a server or a data center integrated with one or more available media.
  • the usable medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape, etc.), an optical medium (such as a DVD, etc.), or a semiconductor medium (such as a solid-state hard disk), and so on.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Geometry (AREA)
  • Computer Graphics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Remote Sensing (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Processing Or Creating Images (AREA)

Abstract

Provided in the present invention are a virtual object display method, a global map update method, and a device, the global map update method comprising: acquiring from a server a global sub-map associated with the location of an electronic device; on the basis of comparison results of a SLAM map in a second coordinate system and the global sub-map in the second coordinate system, or on the basis of comparison results of a SLAM map in a first coordinate system and the global sub-map in the second coordinate system and coordinate system transformation information, acquiring map update information, the map update information being used for indicating the difference of the SLAM map in the second coordinate system or the first coordinate system relative to the global sub-map in the second coordinate system; and sending the map update information to the server, the map update information being used for updating the global map in the second coordinate system in the server. Implementing the present application can achieve highly efficient global map updates, enhancing the user experience.

Description

虚拟物体显示方法、全局地图更新方法以及设备Virtual object display method, global map update method and equipment
本申请要求于2019年11月08日提交到中国专利局、申请号为201911097393.1,以及2019年11月11日提交到中国专利局、申请号为201911099310.2、申请名称为“虚拟物体显示方法、全局地图更新方法以及设备”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。This application is required to be submitted to the Chinese Patent Office on November 8, 2019, with the application number of 201911097393.1, and on November 11, 2019, with the application number of 201911099310. 2. The application name is "Virtual Object Display Method, Global Map The priority of the Chinese patent application of "Update Method and Equipment", the entire content of which is incorporated in this application by reference.
技术领域Technical field
本申请涉及虚拟场景技术领域,尤其涉及虚拟物体显示方法、全局地图更新方法以及电子设备和服务器。This application relates to the technical field of virtual scenes, in particular to virtual object display methods, global map updating methods, electronic equipment and servers.
背景技术Background technique
虚拟现实(Virtual Reality,VR)、增强现实(Augmented Reality,AR)和混合现实(Mixed Reality,MR)技术是近年来新兴的多媒体虚拟场景技术。其中,VR技术是一种可以创建和体验虚拟世界的仿真技术,AR技术是一种可以将虚拟现实和真实世界叠加并进行互动的技术。MR技术是通过合并真实世界和虚拟世界而产生的新的可视化环境,并且在真实世界、虚拟世界和用户之间搭起一个交互反馈的信息回路的综合技术。Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR) technologies are emerging multimedia virtual scene technologies in recent years. Among them, VR technology is a simulation technology that can create and experience a virtual world, and AR technology is a technology that can superimpose and interact between virtual reality and the real world. MR technology is a new visualization environment generated by merging the real world and virtual world, and a comprehensive technology that builds an interactive feedback information loop between the real world, virtual world and users.
在上述虚拟场景技术中,通常采用同时定位与建图(Simultaneous Localization and Mapping,SLAM)技术进行电子设备自身在环境中的定位。SLAM技术具体可实现电子设备(例如手机、VR眼镜等移动电子设备)在环境中从一个未知位置开始移动时,在移动过程中根据位置估计和地图进行自身定位,同时在自身定位的基础上建造增量式地图,以便于后续的定位,采用SLAM技术的系统或模块又可称为空间定位引擎。In the above virtual scene technology, the simultaneous positioning and mapping (Simultaneous Localization and Mapping, SLAM) technology is usually used to locate the electronic device itself in the environment. SLAM technology can specifically realize that when electronic devices (such as mobile phones, VR glasses and other mobile electronic devices) start to move from an unknown location in the environment, they can locate themselves according to location estimates and maps during the movement, and at the same time build on the basis of their own positioning. Incremental map to facilitate subsequent positioning. A system or module using SLAM technology can also be called a spatial positioning engine.
在一些城市级交互式VR或AR应用中,全局地图是虚拟物体的载体,是用户之间互联的桥梁。在实现电子设备的局部坐标系与全局地图的全局坐标系同步后,全局坐标系中的虚拟物体就可以在电子设备上显示。然而,全局地图的生成公司创建的全局地图时往往覆盖率有限,而且在现实环境中城市建筑景观经常会发生变化,环境景观随季节发生变化,环境可视度随光照发生变化,等等。这些因素都会造成全局地图中的地图内容和现实环境有差异,从而导致电子设备的全局定位不准确甚至无法定位。因此,如何有效进行全局地图更新是本领域技术人员亟需解决的技术问题。In some city-level interactive VR or AR applications, the global map is the carrier of virtual objects and the bridge between users. After the local coordinate system of the electronic device is synchronized with the global coordinate system of the global map, the virtual objects in the global coordinate system can be displayed on the electronic device. However, the global map created by the global map generation company often has limited coverage, and the urban architectural landscape often changes in the real environment, the environmental landscape changes with the seasons, and the environmental visibility changes with the illumination, and so on. These factors will cause the map content in the global map to be different from the real environment, which will lead to inaccurate or even inaccurate global positioning of electronic devices. Therefore, how to effectively update the global map is a technical problem that needs to be solved urgently by those skilled in the art.
发明内容Summary of the invention
本申请实施例提供了虚拟物体显示方法、全局地图更新方法以及设备,能够实现高效率的全局地图更新,提升用户使用体验。The embodiments of the present application provide a virtual object display method, a global map update method, and a device, which can implement efficient global map update and improve user experience.
第一方面,本申请实施例提供了一种虚拟物体显示方法,应用于具有显示组件(例如显示屏(例如,触摸屏、柔性屏、曲面屏等,或者光学组件)和摄像头的第一电子设备,第一电子设备可以是手持终端(如手机),VR或AR眼镜,无人机,无人车等。该方法包括:检测到用户打开应用的操作;响应于所述操作,从服务器下载第一全局子地图并存储到所述第一电子设备的即时定位与地图构建(SLAM)系统中;所述第一全局子地图是全局地图中与所述 第一电子设备的位置关联的子地图;在所述显示组件上基于所述第一电子设备的位姿数据来显示虚拟物体(或者,在所述显示组件上显示所述虚拟物体的位置与姿态),所述第一电子设备的位姿数据是所述SLAM系统至少根据所述摄像头采集的视频图像和所述第一全局子地图执行位姿计算得到的,其中所述服务器上的全局地图是根据一个或多个第二电子设备发送的地图更新信息进行更新得到的,其中,所述一个或多个第二电子设备为除所述第一电子设备外的电子设备,第二电子设备例如也可以是手持终端(如手机),VR或AR眼镜,无人机,无人车等。In the first aspect, the embodiments of the present application provide a virtual object display method, which is applied to a first electronic device having a display component (such as a display screen (for example, a touch screen, a flexible screen, a curved screen, etc., or an optical component)) and a camera. The first electronic device can be a handheld terminal (such as a mobile phone), VR or AR glasses, a drone, an unmanned vehicle, etc. The method includes: detecting a user's operation to open the application; in response to the operation, downloading the first The global sub-map is stored in the real-time positioning and mapping (SLAM) system of the first electronic device; the first global sub-map is a sub-map associated with the location of the first electronic device in the global map; The display component displays a virtual object based on the pose data of the first electronic device (or displays the position and posture of the virtual object on the display component), and the pose data of the first electronic device It is obtained by the SLAM system performing pose calculation at least according to the video image collected by the camera and the first global sub-map, wherein the global map on the server is based on the map sent by one or more second electronic devices Update information is obtained by updating, wherein the one or more second electronic devices are electronic devices other than the first electronic device, and the second electronic device may also be a handheld terminal (such as a mobile phone), VR or AR, for example. Glasses, drones, unmanned vehicles, etc.
其中,所谓“打开应用的操作”可以是通过点击、触摸、滑动、或抖动等方式打开应用,也可以是声控或者其它途径打开应用,本申请对此不作限定。举例来说,电子设备检测到用户的触摸操作后,应用中的导航功能被启动,摄像头被启动等等。Among them, the so-called "operation to open the application" can be to open the application by clicking, touching, sliding, or shaking, or by voice control or other means to open the application, which is not limited in this application. For example, after the electronic device detects the user's touch operation, the navigation function in the application is activated, the camera is activated, and so on.
其中,服务器可作为向第一电子设备和第二电子设备的VR或AR或MR应用提供内容和信息支撑的平台。在服务器中存储有全局地图,通常来讲,全局地图是一种大地理范围的高精度的地图,所谓“大地理范围”是相对于电子设备中的SLAM地图所代表的地理范围而言的,例如全局地图可以是由一个或多个第二电子设备生成的多个SLAM地图的地图更新信息按照一定的规则进行融合得到的。相应的,全局子地图表示全局地图中与第一电子设备的位置关联的子地图(亦称为与第一电子设备的位置关联的子地图),也就是说,可以以第一电子设备的实际位置在全局地图中的位置点为起点,获取起点周围预设区域内的地图内容作为所述全局子地图。Among them, the server can be used as a platform for providing content and information support to the VR or AR or MR applications of the first electronic device and the second electronic device. A global map is stored in the server. Generally speaking, the global map is a high-precision map with a large geographic area. The so-called "large geographic area" is relative to the geographic area represented by the SLAM map in the electronic device. For example, the global map may be obtained by fusing map update information of multiple SLAM maps generated by one or more second electronic devices according to certain rules. Correspondingly, the global sub-map refers to the sub-map associated with the location of the first electronic device in the global map (also referred to as the sub-map associated with the location of the first electronic device), that is, the actual location of the first electronic device can be The location point of the location in the global map is the starting point, and the map content in the preset area around the starting point is acquired as the global sub-map.
应当理解的是,本申请不限于用户触发,也可以是第一电子设备自动检测是否符合下载全局子地图(或启动下载全局子地图)的条件,即本申请又一种可能的实现中,第一电子设备也可以通过其他方式来启动下载全局子地图的步骤,例如,通过检测环境光线的变化或者检测环境的变化来启动下载全局子地图的步骤。It should be understood that this application is not limited to user triggering. It can also be that the first electronic device automatically detects whether it meets the conditions for downloading the global sub-map (or starting the download of the global sub-map), that is, in another possible implementation of this application, the first An electronic device may also initiate the step of downloading the global sub-map in other ways, for example, by detecting changes in ambient light or detecting changes in the environment to initiate the step of downloading the global sub-map.
本实施例中,第一电子设备可安装有VR或AR或MR应用等虚拟场景应用,并可基于用户的操作(例如点击、触摸、滑动、抖动、声控等)运行该VR或AR或MR应用。第一电子设备可通过本地摄像头采集环境中的视频图像,结合采集的视频图像以及所下载的全局子地图确定电子设备的当前位姿,进而基于第一电子设备的当前位姿在显示组件上显示虚拟物体的位置与姿态。虚拟物体相应可以为VR场景、AR场景或MR场景中的虚拟物体(即虚拟环境中的物体)。In this embodiment, the first electronic device can be installed with virtual scene applications such as VR or AR or MR applications, and can run the VR or AR or MR applications based on user operations (such as clicking, touching, sliding, shaking, voice control, etc.) . The first electronic device can collect video images in the environment through a local camera, combine the collected video images and the downloaded global submap to determine the current pose of the electronic device, and then display it on the display component based on the current pose of the first electronic device The position and posture of the virtual object. The virtual object may correspondingly be a virtual object in a VR scene, an AR scene, or an MR scene (that is, an object in a virtual environment).
其中,所述地图更新信息用于指示所述第二坐标系下或者所述第一坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异。所述第一坐标系下是所述SLAM系统的SLAM地图的坐标系,所述第二坐标系是所述全局子地图的坐标系,这里的第一或第二仅是为了方便区分而已。The map update information is used to indicate the difference between the SLAM map in the second coordinate system or the first coordinate system with respect to the global sub-map in the second coordinate system. The first coordinate system is the coordinate system of the SLAM map of the SLAM system, the second coordinate system is the coordinate system of the global sub-map, and the first or second here is only for the convenience of distinction.
本申请实施例中,充分利用了各个第二电子设备的计算资源来计算获得地图更新信息,减轻服务器计算负载。而地图更新信息相比起摄像头拍摄的图像数据而言,数据量更少,节省了传输带宽资源,而服务器能够基于一个或多个第二电子设备的地图更新信息快速实现全局地图更新,更新效率高。由于服务器中的全局地图能够被高效、准确地更新,所以也促进了第一电子设备的全局定位的准确度更高,鲁棒性更好。第一电子设备检测到用户打开应用的操作后,通过请求服务器下载全局子地图,利用比SLAM地图精度更高的全局子地图作为视 觉观测输入给SLAM系统,SLAM系统使用全局子地图和采集的视频图像来估计电子设备的位姿,确保在VR或AR或MR应用的长时间(例如超过1分钟)运行中虚拟物体在电子设备的显示位置和方向不会偏移,长时间正确地显示虚拟物体(例如,在视频图像所表示的环境和时长下,以相对准确的方式显示虚拟物体),从而提升用户的使用体验。In the embodiment of the present application, the computing resources of each second electronic device are fully utilized to calculate and obtain map update information, thereby reducing the computing load of the server. Compared with the image data taken by the camera, the map update information has less data volume, which saves transmission bandwidth resources. The server can quickly implement global map update based on the map update information of one or more second electronic devices, and update efficiency high. Since the global map in the server can be updated efficiently and accurately, it also promotes the global positioning of the first electronic device to have higher accuracy and better robustness. After the first electronic device detects the user's operation to open the application, it requests the server to download the global sub-map, and uses the global sub-map with higher accuracy than the SLAM map as the visual observation input to the SLAM system. The SLAM system uses the global sub-map and the collected video The image is used to estimate the pose of the electronic device to ensure that the display position and direction of the virtual object on the electronic device will not shift during the long-term (for example, more than 1 minute) operation of the VR or AR or MR application, and the virtual object will be displayed correctly for a long time (For example, under the environment and duration represented by the video image, the virtual objects are displayed in a relatively accurate manner), thereby enhancing the user experience.
其中,所述第一电子设备的位姿数据可以为第一坐标系下的所述第一电子设备的位姿数据,或者,第二坐标系下的所述第一电子设备的位姿数据;所述第一坐标系下是所述SLAM系统的SLAM地图的坐标系,所述第二坐标系是所述全局子地图的坐标系,这里的第一或第二仅是为了方便区分而已。Wherein, the pose data of the first electronic device may be the pose data of the first electronic device in a first coordinate system, or the pose data of the first electronic device in a second coordinate system; The first coordinate system is the coordinate system of the SLAM map of the SLAM system, the second coordinate system is the coordinate system of the global sub-map, and the first or second here is only for the convenience of distinction.
本文中,构建SLAM地图所采用的坐标系可称为第一坐标系,本文中的第一坐标系在某些应用场景下也可能被称为局部坐标系、SLAM坐标系、相机坐标系或其他某个合适的术语。相应的,电子设备在局部坐标系下所体现出的位姿(位姿数据)可以称为局部位姿。In this article, the coordinate system used to construct the SLAM map can be called the first coordinate system. The first coordinate system in this article may also be called the local coordinate system, the SLAM coordinate system, the camera coordinate system or others in some application scenarios. Some suitable term. Correspondingly, the pose (pose data) embodied by the electronic device in the local coordinate system can be called a local pose.
构建该全局地图所采用的坐标系可称为第二坐标系,本文中的第二坐标系在某些应用场景下也可能被称为全局坐标系、世界坐标系或其他某个合适的术语。相应的,电子设备在全局坐标系下所体现出的位姿(位姿数据)可以称为全局位姿。The coordinate system used to construct the global map may be called the second coordinate system. In some application scenarios, the second coordinate system may also be called the global coordinate system, the world coordinate system, or some other appropriate terminology. Correspondingly, the pose (pose data) embodied by the electronic device in the global coordinate system can be referred to as a global pose.
基于第一方面,在一种实现方案中,所述地图更新信息是所述第二电子设备的SLAM系统根据第二坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果获得的;其中,所述第二坐标系下的SLAM地图是基于所述电子设备的SLAM系统所构建的第一坐标系下的SLAM地图以及所述第一坐标系和所述第二坐标系之间的坐标系变换信息得到的;所述比较结果体现了所述第二坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异,SLAM系统根据所述差异能够进一步获得所述地图更新信息。Based on the first aspect, in an implementation solution, the map update information is a comparison between the SLAM system of the second electronic device according to the SLAM map in the second coordinate system and the global sub-map in the second coordinate system The result is obtained; wherein the SLAM map in the second coordinate system is based on the SLAM map in the first coordinate system constructed by the SLAM system of the electronic device, and the first coordinate system and the second coordinate system The comparison result reflects the difference between the SLAM map in the second coordinate system and the global sub-map in the second coordinate system, and the SLAM system can further Obtain the map update information.
在另一种实现方案中,所述地图更新信息是所述第二电子设备的SLAM系统根据所述第一坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果以及所述坐标系变换信息获得的,所述比较结果体现了所述第一坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异,SLAM系统可将所述差异转换到第二坐标系下去表示,根据第二坐标系下的差异能够进一步获得所述地图更新信息。In another implementation solution, the map update information is a comparison result of the SLAM system of the second electronic device according to the SLAM map in the first coordinate system and the global sub-map in the second coordinate system, and Obtained from the coordinate system transformation information, the comparison result reflects the difference between the SLAM map in the first coordinate system and the global sub-map in the second coordinate system, and the SLAM system can convert the difference to The second coordinate system continues to indicate that the map update information can be further obtained according to the difference in the second coordinate system.
其中,SLAM系统可利用关键帧来构建SLAM地图,SLAM地图可以理解为SLAM系统根据采集设备采集到的环境信息所绘制出的地图,采集设备例如可以包括电子设备中的图像采集装置(例如摄像头或相机)和惯性测量单元(Inertial Measurement Unit,IMU),IMU中可以包括:陀螺仪、加速度计等传感器。Among them, the SLAM system can use key frames to construct a SLAM map. The SLAM map can be understood as a map drawn by the SLAM system according to the environmental information collected by the collection device. The collection device can include, for example, an image collection device in an electronic device (such as a camera or Camera) and Inertial Measurement Unit (IMU). The IMU may include sensors such as gyroscopes and accelerometers.
SLAM地图可以包含以下地图内容:多个关键帧、三角测量的特征点以及关键帧和特征点之间的关联。关键帧可以基于摄像头采集的图像和用来产生图像的相机参数(例如,电子设备在SLAM坐标系中的位姿)而形成。如本文中所用的特征点(即特征)是作为图像的令人感兴趣的或显著的一部分。其中所述特征点可以表示SLAM地图中沿着三维空间的不同的3D地图点以及在3D地图点上的特征描述。每一特征点可具有关联的特征位置。每个特征点可表示3D坐标位置,并且与一或多个描述符相关联。特征点又可称为3D特征、特征点、3D特征点或者其他合适的名字。The SLAM map can contain the following map content: multiple key frames, triangulated feature points, and the association between key frames and feature points. The key frame may be formed based on the image collected by the camera and the camera parameters used to generate the image (for example, the pose of the electronic device in the SLAM coordinate system). A feature point (ie, a feature) as used herein is an interesting or significant part of an image. The feature points may represent different 3D map points along the three-dimensional space in the SLAM map and the feature descriptions on the 3D map points. Each feature point can have an associated feature location. Each feature point can represent a 3D coordinate position and is associated with one or more descriptors. Feature points can also be called 3D features, feature points, 3D feature points or other suitable names.
例如,当电子设备拍摄到一张图像后,是否将该图像作为新的关键帧加入到SLAM地图,可以从新的关键帧自身和新的关键帧与SLAM地图中的已有关键帧之间的关系这两方面来考 虑。也就是说,一方面,新的关键帧自身图像质量要好,比如不能是非常模糊的图像、特征点数量要充足、特征点分布要尽量均匀等等;另一方面,对于新的关键帧需要与SLAM地图中的已有关键帧有少量的共视关系,且大部分特征点是新特征点,以达到既存在约束,又尽量少的信息冗余的效果,比如SLAM地图中的一个邻近关键帧的特征点在该新的关键帧里有10%能观测到,剩余90%观测不到,等等。For example, when an electronic device captures an image, whether to add the image as a new key frame to the SLAM map, you can find the relationship between the new key frame itself and the relationship between the new key frame and the existing key frame in the SLAM map Consider these two aspects. That is to say, on the one hand, the image quality of the new key frame itself should be good, for example, it cannot be a very blurry image, the number of feature points should be sufficient, and the distribution of feature points should be as uniform as possible; on the other hand, the new key frame needs to be The existing key frames in the SLAM map have a small number of common-view relationships, and most of the feature points are new feature points to achieve the effect of both existing constraints and as little information redundancy as possible, such as a neighboring key frame in the SLAM map In the new key frame, 10% of the feature points can be observed, and the remaining 90% cannot be observed, and so on.
其中,3D地图点(或称三维地图点或称三维空间坐标点)表示在三维空间轴线X、Y及Z上的坐标,例如,对于局部坐标系下的SLAM地图,3D地图点表示在局部坐标系三维空间轴线X、Y及Z上的坐标。对于全局坐标系下的SLAM地图,3D地图点表示在全局坐标系三维空间轴线X、Y及Z上的坐标。Among them, 3D map points (or three-dimensional map points or three-dimensional space coordinate points) represent the coordinates on the three-dimensional space axes X, Y, and Z. For example, for a SLAM map in a local coordinate system, the 3D map points are expressed in local coordinates It is the coordinates on the three-dimensional space axis X, Y, and Z. For the SLAM map in the global coordinate system, the 3D map points represent the coordinates on the three-dimensional axis X, Y, and Z of the global coordinate system.
为了描述上的方便,本申请实施例中,所谓“关键帧的3D地图点”或者所谓“非冗余关键帧的3D地图点”中,3D地图点的概念可以包含了二维图像中的视觉特征信息。For ease of description, in the embodiments of the present application, in the so-called "3D map points in key frames" or the so-called "3D map points in non-redundant key frames", the concept of 3D map points may include the vision in the two-dimensional image. Characteristic information.
基于第一方面,在可能的实施方式中,所述地图更新信息包括:由所述第二电子设备的SLAM系统所构建的SLAM地图的目标关键帧对应的位姿数据、三维(3D)地图点以及共视数据,所述共视数据用于指示所述目标关键帧与所述第二电子设备中的第二全局子地图的关键帧之间的共视关系。所述第二全局子地图是全局地图中与所述第二电子设备的位置关联的子地图。Based on the first aspect, in a possible implementation manner, the map update information includes: pose data corresponding to the target key frame of the SLAM map constructed by the SLAM system of the second electronic device, and three-dimensional (3D) map points And common-view data, the common-view data is used to indicate the common-view relationship between the target key frame and the key frame of the second global sub-map in the second electronic device. The second global sub-map is a sub-map associated with the location of the second electronic device in the global map.
具体的,所述目标关键帧为非冗余关键帧,所述非冗余关键帧表示存在于所述第二电子设备的SLAM地图中但不存在于所述第二全局子地图中的关键帧。Specifically, the target key frame is a non-redundant key frame, and the non-redundant key frame represents a key frame that exists in the SLAM map of the second electronic device but does not exist in the second global sub-map .
例如,地图更新信息可以是该目标关键帧被转换到全局坐标系下后再融合进全局子地图后的相关信息(即目标关键帧对应的位姿、共视关系、3D地图点等信息)。即地图更新信息是经融合的全局子地图中相比起原先的全局子地图而言,被更新的那一部分信息。For example, the map update information may be related information after the target key frame is converted to the global coordinate system and then merged into the global sub-map (ie, the pose, common view relationship, 3D map point, etc. information corresponding to the target key frame). That is, the map update information is the updated part of the global sub-map compared to the original global sub-map.
本申请实施例中,地图更新信息只需包含所变化的关键帧位姿、共视关系以及3D地图点,相比原先下载的全局子地图而言数据量会小很多,所以进一步节省了传输带宽资源。第二电子设备将地图更新信息发送给服务器后,服务器就能够基于地图更新信息实现对服务器中的全局地图的更新。这样,减轻服务器计算负载,且提高了全局地图的更新效率和鲁棒性。In the embodiment of this application, the map update information only needs to include the changed key frame pose, common view relationship, and 3D map points. Compared with the previously downloaded global sub-map, the amount of data will be much smaller, so the transmission bandwidth is further saved. Resources. After the second electronic device sends the map update information to the server, the server can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the update efficiency and robustness of the global map are improved.
基于第一方面,在可能的实施方式中,所述地图更新信息包括:存在于所述第二电子设备的SLAM地图中但不存在于所述第二全局子地图中的关键帧。所述第二全局子地图是全局地图中与所述第二电子设备的位置关联的子地图。Based on the first aspect, in a possible implementation manner, the map update information includes: key frames existing in the SLAM map of the second electronic device but not in the second global submap. The second global sub-map is a sub-map associated with the location of the second electronic device in the global map.
本申请实施例中,地图更新信息可以直接为非冗余关键帧,第二电子设备识别出该SLAM地图的非冗余关键帧后,并将非冗余关键帧发送给服务器,服务器再利用非冗余关键帧融合到全局地图,从而也能够实现对服务器中的全局地图的高效率的更新。这样的方案有利于减轻电子设备的计算负载,保证地图融合的准确性。In the embodiment of the present application, the map update information can be directly non-redundant key frames. After the second electronic device recognizes the non-redundant key frames of the SLAM map, it sends the non-redundant key frames to the server, and the server reuses the non-redundant key frames. The redundant key frames are merged into the global map, so that the global map in the server can be updated efficiently. Such a solution is beneficial to reduce the computational load of electronic equipment and ensure the accuracy of map fusion.
基于第一方面,在可能的实施方式中,所述响应于所述操作,从服务器下载第一全局子地图,包括:向服务器发送用于指示所述第一电子设备的初始位置的第一位置指纹信息;从所述服务器接收所述第一全局子地图,所述第一全局子地图对应有第二位置指纹信息,且所述第一位置指纹信息和所述第二位置指纹信息相匹配。通过服务器执行地图匹配操作,提高了匹配效率和精准度,进而有利于减少下载地图的时延。Based on the first aspect, in a possible implementation manner, the downloading the first global submap from the server in response to the operation includes: sending to the server a first position indicating the initial position of the first electronic device Fingerprint information; receiving the first global submap from the server, the first global submap corresponding to second location fingerprint information, and the first location fingerprint information matches the second location fingerprint information. Performing the map matching operation through the server improves the matching efficiency and accuracy, which in turn helps to reduce the delay in downloading the map.
第二方面,本申请实施例提供了一种全局地图更新方法,应用于电子设备,电子设备可 以是手持终端(如手机),VR或AR眼镜,无人机,无人车等。该方法包括:从服务器获取与所述电子设备的位置对应的全局子地图,所述全局子地图是第二坐标系下的全局地图中与所述电子设备的位置关联的子地图;根据所述电子设备的SLAM系统所构建的第二坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果,或者根据所述电子设备的SLAM系统所构建的第一坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果以及所述坐标系变换信息,获得地图更新信息,其中所述第二坐标系下的SLAM地图是基于所述第一坐标系下的SLAM地图以及所述第一坐标系和所述第二坐标系之间的坐标系变换信息得到的,所述地图更新信息用于指示所述第二坐标系下或者所述第一坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异;将所述地图更新信息发送给所述服务器,所述地图更新信息用于更新所述服务器中所述第二坐标系下的全局地图。In the second aspect, the embodiments of the present application provide a global map update method, which is applied to electronic devices, which can be handheld terminals (such as mobile phones), VR or AR glasses, drones, unmanned vehicles, etc. The method includes: obtaining a global sub-map corresponding to the location of the electronic device from a server, the global sub-map being a sub-map associated with the location of the electronic device in a global map in a second coordinate system; The comparison result of the SLAM map in the second coordinate system constructed by the SLAM system of the electronic device and the global sub-map in the second coordinate system, or the comparison result in the first coordinate system constructed by the SLAM system of the electronic device The comparison result of the SLAM map and the global sub-map in the second coordinate system and the coordinate system transformation information obtain map update information, wherein the SLAM map in the second coordinate system is based on the first coordinate system The SLAM map and the coordinate system transformation information between the first coordinate system and the second coordinate system are obtained, and the map update information is used to indicate the second coordinate system or the first coordinate system The difference between the SLAM map in the second coordinate system and the global sub-map in the second coordinate system; the map update information is sent to the server, and the map update information is used to update the second coordinate system in the server Global map.
其中,当方案实现为“根据第二坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果,获得地图更新信息”时,所述地图更新信息用于指示所述第二坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异。也就是说,所述第二坐标系下的SLAM地图是基于第一坐标系下的SLAM地图以及所述第一坐标系和所述第二坐标系之间的坐标系变换信息得到的,所述比较结果体现了所述第二坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异,SLAM系统根据所述差异能够进一步获得所述地图更新信息。Wherein, when the solution is implemented as "according to the comparison result of the SLAM map in the second coordinate system and the global sub-map in the second coordinate system, map update information is obtained", the map update information is used to indicate the second coordinate system. The difference between the SLAM map in the second coordinate system and the global sub-map in the second coordinate system. That is, the SLAM map in the second coordinate system is obtained based on the SLAM map in the first coordinate system and the coordinate system transformation information between the first coordinate system and the second coordinate system. The comparison result reflects the difference between the SLAM map in the second coordinate system and the global sub-map in the second coordinate system, and the SLAM system can further obtain the map update information according to the difference.
当方案实现为“根据第一坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果以及所述坐标系变换信息,获得地图更新信息”时,所述地图更新信息用于指示所述第一坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异。所述比较结果即体现了所述第一坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异。在这里的“差异”是在第一坐标系下所表示的差异,而SLAM系统可以将所述差异基于所述第一坐标系和所述第二坐标系之间的坐标系变换信息进一步转换到第二坐标系下去表示,且根据第二坐标系下的差异能够进一步获得所述地图更新信息。所以,所述地图更新信息能够用于“指示所述差异”。When the solution is implemented as "obtain map update information based on the comparison result of the SLAM map in the first coordinate system and the global submap in the second coordinate system and the coordinate system transformation information", the map update information is used To indicate the difference between the SLAM map in the first coordinate system and the global sub-map in the second coordinate system. The comparison result reflects the difference between the SLAM map in the first coordinate system and the global sub-map in the second coordinate system. The “difference” here is the difference expressed in the first coordinate system, and the SLAM system can further convert the difference to the information based on the coordinate system transformation information between the first coordinate system and the second coordinate system. The second coordinate system is further represented, and the map update information can be further obtained according to the difference in the second coordinate system. Therefore, the map update information can be used to "indicate the difference".
其中,服务器可作为向电子设备的VR或AR或MR应用提供内容和信息支撑的平台。在服务器中存储有全局地图,通常来讲,全局地图是一种大地理范围的高精度的地图,所谓“大地理范围”是相对于电子设备中的SLAM地图所代表的地理范围而言的,例如全局地图可以是由一个或多个电子设备生成的多个SLAM地图按照一定的规则进行整合得到的。相应的,全局子地图表示全局地图中与电子设备的位置关联的子地图,也就是说,可以以电子设备的实际位置在全局地图中的位置点为起点,获取起点周围预设区域内的地图内容作为所述全局子地图。Among them, the server can be used as a platform for providing content and information support to VR or AR or MR applications of electronic devices. A global map is stored in the server. Generally speaking, the global map is a high-precision map with a large geographic area. The so-called "large geographic area" is relative to the geographic area represented by the SLAM map in the electronic device. For example, the global map may be obtained by integrating multiple SLAM maps generated by one or more electronic devices according to certain rules. Correspondingly, the global sub-map refers to the sub-map associated with the location of the electronic device in the global map, that is to say, the actual location of the electronic device can be taken as the starting point to obtain the map in the preset area around the starting point. The content serves as the global sub-map.
本申请实施例中,电子设备从服务器下载全局地图中全局子地图,通过比较全局子地图和电子设备中的SLAM地图,利用电子设备侧的计算资源和运算能力来获得地图更新信息,该地图更新信息能够被用来更新全局子地图中的地图内容。所以电子设备将地图更新信息发送给服务器后,服务器就能够基于地图更新信息实现对服务器中的全局地图更新。这样,减轻服务器计算负载,提高了服务器更新全局地图的效率。而地图更新信息相比起摄像头拍摄的图像数据而言,数据量更少,所以也节省了传输带宽资源。In the embodiment of the present application, the electronic device downloads the global sub-map in the global map from the server, compares the global sub-map with the SLAM map in the electronic device, and obtains map update information by using the computing resources and computing capabilities of the electronic device. The information can be used to update the map content in the global submap. Therefore, after the electronic device sends the map update information to the server, the server can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the efficiency of the server to update the global map is improved. Compared with the image data captured by the camera, the map update information has a smaller amount of data, so the transmission bandwidth resources are also saved.
基于第二方面,在可能的实施方式中,所述第二坐标系下的SLAM地图具体为根据基于所 述坐标系变换信息对所述第一坐标系下的SLAM地图进行变换得到的;所述根据所述电子设备的SLAM系统所构建的第二坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果,获得地图更新信息,包括:通过比较所述第二坐标系下的SLAM地图与所述第二坐标系下的全局子地图,以获得非冗余关键帧,所述非冗余关键帧表示存在于所述第二坐标系下的SLAM地图中但不存在于所述第二坐标系下的全局子地图中的关键帧;将所述非冗余关键帧融合到所述第二坐标系下的全局子地图,得到经融合后的全局子地图;从所述经融合后的全局子地图中获得所述地图更新信息。Based on the second aspect, in a possible implementation manner, the SLAM map in the second coordinate system is specifically obtained by transforming the SLAM map in the first coordinate system based on the coordinate system transformation information; Obtaining map update information according to the comparison result of the SLAM map in the second coordinate system constructed by the SLAM system of the electronic device and the global sub-map in the second coordinate system includes: comparing the second coordinate system Under the SLAM map and the global sub-map under the second coordinate system to obtain non-redundant key frames, the non-redundant key frames indicate that they exist in the SLAM map under the second coordinate system but do not exist in the Key frames in the global sub-map in the second coordinate system; fusing the non-redundant key frames to the global sub-map in the second coordinate system to obtain the fused global sub-map; from the The map update information is obtained from the global sub-map after fusion.
其中,SLAM系统可利用关键帧来构建SLAM地图,SLAM地图可以理解为SLAM系统根据采集设备采集到的环境信息所绘制出的地图,SLAM地图可以包含以下地图内容:多个关键帧、三角测量的特征点以及关键帧和特征点之间的关联。关键帧可以基于摄像头采集的图像和用来产生图像的相机参数(例如,电子设备在SLAM坐标系中的位姿)而形成。如本文中所用的特征点(即特征)是作为图像的令人感兴趣的或显著的一部分。其中所述特征点可以表示SLAM地图中沿着三维空间的不同的3D地图点以及在3D地图点上的特征描述。每一特征点可具有关联的特征位置。每个特征点可表示3D坐标位置,并且与一或多个描述符相关联。Among them, the SLAM system can use key frames to construct a SLAM map. The SLAM map can be understood as a map drawn by the SLAM system according to the environmental information collected by the collection device. The SLAM map can contain the following map content: multiple key frames, triangulation Feature points and the association between key frames and feature points. The key frame may be formed based on the image collected by the camera and the camera parameters used to generate the image (for example, the pose of the electronic device in the SLAM coordinate system). A feature point (ie, a feature) as used herein is an interesting or significant part of an image. The feature points may represent different 3D map points along the three-dimensional space in the SLAM map and the feature descriptions on the 3D map points. Each feature point can have an associated feature location. Each feature point can represent a 3D coordinate position and is associated with one or more descriptors.
其中,3D地图点(或称三维地图点)表示在三维空间轴线X、Y及Z上的坐标,例如,对于局部坐标系下的SLAM地图,3D地图点表示在局部坐标系三维空间轴线X、Y及Z上的坐标。对于全局坐标系下的SLAM地图,3D地图点表示在全局坐标系三维空间轴线X、Y及Z上的坐标。Among them, 3D map points (or three-dimensional map points) represent the coordinates on the three-dimensional space axes X, Y, and Z. For example, for a SLAM map in the local coordinate system, the 3D map points represent the coordinates on the three-dimensional space axes X, Y, and Z of the local coordinate system. Coordinates on Y and Z. For the SLAM map in the global coordinate system, the 3D map points represent the coordinates on the three-dimensional axis X, Y, and Z of the global coordinate system.
为了描述上的方便,本申请实施例中,所谓“关键帧的3D地图点”或者所谓“所谓“非冗余关键帧的3D地图点”中,3D地图点的概念可以包含了二维图像中的视觉特征信息。For the convenience of description, in the embodiments of the present application, in the so-called "3D map points in key frames" or in the so-called "3D map points in non-redundant key frames", the concept of 3D map points may include the two-dimensional image Visual feature information.
本申请一种实施例中,SLAM地图中的关键帧和地图点可以变换到全局坐标系中表示,SLAM地图可以也变换到全局坐标系中表示。在SLAM系统运行过程中,会不断更新SLAM地图,所谓“更新SLAM地图”体现在不断向全局坐标系下的SLAM地图插入关键帧。当有一个新的关键帧插入或者即将要插入到所述SLAM地图时,电子设备判断SLAM系统插入的关键帧是否与下载的全局子地图中的关键帧存在冗余。所谓关键帧的冗余指关键帧关联的3D地图点信息在全局子地图中已经存在,即全局子地图已经保存了当前环境中的视觉信息,不需要再增加该关键帧了。而关键帧不冗余即关键帧关联的3D地图点信息在全局子地图中未存在,全局子地图需要增加该关键帧来实现更新,以便于地图内容与真实环境同步。本申请实施例中,所述全局坐标系下的SLAM地图中不与全局子地图中的关键帧存在冗余的关键帧可称为非冗余关键帧。后续可将所述非冗余关键帧融合到全局坐标系下的全局子地图。In an embodiment of the present application, the key frames and map points in the SLAM map can be transformed into a global coordinate system for representation, and the SLAM map can also be transformed into a global coordinate system for representation. During the operation of the SLAM system, the SLAM map will be continuously updated. The so-called "update of the SLAM map" is embodied in the continuous insertion of key frames into the SLAM map in the global coordinate system. When a new key frame is inserted or is about to be inserted into the SLAM map, the electronic device determines whether the key frame inserted by the SLAM system is redundant with the key frame in the downloaded global sub-map. The so-called redundancy of key frames means that the 3D map point information associated with the key frames already exists in the global sub-map, that is, the global sub-map has saved the visual information in the current environment, and there is no need to add the key frame. The key frame is not redundant, that is, the 3D map point information associated with the key frame does not exist in the global sub-map, and the global sub-map needs to add the key frame to realize the update, so that the map content can be synchronized with the real environment. In the embodiment of the present application, the key frames in the SLAM map in the global coordinate system that are not redundant with the key frames in the global sub-map may be referred to as non-redundant key frames. Subsequently, the non-redundant key frames can be merged into the global sub-map in the global coordinate system.
其中,将所述非冗余关键帧融合到所述全局坐标系下的全局子地图后,所获得所述地图更新信息可以包括非冗余关键帧对应的位姿、共视关系、3D地图点等信息,即地图更新信息是经融合的全局子地图中相比起原先的全局子地图而言,被更新的那一部分信息。所以电子设备可以从所述经融合后的全局子地图中获得所述地图更新信息以及发送给服务器。Wherein, after the non-redundant key frame is merged into the global sub-map in the global coordinate system, the obtained map update information may include the pose, common view relationship, and 3D map point corresponding to the non-redundant key frame. Equal information, that is, the map update information is the part of the updated global sub-map compared to the original global sub-map in the fused global sub-map. Therefore, the electronic device can obtain the map update information from the merged global sub-map and send it to the server.
本申请又一种实施例中,在SLAM系统运行过程中,当有一个新的关键帧插入或者即将要插入到局部坐标系下的SLAM地图时,电子设备判断SLAM系统插入的关键帧是否与下载的全局子地图中的关键帧存在冗余,即是否存在非冗余关键帧,非冗余关键帧表示所述局部坐标系下的SLAM地图中不与全局子地图中的关键帧存在冗余的关键帧。若存在,则可将所述非冗 余关键帧变换到全局坐标系下表示,并将所述全局坐标系下的非冗余关键帧融合到全局坐标系下的全局子地图。In another embodiment of the present application, during the operation of the SLAM system, when a new key frame is inserted or is about to be inserted into the SLAM map in the local coordinate system, the electronic device determines whether the key frame inserted by the SLAM system is the same as the download The key frames in the global sub-map are redundant, that is, whether there are non-redundant key frames. The non-redundant key frames indicate that the SLAM map in the local coordinate system does not have redundancy with the key frames in the global sub-map Keyframe. If it exists, the non-redundant key frame can be transformed into the global coordinate system for representation, and the non-redundant key frame in the global coordinate system can be merged into the global sub-map in the global coordinate system.
其中,将所述全局坐标系下非冗余关键帧融合到所述全局坐标系下的全局子地图后,所获得所述地图更新信息可以包括所述全局坐标系下的非冗余关键帧对应的位姿、共视关系、3D地图点等信息,即地图更新信息是经融合的全局子地图中相比起原先的全局子地图而言,被更新的那一部分信息。所以电子设备可以从所述经融合后的全局子地图中获得所述地图更新信息以及发送给服务器。Wherein, after fusing the non-redundant key frames in the global coordinate system to the global sub-map in the global coordinate system, the obtained map update information may include the corresponding non-redundant key frames in the global coordinate system The position and pose, common view relationship, 3D map points and other information, that is, the map update information is the part of the information that has been updated in the fused global sub-map compared to the original global sub-map. Therefore, the electronic device can obtain the map update information from the merged global sub-map and send it to the server.
本申请实施例中,地图更新信息可以从经融合后的全局子地图中获得,所以相比原先下载的全局子地图而言数据量会小很多,所以进一步节省了传输带宽资源。第二电子设备将地图更新信息发送给服务器后,服务器就能够基于地图更新信息实现对服务器中的全局地图的更新。这样,减轻服务器计算负载,且提高了全局地图的更新效率和鲁棒性。In the embodiment of the present application, the map update information can be obtained from the global sub-map after fusion, so the data volume will be much smaller than the previously downloaded global sub-map, so the transmission bandwidth resources are further saved. After the second electronic device sends the map update information to the server, the server can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the update efficiency and robustness of the global map are improved.
基于第二方面,在可能的实施方式中,所述将所述非冗余关键帧融合到所述第二坐标系下的全局子地图,包括:将所述非冗余关键帧作为新的关键帧节点增加到在所述第二坐标系下的全局子地图;更新所述非冗余关键帧与所述第二坐标系下的全局子地图中的关键帧的共视关系;计算所述非冗余关键帧对应的位姿数据和三维地图点;相应的,所述从所述经融合后的全局子地图中获得所述地图更新信息,包括:从所述经融合后的全局子地图中获取所述非冗余关键帧对应的位姿数据、三维地图点以及共视数据作为所述地图更新信息(即地图更新信息是经融合的全局子地图中相比起原先的全局子地图而言,被更新的那一部分信息),所述共视数据用于指示所述非冗余关键帧与所述第二坐标系下的全局子地图的关键帧之间的共视关系。Based on the second aspect, in a possible implementation manner, the fusing the non-redundant key frame into the global sub-map in the second coordinate system includes: using the non-redundant key frame as a new key The frame node is added to the global sub-map in the second coordinate system; the common view relationship between the non-redundant key frame and the key frame in the global sub-map in the second coordinate system is updated; the non-redundant key frame is calculated; The pose data and the three-dimensional map points corresponding to the redundant key frames; correspondingly, the obtaining the map update information from the fused global sub-map includes: from the fused global sub-map Obtain the pose data, three-dimensional map points, and common view data corresponding to the non-redundant key frame as the map update information (that is, the map update information is the fused global sub-map compared to the original global sub-map , The updated part of the information), the common view data is used to indicate the common view relationship between the non-redundant key frame and the key frame of the global submap in the second coordinate system.
具体的,对于全局子地图中的某一3D地图点,如果该3D地图点能同时被若干个关键帧观测到,那么该若干个关键帧就具有对该3D地图点的共视关系。所以,电子设备可计算出全局子地图中的哪些3D地图点能同时被全局子地图中的至少一个关键帧和非冗余关键帧观测到,从而更新全局子地图中的关键帧和非冗余关键之间的共视关系。Specifically, for a certain 3D map point in the global submap, if the 3D map point can be observed by several key frames at the same time, then the several key frames have a common view relationship with the 3D map point. Therefore, the electronic device can calculate which 3D map points in the global sub-map can be simultaneously observed by at least one key frame and non-redundant key frame in the global sub-map, thereby updating the key frames and non-redundant key frames in the global sub-map The common-view relationship between the keys.
在更新了全局子地图的关键帧共视关系之后,电子设备可利用三角化算法计算新的3D地图点,以及通过BA算法优化非冗余关键帧对应的位姿和3D地图点坐标,从而完成了将非冗余关键帧融合(或称插入)到全局子地图,实现了该全局子地图的更新,并将非冗余关键帧对应的位姿数据、三维地图点以及共视数据作为所述地图更新信息。After updating the key frame common view relationship of the global sub-map, the electronic device can use the triangulation algorithm to calculate new 3D map points, and use the BA algorithm to optimize the pose and 3D map point coordinates corresponding to the non-redundant key frames to complete In order to integrate (or insert) non-redundant key frames into the global sub-map, the global sub-map is updated, and the pose data, three-dimensional map points and common view data corresponding to the non-redundant key frames are used as the Map update information.
本申请实施例中,地图更新信息只需包含所变化的关键帧位姿、共视关系以及3D地图点,相比原先下载的全局子地图而言数据量会小很多,所以进一步节省了传输带宽资源。第二电子设备将地图更新信息发送给服务器后,服务器就能够基于地图更新信息实现对服务器中的全局地图的更新。这样,减轻服务器计算负载,且提高了全局地图的更新效率和鲁棒性。In the embodiment of this application, the map update information only needs to include the changed key frame pose, common view relationship, and 3D map points. Compared with the previously downloaded global sub-map, the amount of data will be much smaller, so the transmission bandwidth is further saved. Resources. After the second electronic device sends the map update information to the server, the server can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the update efficiency and robustness of the global map are improved.
基于第二方面,在可能的实施方式中,所述根据第一坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果以及所述坐标系变换信息,获得地图更新信息,包括:Based on the second aspect, in a possible implementation manner, the map update information is obtained according to the comparison result of the SLAM map in the first coordinate system and the global sub-map in the second coordinate system and the coordinate system transformation information ,include:
通过比较所述第一坐标系下的SLAM地图与所述第二坐标系下的全局子地图,以获得非冗余关键帧,所述非冗余关键帧为存在于所述第一坐标系下的SLAM地图的关键帧,且所述非冗余关键帧经所述坐标系变换信息变换后不存在于所述第二坐标系下的全局子地图;将经所述坐标系变换信息变换后的非冗余关键帧融合到所述第二坐标系下的全局子地图,以获得所述 地图更新信息。所述经所述坐标系变换信息变换后的非冗余关键帧即是所述第二坐标系下的关键帧。By comparing the SLAM map in the first coordinate system with the global sub-map in the second coordinate system, a non-redundant key frame is obtained, where the non-redundant key frame exists in the first coordinate system The key frame of the SLAM map, and the non-redundant key frame is transformed by the coordinate system transformation information and does not exist in the global sub-map in the second coordinate system; the coordinate system transformation information is transformed Non-redundant key frames are fused to the global sub-map in the second coordinate system to obtain the map update information. The non-redundant key frame transformed by the coordinate system transformation information is the key frame in the second coordinate system.
综上可以看到,本申请实施例中,当方案实现为“根据第二坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果,获得地图更新信息”时,所述地图更新信息用于指示所述第二坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异。所述比较结果体现了所述第二坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异,这种差异可以是第二坐标系下的SLAM地图中的非冗余关键帧,即该非冗余关键帧不存在于全局子地图。SLAM系统后续可以将所述非冗余关键帧融合到全局子地图,从而获得地图更新信息(包括非冗余关键帧对应的位姿数据、三维地图点以及共视数据)发给服务器。SLAM系统也可以直接将所述非冗余关键帧作为地图更新信息发给服务器。In summary, in the embodiment of this application, when the solution is implemented as "according to the comparison result of the SLAM map in the second coordinate system and the global sub-map in the second coordinate system, the map update information is obtained", The map update information is used to indicate the difference between the SLAM map in the second coordinate system and the global sub-map in the second coordinate system. The comparison result reflects the difference between the SLAM map in the second coordinate system and the global sub-map in the second coordinate system, and this difference may be non-redundant in the SLAM map in the second coordinate system Key frame, that is, the non-redundant key frame does not exist in the global submap. The SLAM system can subsequently fuse the non-redundant key frames into the global sub-map, thereby obtaining map update information (including pose data, three-dimensional map points, and common view data corresponding to the non-redundant key frames) and sending it to the server. The SLAM system can also directly send the non-redundant key frame as map update information to the server.
当方案实现为“根据第一坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果以及所述坐标系变换信息,获得地图更新信息”时,所述地图更新信息用于指示所述第一坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异。所述比较结果即体现了所述第一坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异。这种差异可以是第一坐标系下的SLAM地图中的非冗余关键帧。SLAM系统可以将该非冗余关键帧转换到第二坐标系下,获得第二坐标系下的非冗余关键帧。后续,SLAM系统可将第二坐标系下的非冗余关键帧融合到全局子地图,从而获得地图更新信息(包括第二坐标系下的非冗余关键帧对应的位姿数据、三维地图点以及共视数据)发给服务器。SLAM系统也可以直接将所述第二坐标系下的非冗余关键帧作为地图更新信息发给服务器。When the solution is implemented as "obtain map update information based on the comparison result of the SLAM map in the first coordinate system and the global submap in the second coordinate system and the coordinate system transformation information", the map update information is used To indicate the difference between the SLAM map in the first coordinate system and the global sub-map in the second coordinate system. The comparison result reflects the difference between the SLAM map in the first coordinate system and the global sub-map in the second coordinate system. This difference may be non-redundant key frames in the SLAM map in the first coordinate system. The SLAM system can convert the non-redundant key frame to the second coordinate system to obtain the non-redundant key frame in the second coordinate system. Later, the SLAM system can merge the non-redundant key frames in the second coordinate system into the global sub-map to obtain map update information (including the pose data corresponding to the non-redundant key frames in the second coordinate system, and the three-dimensional map points And common view data) to the server. The SLAM system may also directly send the non-redundant key frames in the second coordinate system as map update information to the server.
基于第二方面,在可能的实施方式中,所述从服务器获取与所述电子设备的位置对应的全局子地图,包括:向服务器发送所述电子设备的初始位置的指示信息;从所述服务器接收所述全局子地图,所述全局子地图是根据所述电子设备的初始位置确定的。Based on the second aspect, in a possible implementation manner, the obtaining the global submap corresponding to the location of the electronic device from the server includes: sending indication information of the initial location of the electronic device to the server; Receiving the global sub-map, the global sub-map being determined according to the initial position of the electronic device.
其中,所述电子设备的初始位置的指示信息可包括用于指示所述电子设备的初始位置的第一位置指纹信息;所述全局子地图对应有第二位置指纹信息,且所述第一位置指纹信息和所述第二位置指纹信息相匹配。通过服务器执行地图匹配操作,提高了匹配效率和精准度,进而有利于减少下载地图的时延。Wherein, the indication information of the initial position of the electronic device may include first position fingerprint information for indicating the initial position of the electronic device; the global sub-map corresponds to the second position fingerprint information, and the first position The fingerprint information matches the second location fingerprint information. Performing the map matching operation through the server improves the matching efficiency and accuracy, which in turn helps to reduce the delay in downloading the map.
基于第二方面,在可能的实施方式中,在获得地图更新信息之前,还包括:根据采集的视频图像和第一坐标系下的SLAM地图,确定所述电子设备在所述第一坐标系下的SLAM地图中的第一位姿数据;根据所述视频图像和第二坐标系下的全局子地图,确定所述电子设备在所述第二坐标系下的全局子地图中的第二位姿数据;根据所述第一位姿数据和所述第二位姿数据,获得所述SLAM地图的第一坐标系和所述全局地图的第二坐标系之间的所述坐标系变换信息。Based on the second aspect, in a possible implementation manner, before the map update information is obtained, the method further includes: determining that the electronic device is in the first coordinate system according to the collected video image and the SLAM map in the first coordinate system The first pose data in the SLAM map; according to the video image and the global sub-map in the second coordinate system, determine the second pose of the electronic device in the global sub-map in the second coordinate system Data; according to the first pose data and the second pose data, obtain the coordinate system transformation information between the first coordinate system of the SLAM map and the second coordinate system of the global map.
其中,本申请一种实施例中的视频图像也可以直接为一张图像。本申请又一种实施例中的视频图像也可以表示电子设备的摄像头采集的视频图像序列中的第K帧图像,K为大于等于1的整数。也就是说,所述摄像头采集的视频图像可以是一个视频序列(视频流),包括多帧图像,而第K帧图像可以是视频流中的某一帧。Wherein, the video image in an embodiment of the present application may also be directly an image. The video image in another embodiment of the present application may also represent the K-th frame image in the video image sequence collected by the camera of the electronic device, and K is an integer greater than or equal to 1. In other words, the video image collected by the camera may be a video sequence (video stream) including multiple frames of images, and the K-th frame image may be a certain frame in the video stream.
本申请实施例通过根据同一帧分别获得终端在局部坐标系中的位姿和全局坐标系中的位姿,基于这两种位姿就能够获得两种坐标系之间的坐标系变换信息(例如坐标系变换矩阵), 根据这个坐标系变换矩阵就可以实现了两个坐标系同步,这样,原先用局部坐标系来表示的信息(例如局部位姿、图像的特征点、SLAM地图的3D地图点等等)就可以基于坐标系变换矩阵变换到全局坐标系下。这样,就能够实现将局部坐标系下的SLAM地图转换到全局坐标系下,或者将局部坐标系下的非冗余关键帧转换到全局坐标系下。The embodiment of the application obtains the pose of the terminal in the local coordinate system and the pose in the global coordinate system respectively according to the same frame. Based on these two poses, the coordinate system transformation information between the two coordinate systems can be obtained (for example, Coordinate system transformation matrix). According to this coordinate system transformation matrix, the two coordinate systems can be synchronized. In this way, the information originally represented by the local coordinate system (such as local pose, image feature points, SLAM map 3D map points Etc.) Then it can be transformed to the global coordinate system based on the coordinate system transformation matrix. In this way, the SLAM map in the local coordinate system can be converted to the global coordinate system, or the non-redundant key frames in the local coordinate system can be converted to the global coordinate system.
基于第二方面,在可能的实施方式中,所述根据采集的视频图像和第一坐标系下的SLAM地图,确定所述电子设备在所述第一坐标系下的SLAM地图中的第一位姿数据,包括:Based on the second aspect, in a possible implementation manner, the first position of the electronic device in the SLAM map in the first coordinate system is determined according to the collected video image and the SLAM map in the first coordinate system. Pose data, including:
根据所述视频图像、所述第一坐标系下的SLAM地图以及所述电子设备采集的运动数据,获得所述电子设备在所述第一坐标系下的SLAM地图中的第一位姿数据;所述运动数据包括运动速度数据和运动方向数据。Obtaining the first pose data of the electronic device in the SLAM map in the first coordinate system according to the video image, the SLAM map in the first coordinate system, and the motion data collected by the electronic device; The movement data includes movement speed data and movement direction data.
例如,电子设备中设置有IMU,SLAM系统的输入信号包括来自摄像头采集的视频图像、IMU采集的运动数据和局部坐标系下的SLAM地图。IMU高频率地检测电子设备的角速度和线加速度,并对角加速度和线加速度分别积分,进而可计算出电子设备的位姿。摄像头采集的视频图像通过在局部坐标系下的SLAM地图中做匹配,从而也能计算获得电子设备的位姿。那么,基于这两种位姿以一定算法进行运算,就可以获得该第一位姿数据。For example, the electronic device is provided with an IMU, and the input signal of the SLAM system includes the video image collected by the camera, the motion data collected by the IMU, and the SLAM map in the local coordinate system. The IMU detects the angular velocity and linear acceleration of the electronic device at a high frequency, and integrates the angular acceleration and linear acceleration separately, and then can calculate the posture of the electronic device. The video image collected by the camera is matched in the SLAM map in the local coordinate system, so that the position and posture of the electronic device can also be calculated. Then, based on the calculation of these two poses with a certain algorithm, the first pose data can be obtained.
基于第二方面,在可能的实施方式中,所述根据所述视频图像和第二坐标系下的全局子地图,确定所述电子设备在所述第二坐标系下的全局子地图中的第二位姿数据,包括:Based on the second aspect, in a possible implementation manner, the determining the first position of the electronic device in the global sub-map in the second coordinate system according to the video image and the global sub-map in the second coordinate system Two pose data, including:
对所述视频图像进行特征提取,以获得图像特征;将所述图像特征在所述第二坐标系下的全局子地图中进行特征匹配,以获得与所述图像特征匹配的地图特征;根据所述图像特征和所述地图特征,计算得到所述电子设备在所述第二坐标系下的全局子地图中的第二位姿数据。Perform feature extraction on the video image to obtain image features; perform feature matching on the image feature in the global sub-map in the second coordinate system to obtain a map feature that matches the image feature; The image feature and the map feature are calculated to obtain the second pose data in the global sub-map of the electronic device in the second coordinate system.
例如,电子设备对第K帧图像进行特征检测,在第K帧图像中提取出特征的图像位置,特征检测算法不限于FAST、ORB、SIFT、SURF、D2Net、SuperPoint等特征检测方法。然后对每一个检测出的特征进行特征描述,特征描述算法不限于ORB、SIFT、SURF、BRIEF、BRISK、FREAK、D2Net、SuperPoint等特征描述方法,从而形成一个一维向量,用于后续的特征匹配。通过特征匹配,电子设备可从全局子地图中匹配出与第K帧图像最相似的地图内容(例如一帧或多帧关键帧),具体方法例如包括基于BOW、VLAD等传统图像检索方法以及基于NetVLAD、AI的新型图像检索方法。找出与第K帧图像最相似的地图内容后,就可以基于第K帧图像和最相似的地图内容进行位姿估计,例如可采用PnP、EPnP、3D-3D等配准算法,从而可以计算出该第二位姿数据。For example, the electronic device performs feature detection on the K-th frame image, and extracts the image position of the feature in the K-th frame image. The feature detection algorithm is not limited to feature detection methods such as FAST, ORB, SIFT, SURF, D2Net, and SuperPoint. Then perform feature description for each detected feature. The feature description algorithm is not limited to feature description methods such as ORB, SIFT, SURF, BRIEF, BRISK, FREAK, D2Net, SuperPoint, etc., to form a one-dimensional vector for subsequent feature matching . Through feature matching, the electronic device can match the most similar map content (such as one or more key frames) to the K-th frame image from the global submap. Specific methods include, for example, traditional image retrieval methods such as BOW and VLAD, and New image retrieval methods of NetVLAD and AI. After finding the most similar map content to the K-th frame image, you can estimate the pose based on the K-th frame image and the most similar map content. For example, PnP, EPnP, 3D-3D and other registration algorithms can be used to calculate Output the second pose data.
本申请实施例可以实现在电子设备侧,充分利用电子设备的运算能力,计算第一位姿数据和第二位姿数据,提高了处理效率,还能够减轻服务器的计算负担。The embodiments of the present application can be implemented on the side of the electronic device, making full use of the computing power of the electronic device to calculate the first pose data and the second pose data, which improves the processing efficiency and can also reduce the calculation burden of the server.
基于第二方面,在可能的实施方式中,所述根据所述视频图像和第二坐标系下的全局子地图,确定所述电子设备在所述第二坐标系下的全局子地图中的第二位姿数据,包括:向所述服务器发送所述视频图像;从所述服务器接收所述第二位姿数据,所述第二位姿数据是所述服务器根据所述视频图像和所述第二坐标系下的全局地图中与所述电子设备的位置关联的全局子地图进行特征提取和特征匹配确定的。Based on the second aspect, in a possible implementation manner, the determining the first position of the electronic device in the global sub-map in the second coordinate system according to the video image and the global sub-map in the second coordinate system Two pose data, including: sending the video image to the server; receiving the second pose data from the server, the second pose data being the server according to the video image and the first The global sub-map associated with the location of the electronic device in the global map in the two-coordinate system is determined by feature extraction and feature matching.
其中,当视频图像为电子设备的摄像头采集的视频图像序列中的第K帧图像时,所述第K帧图像可以是摄像头拍摄的视频图像序列中的第1帧图像。Wherein, when the video image is the K-th frame image in the video image sequence collected by the camera of the electronic device, the K-th frame image may be the first frame image in the video image sequence captured by the camera.
本实施例中,由于电子设备需要先下载对应区域的全局子地图。下载地图的过程需要花费一定时间。为了加快用户进入应用的速度,可以采用第一次全局位姿估计在服务器端完成的方式。也即是说,应用启动后,第一次全局位姿估计在服务器侧进行,在启动全局位姿估计的同时,服务器相应获取全局子地图并向电子设备传输,提高用户进入应用的速度。用户并不会感知到地图下载流程的时延存在,这样,能够避免了下载时延造成的用户等待,进一步提升用户体验。In this embodiment, since the electronic device needs to download the global submap of the corresponding area first. The process of downloading the map takes a certain amount of time. In order to speed up the user's entry into the application, the first global pose estimation can be done on the server side. In other words, after the application is started, the first global pose estimation is performed on the server side. While starting the global pose estimation, the server obtains the global sub-map and transmits it to the electronic device accordingly, which improves the user's access speed to the application. The user does not perceive the delay of the map downloading process. In this way, the user waiting caused by the download delay can be avoided, and the user experience is further improved.
第三方面,本申请实施例提供了一种全局地图更新方法,应用于服务器,该方法包括:In the third aspect, an embodiment of the present application provides a method for updating a global map, which is applied to a server, and the method includes:
根据电子设备的请求,向所述电子设备发送与所述电子设备的位置对应的全局子地图,所述全局子地图是第二坐标系下的全局地图中与所述电子设备的位置关联的子地图;从所述电子设备接收地图更新信息;所述地图更新信息用于指示所述电子设备的SLAM系统所构建的第一坐标系下或者所述第二坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异根据所述地图更新信息来更新所述全局地图。According to the request of the electronic device, the global sub-map corresponding to the location of the electronic device is sent to the electronic device, and the global sub-map is the sub-map associated with the location of the electronic device in the global map in the second coordinate system. Map; receiving map update information from the electronic device; the map update information is used to indicate that the SLAM map in the first coordinate system or in the second coordinate system constructed by the SLAM system of the electronic device is relative to the The difference of the global sub-map in the second coordinate system updates the global map according to the map update information.
在一种实现方案中,所述地图更新信息用于指示所述电子设备的SLAM系统所构建的所述第二坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异。具体的,所述地图更新信息是所述电子设备的SLAM系统根据第二坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果获得的;其中,所述第二坐标系下的SLAM地图是基于SLAM系统所构建的第一坐标系下的SLAM地图以及所述第一坐标系和所述第二坐标系之间的坐标系变换信息得到的;所述比较结果体现了所述第二坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异,SLAM系统可根据所述差异能够进一步获得所述地图更新信息,并将所述地图更新信息发送给服务器。In an implementation solution, the map update information is used to indicate the difference between the SLAM map in the second coordinate system constructed by the SLAM system of the electronic device and the global sub-map in the second coordinate system . Specifically, the map update information is obtained by the SLAM system of the electronic device according to the comparison result of the SLAM map in the second coordinate system and the global sub-map in the second coordinate system; wherein, the second coordinate The SLAM map under the system is obtained based on the SLAM map in the first coordinate system constructed by the SLAM system and the coordinate system transformation information between the first coordinate system and the second coordinate system; the comparison result reflects The difference between the SLAM map in the second coordinate system and the global sub-map in the second coordinate system, the SLAM system can further obtain the map update information according to the difference, and send the map update information To the server.
在另一种实现方案中,所述地图更新信息用于指示所述电子设备的SLAM系统所构建的所述第一坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异。具体的,所述地图更新信息是所述第二电子设备的SLAM系统根据所述第一坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果以及所述坐标系变换信息获得的,所述比较结果体现了所述第一坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异,SLAM系统可将所述差异转换到第二坐标系下去表示,根据第二坐标系下的差异能够进一步获得所述地图更新信息,并将所述地图更新信息发送给服务器。In another implementation solution, the map update information is used to indicate how the SLAM map in the first coordinate system constructed by the SLAM system of the electronic device is relative to the global sub-map in the second coordinate system difference. Specifically, the map update information is the comparison result of the SLAM system of the second electronic device according to the comparison result of the SLAM map in the first coordinate system and the global sub-map in the second coordinate system and the coordinate system transformation Information obtained, the comparison result reflects the difference between the SLAM map in the first coordinate system and the global sub-map in the second coordinate system, and the SLAM system can convert the difference to the second coordinate system It means that the map update information can be further obtained according to the difference in the second coordinate system, and the map update information can be sent to the server.
本申请实施例中,服务器给电子设备发送全局地图中全局子地图,电子设备通过比较全局子地图和电子设备中的SLAM地图,利用电子设备侧的计算资源和运算能力来获得地图更新信息,该地图更新信息能够被用来更新全局子地图中的地图内容。所以服务器收到电子设备发送的地图更新信息后,就能够基于地图更新信息实现对服务器中的全局地图更新。这样,减轻服务器计算负载,提高了服务器更新全局地图的效率。而地图更新信息相比起摄像头拍摄的图像数据而言,数据量更少,所以也节省了传输带宽资源。In the embodiment of this application, the server sends the global sub-map in the global map to the electronic device, and the electronic device obtains map update information by comparing the global sub-map with the SLAM map in the electronic device and using the computing resources and computing capabilities of the electronic device. The map update information can be used to update the map content in the global submap. Therefore, after the server receives the map update information sent by the electronic device, it can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the efficiency of the server to update the global map is improved. Compared with the image data captured by the camera, the map update information has a smaller amount of data, so the transmission bandwidth resources are also saved.
基于第三方面,在可能的实施方式中,所述地图更新信息表示与非冗余关键帧相关联的信息;所述非冗余关键帧可以表示存在于所述第二坐标系下的SLAM地图中但不存在于第二坐标系下的全局子地图中的关键帧。Based on the third aspect, in a possible implementation manner, the map update information represents information associated with a non-redundant key frame; the non-redundant key frame may represent a SLAM map existing in the second coordinate system Keyframes in the global submap that do not exist in the second coordinate system.
相应的,所述根据所述地图更新信息来更新所述全局地图,包括:将所述地图更新信息加入到所述全局地图。Correspondingly, the updating the global map according to the map update information includes: adding the map update information to the global map.
所述地图更新信息具体包括:所述非冗余关键帧对应的位姿数据、三维地图点以及共视数据,所述共视数据用于指示所述非冗余关键帧与所述第二坐标系下的全局子地图的关键帧之间的所述共视关系。也就是说,所述地图更新信息表示所述非冗余关键被融合进全局子地图后得到的相关信息(例如非冗余关键对应的位姿、共视关系、3D地图点等信息)。即地图更新信息是经融合的全局子地图中相比起原先的全局子地图而言,被更新的那一部分信息。The map update information specifically includes: pose data, three-dimensional map points, and common view data corresponding to the non-redundant key frame, and the common view data is used to indicate the non-redundant key frame and the second coordinate The common view relationship between the key frames of the global sub-map under the system. In other words, the map update information represents related information obtained after the non-redundant key is fused into the global sub-map (for example, information such as the pose, common view relationship, 3D map point, etc.) corresponding to the non-redundant key. That is, the map update information is the updated part of the global sub-map compared to the original global sub-map.
由于地图更新信息只需包含所变化的关键帧位姿、共视关系以及3D地图点,相比原先下载的全局子地图而言数据量会小很多,所以进一步节省了传输带宽资源。服务器收到地图更新信息后,就能够基于地图更新信息实现对服务器中的全局地图的更新。这样,减轻服务器计算负载,且提高了全局地图的更新效率和鲁棒性。Since the map update information only needs to include the changed key frame pose, common view relationship, and 3D map points, the amount of data will be much smaller than that of the previously downloaded global sub-map, so the transmission bandwidth resources are further saved. After the server receives the map update information, it can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the update efficiency and robustness of the global map are improved.
基于第三方面,在可能的实施方式中,所述根据所述地图更新信息来更新所述全局地图,包括:将所述地图更新信息加入到所述全局地图。由于所述地图更新信息是从经融合的全局子地图中获得的,而全局子地图是从服务器的全局地图中取得的,所以服务器也能够直接将所述地图更新信息快速加入到服务器的全局地图中,极大提高了全局地图的更新效率。Based on the third aspect, in a possible implementation manner, the updating the global map according to the map update information includes: adding the map update information to the global map. Since the map update information is obtained from the integrated global sub-map, and the global sub-map is obtained from the server's global map, the server can also directly add the map update information to the server's global map. In, the update efficiency of the global map is greatly improved.
基于第三方面,在可能的实施方式中,所述地图更新信息包括非冗余关键帧,相应的,所述根据所述地图更新信息来更新所述全局地图,包括:将所述非冗余关键帧作为新的关键帧节点增加到在所述全局地图;更新所述非冗余关键帧与所述全局地图中的关键帧的共视关系;计算所述非冗余关键帧对应的位姿数据和三维地图点。Based on the third aspect, in a possible implementation manner, the map update information includes non-redundant key frames, and correspondingly, the update of the global map according to the map update information includes: A key frame is added to the global map as a new key frame node; the common view relationship between the non-redundant key frame and the key frame in the global map is updated; the pose corresponding to the non-redundant key frame is calculated Data and 3D map points.
具体的,对于全局地图中的某一3D地图点,如果该3D地图点能同时被若干个关键帧观测到,那么该若干个关键帧就具有对该3D地图点的共视关系。所以,服务器可计算出全局地图中的哪些3D地图点能同时被全局地图中的至少一个关键帧和所述非冗余关键帧观测到,从而更新全局地图中的关键帧和非冗余关键之间的共视关系。Specifically, for a certain 3D map point in the global map, if the 3D map point can be observed by several key frames at the same time, then the several key frames have a common view relationship with the 3D map point. Therefore, the server can calculate which 3D map points in the global map can be simultaneously observed by at least one key frame in the global map and the non-redundant key frame, thereby updating one of the key frames and the non-redundant key in the global map. The common view relationship between.
在更新了全局地图的关键帧共视关系之后,服务器可利用三角化算法计算新的3D地图点,以及通过BA算法优化非冗余关键帧对应的位姿和3D地图点坐标,从而完成了将非冗余关键帧融合(或称插入)到全局地图,实现了该全局地图的更新。After updating the key frame common view relationship of the global map, the server can use the triangulation algorithm to calculate new 3D map points, and optimize the pose and 3D map point coordinates corresponding to the non-redundant key frames through the BA algorithm, thus completing the transformation The non-redundant key frames are merged (or inserted) into the global map to realize the update of the global map.
本申请实施例中,电子设备通过比较SLAM地图和全局子地图,识别出非冗余关键帧,并将非冗余关键帧发送给服务器,服务器再利用非冗余关键帧融合到全局地图,从而也能够实现对服务器中的全局地图的高效率的更新。这样的方案有利于减轻电子设备的计算负载,保证地图融合的准确性。In the embodiment of this application, the electronic device identifies the non-redundant key frame by comparing the SLAM map and the global sub-map, and sends the non-redundant key frame to the server, and the server uses the non-redundant key frame to merge into the global map, thereby It can also realize the efficient update of the global map in the server. Such a solution is beneficial to reduce the computational load of electronic equipment and ensure the accuracy of map fusion.
基于第三方面,在可能的实施方式中,所述根据电子设备的请求,向所述电子设备发送与所述电子设备的位置对应的全局子地图,包括:接收所述电子设备的初始位置的指示信息;根据所述电子设备的初始位置,在所述全局地图中获取所述与所述电子设备的位置对应的所述全局子地图;将所述全局子地图发送给所述电子设备。Based on the third aspect, in a possible implementation manner, the sending a global sub-map corresponding to the location of the electronic device to the electronic device according to the request of the electronic device includes: receiving the initial location of the electronic device Indication information; acquiring the global sub-map corresponding to the location of the electronic device in the global map according to the initial position of the electronic device; sending the global sub-map to the electronic device.
其中,所述电子设备的初始位置的指示信息可以包括用于指示所述电子设备的初始位置的第一位置指纹信息;所述全局地图中的全局子地图对应有第二位置指纹信息;所述根据所述电子设备的初始位置,在所述全局地图中获取所述与所述电子设备的位置对应的所述全局子地图,包括:根据所述第一位置指纹信息,在所述全局地图中找到具有与所述第一位置指纹信息相匹配的第二位置指纹信息的所述全局子地图。通过服务器执行地图匹配操作,提高了匹配效率和精准度,进而有利于减少下载地图的时延。Wherein, the indication information of the initial location of the electronic device may include first location fingerprint information for indicating the initial location of the electronic device; the global sub-map in the global map corresponds to the second location fingerprint information; According to the initial position of the electronic device, acquiring the global sub-map corresponding to the position of the electronic device in the global map includes: according to the first location fingerprint information, in the global map Finding the global sub-map with second location fingerprint information matching the first location fingerprint information. Performing the map matching operation through the server improves the matching efficiency and accuracy, which in turn helps to reduce the delay in downloading the map.
第四方面,本申请实施例提供了一种用于虚拟物体显示的电子设备,包括:一个或多个处理器、存储器以及一个或多个应用程序、一个或多个计算机程序;所述一个或多个计算机程序被存储在所述存储器中,所述一个或多个计算机程序包括指令,当所述指令被所述电子设备执行时,使得电子设备执行上述第一方面或第一方面任一项可能的实施方式中描述的虚拟物体显示方法。In a fourth aspect, an embodiment of the present application provides an electronic device for displaying virtual objects, including: one or more processors, memories, one or more application programs, and one or more computer programs; the one or more A plurality of computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to execute the first aspect or any one of the first aspects above The virtual object display method described in the possible implementation manners.
第五方面,本申请实施例提供了一种用于全局地图更新的电子设备,包括:一个或多个处理器、存储器,以及一个或多个计算机程序;所述一个或多个计算机程序被存储在所述存储器中,所述一个或多个计算机程序包括指令,当所述指令被所述电子设备执行时,使得电子设备执行上述第二方面或第二方面任一项可能的实施方式中描述的虚拟物体显示方法。In a fifth aspect, an embodiment of the present application provides an electronic device for updating a global map, including: one or more processors, a memory, and one or more computer programs; the one or more computer programs are stored In the memory, the one or more computer programs include instructions, and when the instructions are executed by the electronic device, the electronic device executes the second aspect or any one of the possible implementations described in the second aspect. The virtual object display method.
第六方面,本申请实施例提供了一种用于全局地图更新的服务器,包括:一个或多个处理器、存储器,以及一个或多个计算机程序;所述一个或多个计算机程序被存储在所述存储器中,所述一个或多个计算机程序包括指令,当所述指令被所述电子设备执行时,使得电子设备执行上述第三方面或第三方面任一项可能的实施方式中描述的虚拟物体显示方法。In a sixth aspect, an embodiment of the present application provides a server for global map update, including: one or more processors, memories, and one or more computer programs; the one or more computer programs are stored in In the memory, the one or more computer programs include instructions, and when the instructions are executed by the electronic device, the electronic device executes the foregoing third aspect or any one of the possible implementation manners of the third aspect. Virtual object display method.
第七方面,本申请实施例提供一种芯片,所述芯片包括处理器与数据接口,所述处理器通过所述数据接口读取存储器上存储的指令,执行第一方面或第一方面的任一可能的实施方式中的虚拟物体显示方法。In a seventh aspect, an embodiment of the present application provides a chip. The chip includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface, and executes the first aspect or any of the first aspects. A virtual object display method in a possible implementation manner.
可选地,作为一种实施方式,所述芯片还可以包括存储器,所述存储器中存储有指令,所述处理器用于执行所述存储器上存储的指令,当所述指令被执行时,所述处理器用于执行第一方面或第一方面的任一可能的实施方式中的虚拟物体显示方法。Optionally, as an implementation manner, the chip may further include a memory in which instructions are stored, and the processor is configured to execute instructions stored on the memory. When the instructions are executed, the The processor is configured to execute the virtual object display method in the first aspect or any possible implementation of the first aspect.
第八方面,本申请实施例提供一种芯片,所述芯片包括处理器与数据接口,所述处理器通过所述数据接口读取存储器上存储的指令,执行第二方面或第二方面的任一可能的实施方式中的全局地图更新方法。In an eighth aspect, an embodiment of the present application provides a chip. The chip includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface, and executes the second aspect or any of the second aspect. A global map update method in a possible implementation.
可选地,作为一种实施方式,所述芯片还可以包括存储器,所述存储器中存储有指令,所述处理器用于执行所述存储器上存储的指令,当所述指令被执行时,所述处理器用于执行第二方面或第二方面的任一可能的实施方式中的全局地图更新方法。Optionally, as an implementation manner, the chip may further include a memory in which instructions are stored, and the processor is configured to execute instructions stored on the memory. When the instructions are executed, the The processor is configured to execute the global map update method in the second aspect or any possible implementation of the second aspect.
第九方面,本申请实施例提供一种芯片,所述芯片包括处理器与数据接口,所述处理器通过所述数据接口读取存储器上存储的指令,执行第三方面或第三方面的任一可能的实施方式中的全局地图更新方法。In a ninth aspect, an embodiment of the present application provides a chip. The chip includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface, and executes any of the third aspect or the third aspect. A global map update method in a possible implementation.
可选地,作为一种实施方式,所述芯片还可以包括存储器,所述存储器中存储有指令,所述处理器用于执行所述存储器上存储的指令,当所述指令被执行时,所述处理器用于执行第三方面或第三方面的任一可能的实施方式中的全局地图更新方法。Optionally, as an implementation manner, the chip may further include a memory in which instructions are stored, and the processor is configured to execute instructions stored on the memory. When the instructions are executed, the The processor is configured to execute the global map update method in the third aspect or any possible implementation manner of the third aspect.
第十方面,本申请实施例提供一种用于全局地图更新的电子设备,包括:In a tenth aspect, an embodiment of the present application provides an electronic device for updating a global map, including:
通信模块,用于从服务器获取与所述电子设备的位置对应的全局子地图,所述全局子地图是第二坐标系下的全局地图中与所述电子设备的位置关联的子地图;A communication module, configured to obtain a global sub-map corresponding to the location of the electronic device from a server, the global sub-map being a sub-map associated with the location of the electronic device in the global map in the second coordinate system;
SLAM模块,用于根据所述SLAM模块所构建的第二坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果,或者根据所述SLAM模块所构建的第一坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果以及所述坐标系变换信息,获得地图更新信息,其中所述第二坐标系下的SLAM地图是基于所述第一坐标系下的SLAM地图以及所述第一坐标 系和所述第二坐标系之间的坐标系变换信息得到的;所述地图更新信息用于指示所述第二坐标系下或者所述第一坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异;The SLAM module is used to compare the results of the SLAM map in the second coordinate system constructed by the SLAM module with the global sub-map in the second coordinate system, or according to the first coordinate system constructed by the SLAM module The comparison result of the SLAM map under the second coordinate system and the global sub-map under the second coordinate system and the coordinate system transformation information to obtain map update information, wherein the SLAM map under the second coordinate system is based on the first coordinate The SLAM map in the first coordinate system and the coordinate system transformation information between the first coordinate system and the second coordinate system; the map update information is used to indicate the second coordinate system or the first coordinate The difference between the SLAM map in the second coordinate system and the global sub-map in the second coordinate system;
所述通信模块还用于,将所述地图更新信息发送给所述服务器,所述地图更新信息用于更新所述服务器中所述第二坐标系下的全局地图。The communication module is further configured to send the map update information to the server, and the map update information is used to update the global map in the second coordinate system in the server.
其中,所述SLAM模块可以是本申请的实施例中所描述的SLAM系统,例如可以是本文后文实施例描述的SLAM系统12。Wherein, the SLAM module may be the SLAM system described in the embodiment of the present application, for example, it may be the SLAM system 12 described in the following embodiments of this document.
基于第十方面,在可能的实施方式中,所述第二坐标系下的SLAM地图具体为根据基于所述坐标系变换信息对所述第一坐标系下的SLAM地图进行变换得到的;Based on the tenth aspect, in a possible implementation manner, the SLAM map in the second coordinate system is specifically obtained by transforming the SLAM map in the first coordinate system based on the coordinate system transformation information;
所述SLAM模块具体用于:通过比较所述第二坐标系下的SLAM地图与所述第二坐标系下的全局子地图,以获得非冗余关键帧,所述非冗余关键帧表示存在于所述第二坐标系下的SLAM地图中但不存在于所二坐标系下的全局子地图中的关键帧;将所述非冗余关键帧融合到所述第二坐标系下的全局子地图,得到经融合后的全局子地图;从所述经融合后的全局子地图中获得所述地图更新信息。The SLAM module is specifically configured to: compare the SLAM map in the second coordinate system with the global sub-map in the second coordinate system to obtain a non-redundant key frame, where the non-redundant key frame indicates that there is Key frames in the SLAM map in the second coordinate system but not in the global sub-map in the second coordinate system; fusing the non-redundant key frames to the global sub-map in the second coordinate system Map to obtain a fused global sub-map; obtain the map update information from the fused global sub-map.
基于第十方面,在可能的实施方式中,所述SLAM模块具体用于:将所述非冗余关键帧作为新的关键帧节点增加到在所述第二坐标系下的全局子地图;更新所述非冗余关键帧与所述第二坐标系下的全局子地图中的关键帧的共视关系;计算所述非冗余关键帧对应的位姿数据和三维地图点;从所述经融合后的全局子地图中获取所述非冗余关键帧对应的位姿数据、三维地图点以及共视数据作为所述地图更新信息,所述共视数据用于指示所述非冗余关键帧与所述第二坐标系下的全局子地图的关键帧之间的共视关系。Based on the tenth aspect, in a possible implementation manner, the SLAM module is specifically configured to: add the non-redundant key frame as a new key frame node to the global submap in the second coordinate system; update The common view relationship between the non-redundant key frame and the key frame in the global sub-map in the second coordinate system; calculate the pose data and three-dimensional map points corresponding to the non-redundant key frame; Obtain the pose data, three-dimensional map points, and common view data corresponding to the non-redundant key frame from the fused global sub-map as the map update information, and the common view data is used to indicate the non-redundant key frame The common view relationship with the key frames of the global sub-map in the second coordinate system.
基于第十方面,在可能的实施方式中,所述SLAM模块具体用于:通过比较所述第一坐标系下的SLAM地图与所述第二坐标系下的全局子地图,以得到非冗余关键帧,所述非冗余关键帧为存在于所述第一坐标系下的SLAM地图的关键帧,且所述非冗余关键帧经所述坐标系变换信息变换后不存在于所述第二坐标系下的全局子地图;将经所述坐标系变换信息变换后的非冗余关键帧融合到所述第二坐标系下的全局子地图,获得所述地图更新信息,所述经所述坐标系变换信息变换后的非冗余关键帧是所述第二坐标系下的关键帧。Based on the tenth aspect, in a possible implementation manner, the SLAM module is specifically configured to: compare the SLAM map in the first coordinate system with the global sub-map in the second coordinate system to obtain non-redundant A key frame, the non-redundant key frame is a key frame of the SLAM map existing in the first coordinate system, and the non-redundant key frame does not exist in the first coordinate system after being transformed by the coordinate system transformation information. The global sub-map in the two-coordinate system; the non-redundant key frames transformed by the coordinate system transformation information are fused to the global sub-map in the second coordinate system to obtain the map update information. The non-redundant key frame transformed by the coordinate system transformation information is a key frame in the second coordinate system.
基于第十方面,在可能的实施方式中,所述通信模块具体用于:向服务器发送所述电子设备的初始位置的指示信息;从所述服务器接收所述全局子地图,所述全局子地图是根据所述电子设备的初始位置确定的。Based on the tenth aspect, in a possible implementation manner, the communication module is specifically configured to: send indication information of the initial position of the electronic device to the server; receive the global sub-map from the server, and the global sub-map It is determined according to the initial position of the electronic device.
基于第十方面,在可能的实施方式中,所述电子设备的初始位置的指示信息包括用于指示所述电子设备的初始位置的第一位置指纹信息;所述全局子地图对应有第二位置指纹信息,且所述第一位置指纹信息和所述第二位置指纹信息相匹配。Based on the tenth aspect, in a possible implementation manner, the indication information of the initial location of the electronic device includes first location fingerprint information for indicating the initial location of the electronic device; the global sub-map corresponds to the second location Fingerprint information, and the first location fingerprint information matches the second location fingerprint information.
基于第十方面,在可能的实施方式中所述电子设备还包括全局定位模块和坐标系变化矩阵计算模块;所述SLAM模块具体用于,根据采集的视频图像和第一坐标系下的SLAM地图,确定所述电子设备在所述第一坐标系下的SLAM地图中的第一位姿数据;所述全局定位模块具体用于,根据所述视频图像和第二坐标系下的全局子地图,确定所述电子设备在所述第二坐标系下的全局子地图中的第二位姿数据;所述坐标系变化矩阵计算模块具体用于,根据所述第一位姿数据和所述第二位姿数据,获得所述SLAM地图的第一坐标系和所述全局地图的第二坐标系之间的所述坐标系变换信息。Based on the tenth aspect, in a possible implementation manner, the electronic device further includes a global positioning module and a coordinate system change matrix calculation module; the SLAM module is specifically used to, according to the collected video image and the SLAM map in the first coordinate system , Determine the first pose data of the electronic device in the SLAM map in the first coordinate system; the global positioning module is specifically configured to, according to the video image and the global sub-map in the second coordinate system, Determine the second pose data of the electronic device in the global submap in the second coordinate system; the coordinate system change matrix calculation module is specifically configured to, according to the first pose data and the second The pose data is used to obtain the coordinate system transformation information between the first coordinate system of the SLAM map and the second coordinate system of the global map.
基于第十方面,在可能的实施方式中,所述SLAM模块具体用于:根据所述视频图像、所述第一坐标系下的SLAM地图以及所述电子设备采集的运动数据,获得所述电子设备在所述第一坐标系下的SLAM地图中的第一位姿数据;所述运动数据包括运动速度数据和运动方向数据。Based on the tenth aspect, in a possible implementation manner, the SLAM module is specifically configured to: obtain the electronic device according to the video image, the SLAM map in the first coordinate system, and the motion data collected by the electronic device. The first pose data of the device in the SLAM map in the first coordinate system; the movement data includes movement speed data and movement direction data.
基于第十方面,在可能的实施方式中,所述全局定位模块具体用于,对所述视频图像进行特征提取,以获得图像特征;将所述图像特征在所述第二坐标系下的全局子地图中进行特征匹配,以获得与所述图像特征匹配的地图特征;根据所述图像特征和所述地图特征,计算得到所述电子设备在所述第二坐标系下的全局子地图中的第二位姿数据。Based on the tenth aspect, in a possible implementation manner, the global positioning module is specifically configured to perform feature extraction on the video image to obtain image features; Feature matching in the sub-map to obtain map features that match the image features; according to the image features and the map features, the electronic device in the global sub-map in the second coordinate system is calculated The second pose data.
基于第十方面,在可能的实施方式中,所述通信模块具体用于:向所述服务器发送所述视频图像;从所述服务器接收所述第二位姿数据,所述第二位姿数据是所述服务器根据所述视频图像和所述第二坐标系下的全局地图中与所述电子设备的位置关联的全局子地图进行特征提取和特征匹配确定的。Based on the tenth aspect, in a possible implementation manner, the communication module is specifically configured to: send the video image to the server; receive the second pose data from the server, and the second pose data The server performs feature extraction and feature matching determination based on the video image and the global submap associated with the location of the electronic device in the global map in the second coordinate system.
第十一方面,本申请实施例提供一种用于全局地图的服务器,包括:In an eleventh aspect, an embodiment of the present application provides a server for a global map, including:
通信模块,用于根据电子设备的请求,向所述电子设备发送与所述电子设备的位置对应的全局子地图,所述全局子地图是第二坐标系下的全局地图中与所述电子设备的位置关联的子地图;The communication module is configured to send a global sub-map corresponding to the location of the electronic device to the electronic device according to the request of the electronic device, where the global sub-map is the global map in the second coordinate system and the electronic device The sub-map associated with the location;
所述通信模块还用于,从所述电子设备接收地图更新信息;所述地图更新信息用于指示所述电子设备的SLAM系统所构建的第一坐标系下或者所述第二坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异;The communication module is also used to receive map update information from the electronic device; the map update information is used to indicate the first coordinate system constructed by the SLAM system of the electronic device or the second coordinate system The difference between the SLAM map and the global sub-map in the second coordinate system;
处理模块,用于根据所述地图更新信息更新所述全局地图。The processing module is configured to update the global map according to the map update information.
基于第十一方面,在可能的实施方式中,所述差异信息表示与非冗余关键帧相关联的信息;所述非冗余关键帧表示存在于所述第二坐标系下的SLAM地图中但不存在于第二坐标系下的全局子地图中的关键帧。Based on the eleventh aspect, in a possible implementation manner, the difference information indicates information associated with a non-redundant key frame; the non-redundant key frame indicates that it exists in the SLAM map in the second coordinate system But there is no key frame in the global submap in the second coordinate system.
基于第十一方面,在可能的实施方式中,所述地图更新信息具体包括:所述非冗余关键帧对应的位姿数据、三维地图点以及共视数据,所述共视数据用于指示所述非冗余关键帧与所述第二坐标系下的全局子地图的关键帧之间的共视关系。Based on the eleventh aspect, in a possible implementation manner, the map update information specifically includes: pose data, three-dimensional map points, and common view data corresponding to the non-redundant key frame, and the common view data is used to indicate The common view relationship between the non-redundant key frame and the key frame of the global submap in the second coordinate system.
基于第十一方面,在可能的实施方式中,所述处理模块具体用于:将所述地图更新信息加入到所述全局地图。Based on the eleventh aspect, in a possible implementation manner, the processing module is specifically configured to: add the map update information to the global map.
基于第十一方面,在可能的实施方式中,所述通信模块具体用于:接收所述电子设备的初始位置的指示信息;根据所述电子设备的初始位置,在所述全局地图中获取所述与所述电子设备的位置对应的所述全局子地图;将所述全局子地图发送给所述电子设备。Based on the eleventh aspect, in a possible implementation manner, the communication module is specifically configured to: receive indication information of the initial position of the electronic device; and obtain all information from the global map according to the initial position of the electronic device. The global sub-map corresponding to the location of the electronic device; sending the global sub-map to the electronic device.
基于第十一方面,在可能的实施方式中,所述电子设备的初始位置的指示信息包括用于指示所述电子设备的初始位置的第一位置指纹信息;所述全局地图中的全局子地图对应有第二位置指纹信息;所述处理模块具体用于:根据所述第一位置指纹信息,在所述全局地图中找到具有与所述第一位置指纹信息相匹配的第二位置指纹信息的所述全局子地图。Based on the eleventh aspect, in a possible implementation manner, the indication information of the initial position of the electronic device includes first position fingerprint information for indicating the initial position of the electronic device; the global sub-map in the global map Corresponding to the second location fingerprint information; the processing module is specifically configured to: according to the first location fingerprint information, find the second location fingerprint information matching the first location fingerprint information in the global map The global submap.
第十二方面,本申请实施例提供一种计算机可读存储介质,所述计算机可读介质存储用于设备执行的程序代码,所述程序代码包括用于执行第一方面或者第一方面的任一可能的实施方式中的方法的指令,或者,所述程序代码包括用于执行第二方面或者第三方面的任一可能的实施方式中的方法的指令。In a twelfth aspect, an embodiment of the present application provides a computer-readable storage medium that stores program code for device execution, and the program code includes the program code for executing the first aspect or any of the first aspect. An instruction of the method in a possible implementation manner, or the program code includes an instruction for executing the method in any possible implementation manner of the second aspect or the third aspect.
第十三方面,本发明实施例提供了一种计算机程序产品,该计算机程序产品可以为一个软件安装包,该计算机程序产品包括程序指令,当该计算机程序产品被电子设备执行时,该电子设备的处理器执行前述第一方面或第二方面或者第三方面中的任一可能的实施方式中的方法。In a thirteenth aspect, an embodiment of the present invention provides a computer program product, the computer program product may be a software installation package, the computer program product includes program instructions, when the computer program product is executed by an electronic device, the electronic device The processor executes the method in any possible implementation manner of the foregoing first aspect, second aspect, or third aspect.
可以看到,本申请实施例中,电子设备从服务器下载全局地图中全局子地图,通过比较全局子地图和电子设备中的SLAM地图,利用电子设备侧的计算资源和运算能力来获得地图更新信息,该地图更新信息能够被用来更新全局子地图中的地图内容。所以电子设备将地图更新信息发送给服务器后,服务器就能够基于地图更新信息实现对服务器中的全局地图更新。这样,减轻服务器计算负载,提高了服务器更新全局地图的效率。而地图更新信息相比起摄像头拍摄的图像数据而言,数据量更少,所以也节省了传输带宽资源。It can be seen that, in this embodiment of the application, the electronic device downloads the global sub-map in the global map from the server, and obtains map update information by comparing the global sub-map with the SLAM map in the electronic device and using the computing resources and computing capabilities of the electronic device. , The map update information can be used to update the map content in the global sub-map. Therefore, after the electronic device sends the map update information to the server, the server can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the efficiency of the server to update the global map is improved. Compared with the image data taken by the camera, the map update information has a smaller amount of data, so the transmission bandwidth resources are also saved.
附图说明Description of the drawings
为了更清楚地说明本申请实施例或背景技术中的技术方案,下面将对本申请实施例或背景技术中所需要使用的附图进行说明。In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background art, the following will describe the drawings that need to be used in the embodiments of the present application or the background art.
图1是本申请实施例提供的一种应用架构的示意图;FIG. 1 is a schematic diagram of an application architecture provided by an embodiment of the present application;
图2是本申请实施例提供的一种电子设备的结构示意图;FIG. 2 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;
图3是本申请实施例提供的一种服务器的结构示意图;FIG. 3 is a schematic structural diagram of a server provided by an embodiment of the present application;
图4是本申请实施例提供的一种系统,以及该系统中的电子设备和服务器的结构示意图;4 is a system provided by an embodiment of the present application, and a schematic diagram of the structure of electronic devices and servers in the system;
图5是本申请实施例提供的又一种系统,以及该系统中的电子设备和服务器的结构示意图;FIG. 5 is another system provided by an embodiment of the present application, and a schematic diagram of the structure of electronic devices and servers in the system;
图6是本申请实施例提供的一种地图融合模块的功能实现示意图;FIG. 6 is a schematic diagram of functional realization of a map fusion module provided by an embodiment of the present application;
图7是本申请实施例提供的一种虚拟物体显示方法的流程示意图;FIG. 7 is a schematic flowchart of a method for displaying a virtual object provided by an embodiment of the present application;
图8是本申请实施例提供的一种采用本申请方法实现的场景的示意图;FIG. 8 is a schematic diagram of a scenario implemented by using the method of the present application provided by an embodiment of the present application;
图9是本申请实施例提供的一种全局地图更新方法的流程示意图;FIG. 9 is a schematic flowchart of a method for updating a global map provided by an embodiment of the present application;
图10是本申请实施例提供的又一种全局地图更新方法的流程示意图;FIG. 10 is a schematic flowchart of yet another global map update method provided by an embodiment of the present application;
图11是本申请实施例提供的一种坐标系变换矩阵相关的场景示意图;FIG. 11 is a schematic diagram of a scene related to a coordinate system transformation matrix provided by an embodiment of the present application;
图12是本申请实施例提供的一种描述关键帧之间共视关系的场景示意图;FIG. 12 is a schematic diagram of a scene describing the common view relationship between key frames according to an embodiment of the present application;
图13是本申请实施例提供的又一种采用本申请方法实现的场景的示意图;FIG. 13 is a schematic diagram of yet another scenario implemented by the method of this application provided by an embodiment of this application;
图14是本申请实施例提供的又一种采用本申请方法实现的场景的示意图;FIG. 14 is a schematic diagram of yet another scenario implemented by the method of the present application provided by an embodiment of the present application;
图15是本申请实施例提供的又一种全局地图更新方法的流程示意图;15 is a schematic flowchart of another global map update method provided by an embodiment of the present application;
图16是本申请实施例提供的又一种系统,以及该系统中的电子设备和服务器的结构示意图;FIG. 16 is another system provided by an embodiment of the present application, and a schematic diagram of the structure of electronic devices and servers in the system;
图17是本申请实施例提供的又一种全局地图更新方法的流程示意图。FIG. 17 is a schematic flowchart of another method for updating a global map provided by an embodiment of the present application.
具体实施方式Detailed ways
下面结合本申请实施例中的附图对本申请实施例进行描述。本申请的实施方式部分使用的术语仅用于对本申请的具体实施例进行解释,而非旨在限定本申请。The embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. The terminology used in the implementation mode of this application is only used to explain the specific embodiments of this application, and is not intended to limit this application.
本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”、“第三”等是用于 区别不同对象,而不是用于限定特定顺序。The terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to limit a specific order.
在本申请实施例中,“示例性的”或者“例如”等词用于表示作例子、例证或说明。本申请实施例中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言,使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念。In the embodiments of the present application, words such as "exemplary" or "for example" are used as examples, illustrations, or illustrations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferable or advantageous than other embodiments or design solutions. To be precise, words such as "exemplary" or "for example" are used to present related concepts in a specific manner.
在本申请实施例或所附权利要求书中所使用的单数形式的“一种”、“所述”和“该”也旨在包括多数形式,除非上下文清楚地表示其他含义。还应当理解,本文中使用的术语“和/或”是指并包含一个或多个相关联的列出项目的任何或所有可能组合。如本文所使用的,单数形式的“一”、“某”和“该”旨在也包括复数形式,除非上下文另有明确指示。还将理解,术语“包括”、“具有”、“包含”和/或“含有”在本文中使用时指定所陈述的特征、整数、步骤、操作、要素、和/或组件的存在,但并不排除一个或多个其他特征、整数、步骤、操作、要素、组件和/或其群组的存在或添加。The singular forms of "a", "said" and "the" used in the embodiments of the present application or the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and/or" as used herein refers to and includes any or all possible combinations of one or more associated listed items. As used herein, the singular forms "a", "some" and "the" are intended to also include the plural forms, unless the context clearly dictates otherwise. It will also be understood that the terms "including", "having", "including" and/or "containing" when used herein specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not The existence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof is not excluded.
需要说明的是,在本申请实施例中使用的术语是仅仅出于描述特定实施例的目的,而非旨在限制本申请。It should be noted that the terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application.
参见图1,图1是本申请实施例所提供的一种应用架构。本申请实施例提供的应用架构包括多个电子设备和服务器,如图示中所述多个电子设备包括第一电子设备、一个或多个第二电子设备(图1中以2个第二电子设备为例),所述一个或多个第二电子设备为除所述第一电子设备外的若干个电子设备,各个电子设备和服务器之间可进行通信,例如对于任意电子设备,可通过比如无线保真(wireless-fidelity,Wifi)通信、蓝牙通信、或蜂窝2/3/4/5代(2/3/4/5generation,2G/3G/4G/5G)通信等方式与服务器进行通信。需要说明的是,本文中所谓的“多个第二电子设备”仅用于表示除第一电子设备外的多个其他电子设备,而并不限定该各个第二电子设备的类型是否相同。Refer to Fig. 1, which is an application architecture provided by an embodiment of the present application. The application architecture provided by the embodiments of this application includes multiple electronic devices and servers. As shown in the figure, the multiple electronic devices include a first electronic device, one or more second electronic devices (in Figure 1 there are two second electronic devices). Device as an example), the one or more second electronic devices are several electronic devices other than the first electronic device, and each electronic device and the server can communicate with each other. For example, for any electronic device, you can use, for example, It communicates with the server in wireless-fidelity (Wifi) communication, Bluetooth communication, or cellular 2/3/4/5 generation (2/3/4/5generation, 2G/3G/4G/5G) communication. It should be noted that the “multiple second electronic devices” used herein are only used to indicate multiple other electronic devices except the first electronic device, and it does not limit whether the types of the respective second electronic devices are the same.
对于图1所示的多个电子设备的各个电子设备,可以为配置有摄像头和显示组件的各种类型的设备,例如电子设备可以为手机、平板电脑、笔记本电脑、录像机等终端设备(如图1中以各个电子设备为手机为例),也可以是用于虚拟场景交互的设备,如VR眼镜、AR设备,MR互动设备等,可以为智能手表、智能手环等可穿戴电子设备,还可以是无人驾驶车辆、无人机等载具中的车载设备。本申请实施例对电子设备的具体形式不做特殊限制。For each electronic device of the multiple electronic devices shown in Figure 1, it can be various types of devices equipped with cameras and display components. For example, the electronic devices can be terminal devices such as mobile phones, tablets, laptops, video recorders, etc. (as shown in the figure). In 1, each electronic device is a mobile phone as an example), it can also be a device used for virtual scene interaction, such as VR glasses, AR devices, MR interactive devices, etc., can be wearable electronic devices such as smart watches, smart bracelets, and It can be the on-board equipment in vehicles such as unmanned vehicles and drones. The embodiments of this application do not impose special restrictions on the specific form of the electronic device.
此外,所述多个电子设备中任意电子设备也可能被称为用户装备(UE)、订户站、移动单元、订户单元、无线单元、远程单元、移动设备、无线设备、无线通信设备、远程设备、移动订户站、终端设备、接入终端、移动终端、无线终端、智能终端、远程终端、手持机、用户代理、移动客户端、客户端、或其他某个合适的术语。In addition, any electronic device among the plurality of electronic devices may also be referred to as user equipment (UE), subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device , Mobile subscriber station, terminal equipment, access terminal, mobile terminal, wireless terminal, smart terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term.
服务器具体可以是一台或多台物理服务器(例如图1中示例性地示出了一台物理服务器),也可以是计算机集群,还可以是云计算场景的虚拟机,等等。The server may specifically be one or more physical servers (for example, one physical server is exemplarily shown in FIG. 1), a computer cluster, or a virtual machine in a cloud computing scenario, and so on.
本申请实施例中,所述多个电子设备中的各个电子设备可安装VR或AR或MR应用等虚拟场景应用程序,并可基于用户的操作(例如点击、触摸、滑动、抖动、声控等)运行该VR或AR或MR应用。电子设备可通过本地摄像头和/或传感器采集环境中任意物体的视频图像,根据采集的视频图像在显示组件上显示虚拟物体。虚拟物体相应可以为VR场景、AR场景或MR场景中的虚拟物体(即虚拟环境中的物体)。In the embodiments of the present application, each of the multiple electronic devices can be installed with virtual scene applications such as VR or AR or MR applications, and can be based on user operations (such as clicking, touching, sliding, shaking, voice control, etc.) Run the VR or AR or MR application. The electronic device can collect a video image of any object in the environment through a local camera and/or sensor, and display the virtual object on the display component according to the collected video image. The virtual object may correspondingly be a virtual object in a VR scene, an AR scene, or an MR scene (that is, an object in a virtual environment).
需要说明的是,本申请实施例中,电子设备中的虚拟场景应用可以是电子设备自身内置的应用程序,也可以是用户自行安装的第三方服务商提供的应用程序,本申请实施例不作限定。It should be noted that, in the embodiments of the present application, the virtual scene application in the electronic device may be an application built in the electronic device itself, or may be an application provided by a third-party service provider installed by the user, which is not limited in the embodiment of this application. .
本申请实施例中,所述多个电子设备中各个电子设备还配置有即时定位与地图构建(simultaneous localization and mapping,SLAM)系统,SLAM系统能够实现在完全未知环境中创建地图,并利用该地图进行自主定位、位姿(位置与姿态)确定、导航等。本文中由SLAM系统所构建的地图可简称SLAM地图,SLAM地图可以理解为SLAM系统根据采集设备采集到的环境信息所绘制出的地图,采集设备例如可以包括电子设备中的图像采集装置(例如摄像头或相机)和惯性测量单元(Inertial Measurement Unit,IMU),IMU中可以包括:陀螺仪、加速度计等传感器。In the embodiment of the present application, each of the multiple electronic devices is also equipped with a simultaneous localization and mapping (SLAM) system. The SLAM system can create a map in a completely unknown environment and use the map Carry out autonomous positioning, pose (position and posture) determination, navigation, etc. In this article, the map constructed by the SLAM system can be referred to as the SLAM map. The SLAM map can be understood as a map drawn by the SLAM system according to the environmental information collected by the collection device. The collection device can include, for example, an image collection device in an electronic device (such as a camera). (Or camera) and an inertial measurement unit (Inertial Measurement Unit, IMU). The IMU may include sensors such as gyroscopes and accelerometers.
SLAM地图可以包含以下地图内容:多个关键帧、三角测量的特征点以及关键帧和特征点之间的关联。关键帧可以基于摄像头采集的图像和用来产生图像的相机参数(例如,电子设备在SLAM坐标系中的位姿)而形成。如本文中所用的特征点(即特征)是作为图像的令人感兴趣的或显著的一部分。其中所述特征点可以表示SLAM地图中沿着三维空间的不同的3D地图点(例如三维空间轴线X、Y及Z上的坐标)以及在3D地图点上的特征描述。每一特征点可具有关联的特征位置。每个特征点可表示3D坐标位置,并且与一或多个描述符相关联。The SLAM map can contain the following map content: multiple key frames, triangulated feature points, and the association between key frames and feature points. The key frame may be formed based on the image collected by the camera and the camera parameters used to generate the image (for example, the pose of the electronic device in the SLAM coordinate system). A feature point (ie, a feature) as used herein is an interesting or significant part of an image. The feature points may represent different 3D map points along the three-dimensional space in the SLAM map (for example, the coordinates on the three-dimensional space axes X, Y, and Z) and the feature descriptions on the 3D map points. Each feature point can have an associated feature location. Each feature point can represent a 3D coordinate position and is associated with one or more descriptors.
例如,当电子设备拍摄到一张图像后,是否将该图像作为新的关键帧加入到SLAM地图,可以从新的关键帧自身和新的关键帧与SLAM地图中的已有关键帧之间的关系这两方面来考虑。也就是说,一方面,新的关键帧自身图像质量要好,比如不能是非常模糊的图像、特征点数量要充足、特征点分布要尽量均匀等等;另一方面,对于新的关键帧需要与SLAM地图中的已有关键帧有少量的共视关系,但大部分特征点是新特征点,以达到既存在约束,又尽量少的信息冗余的效果,比如SLAM地图中的一个邻近关键帧的特征点在该新的关键帧里有10%能观测到,剩余90%观测不到,等等。For example, when an electronic device captures an image, whether to add the image as a new key frame to the SLAM map, you can find the relationship between the new key frame itself and the relationship between the new key frame and the existing key frame in the SLAM map Consider these two aspects. That is to say, on the one hand, the image quality of the new key frame itself should be good, for example, it cannot be a very blurry image, the number of feature points should be sufficient, and the feature point distribution should be as uniform as possible; on the other hand, the new key frame needs to be The existing key frames in the SLAM map have a small amount of common view relations, but most of the feature points are new feature points to achieve the effect of both existing constraints and as little information redundancy as possible, such as a neighboring key frame in the SLAM map In the new key frame, 10% of the feature points can be observed, and the remaining 90% cannot be observed, and so on.
构建该SLAM地图所采用的坐标系可称为第一坐标系,本文中的第一坐标系在某些应用场景下也可能被称为局部坐标系、SLAM坐标系、相机坐标系或其他某个合适的术语。为了理解上的方便,后文将主要以“局部坐标系”的名称进行方案的介绍。相应的,电子设备在局部坐标系下所体现出的位姿可以称为局部位姿。The coordinate system used to construct the SLAM map can be called the first coordinate system. The first coordinate system in this article may also be called the local coordinate system, the SLAM coordinate system, the camera coordinate system or some other in some application scenarios. Appropriate terminology. For the convenience of understanding, the following text will mainly introduce the scheme under the name of "local coordinate system". Correspondingly, the pose embodied by the electronic device in the local coordinate system can be called a local pose.
本申请实施例中,服务器可作为向任意电子设备的VR或AR或MR应用提供内容和信息支撑的平台。服务器中存储有全局地图。通常的,全局地图相比单一电子设备中的SLAM地图而言,包含了更大的区域,而且地图内容精度更高,由服务器对全局地图进行维护和更新。本申请实施例中,全局地图可以是由所述多个电子设备中一个或多个电子设备提供的多个地图更新信息按照一定的规则进行整合得到的。In the embodiments of the present application, the server can be used as a platform for providing content and information support to VR or AR or MR applications of any electronic device. A global map is stored in the server. Generally, the global map contains a larger area than the SLAM map in a single electronic device, and the map content is more accurate, and the global map is maintained and updated by the server. In the embodiment of the present application, the global map may be obtained by integrating multiple map update information provided by one or more of the multiple electronic devices according to certain rules.
构建该全局地图所采用的坐标系可称为第二坐标系,本文中的第二坐标系在某些应用场景下也可能被称为全局坐标系、世界坐标系或其他某个合适的术语。为了理解上的方便,后文将主要以“全局坐标系”的名称进行方案的介绍。相应的,电子设备在全局坐标系下所体现出的位姿可以称为全局位姿。The coordinate system used to construct the global map may be called the second coordinate system. In some application scenarios, the second coordinate system may also be called the global coordinate system, the world coordinate system, or some other appropriate terminology. For the convenience of understanding, the following text will mainly use the name "global coordinate system" to introduce the scheme. Correspondingly, the posture embodied by the electronic device in the global coordinate system can be referred to as the global posture.
参见图2,图2示例性地示出了电子设备10的一种结构示意图。电子设备10可以为第 一电子设备和第二电子设备中的至少一者。应该理解的是,本申请实施例示意的结构并不构成对电子设备10的具体限定。在本申请另一些实施例中,电子设备10可以包括比图示更多或更少的部件,或者组合某些部件,或者拆分某些部件,或者不同的部件布置。图中所示出的各种部件可以在包括一个或多个信号处理和/或专用集成电路在内的硬件、软件、或硬件和软件的组合中实现。Referring to FIG. 2, FIG. 2 exemplarily shows a schematic structural diagram of the electronic device 10. The electronic device 10 may be at least one of the first electronic device and the second electronic device. It should be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown, or combine certain components, or split certain components, or arrange different components. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and/or application specific integrated circuits.
如图2所示,电子设备10可包括:芯片310、存储器315(一个或多个计算机可读存储介质)、用户接口322、显示组件323、摄像头324、用于设备定位的定位模块331以及用于通信的收发器332。这些部件可在一个或多个通信总线314上通信。As shown in FIG. 2, the electronic device 10 may include: a chip 310, a memory 315 (one or more computer-readable storage media), a user interface 322, a display component 323, a camera 324, a positioning module 331 for device positioning, and a user interface For communication transceiver 332. These components may communicate on one or more communication buses 314.
芯片310可集成包括:一个或多个处理器311、时钟模块312以及电源管理模块313。集成于芯片310中的时钟模块312主要用于为处理器311提供数据传输和时序控制所需要的计时器,计时器可实现数据传输和时序控制的时钟功能。处理器311可以根据指令操作码和时序信号,执行运算,产生操作控制信号,完成取指令和执行指令的控制。集成于芯片310中的电源管理模块313主要用于为芯片310以及电子设备10的其他部件提供稳定的、高精确度的电压。The chip 310 may integrate: one or more processors 311, a clock module 312, and a power management module 313. The clock module 312 integrated in the chip 310 is mainly used to provide the processor 311 with a timer required for data transmission and timing control, and the timer can realize the clock function of data transmission and timing control. The processor 311 can perform operations according to the instruction operation code and timing signals, generate operation control signals, and complete the control of fetching instructions and executing instructions. The power management module 313 integrated in the chip 310 is mainly used to provide a stable and high-precision voltage for the chip 310 and other components of the electronic device 10.
处理器110又可称为中央处理器(CPU,central processing unit),处理器110具体可以包括一个或多个处理单元,例如:处理器110可以包括应用处理器(application processor,AP),调制解调处理器,图形处理器(graphics processing unit,GPU),图像信号处理器(image signal processor,ISP),控制器,视频编解码器,数字信号处理器(digital signal processor,DSP),基带处理器,和/或神经网络处理器(neural-network processing unit,NPU)等。其中,不同的处理单元可以是独立的器件,也可以集成在一个或多个处理器中。The processor 110 may also be referred to as a central processing unit (CPU, central processing unit). The processor 110 may specifically include one or more processing units. For example, the processor 110 may include an application processor (AP). Tuning processor, graphics processing unit (GPU), image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor , And/or neural-network processing unit (NPU), etc. Among them, the different processing units may be independent devices or integrated in one or more processors.
在一些实施例中,处理器110可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit,I2C)接口,集成电路内置音频(inter-integrated circuit sound,I2S)接口,脉冲编码调制(pulse code modulation,PCM)接口,通用异步收发传输器(universal asynchronous receiver/transmitter,UART)接口,移动产业处理器接口(mobile industry processor interface,MIPI),通用输入输出(general-purpose input/output,GPIO)接口,用户标识模块(subscriber identity module,SIM)接口,和/或通用串行总线(universal serial bus,USB)接口等。In some embodiments, the processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, and a universal asynchronous transmitter/receiver (universal asynchronous) interface. receiver/transmitter, UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and / Or Universal Serial Bus (USB) interface, etc.
存储器315可与处理器311通过总线连接,也可以与处理器311耦合在一起,用于存储各种软件程序和/或多组指令。具体实现中,存储器315可包括高速随机存取的存储器(例如高速缓冲存储器),并且也可包括非易失性存储器,例如一个或多个磁盘存储设备、闪存设备或其他非易失性固态存储设备。存储器315可以存储操作系统,例如ANDROID,IOS,WINDOWS,或者LINUX等嵌入式操作系统。存储器315还用于存储SLAM系统的相关程序。存储器315用于存储数据(例如图像数据、点云数据、地图数据、关键帧数据、位姿数据、坐标系转换信息、地图更新信息等)。存储器315还可以存储通信程序,该通信程序可用于与一个或多个服务器或者其他设备进行通信。存储器315还可以存储一个或多个应用程序。如图示中这些应用程序可包括:AR/VR/MR等虚拟场景应用程序、地图类应用程序、图像管理类应用程序等等。存储器115还可以存储用户界面程序,该用户界面程序可以通过图形化的操作界面将应用程序的内容(如AR/VR/MR等虚拟场景中的虚拟物体)形象逼真的显示出来并通过显示组件 323呈现,以及实现通过菜单、对话框以及按键等输入控件接收用户对应用程序的控制操作。存储器315可以用于存储计算机可执行程序代码,该可执行程序代码包括指令。The memory 315 may be connected with the processor 311 through a bus, or may be coupled with the processor 311, and used to store various software programs and/or multiple sets of instructions. In a specific implementation, the memory 315 may include a high-speed random access memory (such as a cache memory), and may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage. equipment. The memory 315 may store an operating system, such as an embedded operating system such as ANDROID, IOS, WINDOWS, or LINUX. The memory 315 is also used to store related programs of the SLAM system. The memory 315 is used to store data (for example, image data, point cloud data, map data, key frame data, pose data, coordinate system conversion information, map update information, etc.). The memory 315 may also store a communication program, which may be used to communicate with one or more servers or other devices. The memory 315 may also store one or more application programs. As shown in the figure, these applications may include: virtual scene applications such as AR/VR/MR, map applications, image management applications, and so on. The memory 115 can also store a user interface program. The user interface program can vividly display the content of the application program (such as virtual objects in virtual scenes such as AR/VR/MR) through a graphical operation interface and display it through the display component 323 Present, and realize the control operation of the application program received by the user through input controls such as menus, dialog boxes, and buttons. The memory 315 may be used to store computer executable program code, and the executable program code includes instructions.
用户接口322例如可以是触控面板,通过触控面板可检测用户在检测触控面板上的操作指令,用户接口322也可以是小键盘、物理按键或者鼠标。The user interface 322 may be, for example, a touch panel, through which the user's operation instructions on the touch panel can be detected, and the user interface 322 may also be a small keyboard, a physical button, or a mouse.
电子设备10可以包括一个或多个显示组件323。电子设备10可通过显示组件323、芯片310中的图形处理器(GPU)以及应用处理器(AP)等共同实现显示功能。GPU为用于图像处理的微处理器,连接显示组件323和应用处理器。GPU用于执行数学和几何计算,用于图形渲染。显示组件323用于显示系统当前输出的界面内容,例如显示AR/VR/MR等虚拟场景中的图像、视频等,界面内容可包括正在运行的应用程序的界面以及系统级别菜单等,具体可由下述界面元素组成:输入型界面元素,例如按键(Button),文本输入框(Text),滑动条(Scroll Bar),菜单(Menu)等等;以及输出型界面元素,例如视窗(Window),标签(Label),图像,视频,动画等等。The electronic device 10 may include one or more display components 323. The electronic device 10 may jointly implement a display function through the display component 323, a graphics processing unit (GPU) and an application processor (AP) in the chip 310, and so on. The GPU is a microprocessor used for image processing, and is connected to the display component 323 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The display component 323 is used to display the interface content currently output by the system, such as displaying images and videos in virtual scenes such as AR/VR/MR. The interface content can include the interface of the running application and the system-level menu, etc., which can be specified by the following The composition of the interface elements: input interface elements, such as buttons (Button), text input boxes (Text), scroll bars (Scroll Bar), menus (Menu), etc.; and output interface elements, such as windows, labels (Label), images, videos, animations, etc.
显示组件323在具体实现中,可以是显示面板,镜片(例如VR眼镜),投影屏等。显示面板也可以称为显示屏,例如可以是触摸屏、柔性屏、曲面屏等,或者其他光学组件。也即是说在本申请中电子设备具有显示屏时,显示屏可以是触摸屏、柔性屏、曲面屏或者其它形式的屏幕,电子设备的显示屏具有显示图像的功能,至于显示屏的具体材质以及形状本申请对此不作任何限定。In a specific implementation, the display component 323 may be a display panel, lenses (for example, VR glasses), a projection screen, and the like. The display panel may also be called a display screen, for example, it may be a touch screen, a flexible screen, a curved screen, etc., or other optical components. That is to say, when the electronic device has a display screen in this application, the display screen can be a touch screen, a flexible screen, a curved screen or other forms of screen. The display screen of the electronic device has the function of displaying images. As for the specific material of the display screen and The shape is not limited in this application.
例如,当显示组件323包括显示面板时,显示面板可以采用液晶显示屏(liquid crystal display,LCD),有机发光二极管(organic light-emitting diode,OLED),有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode的,AMOLED),柔性发光二极管(flex light-emitting diode,FLED),Miniled,MicroLed,Micro-oLed,量子点发光二极管(quantum dot light emitting diodes,QLED)等。此外,一些具体实现中,可以将用户接口322中的触控面板和显示组件323中的显示面板耦合在一起设置,例如触控面板可设置于显示面板下方,触控面板可用于检测用户通过显示面板输入触控操作(例如点击、滑动,触摸等)时作用于显示面板上的触控压力,显示面板用于进行内容显示。For example, when the display component 323 includes a display panel, the display panel may use a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode or an active matrix. Organic light-emitting diodes (active-matrix organic light emitting diodes, AMOLED), flexible light-emitting diodes (flex light-emitting diodes, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes, QLED) etc. In addition, in some specific implementations, the touch panel in the user interface 322 and the display panel in the display assembly 323 can be coupled together to set up. For example, the touch panel can be set under the display panel, and the touch panel can be used to detect that the user passes through the display panel. The touch pressure that acts on the display panel when a touch operation (such as click, slide, touch, etc.) is input from the panel, and the display panel is used for content display.
摄像头324可以是单目摄像头或双目摄像头或深度摄像头,用于对环境进行拍摄/录像,以获得图像/视频图像。摄像头324所采集的图像/视频图像例如可作为SLAM系统的一种输入数据,或者可通过显示组件323进行图像/视频显示。The camera 324 may be a monocular camera, a binocular camera, or a depth camera, and is used to photograph/video the environment to obtain an image/video image. The image/video image collected by the camera 324 may be used as a kind of input data of the SLAM system, or the image/video image may be displayed through the display component 323, for example.
在一些场景中,也可以把摄像头324视为一种传感器。摄像头324采集的图像可以是IMG格式,也可以是其他格式类型,这里不做限定。In some scenarios, the camera 324 can also be regarded as a sensor. The image collected by the camera 324 may be in IMG format or other format types, which is not limited here.
传感器325可用于采集与电子设备10的状态变化(例如旋转、摆动、移动、抖动等)相关的数据,传感器325所采集的数据例如可作为SLAM系统的一种输入数据。传感器325可包括一种或多种传感器,例如惯性测量单元(Inertial Measurement Unit,IMU),飞行时间(Time of Flight,TOF)传感器等。其中IMU中可进一步包括陀螺仪、加速度计等传感器。陀螺仪可用于测量电子设备运动时的角速度,加速度计用于测量电子设备运动时的加速度。TOF传感器可进一步包括光发射器和光接收器,光发射器可用于向外发射光线,例如激光、红外线、雷达波等,光接收器可用于检测反射的光线,例如反射的激光、红外线、雷达波等。The sensor 325 can be used to collect data related to the state change (such as rotation, swing, movement, jitter, etc.) of the electronic device 10, and the data collected by the sensor 325 can be used as a kind of input data of the SLAM system, for example. The sensor 325 may include one or more sensors, such as an Inertial Measurement Unit (IMU), a Time of Flight (TOF) sensor, and so on. The IMU may further include sensors such as gyroscopes and accelerometers. The gyroscope can be used to measure the angular velocity of the electronic device when it moves, and the accelerometer is used to measure the acceleration of the electronic device when it moves. The TOF sensor can further include a light transmitter and a light receiver. The light transmitter can be used to emit light, such as laser, infrared, and radar waves, and the light receiver can be used to detect reflected light, such as reflected laser, infrared, and radar waves. Wait.
需要说明的,传感器325还可以包括更多的其他传感器,例如惯性传感器,气压计,磁力计,轮速计等。It should be noted that the sensor 325 may also include more other sensors, such as inertial sensors, barometers, magnetometers, wheel speedometers, and so on.
定位模块331用于实现对电子设备10的物理定位,例如用于获取电子设备10的初始位置。定位模块331例如可包括WIFI定位模块、蓝牙定位模块、基站定位模块、卫星定位模块中的一个或多个。其中卫星定位模块中可设置有全球导航卫星系统(Global Navigation Satellite System,GNSS)以辅助定位,GNSS不限于北斗系统、GPS系统、GLONASS系统、Galileo系统。The positioning module 331 is used to implement physical positioning of the electronic device 10, for example, to obtain the initial position of the electronic device 10. The positioning module 331 may include, for example, one or more of a WIFI positioning module, a Bluetooth positioning module, a base station positioning module, and a satellite positioning module. Among them, the satellite positioning module can be equipped with a Global Navigation Satellite System (GNSS) to assist positioning. GNSS is not limited to the Beidou system, GPS system, GLONASS system, and Galileo system.
收发器332用于实现电子设备10与服务器或其他电子设备之间的通信。收发器332集成了发射器和接收器,分别用于发送和接收射频信号。具体实现中,收发器332可包括但不限于:天线系统、RF收发器、一个或多个放大器、调谐器、一个或多个振荡器、数字信号处理器、CODEC芯片、SIM卡和存储介质等。在一些实施例中,还可在单独的芯片上实现收发器332。收发器332例如可支持通过2G/3G/4G/5G等中的至少一种的数据网络通信,和/或支持以下近距无线通信的方式中的至少一种:蓝牙(Bluetooth,BT)通信,无线保真(WirelessFidelity,WiFi)通信,近场通信(Near Field Communication,NFC),红外(Infrared,IR)无线通信,超宽带(UWB,Ultra Wide Band)通信,ZigBee通信。The transceiver 332 is used to implement communication between the electronic device 10 and a server or other electronic devices. The transceiver 332 integrates a transmitter and a receiver, which are used to send and receive radio frequency signals, respectively. In a specific implementation, the transceiver 332 may include, but is not limited to: an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chip, a SIM card, a storage medium, etc. . In some embodiments, the transceiver 332 may also be implemented on a separate chip. The transceiver 332 may, for example, support data network communication through at least one of 2G/3G/4G/5G, etc., and/or support at least one of the following short-range wireless communication methods: Bluetooth (BT) communication, Wireless Fidelity (WiFi) communication, Near Field Communication (NFC), Infrared (IR) wireless communication, Ultra Wide Band (UWB) communication, ZigBee communication.
本申请实施例中,处理器311通过运行存储在存储器315的指令,从而执行电子设备10的各种功能应用以及数据处理,具体的,可执行如图7实施例所示方法步骤,或者可执行如图9,图10,图15或图17实施例中电子设备侧的功能。In the embodiment of the present application, the processor 311 executes various functional applications and data processing of the electronic device 10 by running instructions stored in the memory 315. Specifically, the processor 311 may execute the method steps shown in the embodiment of FIG. 7 or may execute The functions of the electronic device side in the embodiment shown in Figure 9, Figure 10, Figure 15 or Figure 17.
参见图3,图3是本申请实施例提供的一种服务器20的实施方式的结构框图。服务器20可以是图1实施例中所描述的服务器。如图3所示,服务器20包括处理器403、存储器401(一个或多个计算机可读存储介质)、收发器402。这些部件可在一个或多个通信总线404上通信。其中:Referring to FIG. 3, FIG. 3 is a structural block diagram of an implementation manner of a server 20 according to an embodiment of the present application. The server 20 may be the server described in the embodiment of FIG. 1. As shown in FIG. 3, the server 20 includes a processor 403, a memory 401 (one or more computer-readable storage media), and a transceiver 402. These components may communicate on one or more communication buses 404. among them:
处理器403可以是一个或多个中央处理器(Central Processing Unit,CPU),在处理器403是一个CPU的情况下,该CPU可以是单核CPU,也可以是多核CPU。The processor 403 may be one or more central processing units (CPU). In the case where the processor 403 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.
存储器401可与处理器403通过总线连接,也可以与处理器403耦合在一起,用于存储各种软件程序和/或多组指令、以及数据(例如地图数据、位姿数据等)。具体实现中,存储器401包括但不限于是随机存储记忆体(Random Access Memory,RAM)、只读存储器(Read-Only Memory,ROM)、可擦除可编程只读存储器(Erasable Programmable Read Only Memory,EPROM)、或便携式只读存储器(Compact Disc Read-Only Memory,CD-ROM)。The memory 401 may be connected with the processor 403 via a bus, or may be coupled with the processor 403 to store various software programs and/or multiple sets of instructions and data (for example, map data, pose data, etc.). In specific implementation, the memory 401 includes, but is not limited to, random access memory (RAM), read-only memory (Read-Only Memory, ROM), and erasable programmable read-only memory (Erasable Programmable Read Only Memory, EPROM), or portable read-only memory (Compact Disc Read-Only Memory, CD-ROM).
收发器402主要集成了接收器和发射器,其中接收器用于接收电子设备发送的数据(例如请求、图像等),发射器用于向电子设备发送数据(例如地图数据、位姿数据等)。The transceiver 402 mainly integrates a receiver and a transmitter, where the receiver is used to receive data (such as requests, images, etc.) sent by the electronic device, and the transmitter is used to send data (such as map data, pose data, etc.) to the electronic device.
应当理解,上述服务器20仅为本申请实施例提供的一个例子,具体实现中,服务器20可具有比图示更多的部件。It should be understood that the above-mentioned server 20 is only an example provided by the embodiment of the present application. In a specific implementation, the server 20 may have more components than shown in the figure.
本申请具体实施例中处理器403可用于调用存储器401中的程序指令,执行如图9,图10,图15或图17实施例中服务器侧的功能。In the specific embodiment of the present application, the processor 403 may be used to call program instructions in the memory 401, and execute the server-side functions in the embodiment shown in FIG. 9, FIG. 10, FIG. 15 or FIG.
本文所使用的术语“耦合”意指直接连接到、或通过一个或多个居间组件或电路来连接。本文所描述的在各种总线上提供的任何信号可以与其他信号进行时间复用并且在一条或多条 共用总线上提供。另外,各电路元件或软件块之间的互连可被示为总线或单信号线。每条总线可替换地为单信号线,而每条单信号线可替换地为总线,并且单线或总线可表示用于各组件之间的通信的大量物理或逻辑机制中的任一个或多个。The term "coupled" as used herein means directly connected to or connected through one or more intervening components or circuits. Any signal provided on the various buses described herein can be time-multiplexed with other signals and provided on one or more shared buses. In addition, the interconnection between various circuit elements or software blocks may be shown as a bus or a single signal line. Each bus may alternatively be a single signal line, and each single signal line may alternatively be a bus, and a single wire or bus may represent any one or more of a large number of physical or logical mechanisms for communication between components .
在一种现有的全局地图更新方案中,当电子设备向服务器发出全局定位的请求时,电子设备向服务器上传图像,在服务器基于图像执行全局定位,同时触发在服务器进行全局地图的更新。具体的,服务器对电子设备上传的图像提取特征点,将上传图像作为节点,插入到全局地图中,通过全局非线性优化算法来优化全局地图,从而实现全局地图的更新。这种方案中,一方面,电子设备的全局定位请求通常是低频率的,上传图像频次少,所以采用这种方案更新全局地图会比较慢。另一方面,假设电子设备可以频繁向服务器上传图像来更新服务器中的全局地图,那么不仅电子设备要消耗更多的通信资源,当多用户同时上传图像时,服务器负载也会特别大。再一方面,电子设备直接上传图像到服务器会存在隐私泄漏的风险。In an existing global map update solution, when an electronic device sends a global positioning request to the server, the electronic device uploads an image to the server, the server performs global positioning based on the image, and at the same time triggers the update of the global map on the server. Specifically, the server extracts feature points from the image uploaded by the electronic device, inserts the uploaded image as a node into the global map, and optimizes the global map through a global nonlinear optimization algorithm, thereby realizing the update of the global map. In this solution, on the one hand, the global positioning request of the electronic device is usually low-frequency, and the frequency of uploading images is low, so it will be slower to update the global map with this solution. On the other hand, assuming that the electronic device can frequently upload images to the server to update the global map in the server, not only the electronic device consumes more communication resources, but also when multiple users upload images at the same time, the server load will be particularly heavy. On the other hand, if the electronic device directly uploads the image to the server, there is a risk of privacy leakage.
而实施本申请实施例,能够克服现有方案的缺陷,实现高效率的全局地图更新。参见图5,图5是本申请实施例提供的一种系统的结构框图。该系统包括电子设备10和服务器20,电子设备10和服务器20之间可通过各自的收发器实现通信连接。在电子设备10中配置SLAM系统12、数据采集模块13、交互模块14和通信模块11,SLAM系统12、数据采集模块13、交互模块14和通信模块11可以以软件代码的形式存在,在一具体实现中,这些功能模块的数据/程序可被存储于如图2所示的存储器315,并可运行于如图2所示的处理器311。其中:However, implementing the embodiments of the present application can overcome the shortcomings of the existing solutions and achieve high-efficiency global map update. Refer to FIG. 5, which is a structural block diagram of a system provided by an embodiment of the present application. The system includes an electronic device 10 and a server 20, and the electronic device 10 and the server 20 can communicate with each other through respective transceivers. The SLAM system 12, the data acquisition module 13, the interaction module 14, and the communication module 11 are configured in the electronic device 10. The SLAM system 12, the data acquisition module 13, the interaction module 14 and the communication module 11 may exist in the form of software code, in a specific In implementation, the data/programs of these functional modules can be stored in the memory 315 shown in FIG. 2 and can be run on the processor 311 shown in FIG. 2. among them:
通信模块11可利用如图2所示的收发器332,实现与服务器20的通信,具体的,通信模块11被配置成从服务器20获取全局子地图,所述全局子地图是服务器20所存储的全局地图中与电子设备10的位置信息关联的子地图。并将全局子地图存储到电子设备10的SLAM系统12中的全局子地图的数据库123中。The communication module 11 can use the transceiver 332 shown in FIG. 2 to communicate with the server 20. Specifically, the communication module 11 is configured to obtain a global sub-map from the server 20, and the global sub-map is stored by the server 20. A sub-map associated with the location information of the electronic device 10 in the global map. And the global sub-map is stored in the database 123 of the global sub-map in the SLAM system 12 of the electronic device 10.
数据采集模块13被配置成利用如图2所示的传感器325来获得电子设备10的状态数据,利用如图2所示的摄像头324来获得视频图像,以及利用如图2所示的定位模块331来获得电子设备的位置。The data acquisition module 13 is configured to use the sensor 325 shown in FIG. 2 to obtain state data of the electronic device 10, the camera 324 shown in FIG. 2 to obtain video images, and the positioning module 331 shown in FIG. 2 To get the location of the electronic device.
交互模块14被配置成利用如图2所示的用户接口322实现用户操作的检测获得,以及利用如图2所示的显示组件323实现图像/视频/虚拟物体等的显示,例如AR/VR/MR等应用内容的显示。The interaction module 14 is configured to use the user interface 322 shown in FIG. 2 to realize the detection and acquisition of user operations, and to use the display component 323 shown in FIG. 2 to realize the display of images/videos/virtual objects, such as AR/VR/ Display of application content such as MR.
SLAM系统12中的计算模块121被配置成根据摄像头324采集的视频图像和所下载的全局子地图执行位姿计算,以得到电子设备10的位姿数据;交互模块14可基于电子设备10的位姿数据,在显示组件上显示虚拟物体。The calculation module 121 in the SLAM system 12 is configured to perform pose calculation according to the video image collected by the camera 324 and the downloaded global submap to obtain the pose data of the electronic device 10; the interaction module 14 may be based on the position of the electronic device 10. Pose data to display virtual objects on the display component.
SLAM系统12自身所构建的SLAM地图保存在SLAM地图的数据库122中,SLAM系统还被配置成基于电子设备10的位姿数据来更新数据库122中的SLAM地图。The SLAM map constructed by the SLAM system 12 itself is stored in the SLAM map database 122, and the SLAM system is also configured to update the SLAM map in the database 122 based on the pose data of the electronic device 10.
具体实施例中,电子设备10中的各功能模块可相互配合,以执行如图7实施例所示方法中的步骤,或者执行如图9,图10,图15或图17实施例中电子设备侧的功能。In a specific embodiment, the functional modules in the electronic device 10 can cooperate with each other to execute the steps in the method shown in the embodiment shown in FIG. 7, or execute the electronic device in the embodiment shown in FIG. 9, FIG. 10, FIG. 15 or FIG. Side function.
在服务器20中配置通信模块21、地图更新模块22、全局地图的数据库23。通信模块21、处理模块22和全局地图的数据库23可以以软件代码的形式存在。在一具体实现中,这些功 能模块的数据/程序可被存储于如图3所示的存储器401,并可运行于如图3所示的处理器403。其中:The server 20 is configured with a communication module 21, a map update module 22, and a database 23 of the global map. The communication module 21, the processing module 22, and the database 23 of the global map may exist in the form of software code. In a specific implementation, the data/programs of these functional modules can be stored in the memory 401 shown in FIG. 3 and can be run on the processor 403 shown in FIG. 3. among them:
全局地图的数据库23用于存储、维护和更新全局地图。The database 23 of the global map is used to store, maintain and update the global map.
地图更新模块22可被配置成基于电子设备上传的地图更新信息来更新数据库23中的全局地图。The map update module 22 may be configured to update the global map in the database 23 based on the map update information uploaded by the electronic device.
通信模块21可利用如图3所示的收发器402,实现与电子设备10的通信。具体的,通信模块21可将全局子地图发送给电子设备10,以及接收来自电子设备10的地图更新信息。The communication module 21 can use the transceiver 402 as shown in FIG. 3 to realize communication with the electronic device 10. Specifically, the communication module 21 may send the global sub-map to the electronic device 10 and receive map update information from the electronic device 10.
具体实施例中,服务器20中的各功能模块可相互配合,以执行如图9,图10,图15或图17实施例中服务器侧的功能。In a specific embodiment, the functional modules in the server 20 can cooperate with each other to perform the server-side functions in the embodiment shown in FIG. 9, FIG. 10, FIG. 15 or FIG. 17.
参见图5,图5示出了在一种具体实现中,图4所示的电子设备10中的各功能模块进一步可能包含的组件(或称子模块),以及服务器20中的各功能模块进一步可能包含的组件(或称子模块)。需要说明的是,电子设备10中的各功能模块(例如SLAM系统、数据采集模块13,交互模14)进一步可能包含的组件和服务器20中的各功能模块(例如处理模块22)进一步可能包含的组件(或称子模块)仅为本申请实施例的示例,在其他实施例中,上述功能模块还可以包含更多或更少的组件(或称子模块)。Referring to FIG. 5, FIG. 5 shows the components (or sub-modules) that may be further included in each functional module in the electronic device 10 shown in FIG. 4 in a specific implementation, and the functional modules in the server 20 are further The components (or sub-modules) that may be included. It should be noted that the functional modules in the electronic device 10 (for example, the SLAM system, the data acquisition module 13, the interactive module 14) may further include components and the function modules in the server 20 (for example, the processing module 22) may further include The components (or sub-modules) are only examples of the embodiments of the present application. In other embodiments, the above-mentioned functional modules may also include more or fewer components (or sub-modules).
如图5所示,SLAM系统12中的计算模块121进一步包括建图模块1211,位姿估计模块1212,位姿估计模块1212、特征处理模块1213和特征匹配模块1214和地图融合模块1215。此外,在电子设备10中还包括全局定位模块16和软件开发工具包(Software Development Kit,SDK)。全局定位模块16进一步包括图像检索模块161、特征提取模块162、特征匹配模块163和位姿估计模块164。SDK中可包括分别用于全局位姿和局部位姿的数据库以及坐标系变化矩阵计算模块15,SDK还可以调用交互模块14通过显示组件实现显示。As shown in FIG. 5, the calculation module 121 in the SLAM system 12 further includes a mapping module 1211, a pose estimation module 1212, a pose estimation module 1212, a feature processing module 1213, a feature matching module 1214, and a map fusion module 1215. In addition, the electronic device 10 also includes a global positioning module 16 and a software development kit (Software Development Kit, SDK). The global positioning module 16 further includes an image retrieval module 161, a feature extraction module 162, a feature matching module 163, and a pose estimation module 164. The SDK may include a database for the global pose and the local pose respectively, and the coordinate system change matrix calculation module 15. The SDK may also call the interaction module 14 to realize display through display components.
特征处理模块1213可用于与视觉特征处理相关的操作,例如在一种实施例中特征处理模块1213可进一步包括特征检测功能和特征描述功能,特征检测功能是在图像中提取出特征的图像位置,特征描述功能是对每一个检测出的特征进行描述,形成一个一维向量。特征匹配模块1214可用于实现图像特征与地图特征之间特征匹配。位姿估计模块1212可进一步根据特征匹配的结果进行位姿估计。The feature processing module 1213 can be used for operations related to visual feature processing. For example, in an embodiment, the feature processing module 1213 can further include a feature detection function and a feature description function. The feature detection function is to extract the image position of the feature in the image, The feature description function is to describe each detected feature to form a one-dimensional vector. The feature matching module 1214 can be used to implement feature matching between image features and map features. The pose estimation module 1212 may further perform pose estimation according to the result of feature matching.
数据采集模块13(例如IMU传感器)能输出高频率的角速度、线加速度,位姿估计模块1212对角加速度、线加速度分别积分,并结合摄像头拍摄的视频图像进行位姿估计,可计算出电子设备的位置和姿态。位姿估计的结果可作为SLAM系统输出。同时,位姿估计的结果也可作为建图模块1211的输入。建图模块1211在局部坐标系下,创建SLAM系统能感知到的环境地图,即SLAM地图。SLAM系统在空间中不断运行的过程中,不断在创建/更新SLAM地图。在一些实施例中,SLAM地图反过来也可以作为位姿估计模块1212进行位姿估计的输入,提高位姿估计的准确度。The data acquisition module 13 (such as an IMU sensor) can output high-frequency angular velocity and linear acceleration. The pose estimation module 1212 integrates the angular acceleration and linear acceleration separately, and combines the video images taken by the camera to estimate the pose and calculate the electronic equipment Position and posture. The result of pose estimation can be output as SLAM system. At the same time, the result of the pose estimation can also be used as the input of the mapping module 1211. The mapping module 1211 creates an environment map perceivable by the SLAM system under the local coordinate system, that is, the SLAM map. As the SLAM system continues to operate in the space, it is constantly creating/updating SLAM maps. In some embodiments, the SLAM map can also be used as the input of the pose estimation module 1212 to estimate the pose, so as to improve the accuracy of the pose estimation.
在一种实施例中,数据库123中保存有从服务器10下载的全局子地图,位姿估计模块1212被配置成根据所述全局子地图和摄像头324采集的视频图像以及传感器325采集的运动数据执行位姿计算以得到电子设备10的位姿数据,即获得电子设备10的全局位姿,从而实现对电子设备10的位姿的跟踪定位。所述运动数据包括电子设备10的运动速度数据和运动 方向数据,例如加速度、角速度等。具体的,特征处理模块1213可用于提取视频图像的2D特征,位姿估计模块1212可根据所述视频图像的2D特征、全局子地图的3D地图点以及数据采集模块13(例如IMU)采集的运动数据,获得电子设备10在所述全局子地图中的全局位姿。In an embodiment, the database 123 stores a global sub-map downloaded from the server 10, and the pose estimation module 1212 is configured to perform execution based on the global sub-map and the video images collected by the camera 324 and the motion data collected by the sensor 325. The pose calculation is used to obtain the pose data of the electronic device 10, that is, to obtain the global pose of the electronic device 10, so as to realize the tracking and positioning of the pose of the electronic device 10. The movement data includes movement speed data and movement direction data of the electronic device 10, such as acceleration and angular velocity. Specifically, the feature processing module 1213 can be used to extract 2D features of the video image, and the pose estimation module 1212 can be based on the 2D features of the video image, the 3D map points of the global submap, and the movement collected by the data collection module 13 (for example, IMU). Data to obtain the global pose of the electronic device 10 in the global sub-map.
在一些实施例中,SLAM系统的位姿估计模块1212也可用于根据摄像头采集的视频图像和局部坐标系下的SLAM地图,确定电子设备10在局部坐标系下的SLAM地图中的局部位姿。具体的,特征处理模块1213可用于提取所述视频图像中的2D特征,位姿估计模块1212可根据所述视频图像中的2D特征、局部坐标系下的SLAM地图的3D地图点以及数据采集模块13(例如IMU)采集的运动数据,获得电子设备10在所述局部坐标系下的SLAM地图中的局部位姿;所述运动数据包括电子设备10的运动速度数据和运动方向数据,例如加速度、角速度等。In some embodiments, the pose estimation module 1212 of the SLAM system can also be used to determine the local pose of the electronic device 10 in the SLAM map in the local coordinate system based on the video image collected by the camera and the SLAM map in the local coordinate system. Specifically, the feature processing module 1213 can be used to extract 2D features in the video image, and the pose estimation module 1212 can be based on the 2D features in the video image, the 3D map points of the SLAM map in the local coordinate system, and the data collection module. 13 (such as IMU) collected motion data to obtain the local pose of the electronic device 10 in the SLAM map in the local coordinate system; the motion data includes the motion speed data and motion direction data of the electronic device 10, such as acceleration, Angular velocity etc.
全局位姿模块16用于根据所采集的视频图像和全局子地图,确定所述电子设备10在全局子地图中的全局位姿。具体的,可通过图像检索模块161获取视频图像,通过特征提取模块162根据所述视频图像进行特征提取,获得图像特征,通过特征匹配模块163将所述图像特征在全局子地图中进行特征匹配,获得与所述图像特征匹配的地图特征;通过位姿估计模块164根据所述图像特征和所述地图特征,计算获得所述电子设备10在全局子地图中的全局位姿。The global pose module 16 is used to determine the global pose of the electronic device 10 in the global sub-map according to the collected video image and the global sub-map. Specifically, the video image can be obtained by the image retrieval module 161, the feature extraction module 162 can perform feature extraction based on the video image to obtain the image feature, and the feature matching module 163 can perform feature matching on the image feature in the global submap. A map feature matching the image feature is obtained; the pose estimation module 164 calculates and obtains the global pose of the electronic device 10 in the global sub-map according to the image feature and the map feature.
上述局部位姿和全局位姿可分别保存于SDK中的数据库。坐标系变化矩阵计算模块15可配置成根据所述同一视频图像的局部位姿和全局位姿,计算获得所述SLAM地图的局部坐标系和所述全局地图的全局坐标系之间的坐标系变换信息(例如,坐标系变换矩阵)。并将该坐标系变换信息反馈给SLAM系统。The above-mentioned local pose and global pose can be stored in the database in the SDK respectively. The coordinate system change matrix calculation module 15 may be configured to calculate the coordinate system transformation between the local coordinate system of the SLAM map and the global coordinate system of the global map according to the local pose and the global pose of the same video image. Information (for example, coordinate system transformation matrix). And the coordinate system transformation information is fed back to the SLAM system.
SLAM系统中的建图模块1211可被配置成根据电子设备10的位姿数据更新SLAM系统的数据库122中的SLAM地图。具体的,建图模块1211可将根据坐标系变换信息将SLAM地图变换到全局坐标系下,以电子设备10的全局位姿作为在全局坐标系下的SLAM地图中的位姿数据来更新所述全局坐标系下的SLAM地图。The mapping module 1211 in the SLAM system may be configured to update the SLAM map in the database 122 of the SLAM system according to the pose data of the electronic device 10. Specifically, the mapping module 1211 may transform the SLAM map to the global coordinate system according to the coordinate system transformation information, and use the global pose of the electronic device 10 as the pose data in the SLAM map in the global coordinate system to update the SLAM map in the global coordinate system.
SLAM系统运行过程中,建图模块1211会不断创建SLAM地图,体现在不断插入关键帧。SLAM地图中的关键帧和地图点可以变换到全局坐标系中表示。During the operation of the SLAM system, the mapping module 1211 will continue to create SLAM maps, which is reflected in the continuous insertion of key frames. The key frames and map points in the SLAM map can be transformed to the global coordinate system.
地图融合模块1215可用于比较SLAM地图和全局子地图,从而识别出当前的SLAM地图是否存在非冗余关键帧,若存在非冗余关键帧,则将所述非冗余关键帧融合到全局子地图,获得地图更新信息。所述地图更新信息可指示所述全局坐标系下的SLAM地图相对于全局坐标系下的全局子地图的差异。具体实施例中,所述地图更新信息包括:所述非冗余关键帧对应的位姿数据、三维地图点以及共视数据,所述共视数据用于指示所述非冗余关键帧与所述全局子地图的关键帧之间的所述共视关系。The map fusion module 1215 can be used to compare the SLAM map and the global sub-map, thereby identifying whether there are non-redundant key frames in the current SLAM map. If there is a non-redundant key frame, then the non-redundant key frames are merged into the global sub-map. Map, get map update information. The map update information may indicate the difference between the SLAM map in the global coordinate system and the global sub-map in the global coordinate system. In a specific embodiment, the map update information includes: pose data, three-dimensional map points, and common view data corresponding to the non-redundant key frame, and the common view data is used to indicate the non-redundant key frame and the common view data. The common view relationship between the key frames of the global sub-map.
如图6所示,本申请一种实施例中,地图融合模块1215执行地图融合的过程可被描述为:地图融合模块1215判断SLAM系统新插入的关键帧是否与下载的全局子地图中的关键帧存在冗余。关键帧冗余指关键帧关联的3D地图点信息在全局子地图中已经存在,即全局子地图已经保存了当前环境中的视觉信息,不需要再增加关键帧了。关键帧不冗余指关键帧关联的3D地图点信息在全局子地图中未存在(这样的关键帧可简称非冗余关键帧)。如果需要将非冗余关键帧插入到全局子地图,则在全局子地图中增加关键帧节点(无向图结构),并更新全局子 地图中的关键帧之间的共视关系。更新全局子地图的关键帧共视关系之后,利用三角化算法计算新的3D地图点,在通过BA算法优化关键帧的位姿和3D地图点坐标,从而实现在对全局子地图的更新,即获得了经更新的全局子地图(或称经融合的全局子地图)保存于数据库123。经更新的全局子地图可被应用到全局定位模块16进行全局位姿的计算。As shown in Figure 6, in an embodiment of the present application, the process of map fusion performed by the map fusion module 1215 can be described as: the map fusion module 1215 determines whether the key frame newly inserted by the SLAM system is the same as the key in the downloaded global submap The frame is redundant. Keyframe redundancy means that the 3D map point information associated with the keyframe already exists in the global submap, that is, the global submap has saved the visual information in the current environment, and there is no need to add keyframes. The key frame non-redundancy means that the 3D map point information associated with the key frame does not exist in the global sub-map (such key frames may be referred to as non-redundant key frames). If you need to insert non-redundant keyframes into the global submap, add keyframe nodes (undirected graph structure) in the global submap, and update the common view relationship between the keyframes in the global submap. After updating the key frame common view relationship of the global sub-map, use the triangulation algorithm to calculate new 3D map points, and optimize the pose and 3D map point coordinates of the key frames through the BA algorithm, so as to realize the update of the global sub-map, namely The obtained updated global sub-map (or called the fused global sub-map) is stored in the database 123. The updated global sub-map can be applied to the global positioning module 16 to calculate the global pose.
一些实施例中,经更新的全局子地图可用于为服务器20提供地图更新信息(例如非冗余关键帧对应的位姿,3D地图点,共视关系等),即电子设备10可从经更新的全局子地图获取该地图更新信息并发送给服务器20,以便于服务器20利用地图更新信息更新全局地图。In some embodiments, the updated global sub-map can be used to provide the server 20 with map update information (for example, the poses corresponding to non-redundant key frames, 3D map points, common view relationships, etc.), that is, the electronic device 10 can be updated from The global sub-map of obtains the map update information and sends it to the server 20, so that the server 20 uses the map update information to update the global map.
在又一些实施例中,电子设备10也直接将非冗余关键帧发送给服务器20;服务器20再进一步利用非冗余关键帧更新全局地图。经更新的全局子地图可被应用到全局定位模块16进行全局位姿的计算。In still other embodiments, the electronic device 10 also directly sends the non-redundant key frame to the server 20; the server 20 further uses the non-redundant key frame to update the global map. The updated global sub-map can be applied to the global positioning module 16 to calculate the global pose.
具体实施例中,电子设备10中的各功能模块可相互配合,以执行如图7实施例所示方法中的步骤,或者执行如图9,图10,图15或图17实施例中电子设备侧的功能。In a specific embodiment, the functional modules in the electronic device 10 can cooperate with each other to execute the steps in the method shown in the embodiment shown in FIG. 7, or execute the electronic device in the embodiment shown in FIG. 9, FIG. 10, FIG. 15 or FIG. Side function.
在一种实施例中,服务器20中除了地图更新模块22,还包括子地图更新模块221和位置指纹匹配模块222,地图更新模块22可被配置成将所述地图更新信息加入到所述全局地图,实现全局地图的更新;或者被配置成将非冗余关键帧融合到所述全局地图,实现全局地图的更新。位置指纹匹配模块222被配置成根据电子设备10的发送的初始位置的位置指纹信息(这里可称为第一位置指纹信息),在所述全局地图中寻找具有与所述第一位置指纹信息相匹配的位置指纹信息(这里可称为第二位置指纹信息)。子地图处理模块221用于在所述数据库23的全局地图中取出具有所述第二位置指纹信息的全局子地图。该全局子地图也可另外保存于服务器20的数据库,以便于下次快捷取用。服务器20可将具有所述第二位置指纹信息的全局子地图发送给电子设备10。In an embodiment, in addition to the map update module 22, the server 20 also includes a sub-map update module 221 and a location fingerprint matching module 222. The map update module 22 may be configured to add the map update information to the global map. , Realize the update of the global map; or be configured to merge non-redundant key frames into the global map to realize the update of the global map. The location fingerprint matching module 222 is configured to search the global map for the location fingerprint information of the initial location sent by the electronic device 10 (herein may be referred to as the first location fingerprint information). Matched location fingerprint information (here can be referred to as second location fingerprint information). The sub-map processing module 221 is configured to retrieve the global sub-map with the second location fingerprint information from the global map in the database 23. The global sub-map can also be additionally stored in the database of the server 20 for quick access next time. The server 20 may send the global sub-map with the second location fingerprint information to the electronic device 10.
具体实施例中,服务器20中的各功能模块可相互配合,以执行如图9,图10,图15或图17实施例中服务器侧的功能。In a specific embodiment, the functional modules in the server 20 can cooperate with each other to perform the server-side functions in the embodiment shown in FIG. 9, FIG. 10, FIG. 15 or FIG. 17.
基于上文的描述,下面给出本申请实施例提供的一些虚拟物体显示方法。对于下文描述的各方法实施例,为了方便起见,将其都表述为一系列的动作步骤的组合,但是本领域技术人员应该知悉,本申请技术方案的具体实现并不受所描述的一系列的动作步骤的顺序的限制。Based on the above description, some virtual object display methods provided in the embodiments of the present application are given below. For the method embodiments described below, for the sake of convenience, they are all expressed as a combination of a series of action steps. However, those skilled in the art should know that the specific implementation of the technical solution of the present application is not limited to the series of steps described. Restrictions on the order of action steps.
参见图7,图7是本申请实施例提供的一种虚拟物体显示方法的流程示意图,在一些实现中,该方法可应用于具有显示组件和摄像头的第一电子设备。该方法包括但不限于以下步骤:Referring to FIG. 7, FIG. 7 is a schematic flowchart of a virtual object display method provided by an embodiment of the present application. In some implementations, the method may be applied to a first electronic device having a display component and a camera. The method includes but is not limited to the following steps:
S1011、检测到用户打开应用的操作。S1011, an operation of opening an application by a user is detected.
S1012、响应于该操作,从服务器下载第一全局子地图并存储到电子设备的即时定位与地图构建(SLAM)系统中;所述第一全局子地图是全局地图中与所述电子设备的位置关联的子地图。S1012. In response to this operation, download a first global sub-map from the server and store it in the instant positioning and mapping (SLAM) system of the electronic device; the first global sub-map is the position of the electronic device in the global map The associated submap.
S1013、在显示组件上基于电子设备的位姿数据来显示虚拟物体,电子设备的位姿数据是至少根据所述摄像头采集的视频图像和所述第一全局子地图执行位姿计算得到的,其中所述服务器上的全局地图是根据一个或多个第二电子设备发送的地图更新信息进行更新得到的,其中,所述一个或多个第二电子设备为除所述第一电子设备外的电子设备。S1013. Display the virtual object based on the pose data of the electronic device on the display component, where the pose data of the electronic device is obtained by performing pose calculation at least according to the video image collected by the camera and the first global submap, wherein The global map on the server is updated according to map update information sent by one or more second electronic devices, where the one or more second electronic devices are electronic devices other than the first electronic device. equipment.
本申请实施例中,用户在电子设备上输入用于打开应用(application,APP)的操作,例如点击、触摸、滑动、抖动、声控等操作,响应于所述操作,一方面在电子设备的显示组件(例如显示面板或者镜片)上显示应用的界面,另一方面启动从服务器或者其他设备(例如其他终端设备或硬盘、USB等存储介质)下载全局子地图的进程。其中,应用可以是电子设备中安装的AR/VR/MR等应用。In the embodiment of the present application, the user inputs an operation for opening an application (APP) on the electronic device, such as clicking, touching, sliding, shaking, voice control, etc., in response to the operation, on the one hand, the display of the electronic device The application interface is displayed on the component (such as the display panel or the lens), and on the other hand, the process of downloading the global submap from the server or other devices (such as other terminal devices or hard disks, USB and other storage media) is started. Among them, the application may be an application such as AR/VR/MR installed in an electronic device.
本申请一些实施例中,服务器接收来自各个第二电子设备的地图更新信息,基于地图更新信息就可以不断地更新服务器中的全局地图。其中,地图更新信息是第二电子设备根据第二电子设备的SLAM地图与第二电子设备自己所下载的全局子地图的比较结果、以及第二电子设备自己的局部坐标系和全局子地图对应的全局坐标系之间的坐标系变换信息得到的。In some embodiments of the present application, the server receives map update information from each second electronic device, and can continuously update the global map in the server based on the map update information. Wherein, the map update information is based on the comparison result of the second electronic device's SLAM map and the global sub-map downloaded by the second electronic device, and the second electronic device's own local coordinate system and the global sub-map. The coordinate system transformation information between the global coordinate system is obtained.
在一种实施例中,地图更新信息具体可以包括:由所述第二电子设备的SLAM系统所构建的目标关键帧对应的位姿数据、三维地图点(或称3D地图点)以及共视数据,所述共视数据用于指示所述目标关键帧与第二电子设备中的全局子地图的关键帧之间的共视关系。其中,目标关键帧是非冗余关键帧,非冗余关键帧表示存在于所述第二电子设备的SLAM地图中但不存在于第二电子设备中的全局子地图中的关键帧。In an embodiment, the map update information may specifically include: pose data corresponding to the target key frame constructed by the SLAM system of the second electronic device, three-dimensional map points (or 3D map points), and common view data The common view data is used to indicate the common view relationship between the target key frame and the key frame of the global sub-map in the second electronic device. Wherein, the target key frame is a non-redundant key frame, and the non-redundant key frame represents a key frame that exists in the SLAM map of the second electronic device but does not exist in the global sub-map of the second electronic device.
在这样的全局地图更新方案中,充分利用了各个第二电子设备的计算资源来计算获得地图更新信息,减轻服务器计算负载。而地图更新信息相比起摄像头拍摄的图像数据而言,数据量更少,节省了传输带宽资源,而服务器能够基于第二电子设备的地图更新信息快速实现全局地图更新,更新效率高。由于服务器中的全局地图能够被高效、准确地更新,所以也促进了第一电子设备的全局定位的准确度更高,鲁棒性更好。In such a global map update solution, the computing resources of each second electronic device are fully utilized to calculate and obtain map update information, thereby reducing the calculation load of the server. Compared with the image data captured by the camera, the map update information has a smaller amount of data, which saves transmission bandwidth resources. The server can quickly implement global map update based on the map update information of the second electronic device, and the update efficiency is high. Since the global map in the server can be updated efficiently and accurately, it also promotes the global positioning of the first electronic device to have higher accuracy and better robustness.
示例性的,参见图8,在一种可能的实现场景中,以第一电子设备为电子设备10为例,图示中电子设备10为手机,电子设备10中安装有某种虚拟场景应用(例如,可以是带有AR功能的导航地图应用),图8中的(a)示出了电子设备10的显示面板上的一种图形用户界面(graphical user interface,GUI),该GUI为电子设备的桌面101。当电子设备检测到用户点击桌面101上的虚拟场景应用的图标102的操作后,一方面在后台启动下载全局子地图的流程,另一方面,在启动该虚拟场景应用后,在显示面板上显示如图8中的(b)所示的另一GUI,该GUI的用户界面103以AR导航界面为例。该用户界面103上可以包括取景框104。该取景框104内可以实时显示电子设备10所在的真实环境的预览视频流。该预览视频流是通过电子设备10的摄像头拍摄获得的。在该预览视频流上,还叠加有AR应用的虚拟物体。虚拟物体的数量可以是一个或多个,如图8中的(b)所示中,虚拟物体示例性地以导航指示105、互动机器人107为例,导航指示105能够通过方位箭头来实时指示当前位置到达某目的地的导航路线,互动机器人107可用来实现语音对话、语音介绍、或者仅仅作为街边趣味性的展示等等。Exemplarily, referring to FIG. 8, in a possible implementation scenario, taking the first electronic device as the electronic device 10 as an example, the electronic device 10 in the figure is a mobile phone, and a certain virtual scene application is installed in the electronic device 10 ( For example, it may be a navigation map application with AR function), (a) in FIG. 8 shows a graphical user interface (GUI) on the display panel of the electronic device 10, and the GUI is the electronic device Desktop 101. When the electronic device detects that the user has clicked the icon 102 of the virtual scene application on the desktop 101, on the one hand, it starts the process of downloading the global sub-map in the background; on the other hand, it displays on the display panel after starting the virtual scene application. As another GUI shown in (b) in FIG. 8, the user interface 103 of the GUI is an AR navigation interface as an example. The user interface 103 may include a viewing frame 104. The viewing frame 104 can display the preview video stream of the real environment where the electronic device 10 is located in real time. The preview video stream is captured by the camera of the electronic device 10. On the preview video stream, virtual objects of the AR application are also superimposed. The number of virtual objects can be one or more. As shown in Figure 8(b), the virtual objects exemplarily take the navigation indicator 105 and the interactive robot 107 as examples. The navigation indicator 105 can indicate the current position in real time through the azimuth arrow. The navigation route of the location to a certain destination, the interactive robot 107 can be used to implement voice conversations, voice introductions, or just as interesting displays on the street, and so on.
具体的,在电子设备10在后台(处理器)中,根据所获得的全局子地图和摄像头实时拍摄的视频图像,执行位姿计算,从而实时获得电子设备10当前在全局坐标系下的位姿数据(即全局位姿)。AR场景中虚拟物体的位置和姿态可由电子设备10的全局位姿来决定,即电子设备的全局位姿表征了所述虚拟物体的位置与姿态。所以,可以实时基于电子设备10的全局位姿,在该取景框104内显示AR场景中虚拟物体的位置与姿态。Specifically, in the background (processor) of the electronic device 10, the pose calculation is performed according to the obtained global sub-map and the video image captured by the camera in real time, so as to obtain the current pose of the electronic device 10 in the global coordinate system in real time. Data (ie global pose). The position and posture of the virtual object in the AR scene can be determined by the global posture of the electronic device 10, that is, the global posture of the electronic device represents the position and posture of the virtual object. Therefore, the position and posture of the virtual object in the AR scene can be displayed in the viewing frame 104 based on the global pose of the electronic device 10 in real time.
AR应用可借助计算机图形技术和可视化技术生成现实环境中不存在的虚拟物体,并基于 电子设备10当前的全局位姿,将虚拟物体叠加到取景框104内,从而在取景框104内把虚拟物体叠加到取景框104的视频流中。AR applications can use computer graphics and visualization techniques to generate virtual objects that do not exist in the real environment, and based on the current global pose of the electronic device 10, superimpose the virtual objects into the viewing frame 104, thereby placing the virtual objects in the viewing frame 104 It is superimposed on the video stream of the view frame 104.
举例来说,电子设备10通过摄像头捕获到视频图像后,AR应用向AR云服务器发送视频图像,以向AR云服务器请求获取与视频图像对应的待渲染对象,待渲染对象可以包括待渲染对象的标识和/或待渲染对象的名称、元数据等信息。AR云服务器将与视频图像对应的待渲染对象发给电子设备10,电子设备10确定出与该待渲染对象对应的业务规则,利用每个待渲染对象的业务规则渲染对应的待渲染对象,从而生成多个一个或多个AR对象(即虚拟物体),并基于电子设备10的全局位姿,把虚拟物体叠加到取景框104的视频流中。For example, after the electronic device 10 captures a video image through a camera, the AR application sends the video image to the AR cloud server to request the AR cloud server to obtain the object to be rendered corresponding to the video image. The object to be rendered may include the object to be rendered. Identification and/or information such as the name and metadata of the object to be rendered. The AR cloud server sends the object to be rendered corresponding to the video image to the electronic device 10. The electronic device 10 determines the business rule corresponding to the object to be rendered, and uses the business rule of each object to be rendered to render the corresponding object to be rendered, thereby Multiple one or more AR objects (ie virtual objects) are generated, and based on the global pose of the electronic device 10, the virtual objects are superimposed on the video stream of the view frame 104.
当用户手持电子设备10行走一段时间(例如1分钟、5分钟、10分钟等,这里不做限定),电子设备10依然能够持续将虚拟物体准确地叠加在取景框104的视频流中,提升用户使用体验。在一种实施例中,当电子设备10快要移出全局子地图的地理范围时,可以基于电子设备10的位置提前请求下载新的全局子地图,后续电子设备10即可根据新的全局子地图来执行全局位姿估计,从而能进一步避免在两个全局子地图对应的地理范围切换时位姿突变的发生,进一步提升用户使用体验。When the user holds the electronic device 10 and walks for a period of time (for example, 1 minute, 5 minutes, 10 minutes, etc., which are not limited here), the electronic device 10 can continue to accurately superimpose virtual objects on the video stream of the viewfinder 104 to improve the user Use experience. In one embodiment, when the electronic device 10 is about to move out of the geographic range of the global sub-map, it can request to download a new global sub-map in advance based on the location of the electronic device 10, and the subsequent electronic device 10 can use the new global sub-map to download Performing the global pose estimation can further avoid the occurrence of sudden changes in the pose when the geographic ranges corresponding to the two global submaps are switched, and further improve the user experience.
参见图9,图9是本申请实施例提供的一种全局地图更新方法的流程示意图,在一些实现中,该方法从电子设备和服务器侧分别进行描述,该电子设备可以是前文所述的第一电子设备,也可以是前文所述的第二电子设备。该方法包括但不限于以下步骤:Referring to FIG. 9, FIG. 9 is a schematic flowchart of a global map update method provided by an embodiment of the present application. In some implementations, the method is described separately from the electronic device and the server side. The electronic device may be the first described above. An electronic device may also be the second electronic device described above. The method includes but is not limited to the following steps:
S201、服务器基于电子设备的请求,下发全局子地图给电子设备。相应的,电子设备接收该全局子地图。该全局子地图是全局地图中与电子设备的位置关联的子地图。S201. The server issues a global sub-map to the electronic device based on the request of the electronic device. Correspondingly, the electronic device receives the global sub-map. The global sub-map is a sub-map associated with the location of the electronic device in the global map.
S202、电子设备将该全局子地图存储到所述电子设备的SLAM系统中。S202. The electronic device stores the global sub-map in the SLAM system of the electronic device.
S203、电子设备根据自己构建的SLAM地图和全局子地图,获得地图更新信息。S203. The electronic device obtains map update information according to the SLAM map and global sub-map constructed by itself.
其中,该地图更新信息可用于指示在全局坐标系下的SLAM地图相对于全局坐标系下的全局子地图的差异。The map update information can be used to indicate the difference between the SLAM map in the global coordinate system and the global sub-map in the global coordinate system.
在一种实现中,电子设备可利用局部坐标系和全局坐标系两者之间的坐标系变换信息,将自身构建的SLAM地图转换到全局坐标系下,再通过比较全局坐标系下的SLAM地图和全局子地图来获得比较结果(此时的比较结果为全局坐标系下的比较结果),进而根据比较结果来获得地图更新信息。In one implementation, the electronic device can use the coordinate system transformation information between the local coordinate system and the global coordinate system to convert the SLAM map built by itself to the global coordinate system, and then compare the SLAM map in the global coordinate system The comparison result is obtained with the global sub-map (the comparison result at this time is the comparison result in the global coordinate system), and then the map update information is obtained according to the comparison result.
例如,该比较结果可以是全局坐标系下的SLAM地图中的至少一帧关键帧,也就是说,该至少一帧关键帧不存在于全局子地图中,或者与全局子地图中的各个关键帧均有显著差异,那么可以根据该至少一帧关键帧获得地图更新信息,地图更新信息可能是该至少一帧关键帧融合进全局子地图后的相关信息(例如位姿、共视关系、3D地图点等信息),地图更新信息也可能直接就是该至少一帧关键帧。For example, the comparison result may be at least one key frame in the SLAM map in the global coordinate system, that is, the at least one key frame does not exist in the global sub-map, or is different from each key frame in the global sub-map. If there are significant differences, the map update information can be obtained according to the at least one key frame. The map update information may be related information after the at least one key frame is merged into the global sub-map (such as pose, common view relationship, 3D map Point and other information), the map update information may also directly be the at least one key frame.
在又一种实现中,电子设备首先直接根据局部坐标系下的SLAM地图与全局坐标系下的全局子地图获得比较结果(此时的比较结果为局部坐标系下的比较结果),进而利用局部坐标系和全局坐标系两者之间的坐标系变换信息,将比较结果转换到全局坐标系下,从而获得地图更新信息。In another implementation, the electronic device first obtains the comparison result directly according to the SLAM map in the local coordinate system and the global sub-map in the global coordinate system (the comparison result at this time is the comparison result in the local coordinate system), and then uses the local The coordinate system transformation information between the coordinate system and the global coordinate system, and the comparison result is converted to the global coordinate system, thereby obtaining map update information.
例如,该比较结果可以是局部坐标系下的SLAM地图中的至少一帧关键帧,该至少一帧关 键帧被转换到全局坐标系下后也不存在于全局子地图中、或者与全局子地图中的各个关键帧均有显著差异,那么可以根据该至少一帧关键帧获得地图更新信息,地图更新信息可能是该至少一帧关键帧被转换到全局坐标系下后再融合进全局子地图后的相关信息(例如位姿、共视关系、3D地图点等信息),地图更新信息也可能直接就是该至少一帧关键帧转换到全局坐标系下形成的关键帧。For example, the comparison result may be at least one key frame in the SLAM map in the local coordinate system, and the at least one key frame does not exist in the global sub-map after being converted to the global coordinate system, or is different from the global sub-map. Each key frame in has a significant difference, then the map update information can be obtained according to the at least one key frame. The map update information may be that the at least one key frame is converted to the global coordinate system and then merged into the global sub-map Related information (such as pose, common view relationship, 3D map point, etc.), map update information may also be directly the key frame formed by the conversion of the at least one key frame to the global coordinate system.
本文中为了便于理解,上述该至少一帧关键帧还可以称为非冗余关键帧。For ease of understanding herein, the aforementioned at least one key frame may also be referred to as a non-redundant key frame.
S204、电子设备将地图更新信息发送给服务器,相应的,服务器接收该地图更新信息。S204. The electronic device sends the map update information to the server, and correspondingly, the server receives the map update information.
S205、服务器根据该地图更新信息更新服务器中的全局地图。由于地图更新信息能够表征全局地图的子地图中所欠缺的地图内容,所以地图更新信息能够应用来更新服务器中的全局地图。S205. The server updates the global map in the server according to the map update information. Since the map update information can represent the missing map content in the sub-maps of the global map, the map update information can be used to update the global map in the server.
可以看到,本申请实施例中,电子设备从服务器下载全局地图中全局子地图,通过比较全局子地图和电子设备中的SLAM地图,利用电子设备侧的计算资源和运算能力来获得地图更新信息,该地图更新信息能够被用来更新全局子地图中的地图内容。所以电子设备将地图更新信息发送给服务器后,服务器就能够基于地图更新信息实现对服务器中的全局地图更新。这样,减轻服务器计算负载,提高了服务器更新全局地图的效率。而地图更新信息相比起摄像头拍摄的图像数据而言,数据量更少,所以也节省了传输带宽资源。It can be seen that, in this embodiment of the application, the electronic device downloads the global sub-map in the global map from the server, and obtains map update information by comparing the global sub-map with the SLAM map in the electronic device and using the computing resources and computing capabilities of the electronic device. , The map update information can be used to update the map content in the global sub-map. Therefore, after the electronic device sends the map update information to the server, the server can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the efficiency of the server to update the global map is improved. Compared with the image data captured by the camera, the map update information has a smaller amount of data, so the transmission bandwidth resources are also saved.
参见图10,图10是本申请实施例提供的又一种全局地图更新方法的流程示意图,从电子设备和服务器侧分别进行描述。该方法包括但不限于以下步骤:Refer to FIG. 10, which is a schematic flowchart of another global map update method provided by an embodiment of the present application, which is described separately from the electronic device and the server side. The method includes but is not limited to the following steps:
S301、电子设备向服务器发送用于指示所述电子设备的初始位置的第一位置指纹信息。S301. The electronic device sends first location fingerprint information used to indicate the initial location of the electronic device to the server.
本申请实施例中,电子设备向服务器上传位置指纹信息,所述初始位置可能是所述电子设备请求下载地图时电子设备所处的地理位置信息,例如位置指纹信息来源包括GNSS/WiFi/蓝牙/基站等定位方式所测得的初始位置信息、信号强度信息、信号特征信息等等;也可能是在由用户输入的位置信息。In this embodiment of the application, the electronic device uploads location fingerprint information to the server. The initial location may be the location information of the electronic device when the electronic device requests to download the map. For example, the source of the location fingerprint information includes GNSS/WiFi/Bluetooth/ The initial position information, signal strength information, signal characteristic information, etc. measured by the base station and other positioning methods; it may also be the position information input by the user.
S302、服务器从全局地图中,获取与第一位置指纹信息匹配的全局子地图。全局子地图是全局地图中与所述电子设备的位置关联的子地图。S302. The server obtains a global submap matching the fingerprint information of the first location from the global map. The global sub-map is a sub-map associated with the location of the electronic device in the global map.
S303、服务器下发全局子地图给电子设备。相应的,电子设备接收全局子地图。S303. The server delivers the global sub-map to the electronic device. Correspondingly, the electronic device receives the global sub-map.
一种实施例中,服务器根据上述第一位置指纹信息,与服务器预先保存在数据库中的子地图的位置指纹信息做匹配,所述数据库中的子地图为隶属于全局地图的子地图。如果有匹配上的子地图,则将匹配上的子地图传输到电子设备。In an embodiment, the server matches the location fingerprint information of the submap stored in the database in advance by the server according to the above-mentioned first location fingerprint information, and the submaps in the database are submaps belonging to the global map. If there is a matching sub-map, the matching sub-map is transmitted to the electronic device.
又一种实施例中,服务器根据上述第一位置指纹信息,在服务器保存的全局地图中遍历,直到寻找到位置指纹信息匹配的区域,服务器从全局地图中取出该区域作为全局子地图,将全局子地图传输到电子设备。In another embodiment, the server traverses the global map saved by the server according to the above-mentioned first location fingerprint information until it finds an area matching the location fingerprint information, and the server takes the area from the global map as a global sub-map, and uses the global The sub-map is transmitted to the electronic device.
S304、电子设备将该全局子地图存储到所述电子设备的SLAM系统中。S304. The electronic device stores the global sub-map in the SLAM system of the electronic device.
S305、电子设备计算坐标系变换信息,并基于坐标系变换信息对SLAM地图进行变换。S305. The electronic device calculates coordinate system transformation information, and transforms the SLAM map based on the coordinate system transformation information.
具体的,电子设备计算坐标系变换信息的过程可描述如下:电子设备获取摄像头采集的一帧视频图像;然后,一方面,根据该视频图像和SLAM地图,确定电子设备在原先所构建的局部坐标系下的SLAM地图中的局部位姿(这里可称为第一位姿数据)。Specifically, the process of calculating the coordinate system transformation information of the electronic device can be described as follows: the electronic device obtains a frame of video image collected by the camera; then, on the one hand, according to the video image and the SLAM map, determine the local coordinates originally constructed by the electronic device The local pose in the SLAM map under the system (here can be referred to as the first pose data).
例如,电子设备中设置有IMU,SLAM系统的输入信号包括来自摄像头采集的视频图像、IMU采集的运动数据和局部坐标系下的SLAM地图。IMU高频率地检测电子设备的角速度和线加速度,并对角加速度和线加速度分别积分,进而可计算出电子设备的位姿。摄像头采集的视频图像通过在局部坐标系下的SLAM地图中做匹配,从而也能计算获得电子设备的位姿。那么,基于这两种位姿以一定算法进行运算,就可以获得该第一位姿数据。For example, the electronic device is provided with an IMU, and the input signal of the SLAM system includes the video image collected by the camera, the motion data collected by the IMU, and the SLAM map in the local coordinate system. The IMU detects the angular velocity and linear acceleration of the electronic device at a high frequency, and integrates the angular acceleration and linear acceleration separately, and then can calculate the posture of the electronic device. The video image collected by the camera is matched in the SLAM map in the local coordinate system, so that the position and posture of the electronic device can also be calculated. Then, based on the calculation of these two poses with a certain algorithm, the first pose data can be obtained.
又例如,电子设备中除了摄像头和IMU外,还设置有与位姿或运动相关的定位模块(GPS定位、北斗定位、WIFI定位、或基站定位等),那么,SLAM系统还可共同参考摄像头采集的视频图像、IMU采集的运动数据、局部坐标系下的SLAM地图以及定位模块所采集的数据来计算获得该第一位姿数据。For another example, in addition to the camera and IMU, the electronic device is also provided with a positioning module related to pose or movement (GPS positioning, Beidou positioning, WIFI positioning, or base station positioning, etc.), then the SLAM system can also refer to the camera collection The first pose data is calculated by using the video image of the video image, the motion data collected by the IMU, the SLAM map in the local coordinate system, and the data collected by the positioning module.
另一方面,可根据该视频图像和全局子地图,对该视频图像进行特征提取,并与全局子地图进行特征匹配,进而确定电子设备在全局子地图中的全局位姿(这里可称为第二位姿数据)。On the other hand, feature extraction can be performed on the video image based on the video image and the global sub-map, and feature matching can be performed on the global sub-map, so as to determine the global pose of the electronic device in the global sub-map (herein may be referred to as the first Two pose data).
例如,电子设备对视频图像进行特征检测,在视频图像中提取出特征的图像位置,特征检测算法不限于FAST、ORB、SIFT、SURF、D2Net、SuperPoint等特征检测方法。然后对每一个检测出的特征进行特征描述,特征描述算法不限于ORB、SIFT、SURF、BRIEF、BRISK、FREAK、D2Net、SuperPoint等特征描述方法,从而形成一个一维向量,用于后续的特征匹配。通过特征匹配,电子设备可从全局子地图中匹配出与视频图像最相似的地图内容(例如一帧或多帧关键帧),具体方法例如包括基于BOW、VLAD等传统图像检索方法以及基于NetVLAD、AI的新型图像检索方法。特征匹配具体是计算两个特征描述之间的相似程度,对于Float型向量,可以通过欧氏距离计算;对于二值向量,可以通过异或计算。找出与视频图像最相似的地图内容后,就可以基于视频图像和最相似的地图内容进行位姿估计,例如可采用PnP、EPnP、3D-3D等配准算法,从而可以计算出该第二位姿数据。For example, electronic devices perform feature detection on video images and extract feature image positions in video images. Feature detection algorithms are not limited to feature detection methods such as FAST, ORB, SIFT, SURF, D2Net, and SuperPoint. Then perform feature description for each detected feature. The feature description algorithm is not limited to feature description methods such as ORB, SIFT, SURF, BRIEF, BRISK, FREAK, D2Net, SuperPoint, etc., to form a one-dimensional vector for subsequent feature matching . Through feature matching, the electronic device can match the most similar map content (such as one or more key frames) to the video image from the global submap. Specific methods include traditional image retrieval methods such as BOW and VLAD, and NetVLAD, AI's new image retrieval method. Feature matching specifically calculates the degree of similarity between two feature descriptions. For Float-type vectors, it can be calculated by Euclidean distance; for binary vectors, it can be calculated by XOR. After finding the most similar map content to the video image, the pose estimation can be performed based on the video image and the most similar map content. For example, registration algorithms such as PnP, EPnP, 3D-3D, etc. can be used to calculate the second Pose data.
然后,电子设备可根据所述第一位姿数据和所述第二位姿数据,获得所述SLAM地图的第一坐标系和所述全局地图的第二坐标系之间的坐标系变换信息,坐标系变换信息例如可以是坐标系变换矩阵。Then, the electronic device can obtain coordinate system transformation information between the first coordinate system of the SLAM map and the second coordinate system of the global map according to the first pose data and the second pose data, The coordinate system transformation information may be, for example, a coordinate system transformation matrix.
例如,如图11所示,令视频图像在局部坐标系下的位姿(第一位姿数据)为 LT c,在全局坐标系下位姿(第二位姿数据)为 GT c,则 For example, as shown in Figure 11, let the pose (first pose data) of the video image in the local coordinate system be L T c , and the pose (second pose data) in the global coordinate system as G T c , then
GT cGT L LT c     (1) G T c = G T L L T c (1)
GT LGT c( LT c) -1     (2) G T L = G T c ( L T c ) -1 (2)
这里, GT L可表示局部坐标系和全局坐标系之间的坐标系变换矩阵,根据这个坐标系变换矩阵就可以实现了两个坐标系同步,这样,原先用局部坐标系来表示的信息(例如关键帧、局部位姿、图像的特征点、SLAM地图的3D地图点等等)就可以基于坐标系变换矩阵变换到全局坐标系下。原先用局部坐标系来表示的SLAM地图可以基于坐标系变换矩阵变换到全局坐标系下。 Here, G T L can represent the coordinate system transformation matrix between the local coordinate system and the global coordinate system. According to this coordinate system transformation matrix, the two coordinate systems can be synchronized. In this way, the information originally represented by the local coordinate system ( For example, key frames, local poses, image feature points, SLAM map 3D map points, etc.) can be transformed to the global coordinate system based on the coordinate system transformation matrix. The original SLAM map represented by the local coordinate system can be transformed to the global coordinate system based on the coordinate system transformation matrix.
S306、电子设备判断是否存在非冗余关键帧。S306. The electronic device judges whether there is a non-redundant key frame.
本申请实施例中,SLAM地图中的关键帧和地图点可以变换到全局坐标系中表示,SLAM地图可以也变换到全局坐标系中表示。在SLAM系统运行过程中,会不断更新SLAM地图,所谓“更新SLAM地图”体现在不断向SLAM地图插入关键帧。In the embodiment of the present application, the key frames and map points in the SLAM map can be transformed into the global coordinate system for representation, and the SLAM map can also be transformed into the global coordinate system for representation. During the operation of the SLAM system, the SLAM map will be continuously updated. The so-called "update of the SLAM map" is embodied in the continuous insertion of key frames into the SLAM map.
当有一个新的关键帧插入或者即将要到SLAM地图时,电子设备判断SLAM系统插入的关键帧是否与下载的全局子地图中的关键帧存在冗余。所谓关键帧的冗余指关键帧关联的3D地图点信息在全局子地图中已经存在,即全局子地图已经保存了当前环境中的视觉信息,不需要再增加该关键帧了。而关键帧不冗余即关键帧关联的3D地图点信息在全局子地图中未存在,全局子地图需要增加该关键帧来实现更新,以便于地图内容与真实环境同步。When a new key frame is inserted or is about to arrive on the SLAM map, the electronic device determines whether the key frame inserted by the SLAM system is redundant with the key frame in the downloaded global sub-map. The so-called redundancy of key frames means that the 3D map point information associated with the key frames already exists in the global sub-map, that is, the global sub-map has saved the visual information in the current environment, and there is no need to add the key frame. The key frame is not redundant, that is, the 3D map point information associated with the key frame does not exist in the global sub-map, and the global sub-map needs to add the key frame to realize the update, so that the map content can be synchronized with the real environment.
本文中为了便于理解,不与全局子地图中的关键帧存在冗余的关键帧可简称为非冗余关键帧。在一些实施例中,电子设备识别非冗余关键帧的方法例如可以是,检测当前关键帧与电子设备的当前位置的全局子地图中关键帧的地图点关联情况,如果全局子地图上的地图点投影在当前关键帧上,找不到关联的2D特征点,或关联上的2D特征点很少(例如低于50%,需要说明的是,这里仅用于解释说明),则判断该关键帧为非冗余关键帧。如果全局子地图上的地图点投影在当前关键帧上,可以关联上很多的2D特征点(例如大于等于50%,需要说明的是,这里仅用于解释说明),则判断该关键帧不是非冗余关键帧。For ease of understanding in this article, the key frames that are not redundant with the key frames in the global submap can be referred to as non-redundant key frames for short. In some embodiments, the method for the electronic device to identify the non-redundant key frame may be, for example, detecting the association between the current key frame and the map point of the key frame in the global sub-map of the current location of the electronic device, if the map on the global sub-map The point is projected on the current key frame, and no associated 2D feature points can be found, or there are few 2D feature points on the association (for example, less than 50%, it should be noted that this is only used for explanation), then judge the key The frame is a non-redundant key frame. If the map point on the global submap is projected on the current key frame, many 2D feature points can be associated (for example, greater than or equal to 50%, it should be noted that this is only used for explanation), then it is judged that the key frame is not non- Redundant key frames.
S307、若存在非冗余关键帧,则电子设备将非冗余关键帧融合到全局子地图,获得地图更新信息。S307: If there is a non-redundant key frame, the electronic device merges the non-redundant key frame into the global sub-map to obtain map update information.
具体的,当存在非冗余关键帧,则电子设备则在全局子地图中增加关键帧节点(无向图结构),并更新该非冗余关键帧与全局子地图中的关键帧之间的共视关系,如图12所示,图12为用于说明非冗余关键帧与全局子地图中的关键帧之间的共视关系的示意图。对于全局子地图中的某一3D地图点,如果该3D地图点能同时被若干个关键帧观测到,那么该若干个关键帧就具有对该3D地图点的共视关系。所以,电子设备可计算出全局子地图中的哪些3D地图点能同时被全局子地图中的至少一个关键帧和非冗余关键帧观测到,从而更新全局子地图中的关键帧和非冗余关键之间的共视关系(如图示中虚线所示)。Specifically, when there is a non-redundant key frame, the electronic device adds a key frame node (undirected graph structure) in the global sub-map, and updates the difference between the non-redundant key frame and the key frame in the global sub-map. The common view relationship is shown in FIG. 12, which is a schematic diagram for explaining the common view relationship between non-redundant key frames and key frames in the global sub-map. For a certain 3D map point in the global submap, if the 3D map point can be observed by several key frames at the same time, then the several key frames have a common view relationship with the 3D map point. Therefore, the electronic device can calculate which 3D map points in the global sub-map can be simultaneously observed by at least one key frame and non-redundant key frame in the global sub-map, thereby updating the key frames and non-redundant key frames in the global sub-map The common view relationship between the keys (as shown by the dotted line in the figure).
在更新了全局子地图的关键帧共视关系之后,电子设备可利用三角化算法计算新的3D地图点,以及通过BA算法优化非冗余关键帧对应的位姿和3D地图点坐标,从而完成了将非冗余关键帧融合(或称插入)到全局子地图,实现了该全局子地图的更新。After updating the key frame common view relationship of the global sub-map, the electronic device can use the triangulation algorithm to calculate new 3D map points, and use the BA algorithm to optimize the pose and 3D map point coordinates corresponding to the non-redundant key frames to complete In order to integrate (or insert) non-redundant key frames into the global sub-map, the update of the global sub-map is realized.
S308、电子设备将地图更新信息发送给服务器,相应的,服务器接收该地图更新信息。S308. The electronic device sends the map update information to the server, and correspondingly, the server receives the map update information.
S309、服务器根据该地图更新信息更新服务器中的全局地图。S309. The server updates the global map in the server according to the map update information.
本申请实施例中,电子设备所上传的地图更新信息包括全局子地图中所更新的非冗余关键帧对应的位姿、共视数据以及3D地图点。其中所述共视数据用于指示所述非冗余关键帧与所述第二坐标系下的全局子地图的关键帧之间的所述共视关系。即是说,地图更新信息包括全局子地图被下载到电子设备后所变化的关键帧位姿、共视关系以及3D地图点。In the embodiment of the present application, the map update information uploaded by the electronic device includes the pose, common view data, and 3D map points corresponding to the updated non-redundant key frames in the global submap. The common view data is used to indicate the common view relationship between the non-redundant key frame and the key frame of the global submap in the second coordinate system. In other words, the map update information includes the key frame pose, common view relationship, and 3D map points that change after the global sub-map is downloaded to the electronic device.
本申请一种可能的实现场景中,当电子设备判断该电子设备将要离开全局子地图对应的实际地理范围时,会请求服务器下载基于当前位置信息的另一全局子地图。在电子设备在请求所述另一个子地图的之前,或者在电子设备在请求所述另一个子地图的同时,电子设备将地图更新信息上传到服务器。所述上传的地图更新信息相比原先下载的全局子地图而言,因为只需包含所变化的关键帧位姿、共视关系以及3D地图点,所以数据量会小很多。In a possible implementation scenario of the present application, when the electronic device determines that the electronic device is about to leave the actual geographic range corresponding to the global sub-map, it will request the server to download another global sub-map based on the current location information. Before the electronic device requests the another sub-map, or while the electronic device requests the other sub-map, the electronic device uploads the map update information to the server. Compared with the previously downloaded global sub-map, the uploaded map update information only needs to include the changed key frame pose, common view relationship, and 3D map points, so the amount of data is much smaller.
服务器在接收到地图更新信息之后,可直接基于地图更新信息,在服务器的全局地图上更新关键帧共视关系,更新地图点坐标值,更新关键帧位姿,经过这样的更新后,就相当于将相当于已将非冗余关键帧融合到了全局地图,从而实现了全局地图的高效率更新。可以看到,本实施例中电子设备已提前完成了三角化计算、BA优化计算等等计算量大的计算步骤来实现获得地图更新信息,所以服务器使用地图更新信息来更新全局地图的过程中无需执行三角化计算、BA优化计算等等这些计算步骤,进而大大减少了服务器的计算负担。After the server receives the map update information, it can directly update the key frame common view relationship on the server's global map based on the map update information, update the map point coordinate value, and update the key frame pose. After such an update, it is equivalent to It is equivalent to fusing non-redundant key frames to the global map, thus realizing the high-efficiency update of the global map. It can be seen that the electronic device in this embodiment has completed triangulation calculations, BA optimization calculations, and other computationally intensive calculation steps in advance to obtain map update information, so there is no need for the server to use map update information to update the global map. Perform triangulation calculations, BA optimization calculations and other calculation steps, thereby greatly reducing the computing burden of the server.
本申请又一种可能的实现场景中,从用户点击虚拟场景应用到电子设备上传地图更新信息给服务器的过程可以通过如下方式实现:In another possible implementation scenario of this application, the process from the user clicking the virtual scene application to the electronic device uploading map update information to the server can be implemented in the following ways:
如图13所示,以电子设备10为手机为例,电子设备10中安装有某种虚拟场景应用(例如,可以是带有AR功能的导航地图应用),图13中的(a)示出了电子设备10的显示面板上的桌面101。当电子设备检测到用户点击桌面101上的虚拟场景应用的图标102的操作后,一方面,电子设备进入应用,例如在电子设备10的显示面板上显示如图13中的(b)所示的用户界面106,这里的用户界面106中例如包括电子地图的界面以及多个控件,例如图示中包括电子地图控件,卫星地图控件和AR控件。示例性地,电子地图控件为控制应用进入电子地图界面的控件,卫星地图控件为控制应用进入卫星地图界面的控件,AR控件是控制应用进入AR界面的控件。As shown in FIG. 13, taking the electronic device 10 as a mobile phone as an example, a certain virtual scene application (for example, a navigation map application with AR function) is installed in the electronic device 10, and (a) in FIG. 13 shows The desktop 101 on the display panel of the electronic device 10 is displayed. When the electronic device detects that the user has clicked the icon 102 of the virtual scene application on the desktop 101, on the one hand, the electronic device enters the application, for example, the display panel of the electronic device 10 displays as shown in (b) in FIG. 13 The user interface 106, where the user interface 106 includes, for example, an electronic map interface and multiple controls. For example, the illustration includes an electronic map control, a satellite map control, and an AR control. Exemplarily, the electronic map control is a control that controls the application to enter the electronic map interface, the satellite map control is a control that controls the application to enter the satellite map interface, and the AR control is a control that controls the application to enter the AR interface.
另一方面,电子设备10在后台启动下载全局子地图的流程,全局子地图下载完成后,电子设备10在后台进一步执行非冗余关键帧的识别,若存在非冗余关键帧,则可进一步将非冗余关键帧融合到全局子地图,获得地图更新信息。On the other hand, the electronic device 10 starts the process of downloading the global sub-map in the background. After the download of the global sub-map is completed, the electronic device 10 further performs the identification of non-redundant key frames in the background. If there is a non-redundant key frame, it can further Fusion of non-redundant key frames into the global sub-map to obtain map update information.
为了实现地图更新信息的及时上传,在某些合适的时机,在可以在电子设备10的用户界面106中跳出供用户操作的控件,如图13中的(b)所示的控件109,该控件109向用户指示是否需要上传地图更新信息以便于实现服务器侧的全局地图的更新,控件109中例如可包括“拒绝”“同意”两个选项,当电子设备10检测到用户点击“同意”时,则启动地图更新信息上传流程,从而将地图更新信息上传到服务器进行全局地图的更新。而当电子设备10检测到用户点击“拒绝”时,不启动地图更新信息的上传流程。通过这样的设计,既考虑到上传地图更新信息的及时性,又考虑到用户当前共享地图更新信息的意愿(比如用户不想额外耗费流量时可不上传),从而提升了用户的使用体验。In order to realize the timely upload of map update information, at some suitable time, a control for user operation can be popped up in the user interface 106 of the electronic device 10, such as the control 109 shown in (b) in FIG. 13. 109 indicates to the user whether it is necessary to upload map update information in order to update the global map on the server side. For example, the control 109 may include two options of "reject" and "agree". When the electronic device 10 detects that the user clicks "agree", Then the map update information upload process is started, so that the map update information is uploaded to the server to update the global map. When the electronic device 10 detects that the user clicks "reject", it does not start the uploading process of the map update information. Through this design, both the timeliness of uploading the map update information is taken into consideration, and the user's current willingness to share the map update information (for example, the user may not upload it when the user does not want to consume additional data), thereby improving the user experience.
本申请又一种可能的实现场景中,从用户点击虚拟场景应用到电子设备上传地图更新信息给服务器的过程还可以通过如下方式实现:In another possible implementation scenario of this application, the process from the user clicking on the virtual scene application to the electronic device uploading map update information to the server can also be implemented in the following ways:
如图14所示,以电子设备10为手机为例,电子设备10中安装有某种虚拟场景应用(例如,可以是带有AR功能的导航地图应用),图14中的(a)示出了电子设备10的显示面板上的桌面101。当电子设备检测到用户点击桌面101上的虚拟场景应用的图标102的操作后,在电子设备10的显示面板上显示如图14中的(b)所示的用户界面108,用户界面108中包括用于输入账号和密码的文本框,以提示用户通过验证身份来登陆应用。用户界面108还包括待选择控件110。该待选择控件110用于指示是否允许电子设备自动上传地图更新信息。当电子设备10检测到用户点击触发勾选待选择控件110时,则在后台允许地图更新信息的自动上传。相反,用户没有点击触发勾选待选择控件110时,则在后台不允许地图更新信息的自动上传。通过这样的设计,既考虑到用户当前共享地图更新信息的意愿(比如用户不想额 外耗费流量时可不上传),在征得用户同意时,又能在后台自动实现上传地图更新信息,保证了上传地图更新信息的及时性,又避免上传地图更新信息的环节干扰到用户对应用的使用。As shown in FIG. 14, taking the electronic device 10 as a mobile phone as an example, a certain virtual scene application (for example, a navigation map application with AR function) is installed in the electronic device 10, and (a) in FIG. 14 shows The desktop 101 on the display panel of the electronic device 10 is displayed. When the electronic device detects that the user has clicked the icon 102 of the virtual scene application on the desktop 101, it displays a user interface 108 as shown in (b) of FIG. 14 on the display panel of the electronic device 10, and the user interface 108 includes A text box for entering the account and password to prompt the user to log in to the application by verifying their identity. The user interface 108 also includes a control 110 to be selected. The to-be-selected control 110 is used to indicate whether the electronic device is allowed to automatically upload map update information. When the electronic device 10 detects that the user clicks to trigger the check of the to-be-selected control 110, the automatic upload of map update information is allowed in the background. On the contrary, when the user does not click to trigger the check of the to-be-selected control 110, the automatic upload of map update information is not allowed in the background. Through this design, not only the user’s current willingness to share map update information is taken into account (for example, users do not want to upload additional data when they do not want to consume additional data), and when the user’s consent is obtained, the map update information can be automatically uploaded in the background to ensure that the map is uploaded. The timeliness of the update information, and to avoid the link of uploading map update information from interfering with the use of the application by users.
在电子设备10用户点击触发勾选待选择控件11,且检测到用户输入账号和密码进行登录后(电子设备10验证账号和密码正确)后,一方面,电子设备进入应用,例如在电子设备10的显示面板上显示如图14中的(c)所示的用户界面106,这里的用户界面106中例如包括电子地图的界面以及多个控件,例如图示中包括电子地图控件,卫星地图控件和AR控件。示例性地,电子地图控件为控制应用进入电子地图界面的控件,卫星地图控件为控制应用进入卫星地图界面的控件,AR控件是控制应用进入AR界面的控件。After the user of the electronic device 10 clicks the trigger to check the control to be selected 11, and detects that the user enters the account and password to log in (the electronic device 10 verifies that the account and password are correct), on the one hand, the electronic device enters the application, for example, in the electronic device 10 The user interface 106 shown in (c) in Figure 14 is displayed on the display panel of FIG. 14, where the user interface 106 includes, for example, an electronic map interface and multiple controls. For example, the illustration includes an electronic map control, a satellite map control, and AR controls. Exemplarily, the electronic map control is a control that controls the application to enter the electronic map interface, the satellite map control is a control that controls the application to enter the satellite map interface, and the AR control is a control that controls the application to enter the AR interface.
另一方面,电子设备10在后台启动下载全局子地图的流程,全局子地图下载完成后,电子设备10在后台进一步执行非冗余关键帧的识别,若存在非冗余关键帧,则可进一步将非冗余关键帧融合到全局子地图,获得地图更新信息,并在后台将地图更新信息上传到服务器进行全局地图的更新。On the other hand, the electronic device 10 starts the process of downloading the global sub-map in the background. After the download of the global sub-map is completed, the electronic device 10 further performs the identification of non-redundant key frames in the background. If there is a non-redundant key frame, it can further Fusion of non-redundant key frames into the global sub-map to obtain map update information, and upload the map update information to the server in the background to update the global map.
可以看到,本申请实施例中,电子设备从服务器下载全局地图中全局子地图,通过根据同一帧分别获得终端在局部坐标系中的位姿和全局坐标系中的位姿,基于这两种位姿就能够获得两种坐标系之间的坐标系变换信息(例如坐标系变换矩阵),根据这个坐标系变换矩阵就可以实现了两个坐标系同步,这样,原先用局部坐标系来表示的信息(例如局部位姿、图像的特征点、SLAM地图的3D地图点等等)就可以基于坐标系变换矩阵变换到全局坐标系下。通过比较全局坐标系下的SLAM地图和全局子地图和电子设备中的SLAM地图,识别该SLAM地图的非冗余关键帧,利用电子设备侧的计算资源和运算能力来将非冗余关键帧融合到全局子地图中,从而获得地图更新信息,地图更新信息只需包含所变化的关键帧位姿、共视关系以及3D地图点,相比原先下载的全局子地图而言数据量会小很多,所以进一步节省了传输带宽资源。电子设备将地图更新信息发送给服务器后,服务器就能够基于地图更新信息实现对服务器中的全局地图的更新。这样,减轻服务器计算负载,且提高了全局地图的更新效率和鲁棒性。It can be seen that in this embodiment of the application, the electronic device downloads the global sub-map in the global map from the server, and obtains the terminal's pose in the local coordinate system and the pose in the global coordinate system according to the same frame, based on these two The pose can obtain the coordinate system transformation information between the two coordinate systems (such as the coordinate system transformation matrix). According to this coordinate system transformation matrix, the two coordinate systems can be synchronized. In this way, the original local coordinate system is used to express Information (such as local pose, image feature points, 3D map points of SLAM map, etc.) can be transformed to the global coordinate system based on the coordinate system transformation matrix. By comparing the SLAM map and the global sub-map in the global coordinate system with the SLAM map in the electronic device, identify the non-redundant key frames of the SLAM map, and use the computing resources and computing capabilities of the electronic device to integrate the non-redundant key frames Go to the global sub-map to obtain map update information. The map update information only needs to include the changed key frame pose, common view relationship and 3D map points. Compared with the previously downloaded global sub-map, the amount of data will be much smaller. Therefore, the transmission bandwidth resources are further saved. After the electronic device sends the map update information to the server, the server can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the update efficiency and robustness of the global map are improved.
参见图15,图15是本申请实施例提供的又一种全局地图更新方法的流程示意图,从电子设备和服务器侧分别进行描述。图15与前述图10实施例的主要区别在于,图10实施例中的地图更新信息为全局子地图中所变化的关键帧位姿、共视关系以及3D地图点,而图15实施例中的地图更新信息直接为非冗余关键帧。该方法的相关步骤简述如下:Referring to FIG. 15, FIG. 15 is a schematic flowchart of another global map update method provided by an embodiment of the present application, which is described separately from the electronic device and the server side. The main difference between FIG. 15 and the aforementioned embodiment in FIG. 10 is that the map update information in the embodiment in FIG. 10 is the changed key frame pose, common view relationship, and 3D map points in the global sub-map, while in the embodiment in FIG. The map update information is directly non-redundant key frames. The relevant steps of the method are briefly described as follows:
S501、电子设备向服务器发送用于指示所述电子设备的初始位置的第一位置指纹信息。S501: The electronic device sends first location fingerprint information used to indicate the initial location of the electronic device to the server.
S502、服务器从全局地图中,获取与第一位置指纹信息匹配的全局子地图。所述全局子地图是全局地图中与所述电子设备的位置关联的子地图。S502. The server obtains a global submap matching the fingerprint information of the first location from the global map. The global sub-map is a sub-map associated with the location of the electronic device in the global map.
S503、服务器下发全局子地图给电子设备。相应的,电子设备接收全局子地图。S503: The server delivers the global sub-map to the electronic device. Correspondingly, the electronic device receives the global sub-map.
S504、电子设备将该全局子地图存储到所述电子设备的SLAM系统中。S504. The electronic device stores the global sub-map in the SLAM system of the electronic device.
S505、电子设备计算坐标系变换信息,并基于坐标系变换信息对SLAM地图进行变换。S505. The electronic device calculates coordinate system transformation information, and transforms the SLAM map based on the coordinate system transformation information.
S506、电子设备判断是否存在非冗余关键帧。S506: The electronic device judges whether there is a non-redundant key frame.
上述步骤S501-S506的具体内容可与前述图10实施例中步骤S301-S306相同,为了说明书的简洁,这里不在赘述。The specific content of the foregoing steps S501-S506 may be the same as the foregoing steps S301-S306 in the embodiment of FIG. 10. For the sake of brevity of the description, the details are not repeated here.
S507、若存在非冗余关键帧,则电子设备将非冗余关键帧发送给服务器,相应的,服务器接收该非冗余关键帧。S507: If there is a non-redundant key frame, the electronic device sends the non-redundant key frame to the server, and correspondingly, the server receives the non-redundant key frame.
S508、服务器根据所述地图更新信息更新服务器中的全局地图。S508. The server updates the global map in the server according to the map update information.
本实施例中,当有一个新的关键帧插入或者即将要到SLAM地图时,如果电子设备判断该新的关键帧为非冗余关键帧,为了节省电子设备的运算能力,也可以直接将该非冗余关键帧发送给服务器,由服务器执行将非冗余关键帧融合(或称插入)到服务器中的全局地图的过程。In this embodiment, when a new key frame is inserted or is about to arrive on the SLAM map, if the electronic device judges that the new key frame is a non-redundant key frame, in order to save the computing power of the electronic device, it can also be used directly. The non-redundant key frames are sent to the server, and the server executes the process of fusing (or inserting) the non-redundant key frames into the global map in the server.
类似的,服务器可以在全局地图中增加关键帧节点(无向图结构),并更新该非冗余关键帧与全局地图中的关键帧之间的共视关系,即服务器可计算出全局地图中的哪些3D地图点能同时被全局地图中的至少一个关键帧和该非冗余关键帧观测到,从而更新全局地图中的关键帧和非冗余关键之间的共视关系。在更新全局地图的关键帧共视关系之后,服务器可利用三角化算法计算新的3D地图点,以及通过BA算法优化非冗余关键帧对应的位姿和3D地图点坐标,从而完成了将非冗余关键帧融合(或称插入)到全局地图的过程,实现了该全局地图的更新。Similarly, the server can add key frame nodes (undirected graph structure) to the global map, and update the common view relationship between the non-redundant key frame and the key frames in the global map, that is, the server can calculate the global map Which 3D map points can be simultaneously observed by at least one key frame in the global map and the non-redundant key frame, thereby updating the common view relationship between the key frame and the non-redundant key in the global map. After updating the key frame common view relationship of the global map, the server can use the triangulation algorithm to calculate new 3D map points, and optimize the pose and 3D map point coordinates corresponding to the non-redundant key frames through the BA algorithm, thereby completing the non-redundant The process of fusion (or insertion) of redundant key frames into the global map realizes the update of the global map.
可以看到,本申请实施例中,电子设备从服务器下载全局地图中全局子地图,通过根据同一帧分别获得终端在局部坐标系中的位姿和全局坐标系中的位姿,基于这两种位姿就能够获得两种坐标系之间的坐标系变换信息(例如坐标系变换矩阵),根据这个坐标系变换矩阵就可以实现了两个坐标系同步,这样,原先用局部坐标系来表示的信息(例如局部位姿、图像的特征点、SLAM地图的3D地图点等等)就可以基于坐标系变换矩阵变换到全局坐标系下。通过比较全局坐标系下的SLAM地图和全局子地图,识别该SLAM地图的非冗余关键帧,并将非冗余关键帧发送给服务器,服务器再利用非冗余关键帧融合到全局地图,从而也能够实现对服务器中的全局地图的高效率的更新。这样的方案有利于减轻电子设备的计算负载,保证地图融合的准确性。It can be seen that in this embodiment of the application, the electronic device downloads the global sub-map in the global map from the server, and obtains the terminal's pose in the local coordinate system and the pose in the global coordinate system according to the same frame, based on these two The pose can obtain the coordinate system transformation information between the two coordinate systems (such as the coordinate system transformation matrix). According to this coordinate system transformation matrix, the two coordinate systems can be synchronized. In this way, the original local coordinate system is used to express Information (such as local pose, image feature points, 3D map points of SLAM map, etc.) can be transformed to the global coordinate system based on the coordinate system transformation matrix. By comparing the SLAM map and the global sub-map in the global coordinate system, the non-redundant key frames of the SLAM map are identified, and the non-redundant key frames are sent to the server, and the server uses the non-redundant key frames to merge into the global map, thereby It can also realize the efficient update of the global map in the server. Such a solution is beneficial to reduce the computational load of electronic equipment and ensure the accuracy of map fusion.
参见图16,图16示出了在又一种具体实现中,图4所示的电子设备10中的各功能模块进一步可能包含的组件,以及服务器20中的各功能模块进一步可能包含的组件。图16实施例和前述图5实施例的主要区别在于,图16实施例所示功能模块架构中,全局定位模块16的功能配置在服务器20侧实现,即服务器20中还包括图像检索模块161、特征提取模块162、特征匹配模块163和位姿估计模块164。Referring to FIG. 16, FIG. 16 shows the components that may be further included in the functional modules in the electronic device 10 shown in FIG. 4 and the components that may be further included in the functional modules in the server 20 in another specific implementation. The main difference between the embodiment in FIG. 16 and the embodiment in FIG. 5 is that in the functional module architecture shown in the embodiment in FIG. 16, the functional configuration of the global positioning module 16 is implemented on the server 20 side, that is, the server 20 also includes an image retrieval module 161, The feature extraction module 162, the feature matching module 163, and the pose estimation module 164.
服务器20中的全局位姿模块16用于在初始时刻乃至之后的任一时刻,获取电子设备上传的至少一帧视频图像,基于该视频图像计算出电子设备10在全局子地图中的全局位姿(例如为第二位姿数据),并将该全局位姿发送给电子设备10。具体的,可通过图像检索模块161获取电子设备10上传的视频图像序列中的视频图像,通过特征提取模块162根据所述视频图像进行特征提取,获得图像特征,通过特征匹配模块163将所述图像特征在全局子地图中进行特征匹配,获得与所述图像特征匹配的地图特征;通过位姿估计模块164根据所述图像特征和所述地图特征,计算获得所述电子设备10在全局子地图中的全局位姿(例如为第二位姿数据),并将该全局位姿发送给电子设备10。The global pose module 16 in the server 20 is used to obtain at least one frame of video image uploaded by the electronic device at the initial moment or any time thereafter, and calculate the global pose of the electronic device 10 in the global submap based on the video image (For example, the second pose data), and send the global pose to the electronic device 10. Specifically, the video image in the video image sequence uploaded by the electronic device 10 can be obtained through the image retrieval module 161, and the feature extraction module 162 can perform feature extraction according to the video image to obtain the image feature, and the feature matching module 163 can use the feature matching module 163 to perform feature extraction on the video image. The feature is matched in the global sub-map to obtain the map feature that matches the image feature; the pose estimation module 164 calculates that the electronic device 10 is in the global sub-map based on the image feature and the map feature. The global pose of (for example, the second pose data), and the global pose is sent to the electronic device 10.
图16所示的服务器20中的其他功能模块进一步可能包含的组件可类似参考前述图5实 施例中对服务器20的相关描述,为了说明书的简洁,这里不再赘述。The components that may be further included in other functional modules of the server 20 shown in FIG. 16 can be similarly referred to the related description of the server 20 in the embodiment of FIG. 5. For the sake of brevity of the description, the details are not repeated here.
图16所示的电子设备10中的各功能模块进一步可能包含的组件可类似参考前述图5实施例中对电子设备10的相关描述,为了说明书的简洁,这里也不再赘述。The further possible components of each functional module in the electronic device 10 shown in FIG. 16 may be similar to the related description of the electronic device 10 in the embodiment of FIG. 5. For the sake of brevity of the description, the details are not repeated here.
具体实施例中,电子设备10中的各功能模块可相互配合,以执行如图17实施例中电子设备侧的功能。服务器20中的各功能模块可相互配合,以执行如图17实施例中服务器侧的功能。In a specific embodiment, the functional modules in the electronic device 10 can cooperate with each other to perform the functions on the electronic device side in the embodiment of FIG. 17. The functional modules in the server 20 can cooperate with each other to perform the server-side functions in the embodiment of FIG. 17.
参见图17,图17是本申请实施例提供的又一种虚拟物体显示方法的流程示意图,从电子设备和服务器侧分别进行描述。该方法包括但不限于以下步骤:Referring to FIG. 17, FIG. 17 is a schematic flowchart of yet another virtual object display method provided by an embodiment of the present application, which is described separately from the electronic device and the server side. The method includes but is not limited to the following steps:
S601、电子设备向服务器发送用于指示所述电子设备的初始位置的第一位置指纹信息和至少一帧视频图像。S601. The electronic device sends first location fingerprint information and at least one frame of video image used to indicate the initial location of the electronic device to the server.
具体实施例中,为实现电子设备的第一次全局定位,电子设备需要向服务器上传位置指纹信息和至少一张或多张当前采集的视频图像。其中,所述至少一帧视频图像可以是电子设备通过摄像头拍摄的视频图像序列中的视频图像。比如,可以是摄像头拍摄的视频图像序列中的第1帧图像。所述位置指纹信息所指示的初始位置可能是所述电子设备请求下载地图时电子设备所处的地理位置信息,例如位置指纹信息来源包括GNSS/WiFi/蓝牙/基站等定位方式所测得的初始位置信息、信号强度信息、信号特征信息等等;也可能是在由用户输入的位置信息。In a specific embodiment, in order to realize the first global positioning of the electronic device, the electronic device needs to upload location fingerprint information and at least one or more currently collected video images to the server. Wherein, the at least one frame of video image may be a video image in a video image sequence shot by an electronic device through a camera. For example, it may be the first frame image in a video image sequence taken by a camera. The initial location indicated by the location fingerprint information may be the geographic location information of the electronic device when the electronic device requests to download the map. For example, the source of the location fingerprint information includes the initial location measured by GNSS/WiFi/Bluetooth/base station. Location information, signal strength information, signal feature information, etc.; it may also be the location information entered by the user.
在一种具体实现中,电子设备可以将第一位置指纹信息和一帧视频图像一起打包发送给服务器。In a specific implementation, the electronic device may package and send the first location fingerprint information and a frame of video image to the server.
在又一种具体实现中,电子设备也可以将第一位置指纹信息和一帧视频图像分别独立发送给服务器。In another specific implementation, the electronic device may also independently send the first location fingerprint information and a frame of video image to the server.
S602、服务器从全局地图中,获取与第一位置指纹信息匹配的全局子地图。所述全局子地图是全局地图中与所述电子设备的位置关联的子地图。S602. The server obtains a global submap matching the fingerprint information of the first location from the global map. The global sub-map is a sub-map associated with the location of the electronic device in the global map.
一种实施例中,服务器根据上述第一位置指纹信息,与服务器预先保存在数据库中的子地图的位置指纹信息做匹配,所述数据库中的子地图为隶属于全局地图的子地图。如果有匹配上的子地图,则该子地图即为后续需要下发给电子设备的全局子地图。In an embodiment, the server matches the location fingerprint information of the submap stored in the database in advance by the server according to the above-mentioned first location fingerprint information, and the submaps in the database are submaps belonging to the global map. If there is a matching sub-map, the sub-map is a global sub-map that needs to be subsequently issued to the electronic device.
又一种实施例中,服务器根据上述第一位置指纹信息,在服务器保存的全局地图中遍历,直到寻找到位置指纹信息匹配的区域,服务器从全局地图中取出该区域作为全局子地图。In another embodiment, the server traverses the global map saved by the server according to the above-mentioned first location fingerprint information until it finds an area matching the location fingerprint information, and the server takes the area from the global map as a global submap.
S603、服务器根据该视频图像和全局子地图进行位姿计算,获得电子设备在全局子地图中的全局位姿(这里也可称为第二位姿数据)。S603: The server performs pose calculation according to the video image and the global sub-map, and obtains the global pose of the electronic device in the global sub-map (herein may also be referred to as second pose data).
本申请的一种实施例中,第一次计算电子设备的全局位姿可以放在服务器端完成,服务器执行全局位姿计算的过程同样包括图像检索、特征提取、特征匹配、位姿估计等过程,In an embodiment of this application, the first calculation of the global pose of the electronic device can be done on the server side. The process of performing the global pose calculation by the server also includes image retrieval, feature extraction, feature matching, pose estimation, etc. ,
例如,服务器对该视频图像进行特征检测,在该视频图像中提取出特征的图像位置,特征检测算法不限于FAST、ORB、SIFT、SURF、D2Net、SuperPoint等特征检测方法。然后对每一个检测出的特征进行特征描述,特征描述算法不限于ORB、SIFT、SURF、BRIEF、BRISK、FREAK、D2Net、SuperPoint等特征描述方法,从而形成一个一维向量,用于后续的特征匹配。通过特征匹配,服务器可从全局子地图中匹配出与该视频图像最相似的地图内容(例如一帧 或多帧关键帧),具体方法例如包括基于BOW、VLAD等传统图像检索方法以及基于NetVLAD、AI的新型图像检索方法。特征匹配具体是计算两个特征描述之间的相似程度,对于Float型向量,可以通过欧氏距离计算;对于二值向量,可以通过异或计算。找出与视频图像最相似的地图内容后,就可以基于该视频图像和最相似的地图内容进行位姿估计,例如可采用PnP、EPnP、3D-3D等配准算法,从而可以计算出该第二位姿数据。For example, the server performs feature detection on the video image, and extracts the image location of the feature from the video image. The feature detection algorithm is not limited to feature detection methods such as FAST, ORB, SIFT, SURF, D2Net, SuperPoint, etc. Then perform feature description for each detected feature. The feature description algorithm is not limited to feature description methods such as ORB, SIFT, SURF, BRIEF, BRISK, FREAK, D2Net, SuperPoint, etc., to form a one-dimensional vector for subsequent feature matching . Through feature matching, the server can match the most similar map content (such as one or more key frames) to the video image from the global submap. Specific methods include, for example, traditional image retrieval methods such as BOW and VLAD, and NetVLAD, AI's new image retrieval method. Feature matching specifically calculates the degree of similarity between two feature descriptions. For Float-type vectors, it can be calculated by Euclidean distance; for binary vectors, it can be calculated by XOR. After finding the map content that is most similar to the video image, the pose estimation can be performed based on the video image and the most similar map content. For example, registration algorithms such as PnP, EPnP, 3D-3D, etc. can be used to calculate the first image. Two pose data.
S604、服务器在完成位姿估计之后,下发所述电子设备在全局子地图中的全局位姿(第二位姿数据)给电子设备。相应的,电子设备接收该第二位姿数据。电子设备后续可利用该第二位姿数据来计算坐标系变换信息(例如坐标系变换矩阵)。S604: After completing the pose estimation, the server issues the global pose (second pose data) of the electronic device in the global submap to the electronic device. Correspondingly, the electronic device receives the second pose data. The electronic device can subsequently use the second pose data to calculate coordinate system transformation information (for example, a coordinate system transformation matrix).
S605、服务器下发全局子地图给电子设备。相应的,电子设备接收该全局子地图。S605: The server delivers the global sub-map to the electronic device. Correspondingly, the electronic device receives the global sub-map.
S606、电子设备将该全局子地图存储到所述电子设备的SLAM系统中。S606. The electronic device stores the global sub-map in the SLAM system of the electronic device.
S607、电子设备计算坐标系变换信息。S607. The electronic device calculates coordinate system transformation information.
电子设备计算坐标系变换信息的过程可描述如下:电子设备获取视频图像,所述视频图像和S601中所发送给服务器的视频图像为相同图像;然后,一方面,根据该视频图像和SLAM地图,确定电子设备在原先所构建的局部坐标系下的SLAM地图中的局部位姿(这里可称为第一位姿数据)。具体实现过程可类似参考图10实施例的S305中有关于第一位姿数据的描述,为了说明书的简洁,这里不再赘述。The process of the electronic device calculating coordinate system transformation information can be described as follows: the electronic device obtains a video image, and the video image is the same image as the video image sent to the server in S601; then, on the one hand, according to the video image and the SLAM map, Determine the local pose of the electronic device in the SLAM map under the originally constructed local coordinate system (here it may be referred to as the first pose data). The specific implementation process may be similar to the description of the first pose data in S305 of the embodiment in FIG. 10. For the sake of brevity of the description, details are not repeated here.
然后,电子设备可根据所述第一位姿数据和通过S604所获得的第二位姿数据,进一步获得所述SLAM地图的第一坐标系和所述全局地图的第二坐标系之间的坐标系变换信息,坐标系变换信息例如可以是坐标系变换矩阵。关于坐标系变换矩阵的相关内容可类似参考图11实施例的相关描述,为了说明书的简洁,这里也不再赘述。Then, the electronic device can further obtain the coordinates between the first coordinate system of the SLAM map and the second coordinate system of the global map according to the first pose data and the second pose data obtained through S604 The system transformation information, and the coordinate system transformation information may be, for example, a coordinate system transformation matrix. The related content of the coordinate system transformation matrix can be similarly referred to the related description of the embodiment in FIG. 11. For the sake of brevity of the description, the details are not repeated here.
S608、电子设备判断是否存在非冗余关键帧。S608. The electronic device judges whether there is a non-redundant key frame.
S609、若存在非冗余关键帧,则电子设备将非冗余关键帧融合到全局子地图,获得地图更新信息。S609. If there is a non-redundant key frame, the electronic device fuses the non-redundant key frame into the global sub-map to obtain map update information.
S610、电子设备将地图更新信息发送给服务器,相应的,服务器接收该地图更新信息。S610. The electronic device sends the map update information to the server, and correspondingly, the server receives the map update information.
S611、服务器根据所述地图更新信息更新服务器中的全局地图。S611. The server updates the global map in the server according to the map update information.
上述步骤S608-S611的具体实现过程可类似参考图10实施例S306-S309的描述,为了说明书的简洁,这里也不再赘述。The specific implementation process of the foregoing steps S608-S611 may be similar to the description of the embodiments S306-S309 with reference to FIG. 10. For the sake of brevity of the description, the details are not repeated here.
可以看到,本申请实施例中,电子设备需要先下载对应区域的全局子地图,下载地图的过程需要花费一定时间。为了加快用户进入应用的速度,可以采用第一次全局位姿估计在服务器端完成的方式。也即是说,应用启动后,第一次全局位姿估计在服务器侧进行,在启动全局位姿估计的同时,服务器相应获取全局子地图并向电子设备传输,提高用户进入应用的速度。用户并不会感知到地图下载流程的时延存在,这样,能够避免了下载时延造成的用户等待,进一步提升用户体验。电子设备通过比较全局子地图和电子设备中的SLAM地图,利用电子设备侧的计算资源和运算能力来获得地图更新信息,该地图更新信息能够被用来更新全局子地图中的地图内容。所以电子设备将地图更新信息发送给服务器后,服务器就能够基于地图更新信息实现对服务器中的全局地图更新。这样,减轻服务器计算负载,提高了服务器更新全局地图的效率。而地图更新信息相比起摄像头拍摄的图像数据而言,数据量更少,所以也节省了传输带宽资源。It can be seen that in this embodiment of the application, the electronic device needs to download the global sub-map of the corresponding area first, and the process of downloading the map takes a certain amount of time. In order to speed up the user's entry into the application, the first global pose estimation can be done on the server side. In other words, after the application is started, the first global pose estimation is performed on the server side. While starting the global pose estimation, the server obtains the global sub-map and transmits it to the electronic device accordingly, which improves the user's access speed to the application. The user does not perceive the delay of the map downloading process. In this way, the user waiting caused by the download delay can be avoided, and the user experience is further improved. The electronic device obtains map update information by comparing the global sub-map with the SLAM map in the electronic device, and uses the computing resources and computing capabilities of the electronic device to obtain map update information, which can be used to update the map content in the global sub-map. Therefore, after the electronic device sends the map update information to the server, the server can update the global map in the server based on the map update information. In this way, the calculation load of the server is reduced, and the efficiency of the server to update the global map is improved. Compared with the image data captured by the camera, the map update information has a smaller amount of data, so the transmission bandwidth resources are also saved.
在上述实施例中,可以全部或部分地通过软件、硬件、固件或者任意组合来实现。当使用软件实现时,可以全部或者部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令,在计算机上加载和执行所述计算机程序指令时,全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络或其他可编程装置。所述计算机指令可存储在计算机可读存储介质中,或者从一个计算机可读存储介质向另一个计算机可读存储介质传输,例如,所述计算机指令可以从一个网络站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线)或无线(例如红外、微波等)方式向另一个网络站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质,也可以是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质(例如软盘、硬盘、磁带等)、光介质(例如DVD等)、或者半导体介质(例如固态硬盘)等等。In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination. When implemented by software, it can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions, and when the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a network site, computer, server, or data center. Transmission to another network site, computer, server or data center via wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer, and may also be a data storage device such as a server or a data center integrated with one or more available media. The usable medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape, etc.), an optical medium (such as a DVD, etc.), or a semiconductor medium (such as a solid-state hard disk), and so on.
在上述实施例中,对各个实施例的描述各有侧重,某个实施例中没有详述的部分,可以参见其他实施例的相关描述。In the above-mentioned embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in an embodiment, reference may be made to related descriptions of other embodiments.

Claims (46)

  1. 一种虚拟物体显示方法,应用于具有显示组件和摄像头的第一电子设备,其特征在于,所述方法包括:A virtual object display method, applied to a first electronic device having a display component and a camera, characterized in that the method includes:
    检测到用户打开应用的操作;The user's operation to open the application is detected;
    响应于所述操作,从服务器下载第一全局子地图并存储到所述第一电子设备的即时定位与地图构建(Simultaneous Localization and Mapping,SLAM)系统中;所述第一全局子地图是全局地图中与所述第一电子设备的位置关联的子地图;In response to the operation, the first global sub-map is downloaded from the server and stored in the Simultaneous Localization and Mapping (SLAM) system of the first electronic device; the first global sub-map is a global map A sub-map associated with the location of the first electronic device in;
    在所述显示组件上基于所述第一电子设备的位姿数据来显示虚拟物体,所述第一电子设备的位姿数据是所述SLAM系统至少根据所述摄像头采集的视频图像和所述第一全局子地图执行位姿计算得到的,其中所述服务器上的全局地图是根据一个或多个第二电子设备发送的地图更新信息进行更新得到的,其中,所述一个或多个第二电子设备为除所述第一电子设备外的电子设备。A virtual object is displayed on the display component based on the pose data of the first electronic device, and the pose data of the first electronic device is the SLAM system at least according to the video image collected by the camera and the second A global sub-map is obtained by performing pose calculation, wherein the global map on the server is updated according to the map update information sent by one or more second electronic devices, wherein the one or more second electronic devices The device is an electronic device other than the first electronic device.
  2. 根据权利要求1所述的方法,其特征在于,所述地图更新信息是所述第二电子设备的SLAM系统根据第二坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果获得的;其中,所述第二坐标系下的SLAM地图是基于第一坐标系下的SLAM地图以及所述第一坐标系和所述第二坐标系之间的坐标系变换信息得到的;或者,The method according to claim 1, wherein the map update information is obtained by the SLAM system of the second electronic device according to the SLAM map in the second coordinate system and the global sub-map in the second coordinate system. Obtained from the comparison result; wherein the SLAM map in the second coordinate system is obtained based on the SLAM map in the first coordinate system and the coordinate system transformation information between the first coordinate system and the second coordinate system ;or,
    所述地图更新信息是所述第二电子设备的SLAM系统根据所述第一坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果以及所述坐标系变换信息获得的。The map update information is obtained by the SLAM system of the second electronic device according to the comparison result of the SLAM map in the first coordinate system and the global sub-map in the second coordinate system and the coordinate system transformation information .
  3. 根据权利要求1或2所述的方法,其特征在于,所述地图更新信息包括:The method according to claim 1 or 2, wherein the map update information comprises:
    所述第二电子设备的SLAM系统所构建的SLAM地图的目标关键帧对应的位姿数据、三维地图点以及共视数据,所述共视数据用于指示所述目标关键帧与所述第二电子设备中的第二全局子地图的关键帧之间的共视关系,所述第二全局子地图是全局地图中与所述第二电子设备的位置关联的子地图。The pose data, three-dimensional map points, and common view data corresponding to the target key frame of the SLAM map constructed by the SLAM system of the second electronic device, the common view data being used to indicate the target key frame and the second A common view relationship between key frames of a second global sub-map in the electronic device, where the second global sub-map is a sub-map associated with the position of the second electronic device in the global map.
  4. 根据权利要求3所述的方法,其特征在于,所述目标关键帧为非冗余关键帧,所述非冗余关键帧表示存在于所述第二电子设备的SLAM地图中但不存在于所述第二全局子地图中的关键帧。The method according to claim 3, wherein the target key frame is a non-redundant key frame, and the non-redundant key frame indicates that it exists in the SLAM map of the second electronic device but does not exist in all Describe the key frames in the second global submap.
  5. 根据权利要求1或2所述的方法,其特征在于,所述地图更新信息包括:The method according to claim 1 or 2, wherein the map update information comprises:
    存在于所述第二电子设备的SLAM地图中但不存在于所述第二电子设备中的第二全局子地图中的关键帧,所述第二全局子地图是全局地图中与所述第二电子设备的位置关联的子地图。A key frame that exists in the SLAM map of the second electronic device but does not exist in the second global sub-map of the second electronic device, and the second global sub-map is the same as the second global sub-map in the global map. The sub-map associated with the location of the electronic device.
  6. 根据权利要求1-5任一项所述的方法,其特征在于,所述响应于所述操作,从服务器 下载第一全局子地图,包括:The method according to any one of claims 1-5, wherein the downloading the first global submap from the server in response to the operation comprises:
    向服务器发送用于指示所述第一电子设备的初始位置的第一位置指纹信息;Sending the first location fingerprint information used to indicate the initial location of the first electronic device to the server;
    从所述服务器接收所述第一全局子地图,所述第一全局子地图对应有第二位置指纹信息,且所述第一位置指纹信息和所述第二位置指纹信息相匹配。The first global submap is received from the server, the first global submap corresponds to second location fingerprint information, and the first location fingerprint information matches the second location fingerprint information.
  7. 一种全局地图更新方法,应用于电子设备,其特征在于,所述方法包括:A global map update method applied to electronic equipment, characterized in that the method includes:
    从服务器获取与所述电子设备的位置关联的全局子地图,所述全局子地图是第二坐标系下的全局地图中与所述电子设备的位置关联的子地图;Acquiring a global sub-map associated with the location of the electronic device from a server, where the global sub-map is a sub-map associated with the location of the electronic device in the global map in the second coordinate system;
    根据所述电子设备的SLAM系统所构建的第二坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果,或者根据所述电子设备的SLAM系统所构建的第一坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果以及所述坐标系变换信息,获得地图更新信息;其中所述第二坐标系下的SLAM地图是基于所述第一坐标系下的SLAM地图以及所述第一坐标系和所述第二坐标系之间的坐标系变换信息得到的;所述地图更新信息用于指示所述第二坐标系下或者所述第一坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异;According to the comparison result of the SLAM map in the second coordinate system constructed by the SLAM system of the electronic device and the global sub-map in the second coordinate system, or according to the first coordinates constructed by the SLAM system of the electronic device The comparison result of the SLAM map in the second coordinate system and the global sub-map in the second coordinate system and the coordinate system transformation information obtain map update information; wherein the SLAM map in the second coordinate system is based on the first The SLAM map in the coordinate system and the coordinate system transformation information between the first coordinate system and the second coordinate system; the map update information is used to indicate the second coordinate system or the first coordinate system The difference between the SLAM map in the coordinate system and the global sub-map in the second coordinate system;
    将所述地图更新信息发送给所述服务器,所述地图更新信息用于更新所述服务器中所述第二坐标系下的全局地图。The map update information is sent to the server, and the map update information is used to update the global map in the second coordinate system in the server.
  8. 根据权利要求7所述的方法,其特征在于,所述第二坐标系下的SLAM地图具体为基于所述坐标系变换信息对所述第一坐标系下的SLAM地图进行变换得到的;The method according to claim 7, wherein the SLAM map in the second coordinate system is specifically obtained by transforming the SLAM map in the first coordinate system based on the coordinate system transformation information;
    所述根据所述电子设备的SLAM系统所构建的第二坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果,获得地图更新信息,包括:The obtaining map update information according to the comparison result of the SLAM map in the second coordinate system constructed by the SLAM system of the electronic device and the global sub-map in the second coordinate system includes:
    通过比较所述第二坐标系下的SLAM地图与所述第二坐标系下的全局子地图,以获得非冗余关键帧,所述非冗余关键帧表示存在于所述第二坐标系下的SLAM地图中但不存在于所述第二坐标系下的全局子地图中的关键帧;By comparing the SLAM map in the second coordinate system with the global sub-map in the second coordinate system to obtain a non-redundant key frame, the non-redundant key frame indicates that it exists in the second coordinate system Key frames in the SLAM map of but not in the global sub-map in the second coordinate system;
    将所述非冗余关键帧融合到所述第二坐标系下的全局子地图,得到经融合后的全局子地图;Fusing the non-redundant key frames to the global sub-map in the second coordinate system to obtain the fused global sub-map;
    从所述经融合后的全局子地图中获得所述地图更新信息。Obtain the map update information from the fused global sub-map.
  9. 根据权利要求8所述的方法,其特征在于,所述将所述非冗余关键帧融合到所述第二坐标系下的全局子地图,包括:The method according to claim 8, wherein the fusing the non-redundant key frame to the global sub-map in the second coordinate system comprises:
    将所述非冗余关键帧作为新的关键帧节点增加到在所述第二坐标系下的全局子地图;Adding the non-redundant key frame as a new key frame node to the global sub-map in the second coordinate system;
    更新所述非冗余关键帧与所述第二坐标系下的全局子地图中的关键帧的共视关系;Updating the common view relationship between the non-redundant key frames and the key frames in the global sub-map in the second coordinate system;
    计算所述非冗余关键帧对应的位姿数据和三维地图点;Calculating the pose data and three-dimensional map points corresponding to the non-redundant key frames;
    相应的,所述从所述经融合后的全局子地图中获得所述地图更新信息,包括:Correspondingly, the obtaining the map update information from the fused global sub-map includes:
    从所述经融合后的全局子地图中获取所述非冗余关键帧对应的位姿数据、三维地图点以及共视数据作为所述地图更新信息,所述共视数据用于指示所述非冗余关键帧与所述第二坐标系下的全局子地图的关键帧之间的共视关系。Obtain the pose data, three-dimensional map points, and common view data corresponding to the non-redundant key frame from the fused global sub-map as the map update information, and the common view data is used to indicate the non-redundant key frame. The common view relationship between the redundant key frames and the key frames of the global sub-map in the second coordinate system.
  10. 根据权利要求7所述的方法,其特征在于,所述根据第一坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果以及所述坐标系变换信息,获得地图更新信息,包括:8. The method according to claim 7, wherein the map update is obtained according to the comparison result of the SLAM map in the first coordinate system and the global sub-map in the second coordinate system and the coordinate system transformation information Information, including:
    通过比较所述第一坐标系下的SLAM地图与所述第二坐标系下的全局子地图,以得到非冗余关键帧,所述非冗余关键帧为存在于所述第一坐标系下的SLAM地图的关键帧,且所述非冗余关键帧经所述坐标系变换信息变换后不存在于所述第二坐标系下的全局子地图;By comparing the SLAM map in the first coordinate system with the global sub-map in the second coordinate system to obtain a non-redundant key frame, the non-redundant key frame is existing in the first coordinate system The key frame of the SLAM map of, and the non-redundant key frame is transformed by the coordinate system transformation information and does not exist in the global sub-map in the second coordinate system;
    将经所述坐标系变换信息变换后的非冗余关键帧融合到所述第二坐标系下的全局子地图,以获得所述地图更新信息;所述经所述坐标系变换信息变换后的非冗余关键帧是所述第二坐标系下的关键帧。The non-redundant key frames transformed by the coordinate system transformation information are fused to the global sub-map in the second coordinate system to obtain the map update information; the coordinate system transformation information transformed Non-redundant key frames are key frames in the second coordinate system.
  11. 根据权利要求7-10任一项所述的方法,其特征在于,所述从服务器获取与所述电子设备的位置对应的全局子地图,包括:The method according to any one of claims 7-10, wherein the obtaining a global sub-map corresponding to the location of the electronic device from a server comprises:
    向服务器发送所述电子设备的初始位置的指示信息;Sending the indication information of the initial position of the electronic device to the server;
    从所述服务器接收所述全局子地图,所述全局子地图是根据所述电子设备的初始位置确定的。Receiving the global sub-map from the server, the global sub-map being determined according to the initial position of the electronic device.
  12. 根据权利要求11所述的方法,其特征在于,所述电子设备的初始位置的指示信息包括用于指示所述电子设备的初始位置的第一位置指纹信息;所述全局子地图对应有第二位置指纹信息,且所述第一位置指纹信息和所述第二位置指纹信息相匹配。The method according to claim 11, wherein the indication information of the initial position of the electronic device includes first position fingerprint information for indicating the initial position of the electronic device; the global sub-map corresponds to a second Location fingerprint information, and the first location fingerprint information matches the second location fingerprint information.
  13. 根据权利要求7-12任一项所述的方法,其特征在于,在获得地图更新信息之前,还包括:The method according to any one of claims 7-12, wherein before obtaining map update information, the method further comprises:
    根据采集的视频图像和第一坐标系下的SLAM地图,确定所述电子设备在所述第一坐标系下的SLAM地图中的第一位姿数据;Determining the first pose data of the electronic device in the SLAM map in the first coordinate system according to the collected video image and the SLAM map in the first coordinate system;
    根据所述视频图像和第二坐标系下的全局子地图,确定所述电子设备在所述第二坐标系下的全局子地图中的第二位姿数据;Determine the second pose data of the electronic device in the global sub-map in the second coordinate system according to the video image and the global sub-map in the second coordinate system;
    根据所述第一位姿数据和所述第二位姿数据,获得所述SLAM地图的第一坐标系和所述全局地图的第二坐标系之间的所述坐标系变换信息。According to the first pose data and the second pose data, the coordinate system transformation information between the first coordinate system of the SLAM map and the second coordinate system of the global map is obtained.
  14. 根据权利要求13所述的方法,其特征在于,所述根据采集的视频图像和第一坐标系下的SLAM地图,确定所述电子设备在所述第一坐标系下的SLAM地图中的第一位姿数据,包括:The method according to claim 13, wherein the first step of the SLAM map of the electronic device in the first coordinate system is determined based on the collected video image and the SLAM map in the first coordinate system. Pose data, including:
    根据所述视频图像、所述第一坐标系下的SLAM地图以及所述电子设备采集的运动数据,获得所述电子设备在所述第一坐标系下的SLAM地图中的第一位姿数据;所述运动数据包括运动速度数据和运动方向数据。Obtaining the first pose data of the electronic device in the SLAM map in the first coordinate system according to the video image, the SLAM map in the first coordinate system, and the motion data collected by the electronic device; The movement data includes movement speed data and movement direction data.
  15. 根据权利要求13或14所述的方法,其特征在于,所述根据所述视频图像和第二坐 标系下的全局子地图,确定所述电子设备在所述第二坐标系下的全局子地图中的第二位姿数据,包括:The method according to claim 13 or 14, wherein the global sub-map of the electronic device in the second coordinate system is determined according to the video image and the global sub-map in the second coordinate system The second pose data in includes:
    对所述视频图像进行特征提取,以获得图像特征;Performing feature extraction on the video image to obtain image features;
    将所述图像特征在所述第二坐标系下的全局子地图中进行特征匹配,以获得与所述图像特征匹配的地图特征;Performing feature matching on the image feature in the global sub-map in the second coordinate system to obtain a map feature matching the image feature;
    根据所述图像特征和所述地图特征,计算得到所述电子设备在所述第二坐标系下的全局子地图中的第二位姿数据。According to the image feature and the map feature, the second pose data in the global sub-map of the electronic device in the second coordinate system is calculated.
  16. 根据权利要求13或14所述的方法,其特征在于,所述根据所述视频图像和第二坐标系下的全局子地图,确定所述电子设备在所述第二坐标系下的全局子地图中的第二位姿数据,包括:The method according to claim 13 or 14, wherein the global sub-map of the electronic device in the second coordinate system is determined according to the video image and the global sub-map in the second coordinate system The second pose data in includes:
    向所述服务器发送所述视频图像;Sending the video image to the server;
    从所述服务器接收所述第二位姿数据,所述第二位姿数据是所述服务器根据所述视频图像和所述第二坐标系下的全局地图中与所述电子设备的位置关联的全局子地图进行特征提取和特征匹配确定的。The second pose data is received from the server, where the second pose data is associated with the position of the electronic device in the global map in the second coordinate system by the server according to the video image The global sub-map is determined by feature extraction and feature matching.
  17. 一种全局地图更新方法,应用于服务器,其特征在于,所述方法包括:A global map update method applied to a server, characterized in that the method includes:
    根据电子设备的请求,向所述电子设备发送与所述电子设备的位置对应的全局子地图,所述全局子地图是第二坐标系下的全局地图中与所述电子设备的位置关联的子地图;According to the request of the electronic device, the global sub-map corresponding to the location of the electronic device is sent to the electronic device. The global sub-map is the sub-map associated with the location of the electronic device in the global map in the second coordinate system. map;
    从所述电子设备接收地图更新信息;所述地图更新信息用于指示所述电子设备的SLAM系统所构建的第一坐标系下或者所述第二坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异;Receive map update information from the electronic device; the map update information is used to indicate that the SLAM map in the first coordinate system or in the second coordinate system constructed by the SLAM system of the electronic device is relative to the second The difference of the global sub-map in the coordinate system;
    根据所述地图更新信息更新所述全局地图。The global map is updated according to the map update information.
  18. 根据权利要求17所述的方法,其特征在于,所述地图更新信息表示与非冗余关键帧相关联的信息;所述非冗余关键帧表示存在于所述第二坐标系下的SLAM地图中但不存在于第二坐标系下的全局子地图中的关键帧。The method according to claim 17, wherein the map update information represents information associated with a non-redundant key frame; the non-redundant key frame represents a SLAM map existing in the second coordinate system Keyframes in the global submap that do not exist in the second coordinate system.
  19. 根据权利要求18所述的方法,其特征在于,所述地图更新信息具体包括:The method according to claim 18, wherein the map update information specifically comprises:
    所述非冗余关键帧对应的位姿数据、三维地图点以及共视数据,所述共视数据用于指示所述非冗余关键帧与所述第二坐标系下的全局子地图的关键帧之间的共视关系。The pose data, three-dimensional map points, and common view data corresponding to the non-redundant key frame, and the common view data is used to indicate the key of the non-redundant key frame and the global submap in the second coordinate system The common view relationship between frames.
  20. 根据权利要求18或19所述的方法,其特征在于,所述根据所述地图更新信息更新所述全局地图,包括:The method according to claim 18 or 19, wherein the updating the global map according to the map update information comprises:
    将所述地图更新信息加入到所述全局地图。Adding the map update information to the global map.
  21. 根据权利要求17-20任一项所述的方法,其特征在于,所述根据电子设备的请求,向所述电子设备发送与所述电子设备的位置对应的全局子地图,包括:The method according to any one of claims 17-20, wherein the sending a global submap corresponding to the location of the electronic device to the electronic device according to the request of the electronic device comprises:
    接收所述电子设备的初始位置的指示信息;Receiving indication information of the initial position of the electronic device;
    根据所述电子设备的初始位置,在所述全局地图中获取所述与所述电子设备的位置对应的所述全局子地图;Acquiring the global sub-map corresponding to the location of the electronic device in the global map according to the initial location of the electronic device;
    将所述全局子地图发送给所述电子设备。Sending the global submap to the electronic device.
  22. 根据权利要求21所述的方法,其特征在于,所述电子设备的初始位置的指示信息包括用于指示所述电子设备的初始位置的第一位置指纹信息;所述全局地图中的全局子地图对应有第二位置指纹信息;The method according to claim 21, wherein the indication information of the initial position of the electronic device includes first position fingerprint information for indicating the initial position of the electronic device; the global sub-map in the global map Corresponding to the fingerprint information of the second location;
    所述根据所述电子设备的初始位置,在所述全局地图中获取所述与所述电子设备的位置对应的所述全局子地图,包括:The acquiring the global submap corresponding to the location of the electronic device in the global map according to the initial location of the electronic device includes:
    根据所述第一位置指纹信息,在所述全局地图中找到具有与所述第一位置指纹信息相匹配的第二位置指纹信息的所述全局子地图。According to the first location fingerprint information, the global submap with second location fingerprint information matching the first location fingerprint information is found in the global map.
  23. 一种用于虚拟物体显示的电子设备,所述电子设备为第一电子设备,其特征在于,包括:显示组件;摄像头;一个或多个处理器;存储器;多个应用程序;以及一个或多个计算机程序,其中所述一个或多个计算机程序被存储在所述存储器中,所述一个或多个计算机程序包括指令,当所述指令被所述电子设备执行时,使得所述第一电子设备执行以下步骤:An electronic device for displaying virtual objects, the electronic device being a first electronic device, characterized in that it comprises: a display component; a camera; one or more processors; a memory; a plurality of application programs; and one or more Computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the first electronic The device performs the following steps:
    检测到用户打开应用的操作;The user's operation to open the application is detected;
    响应于所述操作,从服务器下载第一全局子地图并存储到所述第一电子设备的即时定位与地图构建(Simultaneous Localization and Mapping,SLAM)系统中;所述第一全局子地图是全局地图中与所述第一电子设备的位置关联的子地图;In response to the operation, the first global sub-map is downloaded from the server and stored in the Simultaneous Localization and Mapping (SLAM) system of the first electronic device; the first global sub-map is a global map A sub-map associated with the location of the first electronic device in;
    在所述显示组件上基于所述第一电子设备的位姿数据来显示虚拟物体,所述第一电子设备的位姿数据是至少根据所述摄像头采集的视频图像和所述第一全局子地图执行位姿计算得到的,其中所述服务器上的全局地图是所述SLAM系统根据一个或多个第二电子设备发送的地图更新信息进行更新得到的,其中,所述一个或多个第二电子设备为除所述第一电子设备外的电子设备。A virtual object is displayed on the display component based on the pose data of the first electronic device, and the pose data of the first electronic device is based on at least the video image collected by the camera and the first global submap Is obtained by performing pose calculation, wherein the global map on the server is updated by the SLAM system according to the map update information sent by one or more second electronic devices, wherein, the one or more second electronic devices The device is an electronic device other than the first electronic device.
  24. 根据权利要求23所述的电子设备,其特征在于,所述地图更新信息是所述第二电子设备的SLAM系统根据第二坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果获得的;其中,所述第二坐标系下的SLAM地图是基于第一坐标系下的SLAM地图以及所述第一坐标系和所述第二坐标系之间的坐标系变换信息得到的;或者,The electronic device according to claim 23, wherein the map update information is based on the SLAM system of the second electronic device according to the SLAM map in the second coordinate system and the global sub-map in the second coordinate system The result of the comparison; wherein, the SLAM map in the second coordinate system is obtained based on the SLAM map in the first coordinate system and the coordinate system transformation information between the first coordinate system and the second coordinate system Of; or,
    所述地图更新信息是所述第二电子设备的SLAM系统根据所述第一坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果以及所述坐标系变换信息获得的。The map update information is obtained by the SLAM system of the second electronic device according to the comparison result of the SLAM map in the first coordinate system and the global sub-map in the second coordinate system and the coordinate system transformation information .
  25. 根据权利要求23或24所述的电子设备,其特征在于,所述地图更新信息包括:The electronic device according to claim 23 or 24, wherein the map update information comprises:
    所述第二电子设备的SLAM系统所构建的SLAM地图的目标关键帧对应的位姿数据、三维地图点以及共视数据,所述共视数据用于指示所述目标关键帧与所述第二电子设备中的第二全局子地图的关键帧之间的共视关系,所述第二全局子地图是全局地图中与所述第二电子设 备的位置关联的子地图。The pose data, three-dimensional map points, and common view data corresponding to the target key frame of the SLAM map constructed by the SLAM system of the second electronic device, the common view data being used to indicate the target key frame and the second A common view relationship between key frames of a second global sub-map in the electronic device, where the second global sub-map is a sub-map associated with the position of the second electronic device in the global map.
  26. 根据权利要求25所述的电子设备,其特征在于,所述目标关键帧为非冗余关键帧,所述非冗余关键帧表示存在于所述第二电子设备的SLAM地图中但不存在于所述第二全局子地图中的关键帧。The electronic device according to claim 25, wherein the target key frame is a non-redundant key frame, and the non-redundant key frame indicates that it exists in the SLAM map of the second electronic device but does not exist in the SLAM map of the second electronic device. The key frame in the second global submap.
  27. 根据权利要求23或24所述的电子设备,其特征在于,所述地图更新信息包括:The electronic device according to claim 23 or 24, wherein the map update information comprises:
    存在于所述第二电子设备的SLAM地图中但不存在于所述第二电子设备中的第二全局子地图中的关键帧,所述第二全局子地图是全局地图中与所述第二电子设备的位置关联的子地图。A key frame that exists in the SLAM map of the second electronic device but does not exist in the second global sub-map of the second electronic device, and the second global sub-map is the same as the second global sub-map in the global map. The sub-map associated with the location of the electronic device.
  28. 根据权利要求23-27任一项所述的电子设备,其特征在于,当所述指令被所述电子设备执行时,使得所述电子设备具体执行以下步骤:The electronic device according to any one of claims 23-27, wherein when the instruction is executed by the electronic device, the electronic device specifically executes the following steps:
    向服务器发送用于指示所述第一电子设备的初始位置的第一位置指纹信息;Sending the first location fingerprint information used to indicate the initial location of the first electronic device to the server;
    从所述服务器接收所述第一全局子地图,所述第一全局子地图对应有第二位置指纹信息,且所述第一位置指纹信息和所述第二位置指纹信息相匹配。The first global submap is received from the server, the first global submap corresponds to second location fingerprint information, and the first location fingerprint information matches the second location fingerprint information.
  29. 一种用于全局地图更新的电子设备,其特征在于,包括:An electronic device used for global map update, which is characterized in that it comprises:
    通信模块,用于从服务器获取与所述电子设备的位置关联的全局子地图,所述全局子地图是第二坐标系下的全局地图中与所述电子设备的位置关联的子地图;A communication module, configured to obtain a global sub-map associated with the location of the electronic device from a server, the global sub-map being a sub-map associated with the location of the electronic device in the global map in the second coordinate system;
    SLAM模块,用于根据所述SLAM模块所构建的第二坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果,或者根据所述SLAM模块所构建的第一坐标系下的SLAM地图与所述第二坐标系下的全局子地图的比较结果以及所述坐标系变换信息,获得地图更新信息;其中所述第二坐标系下的SLAM地图是基于所述第一坐标系下的SLAM地图以及所述第一坐标系和所述第二坐标系之间的坐标系变换信息得到的;所述地图更新信息用于指示所述第二坐标系下或者所述第一坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异;The SLAM module is used to compare the results of the SLAM map in the second coordinate system constructed by the SLAM module with the global sub-map in the second coordinate system, or according to the first coordinate system constructed by the SLAM module The comparison result of the SLAM map under the second coordinate system and the global sub-map under the second coordinate system and the coordinate system transformation information obtain map update information; wherein the SLAM map under the second coordinate system is based on the first coordinate The SLAM map in the first coordinate system and the coordinate system transformation information between the first coordinate system and the second coordinate system are obtained; the map update information is used to indicate the second coordinate system or the first coordinate The difference between the SLAM map under the system and the global sub-map under the second coordinate system;
    所述通信模块还用于,将所述地图更新信息发送给所述服务器,所述地图更新信息用于更新所述服务器中所述第二坐标系下的全局地图。The communication module is further configured to send the map update information to the server, and the map update information is used to update the global map in the second coordinate system in the server.
  30. 根据权利要求29所述的电子设备,其特征在于,所述第二坐标系下的SLAM地图具体为基于所述坐标系变换信息对所述第一坐标系下的SLAM地图进行变换得到的;The electronic device according to claim 29, wherein the SLAM map in the second coordinate system is specifically obtained by transforming the SLAM map in the first coordinate system based on the coordinate system transformation information;
    所述SLAM模块具体用于:The SLAM module is specifically used for:
    通过比较所述第二坐标系下的SLAM地图与所述第二坐标系下的全局子地图,以获得非冗余关键帧,所述非冗余关键帧表示存在于所述第二坐标系下的SLAM地图中但不存在于所二坐标系下的全局子地图中的关键帧;By comparing the SLAM map in the second coordinate system with the global sub-map in the second coordinate system to obtain a non-redundant key frame, the non-redundant key frame indicates that it exists in the second coordinate system The key frames in the SLAM map but not in the global sub-map under all two coordinate systems;
    将所述非冗余关键帧融合到所述第二坐标系下的全局子地图,得到经融合后的全局子地图;Fusing the non-redundant key frames to the global sub-map in the second coordinate system to obtain the fused global sub-map;
    从所述经融合后的全局子地图中获得所述地图更新信息。Obtain the map update information from the fused global sub-map.
  31. 根据权利要求30所述的电子设备,其特征在于,所述SLAM模块具体用于:The electronic device according to claim 30, wherein the SLAM module is specifically configured to:
    将所述非冗余关键帧作为新的关键帧节点增加到在所述第二坐标系下的全局子地图;Adding the non-redundant key frame as a new key frame node to the global sub-map in the second coordinate system;
    更新所述非冗余关键帧与所述第二坐标系下的全局子地图中的关键帧的共视关系;Updating the common view relationship between the non-redundant key frames and the key frames in the global sub-map in the second coordinate system;
    计算所述非冗余关键帧对应的位姿数据和三维地图点;Calculating the pose data and three-dimensional map points corresponding to the non-redundant key frames;
    从所述经融合后的全局子地图中获取所述非冗余关键帧对应的位姿数据、三维地图点以及共视数据作为所述地图更新信息,所述共视数据用于指示所述非冗余关键帧与所述第二坐标系下的全局子地图的关键帧之间的共视关系。Obtain the pose data, three-dimensional map points, and common view data corresponding to the non-redundant key frame from the fused global sub-map as the map update information, and the common view data is used to indicate the non-redundant key frame. The common view relationship between the redundant key frames and the key frames of the global sub-map in the second coordinate system.
  32. 根据权利要求29所述的电子设备,其特征在于,所述SLAM模块具体用于:The electronic device according to claim 29, wherein the SLAM module is specifically configured to:
    通过比较所述第一坐标系下的SLAM地图与所述第二坐标系下的全局子地图,以得到非冗余关键帧,所述非冗余关键帧为存在于所述第一坐标系下的SLAM地图的关键帧,且所述非冗余关键帧经所述坐标系变换信息变换后不存在于所述第二坐标系下的全局子地图;By comparing the SLAM map in the first coordinate system with the global sub-map in the second coordinate system to obtain a non-redundant key frame, the non-redundant key frame is existing in the first coordinate system The key frame of the SLAM map of, and the non-redundant key frame is transformed by the coordinate system transformation information and does not exist in the global sub-map in the second coordinate system;
    将经所述坐标系变换信息变换后的非冗余关键帧融合到所述第二坐标系下的全局子地图,获得所述地图更新信息;所述经所述坐标系变换信息变换后的非冗余关键帧是所述第二坐标系下的关键帧。The non-redundant key frames transformed by the coordinate system transformation information are fused to the global sub-map in the second coordinate system to obtain the map update information; the non-redundant key frames transformed by the coordinate system transformation information The redundant key frame is the key frame in the second coordinate system.
  33. 根据权利要求29-32任一项所述的电子设备,其特征在于,所述通信模块具体用于:The electronic device according to any one of claims 29-32, wherein the communication module is specifically configured to:
    向服务器发送所述电子设备的初始位置的指示信息;Sending the indication information of the initial position of the electronic device to the server;
    从所述服务器接收所述全局子地图,所述全局子地图是根据所述电子设备的初始位置确定的。Receiving the global sub-map from the server, the global sub-map being determined according to the initial position of the electronic device.
  34. 根据权利要求33所述的电子设备,其特征在于,所述电子设备的初始位置的指示信息包括用于指示所述电子设备的初始位置的第一位置指纹信息;所述全局子地图对应有第二位置指纹信息,且所述第一位置指纹信息和所述第二位置指纹信息相匹配。The electronic device according to claim 33, wherein the indication information of the initial position of the electronic device comprises first position fingerprint information for indicating the initial position of the electronic device; the global sub-map corresponds to the first position fingerprint information; Two location fingerprint information, and the first location fingerprint information matches the second location fingerprint information.
  35. 根据权利要求29-34任一项所述的电子设备,其特征在于,所述电子设备还包括全局定位模块和坐标系变化矩阵计算模块;The electronic device according to any one of claims 29-34, wherein the electronic device further comprises a global positioning module and a coordinate system change matrix calculation module;
    所述SLAM模块具体用于,根据采集的视频图像和第一坐标系下的SLAM地图,确定所述电子设备在所述第一坐标系下的SLAM地图中的第一位姿数据;The SLAM module is specifically configured to determine the first pose data of the electronic device in the SLAM map in the first coordinate system according to the collected video image and the SLAM map in the first coordinate system;
    所述全局定位模块具体用于,根据所述视频图像和第二坐标系下的全局子地图,确定所述电子设备在所述第二坐标系下的全局子地图中的第二位姿数据;The global positioning module is specifically configured to determine the second pose data of the electronic device in the global sub-map in the second coordinate system according to the video image and the global sub-map in the second coordinate system;
    所述坐标系变化矩阵计算模块具体用于,根据所述第一位姿数据和所述第二位姿数据,获得所述SLAM地图的第一坐标系和所述全局地图的第二坐标系之间的所述坐标系变换信息。The coordinate system change matrix calculation module is specifically configured to obtain one of the first coordinate system of the SLAM map and the second coordinate system of the global map according to the first pose data and the second pose data The coordinate system transformation information between.
  36. 根据权利要求35所述的电子设备,其特征在于,所述SLAM模块具体用于:The electronic device according to claim 35, wherein the SLAM module is specifically configured to:
    根据所述视频图像、所述第一坐标系下的SLAM地图以及所述电子设备采集的运动数据,获得所述电子设备在所述第一坐标系下的SLAM地图中的第一位姿数据;所述运动数据包括运 动速度数据和运动方向数据。Obtaining the first pose data of the electronic device in the SLAM map in the first coordinate system according to the video image, the SLAM map in the first coordinate system, and the motion data collected by the electronic device; The movement data includes movement speed data and movement direction data.
  37. 根据权利要求35或36所述的电子设备,其特征在于,所述全局定位模块具体用于,The electronic device according to claim 35 or 36, wherein the global positioning module is specifically configured to:
    对所述视频图像进行特征提取,以获得图像特征;Performing feature extraction on the video image to obtain image features;
    将所述图像特征在所述第二坐标系下的全局子地图中进行特征匹配,以获得与所述图像特征匹配的地图特征;Performing feature matching on the image feature in the global sub-map in the second coordinate system to obtain a map feature matching the image feature;
    根据所述图像特征和所述地图特征,计算得到所述电子设备在所述第二坐标系下的全局子地图中的第二位姿数据。According to the image feature and the map feature, the second pose data in the global sub-map of the electronic device in the second coordinate system is calculated.
  38. 根据权利要求35或36所述的电子设备,其特征在于,所述通信模块具体用于:向所述服务器发送所述视频图像;The electronic device according to claim 35 or 36, wherein the communication module is specifically configured to: send the video image to the server;
    从所述服务器接收所述第二位姿数据,所述第二位姿数据是所述服务器根据所述视频图像和所述第二坐标系下的全局地图中与所述电子设备的位置关联的全局子地图进行特征提取和特征匹配确定的。The second pose data is received from the server, where the second pose data is associated with the position of the electronic device in the global map in the second coordinate system by the server according to the video image The global sub-map is determined by feature extraction and feature matching.
  39. 一种用于全局地图的服务器,其特征在于,包括:A server for global map, which is characterized in that it includes:
    通信模块,用于根据电子设备的请求,向所述电子设备发送与所述电子设备的位置对应的全局子地图,所述全局子地图是第二坐标系下的全局地图中与所述电子设备的位置关联的子地图;The communication module is configured to send a global sub-map corresponding to the location of the electronic device to the electronic device according to the request of the electronic device, where the global sub-map is the global map in the second coordinate system and the electronic device The sub-map associated with the location;
    所述通信模块还用于,从所述电子设备接收地图更新信息;所述地图更新信息用于指示所述电子设备的SLAM系统所构建的第一坐标系下或者所述第二坐标系下的SLAM地图相对于所述第二坐标系下的全局子地图的差异;The communication module is also used to receive map update information from the electronic device; the map update information is used to indicate the first coordinate system constructed by the SLAM system of the electronic device or the second coordinate system The difference between the SLAM map and the global sub-map in the second coordinate system;
    处理模块,用于根据所述地图更新信息更新所述全局地图。The processing module is configured to update the global map according to the map update information.
  40. 根据权利要求39所述的服务器,其特征在于,所述地图更新信息表示与非冗余关键帧相关联的信息;所述非冗余关键帧表示存在于所述第二坐标系下的SLAM地图中但不存在于第二坐标系下的全局子地图中的关键帧。The server according to claim 39, wherein the map update information represents information associated with a non-redundant key frame; the non-redundant key frame represents a SLAM map existing in the second coordinate system Keyframes in the global submap that do not exist in the second coordinate system.
  41. 根据权利要求40所述的服务器,其特征在于,所述地图更新信息具体包括:The server according to claim 40, wherein the map update information specifically comprises:
    所述非冗余关键帧对应的位姿数据、三维地图点以及共视数据,所述共视数据用于指示所述非冗余关键帧与所述第二坐标系下的全局子地图的关键帧之间的共视关系。The pose data, three-dimensional map points, and common view data corresponding to the non-redundant key frame, and the common view data is used to indicate the key of the non-redundant key frame and the global submap in the second coordinate system The common view relationship between frames.
  42. 根据权利要求40或41所述的服务器,其特征在于,所述处理模块具体用于:The server according to claim 40 or 41, wherein the processing module is specifically configured to:
    将所述地图更新信息加入到所述全局地图。Adding the map update information to the global map.
  43. 根据权利要求39-42任一项所述的服务器,其特征在于,所述通信模块具体用于:The server according to any one of claims 39-42, wherein the communication module is specifically configured to:
    接收所述电子设备的初始位置的指示信息;Receiving indication information of the initial position of the electronic device;
    根据所述电子设备的初始位置,在所述全局地图中获取所述与所述电子设备的位置对应 的所述全局子地图;Acquiring the global sub-map corresponding to the location of the electronic device in the global map according to the initial location of the electronic device;
    将所述全局子地图发送给所述电子设备。Sending the global submap to the electronic device.
  44. 根据权利要求43所述的服务器,其特征在于,所述电子设备的初始位置的指示信息包括用于指示所述电子设备的初始位置的第一位置指纹信息;所述全局地图中的全局子地图对应有第二位置指纹信息;The server according to claim 43, wherein the indication information of the initial position of the electronic device comprises first position fingerprint information for indicating the initial position of the electronic device; a global sub-map in the global map Corresponding to the fingerprint information of the second location;
    所述处理模块具体用于:根据所述第一位置指纹信息,在所述全局地图中找到具有与所述第一位置指纹信息相匹配的第二位置指纹信息的所述全局子地图。The processing module is specifically configured to: according to the first location fingerprint information, find the global submap with second location fingerprint information matching the first location fingerprint information in the global map.
  45. 一种计算机存储介质,其特征在于,包括计算机指令,当所述计算机指令在电子设备上运行时,使得所述电子设备执行如权利要求1至6中任一项所述的虚拟物体显示方法或者执行如权利要求7至16中任一项所述的全局地图更新方法。A computer storage medium, characterized by comprising computer instructions, which when the computer instructions run on an electronic device, cause the electronic device to execute the virtual object display method according to any one of claims 1 to 6 or The global map update method according to any one of claims 7 to 16 is executed.
  46. 一种计算机存储介质,其特征在于,包括计算机指令,当所述计算机指令在服务器上运行时,使得所述服务器执行如权利要求17至22中任一项所述的全局地图更新方法。A computer storage medium, characterized by comprising computer instructions, when the computer instructions are run on a server, the server is caused to execute the global map update method according to any one of claims 17 to 22.
PCT/CN2020/113321 2019-11-08 2020-09-03 Virtual object display method, global map update method, and device WO2021088497A1 (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN201911097393.1 2019-11-08
CN201911097393 2019-11-08
CN201911099310.2A CN112785700B (en) 2019-11-08 2019-11-11 Virtual object display method, global map updating method and equipment
CN201911099310.2 2019-11-11

Publications (1)

Publication Number Publication Date
WO2021088497A1 true WO2021088497A1 (en) 2021-05-14

Family

ID=75749882

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/113321 WO2021088497A1 (en) 2019-11-08 2020-09-03 Virtual object display method, global map update method, and device

Country Status (2)

Country Link
CN (1) CN112785700B (en)
WO (1) WO2021088497A1 (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113398577B (en) * 2021-05-13 2024-04-09 杭州易现先进科技有限公司 Multi-person AR interaction method and system for offline space
CN114428829A (en) * 2022-01-18 2022-05-03 上汽通用五菱汽车股份有限公司 On-line updating method, device and storage medium for unmanned logistics vehicle map
CN115439536B (en) * 2022-08-18 2023-09-26 北京百度网讯科技有限公司 Visual map updating method and device and electronic equipment
US12048039B2 (en) 2022-08-26 2024-07-23 Htc Corporation Electronic system, control method and non- transitory computer readable storage medium

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120166074A1 (en) * 2010-12-23 2012-06-28 Research In Motion Limited Updating map data from camera images
CN103674011A (en) * 2012-09-19 2014-03-26 联想(北京)有限公司 Instant positioning and map construction device, system and method
CN109086277A (en) * 2017-06-13 2018-12-25 纵目科技(上海)股份有限公司 A kind of overlay region building ground drawing method, system, mobile terminal and storage medium
WO2019010466A1 (en) * 2017-07-07 2019-01-10 Escher Reality, LLC Cloud enabled augmented reality
CN110246182A (en) * 2019-05-29 2019-09-17 深圳前海达闼云端智能科技有限公司 Vision-based global map positioning method and device, storage medium and equipment
CN111177167A (en) * 2019-12-25 2020-05-19 Oppo广东移动通信有限公司 Augmented reality map updating method, device, system, storage and equipment
CN111369628A (en) * 2020-03-05 2020-07-03 南京华捷艾米软件科技有限公司 Multi-camera centralized cooperative SLAM method and system

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120306850A1 (en) * 2011-06-02 2012-12-06 Microsoft Corporation Distributed asynchronous localization and mapping for augmented reality
US20140323148A1 (en) * 2013-04-30 2014-10-30 Qualcomm Incorporated Wide area localization from slam maps
CN107818592B (en) * 2017-11-24 2022-04-01 北京华捷艾米科技有限公司 Method, system and interactive system for collaborative synchronous positioning and map construction
CN109887053B (en) * 2019-02-01 2020-10-20 广州小鹏汽车科技有限公司 SLAM map splicing method and system

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120166074A1 (en) * 2010-12-23 2012-06-28 Research In Motion Limited Updating map data from camera images
CN103674011A (en) * 2012-09-19 2014-03-26 联想(北京)有限公司 Instant positioning and map construction device, system and method
CN109086277A (en) * 2017-06-13 2018-12-25 纵目科技(上海)股份有限公司 A kind of overlay region building ground drawing method, system, mobile terminal and storage medium
WO2019010466A1 (en) * 2017-07-07 2019-01-10 Escher Reality, LLC Cloud enabled augmented reality
CN110246182A (en) * 2019-05-29 2019-09-17 深圳前海达闼云端智能科技有限公司 Vision-based global map positioning method and device, storage medium and equipment
CN111177167A (en) * 2019-12-25 2020-05-19 Oppo广东移动通信有限公司 Augmented reality map updating method, device, system, storage and equipment
CN111369628A (en) * 2020-03-05 2020-07-03 南京华捷艾米软件科技有限公司 Multi-camera centralized cooperative SLAM method and system

Also Published As

Publication number Publication date
CN112785700A (en) 2021-05-11
CN112785700B (en) 2024-09-24

Similar Documents

Publication Publication Date Title
WO2021088498A1 (en) Virtual object display method and electronic device
WO2021088497A1 (en) Virtual object display method, global map update method, and device
US9699375B2 (en) Method and apparatus for determining camera location information and/or camera pose information according to a global coordinate system
US11380012B2 (en) Method and apparatus for visual positioning based on mobile edge computing
CN113790732B (en) Method and device for generating position information
US20240095265A1 (en) Method and apparatus for retrieving 3d map
WO2023051383A1 (en) Device positioning method, device and system
US9596404B2 (en) Method and apparatus for generating a media capture request using camera pose information
WO2022252236A1 (en) Methods and apparatus for encoding and decoding 3d map
WO2022252237A1 (en) Methods and apparatus for encoding and decoding 3d map
WO2022252238A1 (en) 3d map compression method and apparatus, and 3d map decompression method and apparatus
US20240104781A1 (en) 3D Map Compression Method and Apparatus, and 3D Map Decompression Method and Apparatus
US20240119639A1 (en) Apparatus and method for decoding 3d map, and encoded bitstream of 3d map
US20240102826A1 (en) Apparatus and method for encoding 3d map
CN117635697A (en) Pose determination method, pose determination device, pose determination equipment, storage medium and program product

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20885545

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20885545

Country of ref document: EP

Kind code of ref document: A1