KR102695440B1 - Method for determining camera posture and electronic device for the method - Google Patents

Abstract

To determine a camera pose, a first feature point in a first image generated with a first pose of the camera is detected, a first descriptor for the first feature point is generated, and a first spherical model is generated by projecting the first feature point onto a first spherical coordinate system set for the first image based on the coordinates of the first feature point in the first image; a second feature point in a second image generated with a second pose of the camera is detected, a second descriptor for the second feature point is generated, and the second spherical model is generated by projecting the second feature point onto a second spherical coordinate system set for the second image based on the coordinates of the second feature point in the second image; and when the first feature point and the second feature point are matched based on the first descriptor and the second descriptor, the camera pose can be determined based on the second spherical model.

Description

{METHOD FOR DETERMINING CAMERA POSTURE AND ELECTRONIC DEVICE FOR THE METHOD}

The examples below relate to techniques for determining the pose of a camera.

As the level of autonomous driving of vehicles increases, technologies for vehicles to automatically perform parking are being developed. For example, there may be a method in which a vehicle receives a precise map of a parking space using infrastructure installed around the vehicle, and the vehicle performs autonomous parking based on the precise map. This method does not involve the vehicle or driver intervening to create a precise map, and it can generally be created and provided by the person operating the parking space. This method requires a process of creating a precise map in advance, a process in which the operator directly labels information on the precise map, and a process in which a vehicle wishing to use the parking space receives the precise map.

In order to utilize the above method, Lidar is essential for the vehicle, and infrastructure for providing a precise map is also required. In other words, the autonomous parking method using a precise map requires a lot of cost.

One embodiment may provide a method and device for determining a pose of a camera based on a plurality of images.

Another embodiment may provide a method and device for controlling a vehicle based on an image of the current time.

In one embodiment, a camera pose determination method performed by an electronic device comprises: detecting a first feature point in a first image generated with a first pose of a camera and generating a first descriptor for the first feature point; projecting the first feature point onto a first spherical coordinate system set for the first image based on coordinates of the first feature point in the first image to generate a first spherical model, wherein the first feature point is projected into the first feature spherical coordinates; detecting a second feature point in a second image generated with a second pose of the camera and generating a second descriptor for the second feature point; projecting the second feature point onto a second spherical coordinate system set for the second image based on coordinates of the second feature point in the second image to generate a second spherical model, wherein the second feature point is projected into the second feature spherical coordinates; matching the first feature point and the second feature point based on the first descriptor and the second descriptor; and matching the second feature point to the first feature point. In this case, the step of determining the pose of the camera based on the second sphere model is included.

The step of generating the second spherical model may include the steps of setting the second spherical coordinate system having a preset radius centered on the position of the camera, calculating a second reference vector between a center point of the second spherical coordinate system and coordinates of the second feature point in the second image based on calibration information of the camera and rotation and translation information of the camera, and generating the second spherical model by projecting the second reference vector onto the second spherical coordinate system.

The above calibration information may include internal parameters of the camera and the degree of distortion of the lens.

The step of matching the first feature point and the second feature point may include the steps of calculating an initial rotation and translation matrix for the second image, determining first transformed spherical coordinates by projecting the first feature point of the first image onto the second spherical coordinate system based on the initial rotation and translation matrix, determining first transformed image coordinates by projecting the first transformed spherical coordinates onto the coordinate system of the second image, and matching the first feature point and the second feature point in the second image based on the first transformed image coordinates.

The step of matching the first feature point and the second feature point in the second image based on the first transformed image coordinates may include the step of setting a search range including the first transformed image coordinates on the second image, the step of determining the second descriptor corresponding to the first descriptor among one or more feature points within the search range, and the step of matching the second feature point having the second descriptor with the first feature point.

The step of determining the second descriptor corresponding to the first descriptor among one or more feature points within the search range may include the step of calculating Hamming distances between each of the descriptors of the one or more feature points within the search range and the first descriptor, and the step of determining the descriptor having the smallest Hamming distance as the second descriptor.

If the second feature point matches the first feature point, the step of determining the pose of the camera based on the second spherical model may include the steps of calculating a second reference spherical coordinate by projecting a reference feature point in the world coordinate system corresponding to the first feature point into the second spherical coordinate system, calculating a second difference between the second feature spherical coordinate and the second reference spherical coordinate, and generating an optimal rotation and translation matrix by adjusting the initial rotation and translation matrix so that the second difference is minimized.

The above camera attitude determination method may further include a step of generating a vehicle path based on the attitude of the camera.

The above camera pose determination method may further include a step of updating visual simultaneous localization and mapping (SLAM) based on matching of the first feature point and the second feature point.

The above electronic device can be mounted on an autonomous vehicle or a vehicle supporting ADAS (Advanced Driver Assistance Systems).

According to another aspect, an electronic device includes a processor for executing a program for determining a pose of a camera, and a memory for storing the program, wherein the program comprises: a step of detecting a first feature point in a first image generated with a first pose of the camera and generating a first descriptor for the first feature point; a step of projecting the first feature point onto a first spherical coordinate system set for the first image based on coordinates of the first feature point in the first image to generate a first spherical model, wherein the first feature point is projected into the first feature spherical coordinates; a step of detecting a second feature point in a second image generated with a second pose of the camera and generating a second descriptor for the second feature point; a step of projecting the second feature point onto a second spherical coordinate system set for the second image based on coordinates of the second feature point in the second image to generate a second spherical model, wherein the second feature point is projected into the second feature spherical coordinates; and matching the first feature point and the second feature point based on the first descriptor and the second descriptor. Step, and if the second feature point matches the first feature point, a step of determining the pose of the camera based on the second sphere model is performed.

The step of generating the second spherical model may include the step of setting the second spherical coordinate system having a preset radius centered on the position of the camera, the step of calculating a reference vector based on the center point of the second spherical coordinate system and the coordinates of the second feature point within the second image based on calibration information of the camera and rotation and movement information of the camera, and the step of generating the second spherical model by projecting the reference vector onto the second spherical coordinate system.

The above calibration information may include internal parameters of the camera and the degree of distortion of the lens.

The step of matching the first feature point and the second feature point may include the steps of calculating an initial rotation and translation matrix for the second image, determining first transformed spherical coordinates by projecting the first feature point of the first image onto the second spherical coordinate system based on the initial rotation and translation matrix, determining first transformed image coordinates by projecting the first transformed spherical coordinates onto the coordinate system of the second image, and matching the first feature point and the second feature point in the second image based on the first transformed image coordinates.

The step of matching the first feature point and the second feature point in the second image based on the first transformed image coordinates may include the step of setting a search range including the first transformed image coordinates on the second image, the step of determining the second descriptor corresponding to the first descriptor among one or more feature points within the search range, and the step of matching the second feature point having the second descriptor with the first feature point.

The step of determining the second descriptor corresponding to the first descriptor among one or more feature points within the search range may include the step of calculating Hamming distances between each of the descriptors of the one or more feature points within the search range and the first descriptor, and the step of determining the descriptor having the smallest Hamming distance as the second descriptor.

If the second feature point matches the first feature point, the step of determining the pose of the camera based on the second spherical model may include the steps of calculating a second reference spherical coordinate by projecting a reference feature point in the world coordinate system corresponding to the first feature point into the second spherical coordinate system, calculating a second difference between the second feature spherical coordinate and the second reference spherical coordinate, and generating an optimal rotation and translation matrix by adjusting the initial rotation and translation matrix so that the second difference is minimized.

The above program may further perform a step of generating a path of the vehicle based on the attitude of the camera.

The above program may further include a step of updating a visual simultaneous localization and mapping (SLAM) map based on the matching of the first feature point and the second feature point.

The above electronic device can be mounted on an autonomous vehicle or a vehicle supporting ADAS (Advanced Driver Assistance Systems).

A method and device for determining a pose of a camera based on multiple images can be provided.

A method and device for controlling a vehicle based on the attitude of a camera may be provided.

FIG. 1 illustrates a vehicle including an electronic device according to an example.
Figure 2 is a block diagram of an electronic device according to one embodiment.
FIG. 3 is a flowchart of a method for determining a pose of a camera according to one embodiment.
Figure 4 is a flowchart of a method for generating a first sphere model according to an example.
Figure 5 illustrates a sphere model generated based on a reference vector according to an example.
FIG. 6 is a flowchart of a method for matching a first feature point of a first image and a second feature point of a second image according to an example.
FIG. 7 is a flowchart of a method for matching a first feature point and a second feature point based on a first technician and a second technician according to an example.
FIG. 8 illustrates a method for determining first transformed image coordinates by projecting first transformed spherical coordinates for a first feature point according to an example into the coordinate system of a second image.
FIG. 9 illustrates a search range including first transformed image coordinates set on a second image according to an example.
Figure 10 is a flowchart of a method for determining a second technician according to an example.
FIG. 11 is a flowchart of a method for determining a camera pose based on a first sphere model and a second sphere model according to an example.
Figure 12 illustrates a method for calculating the difference between feature sphere coordinates and reference sphere coordinates according to an example.
Figure 13 illustrates additional operations performed based on the determined camera pose according to an example.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, since various modifications may be made to the embodiments, the scope of rights of the patent application is not limited or restricted by these embodiments. It should be understood that all modifications, equivalents, or substitutes to the embodiments are included in the scope of rights.

The terms used in the examples are for the purpose of description only and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, the terms "comprises" or "has" and the like are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood to not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments belong. Terms defined in commonly used dictionaries, such as those defined in common usage dictionaries, should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense unless expressly defined in this application.

In addition, when describing with reference to the attached drawings, the same components will be given the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted. When describing an embodiment, if it is determined that a specific description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

Also, in describing components of the embodiments, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only intended to distinguish the components from other components, and the nature, order, or sequence of the components are not limited by the terms. When it is described that a component is "connected," "coupled," or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but another component may also be "connected," "coupled," or "connected" between each component.

Components included in one embodiment and components that have common functions will be described using the same names in other embodiments. Unless otherwise stated, descriptions made in one embodiment can be applied to other embodiments, and specific descriptions will be omitted to the extent of overlap.

FIG. 1 illustrates a vehicle including an electronic device according to an example.

According to one embodiment, the electronic device (110) may be included in a vehicle (100) that supports autonomous driving or a vehicle (100) that supports an advanced driver assistance system (ADAS).

According to one aspect, the vehicle (100) can drive in an autonomous mode according to a recognized driving environment even in a situation where there is little or no input from a driver. The driving environment can be recognized through one or more sensors attached or installed to the vehicle (100). For example, the one or more sensors can include a camera, a lidar, a radar, and a voice recognition sensor, but are not limited to the examples described. The driving environment can include a road, a condition of the road, a type of lane, the presence or absence of surrounding vehicles, a distance from a nearby vehicle, the weather, the presence or absence of obstacles, and the like, but are not limited to the examples described.

The vehicle (100) recognizes the driving environment and generates an autonomous driving path suitable for the driving environment. The autonomous vehicle controls internal and external mechanical elements to follow the autonomous driving path. The vehicle (100) can periodically generate an autonomous driving path.

According to another aspect, the vehicle (100) can assist the driver's driving by using ADAS. ADAS includes an Autonomous Emergency Braking (AEB) system that reduces speed or stops the vehicle by itself without the driver stepping on the brakes when there is a risk of collision, a Lane Keep Assist System (LKAS) that controls the driving direction when the vehicle departs from the lane to maintain the lane, an Advanced Smart Cruise Control (ASCC) that automatically maintains a distance from the vehicle in front while driving at a preset speed, an Active Blind Spot Detection (ABSD) system that detects a risk of collision in a blind spot and assists in a safe lane change, and an Around View Monitor (AVM) system that visually displays the situation around the vehicle.

The electronic device (110) included in the vehicle (100) controls the mechanical device of the vehicle (100) to enable autonomous driving or assist driving of the driver, and can be used in ECUs, BCMs, and various types of controllers or sensors other than those described in the embodiments.

According to one embodiment, the electronic device (110) may generate an image by photographing one side (e.g., the front) of the vehicle (100) using a camera. The electronic device (110) may detect feature points of an object in the image and generate a map around the vehicle (100) based on the detected feature points. For example, the map may be generated based on SLAM (Simultaneous Localization And Mapping). For example, the map may be generated based on VSLAM (visual SLAM) performed based on visual image information. The map generated based on VSLAM may be implemented in the form of a point cloud, and points in the map may correspond to feature points of each of a plurality of frames. The coordinate system of the map may be a world coordinate system. The electronic device (110) may generate a path along which the vehicle (100) drives (or parks) using the map, and control the movement of the vehicle (110) so that the vehicle (110) moves along the path.

In one embodiment, the map can be generated based on one or more images. The one or more images can include images taken at different viewpoints at the same time and images taken at different times. For example, the map can be generated in the form of a point cloud that displays feature points in the image in the world coordinate system of the map. If a new feature point that does not correspond to a feature point (or point) in the existing map is detected in the image, the map can be updated by adding the new feature point to the map.

In order for the electronic device (110) to control the movement of the vehicle (100), the current attitude of the vehicle (100) must be accurately determined. For example, the electronic device (110) can determine the attitude of the vehicle (100) by determining the attitude of the camera based on the captured image. A method for determining the attitude of the camera is described in detail below with reference to FIGS. 2 to 12.

Figure 2 is a block diagram of an electronic device according to one embodiment.

According to one aspect, an electronic device (200) includes a communication unit (210), a processor (220), a memory (230), and a camera (240). The electronic device (200) (e.g., the electronic device (110) of FIG. 1) may be included in a vehicle (e.g., the vehicle (100) of FIG. 1). For example, the electronic device (200) may be a device such as an electronic control unit (ECU) or a body control module (BCM). As another example, the electronic device (200) may be an independent device connected to an ECU/BCM.

The communication unit (210) is connected to the processor (220), memory (230), and camera (240) to transmit and receive data. The communication unit (210) can be connected to another external device to transmit and receive data. Hereinafter, the expression "transmitting and receiving" "A" can indicate transmitting and receiving "information or data representing A."

The communication unit (210) may be implemented as a circuitry within the electronic device (200). For example, the communication unit (210) may include an internal bus and an external bus. As another example, the communication unit (210) may be an element that connects the electronic device (200) and an external device. The communication unit (210) may be an interface. The communication unit (210) may receive data from an external device and transmit the data to the processor (220) and the memory (230).

The processor (220) processes data received by the communication unit (210) and data stored in the memory (230). The “processor” may be a data processing device implemented as hardware having a circuit having a physical structure for executing desired operations. For example, the desired operations may include code or instructions included in a program. For example, the data processing device implemented as hardware may include a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an ASIC (Application-Specific Integrated Circuit), and an FPGA (Field Programmable Gate Array).

The processor (220) executes computer-readable code (e.g., software) stored in memory (e.g., memory (230)) and instructions issued by the processor (220).

The memory (230) stores data received by the communication unit (210) and data processed by the processor (220). For example, the memory (230) may store a program (or application, software). The stored program may be a set of syntaxes that are coded to determine the attitude of the camera and can be executed by the processor (220).

According to one aspect, the memory (230) may include one or more volatile memory, nonvolatile memory, and random access memory (RAM), flash memory, a hard disk drive, and an optical disk drive.

The memory (230) stores a set of instructions (e.g., software) for operating the electronic device (200). The set of instructions for operating the electronic device (200) is executed by the processor (220).

The camera (240) generates an image by photographing a scene. The camera (240) may include multiple cameras. For example, the multiple cameras may be positioned respectively on the front, rear, left, and right sides of the vehicle.

According to one embodiment, the lens of the camera (240) may be a fish-eye lens.

The communication unit (210), processor (220), memory (230), and camera (240) are described in detail with reference to FIGS. 3 to 13 below.

FIG. 3 is a flowchart of a method for determining a pose of a camera according to one embodiment.

The steps (310 to 360) below are performed by the electronic device (200) described above with reference to FIG. 2.

In step (310), the electronic device (200) detects a first feature point in a first image generated in a first pose of the camera (240) and generates a first descriptor for the first feature point. One or more feature points may be detected based on the first image. Any one of the feature points may be referred to as the first feature point. The first image may be an image captured at a first viewpoint.

In step (320), the electronic device (200) generates a first sphere model by projecting the first feature point onto a first spherical coordinate system set for the first image based on the coordinates of the first feature point in the first image. The first sphere model can be generated by projecting one or more feature points of the detected first image onto the first spherical coordinate system. The coordinates of the first feature point projected onto the first spherical coordinate system can be referred to as first feature sphere coordinates. The first spherical coordinate system can be set based on the center of a sphere determined based on the position of the camera (240) at the first viewpoint and a preset radius. For example, the radius can be preset according to the degree of distortion of a lens of the camera (240).

The method of generating the first sphere model is described in detail below with reference to FIGS. 4 and 5.

In step (330), the electronic device (200) detects a second feature point in a second image generated in a second pose of the camera (240) and generates a second descriptor for the second feature point. One or more feature points may be detected based on the second image. Any one of the feature points may be referred to as the second feature point. The second image may be an image captured at a second time point that follows the first time point.

In step (340), the electronic device (200) generates a second spherical model by projecting the second feature point onto a second spherical coordinate system set for the second image based on the coordinates of the second feature point in the second image. The second spherical model can be generated by projecting one or more feature points of the detected second image onto the second spherical coordinate system. The coordinates of the second feature point projected onto the second spherical coordinate system can be referred to as second feature spherical coordinates. The second spherical coordinate system can be set based on a center of a sphere determined based on a position of the camera (240) at the second viewpoint and a preset radius.

In step (350), the electronic device (200) matches the first feature point and the second feature point based on the first descriptor and the second descriptor. For example, the first feature point and the second feature point can be matched by projecting the first feature point into the coordinate system of the second image. A method of matching the first feature point and the second feature point is described in detail with reference to FIGS. 6 to 10 below.

In step (360), the electronic device (200) determines the pose of the camera (240) based on the first sphere model and the second sphere model when the first feature point and the second feature point are matched. A method for determining the pose of the camera (240) is described in detail below with reference to FIGS. 11 and 12.

Figure 4 is a flowchart of a method for generating a first sphere model according to an example.

According to one embodiment, step (320) described above with reference to FIG. 3 may include steps (410 to 430).

In step (410), the electronic device (200) sets a first spherical coordinate system having a preset radius centered on the position of the camera (240).

In step (420), the electronic device (200) calculates a first reference vector between the center point of the first spherical coordinate system and the coordinates of the first feature point in the first image based on the calibration information of the camera (240) and the rotation and translation information of the camera (240). For example, the first image may be set on the first spherical coordinate system based on the calibration information of the camera (240) and the rotation and translation information of the camera (240), and the first reference vector may be calculated based on the center point of the first spherical coordinate system and the coordinates of the first feature point in the first image.

In step (430), the electronic device (200) generates a first spherical model by projecting the first reference vector onto the first spherical coordinate system. One or more feature points within the first image may be projected onto the first spherical coordinate system.

Figure 5 illustrates a sphere model generated based on a reference vector according to an example.

According to one embodiment, the electronic device (200) can set a center point (501) based on a position of the camera (240) at a first viewpoint, and set a first spherical coordinate system (502) having a preset radius based on the center point (501). The electronic device (400) can set a first image (510) on the first spherical coordinate system based on calibration information of the camera (240) and rotation and movement information of the camera (240). The electronic device (200) can calculate a first reference vector (520) based on coordinates (511) of the center point (501) and the first feature point in the first image. The electronic device (200) can generate a first spherical model by projecting the first reference vector (520) onto the first spherical coordinate system (502). For example, a first feature sphere coordinate (531) corresponding to a first feature point can be determined by extending (530) a first reference vector (520) onto a first sphere coordinate system (502).

The above description relates to a method for generating a first sphere model, but can also be replaced with a description of a method for generating a second sphere model, so any redundant description is omitted below.

FIG. 6 is a flowchart of a method for matching a first feature point of a first image and a second feature point of a second image according to an example.

According to one embodiment, step (350) described above with reference to FIG. 3 may include steps (610 to 640).

In step (610), the electronic device (200) calculates an initial rotation and translation matrix [R|t] for the second image.

In one embodiment, when the second image is the k-th frame, an initial rotation and translation matrix for the second image can be calculated (or estimated) based on the rotation and translation matrix of the k-2-th frame and the rotation and translation matrix of the k-1-th frame. For example, an initial rotation and translation matrix for the current frame (i.e., the second image) can be calculated based on a change trend of the rotation and translation matrices of previous frames.

In another embodiment, if the k-th frame is an initial frame, such as the second frame, the feature points of the first frame and the second frame are matched, and an essential matrix can be computed using the matching pair information. By decomposing the essential matrix, the initial rotation and translation matrices of the camera can be computed.

In step (620), the electronic device (200) determines the first transformed spherical coordinates by projecting the first feature point of the first image onto the second spherical coordinate system based on the initial rotation and translation matrix. For example, the electronic device (200) can project the coordinates on the first spherical coordinate system corresponding to the first feature point into coordinates on the second spherical coordinate system based on the initial rotation and translation matrix.

In step (630), the electronic device (200) determines the first transformed image coordinates by projecting the first transformed spherical coordinates into the coordinate system of the second image. The coordinate system of the second image may be a plane of the second image. A method for determining the first transformed image coordinates is described in detail below with reference to FIG. 7.

In step (640), the electronic device (200) matches the first feature point and the second feature point based on the first transformed image coordinates, the first descriptor, and the second descriptor. A method of matching the first feature point and the second feature point based on the first transformed image coordinates, the first descriptor, and the second descriptor is described in detail below with reference to FIGS. 8 to 10.

FIG. 7 is a flowchart of a method for matching a first feature point and a second feature point based on a first technician and a second technician according to an example.

According to one embodiment, the electronic device (200) can determine the first transformed image coordinates (730) by projecting the first transformed spherical coordinates (720) (corresponding to the first feature point) determined on the second spherical coordinate system (702) centered on the center point (701) into the coordinate system of the second image (710).

FIG. 8 illustrates a method for determining first transformed image coordinates by projecting first transformed spherical coordinates for a first feature point according to an example into the coordinate system of a second image.

According to one embodiment, step (640) described above with reference to FIG. 6 may include steps (810 to 830).

In step (810), the electronic device (200) sets a search range including the first transformed image coordinates on the second image. For example, a search range having a certain radius centered on the first transformed image coordinates may be set on the second image.

According to one embodiment, the radius of the search range may be set differently based on the first transformed image coordinates. For example, the radius may be set larger as the first transformed image coordinates are located at the outer edge of the second image.

In step (820), the electronic device (200) determines a second descriptor corresponding to the first descriptor among one or more feature points (i.e., second feature points) within the search range. A method for determining the second descriptor corresponding to the first descriptor is described in detail below with reference to FIG. 10.

At step (830), the electronic device (200) matches a second feature point having a second descriptor with the first feature point.

FIG. 9 illustrates a search range including first transformed image coordinates set on a second image according to an example.

In the embodiment described with reference to FIG. 7, a search range (910) including the first transformed image coordinates (730) may be set on the second image (710). The size of the search range (910) may be determined based on the first transformed image coordinates (730). For example, the search range (910) may be set larger as the first transformed image coordinates (730) are located further out from the second image (710).

Figure 10 is a flowchart of a method for determining a second technician according to an example.

According to one embodiment, step (820) described above with reference to FIG. 8 may include steps (1010 and 1020).

In step (1010), the electronic device (200) calculates Hamming distances between each of the descriptors of one or more feature points within a search range (e.g., the search range (910) of FIG. 9) and the first descriptor.

In step (1020), the electronic device (200) determines the descriptor with the smallest Hamming distance as the second descriptor.

FIG. 11 is a flowchart of a method for determining a camera pose based on a first sphere model and a second sphere model according to an example.

According to one embodiment, step (360) described above with reference to FIG. 3 may include steps (1110 to 1150). For example, step (1110) may be performed after step (1020) described above with reference to FIG. 10 is performed.

In step (1110), the electronic device (200) calculates a second reference spherical coordinate by projecting a reference feature point of the world coordinate system corresponding to the first feature point into a second spherical coordinate system. For example, the world coordinate system may be a coordinate system of a pre-generated (or updated) map.

In step (1120), the electronic device (200) calculates a second difference between the second feature sphere coordinates and the second reference sphere coordinates. A method for calculating the second difference is described in detail below with reference to FIG. 12.

In step (1130), the electronic device (200) generates an optimal rotation and translation matrix by adjusting the initial rotation and translation matrix so that the second difference is minimized. A method for generating an optimal rotation and translation matrix is described in detail below with reference to FIG. 12.

Steps (1110 to 1130) may be repeatedly performed as many times as the number of feature points in the first image. That is, steps (1110 to 1130) may be performed for each of all feature points in the first image. For example, in step (1130), the electronic device (200) may generate an optimal rotation and translation matrix by repeatedly adjusting the initial rotation and translation matrix so that the calculated plurality of second differences are minimized.

In one embodiment, not only the basic feature points on the map generated based on the first image, but also the feature points on the map generated based on the previous image of the first image can be matched with the second feature points. Using the above matching, the rotation and translation matrices can be further optimized. That is, additional previous images can be used to obtain more accurate rotation and translation matrices.

Figure 12 illustrates a method for calculating the difference between feature sphere coordinates and reference sphere coordinates according to an example.

In one embodiment, general graph optimization (g2o) can be used to optimize the rotation and translation matrices. For example, the mathematical expression for reprojection error used in g2o can be modified to fit the spherical model.

[Mathematical formula 1]

In [Mathematical Formula 1], p represents the pose of the camera, i represents the ith feature point among all feature points, M _i (1210) is the coordinate of the reference feature point in the world coordinate system, m _i (1211) is the coordinate projected on the second spherical coordinate system (1202) centered on the center point (1201) of M _i (1210), Proj(M _i )(1220) is the second feature spherical coordinate projected on the second feature point in the second spherical coordinate system (1202), represents the reprojection error value for p. The difference between m _i (1211) and Proj(M _i )(1220) can be computed as the second difference (1230).

[Mathematical expression 2] may be a mathematical expression of the Jacobian error obtained by partially differentiating [Mathematical expression 1] with respect to the camera pose.

[Mathematical formula 2]

[Mathematical formula 2] can be expressed by [Mathematical Formula 3] below.

[Mathematical formula 3]

In [Mathematical Formula 3], is a function that converts the reference feature points of the world coordinate system into the camera coordinate system using the camera's pose. is a function that projects feature points from the camera coordinate system to the spherical coordinate system. , r is the radius of the sphere, and X, Y, and Z can be the coordinates of the reference feature points converted to the camera's coordinate system.

Since the camera pose is estimated by utilizing the positional relationship between three-dimensional spherical models using the mathematical formulas above, the estimation error can be reduced compared to existing methods (e.g., cubemap SLAM) that estimate the camera pose by utilizing the positional relationship between two planar models.

Figure 13 illustrates additional operations performed based on the determined camera pose according to an example.

Each of the steps (1310 and 1320) below may be performed after step (360) is performed.

In step (1310), the electronic device (200) generates a path for the vehicle based on the attitude of the camera. For example, the electronic device (200) can determine the position and attitude of the camera (or vehicle) on the map based on the attitude of the camera, and generate a path to the destination based on this. The electronic device (200) can control the vehicle to proceed along the path.

In step (1320), the electronic device (200) updates the map of the visual SLAM based on the matching of the first feature point and the second feature point. For example, the electronic device (200) can update the map by displaying on the map additional feature points that do not match feature points detected by the previous image among the feature points of the second image. The electronic device (200) can add the additional feature points to the map based on the pose of the camera at the current time and the coordinates of the additional feature points in the second image. The SLAM performed using the camera pose determination method described with reference to FIGS. 2 to 12 can be named O2RB (Omnidirectional Oriented FAST and Rotated BRIEF) SLAM.

The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known to and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, flash memories, etc. Examples of the program commands include not only machine language codes generated by a compiler but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

The software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and/or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal waves, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over network-connected computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

Although the embodiments have been described with limited drawings as described above, those skilled in the art can apply various technical modifications and variations based on the above. For example, appropriate results can be achieved even if the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or are replaced or substituted by other components or equivalents.

Therefore, other implementations, other embodiments, and equivalents to the claims are also included in the scope of the claims described below.

200: Electronic devices
210: Communications Department
220: Processor
230: Memory
240: Camera

Claims (21)

A method for determining camera attitude, performed by an electronic device,
A step of detecting a first feature point in a first image generated in a first pose of a camera and generating a first descriptor for the first feature point;
A step of generating a first spherical model by projecting the first feature point onto a first spherical coordinate system set for the first image based on coordinates of the first feature point within the first image, wherein the first feature point is projected onto the first feature spherical coordinate;
A step of detecting a second feature point in a second image generated in a second pose of the camera and generating a second descriptor for the second feature point;
A step of generating a second spherical model by projecting the second feature point onto a second spherical coordinate system set for the second image based on the coordinates of the second feature point within the second image, wherein the second feature point is projected onto the second feature spherical coordinates;
A step of matching the first feature point and the second feature point based on the first technician and the second technician; and
A step of determining the pose of the camera based on the second sphere model when the second feature point matches the first feature point.
Including,
The step of matching the first feature point and the second feature point is:
A step of calculating an initial rotation and translation matrix for the second image;
A step of determining a first transformed spherical coordinate by projecting the first feature point of the first image onto the second spherical coordinate system based on the initial rotation and translation matrix;
A step of determining the first transformed image coordinate by projecting the first transformed spherical coordinate into the coordinate system of the second image; and
A step of matching the first feature point and the second feature point in the second image based on the first transformed image coordinates.
Including,
If the second feature point matches the first feature point, the step of determining the pose of the camera based on the second sphere model is:
A step of calculating a second reference spherical coordinate by projecting a reference feature point of the world coordinate system corresponding to the first feature point into the second spherical coordinate system;
a step of calculating a second difference between the second feature sphere coordinates and the second reference sphere coordinates; and
A step of generating an optimal rotation and translation matrix by adjusting the initial rotation and translation matrix so that the second difference is minimized.
Including,
How to determine camera pose.
In the first paragraph,
The step of generating the above second sphere model is:
A step of setting the second spherical coordinate system having a preset radius centered on the position of the camera;
A step of calculating a second reference vector between the center point of the second spherical coordinate system and the coordinates of the second feature point in the second image based on the calibration information of the camera and the rotation and movement information of the camera; and
A step of generating the second sphere model by projecting the second reference vector onto the second sphere coordinate system.
Including,
How to determine camera pose.
In the second paragraph,
The above calibration information includes the internal parameters of the camera and the degree of distortion of the lens.
How to determine camera pose.
delete In the first paragraph,
The step of matching the first feature point and the second feature point in the second image based on the first transformed image coordinates is:
A step of setting a search range including the first transformed image coordinates on the second image;
A step of determining the second descriptor corresponding to the first descriptor among one or more feature points within the search range; and
A step of matching the second feature point having the second descriptor with the first feature point.
Including,
How to determine camera pose.
In paragraph 5,
The step of determining the second descriptor corresponding to the first descriptor among one or more feature points within the search range is:
A step of calculating Hamming distances between each of the descriptors of one or more feature points within the search range and the first descriptor; and
A step of determining the descriptor with the smallest Hamming distance as the second descriptor
Including,
How to determine camera pose.
delete In the first paragraph,
Step of generating a vehicle path based on the attitude of the above camera
Including more,
How to determine camera pose.
In the first paragraph,
A step of updating a map of visual simultaneous localization and mapping (SLAM) based on matching of the first feature point and the second feature point.
Including more,
How to determine camera pose.
In the first paragraph,
The above electronic device is installed in a vehicle that supports autonomous vehicles or ADAS (Advanced Driver Assistance Systems).
How to determine camera pose.
A computer-readable recording medium containing a program for performing the method of any one of claims 1, 2, 3, 5, 6, 8, 9, and 10.
Electronic devices,
a processor that executes a program that determines the attitude of the camera; and
Memory for storing the above program
Including,
The above program is,
A step of detecting a first feature point in a first image generated in a first pose of a camera and generating a first descriptor for the first feature point;
A step of generating a first spherical model by projecting the first feature point onto a first spherical coordinate system set for the first image based on coordinates of the first feature point within the first image, wherein the first feature point is projected onto the first feature spherical coordinate;
A step of detecting a second feature point in a second image generated in a second pose of the camera and generating a second descriptor for the second feature point;
A step of generating a second spherical model by projecting the second feature point onto a second spherical coordinate system set for the second image based on the coordinates of the second feature point within the second image, wherein the second feature point is projected onto the second feature spherical coordinates;
A step of matching the first feature point and the second feature point based on the first technician and the second technician; and
A step of determining the pose of the camera based on the second sphere model when the second feature point matches the first feature point.
To perform,
The step of matching the first feature point and the second feature point is:
A step of calculating an initial rotation and translation matrix for the second image;
A step of determining a first transformed spherical coordinate by projecting the first feature point of the first image onto the second spherical coordinate system based on the initial rotation and translation matrix;
A step of determining the first transformed image coordinate by projecting the first transformed spherical coordinate into the coordinate system of the second image; and
A step of matching the first feature point and the second feature point in the second image based on the first transformed image coordinates.
Including,
If the second feature point matches the first feature point, the step of determining the pose of the camera based on the second sphere model is:
A step of calculating a second reference spherical coordinate by projecting a reference feature point of the world coordinate system corresponding to the first feature point into the second spherical coordinate system;
a step of calculating a second difference between the second feature sphere coordinates and the second reference sphere coordinates; and
A step of generating an optimal rotation and translation matrix by adjusting the initial rotation and translation matrix so that the second difference is minimized.
Including,
Electronic devices.
In Article 12,
The step of generating the above second sphere model is:
A step of setting the second spherical coordinate system having a preset radius centered on the position of the camera;
A step of calculating a second reference vector between the center point of the second spherical coordinate system and the coordinates of the second feature point in the second image based on the calibration information of the camera and the rotation and movement information of the camera; and
A step of generating the second sphere model by projecting the second reference vector onto the second sphere coordinate system.
Including,
Electronic devices.
In Article 13,
The above calibration information includes the internal parameters of the camera and the degree of distortion of the lens.
Electronic devices.
delete In Article 12,
The step of matching the first feature point and the second feature point in the second image based on the first transformed image coordinates is:
A step of setting a search range including the first transformed image coordinates on the second image; and
A step of determining the second descriptor corresponding to the first descriptor among one or more feature points within the search range;
A step of matching the second feature point having the second descriptor with the first feature point.
Including,
Electronic devices.
In Article 16,
The step of determining the second descriptor corresponding to the first descriptor among one or more feature points within the search range is:
A step of calculating Hamming distances between each of the descriptors of one or more feature points within the search range and the first descriptor; and
A step of determining the descriptor with the smallest Hamming distance as the second descriptor
Including,
Electronic devices.
delete In Article 12,
The above program is,
Step of generating a vehicle path based on the attitude of the above camera
to perform more,
Electronic devices.
In Article 12,
The above program is,
A step of updating a map of visual simultaneous localization and mapping (SLAM) based on matching of the first feature point and the second feature point.
Including more,
Electronic devices.
In Article 12,
The above electronic device is installed in a vehicle that supports autonomous vehicles or ADAS (Advanced Driver Assistance Systems).
Electronic devices.

Images