KR102733090B1 - Apparatus and method for obtaining lane information - Google Patents

Abstract

A method for obtaining lane information according to one embodiment of the present invention includes: a step of estimating a position of a vehicle based on an image of the surroundings of the vehicle; a step of matching a lane corresponding to the estimated position of the vehicle in a precise map to a lane in the image of the surroundings; a step of determining the matched lane as a valid lane using matching information of the lane; a step of fitting the determined valid lane; a step of obtaining polynomial function coefficients for the fitted valid lane as correct data of the valid lane used for learning for generating a model for obtaining lane information; and a step of labeling the polynomial function coefficients for the fitted valid lane in the image of the surroundings.

Description

{APPARATUS AND METHOD FOR OBTAINING LANE INFORMATION}

The present invention relates to a lane information acquisition device and method for acquiring information on a lane on a road on which a vehicle is driving.

In general, a vehicle refers to a transportation device that runs on roads or tracks using fossil fuels, electricity, etc. as a power source.

Vehicles have evolved to provide drivers with a variety of functions as technology has developed. In particular, with the trend toward electrification of vehicles, vehicles equipped with active safety systems (ASS: Active Safety Systems) that operate to prevent accidents immediately before or at the moment of an accident have emerged.

Furthermore, research is being actively conducted on vehicles equipped with advanced driver assistance systems (ADAS) that actively provide information on the driving environment, such as vehicle status, driver status, and surrounding environment, to reduce the burden on drivers and increase convenience.

Among advanced driver assistance systems, the Lane Departure Warning System (LDWS) and the Lane Keeping Assist System (LKAS) obtain lane information based on images of the vehicle's surroundings and control the vehicle's driving using the obtained driving lane information.

Korean Patent Publication, No. 10-2017-0097435 (Published on August 28, 2017)

The problem to be solved by the present invention is to provide a lane information acquisition device and method that generates a lane information acquisition model by learning a polynomial function fitting result of an effective lane determined based on a matching result of a vehicle's surrounding image and a lane in a precise map.

However, the problems to be solved by the present invention are not limited to those mentioned above, and other problems to be solved that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present invention pertains from the description below.

A method for obtaining lane information according to one embodiment of the present invention includes: a step of estimating a position of a vehicle based on an image of the surroundings of the vehicle; a step of matching a lane corresponding to the estimated position of the vehicle in a precise map to a lane in the image of the surroundings; a step of determining the matched lane as a valid lane using matching information of the lane; a step of fitting the determined valid lane; a step of obtaining polynomial function coefficients for the fitted valid lane as correct data of the valid lane used for learning for generating a model for obtaining lane information; and a step of labeling the polynomial function coefficients for the fitted valid lane in the image of the surroundings.

In addition, the step of determining the matched lane as the valid lane may determine the valid lane having the minimum average pixel coordinate error from candidate lanes in which the average pixel coordinate error between the lane corresponding to the estimated vehicle location in the precision map and the lane in the surrounding image among the matched lanes is less than or equal to a first threshold value.

In addition, the step of determining the matched lane as the valid lane may determine the valid lane in which the number of pixels in which the coordinate error between the lane corresponding to the estimated vehicle location in the precision map and the lane in the surrounding image is greater than or equal to a second threshold value is less than or equal to a third threshold value from the candidate lane with the minimum average pixel coordinate error.

In addition, the step of obtaining the polynomial function coefficients for the fitted valid lane as the correct answer data of the valid lane may obtain the polynomial function coefficients when the average fitting error for the fitted polynomial function is less than or equal to a fourth threshold value.

In addition, the step of obtaining the coefficients of the polynomial function for the fitted valid lane as the correct data of the valid lane may obtain the coefficients of the polynomial function in which the number of pixels having a fitting error greater than or equal to a fifth threshold value among the fitted polynomial functions in which the average fitting error is less than or equal to the fourth threshold value is less than or equal to a sixth threshold value.

In addition, the method further includes a step of learning the labeled valid lane to generate the lane information acquisition model, and the step of generating the lane information acquisition model can learn the valid lane so that an error between output data output by the lane information acquisition model into which the valid lane is input and the polynomial function coefficient labeled to the valid lane is reduced.

In addition, the step of labeling the polynomial function coefficients in the valid lanes may include labeling the polynomial function coefficients by merging at least one of the valid lanes in the surrounding image acquired at a first time point, the valid lanes in the surrounding image acquired at a second time point before the first time point, and the valid lanes in the surrounding image acquired at a third time point after the first time point.

In addition, the step of generating the lane information acquisition model can generate the lane information acquisition model by learning the location information of the vehicle together with the labeled valid lane.

In addition, the step of estimating the position of the vehicle may include the step of obtaining initial position information of the camera by matching a landmark of the precision map corresponding to the initial position information of the vehicle based on GPS to the surrounding image captured by the camera of the vehicle; the step of estimating the position of the camera based on matching information between the landmark of the precision map and the image corresponding to each of a plurality of candidate position information sampled based on the initial position information of the camera and the driving information of the vehicle; and the step of estimating the position of the vehicle based on the estimated position of the camera.

In addition, the method may further include a step of extracting the target lane from a surrounding image; and a step of inputting the extracted target lane into the lane information acquisition model to obtain the polynomial function coefficients.

A lane information acquisition method according to another embodiment of the present invention includes the steps of extracting a target lane from a target surrounding image of a vehicle; and the step of inputting the extracted target lane into a lane information acquisition model to acquire polynomial function coefficients as a fitting result of the target lane, wherein the lane information acquisition model is generated by learning a fitting result of an effective lane determined according to matching information between lanes in the surrounding image of the vehicle and lanes corresponding to the location of the vehicle in a detailed map.

According to one embodiment of the present invention, a lane information acquisition device includes a fitting unit which estimates a position of a vehicle based on an image of the surroundings of the vehicle, matches a lane corresponding to the estimated position of the vehicle in a precise map with a lane in the image of the surroundings, and fits an effective lane determined based on matching information of the lane to acquire polynomial function coefficients for the fitted effective lane; and a model generation unit which labels the effective lane with the polynomial function coefficients and generates a lane information acquisition model which learns the labeled effective lane and outputs polynomial function coefficients for a target lane.

According to an embodiment of the present invention, more accurate lane information can be obtained by excluding the user's experience or intuition when fitting lanes. In particular, since a lane information acquisition model is created and used through machine learning, the accuracy of the output lane information can be increased as the learning DB increases.

In addition, by using the detected driving lane information as an input value for the lane departure warning system and the lane keeping assistance system, more precise vehicle control may be possible.

Figures 1 and 2 are control block diagrams of a lane information acquisition system according to various embodiments.
FIG. 3 is a flowchart of a method for obtaining coefficients of a polynomial function for an effective lane among lane information obtaining methods according to one embodiment of the present invention.
FIG. 4 is a drawing for explaining the range of lanes extracted from a precision map and the range of lanes extracted from a surrounding image according to one embodiment of the present invention.
FIG. 5 is a drawing illustrating a lane extracted from a surrounding image according to one embodiment of the present invention.
Figure 6 is a diagram illustrating a case where a lane extracted from a precision map is matched to a surrounding image of Figure 5.
Figure 7 is a drawing for explaining the matching error according to the matching of Figure 6.
FIG. 8 is a drawing for explaining a method for determining a fitting range based on a distance from a vehicle according to one embodiment of the present invention.
FIG. 9 is a drawing for explaining a method for generating a lane information acquisition model among lane information acquisition methods according to one embodiment of the present invention.
FIG. 10 is a drawing for explaining a method for obtaining coefficients of a polynomial function for a target lane among lane information obtaining methods according to one embodiment of the present invention.

The advantages and features of the present invention, and the methods for achieving them, will become clear with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and these embodiments are provided only to make the disclosure of the present invention complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

In describing embodiments of the present invention, if it is judged that a detailed description of a known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of functions in the embodiments of the present invention, and these may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification.

Figures 1 and 2 are control block diagrams of a lane information acquisition system according to various embodiments.

Referring to FIG. 1, a lane information acquisition system (1) according to one embodiment may be composed of a vehicle (V) and a lane information acquisition device (100).

A vehicle (V) may refer to a means of transportation that can move people, objects, or animals from one location to another while running along a road or track. According to one embodiment, a vehicle (V) may include a three-wheeled or four-wheeled vehicle, a two-wheeled vehicle such as a motorcycle, construction equipment, a motor-driven bicycle, a bicycle, and a train running on a track.

The vehicle (V) of Fig. 1 is equipped with a GPS module and can receive satellite signals including navigation data from at least one GPS (Global Position System) satellite. Based on the satellite signals, the vehicle (V) can obtain the current location of the GPS-based vehicle (V) and the direction of travel of the vehicle (V). At this time, the GPS module can receive multiple satellite signals from each of multiple GPS satellites.

In addition, the vehicle (V) of FIG. 1 may store a general map and/or a precise map in advance. Here, the general map may mean a map including road information expressed by links and nodes. In addition, the precise map may mean a map including more information than the general map, has high accuracy for safe and precise vehicle (V) control, and includes information on not only the flat position of the road but also the elevation, slope, curvature, etc. In addition, the precise map may mean a map including more information on road facilities such as lanes, signs, traffic lights, and guardrails.

In addition, the vehicle (V) of Fig. 1 may be equipped with an Advanced Driver Assistance System (ADAS). Here, the advanced driver assistance system may mean a system that provides driving environment information such as vehicle (V) status, driver status, and surrounding environment information, or actively controls the vehicle (V) to reduce the burden on the driver and increase convenience.

For example, the vehicle (V) may be equipped with at least one of a Lane Departure Warning System (LDWS) and a Lane Keeping Assist System (LKAS). However, the advanced driver assistance system equipped in the vehicle (V) is not limited to the examples described above.

An advanced driver assistance system mounted on a vehicle (V) may include a detection means for detecting a driving environment of the vehicle (V). According to one embodiment, the detection means may include a radar that irradiates a pulse around the vehicle (V) and receives an echo pulse reflected from an object located in a corresponding direction to detect the driving environment, a LiDAR that irradiates a laser around the vehicle (V) and receives a laser reflected from an object located in a corresponding direction, and/or an ultrasonic sensor that irradiates an ultrasonic wave around the vehicle (V) and receives an echo ultrasonic wave reflected from an object located in a corresponding direction.

In addition, the advanced driver assistance system may include a camera as a sensing means. The camera may be installed to face the front, side, and/or rear of the vehicle (V) and may capture surrounding images in the corresponding direction. The captured surrounding images may be used as a basis for obtaining information on objects around the vehicle (V) as well as lanes and signs through an image processing process.

Meanwhile, the above-described lane departure warning system and/or lane keeping assistance system can control the driving of the vehicle (V) based on lane information around the vehicle. At this time, the lane information used for driving control of the vehicle (V) can be obtained from an image of the vehicle (V)'s surroundings captured by a camera mounted on the vehicle (V).

One such method is to detect lanes from the front image using machine learning such as deep learning, assume the area around the vehicle as flat ground, convert the lane coordinates to the vehicle coordinate system, and calculate the polynomial function coefficients for the fitted lanes. At this time, the lane results detected from multiple consecutive images can be merged to increase the accuracy of the polynomial function coefficients.

However, since the body shakes depending on the ground condition and the camera attitude angle changes in real time, there is the inconvenience of having to perform a complex process such as online camera calibration for each continuous image.

In addition, in order to create a lane detection model through learning of surrounding images, it is necessary to create correct answer data that is labeled in the learning data. This correct answer data can be used by a person to mark the lane area by directly checking the surrounding images with the naked eye. In this case, the accuracy and reliability of the acquired lane information may be reduced.

To solve this problem, a lane information acquisition system (1) according to one embodiment of the present invention can generate a lane information acquisition model by learning the polynomial function fitting result of an effective lane determined based on the matching result of the surrounding image of the vehicle (V) and the lane in the precision map.

Specifically, the lane information acquisition device (100) can acquire lane information using information received from the vehicle (V). Here, the lane information means the result of fitting a lane in a surrounding image with a polynomial function, and may include coefficients of the fitted polynomial function.

In order to obtain lane information, the lane information obtaining device (100) can exchange information by communicating with the vehicle (V) using various known communication methods. The estimated position obtaining device (100) of the vehicle (V) according to one embodiment can communicate with the vehicle (V) via a base station by adopting a known communication method such as CDMA, GSM, W-CDMA, TD-SCDMA, WiBro, LTE, EPC, etc. In contrast, the estimated position obtaining device (100) of the vehicle (V) according to another embodiment can communicate with the vehicle (V) within a predetermined distance by adopting a communication method such as Wireless LAN, Wi-Fi, Bluetooth, Zigbee, WFD (Wi-Fi Direct), UWB (Ultra wideband), Infrared Data Association (IrDA), BLE (Bluetooth Low Energy), NFC (Near Field Communication). However, the method by which the lane information acquisition device (100) communicates with the vehicle (V) is not limited to the above-described embodiment.

A lane information acquisition device (100) can match lanes in a surrounding image of a vehicle (V) with lanes in a detailed map, fit a polynomial function to an effective lane determined based on the matching result, and then learn the fitted polynomial function coefficients to generate a lane information acquisition model. In addition, the lane information acquisition device (100) can input a target lane extracted from a surrounding image into the generated lane information acquisition model to obtain polynomial function coefficients for the target lane.

To this end, the lane information acquisition device (100) may include a learning unit (110) and an inference unit (120).

The learning unit (110) can generate a lane information acquisition model by learning the polynomial function fitting result of the valid lane determined based on the matching result of the surrounding image of the vehicle (V) and the lane in the precision map. Referring to Fig. 1, the learning unit (110) can include a fitting unit (111) and a model generation unit (112).

The fitting unit (111) can obtain polynomial function coefficients from the polynomial function fitting result of the valid lane determined based on the matching result of the surrounding image of the vehicle (V) and the lane in the precision map. Specifically, the fitting unit (111) estimates the position of the vehicle (V) based on the surrounding image of the vehicle (V), matches the lane corresponding to the estimated position of the vehicle (V) in the precision map to the lane in the surrounding image, and fits the determined valid lane based on the matching information of the lane, thereby obtaining polynomial function coefficients for the fitted valid lane.

The model generation unit (112) can learn the fitting result to generate a lane information acquisition model. It can label polynomial function coefficients for valid lanes, learn the labeled valid lanes, and generate a lane information acquisition model that outputs polynomial function coefficients for target lanes.

The inference unit (120) can input a target lane extracted from a surrounding image into the generated lane information acquisition model to obtain a polynomial function coefficient for the target lane. Referring to Fig. 1, the inference unit (120) can include a lane extraction unit (121) and a coefficient acquisition unit (122).

The lane extraction unit (121) can extract a target lane from a surrounding image input from the outside. In addition, the coefficient acquisition unit (122) can input the extracted target lane into a lane information acquisition model to obtain a polynomial function coefficient.

Each component of the lane information acquisition device (100) according to the above-described embodiment may be implemented as a computational device including a microprocessor, and may be implemented as, for example, a central processing unit (CPU), a graphic processing unit (GPU), etc. Alternatively, it may be possible for a plurality of components constituting the lane information acquisition device (100) to be implemented as a single SOC (System On Chip).

Meanwhile, in FIG. 1, a case is exemplified where a lane information acquisition device (100) is installed separately from a vehicle (V) to configure a lane information acquisition system (1), but, alternatively, it may also be possible for the lane information acquisition device (100) to be installed inside the vehicle (V).

Referring to FIG. 2, a lane information acquisition system (1) according to another embodiment may be configured with a vehicle (V) including a lane information acquisition device (100). However, except for the method in which the lane information acquisition device (100) is provided, the operation method of the lane information acquisition system (1) of FIG. 2 and the lane information acquisition system (1) of the vehicle (V) of FIG. 2 is the same.

Hereinafter, a lane information acquisition method according to an embodiment of the present invention performed by the lane information acquisition device (100) described above will be described with reference to FIGS. 3 to 10. First, a method for obtaining coefficients of a polynomial function for an effective lane performed by the fitting unit (111) described above will be described with reference to FIGS. 3 to 8, and a method for generating a lane information acquisition model performed by the learning unit (110) described above will be described with reference to FIG. 9. Then, a method for obtaining coefficients of a polynomial function for a target lane performed by the inference unit (120) described above will be described with reference to FIG. 10.

FIG. 3 is a flowchart of a method for obtaining coefficients of a polynomial function for an effective lane among lane information obtaining methods according to one embodiment of the present invention, FIG. 4 is a diagram for explaining a range of lanes extracted from a precision map and a range of lanes extracted from a surrounding image according to one embodiment of the present invention, FIG. 5 is a diagram exemplifying lanes extracted from a surrounding image according to one embodiment of the present invention, FIG. 6 is a diagram exemplifying a case of matching a lane extracted from a precision map to the surrounding image of FIG. 5, FIG. 7 is a diagram for explaining a matching error according to the matching of FIG. 6, and FIG. 8 is a diagram for explaining a method for determining a fitting range based on a distance from a vehicle according to one embodiment of the present invention.

As described above, the fitting unit (111) of the lane information acquisition device (100) according to one embodiment can acquire polynomial function coefficients according to the fitting of effective lanes in order to generate a lane information acquisition model.

To this end, the fitting unit (111) can estimate the location of the vehicle (V) based on satellite signals and surrounding images (S100). As an example for estimating the location of the vehicle (V), the fitting unit (111) can first match the landmark of the precise map corresponding to the initial location information of the vehicle (V) based on GPS to the surrounding images captured by the camera of the vehicle (V) to obtain the initial location information of the camera.

In order to obtain the initial position information of the camera, the fitting unit (111) can obtain the initial attitude angle of the vehicle (V) by using the initial position of the GPS-based vehicle (V) received from the vehicle (V). Then, the fitting unit (111) can determine a first region of interest based on the initial position information of the GPS-based vehicle (V) in the precision map. Once the first region of interest is determined, the fitting unit (111) can match a first landmark for a lane existing in the first region of interest to an image captured by the camera to obtain a rotation matrix R for the initial attitude angle of the camera. In addition, the fitting unit (111) can determine a second region of interest based on the initial position information of the GPS-based vehicle (V) in the precision map, and match a second landmark other than a lane existing in the second region of interest to an image based on the initial attitude angle of the camera to obtain a translation matrix T for the initial position of the camera.

After obtaining the initial position information of the camera, the fitting unit (111) can obtain the estimated position information of the camera by using the initial position of the camera as an input value. To this end, the fitting unit (111) can sample a plurality of candidate position information around the initial position information of the camera, and obtain the estimated position information of the camera by using a particle filter. Then, the fitting unit (111) can assign a weight to each of the plurality of candidate position information based on a matching error between a landmark of a precision map and an image corresponding to each of the plurality of candidate position information. After assigning the weight, the fitting unit (111) newly samples the plurality of candidate position information by using the plurality of candidate position information to which the weight has been assigned, and if the standard deviation of the newly sampled plurality of candidate position information is less than or equal to a reference standard deviation, the average value of the newly sampled plurality of candidate position information can be acquired as the estimated position information of the camera.

When the estimated position information of the camera is acquired, the fitting unit (111) can verify the validity of the estimated position information of the camera, and then acquire the estimated position information of the vehicle (V) based on the valid estimated position information of the camera. To this end, the fitting unit (111) can first confirm a second landmark other than a lane among the landmarks of the precise map corresponding to the estimated position information of the camera, and acquire a weight of the estimated position information of the camera corresponding to the matching error of each of the second landmarks. When the weight corresponding to the matching error of each of the second landmarks is acquired, the fitting unit (111) can determine whether the number of weights greater than a reference value is greater than a reference number, and determine the estimated position information of the camera to be valid. Through this, the fitting unit (111) can acquire the estimated position information of the vehicle (V) using the estimated position information of the camera.

In Fig. 3, a case in which the position of a vehicle (V) is estimated based on a satellite signal and surrounding images is exemplified, but a lane information acquisition method according to another embodiment can also estimate the position information of the current vehicle (V) on a high-precision map using RTK/INS equipment.

When the position of the vehicle (V) is estimated based on the estimated position information of the vehicle (V), the fitting unit (111) can extract a lane corresponding to the estimated position of the vehicle (V) from a precise map (S110). Referring to FIG. 4, the fitting unit (111) can set a first region of interest S M including a rear _bumper of the vehicle (V) and extending from the front wheel of the vehicle (V) to the front of the vehicle (V), that is, to a predetermined maximum threshold distance in the longitudinal direction, and extract pixel coordinates of a lane within the first region of interest S _M.

Next, the fitting unit (111) can fit the lane by matching the extracted lane to the surrounding image (S120). Prior to this, the fitting unit (111) can extract the lane in the surrounding image captured by the camera mounted on the vehicle (V). According to one embodiment, the fitting unit (111) can extract the lane from the surrounding image by using a lane extraction model generated by learning learning data in which the location of the lane area is labeled in the learning image. Fig. 5 illustrates four lanes extracted from the surrounding image by the fitting unit (111).

Referring again to FIG. 4, the camera mounted on the vehicle (V) obtains a peripheral image for the second region of interest S _C formed by the angle of view, and the fitting unit (111) can extract pixel coordinates of a lane within the second region of interest S _C from the peripheral image. At this time, since the first region of interest S _M and the second region of interest S _C have a partially overlapping area, the same lane may be extracted from each of the precision map and the peripheral image.

Accordingly, the fitting unit (111) can match the lane extracted from the precision map to the lane of the surrounding image. Specifically, the fitting unit (111) can perform lane matching by projecting the pixel coordinates of the lane extracted from the precision map onto the surrounding image according to mathematical expression 1.

Here, m is a pixel coordinate value of a lane projected onto a surrounding image, K is a camera internal parameter matrix, R is a rotation matrix that transforms a 3D pixel coordinate of a lane on a precision map into a camera coordinate system, and T is a translation matrix that transforms a 3D pixel coordinate of a lane on a precision map into a camera coordinate system. In addition, R and T can be obtained based on the position estimation result of the vehicle (V) performed in step S100 and the rotation/translation transformation matrix between the camera-vehicle (V) coordinate systems.

In Fig. 6, a case is exemplified where multiple lanes extracted from a precision map are matched to four lanes extracted from a surrounding image. At this time, it can be confirmed that an error occurs in the matching result between lane number 4 in the surrounding image and the lane in the precision map. If a matching error exists, the fitting result for the matched lane may also be inaccurate, so the fitting unit (111) can determine the validity of the matched lane prior to fitting.

Specifically, the fitting unit (111) according to one embodiment can determine a valid lane by checking whether the average pixel coordinate error between the lane corresponding to the estimated location of the vehicle (V) in the precision map among the matched lanes and the lanes in the surrounding image is less than or equal to a first threshold value. Referring to FIG. 7, the fitting unit (111) can check the matching error d _e between any one pixel P _M of the lane extracted from the precision map and the corresponding pixel P _I of the lane extracted from the surrounding image.

Next, the fitting unit (111) can check the matching error of each of the remaining pixels of the lane extracted from the precision map and the corresponding pixels of the lane extracted from the surrounding image, and then obtain an average pixel coordinate error, which is an average of them. After obtaining the average pixel coordinate error, the fitting unit (111) can check whether the obtained average pixel coordinate error is less than or equal to a first threshold value, and determine a matched lane whose average pixel coordinate error is less than or equal to the first threshold value as a valid lane. Here, the first threshold value can mean the maximum value of the average pixel coordinate error for the matched lane that can be determined as a valid lane.

In addition to the above-described process, the fitting unit (111) according to another embodiment can check the number of pixels in which the pixel coordinate error between the lane extracted from the precision map and the lane extracted from the surrounding image is greater than or equal to a second threshold value for each candidate lane with the minimum average pixel coordinate error. Here, the second threshold value can mean the minimum value of the pixel coordinate error for the matched lane that can determine the pixel coordinate of the lane as invalid.

Next, the fitting unit (111) may determine a matching lane in which the number of pixels having a pixel coordinate error greater than or equal to a second threshold value is less than or equal to a third threshold value as a valid lane. Here, the third threshold value may mean the maximum number of invalid pixels for a matched lane that can be determined as a valid lane.

When a valid lane is determined through the above-described process, the fitting unit (111) can fit the valid lane (S120). Here, the valid lane may mean a lane extracted from a precision map among matching lanes judged to be valid. Since the polynomial function coefficients obtained from the lane fitting result are labeled as correct answer data in machine learning such as deep learning, the fitting unit (111) can select a lane extracted from a precision map with higher accuracy than a lane extracted from a surrounding image as a valid lane.

Specifically, the fitting unit (111) can perform curve fitting after converting each pixel coordinate constituting the valid lane into a vehicle (V) coordinate system. Specifically, the fitting unit (111) can fit the valid lane according to mathematical expression 2.

After fitting the valid lane with a polynomial function according to mathematical expression 2, the fitting unit (111) can obtain coefficients of the polynomial function for the fitted valid lane (S130). Specifically, the fitting unit (111) can obtain coefficients a, b, c, and d of mathematical expression 2. With respect to the obtained coefficients, 6a may denote a curvature differential value of the lane, 2b may denote curvature, arctan(c) may denote a direction value, and d may denote an offset value. As a result, the obtained polynomial function coefficients may be used as input values for a lane departure warning system (1), a lane keeping assistance system (1), and the like.

In addition, the fitting unit (111) according to one embodiment may determine the validity of the fitted polynomial function and obtain coefficients only for valid polynomial functions. As described in FIG. 4, the fitting unit (111) may set a region of interest from the front wheel of the vehicle (V) to the front of the vehicle (V), i.e., a predetermined maximum threshold distance in the longitudinal direction, and extract lane coordinates within the region of interest. The fitting unit (111) may match the extracted lane coordinates to lanes within the surrounding image.

At this time, the fitting part (111) may have an increasing matching error as the pixel of the lane is farther from the vehicle (V). Fig. 8 illustrates a case where the matching error of the pixel coordinates having the hatched area among the pixel coordinates of the matched lanes arranged along the lane L _f within the maximum critical distance d ₁ is large.

In consideration of this, the fitting unit (111) can first perform polynomial function fitting based on pixels of the lane within the maximum threshold distance d ₁ , and then check the error of the output value of the polynomial function from each pixel coordinate of the actual lane. The fitting unit (111) can determine that the fitted polynomial function is valid if the average error for each pixel coordinate of the lane is less than or equal to the fourth threshold value.

On the other hand, if the average error for each pixel coordinate of the lane is greater than the fourth threshold value, the fitting unit (111) can re-fit only for pixels of the lane within a range less than the maximum threshold distance, and determine the validity of the fitting result. Here, the fourth threshold value can mean the maximum value of the average error for pixels of the actual lane that can determine the fitted polynomial function as valid.

Figure 8 illustrates an example of performing polynomial function fitting by limiting the pixel range of the lane used for fitting from the maximum critical distance d ₁ to d ₂ .

In addition to the above-described process, the fitting unit (111) according to another embodiment may determine as valid a polynomial function in which the number of pixels in which the error for each pixel coordinate of each lane is greater than or equal to a fifth threshold value is less than or equal to a sixth threshold value, among candidate polynomial functions in which the average error for each pixel coordinate of each lane is less than or equal to a fourth threshold value. Here, the fifth threshold value may mean the minimum value of the error for each pixel of the actual lane that can determine the polynomial function as invalid, and the sixth threshold value may mean the maximum value of the number of invalid pixels for each pixel of the actual lane that can determine the polynomial function as valid.

If the fitted polynomial function is determined to be valid according to the above-described process, the fitting unit (111) can obtain the coefficients of the polynomial function.

So far, a method for obtaining coefficients of a polynomial function for a valid lane performed by the fitting unit (111) has been described. Hereinafter, a method for generating a lane information acquisition model performed by the learning unit (110) described above will be described with reference to FIG. 9.

FIG. 9 is a drawing for explaining a method for generating a lane information acquisition model among lane information acquisition methods according to one embodiment of the present invention.

As described above, the learning unit (110) of the lane information acquisition device (100) according to one embodiment can learn an effective lane based on the polynomial function coefficients according to the fitting acquired by the fitting unit (111) to generate a lane information acquisition model.

To this end, the learning unit (110) can extract the same valid lane from the surrounding image acquired at the first point in time, the surrounding image acquired at the second point in time before the first point in time, and the surrounding image acquired at the third point in time after the first point in time (S200). Specifically, the learning unit (110) according to one embodiment can extract the valid lane from the surrounding image acquired at the first point in time, extract the same valid lane as the previously extracted valid lane from the surrounding image acquired at the second point in time before the first point in time, extract the same valid lane as the previously extracted valid lane from the surrounding image acquired at the third point in time after the first point in time, and then merge the valid lanes extracted from the surrounding images acquired at different points in time.

In Fig. 9, a case is exemplified where a valid lane corresponding to a first point in time is merged with a valid lane corresponding to a second point in time and a valid lane corresponding to a third point in time, but it may also be possible to merge a valid lane corresponding to a first point in time with at least one of a valid lane corresponding to a second point in time and a valid lane corresponding to a third point in time.

Next, the learning unit (110) can label the extracted valid lanes with polynomial function coefficients (S210). Specifically, the learning unit (110) can label the extracted and merged valid lanes with polynomial function coefficients obtained according to the fitting of the corresponding valid lanes. As described above, the polynomial function coefficients obtained according to the fitting of the valid lanes are reliable as values obtained from a precise map, so the learning unit (110) can label the polynomial function coefficients as correct data to the valid lanes.

If the learning unit (110) follows supervised learning, the learning unit (110) can label polynomial function coefficients for all extracted valid lanes. On the other hand, if the learning unit (110) follows semisupervised learning, the learning unit (110) can label polynomial function coefficients for only some of the extracted valid lanes, and learn by using both labeled and unlabeled valid lanes.

Finally, the learning unit (110) can learn the labeled valid lanes to generate a lane information acquisition model (S220). Here, the lane information acquisition model means a model that outputs polynomial function coefficients according to polynomial function fitting of a target lane in a surrounding image when the target lane is input, and the lane information acquisition model according to one embodiment may be a neural network-based model. The neural network-based lane information acquisition model may be implemented as a neural network model such as a CNN (Convolution Neural Network), a DNN (Deep Neural Network), an RNN (Recurrent Neural Network), or a BRDNN (Bidirectional Recurrent Deep Neural Network).

A lane information acquisition model according to one embodiment may be composed of an input layer into which a valid lane is input, an output layer into which a polynomial function coefficient corresponding to the input valid lane is output, and a plurality of hidden layers provided between the input layer and the output layer. The output layer may form a fully connected network with the last hidden layer among the plurality of hidden layers. At this time, the output layer of the artificial neural network may be composed of nodes for coefficients of an Nth-order polynomial function, and each node may mean a respective coefficient in the polynomial function coefficients.

If the lane information acquisition model is implemented as a CNN model, multiple hidden layers can be composed of multiple convolution layers and subsampling layers. The convolution layer can obtain a feature vector for the effective lane by applying various convolution kernels to the input effective lane. In this way, the convolution can play a role like a template that extracts features for high-dimensional effective lanes. The subsampling layer is a neuron layer that reduces the dimension of the obtained feature vector, and can lower the spatial resolution for the feature vector generated by the convolution layer. Through this process, the lane information acquisition model can reduce the complexity of the problem. The convolution layer and the subsampling layer are repeatedly arranged in the hidden layer, so that the above-described process can be repeatedly performed. The learning unit (110) can readjust the parameters of the artificial neural network using back propagation in a direction that minimizes the error between the output and the correct answer, and repeatedly train the artificial neural network using the above-described input until the output converges to the correct answer.

For example, the learning unit (110) according to one embodiment can train an artificial neural network by using as input data the i-th valid lane detected in the k-th surrounding image and the valid lane in the surrounding image before or after the k-th surrounding image that is identical to the i-th valid lane. In addition, the learning unit (110) according to another embodiment can train an artificial neural network by adding position/position information of a vehicle corresponding to the surrounding image before or after the k-th surrounding image based on the k-th vehicle coordinate system to the above-described input data.

In addition, the learning unit (110) according to another embodiment may train the artificial neural network so that the output data output by the artificial neural network includes the polynomial function coefficient for the ith valid lane detected in the kth surrounding image. In addition, the learning unit (110) according to another embodiment may train the artificial neural network so that, in addition to the output data described above, the artificial neural network outputs the polynomial function coefficient for the valid lane detected in the surrounding image before or after the kth surrounding image.

At this time, the learning unit (110) can obtain polynomial function coefficients that minimize loss for the loss function of mathematical expression 3.

Here, a _g , b _g , c _g , and d _g represent correct answer data for polynomial function coefficients, and a, b, c, and d may represent output data.

According to one embodiment, the learning unit (110) can learn the valid lane by inputting the valid lane and corresponding the polynomial function coefficient according to the fitting result of the valid lane, according to the supervised learning method. That is, the learning unit (110) can learn the relationship between the input valid lane and the correct polynomial function coefficient, thereby generating a lane information acquisition model that outputs the polynomial function coefficient according to the fitting result of the input valid lane.

In contrast, the learning unit (110) according to another embodiment learns by using some labeled valid lanes and the remaining unlabeled valid lanes together.

A lane information acquisition model may be generated by semi-supervised learning. Alternatively, a lane information generation model may be generated by reinforcement learning that utilizes feedback on whether the learning result of the learning unit (110) according to another embodiment is correct.

After learning is completed, the learning unit (110) uses the inference process to input data in the same format as the data used in the learning process into the neural network.

In addition, the learning unit (110) may learn the location information of the estimated vehicle (V) as input in addition to the valid lane. That is, the learning unit (110) may generate a lane information acquisition model that takes the location information of the estimated vehicle (V) and the valid lane corresponding to the location information as input, and outputs the polynomial function coefficients according to the fitting result of the corresponding valid lane.

In addition, when a valid lane corresponding to a first time point is merged with at least one of a valid lane corresponding to a second time point and a valid lane corresponding to a third time point, the learning unit (110) may learn the valid lane so that the lane information acquisition model can output polynomial function coefficients for the valid lane corresponding to at least one of the second time point and the third time point as well as the first time point. For example, when a valid lane corresponding to a first time point is merged with a valid lane corresponding to a second time point, the learning unit (110) may learn the merged valid lane and generate a lane information acquisition model that outputs polynomial function coefficients for the valid lane corresponding to the first time point and polynomial function coefficients for the valid lane corresponding to the second time point. In contrast, when a valid lane corresponding to a first time point is merged with a valid lane corresponding to a third time point, the learning unit (110) may learn the merged valid lane and generate a lane information acquisition model that outputs polynomial function coefficients for the valid lane corresponding to the first time point and polynomial function coefficients for the valid lane corresponding to the third time point.

So far, a method for generating a lane information acquisition model performed by the learning unit (110) has been described. Hereinafter, a method for obtaining coefficients of a polynomial function for a target lane performed by the above-described inference unit (120) will be described with reference to FIG. 10.

FIG. 10 is a drawing for explaining a method for obtaining coefficients of a polynomial function for a target lane among lane information obtaining methods according to one embodiment of the present invention.

As described above, the inference unit (120) of the lane information acquisition device (100) can input the target lane extracted from the surrounding image into the generated lane information acquisition model to obtain polynomial function coefficients for the target lane.

To this end, the inference unit (120) can first obtain a surrounding image of the vehicle (V) (S300). If the lane information acquisition device (100) is provided separately from the vehicle (V), the inference unit (120) can obtain a surrounding image of the vehicle (V) received by the communication means of the lane information acquisition device (100). In contrast, if the lane information acquisition device (100) is provided integrally with the vehicle (V), the inference unit (120) can obtain a surrounding image directly from the camera of the vehicle (V) or obtain a surrounding image obtained by the camera by the internal communication means of the vehicle (V).

After acquiring the surrounding image, the inference unit (120) can extract a target lane from the acquired surrounding image (S310). Here, the target lane may mean a lane for which information is to be acquired in the surrounding image, and according to one embodiment, there may be one or more target lanes.

Specifically, the lane extraction unit (121) of the inference unit (120) can extract a target lane from a surrounding image by using a lane extraction model generated by learning learning data in which the location of the lane area is labeled in the learning image. At this time, the inference unit (120) according to one embodiment can use a lane extraction model separate from the fitting unit (111). In contrast, the inference unit (120) according to another embodiment can share a lane extraction model with the fitting unit (111), and can further be implemented as an integral part with the fitting unit (111).

Finally, the inference unit (120) can input the target lane into the lane information acquisition model to obtain polynomial function coefficients for the target lane (S320). Specifically, the coefficient acquisition unit (122) of the inference unit (120) can input the target lane extracted from the surrounding image into the lane information acquisition model generated by the learning unit (110) and obtain polynomial function coefficients output from the lane information acquisition model.

If the learning unit (110) learns not only the valid lane but also the location information of the estimated vehicle (V) to generate a lane information acquisition model, the coefficient acquisition unit (122) can input the location information of the current vehicle (V) along with the target lane into the lane information acquisition model to acquire polynomial function coefficients for the target lane. The polynomial function coefficients acquired in this way include information about the curvature differential value of the target lane, the curvature of the target lane, the direction value of the target lane, and the offset value, and thus can be used as input values for a lane departure warning system (1), a lane keeping assistance system (1), etc. in the future.

The lane information acquisition device and method according to the above-described embodiment can acquire more accurate lane information by excluding the user's experience or intuition when fitting the lane. In particular, since a lane information acquisition model is created and used through machine learning, the accuracy of the output lane information can be increased as the learning DB increases.

In addition, by using the detected driving lane information as an input value for the lane departure warning system and the lane keeping assistance system, more precise vehicle control may be possible.

Meanwhile, each step included in the lane information acquisition method according to the above-described embodiment can be implemented in a computer-readable recording medium that records a computer program programmed to perform these steps.

Additionally, each step included in the lane information acquisition method according to the above-described embodiment may be implemented as a computer program programmed to perform these steps.

The above description is merely an illustrative description of the technical idea of the present invention, and those skilled in the art will appreciate that various modifications and variations may be made without departing from the essential quality of the present invention. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention but to explain it, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the rights of the present invention.

According to one embodiment, the lane information acquisition device and method described above can be used in various fields such as at home or at industrial sites, and thus has industrial applicability.

1: Lane information acquisition system
100: Lane information acquisition device
V: vehicle

Claims (13)

A step of estimating the location of a vehicle based on an image of the surroundings of the vehicle;
A step of matching a lane corresponding to the estimated location of the vehicle within the precision map to a lane within the surrounding image;
A step of determining the matched lane as a valid lane by using the matching information of the lane;
A step of fitting the determined valid lane;
A step of obtaining polynomial function coefficients for the fitted valid lane as correct data of the valid lane used for learning to generate a lane information acquisition model that outputs polynomial function coefficients for the target lane by inputting a target lane extracted from an image surrounding the vehicle; and
A step of labeling the polynomial function coefficients in the fitted valid lanes within the surrounding image.
How to obtain lane information. In paragraph 1,
The step of determining the above-mentioned matched lane as the valid lane is:
Among the matched lanes, the candidate lanes in which the average pixel coordinate error between the lane corresponding to the estimated vehicle position in the precision map and the lanes in the surrounding image is less than or equal to a first threshold value are determined as the valid lane having the minimum average pixel coordinate error.
How to obtain lane information. In the second paragraph,
The step of determining the above-mentioned matched lane as the valid lane is:
The valid lane is determined by determining the number of pixels in which the coordinate error between the lane corresponding to the estimated vehicle position in the precision map and the lane in the surrounding image is greater than or equal to a second threshold value from the candidate lane with the minimum average pixel coordinate error and is less than or equal to a third threshold value.
How to obtain lane information. In paragraph 1,
The step of obtaining the polynomial function coefficients for the fitted valid lane as the correct answer data of the above valid lane is:
If the average fitting error for the fitted polynomial function is less than or equal to the fourth threshold value, the polynomial function coefficient is obtained.
How to obtain lane information. In paragraph 4,
The step of obtaining the polynomial function coefficients for the fitted valid lane as the correct answer data of the above valid lane is:
Obtain coefficients of the polynomial function in which the number of pixels having a fitting error greater than or equal to a fifth threshold among the fitted polynomial functions in which the average fitting error is less than or equal to the fourth threshold is less than or equal to a sixth threshold.
How to obtain lane information. In paragraph 1,
Further comprising a step of learning the labeled valid lane and generating the lane information acquisition model,
The step of generating the above lane information acquisition model is:
The valid lane is learned so that the error between the output data output by the lane information acquisition model into which the valid lane is input and the polynomial function coefficient labeled to the valid lane is reduced.
How to obtain lane information. In paragraph 1,
The step of labeling the polynomial function coefficients in the above valid lanes is:
Labeling the polynomial function coefficients by merging at least one of the valid lanes in the surrounding image acquired at a first time point, the valid lanes in the surrounding image acquired at a second time point before the first time point, and the valid lanes in the surrounding image acquired at a third time point after the first time point.
How to obtain lane information. In paragraph 1,
The step of generating the above lane information acquisition model is:
The lane information acquisition model is created by learning the location information of the vehicle along with the labeled valid lane.
How to obtain lane information. In paragraph 1,
The step of estimating the location of the above vehicle is:
A step of obtaining initial location information of the camera by matching a landmark of the precision map corresponding to the initial location information of the vehicle based on GPS to the surrounding image captured by the camera of the vehicle;
A step of estimating the position of the camera based on matching information between the landmark of the precision map and the image corresponding to each of a plurality of candidate position information sampled based on the initial position information of the camera and the driving information of the vehicle; and
A step of estimating the position of the vehicle based on the position of the estimated camera.
How to obtain lane information. In paragraph 1,
Further comprising a step of generating learning data of the lane information acquisition model based on the labeled valid lane.
How to obtain lane information. A step of extracting a target lane from an image surrounding a target of a vehicle; and
It includes a step of inputting the extracted target lane into a lane information acquisition model to obtain polynomial function coefficients as a fitting result of the target lane.
The above lane information acquisition model is,
The correct answer data is generated by learning the polynomial function coefficients obtained by fitting the valid lanes determined based on the matching information between the lanes in the surrounding images of the vehicle and the lanes corresponding to the location of the vehicle in the detailed map.
How to obtain lane information. A fitting unit that estimates the location of the vehicle based on an image of the surroundings of the vehicle, matches a lane corresponding to the estimated location of the vehicle in a precision map to a lane in the image of the surroundings, and fits an effective lane determined based on the matching information of the lane, and obtains polynomial function coefficients for the fitted effective lane as correct data of the effective lane used for learning to generate a lane information acquisition model that outputs polynomial function coefficients for the target lane when a target lane extracted from the image of the surroundings of the vehicle is input; and
A model generation unit is included that labels the polynomial function coefficients on the above valid lanes and generates a lane information acquisition model that learns the labeled valid lanes and outputs polynomial function coefficients for the target lane.
Lane information acquisition device. A program stored on a computer-readable recording medium programmed to perform each step according to any one of the methods described in claims 1 to 11.