JP4584120B2 - Road marking line detection device, road marking line detection method, road marking line detection program - Google Patents

Description

  The present invention relates to a road marking line detection device that detects a marking line on a road based on an image photographed by a photographing means.

  2. Description of the Related Art Conventionally, there is known an image edge detection device that obtains the directionality and intensity of a white line edge using a plurality of mask filters having directivity for image data input from an imaging device (for example, a patent) Reference 1).

  Also, a vehicle that measures the edge point count maximum value of an edge component representing a straight line component, recognizes it as a broken line lanemer when the maximum value changes periodically, and recognizes it as a solid line lane marker when it is a constant constant value A traveling lane recognition device is known (see, for example, Patent Document 2).

Furthermore, a vehicular travel lane detection device that selects only those recognized as white lines from the detected white line candidate points and calculates and stores the detection rate of the selected white line candidate points is known ( For example, see Patent Document 3). The apparatus determines the type of white line based on the stored white line candidate point detection rate.
Japanese Patent Laid-Open No. 5-298446 JP-A-8-320997 JP 2001-14595 A

  By the way, in addition to the white line, a three-dimensional botsdot, cat's eye, or the like may be used as a lane marking that divides the road. In these three-dimensional botsdots and cat's eyes, the shape of the photographed image when photographed by a camera or the like changes depending on the illumination environment such as how the light hits the sun.

  However, in the conventional techniques shown in Patent Documents 1 to 3, edge detection is mainly performed by a line filter, and the shape on a captured image changes depending on an illumination environment such as a three-dimensional botsdot or cat's eye. In consideration of the above, no lane marking is detected. Therefore, there is a possibility that the detection accuracy of the lane marking on the road is lowered.

  The present invention has been made to solve such problems, and has as its main object to detect a marking line on a road with high accuracy.

In order to achieve the above object, one embodiment of the present invention provides:
A road lane marking detection device that detects a predetermined lane marking on a road based on a photographed image photographed by a photographing means,
A structural element setting means for setting a structural element in a morphological operation based on the shape of the predetermined lane marking;
Based on the structural element set by the structural element setting means, a contracted image is generated on the captured image captured by the capturing means to generate a contracted image, and the generated contracted image is expanded. Opening processing means for performing processing to generate an Opening image;
Top hat conversion processing means for generating a difference image obtained by subtracting the Opening image generated by the Opening processing means from the captured image captured by the imaging means;
It is provided with the road lane marking detection apparatus characterized by the above-mentioned.

  According to this aspect, the opening processing means performs a contraction process and an expansion process on the captured image using a structural element set based on the shape of a predetermined partition line, and generates an opening image. Further, the top hat conversion means generates a difference image based on the generated opening image. Thereby, the lane marking on the road can be detected with high accuracy.

  In this aspect, it is preferable that the image processing apparatus further includes a noise removing unit that removes noise from the difference image generated by the top hat conversion processing unit. By removing noise from the difference image generated by the top hat conversion processing means, it is possible to detect a lane marking on the road with higher accuracy. Note that the noise removing unit removes noise by performing the shrinkage process and the expansion process on the difference image, for example.

  Furthermore, in this aspect, it is preferable that the structural element setting unit sets the structural element according to a shape of the predetermined partition line on the captured image captured by the imaging unit. For example, the structural element setting means sets the structural element so that a predetermined partition line is erased during the opening process by the opening processing means.

  In this aspect, the predetermined lane marking is, for example, a bots dot, a cat's eye, or a white line.

  In this aspect, the vehicle may further include travel support means for performing travel support of the vehicle based on the predetermined lane marking. Safety of the vehicle is improved by performing driving support of the vehicle based on a predetermined lane marking detected with high accuracy.

  In this one aspect, the driving support means includes, for example, lane keeping control for causing the vehicle to travel in the lane line, or lane departure warning control for warning the user when the vehicle departs from the lane line. Do.

Note that one embodiment of the present invention for achieving the above object is as follows.
A road lane marking detection method for detecting a predetermined lane marking on a road based on a photographed image photographed by a photographing means,
A structural element setting step for setting a structural element in a morphological operation based on the shape of the predetermined partition line;
A contraction process is performed on the captured image captured by the capturing unit using the structural element set in the structural element setting step to generate a contracted image, and the generated contracted image is expanded. An Opening processing step that performs processing to generate an Opening image;
Top hat conversion processing step for generating a difference image obtained by subtracting the Opening image generated by the Opening processing step from the captured image captured by the imaging means;
A noise removal step of removing noise from the difference image generated by the top hat conversion processing step;
A road lane marking detection method characterized by comprising:

In order to achieve the above object, one embodiment of the present invention provides
A road lane marking detection program for detecting a predetermined lane marking on a road based on a photographed image photographed by a photographing means,
A structural element setting procedure for setting a structural element in a morphological operation based on the shape of the predetermined partition line;
A contraction process is performed on the captured image captured by the imaging unit using the structural element set by the structural element setting procedure to generate a contracted image, and the generated contracted image is expanded. Opening processing procedure to generate an Opening image by processing,
Top hat conversion processing procedure for generating a difference image obtained by subtracting the Opening image generated by the Opening processing procedure from the captured image captured by the imaging unit;
A noise removal procedure for removing noise from the difference image generated by the top hat conversion processing procedure;
This may be a road lane marking detection program.

  According to the present invention, a lane marking on a road can be detected with high accuracy.

  Hereinafter, the best mode for carrying out the present invention will be described with reference to the accompanying drawings. Note that the basic concept of the image processing apparatus, the main hardware configuration, the operating principle, the basic control method, and the like are known to those skilled in the art, and thus detailed description thereof is omitted.

  FIG. 1 is a block diagram illustrating an example of a system configuration of a road marking line detection apparatus according to an embodiment of the present invention. The road lane marking detection device 1 according to the present embodiment is based on a photographed image taken by a photographing means 3 such as a camera, and a predetermined lane marking (lane marker) such as a white line on the road, Botts Dots, and Cat's Eye. ) Perform detection. The white line includes any line that divides the road, such as a line of an arbitrary color such as a yellow line, a solid line, a broken line, a dotted line, and a double line.

  The botsdots 51 is a lane marking mainly used in North America, and is formed by arranging a plurality of solid disks having a diameter of about 100 mm made of ceramic, for example, and embedded in the road surface (FIG. 2A). On the other hand, the cat's eye 53 is a dividing line in which a plurality of reflectors formed in a substantially rectangular shape are arranged, and has a characteristic of reflecting incident light in the same direction (FIG. 2B). For example, in Japan, a lane marking composed only of the cat's eye 53 is used on a curved road other than a highway, and in North America, it is used not only on a curved road but also on a straight road. Both the bots dot 51 and the cat's eye 53 are arranged in a state of slightly protruding from the road surface.

  The road lane marking detection apparatus 1 includes a structural element setting unit 5a that sets a one-dimensional, two-dimensional, or three-dimensional structural element B in a morphological operation based on the shape of a predetermined lane marking. Note that the morphological operation is a well-known technique detailed in, for example, “Morphology” (Corona). Specifically, in the morphology, only elements having a specific geometric structure are selectively selected from the original image (binary image or grayscale image) by a set-theoretic operation using a preset structural element B. Can be extracted.

  Based on the structural element B set by the structural element setting unit 5a, the structural element setting unit 5a performs a contraction process (Erosion) on the captured image photographed by the photographing unit 3 to generate a contracted image. An opening processing means 5b for generating an opening image by performing dilation processing on the generated contracted image is connected. The opening processing means 5b includes a top-hat transform processing means (Top-Hat Transform) means for generating a difference image obtained by subtracting the opening image generated by the opening processing means 5b from the photographed image taken by the photographing means 3. 5c is connected.

  The photographing means 3 is arranged on the upper part of the front window (for example, the rear side of the rearview mirror) of the host vehicle, and has a camera for photographing images around the vehicle such as the front, rear, and side of the vehicle.

  The camera 3 can be attached to any position of the vehicle as long as it can photograph the surroundings of the vehicle. The camera 3 is, for example, a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) camera having a predetermined number of pixels (5 million pixels or the like).

  The camera 3 is connected to an image processing ECU (Electronic Control Unit) 5 including the above-described opening processing means 5b, top hat conversion processing means 5c, and the like. The captured image captured by the camera 3 is transmitted to the image processing ECU 5.

  Note that the image processing ECU 5, a travel support ECU 30, an EMPS-ECU 35, an EFI-ECU 37, and a meter ECU 39, which will be described later, are composed of a microcomputer, which executes various processes according to a program and controls each part of the apparatus. The CPU has a ROM for storing an execution program for the CPU, a RAM (Random Access Memory) for storing image data, calculation results, and the like, a timer, a counter, an input / output interface, and the like. The CPU, ROM, RAM, and input / output interface are connected to each other by a data bus.

  The input / output interface includes an A / D (Analog / Digital) converter that converts an analog signal into a digital signal, a D / A (Digital / Analog) converter that converts a digital signal into an analog signal, and the like. Furthermore, the structural element setting means 5a, opening processing means 5b, top hat conversion processing means 5c, timer, and counter described above are stored in the ROM and realized by a program executed by the CPU.

  A captured image captured by the camera 3 is converted into digital data by an A / D converter of an input / output interface of the image processing ECU 5 and stored in a RAM (for example, a screen frame memory). The CPU reads a captured image (for example, an image for one screen) from the RAM as necessary, and performs image processing such as the above-described opening processing and top hat conversion processing.

  The structural element setting unit 5a performs setting of the structural element B in the morphology according to the shape of a predetermined lane marking on the captured image captured by the camera 3. Note that the actual shape (dimensions, etc.) of predetermined lane markings such as botsdots 51, cat's eyes 53, and white lines are stored in advance in the ROM, but the shape of an arbitrary lane marking is set via the input / output device 7. Or it can be changed. For example, the shape of the botsdots 51 is stored in the ROM as a disk shape of about 100 mm.

  The structural element setting unit 5a sets the structural element B based on the shape of a predetermined partition line stored in the ROM. The structural element setting unit 5a sets the structural element B so that a predetermined division line disappears by, for example, opening processing.

  Next, a method for setting the structural element B will be described in detail.

  In the shooting by the camera 3, the apparent size and shape change on the front side and the back side as shown in FIG. 3 due to the influence of the position of the camera 3. Therefore, it is necessary to set the structural element B according to this change. Arise. As a method of setting the structural element B for an object whose size and shape change in appearance, for example, there is a method of performing a geometric transformation of an image to generate a bird's-eye view image.

  In other words, this is a method of aligning the sizes of objects in the distance and in the vicinity by stretching the image in the horizontal direction as it goes further. In this case, only one structural element B needs to be set. As shown in FIG. 4, when the botsdot is 10 cm, it is assumed that the image is displayed with 5 pixels on the image, and a structural element B of (5 + α) pixels larger than that is set. Here, α is set to 1 to 3, for example. Further, the enlargement magnification can be easily calculated based on the position of the camera 3 or the like.

  Furthermore, as a method of setting the structural element B for the object that apparently changes in size and shape, a method of reducing the size of the structural element B as it goes farther as shown in FIG. May be used.

  Compared with the method for generating the bird's-eye image, this method does not require image processing for generating the bird's-eye image, and only calculates and reduces the reduction rate of the structural element B. Can be kept low. As the size of the structural element B goes farther, it becomes smaller vertically, but since it is a one-dimensional process, it actually becomes smaller only in the horizontal direction. Therefore, the vertical length is 1 pixel both in the distance and in the vicinity.

  In addition, the setting method of the structural element B may be varied depending on the detection target. Hereinafter, a method for setting the detection target according to the detection target will be described in detail.

  When the detection object is a botsdot, since it is a three-dimensional object, the appearance differs depending on how the light strikes (sun angle, weather, etc.) as shown in FIG. Since both the botsdots and the cat's eyes are very small, they appear to have substantially the same characteristics on the photographed image. Therefore, here, description will be made assuming that these are the same object.

  The size of the botsdots apparently changes from “a state like a point” to a case of “having a width of several pixels in the horizontal direction”. The nature of the top-hat transformation is basically to “leave something smaller than the structural element B”. Therefore, if the size of the structural element B is 10 cm + α, which is the size of the botsdots, it will apparently be 10 cm or less. If you can see it, you can deal with it. The size of α is set by actual experiments, but it is preferable to set it to an appropriate size that is not too large so that unnecessary objects such as noise do not remain.

  Botsdots usually appear smaller than 10 cm, but rarely appear larger than this. Therefore, α may be set to 1 to 2 cm. Then, how many pixels 10 cm + α is at an arbitrary position on the line (scanning line) is calculated, and the size of the structural element B is varied for each line.

  Next, a case where the detection target is a white line will be described.

  For example, as shown in FIG. 6, there is a case where one side of the lane is constituted by a white line and the other side is constituted by a botsdot. In this case, by matching the size of the structural element B with the white line, it is possible to detect the botsdots and the white line at the same time. In addition, since the white line in the United States is comparatively thin with 15-25 cm, it is not necessary to enlarge the structural element B so much. Therefore, there is no possibility of detecting unnecessary objects such as vehicles, road shoulders, and noise. It is also possible to detect white lines in Japan and Europe without botsdots with this method.

  Note that the object may be detected by changing the detection method on one side and the other side. For example, conventional edge detection may be performed on one side, top hat conversion may be performed on the other side, and the size of the structural element B may be changed on one side and the other side (for example, one side The side is 30 cm and the other side is 12 cm). Thereby, detection accuracy can be improved more.

  The opening processing means 5b performs shading correction on the captured image read from the RAM in order to remove luminance unevenness caused by the optical system. The standardization by this shading correction is to calculate a moving average in a predetermined range before and after the line of interest (scanning line) and subtract or divide this moving average from the signal of the line of interest, as is conventionally known. . The photographed image after the shading correction is, for example, a grayscale image (grayscale image) with 256 gradations, so that a binarized image can be obtained by performing binarization processing using a predetermined threshold.

  The Opening processing means 5b performs an Opening process that combines the above-described contraction process and expansion process on a grayscale or binarized captured image (hereinafter referred to as an initial process image).

  Next, the concept of contraction processing and expansion processing will be described.

  FIG. 7A is a diagram showing an arbitrary horizontal line (scanning line) and the structural element B on the initial processing image. On one line L, for example, a white line having a width of 6 pixels, noise of 2 pixels, and botsdots of 3 pixels are displayed in this order. The structural element B has a height of 1 pixel and a horizontal width of 5 pixels, for example.

  First, an arbitrary line L on the initial processing image is extracted, and the structural element B is moved along the inside of the line. At this time, as shown in FIG. 7B, it can be seen that the structural element B cannot enter the narrow portion (portion smaller than the structural element B) S of the line. Next, when the center line (solid line) of the movement trajectory of the structural element B when the structural element B is moved along the inside of the line is drawn, a narrow portion (dotted line) S as shown in FIG. Is erased, noise and botsdots are erased on the line, and only the white line remains. The above process corresponds to the opening process.

  Furthermore, when a line in which only the white line remains is drawn from the initial line L, the white line (dotted line) is erased and only noise and botsdots smaller than the structural element B can be detected as shown in FIG. This process corresponds to a top hat conversion process. As described above, although one arbitrary line in the horizontal direction on the initial processing image has been described, the same processing is performed for all the lines on the initial processing image.

  By these opening process and top hat conversion process, unlike the conventional edge filter process, the shape of the bots dot smaller than the structural element B can be detected in the state before the process. Note that the actual opening process and top hat conversion process have substantially the same effects, although the processing contents are different from the trace of the structural element B described above.

  Next, an operation on an actual processed image when the above-described opening process and top hat conversion process are performed will be described.

  When contraction processing is performed on the initial processing image shown in FIG. 8A using the structural element B, noise and botsdots smaller than the structural element B are eliminated as shown in FIG. The white line that is substantially the same as is contracted to 1 pixel in width. In the above line, the horizontal width of the white line is 6 pixels, but in the initial processing screen, the horizontal width is 5 pixels, which is the same as the structural element B.

  Next, when expansion processing is performed on the contracted image (FIG. 8B), the white line is restored from 1 pixel to 5 pixels as shown in FIG. 8C, but noise and botsdots are reduced. It remains erased and cannot be restored. After that, by subtracting the shrinking and expansion processed Opening images from the initial processing image, a difference image in which only noise (less than 5 pixels) smaller than the structural element B and Botsdots are displayed is generated (see FIG. 8 (d)).

  In addition, although noise remains in the difference image in addition to the botsdots, when the luminance of the noise is lower than that of the bottsdots, the noise can be easily removed by a filter set to a predetermined luminance, for example. On the other hand, when the brightness of noise is higher than the brightness of botsdots, only botsdots can be easily detected by Hough transform or the like that detects only a linearly arranged pattern.

  In the above description, the structural element B having a height of 1 pixel and a horizontal width of 5 pixels is used. For example, a structural element (two-dimensional structural element) having a circular shape (for example, a diameter of 4 pixels) larger than Botsdots. B may be used. That is, it is possible to use the structural element B having an arbitrary shape according to the shape of the dividing line such as a white line, a cat's eye, or a bots dot detected from the initial processing image.

  Next, morphological operations such as the above-described contraction process, expansion process, and opening process performed by the opening processing unit 5b will be described.

  In the morphological operation, a set-theoretic operation is performed using a preset structural element B. First, an auxiliary operation that is a premise of a basic operation of morphology is defined by the following equations (1) and (2).

  Inversion (symmetry) with respect to the origin is expressed by the following equation (1).

なお、上記（１）式において、Aは２次元空間における集合とし、aは集合Aに属する要素とする。また、要素とは２次元空間内の集合Aを構成する座標点を意味し、aは原点からのベクトルを意味する。例えば、２値画像において、画像の背景を構成する各画素の画素値は０となり、その背景上に描かれた図形を構成する各画素の画素値は１となる。画素値が１となる画素が要素aとなる。

In the above equation (1), A is a set in a two-dimensional space, and a is an element belonging to the set A. An element means a coordinate point that constitutes a set A in a two-dimensional space, and a means a vector from the origin. For example, in a binary image, the pixel value of each pixel constituting the background of the image is 0, and the pixel value of each pixel constituting the graphic drawn on the background is 1. A pixel having a pixel value of 1 is element a.

  Translation (for example, translating A by b) is expressed by the following equation (2).

また、モルフォロジー演算において、下記（３）式、（４）式、及び（５）式に示すMinkowski和とMinkowski差が基本演算となる。さらに、これら基本演算を組合せることにより、種々のモルフォロジー演算が構成される。

In the morphological operation, the Minkowski sum and the Minkowski difference shown in the following equations (3), (4), and (5) are basic operations. Furthermore, various morphological operations are configured by combining these basic operations.

  Minkowski sum, for example, consists of all combinations of elements when there are two sets A and B, and the sum (a + b) of element a of set A and element b of set B is considered It refers to a set and is represented by the following formula (3) or (4).

なお、上記（３）式を上記（２）式で示す平行移動の定義で表すと、下記（４）式で表される。

In addition, when the said (3) Formula is represented by the definition of the parallel movement shown by the said (2) Formula, it represents with the following (4) Formula.

一方、Minkowski差とは、集合Bの全て要素bに対して、（x−b）がAの要素となるようなxの集合であり、下記（５）式で表される。

On the other hand, the Minkowski difference is a set of x such that (x−b) is an element of A with respect to all elements b of the set B, and is expressed by the following equation (5).

なお、上記（５）式を上記（２）式で示す平行移動の定義で表すと、上記（５）式は下記（６）式で表すことができる。

In addition, when said (5) Formula is represented by the definition of the parallel movement shown by said (2) Formula, said (5) Formula can be represented by the following (6) Formula.

Opening処理手段５ｂは、（１）式乃至（６）式に基づいて、モルフォロジーの基本演算である下記（７）式乃至（９）式により、膨張処理、収縮処理、これらを組合せたOpening処理を行う。

Based on the equations (1) to (6), the opening processing means 5b performs the expansion processing, the contraction processing, and the opening processing combining these by the following equations (7) to (9) that are basic operations of morphology. Do.

  The Opening processing means 5b performs Dilation (expansion processing, shifted overlap) using the following equation (7).

また、Opening処理手段５ｂは、下記（８）式を用いて、Erosion（収縮処理、掻き取り、侵食）を行う。

The opening processing means 5b performs Erosion (contraction processing, scraping, erosion) using the following equation (8).

さらに、Opening処理手段５ｂは、下記（９）式を用いて、DilationとErosionとを組み合わせたOpening処理を行う。

Further, the opening processing means 5b performs an opening process combining Dilation and Erosion using the following equation (9).

なお、上記Bは構造要素であり、有界集合である。上述したように、構造要素設定手段５ａにより構造要素Bを設定、変更することで、Opening処理手段５ｂは、例えば処理画像上のボッツドッツ及びノイズを消去し、白線のみが抽出されたOpening画面を生成する。

Note that B is a structural element and a bounded set. As described above, by setting and changing the structural element B by the structural element setting unit 5a, the opening processing unit 5b generates, for example, an opening screen in which only white lines are extracted by erasing bots dots and noise on the processed image. To do.

  The top hat conversion processing unit 5c generates a difference image obtained by subtracting the opening image generated by the opening processing unit from the initial processing image (for example, an image from which botsdots and noise are deleted and only a white line is extracted). Note that, for example, only botsdots and noise are extracted from the generated difference image.

  The noise removing unit 5d performs an opening process on the difference image generated by the top hat conversion processing unit 5c, using the structural element B ′ set by the structural element setting unit 5a, and removes noise. , Only predetermined lane markings such as botsdots 51 are extracted.

  By this opening process, noise smaller than the structural element B ′ is removed, and only botsdots larger than B ′ remain. However, an absolute condition is that the size of the botts dots is not the same as the noise so that the botts dots are not accidentally removed. In addition, since an extremely small size (for example, about 1 pixel) usually exists on the image, this process can suppress a decrease in detection accuracy due to noise. Furthermore, noise can be easily removed by simply changing the size or shape of the structural element without using an additional algorithm for noise removal. In the above, the case where the botsdots are detected has been mainly described. However, since the cat's eye is substantially the same as the processing contents of the botsdots, a detailed description thereof will be omitted.

  Next, an actual processing procedure in the detection of botsdots will be described in detail.

  Since botsdots are only 10 cm in diameter, they will be almost invisible in the far field. Therefore, unnecessary processing is suppressed by avoiding detection in areas where 10 cm botsdots appear to be 2 to 3 pixels or less (for example, it is difficult to distinguish from 1 to 2 pixels from noise). can do. It should be noted that an optimum distance can be obtained by experiment or the like to determine how far the object is to be detected.

  Further, since the botsdots appear only small even in a close region, it is preferable not to perform thinning of the processing line that is performed in a normal LKA. Even if the thinning-out process is not performed, the above-described detection process of the far region is not performed, so that the calculation cost does not increase.

  FIG. 9A is a diagram illustrating a state in which a shoulder, a bots dot, a vehicle, noise, and a bots dot are photographed on an image. FIG. 9B is a diagram showing a result of extracting one line on the image of FIG. The following description will be made using a one-dimensional structural element and a binarized image.

  First, as shown in FIG. 9B, one line on the image of FIG. 9A is extracted. In FIG. 9A, the vehicle and the road shoulder are gray, but are binarized, and are therefore white.

  Next, a pixel size of 10 cm (bottom dot width) on this line is calculated, and the length of the structural element B is set. For example, if 10 cm on this line is 5 pixels, a structural element B having a width of about 7 pixels larger than that is set.

  Next, when contraction processing is performed on this signal, the result is as shown in FIG. Furthermore, when the expansion process is performed on FIG. 10A, the result is as shown in FIG. If the signal after the Opening process shown in FIG. 10B is subtracted from the input signal shown in FIG. 9B, a top-hat converted signal shown in FIG. 11A is obtained.

  That is, it can be seen that the vehicle and the shoulder are deleted from the input image, and only noise and botsdots remain. Here, the noise (signal noise, road surface noise) smaller than the structural element B ′ is deleted by contraction and expansion (Opening processing). A feature of the Opening process is that an element smaller than the structural element B ′ can be removed. Accordingly, when a structural element (for example, 1 × 3 pixels) B ′ having a size smaller than the botsdots is prepared and the opening process is performed on FIG. 11A, the result is as shown in FIG. When it is desired to detect a botsdot having a width of about 3 pixels, the size of the structural element B ′ is preferably smaller than 1 × 3, and the optimum size can be set by experiment.

  As a result, if the apparatus is provided with expansion processing, contraction processing, and subtraction processing using structural elements B ′ having different sizes, it is possible to perform detection and noise removal without having to incorporate different processing. In this noise removal process, if the size of the botts dot is the same as the noise (for example, 1 pixel), the bots dot is also deleted, but the distinction between “dot bots dot” and “point noise” is easy. Because it is not, there is no problem even if it is actually erased.

  As described above, when a series of processing is applied to all the processing lines, an image in which only the botsdots remain can be obtained as shown in FIG.

  Next, setting and processing of the structural element B in an image in which white lines are mixed will be described in detail.

  FIG. 13A is a diagram illustrating a state in which a road shoulder, a white line, a vehicle, noise, and botsdots are photographed on an image. FIG. 13B is a diagram illustrating a result of extracting one line on the image illustrated in FIG. The following description will be made using a one-dimensional structural element and a binarized image.

  Similarly to the above, when one line is extracted from FIG. 13A, the luminance value is as shown in FIG. Next, a pixel size of 15 cm (white line width) on this line is calculated, and the length of the structural element B is set. For example, if 15 cm in this line is 7 pixels, a structural element B of about 9 pixels larger than that is prepared.

  Furthermore, when contraction processing is performed on this signal, the result is as shown in FIG. Subsequently, when an expansion process is performed on FIG. 14A, the result is as shown in FIG. When the signal after the opening process shown in FIG. 14B is subtracted from the input signal shown in FIG. 13B, a top hat converted signal as shown in FIG. 15A is obtained.

  The vehicle and the road shoulder are deleted from the input signal shown in FIG. 13 (b), and only noise, botsdots, and white lines remain as shown in FIG. 15 (a). Here, as shown in FIG. 15B, only noise is removed by performing the same contraction process and expansion process on noise (signal noise, road surface noise) smaller than the structural element B ′. be able to. The structural element B ′ is 1 × 3 pixels as described above.

  This makes it possible to simultaneously perform botsdot detection, white line detection, and noise removal. However, as described above, when the number of unnecessary objects to be detected increases because the structural element B ′ is enlarged, a new noise removal method may be applied. Further, when the width of the white line corresponds to a country that frequently changes (for example, branching or merging) depending on the situation, the width of the structural element B may be dynamically varied.

  By applying the above-described series of processing to all lines, an image in which only bot dots and white lines remain as shown in FIG. 16 can be obtained.

  The case where the one-dimensional structural element B is used for the binarized image has been described above. Next, the one-dimensional structural element B is applied to the grayscale image (grayscale image, monochrome multi-valued image). The case where it is used will be described.

  When one line of the actual photographed image (grayscale image) is extracted, it can be seen that the luminance value is greatly undulated as shown in FIG. The brightness value of the white line is not flat at the top, and has some variation. Moreover, since the botsdots are spherical, the luminance value is a mountain. The noise brightness value may be lower or higher than the brightness values of the botsdots and the white line.

  With respect to FIG. 17, the lane marking is detected in the same procedure as described above. Note that the size of the structural element B is 1 × 9 pixels in order to detect a white line. When the contraction process is performed on FIG. 17, the result is as shown in FIG. Further, when the expansion process is performed on FIG. 18A, the result is as shown in FIG. In FIG. 17, the vehicle larger than the structural element B is a vehicle. When the subtraction processing from FIG. 17 to FIG. 18B is performed, the top hat conversion processing result of FIG. 19 is obtained. In FIG. 19, it can be seen that the signal indicating the vehicle has fallen to a low level, while the white line, the botsdots, and the noise remain while maintaining the shape of the input signal.

  FIG. 20 shows the concept of the contraction process and the expansion process in FIG. Specifically, as shown in FIG. 20, the structural element B having the same width of 9 pixels is traced sequentially from the left side (or right side) with respect to FIG. Indicated. This thick line arrow has the same shape as the result of the opening process in FIG. 18B, and the upper part (area where the structural element B cannot enter) is the same as the result of the top hat conversion process in FIG.

  Furthermore, when the opening process is performed using the (1 × 5) structural element B ′ and noise is removed, the result is as shown in FIG.

  In FIG. 21, it can be seen that the noise is eliminated and only the white line and botsdot signals remain. In this way, by selecting an appropriate size of the structural element B in the grayscale image, it is possible to detect botsdots and white lines.

  Next, a detection method using the multidimensional structural element B will be described. First, processing using the two-dimensional structural element B will be described. Here, as shown in FIG. 22, a case will be described in which processing is performed using a two-dimensional structural element (two-dimensional signal) B for an image photographed progressively or non-interlaced.

  Note that the following advantages can be considered when processing is performed using a two-dimensional signal. When repair marks, wrinkle streaks, rain streaks, etc., equal to the width of the bots dot are present on the image shown in FIG. 23A, the shape is completely different, but the one-dimensional signal cannot be distinguished. (FIG. 23B). Furthermore, since the horizontal width is the same, these unnecessary items remain even if the top hat conversion process is performed. However, even in such a situation, if the two-dimensional structural element B is used, the unnecessary items can be easily distinguished.

  First, a two-dimensional structural element B is set by paying attention to the structural difference between a detection target (botts dot) and an unnecessary object. When the horizontal width of the detection target and the horizontal width of the unnecessary object are the same, the structurally different points include, for example, the vertical width (in the depth direction), the difference between a circle and a quadrangle. Here, assuming that the ridges and the botsdots have shapes as shown in FIG. 24, the shapes are set so that the bottsdots are smaller than the two-dimensional structural element B.

  In the concept of the top hat conversion process, those that can enter the structural element B are deleted, and those that cannot enter remain. The structural element (5 × 5) B in FIG. 24 can enter (can be overlapped) the ridges, but the botsdots have corners (substantially circular), so they can enter even 5 pixels in the same size as the structural element B. Absent.

  A process in which this structural element B is used for detection will be described below. It is assumed that the left mountain in FIG. 25A is a ridge and the right mountain is a botsdot. When the contraction process is performed on this FIG. 25A using the structural element B of FIG. 24, the result is as shown in FIG. 25B. When the expansion process is performed on FIG. 26 (a). As can be seen from FIG. 26 (a), the botsdots smaller than the structural element B are eliminated at the stage of the shrinking process, and soot larger than the structural element B remains.

  When FIG. 26A is subtracted from the input signal of FIG. 25A, the top hat conversion processing result of FIG. 26B is obtained. The brightness value of the eyelid shown in FIG. 26 (a) is significantly lower than the brightness value of the eyelid shown in FIG. 25 (a) (during input), but the brightness value of Botsdots is substantially the same as that during input. It remains in a state where its shape and height are maintained.

  As described above, by using the two-dimensional processing as compared with the one-dimensional processing, the choice of the shape of the structural element B increases, and the design more suitable for the detection target becomes possible. In addition, when an image as shown in FIG. 27A is obtained, only a solid line can be detected by setting a two-dimensional structural element B and performing an opening process, and a top hat conversion process is performed. Thus, only the broken line can be detected.

  Thereby, improvement in detection accuracy can be expected even in an image in which a composite line is photographed. For example, in an image obtained by photographing a composite line as shown in FIG. 27B, conventionally, special processing such as pattern matching is performed by observing the shape of the composite line. However, due to a matching error or the like, the correct lane cannot be selected, and the solid line and the broken line are detected alternately, causing the vehicle to fluctuate.

  On the other hand, if the above process is applied, if it is desired to travel along the arrow X1, it is possible to detect the lane without performing complicated processing by leaving only the broken line and deleting the solid line in the top hat conversion process. . Further, when the vehicle travels along the arrow X2, only the solid line needs to remain by performing the opening process. In this way, by removing unnecessary objects from the image as necessary, it is not necessary to perform complicated determination and recognition processing in the subsequent stage.

  Next, a processing flow by the road marking line detection apparatus 1 according to the present embodiment will be described. FIG. 28 is a flowchart illustrating an example of a processing flow of the road lane marking detection device 1. The processing routine shown in FIG. 28 is repeatedly executed every predetermined minute time.

  As shown in FIG. 28, image information captured by the camera 3 is transmitted to the image processing ECU 5 and stored in the RAM.

  Next, the image processing ECU 5 reads the captured image stored in the RAM, performs shading correction, binarization processing, etc., and generates an initial processed image (S100).

  Thereafter, the opening processing means 5a of the image processing ECU 5 performs a contraction process and an expansion process (Opening process) on the initial process image using a predetermined structural element B to generate an opening image (S110).

  The top hat conversion processing unit 5c generates a difference image obtained by subtracting the opening image generated by the opening processing unit 5b from the initial processing image (S120).

  Next, the noise removing unit 5d removes noise from the difference image generated by the top hat conversion processing unit 5c using a filter or the like, and detects predetermined dividing lines such as botsdots and cat's eyes (S130).

  As described above, in the road lane marking detection device 1 according to the present embodiment, the opening processing means 5b performs the opening processing using the structural element B set based on the shape of a predetermined lane marking such as botsdots, cat's eyes, white lines, etc. And generate an Opening image. Next, the top hat conversion processing means 5c performs a top hat conversion process on the generated opening image to generate a difference image. After that, the noise removing unit 5d removes noise from the difference image generated by the top hat conversion processing unit 5c, and detects a predetermined dividing line such as a bots dot, a cat's eye, or a white line. Thereby, predetermined division lines, such as a bots dot, a cat's eye, and a white line, can be detected with high precision.

  As mentioned above, although the best mode for carrying out the present invention has been described using one embodiment, the present invention is not limited to such one embodiment, and within the scope not departing from the gist of the present invention, Various modifications and substitutions can be made to the above-described embodiment.

  For example, in the above-described embodiment, the vehicle travel assistance may be performed based on a predetermined lane line detected by the road segment detection device 1.

  In this case, the image processing ECU 5 of the road marking line detection apparatus 1 is connected to a travel support ECU 30 that performs vehicle travel support via a communication line (FIG. 29). The image processing ECU 5 transmits the detected predetermined marking line to the travel support ECU 30 via a communication line.

  The driving support ECU 30 is connected to a main switch 31 that turns on or off the main power supply for driving support. The on / off state of the main switch 31 is transmitted to the image processing ECU 5 via the communication line.

  The driving support ECU 30 is connected to a communication line 33 such as a vehicle LAN (Local Area Network). The vehicle LAN 33 is connected to an electric power steering ECU (EMPS-ECU) 35 that controls the assist force applied to the steering. Furthermore, an engine ECU (EFI-ECU) 37 that controls the engine in an integrated manner and a meter ECU 39 that controls display of a meter panel and the like are connected to the vehicle LAN 33. These travel support ECU 30, EMPS-ECU 35, EFI-ECU 37, and meter ECU 39 perform bidirectional data communication based on the CAN protocol or the like via the vehicle LAN 33.

  The EMPS-ECU 35 is connected to a steering actuator 35a such as an electric motor that applies assist force to the steering. The EMPS-ECU 35 drives and controls the steering actuator 35a based on the torque value (current value) transmitted from the travel support ECU 30.

  A vehicle speed sensor 37a for detecting the vehicle speed and a steering angle sensor 37b for detecting the steering angle of the steering are connected to the EFI-ECU 37. The vehicle speed detected by the vehicle speed sensor 37 a and the steering angle detected by the steering angle sensor 37 b are transmitted to the travel support ECU 30 via the EFI-ECU 37 and the vehicle LAN 33.

  The driving support ECU 30 predicts the position of the subsequent vehicle based on the lane line data transmitted from the road lane line detection device 1 via the communication line and the driving situation of the driver, and the own vehicle deviates from the lane line. When it is determined that there is a possibility of the alarm, an alarm signal is transmitted to the meter ECU 39 (lane departure control). When the meter ECU 39 receives an alarm signal from the driving support ECU 30, the meter ECU 39 generates a warning sound in the buzzer 39a and / or turns on the warning lamp 39b.

  Further, the driving support ECU 30 calculates the estimated position of the host vehicle based on the lane line data transmitted from the road lane line detection device 1 via the communication line. Further, the driving support ECU 30 determines that the own vehicle is an abbreviation of the lane marking based on the calculated estimated position of the own vehicle, the vehicle speed detected by the vehicle speed sensor 37a, and the steering angle detected by the steering angle sensor 37b. The assist steering force (current value) required for traveling in the center is calculated and transmitted to the EMPS-ECU 35 (lane keeping control). The EMPS-ECU 35 controls the steering actuator 35a based on the assist steering force transmitted from the travel support ECU 30.

  As described above, the driving assistance ECU 30 performs the lane departure control and / or the lane keeping control based on the lane line detected by the road lane line detection device 1. As a result, each of these controls can be performed with high accuracy.

  The present invention can be used, for example, in a road lane marking detection apparatus that detects a lane marking on a road. The appearance, weight, size, running performance, etc. of the vehicle to be mounted are not limited.

It is a block diagram which shows an example of the system configuration | structure of the road marking line detection apparatus which concerns on one Example of this invention. (A) It is the figure which looked at the road from upper direction, and is a figure which shows a botsdot. (B) It is the figure which looked at the road from upper direction, and is a figure which shows a cat's eye. It is a figure which shows the change of the apparent magnitude | size in a distant place and the vicinity. It is the bird's-eye view created by giving a geometric change to an image. It is a figure which shows the method of reducing the magnitude | size of a structural element as it goes to a distant place. It is a figure which shows the state from which the kind of lane marking differs in one side and the other side. (A) It is a figure which shows the arbitrary 1 line and structural element of the horizontal direction on an initial process image. (B) It is a figure which shows the state which moved the structural element along the inner side of arbitrary 1 lines of an initial process image. (C) It is a figure which shows the centerline of the movement locus | trajectory of a structural element when moving a structural element along the inner side of a line. (D) It is a figure which shows the state from which the white line was erase | eliminated and only the noise smaller than a structural element and botsdots were detected. (A) It is a figure which shows the initial processing image on which the white line, the noise, and the botsdots were displayed. (B) It is a figure which shows the state where the noise smaller than a structural element and botsdots were erase | eliminated, and the white line substantially the same as a structural element contracted. (C) It is a figure which shows the state from which the white line was decompress | restored and noise and botsdots were erase | eliminated. (D) It is a figure which shows the difference image by which only the noise smaller than a structural element and botsdots were displayed. (A) It is a figure which shows the state by which the road shoulder, the botsdots, the vehicle, the noise, and the botsdots were image | photographed on the image. (B) It is a figure which shows the result of having extracted 1 line on the image of Fig.9 (a). (A) It is a figure which shows the state of the luminance value after a shrinkage | contraction process. (B) It is a figure which shows the state of the luminance value after an expansion process. (A) It is a figure which shows the state of the luminance value after a top hat conversion process. (B) It is a figure which shows the state of the luminance value after noise removal. It is a figure which is the final image which performed the process with respect to all the lines, and shows the image which only the botsdot remained. (A) It is a figure which shows the state by which the road shoulder, the white line, the vehicle, the noise, and the botsdot were image | photographed on the image. (B) It is a figure which shows the result of having extracted 1 line on the image shown to Fig.13 (a). (A) It is a figure which shows the state of the luminance value after a shrinkage | contraction process. (B) It is a figure which shows the state of the luminance value after an expansion process. (A) It is a figure which shows the state of the luminance value after a top hat conversion process. (B) It is a figure which shows the state of the luminance value after noise removal. It is a figure which is the final image which performed the process with respect to all the lines, and shows the image which only the white line and the botsdot remained. It is a figure which shows the signal which extracted one line from the grayscale image. (A) It is a figure which shows the state of the luminance value after a shrinkage | contraction process. (B) It is a figure which shows the state of the luminance value after an expansion process. It is a figure which shows the state of the luminance value after a top hat conversion process. It is a figure which shows the state of the conceptual process of a contraction process and an expansion process. It is a figure which shows the state which detected only the white line and the botsdot. It is a figure which shows the image image | photographed by progressive or non-intern. (A) It is a figure which shows the state in which the repair trace equal to the width of a bots dot, the stirrup line, the rain line, etc. existed on the image. (B) It is a figure which shows the brightness | luminance value when the repair trace equal to the width of a bots dot, the stirrup line, the rain line, etc. exist on the image. It is a figure which shows the shape of a bottling dot and a structural element. (A) It is a figure which shows the luminance value of an input image, and is a figure which shows the luminance value of a rudder and a botsdot. (B) It is a figure which shows the state of the luminance value after a shrinkage | contraction process. (A) It is a figure which shows the state of the luminance value after an expansion process. (B) It is a figure which shows the state of the luminance value after a top hat conversion process. (A) It is a figure which shows the image by which the continuous line and the broken line were image | photographed. (B) It is a figure which shows the image by which the compound line was image | photographed. It is a flowchart which shows an example of the control processing flow of a road marking line detection apparatus. It is a block diagram which shows an example of the driving assistance system by which the road lane marking detection apparatus which concerns on one Example of this invention is mounted.

Explanation of symbols

1 Road marking line detection device 3 Imaging means 5 Image processing ECU
5a Structural element setting means 5b Opening processing means 5c Top hat conversion processing means 5d Noise removing means 30 Driving support ECU

Claims (10)

A road lane marking detection device that detects a predetermined lane marking on a road based on a photographed image photographed by a photographing means,
A structural element setting means for setting a structural element in a morphological operation based on the shape of the predetermined lane marking;
Based on the structural element set by the structural element setting means, a contracted image is generated on the captured image captured by the capturing means to generate a contracted image, and the generated contracted image is expanded. Opening processing means for performing processing to generate an Opening image;
Top hat conversion processing means for generating a difference image obtained by subtracting the Opening image generated by the Opening processing means from the captured image captured by the imaging means;
Noise removing means for removing noise by performing contraction processing and expansion processing using a structural element smaller than the structural element used in the opening processing means from the difference image generated by the top hat conversion processing means; Prepared,
The road lane marking detection apparatus characterized in that the structural element setting means sets the size of the structural element as it goes farther. A road lane marking detection device that detects a predetermined lane marking on a road based on a photographed image photographed by a photographing means,
A structural element setting means for setting a structural element in a morphological operation based on the shape of the predetermined lane marking;
Based on the structural element set by the structural element setting means, the contracted image is obtained by performing a contraction process on the bird's-eye view image obtained by enlarging the image in the lateral direction as the distance from the captured image captured by the imaging means increases. Opening processing means for generating an Opening image by performing expansion processing on the generated contracted image,
Top-hat transform processing for generating a difference image obtained by subtracting the opening image generated by the opening processing unit from a bird's-eye image obtained by extending the image in the horizontal direction as the distance from the captured image captured by the imaging unit increases. Means,
Noise removing means for removing noise by performing contraction processing and expansion processing using a structural element smaller than the structural element used in the opening processing means from the difference image generated by the top hat conversion processing means;
A road marking line detection apparatus comprising: The road marking line detection device according to claim 1 or 2,
The structural element setting unit determines a structural element based on the number of pixels corresponding to 10 cm plus 1 to 2 cm when the predetermined division line on the captured image captured by the imaging unit is a botsdot. A road lane marking detection apparatus, wherein, when the predetermined lane marking on the photographed image is a white line, a structural element is determined based on the number of pixels corresponding to 30 cm. The road marking line detection device according to any one of claims 1 to 3,
The road marking line detection device, wherein the predetermined marking line is a bots dot, a cat's eye, or a white line. The road marking line detection device according to any one of claims 1 to 4,
A road lane marking detection apparatus, further comprising travel support means for supporting driving of a vehicle based on the predetermined lane marking. The road marking line detection device according to claim 5,
The road is characterized in that the driving support means performs lane keeping control in which a vehicle travels in the lane line, or lane departure warning control in which a warning is given to a user when the vehicle departs from the lane line. A lane marking detector. A road lane marking detection method for detecting a predetermined lane marking on a road based on a photographed image photographed by a photographing means,
A structural element setting step for setting a structural element in a morphological operation based on the shape of the predetermined partition line;
A contraction process is performed on the captured image captured by the capturing unit using the structural element set in the structural element setting step to generate a contracted image, and the generated contracted image is expanded. An Opening processing step that performs processing to generate an Opening image;
Top hat conversion processing step for generating a difference image obtained by subtracting the Opening image generated by the Opening processing step from the captured image captured by the imaging means;
A noise removal step of removing noise by performing shrinkage processing and expansion processing using a structural element smaller than the structural element used in the opening processing step from the difference image generated by the top hat transformation processing step;
With
The structural element setting step is a step of setting the size of the structural element to be smaller as it goes farther. A road lane marking detection method for detecting a predetermined lane marking on a road based on a photographed image photographed by a photographing means,
A structural element setting step for setting a structural element in a morphological operation based on the shape of the predetermined partition line;
A contracted image is obtained by performing a contraction process on the bird's-eye view image obtained by enlarging the image in the lateral direction as the distance from the captured image captured by the capturing unit increases using the structural element set in the structural element setting step. And an opening processing step for generating an opening image by performing expansion processing on the generated contracted image,
Top-hat conversion processing for generating a difference image by subtracting the opening image generated by the opening processing step from the bird's-eye view image obtained by extending the image in the lateral direction as the distance from the captured image captured by the imaging unit increases. Steps,
A noise removal step of removing noise by performing shrinkage processing and expansion processing using a structural element smaller than the structural element used in the opening processing step from the difference image generated by the top hat transformation processing step;
A method of detecting a road marking line, comprising: A road lane marking detection program for causing a computer to detect a predetermined lane marking on a road based on a photographed image photographed by a photographing means,
In the computer,
Based on the shape of the predetermined lane marking, a structural element setting procedure for setting the structural element in the morphological operation so that the size of the structural element becomes smaller toward the distance ,
A contraction process is performed on the captured image captured by the imaging unit using the structural element set by the structural element setting procedure to generate a contracted image, and the generated contracted image is expanded. Opening processing procedure to generate an Opening image by processing,
Top hat conversion processing procedure for generating a difference image obtained by subtracting the Opening image generated by the Opening processing procedure from the captured image captured by the imaging unit;
A noise removal procedure for removing noise by performing contraction processing and expansion processing using a structural element smaller than the structural element used in the opening processing procedure from the difference image generated by the top hat transformation processing procedure;
Road lane marking detection program to execute A road lane marking detection program for causing a computer to detect a predetermined lane marking on a road based on a photographed image photographed by a photographing means,
In the computer,
A structural element setting procedure for setting a structural element in a morphological operation based on the shape of the predetermined partition line;
A contracted image is obtained by performing a contraction process on the bird's-eye view image obtained by enlarging the image in the lateral direction as the distance from the captured image captured by the capturing unit increases using the structural element set by the structural element setting procedure. Opening processing procedure for generating an Opening image by performing expansion processing on the generated contracted image,
Top-hat conversion processing for generating a difference image by subtracting the opening image generated by the opening processing procedure from the bird's-eye image obtained by extending the image in the lateral direction as the distance from the captured image captured by the imaging unit increases. Procedure and
A noise removal procedure for removing noise by performing contraction processing and expansion processing using a structural element smaller than the structural element used in the opening processing procedure from the difference image generated by the top hat transformation processing procedure;
Road lane marking detection program to execute

Images