JP5208790B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for realizing the image processing apparatus on a computer, which generate an image signal for appropriately displaying an image around a vehicle.

  2. Description of the Related Art In recent years, a technique for prompting a driver to perform an appropriate driving operation by displaying an image around a vehicle captured by an in-vehicle camera on a monitor installed in a vehicle interior is known. In-vehicle cameras used in such applications are generally attached to an accurate position of the vehicle body at an accurate angle so that a desired region (space) around the vehicle can be appropriately displayed.

  However, the attachment position and the attachment angle may be shifted due to an attachment mistake or aging deterioration when attaching the in-vehicle camera to the vehicle body. Technologies for solving such problems are disclosed in, for example, Patent Documents 1 to 4.

  Patent Document 1 includes a first matrix calculation unit, a calibration point specifying unit, a second matrix calculation unit, a third matrix calculation unit, and a success / failure determination unit. An in-vehicle camera calibration device that calibrates an attached in-vehicle camera and determines success or failure is disclosed. Here, the first matrix calculation unit calculates the first matrix based on the coordinates in the reference coordinate system of the calibration points of the calibration indices arranged at least at two different locations. Further, the calibration point specifying unit specifies the calibration point from the captured image of the in-vehicle camera. And a 2nd matrix calculating part calculates a 2nd matrix based on the coordinate of the calibration point on the picked-up image of a vehicle-mounted camera. The third matrix calculation unit calculates a rotation matrix indicating the rotation state of the in-vehicle camera based on the first matrix and the second matrix. Then, the success / failure determination unit determines the success or failure of the calibration of the in-vehicle camera based on the captured image of the in-vehicle camera.

  Patent Document 2 discloses an image processing apparatus capable of reducing the memory capacity of a memory constituting the image processing apparatus and correcting optical axis deviation with respect to the video camera. The image processing apparatus disclosed in Patent Literature 2 includes a video camera, a memory that stores the image pickup signal, and a CPU that performs predetermined processing on the read image pickup signal. The image processing apparatus disclosed in Patent Document 2 further includes a selection writing unit that selects only a predetermined portion of the imaging signal in response to a selection command from the CPU and writes only the selected imaging signal in the memory.

  Further, Patent Literature 3 discloses an optical axis deviation detection device for an in-vehicle camera that can detect an optical axis deviation of the in-vehicle camera. The on-vehicle camera optical axis deviation detection device disclosed in Patent Literature 3 detects the optical axis deviation of the on-vehicle camera by the following method. That is, the in-vehicle camera is attached to the vehicle so that a part of the vehicle body such as a rear bumper enters the field of view. Here, the position where the rear bumper or the like should be observed in the image captured by the in-vehicle camera is stored in advance in the data ROM as a template image. When detecting the optical axis deviation of the vehicle-mounted camera, the position of the rear bumper or the like in the actual image captured by the vehicle-mounted camera is compared with the position of the rear bumper or the like in the template image. And when the position shift more than a predetermined value has arisen, it determines with the optical axis shift having arisen in the vehicle-mounted camera.

  Further, Patent Document 4 discloses an in-vehicle camera inspection device that can promptly give a warning to an occupant when the in-vehicle camera mounted on the vehicle is out of focus, and allows the occupant to reliably discriminate the displacement and dirt of the in-vehicle camera. An in-vehicle camera inspection device is disclosed. The in-vehicle camera inspection device disclosed in Patent Document 4 is configured as follows. That is, the focus determination unit performs focus determination on the captured image captured by the imaging unit. Focus determination is performed by edge extraction. Here, the reference image is stored in the storage unit. Then, the reference image and the captured image are collated to determine the deviation of the installation posture of the imaging unit. In addition, when a non-change area is detected even though the vehicle is moving, it is determined that there is dirt attached to the lens surface of the imaging unit. When it is out of focus, when the installation posture is deviated, or when it is determined that dirt is attached to the lens surface, the warning unit issues a warning to the occupant.

JP 2008-131250 A JP 2002-94977 A JP 2004-1658 A JP 2007-38773 A

  However, the in-vehicle camera calibration device disclosed in Patent Document 1 is a device for calibrating the in-vehicle camera. For this reason, the vehicle-mounted camera itself needs to have a special mechanism for calibrating the vehicle-mounted camera. The image processing apparatus disclosed in Patent Document 2 is an image processing apparatus that can perform optical axis misalignment correction, and the image processing apparatus itself needs to include a mechanism for performing optical axis misalignment correction. there were. The on-vehicle camera optical axis deviation detection device disclosed in Patent Document 3 is a device that only detects the optical axis deviation of the on-vehicle camera. Furthermore, the in-vehicle camera inspection device disclosed in Patent Document 4 determines whether or not there is a deviation in the focus of the in-vehicle camera, a deviation in the installation posture of the in-vehicle camera, dirt on the lens, and the like. It is a device that only issues warnings to passengers. As described above, none of the apparatuses disclosed in Patent Documents 1 to 4 is an apparatus that can appropriately display an image around the vehicle without providing a special mechanism for calibrating the in-vehicle camera in the apparatus itself. It was. Therefore, there has been a strong demand for a technique for appropriately displaying an image around the vehicle with a simple configuration.

  The present invention has been made in view of the above problems, and realizes an image processing apparatus, an image processing method, and the image processing apparatus for generating an image signal for appropriately displaying an image around a vehicle on a computer. It aims at providing the program for.

In order to achieve the above object, an image processing apparatus according to the first aspect of the present invention provides:
A predetermined that defines an area of the predetermined space on the captured image based on a position of the part of the vehicle on a captured image obtained by capturing an imaging space including a predetermined space including a part of the vehicle Storage means for storing spatial position definition information;
A specifying means for specifying a position of a part of the vehicle on the captured image;
An image indicating the predetermined space from the captured image based on the predetermined space position definition information stored in the storage unit and a position of a part of the vehicle on the captured image specified by the specifying unit. Extraction means for extracting the part;
Generating means for generating an image signal for causing the display device to display the image portion extracted by the extracting means ;
The captured image is a moving image obtained by imaging the imaging space while the vehicle is moving by an imaging device installed in the vehicle,
The specifying means specifies a position of an image portion where a change in luminance of a pixel on the moving image is small as a position of a part of the vehicle;
It is characterized by that.

  The generating unit generates an image signal for displaying the image portion extracted by the extracting unit at a certain position on the screen of the display device and displaying a guide line superimposed on the certain position on the screen. May be.

In order to achieve the above object, an image processing method according to a second aspect of the present invention includes:
An image processing method executed by an image processing apparatus including a storage unit, an identification unit, an extraction unit, and a generation unit,
The storage means
A predetermined that defines an area of the predetermined space on the captured image based on a position of the part of the vehicle on a captured image obtained by capturing an imaging space including a predetermined space including a part of the vehicle Store spatial location definition information,
The image processing method includes:
A specifying step in which the specifying unit specifies a position of a part of the vehicle on the captured image;
The extraction unit is configured to extract the predetermined space from the captured image based on the predetermined spatial position definition information stored in the storage unit and a position of a part of the vehicle on the captured image specified by the specifying unit. An extraction step of extracting an image portion indicating the space of
The generating unit includes a generating step of generating an image signal for causing the display device to display the image portion extracted by the extracting unit ;
The captured image is a moving image obtained by imaging the imaging space while the vehicle is moving by an imaging device installed in the vehicle,
In the specifying step, the specifying unit specifies a position of an image portion where a change in luminance of a pixel on the moving image is small as a position of a part of the vehicle.
It is characterized by that.

In order to achieve the above object, a program according to the third aspect of the present invention provides:
Computer
A predetermined that defines an area of the predetermined space on the captured image based on a position of the part of the vehicle on a captured image obtained by capturing an imaging space including a predetermined space including a part of the vehicle Storage means for storing spatial position definition information;
A specifying means for specifying a position of a part of the vehicle on the captured image;
An image indicating the predetermined space from the captured image based on the predetermined space position definition information stored in the storage unit and a position of a part of the vehicle on the captured image specified by the specifying unit. Extraction means for extracting the part,
Generating means for generating an image signal for causing the display device to display the image portion extracted by the extracting means;
A program that functions as
The captured image is a moving image obtained by imaging the imaging space while the vehicle is moving by an imaging device installed in the vehicle,
The specifying means specifies a position of an image portion where a change in luminance of a pixel on the moving image is small as a position of a part of the vehicle;
It is characterized by that.

  According to the image processing device, the image processing method, and the program according to the present invention, it is possible to generate an image signal for appropriately displaying an image around the vehicle.

1 is a configuration diagram of an image processing system including an image processing apparatus according to an embodiment of the present invention. It is a figure which shows the example of arrangement | positioning of each module with which an image processing system is provided. It is a block diagram for demonstrating the function of an image processing apparatus. It is the 1st figure showing typically signs that a camera picturizes an image. It is a 1st figure which shows typically the image displayed on a display apparatus. It is a 2nd figure which shows a mode that a camera captures an image typically. It is a 2nd figure which shows typically the image displayed on a display apparatus. It is a flowchart which shows the vehicle rear space display process which an image processing system performs. It is a flowchart which shows the detail of the rear bumper position specific process shown in the flowchart of FIG. It is a figure for demonstrating the identification method of the position of the upper end part of a rear bumper. It is a figure for demonstrating the method for an image processing apparatus to extract an imaging target image from a captured image. It is a figure which shows the synthesized image displayed on a display apparatus. It is a figure for demonstrating a feature point. It is a figure for demonstrating the method for the image processing apparatus which concerns on a modification to extract an imaging target image from a captured image. It is a 1st figure which shows the synthesized image which shows the rough standard of distance. It is a 2nd figure which shows the synthesized image which shows the rough standard of distance.

  Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings.

  First, the configuration of an image processing system including an image processing apparatus according to an embodiment of the present invention will be described with reference to FIG.

  As shown in FIG. 1, the image processing system 100 includes a camera 10, an illumination light source 20, an image processing device 30, a display device 50, and a vehicle speed sensor 60.

  The camera 10 captures an imaging space including a predetermined space including a part of the vehicle (hereinafter referred to as “imaging target”) at regular intervals (for example, 1/30 seconds), and is obtained by imaging. The obtained gradation images (hereinafter referred to as “captured images”) are sequentially output. The camera 10 outputs the captured image as an analog image signal, for example. The camera 10 is composed of, for example, a CCD (Charge Coupled Device) camera or the like. The illumination light source 20 illuminates the imaging target.

  The display device 50 displays an image according to the image signal supplied from the image processing device 30. The display device 50 includes, for example, an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube).

  The vehicle speed sensor 60 converts the speed of the vehicle into an electrical signal and outputs it. The vehicle speed sensor 60 is composed of, for example, a rotary encoder.

  The image processing apparatus 30 extracts an image portion corresponding to the imaging target from the captured image supplied from the camera 10 and synthesizes a reference line or the like with the extracted image portion (hereinafter referred to as “imaging target image”). An image signal for causing the display device 50 to display the composite image obtained in this manner is generated. For example, as shown in FIG. 1, the image processing apparatus 30 includes an A / D (Analog / Digital) converter 31, an image memory 32, a ROM (Read Only Memory) 33, and a CPU (Central Processing Unit) 34. , A RAM (Random Access Memory) 35, a display control unit 36, a light source control unit 37, a setting memory 38, an operation device 39, and a sensor input unit 40.

  The A / D converter 31 converts the analog image signal supplied from the camera 10 into a digital image signal and outputs it.

  The image memory 32 stores a captured image expressed by the digital image signal output from the A / D converter 31. The image memory 32 also stores an imaging target image and a composite image. The image memory 32 is composed of, for example, an SRAM (Static Random Access Memory).

  The ROM 33 stores a program for controlling the operation of the CPU 34. The CPU 34 controls the overall operation of the image processing apparatus 30. The CPU 34 processes the captured image supplied from the camera 10 by executing a program stored in the ROM 33. The RAM 35 functions as a work area for the CPU 34.

  The display control unit 36 generates an image signal from the composite image stored in the image memory 32 under the control of the CPU 34 and supplies the image signal to the display device 50. The light source control unit 37 controls turning on / off of the illumination light source 20.

  The setting memory 38 stores setting data necessary for the CPU 34 to execute a vehicle rear space display process to be described later.

  The operation device 39 receives an operation input from the driver and supplies an operation signal corresponding to the received operation input to the CPU 34. The operation device 39 includes various buttons. By operating these buttons, the driver displays the state behind the vehicle on the display device 50 or ends the display.

  The sensor input unit 40 receives an input of an electrical signal supplied from the vehicle speed sensor 60 and stores vehicle speed data indicated by the electrical signal in the RAM 35 or the like.

  Next, with reference to FIG. 2, how each module included in the image processing system 100 is installed in which part of the vehicle will be described. In addition, the installation location of each module demonstrated here is an illustration.

  For example, the camera 10 is installed in a vehicle body portion near the upper end portion of the license plate behind the vehicle 1000. The camera 10 is installed at a position and an angle at which a space obliquely below the rear of the vehicle 1000 can be imaged together with the upper part of the rear bumper 200. The illumination light source 20 is installed in a vehicle body part near the camera 10. The illumination light source 20 is installed at a position and an angle at which the space captured by the camera 10 can be illuminated.

  For example, the image processing device 30 is installed in front of the gear box so that a driver sitting in a driver's seat can operate the operation device 39. The display device 50 is installed in front of the gear box so that, for example, a driver sitting in the driver's seat can see the composite image. The vehicle speed sensor 60 is installed in the vicinity of the shaft portion of the front wheel, for example.

  Next, functions of the image processing apparatus 30 according to the embodiment of the present invention will be described with reference to FIG.

  As illustrated in FIG. 3, the image processing apparatus 30 includes a storage unit 110, a specification unit 120, an extraction unit 130, and a generation unit 140. The image processing device 30 processes the captured image supplied from the camera 10 to generate an image signal, and supplies the generated image signal to the display device 50.

  The camera 10 captures an imaging space including a predetermined space including a part of the vehicle to acquire a captured image, and supplies the acquired captured image to the specifying unit 120 and the extraction unit 130. A part of the vehicle is, for example, an upper end portion of the rear bumper 200, and the predetermined space is a space including, for example, a space obliquely below the rear of the vehicle 1000.

  The storage unit 110 stores predetermined space position definition information. Here, the predetermined space position definition information is information that defines an area indicating a predetermined space on the captured image on the basis of the position of a part of the vehicle on the captured image. The captured image is an image obtained by capturing an imaging space including a predetermined space including a part of the vehicle. For example, the predetermined space position definition information defines a space obliquely below the vehicle 1000 on the captured image and an area indicating the upper end portion of the rear bumper 200 on the basis of the position of the upper end portion of the rear bumper 200 on the captured image. Information. The storage unit 110 is configured by the setting memory 38, for example.

  The specifying unit 120 specifies the position of a part of the vehicle on the captured image generated by the camera 10. For example, the specifying unit 120 specifies the position of the upper end portion of the rear bumper 200 on the captured image. The specifying unit 120 is configured by the CPU 34, for example.

  The extraction unit 130 is an image indicating a predetermined space from the captured image based on the predetermined space position definition information stored in the storage unit 110 and the position of a part of the vehicle on the captured image specified by the specifying unit 120. Extract the part. The extraction unit 130 extracts, for example, an image portion showing a space obliquely below the rear of the vehicle 1000 and an upper end portion of the rear bumper 200 from the captured image. The extraction unit 130 is configured by the CPU 34, for example.

  The generation unit 140 generates an image signal for displaying the image portion extracted by the extraction unit 130 on a predetermined display device. The generation unit 140 generates, for example, an image signal for causing the display device 50 to display an image portion indicating a space obliquely below the rear of the vehicle 1000.

  The display device 50 displays an image indicated by the image signal supplied from the generation unit 140. For example, the display device 50 displays an image of a space obliquely below the rear of the vehicle 1000 indicated by the image signal supplied from the generation unit 140.

  Here, the function of the image processing apparatus 30 will be briefly described with reference to FIGS. 4A, 4B, 5A, and 5B.

  FIG. 4A is a diagram schematically illustrating how the camera 10 captures an imaging target. As shown in FIG. 4A, assuming that the vertical angle of view of the camera 10 (hereinafter referred to as “camera full angle of view”) is DegA, the vertical range captured by the camera 10 is two broken lines sandwiching DegA. It is the range between the line segments. When the vertical field angle of the display device 50 (hereinafter referred to as “screen display angle”) is DegB, the vertical range displayed on the display device 50 in the vertical range captured by the camera 10 is as follows. , The range is sandwiched between two solid line segments sandwiching DegB. In FIG. 4A, the camera 10 captures an image of a space (imaging region) centered on the direction indicated by the alternate long and short dash line. Further, the horizontal range imaged by the camera 10 is also limited to a predetermined range. Here, the range captured by the camera 10 includes the upper end portion of the rear bumper 200 that is a reference object for determining which range of the captured image is displayed on the display device 50. In the example shown in FIG. 4A, a mark 401 is located at a position 50 cm behind the vehicle 1000 on the road surface, a mark 402 is located 100 cm behind the vehicle 1000 on the road surface, and a mark 403 is located 200 cm behind the vehicle 1000 on the road surface. Are arranged, and marks 401 to 403 are included in the imaging range.

  FIG. 4B is a diagram schematically illustrating an image displayed on the display device 50. In FIG. 4B, the captured image generated by the camera 10 is an image shown in a portion surrounded by a dashed frame 410. On the other hand, in FIG. 4B, the composite image displayed on the display device 50 is an image shown in a portion surrounded by a thick frame 420. The synthesized image is an image obtained by synthesizing a predetermined reference line or the like with an imaging target image obtained by extracting a part of the captured image. Which part of the captured image is extracted as the captured image is based on the position of the reference object included in the captured image. In the present embodiment, the vertical position of the upper end of the rear bumper 200 is located in the captured image, and the space below the vehicle 1000 to be displayed on the display device 50 is defined as the upper end of the rear bumper 200. Based on the predetermined spatial position definition information defined with reference to the vertical position of the part, it is determined which part is to be extracted from the captured image as the captured image.

  In the example shown in FIG. 4B, the image in the range indicated by DegA is a captured image, and the image in the range indicated by DegB is a composite image. In the composite image, a reference line 441 indicating the road surface position 50 cm behind the vehicle 1000, a reference line 442 indicating the road surface position 100 cm behind the vehicle 1000, and a reference line 443 indicating the road surface position 200 cm behind the vehicle 1000 are shown. Is included. As described above, the mark 401 is located at a position 50 cm behind the vehicle 1000 on the road surface, the mark 402 is located 100 cm behind the vehicle 1000 on the road surface, and the mark 403 is located 200 cm behind the vehicle 1000 on the road surface. Each is arranged. For this reason, the composite image is an image displayed such that the mark 401 is arranged on the guide line 441, the mark 402 is placed on the guide line 442, and the mark 403 is placed on the guide line 443. A rear bumper 200 is shown below the composite image. In FIG. 4B, an image portion corresponding to the rear bumper 200 is clearly shown by hatching.

  FIG. 5A is a diagram schematically illustrating how the camera 10 captures an imaging target. In the example illustrated in FIG. 5A, the camera 10 is installed in the vehicle 1000 at an angle such that the lower space is the imaging range as compared to the example illustrated in FIG. 4A. Other than the installation angle of the camera 10 is the same as FIG.

  FIG. 5B is a diagram schematically illustrating an image displayed on the display device 50. Since the range in the vertical direction captured by the camera 10 is different between the example illustrated in FIG. 4A and the example illustrated in FIG. 5A, the captured image illustrated in FIG. 5B is different from the captured image illustrated in FIG. 4B. As shown in FIG. 5B, the composite image is the same as the composite image shown in FIG. 4B. This is because the example illustrated in FIG. 4B and the example illustrated in FIG. 5B are configured such that the same image to be captured is extracted from the captured image from different captured images.

  That is, in a conventional imaging device, when an imaging target is to be displayed on a display device, it is necessary to install a camera on the vehicle at an accurate position and an accurate angle at which the imaging target can be imaged. However, there may be a case where the installation position or the installation angle of the camera is deviated due to an installation error at the time of installation of the camera or aging deterioration. In this case, the imaging target cannot be accurately displayed on the display device. In contrast, in the image processing system 100, a wide area including the imaging target is captured by the camera 10 to generate a captured image, and the display device 50 displays only a portion corresponding to the imaging target in the captured image. Thereby, according to the image processing system 100, the imaging target can be accurately displayed on the display device 50 even when there is a slight shift in the installation position or the installation angle of the camera 10. The camera 10 is installed at a position and an angle at which the reference object is included in the imaging range.

  Next, the vehicle rear space display process executed by the image processing system 100 will be described using the flowchart shown in FIG.

  The image processing system 100 shown in FIG. 6 periodically (for example, every 1/30 second) after receiving a request to start display of the space behind the vehicle until receiving a request to end the display. Car rear space display processing is executed.

  First, the camera 10 images the space behind the vehicle (step S101). And CPU34 produces | generates a captured image based on the signal supplied from the camera 10 (step S102). Specifically, the CPU 34 supplies a captured image expressed by an analog image signal for one frame output from the camera 10 to the A / D converter 31, and outputs one frame for output from the A / D converter 31. The captured image represented by the digital image signal is stored in the image memory 32.

  Next, the CPU 34 executes a rear bumper position specifying process (step S103). The rear bumper position specifying process will be described in detail with reference to the flowchart shown in FIG. The rear bumper position specifying process is a process of specifying the vertical position of the upper end portion of the rear bumper 200 in the captured image.

  First, the vehicle speed of the vehicle 1000 is acquired from the vehicle speed sensor (step S201). Next, the CPU 34 determines whether or not the vehicle 1000 is moving (step S202). Specifically, the CPU 34 determines that the vehicle 1000 is moving when the absolute value of the vehicle speed acquired in step S201 is equal to or greater than a predetermined value, and the absolute value of the vehicle speed acquired in step S201 is a predetermined value. If it is less than that, it is determined that the vehicle 1000 is not moving.

  When determining that the vehicle 1000 is not moving (step S202: NO), the CPU 34 detects the position of the upper end portion of the rear bumper 200 by edge detection (step S203), and ends the rear bumper position specifying process. As a method of detecting the position of the upper end portion of the rear bumper 200 by edge detection, a conventional edge detection technique can be employed. A conventional edge detection technique is disclosed, for example, in FIG. 4 of Patent Document 3 (Japanese Patent Application Laid-Open No. 2004-1658) as a technique for detecting a boundary line of a rear bumper. Note that the position of the upper end portion of the rear bumper 200 is strictly the position of the boundary line of the upper end of the rear bumper 200 in the captured image.

  On the other hand, if it is determined that the vehicle 1000 is moving (step S202: YES), the pixel to be checked is set to the leftmost pixel in the first row from the bottom (step S204). Here, the pixel to be checked is a pixel to be checked for a difference between the luminance in the previous frame and the luminance in the current frame among the pixels constituting the captured image. In the present embodiment, the pixel to be checked is scanned from left to right in the horizontal direction and from bottom to top in the vertical direction.

  Next, the CPU 34 acquires the luminance of the pixel to be checked in the current frame (step S205). Specifically, the CPU 34 reads out the captured image of the current frame among the captured images stored in the image memory 32, and acquires the luminance of the check target pixel among the pixels constituting the captured image. Similarly, the CPU 34 acquires the luminance of the pixel to be checked in the previous frame (step S206). Specifically, the CPU 34 reads a captured image of a frame one frame before the captured image stored in the image memory 32, and acquires the luminance of the pixel to be checked among the pixels constituting the captured image. In addition, when each pixel is comprised from 3 colors of Red, Green, and Blue, CPU34 acquires the brightness | luminance of 3 colors, respectively.

  Then, the CPU 34 determines whether or not the absolute value of the luminance difference is greater than or equal to a predetermined value (step S207). Specifically, the CPU 34 determines whether or not the absolute value of the difference between the luminance acquired in step S205 and the luminance acquired in step S206 is a predetermined value or more, and stores the determination result in the RAM 35. For example, when the luminance of each pixel is indicated by a value of 0 to 255, the predetermined value is set to a value of about 5 to 50. When each pixel is composed of three colors of Red, Green, and Blue, the CPU 34 takes the absolute value of the luminance difference for each color, and the sum of the absolute values of the luminance differences for each color is greater than or equal to a predetermined value. It is determined whether or not there is.

  When the CPU 34 finishes the process of determining whether or not the absolute value of the luminance difference is greater than or equal to a predetermined value (step S207), it determines whether or not the pixel to be checked is the rightmost pixel (step S208). . When the CPU 34 determines that the pixel to be checked is not the rightmost pixel (step S208: NO), the CPU 34 sets the pixel to be checked one right pixel (step S209), and the pixel to be checked in the current frame. The process returns to the process of acquiring the brightness (step S205).

  On the other hand, if the CPU 34 determines that the pixel to be checked is the rightmost pixel (step S208: YES), the CPU 34 determines whether or not the proportion of pixels whose absolute value of luminance difference is equal to or greater than a predetermined value is less than the predetermined proportion. (Step S210). Specifically, the CPU 34 has a predetermined ratio (the number of pixels in which the absolute value of the luminance difference is equal to or larger than a predetermined value among the pixels included in the row from the row including the pixel to be checked to the second lower row). For example, it is determined whether it is less than 80%. If the CPU 34 determines that the ratio of the pixels whose luminance difference absolute value is equal to or larger than the predetermined value is not less than the predetermined ratio (step S210: NO), the CPU 34 sets the pixel to be checked to the leftmost pixel on one line. (Step S211), the process returns to the process of acquiring the luminance of the pixel to be checked in the current frame (Step S205). On the other hand, when the CPU 34 determines that the ratio of the pixels whose luminance difference absolute value is equal to or larger than the predetermined value is less than the predetermined ratio (step S210: YES), the RAM 35 stores information indicating the row including the pixel to be checked. (Step S212), and the rear bumper position specifying process is terminated. Note that the row including the pixel to be checked is treated as a row indicating the position of the upper end portion of the rear bumper 200.

  Here, how the position of the upper end portion of the rear bumper 200 on the captured image is specified in the rear bumper position specifying process will be described with reference to FIG.

  In FIG. 8, the captured image is an image shown in a portion surrounded by a broken line frame 410. On the other hand, in FIG. 8, the imaging target image is a portion surrounded by a thick frame 420. In FIG. 8, a portion corresponding to the rear bumper 200 is hatched. Here, both the camera 10 and the rear bumper 200 are fixed to the vehicle 1000, and the position of the rear bumper 200 on the captured image does not change even when the vehicle 1000 is moving. For this reason, the portion corresponding to the rear bumper 200 has a small change in luminance between the current frame and the previous frame. On the other hand, in a portion other than the portion corresponding to the rear bumper 200 on the captured image, a change in luminance is generally large between the current frame and the previous frame.

  Here, in FIG. 8, a portion surrounded by a thick frame 450 is a portion indicating a pixel to be checked, and corresponds to one pixel. First, the pixel to be checked is set to the leftmost pixel of the bottom row (k1) in step S204. Then, the pixels to be checked are shifted to the right pixel one by one in step S209. When the pixel to be checked is the rightmost pixel, the pixel to be checked is set to the leftmost pixel in the next row in step S211. In addition, before the pixel to be checked is set as the leftmost pixel in the upper row, in step S210, the luminance of pixels included in the row from the row including the pixel to be checked to the second lower row is determined. It is checked whether or not the ratio of pixels whose absolute value of the difference is greater than or equal to a predetermined value is less than the predetermined ratio.

  In the present embodiment, pixels whose absolute value of luminance difference is less than a predetermined value are set as still pixels, and the ratio of still pixels in one row is not checked, but the ratio of still pixels in one row is equal to three rows. The reason for checking the ratio of the moving average of is to remove noise. In FIG. 8, in the rows from k1 to k6, the ratio of still pixels is 100%, the ratio of still pixels in the k7 row is 90%, the ratio of still pixels in the k8 row is 80%, and k9 The ratio of still pixels in row k is 75%, the ratio of still pixels in row k10 is 70%, the ratio of still pixels in row k11 is 60%, and the ratio of still pixels in row k12 is 25%, the proportion of still pixels in the row k13 is 5%, and the proportion of still pixels in the rows after k14 is 0%. Therefore, for example, after checking the pixel to be checked at the right end of k7, whether or not 96.7%, which is the average value of the ratio of still pixels for each of the rows k5 to k7, is less than 80%. Determined. Similarly, after checking the pixel to be checked at the right end of k8, it is determined whether 90%, which is the average value of the ratio of still pixels for each of the rows k6 to k8, is less than 80%. After checking the pixel to be checked at the right end of k9, it is determined whether or not 81.7% that is the average value of the ratio of still pixels for each of the rows k7 to k9 is less than 80%. After checking the rightmost pixel to be checked, it is determined whether or not 75%, which is the average value of the ratio of still pixels for each of the rows k8 to k10, is less than 80%. Then, k9, which is a row determined for the first time that the average value of the ratio of still pixels is less than 80%, is stored as a row corresponding to the upper end portion of the rear bumper 200 in step S212.

  Note that the ratio of still pixels for three rows may be directly determined without determining the ratio of still pixels for each row. In addition, for the first and second lines, it is not necessary to execute processing for determining whether or not the ratio of still pixels is less than a predetermined ratio, and the ratio of still pixels in only existing rows is averaged. Then, it may be determined whether the ratio is less than a predetermined ratio.

  Hereinafter, the description returns to the flowchart. When completing the rear bumper position specifying process (step S103), the CPU 34 acquires predetermined space position definition information stored in advance in the setting memory 38 (step S104). As described above, the predetermined space position definition information is information that defines, for example, the position of the space obliquely below the rear of the vehicle 1000 in the captured image with reference to the position of the upper end portion of the rear bumper 200 in the captured image. . For example, when the captured image is cut out from the captured image so that the upper end of the rear bumper 200 is positioned at 20% from below the captured image, information indicating 20% can be used as the predetermined spatial position definition information. .

  When the CPU 34 completes the acquisition of the predetermined space position definition information (step S104), the captured image is obtained based on the position of the upper end portion of the rear bumper 200 specified in step S103 and the predetermined space position definition information acquired in step S104. An image to be captured is extracted from (step S105). As described above, when the predetermined space position definition information is 20%, as shown in FIG. 9A, the captured image is such that the position of the upper end of the rear bumper 200 is 20% from the lower side of the image to be captured. The image to be captured is cut out from the image.

  Next, the CPU 34 combines the reference lines 441 to 443 with the imaging target image extracted in step S105 to generate a combined image (step S106). And CPU34 supplies the image signal which shows the synthesized image produced | generated in step S106 to the display apparatus 50, and displays a synthesized image on the display apparatus 50 (step S107). FIG. 9B shows an example of a composite image. CPU34 complete | finishes a vehicle back space display process, after complete | finishing the process (step S107) which displays a synthesized image.

  According to the image processing apparatus 30 according to the present embodiment, an image of a desired region can be appropriately displayed even when the installation position and the installation angle of the camera 10 are shifted in the vertical direction. In addition, the position on the screen where the reference line is displayed is set to a fixed position regardless of the fixed position and fixed angle of the camera 10. For this reason, it is not necessary to perform processing such as recalculating the position on the screen where the reference line is displayed in accordance with the amount of deviation.

(Modification)
In the above embodiment, the vertical range of the captured image captured by the camera 10 is set to a range wider than the vertical range of the composite image (or the image to be captured) displayed on the display device 50, and the image is captured. In the example, the vertical display range is determined based on the position of the upper end of the rear bumper 200 in the image, and only the determined vertical range of the captured image is displayed on the display device 50. However, according to the present invention, the horizontal range of the captured image is set to be wider than the horizontal range of the composite image, and the horizontal display range is set based on the position of the predetermined reference object in the captured image. It is also possible to display only the determined horizontal range in the captured image on the display device 50. Also, the vertical range of the captured image may be set to a range wider than the vertical range of the composite image, and the horizontal range of the captured image may be set to a range wider than the horizontal range of the composite image. Good. As described above, by setting the range of the captured image wider than the range of the composite image, an image of a desired region can be appropriately displayed even when the installation position and the installation angle of the camera 10 are shifted.

  For example, the imaging target image is extracted from the captured image so that the feature points detected by edge detection based on the captured image are displayed at the desired vertical position and the desired horizontal position on the imaging target image. Can be configured to. As shown in FIG. 10A, it is assumed that a boundary line 451 at the upper end of the rear bumper 200 is detected by edge detection based on the captured image. At this time, it is assumed that the point having the lowest vertical position among the points on the boundary line 451 is the feature point 452.

  Here, a predetermined spatial position definition is defined in the setting memory 38 so that the position of the feature point 452 on the image to be captured is 20% from below in the vertical direction and 50% from the left in the horizontal direction. Assume that information is stored. At this time, the CPU 34 extracts the imaging target image from the captured image so that the position of the feature point 452 in the imaging target image is 20% from below in the vertical direction and 50% from the left in the horizontal direction. .

  FIG. 10B shows the relationship between the captured image and the captured image. In FIG. 10B, the captured image is an image shown in a part surrounded by a broken line frame 410, and the imaging target image is an image shown in a part surrounded by a thick line frame 420. The position of the feature point 452 in the captured image is 20% from below and 50% from the left. In this manner, the imaging target image is extracted from the captured image so that the position of the feature point 452 in the imaging target image becomes a position defined by the predetermined spatial position definition information. Note that the sizes of the captured image and the captured image can be adjusted as appropriate. For example, the ratio of the size of the captured image to the size of the image to be captured is increased as the predicted displacement of the installation position or the installation angle of the camera 10 on the vehicle 1000 is larger.

  In the present embodiment, a composite image indicating a distance from the vehicle 1000 (a distance from the camera 10) is generated by combining a narrow reference line with the image to be captured. However, as shown in FIG. 11, the standard of distance from the vehicle 1000 may be indicated using a wide standard line. In the example shown in FIG. 11, a reference line 461 indicating a position 50 cm from the vehicle 1000, a reference line 462 indicating a position 100 cm from the vehicle 1000, and a reference line 463 indicating a position 200 cm from the vehicle 1000 are shown. If the mounting accuracy of the bumper is not good, the reference line may not be displayed accurately. However, by widening the width of the reference line to some extent in this way, it is possible to allow the driver to perform a driving operation with a margin. Note that the color and pattern of each guide line can be appropriately adjusted and displayed. Typically, the closer the distance from the vehicle 1000 indicated by the reference line, the more noticeable colors and patterns can be drawn to attract the driver's attention. For example, the guide line 461 can be red, the guide line 462 can be yellow, and the guide line 463 can be blue. When the width of the reference line is increased, the reference line may be displayed semi-transparently so that the background image is not completely hidden.

  In the present embodiment, a combined image indicating a distance from the vehicle 1000 (a distance from the camera 10) is generated by combining a reference line with the image to be captured. However, as an indication of the distance from the vehicle 1000, a reference area may be displayed instead of the reference line. In the example shown in FIG. 12, a region 471 indicating a region near 50 cm from the vehicle 1000, a region 472 indicating a region near 100 cm from the vehicle 1000, and a region 473 indicating a region near 200 cm from the vehicle 1000 are shown. In this way, by indicating the rough distance from the vehicle 1000, it is possible to allow the driver to perform a driving operation with a margin. Note that, similarly to the reference line, the color and pattern of each region can be appropriately adjusted and displayed. For example, the area 471 can be displayed in red as a dangerous area, the area 472 can be displayed in yellow as a caution area, and the area 473 can be displayed in blue as a safety area.

  In the above embodiment, when the ratio of pixels with a large change in luminance is obtained, the ratio is obtained for pixels for three rows, but the ratio may be obtained for pixels for one row or two rows. You may obtain | require a ratio about the pixels for a line or more. Moreover, although the example which calculates | requires the difference of a brightness | luminance for every pixel was shown in the said embodiment, the reference | standard thing is calculated | required by calculating | requiring the sum total of the brightness | luminance for every block comprised from several pixels, and calculating | requiring the brightness | luminance difference for every block. (Stationary object) may be specified. Further, the reference object may be specified by obtaining the sum of luminance for each row and obtaining the difference in luminance for each row.

  In the above embodiment, the difference in luminance between the current frame and the previous frame is obtained, but the frame to be compared can be changed as appropriate. For example, a luminance difference may be obtained between the current frame and 10 frames before. In addition, a portion where the luminance does not change continuously over several frames may be specified as a portion indicating the reference object.

  In the above-described embodiment, an example in which a composite image in which a reference line or the like is combined with an imaging target image extracted from the captured image is displayed. However, the captured image itself may be displayed.

  In the above embodiment and the modification, the predetermined space position definition information indicates the position of the reference object on the composite image in% display. However, the predetermined spatial position definition information includes, for example, the imaging target image of the imaging target depending on the position of the predetermined portion of the reference object on the composite image, the number of rows from the bottom or top, or the column from the left or right It may be information defining a position on the top. The number of rows or the number of columns is synonymous with the number of pixels. The reference object may indicate a reference line as in the embodiment, or may indicate a reference point as in the modification.

  In the above embodiment, a portion with a small difference in brightness in the captured image is specified as a portion corresponding to the rear bumper. Further, in the modification, the method of detecting the upper end portion of the rear bumper by edge detection and specifying the portion corresponding to the rear bumper in the captured image is shown. However, the method for specifying the portion corresponding to the reference object (rear bumper) in the captured image is not limited to these methods. For example, information indicating the color of the rear bumper may be stored in advance in the setting memory 38 or the like, and a pixel portion indicating the color in the captured image may be specified as a portion corresponding to the rear bumper.

  Note that the image processing apparatus according to the present invention can be realized using a normal computer system, not a dedicated system. For example, a program for performing the above operation on a computer capable of inputting an image includes a flexible disk, a CD-ROM (Compact Disk Read-Only Memory), a DVD (Digital Versatile Disk), an MO (Magneto Optical Disk), etc. An image processing apparatus that performs the above-described processing may be configured by storing and distributing in a computer-readable recording medium and installing the recording medium in a computer system.

  Furthermore, the program may be stored in a disk device or the like included in a server device on the Internet, and may be downloaded onto a computer by being superimposed on a carrier wave, for example.

  The present invention can be used for, for example, an in-vehicle imaging device.

DESCRIPTION OF SYMBOLS 10 Camera 20 Illumination light source 30 Image processing apparatus 31 A / D converter 32 Image memory 33 ROM
34 CPU (identifying means, extracting means, generating means)
35 RAM
36 Display Control Unit 37 Light Source Control Unit 38 Setting Memory (Storage Unit)
39 Operating device 40 Sensor input unit 50 Display device 60 Vehicle speed sensor 100 Image processing system 110 Storage unit 120 Identification unit 130 Extraction unit 140 Generation unit 200 Rear bumper 1000 Vehicle

Claims (4)

A predetermined that defines an area of the predetermined space on the captured image based on a position of the part of the vehicle on a captured image obtained by capturing an imaging space including a predetermined space including a part of the vehicle Storage means for storing spatial position definition information;
A specifying means for specifying a position of a part of the vehicle on the captured image;
An image indicating the predetermined space from the captured image based on the predetermined space position definition information stored in the storage unit and a position of a part of the vehicle on the captured image specified by the specifying unit. Extraction means for extracting the part;
Generating means for generating an image signal for causing the display device to display the image portion extracted by the extracting means ;
The captured image is a moving image obtained by imaging the imaging space while the vehicle is moving by an imaging device installed in the vehicle,
The specifying means specifies a position of an image portion where a change in luminance of a pixel on the moving image is small as a position of a part of the vehicle;
An image processing apparatus. The generation unit generates an image signal for displaying the image portion extracted by the extraction unit at a certain position on the screen of the display device and displaying a guide line superimposed on the certain position on the screen. ,
The image processing apparatus according to claim 1 . An image processing method executed by an image processing apparatus including a storage unit, an identification unit, an extraction unit, and a generation unit,
The storage means
A predetermined that defines an area of the predetermined space on the captured image based on a position of the part of the vehicle on a captured image obtained by capturing an imaging space including a predetermined space including a part of the vehicle Store spatial location definition information,
The image processing method includes:
A specifying step in which the specifying unit specifies a position of a part of the vehicle on the captured image;
The extraction unit is configured to extract the predetermined space from the captured image based on the predetermined spatial position definition information stored in the storage unit and a position of a part of the vehicle on the captured image specified by the specifying unit. An extraction step of extracting an image portion indicating the space of
The generating unit includes a generating step of generating an image signal for causing the display device to display the image portion extracted by the extracting unit ;
The captured image is a moving image obtained by imaging the imaging space while the vehicle is moving by an imaging device installed in the vehicle,
In the specifying step, the specifying unit specifies a position of an image portion where a change in luminance of a pixel on the moving image is small as a position of a part of the vehicle.
An image processing method. Computer
A predetermined that defines an area of the predetermined space on the captured image based on a position of the part of the vehicle on a captured image obtained by capturing an imaging space including a predetermined space including a part of the vehicle Storage means for storing spatial position definition information;
A specifying means for specifying a position of a part of the vehicle on the captured image;
An image indicating the predetermined space from the captured image based on the predetermined space position definition information stored in the storage unit and a position of a part of the vehicle on the captured image specified by the specifying unit. Extraction means for extracting the part,
Generating means for generating an image signal for causing the display device to display the image portion extracted by the extracting means;
A program that functions as
The captured image is a moving image obtained by imaging the imaging space while the vehicle is moving by an imaging device installed in the vehicle,
The specifying means specifies a position of an image portion where a change in luminance of a pixel on the moving image is small as a position of a part of the vehicle;
A program characterized by that.

Images