JP6123377B2 - Image processing apparatus, object detection method, and object detection program - Google Patents

Description

  The present invention relates to a technique for detecting an object located in a detection area by processing a distance image of the detection area, which is picked up by receiving reflected light of light irradiated to the detection area.

  Conventionally, in order to prevent passengers from falling from the station platform into the track, railway companies have been proceeding with installation of fall prevention fences along the side edges of the station platform. This fall prevention fence needs to secure a passage for passengers to get on and off the train stopped at the station platform.

  As a fall prevention fence, a slide door that slides horizontally and opens and closes at a position facing a train door (vehicle door) stopped at a station platform (see Patent Document 1), or both sides of an entrance / exit A fixed column is erected on the fixed column, and a plurality of stop bars are provided between the movable columns that are slidable in the vertical direction with respect to the fixed column. See).

  The fall prevention fence is temporarily opened to secure a passage for passengers after the train stops at the station platform. The train departs after passengers get on and off and the fall prevention fence is closed. In order to ensure safety when the train departs, the presence or absence of an object located between the train and the fall prevention fence is detected by a sensor or the like. Specifically, an object located between the train and the fall prevention fence after passengers getting on and off the train stopped at the station platform is closed and the door of the train and the fall prevention fence are closed. When is detected, the station staff or the like is confirmed without starting the train.

  As a sensor for detecting an object located between the train and the fall prevention fence, a transmission type or a reflection type photoelectric sensor is used from the viewpoint of preventing a blind spot where the presence or absence of the object cannot be detected. A configuration using an image sensor is more common than a configuration.

JP 2000-16280 A International Publication No. 2011/024612

  However, in a configuration using an image sensor, a pixel affected by raindrops (raindrop effect pixel) may occur during rain. The output of the raindrop-affected pixels is noise. Therefore, as the number of raindrop-affected pixels increases, the possibility that an object is erroneously detected increases (in fact, the possibility that an object is detected even though no object exists) increases.

  When an object located between the train and the fall prevention fence is detected, the train driver does not start the train until the station staff or the like confirms safety. Therefore, if an object (an object that does not actually exist) located between the train and the fall prevention fence is erroneously detected during the rain due to the output of the raindrop influence pixel that is noise, the train stops at the station. Time is wasted and disturbed. Moreover, the work burden of the station staff who confirms safety will be increased.

  An object of the present invention is to provide a technique capable of sufficiently suppressing erroneous detection of an object due to the generation of raindrop-affected pixels affected by raindrops during rain.

  The image processing apparatus of the present invention is configured as follows in order to solve the above-described problems and achieve the object.

  The image acquisition unit acquires, as a pair of captured images, a distance image of the detection area and a received light intensity image captured by the imaging device irradiating the detection area with light and receiving the reflected light.

  In addition, the image acquisition part should just be the structure which acquires the distance image of a detection area at least.

  For example, a known TOF (Time Of Flight) camera may be used as the imaging device, a light source that emits laser light, a light receiving element that receives reflected light, and laser light emitted from the light source in the detection area. It is good also as a structure which has the scanning part to scan. The imaging device may be configured so that the distance image of the detection area and the received light intensity image can be captured at the same timing.

  Note that the TOF camera has been attracting attention because it can obtain a distance image with a single eye and can be downsized at a lower cost than a stereo image processing method.

  Moreover, the image acquisition part should just be the structure which acquires the distance image of a detection area at least (it does not acquire a received light intensity image). In this case, the imaging device may be configured to capture a distance image of the detection area.

  The raindrop-affected pixel determination unit uses a distance image acquired by the image acquisition unit as a pair of captured images and each pixel of the received light intensity image, and the raindrop affected by the raindrop using the distance and received light intensity acquired at that pixel. It is determined whether the pixel is an influence pixel. The raindrop-affected pixels receive, for example, rain pixels that have detected reflected light from raindrops, or raindrop-attached pixels that have been affected by raindrops attached to the imaging lens (reflected light transmitted through the raindrops that have adhered to the imaging lens) Pixel).

  The pixel referred to in the present invention may be one light receiving element (1 pixel) of the image pickup element, or a plurality of adjacent light receiving elements (for example, two light receiving elements (2 × 2 pixels) in the vertical and horizontal directions). The block comprised by may be sufficient. The image processing apparatus according to the present invention processes the distance image and the received light intensity image in units of pixels referred to herein (not necessarily in units of light receiving elements).

  Reflected light from raindrops detected during rainfall is reflected at a relatively close position. Considering the reflectance of raindrops, the reflected light having a light reception intensity greatly different from that of the raindrops is not likely to be reflected by raindrops but is likely to be reflected light reflected by an object. Therefore, the determination accuracy can be improved by determining whether or not the pixel is a rain pixel in which reflected light from the raindrop is detected using the distance and the received light intensity acquired in the pixel.

  Further, the inventor of the present application has confirmed the tendency of distance and received light intensity by experiment for raindrop-attached pixels affected by raindrops attached to the imaging lens. It was confirmed that the received light intensity of the raindrop adhesion pixel was lowered. It was confirmed that most of the raindrop-attached pixels were within a certain range. This is considered to be a phenomenon caused by attenuation of the amount of reflected light accompanying transmission of raindrops attached to the imaging lens. In addition, it was confirmed that most of the raindrop adhesion pixels are within a range where the distance changes. This is considered to be a phenomenon that occurs when reflected light is refracted during the transmission of raindrops attached to the imaging lens, and pixels that receive the reflected light shift to adjacent pixels or neighboring pixels.

  Therefore, by determining whether or not the pixel is a raindrop-attached pixel affected by the raindrop attached to the imaging lens based on the above-described tendency of the distance and the received light intensity confirmed by the experiment on the raindrop-attached pixel, the determination accuracy is increased. It can be improved.

  It should be noted that the tendency of the change in distance and the change in the received light intensity in the raindrop-attached pixels affected by the raindrops attached to the imaging lens is considered to vary depending on factors such as the characteristics of the image acquisition unit and the imaging environment. It is preferable to set a determination criterion for determining whether the pixel is a raindrop-attached pixel by confirming these tendencies at the time of installation.

  In addition, whether or not the pixel is a raindrop-affected pixel can be determined using one of the distance and the received light intensity acquired in the pixel. However, in order to ensure sufficient determination accuracy, both the distance and the received light intensity are set. It is preferable to use it. In other words, the image acquisition unit is preferably configured to acquire the distance image of the detection area and the received light intensity image as a pair of captured images (the imaging device has the same timing for the distance image of the detection area and the received light intensity image). It is preferable that the image can be captured with a

The object detection unit processes the detection distance image using the determination result of the raindrop-affected pixel determination unit, and detects an object imaged in the detection distance image. The object detection unit can detect an object located in the detection area without being affected by the raindrop-affected pixels that are noise by not using the raindrop-affected pixels as pixels that have detected reflected light from the object. .
Moreover, based on the pixel number which the raindrop influence pixel determination part determined to be a raindrop influence pixel, it is determined whether a rain state determination part is a rain state. Further, the object detection unit increases the lower limit of the size of the object to be detected when the rain state determination unit determines that it is raining than when the rain state determination unit determines that it is not raining. Therefore, even if raindrop-affected pixels are concentrated to some extent during rain, object detection errors in the object detection unit can be sufficiently suppressed.
The reference image storage unit may store the reference distance image of the detection area and the reference light reception intensity image as a pair of reference images. In this case, the raindrop-affected pixel determination unit determines whether each pixel is a raindrop-affected pixel by using the distance image and the received light intensity image acquired by the imaging device and acquired by the image acquisition unit as a pair of detection images. The object detection unit may be configured to generate a difference image between the reference distance image and the distance image of the detection image and detect the imaged object.

  As a result, structures (posts, walls, etc.) located in the detection area, trains, cars, etc. that are stopped in the detection area are used as the background in the detection area, and these are not detected as objects.

In addition, the raindrop-affected pixel determination unit determines the reflected light from the raindrop for each pixel based on the difference in the received light intensity calculated using the received light intensity of the pixel in the reference received light intensity image and the received light intensity image of the detection image. It is good also as a structure which determines whether it is the detected rain pixel.

  Furthermore, the raindrop-affected pixel determination unit uses, for each pixel, the distance acquired in the pixel in the distance image of the detection image and the light reception intensity of the pixel in the reference light reception intensity image and the light reception intensity image of the detection image. Based on the difference in the received light intensity calculated in the above, it may be configured to determine whether or not the pixel is a rain pixel in which reflected light from the raindrop is detected.

  The reflectance of raindrops varies depending on the reflectance of an object (including the background) located behind the raindrop. Therefore, by using the difference in the received light intensity calculated by using the received light intensity of the pixel in the reference received light intensity image and the received light intensity image of the detection image, it is determined whether the pixel is a rain pixel in which reflected light from the raindrop is detected. The determination can be made in consideration of the reflectance of an object (including the background) located behind the raindrop. Accordingly, it is possible to further improve the accuracy of determining whether or not the pixel is a rain pixel that has detected reflected light from raindrops, and as a result, it is possible to further improve the accuracy of detecting an object located in the detection area.

  Further, as described above, it has been confirmed that most of the raindrop adhesion pixels are within a range where the amount of decrease in received light amount is within a certain range and within a range where there is a change in distance.

  Therefore, a pixel in which the difference in received light intensity calculated using the received light intensity of the pixel between the reference received light intensity image and the received light intensity image of the detection image is within a set range (within a range in which the amount of decrease in received light amount falls). Is determined as a raindrop-attached pixel, the determination accuracy of the raindrop-attached pixel can be improved, and as a result, the detection accuracy of an object located in the detection area can be further improved.

  In addition, by determining a pixel in which the difference in the distance between the reference light intensity image and the light intensity image of the detection image is within a set range (within a range in which the change in distance is within) as a raindrop adhesion pixel Therefore, it is possible to improve the determination accuracy of the raindrop adhesion pixels, and as a result, it is possible to further improve the detection accuracy of the object located in the detection area.

  Whether or not the pixel is a raindrop-attached pixel may be determined using either the difference in distance or the difference in received light intensity, but is determined using both the difference in distance or the difference in received light intensity. Is preferred.

  In addition, the reference image generation unit captures the reference distance image based on the distance image for one or a plurality of frames and the received light intensity image captured by the imaging device at a predetermined reference image acquisition timing and acquired by the image acquisition unit, and A configuration may be used in which the reference received light intensity image is generated as a pair of reference images.

  In particular, by generating a reference image based on multiple frames of distance images and received light intensity images, noise contained in this reference image can be suppressed, and as a result, detection accuracy of objects located within the detection area Can be further improved.

Further, the object detection unit may determine the lower limit of the size of the object to be detected according to the number of pixels that the raindrop influence pixel determination unit has determined to be raindrop influence pixels, that is, the amount of rainfall. The number of raindrop-affected pixels increases as the amount of rainfall increases. Therefore, not only is it possible to prevent erroneous detection of raindrop-affected pixels, which are noise, as an object located within the detection area when the rainfall is relatively high, but also the reflection from the object when the rainfall is relatively low. A plurality of adjacent pixels that have detected light are erroneously determined to be raindrop-affected pixels, and as a result, occurrence of a situation in which an object located in the detection area is missed can be suppressed.

Further, the object detecting method according to the present invention, the image acquisition unit of the above, the raindrop impact pixel determining unit, object detection unit, and an invention for executing the process corresponding to the configuration of the rainfall state determination unit to the computer.

Further, the object detecting program according to the present invention, by installing the computer, the image acquisition unit of the above, the raindrop pixel determining unit, object detection unit, and the processing corresponding to the configuration of the rainfall state determination unit, in the computer It is an invention to be executed.

  According to the present invention, it is possible to sufficiently suppress erroneous detection of an object due to the generation of raindrop-affected pixels that are affected by raindrops during rainfall.

It is the schematic which shows the station platform in which the fall prevention fence is installed. It is a block diagram which shows the structure of the principal part of an image processing apparatus. It is a figure explaining the known principle which calculates | requires the distance to an object. It is a figure which shows the example of attachment of a TOF camera. It is a flowchart which shows operation | movement of an image processing apparatus. It is a flowchart which shows a reference | standard image acquisition process. It is a flowchart which shows an object detection process. It is a figure explaining the determination method of whether it is a rain pixel. It is a figure which shows the experimental result concerning a raindrop adhesion pixel. It is a figure explaining the determination method of whether it is a raindrop adhesion pixel. It is a flowchart which shows the rain condition determination process concerning another example.

  Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described.

  The image processing apparatus according to this example has a fall prevention fence installed along the side edge of the station platform between the train and the train to prevent passengers from falling from the station platform into the track. The space is the detection area. The image processing apparatus detects an object located in this detection area.

  First, the fall prevention fence installed along the side edge part of the station platform will be briefly described.

  FIG. 1 is a schematic diagram showing a station platform where a fall prevention fence is installed. FIG. 1A is an overhead view of the station platform, and FIG. 1B is a plan view of the station platform viewed from the track side. The fall prevention fence includes a housing 1 that functions as a door pocket, and a sliding door 2 that is slidably attached to the housing 1. FIG. 1 shows a state in which the slide door 2 is closed. The slide door 2 is provided at a position where each door of a train that stops at the installed station platform is opposed. When the sliding door 2 is opened, it is housed in the housing 1 (door pocket). The slide door 2 slides in the left-right direction in FIG.

  The detection area where the image processing apparatus according to this example detects an object is between the fall prevention fence and the track at the position where the slide door 2 is provided. In this example, the image processing apparatus is provided in each slide door 2.

  In addition, although the thing shown by the patent document 1 mentioned above is illustrated as a fall prevention fence here, the thing shown by the patent document 2 mentioned above may be sufficient.

  FIG. 2 is a block diagram showing the configuration of the main part of the image processing apparatus according to this example. The image processing apparatus 10 includes a control unit 11, an image sensor unit 12, an image processing unit 13, and an output unit 14. The image processing apparatus 10 can use an information processing apparatus such as a personal computer having the above-described configuration as hardware. The information processing apparatus used as hardware executes the processing described later (processing according to the flowcharts shown in FIGS. 5, 6, 7, and 11) by installing the object detection program referred to in the present invention.

  The control unit 11 controls the operation of each part of the main body of the image processing apparatus 10.

  The image sensor unit 12 includes a TOF (Time Of Flight) camera that captures a distance image of the detection area and a received light intensity image by irradiating the detection area with infrared light and receiving the reflected light. This TOF camera corresponds to the imaging device referred to in the present invention.

  The TOF camera has a light source that irradiates infrared light to a detection area (imaging area) and an imaging element (n × m pixel imaging element) in which n × m light receiving elements are arranged in a matrix. The TOF camera measures the time (flight time) from irradiating infrared light to a detection area until receiving reflected light for each pixel. The TOF camera obtains the distance to the object (the reflection surface of the light-reflected target object) by measuring the phase difference between the light applied to the detection area and the received reflected light.

Explaining the known principle for obtaining the distance to the object, the light emitted from the light source uses light whose intensity is modulated. When the reflected light from the detection area is received, the modulation phase is shifted according to the propagation distance. The phase of the irradiated light is monitored by directly receiving a part of the light from the light source by a part of the light receiving element, and the deviation from the phase of the light received as reflected light is obtained. As shown in FIG. 3A, the known principle for obtaining the phase shift is the received light signal (A0, A2) sampled every T / 2 period with respect to the modulation period T of the irradiation light, and further FIG. ), The phase shift amount φ is calculated based on the propagation distance based on the received light signals (A1, A3) sampled at the timing shifted by T / 4. The amount of phase shift φ is
φ = arctan {(A3-A1) / (A0-A2)}
Can be calculated.

Further, the distance D from the phase shift φ obtained here to the object can be obtained. The distance D to the object is
D = Lmax × φ / 2π
Can be calculated. Here, Lmax is the distance to the object when φ = 2π (maximum measurement distance), and Lmax is 7.5 m if the modulation frequency is 20 MHz and 15 m if 10 MHz.

  In the description of the image pickup element, two adjacent light receiving elements (four pixels in total) are arranged as one set, and this is handled as one pixel. Each light receiving element (pixel) accumulates the photoelectrically converted charge at the sampling timing shifted every T / 4 period. As a result, the light receiving signals A0, A1, A2, and A3 described above can be obtained based on the accumulated charge for each T / 4 period. As another known technique, there is a method in which two light receiving elements are treated as one pixel and the above-described light receiving signals A0 to A3 are obtained.

As described above, the “pixel” is a block of light receiving elements (pixels), which is a unit when performing image processing to detect an object by obtaining a phase shift or distance, and It may be a single light receiving element (1 pixel) or a block composed of a plurality of adjacent light receiving elements (for example, two light receiving elements (2 × 2 pixels) vertically and horizontally). The image processing apparatus according to the present invention processes the distance image and the received light intensity image in units of pixels referred to herein (not necessarily in units of light receiving elements).

  In actual image processing in a TOF camera, the amount of light received for one period of the above-described modulation of irradiation light is too small in quantity, so the exposure time of the camera is set appropriately and the amount of charge accumulated during that period is used. The phase shift is calculated to find the distance to the object. The distance obtained by associating the distance to the reflecting surface for each pixel by collecting all the distance information for each pixel (meaning a block as a unit for image processing as described above). Get an image. In addition, the TOF camera collects all accumulated charges for a predetermined period (for a plurality of cycles) for each pixel, thereby obtaining a received light intensity image in which the intensity of the reflected light (reflected light amount) received by the pixel is associated with each pixel. get. The TOF camera can obtain a distance image of the detection area and a received light intensity image captured at the same exposure timing (exposure period).

  Instead of the above-described TOF camera, an imaging device having a light source that emits laser light, a light receiving element that receives reflected light, and a scanning unit that scans laser light emitted from the light source in a detection area is used. May be. The imaging device included in the image sensor unit 12 may have any configuration as long as the distance image of the detection area and the received light intensity image can be captured at the same exposure timing (exposure period). This TOF camera can capture, for example, a distance image of about 5 to 10 frames per second and a received light intensity image (a pair of captured images).

  FIG. 4 is a diagram illustrating an example of mounting the TOF camera. The TOF camera is attached relatively above the housing 1 so that the detection area shown in FIG. 1 is within the imaging area, and the imaging direction is directed obliquely downward. The TOF camera is attached to the track side from the slide door 2.

Also, if the TOF camera can fit the detection area within the imaging area, as shown in FIG. 4, do not attach it to the case 1 of the fall prevention fence, and attach it to a column or the like standing on the station platform. Also good.

  When the image processing unit 13 processes the detection area distance image and the received light intensity image captured by the image sensor unit 12 at the same timing as a pair of detection images (detection distance image and detection received light intensity image). Otherwise, it may be used to generate a pair of reference images (reference distance image and reference received light intensity image). The image processing unit 13 includes a memory (not shown) that stores the generated pair of reference images. This memory has a configuration corresponding to the reference image storage unit referred to in the present invention. The image processing unit 13 processes the pair of detection images using the pair of reference images stored in the memory, and detects an object located in the detection area. Details of the object detection process will be described later.

  The output unit 14 outputs the detection result of the object in the image processing unit 13 to a connected fall prevention fence or an alarm device. When an object detection result is input, a device that has detected the object is notified by a warning sound or turned on a warning light to notify the station staff of that fact (the object has been detected). Inform.

  Hereinafter, the operation of the image processing apparatus 10 will be described. FIG. 5 is a flowchart showing the operation of the image processing apparatus.

  As described above, the fall prevention fence closes the slide door 2 when the train is not stopped at the station platform. The image processing apparatus 10 executes the process shown in FIG. 5 every time the train stops at the station platform.

  When the train stops at the station platform, the image processing apparatus 10 is a pair of captured images (distance image, distance image, and image) captured by the image sensor unit 12 with the TOF camera at this timing (corresponding to the first timing in the present invention). Then, a reference distance image and a reference received light intensity image (a pair of reference images) based on the received light intensity image) are generated, and a reference image acquisition process is executed (s1). This reference image acquisition process uses a pair of captured images captured by the TOF camera in a state where the door of the train stopped at the station platform and the slide door 2 of the fall prevention fence are closed. That is, the pair of reference images can be used as a background image in which no object exists in the detection area. Further, the pair of reference images are images with a background of a train stopped at the station platform.

  When the image processing apparatus 10 generates and acquires a pair of reference images, the door of the train stopped at the station platform and the slide door 2 of the fall prevention fence are opened, and passengers can get on and off the train. When the passengers get on and off the train, the door of the train stopped at the station platform and the slide door 2 of the fall prevention fence are closed.

  The image processing apparatus 10 has an image sensor unit 12 at a timing (corresponding to a second timing referred to in the present invention) at which the door of the train stopped at the station platform and the slide door 2 of the fall prevention fence are closed. Performs a detection image acquisition process for acquiring a pair of captured images (distance image and received light intensity image) captured by the TOF camera as a pair of detection images (s2).

  The image processing apparatus 10 performs an object detection process for detecting an object located in the detection area using the pair of reference images generated and acquired in s1 and the pair of detection images acquired in s2 (s3). . In s3, it is a process which detects whether the object which is not imaged by the pair of reference images acquired by s1 is imaged by the pair of detection images acquired by s2. Therefore, a train stopped at the station platform, a support post installed at the station platform, a structure such as a fall prevention fence, etc. is not detected as an object in s3.

  The image processing apparatus 10 outputs the detection result of the object detection processing relating to s3 at the output unit 14 (s4).

  If an object located in the detection area is detected, the device to which the detection result is input performs a warning notification or the like, and notifies the station staff, the train driver, or the like to that effect. If the object located in the detection area is not detected in the object detection process in the image processing apparatus 10, the train driver starts the train from the station platform. On the other hand, if an object located within the detection area is detected, the station staff confirms and then starts a train from the station platform.

  An operation management system or the like that manages the operation of the train instructs the image processing apparatus 10 to start timing of the reference image acquisition process related to s1 and start timing of the detection image acquisition process related to s2. What is necessary is just composition. In addition, these start timing instructions may be input by a train driver or a station staff.

  Next, the reference image acquisition process according to s1, the detection image acquisition process according to s2, and the object detection process according to s3 will be described in detail.

  FIG. 6 is a flowchart showing the reference image acquisition process.

  The image sensor unit 12 captures a pair of captured images (distance image and received light intensity image) with a TOF camera for a preset number of frames (for example, 5 frames) (s11). The image processing unit 13 generates a reference distance image using the distance image of the number of frames captured in s11 (s12). In s12, a distance image in which an average value of distances of frames corresponding to the pixel is associated is generated for each pixel, and this is acquired as a reference distance image.

  Further, the image processing unit 13 generates a reference light reception intensity image using the light reception intensity images of the number of frames captured in s11 (s13). In s13, for each pixel, a light reception intensity image is generated in which the average value of the light reception intensity of each frame corresponding to the pixel is associated, and this is acquired as a reference light reception intensity image.

  The order in which the processes related to s12 and s13 are executed may be opposite to the above.

  The image processing unit 13 stores the reference distance image generated and acquired in s12 and the reference received light intensity image generated and acquired in s13 in the image memory (not shown) as a pair of reference images (s14).

  As described above, in this example, the reference distance image is generated from the distance images of a plurality of frames, and the reference light reception intensity image is generated from the light reception intensity images of the plurality of frames. A suppressed reference distance image and a reference received light intensity image can be generated and acquired.

  The image processing unit 13 sets the set number of frames as one frame, and stores the distance image and the received light intensity image captured by the image sensor unit 12 in the image memory as the reference distance image and the reference received light intensity image. It is good also as a structure.

  The detection image acquisition processing according to s2 is a pair of detection images obtained by capturing a pair of captured images (distance image and light reception intensity image) with the TOF camera and using the images as a detection distance image and detection light reception intensity image. It is a process to acquire.

  FIG. 7 is a flowchart showing the object detection process. The image processing unit 13 performs, for each pixel of the detection image, a rain pixel determination process that determines whether the pixel is a pixel that receives reflected light from raindrops (hereinafter referred to as a rain pixel) (s21). ).

  As described above, the meaning of the “pixel” of the rain pixel is a block of light receiving elements (pixels) as a unit when performing image processing. However, this block may be one light receiving element (1 pixel) of the image pickup device, or is constituted by a plurality of adjacent light receiving elements (for example, two light receiving elements (2 × 2 pixels) each in vertical and horizontal directions). It may be a block.

  The rain pixel determination process in s21 is performed by any one of the following methods (1) to (3).

  (1) Focusing on the fact that the reflected light from the raindrop is reflected at a relatively close position, as shown in FIG. 8A, the distance associated with the detection distance image (FIG. 7A). A pixel whose horizontal axis is less than a predetermined distance D1 is determined as a rain pixel, and a pixel whose distance is greater than or equal to a predetermined distance D1 is determined not to be a rain pixel.

  (2) In consideration of the reflectance of raindrops, as shown in FIG. 8B, a determination straight line for determining whether or not it is a rain pixel is determined in advance by a linear function of distance and received light intensity. The distance associated with the detection distance image (horizontal axis in FIG. 8B) and the light reception intensity associated with the detection light reception intensity image (vertical axis in FIG. 8B). Based on the plotted points, pixels located in the hatched area in FIG. 8B are determined as rain pixels, and pixels not located in the hatched area are determined not to be rain pixels.

  (3) Further, it may be considered that the reflectance of the raindrop changes depending on the reflectance of the reflecting surface located after the raindrop. As shown in FIG. 8C, a linear function of a distance and a difference in received light intensity (absolute value of a difference between a received light intensity in a detection received light intensity image and a received light intensity in a reference received light intensity image) is a rain pixel. A determination straight line for determining whether or not is determined in advance. Then, the absolute value of the difference between the distance associated with the detection distance image (horizontal axis in FIG. 8C) and the light reception intensity in the detection light reception intensity image and the light reception intensity in the reference light reception intensity image (FIG. 8). A pixel plotted based on the vertical axis in (C) is determined as a rain pixel if the pixel is located in the area indicated by hatching in FIG. 8C, and a pixel not located in the area indicated by the hatching is determined. It is determined that it is not a rain pixel.

  8A, 8 </ b> B, and 8 </ b> C, areas to be determined as rain pixels include the imaging environment of the detection area, the imaging characteristics of the image sensor unit 12 (for example, the pixel density of the TOF camera, the imaging lens, and the like). Therefore, adjustment is made when the image processing apparatus 10 is installed.

  As described above, the difference in received light intensity here is the absolute value of the difference in received light intensity of the corresponding pixels in the reference received light intensity image and the detected received light intensity image. The reason why the absolute value is used is that the pixel of the reference light reception intensity image is a rain pixel that detects the reflected light from the raindrop, and the corresponding pixel of the detection light reception intensity image is the pixel that does not detect the reflected light from the raindrop. This is to consider the case. Further, in both the reference light reception intensity image and the detection light reception intensity image, it is highly likely that the pixel that has detected the reflected light from the raindrop is determined to be the background in the differential image generation process described later.

  The accuracy of the rain pixel determination process is improved in the order of (1), (2), and (3). In addition, in the rain pixel determination process, the calculation amount increases in the order of (1), (2), and (3), so the processing time increases. Whether the rain pixel determination process is performed by any of the methods (1) to (3) described above may be determined in consideration of accuracy and processing time.

  Next, the image processing unit 13 determines, for each pixel of the detection image, whether the pixel is a pixel affected by raindrops attached to the imaging lens (hereinafter referred to as raindrop-attached pixels). An attached pixel determination process is performed (s22).

  The raindrop adhesion pixel determination process according to s22 is performed by any of the following methods (4) to (6).

  The inventor of the present application confirmed, through experiments, the tendency of changes in distance and received light intensity for raindrop-attached pixels affected by raindrops attached to the imaging lens. It was confirmed that the received light intensity of the raindrop adhesion pixel was lowered. In addition, it was confirmed that most of the raindrop adhesion pixels fall within a certain range. This is considered to be a phenomenon caused by attenuation of the amount of reflected light accompanying transmission of raindrops attached to the imaging lens.

  FIG. 9A shows a measurement result obtained by measuring the amount of change in the amount of received light applied to a pixel having raindrops attached to the imaging lens. The bar graph indicates the number of pixels with respect to the amount of change in received light intensity. Further, the line graph shows the cumulative ratio (percentage) of pixels with respect to the amount of change in received light intensity in raindrop-affected pixels. In this experiment, as shown in FIG. 9A, it was confirmed that about 94% of the raindrop-attached pixels had a change amount of received light intensity of 100 or less. However, the amount of change in the received light intensity referred to here depends on the environment in which the inventor conducted the experiment, and it goes without saying that the numerical value varies depending on the environment.

  It was also confirmed that most of the raindrop adhesion pixels fall within a range where the distance changes. This is considered to be a phenomenon that occurs when reflected light is refracted during the transmission of raindrops attached to the imaging lens, and pixels that receive the reflected light shift to adjacent pixels or neighboring pixels.

  FIG. 9B shows a measurement result obtained by measuring a change amount of a distance applied to a pixel having raindrops attached to the imaging lens. The bar graph indicates the number of pixels with respect to the amount of change in distance. In addition, the line graph indicates a cumulative ratio (percentage) of pixels with respect to the distance change amount in the raindrop-affected pixels. In this experiment, as shown in FIG. 9B, it was confirmed that about 73% of the raindrop adhesion pixels had a distance change amount of 350 mm or less. It was also confirmed that the percentage of raindrop-attached pixels whose distance change amount was 150 mm or less was about 3%. However, the amount of change in distance referred to here depends on the environment in which the inventor conducted the experiment, and it goes without saying that the numerical value varies depending on the environment.

  Therefore, by determining whether or not the pixel is a raindrop-attached pixel affected by the raindrop attached to the imaging lens based on the above-mentioned distance and the tendency of the change in received light intensity confirmed by the experiment on the raindrop-attached pixel, The determination accuracy can be improved.

  (4) In the reference received light intensity image and the detected received light intensity image, the absolute value of the difference in received light intensity is equal to or less than P1 shown in FIG. 10A (in the above experiment, for example, P1 = 100). The pixel is determined to be a raindrop-attached pixel, and the other pixels are determined not to be raindrop-attached pixels.

  (5) In the reference distance image and the detection distance image, the absolute values of the difference in distance are Dmin and Dmax shown in FIG. 10B (in the above experiment, for example, Dmin = 150 and Dmax = 350, for example). ) Are determined as raindrop-attached pixels, and other pixels are determined not to be raindrop-attached pixels.

  (6) In the reference received light intensity image and the detected received light intensity image, the absolute value of the difference in received light intensity is equal to or less than P1 shown in FIG. 10C, and in the reference distance image and the detected distance image, A pixel whose absolute value of the difference in distance is between Dmin and Dmax shown in FIG. 10C is determined as a raindrop-attached pixel, and the other pixels are determined not to be raindrop-attached pixels.

  10A, 10B, and 10C, P1, Dmin, and Dmax for determining an area to be determined as a raindrop adhesion pixel are the imaging environment of the detection area and the imaging characteristics of the image sensor unit 12 (for example, Since it varies depending on the pixel density of the TOF camera and the focal length of the imaging lens, it is adjusted when the image processing apparatus 10 is installed.

  The accuracy of this raindrop adhesion pixel determination process is improved in the order of (4), (5), and (6). The reason for using the absolute value of the difference in the received light intensity of the corresponding pixel and the absolute value of the difference in the distance of the corresponding pixel in the raindrop adhesion pixel determination process is the same as the rain pixel determination process described above. This is because the pixel of the intensity image is a raindrop-attached pixel and the corresponding pixel of the detection light-receiving intensity image is not a raindrop-attached pixel.

  The image processing unit 13 performs a rain state determination process for determining whether it is a rain state (s23). In s23, if the total number of pixels determined to be rain pixels in s21 and pixels determined to be raindrop adhesion pixels in s22 is equal to or greater than a predetermined threshold pixel number S, it is determined to be a rain state. If the total number of pixels determined to be rain pixels in s21 and pixels determined to be raindrop adhesion pixels in s22 is less than a predetermined threshold pixel number S, it is determined that the rain state is not present.

  If the image processing unit 13 determines that it is not raining in s23, it sets the lower limit of the size of the object to be detected to A1 (s24). If the image processing unit 13 determines that it is raining in s23, it sets the lower limit of the size of the object to be detected to A2 (A1 <A2) larger than A1 (s25). The unit of A1 and A2 is the number of pixels.

  The image processing unit 13 generates a difference image between the reference distance image generated and acquired in s12 and the detection distance image acquired in s2 (s26). In s26, a binary value in which the pixel value of the pixel having the same distance, the pixel determined to be a rain pixel in s22, and the pixel value determined to be a raindrop adhesion pixel in s23 is set to “0”, and the pixel values of other pixels are set to “1”. Generate an image.

The image processing unit 13 detects an object captured in the difference image generated in s26 (s27). In s27, in the difference image generated in s26, grouping is performed in which a set of pixels having a pixel value “1” located in the vicinity is set as one group. For each group, the image processing unit 13 determines that the group is an object if the number of pixels in the group is larger than the lower limit (A1 or A2) of the object size set in s24 or s25, and if the number is smaller, the image processing unit 13 determines noise. (It is not an object.)

  As described above, the image processing apparatus 10 according to this example includes an object that accompanies generation of raindrop-affected pixels such as rain pixels that detect reflected light from raindrops or raindrop-attached pixels that are affected by raindrops attached to the imaging lens. Can be sufficiently suppressed. That is, even when it is raining, erroneous detection of objects can be sufficiently suppressed.

  Further, the image processing apparatus 10 may be configured to output not only the object detection result in s4 but also the determination result of the rainfall state in s23.

  In addition, since the image processing apparatus 10 is configured to change the lower limit (A1 or A2) of the size of the object detected in s27 depending on whether or not it is in a rainy state, rain pixels for some pixels in s21 Prevents the object from being erroneously detected due to this misjudgment, even if a misjudgment that the pixel is not a rain pixel or a misjudgment that a raindrop adhering pixel is not a raindrop adhering pixel for some pixels occurs in s22 it can. Conversely, when it is not raining, the neighboring pixels that detected the reflected light from the object are erroneously determined to be raindrop-affected pixels, and a situation may occur in which an object located in the detection area is missed. Is also suppressed.

In the above example, the lower limit of the size of the object detected in s27 is set in two stages depending on whether it is raining. However, the lower limit of the size of the object is set in the rain pixel in s21. You may make it set according to the total of the pixel determined to exist and the pixel determined to be a raindrop adhesion pixel by s22. For example, when the total number of pixels determined to be rain pixels in s21 and pixels determined to be raindrop adhesion pixels in s22 is less than X, the size of the object detected in s27 is set to A1, and s21 When the total number of pixels determined to be rain pixels in step s22 and pixels determined to be raindrop adhesion pixels in s22 is equal to or greater than X, the size A of the object detected in s27 is
A = α × (number of pixels determined to be rain pixels in s21) + A1
(Where α is a predetermined proportionality constant, rounded down to the decimal point of the calculated A)
It is good also as a structure calculated and set by these. In this way, the lower limit of the size of the object detected in s27 can be set according to the amount of rainfall.

  The number of raindrop-affected pixels increases as the amount of rainfall increases. Therefore, it is possible not only to prevent erroneous detection of a plurality of adjacent raindrop-affected pixels as an object located in the detection area when the rainfall is relatively high, but also from the object when the rainfall is relatively low. A plurality of adjacent pixels that have detected reflected light are erroneously determined to be raindrop-affected pixels, and as a result, the occurrence of a situation in which an object located in the detection area is missed is further suppressed.

  In the above example, the rain state is determined according to the detection processing of the object located in the detection area. However, for example, it may be performed in a time zone when the train is not stopped at the platform. In this case, the detection area may be continuously captured by the TOF camera, and the process shown in FIG. 11 may be performed.

  The image processing apparatus 10 captures a pair of captured images (distance image and received light intensity image) with a TOF camera (s31). The image processing unit 13 generates and acquires a pair of reference images based on the latest captured image of the set number of frames including the pair of captured images captured this time (s32). s32 is the same processing as s1 (s11-s14) mentioned above.

  The image processing unit 13 acquires the pair of captured images captured this time as detection images (s33). The image processing unit 13 performs the raindrop influence pixel determination process described above (s34), and determines whether or not it is a raining state (s35). The process concerning s34 and s35 is the same process as s21-s23 mentioned above.

  Also, in s35, it may be determined that it is in a rainy state when a frame in which the total number of pixels determined to be raindrop-affected pixels is equal to or greater than a predetermined threshold pixel number S continues for a set frame or more. Good. In s35, if the average value of the total number of pixels determined to be raindrop-affected pixels in the captured image of the latest N frames is greater than or equal to a predetermined threshold pixel number S, it is determined that the rain state is present. It may be.

  In the above example, the present invention is described by taking the image processing apparatus 10 as an example where the detection area for detecting an object is between the fall prevention fence and the track at the position where the slide door 2 is provided. However, for example, it can be used for purposes such as detecting an intruder by setting the entrance to a factory or a condominium as a detection area. In this case, when an intruder is detected, it may be configured to notify the security guard staying in the security room or the management room to that effect.

DESCRIPTION OF SYMBOLS 10 ... Image processing apparatus 11 ... Control part 12 ... Image sensor part 13 ... Image processing part 14 ... Output part

Claims (17)

An image acquisition unit that acquires a distance image of the detection area captured by irradiating the detection area with light and receiving reflected light, and a received light intensity image as a pair of captured images;
Whether the image is a raindrop-affected pixel affected by raindrops using the distance image acquired as a pair of captured images by the image acquisition unit, the distance acquired at each pixel of the received light intensity image, and the received light intensity. A raindrop influence pixel determination unit for determining;
An object detection unit that processes the distance image using the determination result of the raindrop influence pixel determination unit and detects an object captured in the distance image;
A rain state determination unit that determines whether or not it is a rain state based on the number of pixels that the rain drop effect pixel determination unit determines to be the rain drop effect pixel;
The object detection unit is an image processing apparatus in which the lower limit of the size of an object to be detected is larger when the determination by the rain state determination unit is a rain state than when the determination is not a rain state . An image acquisition unit for acquiring a distance image of the detection area captured by the imaging device irradiating the detection area with light and receiving the reflected light;
A raindrop-affected pixel determination unit that determines whether or not the pixel is a raindrop-affected pixel affected by a raindrop, using the distance acquired in each pixel of the distance image acquired by the image acquisition unit;
An object detection unit that processes the distance image using the determination result of the raindrop influence pixel determination unit and detects an object captured in the distance image;
A rain state determination unit that determines whether or not it is a rain state based on the number of pixels that the rain drop effect pixel determination unit determines to be the rain drop effect pixel;
The object detection unit is an image processing apparatus in which the lower limit of the size of an object to be detected is larger when the determination by the rain state determination unit is a rain state than when the determination is not a rain state . An image acquisition unit that acquires a distance image of the detection area captured by irradiating the detection area with light and receiving reflected light, and a received light intensity image as a pair of captured images;
A raindrop effect pixel determination unit that determines whether the pixel is a raindrop effect pixel affected by raindrops, using the received light intensity acquired in each pixel of the received light intensity image acquired by the image acquisition unit;
An object detection unit that processes the distance image acquired by the image acquisition unit using the determination result of the raindrop influence pixel determination unit and detects an object captured in the distance image;
A rain state determination unit that determines whether or not it is a rain state based on the number of pixels that the rain drop effect pixel determination unit determines to be the rain drop effect pixel;
The object detection unit is an image processing apparatus in which the lower limit of the size of an object to be detected is larger when the determination by the rain state determination unit is a rain state than when the determination is not a rain state . A reference image storage unit that stores a reference distance image of the detection area and a reference light reception intensity image as a pair of reference images,
The raindrop-affected pixel determination unit sets the distance image acquired by the image acquisition unit and the received light intensity image as a pair of detection images, determines whether each pixel is the raindrop-effect pixel,
The image processing apparatus according to claim 1, wherein the object detection unit generates a difference image between the reference distance image and a distance image of the detection image, and detects an imaged object.   The raindrop-affected pixel determination unit is configured to determine, for each pixel, the raindrop-affected pixel based on a difference in received light intensity calculated using the received light intensity of the pixel in the reference received light intensity image and the received light intensity image of the detection image. The image processing apparatus according to claim 4, wherein it is determined whether or not there is any.   The raindrop influence pixel determination unit uses, for each pixel, the distance of the pixel in the distance image of the detection image, and the light reception intensity of the pixel in the reference light reception intensity image and the light reception intensity image of the detection image. 6. It is determined whether or not the pixel is a rain pixel in which reflected light from a raindrop is detected based on the calculated difference in received light intensity, and the pixel determined to be the rain pixel is the raindrop-affected pixel. The image processing apparatus described.   The raindrop-affected pixel determination unit, for each pixel, the distance difference calculated using the distance of the pixel in the reference distance image and the distance image of the detection image, the reference light reception intensity image, and the detection image Based on the difference in the received light intensity calculated using the received light intensity of the pixel with respect to the received light intensity image, it is determined whether the pixel is a raindrop-attached pixel affected by the raindrop attached to the imaging lens of the image acquisition unit The image processing apparatus according to claim 5, wherein a pixel determined to be the raindrop-attached pixel is the raindrop-affected pixel. A reference image for generating a pair of the reference images based on the distance image of the detection area and the received light intensity image captured by the imaging device and acquired by the image acquisition unit at a predetermined reference image acquisition timing With a generator,
The image processing apparatus according to claim 4, wherein the reference image storage unit updates the pair of stored reference images to the pair of reference images generated by the reference image generation unit.   The reference image generation unit captures a pair of the reference images based on the distance image and the received light intensity image for a plurality of frames captured by the imaging device and acquired by the image acquisition unit at the reference image acquisition timing. The image processing apparatus according to claim 8, which generates It said object detection unit, the lower limit of the size of the object to be detected is determined in accordance with the number of pixels the raindrop impact pixel determination unit determines that said raindrop impact pixels, according to any one of claims 1 to 9 Image processing device. The image processing apparatus according to any one of claims 1 to 10 having an output unit for outputting a detection result of the object by the object detection unit. Computer
An image acquisition step of acquiring a distance image of the detection area captured by irradiating the detection area with light and receiving the reflected light, and a received light intensity image as a pair of captured images;
The distance image acquired as a pair of captured images by the image acquisition step, and the distance obtained at each pixel of the received light intensity image, and using the received light intensity, whether a raindrop impact pixels affected by raindrop A raindrop influence pixel determination step for determining,
An object detection step of processing the distance image using the determination result of the raindrop influence pixel determination step and detecting an object captured in the distance image;
A rain state determination step for determining whether or not it is a rain state based on the number of pixels determined to be the rain drop effect pixel in the rain drop influence pixel determination step,
The object detection step is an object detection method in which the lower limit of the size of an object to be detected is set to be larger when the determination by the rain state determination unit is a rain state than when the determination is not a rain state . Computer
An image acquisition step of acquiring a distance image of the detection area captured by the imaging device irradiating the detection area with light and receiving the reflected light;
By using the distance obtained at each pixel of the distance image acquired in the image acquisition step, if a raindrop impact pixel determination step of determining whether a raindrop impact pixels affected by the raindrops,
An object detection step of processing the distance image using the determination result of the raindrop influence pixel determination step and detecting an object captured in the distance image;
A rain state determination step for determining whether or not it is a rain state based on the number of pixels determined to be the rain drop effect pixel in the rain drop influence pixel determination step,
The object detection step is an object detection method in which the lower limit of the size of an object to be detected is set to be larger when the determination by the rain state determination unit is a rain state than when the determination is not a rain state . Computer
An image acquisition step of acquiring a distance image of the detection area captured by irradiating the detection area with light and receiving the reflected light, and a received light intensity image as a pair of captured images;
Using the acquired received light intensity in each pixel of the acquired received light intensity image as a pair of captured images by the image acquisition step, a raindrop impact pixel determination step of determining whether the raindrop impact pixels affected by raindrop ,
An object detection step of processing the distance image using the determination result of the raindrop influence pixel determination step and detecting an object captured in the distance image;
A rain state determination step for determining whether or not it is a rain state based on the number of pixels determined to be the rain drop effect pixel in the rain drop influence pixel determination step,
The object detection step is an object detection method in which the lower limit of the size of an object to be detected is set to be larger when the determination by the rain state determination unit is a rain state than when the determination is not a rain state . An imaging device irradiated with light detection area, an image acquiring range image of the detection area captured by receiving the reflected light, and the light-receiving intensity image as a pair of captured images,
The distance image acquired as a pair of captured images by the image acquisition step, and the distance obtained at each pixel of the received light intensity image, and using the received light intensity, whether a raindrop impact pixels affected by raindrop A raindrop influence pixel determination step for determining,
An object detection step of processing the distance image using the determination result of the raindrop influence pixel determination step and detecting an object captured in the distance image;
An object detection program for causing a computer to execute a rain state determination step for determining whether or not it is a rain state based on the number of pixels determined to be the rain drop effect pixel in the rain drop effect pixel determination step,
The object detection step is an object detection program in which the lower limit of the size of an object to be detected is a step in which, when the determination by the rain state determination unit is a rain state, the determination is larger than when the determination is not a rain state . An image acquisition step of acquiring a distance image of the detection area captured by the imaging device irradiating the detection area with light and receiving the reflected light;
A raindrop influence pixel determination step for determining whether or not a raindrop influence pixel is affected by a raindrop, using the distance acquired in each pixel of the distance image acquired by the image acquisition step;
An object detection step of processing the distance image using the determination result of the raindrop influence pixel determination step and detecting an object captured in the distance image;
An object detection program for causing a computer to execute a rain state determination step for determining whether or not it is a rain state based on the number of pixels determined to be the rain drop effect pixel in the rain drop effect pixel determination step,
The object detection step is an object detection program in which the lower limit of the size of an object to be detected is a step in which, when the determination by the rain state determination unit is a rain state, the determination is larger than when the determination is not a rain state . An image acquisition step of acquiring a distance image of the detection area captured by irradiating the detection area with light and receiving the reflected light, and a received light intensity image as a pair of captured images;
A raindrop-affected pixel determining step for determining whether the pixel is a raindrop-affected pixel affected by raindrops using the received-light intensity acquired in each pixel of the received-light-intensity image acquired as a pair of captured images in the image acquisition step; ,
An object detection step of processing the distance image using the determination result of the raindrop influence pixel determination step and detecting an object captured in the distance image;
An object detection program for causing a computer to execute a rain state determination step for determining whether or not it is a rain state based on the number of pixels determined to be the rain drop effect pixel in the rain drop effect pixel determination step,
The object detection step is an object detection program in which the lower limit of the size of an object to be detected is a step in which, when the determination by the rain state determination unit is a rain state, the determination is larger than when the determination is not a rain state .

Images