JP4762880B2 - Image processing system - Google Patents

Description

  The present invention relates to an image processing system that estimates noise intensity of an original image.

Patent Document 1 discloses a stereo-type vehicle exterior monitoring device that calculates a parallax in units of pixel blocks by a stereo matching process using a pair of images obtained by capturing a scene outside the vehicle, and monitors conditions outside the vehicle based on the parallax. Has been. In this monitoring apparatus, a pixel block having a large luminance change is identified by comparing a luminance change amount between adjacent pixels in the pixel block with a luminance determination threshold for each pixel block that is a unit for calculating parallax. . Only the parallax related to the specified pixel block is used as effective distance data used in object recognition, and the others are invalidated. For example, the luminance judgment threshold is set to a large value during the day when an excessive number of distance data is likely to be calculated, and is set to a small value at night when the number of distance data tends to be insufficient. It is automatically adjusted according to the number of distance data. Thus, by adjusting the luminance determination threshold according to the monitoring environment (imaging environment), it is possible to stably secure the necessary number of distance data without depending on the environment.
JP 2001-67484 A

  However, in Patent Document 1 described above, no consideration is given to noise superimposed on the original image.

  Therefore, an object of the present invention is to provide a novel technique that can estimate the noise intensity of an original image from a corrected image obtained by performing correction processing on the original image.

  In order to solve this problem, the present invention has a determination unit, a reverse correction unit, and a noise estimation unit, and estimates the noise intensity of the original image from the corrected image obtained by performing correction processing on the original image. An image processing system is provided. The determination unit extracts a plurality of pixel blocks having a small luminance change as a first pixel block by comparing a luminance change amount between each adjacent pixel in the pixel block with a luminance determination threshold value in the corrected image. The reverse correction unit performs reverse correction processing of the correction processing on the plurality of first pixel blocks extracted by the determination unit. The noise intensity estimation unit generates a histogram of the luminance change amount between adjacent pixels for the plurality of first pixel blocks subjected to the reverse correction process by the reverse correction unit, and the luminance change amount on the histogram is 0. An appearance frequency distribution near a certain origin is estimated as a noise intensity distribution of the original image.

  Here, in the present invention, the noise intensity estimation unit sets a threshold level for the appearance frequency according to the peak height of the appearance frequency existing in the vicinity of the origin, and the appearance of the threshold level. It is preferable to calculate a noise intensity estimated value based on a luminance change amount having a frequency. In this case, the noise intensity estimator obtains a noise intensity estimated value smaller than the integer level by an interpolation operation using a pair of adjacent brightness change amounts as input, one of which exceeds the threshold frequency appearance frequency and the other is less than the other. You may calculate with resolution.

  In the present invention, an adjustment unit may be further provided that adjusts the luminance determination threshold value according to the noise intensity estimation value calculated by the noise intensity estimation unit. In this case, the adjustment unit preferably increases the luminance determination threshold value as the noise intensity estimation value increases.

  In the present invention, it is preferable that the determination unit does not extract a pixel block whose average luminance is equal to or lower than the lower limit value and a pixel block whose average luminance is equal to or higher than the upper limit value as the first pixel block.

  Further, in the present invention, a stereo matching unit that calculates distance data in units of pixel blocks by performing stereo matching processing using a pair of images as input may be further provided. In this case, the determination unit extracts the pixel block having a large luminance change as the second pixel block by comparing the luminance change amount between adjacent pixels with the luminance determination threshold in the corrected image. It is preferable to specify the distance data calculated for the pixel block as effective distance data used when performing object recognition.

  Appearance frequencies are concentrated near the origin where the amount of change in luminance is 0 as the characteristics of a histogram using pixel blocks with little change in luminance as samples. Theoretically, it appears only at the origin, but since it is actually affected by noise, a peak shape is formed with the approximate origin at the center. On the other hand, when there are some edges due to the object in the pixel block, these edges are not included in the peak shape near the origin and appear as another peak at a position relatively far from the vicinity of the origin. In view of such characteristics, the appearance frequency distribution near the origin in the histogram can be estimated as the noise intensity distribution of the original image.

  FIG. 1 is a block diagram of a stereo object recognition apparatus to which an image processing system is applied. This object recognizing device is mainly composed of a stereo camera 1, an image processing system 2, and an object recognizing unit 3, and recognizes a position in real space of a detection target imaged in an image. The stereo camera 1 has a main camera 1a and a sub camera 1b, and outputs a pair of original images (light reception images) in time series by repeating imaging at a predetermined interval. As its main function, the image processing system 2 performs stereo matching processing using a pair of images taken in time series as input, and generates distance data. The object recognition unit 3 recognizes the detection target based on the distance data generated by the image processing system 2 (in practice, effective distance data described later), and calculates the position in the real space.

  The image processing system 2 includes a correction unit 4, a stereo matching unit 5, a determination unit 6, a reverse correction unit 7, a noise intensity estimation unit 8, and an adjustment unit 9. This processing system 2 is characterized in that feedback mechanisms 7 to 9 are newly added to the basic processing mechanisms 4 to 6, and the latter makes it possible to estimate a noise intensity and a luminance determination threshold value DCDXth based thereon. Adjustment function is realized.

  The correction unit 4 performs various correction processes on the pair of original images output from the stereo camera 1 to generate a corrected image. The correction process includes all processes that cause modulation of noise superimposed on the original image. For example, luminance correction (gamma correction, etc.) of the original image, correction according to the CCD characteristics in the cameras 1a and 1b, Examples include correction according to the ambient temperature and various filter processes. These correction processes are performed by a table look-up method (that is, a method in which correction is performed with reference to a table prepared in advance), a numerical operation using a predetermined mathematical expression, or the like.

  The stereo matching unit 5 performs a stereo matching process using the pair of corrected images output from the correction unit 4 as input, and calculates distance data dij (parallax dij) for each pixel block. As shown in FIG. 2, the unit for calculating the parallax dij is a pixel block PBij obtained by dividing a reference image (corrected image of the main camera 1a) of the main camera 1a vertically and horizontally (a small area constituting a part of the image). ). For example, when the size of the reference image is 512 × 200 pixels and the size of one pixel block PBij is 4 × 4, a parallax group corresponding to the maximum number of pixel blocks PBij (128 × 50) from a captured image corresponding to one frame. Is calculated. The parallax dij is a horizontal shift amount with respect to the pixel block PBij that is a calculation unit, and has a large correlation with the distance to the object projected on the pixel block PBij. That is, the closer the object projected in the pixel block PBij is to the stereo camera 1, the larger the parallax dij of this pixel block PBij, and the smaller the object, the smaller the parallax dij (if the object is infinitely far away, the parallax dij Becomes 0). When calculating the parallax dij related to a certain pixel block PBij (correlation source), a region (correlation destination) having a correlation with the luminance characteristic of the pixel block PBij is specified in the comparison image (corrected image of the sub camera 1b). As described above, the distance from the stereo camera 1 to the object appears as a horizontal shift amount between the reference image and the comparison image. Therefore, when searching for the correlation destination in the comparison image, it is only necessary to search on the same horizontal line (epipolar line) as the j coordinate of the pixel block PBij as the correlation source. The stereo matching unit 5 sequentially shifts the correlation between the correlation source and the correlation destination candidates while shifting one pixel at a time on the epipolar line within a predetermined search range set based on the i coordinate of the correlation source. Evaluate (stereo matching). The correlation between the two pixel blocks can be evaluated using a well-known correlation evaluation value such as a luminance difference absolute sum or a luminance difference square sum. In principle, the amount of horizontal shift of the correlation destination (one of the correlation destination candidates) determined to have the highest correlation is the parallax dij of the pixel block PBij. The position (X, Y, Z) in the real space of the object projected in the pixel block PBij has the coordinates (i, J) on the image plane of the pixel block PBij and its parallax dij as inputs. It is uniquely identified by coordinate transformation.

  The determination circuit 6 refers to one corrected image (for example, a reference image) and evaluates the luminance edge (luminance change) in the horizontal direction in the pixel block PBij, thereby calculating the distance data dij calculated in the stereo matching unit 5. Is subjected to filtering processing. FIG. 3 is an explanatory diagram of luminance edges in the pixel block PBij. In a certain pixel block PBij, luminance change amounts (absolute values) Δp1 to Δp16 of pixel pairs adjacent in the horizontal direction are calculated. However, for the leftmost pixel column (p11, p12, p13, P14), the luminance change amount (Δp1, Δp5, Δp9, Δp13) is calculated from the rightmost pixel column of the pixel block adjacent to the left side. . Next, of these luminance change amounts Δp1 to Δp16, the number of luminance determination threshold values DCDXth or more (that is, the number of luminance edges) is counted. The luminance determination threshold value DCDXth is appropriately set by the adjustment unit 9 by feedback control described later. The determination unit 6 counts the number of luminance change amounts Δp having a value equal to or greater than the luminance determination threshold value DCDXth for each pixel block PBij as the number of edges (the number of edges is in the range of 0 to 16).

  The determination unit 6 extracts pixel blocks PBij having a predetermined number of edges (for example, 4) or more as effective blocks, and outputs distance data dij related to effective blocks to the object recognition unit 3 as effective distance data d′ ij. . Even if a pixel block PBij with little change in luminance in the horizontal direction, in other words, a pixel block PBij with little luminance feature (texture), is matched in the stereo matching unit 5, the reliability of the parallax dij Are often low. Thus, the reliability of the effective distance data d′ ij is improved by invalidating the parallax dij for the pixel block PBij having no texture. The object recognition unit 3 recognizes the detection target based on the highly reliable effective distance data d′ ij output from the determination unit 6.

  In addition, the determination unit 6 extracts a plurality of pixel blocks PBij (those having a small change in luminance and no texture) having a predetermined number (for example, 0 or 1) of edges as invalid blocks IB, and relate to the invalid blocks IB. Information (including luminance change amount Δp) is output to the reverse correction unit 7. In the present embodiment, the intensity of noise superimposed on the original image is estimated by statistical processing using a plurality of invalid blocks IB as samples. From the viewpoint of the manifestation of a statistical tendency, it is preferable to ensure a large number of samples dispersed in the screen. In addition, when the scenery to be imaged is fixed to some extent, such as in a front monitoring system mounted on a car, a window is set in an image area where there is not much texture (for example, a portion where a road is projected). It is preferred to extract the sample within the window.

  When extracting the invalid block IB, the determination unit 6 may consider the luminance average of the pixel block PBij in addition to the determination based on the number of edges described above. For example, a pixel block PBij whose average luminance is lower than the lower limit value (for example, 0 at 256 gradations) and a pixel block PB whose average luminance is higher than the upper limit value (for example, 255 at 256 gradations) are not extracted as invalid blocks IB. It is like that. The reason for considering the luminance average is to avoid the noise amplitude of the pixel block PBij that is completely saturated or the like, so that it cannot be measured. Further, in addition to these, it may be taken into account that the distance data dij of the upper and lower pixel blocks PBij are not continuous (since this state may be a mismatch due to noise rather than an object).

  The reverse correction unit 7 performs reverse correction processing of the correction processing performed by the correction unit 4 on each of the plurality of invalid blocks IB. FIG. 4 is a characteristic diagram showing the relationship between the luminance and the noise amplification factor, and FIG. 4A is an example of the characteristic of the corrected image. It is preferable that the noise amplification factor of the recognition image (the input image to the object recognition unit 3) with respect to the received light intensity is constant over the entire luminance as shown in FIG. However, as shown in FIG. 6A, in a corrected image subjected to correction according to CCD characteristics, gamma correction, or the like, the noise amplification factor is not constant but nonlinear. As described above, it is difficult to effectively estimate the noise intensity superimposed on the original image in the state where the noise is modulated (corrected image). Therefore, as the preprocessing, the characteristic / correction is removed from the corrected image, and the state (b) in FIG. 10 (nonlinear noise characteristic of the corrected image) is changed to the state (a) in FIG. Processing to return to the normal noise characteristic), that is, reverse correction processing must be performed.

  Since the correction content in the correction unit 4 (that is, all processes affecting the noise from the CCD noise characteristic to the recognized image) is known, the content of the reverse correction in the reverse correction unit 7 is also uniquely specified based on this. Is done. Specifically, when the previous correction is a table look-up method, an inverse conversion table corresponding to this correction can be performed by preparing an inverse conversion table of the table used at the time of correction. Further, when the previous correction is a numerical calculation, if an inverse conversion formula of the mathematical formula used at the time of correction is prepared, reverse correction corresponding to this correction can be performed. The inverse correction block RB generated by such inverse correction processing is output to the noise intensity estimation unit 8.

  The noise intensity estimation unit 8 uses a plurality of inverse correction blocks RB as sample blocks, and generates a histogram of the luminance change amount Δp between adjacent pixels as shown in FIG. From N sample blocks, (N × 16) samples of luminance change amount Δp are obtained. These samples are sequentially voted in a voting space with the absolute value | Δp | As a characteristic of the histogram obtained by this, the appearance frequency is concentrated in the vicinity of the origin where the luminance change amount | Δp | = 0. Since a block without texture is used as a sample, it should theoretically appear only at the origin, but in reality there is an influence of noise, so a peak shape is formed around the origin. On the other hand, when there are some edges due to the object in the block, these edges are not included in the peak shape near the origin and appear as another peak at a position relatively far from the vicinity of the origin. For the reasons described above, the appearance frequency distribution near the origin in the histogram can be estimated as the noise intensity distribution of the original image. In the example shown in the figure, the range of the luminance change amount from 0 to | Δp3 | is specified as the noise intensity distribution.

  In the present embodiment, the histogram is generated on the basis of the absolute value of the luminance change amount Δp. However, a histogram that also considers the sign of the luminance change amount Δp may be used.

  In addition, the noise intensity estimation unit 8 calculates a noise intensity estimated value N that is a reference value of the noise intensity based on the noise intensity distribution. Specifically, first, the peak height of the appearance frequency existing in the noise intensity distribution (near the origin on the histogram) is specified. In the example shown in the figure, since the luminance change amount | Δp | = 0 is the peak, the peak height is the appearance frequency k0. Next, a threshold level kth relating to the appearance frequency is set according to the peak height K0. As this threshold level kth, for example, a value (k0 × δ) obtained by multiplying a predetermined ratio δ (for example, 25%) based on the peak height k0 can be used. Then, a noise intensity estimated value N is calculated on the basis of the luminance change amount that becomes the threshold level kth.

  In FIG. 5, the value | Δp3 | which is lower than the threshold level Kth among the adjacent brightness change amounts | Δp2 | and | Δp3 | N may be used. Alternatively, the value | Δp2 | that exceeds the threshold level Kth may be used. In this case, the noise intensity estimated value N is an integer value. When it is desired to obtain the noise intensity estimated value N with a resolution (decimal level) finer than the integer level, a pair of adjacent luminance change amounts | Δp2 | and | Δp3 | The interpolation calculation (for example, linear interpolation) is performed, and the interpolation value may be used. The calculated noise intensity estimated value N is output to the adjustment unit 9.

  The adjustment unit 9 adjusts the luminance determination threshold value DCDXth according to the noise intensity estimated value N, and feeds it back to the determination unit 6. The luminance determination threshold value DCDXth is set to a larger value as the noise intensity estimated value N increases. For example, a new luminance determination threshold value DXCDth can be calculated by multiplying the noise intensity estimation value N by a constant value α as a gain and multiplying the current luminance determination threshold value DCDXth by this gain. A large noise intensity estimated value N means that the current luminance determination threshold value DCDXth makes a loose determination and the cancellation of noise in the image is not sufficient. In this case, by setting the luminance determination threshold value DCDXth to a value larger than the current value, the determination criterion of the luminance edge in the determination unit 6 is made stricter than the current value (this enhances noise cancellation). ). In order to suppress a rapid change in the luminance determination threshold value DCDXth, it is preferable to use the noise intensity estimated value N after performing a temporal and spatial smoothing process.

  According to the present embodiment, using the histogram characteristic that the appearance frequency due to noise is concentrated near the origin where the luminance change amount is 0, and the appearance frequency due to the edge appears at a position away from the origin, The appearance frequency distribution near the origin can be estimated as the noise intensity distribution of the original image.

  Further, the accuracy of the effective distance data d′ ij used in object recognition is improved by calculating the noise intensity estimated value N based on the noise intensity distribution and using this to feedback control the luminance determination threshold value DCDXth. Can be achieved. As another use of the noise intensity estimated value N, it may be used to calculate the reliability (disparity variation degree) of the distance data dij.

  In the above-described embodiment, the amount of change in the brightness of the pixel block PBij (the strength of the texture) is evaluated by comparing the amount of change Δp in the horizontal direction of the pixel block PBij with the brightness determination threshold value DCDXth. Yes. However, the present invention is not limited to this, and the evaluation may be performed based on the luminance change amount in the vertical direction or the luminance change amounts in both the vertical direction and the horizontal direction.

Block diagram of a stereo object recognition device to which an image processing system is applied Explanatory drawing of the pixel block set on the image plane Explanatory diagram of luminance edge in pixel block Characteristic diagram showing the relationship between brightness and noise gain The figure which shows the histogram of the luminance variation between adjacent pixels

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Stereo camera 1a Main camera 1b Sub camera 2 Image processing system 3 Object recognition part 4 Correction | amendment part 5 Stereo matching part 6 Judgment part 7 Inverse correction | amendment part 8 Noise intensity estimation part 9 Adjustment part

Claims (7)

In an image processing system that estimates noise intensity of an original image from a corrected image obtained by performing correction processing on the original image,
A determination unit that extracts a plurality of pixel blocks having a small luminance change as a first pixel block by comparing a luminance change amount between each adjacent pixel in the pixel block with a luminance determination threshold in the corrected image;
A reverse correction unit that performs reverse correction processing of the correction processing on the plurality of first pixel blocks extracted by the determination unit;
For the plurality of first pixel blocks that have been subjected to reverse correction processing by the reverse correction unit, a histogram of the luminance change amount between adjacent pixels is generated, and a luminance change amount on the histogram near the origin that is zero An image processing system comprising: a noise intensity estimation unit that estimates an appearance frequency distribution as a noise intensity distribution of the original image.   The noise intensity estimation unit sets a threshold level related to the appearance frequency according to the peak height of the appearance frequency existing in the vicinity of the origin, and changes in luminance having the appearance frequency of the threshold level. The image processing system according to claim 1, wherein an estimated noise intensity value is calculated based on the quantity.   The noise intensity estimation unit is configured to make the noise intensity estimation value finer than the integer level by an interpolation operation using a pair of adjacent brightness change amounts as input, one of which exceeds the frequency of appearance of the threshold level and is less than the other. The image processing system according to claim 2, wherein the image processing system calculates the resolution.   4. The image according to claim 1, further comprising: an adjustment unit that adjusts the luminance determination threshold value according to the noise intensity estimation value calculated by the noise intensity estimation unit. 5. Processing system.   The image processing system according to claim 4, wherein the adjustment unit increases the luminance determination threshold value as the noise intensity estimation value increases.   2. The determination unit according to claim 1, wherein the determination unit does not extract a pixel block whose average luminance is equal to or lower than a lower limit value and a pixel block whose average luminance is equal to or higher than an upper limit value as the first pixel block. Image processing system. A stereo matching unit that calculates distance data in units of pixel blocks by performing a stereo matching process using a pair of images as input, and
The determination unit extracts a pixel block having a large luminance change as a second pixel block by comparing a luminance change amount between the adjacent pixels with the luminance determination threshold in the corrected image. The image processing system according to claim 1, wherein the distance data calculated for two pixel blocks is specified as effective distance data used when performing object recognition.

Images