DE102006001634B3 - Creation of distance-image from correspondence between pixels from two cameras, by illuminating with random grid, and comparing brightness ratios of pixels - Google Patents

Abstract

The scene is illuminated with a random or pseudo-random grid and a first picture is taken with both cameras. The scene is then illuminated and a second picture taken with both cameras. A brightness ratio is calculated pixel-wise for both cameras from the two pictures. The correspondence between pixels is established based on the comparison of the brightness ratios of pixels from different cameras. An independent claim is included for an apparatus for creating a distance-image of a scene.

Description

The The invention relates to a method and arrangements for obtaining a Distance image. Distance images encode in contrast to conventional ones Images encoding gray values or colors, the distance from Object points from the sensor (i.a.a camera) or the height of the object points relative to a plane. Technical applications can be found i.a. in assembly control, robotics, metrology, archeology, Garment Industry, Biometrics, Medicine, Reverse Engineering.

An overview of used Process with a table of commercially available systems is described in [1] given, in addition for the procedures see also [2].

The The method disclosed here concerns trangulation with stereo cameras. For stereo methods, the most difficult problem is the production the correspondence of the pixels. If the correspondence is known, so can according to known mathematical methods (see, for example, [3]) corresponding distance image are calculated.

to Making the correspondence are classic image analysis methods based on the extraction of contoured or blob-like features used; However, a match is due to potential problems in the feature extraction. not really sure, besides, between the characteristics estimated or interpolated. To work around these issues, used one additionally structured light.

at regularly repeating light structures such as e.g. the widespread striped patterns Ambiguities occur which are eliminated with the coded light approach can be. Usually is working with a single camera, the geometry the light source itself is used for triangulation. An overview gives [4].

WHERE 2005/010825 A2 shows a method for creating a distance image, in which a first illumination of the scene ("object") with a random pattern ("randomgrid") and a second Lighting with a so-called "striped grid "is carried out Here, a first and a second image are taken, each with a camera. In a first step after taking the first pictures with both cameras will be the correspondence between coordinates of the two first pictures determined. After taking the second pictures will used the correspondence determined in the first step, pattern strips to identify between the two second shots of the scene. After all become, for example, by means of triangulation, the 3D coordinates of the scene determined.

To US Pat. No. 6,542,250 B1 At least two patterns that can be random patterns are used. In an iterative process, starting from an initial estimate, point-by-point forward calculation of spatial coordinates to image coordinates of two cameras is realized, with refinement at each iteration step.

It is also using individual textured lighting patterns and texturelementeweiser Correspondence determination worked, e.g. with the systems of the companies 3Q / 3DMD ([5]). This has the disadvantage that the texture elements different perspectives in different ways (gloss, Shading, shape) and therefore are difficult to evaluate and thus can give unsafe results. The procedures are therefore only on not too heavily structured Surfaces with wide reflection lobe used (matte surfaces). After [6] will be the problem through a specially chosen Shape of features defused: line elements by chance changing intensity and orientation; it is allowed only elements with matching orientation and intensity with each other be paired. The disadvantage is, a) that the intensity is different Considered directions, b) that these elements for a reliable evaluation have to have a certain extent and therefore a pixel-wise correspondence formation again an interpolation between the features required.

To [8] pp. 841-842, Paragraph 5.1, with a camera several pictures of a scene with recorded different lighting patterns and then for each Picture each the brightness ratio ("intensity ratio") relative to one constant illumination calculated ("codification based on gray levels").

A special form of coded light is the coding by a color pattern ("rainbow") running continuously over the field of view, with a single camera, after JP 61075210 (1986) or US 5675407 (1997). One problem is the high price for the projector, realized eg via a "linear variable wavelength filter".

To US 6028672 (2000), the triangulation does not occur between the camera and the projector, but between two cameras with color matching on epipolar lines of the two cameras. This defuses the demand for geometrically precise color projection, because any color pattern can be used (see column 6, line 45), but there are still fundamental problems with color evaluation.

These should be after US 6556706 (2003, same applicant) are reduced by various measures, including the fact that the light projection disk-wise only light of a wavelength contains (see, for example, column 5, line 35: "..impose a, single spectral light condition 'to the projector").

The same applicant has also proposed arrangements with a rotating laser slit, with a camera, wherein the light intensity varies depending on the angle, rising in an image acquisition and falling in an image acquisition (hereinafter referred to in opposite directions). About the intensity ratio of a pixel from the images, a clear assignment to the associated projection angle can be made. The disadvantage is the mechanical movement and the required precision in the projection; US 6,600,168 (2003).

A static arrangement for generating over the field of view opposite illumination patterns, with a camera is in US 6897946 (2005), generated by two closely spaced, slightly rotated lamps with reflectors.

Another arrangement for generating over the field of view opposite illumination patterns is in US 6618123 (2003), with a field of LEDs that are controlled in such a way that opposing illumination patterns arise across the field of view (see, for example, p 4 ). The arrangement works with a single camera and requires homogeneous light lobes and a complex calibration with a flat plate (see eg column 8).

One general disadvantage of methods with contiguous over the field of view, opposing Illumination patterns is that when plane template the brightness values of adjacent pixels differ only slightly and the exact local assignment becomes difficult.

Around nevertheless a good assignment (and thus good distance measurement) too reach, must accurate and high resolution Transducer can be used. To the lighting dynamics with low Brightness values well exploit and also dark surface areas good to evaluate, must at the ratio calculation be divided by small numbers; so it has to be. at small Pay enough significant bits to disposal stand. When using two cameras, they must be exactly on top of each other be coordinated.

aim The invention is the provision of a method and a Device, the o.g. Avoid disadvantages.

The Task is solved after the independent Claims.

Under brightness ratio is the mathematical relationship two brightnesses h1 and h2, ie h1 / h2 or h2 / h1 meant, or another ratio formation in this sense, e.g. (h1 - g) / (h2 - g), where g is a previously measured background brightness in ambient light.

accidental or pseudo-random patterns will become random patterns in the following called. Under pseudo-random patterns are also determined here Patterns subsumed within a certain local area have no repetitions, about a minimum length seen. The minimum length must be there be at least that big as it is under consideration the picture relations the maximum disparity corresponds to corresponding pixels.

For explanation a numerical example:

The Number sequence with values 0, 1, 2, 3 (generated with a digital Pseudo-random generator) is repeated from the minimum length of 18 values (character "^"), in it come however no repetitions of local areas of length 3 (or more).

Especially Preferably, the lighting is done in the two shots each at the same solid angle, or at least approximately. This will ensure that the relationship of two measured brightness values only by the projection of the Pattern is determined and independent of color, brightness, surface roughness and surface tilt of the object and especially important, regardless of the viewing angle, and for that both cameras is the same.

For example is from the 2D measuring technique with image processing known that workpiece edges from lighting technology establish depending on the severity (Rounding, chamfer ...) can provide different measurement results. By independence from the surface inclination Here such problems are eliminated.

Prefers is not just one, but at least two random or Pseudo-random patterns worked; Such solutions are depending on the technical Approach (s.u.) easier to implement and also provide more significant Measurement results (stronger local Structuring according to ratio formation).

The mathematical preparation of the correspondence can be done as in the above-mentioned document US 6082672 In contrast, however, not on the basis of a comparison of colors of a passing over the field of view color gradient, but due to the comparison of the brightness ratios (quotient) of pixels of the two images with different random patterns.

at the correspondence determination is omitted the above-described problem of the exact tuning of the cameras on top of each other, if not just the ratios of the brightness values compares but local histories the circumstances the brightness values, which, for example, by piecewise normalized one-dimensional correlation. along the epipolar lines can realize.

Once the correspondence is made, the distance image can be calculated by known methods (see eg [3] or the above-mentioned font) US 6082672 ).

at non-binary Random patterns, except for unlikely special cases, are one clear correspondence determination pixel by pixel and via interpolation even subpixelwise possible.

The solution avoids the above-cited problems with color evaluation, as well as the above-described general disadvantage of methods with over the Field of view of continuous, opposing lighting patterns: the random patterns allowed have locally high contrast. The solution is based on the knowledge the existence locally continuous course, as previously proposed, not is required. it is sufficient, if, at corresponding points, the COURSE OF O.G. RELATIONSHIP the brightness values are the same or even only the same. It may only not locally periodically recurring courses of relationships occur to avoid ambiguity. For pseudo-random patterns this is over the minimum length closed, unlikely for random patterns due to random nature. Nevertheless, occasional cases may be due to repetition with at least Another random pattern is virtually excluded.

The above problems with textured lighting patterns with texturelementeweiser correspondence determination are by the pixel-wise works and over the ratio calculation avoided; an image analysis to determine the position of the texture elements eliminated.

on the other hand becomes due to the texture-like random pattern, with pixelwise Money bill the o.g. Problem in opposite directions Illumination pattern over The field of vision avoids that with a flat original the brightness values little difference between neighboring pixels: due to the random nature This case can only be for single or very little directly adjacent pixels occur.

The method can advantageously be realized several times with different illumination units, each consisting of one or more projection units, with fusion of the results, local of the global realized on the basis of security measures (eg due to contrast, hiding results from brightness overrun regions) or due to averaging or majority voting. A particularly practical advantage in this case is the automatic suppression of local overmodulation with local gloss (gloss angle condition only fulfilled for one of the lighting units).

pixel way in the sense of the claims is called for each a camera that measures the brightness of pixels of the same image coordinates in relation to each other be set (or at least approximately same; local averages can be useful, e.g. For Subpixel algorithms, a systematic small offset may prove to be prove necessary). For dynamic scenes, e.g. with moving workpieces, are according to the displacement vector field (in the image area) offset pixels in proportion. In simple make (known workpiece displacement, e.g. on conveyor belt, langbrennweitige viewing, small height differences) is the displacement vector field known in advance. With decreasing focal length and with increasing Height differences however, the displacement vector field is increasingly of the end result, the height the object points, depending. The solution The vicious circle should be possible with iterative methods (Estimate of the displacement vector field with increasing accuracy, optimization with a quality measure to minimizing mean depth variation and similarity to be maximized from corresponding ratios) - a mathematical Challenge, but in the Applicant's opinion, no fundamentally unsolvable problem.

technical realization of random patterns:

A first technical realization (without figure) is the use of a Beamers (micromirror array) for programmable projection of Pseudorandom patterns.

The following are based 1 to 11 described several solutions according to the invention, wherein those without mechanical movement are preferred.

The Arrangements are preferably made such that the random patterns in the picture roughly along the epipolar lines; across to it is it rather cheap if only slight brightness variations are present (important for inaccurate Camera calibration).

2 shows a scene with a document 1a on which a workpiece 1b lies. document 1a and workpiece 1b become short as an object 1 designated. It is the height of the surface points of the base and workpiece, in short, the object points to determine. Of course, the scene can only consist of a section of a workpiece. The scene is about two stereo cameras 2 considered and an approximately punktlichtförmige light source 3 illuminated. In the illumination beam path is a mask 4 , locally different transparent in the form of a pseudo-random pattern. The pattern preferably consists of stripes which, in the images, are roughly transverse to the epipolar lines. The mask can be changed by electrical control, for example as known from liquid crystal displays; In this way, different patterns can be realized with known display technology. It is thus also possible to realize interwoven different color patterns. The light source should be as point-light as possible in order to avoid too strong superposition of rays of different intensity. Out 3 on the left you can see how else at one point light and dark spots can overlap in the projection. However, a certain smearing (blurring) of the imaged pattern does not matter in principle. An improvement shows 3 right, by classic illustration with a condenser 6 and a lens 7 , Here too, however, attention must be paid to the smallest possible aperture (depth of field). The projection unit 5 consists here of lighting and controllable mask, if necessary with condenser and lens.

It is of course possible to work with several projection units in time and to merge the results (local averaging or selection based on quality measures); should they be active at the same time, then, as in 4 shown to make sure that the projected patterns coincide as well as possible, otherwise it comes as in 3 left, to overlay light and dark sample spots.

5 shows an arrangement with two closely adjacent, approximately point-shaped light sources, such as LEDs whose light cone radiate different brightness patterns. One strives for the production of LEDs as homogeneous as possible light cone, due to irregularities of the surfaces (lens effects), material inclusions air bubbles, geometry deviations, etc., however, result in more or less random inhomogeneous structures. These dirt effects 10 can be used to coincidence Project to project. Of course, some of these effects can also be intentionally induced. One can achieve the same effect instead of dirt effects by intention of an appropriately structured mask. On the one hand, the two LEDs now project different random patterns but, on the other hand, due to their proximity, project approximately from the same solid angle. In the drawing has the front LED 8th a highly structured, the rear 9 a homogeneous cone of light. The projection unit 5 consists here of both LEDs and possibly mask.

6 shows arrangements with several projection units. It is essential that the light cones of the projection units, if they are switched on at the same time, either do not overlap or only slightly overlap, or that they correspond to 4 are coordinated. Line-shaped projection units according to 7 at the bottom, they should be oriented transversely to the epipolar lines; As mentioned above, the random patterns in the image should be roughly along the epipolar lines and have only slight variations in brightness across them.

7 demonstrates the moiré principle, to introduce the solution 8th , 7 shows up a regular grid 11 , including another regular grid with slightly different lattice constants. By overlaying the grid 11 and 12 creates an interference pattern (middle). A slight shift of the grids against each other roughly changes the phase of the reference pattern (below). 8th shows in the upper double row the superposition of a uniform grid 11 with a grid 13 that is opposite lattice 11 has pseudo-random deviations of the local lattice constants. Overlaying creates an appropriate pseudo-random pattern here. The technically interesting thing about this is that a very small change in the relative position of the gratings results in a rough change of the pseudo-random interference patterns ( 8th second double row). So you can with a very small movement of a grid, for example by Piezo elements, produce gross changes to pseudo-random patterns. The third and fourth duplexes show the same principle, superimposing two different pseudo-random patterns of equal or nearly equal lattice constants, with random phase shifts of 180 degrees (preferably so that no two opaque lines may follow each other).

A slight twisting of the patterns against each other results in a moire, across to the desired Moire, However, this is "long-wave" and therefore not disturbing.

8a shows in addition that even when superimposing simple, pseudo-random bar patterns, different random patterns can be generated by a shift. Here, however, a larger shift is required than in moiré technique; the latter is therefore preferred by the applicant.

9 shows a solution with a movable mask 14 with a fixed pseudo random pattern 15 with the mask moving in the illumination cone. The two images are taken at different times in such a way that they effectively take different lighting patterns. In the drawing, especially the mask is rotated, it can of course just oscillate linearly or rotationally. It is only necessary to take care with the image recordings that they take place for both cameras exactly at the same time and with the same integration intervals. This requirement does not apply if the mask through. a mechanical device can assume two discrete fixed positions.

1 : shows a preferred solution without movement. The random patterns are here, as in 8th in Moirétechnik realized by superposition of gratings, wherein at least one of the lattice constant varies in a pseudo-random manner. According to the invention, the grids are now at a distance d from one another so that different moiré patterns result when illuminated from different directions, without any mechanical movement being required. For example 1 the grids are roughly arranged transversely to the epipolar lines, so that the moire random patterns are roughly transverse to the epipolar lines. In the example, the upper mask is 21 uniform, the lower mask 22 structured in a pseudorandom way. The scene is illuminated by a projection unit, which consists of two approximately punctiform or linear or, by the composition of point sources, linear light sources. 1 shows two light sources 23 , consisting of two rows each 26 from point sources 27 , eg LEDs, which together form approximately approximately linear light sources; they are advantageous at least approximately aligned as the grid. To generate two different pseudorandom patterns, the rows of a projection unit are turned on one at a time. This creates different patterns in the scene. The big advantage of this approach is that a) no movement is required, b) the resulting patterns are drastically different, that c) the illumination is effectively made from almost the same solid angle (important, so) and that d) the characteristic the pseudo-random pattern can be adapted to the application.

With several projection units 23 , as in 1 In addition, with the same grids, a (again different) pair of pseudo-random patterns can be generated, which, however, arises when illuminated from a deliberately different spatial direction. Thus, a further, independent evaluation with fusion of the results can be realized, as described above for different lighting units; Advantages see also there.

The with inventive solutions described illuminated patterns, including moire-forming patterns Naturally in analogous way instead of transmitted light entprechende incident light solutions with locally diverse reflecting mirror surfaces.

10 shows a mask, which consists of a transparent container 16 exists in which stochastisch partially transparent particles 17 move, eg in a liquid. Such an arrangement could play a role, for example, in microscopy. Again, make sure that the images are synchronized with each other.

Of course, the solutions can be after 9 and 10 as well as the after 2 complemented by the classical projection arrangement with condenser and objective ( 3 ). This also applies to the arrangement 1 , where you will put the focus plane between the two patterns and a suitable beam path ensures that comes in the two images the effective light source (analogous to the pupil of a recording device) of the projection assembly to lie at two slightly different locations (eg with two laterally offset Light sources in front of the condenser, the light source is usually imaged in the imaging lens).

Another solution with movement (not shown) is lighting elements with random or poseudo randomly structured light cone, such as 5 , No. 8, possibly ordered by 6 , to move, eg by vibration of the attachment and possibly additionally by flexible suspension of the lighting elements.

11 shows a scanner solution: a slit of light is placed on a deflector 18 projected, in the figure a rotary polygon as an example, and from there to the scene. The light source 3 Here, for example, a laser light source with cylindrical lens for beam expansion (not shown), as usual with laser scanners. The light source is brightness modulated in a random or pseudo-random manner. Integration time of the cameras and deflection speed are coordinated so that the scene in the area of interest is covered within an integration interval. There are two images per camera realized. The cameras do not need to be tuned with the deflection unit. This is an advantage over the approach described above US 6,600,168 , The random generator does not need to be synchronized with either the cameras or the deflection unit.

to Separation of the two lights are color and polarization though possible, but are not preferred; Color because of the problems quoted above in color evaluation, polarization due to possible viewing angle dependent polarization effects, especially on non-metallic surfaces (near the Brewster Angle).

to Realization with color happen the first and the second illumination with different colors, preferably simultaneously, and the Image capture with color cameras to separate the color channels.

to Realization with polarization happen the first and the second Illumination with different polarization, preferably simultaneously, and in image acquisition the channels are separated by beam splitters (e.g. Beam splitter cube, semipermeable Mirror) and downstream polarization filters separated.

Advantages:

The Procedure also works for colored objects, even with strong color saturation (a problem with the Coding with color samples, see [1]).

It are for the creation of a complete distance image only 2 shots with Standard cameras required.

In classical stereo methods, correspondence can only be calculated for brightness discontinuities, for example at object edges. With the methods presented here, this restriction does not apply; for the Korres In addition, a complex image analysis for determining picture elements (edges, corners, blobs, etc.) is unnecessary.

One The main advantage of the procedure is that local discontinuities of the Brightness, by coloring, imprint, Dirt, voids, marks, processing marks, oil film and Like. Do not matter (they can for the correspondence u.U. even helpful). This also applies to natural structures, like them e.g. occur in textiles.

The above problem of division by small numbers (dark surface section and small brightness) plays a minor role here so far, as due to the random structure only small local areas of it can be affected, which is why from this only small disparity errors result in the correspondence determination.

One application-technical advantage is that the lighting does not need to be calibrated. So you can on the one hand a Unit with permanently connected precalibrated stereo cameras use, on the other hand, one or more lighting units, consisting of one or more projection units, which depend on the concrete task can be mounted anywhere in the room (Accessibility, Avoiding gloss angles and shadows) without any lighting calibration and without strict demand for a sharp mapping of the random patterns; the lighting units need just "hit" the objects roughly.

Of course you can a 2-camera stereo arrangement also with a single camera in various positions can be realized, e.g. on a robot arm.

Of course it affects the same procedure with arrangements with more than two cameras, with Stereo evaluation between camera pairs; if necessary with fusion results due to safety measures (e.g., due to contrast, Hide results from brightness overdriven regions) or Majority decisions.

Of course you can the described procedure also several times in different position relative to the object surface be realized, e.g. with the object or with cameras and if necessary also lighting on a robotic arm, with subsequent fusion the Results.

Of course they can Here also described method for a combination of distance images be used with gray value images or color images. This is e.g. in body scanners for purposes the visualization usual.

Non-patent literature:

• [1] Review of 20 years of range sensor development. journal of Electronic Imaging, 13 (1): 231-240. Jan. 2004. National Research Council Canada.
• [2] Paul J. Besl: Active, Optical Range Imaging Sensors. Machine Vision and Applications. (1988) 1: 127-152.
• [3] Yi Ma, Stefano Soatto, Jana Kosecka, S. Shankar Sastry: An Invitation to 3D, Springer Publishing 2004.
• [4] J. Battle, E. Mouaddib, J. Salvi: Recent Progress in Coded Structured Light as a Technique to Solve the Correspondence Problem. A survey. Pattern Recognition, Vol. 31, no. 7, p 963-982, 1998.
• [5] http://www.3dmd.com/AboutUs/Technology.asp, downloaded on Jan. 7, 2006
• [6] D. Viejo, J.M. Sa'z, M.A. Cazorla, F. Escolano: Active Stereo Based Compact Mapping. Proc. of the IEEE / RSJ Intern. Conf. on Intell. Robots and Systems. Canada, August 2005

Claims (18)

Method for creating a distance image from the correspondence of pixels of the images of a first camera and a second camera in stereo arrangement, characterized by a first illumination of the scene with a random or pseudo random pattern with a first picture taken with both cameras, a second one Lighting the scene, taking a second picture with both Cameras, for both cameras pixel by pixel calculation of a brightness ratio from the first and the second picture, making the correspondence of pixels due to comparison of the brightness ratios of pixels of different cameras. The method of claim 1, with the second illumination by another random or pseudo-zu case pattern. The method of claim 1, wherein the comparison of brightness ratios on associated epipolar lines of different cameras takes place. The method of claim 3, wherein the correspondence is produced by comparing the values of piecewise, one-dimensional, normalized correlation of the brightness ratios along the epipolar lines. Method according to one of the preceding claims, wherein the first and the second lighting with different colors takes place, preferably simultaneously, and color cameras are used. Method according to one of the preceding claims, wherein the first and the second illumination with different polarization takes place and wherein the image recording the separation of the channels Beam splitter and downstream polarization filter happens. Apparatus for creating a distance image of a Scene consisting of at least two cameras in stereo arrangement and one Image evaluation unit for evaluating the camera images and at least a projection unit for the lighting of the scene, wherein with the projection unit at least two different brightness patterns can be projected, of which at least one is a random or pseudo-random pattern, and where with each the cameras taken a picture with each of the brightness patterns and with the image evaluation unit for each of the cameras pixel by pixel The relationship the for calculated different brightness patterns measured brightness values can be. Device according to the preceding claim, characterized characterized in that at least two brightness patterns random or Are pseudo-random patterns. Device according to one of claims 7 to 8, characterized by a projection unit with two superimposed grids facing each other have a distance, where at least one is a pseudorandom structure and with lighting having at least two light sources, which can shine through the grids or illuminate them, and thereby giving different patterns. Device according to one of claims 7 to 8, characterized by a projection unit with two superimposed grids to each other have a distance, wherein at least one of the phase and / or the Frequency in pseudo random Way, and with a lighting with at least two light sources, the through the grids or illuminate them, and with different moire patterns result. Device according to one of claims 7 to 8, characterized by a locally differently transparent mask in the form of a Pseudo-random pattern, which can be changed by electrical control is. Device according to one of claims 7 to 8 characterized by one or more projection units, each consisting of at least 2 close approximate punctiform or linear Light sources whose light cone different brightness patterns radiate. Device according to one of claims 7 to 8, characterized by a projection unit with two mutually movable, superimposed Patterns, wherein at least one has a pseudo-random structure. Device according to one of claims 7 to 8, characterized by a projection unit with two mutually movable superimposed Lattice patterns, wherein at least one of the phase and / or the Frequency in pseudo-random Way varies. Device according to one of claims 7 to 8 characterized by a projection unit with a locally different transparent Mask in the form of a pseudo-random pattern, which can be moved in rotation or in translation. Device according to one of claims 7 to 8 characterized through a projection unit with a mask consisting of a transparent container exists, in which stochastisch partially transparent particles move. Device according to one of claims 7 to 8 characterized by a projection unit of at least one approximate punctate or linear Light source whose light cone emit brightness pattern with fixed pseudo-randomness patterns, wherein the light sources are moved by vibration. Device according to one of claims 7 to 8 characterized through one over the scene moving slit of light that in random or pseudorandom Is brightness modulated.