EP2094073A1

EP2094073A1 - Method and arrangement for the steering of a vehicle

Info

Publication number: EP2094073A1
Application number: EP07846679A
Authority: EP
Inventors: Nico Correns; Enrico Geissler; Michael Rode; Christoph Nieten; Tobias Neumann; Ruediger Kuehnle
Original assignee: Carl Zeiss MicroImaging GmbH
Current assignee: Carl Zeiss Microscopy GmbH
Priority date: 2006-11-27
Filing date: 2007-11-20
Publication date: 2009-09-02
Also published as: US20100063681A1; DE102006055858A1; WO2008064800A1; JP2010510918A

Abstract

The invention relates to a method by which image data from the terrain lying in front of a vehicle (1) in the direction of travel are detected, and from which data steering commands to influence the direction and/or the speed of travel are generated. The invention relates further to an arrangement for steering an agricultural vehicle (1) according to this method. According to the invention, the problem is solved in that prominent objects (3) are selected by means of the image data, the distance between the vehicle (1) and the prominent objects (3) is determined, and the steering commands are generated from the image data which correspond to the objects and from the changes of distance between the vehicle (1) and the objects (3).

Description

Method and arrangement for controlling a vehicle

The invention relates to a method in which image information is detected from the lying in front of a vehicle in the direction of travel terrain and from it control commands for influencing the direction of travel and / or the vehicle speed are generated. Furthermore, the invention relates to an arrangement for controlling an agricultural vehicle according to this method.

The driver, particularly an agricultural vehicle, such as a tractor, combine harvester, or forage harvester, is usually entrusted with a variety of other tasks in addition to steering his vehicle, for example control of the ejector arm, monitoring of the filling in the grain tank, control of the settings of a field sprayer, a plow or thresher.

To relieve the driver in these tasks, automatic steering devices have already been developed. Accurate tracking is important to avoid damage to crops and yield losses, and to optimize the overlapping of crop rows, for example, during grain harvesting, with the aim of saving time and fuel costs, thereby increasing the efficiency of agricultural land management ,

It is known to realize the automatic control of a vehicle based on a GPS-based arrangement. There is a GPS receiver on the vehicle, and By means of software, the dimensions of the field and an optimal route to cope with the planned work are calculated, used to generate control commands and transmitted to the steering device.

A disadvantage of this principle is the inadequate for many applications accuracy: For example, to drive a field on which the lanes between the rows of plants just the tire widths and the track width of the vehicle are appropriate, not to damage the seedlings, a maximum lane deviation of 8 cm allowed. This is not achievable with a satellite-only GPS arrangement.

To increase the accuracy, differential GPS systems have already been developed for the aforementioned purpose. The receiver on the vehicle is not only supplied with the satellite signals, but also with information from a stationary transmitting station.

In this way it is possible to improve the accuracy of tracking to about ± 5 cm. Such a system is described, for example, in US 2003/0208311 A1.

In US 6,539,303 a further procedure is disclosed, are automatically controlled in the mutually parallel moving vehicles.

The two systems presented, however, have in common that a very accurate knowledge of the location of obstacles or predetermined lanes must be present. However, when generating steam from crops, such information is not available. Furthermore, a spatial drift of the system used must be tracked, for example due to the time between sowing and harvest.

There are further developed methods and arrangements for controlling vehicles in which information about the terrain surrounding the vehicle is used to generate the control commands, as described for example in US 6,278,918 and US 6,686,951.

The terrain in front of the vehicle is observed with the help of a camera, and in a subsequent image analysis, the occurring structures are evaluated with different algorithms. For example, for the detection of swathing, a distinction is made between the swath and the mown stubble field on the basis of color differences.

The problem is that the colors of swaths and subsoil are dependent on conditions such as wetness and dryness, so that the color difference is often too small. The evaluation of the recorded images is also dependent on the ambient light. Shadows, in particular the shadow of the vehicle in question, generate a high degree of dynamics in the signals to be processed, thereby making automatic interpretation considerably more difficult.

In the publication "Schwadabtastung with ultrasound" in the journal Agricultural Engineering Issue 5/1993, from page 266, a procedure is described in which a plurality of ultrasonic sensors are arranged in series parallel to the ground Each of these sensors determines the distance to the ground and provides a " Pixel "of a height profile, the off the distances determined by all involved sensors is obtained. Each scan provides two-dimensional height profile information. With repeated scans during the movement of the vehicle, a three-dimensional information of the height profile is then obtained.

According to this principle, only relatively coarse structures can be detected, such as the height and width of a swath lying on the scanned ground.

Other known devices use a stereo camera for image acquisition, as disclosed, for example, in EP 1 473 673. By analyzing two images obtained from different perspectives from the vehicle, a three-dimensional profile of the surface is created by translating the two-dimensional deviations from identical objects on both images into a spatial depth.

The disadvantage here is that the positions of the two cameras must be known very well and may not change on the vehicle.

In an arrangement using a laser scanner-based distance measuring device as described in US Pat. No. 6,389,785, a terrain line perpendicular to the direction of travel is scanned point by point and its distance is measured. During locomotion of the vehicle, the measurement data is assembled into a three-dimensional profile.

Due to the scanner, moving parts are used for image acquisition. Especially those on a farmer's Shock and vibration occurring therefore easily lead to interference of the measuring system and also limit the life of the scanner.

Based on this prior art, the object of the invention is to further develop the method for automatically controlling a vehicle, in particular a vehicle for agriculture. The object of the invention is also to provide an arrangement which enables improved control of an agricultural vehicle with regard to the efficiency of the management of agricultural land.

According to the invention, the object is achieved with a method for controlling a vehicle in which image information from the lying in the direction of travel in front of the vehicle terrain and generated from these image information control commands for influencing the direction of travel and / or driving speed are achieved in that based on the image information distinctive object structures are selected, the distance between the vehicle and the distinctive object structures is determined repeatedly and the control commands are generated from the image information corresponding to the object structures and the distance changes between the vehicle and the object structures.

Certain plants, rows of plants, furrows, swaths of crop, roadsides or lanes can be used as distinctive object structures, and the control commands are generated to influence the direction of travel relative to such object structures. In other words, the image information obtained from the distinctive object structures and the distance changes between the vehicle and the object structures related to this image information are converted into control commands which influence the steering of the vehicle and correct the direction of travel in such a way that the wheels of the vehicle automatically in intended tracks, for example, in precise alignment between the rows of plants, run without the driver has to intervene.

In an advantageous embodiment of the method according to the invention, control commands for influencing the direction of travel or driving speed for the purpose of avoiding obstacles are generated. This serves in particular to recognize objects that can endanger the vehicle due to their size and nature. It is also possible to detect traveling objects, such as other vehicles or persons, and to generate control commands for influencing the direction of travel or driving speed and / or warning signals from these image information.

In a further advantageous embodiment, control commands are also generated to influence the driving speed or the direction of travel in order to influence the distance and the orientation of the vehicle to be controlled to other vehicles. Thus, it is for example possible to synchronize the driving speed or direction of travel of a harvesting vehicle to the driving speed or direction of one or more vehicles for transporting the crop. The transfer of the crop from the harvester to changing transport vehicles can be done without interrupting the harvesting process, so that the Harvesting vehicle is effectively utilized. This method is particularly suitable for use in mobile shredders or grain harvest.

For example, many agricultural machines have several agricultural machines in one field at the same time. This can be a combination of combine harvesters slightly harvesting crops from one another, or harvesters that load trailers on tractors that move parallel to the machine. In all these cases, situations in which the direction of travel and speed of each machine are important to each other to ensure a smooth flow.

Depending on the embodiment of the method according to the invention, the image information is obtained with light from the infrared range and / or from the visual range. Thus, it is possible to spectrally resolve the image information due to the evaluation of certain colors, for example, to improve the swath detection or to obtain information about the density of a swath. The control of an agricultural vehicle is carried out depending on the amount and quality of the crop.

In a further embodiment of the method according to the invention, it is provided to generate acoustic warning signals for the vehicle driver at the same time as the control commands or else separately, which can then control the effectiveness of the automatic control and the corrective measures. In an arrangement for controlling an agricultural vehicle, which is equipped with a drive for moving the vehicle, a steering device for determining the direction of travel, an acceleration and braking device for influencing the speed of movement, an optical device for detecting image information from the in the direction of travel before the Vehicle lying terrain, and with an evaluation and processing device for generating control commands for the steering device and / or for the acceleration and braking device from the image information are provided according to the invention:

Means for selecting image information corresponding to distinctive object structures, and a distance measuring device for clocked, repeated determination of the distance between the vehicle and these object structures, wherein the evaluation and processing means for generating the control commands from the image information corresponding to the distinctive object structures, and the Distance changes between the vehicle and the object structures is formed.

Advantageously, the optical device has means for detecting the image information originating from object structures lying in the space in front of the agricultural vehicle on a straight line, this straight line with the direction of travel an angle of α ≠ 0 °, preferably α ~ 90 °. Alternatively, the optical device has means for detecting the image information originating from object structures lying in the space in front of the agricultural vehicle in a surface, said surface with the direction of travel of the vehicle an angle of α ≠ 0, preferably α «90 ° , includes.

The optical device for acquiring the image information is equipped with individual optical sensors, optical sensor lines and / or optical sensor arrays. At least one of the sensors is designed as a phase-sensitive sensor and provided for detecting the changes in distance.

In this regard, a runtime camera is advantageously used as an optical device. While a conventional camera is equipped with optical sensors that capture only the brightness of pixels, the runtime camera has instead of or in addition to these sensors via phase-sensitive sensors that measure not only the brightness but also the duration of the light that carries the image information, and thus distance measurements enable. This is done with the aid of a separate, modulated light source for illuminating the objects to which the distance is to be measured. Thus, the runtime camera not only obtains image information of objects, but also distance values to these objects.

When using a runtime camera, the terrain is illuminated in front of the vehicle with a light source, which is advantageously integrated in the runtime camera, and which is modulated sinusoidally, for example, with a frequency f. The light propagating at the speed of light c Ie has the period 1 / f. The distance z to a selected illuminated object or to an illuminated object structure is calculated from the measurement of the phase shift, which occurs during the propagation time of the light, according to the function z = - * - ^J - with c the speed of light -

2 / 2π speed, f the modulation frequency and φ the phase shift. The distance is thus determined on the basis of the phase shift φ.

This distance measuring method, which is also known by the term "time-of-flighf principle", is described in detail, for example, in the periodical "Elektronik", WEK Fachzeitschriften-Verlag GmbH, 2000, No. 12, under the title "Photomixdetektor captures 3D images" The description contains the dissertation "Examination and Development of Modulation Term-based SD Viewing Systems", presented by Horst G. Heinold, the Department of Electrical Engineering and Computer Science of the University of Siegen. A more detailed explanation is therefore not required here at this point.

The image capture is designed in the inventive arrangement so that, for example, for a swath detection, a resolution of about 10 cm to 20 cm is possible. By contrast, when acquiring image information from plant rows, the arrangement is advantageously designed for accuracies of 5 cm to 10 cm.

Cylindrical lenses or prisms may advantageously be arranged upstream of the optical device for the purpose of achieving an unevenly distributed optical resolution in the acquisition of the image information. The cylinder liners sen or prisms formed so that certain sensor areas of the optical device capture image information from the terrain in front of the vehicle with higher resolution than the other sensor areas.

The invention will be explained in more detail with reference to an embodiment. In the accompanying drawings show:

Fig.l symbolically the side view of an agricultural vehicle in a cultivated field,

2 shows the agricultural vehicle from Fig.l in a plan view,

3 is a block diagram of the signal flow in the extraction of image and distance information and their implementation in control commands.

The agricultural vehicle 1 in Fig.l comprises the following, not shown in detail in the drawing devices: a drive for locomotion, a steering device for influencing the direction of movement, an acceleration and braking device for influencing the speed of movement, a runtime camera 2, which is used to capture image information distinctive object structures 3, which lie in the direction of travel in front of the vehicle, and for clocked, repeated determination of the distance between the vehicle 1 and the object structures 3 is formed, and an evaluation and processing device for generating control commands for the steering device and for the acceleration and braking device from the image information and the distance changes.

The runtime camera 2 has in the Fig.l side view shown a viewing area 4 with a viewing angle ß of about 15 ° to 40 °. The viewing angle β is dependent on the purpose of the agricultural vehicle 1 and predetermined by constructive measures. Thus, for example, when using the agricultural vehicle 1 for maize harvest, a viewing angle β = 15 ° is sufficient, while in the tracking of plant rows, for example in order to be able to recognize splitting rows, a larger viewing angle β is desirable.

2 shows a top view of the illustration according to FIG. 1, from which the lateral viewing angle γ of the viewing area 4 of the transit time camera 2 can be seen, which lies for example in the order of magnitude of 40 ° to 140 °. The viewing angle γ is also dependent on the use of the agricultural vehicle 1 and predetermined by constructive measures. Thus, for example, a viewing angle γ = 40 ° suffices for the detection of swaths, while in the case of an agricultural vehicle 1 designed as a combine harvester, a viewing angle γ is required which covers the entire width of the cutting mechanism.

The runtime camera 2 includes an array of sensors, of which at least one, but preferably a plurality, particularly preferably are all phase-sensitive and provide both brightness and distance signals, and whose signal inputs are connected to the preferably internal and external signals. half of the vehicle 1 accommodated evaluation and processing device are connected.

When operating the arrangement, the runtime camera 2 takes pictures of the lying in the direction of travel in front of the vehicle terrain and transmits the image information obtained, the arrangement of the sensors according to one or two-dimensional, caused by the object structures 3 brightness distribution to the evaluation and processing device.

As the vehicle 1 moves, the distance a between the transit time camera 2 or the vehicle 1 and the object structures 3 detected by the transit time camera 2 in its field of view 4 changes as a function of the travel speed.

Thus, in addition to the brightness values, information about the distances a from the object structures is also obtained from the phase-sensitive sensors in accordance with the "time-of-flight principle." At short intervals of time, which specifies a clock that is advantageously integrated into the runtime camera 2 This information is constantly updated and on this basis in the evaluation and processing device continuously generates a three-dimensional image of the space in front of the vehicle 1. It goes without saying that the clocks per unit of time are many times higher than those in the same time unit of Vehicle 1 traveled distance.

In order to obtain the information about the distances a, phase-sensitive sensors are provided, which lie either in one line or span a surface, so that image information mation of structures of several objects are detected, which are in a row, or image information of structures of multiple objects are detected, which lie in a plane.

The three-dimensional image of the space in front of the vehicle 1 is permanently evaluated in order to recognize the striking object structures 3 from image to image. Depending on the application of the arrangement according to the invention, such object structures 3 correspond to swaths, plant rows, cut edges in the grain harvest, regions with bent over cereals or even rows of maize or soybean plants. For example, plant rows 5 are shown in FIG. 2 which extend parallel to the direction of travel of the vehicle 1 and which are separated from each other by furrows which serve as lanes 6.

From the image to image comparison, the information about the change of the position of the vehicle 1 relative to the object structures 3 are obtained by the evaluation and processing device in addition to the information about the distance changes and derived therefrom correcting control commands for the steering device and the acceleration and braking device , transmitted to this and so affects the direction of travel and the speed of movement of the vehicle 1.

In a further embodiment of the arrangement according to the invention, the evaluation and processing device is also designed for color evaluation in the three-dimensional image, so that object structures 3 are distinguishable not only in terms of their size or shape, but also because of their particular color. For example, based on the information obtained, the plant rows 5 can be precisely distinguished from the lanes 6. For a particularly high accuracy in the corrective influence on the direction of movement of the vehicle 1 is achieved.

Configurations are also conceivable, for example for evaluating the three-dimensional image with regard to a possible inclined position of the vehicle 1 due to unevenness in the floor.

It is also conceivable to supplement the information arriving at the evaluation and processing device with information from a GPS system or also from a differential GPS system, thereby achieving further precision.

3 schematically shows a block diagram of the arrangement according to the invention, which itself is explained on the basis of the designations contained therein and the signal flow directions marked by arrows.

In contrast to the relevant methods and arrangements known from the prior art, in which a three-dimensional profile of the space in front of the vehicle is generated by means of a scanning optical system, the method according to the invention and the arrangement according to the invention forego scanning optical systems. This makes it possible to record the multiplicity of two-dimensional image information in parallel over time and, in contrast to the prior art, to simultaneously generate the three-dimensional profile. The method according to the invention, in particular the exemplary arrangement according to the invention, can furthermore be used to detect obstacles in the field of vision in front of the vehicle. Because the system creates a three-dimensional terrain profile, it is able to detect objects that are potentially hazardous to the machine due to their size, such as large stones, trees, other vehicles unexpectedly crossing the direction of travel.

In addition, relative velocities can be detected very accurately based on the obtained image and distance information. In particular, movements of an agricultural machine relative to an obstacle (eg another agricultural machine) in the direction of travel, the time-based image information acquisition has the advantage of having direct information about the distance, while the conventional procedure can only make a statement about the distance due to size changes in the image ,

The three-dimensional profile data of the space in front of the agricultural vehicle can, for example, continue to be used to adapt the tools coupled to the vehicle to the terrain profile. For example, the cutting unit of a combine harvester can be automatically height-adjusted to prevent contact with the ground or contact with a low obstacle and thus damage. The cutting unit can always be adjusted in height so that an optimal distance from the ground is maintained. The invention also includes embodiments in which the runtime camera is oriented so that a three-dimensional profile is obtained not only from a terrain section in the direction of travel, but also of terrain sections opposite or laterally to the direction of travel. Thus, for example, advantageously an adjacent vehicle can be monitored with regard to location, distance and / or relative speed, and on this basis an adaptation of the driving speed and direction of travel of the controlled vehicle can be carried out.

LIST OF REFERENCE NUMBERS

1 vehicle

2 runtime camera

3 object structure

4 viewing area

5 rows of plants

6 lanes

a distance f frequency

Claims

claims

1. A method for controlling a vehicle, in which image information from the lying in the direction of travel in front of the vehicle terrain and generated from these image information control commands for influencing the direction of travel and / or the vehicle speed, characterized in that based on the image information distinctive object structures (3) repeatedly, the distance (a) between the vehicle and the object structures (3) is determined, and the control commands are generated from the image information corresponding to the object structures (3) and the distance changes between the vehicle and the object structures (3) ,

2. The method of claim 1, wherein as a distinctive object structures (3) certain plants, rows of plants (5), furrows, swaths of crop, roadsides or lanes (6) are used.

3. The method of claim 1 or 2, are generated in the control commands for influencing the direction of travel for tracking relative to the object structures.

4. The method according to any one of claims 1 to 3, are generated in the control commands for influencing the direction of travel or speed for the purpose of avoiding obstacles.

5. The method according to any one of the preceding claims, wherein the control commands for influencing the driving speed or direction of travel of the vehicle for the purpose of synchronization to the driving speed or direction of at least one further vehicle are generated.

6. The method according to any one of the preceding claims, are generated in the control commands for influencing the driving speed for the purpose of influencing and / or keeping constant the work of agricultural machinery, which are coupled to the vehicle.

7. The method according to any one of the preceding claims, wherein the image information obtained with light in the infrared range and / or in the visual range.

8. The method according to any one of the preceding claims for controlling an agricultural vehicle coupled with harvesters (1), wherein the control commands are generated in dependence on the amount and the quality of the crop.

9. An arrangement for controlling an agricultural vehicle (1), comprising a drive for locomotion of the vehicle, a steering device for determining the direction of travel, an acceleration and braking device for influencing the speed of movement, an optical device for detecting image information from the in the direction of travel before the Vehicle (1) lying terrain, and an evaluation and processing device for generating control commands from the image information for the steering device, for the acceleration and braking device and / or for implements, which are coupled to the agricultural vehicle (1), characterized by

Means for acquiring the image information of distinctive object structures (3) in the field, and a distance measuring device for clocked, repeated determination of the distance (a) between the vehicle (1) and these object structures (3), wherein the evaluation and processing device for generating the Control commands from the image information, which correspond to the object structures (3), and the distance changes between the vehicle (1) and the object structures (3) is formed.

10. Arrangement according to claim 8, in which the optical device has means for detecting the image information originating from object structures which lie in the space in front of the agricultural vehicle on a straight line, this straight line with the direction of movement an angle of α ≠ 0 °, preferably α "90 °.

11. The arrangement of claim 10, wherein the optical means comprises means for detecting the image information derived from object structures lying in the space in front of the agricultural vehicle in a surface, said surface with the direction of travel of the vehicle an angle of α ≠ 0th , preferably α "90 °, includes.

12. Arrangement according to one of claims 8 to 12, wherein the optical device for detecting the image information with individual optical sensors, optical sensor rows and / or optical sensor arrays is equipped.

13. Arrangement according to claim 13, wherein at least one of the sensors is designed as a phase-sensitive sensor and is provided for detecting the changes in distance.

14. Arrangement according to one of claims 8 to 14, wherein the optical device for detecting the image information on the vehicle (1) is fixed and during the locomotion of the vehicle (1) is in relative rest to this.

15. Arrangement according to one of claims 8 to 15, wherein the optical means are arranged in front of cylindrical lenses or prisms to cause an unevenly distributed optical resolution in the detection of the image information.

16. Arrangement according to claim 16, wherein the cylindrical lenses or prisms are formed so that certain sensor areas of the optical device capture image information from the terrain in front of the vehicle with higher resolution than the other sensor areas.