DE102019125612A1 - Method for the computer-implemented simulation of an optical sensor in a virtual environment - Google Patents

Abstract

The invention relates to a method for the computer-implemented simulation of an optical sensor (6) in a virtual environment, the optical sensor (6) providing depth information, with the following steps:
(S100) Reading in a value (W) indicative of a number of a plurality of partial frustums (F1, F2, F3) of a frustum (F) of the sensor (6) each with a near cutting plane (NS1, NS2, NS3) and one far cutting plane (FS1, FS2, FS3),
(S200) Forming the partial frustums (F1, F2, F3) according to the value (W) by arranging the partial frustums (F1, F2, F3) one behind the other along an optical axis (A) of the optical sensor (6), and
(S1300) Filling a Z-buffer (8) with depth data (TD) which have been obtained on the basis of the application of at least one partial frustum (F1, F2, F3).

Description

The invention relates to a method for the computer-implemented simulation of an optical sensor in a virtual environment, the optical sensor providing depth information. The invention also relates to a computer program product and a system for computer-implemented simulation of such a LIDAR sensor.

Motor vehicles can be designed for so-called autonomous driving. An autonomously driving motor vehicle is a self-driving motor vehicle that can drive, control and park without the influence of a human driver (highly automated or autonomous driving). In the event that no manual control is required on the part of the driver, the term robot car is also used. Then the driver's seat can remain empty; possibly the steering wheel, brake and accelerator pedals are not available.

Autonomous vehicles of this type can use various sensors to detect their surroundings and use the information obtained to determine their own position and that of other road users, navigate to a destination in cooperation with the navigation software and avoid collisions on the way there.

Such an optical sensor can be a so-called LIDAR sensor. LIDAR (abbreviation for light detection and ranging), describes a radar-related method for optical distance and speed measurement as well as for remote measurement. Instead of electromagnetic waves as with RADAR, laser beams are used.

LIDAR sensors offer the advantage that they can scan a 360-degree area around the motor vehicle with high resolution and speed. Typically, a LIDAR sensor uses an array of laser-based sensors (e.g. 64) that rotate at high speed (several 100 n / min). The LIDAR sensor is then able to detect obstacles hit by a laser beam. It is thus possible to determine the coordinates of every hit or of every object in the vicinity of the motor vehicle. By evaluating this LIDAR data, it is also possible to obtain information about the topology of the terrain in the area around the motor vehicle.

Furthermore, such an optical sensor can be a 3D camera or stereo camera. 3D cameras are camera systems that allow the visual representation of distances from an entire scene. Stereo cameras, for example, usually have two objects attached next to each other and, when triggered, enable the two stereoscopic half-images required for 3D images to be recorded at the same time.

In order to test such automated driving, the motor vehicles are tested in the real world. However, this is an expensive process and the risk of accidents is high. In order to avoid accidents and at the same time reduce costs, tests in the computer-generated virtual environments, such as tests in virtual cities, are necessary. The VR technology (Virtual Reality Technology) together with a virtual environment opens up many possibilities. The main advantage of VR technology is that it allows a user, such as an engineer, to be part of the tests, to interact with the test scenario or the configuration parameters.

From the US 8 665 260 B2 , of the US 7 525 542 B2 , of the CN 104 933 758 A and US 2016/0210775 A1, methods for computer-implemented simulation of an optical sensor in a virtual environment are known.

There is a need to show ways in which a computer-implemented simulation of an optical sensor in a virtual environment that provides depth information can be improved.

• Einlesen eines Wertes indikativ für eine Anzahl einer Mehrzahl von Teil-Frustums eines Frustums des Sensors mit je einer nahen Schnittebene und einer fernen Schnittebene,
• Bilden der Teil-Frustums gemäß dem Wert durch Anordnungen der Teil-Frustum hintereinander entlang einer optischen Achse des optischen Sensors, und
• Füllen eines Z-Puffers mit Tiefendaten, die auf Basis der Anwendung zumindest eines Teil-Frustums gewonnen wurden.

The object of the invention is achieved by a method for the computer-implemented simulation of an optical sensor in a virtual environment, the optical sensor providing depth information, with the following steps:

• Reading in a value indicative of a number of a plurality of partial frustums of a frustum of the sensor, each with a near cutting plane and a distant cutting plane,
• Forming the partial frustums according to the value by arranging the partial frustums one behind the other along an optical axis of the optical sensor, and
• Filling a Z-buffer with depth data that was obtained on the basis of the application of at least a partial frustum.

In other words, the frustum of the sensor, which can be a LIDAR sensor or a 3D camera, is divided into a predetermined plurality of partial frustums, with each partial frustum having a (virtual) area sensor for monitoring the respective partial frustum is assigned. Only objects in the respective partial frustum between the respective near cutting plane and the far cutting plane are recorded; objects lying outside the partial frustum are not taken into account. By providing a plurality of partial frustums, each with a near cutting plane and a distant one Cutting plane, the distance between the respective near and far cutting planes can be minimized, which increases the accuracy of the depth data. The depth data obtained in this way can then be used to render a scene in the virtual environment. In this way, a computer-implemented simulation of an optical sensor in a virtual environment that provides depth information can be improved.

According to one embodiment, the further steps

• Aussenden einer vorbestimmten Anzahl von Strahlen von dem Sensor zu einem Objekt,
• Bestimmen des Strahls der vorbestimmten Anzahl von Strahlen, der das Objekt trifft und die geringste Distanz zu dem Sensor aufweist,
• Bestimmen einer Position einer nahen Schnittebene unter Auswertung des Strahls mit der geringsten Distanz zu dem Sensor,
• Bestimmen eines weiteren Strahls der vorbestimmten Anzahl von Strahlen, der das Objekt trifft und die größte Distanz zu dem Sensor aufweist, und
• Bestimmen einer Position einer fernen Schnittebene unter Auswertung des Strahls mit der größten Distanz zu dem Sensor ausgeführt.

Detecting whether an object is located within a partial frustum, and following the detection of an object within the partial frustum:

• Sending out a predetermined number of beams from the sensor to an object,
• Determining the beam of the predetermined number of beams that hits the object and is the closest to the sensor,
• Determining a position of a near cutting plane while evaluating the beam with the smallest distance to the sensor,
• Determining a further beam of the predetermined number of beams which strikes the object and is the greatest distance from the sensor, and
• Determination of a position of a distant cutting plane carried out by evaluating the beam with the greatest distance to the sensor.

In other words, the respective position of the near cutting plane and a far cutting plane along the optical axis of the sensor is adapted if an object has been detected in the partial frustum that is delimited by the respective near cutting plane and a far cutting plane. Both the near cutting plane and the far cutting plane are then adjusted by shifting along the optical axis of the sensor, specifically in relation to the extension or expansion of the object in relation to the optical axis of the sensor. The extent of the object is determined by emitting a plurality of simulated beams, such as laser beams, from the sensor and determining the positions of impact on the object. In this way, non-linearities of the z-buffering of depth information can be counteracted, so that the accuracy of the depth information is further increased.

• Vergleichen der geringsten Distanz mit einem vorbestimmten Nah-Grenzwert, und
• Bestimmen einer Position einer nahen Schnittebene unter Auswertung des Nah-Grenzwertes, wenn die geringste Distanz kleiner als der Nah-Grenzwert ist ausgeführt.

According to a further embodiment, the further steps

• Comparing the smallest distance with a predetermined near limit value, and
• Determining a position of a near cutting plane while evaluating the near limit value when the smallest distance is smaller than the near limit value is carried out.

In this way, a minimum distance below which the sensor is unable to detect an object can be taken into account in a particularly simple manner. Furthermore, the case can be taken into account that an object extends with its extension or extent in relation to the optical axis of the sensor in an area below the minimum distance, i.e. the object is partly too close to the sensor.

• Vergleichen der größten Distanz mit einem vorbestimmten Fern-Grenzwert, und
• Bestimmen einer Position einer fernen Schnittebene unter Auswertung des Fern-Grenzwertes, wenn die größte Distanz größer als der Fern-Grenzwert ist ausgeführt.

According to a further embodiment, the further steps

• Comparing the greatest distance to a predetermined far limit, and
• Determining a position of a distant cutting plane while evaluating the distant limit value when the greatest distance is greater than the distant limit value is carried out.

A maximum distance above which the sensor is not able to detect an object can thus be taken into account in a particularly simple manner in an analogous manner. Furthermore, the case can be taken into account that an object extends with its extension or extent in relation to the optical axis of the sensor in an area above the maximum distance, i.e. the object is partly too far away from the sensor.

The invention also includes a computer program product and a system for computer-implemented simulation of such a sensor.

• 1 in schematischer Darstellung Komponenten eines Systems zur computerimplementierten Simulation eines optischen Sensors in einer virtuellen Umgebung.
• 2 in schematischer Darstellung weitere Details einer computerimplementierten Simulation eines optischen Sensors in einer virtuellen Umgebung.
• 3 in schematischer Darstellung einen Verfahrensablauf zum Betrieb des in 1 gezeigten Systems.

The invention will now be explained with reference to a drawing. Show it:

• 1 a schematic representation of components of a system for the computer-implemented simulation of an optical sensor in a virtual environment.
• 2 a schematic representation of further details of a computer-implemented simulation of an optical sensor in a virtual environment.
• 3rd a schematic representation of a process sequence for operating the in 1 shown system.

It gets on first 1 Referenced.

A system is shown 2 for the computer-implemented simulation of an optical sensor 6th in a virtual environment.

Virtual reality, or VR for short, is the representation and simultaneous perception of reality and its physical properties in a real-time, computer-generated, interactive virtual environment.

In order to create a feeling of immersion, special output devices (not shown), such as virtual reality headsets, are used to represent the virtual environment. In order to convey a spatial impression, two images are generated and displayed from different perspectives (stereo projection).

Special input devices (not shown), such as a 3D mouse, data glove or flystick, are required for interaction with the virtual world. The flystick is used for navigation with an optical tracking system, whereby infrared cameras permanently transmit the position in space to the system by capturing markers on the flystick 2 report so that a user can move freely without cabling. Optical tracking systems can also be used to record tools and complete human models so that they can be manipulated in real time within the VR scenario.

Some input devices provide the user with force feedback on the hands or other parts of the body (force feedback) so that the user can orientate himself through the haptics and sensors as a further sensory perception in the virtual environment.

In addition, software specially developed for this purpose is required to create a virtual environment. The software must be able to calculate complex three-dimensional worlds in real time, i.e. with at least 25 frames per second, separately in stereo for the left and right eyes of the user. This value varies depending on the application - a driving simulation, for example, requires at least 60 images per second to avoid nausea (simulator sickness).

The optical sensor 6th provides depth information and in the present exemplary embodiment can be a LIDAR sensor or a 3D camera or stereo camera.

From the other components of the system 2 are in the 1 a rendering engine 4th , Area sensors 6a , 6b , 6c of the sensor 6th and a z-buffer 8 is shown.

The system 2 who have favourited the rendering engine 4th , the sensor 6th with the area sensors 6a , 6b, 6c and / or the z-buffer 8 can have hardware and / or software components for their tasks and / or functions described below.

The rendering engine 4th is in the present embodiment a real-time graphics engine that uses Z-buffering, such as OpenGL or DirectX.

The rendering engine 4th is part of a computer program or computer hardware, a so-called engine, which is designed for the representation of the virtual environment with objects, an environment and people in the form of a computer graphic.

Z-buffering (also depth buffering or depth storage method) is understood to mean a computer graphics method for calculating occlusion, i.e. to determine the three-dimensional areas in the computer graphics that are visible to a user in the virtual environment. Through depth information in the form of depth data TD In the z-buffer 8, the method determines pixel by pixel which elements of a scene have to be drawn and which are covered.

This approach is known to be fast compared to other methods, such as ray tracing.

It is now additionally on 2 Referenced.

An optical sensor to be simulated is shown 6th with his frustration F. .

The frustum F. in computer graphics is the mathematical representation of a 3D universe on a screen.

Starting from the sensor 6th becomes an object through the vanishing point perspective O in depth, ie along the optical axis A. of the sensor 6th , smaller. A user gets the impression that he is looking at one end (the stump) of a cone. In 3D graphics, one also speaks of the visual volume. It is a “truncated pyramid” that represents the volume that can be seen by the user. A so-called frustum matrix depicts the 3D world on the two-dimensional screen.

The frustum F. thus has the shape of a truncated pyramid, the base of which in the present exemplary embodiment is in the direction of the axis A. last distant cutting plane FS3 . In the direction of the axis A. beyond the last distant cutting plane FS3 nothing should be visualized by rendering anymore.

In the present exemplary embodiment, the one in the direction of the axis A. In contrast, the first close cutting plane NS1 corresponds to a position of the user who is looking in the direction of the pyramid base. Everything in the pyramid between the first close cutting plane NS1 and the sensor 6th lies behind the user. The side walls of the pyramid are the levels that represent the edge of the screen.

To be in the field of vision of the sensor 6th to lie, the object must O within the frustration F. lie.

In the present exemplary embodiment, the rendering engine is 4th of the system 2 designed to have a predetermined value W. indicative of a number of a plurality of partial frustums F1 , F2 , F3 of frustration F. of the sensor 6th read in and one of the value W. corresponding number of partial frustums F1 , F2 , F3 to build.

In doing so, each has partial frustum F1 , F2 , F3 one near section plane NS1 each, NS2 , NS3 and a distant cutting plane FS1 , FS2 , FS3 on.

The rendering engine 4th arranges the part-frustums so formed F1 , F2 , F3 along the optical axis A. lying one behind the other. In the present embodiment, the near cutting plane falls NS2 of the second part frustum F2 and the far cutting plane FS1 of the first part frustration F1 together. Likewise fall the near cutting plane NS3 the third part frustum F3 and the far cutting plane FS2 of the second part frustum F2 together.

Every part frustum F1 , F2 , F3 is then an area sensor each 6a , 6b , 6c of the sensor 6th assigned.

The rendering engine 4th is also designed to respond to a detection of the object O in e.g. the partial frustum F1 a predetermined number of simulated beams ST from the sensor 6th , in the present exemplary embodiment from the area sensor 6a , to an object O to send out.

Furthermore is the rendering engine 4th designed to select the beam from the number of beams ST to determine who the object O hits and the smallest distance D1 to the sensor 6th having. The near cutting plane NS1 of the first partial frustum then becomes at the level of this position F1 arranged.

The rendering engine determines in the same way 4th plus another ray of rays ST who is the object O also hits and the greatest distance D2 to the sensor 6th having. The far cutting plane is then at the height of this position FS1 of the first part frustration F1 arranged.

In other words, the partial distance x1 of the total distance x is reduced. It can be provided that the partial distance x2 is adjusted accordingly, so that there are no gaps between the partial frustums F1 and F2 arise. In other words, the partial distance x2 of the total distance x is enlarged accordingly.

Furthermore is the rendering engine 4th designed for the smallest distance D1 with a predetermined near limit value NG to compare and the near cutting plane NS1 at the level of the position of the near limit value NG to arrange. So there can be a minimum distance below which the area sensor 6a cannot detect any object, can be taken into account particularly easily.

In an analogous way to achieve a maximum distance above which the range sensor 6a Cannot detect an object, it is provided that the greatest distance is taken into account 2 with a predetermined remote limit value FG is compared and at the level of this position the distant cutting plane FS1 of the first part frustration F1 is arranged.

Finally, the rendering engine 4 then fills the Z-buffer 8th with depth data TD that is based on the application of the partial frustum F1 were won.

It is now additionally on 3rd Reference is made to a process flow for operating the in 1 shown system 2 to explain.

In a first step S100 reads the rendering engine 4th of the system 2 the value W. indicative of a number of a plurality of partial frustums F1 , F2 , F3 of frustration F. of the sensor 6th on.

In a further step S200 forms the rendering engine 4th of the system 2 the part frustums F1 , F2 , F3 according to the value W. by arrangements of the part-frustums F1 , F2 , F3 one behind the other along the optical axis A. .

In a further step S300 captures the rendering engine 4th of the system 2 whether there is an object O within a partial frustration F1 , F2 , F3 is located.

If this is the case, sends in a further step S400 the rendering engine 4th of the system 2 a predetermined number of beams ST from the sensor 6th to the object O out.

In a further step S500 determines the rendering engine 4th of the system 2 the beam of the predetermined number of beams ST who is the object O hits and the smallest distance D1 to the sensor 6th having.

In a further step S600 compares the rendering engine 4th of the system 2 the smallest distance D1 with the predetermined near limit value NG .

When the slightest distance D1 smaller than the near limit value NG is determined in a further step S700 the rendering engine 4th of the system 2 the position of the near cutting plane NS1, NS2 , NS3 while evaluating the near limit value NG .

If, on the other hand, the smallest distance D1 greater than the near limit value NG is determined in a further step S800 the rendering engine 4th of the system 2 the position of the near cutting plane NS1, NS2 , NS3 evaluating the smallest distance D1 .

In a further step S900 determines the rendering engine 4th of the system 2 the beam of the predetermined number of beams ST who is the object O meets and the greatest distance D2 to the sensor 6th having,

In a further step S1000 compares the rendering engine 4th of the system 2 the greatest distance D2 with the predetermined remote limit value FG .

When the greatest distance D2 greater than the remote limit FG is determined in a further step S1100 the rendering engine 4th of the system 2 the position of the far cutting plane FS1 , FS2 , FS3 while evaluating the remote limit value FG .

If, on the other hand, the greatest distance D2 less than the remote limit FG is determined in a further step S1200 the rendering engine 4th of the system 2 the position of the far cutting plane FS1 , FS2 , FS3 evaluating the greatest distance D2 .

Finally, it fills in another step S1300 the rendering engine 4th of the system 2 the Z-buffer 8th with depth data TD that is based on the application of the partial frustum F1 , F2 , F3 were won.

In a departure from the present exemplary embodiment, the sequence of the steps can also be different. Furthermore, several steps can also be carried out at the same time or simultaneously. Furthermore, different from the present exemplary embodiment, individual steps can also be skipped or omitted.

A computer-implemented simulation of an optical sensor 6th in a virtual environment that provides depth information.

List of reference symbols

2

system

4th

Rendering engine

6th

sensor

6a

Area sensor

6b

Area sensor

6c

Area sensor

8th

z buffer

A.

axis

D1

smallest distance

D2

greatest distance

F.

Frustration

F1

Partial frustum

F2

Partial frustum

F3

Partial frustum

FG

Remote limit value

FS1

distant cutting plane

FS2

distant cutting plane

FS3

distant cutting plane

NG

Near limit value NS1 near cutting plane

NS2

near cutting plane

NS3

near cutting plane

O

object

ST

Rays

TD-

Depth data

W.

value

x

Total distance

x1

Partial spacing

x2

Partial spacing

x3

Partial spacing

S100

step

S200

step

S300

step

S400

step

S500

step

S600

step

S700

step

S800

step

S900

step

S1000

step

S1100

step

S1200

step

S1300

step

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• US 8665260 B2 [0008]
• US 7525542 B2 [0008]
• CN 104933758 A [0008]

Claims (9)

Method for the computer-implemented simulation of an optical sensor (6) in a virtual environment, the optical sensor (6) providing depth information, with the following steps: (S100) Reading in a value (W) indicative of a number of a plurality of partial frustums (F1, F2, F3) of a frustum (F) of the sensor (6), each with a near cutting plane (NS1, NS2, NS3) and one far cutting plane (FS1, FS2, FS3), (S200) Forming the partial frustums (F1, F2, F3) according to the value (W) by arranging the partial frustums (F1, F2, F3) one behind the other along an optical axis (A) of the optical sensor (6), and (S1300) Filling a Z-buffer (8) with depth data (TD) which have been obtained on the basis of the application of at least one partial frustum (F1, F2, F3). Procedure according to Claim 1 , with the further steps: (S300) Detection of whether an object (O) is located within a partial frustum (F1, F2, F3), and for detection of an object (O) within the partial frustum (F1, F2 , F3) carrying out the steps: (S400) emitting a predetermined number of beams (ST) from the sensor (6) to an object (O), (S500) determining the beam of the predetermined number of beams (ST), which the Object (O) hits and has the smallest distance (D1) to the sensor (6), (S800) determining a position of a near cutting plane (NS1, NS2, NS3) while evaluating the beam with the smallest distance (D1) to the sensor (6), (S900) determining a further beam of the predetermined number of beams (ST) which hits the object (O) and is the greatest distance (D2) from the sensor (6), and (S1200) determining a position of a far cutting plane (FS1, FS2, FS3) while evaluating the beam with the greatest distance (D2) to the sensor (6). Procedure according to Claim 2 , with the further steps: (S600) comparing the smallest distance (D1) with a predetermined near limit value (NG), and (S700) determining a position of a near cutting plane (NS1, NS2, NS3) while evaluating the near limit value ( NG), if the smallest distance (D1) is smaller than the near limit value (NG). Procedure according to Claim 2 or 3rd , (S1000) comparing the greatest distance (D2) with a predetermined far limit value (FG), and (S1100) determining a position of a far cutting plane while evaluating the far limit value (FG) if the greatest distance (D2) is greater than is the remote limit value (NG). Computer program product designed to carry out a method according to one of the Claims 1 to 4th . System (2) for the computer-implemented simulation of an optical sensor (6) in a virtual environment, the optical sensor (6) providing depth information, the system (2) being designed to generate a value (W) indicative of a number of a plurality of Partial frustums (F1, F2, F3) of a frustum (F) of the sensor (6) each with a near cutting plane (NS1, NS2, NS3) and a distant cutting plane (FS1, FS2, FS3) to be read in to form the Frustums (F1, F2, F3) according to the value (W) by arranging the partial frustums (F1, F2, F3) one behind the other along an optical axis (A) of the optical sensor (6), and for filling a Z-buffer ( 8) with depth data (TD) obtained on the basis of the application of at least one partial frustum (F1, F2, F3). System (2) according to Claim 6 , the system (2) being designed to detect whether an object (O) is located within a partial frustum (F1, F2, F3), and to detect an object (O) within the partial frustum ( F1, F2, F3) to emit a predetermined number of beams (ST) from the sensor (6) to an object (O), to determine the beam of the predetermined number of beams (ST) that hits the object (O) and the smallest distance (D1) to the sensor (6) to determine a position of a near cutting plane (NS1, NS2, NS3) by evaluating the beam with the smallest distance (D1) to the sensor (6), another beam of the to determine a predetermined number of beams (ST) which hits the object (O) and the greatest distance (D2) to the sensor (6) and a position of a distant cutting plane (FS1, FS2, FS3) while evaluating the beam with the to determine the greatest distance (D2) to the sensor (6). System (2) according to Claim 6 or 7th , the system (2) being designed to compare the smallest distance (D1) with a predetermined near limit value (NG) and assign a position to a near cutting plane (NS1, NS2, NS3) while evaluating the near limit value (NG) determine if the smallest distance (D1) is less than the near limit value (NG). System (2) according to Claim 6 , 7th or 8th , the system (2) being designed to compare the greatest distance (D2) with a predetermined remote limit value (FG) and to determine a position of a remote cutting plane while evaluating the remote Limit value (FG) to be determined if the greatest distance (D2) is greater than the remote limit value (NG).

Images