DE102018121317A1 - Method and device for estimating direction information conveyed by a free space gesture for determining user input at a human-machine interface - Google Patents

Abstract

The invention relates to a method and a device for estimating direction information conveyed by a free space gesture for determining a user input at a human-machine interface, in particular a vehicle. Furthermore, the invention relates to a computer program that is configured to execute the method. The method comprises: receiving sensor-acquired measurement data which represent a free space gesture carried out by a user with respect to the human-machine interface as a spatially multidimensional inhomogeneous point cloud in a correspondingly multidimensional space; Determining, on the basis of the measurement data, density data which represent a density distribution corresponding to the point cloud, in which in each case density information is assigned to the individual points of the point cloud, which identifies a local point density of the point cloud in the vicinity of the respective point; Applying a dimension-reducing approximation method to the density data to determine estimated data that estimate an estimate of a one-dimensional direction curve that characterizes at least one direction aspect of the density distribution; represent; and generating and providing directional data which represent directional information which is determined by the spatial course of the directional curve represented by the estimated data in the superordinate multidimensional space.

Description

The present invention relates to a method and a device for estimating direction information conveyed by a free space gesture for determining a user input at a human-machine interface, in particular a vehicle. Furthermore, the invention relates to a computer program that is configured to execute the method.

While classic human-machine interfaces (abbreviated “MMS” or more frequently “MMI”) are regularly based on a user's contact-based interaction with a corresponding control element, for example a physical switch or a touch-sensitive surface or display device, etc., MMI are now known, in which a static or dynamic gesture of a human user, performed in free space without physical contact with an MMI device, is sensed. This sensory detection of such gestures, also known as “free space gestures”, can be carried out, for example, using a suitable camera. Then, by means of mathematical methods, in particular image recognition methods, conclusions are drawn from the sensor-acquired data on a user input intended by means of the recorded gesture.

In particular in connection with vehicle-related applications, such MII capable of recognizing free space gestures can be used to make inputs for operating the vehicle or a subsystem thereof. Furthermore, it is also possible for the free space gestures to relate to the surroundings of the vehicle, such as pointing to an interesting object that is visible from the vehicle and external to the vehicle, and the recognition of such a free space gesture serves to provide an input for a system which is intended to provide information relating to the object or causes the vehicle to react to it.

For the implementation of such contactless MMI, gesture recognition systems are known in which calibrated 2D sensors are used in order to detect and track points associated with certain features along the contour of a user's hand, in particular from above and from the side. On the basis of the detected points, a pointing direction associated with the gesture carried out in three-dimensional space can then be estimated. Such systems are dependent on the recognition of clearly distinguishable features (points) of the human hand and require fixed and calibrated hardware with at least two 2D sensors.

In addition, methods are known in which body-related information is used to infer a pointing direction expressed by a gesture. In most cases, a head position and a hand position are detected and a display direction is estimated based on this. Alignments or even movements of the fingers of the hand are not detected or taken into account.

The present invention is based on the object of further improving the recognition of free space gestures and in particular the determination of directional information conveying them.

This object is achieved in accordance with the teaching of the independent claims. Various embodiments and developments of the invention are the subject of the dependent claims.

A first aspect of the invention relates to a, in particular computer-implemented, method for estimating direction information conveyed by a free space gesture for determining a user input at a human-machine interface, in particular a vehicle. The method comprises: (i) receiving sensor-acquired measurement data which a free space gesture carried out by a user with respect to the human-machine interface, in particular by means of arm, hand and / or finger, as spatially multidimensional, i.e. Represent 2-dimensional or 3-dimensional, inhomogeneous point cloud in a correspondingly multidimensional space; (ii) determining, on the basis of the measurement data, density data which represent a density distribution corresponding to the point cloud, in which in each case density information is assigned to the individual points of the point cloud, which identifies a local point density of the point cloud in the vicinity of the respective point; (iii) applying a dimension-reducing approximation method to the density data to determine estimated data which is an estimate of a one-dimensional direction curve characterizing at least one direction aspect of the density distribution; represent; and (iv) generation and provision of directional data which represent directional information which is determined by the spatial course of the directional curve represented by the estimated data in the superordinate multidimensional space.

“Direction information” in the sense of the invention is to be understood in particular to mean information that indicates at least one specific, selected direction in space. Optionally, it can optionally also specify its temporal dependency.

A “point cloud” in the sense of the invention is to be understood in particular as a set of points of a vector space, in particular of the three-dimensional space or of a two-dimensional subspace thereof, which has an unorganized spatial structure (“cloud”). A point cloud is described by the points contained therein, each of which can be detected in particular by their spatial coordinates in a coordinate system assigned to the vector space.

A “direction curve” in the sense of the invention is to be understood in particular as a one-dimensional curve or in particular a one-dimensional direction curve in a higher-dimensional space, which has at least one directional aspect characterized by it. In the simplest case, the direction curve can represent a straight line as a one-dimensional direction curve, the direction of which at the same time indicates the direction aspect. In the general case, however, it can also have any curvature, including multiple curvatures. The directional curve is preferably at least for the most part, ideally everywhere, continuous and differentiable.

In the solution according to the invention, the desired directional information is consequently not determined directly from the point distribution of the recorded point cloud, but instead a density distribution representing a probability distribution is first derived from the point distribution, on the basis of which a one-dimensional directional curve is determined only by means of the dimension-reducing approximation method mentioned, which determines the characterizes the desired directional information. In this way, the achievable reliability and accuracy in the estimation-based determination of directional information from point clouds can be increased. In addition, the method requires only relatively small computing resources, so that it is particularly suitable for real-time capable applications. In addition, the method is independent of a specific orientation of the sensor used to record the measurement data, for example a camera, which makes it particularly well suited for determining, in particular also finger-based, gestures of pointer clearance within a vehicle, since the sensor can be provided in very different positions there can.

Preferred embodiments of the method are described below, each of which, as far as this is not expressly excluded or technically impossible, can be combined with one another as well as with the other described other aspects of the invention or used as corresponding embodiments of the latter.

In some embodiments, the sensory multidimensional detection of the free space gesture carried out by a user with respect to the human-machine interface takes place by means of a TOF camera system. A “TOF camera system” in the sense of the invention is to be understood as a 3D camera system that measures distances on the basis of a time of flight (“TOF”) method. The measuring principle is based on the fact that the scene to be recorded is illuminated by means of a light pulse, and the camera measures for each pixel the time it takes for the light to reach the object and back again. Due to the constant speed of light, this time is directly proportional to the distance. The camera thus supplies the distance of the object depicted on it for each pixel. The use of a TOF camera system represents a particularly effective and high-resolution implementation option for the at least one sensor required for sensory detection of the free space gesture.

In some embodiments, the multi-dimensional space is three-dimensional, and the measurement data represent a point cloud that depicts the detected free space gesture in three-dimensional, 3D.

In a first variant, a density distribution corresponding to this is determined directly for the 3D point cloud by assigning density information to the individual points of the 3D point cloud, which identifies a local density of the 3D point cloud in the vicinity of the respective point, whereby for the 3D point clouds representing the resulting 3D density distribution and associated density data are generated. A dimension-reducing image of the 3D point cloud on one or in particular several point clouds of a lower dimension is not necessary in this variant, so that corresponding method steps can be saved.

In a second, alternative variant, the determination of the density data comprises: (i) dimension-reducing mapping of the 3D point cloud onto a plurality N, with N> 1, each spatially two-dimensional, 2D, point clouds, each of which is defined in one of N associated 2D subspaces of the multidimensional space, at least two of which are not parallel to one another in the superordinate 3D space; and (ii) determining, for each of the N 2D point clouds, a density distribution corresponding thereto, in which the individual points of the respective 2D point cloud are each assigned density information which is a local density of the respective 2D point cloud in the vicinity of the respective point indicates, for each of the 2D point clouds the associated resulting 2D density distribution representing assigned density data are generated. Furthermore, the approximation method is used separately for each of the 2D density distributions represented by the respective density data for estimating a respective one-dimensional, 1D, direction curve characterized by them, whereby for each of the 2D density distributions the associated 1 D direction curve representing assigned estimation data is generated . In addition, the directional information is determined on the basis of a, in particular weighted or unweighted, averaging of at least two, preferably all, of the individual 1D directional curves represented by the respective estimation data in the superordinate 3D space in order to obtain directional information related to the 3D space .

The latter can be done in particular by means of gradient determination with regard to a 1D direction curve resulting from the averaging. These, as well as any further embodiments based on them, have the advantage that the complexity of the subsequent calculations can be significantly reduced due to the dimensional reduction carried out, since overall less and only two-dimensional data have to be processed. In particular, the performance can thus be increased and thus the real-time capability of the method can be achieved or improved.

In some of these embodiments according to the second variant, the dimension-reducing mapping has a projection mapping along one of the three dimensions of the 3D point cloud represented by the measurement data onto a corresponding 2D point cloud in a corresponding one of the N 2D subspaces. The dimension reduction to two dimensions is thus carried out by means of projection. For example, when using a Cartesian coordinate system, the projection can take place in particular along one of the axes of this coordinate system, so that, for example, all points of the point cloud in the through the coordinate axes X . Y and Z spanned three-dimensional space along the Z axis on the X / Y plane. Of course, this is only an example and should not be understood as a limitation. In particular, the projection can take place along any projection direction and instead of a Cartesian coordinate system, another coordinate system of the same dimension can also be used to represent the coordinates of the points.

In some embodiments, determining the density data further comprises: (i) spatial quantizing the multidimensional point cloud represented by the measurement data
or, if applicable, each of the 2D point clouds obtained therefrom by means of a respective dimension-reducing image, and (ii) determining the density data on the basis of a resulting density distribution of the points of the corresponding quantized point cloud. Instead of a continuous or fixed image depending on the discrete resolution of the corresponding sensor system generated the measurement data, the point cloud is now mapped onto a predefined spatial grid and thus quantized. The individual cells of the grid can be referred to as a “voxel” in the three-dimensional case and as a “pixel” in the two-dimensional case, particularly when using a rectangular or even cube grid. The quantization in turn simplifies the complexity of the subsequent calculations and in particular enables simple indexing for the voxels or the pixels instead of coordinate information. In particular, the complexity reduction can include that instead of floating point arithmetic, the simpler and less complex integer arithmetic can now be used. This simplifies and speeds up use, particularly in embedded systems. In addition, the type of quantization allows the accuracy of the system to be adjusted, in particular even increased, since the type of quantization enables an adjustable trade-off between accuracy and speed (coarse grid = fast but inaccurate, fine grid = slower but more precise).

In some developments of these embodiments, the quantization can additionally relate to a value assigned to the respective voxel or pixel, which is determined on the basis of the points of the point cloud present therein. The quantization can in particular be carried out in binary form, so that in the course of quantization the value of a voxel or pixel, if its number of points lies below a predetermined threshold, is set to a first of the two binary values, for example to "0", while for points above the threshold this value is set to the second of the two binary values, for example "1". If the number of points is equal to the threshold, the value can be set so that it is set to one of the binary values in a predefined manner. This way the number of Points to be processed further can be reduced further by using only those voxels or pixels with the second value in the further process, which further increases the performance of the process.

In some of these embodiments, in the case of spatial quantization of a 2D point cloud obtained by means of a respective dimension-reducing image, the density distribution for each point of the resulting 2D point cloud is determined on the basis of the number of those points of the multidimensional point cloud represented by the measurement data, which are according to the projection image are mapped onto the same quantized point of the 2D point cloud. If, in the case of dimension-reducing imaging, many points are mapped onto the same pixel of the resulting 2D point cloud by means of the projection, this pixel receives a high value corresponding to the number of these points or derived therefrom, while in the opposite case if only a few points are on the pixel are shown, this value is correspondingly lower. In this way, the original point cloud can be converted into a two-dimensional density distribution, which characterizes the resulting respective 2D point cloud, in an effective and easy to carry out and thus high-performance manner.

In some embodiments, the determination of the density data further comprises filtering the respective point cloud, on the basis of which the respective density distribution represented by the respective density data by means of, preferably multiple, folding of the density distribution with a respective filter core, which in particular can have a low-pass property and in particular Smoothing core can be determined. In one variant, the filter core can in particular be a Gaussian filter core. With the aid of this filtering, the original density distribution, as it results in particular from the quantization, is smoothed, which in particular can improve the applicability of the subsequent method steps and the quality of the resulting estimation results for the directional information. Without the filtering, the information between the points would not be interpolated, with the result that significantly less information would be available for the subsequent method steps, in particular for a PCA (described below), which in some cases could lead to less reliable results. Another advantage of filtering is that individual "wrong" points (e.g. caused by measurement noise or incorrect measurements) are smoothed and thus have less influence on the further process, especially e.g. has a PCA. The noise usually associated with TOF sensors in particular is thus suppressed by means of the filtering and the quality of the result can thus be increased.

In some embodiments, a main component analysis, PCA, or a regression method is used as the dimension-reducing approximation method for determining the respective estimation data.

The PCA is a particularly suitable and performant approximation method within the scope of the method according to the invention. The PCA can in particular be weighted so that smaller main components resulting therefrom are neglected for the resulting determination of the directional data. In particular, in the simplest case, the PCA can be terminated immediately after the determination of the first main component, so that a determination of the further main components is no longer necessary. The directional information represented by the directional data is then determined on the basis of only the largest main component ascertained by means of the PCA (i.e. those components with the greatest intrinsic value determined as part of the PCA). This enables a particularly performant and simple implementation.

mit $${\sigma }_{cc}={{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i,j\right)}\left(i-{\mu }_c\right)}}^2$$

$${\sigma }_{cr}={\sigma }_{rc}={\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i,j\right)}\left(i-{\mu }_c\right)}\left(j-{\mu }_r\right)$$

$${\sigma }_{rr}={{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i,j\right)}\left(i-{\mu }_r\right)}}^2$$

und $${\mu }_c=\frac{1}{{\displaystyle {\sum }_i{\displaystyle {\sum }_{j}F\left(i,j\right)}}}{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i,j\right)}}\cdot i$$

$${\mu }_r=\frac{1}{{\displaystyle {\sum }_i{\displaystyle {\sum }_{j}F\left(i,j\right)}}}{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i,j\right)}}\cdot i$$

wobei $$F\left(i,j\right)={\displaystyle \sum _n{\displaystyle \sum _{m}P\left(i+n,j+m\right)}\cdot K\left(n,m\right)},$$

und wobei (i,j) jeweils Indexpaare zur Indizierung der Zellen des quantisierten 2D-Unterraums bzw. ihrer zugeordneten ungefilterten Dichtwerte P(i,j) bzw. mittels Faltung mit dem Kern Kgefilterten Dichtwerte F(i,j) sind, und die Indizierung des Kerns K mit den Indexpaaren (n,m) erfolgt.In some embodiments, in which a PCA with respect to a quantized and filtered, two-dimensional density distribution F (i, j) the covariance matrix, M, the PCA receives the following initial values: $$M=\left[\begin{array}{cc}{\sigma }_{cc}& {\sigma }_{cr}\\ {\sigma }_{rc}& {\sigma }_{rr}\end{array}\right]$$

With $${\sigma }_{cc}={{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i.j\right)}\left(i-{\mu }_c\right)}}^2$$

$${\sigma }_{cr}={\sigma }_{rc}={\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i.j\right)}\left(i-{\mu }_c\right)}\left(j-{\mu }_r\right)$$

$${\sigma }_{rr}={{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i.j\right)}\left(i-{\mu }_r\right)}}^2$$

and $${\mu }_c=\frac{1}{{\displaystyle {\sum }_i{\displaystyle {\sum }_{j}F\left(i.j\right)}}}{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i.j\right)}}\cdot i$$

$${\mu }_r=\frac{1}{{\displaystyle {\sum }_i{\displaystyle {\sum }_{j}F\left(i.j\right)}}}{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i.j\right)}}\cdot i$$

in which $$F\left(i.j\right)={\displaystyle \sum _n{\displaystyle \sum _{m}P\left(i+n.j+m\right)}\cdot K\left(n.m\right)}.$$

and where (i, j) are index pairs for indexing the cells of the quantized 2D subspace or their associated unfiltered density values P (i, j) or density values filtered by means of convolution with the core K, and the indexing of the core K with the index pairs (n, m).

In some embodiments, the method further comprises: (i) segmenting the respective point cloud provided for determining the associated density data therefrom into a plurality of different segments on the basis of a recognition of different body parts of the user represented by the point cloud; (ii) selecting a real subset of the segments by means of a selection criterion which is defined as a function of the one or more body parts intended for the execution of the free space gesture; and (iii) determining the associated density data only based on the portion of the point cloud represented by the selected subset of the segments. In this way it is possible to determine the desired directional information solely on the basis of a section of the sensor-acquired point cloud. This is particularly advantageous if the point cloud also represents one or more other body parts or sections thereof beyond a body part of the user that is relevant for determining the directional information. The segmentation also allows, on the one hand, to avoid unnecessary processing of the portions of the point cloud that are not relevant in this sense, and on the other hand increases the required reliability and accuracy of the results, since these cannot be falsified by irrelevant portions of the point cloud.

Thus, in some embodiments, the point cloud represented in the measurement data can correspond in particular to a free space gesture carried out with at least part of the upper limbs of a user. The segmentation contains at least one selection segment that corresponds only to one or more fingers of the user, and the selection criterion is defined in such a way that the subset of the segments determined thereby contains at least, preferably exclusively, this selection segment. In spite of the sensory detection of a larger part of the upper limbs, the segmentation can be focused solely on aspects of the open space gesture carried out by one or more fingers of the user. In particular, a pointing direction expressed with the finger or fingers can be recognized reliably, with high performance and / or with high accuracy.

In some embodiments, the measurement data represent a temporal development of the free space gesture as a spatially multidimensional and at the same time correspondingly time-dependent point cloud. In addition, the procedural determination of a directional information conveyed by the free space gesture is carried out several times, in each case for a different point in time with respect to the time-dependent point cloud. The directional information is generated and provided in such a way that it represents the temporal development of the directional information resulting from this multiple determination. In this way, a temporal development of the detected free space gesture can also be tracked, for example a temporal change in direction information, such as a pointing direction, expressed by the free space gesture.

In some embodiments (i), the measurement data in M + 1 dimensions represent a temporal development of the free space gesture as a spatially M-dimensional and at the same time correspondingly time-dependent point cloud, with M> 1, (ii) the determination of the density data is a dimension-reducing mapping of the data from the measurement data in M spatial and an additional temporal dimension represented point cloud in advance with elimination of the temporal dimension to an at least two-dimensional space in order to obtain a representation of a 2D or 3D space area corresponding to the free space gesture; and (iii) the directional data are generated and provided such that the directional information represented by them, which is determined by the spatial course of the directional curve represented by the estimated data in the superordinate multidimensional space, is defined by the directional curve and at least partially included by it Indicates 2D or 3D space. The dimension-reducing mapping of the point cloud represented by the measurement data in M spatial and an additional temporal dimension can be carried out in particular by means of projection along the temporal dimension and, in the case of M = 3, optionally along one of the spatial dimensions. In this way, the method can be expanded or configured in such a way that it can be used to estimate direction information conveyed by a free space gesture for determining user input at a human-machine interface, the direction information being replaced by or in addition to a pointing direction Free space gesture identifies a defined spatial area, and thus in particular describes a region, ie a 2D or 3D spatial area.

This is to be explained using a short example: Suppose the user performs a three-dimensional pointing gesture (M = 3), in which he moves the hand or arm in a circular motion, so that the pointing direction has a time course in which it describes a cone . The cone tip is (i) the finger root (in the case of a pure finger movement) or (ii) the elbow in the case of a forearm movement emanating from it) or (iii) the shoulder joint (in the case of a movement of the entire arm emanating from it), and the region desired by the user is located inside the cone. The point cloud can then first be described in M + 1 = 4 dimensions with three space and one time dimension and a dimension-reducing projection along the time dimension and optionally one of the space dimensions onto a three- or two-dimensional space. The resulting point cloud in this reduced space could then (e.g. for the above gesture) have a shape or behavior that at least partially includes a spatial area, e.g. in which it has an at least largely closed ring shape. Proceeding from this, corresponding density data can then be generated in accordance with the method and, in turn, directional information can be generated from this, which describes the region outlined by the free space gesture. The applicability of the method to the estimation of pure pointing gestures to the estimation of spatial gestures can thus be expanded.

In some embodiments, determining a direction information conveyed by the free space gesture on the basis of the estimated data comprises determining at least one direction vector lying tangential to the direction curve represented by the estimated data. This can be done in particular by means of a derivation in the sense of a mathematical differentiation or gradient formation with respect to the direction curve. In this way, directional information characterizing it can be obtained in a simple manner from the directional curve, which at the same time corresponds to an estimate of the directional information conveyed by the free space gesture.

A second aspect of the invention relates to a device for estimating direction information conveyed by a free-space gesture for determining a user input at a human-machine interface, in particular a vehicle, the device being set up to carry out the method according to the first aspect of the invention. The device can be, in particular, a control device for a vehicle, in particular a motor vehicle, the control device itself having the man-machine interface mentioned or being set up to be signal-connected to it for receiving the measurement data.

A third aspect of the invention relates to a computer program configured to carry out the method of the first aspect of the invention. The computer program can in particular be stored on a non-volatile data carrier. This is preferably a data carrier in the form of an optical data carrier or a flash memory module. This can be advantageous if the computer program as such is to be traded independently of a processor platform on which the one or more programs are to be executed. In another implementation, the computer program can be present as a file on a data processing unit, in particular on a server, and via a data connection, for example the Internet or a dedicated data connection, such as a proprietary or local network. be downloadable. In addition, the computer program can have a plurality of interacting individual program modules.

The device according to the second aspect of the invention can accordingly have a program memory in which the computer program is stored. Alternatively, the device can also be set up to access a computer program which is available externally, for example on one or more servers or other data processing units, in particular in order to exchange data with it, which are used during the course of the method or computer program or outputs of the computer program represent.

The features and advantages explained above in relation to the first aspect of the invention also apply accordingly to the further aspects of the invention.

Further advantages, features and possible uses of the present invention result from the following detailed description in connection with the figures.

• 1 schematisch ein System zur Schätzung einer Richtungsinformation einschließlich einer Schätzvorrichtung, gemäß verschiedener Ausführungsformen der Erfindung;
• 2 ein Flussdiagramm zur Veranschaulichung einer bevorzugten Ausführungsform des erfindungsgemäßen Verfahrens;
• 3 zum Zwecke der Illustration des Segmentierungsprozesses S1 aus 2 eine zweidimensionale Darstellung einer durch entsprechende, durch Aufnahme eines menschlichen Arms gewonnene Messdaten repräsentierten beispielhaften Punktwolke, wobei ein zu den Fingern des Arms korrespondierendes Segment gekennzeichnet ist;
• 4 graphische Darstellungen zur Veranschaulichung von während des Verfahrens nach 2 auftretenden und aus den Messdaten abgeleiteten Daten; und
• 5 eine beispielhafte Illustration einer Interpretation einer erfassten Freiraumgeste als Raumgeste zur Schätzung von Richtungsinformation, die einen durch die Freiraumgeste ausgewählten 2D-Flächenbereich definiert, gemäß Ausführungsformen der Erfindung.

It shows

• 1 schematically shows a system for estimating a direction information including an estimation device, according to various embodiments of the invention;
• 2 a flowchart to illustrate a preferred embodiment of the method according to the invention;
• 3 for the purpose of illustrating the segmentation process S1 out 2 a two-dimensional representation of an exemplary point cloud represented by corresponding measurement data obtained by recording a human arm, wherein a segment corresponding to the fingers of the arm is identified;
• 4 graphic representations to illustrate during the process 2 occurring data and derived from the measurement data; and
• 5 an exemplary illustration of an interpretation of a detected free space gesture as a spatial gesture for estimating directional information, which defines a 2D surface area selected by the free space gesture, according to embodiments of the invention.

The same reference numbers are used throughout the figures for the same or corresponding elements of the invention.

1 schematically shows a system 1 for estimating direction information according to various embodiments of the invention. The system 1 has an estimation device 2 as well as a sensor 3 for three-dimensional detection of a free space gesture carried out by a user of the system. The estimator 2 can in particular be a computer, in particular with a processor platform 2a and a program and data memory 2 B and a data output 2c for outputting output data determined by the computer, in particular in the form of directional information. the sensor 3 can in particular have a TOF (time of flight) camera system. Other types of sensors, in particular sensors based on ultrasound or radar measurement, can also be used instead or in combination therewith. In the program memory 2 B a computer program can be stored, which can consist of one or more program modules and which is configured when it runs on the processor platform 2a the estimator 2 to cause the inventive method, for example as follows based on the 2 to 5 described to perform.

This in 2 The flowchart shown serves to illustrate a preferred embodiment of the method according to the invention for estimating direction information conveyed by a free space gesture for determining a user input at a human-machine interface, which in particular the system 1 correspond or can have one. The actual estimation process, which exemplifies the processes here S1 to S8 involves a measuring process S0 ahead, with the sensor, especially with a measuring sensor 3 according to 1 , Measurement data MD are generated, which represent a spatially three-dimensional point cloud, which in turn represents a free space gesture recorded by the user and carried out during the measurement. 3 shows a two-dimensional representation of an exemplary 3D point cloud obtained in this way, which represents parts of the upper arm, the forearm and the hand of an arm of a user performing the free space gesture.

In the embodiment of the estimation method described here, the measurement data MD represented three-dimensional point cloud as part of preprocessing, initially as part of a segmentation process S1 segmented to segment data using one or more known methods, for example using a “random forest” classification method SD to obtain the section of the measurement data relevant to the further course of the process MD correspond. In 3 is such a cutout, which corresponds to the fingers of the hand, marked with a black box.

Then the segment data SD represented point cloud in a projection process S2 in N = 2 two-dimensional point clouds are shown. This is advantageously done by means of projection along one of the three spatial dimensions, here by way of example - without this being understood as a restriction - when using a Cartesian coordinate system, on the one hand along the Z direction, so that a two-dimensional projection of the point cloud onto the XY plane results, and on the other hand along the Y direction, so that there is accordingly a second projection on the XZ plane. The 4 (a) respectively. 4 (e) graphically show these two-dimensional, from the segment data SD resulting projections, which are each represented by corresponding projection data BD _xy or BD _xz .

A quantization process follows S3 , in which each of the 2D point clouds represented by the projection data BD _xy or BD _{xz is} quantized according to a predefined spatial grid, so that now only predefined spatial positions are defined for the points (pixels). The two quantized 2D point clouds, which are exemplary in the 4 (b) respectively. 4 (f) are represented graphically, are represented by corresponding image data ID _xy or ID _xz . The image data can thus be represented by corresponding data fields as follows: $$I_{x.y}\left(i.j\right)und{I}_{x2}\left(i.j\right)mitder2D-Pixelindi\mathrm{e.g.}ierunGi=\mathrm{1,}....kundj=\mathrm{1,}....l$$

The quantization process S3 concludes a preprocessing section of the method that precedes the actual estimation of the direction information.

This estimation is carried out in an estimation section of the method, in which, first of all, ID _xy or ID _xz from the quantized image data in a probability estimation process S4 Corresponding density data PD _xy or PD _{xz can be} obtained. For this purpose, a density value is assigned to each pixel of the two quantized 2D point clouds, which is equal to the number of points, which according to the image data by means of the preceding processes S2 and S3 were mapped onto the corresponding pixel. The density data can thus be represented as follows by corresponding data fields: $$P_{x.y}\left(i.j\right)und{P}_{x\mathrm{e.g.}}\left(i.j\right)miti=\mathrm{1,}....kundj=\mathrm{1,}....l$$

The 4 (c) respectively. 4 (g) represent graphically such quantized 2D data fields in which the value of a pixel is represented by a corresponding gray value for the purpose of illustration, a darker color corresponding to a higher probability. Instead of the number of points, it is also conceivable to choose a dependent or derived other value which is suitable for at least indirectly characterizing this number of points.

sein kann, wie folgt gefiltert werden: $$F\left(i,j\right)={\displaystyle \sum _n{\displaystyle \sum _{m}P\left(i+n,j,j+m\right)}}\cdot K\left(n,m\right).$$

A filter process now follows S5 , in which the density data PD _xy or PD _{xz is} folded by means of a core K, which is in particular a Gaussian 2D core, for example of the form: $$K\left(n.m.\sigma \right)=a<figure-callout\; id="2"\; label="\cdot \; e\; n"\; filenames="DE102018121317A1\_0026.png,DE102018121317A1\_0027.png"\; state="\{\{state\}\}">\cdot \; </figure-callout>e^{\frac{n^{}}{}}{\frac{{}^2+{\mathrm{<figure-callout\; id="2"\; label="m"\; filenames="DE102018121317A1\_0026.png,DE102018121317A1\_0027.png"\; state="\{\{state\}\}">m</figure-callout>}}^2}{2{\sigma }^Z}}^{}$$

can be filtered as follows: $$F\left(i.j\right)={\displaystyle \sum _n{\displaystyle \sum _{m}P\left(i+n.j.j+m\right)}}\cdot K\left(n.m\right),$$

The core is indexed K with the index pairs (n, m). The core K has the variance σ that can be selected as a parameter and a scaling factor a, which is selected such that it normalizes the sum of all core elements to 1. The filtering, which in particular preferably represents smoothing, results corresponding filter data FD _xy or FD _xz, for example in the 4 (d) respectively. 4 (h) are represented graphically as corresponding filtered pixel clouds F _xy (i, j) or F _xz (i, j).

In a subsequent approximation process S6 , which is carried out with the aid of a dimension-reducing approximation method, which in the present example is chosen as the main component analysis (PCA), then corresponding filter data are obtained from the filter data ED won, each representing a direction curve, which in the 4 (d) respectively. 4 (h) is shown as a corresponding white arrow. In the simplest case, only the largest main component determined by means of the PCA is taken into account, which defines the direction and position of the corresponding white arrow.

mit $${\sigma }_{cc}={{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i,j\right)}\left(i-{\mu }_c\right)}}^2$$

$${\sigma }_{cr}={\sigma }_{rc}={\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i,j\right)}\left(i-{\mu }_c\right)}\left(j-{\mu }_r\right)$$

$${\sigma }_{rr}={{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i,j\right)}\left(i-{\mu }_r\right)}}^2$$

und $${\mu }_c=\frac{1}{{\displaystyle {\sum }_i{\displaystyle {\sum }_{j}F\left(i,j\right)}}}{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i,j\right)}}\cdot i$$

$${\mu }_r=\frac{1}{{\displaystyle {\sum }_i{\displaystyle {\sum }_{j}F\left(i,j\right)}}}{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i,j\right)}}\cdot i$$

wobei $$F\left(i,j\right)={\displaystyle \sum _n{\displaystyle \sum _{m}P\left(i+n,j+m\right)}\cdot K\left(n,m\right)},$$

und wobei (i,j) jeweils Indexpaare zur Indizierung der Zellen des quantisierten 2D-Unterraums bzw. ihrer zugeordneten ungefilterten Dichtwerte P(i,j) bzw. mittels Faltung mit dem Kern K gefilterten Dichtwerte F(i,j) sind, und die Indizierung des Kerns K mit den Indexpaaren (n,m) erfolgt.If a PCA is used with regard to the quantized and filtered, two-dimensional density distribution F (i, j), the covariance matrix, M , the PCA in particular received the following initial values: $$M=\left[\begin{array}{cc}{\sigma }_{cc}& {\sigma }_{cr}\\ {\sigma }_{rc}& {\sigma }_{rr}\end{array}\right]$$

With $${\sigma }_{cc}={{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i.j\right)}\left(i-{\mu }_c\right)}}^2$$

$${\sigma }_{cr}={\sigma }_{rc}={\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i.j\right)}\left(i-{\mu }_c\right)}\left(j-{\mu }_r\right)$$

$${\sigma }_{rr}={{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i.j\right)}\left(i-{\mu }_r\right)}}^2$$

and $${\mu }_c=\frac{1}{{\displaystyle {\sum }_i{\displaystyle {\sum }_{j}F\left(i.j\right)}}}{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i.j\right)}}\cdot i$$

$${\mu }_r=\frac{1}{{\displaystyle {\sum }_i{\displaystyle {\sum }_{j}F\left(i.j\right)}}}{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i.j\right)}}\cdot i$$

in which $$F\left(i.j\right)={\displaystyle \sum _n{\displaystyle \sum _{m}P\left(i+n.j+m\right)}\cdot K\left(n.m\right)}.$$

and where (i, j) each index pairs for indexing the cells of the quantized 2D subspace or their associated unfiltered density values P (i, j) or by means of folding with the core K filtered density values F (i, j), and the indexing of the core K with the index pairs (n, m) takes place.

From the estimated data ED can eventually be in a directional process S8 the desired direction information is derived and by means of corresponding direction data RD be represented. In particular, this can be done in such a way that the estimated data ED represented one-dimensional direction curves (white arrows), which are defined in the respective 2D space, are transferred to a common three-dimensional space by expanding the two 2D spaces by the missing third dimension, and the estimated data ED then averaged in the common 3D space (this corresponds to a message from the two white arrows) in order to obtain a resulting direction curve. In the more general case, the estimated data represent ED In each case a curved direction curve, so that after averaging the two direction curves, a generally likewise resulting curve is created in 3D space, from which the desired direction information can be obtained, for example Gradient formation can be determined at a desired point in the resulting direction curve or by linear regression.

The above with reference to the 2 to 4 The method described by way of example can, in particular, also be carried out for different points in time of the execution of the free space gesture, so that time-dependent directional information is obtained and the time dynamics of a free space gesture can also be recorded.

5 finally shows an exemplary illustration of an interpretation of a detected free space gesture as a spatial gesture for estimating directional information, which defines a 2D surface area selected by the free space gesture, according to embodiments of the invention.

Suppose the user executes a three-dimensional pointing gesture (M = 3), in which he moves the hand or arm in a circular motion, so that the pointing direction has a temporal course (here marked by the five times t ₀ to t ₄ ), in which she a cone C describes. The cone tip T is (i) the finger root (in the case of a pure finger movement) or (ii) the elbow in the case of a forearm movement emanating from it) or (iii) the shoulder joint (in the case of a movement of the entire arm emanating from it), and that of the The desired region is located inside the cone, optionally also in its projection along the cone axis towards even greater distances from T , The point cloud can then first be described in M + 1 = 4 dimensions with three space and one time dimension and a dimension-reducing projection along the time dimension and optionally one of the space dimensions onto a three- or two-dimensional space. The resulting point cloud in this reduced space could then (for example for the above gesture) have a shape or a behavior that has a spatial area, for example the two-dimensional area or region shown with hatching A , at least partially includes, for example, in which, as in 5 shown has an at least largely closed ring shape. Proceeding from this, corresponding density data can then be obtained according to the method FD and from this in turn a direction information or corresponding direction data RD generate the region outlined by the open space gesture A describes or represent. In particular, this can also be used to create an object X that is in the region A is to be selected using the free space gesture. The object can correspond in particular to an operating element of a human-machine interface or another element of vehicle equipment. Objects outside the vehicle are also conceivable. The applicability of the method to the estimation of pure pointing gestures to the estimation of spatial gestures can thus be expanded.

While at least one exemplary embodiment has been described above, it should be noted that a large number of variations exist. It should also be noted that the exemplary embodiments described are only non-limiting examples and are not intended to limit the scope, applicability, or configuration of the devices and methods described herein. Rather, the foregoing description will provide those skilled in the art with a guide to implementing at least one example embodiment, it being understood that various changes in the operation and arrangement of the elements described in an example embodiment may be made without departing from the scope of the appended claims deviated subject and its legal equivalents.

Reference list

1

System for estimating directional information

2

Estimator

2a

Processor platform

2 B

Program and data storage

2c

Data output

3

TOF camera system

MD

Measurement data

SD

Segment or segment data representing this

BD

Projection data

ID

Image data

PD

Density data

FD

Filter data

ED

Estimated data

RD

Directional data

A

2D area selected using free space gesture

C

cone

T

Cone tip

X

object enclosed by the selected 2D surface

t ₀ -t ₄

Sequence of different times corresponding to the execution of the free space gesture

Claims (17)

Method for estimating direction information conveyed by a free space gesture for determining a user input at a human-machine interface (1), in particular a vehicle, the method comprising: Receiving sensor-acquired measurement data (MD) which represent a free space gesture carried out by a user with respect to the human-machine interface as a spatially multidimensional inhomogeneous point cloud in a correspondingly multidimensional space; Determining, on the basis of the measurement data, density data (PD) which represent a density distribution corresponding to the point cloud, in which in each case density information is assigned to the individual points of the point cloud, which identifies a local point density of the point cloud in the vicinity of the respective point; Applying a dimension-reducing approximation method to the density data to determine estimated data (ED), which estimate an estimate of a one-dimensional direction curve characterizing at least one direction aspect of the density distribution; represent; and Generation and provision of directional data (RD) which represent directional information which is determined by the spatial course of the directional curve represented by the estimated data in the superordinate multidimensional space. Procedure according to Claim 1 , wherein: the multidimensional space is three-dimensional, and the measurement data represent a spatially three-dimensional, 3D, point cloud representing the detected free space gesture; determining the density data comprises: - dimension-reducing mapping of the 3D point cloud to a plurality N, with N> 1, each spatially two-dimensional, 2D, point clouds (BD), each of which is defined in one of N associated 2D subspaces of the multidimensional space, at least two of which are not parallel to one another in the superordinate 3D space; and determining for each of the N 2D point clouds a density distribution (PD) corresponding thereto, in which the individual points of the respective 2D point cloud are each assigned density information which characterizes a local density of the respective 2D point cloud in the vicinity of the respective point , wherein for each of the 2D point clouds the respective resulting 2D density distribution representing assigned density data are generated; the approximation method is used separately for each of the 2D density distributions represented by the respective density data to estimate a respective one-dimensional, 1D, directional curve characterized by them, wherein for each of the 2D density distributions the associated estimated data representing the respective 1D directional curve are generated; and the determination of the direction information (RD) is carried out on the basis of averaging at least two of the individual 1D direction curves represented by the respective estimation data (ED) in the superordinate 3D space in order to obtain direction information related to the 3D space. Procedure according to Claim 2 , wherein the dimension-reducing mapping has a projection mapping along one of the three dimensions of the 3D point cloud represented by the measurement data onto a corresponding 2D point cloud (BD) in an associated one of the N 2D subspaces. Method according to one of the preceding claims, wherein the determination of the density data further comprises: Spatial quantization of the multidimensional point cloud represented by the measurement data or, if applicable, each of the 2D point clouds obtained therefrom by means of a respective dimension-reducing image, and determination of the density data (PD) on the basis of a resulting density distribution of the points of the corresponding quantized point cloud (ID). Procedure according to Claim 4 , wherein in the case of the spatial quantization of a 2D point cloud obtained by means of a respective dimension-reducing image, the density distribution for each point of the resulting 2D point cloud is determined on the basis of the number of those points of the multidimensional point cloud represented by the measurement data, which according to the projection image on the same quantized point of the 2D point cloud. Method according to one of the preceding claims, wherein the determination of the density data (PD) further comprises: Filtering the respective point cloud, from which the respective density distribution represented by the respective density data is determined, by folding the density distribution with a respective filter core that has a low-pass property. The method of 6, wherein the filter core is a Gaussian filter core. Method according to one of the preceding claims, wherein a main component analysis, PCA, or a regression method is used as the dimension-reducing approximation method for determining the respective estimation data. Procedure according to Claim 8 , wherein if a PCA is used, the direction information represented by the direction data (RD) is determined on the basis of only the largest main component determined by means of the PCA. Procedure according to Claim 8 or 9 , where a PCA is used with respect to a quantized and filtered, two-dimensional density distribution F (i, j) the covariance matrix, M, the PCA receives the following initial values: $$M=\left[\begin{array}{cc}{\sigma }_{cc}& {\sigma }_{cr}\\ {\sigma }_{rc}& {\sigma }_{rr}\end{array}\right]$$

With $${\sigma }_{cc}={{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i.j\right)}\left(i-{\mu }_c\right)}}^2$$

$${\sigma }_{cr}={\sigma }_{rc}={\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i.j\right)}\left(i-{\mu }_c\right)}\left(j-{\mu }_r\right)$$

$${\sigma }_{rr}={{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i.j\right)}\left(i-{\mu }_r\right)}}^2$$

and $${\mu }_c=\frac{1}{{\displaystyle {\sum }_i{\displaystyle {\sum }_{j}F\left(i.j\right)}}}{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i.j\right)}}\cdot i$$

$${\mu }_r=\frac{1}{{\displaystyle {\sum }_i{\displaystyle {\sum }_{j}F\left(i.j\right)}}}{\displaystyle \sum _i{\displaystyle \sum _{j}F\left(i.j\right)}}\cdot i$$

in which $$F\left(i.j\right)={\displaystyle \sum _n{\displaystyle \sum _{m}P\left(i+n.j+m\right)}\cdot K\left(n.m\right)}.$$

and where (i, j) are index pairs for indexing the cells of the quantized 2D subspace or their associated unfiltered density values P (i, j) or density values filtered by means of convolution with the core K, and the The core K is indexed with the index pairs (n, m). Method according to one of the preceding claims, further comprising: Segmenting the respective point cloud provided for determining the associated density data therefrom into several different segments on the basis of a recognition of different body parts of the user depicted by the point cloud; Selecting a real subset of the segments by means of a selection criterion which is defined as a function of the one or more body parts which are intended for carrying out the free space gesture; and Determination of the associated density data only on the basis of the part (SD) of the point cloud represented by the selected subset of the segments. Procedure according to Claim 11 , wherein the point cloud represented in the measurement data (MD) corresponds to a free space gesture carried out with at least part of the upper limbs of a user; the segmentation contains at least one selection segment (SD) that only corresponds to one or more fingers of the user, and the selection criterion is defined in such a way that the subset of the segments determined thereby contains at least this selection segment. Method according to one of the preceding claims, wherein the measurement data represent a temporal development of the free space gesture as a spatially multidimensional and at the same time time-dependent point cloud; and the procedural determination of a directional information conveyed by the free space gesture takes place several times, in each case for a different point in time with respect to the time-dependent point cloud; and the direction data (RD) are generated and provided in such a way that they represent the temporal development of the direction information resulting from this multiple determination. Method according to one of the preceding claims, wherein: the measurement data in M + 1 dimensions represent a temporal development (t ₀ -t ₄ ) of the free space gesture as a spatially M-dimensional and at the same time time-dependent point cloud with M>1; The determination of the density data is preceded by a dimension-reducing mapping of the point cloud represented by the measurement data in M spatial and an additional temporal dimension and the elimination of the temporal dimension to an at least two-dimensional space in order to represent a 2D or 3D space area corresponding to the free space gesture ( A) get; and the directional data (RD) are generated and provided such that the directional information represented by them, which is determined by the spatial course of the directional curve represented by the estimated data in the superordinate multidimensional space, is defined by the directional curve and at least partially included by it Characterizes the area. Method according to one of the preceding claims, wherein determining a direction information conveyed by the free space gesture on the basis of the estimated data comprises determining at least one direction vector lying tangentially to the direction curve represented by the estimated data. Device (1) for estimating a directional information conveyed by a free space gesture for determining a user input at a human-machine interface, in particular a vehicle, the device being set up to carry out the method according to one of the preceding claims. Computer program that is configured, the method according to one of the Claims 1 to 15 to execute.

Images