WO2024184519A1

WO2024184519A1 - Method for three-dimensional reconstruction of a mobile element

Info

Publication number: WO2024184519A1
Application number: PCT/EP2024/056207
Authority: WO
Inventors: Alain René SIMON; François Edouard Michel DUPONT
Original assignee: Thales
Priority date: 2023-03-08
Filing date: 2024-03-08
Publication date: 2024-09-12
Also published as: FR3146517A1

Abstract

The method is implemented by an optronic system (16) comprising an imager and a computer. The method comprises the following steps: a. carrying out tracking of a mobile element of the scene such that the imager of the optronic system acquires several successive images of the mobile element seen at different angles according to the movement of the mobile element relative to the optronic system, and b. three-dimensionally reconstructing the mobile element as a function of the acquired images to obtain a three-dimensional model of the mobile element.

Description

DESCRIPTION

TITLE: Method for three-dimensional reconstruction of a moving element

The present invention relates to a method for three-dimensional reconstruction of a mobile element in a scene. The present invention also relates to a corresponding optronic system and a platform comprising such an optronic system.

When carrying out a mission, for example observation, it is of interest to detect and locate elements of the scene, or even to obtain more precise information on these elements, in particular three-dimensional (3D) information describing the shape of these elements.

For this purpose, reconstruction methods using passive sensors are known. However, these methods are relatively complex and take a long time to set up. In particular, in the case of a single sensor, these methods impose sensor trajectory constraints in order to have a sufficient base and a time delay that can be significant (it takes about a minute to travel 20 km at Machl, for example, using a transverse movement) to cover the base distance. In the case of several delocalized sensors, these methods require a network infrastructure and communications between the sensors, which complicates the organization, logistics and autonomy in terms of decision-making. In addition, matching the common parts of the element between the views of the different sensors is not always easy.

Other approaches rely on the use of active sensors, such as lidars or radars. However, these methods do not allow for discrete operation. In addition, they require significant energy consumption for the production of signals by the sensor.

There is therefore a need for a method enabling an optronic system to reconstruct in three dimensions non-cooperating moving elements, potentially at a great distance from the optronic system, in a simple, rapid and discreet manner, and without imposing displacement constraints, specific for the reconstruction, on the optronic system.

For this purpose, the present description relates to a method for three-dimensional reconstruction of a mobile element in a scene, the method being implemented by an optronic system comprising an imager and a computer, the method comprising the following steps: a. setting up a tracking of a mobile element of the scene so that the imager of the optronic system acquires several successive images of the mobile element seen from different angles depending on the movement of the mobile element relative to the optronic system, and b. three-dimensional reconstruction of the mobile element based on the acquired images to obtain a three-dimensional model of the mobile element.

Thus, the proposed method makes it possible to address the limitations of the state of the art, by intertwining the automatic reconstruction and tracking capabilities of mobile objects. The quality of the information produced is based in particular on opportunity events, even if it is not rare for a vehicle to change direction in its movement, since the relative rotation of a non-cooperating object with respect to the optronic system is not controlled over a short time (<10 seconds) or for relative movements of the approach/distance type and at a great distance from the object. The tracking remains permanent and its quality is sufficient, and this particularly in the absence of any rotation event of the object even if the information produced during the reconstruction is likely to maintain the tracking in rare more critical conditions where it could have been interrupted.

According to particular modes of implementation, the method comprises one or more of the following characteristics, taken in isolation or in all technically possible combinations: the reconstruction step is triggered automatically when the acquired images comprise at least two images of the mobile element taken at angles compatible with the three-dimensional reconstruction;

- the method comprises a step of determining the geographical position and the speed of movement of the mobile element relative to the optronic system as a function of the image acquired at the current time, of at least one image acquired at a previous time, and of the three-dimensional model of the mobile element;

- the method comprises updating the shooting and detection conditions of the mobile element according to primitives of the images used to obtain the three-dimensional model;

- the method comprises calculating the precisions on the determined geographical position and speed of movement, and possibly on the conditions of taking pictures of the images; the tracking of the mobile element is updated according to the last position and the last speed determined for the mobile element, and of the three-dimensional model of the mobile element; - the method comprises, over time, a step of updating the three-dimensional model as a function of the last image acquired by the imager; the three-dimensional model of the mobile element is obtained by extracting primitives characteristic of the mobile element in the images acquired by the imager and by matching the extracted primitives between said images; the three-dimensional model of the mobile element is also obtained by extracting primitives characteristic of the scene in the images acquired by the imager and by matching the extracted primitives between said images; the three-dimensional model obtained is displayed on a screen; the reconstruction step comprises determining a texture for the mobile element as a function of the acquired images, the three-dimensional model obtained being a model reproducing a texture for the mobile element; and

- the optronic system further comprises a distance evaluation unit capable of determining a distance between the mobile element and the optronic system, the evaluation unit comprising a rangefinder and/or a digital terrain model.

The present invention also relates to an optronic system comprising an imager and a computer, the optronic system being configured to implement a method as described above.

The present invention also relates to a platform, such as a vehicle, comprising such an optronic system.

Other features and advantages of the invention will become apparent from reading the following description of embodiments of the invention, given by way of example only and with reference to the drawings which are:

- [Fig 1] figure 1, a schematic representation of a scene in which a mobile element (car on a road) and an optronic system carried by a platform evolve,

- [Fig 2] figure 2, a schematic representation of an example of an optronic system comprising an imager, a calculator and optionally a distance evaluation unit, and

- [Fig 3] Figure 3, a flowchart of an example of implementation of a three-dimensional reconstruction method of a mobile element in a scene.

A scene 10 is illustrated as an example in Figure 1. A scene designates a theater of operations, that is, the place where an action takes place. The scene is therefore a large space with sufficient dimensions to allow an action to take place. The stage is typically an outdoor space.

The scene 10 comprises at least one element 12 that is mobile in the scene 10, that is, moving in the scene 10. The mobile element 12 is, for example, an object, such as a vehicle, or a living being, such as an animal or a human. In the example of FIG. 1, the mobile element 12 is a car moving on a road.

A platform 14, on which an optronic system 16 is mounted, is also illustrated in FIG. 1.

The platform 14 is mobile or fixed. In particular, when the platform 14 is mobile, the trajectory of the platform 14 is not modified specifically for the implementation of the reconstruction method which will be described in the remainder of the description.

The platform 14 is, for example, a vehicle, such as a land, sea or air vehicle (aircraft, such as an airplane or a drone). In the example illustrated by FIG. 1, the platform 14 is a drone. Alternatively, the platform 14 is a ground installation.

The platform 14 preferably has at least one position and attitude measuring device, making it possible to obtain the position and attitude of the optronic system 16, as well as to date these measurements. The measuring device is, for example, an inertial unit, or a GNSS (Global Navigation Satellite System). Alternatively, the measuring device is integrated into the optronic system 16.

The optronic system 16 is configured to implement a three-dimensional reconstruction method of mobile elements of the scene 10, such as the mobile element 12.

As schematically illustrated in FIG. 2, the optronic system 16 comprises an imager 20 and a calculator 22. Optionally, the optronic system 16 further comprises a distance evaluation unit 24.

The imager 20 (also referred to as “sensor” in the description) is a passive sensor capable of acquiring images of scene 10.

The imager 20 has a line of sight that can be oriented according to commands issued by the computer 22, making it possible in particular to track a mobile element 12 in the scene 10. In particular, the imager 20 comprises, for example, an actuator making it possible to orient the imager 20. The actuator is preferably automatic in nature. The actuator thus comprises an automatic attitude control system for the line of sight of the optronic system, possibly itself entangled with the object tracking/tracking function. The imager 20 comprises at least one camera. Preferably, the imager 20 comprises a single camera, which makes it possible to dispense with a communication network between different sensors. Alternatively, the imager 20 is formed by a set of cameras.

The imager 20 comprises at least one optical channel. The optical channel is suitable for producing a digital video at a sufficient rate with respect to the rotation speed of the mobile element 12. For nighttime implementation, an infrared detector is preferred, whereas during the day a visible channel, generally more resolved and having richer textures, is preferred.

Advantageously, the imager 20 has several optical channels, in different spectral bands and/or different fields of vision, suitable for acquiring simultaneous videos. Thus, the information obtained on the different channels can be merged to enrich/improve the processing results. The use of a multi-spectral camera also helps to characterize the nature of the materials of the mobile element 12 and to enrich its visual representation.

The calculator 22 comprises a calculation unit, and preferably also a display unit (screen) and a human-machine interface.

The computing unit includes, for example, a processor and memories.

In one example, the computing unit interacts with a computer program product that includes an information medium. The information medium is a medium readable by the computing unit. The readable information medium is a medium suitable for storing electronic instructions and capable of being coupled to a bus of a computer system. The computer program product including program instructions is stored on the information medium.

The computer program is loadable onto the computing unit and causes the implementation of a three-dimensional reconstruction method of a mobile element 12 in a scene 10, when the computer program is implemented on the computing unit as will be described in the remainder of the description.

The distance evaluation unit 24 comprises, for example, either a rangefinder, such as a laser rangefinder, or a digital terrain model (DTM) associated with ray tracing processing, or both sources of information. In particular, a DTM is data representing the ground surface of the scene 10. The DTM makes it possible to retrieve the altitude of a point with known coordinates in latitude and longitude, or to measure a distance from the sensor to the scene 10 by ray tracing processing. The DTM is, for example, incorporated in software in a memory of the computer 22. In the case of a system benefiting from both a rangefinder and a DTM, the two distances obtained are, for example, merged according to their respective covariances. This operation allows the first of all to assess whether the mobile object is moving well on the ground surface, by means of a statistical test on the two distances and their variances, and in this case, to obtain a better evaluation of the distance.

The operation of the optronic system 16 leading to the implementation of a three-dimensional reconstruction method of a mobile element 12 in a scene 10 will now be described with reference to the flowchart in FIG. 3.

The reconstruction method comprises a step 1 10 of tracking a mobile element 12 of the scene 10. The tracking is carried out so that the imager 20 of the optronic system 16 acquires several successive images (at least two) of the mobile element 12 seen from different angles depending on the movement of the mobile element 12 relative to the optronic system 16.

Preferably, the mobile element 12 has a rotational movement relative to the optronic system 16 during acquisitions, which makes it possible to image different faces of the mobile element 12. Such rotational movements (as well as more generally the evolution of the shooting angles) are analyzed during tracking. This allows the triggering of the reconstruction when the conditions on the feasibility of the 3D calculations are met as will be described in the remainder of the description.

The tracking of the mobile element 12 makes it possible to maintain the mobile element 12 in the field of vision of the imager 20, and advantageously close to the center of the image if telemetry is used. The tracking is, for example, carried out by servocontrols making it possible to control the line of sight of the imager 20 (i.e. the orientation of the imager 20) as a function of inertial data from the optronic system 16 (obtained via the position and attitude measuring device). The orientation of the imager 20 is, for example, carried out manually, or automatically by coupling with the tracking processing.

The tracking also makes it possible to locate the mobile element 12 on the successive images, and this in a precise and robust manner. Such tracking is, for example, based on a classic image processing algorithm compatible with the frame rate. For example, the algorithm is an ATDR algorithm (for “automatic target detection and recognition”). In addition or as a variant, the algorithm implements a correlation approach or a Siamese neural network type approach.

Thus, the tracking makes it possible to obtain two-dimensional information on the mobile element 12, namely its location, its kinematics, and its contour. The images obtained are preferably associated with auxiliary image data characterizing the geographic position and attitude of the optronic system 16 during the acquisition of said images.

Figure 1 illustrates an example of image acquisition of a mobile element 12 at different times t1, t2, t3 and t4, making it possible to obtain images of the mobile element 12 from different viewing angles.

The reconstruction method comprises a step 120 of three-dimensional reconstruction of the mobile element 12 based on the images acquired during tracking, to obtain a three-dimensional model of the mobile element 12.

Preferably, the reconstruction step 120 is triggered automatically when the acquired images comprise at least two images of the mobile element 12 taken at angles compatible with the three-dimensional reconstruction, that is to say that at least one different face of the mobile element 12 is imaged on at least two images.

The reconstruction of the three-dimensional model is, for example, carried out by determining the 2D-3D optical flow or by correspondence between characteristic points on the different images of the mobile element 12. The choice of technique depends in particular on the level of resolution of the image and the spectral band.

In an exemplary implementation, the reconstruction step 120 comprises the extraction of primitives (points or set of points) characteristic of the mobile element 12 in the images acquired by the imager 20, and the matching of the extracted primitives between said images to obtain the three-dimensional model.

Preferably, the reconstruction step 120 also comprises the extraction of characteristic primitives of the scene 10 in the images acquired by the imager 20, and the matching of the extracted primitives between said images to improve the knowledge of the shooting parameters and consequently the quality of the reconstruction (3D model, kinematics, etc.). This makes it possible to enrich the observations and to increase the constraints in the estimation of the model, as well as to improve the values of the shooting parameters carried by the auxiliary data of the image. In an optional exemplary implementation, the three-dimensional model is also determined as a function of a determined distance between the mobile element 12 and the optronic system 16. The distance is determined by the evaluation unit. Thus, optionally, in one example, when the evaluation unit comprises a rangefinder, the distance is determined via the rangefinder. When the evaluation unit comprises a digital terrain model, the distance is, for example, determined by a ray tracing method. In this case, the distance obtained is the distance between the position of the optronic system 16 and the intersection of a predetermined half-line with the ground of a digital terrain model. The predetermined half-line passes through the position of the optronic system 16 and has the orientation of the line of sight of the imager 20 of the optronic system 16.

Generally speaking, the shooting conditions (CPDV) and the distance allow to build a 3D model to scale with the right dimensions, note that in the absence of a rangefinder the DTM would allow to have a sufficient scale quality (let's say better than the class of 5/100). Note that the quality of the CPDV and the distance also conditions that of the kinematics a distance precision of X% hardly allows to determine the geographical position and the speed of the object to better than x%.

Preferably, the reconstruction step 120 comprises determining a texture for the mobile element 12 based on the acquired images. The texture is the appearance of the surface of the mobile element 12 (smooth, rough, grainy, etc.). The three-dimensional model obtained is a model reproducing a texture for the mobile element 12.

Preferably, the three-dimensional model obtained is displayed on a screen. For example, the model is displayed in the form of points or facets, with and without texture. The display is carried out for example from a direction fixed by default or chosen by the user, possibly in the form of a video presenting different images under variable directions historicizing the angular domain on which the object could be reconstructed until then.

Preferably, the reconstruction step 120 also comprises the determination of the 3D position and the 3D movement speed of the mobile element 12 relative to the optronic system 16 as a function of the acquired images and the three-dimensional model of the mobile element 12. The three-dimensional model makes it possible, in fact, to complete the two-dimensional positions and kinematics of the mobile element 12, obtained during tracking.

Thus, the tracking step 110 operates independently, preferably at the image acquisition rate, while the reconstruction step 120 is only triggered punctually (for example if conditions are met). In particular, the tracking is implemented in parallel with the possible reconstruction and is not modified regardless of the state of the reconstruction (inactive, initialization or maintenance).

Preferably, when a reconstruction has taken place, the tracking is enriched with the results of the reconstruction. In particular, the tracking function implementing the tracking takes into account, for example, the last position and the last speed determined for the mobile element 12, as well as the three-dimensional model of the mobile element 12.

In particular, the three-dimensional model makes it possible to better characterize the dimensions of the mobile element 12, its own rotations and its kinematics, and thus to feed the automatic tracking and recognition algorithm. Tracking thus benefits from spatial segmentation and better prediction of the appearance of the mobile element 12 on the current image, which eliminates some cases of disconnection in difficult conditions. Tracking also benefits from better characterization of the 3D kinematics of the mobile element 12 and its 3D rotation. Automatic recognition is improved with the measurements of the 3D model.

Preferably, the method comprises a step 130 of updating the three-dimensional model over time based on the current images of the mobile element 12 acquired by the imager 20. Indeed, the 3D reconstruction is an incremental method which is enriched with image information over time both by the angular variability of the reconstruction and by the resolution of the model which can be reconstructed. The three-dimensional model is, thus, either the same model as the previous model, or a better resolved model (shape with more details) than the previous model. This can also allow a “progressive unmasking” of certain parts of the element considered. Preferably, the updated three-dimensional model is displayed on a screen.

Preferably, the update step 130 comprises updating the shooting and detection conditions of the mobile element according to primitives of the images used to obtain the three-dimensional model. For this, a triangularization is for example carried out.

Preferably, the updating step 130 comprises updating over time the 3D geographic position and the 3D movement speed of the mobile element 12 relative to the optronic system 16 as a function of the image acquired at the current time, of at least one image acquired at a previous time, and of the three-dimensional model of the mobile element 12. Preferably, data relating to the precision of the 3D geographic position and of the 3D speed are also determined.

Preferably, the update step 130 also includes the calculation of the details on the determined geographic position and speed of movement, and possibly on the conditions for taking pictures of the images.

Thus, the reconstruction method described makes it possible to reconstruct a mobile element 12 in three dimensions, under the following operating conditions: by means of the imager 20 of the optronic system 16 tracking the mobile element 12 (this means that the mobile element 12 is maintained in the image of the optronic video independently of the respective movements of the mobile element 12 and of the optronic system 16). even with imperfect measurements of the positions of the optronic system 16, the attitudes of the images and the internal parameters of the optronic channel used. by detecting, in a timely manner, rotational movements of the mobile element 12. on a non-cooperating element, therefore without the possibility of acting on the presentation of the mobile element 12 relative to the imager 20. in critical acquisition conditions in terms of the ratio between the base of the shooting vertices at the distance of the optronic system 16 to the mobile element 12, in a short time as soon as there is an opportunity for rotation of the object on itself.

Thus, the reconstruction method described makes it possible to reconstruct in 3D the external shape of a mobile element in a scene 10, with a level of detail depending on the resolution of the images, even if the element is non-cooperative and evolves at a great distance (several km and even several tens of km) from the optronic system 16, therefore with an unfavorable base distance ratio (typically less than 0.1). The reconstruction can be carried out with a single passive sensor (therefore discretely and without communication between sensors), and in a short time of the order of 0 to 10 seconds. Such a method also makes it possible to characterize the kinematics of the mobile element 12.

The reconstruction is all the more complete as the mobile element 12 moves relative to the optronic system 16, and if necessary performs rotational movements. The reconstruction is all the more precise as the images are of better resolution, conditioned by the characteristics of the optronic channel used in the system, as well as the distance to the object. Furthermore, the precision of the model is also improved by the redundancy of the images used for the reconstruction.

The proposed 3D reconstruction process of a mobile element is called optronic inverse reconstruction (RIO), since unlike most classic 3D reconstruction techniques which exploit the movement of the sensor to reconstruct a scene 10, the 3D reconstruction problem of an element here exploits the mobility of the element, the sensor being able to be extremely fixed during the acquisitions.

The three-dimensional model obtained also makes it possible to improve the robustness of other algorithms implemented by the optronic system 16 (tracking, tracking, etc.), as well as the visual decision of the operators. The method also makes it possible to improve the visualization, allowing better resolution of the mobile element 12, and the possibility of detaching the mobile element 12 from the scene 10, of presenting it under different faces with complementary visual clues useful for its identification (reduces the delay and improves the analysis).

The method is also applicable for the simultaneous tracking and reconstruction of several moving elements of the scene 10, as long as these elements remain visible from the same line of sight for the imager 20.

Preferably, the method comprises, either at each instant (in real time) or with a set of images, an update of the shooting parameters by benefiting from the correspondences between the primitives extracted from the current image and the fixed primitives of the scene previously extracted by means of a process of the simultaneous mapping and localization (SLAM) type or of the aero-triangulation type.

Preferably, the recognition of a vehicle-type object could, for example, give rise to a 3D model completion treatment by symmetries if we wish to propose a more exhaustive vision of the reconstructed partial model. This completion by symmetrisations is limited to duplicating the extracted shapes and structures on the opposite side face of the vehicle.

The process is particularly suitable for the following industrial applications:

- vehicle reconstruction from a sensor positioned on another vehicle.

- helps to better identify a moving element by adding a spatial dimension and improving the resolution thanks to the multiplicity of images (super resolution).

- characterization of the kinematics of the mobile element in terms of speed and 3D orientation over time.

Those skilled in the art will understand that the embodiments and variants of the description can be combined provided that they are technically compatible.

Claims

1. A method for three-dimensional reconstruction of a mobile element (12) in a scene (10), the method being implemented by an optronic system (16) comprising an imager (20) and a computer (22), the method comprising the following steps: a. setting up tracking of a mobile element (12) of the scene (10) such that the imager (20) of the optronic system (16) acquires several successive images of the mobile element (12) seen from different angles as a function of the movement of the mobile element (12) relative to the optronic system (16), and b. three-dimensional reconstruction of the mobile element (12) as a function of the acquired images to obtain a three-dimensional model of the mobile element (12).

2. Method according to claim 1, in which the reconstruction step is triggered automatically when the acquired images comprise at least two images of the mobile element (12) taken at angles compatible with the three-dimensional reconstruction.

3. Method according to claim 1 or 2, in which the method comprises a step of determining the geographical position and the speed of movement of the mobile element (12) relative to the optronic system (16) as a function of the image acquired at the current time, of at least one image acquired at a previous time, and of the three-dimensional model of the mobile element (12).

4. Method according to any one of claims 1 to 3, in which the method comprises updating the conditions for taking pictures and detecting the moving element as a function of primitives of the images used to obtain the three-dimensional model.

5. Method according to claim 3 or 4, in which the method comprises calculating the details of the determined geographical position and speed of movement, and possibly of the conditions for taking the images.

6. Method according to any one of claims 1 to 5, in which the tracking of the mobile element (12) is updated according to the last position and the last speed determined for the mobile element (12), and the three-dimensional model of the mobile element (12).

7. Method according to any one of claims 1 to 6, in which the method comprises, over time, a step of updating the three-dimensional model as a function of the last image acquired by the imager (20).

8. Method according to any one of claims 1 to 7, in which the three-dimensional model of the mobile element (12) is obtained by extracting characteristic primitives of the mobile element (12) in the images acquired by the imager (20) and by matching the extracted primitives between said images.

9. Method according to claim 8, in which the three-dimensional model of the mobile element (12) is also obtained by extracting characteristic primitives of the scene (10) in the images acquired by the imager (20) and by matching the extracted primitives between said images.

10. Method according to any one of claims 1 to 9, in which the three-dimensional model obtained is displayed on a screen.

11. Method according to any one of claims 1 to 10, in which the reconstruction step comprises determining a texture for the mobile element (12) based on the acquired images, the three-dimensional model obtained being a model reproducing a texture for the mobile element (12).

12. Method according to any one of claims 1 to 11, in which the optronic system (16) further comprises a distance evaluation unit (24) capable of determining a distance between the mobile element (12) and the optronic system (16), the evaluation unit (24) comprising a rangefinder and/or a digital terrain model.

13. Optronic system (16) comprising an imager (20) and a computer (22), the optronic system (16) being configured to implement a method according to any one of claims 1 to 12.

14. Platform (14), such as a vehicle, comprising an optronic system (16) according to claim 13.