DE102019220335A1 - SEMANTIC SEGMENTATION USING DRIVER ATTENTION INFORMATION - Google Patents

Abstract

Method for generating trained semantic segmentation network models and operating vehicles using the model. An exemplary method includes an exterior view camera configured to capture images that represent an artificial representation of the driver's view, a driver-facing camera configured to capture a driver's eye movements, and electronic control. The electronic control is designed to receive images from the cameras; calibrate the image of the driver's eye movements with the artificial driver's vision; generate a pixel-weighted thermal map of the calibrated images; generate a trained semantic neural segmentation network model and a trained attention-based neural network model using the pixel-weighted heat map and the artificial driver view; and operate the vehicle using the trained semantic neural segmentation network model and the trained attention-based neural network model.

Description

CROSS REFERENCE TO RELATED PATENT APPLICATIONS

This patent application claims priority over the provisional one U.S. Patent Application No. 62 / 786,711 , filed on December 31, 2018, the entire contents of which are incorporated by reference.

AREA

Embodiments relate, inter alia, to the acquisition of training data for an attention-based neural network model and the use of the trained model for autonomous driving functions.

BACKGROUND

Modern vehicles have an autonomous or semi-autonomous driving functionality that uses semantic segmentation network models. These segmentation network models are trained, for example, with regard to object identification and identifications that are assigned to each pixel in the defined objects.

SUMMARY

The importance of pixels is ignored during training of the segmentation network models, and each pixel is viewed the same. In reality, pixels have different relevance, and some pixels should be considered more important than others. Even if segmentation network models are known, the training of segmentation network models, including the use of weighted pixels for representing a different importance, is either not available or is not implemented when using driver-related eye tracking.

Embodiments described herein provide, among other things, a system and method for using a semantic segmentation model that is trained with data that is referenced to incorporate pixel weighting based on importance.

One embodiment provides a system for generating a trained semantic neural segmentation network model and a trained attention-based neural network model for operating a vehicle. The system includes an exterior view camera, a driver-facing camera, and one or more electronic controls. The one or more electronic controls are designed to receive an image from the exterior view camera and an image of a driver's eye movements from the camera directed at the driver. The one or more electronic controls are configured to calibrate the image of the driver's eye movements with the image from the outside view camera to produce calibrated images that are used to generate a pixel-weighted thermal map of the calibrated images. The one or more electronic controls are also configured to generate the trained semantic neural segmentation network model using the image from the outside view camera, and to generate the trained attention-based neural network model using the pixel-weighted thermal map. The one or more electronic controls are designed to operate the vehicle using the trained semantic neural segmentation network model and the trained attention-based neural network model.

Another embodiment provides a method for generating a trained semantic neural segmentation network model and a trained attention-based neural network model for operating a vehicle. The method includes receiving, via one or more controls, an image from an outside view camera and an image of a driver's eye movements from a camera directed at the driver. The method also includes calibrating, via the one or more electronic controls, the image of the driver's eye movements with the image from the exterior view camera to produce calibrated images that are used to generate a pixel-weighted thermal map of the calibrated images. The method also includes creating, via the one or more electronic controls, the trained semantic neural segmentation network model using the image from the outside view camera and generating the trained attention-based neural network model using the pixel-weighted thermal map. The method also includes operating, via the one or more electronic controls, the vehicle using the trained semantic neural segmentation network model and the trained attention-based neural network model.

Other aspects, features, and embodiments will become apparent upon consideration of the detailed description and accompanying drawings.

Figure list

• 1 10 is a block diagram of a system for improving the acquisition of training data for a semantic segmentation model according to an embodiment.
• 2nd is a block diagram of a first electronic control of the system of 1 according to one embodiment.
• 3rd FIG. 4 is a block diagram of a second electronic control of the system of FIG 1 according to one embodiment.
• 4th Figure 3 is a block diagram of an exterior view camera of the system of 1 according to one embodiment.
• 5 FIG. 4 is a block diagram of a driver-facing camera of the system of FIG 1 according to one embodiment.
• 6 FIG. 10 is a flow diagram of a method of using an image calibration engine of the electronic controller of FIG 2nd to, according to one embodiment, input from the cameras of 1 and 2nd to receive and generate a pixel-weighted heat map based on the input.
• 7 FIG. 10 is a flow diagram of a method of using an attention-based neural network model trainer of the electronic controller of FIG 3rd to integrate the pixel-weighted thermal map and the exterior view image with a cost function used in training the attention-based neural network model according to one embodiment.
• 8th FIG. 10 is a flow diagram of a method of using a semantic neural segmentation network model trainer of the electronic controller of FIG 3rd to integrate the exterior view image while training the semantic neural segmentation network model according to one embodiment.
• 9 10 is a block diagram of a system for operating an autonomous vehicle using a semantic segmentation classifier according to an embodiment.
• 10th FIG. 10 is a flowchart of a method for using a vehicle operating engine of the third electronic controller of FIG 9 to receive an exterior view image from a second exterior view camera and process the image by trained models to operate a vehicle.
• 11 Fig. 3 is a conceptual illustration of a labeled thermal map used in the process of 6 was generated to be used in training the attention-based neural network model.

DETAILED DESCRIPTION

Before explaining any embodiments in detail, it should be understood that the application of this disclosure is not intended to be limited to the construction details and component arrangements set forth in the description below or illustrated in the drawings below. Embodiments are also designed for other configurations and for being practically used or implemented in various ways.

Multiple hardware and software based devices as well as multiple different structural components can be used to implement different embodiments. In addition, embodiments may include hardware, software, and electronic components or modules that, for purposes of discussion, may be illustrated and described as if most of the components were implemented in hardware alone. However, those of ordinary skill in the art, based on reading this detailed description, would recognize that, in at least one embodiment, the electronics-based aspects of the invention may be implemented in software (e.g., stored on a non-transitory computer readable medium) software executable by one or more processors is. For example, “control units” and “controls” described in the description can include one or more electronic processors, one or more memory modules with a non-transitory computer-readable medium, one or more communication interfaces, one or more application-specific integrated circuits (ASICs) ) and various connections (for example a system bus or one or more networks) that connect the various components.

1 illustrates a vehicle system 100 for calibrating images outside of the vehicle with detected driver eye movements that are used to train an attention-based neural network model. The system 100 includes a vehicle 105 . The vehicle 105 Even though it is illustrated as a four-wheeled vehicle, it can include various types of vehicle designs. For example, the vehicle 105 an automobile, a motorcycle, a truck, a bus, trade a half tractor and other vehicles. In the illustrated example, the vehicle is facing 105 several hardware components, including a first electronic control 110 , a first exterior view camera 115 and a camera aimed at the driver 120 . The first electronic control 110 is communicative with the first exterior view camera using one or more connections, for example direct connections and network connections (for example a control network or CAN bus) 115 , the camera aimed at the driver 120 and a second electronic control 125 connected. Wired and wireless connections are possible.

The first exterior view camera 115 is designed to capture images outside the vehicle. The camera aimed at the driver 120 is designed to record driver eye movements. The first exterior view camera 115 and the camera aimed at the driver 120 provide data (images) for the first electronic control 110 of the system 100 ready.

The second electronic control is in the illustrated example 125 outside the vehicle 105 and is communicative with the first electronic controller over one or more communication links and, in this example, over a wireless link 110 connected. However, the second electronic control can 125 also in the vehicle system 100 located and connected via direct connections or via a vehicle network or a bus.

2nd is a block diagram of the first electronic control 110 of the system of 1 . The first electronic control 110 has multiple electrical and electronic components that provide power, operational control, and protection for the components and modules in the first electronic control 110 provide. The first electronic control 110 has, among other things, a first electronic processor 205 (for example a programmable electronic microprocessor, a microcontroller or a similar device) and a first memory 210 on. The first store 210 is, for example, a non-transitory, machine-readable memory. The first electronic control also has a first communication interface 215 on.

The first electronic processor 205 is communicative with the first memory 210 and the first communication interface 215 connected. The first store 210 has an image calibration engine 220 on. The image calibration engine 220 For example, software or a set of computer-readable instructions that input from the first exterior camera 115 and the camera aimed at the driver 120 calibrated. After the input has been calibrated, it is the first electronic control 110 designed for calibrated input and input from the first exterior camera 115 for the second electronic control 125 to provide. In other embodiments, the image calibration engine may itself 220 located in the second electronic control 125. In embodiments in which the image calibration engine 220 in the second electronic control 125 is the first electronic control 110 the input from the first outside view camera 115 and the camera aimed at the driver 120 directly for the second electronic control 125 ready. The first electronic processor 205 , in coordination with that in the first store 210 stored software (e.g. the software described above), and the first communication interface 215 are designed to implement one or more of the methods described here.

The first electronic control 110 can be implemented in several independent controls (e.g. programmable electronic controls), each designed to perform specific functions or sub-functions. In addition, the first electronic control 110 Include sub-modules that include additional electronic processors, memory, or application-specific integrated circuits (ASICs) to handle communication functions, process signals, and use the methods listed below. In other embodiments, the first electronic controller 110 additional, fewer or different components.

3rd is a block diagram of the second electronic control 125 of the vehicle 105 . The second electronic control 125 is, in general, similar to the first electronic control 110 . Therefore, not all details of their architecture and their connections to other components are described below. The second electronic control 125 among other things, has a second electronic processor 305 (e.g., a programmable electronic microprocessor, microcontroller, or similar device) and a second memory 310 on. The second store 310 is, for example, a non-transitory, machine-readable memory. The second electronic control also has a second communication interface 315 on.

The second electronic processor 305 is communicative with the second memory 310 and the second communication interface 315 connected. The second store 310 has an attention-based neural network model trainer 320 and a semantic neural segmentation network model trainer 325 on. With the attention-based neural network model trainer 320 For example, it can be software or a set of computer-readable instructions that input from the first electronic controller 110 receives and trains an attention-based neural network model with the received input to generate a trained attention-based neural network model. With the semantic segmentation network model trainer 325 For example, it can be software or a set of computer-readable instructions that input from the first electronic controller 110 receives and trains a semantic neural segmentation network model with the received input to generate a trained semantic neural segmentation network model. In the illustrated example, the input comes from the first electronic control 110 that are in the vehicle 105 located. However, in other embodiments, multiple vehicles of the second electronic control can 125 Send entries.

Like the first electronic control 110 can also use the second electronic control 125 be implemented in several independent controls, each designed to perform specific functions or sub-functions. In addition, the second electronic control 125 Include sub-modules that include additional electronic processors, memory, or application-specific integrated circuits (ASICs) to handle communication functions, process signals, and use the methods listed below. In addition, the second electronic control 125 additional, fewer or different components than those shown.

4th Fig. 3 is a block diagram of the first outside view camera 115 of the vehicle 105 . The first exterior view camera 115 has, among other things, a third communication interface 405 , a first image signal processor 410 and a first lens and image sensor assembly 415 on. In the system 100 The camera is a rear-facing camera that is designed in such a way that the images provided resemble what the driver looks through the windshield of the vehicle 105 looks outside.

5 is a block diagram of the driver-facing camera 120 of the vehicle 105 . The camera aimed at the driver 120 has, among other things, a fourth communication interface 505 , a second image signal processor 510 and a second lens and image sensor assembly 515 on. As the name implies, the camera is aimed at the driver 120 faces the driver and is designed to track the driver's eye movements. As explained in more detail below, the tracked eye movements are used to capture images from the first outside view camera 115 to calibrate.

Images from the first exterior camera 115 and the camera aimed at the driver 120 are captured for the image calibration engine 220 provided. 6 illustrates an exemplary method 600 the image calibration engine 220 . The image calibration engine 220 controls the calibration of the images and generates a calibrated heat map. Even if semantic segmentation network models are currently only trained with labels that categorize objects in an image and predictions as to which label each pixel is relevant, a semantic segmentation model is created in this way, in which each pixel or group of pixels in one Object has the same relevance. However, a human mind does not process every object in a field of view with the same relevance. Thus, looking at a driver's eye movement in relation to the pictures taken by the first exterior view camera 115 are detected, advantageous and provides a second level of relevance or information of interest for pixels in an object contained in images by the first exterior view camera 115 are recorded.

In step 605 receives the first electronic processor 205 a first picture from the first exterior view camera 115 . The first image is used as an artificial representation of the driver's view. In step 610 a second image of the driver's eye movement is received. Next, a calibration is performed to determine which pixels correspond to the driver's eye movement (step 615 ). The calibration of pixels for eye movement is accomplished by pixel weighting based on the center of the driver's gaze (ie, where the driver is looking). Known techniques for determining the center of a person's gaze can be used in described embodiments. Referring to the weighting, in one example, when the driver looks at a pedestrian crossing the street, the pedestrian is weighted more than a bird flying in the sky. In one case, each pixel in the image is assigned a weight by the outside view camera based on the image of the driver's eye movements. The calibrated images are then sent from the first electronic processor 205 processed to generate a pixel weighted thermal map (step 620 ). In one example there is the pixel-weighted heat map Weightings by the darkness of color. The darker the color in the thermal map, the higher the weighting of the corresponding pixel. In other embodiments, the weighting of pixels is represented in other shapes. For example, a shape can be a number location of pixels with corresponding weighting values. As soon as the images are calibrated, the representation of the calibrated images (for example the heat map) for the second electronic control 125 provided (step 625 ). In one embodiment, the representation of the calibrated images after generation for the second electronic control 125 to be provided. However, in other embodiments, the calibrated images can be stored in a data storage location and at a later time to the second electronic controller 125 be transmitted.

7 illustrates an exemplary method 700 of the attention-based neural network model trainer 320 . In step 705 receives the attention-based neural network model trainer 320 a pixel-weighted thermal map and an exterior view image taken by the first exterior view camera 115 has been recorded. As noted, semantic segmentation models are currently trained using labels to classify pixels in object groups and do not take pixel weighting into account. In the exemplary embodiment, the pixel-weighted thermal map is used as a classification identifier and the exterior view image is used as input. Training can be done using a cross entropy cost function. The second electronic processor 305 trains the model using a cost function that includes the weighting of pixels represented by the pixel-weighted thermal map (step 710 ).

8th illustrates an exemplary method 800 of the semantic neural segmentation network model trainer 325 . In step 805 receives the semantic neural segmentation network model trainer 325 an exterior view image from the first exterior view camera 115 . The second electronic processor 305 trains the model using a cross entropy cost function that includes the exterior view image (step 810 ). Although a cross entropy cost function is used in one example, another cost function may be used in other embodiments. The trained semantic neural segmentation network model provides a mechanism for analyzing images and categorizing segmentation features in an image.

After the trained semantic neural segmentation network model and the trained attention-based neural network model have been generated, vehicles with autonomous driving functionality can integrate the models into autonomous and semi-autonomous driving systems, and a vehicle can be operated according to the models. 9 illustrates an automated vehicle system 900 for operating an automated vehicle. In the illustrated example, the automated vehicle system includes 900 a similar architecture to the vehicle 105 from 1 . However, the automated vehicle system 900 not a camera aimed at the driver. The hardware components in the automated vehicle system 900 have a third electronic control 905 and a second outside view camera 910 on. The third electronic control 905 is communicative with the second exterior camera 910 connected.

The second exterior view camera 910 is designed to capture images outside of the automated vehicle. The second exterior view camera 910 provides data (pictures) for the third electronic control 905 of the system 900 ready. The third electronic control 905 among other things, has a third electronic processor 915 (e.g., a programmable electronic microprocessor, microcontroller, or similar device) and a third memory 920 on communication links between the third electronic processor 915 and the third memory 920 are similar to the connections between the electronic processors described above and the memory. The third store 920 is, for example, a non-transitory, machine-readable memory. The third store 920 has a vehicle operating engine 930 on. The vehicle operating engine 930 For example, software or a set of computer-readable instructions that input from the second exterior camera 910 processed and generated image features for autonomous and semi-autonomous driving systems. The third electronic processor 915 is in coordination with that in the third memory 920 stored software (for example, the software described above), designed to implement one or more of the methods described herein.

10th illustrates an exemplary method 1000 the vehicle operating engine 930 , which controls autonomous and semi-autonomous driving systems, which have so-called advanced driver assistance systems (ADASs). Examples of ADASs include lane detection and lane keeping systems, collision warning and collision avoidance systems, object detection systems and free space detection systems. In step 1005 becomes one of the second exterior view camera 910 captured external view image from a third electronic control 905 receive. The exterior view image is then processed by the trained semantic neural segmentation network model, which is in process 800 was trained. Processing the exterior view image with the semantic neural segmentation network model extracts segmentation features (step 1010 ). In one example, segmentation feature extraction identifies the presence of a vehicle. The exterior view image is also processed by the trained attention-based neural network model that is in process 700 was trained. A pixel-weighted heat map is generated based on the processing of the exterior view image with the attention-based neural network model (step 1015 ). The pixel-weighted heat map represents a prediction made by the attention-based neural network model, which indicates the importance of pixels in an image based on training based on data from a driver's eye movements. After the pixel-weighted heat map and the segmentation features have been obtained, the features and the heat map are chained (step 1020 ). Next, the chained segmentation features and the thermal map are processed with a modified cost function of a segmentation classifier (step 1025 ). In some embodiments, the segmentation classifier's cost function may be modified to accept both inputs without the inputs being chained. The segmentation classifier uses a cross entropy cost function that includes the weighting of pixels represented by the pixel-weighted thermal map and the segmentation features. An example of a cross entropy cost function is as follows: $${\displaystyle {\sum }_{pixel}{\displaystyle {\sum }_{class}{y}_{true}\text{log}\left(y_{predict}\right)}}$$

w_pixel ist hierbei die mit jedem Pixel verknüpfte Gewichtung, die basierend auf der Gewichtung bestimmt wird, welche als Teil der Korrelation von Blickinformationen von der auf den Fahrer gerichteten Kamera 120 zu den Bildern von der ersten Außenkamera 115 durchgeführt wird, bestimmt durch die Kalibrierungsengine 220 in Verfahren 600. Als Nächstes wird das Fahrzeug gemäß der Verarbeitung der verketteten Segmentierungsmerkmale und der pixelgewichteten Wärmekarte mit dem Segmentierungsklassifikator betrieben.Using the pixel weighted heat map, equation 1 is changed as follows: $${\displaystyle {\sum }_{pixel}{\displaystyle {\sum }_{class}{y}_{true}{\text{w}}_{pixel}\text{log}\left(y_{predict}\right)}}$$

w _pixel is the weighting associated with each pixel, which is determined based on the weighting, which is part of the correlation of gaze information from the camera directed at the driver 120 to the pictures from the first outdoor camera 115 is performed, determined by the calibration engine 220 in proceedings 600 . Next, the vehicle is operated with the segmentation classifier according to the processing of the concatenated segmentation features and the pixel weighted thermal map.

Although a cross entropy cost function is used in one example, another cost function may be used and altered in other embodiments to include the weighting of pixels provided by the image calibration engine 220 be determined.

The weighting of pixels determined by the image calibration as indicated above is presented as a pixel weighted thermal map. 11 Figure 3 is an illustration of a pixel weighted thermal map 1100 according to one embodiment. In the example provided, the degree of shading of a region represents the weighting of the pixels in the region. For example, the area has no shading 1105 a very little to no weighting, while the most shaded region 1110 has the highest weight. In the illustrated example, pixels are grouped into regions with an assigned weight. However, the pixels in the group may have actual weights that are within a tolerance of the assigned weight, for example within 1% of the assigned weight. In other embodiments, there are no regions and the weighting of the pixels are actual values received by calibrating a driver's eye movements with the exterior view image.

Adding weights to these pixels improves autonomous and semi-autonomous driving functionality, such as ignoring objects that are less relevant and taking action (such as braking, controlling, or accelerating) based on objects that are more relevant.

Even if a certain sequence of steps is given in the procedures that are described in 6-8 and 10th As illustrated, the timing, sequence, and inclusion of steps may vary without compromising the purpose and advantages of the examples provided.

Thus, the embodiments described herein provide, among other things, a system and method for capturing and calibrating images to produce a representation of calibrated images that indicates a weighting of pixels based on relevance. Various features and advantages of the embodiments are set out in the claims below.

QUOTES INCLUDE IN THE DESCRIPTION

This list of documents listed by the applicant has been generated automatically and is only included for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• US 62786711 [0001]

Claims (17)

A system for generating a trained semantic neural segmentation network model and a trained attention-based neural network model for operating a vehicle, the system comprising: an outside view camera; a camera aimed at the driver; one or more electronic controls, designed for: receive an image from the outside view camera; receive an image of a driver's eye movements from the driver-facing camera; calibrate the image of the driver's eye movements with the image from the outside view camera to produce calibrated images; generate a pixel-weighted thermal map of the calibrated images; generate the trained semantic neural segmentation network model using the image from the outside view camera; generate the trained attention-based neural network model using the pixel-weighted thermal map and the image from the outside view camera; and operate the vehicle using the trained semantic neural segmentation network model and the trained attention-based neural network model. System according to Claim 1 wherein a first electronic controller receives the image from the outside view camera; receives the image of a driver's eye movements from the driver-facing camera; the image of the driver's eye movements calibrated with the image from the outside view camera to produce calibrated images; and generates the pixel-weighted thermal map of the calibrated images, and wherein a second electronic controller receives the pixel-weighted thermal map of the calibrated images and the image from the outside view camera. System according to Claim 2 , wherein the second electronic controller is designed to generate the trained semantic neural segmentation network model and the trained attention-based neural network model with a cost function that includes the pixel-weighted heat map. System according to Claim 3 , wherein the second electronic controller is further configured to train a semantic segmentation classifier with a cost function that includes the pixel-weighted heat map. System according to Claim 2 , wherein the second electronic control is designed to receive pixel-weighted heat cards from several vehicles. System according to Claim 1 , with the outside view camera being a front-facing camera. System according to Claim 1 , wherein the one or more electronic controls are configured to calibrate the driver's eye movement image with the image from the exterior view camera by assigning a weight to each pixel in the image from the exterior view camera based on the image of the driver's eye movement . System according to Claim 1 , wherein the generation of the trained attention-based neural network model includes using the pixel-weighted heat map as a classification identifier and the exterior view image as an input. A method for generating a trained semantic neural segmentation network model and a trained attention-based neural network model for operating a vehicle, the method comprising: Receiving, via one or more electronic controls, an image from an outside view camera; Receiving, via the one or more electronic controls, an image of a driver's eye movements from a camera directed at the driver; Calibrating, via the one or more electronic controls, the image of the driver's eye movements with the image from the exterior view camera to produce calibrated images; Generating, via the one or more electronic controls, a pixel-weighted thermal map of the calibrated images; Generating, via the one or more electronic controls, the trained semantic neural segmentation network model using the image from the exterior view camera; Generating, via the one or more electronic controls, the trained attention-based neural network model using the pixel-weighted thermal map and the image from the exterior view camera; and Operate, via the one or more electronic controls, the vehicle using the trained semantic neural segmentation network model and the trained attention-based neural network model. Procedure according to Claim 9 , in which receiving an image from the outside view camera; receiving an image of a driver's eye movements from the driver-facing camera; calibrating the image of the driver's eye movements with the image from the outside view camera to produce calibrated images; and generating a pixel weighted thermal map of the calibrated images via a first electronic controller, and the method further comprising providing, via a first electronic controller, the pixel weighted thermal map of the calibrated images for a second electronic controller. Procedure according to Claim 10 , wherein the operation, via the one or more electronic controls, of the vehicle using the trained semantic neural segmentation network model and the trained attention-based neural network model is carried out via the second electronic control. Procedure according to Claim 11 , further comprising training, via the second electronic control, a semantic segmentation classifier with a cost function that includes the pixel-weighted heat map. Procedure according to Claim 10 , further comprising receiving multiple vehicles that provide pixel-weighted thermal maps. Procedure according to Claim 9 wherein calibrating the image of the driver's eye movements with the image from the exterior view camera further comprises assigning a weight to each pixel in the image from the exterior view camera based on the image of the driver's eye movements. Procedure according to Claim 9 , wherein the generation of the trained attention-based neural network model includes using the pixel-weighted heat map as a classification identifier and the exterior view image as an input. A system for operating a vehicle using a trained semantic neural segmentation network model and a trained attention-based neural network model, the system comprising: an outside view camera; one or more electronic controls, designed for: receive an image from the outside view camera; process the image from the outside view camera with the trained semantic neural segmentation network model to extract segmentation features; process the image from the exterior view camera with the trained attention-based neural network model to generate a pixel-weighted thermal map; to chain the segmentation features with the pixel-weighted thermal map; process the chained segmentation features and the pixel-weighted heat map with a segmentation classifier; and operate the vehicle according to the processing of the chained segmentation features and the pixel-weighted heat map with the segmentation classifier. System according to Claim 16 , wherein processing the concatenated segmentation features and the pixel-weighted thermal map with the segmentation classifier includes using a modified cost function that incorporates the pixel-weighted thermal map.

Images