Nothing Special   »   [go: up one dir, main page]
Skip to main content
EURASIP Journal on Advances in Signal Processing
Download PDF

Fusion of hyperspectral and panchromatic images using multiresolution analysis and nonlinear PCA band reduction

EURASIP Journal on Advances in Signal Processing volume 2012, Article number: 207 (2012) Cite this article

Abstract

This article presents a novel method for the enhancement of the spatial quality of hyperspectral (HS) images through the use of a high resolution panchromatic (PAN) image. Due to the high number of bands, the application of a pan-sharpening technique to HS images may result in an increase of the computational load and complexity. Thus a dimensionality reduction preprocess, compressing the original number of measurements into a lower dimensional space, becomes mandatory. To solve this problem, we propose a pan-sharpening technique combining both dimensionality reduction and fusion, making use of non-linear principal component analysis (NLPCA) and Indusion, respectively, to enhance the spatial resolution of a HS image. We have tested the proposed algorithm on HS images obtained from CHRIS-Proba sensor and PAN image obtained from World view 2 and demonstrated that a reduction using NLPCA does not result in any significant degradation in the pan-sharpening results.

Introduction

Generally, for satellite images, the highest spatial resolution is captured by the panchromatic (PAN) image. However the drawback of the PAN image is that it has no spectral information beyond that which is averaged within the bandpass of the PAN image. Unlike a PAN image, multispectral (MS) and in particular hyperspectral (HS) satellite images cover a wider spectral range with moderate to high resolution. However, as compared to MS images, HS images have a better spectral resolution, that may result in a very high number of bands having low spatial resolution. For better utilization and interpretation, HS images having both high spectral and spatial resolution are desired. This can be achieved by making use of a high spatial resolution PAN image along with low resolution HS images in the context of pan-sharpening.

Pan-sharpening, or image fusion, is the process of improving the spatial quality of a low spatial resolution image (HS or MS) by fusing it with a high resolution PAN image[1, 2]. One of the main challenges in image fusion is to improve the spatial resolution i.e. spatial details, while preserving the original spectral information. This requires addition of spatial details to each band of the image. Due to the high number of bands the pan-sharpening of HS images results in an increased computational load and complexity. Thus a dimensionality reduction preprocess, compressing the original number of measurements into a lower dimensional space, becomes mandatory[3]. Among image fusion methods, the most popular are those based on the substitution approach, such as intensity-hue-saturation transformation (IHS) and principal component analysis (PCA)[4]. Among these, PCA approaches are commonly applied to HS images. PCA approaches are based on the assumption that the first principal component PC collects the information that is common to all the bands. The fusion is achieved by substituting the first PC band with the PAN image, whose histogram has previously been matched with that of PC band. In this way the spatial information is encapsulated in the histogram matched PAN image, while the spectral information that is specific to each spectral band is contained in the other principal components. In this case the dimensionality reduction is performed discarding the less relevant principal components. This means that the image resulting from the inverse transformation will not have the same information of the original one, resulting in a strong spectral distortion. In this article, we propose a new approach to enhance the spatial resolution of a HS image combining both non-linear principal component analysis (NLPCA) and Indusion for dimensionality reduction and fusion, respectively. In particular NLPCA is applied to reproject the original data into a lower space, thus the derived nonlinear principal components are then enhanced according to the Indusion process. Finally, the inverse NLPCA reprojects the enhanced components back to the original dimensionality, resulting in a spatially enhanced HS image having a similar spectral characteristic of the original one.

The article is organized as follows. In Sections “Dimensionality reduction” and “Image fusion” the NLPCA and Indusion are described, respectively. Section “Experimental results” presents experimental results obtained by applying the proposed approach to three different datasets, while conclusions are drawn in Section “Conclusion”.

Dimensionality reduction

The main difficulty in processing HS images is that the number of bands can vary from several tens to several hundreds. Applying a pan-sharpening technique to each band of the HS image, can lead to a dramatic increase of the computational time of the entire process. Hence, while enhancing the spatial resolution of a HS image, it is generally desirable to reduce the number of bands. Another important property is that the reduction method should allow a complete reconstruction of the original spectral information content. Consequently, the dimensionality reduction should seek to avoid loosing relevant spectral information from the original dataset.

In the literature, many methods have been developed to tackle the issue of high dimensionality of HS data[5]. Summarizing, we may say that dimensionality reduction methods can be grouped into two classes: “feature-selection” algorithms (which suitably select a sub-optimal subset of the original set of features while discarding the remaining ones) and “feature extraction” by data transformation (which projects the original feature space onto a lower dimensional subspace that preserves most of the information)[6].

Feature selection techniques can be generally considered as a combination of both a search algorithm and a criterion function[7–11]. The solution to the feature selection problem is offered by the search algorithm, which generates a subset of features and compares them on the basis of the criterion function. On the other hand, feature extraction techniques seek to reduce the dimensionality of the data by mapping the feature space onto a new lower-dimensional space. While feature selection is a more simple and direct approach, feature extraction methods can be more effective in representing the information content in a lower dimensionality domain. Nevertheless the loss of information derived from a feature selection approach does not allow a good reconstruction of the original dataset. For this reason it is not recommended to integrate feature selection methods into pan-sharpening processes.

The most common techniques to reduce the number of bands are the minimum noise fraction (MNF) transform, where an operator calculates a set of transformed features according to a signal-to-noise ratio optimization criterion, PCA, where a set of uncorrelated transformed features is generated and also independent component analysis (ICA), where a computational method for separating a multivariate signal into additive subcomponents supposing the mutual statistical independence of the non-Gaussian source signals[12, 13]. For these techniques, the dimensionality reduction is performed by discarding the components with the lowest information content. Also, for these techniques, the components obtained are linearly uncorrelated but the physical representation of the image may be lost. Moreover, being linear methods, ICA, PCA and MNF assume that the observed data set is composed of linear combinations of certain basis. In[14, 15], it has been demonstrated that the nonlinear version of the common PCA, namely kernel PCA (KPCA) is capable of capturing part of the high order statistics, thus providing more information from the original data set than the PCA. In this case, the dimensionality reduction is once again performed by discarding the less relevant components. Other approaches, are based upon the characteristic of HS images having adjacent bands that are spectrally highly correlated[7, 8].

In this article, we propose to perform the dimensionality reduction by using NLPCA, commonly referred to as nonlinear generalization of standard PCA.

Non-linear principal component analysis, originally introduced by Kramer[16], is based on multi-layer perceptrons (MLP) commonly referred to as autoassociative neural network (AANN) or as autoencoder[17, 18]. The AANNs are NNs of a conventional type, i.e. featuring feedforward connections and sigmoidal nodal transfer functions, trained by backpropagation or similar algorithms. The particular network architecture used employs three hidden layers, including an internal bottleneck layer of smaller dimension than either the input or the output (Figure1). The network is trained to perform identity mapping, where the input x has to be equal to the output x ̂ . Training an AANN is not an easy task because of the bottleneck layer where the data have to be projected or compressed into a lower dimensional space Z. Since there are fewer units in the bottleneck layer than the output, the bottleneck nodes must represent or encode the information obtained from the inputs for the subsequent layers to reconstruct the input.

Figure 1
figure 1

Auto-associative neural networks scheme used for feature reduction.

Full size image

The AANN, as shown in Figure2, can be divided into two parts: the first part represents the encoding or extraction function Fencode:X→Z, while the second part represents the inverse function, called decoding function Fdecode:Z→X ̂. After the training of the AANN the NLPCs can be extracted from the extraction subnet, while the reconstruction can be performed by the decoding subnet. A topology with three hidden layers enables the AANN to perform non-linear mapping functions. In fact, if we design an AANN with only one hidden layer of linear nodes, the projection into the Z-dimensional subspace would correspond exactly to linear PCA. Also if the activation functions in the bottleneck nodes were sigmoidal, the projection into the sub-space would still be severely constrained; only linear combinations of the inputs compressed by the sigmoid into the range [−1,1] could be represented. Therefore the performance of an AANN with only one internal layer of sigmoidal nodes is often no better than linear PCA [81]. The proposed AANN algorithms can be trained by minimizing the sum-of-squares error of the form:

E= 1 2 ∑ n = 1 N ∑ k = 1 d { y k ( x n ) − x k n } 2
(1)
Figure 2
figure 2

The two subnets performing the encode (left) and the decode (right) functions.

Full size image

where y k (k=1,2,…,d) is the output vector. The non-linear activation function σ(x) can be any continuous and monotonically increasing function with σ(x)→1 as x→ + ∞ and u(x)→0 as x→−∞. In this article the chosen function is the sigmoid and is applied elementwise:

σ(x)= 1 1 + e − x
(2)

One of the main tasks designing the AANN is the selection of the number of nodes in the hidden layers and in particular in the bottleneck layer that minimizes the loss of information of the entire network. This problem has been solved by a grid search algorithm that varies recursively the number of nodes and evaluates the respective error. The topology with the lowest error is then selected. A general evaluation of the performance of this method, in terms of time and memory, cannot be addressed because it is mainly dependent on the number of nodes of the network considered in each iteration. However usually each iteration of the grid search required an average time that is less than a minute, resulting in a computational time that was less than 10 min for each of the experiments described in the following sections.

Compared to linear reduction techniques, NLPCA has many advantages. First of all, while PCA and MNF can detect and discard linear correlations among spectral bands, NLPCA detects both linear and nonlinear correlations. In this case the dimensionality reduction is performed by discarding the less relevant components, meaning that the reconstruction of the original image would be extremely affected in terms of loss of information. Moreover the first PCA usually retains information from all the original spectral bands, meaning that it may include features within the spectral bandpass of the PAN image and will thus result in strong spectral distortions. On the other hand, the information content in the NLPCs is somehow focused on relevant spectral signatures. For instance, in a HS image acquired on a landscape with water, vegetation and manmade structures, one of the derived NLPCs will contain all the information related to the vegetation, while another one will have only information about water surfaces. This allows NLPCA to be significantly more effective than PCA or MNF in the inverse operation of reconstruction of the original spectral information[19]. Results similar to NLPCA can be obtained through the use of KPCA, however, as for the linear approaches, also in this case the dimensionality reduction is performed by discarding less relevant components. This let the NLPCA to be preferred for the dimensionality reduction processing[15].

In this article we propose the use of NLPCA for dimensionality reduction. The NLPCs obtained from the extraction function will be used as input for the pan-sharpening task. After enhancing the NLPCs, the enhanced HS image is eventually obtained with the decoding subnet of the AANN.

Image fusion

The fusion of HS and PAN images is a useful technique for enhancing the spatial quality of low-resolution HS images. Generally, the fusion process can be subdivided into two steps. In the first step, the low resolution image is scaled up to the same size as the PAN image. Next, fusion is achieved by adding high-frequency content of the PAN image to the HS image. Literature on pan-sharpening methods is rich with diversity, encompassing methods based upon the use of discrete wavelet transform (DWT)[20, 21], Laplacian Pyramids[22], PCA transform[4], and IHS transform[23]. The latter two methods fall in the category of component substitution method and result in fused images having high spatial quality but suffering from spectral distortions[24, 25]. The images fused using DWT or Laplacian Pyramid based methods are not as sharp as component substitution methods but they are spectrally consistent[24].

In this article, we propose to use Indusion for pan-sharpening. Introduced in[26], Indusion is a pan-sharpening approach derived from the Induction scaling technique. The Induction technique, considers enlargement as the inverse problem of reduction. This yields the condition that an enlarged image should, when reduced, give the initial image back. This condition is called the reduction constraint. Assuming I as the original image and R the reduction filter, with a reduction ratio a so that the upscaled image is I1/a, the reduction constraint can be written as:

I 1 / a ∗ R ↓a=I.
(3)

For a given image and a reduction filter, there is a set of enlarged images, called the induced set, that verifies the reduction constraint, and is defined as:

Ω I ={X∣[X∗R]↓a=I}.
(4)

Induction, initially developed for image magnification[27], simply consists of projecting an upscaled image J, not adhering to the reduction constraint, onto the induced set Ω so as to obtain an induced image K that belongs to the induced set. Indusion process, deriving its name from Induction and Fusion, defines the induced image as:

K=J−[J∗R]↓a↑a∗A+[I]↑a∗A.
(5)

Where R and A are the Cohen–Daubechies–Fauveau (CDF) 9/7 tap bi-orthogonal filter pair, respectively[28], and J is the upscaled version of the initial image I that does not adhere to the reduction constraint.

Applied to remote sensing, the concept of image fusion consists in the extraction of the high-frequency information from the PAN image and adding it to the upscaled low-resolution image. The idea of Indusion is to replace the unconstrained upscaled image by the PAN image since, we want the high-frequency information of the PAN image to be added to the upscaled NLPCA image obtained from the original HS dataset.

K=PAN−[[PAN∗R]↓a]↑a∗A+[I]↑a∗A.
(6)

The Indusion algorithm has been successfully tested on true and simulated images[26]. The main intent of this article is to evaluate the effectiveness of the Indusion approach combined with a NLPCA dimensionality reduction applied to HS images. In particular the original HS image is reduced in dimensionality by means of NLPCA, resulting in few nonlinear principal components. Then according to the above described Indusion method the high resolution spatial details contained in the PAN image are injected into the nonlinear components. Finally the spatially enhanced components are reprojected back to the original space using the inverse NLPCA. The result is a spatially enhanced HS image having the same spectral characteristics as the original one.

The implementation of the proposed algorithm is as follows.

  1. 1.

    Design/training of the AANN (selection of the best topology);

  2. 2.

    Extraction of the NLPCs through the encoding function;

  3. 3.

    Downscaling of the PAN image using CDF9 filter to fit the size of the NLPCs;

  4. 4.

    Perform histogram matching between the downscaled PAN and the NLPCs;

  5. 5.

    Upscale the NLPCs and the histogram matched PAN using CDF7 filter;

  6. 6.

    Perform histogram matching between the original PAN and the upscaled NLPCs;

  7. 7.

    Obtain the difference between histogram matched original PAN and the histogram matched upscaled PAN;

  8. 8.

    Add the previously obtained difference to the upscaled NLPCs;

  9. 9.

    Reconstruction of the original spectral bands through the decoding function.

A complete scheme of the method is depicted in Figure3.

Figure 3
figure 3

Complete flowchart of the fusion process.

Full size image

Experimental results

In this article the proposed method was applied to three different images having increasing complexity. In Section “WorldView-2 dataset”, we discuss the results of applying the Indusion approach to a WorldView-2 dataset to assess the accuracy of the fusion method. In Section “CHRIS-Proba dataset”, the NLPCA + Indusion approach is discussed in the context of the fusion of a CHRIS-Proba dataset and a QuickBird PAN image. The last section evaluates the reduction-fusion algorithm on a Hyperion image. While the enlargement of a MS image such as provided by WorldView-2 satellite, can be an easy task, pan-sharpening of HS imagery can be more complex not only because of the huge number of HS bands, but also because the spectral coverage of the PAN image does not match the wavelength acquisition range of the HS bands. Moreover a difference between the WorldView-2 dataset and the CHRIS-proba and Hyperion datasets is that for the latter two datasets the PAN image has been acquired by a different satellite sensor i.e. QuickBird, having different acquisition dates and geometry. A test of the spectral fidelity of the fusion process is introduced using PAN images not covering the spectral range of the HS bands, and this is verified by pan-sharpening CHRIS-proba and Hyperion images using a PAN image from the QuickBird sensor.

Once the pansharpened images have been obtained, the next phase is their quality assessment. Evaluating the quality of a fusion process is not a trivial task. For the quantitative quality assessment, it is generally recommended to make use of the synthesis property as proposed by Wald[1]. This means that both the HS and the PAN images are degraded to a lower resolution and then pan-sharpening is performed. This is done so that the resultant pansharpened image is at the same resolution as the starting reference and hence statistical analysis can be made between the reference and the pansharpened images. If the reference image is at the same resolution as the fused image, we can perform universal image quality index (UIQI), relative dimensional global error (ERGAS) and spectral angle mapper (SAM) calculations between the pansharpened and the reference image[1, 29]. UIQI can be seen as a ratio between the original image and the enhanced one, where 1 is the ideal value. ERGAS and SAM will both produce positive values with an ideal value of 0. However, values that are around 3 are referred to a good image enhancement. SAM is a useful measure of the spectral quality introduced by the fusion process, while ERGAS and UIQI measure both spectral and spatial quality. On the other hand, if we reduce our images to a lower resolution there will be no significant information left in the images and hence the pansharpened images would not be at a good resolution. For this reason a qualitative analysis, through visual inspection, will also be discussed without reducing the spatial resolution of the images before the fusion process.

WorldView-2 dataset

WorldView-2, launched October 2009, is a high-resolution 8-band MS satellite. WorldView-2 provides PAN and MS images at 50 cm and 2 m spatial resolution, respectively. The spectral coverages are 450–800 nm for the PAN images and 400–1,040 nm for the MS image. The image used in this experiment was collected over Tor Vergata area, in the south-east part of Rome, Italy, on February 2010. The landscape represented is diverse, with large pastures, industrial and dense urban areas.

In this first experiment, we evaluate the quality of the fusion method without any dimensionality reduction. This experiment aims to validate the ability of the Indusion technique for image enhancement and also to provide a benchmark for the following experiments. Following the processing steps proposed in[26], the PAN image (Figure4) is downscaled from 0.5 to 1 m, and then from 1 to 2 m using CDF9 filter coefficient. In the same way, the reduced MS (Figure5) is upscaled from 2 to 1 m and again from 1 to 0.5 m pixel size with a CDF7 filter. Then the histogram of the PAN downscaled at 2 m is matched to the MS at 2 m, resulting in eight histogram matched PAN images at 2 m resolution. The same procedure is applied between the PAN at 0.5 m and the MS at 0.5 m, producing eight histogram matched PAN images at 0.5 m resolution. The set of eight histogram matched PAN images at 2 m is then upscaled back to 1 m and then to 0.5 m using CDF7 filter. Finally the difference between the two series of eight histogram matched PAN images is added to the upscaled MS image.

Figure 4
figure 4

WorldView-2 PAN band.

Full size image
Figure 5
figure 5

False color composite obtained combining NIR1, RED and GREEN bands of the original WorldView-2 image.

Full size image

On a visual analysis, the fused images obtained from the Indusion approach appear to be very sharp, as can be observed comparing Figures5 and6. As for the quantitative analysis, the indices UIQI, ERGAS and SAM were calculated and reported in Table1. The results appear to be very appreciable with values close to the optimum.

Figure 6
figure 6

False color composite obtained combining NIR1, RED and GREEN bands of the fused WorldView-2 image.

Full size image
Table 1 Quality indexes for the fusion process applied on WorldView-2
Full size table

CHRIS-Proba dataset

In the second test a CHRIS-Proba dataset and a QuickBird PAN image at 1m have been used for testing the proposed fusion technique. The two datasets were acquired during different periods of 2006 over the Tor Vergata area, south-east of Rome, Italy.

The PRoject for On Board Autonomy (PROBA) spacecraft is a mini-satellite operated by the European Space Agency. The main payload of PROBA is the Compact High Resolution Imaging s Spectrometer (CHRIS) sensor which can acquire up to 63 spectral bands (400–1,050 nm) with nominal spatial resolutions of 17 or 34 m at nadir, depending on the acquisition mode. The CHRIS sensor acquires data in five different modes (aerosol, land cover, vegetation and coastal zones), varying the number and location of spectral bands, the spatial resolution and the width of the swath. Thus, the CHRIS-PROBA system acquires HS images of the same scene with five different view angles during a single overpass: +55°, +36°, 0°, -36° and -55°. To test the proposed approach a 0° image, acquired in Land-Cover mode (18 bands), was used. The spectral ranges of the two images are very similar, 438–1,035 nm for CHRIS and 405–1,053 nm for the QuickBird-PAN, respectively. Also the angles of view of the two images are very similar, thus avoiding any geometric or registration distortions between the two images. The CHRIS image was atmospherically corrected and accurately co-registered to the PAN image for testing. In particular, the atmospheric correction consists of estimating the contribution of the atmospheric effects in terms of the total irradiance, the direct and diffuse transmittance, and the radiance due to the scattering. The estimations have been produced starting from simulations carried out with the suite of libraries libRadtran for the simulation of the radiative transfer balance[30]. A subset of 216×216 pixels for the CHRIS image and 864×864 for the PAN image was selected according to the overlapping areas of the two acquisitions. To obtain an enlargement ratio of 4, that, respect to the description of the Indusion method, offers the best tradeoff between spatial enhancement and spectral distortion[26], the CHRIS image and the PAN image have been degraded to the spatial resolutions of 20 and 5 m, respectively adhering to the consistency criteria of quality assessment as proposed by Wald[1].

The dimensionality reduction phase was performed by extracting the nonlinear principal components. To detect the best topology configuration for the AANN, a grid-search approach is performed. The best solution was found with 18 inputs/output, 9 nodes both in the outer hidden layers and 3 nodes in the bottleneck. The mean square error (MSE) obtained after 500 training cycles was 0.003. The three NLPCs are then enhanced with a ratio of 4, following the Indusion approach, fusing them with the PAN image. Figures7,8 and9 report a false color representation of the original image, the panchromatic image and of the fused image, respectively. Figures10 and11 show a RGB composite obtained combining the three NLPC from the original CHRIS-Proba image and the result of the Indusion process applied to the three NLPCs, respectively. The pansharpened image shown in Figure9 is obtained by performing an inverse NLPCA process on the results shown in Figure11.

Figure 7
figure 7

False color composite of the original CHRIS-Proba image obtained combining the 709nm 697nm and 668nm bands for R, G and B, respectively.

Full size image
Figure 8
figure 8

PAN image used in the fusion process with the CHRIS-Proba image.

Full size image
Figure 9
figure 9

False color composite of the fusion process obtained combining the 709nm 697nm and 668nm bands for R, G and B, respectively.

Full size image
Figure 10
figure 10

RGB composite obtained using the three original NLPCs obtained from the CHRIS-Proba data.

Full size image
Figure 11
figure 11

RGB composite obtained using the three fused NLPCs.

Full size image

Also in this experiment, the qualitative analysis was evaluated by visual inspection. While to better understand the effectiveness of the proposed method, the quantitative analysis is carried out in three different ways. Firstly the quality indexes are calculated to evaluate the ability of AANN to reconstruct the original spectral bands after the reduction into three NLPCs. This means that the evaluation is performed between the original spectral bands and the bands reconstructed considering the complete AANN topology. A second measurement is performed to evaluate the accuracy of the fusion phase, comparing the original NLPCs with the enhanced ones. Finally the quality indexes are calculated on the complete end-to-end process. As reported in Table2, the quality indexes are derived not only for the entire scene but also for the different areas in the image representing three main types (industrial buildings, dense residential areas and pastures). For the sake of comparison, Table3 reports the quality index values for a dimensionality reduction performed using the first three components of the PCA. The table clearly shows that a comparison of pansharpened images with the reference HS image reveals that all three SAM, ERGAS and UIQI values for the NLPCA and Indusion method are better than PCA based and Indusion method pan-sharpening methods.

Table 2 Quality indexes for the proposed fusion process applied on the entire CHRIS-Proba image (complete image) and on three different land cover types (pasture, industrial and dense urban fabric)
Full size table
Table 3 Quality indexes values for the reconstruction of the CHRIS-Proba image obtained by PCA and NLPCA, respectively
Full size table

From a qualitative visual analysis, the fused image obtained with the proposed approach appears to be very sharp and spectrally consistent with the original image. Also the quantitative analysis shows very good values for the pan-sharpening image. It is important to notice that, despite having an overall good quality, the index values for the Pasture and Industrial areas are very good, while the worst result was obtained on the dense urban fabric. This effect can be explained because of the different geometry of acquisition of the two images. In particular, the very high reflectance values of industrial buildings, under different illumination conditions, may result in spectral distortions in the enlarged image. Another source of distortion came from objects that are present only in one of the two images due to different dates of acquisition. This is particularly clear for the burned areas and the cars in the parking lots, as depicted on Figure12. Aside from the expected distortions introduced by errors in the registration phase and differences in terms of angles of view, there are also other negative contributions introduced by objects that are detected by the PAN image but not in the HS image and result in mixed spectral signatures. In this case, even if the spectral signatures of small objects can be identified in the low resolution HS image, it is still not possible to determine their spatial distribution. A possible solution to this problem has been proposed in[31].

Figure 12
figure 12

Detail of the false color composites of the original image and the enhanced one, obtained combining the 709nm 697nm and 668nm bands for R, G and B, respectively.

Full size image

A further quality assessment of the proposed approach evaluates the spectral distortion introduced by the entire process. In particular, a comparison of the enhanced spectra with the original spectra has been made. In this case the comparison was set up on three different areas representing, respectively an industrial building, a residential building and a grassland area, as reported on Figure13. Analyzing the original and the enhanced spectra related to the grassland pixel, it can be noted that the process produced a spectrum very close to the original one. This accuracy can be quantified by the value of the root mean square error (RMSE), the mean difference and the standard deviation computed between the values of the original and the enhanced spectra reported in Table4. A similar result is obtained with the industrial building pixel. As for the residential building pixel, the comparison of the original and enhanced spectra shows a non perfect reconstruction (Figure14). These spectral distortions can be explained by the presence in the PAN image of features that are not detected by the HS image. In particular, the spectral signature of the grassland is almost the same over the entire field, independently from the spatial resolution of the sensor, while for the buildings, the signature is more heterogeneous. In this case the PAN image does not introduce further information content and the spectral signature of the grassland, on the other hand many features detected only by the PAN image influenced the enhanced spectrum. Also, the spectral signature of the industrial building can be considered almost uniform over the entire surface, non generating any significant distortions. On the other hand, during the fusion process, the high resolution image introduced a great amount of spatial information that produced spectral distortions that are particularly evident on buildings pixels, as reported in Table4.

Figure 13
figure 13

Quality assessment areas noted in yellow, green and cyan for pasture, industrial and dense urban fabric, respectively.

Full size image
Figure 14
figure 14

Spectra of three different test areas: Grassland original spectrum (a) and enhanced spectrum (b), Industrial building original spectrum (c) and enhanced spectrum (d), residential building spectrum (e) and enhanced spectrum (f).

Full size image
Table 4 RMSE, mean difference and Standard deviation computed between the values of original and enhanced spectra for three different test areas (Grassland: 20 samples; Industrial: 8 samples; Residential: 12 samples)
Full size table

Hyperion dataset

The last experiment shows the results of the proposed technique applied to a Hyperion dataset acquired in 2002. Hyperion is a grating imaging spectrometer providing 220 HS bands (from 0.4 to 2.5 μ m) with a 30 m spatial resolution. Each image covers a 7.5 km by 100 km land area and provides detailed spectral mapping across all 220 channels with high radiometric accuracy. The test area was selected over the Rome city centre, with a landscape characterized mainly by dense urban areas and sparse vegetation. Before the extraction of the NLPCs, noisy bands not containing relevant information were discarded from the original dataset, resulting in 168 spectrally unique and good quality bands[32]. In this experiment, two different PAN images were used to evaluate the spectral distortion introduced by these images. In a first attempt, we fused the Hyperion image with a QuickBird PAN image.

The ratio between the spatial resolution of the two images has to be 4. Consequently the QuickBird image was downscaled to the 7.5 m spatial resolution. As for the previous experiment, the dimensionality reduction phase is performed by a grid search of the best AANN topology. The MSE is found to be 0.012, and is obtained with the following topology: 168 input/output nodes, 3 nodes in the bottleneck layer and 75 in the other hidden layers, resulting in 3 NLPCs. Following the Indusion approach the three NLPCs are fused with the PAN image and the three fused NLPCs are then reprojected back into the original 168 spectral bands. The PAN image, original Hyperion image, the three NLPCs before and after the fusion phase and the enlarged Hyperion image resulting from the entire process are featured on In Figures15,16,17,18 and19.

Figure 15
figure 15

Quickbird PAN image used in the fusion process with the Hyperion image.

Full size image
Figure 16
figure 16

False color composite of the original Hyperion image obtained combining the 925, 823 and 721 nm bands for R, G and B respectively.

Full size image
Figure 17
figure 17

False color composite of the fusion process obtained combining the 925, 823 and 721 nm bands of Hyperion for R, G and B, respectively.

Full size image
Figure 18
figure 18

RGB composite obtained using the three original NLPCs obtained from the Hyperion data.

Full size image
Figure 19
figure 19

RGB composite obtained using the three fused NLPCs derived from Hyperion image.

Full size image

From a visual analysis, the resulting image appears to be sharper than the original and it can also be noted that the road network, that was not visible in the original image, is now clearly defined. Analyzing the values of quality indexes in Table5, it can be noted that the dimensionality reduction phase, performed through the NLPCA, does not produced any relevant distortion. On the other hand, evaluation of the quality of the enhanced NLPCs obtained by the Indusion process returned unacceptable values. These results are generated mainly by the different acquisition angles and the consequent imperfect registration between HS and PAN images. An enhancement of the overall quality of the enlarged image can be obtained using a different PAN image.

Table 5 Quality indexes for the fusion process applied on Hyperion
Full size table

The best case scenario requires a PAN image acquired by the same satellite at the same moment. For this reason we choose a PAN image acquired by ALI instrument in 2002, onboard the EO1 satellite (Figure20). The spectral coverage of the ALI-PAN image covers the visible range (0.48–0.69 μ m), with a spatial resolution of 10m. Since this PAN image was acquired on a different date than the HS image, there are isolated clouds that are not present in the HS image. This does not allow to select the same reference area. Only a subset of the area is selected as depicted in Figure21. Moreover, to respect the algorithm requirement to have a ratio of 4, we have degraded the HS image resolution to 40 m.

Figure 20
figure 20

ALI PAN image.

Full size image
Figure 21
figure 21

False color composite of the original Hyperion image obtained combining the 518, 620 and 721 nm f bands or R, G and B, respectively.

Full size image

The dimensionality reduction was performed using the same AANN of the previous case, while the fusion of the three NLPCs with the new PAN image was obtained with the Indusion approach. From the Table5 it can be noted that the use of a new PAN image, acquired by the same satellite, provides better values for the quality indexes, demonstrating the effectiveness of the proposed approach. Also from a visual analysis, the pansharpened image obtained with ALI-PAN (Figure22) seems to be sharper than when a QuickBird-PAN image was used for pan-sharpening. As for the dimensionality reduction, it can be noted that the quality of the transformed image is very similar to the previous one. In Figure 23a-f is also reported a comparison of the enhanced spectra with the original spectra. Also in this case the comparison was set up on three different areas representing respectively an residential building, a industrial building and a vegetation area. Apart from distortions introduced by features detected by the PAN image and not present in the low resolution one, it is possible to note also some spectral distortions in the vegetation area. The selected vegetation area is mainly composed of sparse deciduous trees, so the spectral distortion can be explained by the different seasonal growth cycle of the trees spotted by the two sensors.

Figure 22
figure 22

False color composite of the fusion process obtained combining the 518, 620 and 721 nm bands for R, G and B, respectively.

Full size image
Figure 23
figure 23

Spectra of three different test areas: residential building spectrum (a) and enhanced spectrum (b), industrial building original spectrum (c) and enhanced spectrum (d) and vegetation original spectrum (e) and enhanced spectrum (f).

Full size image

Conclusion

In this, we have presented a novel approach combining dimensionality reduction and a pan-sharpening technique for spatial quality improvement of HS images while preserving the spectral quality of the original HS image. The proposed method introduces dimensionality reduction of HS images based on the nonlinear generalization of standard PCA. The nonlinear principal components were obtained by an AANN. The use of the Indusion technique has been investigated in the framework of pan-sharpening. The innovation proposed by this technique relies in fusing NLPCs with the PAN images instead of the spectral bands to reduce the computational load of the pan-sharpening process. In the experimental section the Indusion was first tested on a WorldView-2 image to assess the performance of the fusion algorithm, while Indusion combined with NLPCA was tested on a two HS images, a CHRIS-Proba and a Hyperion dataset, respectively. These two latter experiments were carried out fusing NLPCs with PAN images collected under different geometries and for images acquired at different dates. Moreover in both cases the PAN image does not cover the same spectral range as covered by the original HS bands. A further experiment has been made fusing a Hyperion image with a PAN image acquired by the same satellite and having the same geometry as the HS image. Being the proposed method applied to real data, it is important to consider that many sources of spectral distortion are introduced, such as error in the registration phase and differences in terms of angles of view. Moreover, it has to be considered that there are also other negative contributions introduced by objects that are detected by the PAN image but not in the HS image where their spectral signature results to be mixed with the signatures of the surrounding objects. Finally there are also some objects in one image that are not present in the other due to the different dates of acquisition. However, apart from these negative contributions, the results demonstrated a good behavior of the proposed method in mitigating the spectral distortions. Aside from the benefits of using Indusion for image fusion, the use of NLPCA for dimensionality reduction results in a better reconstruction of the original HS image. However, one main drawback on the use of this technique relies on the necessity to perform a grid search to find the neural network topology that may lead to the best performances. In any case, the UIQI and ERGAS quality indexes quantitatively demonstrate the good performance of the proposed method on CHRIS-Proba and a Hyperion images. Visually, Indusion produced sharp and spectrally consistent images while NLPCA reduced the original dataset dimensionality minimizing the introduction of further spectral distortions while speeding up the pan-sharpening process.

References

  1. Wald L: Data Fusion. Definitions and Architectures—Fusion of Images of Different Spatial Resolutions. Presses de l’Ecole, Ecole des Mines de Paris, Paris, France,; 2002.

    Google Scholar 

  2. Palsson F, Sveinsson JR, Benediktsson J, Aanaes H: Classification of Pansharpened Urban Satellite Images. IEEE J. Sel. Topics Appl. Earth Observations and Remote Sens 2012, 5: 281-297.

    Article  Google Scholar 

  3. Eismann MT, Hardie RC: Hyperspectral resolution enhancement using high resolution multispectral imagery with arbitrary response function. IEEE Trans. Geosci. Remote Sens 2005., 43(3):

    Google Scholar 

  4. Chavez PS, Sides SC, Anderson JA: Comparison of three different methods to merge multiresolution and multispectral data: landsat TM and SPOT panchromatic. Photogram. Eng. Remote Sens. 1991, 57(3):295-303.

    Google Scholar 

  5. Serpico B, D’Inca M, Melgani F, Moser G: Comparison of feature reduction techniques for classification of hyperspectral remote sensing data universal image quality index. Proc. SPIE Image Signal Process Remote Sens 2003, 4885(8):347-358.

    Google Scholar 

  6. Lee C, Landgrebe DA: Feature extraction based on decision boundaries. IEEE Trans. Pattern Anal. Mach. Intell 1993, 15(4):388-400. 10.1109/34.206958

    Article  Google Scholar 

  7. Mitra P, Murthy CA, Pal SK: Unsupervised feature selection using feature similarity. IEEE Trans. Pattern Anal. Mach. Intell 2002, 24(3):301-312. 10.1109/34.990133

    Article  Google Scholar 

  8. Plaza A, Martinez P, Plaza J, Perez R: Dimensionality reduction and classification of hyperspectral image data using sequences of extended morphological transformations. IEEE Trans. Geosci. Remote Sens 2005, 43(3):466-479.

    Article  Google Scholar 

  9. He Y, Du Q, Chen G: Unsupervised Hyperspectral Band Selection Using Graphics Processing Units. IEEE J. Selected Topics in Appl. Earth Observations and Remote Sens 2011, 4: 660-668.

    Article  Google Scholar 

  10. Sen J, Zhen J, Yuntao Q, Linlin S: Unsupervised Band Selection for Hyperspectral Imagery Classification Without Manual Band Removal. IEEE J. Selected Topics Appl. Earth Observations and Remote Sens 2012, 5: 531-543.

    Article  Google Scholar 

  11. Chein-I C, Su W, Keng-Hao L, Mann-Li C, Chinsu L: Progressive Band Dimensionality Expansion and Reduction Via Band Prioritization for Hyperspectral Imagery. IEEE J. Selected Topics Appl. Earth Observations and Remote Sens 2011, 4: 591-614.

    Article  Google Scholar 

  12. Jutten C, Herault J: Blind separation of sources, part I: an adaptive algorithm based on neuromimetic architecture. Signal Process 1991, 24: 1-10. 10.1016/0165-1684(91)90079-X

    Article  MATH  Google Scholar 

  13. Green AA, Berman M, Switze RP, Craig MD: A transformation for ordering multispectral data in terms of image quality with implications for noise removal. IEEE Trans. Geosci. Remote Sens 1988, 26: 65-74. 10.1109/36.3001

    Article  Google Scholar 

  14. Fauvel M, Chanussot J, Benediktsson JA: Kernel principal component analysis for the classification of hyperspectral remote sensing data over urban areas. EURASIP J. Adv. Signal Process 2009, 2009: 1-14.

    Article  Google Scholar 

  15. Licciardi G, Marpu PR, Chanussot J, Benediktsson JA: Linear versus nonlinear PCA for the classification of hyperspectral data based on the extended morphological profiles. IEEE Geosci. Remote Sens. Lett 2012, 9: 447-451.

    Article  Google Scholar 

  16. Kramer MA: Nonlinear principal component analysis using autoassociative neural networks. AIChE J 1991, 37: 233-243. 10.1002/aic.690370209

    Article  Google Scholar 

  17. Bishop C: Neural Networks for Pattern Recognition. Oxford University Press, London; 1995.

    MATH  Google Scholar 

  18. Scholz M, Kaplan F, Guy L, Kopka J, Selbig J: Non-linear PCA: a missing data approach. Bioinformatics 2005, 21: 3887-3895. 10.1093/bioinformatics/bti634

    Article  Google Scholar 

  19. Licciardi G, Frate FD: Pixel unmixing in hyperspectral data by means of neural networks. IEEE Trans. Geosci. Remote Sens 2011, 49: 4163-4172.

    Article  Google Scholar 

  20. Nunez J, Otazu X, Fors O, Prades A, Pala V, Arbiol R: Multiresolution-based image fusion with additive wavelet decomposition. Trans. Geosci. Remote Sens 1999, 37(3):1204-1211. 10.1109/36.763274

    Article  Google Scholar 

  21. Ranchin T, Wald L: Fusion of high spatial and spectral resolution images: the ARSIS concept and its implementation. Photogramm. Eng. Remote Sens 2000, 66: 49-61.

    Google Scholar 

  22. Aiazzi B, Alparone L, Baronti S, Garzelli A: Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis. IEEE Trans. Geosci. Remote Sens 2002, 40: 2300-2312. 10.1109/TGRS.2002.803623

    Article  Google Scholar 

  23. Tu TM, Huang PS, Hung CL, Chang CP: A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery. IEEE Geosci. Remote Sens. Lett 2004, 1(4):309-312. 10.1109/LGRS.2004.834804

    Article  Google Scholar 

  24. Alparone L, Wald L, Chanussot J, Thomas C, Gamba P, Bruce LM: Comparison of pansharpening algorithms: Outcome of the 2006 GRS-S data fusion contest. IEEE Trans. Geosci. Remote Sens 2007, 45(10):3012-3021.

    Article  Google Scholar 

  25. Thomas C, Ranchin T, Wald L, Chanussot J: Synthesis of Multispectral Images to High Spatial Resolution: A Critical Review of Fusion Methods Based on Remote Sensing Physics. IEEE Trans Geoscience and Remote Sens 2008, 46: 1301-1312.

    Article  Google Scholar 

  26. Khan MM, Chanussot J, Condat L, Montanvert A: Indusion: fusion of multispectral and panchromatic images using the induction scaling technique. IEEE Geosci. Remote Sens. Lett 2008, 5: 98-102.

    Article  Google Scholar 

  27. Condat L, Montanvert A: A framework for image magnification: induction revisited. ICASSP 2005. pp. 845–848

    Google Scholar 

  28. Cohen A, Daubechies I, Feauveau JC: Biorthogonal bases of compactly supported wavelets. Commun. Pure Appl. Math 1992, 45(5):485-560. 10.1002/cpa.3160450502

    Article  MathSciNet  MATH  Google Scholar 

  29. Wang Z, Boviki AC: A universal image quality index. IEEE Signal Process. Lett 2002, 9(3):81-84.

    Article  Google Scholar 

  30. Mayer B, Kylling A: Technical note: the libRadtran software package for radiative transfer calculations—description and example of use. Atmos. Chem. Phys 2005, 5: 1855-1877. 10.5194/acp-5-1855-2005

    Article  Google Scholar 

  31. Villa A, Chanussot J, Benediktsson J, Jutten C: Spectral unmixing for the classification of hyperspectral images at a finer spatial resolution. IEEE J. Sel. Top. Signal Process 2011, 5(3):521-533.

    Article  Google Scholar 

  32. Datt B, McVicar TR, Van Niel TG, Jupp DLB, Pearlman JS: Preprocessing EO-1 hyperion hyperspectral data to support the application of agricultural indexes. IEEE Trans. Geosci. Remote Sens 2003, 41(6):1246-1259. 10.1109/TGRS.2003.813206

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. GIPSA-lab, CNRS, University of Grenoble, Grenoble, France

    Giorgio Antonino Licciardi, Jocelyn Chanussot, Annick Montanvert, Laurent Condat & Christian Jutten

  2. SEECS, National University of Sciences and Technology, Islamabad, Pakistan

    Muhammad Murtaza Khan

Authors
  1. Giorgio Antonino Licciardi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Muhammad Murtaza Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jocelyn Chanussot
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Annick Montanvert
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Laurent Condat
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Christian Jutten
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Giorgio Antonino Licciardi.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2

Authors’ original file for figure 3

Authors’ original file for figure 4

Authors’ original file for figure 5

Authors’ original file for figure 6

Authors’ original file for figure 7

Authors’ original file for figure 8

Authors’ original file for figure 9

Authors’ original file for figure 10

Authors’ original file for figure 11

Authors’ original file for figure 12

Authors’ original file for figure 13

Authors’ original file for figure 14

Authors’ original file for figure 15

Authors’ original file for figure 16

Authors’ original file for figure 17

Authors’ original file for figure 18

Authors’ original file for figure 19

Authors’ original file for figure 20

Authors’ original file for figure 21

Authors’ original file for figure 22

Authors’ original file for figure 23

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Licciardi, G.A., Khan, M.M., Chanussot, J. et al. Fusion of hyperspectral and panchromatic images using multiresolution analysis and nonlinear PCA band reduction. EURASIP J. Adv. Signal Process. 2012, 207 (2012). https://doi.org/10.1186/1687-6180-2012-207

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/1687-6180-2012-207

Keywords