, Chapter 1
Mix-Unmix Pan-Sharpener –
Novel Pan-Sharpening Method
Based on Mixing Constituent
Multispectral Bands and Unmixing
Panchromatic Band
Thomas Ngigi, Eunice Nduati, Wei Xianhu and Marlena Götza
Abstract
A panchromatic band (Pan-band) spectrally covers a number of the other bands
(multispectral-bands, MS). The Pan-band is of higher spatial resolution than the MS.
The respective advantages of the two are combined through pan-sharpening with the
resultant image adopting the higher spatial resolution of the Pan-band and the colour
information of the MS. Various techniques have evolved but most of them cannot
pan-sharpen more than three MS, and none of them can pan-sharpen more than three
MS at a go, nor pan-sharpen a multispectral image not geographically covered by the
Pan-band. This novel concept overcomes the first problem. The sequel to this chapter
will address the second problem through reverse pan-sharpening. The concept argues
that for a given pixel in the Pan-band, the strata of digital numbers (DNs) in the MS
combine to give rise to a panchromatic-DN. The concept estimates respective
coefficients of strata of DNs in the encompassed bands corresponding to pure blocks
of pixels in the Pan-band. On the basis of the coefficients, encompassed bands’ DN
contributions to the panchromatic-DN are computed from the Pan-band DN. The
resultant DN contributions are regressed on the MS-DNs and one of the encompassed
MS pan-sharpened on the basis of its model. The other multi-spectral bands are pan-
sharpened through it.
Keywords: pan-sharpening, image fusion, mix-unmix classifier, geographical
inadequacy of panchromatic band, multi-spectral imagery, hyper-spectral imagery
1. Introduction
All Earth observation satellites are multi-band. In most cases, there is a band
that collectively covers some of the other bands. The band covering the collective
spectral ranges covered by individual bands is called the panchromatic band. The
1
,Digital Image Processing – Latest Advances and Applications
other bands are collectively referred to as multi-spectral bands. The panchromatic
band always has a higher spatial resolution than the multispectral bands it encom-
passes. For instance, for Geoeye-1, the panchromatic band covers the spectral range
450 nm–800 nm encompassing the blue (450 nm–510 nm), green (510 nm–580 nm),
red (655 nm–690 nm), and partially near-infra red (780 nm–920 nm) bands. The
spatial resolution of the panchromatic band is 0.41 m against 1.65 m for the multi-
spectral bands.
A panchromatic band does not provide colour information as it cannot be viewed
in combination with the other bands due to differing spatial resolution. The encom-
passed multispectral bands when viewed in combination, made possible by a com-
mon spatial resolution, provide colour information in the RGB colour space through
colour combination. However, due to the higher spatial resolution, the panchromatic
band provides greater spatial detail than the multispectral bands. For a variety of
applications, an image with both high spatial and spectral resolutions would be
desirable. The higher spatial detail of the panchromatic band and colour information
of the multispectral bands can be combined through the process of image fusion by
pan-sharpening [1], defined pan-sharpening as fusion of a multi-spectral image and
panchromatic data aimed at generating an outcome with the same spatial resolution
of the panchromatic data and the spectral resolution of the multispectral image [2],
defined it as merging the spatial and spectral information of the source images into a
fused one, which has a higher spatial and spectral resolution and is more reliable for
downstream tasks compared with any of the source images.
Pan-sharpening may also be across multispectral imagery [3, 4], discussed fusion
of hyperspectral and multispectral imagery, noting that: the latter has detailed spectral
information for fine classification and identification of land-cover but the narrow
bandwidths limit absorption of reflected surface energy; the former has limited repre-
sentation of spectral information making it difficult to characterise the spectral features
but the higher spatial resolution allows for more complete description of the morphol-
ogy and distribution of land-cover. Fusion of the two provides high spatial resolution
and detailed spectral information for value-addition to remotely sensed data.
There have been numerous studies on pan-sharpening and development of
pan-sharpening algorithms with the aim of preserving spectral information while
increasing the spatial resolution of low-resolution multispectral data. However, there
seems to be no universal categorisation criteria of pan-sharpening techniques. For
instance [1], categorises the techniques into (i) Component substitution, (ii) Multi-
resolution analysis, and (iii) third generation techniques e.g. variational optimization
and machine learning techniques; [2] has (i) Component substitution, (ii) Multi-
resolution analysis, (iii) Degradation model, (iv) Deep neural network; [5] has (i)
Component substitution-, (ii) Relative spectral contribution-, (iii) High-frequency
injection-, (iv) Methods based on the statistics of the image-, (v) Multiresolution-
family. Despite the disharmony, two classical techniques are common to all reviewers:
component substitution and multi-resolution analysis. [6–12] give further detailed
discussions on pan-sharpening techniques.
2. Classical pan-sharpening techniques
A need arises to harmonise the criteria for categorisation of pan-sharpening
techniques. This chapter cannot cover all the uttered categories of pan-sharpening
techniques.
2
, Mix-Unmix Pan-Sharpener – Novel Pan-Sharpening Method Based...
ITexLi.1003721
2.1 Component substitution techniques
Component Substitution (CS) techniques include among others, The Brovey
transform, Intensity-Hue-Saturation (IHS) and Principal Component Analysis (PCA)
methods [13]. As a requirement for all the aforementioned methods, the images must be
co-registered and resampled, the quality of which impacts on the results of pan-sharp-
ening [13, 14]. The IHS method is the most popular of the three and involves conversion
of a three-band image from the RGB colour space to the IHS colour space [15]. However,
the technique has been shown to produce fused images with spectral and colour distor-
tions [15, 16]. Modifications to the IHS method have included extension from three
bands to four bands with the inclusion of near-infrared and the incorporation of weight-
ing coefficients on the green and blue bands to minimise the difference between I and
the panchromatic band [17]. Recent studies and developments have led to the develop-
ment of other CS-based techniques and algorithms including the Fast Spectral Response
Function (FSRF), the GIHS with Genetic Algorithm (GIHS-GA), the GIHS with
Trade-off Parameter (GIHS-TP), University of New Brunswick (UNB)-Pan-sharpen
and the Weighted Sum Image Sharpening (WSIS) techniques [16]. The aforementioned
methods were evaluated using a uniform data set in the 2006 GRS-S Data-Fusion contest
and were generally found to have various shortcomings including poor colour synthesis,
missing colours and features, and blurred contours and shapes [16].
2.2 Multiresolution analysis techniques (MRA)
MRA methods, also referred to as wavelet-based fusion methods, include but are
not limited to Generalised Laplacian Pyramid (GLP), Discrete Wavelet Transform
(DWT), Contour-let transform, and Additive Wavelets Luminance Proportional
(AWLP) techniques [6, 18]. These techniques employ the basic principle of extraction
of spatial detail information from the high-resolution panchromatic image, which is
not present in the low-resolution multispectral images and injecting it into the latter
[6]. These methods preserve spectral characteristics of the multispectral image better
than CS methods but tend to have less spatial information [15, 18].
Thus, depending on the application, it seems the trade-off is between the pres-
ervation of spectral information with limited increase in spatial resolution and
distorted spectral information with increasing spatial resolution. Hybrid techniques
aim to overcome the limitations of the CS- and MRA-based techniques, to preserve
the spectral and spatial information content of the input images [16].
3. Materials and methods
This chapter introduces a novel pan-sharpening technique, the mix-unmix
pan-sharpener, based on the Mix-unmix Classifier [19, 20]. The classifier was devel-
oped to overcome the problem of under-determinacy in linear spectral unmixing.
Optical panchromatic and multispectral imagery are utilised in demonstration of the
Mix-unmix Pan-sharpener.
3.1 Theory of the mix-unmix pan-sharpener
For a given sensor, a panchromatic band covers a number of multi-spectral bands.
3
The former has a higher spatial resolution than the latter. One multispectral pixel
Mix-Unmix Pan-Sharpener –
Novel Pan-Sharpening Method
Based on Mixing Constituent
Multispectral Bands and Unmixing
Panchromatic Band
Thomas Ngigi, Eunice Nduati, Wei Xianhu and Marlena Götza
Abstract
A panchromatic band (Pan-band) spectrally covers a number of the other bands
(multispectral-bands, MS). The Pan-band is of higher spatial resolution than the MS.
The respective advantages of the two are combined through pan-sharpening with the
resultant image adopting the higher spatial resolution of the Pan-band and the colour
information of the MS. Various techniques have evolved but most of them cannot
pan-sharpen more than three MS, and none of them can pan-sharpen more than three
MS at a go, nor pan-sharpen a multispectral image not geographically covered by the
Pan-band. This novel concept overcomes the first problem. The sequel to this chapter
will address the second problem through reverse pan-sharpening. The concept argues
that for a given pixel in the Pan-band, the strata of digital numbers (DNs) in the MS
combine to give rise to a panchromatic-DN. The concept estimates respective
coefficients of strata of DNs in the encompassed bands corresponding to pure blocks
of pixels in the Pan-band. On the basis of the coefficients, encompassed bands’ DN
contributions to the panchromatic-DN are computed from the Pan-band DN. The
resultant DN contributions are regressed on the MS-DNs and one of the encompassed
MS pan-sharpened on the basis of its model. The other multi-spectral bands are pan-
sharpened through it.
Keywords: pan-sharpening, image fusion, mix-unmix classifier, geographical
inadequacy of panchromatic band, multi-spectral imagery, hyper-spectral imagery
1. Introduction
All Earth observation satellites are multi-band. In most cases, there is a band
that collectively covers some of the other bands. The band covering the collective
spectral ranges covered by individual bands is called the panchromatic band. The
1
,Digital Image Processing – Latest Advances and Applications
other bands are collectively referred to as multi-spectral bands. The panchromatic
band always has a higher spatial resolution than the multispectral bands it encom-
passes. For instance, for Geoeye-1, the panchromatic band covers the spectral range
450 nm–800 nm encompassing the blue (450 nm–510 nm), green (510 nm–580 nm),
red (655 nm–690 nm), and partially near-infra red (780 nm–920 nm) bands. The
spatial resolution of the panchromatic band is 0.41 m against 1.65 m for the multi-
spectral bands.
A panchromatic band does not provide colour information as it cannot be viewed
in combination with the other bands due to differing spatial resolution. The encom-
passed multispectral bands when viewed in combination, made possible by a com-
mon spatial resolution, provide colour information in the RGB colour space through
colour combination. However, due to the higher spatial resolution, the panchromatic
band provides greater spatial detail than the multispectral bands. For a variety of
applications, an image with both high spatial and spectral resolutions would be
desirable. The higher spatial detail of the panchromatic band and colour information
of the multispectral bands can be combined through the process of image fusion by
pan-sharpening [1], defined pan-sharpening as fusion of a multi-spectral image and
panchromatic data aimed at generating an outcome with the same spatial resolution
of the panchromatic data and the spectral resolution of the multispectral image [2],
defined it as merging the spatial and spectral information of the source images into a
fused one, which has a higher spatial and spectral resolution and is more reliable for
downstream tasks compared with any of the source images.
Pan-sharpening may also be across multispectral imagery [3, 4], discussed fusion
of hyperspectral and multispectral imagery, noting that: the latter has detailed spectral
information for fine classification and identification of land-cover but the narrow
bandwidths limit absorption of reflected surface energy; the former has limited repre-
sentation of spectral information making it difficult to characterise the spectral features
but the higher spatial resolution allows for more complete description of the morphol-
ogy and distribution of land-cover. Fusion of the two provides high spatial resolution
and detailed spectral information for value-addition to remotely sensed data.
There have been numerous studies on pan-sharpening and development of
pan-sharpening algorithms with the aim of preserving spectral information while
increasing the spatial resolution of low-resolution multispectral data. However, there
seems to be no universal categorisation criteria of pan-sharpening techniques. For
instance [1], categorises the techniques into (i) Component substitution, (ii) Multi-
resolution analysis, and (iii) third generation techniques e.g. variational optimization
and machine learning techniques; [2] has (i) Component substitution, (ii) Multi-
resolution analysis, (iii) Degradation model, (iv) Deep neural network; [5] has (i)
Component substitution-, (ii) Relative spectral contribution-, (iii) High-frequency
injection-, (iv) Methods based on the statistics of the image-, (v) Multiresolution-
family. Despite the disharmony, two classical techniques are common to all reviewers:
component substitution and multi-resolution analysis. [6–12] give further detailed
discussions on pan-sharpening techniques.
2. Classical pan-sharpening techniques
A need arises to harmonise the criteria for categorisation of pan-sharpening
techniques. This chapter cannot cover all the uttered categories of pan-sharpening
techniques.
2
, Mix-Unmix Pan-Sharpener – Novel Pan-Sharpening Method Based...
ITexLi.1003721
2.1 Component substitution techniques
Component Substitution (CS) techniques include among others, The Brovey
transform, Intensity-Hue-Saturation (IHS) and Principal Component Analysis (PCA)
methods [13]. As a requirement for all the aforementioned methods, the images must be
co-registered and resampled, the quality of which impacts on the results of pan-sharp-
ening [13, 14]. The IHS method is the most popular of the three and involves conversion
of a three-band image from the RGB colour space to the IHS colour space [15]. However,
the technique has been shown to produce fused images with spectral and colour distor-
tions [15, 16]. Modifications to the IHS method have included extension from three
bands to four bands with the inclusion of near-infrared and the incorporation of weight-
ing coefficients on the green and blue bands to minimise the difference between I and
the panchromatic band [17]. Recent studies and developments have led to the develop-
ment of other CS-based techniques and algorithms including the Fast Spectral Response
Function (FSRF), the GIHS with Genetic Algorithm (GIHS-GA), the GIHS with
Trade-off Parameter (GIHS-TP), University of New Brunswick (UNB)-Pan-sharpen
and the Weighted Sum Image Sharpening (WSIS) techniques [16]. The aforementioned
methods were evaluated using a uniform data set in the 2006 GRS-S Data-Fusion contest
and were generally found to have various shortcomings including poor colour synthesis,
missing colours and features, and blurred contours and shapes [16].
2.2 Multiresolution analysis techniques (MRA)
MRA methods, also referred to as wavelet-based fusion methods, include but are
not limited to Generalised Laplacian Pyramid (GLP), Discrete Wavelet Transform
(DWT), Contour-let transform, and Additive Wavelets Luminance Proportional
(AWLP) techniques [6, 18]. These techniques employ the basic principle of extraction
of spatial detail information from the high-resolution panchromatic image, which is
not present in the low-resolution multispectral images and injecting it into the latter
[6]. These methods preserve spectral characteristics of the multispectral image better
than CS methods but tend to have less spatial information [15, 18].
Thus, depending on the application, it seems the trade-off is between the pres-
ervation of spectral information with limited increase in spatial resolution and
distorted spectral information with increasing spatial resolution. Hybrid techniques
aim to overcome the limitations of the CS- and MRA-based techniques, to preserve
the spectral and spatial information content of the input images [16].
3. Materials and methods
This chapter introduces a novel pan-sharpening technique, the mix-unmix
pan-sharpener, based on the Mix-unmix Classifier [19, 20]. The classifier was devel-
oped to overcome the problem of under-determinacy in linear spectral unmixing.
Optical panchromatic and multispectral imagery are utilised in demonstration of the
Mix-unmix Pan-sharpener.
3.1 Theory of the mix-unmix pan-sharpener
For a given sensor, a panchromatic band covers a number of multi-spectral bands.
3
The former has a higher spatial resolution than the latter. One multispectral pixel
