Underwater Image Processing: State of the Art of Restoration and Image Enhancement Methods

The underwater image processing area has received considerable attention within the last decades, showing important achievements. In this paper we review some of the most recent methods that have been specifically developed for the underwater environment. These techniques are capable of extending the range of underwater imaging, improving image contrast and resolution. After considering the basic physics of the light propagation in the water medium, we focus on the different algorithms available in the literature. The conditions for which each of them have been originally developed are highlighted as well as the quality assessment methods used to evaluate their performance.

In order to deal with underwater image processing, we have to consider first of all the basic physics of the light propagation in the water medium. Physical properties of the medium cause degradation effects not present in normal images taken in air. Underwater images are essentially characterized by their poor visibility because light is exponentially attenuated as it travels in the water and the scenes result poorly contrasted and hazy. Light attenuation limits the visibility distance at about twenty meters in clear water and five meters or less in turbid water. The light attenuation process is caused by absorption (which removes light energy) and scattering (which changes the direction of light path). The absorption and scattering processes of the light in water influence the overall performance of underwater imaging systems. Forward scattering (randomly deviated light on its way from an object to the camera) generally leads to blurring of the image features. On the other hand, backward scattering (the fraction of the light reflected by the water towards the camera before it actually reaches the objects in the scene) generally limits the contrast of the images, generating a characteristic veil that superimposes itself on the image and hides the scene. Absorption and scattering effects are due not only to the water itself but also to other components such as dissolved organic matter or small observable floating particles. The presence of the floating particles known as "marine snow" (highly variable in kind and concentration) increase absorption and scattering effects. The visibility range can be increased with artificial lighting but these sources not only suffer from the difficulties described before (scattering and absorption), but in addition tend to illuminate the scene in a non uniform fashion, producing a bright spot in the center of the image with a poorly illuminated area surrounding it. Finally, as the amount of light is reduced when we go deeper, colors drop off one by one depending on their wavelengths. The blue color travels the longest in the water due to its shortest wavelength, making the underwater images to be dominated essentially by blue color. In summary, the images we are interested on can suffer of one or more of the following problems: limited range visibility, low contrast, non uniform lighting, blurring, bright artifacts, color diminished (bluish appearance) and noise. Therefore, application of standard computer vision techniques to underwater imaging requires dealing first with these added problems.

The image processing can be addressed from two different points of view: as an image restoration technique or as an image enhancement method:

1. (i)

   The image restoration aims to recover a degraded image using a model of the degradation and of the original image formation; it is essentially an inverse problem. These methods are rigorous but they require many model parameters (like attenuation and diffusion coefficients that characterize the water turbidity) which are only scarcely known in tables and can be extremely variable. Another important parameter required is the depth estimation of a given object in the scene.

2. (ii)

   Image enhancement uses qualitative subjective criteria to produce a more visually pleasing image and they do not rely on any physical model for the image formation. These kinds of approaches are usually simpler and faster than deconvolution methods.

In what follows we give a general view of some of the most recent methods that address the topic of underwater image processing providing an introduction of the problem and enumerating the difficulties found. Our scope is to give the reader, in particular who is not an specialist in the field and who has a specific problem to address and solve, the indications of the available methods focusing on the imaging conditions for which they were developed (lighting conditions, depth, environment where the approach was tested, quality evaluation of the results) and considering the model characteristics and assumptions of the approach itself. In this way we wish to guide the reader so as to find the technique that better suits his problem or application.

In Section 2 we briefly review the optical properties of the light propagation in water and the image formation model of Jaffe-McGlamery, following in Section 3 with a report of the image restoration methods that take into account this image model. In Section 4, works addressing image enhancement and color correction in underwater environment are presented. We include a brief description of some of the most recent methods. When possible, some examples (images before and after correction) that illustrate these approaches are also included. Section 5 considers the lighting problems and Section 6 focuses on image quality metrics. Finally the conclusions are sketched in Section 7.

In this section we focus on the special transmission properties of the light in the water. Light interacts with the water medium through two processes: absorption and scattering. Absorption is the loss of power as light travels in the medium and it depends on the index of refraction of the medium. Scattering refers to any deflection from a straight-line propagation path. In underwater environment, deflections can be due to particles of size comparable to the wavelengths of travelling light (diffraction), or to particulate matter with refraction index different from that of the water (refraction).

According to the Lambert-Beer empirical law, the decay of light intensity is related to the properties of the material (through which the light is travelling) via an exponential dependence. The irradiance E at position r can be modeled as:

(1)

where is the total attenuation coefficient of the medium. This coefficient is a measure of the light loss from the combined effects of scattering and absorption over a unit length of travel in an attenuation medium. Typical attenuation coefficients for deep ocean water, coastal water and bay water are 0.05 , 0.2 , and 0.33 , respectively.

Assuming an isotropic, homogeneous medium, the total attenuation coefficient can be further decomposed as a sum of two quantities and , the absorption and scattering coefficients of the medium, respectively:

(2)

The total scattering coefficient is the superposition of all scattering events at all angles through the volume scattering function (this function gives the probability for a ray of light to be deviated of an angle from its direction of propagation)

(3)

The parameters , , , and represent the inherent properties of the medium and their knowledge should theoretically permit us to predict the propagation of light in the water. However, all these parameters depend on the location (in a three dimensional space) and also on time. Therefore, the corresponding measurements are a complex task and computational modeling is needed.

McGlamery [1] laid out the theoretical foundations of the optical image formation model while Jaffe [2] extended the model and applied it to design different subsea image acquisition systems. Modeling of underwater imaging has also been carried out by Monte Carlo techniques [3].

In this section we follow the image formation model of Jaffe-McGlamery. According to this model, the underwater image can be represented as the linear superposition of three components (see Figure 1). An underwater image experiment consists of tracing the progression of light from a light source to a camera. The light received by the camera is composed by three components: (i) the direct component (light reflected directly by the object that has not been scattered in the water), (ii) the forward-scattered component (light reflected by the object that has been scattered at a small angle) and (iii) the backscatter component (light reflected by objects not on the target scene but that enters the camera, for example due to floating particles). Therefore, the total irradiance reads:

(4)

Spherical spreading and attenuation of the source light beam is assumed in order to model the illumination incident upon the target pane. The reflected illumination is then computed as the product of the incident illumination and the reflectance map. Assuming a Lambertian reflector, geometric optics is used to compute the image of the direct component in the camera plane. The reflected light is also small scattered on its way to the camera. A fraction of the resultant blurred image is then added to the direct component. The backscatter component is the most computationally demanding to calculate. The model partitions 3-dimensional space into planes parallel to the camera plane, and the radiation scattered toward the camera is computed superposing small volume elements weighted by an appropriate volume scattering function. The detail derivation of each of the components of (4) can be found in [2]. We report here the final results, as they appear in Jaffe's article. The direct component results (see Figure 2 for the coordinate system)

(5)

where is the irradiance on the scene surface at point , is the distance from to the camera and the function represents the surface reflectance map. We note that and typical values for objects of oceanographic interest are [4]. The camera system is characterized by (-number of the lens), (lens transmittance) and (focal length). The angle is the angle between the reflectance map and a line between the position and the camera. The forward scatter component is calculated from the direct component via the convolution operator with a point spread function ; and its derivation is valid under the small angle scattering approximation

(6)

where the function g is given by

(7)

with an empirical factor such that and a damping function determined empirically. indicates the inverse Fourier transform and is the radial frequency. Experimental measurements of the point spread function validate the use of the small angle scattering theory [5, 6]. For the calculation of the backscatter component the small angle approximation is no longer valid as the backscattered light enters the camera from a large distribution of angles. The model takes into account the light contributions from the volume of water between the scene and the camera. The three dimensional space is divided into a large number of differential volumes . The backscatter component is a linear superposition of these illuminated volumes of water, weighted by the volume scattering function

(8)

where is the direct component of the backscattered irradiance and is evaluated as

(9)

with the thickness of the backscattering volume , the distance from a point in the camera to the center of the backscatter slab; is the volume scattering function and is the irradiance in the three dimensional space propagating away from the light source.

The three components of underwater optical imaging: direct component (straight line), forward component (dashed line) and backward scatter component (dash-dot line).

Full size image

Coordinate system of the Jaffe-McGlamery model.

Full size image

In Jaffe's work [2, 7] the relationship between image range, camera light separation and the limiting factors in underwater imaging are considered. If only short ranges are desired (one attenuation length), a simple conventional system that uses close positioning of camera and lights can yield good results but these configurations are contrast-limited at greater ranges. If longer distances are desired (2-3 attenuation lengths), systems with separated camera and lights are preferred but backscattering problems appear as the distance increases. For greater distances more sophisticated technology is required, like for example, laser range-gated systems and synchronous scan imaging.

A possible approach to deal with underwater images is to consider the image transmission in water as a linear system [8].

Image restoration aims at recovering the original image from the observed image using (if available) explicit knowledge about the degradation function (also called point spread function PSF) and the noise characteristics :

(10)

where denotes convolution. The degradation function includes the system response from the imaging system itself and the effects of the medium (water in our case). In the frequency domain, we have:

(11)

where are spatial frequencies and ,,, and are Fourier transforms of , , , and , respectively. The system response function in the frequency domain is referred as the optical transfer function (OTF) and its magnitude is referred as modulation transfer function (MTF). Usually, the system response is expressed as a direct product of the optical system itself and the medium:

(12)

The better the knowledge we have about the degradation function, the better are the results of the restoration. However, in practical cases, there is insufficient knowledge about the degradation and it must be estimated and modeled. In our case, the source of degradation in underwater imaging includes turbidity, floating particles and the optical properties of light propagation in water. Therefore, underwater optical properties have to be incorporated into the PSF and MTF. The presence of noise from various sources further complicates these techniques.

Recently, Hou et al. [9–11] incorporated the underwater optical properties to the traditional image restoration approach. They assume that blurring is caused by strong scattering due to water and its constituents which include various sized particles. To address this issue, they incorporated measured in-water optical properties to the point spread function in the spatial domain and the modulation transfer function in frequency domain. The authors modeled for circular symmetrical response systems (2-dimensional space) as an exponential function

(13)

The exponent, , is the decay transfer function obtained by Wells [12] for the seawater within the small angle approximation

(14)

where is the mean square angle, and are the total scattering and attenuation coefficients, respectively. The system (camera/lens) response was measured directly from calibrated imagery at various spatial frequencies. In water optical properties during the experiment were measured: absorption and attenuation coefficients, particle size distributions and volume scattering functions. The authors implemented an automated framework termed Image Restoration via Denoised Deconvolution. To determine the quality of the restored images, an objective quality metric was implemented. It is a wavelet decomposed and denoised perceptual metric constrained by a power spectrum ratio (see Section 6). Image restoration is carried out and medium optical properties are estimated. Both modeled and measured optical properties are taken into account in the framework. The images are restored using PSFs derived from both the modeled and measured optical properties (see Figure 3).

Image taken at 7. 5 m depth in Florida. The original (a), the restored image based on measured MTF (b) and the restored image based on modeled MTF (c). Courtesy of Hou et al. [9].

Full size image

Trucco and Olmos [13] presented a self-tuning restoration filter based on a simplified version of the Jaffe-McGlamery image formation model. Two assumptions are made in order to design the restoration filter. The first one assumes uniform illumination (direct sunlight in shallow waters) and the second one is to consider only the forward component of the image model as the major degradation source, ignoring back scattering and the direct component . This appears reasonable whenever the concentration of particulate matter generating backscatter in the water column is limited. A further simplification considers the difference of exponentials in the forward scatter model (6) as an experimental constant (with typical values between 0.2 and 0.9)

(15)

Within these assumptions, from (7), a simple inverse filter in the frequency domain is designed as follows (the parameter is approximated by )

(16)

Optimal values of these parameters were estimated automatically for each individual image by optimizing a quality criterion based on a global contrast measure (optimality is defined as achieving minimum blur). Therefore, low-backscatter and shallow-water conditions represent the optimal environment for this technique. The authors assessed both qualitative (by visual inspection) and quantitatively the performance of the restoration filter. They assessed quantitatively the benefits of the self-tuning filer as preprocessor for image classification: images were classified as containing or not man-made objects [14, 15]. The quantitative tests with a large number of frames from real videos show an important improvement to the classification task of detecting man-made objects on the seafloor. The training videos were acquired under different environments: instrumented tank, shallow and turbid waters conditions in the sea.

Liu et al. [16] measured the PSF and MTF of seawater in the laboratory by means of the image transmission theory and used Wiener filters to restore the blurred underwater images. The degradation function is measured in a water tank. An experiment is constructed with a slit image and a light source. In a first step, one dimensional light intensity distribution of the slit images at different water path lengths is obtained. The one dimensional PSF of sea water can be obtained by the deconvolution operation. Then, according to the property of the circle symmetry of the PSF of seawater, the 2-dimensional PSF can be calculated by mathematical method. In a similar way, MTFs are derived. These measured functions are used for blurred image restoration. The standard Wiener deconvolution process is applied. The transfer function reads

(17)

where and are the power spectrum of noise and original image, respectively, and is the conjugate matrix of (measured result as previously described). Noise is regarded as white noise, and is a constant that can be estimated form the blurred images with noise while is estimated as

(18)

where is the power spectrum of the blurred image. Then, the spectrum of the restored image is

(19)

Also parametric Wiener filter is used by the authors and both deconvolution methods are compared.

Schechner and Karpel [17] exploit the polarization effects in underwater scattering to compensate for visibility degradation. The authors claim that image blur is not the dominant cause for image contrast degradation and they associate underwater polarization with the prime visibility disturbance that they want to delete (veiling light or backscattered light). The Jaffe-McGlamery image formation model is applied under natural underwater lighting exploiting the fact that veiling light is partially polarized horizontally [18]. The algorithm is based on a couple of images taken through a polarizer at different orientations. Even when the raw images have very low contrast, their slight differences provide the key for visibility improvement. The method automatically accounts for dependencies on object distance, and estimates a distance map of the scene. A quantitative estimate for the visibility improvement is defined as a logarithmic function of the backscatter component. Additionally, an algorithm to compensate for the strong blue hue is also applied. Experiments conducted in the sea show improvements of scene contrast and color correction, nearly doubling the underwater visibility range. In Figure 4 a raw image and its recovered version are shown.

Underwater scene at the Red-Sea at 26 m below the water surface. Left, raw image; right, recovered image. Image courtesy of Schechner and Karpel [17].

Full size image

Recently, Treibitz and Schechner [19] used a similar polarization-based method for visibility enhancement and distance estimation in scattering media. They studied the formation of images under wide field (non-scanning) artificial illumination. Based on backscattered light characteristics (empirically obtained) they presented a visibility recovery approach which also yields a rough estimate of the 3D scene structure. The method is simple and requires compact hardware, using active wide field polarized illumination. Two images of the scene are instantly taken, with different states of a camera-mounted polarizer. The authors used the approach to demonstrate recovery of object signals and significant visibility enhancement in experiments in various sea environments at night. The distance reconstruction is effective in a range of 1-2 m. In Figure 5, an underwater image taken in the Mediterranean sea with two articial light sources is shown together with the corresponding de-scattered image result [19].

Raw image (a), De-scattered image (b) [19]. From http://webee.technion.ac.il/~yoav/research/active-descattering.html.

Full size image

These methods make total abstraction of the image formation process, and no a priori knowledge of the environment is needed (do not use attenuation and scattering coefficients for instance). They are usually simpler and faster than the image restoration techniques.

Regarding color correction, as depth increases, colors drop off one by one depending on their wavelength. First of all, red color disappears at the depth of 3 m approximately. At the depth of 5 m, the orange color is lost. Most of the yellow goes off at the depth of 10 m and finally the green and purple disappear at further depth. The blue color travels the longest in the water due to its shortest wavelength. The underwater images are therefore dominated by blue-green color. Also the light source variations will affect the color perception. As a consequence, a strong and non uniform color cast will characterize the typical underwater images.

Bazeille et al. [20, 21] propose an algorithm to pre-process underwater images. It reduces underwater perturbations and improves image quality. It is composed of several successive independent processing steps which correct non uniform illumination (homorphic filtering), suppress noise (wavelet denoising), enhance edges (anisotropic filtering) and adjust colors (equalizing RGB channels to suppress predominant color). The algorithm is automatic and requires no parameter adjustment. The method was used as a preliminary step of edge detection. The robustness of the method was analyzed using gradient magnitude histograms and also the criterion used by Arnold-Bos et al. [22] was applied. This criterion assumes that a well-contrasted and noise-free image has a distribution of the gradient magnitude histogram close to exponential and it attributes a mark from zero to one. In Figure 6 pairs of images are shown before and after Bazeille et al'. processing [20].

Pairs of images before (a) and after (b) Bazeille et al.' processing. Image courtesy of Bazeille et al. [20].

Full size image

Chambah et al. [23] proposed a color correction method based on ACE model, an unsupervised color equalization algorithm developed by Rizzi et al. [24]. ACE is a perceptual approach inspired by some adaptation mechanisms of the human vision system, in particular lightness constancy and color constancy. ACE was applied on videos taken in aquatic environment that present a strong and non uniform color cast due to the depth of the water and the artificial illumination. Images were taken from the tanks of an aquarium. Inner parameters of the ACE algorithm were properly tuned to meet the requirements of image and histogram shape naturalness and to deal with these kinds of aquatic images. In Figure 7 two example original images and their restored ACE version are shown.

Original images (a), after correction with ACE (b). Image courtesy of Chambah et al. [23].

Full size image

Iqbal et al. [25] presented an underwater image enhancement method using an integrated color model. They proposed an approach based on slide stretching: first, contrast stretching of RGB algorithm is used to equalize the color contrast in the images. Second, saturation and intensity stretching of HSI is applied to increase the true color and solve the problem of lighting. The blue color component in the image is controlled by the saturation and intensity to create the range from pale blue to deep blue. The contrast ratio is therefore controlled by decreasing or increasing its value. In Figure 8 two example images before and after Iqbal et al'. technique are shown.

Original images (a), images after enhancement using Iqbal et al'. technique (b). Image courtesy of Iqbal et al. [25].

Full size image

Arnold Bos et al. [22, 26] presented a complete preprocessing framework for underwater images. They investigated the possibility of addressing the whole range of noises present in underwater images by using a combination of deconvolution and enhancement methods. First, a contrast equalization system is proposed to reject backscattering, attenuation and lighting inequalities. If is the original image and its low-pass version, a contrast-equalized version of is . Contrast equalization is followed by histogram clipping and expansion of the image range. The method is relevant because backscattering is a slowly varying spatial function. Backscattering is considered as the first noise addressed in the algorithm but contrast equalization also corrects the effect of the exponential light attenuation with distance. Remaining noises corresponding to sensor noise, floating particles and miscellaneous quantification errors are suppressed using a generic self-tuning wavelet-based algorithm. The use of the adaptive smoothing filter significantly improves edge detection in the images. Results on simulated and real data are presented.

The color recovery is also analyzed by Torres-Mendez and Dudek [27] but from a different perspective: it is formulated as an energy minimization problem using learned constraints. The idea, on which the approach is based, is that an image can be modeled as a sample function of a stochastic process known as Markov Random Field. The color correction is considered as a task of assigning a color value to each pixel of the input image that best describes its surrounding structure using the training image patches. This model uses multi-scale representations of the color corrected and color depleted (bluish) images to construct a probabilistic algorithm that improves the color of underwater images. Experimental results on a variety of underwater scenes are shown.

Ahlen et al. [28] apply underwater hyperspectral data for color correction purposes. They develop a mathematical stability model which gives a value range for wavelengths that should be used to compute the attenuation coefficient values that are as stable as possible in terms of variation with depth. Their main goal is to monitor coral reefs and marine habitats. Spectrometer measurements of a colored plate at various depths are performed. The hyperspectral data is then color corrected with a formula derived from Beer's law

(20)

where is the pixel intensity in the image for depth and is the corresponding attenuation coefficient calculated from spectral data. In this way, they obtain images as if they were taken at a much shallower depth than in reality. All hyperspectral images are "lifted up" to a depth of 1.8 m, where almost all wavelengths are still present (they have not been absorbed by the water column). The data is finally brought back into the original RGB space.

Another approach to improve color rendition is proposed by Petit et al. [29]. The method is based on light attenuation inversion after processing a color space contraction using quaternions. Applied to the white vector in the RGB space, the attenuation gives a hue vector characterizing the water color

(21)

where , , and are the attenuation coefficients for red, green and blue wavelengths, respectively. Using this reference axis, geometrical transformations into the color space are computed with quaternions. Pixels of water areas of processed images are moved to gray or colors with a low saturation whereas the objects remain fully colored. In this way, objects contrasts result enhanced and bluish aspect of images is removed. Two example images before and after correction by Petit et al'. algorithm are shown in Figure 9.

Original image (a), corrected by Petit et al.' algorithm (b). Image courtesy of Petit et al. [29].

Full size image

In this section we summarize the articles that have been specifically focused on solving lighting problems. Even if this aspect was already taken into account in some of the methods presented in the previous sections, we review here the works that have addressed in particular this kind of problem, proposing different lighting correction strategies.

Garcia et al. [30] analyzed how to solve the lighting problems in underwater imaging and reviewed different techniques. The starting point is the illumination-reflectance model, where the image sensed by the camera is considered as a product of the illumination i(x,y), the reflectance function and a gain factor plus an offset term :

(22)

The multiplicative factor due to light sources and camera sensitivity can be modeled as a smooth function (the offset term is ignored). In order to model the non-uniform illumination, a Gaussian-smoothed version of the image is proposed. The smoothed image is intended to be an estimate of how much the illumination field (and camera sensitivity) affects every pixel. The acquired image is corrected by a point-by-point division by the smoothed image , giving rise to an estimate of ideal image

(23)

where is a normalization constant. Next, the contrast of the resulting image is emphasized, giving rise to an equalized version of .

Some authors compensate for the effects of non-uniform lighting by applying local equalization to the images [31, 32]. The non uniform of lighting demands a special treatment for the different areas of the image, depending on the amount of light they receive. The strategy consists in defining annxn neighborhood, computing the histogram of this area and applying an equalization function but modifying uniquely the central point of the neighborhood [33]. A similar strategy is used in Zuidervel [34].

An alternative model consists of applying homomorphic filtering [30]. This approach assumes that the illumination factor varies smoothly through the field of view; generating low frequencies in the Fourier transform of the image (the offset term is ignored). Taking the logarithm of (22), the multiplicative effect is converted into an additive one

(24)

Taking the Fourier transform of (24) we obtain

(25)

where , , and are the Fourier transforms of ,, and , respectively. Low frequencies can be suppressed by multiplying these components by a high pass homomorphic filter given by

(26)

where is the cutoff frequency, is a multiplicative factor and is an offset term. This filter not only attenuates non uniform illumination but also enhances the high frequencies, sharpening the edges.

Rzhanov et al. [35] disregards the multiplicative constant , considering the lighting of the scene as an additive factor which should be subtracted from the original image

(27)

where is a two dimensional polynomial spline and is a normalization constant.

Garcia et al. [30] tested and compared the different lighting-corrections strategies for two typical underwater situations. The first one considers images acquired in shallow waters at sun down (simulating deep ocean). The vehicle carries its own light producing a bright spot in the center of the image. The second sequence of images was acquired in shallow waters on a sunny day. The evaluation methodology for the comparisons is qualitative. The best results have been obtained by the homomorphic filtering and the point-by-point correction by the smoothed image. The authors emphasize that both methods consider the illumination field is multiplicative and not subtractive.

In the last years many different methods for image quality assessment have been proposed and analyzed with the goal of developing a quality metric that correlates with perceived quality measurements (for a detailed review see [36]). Peak Signal to Noise Ratio and Mean Squared Error are the most widely used objective image quality/distortion metrics. In the last decades however, a great effort has been made to develop new objective image quality methods which incorporate perceptual quality measures by considering human visual system characteristics. Wang et al. [37] propose a Structural Similarity Index that does not treat the image degradation as an error measurement but as a structural distortion measurement.

The objective image quality metrics are classified in three groups: full reference (there exists an original image with which the distorted image is to be compared), no-reference or "blind" quality assessment and reduced-reference quality assessment (the reference image is only partially available, in the form of a set of extracted features).

In the present case of underwater image processing, no original image is available to be compared, and therefore, no-reference metrics are necessary. Within the above cited methods for enhancement and restoration, many of the authors use subjective quality measurements to evaluate the performance of their methods. In what follows we focus on the quantitative metrics used by some of the authors to evaluate the algorithm performance and image quality in the specific case of underwater images.

Besides visual comparison, Hou and Weidemann [38] also propose an objective quality metric for the scattering-blurred typical underwater images. The authors measure the image quality by its sharpness using the gradient or slope of edges. They use wavelet transforms to remove the effect of scattering when locating edges and further apply the transformed results in restraining the perceptual metric. Images are first decomposed by a wavelet transform to remove random and medium noise. Sharpness of the edges is determined by linear regression, obtaining the slope angle between grayscale values of edge pixels versus location. The overall sharpness of the image is the average of measured grayscale angles weighted by the ratio of the power of the high frequency components of the image to the total power of the image (WGSA metric). The metric has been used in their automated image restoration program and the results demonstrate consistency for different optical conditions and attenuation ranges.

Focusing on underwater video processing algorithms, Arredondo and Lebart [39] propose a methodology to quantitative assess the robustness and behavior of algorithms in face of underwater noises. The principle is to degrade test images with simulated underwater perturbations and the focus is to isolate and assess independently the effects of the different perturbations. These perturbations are simulated with varying degrees of severity. Jaffe and McGlamery' model is used to simulate blur and unequal illumination. Different levels of blurring are simulated using the forward-scattered component of images taken at different distances from the scene: in (6) is increased varying from to meters to the scene at intervals . The non-uniform lighting is simulated placing the camera at distances between and meters, at intervals of . In order to isolate the effect of non-uniform lighting, only the direct component is taken into account. The lack of contrast is simulated by histogram manipulation. As a specific application, different optical flow algorithms for underwater conditions are compared. A well known ground-truth synthetic sequence is used for the experiments. The true motion of the sequence is known and it is possible to measure quantitatively the effect of the degradations on the optical flow estimates. In [39] different methods available are compared. The angular deviation between the estimated velocity and the correct one is measured. An attenuation coefficient typical of deep ocean is used. It is shown that the angular error increases linearly with the Gaussian noise for all the methods compared.

In order to assess the quality of their adaptive smoothing method for underwater image denoising, Arnold-Bos et al. [26] proposed a simple criterion based on a general result by Pratt [40]: for most well contrasted and noise free images, the distribution of the gradient magnitude histogram is closely exponential, except for a small peak at low gradients corresponding to homogeneous zones. They define a robustness index between 0 and 1 (it is linked to the variance of the linear regression of the gradient magnitude histogram) that measures the closeness of the histogram with an exponential distribution. The same index was also used by Bazeille et al. [20] to evaluate the performance of their algorithm.

In Table 1 we summarize the articles above reviewed indicating the model assumptions and imaging conditions for which they have been developed and tested as well as the image quality assessment method used to evaluate the corresponding results.

To make a quantitative comparison of the above cited methods, judging which of them gives the best/worst results is beyond the scope of this article. In fact, in order to do such a quantitative comparison of results, a common data base should be available in order to test the corresponding algorithms according to specific criteria. To our knowledge, no such underwater database exist at present and therefore, to build this database could be one of the future research lines from which the underwater community would certainly beneficiate. However, we have pointed out how each of the algorithms has been evaluated by the own authors: subjectively (by visual inspection) or objectively (by the implementation of an objective image quality measure). The majority of the algorithms here reviewed have been evaluated using subjective visual inspection of their results.

The difficulty associated with obtaining visibility of objects at long or short distance in underwater scenes presents a challenge to the image processing community. Even if numerous approaches for image enhancement are available, they are mainly limited to ordinary images and few approaches have been specifically developed for underwater images. In this article we have reviewed some of them with the intention of bringing the information together for a better comprehension and comparison of the methods. We have summarized the available methods for image restoration and image enhancement, focusing on the conditions for which each of the algorithms has been originally developed. We have also analyzed the methodology used to evaluate the algorithms' performance, highlighting the works where a quantitative quality metric has been used.

As pointed by our analysis, to boost underwater imaging processing, a common suitable database of test images for different imaging conditions together with standard criteria for qualitative and/or quantitative assessment of the results is still required.

Nowadays, leading advancements in optical imaging technology [41, 42] and the use of sophisticated sensing techniques is rapidly increasing the ability to image objects in the sea. Emerging underwater imaging techniques and technologies make it necessary to adapt and extend the above cited methods to, for example, handle data from multiple sources that can extract 3-dimensional scene information. On the other hand, studying the vision system of underwater animals (their physical optics, photoreceptors and neurophysiological mechanisms) will certainly give us new insights to the information processing of underwater images.

The authors acknowledge Gianluigi Ciocca for a critical reading of the manuscript and the reviewers for their critical observations. We also gratefully acknowledge F. Petit, W. Hou, K. Iqbal, Y. Schechner, A. Rizzi, M. Chambah and S. Bazeille who kindly made their figures available upon our request.

Authors and Affiliations

1. Department of Informatics, Systems and Communication (DISCo), University of Milano-Bicocca, Viale Sarca 336, 20126, Milan, Italy

   Raimondo Schettini & Silvia Corchs

Corresponding author

Correspondence to Silvia Corchs.

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions