Open AccessArticle

Surface ALbedo VALidation (SALVAL) Platform: Towards CEOS LPV Validation Stage 4—Application to Three Global Albedo Climate Data Records

Jorge Sánchez-Zapero

^1,*

Enrique Martínez-Sánchez

¹,

Fernando Camacho

¹,

Zhuosen Wang

^2,3,

Dominique Carrer

⁴,

Crystal Schaaf

⁵,

Francisco Javier García-Haro

⁶

Jaime Nickeson

^7,8 and

Michael Cosh

⁹

Earth Observation Laboratory (EOLAB), C/ Savina 8–A4, 46980 Paterna, València, Spain

Earth System Science Interdisciplinary Center, University of Maryland College Park, College Park, MD 20742, USA

Terrestrial Information Systems Laboratory, NASA Goddard Space Flight Center, Greenbelt, MD 20771, USA

⁴

CNRM––Centre National de Recherches Météorologiques (CNRM), Université de Toulouse, Météo France, CNRS, 42 Avenue Gaspard Coriolis, 31057 Toulouse, France

⁵

School for the Environment, University of Massachusetts Boston, Boston, MA 02125, USA

⁶

Earth Physics and Thermodynamics Department, Faculty of Physics, Universitat de València, Dr. Moliner, 46100 Burjassot, València, Spain

⁷

Science Systems and Applications, Inc., 10210 Greenbelt Rd., Lanham, MD 20706, USA

⁸

NASA Goddard Space Flight Center (GSFC), Biospheric Sciences Laboratory, 8800 Greenbelt Rd., Greenbelt, MD 20771, USA

⁹

USDA Agricultural Research Service, Hydrology and Remote Sensing Laboratory, 10300 Baltimore Ave., Beltsville, MD 20705, USA

Author to whom correspondence should be addressed.

Remote Sens. 2023, 15(4), 1081; https://doi.org/10.3390/rs15041081

Submission received: 11 January 2023 / Revised: 9 February 2023 / Accepted: 11 February 2023 / Published: 16 February 2023

(This article belongs to the Section Biogeosciences Remote Sensing)

Download

Browse Figures

Graphical abstract
"> Figure 1
Global distribution of 99 REALS and 720 LANDVAL (including 19 desert calibration) sites used for product intercomparison and direct validation, respectively. LADNVAL are displayed as per biome type: EBF stands for evergreen broadleaved forest, DBF for deciduous broadleaved forest, NLF for needle-leaf forests, OF for other forests, CUL for cultivated, HER for herbaceous, SHR for shrublands, and SBA for sparse and bare areas. "> Figure 2
Example of variogram fitting and ST estimation over two different sites, Desert Rock (DRAK) and Talladega National Forest (TALL). "> Figure 3
Evolution of Number of sites (top left), Number of samples (top right), and RMSD (bottom) of the comparison of MCD43A3 C6.1 versus REALS sites as a function of the ST score in the 2000–2020 period. "> Figure 4
Stability (left) and Accuracy (right) requirement levels as a function of SA values. "> Figure 5
Global maps of the percentage of missing values for C3S V2 (top), GLASS V4 (center), and MCD43A3 C6.1 (bottom) for all pixels (left), and only best quality pixels (right) in the 2003 year, evaluated over the 720 LANDVAL sites. "> Figure 6
Temporal evolution of the percentage of LANDVAL missing data in the 2003 for C3S V2 (red), GLASS V4 (green), and MCD43A3 C6.1 (blue) considering all pixels (top), and only best quality pixels (bottom) according to quality flags. "> Figure 7
Global spatial distribution of average residuals (left) and number of residuals that reach the requirements (right) for C3S V2 vs. GLASS V4 (top), C3SV2 vs. MCD43A3 C6.1 (middle), and GLASS V4 vs. MCD43A3 C6.1 (bottom). Evaluation in the 2001–2005 period with a 10-day temporal frequency over LANDVAL sites using best quality total shortwave black-sky albedo (AL-DH-BB) retrievals. "> Figure 8
Examples of albedo temporal variations in C3S V2 (red), GLASS V4 (green), and MCD43A3 C6.1 (blue) satellite products (all quality pixels not just high quality) and ground data (black dots) from 4 REALS sites during the 2001–2005 period. "> Figure 9
Examples of albedo temporal variations in C3S V2 (red), GLASS V4 (green), and MCD43A3 C6.1 (blue) satellite products (all quality pixels not just high quality) and ground data (black dots) from 4 REALS sites during the 2014–2019 period. "> Figure 10
Histogram of the smoothness (δ) for C3S V2 (red), GLASS V4 (green), and MCD43A3 C6.1 (blue) total shortwave black-sky albedo products. Evaluation in the 2001–2005 period over LANDVAL sites considering all pixels. "> Figure 11
Scatterplots (X-Axis: retrieval for a given date, Y-axis: retrieval for equivalent date of the following year) of the inter-annual precision of C3S V2 (top left), GLASS V4 (top right), and MCD43A3 C6.1 (bottom) products. Evaluation for total shortwave black-sky retrievals over LANDVAL sites in the 2001–2005 period considering all pixels. "> Figure 12
Scatterplots of C3S V2 vs. GLASS V4 (top), C3S V2 vs. MCD43A3 C6.1 (middle), and GLASS V4 vs. MCD43A3 C6.1 (bottom) for all pixels (left) and best quality pixels (right) for black-sky total shortwave retrievals in the 2001–2005 period over LANDVAL sites. "> Figure 13
Examples of temporal profiles of C3S V2 (red), GLASS V4 (green), and MCD43A3 C6.1 (blue) for black-sky total shortwave albedos over calibration sites of LANDVAL in the 2001–2010 period for best quality pixels. Dashed lines represent the linear regression of each product trend. Mean slope value corresponds to the mean slope considering all calibration sites. "> Figure 14
Box plots of the slope per decade (2001–2010) for C3S V2 (red), GLASS V4 (green) and MCD43A3 C6.1 (blue) for black-sky albedos where each box stretches from the 25th percentile to the 75th percentile of the data and whiskers include 99.3% of the coverage data (±2.7 σ). Ouliers are represented by rhombus. Red lines/crosses represent median/mean values. Computation over desert calibration sites. "> Figure 15
Direct validation of C3S V2 (top), GLASS V4 (middle), and MCD43A3 C6.1 (bottom) blue-sky albedo satellite products vs. REALS ground values during the 2000–2019 period for all pixels (left) and only best quality pixels (right). Green, blue, red, and orange points represent forest, crop, shrublands/herbaceous, and desert biome types, respectively. "> Figure A1
Snapshot of SALVAL configuration step 1. “*” stands for mandatory fields. "> Figure A2
Snapshot of SALVAL configuration step 2. "> Figure A3
Snapshot of SALVAL configuration step 3. "> Figure A4
Snapshot of SALVAL configuration step 4. ">

Versions Notes

Abstract

The Surface ALbedo VALidation (SALVAL) online platform is designed to allow producers of satellite-based albedo products to move to operational validation systems. The SALVAL tool integrates long-term satellite products, global in situ datasets, and community-agreed-upon validation protocols into an online and interactive platform. The SALVAL tool, available on the ESA Cal/Val portal, was developed by EOLAB under the framework outlined by the Committee on Earth Observation Satellites (CEOS) Working Group on Calibration and Validation (WGCV) Land Product Validation (LPV) subgroup, and provides transparency, consistency, and traceability to the validation process. In this demonstration, three satellite-based albedo climate data records from different operational services were validated and intercompared using the SALVAL platform: (1) the Climate Change Service (C3S) multi-sensor product, (2) the NASA MODIS MCD43A3 product (C6.1) and (3) Beijing Normal University’s Global LAnd Surface Satellites (GLASS) version 4 products. This work demonstrates that the three satellite albedo datasets enable long-term reliable and consistent retrievals at the global scale, with some discrepancies between them associated with the retrieval processing chain. The three satellite albedo products show similar uncertainties (RMSD = 0.03) when comparing the best quality retrievals with ground measurements. The SALVAL platform has proven to be a useful tool to validate and intercompare albedo datasets, allowing them to reach stage 4 of the CEOS LPV validation hierarchy.

Keywords:

surface albedo; validation; MODIS; MCD43; C3S; SPOT/VGT; PROBA-V; GLASS; CEOS LPV

Graphical Abstract

1. Introduction

Land Surface Albedo (SA), defined as the ratio of the radiant flux reflected from the Earth’s land surface to the incident flux on it, is a parameter of critical importance in understanding both climate and vegetation dynamics [1], and plays a significant role in quantifying the surface energy balance and parameterizing global and regional climate models [2]. SA, established as an Essential Climate Variable (ECV) by the Global Climate Observing System (GCOS) [3], connects the land surface and climate system through the regulation of the shortwave energy exchange [4,5,6].

Many SA products derived from satellites have been developed and made available to the user community over the last 40 years [7,8] thanks to the availability of well-characterized and calibrated satellite data, and pressing demands from the user community to have access to consistent and ready to use products. The current state-of-the-art method for the computation of surface albedo is to use a Bidirectional Reflectance Distribution Function (BRDF) kernel-based approach for the development of hemispherical albedo products from a number of multi-angular space-borne sensors. This approach is a pragmatic and cost-effective solution to the surface brightness inversion problem of operational programs that are constrained by the need to process extensive amounts of satellite data in near real time. Such an approach was adopted for near real time retrieval of albedo from ADEOS/POLDER [9], Terra + Aqua/MODIS [10,11], MSG/SEVIRI, and MetOp/AVHRR [12,13,14,15,16] in the framework of the Land Surface Analysis Satellite Application Facility (LSA SAF) [17,18], from SPOT/VGT and PROBA-V in Copernicus Global Land Service (CGLS) [19] and Copernicus Climate Change Service (C3S) [14,20], and recently adopted for Sentinel-3 data [21] in the C3S.

Due to the multitude of albedo products available, users face a complex situation because of the spatial and temporal discrepancies among them [13,22,23,24]. Therefore, product quality needs to be assessed, the product’s compliance with requirements must be known, and the user should know to what extent a product is suitable for their specific application [25]. Thus, the availability of reliable in situ data for direct validation of remote sensing ECV products is a key scientific requisite for users to make effective decisions [26] about the utility of a given ECV.

The Land Product Validation (LPV) subgroup [27] of the Committee for Earth Observation Satellites (CEOS) Working Group on Calibration and Validation (WGCV) (the so-called CEOS LPV) coordinates the quantitative validation of satellite-derived land products, focusing on standardized intercomparisons and evaluation across products from different satellites, algorithms, and agency sources. Validation is defined as the process of independently assessing and evaluating the quality of the data products from the system outputs [28]. In terms of satellite-based land products, validation refers to the assessment and quantification of their accuracy and uncertainties via analytical comparisons with reference datasets. In 2019, the CEOS LPV subgroup compiled a global surface albedo product validation best practices protocol [29], a community-agreed-upon document on validation best practices for satellite-based albedo products. The validation protocol is mainly based on two strategies: comparison of satellite products versus in situ data (direct validation) and intercomparison of satellite products (indirect validation).

A hierarchical approach that identifies four land product validation stages (Table 1) was adopted by CEOS LPV, following CEOS validation principles [30]. Stage 3 implies that uncertainties are evaluated over a significant set of locations (>30), using community consensus protocols, where spatial and temporal consistency is evaluated over globally representative locations. Stage 4 includes systematic and regular update of stage 3 validation results when new product versions are released, or as the time series expands. Stage 4 implies an operational validation that ensures time series are systematically validated.

A current review of the literature revealed only a few long-term global SA validation exercises [31]. Among them, global albedo products from MODIS observations [22,32,33], PROBA-V [24], SPOT/VGT, and EPS/AVHRR [15] were validated to the CEOS LPV stage 3 hierarchy level. However, in most cases, the existing validation results are not directly comparable, as validation practices are rather diverse in terms of methods, reference data and locations, lack of traceability and transparency, spatiotemporal coverage, scaling, metrics and target accuracies, resulting in a considerable lack of consistency in validation outputs [8]. These inconsistencies support the movement of the future validation activities into an operational validation workflow that would align all the various aspects, thus allowing for consistent and traceable validation of the existing satellite-based albedo products.

An operational validation workflow to validate remote sensing global terrestrial ECV products should consider, at least, four key components [8]: (1) the long-term Climate Data Records (CDRs) of satellite-based ECVs; (2) a set of representative, reliable, and globally distributed in situ measurements; (3) a suitable standard assessment framework based on community-agreed-upon validation best practice protocols; and (4) an online validation platform that provides open-access tools to generate standardized validation reports.

In this context, the Surface ALbedo VALidation (SALVAL) web tool [34] has been developed, integrating these four key components, in order to facilitate the evaluation of global albedo products derived from satellite data in accordance with the CEOS LPV validation good practice [29]. The main objectives of the SALVAL tool are: (i) to provide transparency and traceability (i.e., reproducibility) to the validation process; (ii) to integrate the protocols and metrics from the CEOS LPV albedo product validation best practices document [29] into a tool within which the user can analyze the different validation criteria; (iii) to provide a platform where new versions or new products can be consistently evaluated; and (iv) to facilitate the update of validation results and to achieve CEOS LPV validation stage 4. The SALVAL tool currently includes existing CDRs and a representative network of ground observations, and, additionally, it also allows for user-friendly result updates for new products or periods.

The objective of this paper is two-fold: (1) to introduce the SALVAL tool and functionalities; and (2) to show an application of validation and intercomparison of three existing surface albedo CDRs: the NASA’s MCD43A3 Collection 6.1 (C6.1), C3S SPOT/VGT and PROBA-V V2, and Beijing Normal University’s (BNU) GLASS albedo products. The remainder of this paper is structured as follows: Section 2 describes the functionalities of the SALVAL tool; Section 3 presents the specifications and validation results from an intercomparison and validation exercise including these three remote sensing SA products; and Section 4 and Section 5 discuss the results and provide the conclusions, respectively.

2. Methods and Datasets: The SALVAL Tool

2.1. Validation Methodology

The validation methodology follows the CEOS LPV best practices protocol for the validation of satellite-derived albedo products [29]. The methods are divided into four categories: product intercomparison, direct validation, precision, and stability. Each category has several sub-criteria, including completeness, spatial consistency, temporal consistency, accuracy, precision, and stability. Table 2 describes the validation methods associated with each criterion. The definitions of the completeness, precision, uncertainty, and accuracy are drawn from the experimental recommendations of the Joint Committee for Guides in Metrology (JCGM) regarding the expression of uncertainty in measurement [35] and from the GCOS [36]. The summary of accuracy, precision, and uncertainty (APU) validation metrics is provided in Table 3. The SALVAL tool allows users the possibility of directly visualizing the results by category and criteria, or they can generate a validation report document of the results.

The validation exercise is based on two main approaches: indirect validation (i.e., satellite product intercomparison) and direct validation. The indirect validation offers a means of assessing the discrepancies (systematic or random) between products and allows the evaluation of the metrics at a global scale due to the limited availability of ground measurements. For satellite-derived albedo, direct validation involves comparison of satellite products with the albedo measured from in situ tower-based instruments. Direct validation enables the assessment of uncertainties, and it may be argued that only such methods can be considered actual validation in the field of remote sensing [23]. SALVAL uses two different sampling strategies for product intercomparison and direct validation, as described below. The list of sites is available at the CEOS Cal/Val portal [37].

The quantitative and qualitative product-to-product intercomparisons are performed over the 720-site land validation (LANDVAL) network [17,28] (shown in Figure 1), which was designed to globally represent the variability of land surface types. This network also includes 19 well-known desert calibration sites [29] for the precision and stability evaluation. Product intercomparison of different satellite products needs a common spatial and temporal sampling. The comparison is performed at 3 km × 3 km spatial support area with the aim of reducing the co-registration errors between products and differences in their sensor point spread function which determines the actual footprint of the data [38,39]. The temporal frequency used corresponds to that of the products under evaluation, which are compared with the closest date of the reference satellite products.

The 99 sites of the Representativeness-Evaluated ALbedo Stations (REALS) dataset (also depicted in Figure 1) are used for the direct point-to-pixel validation (see Section 2.3). In order to compare tower in situ measurements to satellite-based albedo products, the generation of clear blue-sky satellite albedo [40] is performed as the weighted average of black-sky and white-sky retrieved albedos by the fraction of diffuse downwelling shortwave radiation from the ground station at a particular illumination and atmospheric condition. The test of spatial representativeness of the in situ albedometer footprints was performed for the satellite pixel resolution of interest according to in situ measurement standards [41,42]. This exercise is usually performed at a resolution of 1 km considering only sites that are homogeneous over a footprint area of at least a 1 km² around the albedo in situ station. More details about the test of the spatial representativeness can be found below in Section 2.3. Temporal averages of daily ground data are computed to allow comparison with satellite products generated in composited time intervals.

It is important to highlight the fact that the sampling of LANDVAL and REALS sites allows the validation exercise to reach CEOS LPV stage 3, as uncertainties can be quantified over a significant set of in situ stations (99 REALS) and the spatial and temporal consistency with similar satellite datasets can be evaluated over globally representative locations (LANDVAL).

2.2. Satellite Datasets

Currently, the SALVAL platform incorporates satellite-based albedo datasets from existing programs, such as NASA MODIS [43], Copernicus C3S [20], BNU GLASS [44], or GlobAlbedo [45]. Additionally, the tool also allows for importing either a new product dataset or expanding the temporal coverage of existing products.

In this work, we selected three satellite-based CDRs with around 20 years of data from existing operational programs: NASA MCD43A3 C6.1, C3S multi-sensor V2, and BNU GLASS. These products include the total shortwave domain [0.3 μm, 4 μm], which is the most relevant albedo quantity in terms of energy budget. The total shortwave domain also includes visible [0.4 μm, 0.7 μm] and near-infrared [0.7 μm, 4 μm]. Additionally, different definitions of satellite albedo products exist according to the domain of directional integration [46]: the directional-hemispherical reflectance (DHR) or black-sky albedo (BSA or AL-DH), and the bi-hemispherical reflectance (BHR) or white-sky albedo (WSA or AL-BH). BSA is defined as the ratio of the radiant flux for light reflected by a unit surface area into the view hemisphere to the illumination radiant flux, when the surface is illuminated with a parallel beam of light from a single direction [47]. WSA is the ratio of the radiant flux reflected from a unit surface area into the whole hemisphere to the incident radiant flux of hemispherical angular extent [48]. The combination of both BSA and WSA in relation to the proportion of sky irradiance provides the actual albedo value, also called blue-sky albedo [40].

The three satellite albedo products provide both BSA at local solar noon and WSA for three broadband ranges (visible, NIR, and total shortwave).

2.2.1. NASA MCD43A3 C6.1

The MODIS BRDF/Albedo MCD43A3 C6.1 dataset, available from the LPDAAC [49], produces albedo quantities at a resolution of 500 m in a sinusoidal projection. These quantities have been produced daily since 2000 with a synthesis period of 16 days, using data from both the Terra and Aqua satellites. The MODIS albedo algorithm uses atmospherically corrected cloud-free reflectance data (the MOD/MYD09 product) to establish the best fit to a linear kernel-driven BRDF model. Observations flagged as “cloud”, “cirrus high” or “aerosol high”, or “very high solar zenith angles” are not utilized. The parametric BRDF model uses the Ross_Thick kernel for volumetric scattering and the Li_Sparse_Reciprocal kernel for geometrical scattering [47,50]. A full retrieval of the model is attempted if there are at least seven or more high-quality observations that are well-distributed over the viewing hemisphere during the 16-day synthesis period. When the number of observations is strictly less than 7 and strictly greater than 2, or if observations are not well sampled or do not fit the BRDF model well, though the number of observations is larger than 7, a backup algorithm (magnitude inversion) with prior information is used and the values are designated with a lower quality flag. A fill value is stored if the number of observations is strictly less than 3. Snow and snow-free albedos are processed separately depending on the ground condition of the day of interest. In addition, products at 30 arc second and 0.05 degree resolutions on a geographic lat/lon projection are also available for ease of use by modelers. Separate 30 arc second snow-free gap-filled products (MCD43GF) are also accessible [51]. The BRDF model parameters are then used for estimating spectral albedos from angular integration. The broadband albedos are then computed using the spectral to broadband conversion approach [52]. The MCD43 C6.1 products use an improved backup database [53], which is pixel-based, updated from the latest high quality full inversion, as opposed to the land cover-based BRDF database used in the previous Collection 5. In this work, the updated C6.1 version is being used, which incorporates the latest improved calibration coefficients and surface reflectance values.

MCD43A3 C6.1 SA products have reached CEOS LPV validation stage 3 [53]. Existing studies of the previous MCD43 Collections 5 and 6 indicate that the accuracy of the MODIS shortwave broadband albedo met the GCOS accuracy requirements (Max [5%, 0.0025]) for both snow-free and snow-covered surfaces [41,42,54,55].

2.2.2. C3S Multi-Sensor V2

The C3S V2 products, which are available in the C3S Climate Data Store (CDS, [56]), provide a CDR of global albedo estimates in a nearly 40-year record of satellite observations from 1981 to 2020, using multiple input datasets: NOAA/AVHRR from 1981 to 2005 (around 4 km pixel size), SPOT/VGT from 1998 to 2014 (1 km pixel size), and PROBA-V from 2014 to 2020 (1 km pixel size). The temporal frequency of the products is 10 days, built with a compositing window of 20 days. The dates of production are the 10th, 20th, and final days of each month.

C3S V2 version builds on V1 [57] and adds a multi-sensor aspect to the albedo products delivered so far. The retrieval algorithm [14,58] uses Top-of-Canopy (TOC) reflectance resulting from both the harmonized pixel classification approach (cloud, snow, and shadow pixels) and the Simplified Method for Atmospheric Correction (SMAC) algorithm [59], to improve satellite cross-consistency. Additionally, a spectral harmonization is performed, by creating TOC reflectance values as if they had been acquired with SPOT/VGT. The harmonized TOC reflectance values are processed to determine the coefficients of a semi-empirical kernel-based reflectance model, which accounts for the complete angular dependence of the bi-directional reflectance factor. This inversion step is performed using prior information and a climatology of surface BRDF. To estimate albedo using a multi-sensor time series, a second harmonization was conducted, using BRDF climatology data from SPOT/VGT. The algorithm [14] relies on a similar method to that of previous versions (Kalman filters and BRDF model fit) but with the addition of a reference BRDF (climatology derived from VGT BRDF) to (i) reduce the gaps in the time series, and (ii) introduce multi-sensor information in each albedo estimation to increase homogeneity among the datasets derived from each sensor. In the final steps, the spectral albedo values are determined from the angular integrals of the model functions using the retrieved parameter, and the narrow-to-broadband conversion is performed with a linear regression formula. A different set of narrow-to-broadband conversion coefficients is applied for snow-free pixels and for pixels flagged as “snow/ice” in the input data status map.

An independent scientific quality assessment of the C3S multi-sensor V2 products was performed [60], considering the whole CDR (1981–2020), achieving CEOS LPV validation stage 2. The V2 time series was demonstrated to be more consistent compared with V1 when changing input data in the transitions from AVHRR to SPOT/VGT and from SPOT/VGT to PROBA-V. Direct validation results of SPOT/VGT SA V2 showed positive bias (12.5%) and overall uncertainty (RMSD) of 0.048 in the comparison with albedo measurements from 15 homogeneous FLUXNET stations (2000–2005 period). The comparison of C3S PROBA-V SA V2 with field data for 20 homogeneous sites from the Copernicus Ground-Based Observations for Validation (GBOV) program (2014–2018 period) also showed positive bias (9.1%) and RMSD of 0.039

2.2.3. BNU GLASS V4

The Global LAnd Surface Satellites [61] (GLASS) product suite provides albedo products from 1981 to 2019 with a temporal resolution of 8 days, and is available from BNU [44]. Products from 1981 to 1999 are derived from AVHRR data, with a spatial resolution of 0.05° in the global Climate Modelling Grid (CMG) projection. The GLASS albedo products from 2000 to 2019 are derived from MODIS data, with a spatial resolution of 1 km in a tile-based sinusoidal projection.

The GLASS albedo products are generated in two steps. First, the albedo is directly retrieved from remote sensing data by employing the second simulation of a satellite signal in the solar spectrum (6S) atmospheric radiative transfer model (RTM) to simulate the TOA directional reflectance, calculating the broadband albedos based on the POLDER BRDF dataset, and then establishing a relationship between the TOA reflectance and surface broadband albedo using an angular bin regression method [62]. Intermediate products from this first step are merged to generate a unique and gap-filled final product based on the Statistics-based Temporal Filtering (STF) algorithm [63]. Additionally, the current version 4 data have been improved with respect to previous versions: the snow/ice BRDF model has been updated [64] and a water surface BRDF model has been adopted for the ocean surface as well as for mixed pixels of water/sea ice [65].

The preliminary evaluation of the GLASS albedo product [22] showed that it is a gapless, long-term, continuous and self-consistent dataset with an accuracy similar to that of the MODIS MCD43 C5 product. Recent validation efforts for version 4 [61] showed that products are consistent with MODIS MCD43A3 C6 over snow-free pixels, and overall uncertainty (RMSD) of 0.052 was found in comparison with tower-based observations from 53 spatially homogeneous global sites. The RMSD was reduced to 0.037 for snow-free pixels.

2.2.4. Summary and Quality Flags

Table 4 summarizes the main features of the three SA products used in this study. Both MCD43A3 and C3S are based on a BRDF model resulting from the combination of two of the same models for defining the volumetric (Ross_Thick) and geometric (Li_Sparse_Reciprocal) kernels [47]. Unlike the MCD43A3 and C3S albedo products, which are based on inversions of BRDF model parameters, the GLASS albedo products are based on the direct-estimation method and represent surface albedo under general clear-sky atmospheric conditions.

The production frequency of C3S V2 and GLASS V4 is 10 and 8 days, respectively. MCD43A3 C6.1 products are originally produced daily, but are ingested into the SALVAL platform at a temporal frequency of 5 days, due to the limitations in memory storage and processing capabilities of the server.

The quality flag information for each product was used to filter low quality pixels (Table 5), and the SALVAL tool provides the results for both cases of “best quality” retrievals, and all retrieved valid pixels (quality flag not considered).

For MCD43A3 only “best quality” full BRDF inversions were considered, and magnitude BRDF inversions were discarded. Higher confidence is expected for a “full inversion” retrieval that is performed under good sampling of the viewing and illumination geometry for a grid location. The “magnitude inversion” is a backup algorithm, which performs generally well, but relies on prior estimates of the BRDF when insufficient observations are available to fully sample the viewing and illumination geometry. On the other hand, the C3S V2 pixels where the algorithm failed were not considered in this validation exercise. Additionally, two ancillary variables were also considered: the uncertainty (ERR) and the mean age (AGE, in number of days) of the observations used to produce the SA. The C3S V2 pixels with associated uncertainty of greater than 0.2 and an AGE greater than 20 were discarded, indicating excessive use of prior information [60]. Finally, in the case of GLASS V4, only pixels classified as “good overall uncertainty” were considered as “best quality”, while pixels flagged as “acceptable”, “with uncertainty” or “fill value” resulting from gap-filling methods were not considered.

2.3. Representativeness-Evaluated ALbedo Stations (REALS) Dataset

REALS is a database of sites with the objective to generate an extensive in situ dataset for direct validation purposes. The database has been defined as a combination of 99 sites with availability of ground data in the 2000–2020 period from existing networks and initiatives, such as Ground-Based Observations for Validation (GBOV) [66], Flux Network (FLUXNET) [67], the National Science Foundation’s National Ecological Observatory Network (NEON) [68], European Fluxes Database Cluster (EFDC) [69], Integrated Carbon Observation System (ICOS) [70], and Australia’s Land Ecosystem Observatory or Terrestrial Ecosystem (TERN) [71]. Some GBOV, FLUXNET, and EFDC sites incorporate measurements from the Baseline Surface Radiation Network (BSRN) [72] and its U.S. component known as the Surface Radiation Budget (SURFRAD) Network [73]. BSRN is considered the gold standard of albedo measurements according to the GCOS [36] and CEOS LPV albedo best practices protocol [29]. It is worth noting that 23 of these REALS sites are considered “Super Sites” endorsed by the CEOS LPV subgroup, meaning that they are well-characterized (canopy structure and biogeophysical variables) following well-established protocols, and are active in long-term operation, supported by appropriate funding and infrastructural capacity.

The albedo measured from a tower covers a circular footprint (dependent upon the tower height) that should be ideally equivalent to the pixel size of satellite estimation. However, satellite footprints are often much larger than the tower footprints. For that reason, the representativeness of the measurement, both within the tower footprint and in the surrounding landscape, is evaluated through geostatistical indices based on a semivariogram model [74,75], following current state-of-the-art protocols [33,41,42] and CEOS LPV recommendations. Four different geostatistical attributes have been used for this evaluation: relative coefficient of variation (R_CV), scale requirement index (R_SE), relative strength of the spatial correlation (R_ST), and relative proportion of structural variation (R_SV). They are combined in a condensed indicator of spatial representativeness: the standard score (ST, see Equation (1)). For those situations where the semivariogram estimator does not provide a good fit with a semi-spherical variogram, the first order score (RAW, see Equation (2)) can be adopted to evaluate the spatial representativeness [41]. Both scores are directly proportional to the representativeness or relative homogeneity of a site, so a higher score means that a ground site (point) is more suitable to be comparable to satellite-based measurements (pixel). Note that the cover does not have to be uniform, and can be a heterogeneous landscape, as long as that heterogeneous landscape is similar both within the tower footprint and the surrounding landscape.

ST = {(\frac{| R_{CV} | + | R_{ST} | + | R_{SV} |}{3} + R_{SE})}^{- 1}

(1)

RAW = {| 2 R_{CV} |}^{- 1}

(2)

The methodology adopted for the evaluation of the representativeness of the sites is based on the estimation of the spherical semivariogram for different spatial resolutions (1, 1.5, and 3 km²). When the semivariogram has been estimated, geostatistical indices are calculated in order to quantify the level of representativeness of a site.

The spatial representativeness is estimated for each site of the REALS sites in different temporal conditions (leaf-off season and leaf-on season) using high-resolution Sentinel-2 imagery [76] for the B8 band, which is the most spectrally representative of the total shortwave [77]. Figure 2 shows an example of variogram fitting and ST estimation over two different sites of the REALS database, Desert Rock (DRAK) and Talladega National Forest (TALL). These results show more homogeneity or spatial representativeness in the case of TALL (ST = 8) in the leaf-on period than in DRAK (no seasonality) (ST = 0.96). Appendix A describes the ST summary of the REALS sites.

In order to choose a ST threshold for filtering non-representative or differing heterogeneous sites, an analysis of the variation in RMSD (uncertainty) of number of sites and samples between the NASA MCD43A3 C6.1 product and REALS sites was performed for the available period (2000–2020). Figure 3 shows the evolution of number of sites, number of samples, and RMSD as function of the ST score for the comparison between the MCD43A3 C6.1 satellite-derived product and REALS in situ measurements for the 2000–2020 period. According to the results, the RMSD tends to decrease when the ST threshold increases, but the numbers of sites and samples decrease. For this reason, a threshold of 1.5 for ST was selected as the filter in the REALS database because the RMSD tends to be stable at this score and the number of sites and samples discarded is reasonable. This threshold is similar to that used in previous works [24,33].

The main characteristics of REALS sites and ST scores for each site are summarized in Appendix A.

2.4. SALVAL Functionalities and Configuration

The SALVAL tool has three main functionalities: (i) to select both the product to be evaluated and the reference products based on existing datasets or importing a new test candidate product; (ii) to configure the validation exercise by selecting a set of user requirements, spectral region, spatial domain, and temporal domain of the study; and (iii) to run the validation exercise according to the previously set configuration, based on the protocols and metrics implemented from the CEOS LPV albedo protocol [29]. More information about the use of the tool can be found in the SALVAL user guide [34] or in Appendix B.

After the selection of the products to be evaluated, SALVAL requires the configuration of the validation exercise. Different sets of configurations should be introduced by the user, such as the period, the albedo type (black-sky or white-sky albedo) for product intercomparison, the user requirements for the evaluation of stability and accuracy, and the spatial region of the study (global or continental region).

The stability and accuracy results were evaluated against three predefined requirement levels (optimal, target, and threshold), which are used for both BSA and WSA products in all spectral broadband domains. We used default values within the SALVAL tool (Table 6), which are based on a review of the existing user requirements for measuring global climate change [78], and from the Global Climate Observing System (GCOS) [36] and the World Meteorological Organization (WMO) [79]. The optimal accuracy level (Max [5%, 0.0025]) was selected according to the GCOS uncertainty threshold and is equivalent to the WMO goal. The target level is equivalent to the WMO breakthrough level, which is an intermediate level between “goal” and “threshold” (i.e., if achieved, would result in a significant improvement for the targeted application). The breakthrough level may be considered as an optimum, from a cost–benefit point of view, when planning or designing observing systems. Poor performances of the product correspond to values above the threshold levels (WMO minimum requirement). Figure 4 displays the selected uncertainty levels as a function of the product values.

In the recent update of GCOS requirements [80], a new goal uncertainty level of 3% was defined. The SALVAL platform allows modifying the predefined requirements as a function of the user needs.

The defined temporal domain is the 20-year period from 2000 to 2019. For this period, NASA MCD43A3 C6.1 and GLASS are based on MODIS data whereas C3S multi-sensor V2 products are based on input data from two different platforms (SPOT/VGT for 2000 to May 2014, and PROBA-V from May 2014 to 2019). The whole period is used for direct validation with ground data, and 10 years (2001–2010) is used for the stability evaluation. Product intercomparison and completeness are based on five years (2001–2005) of data, as the SALVAL platform is restricted for those exercises due to computational constraints. The spatial domain covers the whole globe and retrievals from all LANDVAL and REALS sites are included in the analysis.

Results are displayed for total shortwave black-sky albedo (AL-DH-BB) for product intercomparison. For the direct evaluation with in situ measurements, the SALVAL platform computes satellite blue-sky albedos as the weighted average of total shortwave black-sky and white-sky albedos by the fraction of diffuse downwelling shortwave radiation from the ground station.

3. Results

3.1. Product Completeness

The three products (C3S V2, GLASS V4, and MCD43A3 C6.1) show a similar spatial distribution of missing data when all pixels are considered (left side in Figure 5), with gaps mainly located at northern regions. However, when quality flags (see Table 5) are used (right side in Figure 5), MCD43A3 C6.1 is more restrictive, if only pixels based on high-quality full BRDF inversions are considered, compared to C3S V3 and GLASS V4.

The temporal evolution of missing values (Figure 6) is displayed at the different temporal resolutions being used in this effort (i.e., 10, 8, and 5 days for C3S, GLASS, and MCD43A3, respectively) showing similar trends, with the highest percentage of missing data during wintertime in the northern hemisphere (December and January). Products based on MODIS (MCD43A3 C6.1 and GLASS V4) and VGT (C3S V2) show maximum percentages of LANVAL missing data, typically around 20% and 10%, respectively. When only best quality observations are considered according to quality flags, the most (around 80%) missing values are found for MCD43A3 C6.1, and around 30% missing values are found for the other products (C3S V2 and GLASS V4) that incorporate gap-filling techniques in their algorithms.

3.2. Spatial Consistency

The spatial consistency is quantitatively assessed through the global distribution of residuals between pair of products. The residual represents the remaining discrepancies regarding the general trend between products, which means that systematic differences are not considered, depicting more clearly the patterns associated with the spatial distribution of retrievals [81]. Two products are considered spatially consistent when the residual lies within predefined uncertainty requirements. Figure 7 shows the spatial distribution of residuals (average value for 2001–2005 period) between pairs of evaluated products (C3S V2, GLASS V4 and MCD43A3 C6.1). The percentages of residuals within the predefined requirements are summarized in Table 7.

Most residuals between pairs of products are within ±0.015, demonstrating overall good spatial consistency among them. The comparison between GLASS and MCD43A3 shows the higher percentage (around 90%) of evaluated samples within optimal requirements. The comparison of C3S and GLASS seems to display the better spatial consistency, with 98% of cases within target level and almost no non-compliant cases (0.3%). The results show that, in all comparisons, more than 90% of residuals are within target requirements, and typically less than 5% of cases are non-compliant.

3.3. Temporal Consistency

To assess the realism of the seasonal and inter-annual temporal variations, examples of temporal profiles of C3S V2, GLASS V4, and MCD43A3 C6.1 satellite products are qualitatively compared with ground data over REALS sites. It should be noted that two different time periods are displayed due to the different input data used to retrieve C3S V2 products: 2001–2005 (Figure 8, SPOT/VGT) and 2014–2019 (Figure 9, PROBA-V).

Good temporal agreement is found between satellite products and ground data over forest sites in the 2001–2005 period, properly reproducing the different situations: rapid changes due to snow events (CA_Oas, RU_Fyo), stable values over long periods (NL_Loo, RU_Fyo), and variation due to phenological changes (AUS_TUMB). C3S V2 products based on SPOT/VGT typically tend to provide higher values than ground data (and other satellite products) for most snow-free cases. GLASS V4 shows, on the other hand, some unexpected peaks (e.g., NL_Loo in January 2004) which are not captured by the other satellite products and ground data.

Good temporal agreement is also found between satellite retrievals and in situ data in the 2014–2019 period for different biome types, following similar temporal trajectories. C3S V2 (based on PROBA-V for this period) shows an overestimation compared with ground truth and the other satellite products for USA_SFSD (cropland) and SCBI (forest) sites, but better accordance with ground data over AUS_CPRM (grassland) and AU_ASM (forest) than GLASS V4 and MCD43A3 C6.

The snow episodes were correctly reported by the three satellite products in most cases but MCD43A3 C6.1 reaches typically higher values, which are more consistent with daily ground observations. Some spurious events (e.g., February 2015 and 2016 in SCBI) were not captured by C3S V2, which could be attributed to its larger temporal composite and lower production frequency, and to the more conservative approach of the PROBA-V cloud-masking algorithm [24,82].

3.4. Intra-Annual Precision

Intra-annual precision (so-called smoothness) corresponds to temporal noise assumed to have no serial correlation within a season and is quantitatively assessed as the anomaly between the product value for one date and the linear estimate based on its neighbors [83]. Figure 10 shows the Probability Density Function (PDF) of the intra-annual precision for C3S V2, GLASS V4, and MCD43A3 C6.1 products. The median values (indicative of intra-annual precision) of each product are summarized in Table 8.

Smoothness histograms reveal that most values are below 0.005, which demonstrates that the three satellite products show high precision. The temporal resolution of each product directly impacts their smoothness, as MCD43A3 C6.1 (smoothness calculated at 5 days temporal step rather than the actual daily temporal resolution) shows better intra-annual precision than GLASS V4 (8 days) and C3S V2 (10 days).

3.5. Inter-Annual Precision

Inter-annual precision (Figure 11, Table 9) for each satellite product under study is assessed by comparison of retrievals for consecutive years over 19 desert calibration sites [84] during 5 years of data (2001–2005). The best inter-annual precision (i.e., MAD, CEOS LPV best practice) is observed for GLASS V4 (MAD = 0.002, 0.55%), while the C3S VGT V2 product provides the worst results (MAD = 0.007, 1.64%), with median absolute deviations of 1.64%. MCD43A3 C6.1 shows intra-annual precision better than 1% (MAD = 0.004, 0.84%).

3.6. Overall Spatio-Temporal Consistency

The overall consistency between a pair of satellite products is evaluated by reporting several metrics indicative of the goodness of the fit, such as accuracy (mean bias, B, and median deviation, MD), precision (standard deviation, STD, and median absolute deviation, MAD) and uncertainty (RMSD). Scatterplots (Figure 12) and validation metrics between products are computed over LANDVAL sites for the period of 2001–2005 and two different postulates: taking into account all pixels (Table 10) and only considering best quality pixels (Table 11).

The best agreement in terms of accuracy and uncertainty is found between MCD43A3 C6.1 and GLASS V4, with almost no bias and RMSD of 0.043 (21%). Improved results are found when considering best quality retrievals (RMSD of 0.026 (12.4%)). The C3S V2 product tends to provide systematically higher values compared to MODIS-based products (MCD43A3 C6.1 and GLASS V4), with mean bias of 7–8% (0.015–0.017) and MD of 11% (0.024) when all pixels are considered. The uncertainty between C3S V2 and the MODIS-based products significantly improves when only best quality retrievals are considered (mainly in the comparison with MCD43A3 C6.1) but a large bias is found, indicating systematic positive differences.

3.7. Stability

Stability is the extent to which a product remains constant over a long period, typically a decade or more [36]. Temporal stability can be also defined as the change in bias over a predefined time period [85], and stability can be estimated as the slope of a linear regression for the bias over time [86]. In SALVAL implementation, pseudo-invariant desert calibration sites [84] are used for stability evaluation, and the slope of albedo values per decade is provided as an indicator of stability. As desert sites are supposed to experience very little temporal variation, variation in albedo time series can be considered to be equivalent to evaluation of the bias over time.

Figure 13 displays some examples of C3S V2, GLASS V4, and MCD43A3 C6.1 temporal profiles in the 2001–2010 period for some selected desert calibration sites. The box plots of the decadal slopes over all calibration sites are displayed in Figure 14. The three products show some seasonality that could be attributed to changes in illumination, but no deviations at the long-term scale. Furthermore, C3S V2 shows more noise in the signal compared with GLASS V4 and MCD43A3 C6.1, as previously revealed in its worst intra-annual (see Section 3.4) and inter-annual (see Section 3.5) precision. This noise could be partly attributed to the BRDF climatology data used as prior information in the C3S V2 algorithm, which presents the same behavior. Median slopes in all calibration sites revealed no deviation for C3S V2 and GLASS V4 and a very slight positive slope (i.e., 0.003, which is equivalent to 0.6%) for MCD43A3 C6.1, largely fulfilling the GCOS requirements in terms of stability (1%).

3.8. Direct Validation

Direct validation involves the comparison of satellite retrievals with albedo measured from tower-based instruments (REALS dataset). Figure 15 shows the scatterplots between blue-sky satellite albedo quantities and REALS during the 2000–2019 period, taking into account all quality pixels (left side), and only best quality pixels (right side), where most of the outliers are removed. The main statistics resulting from the direct validation are summarized in Table 12 (all pixels) and Table 13 (best quality pixels).

The overall accuracy (median error) of C3S V2 is 15%, with a systematic tendency to provide higher values than ground measurements (mean bias of 12.2%). GLASS V4 and MCD43A3 C6.1 show betters results, and an opposite sign of differences (mean bias of −2.5%). In terms of overall uncertainty, the three satellite products provide similar results in the comparison with in situ data, with RMSD of around 0.04 when all pixels are considered in the comparison, and RMSD around 0.03 when only best quality retrievals are contemplated. However, C3S V2 shows a lower percentage of cases within the predefined accuracy levels than GLASS V4 and MCD43A3 C6.1.

The MAR relationship of the best quality pixels indicates a tendency of all satellite products to overestimate ground data for the lowest albedo values (mainly dominated by forests) and the opposite trend for higher albedo values (sparse vegetation), with slopes lower than 0.7 in the line comparing results to other validation studies over a significant set of locations [15,24,33].

4. Discussion

The three products under study (MCD43A3 C6.1, C3S V2, and CLASS V4) show remarkably good completeness, with missing data mainly located over northern regions and wintertime, typically affected by persistent clouds. The three products introduce different techniques to improve the spatiotemporal continuity: a poorer quality back-up algorithm is used in the case of MCD43A3 C6.1, a prior climatology of BRDF data is used in the case of C3S V2, and gap-filling techniques are used in the case of GLASS. When considering best quality pixels, MCD43A3 C6.1 is the most restrictive product, as only full retrievals of the model are provided when at least 50% of high-quality observations are well-distributed over the viewing hemisphere during the 16-day synthesis period.

In terms of spatial consistency (i.e., residuals), all combinations between pairs of products largely meet uncertainty requirements, with more than 70% of global cases achieving optimal level of consistency (residuals typically lower than 0.015). As expected, the best spatial consistency between pairs of products was found between MCD43A3 C6.1 and GLASS, as they are based on data from the same MODIS instruments on board Terra and Aqua, whereas C3S products are retrieved using SPOT/VGT or PROBA-V depending on the temporal range. Different spectral response functions among the instruments show dissimilarities in band location, band width, and response percentage of input signal over similar spectral channels [87]. The main discrepancies are typically located over equatorial areas and northern regions, which can be explained by cloud contamination and differences in the pre-processing chain. The underestimation of C3S products over snow targets [60] is another reason for the discrepancies over northern areas.

The three satellite products provide good temporal agreement among them and in comparison with in situ data. The better temporal resolution of MCD43A3 C6.1 allows us to capture smoother temporal variations than is possible from C3S V2 and GLASS V4, despite the incorporation of gap-filling techniques in the C3S and GLASS algorithms, and the inclusion of a temporal smoothing method in GLASS V4.

The evaluation of the APU metrics from the indirect evaluation also indicated that the best agreement is found between GLASS V4 and MCD43A3 C6.1. More than 77.5% of best quality observations lie within the optimal uncertainty level (Max [5%, 0.0025]) when the same sensor is used (i.e., GLASS V4 versus MCD43A3 C6.1). C3S V2 provides larger differences from both MODIS-based products, but satisfactory results, with systematically higher values of around 10%.

As a corollary, we can conclude that the use of a different sensor is the most important factor contributing to discrepancies among products. Nevertheless, the inversion algorithm is another contributing factor to differences between products. C3S V2 and MCD43A3 C6.1 make use of semi-empirical linear kernel-driven models to first retrieve BRDF coefficients and then compute surface albedo by angular and spectral integration. By comparison, GLASS adopts the angular bin and STF algorithm, and incorporates improvements in the inversion of snow and ice using an asymptotic radiative transfer model [88]. Additionally, the different spectral integration approach also contributes to differences between products. MODIS and GLASS adopt the same broadband albedo range and narrow-to-broadband conversion algorithm [89]. C3S products are computed over slightly different broadband albedo intervals and a different conversion algorithm [14].

The direct validation showed systematic positive bias of around 10% for C3S (SPOT/VGT and PROBA-V) V2 products for the period under study (2000–2019), in line with that found for previous C3S V1 versions [15,24], where positive bias of 11.5% was also reported. GLASS V4 and MCD43A3 C6.1 showed the opposite sign of differences, but improved results (with mean bias of around 6% and median deviation of 3%).

The comparison of satellite-based surface albedo estimates versus ground measurements indicates the difficulty in complying with existing user uncertainty requirements. Typically, less than 20% of satellite-based best quality retrievals actually achieve the GCOS target (Max [5%, 0.0025]) and the WMO goal requirements in terms of accuracy. By comparison, the three satellite products investigated largely accomplished stability optimal requirements (Max [1%, 0.001]).

5. Conclusions

This paper demonstrates the functionality of the SALVAL online platform to validate currently available operational albedo products. A validation and intercomparison exercise was conducted on three long-term global products generated by C3S, MODIS, and GLASS. Completeness, spatiotemporal consistency, precision, and accuracy were evaluated. Results from the SALVAL tool indicate that the three datasets under evaluation provide long-term reliable and highly consistent retrievals at a global scale. Discrepancies between products are primarily associated with differences in the retrieval processing chain: different input data sensors, pre-processing and atmospheric corrections, and inversion algorithms.

The CEOS LPV validation stage assigned to these global satellite albedo products is currently stage 3, which means that direct validation with in situ data or other reference datasets is performed over a significant set of locations and time periods representing global conditions. Thanks to the availability of the SALVAL online platform, the four main components [8] for an operational validation system of satellite-based surface albedo products have been integrated: long term satellite products, a global in situ dataset, the CEOS LPV validation best practices protocol, and an online validation platform. The SALVAL tool provides the potential functionality to achieve CEOS LPV validation stage 4 as it is also designed to accommodate regular updates of the validation results, providing albedo ECVs the readiness level for ongoing operational validation.

SALVAL provides transparency, consistency, and traceability to the validation process. The tool is available within the CEOS Cal/Val Portal [90], and offers a way to contribute to and collaborate with the greater the scientific community, thus allowing new products or ground reference datasets to be incorporated into the tool.

Author Contributions

J.S.-Z. is the main author and wrote the majority of the manuscript. J.S.-Z. and F.C. conceived and conceptualized the work and contributed to all phases of the investigation. J.S.-Z. initially developed the software and E.M.-S. developed the final software tool and generated the REALS database. Z.W. extracted the satellite dataset of MODIS MCD43A3 C6.1. All authors contributed to the discussion of results. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Data Availability Statement

SALVAL tool is available online at www.salval.eolab.es.

Acknowledgments

The authors would like to thank the different site networks and services dedicated to providing all the ground measurements that are integrated in the REALS dataset (GBOV, FLUXNET, NEON, EFDC, ICOS, TERN, BSRN, SURFRAD). Thanks are due to ESA for publishing the SALVAL tool through the CEOS WGCV Cal/Val Portal. Thanks to the CEOS WGCV LPV subgroup for its contribution to validation practices implemented in SALVAL. This research was supported in part by the U.S. Department of Agriculture, Agricultural Research Service. USDA is an equal opportunity provider and employer.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:

ADEOS	ADvanced Earth Observing Satellite
AL-BH	Bi-Hemispherical ALbedos
AL-DH	Directional-Hemispherical Albedos
APU	Accuracy, Precision and Uncertainty
AVHRR	Advanced Very High Resolution Radiometer
B	Mean Bias
BHR	Bi-Hemispherical Reflectance
BNU	Beijing Normal University’s
BRDF	Bidirectional Reflectance Distribution Function
BSA	Black-Sky Albedo
BSRN	Baseline Surface Radiation Network
C3S	Copernicus Climate Change Service
Cal/Val	Calibration/Validation
CDR	Climate Data Record
CDS	Climate Data Store
CEOS	Committee on Earth Observation Satellites
CGLS	Copernicus Global Land Service
CMG	Climate Modelling Grid
CUL	CULtivated
DBF	Deciduous Broadleaved Forest
DHR	Directional-Hemispherical Reflectance
EBF	Evergreen Broadleaved Forest
ECV	Essential Climate Variable
EFDC	European Fluxes Database Cluster
EOLAB	Earth Observation LABoratory
EPS	EUMETSAT Polar System
ESA	European Space Agency
FLUXNET	FLUXes NETwork
GBOV	Ground-Based Observations for Validation
GCOS	Global Climate Observing System
GLASS	Global LAnd Surface Satellites
GSD	Ground Sampling Distance
HER	HERbaceous
ICOS	Integrated Carbon Observation System
JCGM	Joint Committee for Guides in Metrology
LANDVAL	LAND VALidation network
LPDAAC	Land Processes Distributed Active Archive Center
LPV	Land Product Validation subgroup
MAD	Median Absolute Deviation
MAR	Major Axis Regression
MCD43	TERRA + AQUA MODIS BRDF/Albedo/NBAR Product
MD	Median Deviation
MetOp	Polar-orbiting Meteorological satellites
MODIS	MODerate resolution Imaging Spectroradiometer
MSG	Meteosat Second Generation
N	Number of samples
NASA	National Aeronautics and Space Agency
NEON	National Science Foundation’s National Ecological Observatory Network
NIR	Near-Infrared
NLF	Needle-Leaf Forest
NOAA	National Oceanic and Atmospheric Administration
OF	Other Forests
PDF	Probability Density Function
POLDER	POLarization and Directionality of the Earth’s Reflectances
PROBA-V	Project for Onboard Autonomy satellite, the V standing for vegetation
R	Correlation coefficient
R_CV	Relative Coefficient of Variation
REALS	Representativeness-Evaluated ALbedo Stations
RMSD	Root Mean Square Deviation
R_SE	Scale REequirement index
R_ST	Relative STrength of the spatial correlation
R_SV	Relative proportion of Structural Variation
RTM	Radiative Transfer Model
SA	Surface Albedo
SALVAL	Surface ALbedo VALidation tool
RAW	First order score
SBA	Sparse and Bare Areas
SEVIRI	Spinning Enhanced Visible and Infrared Imager
SHR	SHRublands
SMAC	Simplified Method for Atmospheric Correction
SPOT	Satellites for the Observation of the Earth
ST	STandard score
STD	Standard deviation
STF	Statistics-based Temporal Filtering
SURFRAD	SURFace RADiation budget network
SW	ShortWave
TERN	Australia’s Land Ecosystem Observatory or Terrestrial Ecosystem
TOA	Top-Of-Atmosphere
TOC	Top-Of-Canopy
VGT	VeGeTation sensor
WGCV	Working Group on Calibration and Validation
WMO	World Meteorological Organization
WSA	White-Sky Albedo

Appendix A. REALS Sites’ Characteristics and ST Scores

Table A1. Characteristics and ST scores of REALS sites.

ID	Code	Latitude	Longitude	Name	Network	Class	ST Leaf-Off	ST Leaf-On
1	USA_BOND	40.05192	−88.37309	Bondville	SURFRAD, GBOV	Croplands	1.52	1.58
2	USA_BAOR	40.05005	−105.00387	Boulder	BSRN, GBOV	Croplands	1.29	2.98
3	BEL_BRAS	51.30761	4.51984	Brasschaat	FLUXNET, GBOV(LPV SuperSite)	Forest	19.36	10.42
4	NET_CABA	51.97100	4.92700	Cabauw	BSRN, GBOV	Grass/shrub	13.86	6.65
5	AUS_CPRM	−34.00270	140.58771	Calperum	OZFLUX, TERN, GBOV(LPV SuperSite)	Grass/shrub	2.72	2.83
6	USA_DRAK	36.62418	−116.01990	Desert Rock	SURFRAD, GBOV	Desert	0.96	0.96
7	USA_FPEK	48.30783	−105.10170	Fort Peck	SURFRAD, GBOV	Grass/shrub	1.85	1.60
8	GER_GEBE	51.10010	10.91430	Gebesee	FLUXNET, GBOV	Croplands	1.08	1.22
9	NAM_GOBA	−23.56184	15.04131	Gobabeb	BSRN, GBOV(LPV SuperSite)	Desert	0.95	0.87
10	USA_GCMK	34.25505	−89.87360	Goodwin Creek	SURFRAD, GBOV	Forest	2.92	1.96
11	FRA_GRIG	48.84420	1.95191	Grignon	FLUXNET, GBOV	Croplands	1.04	1.05
12	FRA_GUYA	5.27877	−52.92486	Guyaflux	FLUXNET, GBOV(LPV SuperSite)	Forest	5.47	5.47
13	GER_HAIN	51.07920	10.45220	Hainich	FLUXNET, GBOV(LPV SuperSite)	Forest	6.84	18.17
14	USA_NRFT	40.03287	−105.54690	Niwot Ridge Forest	FLUXNET, GBOV	Forest	4.06	n/a
15	ITA_RENO	46.58690	11.43370	Renon	FLUXNET, GBOV	Forest	1.45	1.79
16	USA_PSUS	40.72012	−77.93085	Rock Springs	SURFRAD, GBOV	Forest	1.04	2.96
17	USA_SFSD	43.73403	−96.62331	Sioux Falls SurfRad	SURFRAD, GBOV	Croplands	1.85	2.11
18	USA_SGP	36.60575	−97.48876	Southern Great Plains	SURFRAD, GBOV	Croplands	1.02	0.80
19	USA_TBLN	40.12498	−105.23680	Table Mountain	SURFRAD, GBOV	Desert	2.24 ⁽*⁾	2.24 ⁽*⁾
20	AUS_TUMB	−35.65652	148.15163	Tumbarumba	OZFLUX, TERN, GBOV (LPV SuperSite)	Forest	11.65	11.65
21	LENO	31.85388	−88.16122	Lenoir Landing	NEON	Forest	2.33	4.96
22	TALL	32.95046	−87.39327	Talladega National Forest	NEON(LPV SuperSite)	Forest	103.65	8.00
23	BONA	65.15401	−147.50258	Caribou-Poker	NEON	Forest	n/a	2.78
24	DEJU	63.88112	−145.75136	Delta Junction	NEON	Forest	n/a	3.77
25	HEAL	63.87569	−149.21334	Healy	NEON	Grass/shrub	n/a	1.42
26	TOOL	68.66109	−149.37047	Toolik	NEON	Grass/shrub	n/a	1.28
27	SRER	31.91068	−110.83549	Santa Rita Experimental Range	NEON	Grass/shrub.	5.92	4.29
28	SOAP	37.03337	−119.26219	Soaproot Saddle	NEON	Forest	19.48	10.58
29	TEAK	37.00583	−119.00602	Lower Teakettle	NEON	Forest	25.17	8.46
30	CPER	40.81550	−104.7456	Central Plains Experimental Range	NEON (LPV SuperSite)	Grass/shrub	1.12	0.98
31	NIWO	40.05425	−105.58237	Niwot Ridge Mountain Research Station	NEON	Forest	0.71	0.88
32	STER	40.46190	−103.02930	Sterling	NEON	Croplands	1.05	0.92
33	DSNY	28.12504	−81.43620	Disney Wilderness Preserve	NEON	Croplands	1.34	1.51
34	OSBS	29.68927	−81.99343	Ordway-Swisher Biological Station	NEON(LPV SuperSite)	Forest	0.65	0.61
35	JERC	31.19484	−84.46861	Jones Ecological Research Center	NEON	Forest	12.99	4.83
36	KONA	39.11044	−96.61295	Konza Prairie Biological Station–Relocatable	NEON	Grass/shrub	1.60	1.26
37	KONZ	39.10077	−96.56309	Konza Prairie Biological Station	NEON	Grass/shrub	4.37	1.26
38	UKFS	39.04043	−95.19215	The University of Kansas Field Station	NEON	Forest	0.55	10.60
39	SERC	38.89008	−76.56001	Smithsonian Environmental Research Center	NEON	Forest	2.64	4.13
40	HARV	42.53690	−72.17266	Harvard Forest	NEON(LPV SuperSite)	Forest	40.01	6.32
41	UNDE	46.23388	−89.53725	UNDERC	NEON	Forest	2.29	2.08
42	BART	44.06388	−71.28731	Bartlett Experimental Forest	NEON(LPV SuperSite)	Forest	6.50	3.04
43	JORN	32.59068	−106.84254	Jornada LTER	NEON	Grass/shrub	0.83	1.04
44	DCFS	47.16165	−99.10656	Dakota Coteau Field School	NEON	Grass/shrub	0.87	1.18
45	NOGP	46.76972	−100.91535	Northern Great Plains Research Laboratory	NEON	Grass/shrub	1.74	1.43
46	OAES	35.41059	−99.05879	Klemme Range Research Station	NEON	Grass/shrub	1.04	1.41
47	GUAN	17.96955	−66.86870	Guanica Forest	NEON(LPV SuperSite)	Forest	9.75	9.75
48	LAJA	18.02125	−67.07690	Lajas Experimental Station	NEON	Grass/shrub	1.35	1.23
49	GRSM	35.68896	−83.50195	Great Smoky Mountains National Park	NEON	Forest	7.39	4.27
50	ORNL	35.96412	−84.28260	Oak Ridge	NEON(LPV SuperSite)	Forest	13.12	1.46
51	MOAB	38.24833	−109.38827	Moab	NEON(LPV SuperSite)	Grass/shrub	0.43	1.19
52	ONAQ	40.17759	−112.45244	Onaqui	NEON	Grass/shrub	1.30	1.59
53	MLBS	37.37828	−80.52484	Mountain Lake Biological Station	NEON(LPV SuperSite)	Forest	7.41	1.55
54	SCBI	38.89292	−78.1395	Smithsonian Conservation Biology Institute	NEON (LPV SuperSite)	Forest	2.51	13.86
55	ABBY	45.76243	−121.24700	Abby Road	NEON	Forest	2.42	7.30
56	WREF	45.82049	−121.95191	Wind River Experimental Forest	NEON	Forest	6.17	5.76
57	STEI	45.50894	−89.58637	Steigerwaldt Land Services	NEON(LPV SuperSite)	Forest	6.44	1.84
58	TREE	45.49369	−89.58571	Treehaven	NEON	Forest	8.10	6.44
59	AT-Neu	47.11667	11.3175	Neustift	FLUXNET	Grass/shrub	1.14	1.86
60	CA-Gro	48.2167	−82.1556	Ontario–Groundhog River, Boreal Mixedwood Forest	FLUXNET	Forest	6.32	4.91
61	CA-Oas	53.62889	−106.19779	Saskatchewan–Western Boreal, Mature Aspen	FLUXNET	Forest	27.82	9.18
62	CA-Obs	53.98717	−105.11779	Saskatchewan–Western Boreal, Mature Black Spruce	FLUXNET	Forest	7.98	3.23
63	CA-Qfo	49.6925	−74.34206	Quebec–Eastern Boreal, Mature Black Spruce	FLUXNET	Forest	1.40	1.47
64	CZ-BK1	49.50208	18.53688	Bily Kriz forest	FLUXNET(LPV SuperSite)	Forest	4.63	7.44
65	DE-Lnf	51.32822	10.3678	Leinefelde	FLUXNET	Forest	13.88	3.06
66	DE-Tha	50.96256	13.56515	Tharandt	FLUXNET(LPV SuperSite)	Forest	5.51	2.86
67	FR-Gri	48.84422	1.95191	Grignon	FLUXNET	Croplands	n/a	n/a
68	FR-LBr	44.71711	−0.7693	Le Bray	FLUXNET	Forest	10.82	1.59
69	FR-Pue	43.7413	3.5957	Puechabon	FLUXNET(LPV SuperSite)	Forest	1.22	1.22
70	GH-Ank	5.26854	−2.69421	Ankasa	FLUXNET	Forest	17.71	17.71
71	IT-Col	41.84936	13.58814	Collelongo	FLUXNET(LPV SuperSite)	Forest	1.63	1.44
72	IT-MBo	46.01468	11.04583	Monte Bondone	FLUXNET	Grass/shrub	2.03	1.26
73	IT-SR2	43.73202	10.29091	San Rossore 2	FLUXNET	Forest	13.04	12.66
74	NL-Hor	52.24035	5.0713	Horstermeer	FLUXNET	Grass/shrub	0.60	0.60
75	NL-Loo	52.16658	5.74356	Loobos	FLUXNET(LPV SuperSite)	Forest	29.14	1.55
76	RU-Fyo	56.46153	32.92208	Fyodorovskoye	FLUXNET(LPV SuperSite)	Forest	17.98	119.73
77	SN-Dhr	15.40278	−15.43222	Dahra	FLUXNET(LPV SuperSite)	Grass/shrub	1.03	0.83
78	US-Me2	44.4523	−121.5574	Metolius mature ponderosa pine	FLUXNET	Forest	0.79	2.18
79	US-UMd	45.5625	−84.6975	UMBS Disturbance	FLUXNET	Forest	0.69	0.80
80	US-Var	38.4133	−120.9507	Vaira Ranch- Ione	FLUXNET	Grass/shrub	4.84	2.58
81	ES-Cpa	39.22417	−0.90305	Cortes de Pallas	EFDC	Grass/shrub	6.88	4.70
82	ES-ES2	39.27556	−0.31528	El Saler-Sueca	EFDC	Croplands	5.36	4.68
83	ES-LMa	39.9415	−5.77336	Las Majadas del Tietar	EFDC	Grass/Shrub	1.66	1.24
84	DE-HoH	52.08656	11.22235	Hohes Holz	ICOS (LPV SuperSite)	Forest	6.95	5.28
85	SE-Svb	64.25611	19.7745	Svartberget	ICOS (LPV SuperSite)	Forest	1.11	1.11
86	FI-Hyy	61.84741	24.29477	Hyytiala	FLUXNET (LPV SuperSite)	Forest	1.37	1.37
87	DE-RuS	50.86591	6.44714	Selhausen Juelich	FLUXNET, ICOS (LPV SuperSite)	Croplands	1.82	1.40
88	AU_ASM	−22.2828	133.2493	Alice Springs Meller	TERN (LPV SuperSite)	Forest	8.88	6.78
89	AU_Boy	−32.477093	116.93856	Boyaginj Wandoo Woodland	TERN (SuperSite)	Forest	0.72	0.33
90	AU_Cum	−33.61528	150.72361	Cumberland Plain	TERN (LPV SuperSite)	Forest	6.18	1.04
91	AU_DRF	−16.23819	145.42715	Daintree Rainforest	TERN (SuperSite)	Forest	13.17	4.53
92	AU_Gin	−31.37635	115.71377	Gingin Banksia Woodland	TERN (SuperSite)	Forest	1.74	0.97
93	AU_GWW	−30.1914	120.65416	Great Western Woodlands	TERN (LPV SuperSite)	Forest	23.87	1.79
94	AU_LiS	−13.17904	130.79455	Litchfield Savanna	TERN (LPV SuperSite)	Forest	34.74	7.66
95	AU_RCR	−17.11747	145.63014	Robson Creek Rainforest	TERN (LPV SuperSite)	Forest	17.90	28.67
96	AU_SPU	−27.38806	152.87778	Samford Peri-Urban	TERN (SuperSite)	Forest	14.49	4.71
97	AU_Wrr	−43.09502	146.65452	Warra Tall Eucalypt	TERN (LPV SuperSite)	Forest	3.76	3.30
98	AU_WSE	−37.4222	144.0944	Wombat Stringybark Eucalypt	TERN (LPV SuperSite)	Forest	8.34	13.02
99	AU_WDE	−36.6732	145.0294	Whroo Dry Eucalypt	TERN (SuperSite)	Forest	4.15	91.64

(*) For those cases, RAW score (see Equation (2)) was adopted due to the semivariogram estimator does not provide a good fit with a semi-spherical variogram (i.e., ST score cannot be computed).

Appendix B. Using the SALVAL Tool

Follow these steps to start using SALVAL. More technical details about tool functionalities, satellite reference products, in situ datasets, etc., can be found in the SALVAL user guide [34].

(1): Sign up to start using the SALVAL Tool.

Figure A1. Snapshot of SALVAL configuration step 1. “*” stands for mandatory fields.

(2): Specify the product being validated and the reference products. Select from the existing database of products or import new products.

Figure A2. Snapshot of SALVAL configuration step 2.

(3): Define the input product: time period, albedo type, requirements, and spatial coverage.

Figure A3. Snapshot of SALVAL configuration step 3.

(4): Visualize the validation results for different criteria, or generate a standardized validation report in PDF.

Figure A4. Snapshot of SALVAL configuration step 4.

(5): Enjoy the interactive validation process (see below Direct Validation type results).

References

Dickinson, R.E. Land processes in climate models. Remote Sens. Environ. 1995, 51, 27–38. [Google Scholar] [CrossRef]
Liang, S. A direct algorithm for estimating land surface broadband albedos from MODIS imagery. IEEE Trans. Geosci. Remote Sens. 2003, 41, 136–145. [Google Scholar] [CrossRef]
WMO; United Nations Educational, Scientific and Cultural Organization; Intergovernmental Oceanographic Commission; United Nations Environment Programme; International Science Council. The 2022 GCOS Implementation Plan (GCOS-244). Available online: https://gcos.wmo.int/en/publications/gcos-implementation-plan2022 (accessed on 10 November 2022).
Zhou, Y.; Wang, D.; Liang, S.; Yu, Y.; He, T. Assessment of the Suomi NPP VIIRS Land Surface Albedo Data Using Station Measurements and High-Resolution Albedo Maps. Remote Sens. 2016, 8, 137. [Google Scholar] [CrossRef] [Green Version]
Wang, D.; Liang, S.; Zhou, Y.; He, T.; Yu, Y. A New Method for Retrieving Daily Land Surface Albedo from VIIRS Data. IEEE Trans. Geosci. Remote Sens. 2017, 55, 1765–1775. [Google Scholar] [CrossRef]
Wang, D.; Liang, S.; He, T.; Yu, Y. Direct estimation of land surface albedo from VIIRS data: Algorithm improvement and preliminary validation. J. Geophys. Res. Atmos. 2013, 118, 12577–12586. [Google Scholar] [CrossRef]
Balsamo, G.; Agusti-Panareda, A.; Albergel, C.; Arduini, G.; Beljaars, A.; Bidlot, J.; Bousserez, N.; Boussetta, S.; Brown, A.; Buizza, R.; et al. Satellite and In Situ Observations for Advancing Global Earth Surface Modelling: A Review. Remote Sens. 2018, 10, 2038. [Google Scholar] [CrossRef] [Green Version]
Bayat, B.; Camacho, F.; Nickeson, J.; Cosh, M.; Bolten, J.; Vereecken, H.; Montzka, C. Toward operational validation systems for global satellite-based terrestrial essential climate variables. Int. J. Appl. Earth Obs. Geoinf. 2021, 95, 102240. [Google Scholar] [CrossRef]
Leroy, M.; Deuzé, J.L.; Bréon, F.M.; Hautecoeur, O.; Herman, M.; Buriez, J.C.; Tanré, D.; Bouffiès, S.; Chazette, P.; Roujean, J.L. Retrieval of atmospheric properties and surface bidirectional reflectances over land from POLDER/ADEOS. J. Geophys. Res. Atmos. 1997, 102, 17023–17037. [Google Scholar] [CrossRef]
Schaaf, C.B.; Gao, F.; Strahler, A.H.; Lucht, W.; Li, X.; Tsang, T.; Strugnell, N.C.; Zhang, X.; Jin, Y.; Muller, J.P.; et al. First operational BRDF, albedo nadir reflectance products from MODIS. Remote Sens. Environ. 2002, 83, 135–148. [Google Scholar] [CrossRef] [Green Version]
Strahler, A.H.; Muller, J.-P.; MODIS Science Team Members. MODIS BRDF/Albedo Product: Algorithm Theoretical Basis Document Version 5.0. Available online: https://modis.gsfc.nasa.gov/data/atbd/atbd_mod09.pdf (accessed on 15 February 2023).
Geiger, B.; Carrer, D.; Franchistéguy, L.; Roujean, J.L.; Meurey, C. Land surface albedo derived on a daily basis from meteosat second generation observations. IEEE Trans. Geosci. Remote Sens. 2008, 46, 3841–3856. [Google Scholar] [CrossRef]
Carrer, D.; Roujean, J.L.; Meurey, C. Comparing operational MSG/SEVIRI Land Surface albedo products from Land SAF with ground measurements and MODIS. IEEE Trans. Geosci. Remote Sens. 2010, 48, 1714–1728. [Google Scholar] [CrossRef]
Carrer, D.; Pinault, F.; Lellouch, G.; Trigo, I.F.; Benhadj, I.; Camacho, F.; Ceamanos, X.; Moparthy, S.; Munoz-Sabater, J.; Schüller, L.; et al. Surface Albedo Retrieval from 40-Years of Earth Observations through the EUMETSAT/LSA SAF and EU/C3S Programmes: The Versatile Algorithm of PYALUS. Remote Sens. 2021, 13, 372. [Google Scholar] [CrossRef]
Lellouch, G.; Carrer, D.; Vincent, C.; Pardé, M.; Frietas, S.C.; Trigo, I.F. Evaluation of Two Global Land Surface Albedo Datasets Distributed by the Copernicus Climate Change Service and the EUMETSAT LSA-SAF. Remote Sens. 2020, 12, 1888. [Google Scholar] [CrossRef]
Carrer, D.; Moparthy, S.; Lellouch, G.; Ceamanos, X.; Pinault, F.; Freitas, S.C.; Trigo, I.F. Land surface albedo derived on a ten daily basis from Meteosat Second Generation Observations: The NRT and climate data record collections from the EUMETSAT LSA SAF. Remote Sens. 2018, 10, 1262. [Google Scholar] [CrossRef] [Green Version]
Trigo, I.F.; Dacamara, C.C.; Viterbo, P.; Roujean, J.-L.; Olesen, F.; Barroso, C.; Camacho-de-Coca, F.; Carrer, D.; Freitas, S.C.; García-Haro, J.; et al. The Satellite Application Facility for Land Surface Analysis. Int. J. Remote Sens. 2011, 32, 2725–2744. [Google Scholar] [CrossRef]
Land Surface Analysis (LSA-SAF) of EUMETSAT. Available online: https://landsaf.ipma.pt/en/ (accessed on 10 November 2022).
Copernicus Global Land Service (CGLS) portal. Available online: https://land.copernicus.eu/global/index.html (accessed on 10 November 2022).
Copernicus Climate Change Service (C3S). Available online: https://climate.copernicus.eu/ (accessed on 10 November 2022).
Sanchez-Zapero, J.; Camacho, F.; Leon-Tavares, J.; Martinez-Sanchez, E.; Gorrono, J.; Benhadj, I.; Tote, C.; Swinnen, E.; Munoz-Sabater, J. Prototype for Surface Albedo Retrieval Based on Sentinel-3 OLCI and SLSTR Data in the Framework of Copernicus Climate Change. In Proceedings of the 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, Brussels, Belgium, 11–16 July 2021; pp. 2377–2380. [Google Scholar]
Liu, Q.; Wang, L.; Qu, Y.; Liu, N.; Liu, S.; Tang, H.; Liang, S. Preliminary evaluation of the long-term GLASS albedo product. Int. J. Digit. Earth 2013, 6, 69–95. [Google Scholar] [CrossRef]
Liu, Y.; Wang, Z.; Sun, Q.; Erb, A.M.; Li, Z.; Schaaf, C.B.; Zhang, X.; Román, M.O.; Scott, R.L.; Zhang, Q.; et al. Evaluation of the VIIRS BRDF, Albedo and NBAR products suite and an assessment of continuity with the long term MODIS record. Remote Sens. Environ. 2017, 201, 256–274. [Google Scholar] [CrossRef]
Sánchez-Zapero, J.; Camacho, F.; Martínez-Sánchez, E.; Lacaze, R.; Carrer, D.; Pinault, F.; Benhadj, I.; Muñoz-Sabater, J. Quality Assessment of PROBA-V Surface Albedo V1 for the Continuity of the Copernicus Climate Change Service. Remote Sens. 2020, 12, 2596. [Google Scholar] [CrossRef]
Zeng, Y.; Su, Z.; Calvet, J.C.; Manninen, T.; Swinnen, E.; Schulz, J.; Roebeling, R.; Poli, P.; Tan, D.; Riihelä, A.; et al. Analysis of current validation practices in Europe for space-based climate data records of essential climate variables. Int. J. Appl. Earth Obs. Geoinf. 2015, 42, 150–161. [Google Scholar] [CrossRef]
Nightingale, J.; Mittaz, J.P.D.; Douglas, S.; Dee, D.; Ryder, J.; Taylor, M.; Old, C.; Dieval, C.; Fouron, C.; Duveau, G.; et al. Ten Priority Science Gaps in Assessing Climate Data Record Quality. Remote Sens. 2019, 11, 986. [Google Scholar] [CrossRef] [Green Version]
LPV (Land Product Validation). Subgroup CEOS Validation Hierarchy 2019. Available online: https://lpvs.gsfc.nasa.gov/ (accessed on 2 March 2022).
Justice, C.; Belward, A.; Morisette, J.; Lewis, P.; Privette, J.; Baret, F. Developments in the’validation’of satellite sensor products for the study of the land surface. Int. J. Remote Sens. 2000, 21, 3383–3390. [Google Scholar] [CrossRef]
Wang, Z.; Schaaf, C.; Lattanzio, A.; Carrer, D.; Grant, I.; Roman, M.; Camacho, F.; Yang, Y.; Sánchez-Zapero, J. Global Surface Albedo Product Validation Best Practices Protocol. Version 1.0. In Good Practices for Satellite-Derived Land Product Validation (p. 45): Land Product Validation Subgroup (WGCV/CEOS); Wang, Z., Nickeson, J., Román, M., Eds.; Available online: https://lpvs.gsfc.nasa.gov/PDF/CEOS_ALBEDO_Protocol_20190307_v1.pdf (accessed on 1 March 2022).
Nightingale, J.; Schaepman-Strub, G.; Nickeson, J.; Baret, F.; Herold, M. Assessing Satellite-Derived Land Product Quality for Earth System Science Applications: Overview of the CEOS LPV Sub-Group. In Proceedings of the 34th International Symposium on Remote Sensing of Environment, Sydney, NSW, Australia, 10–15 April 2011. [Google Scholar]
Mayr, S.; Kuenzer, C.; Gessner, U.; Klein, I.; Rutzinger, M. Validation of Earth Observation Time-Series: A Review for Large-Area and Temporally Dense Land Surface Products. Remote Sens. 2019, 11, 2616. [Google Scholar] [CrossRef] [Green Version]
Liang, S.; Fang, H.; Chen, M.; Shuey, C.J.; Walthall, C.; Daughtry, C.; Morisette, J.; Schaaf, C.; Strahler, A. Validating MODIS land surface reflectance and albedo products: Methods and preliminary results. Remote Sens. Environ. 2002, 83, 149–162. [Google Scholar] [CrossRef]
Cescatti, A.; Marcolla, B.; Santhana Vannan, S.K.; Pan, J.Y.; Román, M.O.; Yang, X.; Ciais, P.; Cook, R.B.; Law, B.E.; Matteucci, G.; et al. Intercomparison of MODIS albedo retrievals and in situ measurements across the global FLUXNET network. Remote Sens. Environ. 2012, 121, 323–334. [Google Scholar] [CrossRef] [Green Version]
EOLAB. Surface ALbedo VALidation (SALVAL) Tool. Available online: https://salval.eolab.es/ (accessed on 10 November 2022).
JCGM-GUM Joint Committee for Guides in Metrology (JCGM)—Guides to the Expression of Uncertainty in Measurement (GUM). [ISO/IEC Guide 98—Part 3, 2008]. Available online: https://www.iso.org/sites/JCGM/GUM-introduction.htm (accessed on 10 November 2022).
GCOS-154 Systematic Observation Requirements for Satellite-Based Data Products for Climate. Supplemental Details to the Satellite-Based Component of the “Implementation Plan for the GCOS in Support of the UNFCCC”. Available online: https://library.wmo.int/doc_num.php?explnum_id=3710 (accessed on 10 April 2022).
SALVAL Sampling—CalValPortal. Available online: https://calvalportal.ceos.org/sampling (accessed on 19 December 2022).
Camacho, F.; Cernicharo, J.; Lacaze, R.; Baret, F.; Weiss, M. GEOV1: LAI, FAPAR essential climate variables and FCOVER global time series capitalizing over existing products. Part 2: Validation and intercomparison with reference products. Remote Sens. Environ. 2013, 137, 310–329. [Google Scholar] [CrossRef]
Sánchez, J.; Camacho, F.; Lacaze, R.; Smets, B. Early validation of PROBA-V GEOV1 LAI, FAPAR and FCOVER products for the continuity of the copernicus global land service. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.—ISPRS Arch. 2015, 40, 93–100. [Google Scholar] [CrossRef] [Green Version]
Lewis, P.; Barnsley, M. Influence of the sky radiance distribution on various formulations of the earth surface albedo. In Proceedings of the 6th International Symposium on Physical Measurements and Signatures in Remote Sensing, ISPRS, Val d’Isère, France, 17–22 January 1994; pp. 707–715. [Google Scholar]
Román, M.O.; Schaaf, C.B.; Woodcock, C.E.; Strahler, A.H.; Yang, X.; Braswell, R.H.; Curtis, P.S.; Davis, K.J.; Dragoni, D.; Goulden, M.L.; et al. The MODIS (Collection V005) BRDF/albedo product: Assessment of spatial representativeness over forested landscapes. Remote Sens. Environ. 2009, 113, 2476–2498. [Google Scholar] [CrossRef] [Green Version]
Román, M.O.; Schaaf, C.B.; Lewis, P.; Gao, F.; Anderson, G.P.; Privette, J.L.; Strahler, A.H.; Woodcock, C.E.; Barnsley, M. Assessing the coupling between surface albedo derived from MODIS and the fraction of diffuse skylight over spatially-characterized landscapes. Remote Sens. Environ. 2010, 114, 738–760. [Google Scholar] [CrossRef]
MODIS Data Products. Available online: https://modis.gsfc.nasa.gov/data/dataprod/ (accessed on 10 November 2022).
Global LAnd Surface Satellite (GLASS). Available online: http://www.glass.umd.edu/ (accessed on 1 April 2022).
Globalbedo Portal. Available online: http://www.globalbedo.org/ (accessed on 10 November 2022).
Schaepman-Strub, G.; Schaepman, M.E.; Painter, T.H.; Dangel, S.; Martonchik, J.V. Reflectance quantities in optical remote sensing-definitions and case studies. Remote Sens. Environ. 2006, 103, 27–42. [Google Scholar] [CrossRef]
Lucht, W.; Schaaf, C.B.; Strahler, A.H. An algorithm for the retrieval of albedo from space using semiempirical BRDF models. IEEE Trans. Geosci. Remote Sens. 2000, 38, 977–998. [Google Scholar] [CrossRef] [Green Version]
Shuai, Y.; Tuerhanjiang, L.; Shao, C.; Gao, F.; Zhou, Y.; Xie, D.; Liu, T.; Liang, J.; Chu, N. Re-understanding of land surface albedo and related terms in satellite-based retrievals. Big Earth Data 2020, 4, 45–67. [Google Scholar] [CrossRef]
Schaaf, C.; Wang, Z. MODIS/Terra+Aqua BRDF/Albedo Daily L3 Global—500m V061 [Data set]. NASA EOSDIS Land Processes DAAC. Available online: https://lpdaac.usgs.gov/products/mcd43a3v061/ (accessed on 20 December 2022).
Lucht, W.; Lewis, P. Theoretical noise sensitivity of BRDF and albedo retrieval from the EOS-MODIS and MISR sensors with respect to angular sampling. Int. J. Remote Sens. 2000, 21, 81–98. [Google Scholar] [CrossRef]
Sun, Q.; Wang, Z.; Li, Z.; Erb, A.; Schaaf, C.B. Evaluation of the global MODIS 30 arc-second spatially and temporally complete snow-free land surface albedo and reflectance anisotropy dataset. Int. J. Appl. Earth Obs. Geoinf. 2017, 58, 36–49. [Google Scholar] [CrossRef]
Liang, S.A.H.S.C.W. Retrieval of Land Surface Albedo from Satellite Observations: A Simulation Study. J. Appl. Meteorol. 1999, 38, 712–725. [Google Scholar] [CrossRef]
Wang, Z.; Schaaf, C.B.; Sun, Q.; Shuai, Y.; Román, M.O. Capturing rapid land surface dynamics with Collection V006 MODIS BRDF/NBAR/Albedo (MCD43) products. Remote Sens. Environ. 2018, 207, 50–64. [Google Scholar] [CrossRef]
Wang, Z.; Schaaf, C.B.; Chopping, M.J.; Strahler, A.H.; Wang, J.; Román, M.O.; Rocha, A.V.; Woodcock, C.E.; Shuai, Y. Evaluation of Moderate-resolution Imaging Spectroradiometer (MODIS) snow albedo product (MCD43A) over tundra. Remote Sens. Environ. 2012, 117, 264–280. [Google Scholar] [CrossRef]
Wang, Z.; Schaaf, C.B.; Strahler, A.H.; Chopping, M.J.; Román, M.O.; Shuai, Y.; Woodcock, C.E.; Hollinger, D.Y.; Fitzjarrald, D.R. Evaluation of MODIS albedo product (MCD43A) over grassland, agriculture and forest surface types during dormant and snow-covered periods. Remote Sens. Environ. 2014, 140, 60–77. [Google Scholar] [CrossRef] [Green Version]
Climate Data Store of Copernicus Climate Change Service. Available online: https://cds.climate.copernicus.eu/#!/home (accessed on 10 May 2020).
Carrer, D.; Pinault, F.; Ramon, D.; Benhadj, I.; Swinnen, E. Algorithm Theoretical Basis Document (ATBD) of CDR SPOT/VGT Surface Albedo v1.0 (Official Reference Number Service Contract: 2018/C3S_312b_Lot5_VITO/SC1). Available online: https://datastore.copernicus-climate.eu/documents/satellite-albedo/D1.3.3-v1.0_ATBD_CDR-ICDR_SA_PROBAV_v1.0_PRODUCTS_v1.0.2.pdf (accessed on 1 November 2022).
Carrer, D.; Pinault, F.; Bigeard, G.; Ramon, D.; Jolivet, D.; Kirches, G.; Brockmann, C.; Boettcher, M.; Benhadj, I. Algorithm Theoretical Basis Document Multi sensor CDR Surface Albedo v2.0 (Official Reference Number Service Contract: 2018/C3S_312b_Lot5_VITO/SC1). Available online: https://datastore.copernicus-climate.eu/documents/satellite-albedo/D1.3.4-v2.0_ATBD_CDR_SA_MULTI_SENSOR_v2.0_PRODUCTS_v1.1.pdf (accessed on 10 November 2022).
Rahman, H.; Dedieu, G. SMAC: A simplified method for the atmospheric correction of satellite measurements in the solar spectrum. Int. J. Remote Sens. 1994, 15, 123–143. [Google Scholar] [CrossRef]
Sánchez-Zapero, J.; Martínez-Sánchez, E.; Camacho, F.; León-Tavares, J. Product Quality Assessment Report Multi-sensor Surface Albedo v2.0 (Official Reference Number Service Contract: 2018/C3S_312b_Lot5_VITO/SC1). Available online: https://datastore.copernicus-climate.eu/documents/satellite-albedo/D2.3.4-v2.0_PQAR_CDR_SA_MULTI_SENSOR_v2.0_PRODUCTS_v1.1.pdf (accessed on 10 November 2022).
Liang, S.; Cheng, J.; Jia, K.; Jiang, B.; Liu, Q.; Xiao, Z.; Yao, Y.; Yuan, W.; Zhang, X.; Zhao, X.; et al. The Global LAnd Surface Satellite (GLASS) product suite. Bull. Am. Meteorol. Soc. 2020, 102, E323–E337. [Google Scholar] [CrossRef]
Qu, Y.; Liu, Q.; Liang, S.; Wang, L.; Liu, N.; Liu, S. Direct-estimation algorithm for mapping daily land-surface broadband albedo from modis data. IEEE Trans. Geosci. Remote Sens. 2014, 52, 907–919. [Google Scholar] [CrossRef]
Liu, N.F.; Liu, Q.; Wang, L.Z.; Liang, S.L.; Wen, J.G.; Qu, Y.; Liu, S.H. A statistics-based temporal filter algorithm to map spatiotemporally continuous shortwave albedo from MODIS data. Hydrol. Earth Syst. Sci. 2013, 17, 2121–2129. [Google Scholar] [CrossRef] [Green Version]
Qu, Y.; Liang, S.; Liu, Q.; Li, X.; Feng, Y.; Liu, S. Estimating Arctic sea-ice shortwave albedo from MODIS data. Remote Sens. Environ. 2016, 186, 32–46. [Google Scholar] [CrossRef]
Feng, Y.; Liu, Q.; Qu, Y.; Liang, S. Estimation of the Ocean Water Albedo From Remote Sensing and Meteorological Reanalysis Data. IEEE Trans. Geosci. Remote Sens. 2016, 54, 850–868. [Google Scholar] [CrossRef]
Ground-Based Observations for Validation (GBOV) of Copernicus Global Land Products Site. Available online: https://land.copernicus.eu/global/gbov (accessed on 3 November 2022).
FLUXNET. The Data Portal serving the FLUXNET Community. Available online: https://fluxnet.org/ (accessed on 10 November 2022).
NSF NEON. Open Data to Understand our Ecosystems. Available online: https://www.neonscience.org/ (accessed on 10 November 2022).
European Fluxes Database Cluster. Available online: http://www.europe-fluxdata.eu/ (accessed on 10 November 2022).
ICOS—Integrated Carbon Observation System. Available online: https://www.icos-cp.eu/ (accessed on 10 November 2022).
TERN—Australia’s Terrestrial Ecosystem Research Network. Available online: https://www.tern.org.au/ (accessed on 10 November 2022).
Ohmura, A.; Dutton, E.G.; Forgan, B.; Fröhlich, C.; Gilgen, H.; Hegner, H.; Heimo, A.; König-Langlo, G.; McArthur, B.; Müller, G.; et al. Baseline Surface Radiation Network (BSRN/WCRP): New Precision Radiometry for Climate Research. Bull. Am. Meteorol. Soc. 1998, 79, 2115–2136. [Google Scholar] [CrossRef]
ESRL Global Monitoring Laboratory—Global Radiation and Aerosols. SURFRAD. Available online: https://gml.noaa.gov/grad/surfrad/ (accessed on 20 December 2022).
Matheron, G. Principles of geostatistics. Econ. Geol. 1963, 58, 1246–1266. [Google Scholar] [CrossRef]
Hohn, M.E. An Introduction to Applied Geostatistics: By Edward H. Isaaks and R. Mohan Srivastava, 1989, Oxford University Press, New York, 561 p., ISBN 0-19-505012-6, ISBN 0-19-505013-4. Comput. Geosci. 1991, 17, 471–473. [Google Scholar] [CrossRef]
Copernicus Sentinel-2 Mission. Available online: https://sentinel.esa.int/web/sentinel/missions/sentinel-2 (accessed on 20 December 2022).
Bonafoni, S.; Sekertekin, A. Albedo Retrieval from Sentinel-2 by New Narrow-to-Broadband Conversion Coefficients. IEEE Geosci. Remote Sens. Lett. 2020, 17, 1618–1622. [Google Scholar] [CrossRef]
Ohring, G.; Wielicki, B.; Spencer, R.; Emery, B.; Datla, R. Satellite Instrument Calibration for Measuring Global Climate Change: Report of a Workshop. Bull. Am. Meteorol. Soc. 2005, 86, 1303–1314. [Google Scholar] [CrossRef]
World Meteorological Organization (WMO) Requirements for Earth Surface Albedo. Available online: https://www.wmo-sat.info/oscar/variables/view/54 (accessed on 10 April 2020).
The 2022 GCOS ECVs Requirements (GCOS 245). Available online: https://library.wmo.int/index.php?lvl=notice_display&id=22135#.Y5eLMofMI2whttps://library.wmo.int/doc_num.php?explnum_id=11318 (accessed on 20 December 2022).
Sánchez-Zapero, J.; Camacho, F. Product Quality Assessment Report (PQAR) of CDR and ICDR Surface Albedo v1.0 based on PROBA-V (Official Reference Number Service Contract: 2018/C3S_312b_Lot5_VITO/SC1). Available online: https://cds.climate.copernicus.eu/cdsapp#!/dataset/satellite-albedo?tab=doc (accessed on 9 April 2022).
Iannone, R.Q.; Niro, F.; Goryl, P.; Dransfeld, S.; Hoersch, B.; Stelzer, K.; Kirches, G.; Paperin, M.; Brockmann, C.; Gomez-Chova, L.; et al. Proba-V cloud detection Round Robin: Validation results and recommendations. In Proceedings of the 2017 9th International Workshop on the Analysis of Multitemporal Remote Sensing Images, MultiTemp, Bruges, Belgium, 27–29 June 2017; pp. 1–8. [Google Scholar] [CrossRef]
Weiss, M.; Baret, F.; Block, T.; Koetz, B.; Burini, A.; Scholze, B.; Lecharpentier, P.; Brockmann, C.; Fernandes, R.; Plummer, S.; et al. On line validation exercise (OLIVE): A web based service for the validation of medium resolution land products. application to FAPAR products. Remote Sens. 2014, 6, 4190–4216. [Google Scholar] [CrossRef] [Green Version]
Lacherade, S.; Fougnie, B.; Henry, P.; Gamet, P. Cross calibration over desert sites: Description, methodology, and operational implementation. IEEE Trans. Geosci. Remote Sens. 2013, 51, 1098–1113. [Google Scholar] [CrossRef]
Merchant, C.J. Thermal remote sensing of sea surface temperature. Therm. Infrared Remote Sens. Sens. Methods Appl. 2013, 17, 287–313. [Google Scholar]
Fell, F.; Bennartz, R.; Loew, A. Validation of the EUMETSAT Geostationary Surface Albedo Climate Data Record -2- (ALBEDOVAL-2). Available online: https://www.eumetsat.int/website/home/Data/TechnicalDocuments/index.html (accessed on 12 April 2020).
Gu, L.; Shuai, Y.; Shao, C.; Xie, D.; Zhang, Q.; Li, Y.; Yang, J.; Gu, L.; Shuai, Y.; Shao, C.; et al. Angle Effect on Typical Optical Remote Sensing Indices in Vegetation Monitoring. Remote Sens. 2021, 13, 1699. [Google Scholar] [CrossRef]
Kokhanovsky, A.A.; Breon, F.M. Validation of an analytical snow BRDF model using PARASOL multi-angular and multispectral observations. IEEE Geosci. Remote Sens. Lett. 2012, 9, 928–932. [Google Scholar] [CrossRef]
Liang, S. Narrowband to broadband conversions of land surface albedo I Algorithms. Remote Sens. Environ. 2001, 76, 213–238. [Google Scholar] [CrossRef]
CalVal Portal: SALVAL Tool. Available online: https://calvalportal.ceos.org/salval (accessed on 14 November 2022).

Figure 1. Global distribution of 99 REALS and 720 LANDVAL (including 19 desert calibration) sites used for product intercomparison and direct validation, respectively. LADNVAL are displayed as per biome type: EBF stands for evergreen broadleaved forest, DBF for deciduous broadleaved forest, NLF for needle-leaf forests, OF for other forests, CUL for cultivated, HER for herbaceous, SHR for shrublands, and SBA for sparse and bare areas.

Figure 2. Example of variogram fitting and ST estimation over two different sites, Desert Rock (DRAK) and Talladega National Forest (TALL).

Figure 3. Evolution of Number of sites (top left), Number of samples (top right), and RMSD (bottom) of the comparison of MCD43A3 C6.1 versus REALS sites as a function of the ST score in the 2000–2020 period.

Figure 4. Stability (left) and Accuracy (right) requirement levels as a function of SA values.

Figure 5. Global maps of the percentage of missing values for C3S V2 (top), GLASS V4 (center), and MCD43A3 C6.1 (bottom) for all pixels (left), and only best quality pixels (right) in the 2003 year, evaluated over the 720 LANDVAL sites.

Figure 6. Temporal evolution of the percentage of LANDVAL missing data in the 2003 for C3S V2 (red), GLASS V4 (green), and MCD43A3 C6.1 (blue) considering all pixels (top), and only best quality pixels (bottom) according to quality flags.

Figure 7. Global spatial distribution of average residuals (left) and number of residuals that reach the requirements (right) for C3S V2 vs. GLASS V4 (top), C3SV2 vs. MCD43A3 C6.1 (middle), and GLASS V4 vs. MCD43A3 C6.1 (bottom). Evaluation in the 2001–2005 period with a 10-day temporal frequency over LANDVAL sites using best quality total shortwave black-sky albedo (AL-DH-BB) retrievals.

Figure 8. Examples of albedo temporal variations in C3S V2 (red), GLASS V4 (green), and MCD43A3 C6.1 (blue) satellite products (all quality pixels not just high quality) and ground data (black dots) from 4 REALS sites during the 2001–2005 period.

Figure 9. Examples of albedo temporal variations in C3S V2 (red), GLASS V4 (green), and MCD43A3 C6.1 (blue) satellite products (all quality pixels not just high quality) and ground data (black dots) from 4 REALS sites during the 2014–2019 period.

Figure 10. Histogram of the smoothness (δ) for C3S V2 (red), GLASS V4 (green), and MCD43A3 C6.1 (blue) total shortwave black-sky albedo products. Evaluation in the 2001–2005 period over LANDVAL sites considering all pixels.

Figure 11. Scatterplots (X-Axis: retrieval for a given date, Y-axis: retrieval for equivalent date of the following year) of the inter-annual precision of C3S V2 (top left), GLASS V4 (top right), and MCD43A3 C6.1 (bottom) products. Evaluation for total shortwave black-sky retrievals over LANDVAL sites in the 2001–2005 period considering all pixels.

Figure 12. Scatterplots of C3S V2 vs. GLASS V4 (top), C3S V2 vs. MCD43A3 C6.1 (middle), and GLASS V4 vs. MCD43A3 C6.1 (bottom) for all pixels (left) and best quality pixels (right) for black-sky total shortwave retrievals in the 2001–2005 period over LANDVAL sites.

Figure 13. Examples of temporal profiles of C3S V2 (red), GLASS V4 (green), and MCD43A3 C6.1 (blue) for black-sky total shortwave albedos over calibration sites of LANDVAL in the 2001–2010 period for best quality pixels. Dashed lines represent the linear regression of each product trend. Mean slope value corresponds to the mean slope considering all calibration sites.

Figure 14. Box plots of the slope per decade (2001–2010) for C3S V2 (red), GLASS V4 (green) and MCD43A3 C6.1 (blue) for black-sky albedos where each box stretches from the 25th percentile to the 75th percentile of the data and whiskers include 99.3% of the coverage data (±2.7 σ). Ouliers are represented by rhombus. Red lines/crosses represent median/mean values. Computation over desert calibration sites.

Figure 15. Direct validation of C3S V2 (top), GLASS V4 (middle), and MCD43A3 C6.1 (bottom) blue-sky albedo satellite products vs. REALS ground values during the 2000–2019 period for all pixels (left) and only best quality pixels (right). Green, blue, red, and orange points represent forest, crop, shrublands/herbaceous, and desert biome types, respectively.

Table 1. The CEOS LPV validation stages [27].

Stage	Description
0	No validation. Product accuracy has not been assessed. Product considered beta.
1	Product accuracy is assessed from a small (typically < 30) set of locations and time periods by comparison with in situ or other suitable reference data.
2	Product accuracy is estimated over a significant (typically > 30) set of locations and time periods by comparison with reference in situ or other suitable reference data. Spatial and temporal consistency of the product, and its consistency with similar products, has been evaluated over globally representative locations and time periods. Results are published in the peer-reviewed literature.
3	Uncertainties in the product and its associated structure are well quantified over a significant (typically > 30) set of locations and time periods representing global conditions by comparison with reference in situ or other suitable reference data. Validation procedures follow community-agreed-upon good practices. Spatial and temporal consistency of the product, and its consistency with similar products, has been evaluated over globally representative locations and time periods. Results are published in the peer-reviewed literature.
4	Validation results for stage 3 are systematically updated when new product versions are released or as the inter-annual time series expands. When appropriate for the product, uncertainties in the product are quantified using fiducial reference measurements over a global network of sites and time periods (if available).

Table 2. Summary of SALVAL validation methods.

Category	Criteria	Methods
Product inter- comparison	Completeness	Gap size distribution (spatial and temporal) and gap length.
	Spatial consistency	Mean residual and mean difference maps, percentage of cases within requirements
	Temporal consistency	Temporal profiles and histograms of cross-correlation
	Overall analysis	Product histograms, difference histograms, scatterplots (APU validation metrics) and box plots of bias and Root Mean Square Deviation (RMSD) per bin
Direct validation	Temporal realism	Temporal evolution of the satellite-derived products vs ground data
Direct validation	Overall analysis	Scatterplots (APU validation metrics)
Precision	Intra-annual precision	Median 3-point difference (smoothness)
Precision	Inter-annual precision	Median absolute deviation over desert calibration sites
Stability	Stability	Slope of the 10-year linear regression over desert calibration sites

Table 3. APU validation metrics.

Statistics	Comment
N	Number of samples. Indicative of the strength of the validation.
B	Mean Bias. Difference between average values of x and y. Indicative of accuracy and offset. Bias (%) is the relative mean bias between the average of x and y.
MD	Median (i.e., 50th percentile) deviation between x and y. MD is the CEOS LPV good practice reporting of the accuracy. MD (%) is the relative MD between the average of x and y.
STD	Standard deviation of the pair differences. Indicates precision. STD (%) is the relative STD between the average of x and y.
MAD	Median (i.e., 50th percentile) absolute deviation between x and y. MAD is the CEOS LPV good practice indicator of precision. MAD (%) is the relative MAD between the average of x and y.
RMSD	Root Mean Square Deviation. RMSD is the square root of the average of squared errors between x and y. The RMSD is the CEOS LPV good practice reporting of uncertainty. RMSD (%) is the relative RMSD between the average of x and y.
R	Correlation coefficient. Indicates descriptive power of the linear accuracy test. Pearson coefficient is used.
MAR	Slope and offset of the Major Axis Regression (MAR) linear fit. Indicates possible bias.

Table 4. Characteristics of the global remote sensing SA products under study. GSD stands for the Ground Sampling Distance.

Product	Satellite /Sensor	Methodology	Broadband Definition	Frequency /Period	GSD /Projection	Reference
NASA MCD43A3 C6.1	TERRA + AQUA /MODIS	BRDF model inversion and angular/spectral integration	visible [0.3–0.7 μm] NIR [0.7–5.0 μm] total SW [0.3–5.0 μm]	Daily ⁽*⁾ /16 days	500 m /Sinusoidal	[10]
C3S V2	SPOT /VGT PROBA /VGT	BRDF model inversion and angular/spectral integration	visible [0.4–0.7 µm] NIR [0.7–4 µm] total SW [0.3–4.0 µm]	10 days /20 days using prior climatology BRDF	1 km /Plate Carrée	[14,58]
GLASS V4	TERRA + AQUA /MODIS	RTM + gap-filling	visible [0.3–0.7 μm] NIR [0.7–5.0 μm] total SW [0.3–5.0 μm]	8 days /16 days	1 km /Sinusoidal	[61]

(*) MCD43A3 C6.1 products are produced daily but are ingested into the SALVAL tool at a temporal frequency of 5 days.

Table 5. Quality flag information used to filter pixels flagged as “low quality”.

Product	Quality Control Used as “Best Quality”	Quality Control Used to Discard Pixels
MCD43A2 C6.1	Full BRDF inversion	Magnitude inversion
C3S V2	Land (bits 0–1 QFLAG) Normally processed (bit 7 QFLAG) ERR $\leq$ 0.2 AGE $\leq$ 20	Sea and continental water (bits 0–1 QFLAG) Algorithm Failed (bit 7 QFLAG) ERR > 0.2 AGE > 20
GLASS V4	Overall uncertainty ‘best quality’	Overall uncertainty ‘acceptable’, ‘with uncertainty’ or ‘fill value’

Table 6. Predefined accuracy and stability requirement levels used for SA validation.

	Optimal	Target	Threshold
Accuracy requirement	Max [5%, 0.0025]	Max [10%, 0.01]	Max [15%, 0.015]
Stability requirement	Max [1%, 0.001]	Max [2%, 0.002]	Max [3%, 0.003]

Table 7. Percentage of mean residuals between pair of products (C3S V2, GLASS V4 and MCD43A3 C6.1) within each level of uncertainty requirements. Evaluation in the 2001–2005 period over LANDVAL sites using best quality total shortwave black-sky albedo (AL-DH-BB) retrievals.

Residual	Optimal	Target	Threshold	Non-Compliance
C3S V2 vs. GLASS V4	83.5%	98.1%	99.7%	0.3%
C3S V2 vs. MCD43A3 C61	72.8%	91.8%	95.4%	4.6%
GLASS V4 vs. MCD43A3 C61	89.6%	94.5%	94.9%	5.1%

Table 8. Summary of median δ for C3S V2, GLASS V4, and MCD43A3 C6.1 total shortwave black-sky albedo products. Evaluation in the 2001–2005 period over LANDVAL sites considering all pixels.

Satellite Product	Median δ
C3S V2	0.0022
GLASS V4	0.0014
MCD43A3 C6.1	0.0008

Table 9. Inter-annual precision indicator (median absolute deviation between two consecutive years) of C3S V2, GLASS V4, and MCD43A3 C6.1 products. Evaluation for total shortwave black-sky albedo retrievals over LANDVAL sites in the 2001–2005 period considering all pixels.

	C3S V2	GLASS V4	MCD43A3 C6.1
Inter-annual precision: median absolute deviation	0.007 (1.64%)	0.002 (0.55%)	0.004 (0.84%)

Table 10. Summary of the main statistics for scatterplots between C3S V2 vs. GLASS V4, C3S V2 vs. MCD43A3 C6.1, and GLASS V4 vs. MCD43A3 C6.1 for black-sky total shortwave retrievals in the 2001–2005 period over LANDVAL sites. All quality pixels are taken into account.

	C3S V2 vs. GLASS V4	C3S V2 vs. MCD43A3 C6.1	GLASS V4 vs. MCD43A3 C6.1
N	122086	115912	145694
R	0.91	0.89	0.94
MAR	y = 0.87x + 0.04	y = 0.83x + 0.05	y = 0.96x + 0.01
B	0.017 (8.1%)	0.015 (7.1%)	−0.001 (−0.3%)
MD	0.024 (11.4%)	0.024 (11.2%)	<0.001 (0.2%)
STD	0.052 (24.8%)	0.062 (28.7%)	0.043 (21.0%)
MAD	0.027 (12.7%)	0.027 (12.8%)	0.006 (3.0%)
RMSD	0.055 (26.1%)	0.064 (29.6%)	0.043 (21.0%)
%Optimal	10.2	9.6	61.8
%Target	28.4	29.4	83.7
%Threshold	49.4	49.7	90.9

Table 11. Summary of the main statistics for scatterplots between C3S V2 vs. GLASS V4, C3S V2 vs. MCD43A3 C6.1, and GLASS V4 vs. MCD43A3 C6.1 for black-sky total shortwave retrievals in the 2001–2005 period over LANDVAL sites. Only best quality pixels are taken into account.

	C3S V2 vs. GLASS V4	C3S V2 vs. MCD43A3 C6.1	GLASS V4 vs. MCD43A3 C6.1
N	102857	52954	69280
R	0.90	0.99	0.97
MAR	y = 0.94x + 0.03	y = 1.03x + 0.02	y = 1.01x + 0.00
B	0.021 (11.0%)	0.022 (10.6%)	>−0.001 (−0.0%)
MD	0.025 (13.0%)	0.022 (10.3%)	−0.001 (−0.5%)
STD	0.037 (19.5%)	0.013 (6.2%)	0.026 (12.4%)
MAD	0.026 (13.5%)	0.022 (10.3%)	0.005 (2.3%)
RMSD	0.042 (22.4%)	0.026 (12.3%)	0.026 (12.4%)
%Optimal	9.8	12.4	77.5
%Target	28.7	43.1	95.8
%Threshold	50.6	70.4	98.7

Table 12. Summary of the main statistics for the direct validation of C3S V2, GLASS V4, and MCD43A3 C6.1 satellite blue-sky albedo products vs. blue-sky albedo ground values from the REALS dataset in the 2000–2019 period. All pixels were taken into account.

	C3S V2	GLASS V4	MCD43A3 C6.1
N	12067	12067	12067
R	0.63	0.61	0.60
MAR	y = 0.68x + 0.06	y = 0.93x + 0.01	y = 1.05x−0.01
B	0.017 (12.2%)	−0.003 (−2.5%)	−0.003 (−2.5%)
MD	0.021 (14.9%)	<0.001 (−0.1%)	−0.002 (−1.2%)
STD	0.040 (27.8%)	0.043 (32.7%)	0.047 (35.2%)
MAD	0.029 (20.4%)	0.017 (13.2%)	0.017 (13.2%)
RMSD	0.043 (30.4%)	0.043 (32.8%)	0.047 (35.3%)
%Optimal	12.6	20.6	18.1
%Target	24.5	38.3	37.0
%Threshold	46.8	63.1	64.7

Table 13. Summary of the main statistics for the direct validation of C3S V2, GLASS V4, and MCD43A3 C6.1 satellite blue-sky albedo products vs. blue-sky albedo ground values from the REALS dataset in the 2000–2019 period. Only best quality pixels were taken into account.

	C3S V2	GLASS V4	MCD43A3 C6.1
N	4598	4598	4598
R	0.68	0.74	0.76
MAR	y =0.66x + 0.06	y = 0.61x + 0.04	y = 0.65x + 0.04
B	0.014 (9.7%)	−0.008 (−6.2%)	−0.008 (−5.7%)
MD	0.017 (11.7%)	−0.004 (−2.9%)	−0.005 (−3.8%)
STD	0.032 (22.2%)	0.030 (22.3%)	0.029 (21.6%)
MAD	0.024 (16.7%)	0.013 (10.1%)	0.015 (11.3%)
RMSD	0.035 (24.2%)	0.031 (23.2%)	0.030 (22.4%)
%Optimal	16.8	27.5	20.9
%Target	32.2	48.1	43.3
%Threshold	56.8	72.0	73.4

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

Share and Cite

MDPI and ACS Style

Sánchez-Zapero, J.; Martínez-Sánchez, E.; Camacho, F.; Wang, Z.; Carrer, D.; Schaaf, C.; García-Haro, F.J.; Nickeson, J.; Cosh, M. Surface ALbedo VALidation (SALVAL) Platform: Towards CEOS LPV Validation Stage 4—Application to Three Global Albedo Climate Data Records. Remote Sens. 2023, 15, 1081. https://doi.org/10.3390/rs15041081

AMA Style

Sánchez-Zapero J, Martínez-Sánchez E, Camacho F, Wang Z, Carrer D, Schaaf C, García-Haro FJ, Nickeson J, Cosh M. Surface ALbedo VALidation (SALVAL) Platform: Towards CEOS LPV Validation Stage 4—Application to Three Global Albedo Climate Data Records. Remote Sensing. 2023; 15(4):1081. https://doi.org/10.3390/rs15041081

Chicago/Turabian Style

Sánchez-Zapero, Jorge, Enrique Martínez-Sánchez, Fernando Camacho, Zhuosen Wang, Dominique Carrer, Crystal Schaaf, Francisco Javier García-Haro, Jaime Nickeson, and Michael Cosh. 2023. "Surface ALbedo VALidation (SALVAL) Platform: Towards CEOS LPV Validation Stage 4—Application to Three Global Albedo Climate Data Records" Remote Sensing 15, no. 4: 1081. https://doi.org/10.3390/rs15041081

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Menu