High Throughput In vivo Analysis of Plant Leaf Chemical Properties Using Hyperspectral Imaging

Hyperspectral camera and imaging chamber

The hyperspectral imaging system is the part of high throughput phenotyping greenhouse (Scanalyzer3D, LemnaTec GmbH, Aachen, Germany) installed at the University of Nebraska-Lincoln. The hyperspectral camera is an extended VNIR (visible and near infrared), push-broom type imaging spectrometer (Headwall Photonics, Fitchburg, MA, USA). Both plants and the camera remain still during image acquisition. A scanning mirror inside the imaging system is rotated to sequentially expose each line of image from the top to the bottom of the imaging chamber. The camera has a nominal spectral range from 550 to 1,700 nm (the green-red portion of VIS, the entire NIR, and the first part of SWIR). The spectral interval of each image band is 4.77 nm, giving a total of 243 image bands for each hyperspectral image cube. In the spatial domain, the imaging detector array is 320-pixel wide. The mirror's incremental angle at each step is configured such that the pixel size at the scanning (vertical) direction is 5 mm. Each hyperspectral image is consisted of 500 scan lines, which covers the height of the imaging chamber (2,500 mm = 5 × 500). The imaging system is placed in the imaging chamber such that the scan line matches the entire width of the imaging chamber (1,600 mm). Therefore, the image pixel size on the horizontal direction is also 5 mm (1,600/320, meaning square pixels). The imaging chamber is illuminated by two banks of halogen lamps (35W, color temperature 2,600 K), one at the back of the imaging chamber behind the hyperspectral imaging system and the other on the top. The interior view of the hyperspectral imaging chamber during image acquisition is shown in Figure 1, as well as a 3D rendering of the chamber showing its relative dimension. To reiterate the setup of the hyperspectral system: 500 pixels vertical (5 mm per pixel) × 320 pixels horizontal (5 mm per pixel) × 243 spectral bands (4.7 nm wide from 550 to 1,700 nm).

Experiment design and data collection

Sixty maize (Zea mays, B73 inbred) plants and 60 soybean (Glycine max, Thorne variety) plants were used in this study. Seeds were germinated and grown in 5.7 L pots having a diameter of 22 cm and a height of 20 cm filled with germination mix (Fafard). The temperature in the greenhouse was maintained between 25 and 27°C during daytime and 20–22°C during the night time. Relative humidity was maintained at ~60%. The daily light intensity resulting from natural sunlight and the supplemental LED peaked at ~350 μmol/m2/s photosynthetically active radiation. The supplemental LED had a lighting period set to 12 h (7:00 to 19:00 h). Plant growth and data collection occurred in June and July of 2016.

The maize and soybean plants were divided equally into two different experiments. The first was a water limitation experiment including 30 maize and 30 soybean plants. The purpose of this experiment was to create a wide range of plant leaf water contents for the testing of hyperspectral imaging for plant water content measurement. These 60 plants were further divided into two groups: a control group and a water limitation group. For the control group, water was added daily to each pot to field capacity (water potential −33 kilo Pascal). For the water limitation group, water was added daily to each pot to field capacity until 20 Days after Sowing (DAS) for maize and 27 DAS for soybean. This was to ensure that non-stressed plants were established before they were subject to water limitation. After that, only limited amount of water was added to soybean plants (to prevent complete dry out) whereas no water was given to the maize plants.

The remaining 30 maize and 30 soybean plants were used in the second nutrient stress experiment. These maize plants were divided into groups of 10 plants each and were subjected to three different nutrient levels: low, medium, and high. The same procedure was also applied to the soybean plants. The purpose was to create a large range of nutrient levels (in particular nitrogen, phosphorus, and potassium) in plant leaves for the testing of the hyperspectral imaging system. The treatments were administered using a slow release commercial fertilizer (Osmocote Pro 5-6 mo. 17-5-11 with improved micronutrients). The application rate was 20 and 7.5 g per kg of soil for high and medium levels, respectively; while no fertilizer was applied to the low nutrient group.

Data collection was done between 27 and 64 DAS, during which the maize and soybean plants spanned several developmental stages. The plants under water limitation were sampled earlier (until 51 DAS) so that the plants under severe water limitation can be sampled before complete dry out. Most of the plants in the nutrient stress experiment were therefore sampled toward the end of the experiment. This was beneficial since it allowed these plants to grow to larger total biomass and measurable nutrient deficiency was developed for the plants receiving medium or low nutrients.

Data collection and imaging was conducted on 10 different days. On each day, 12 plants were selected. The plants were placed on Scanalyzer3D's conveyor belt and transported to system's imaging chamber where the hyperspectral images were acquired. Immediately following each plant, an empty pot carrier was also sent to the imaging chamber and a blank image was taken as the reference image. This reference image was used to normalize the raw digital numbers in plant images to a common range between 0 and 1. This normalization step helped to account for the short-term and long-term variations in halogen lamps' energy output and imaging detector responses.

After image acquisition, the plants were cut above the soil line to determine the fresh weight (W_fresh) of plant shoot. The harvested plant shoot materials were placed in a walk-in drying oven at 50°C for over 72 h to obtain dry weight (W_dry). The actual drying time was determined by repeatedly weighing the dry plant material every 24 h until two consecutive measurements were found to be equal. Plant water content was calculated WC = (W_fresh − W_dry)/W_fresh × 100%. The dried plant leaves were sent to a commercial lab (Midwest Laboratories, Omaha, NE) where the samples were ground, homogenized, and analyzed to determine the concentration of macronutrients including total nitrogen (N), phosphorus (P), potassium (K), magnesium (Mg), calcium (Ca), and sulfur (S), and micronutrients including sodium (Na), iron (Fe), manganese (Mn), boron (B), copper (Cu), and zinc (Zn). N was analyzed by Dumas method with a LECO FP428 nitrogen analyzer (AOAC method 968.06). All other elements were analyzed with microwave nitric acid digestion followed by inductively coupled plasma spectroscopy (AOAC method 985.01).

Hyperspectral image analysis and data analysis

Raw hyperspectral image data were in 16 bit BIL (Band Interleaved by Line) format. A customized function was written to convert both the raw image data (including both 120 plant images and 120 reference images) into 3D image cubes. The plant image cubes were then divided by the reference image cubes through a band-by-band operation, which converted plant image pixels from the raw intensity numbers to a fractional number (between 0 and 1) analog to leaf reflectance. Two image bands at 705 and 750 nm (I₇₀₅ and I₇₅₀) were extracted from the image cubes to calculate an NDVI image as (I₇₅₀−I₇₀₅)/(I₇₅₀+I₇₀₅). This NDVI image was found very effective to segment plants from the background by setting a universal threshold of 0.20. After segmentation, the binary image was used to extract plant pixels from all image bands along the wavelength. Extracted plant pixels from each image band were then averaged to obtain the average reflectance of the plants. Through these steps, the hyperspectral image was reduced to a 243 data-point spectral curve that represented the average apparent reflectance of each plant from 550 to 1,700 nm. Figure 2 summarizes the image processing procedure and the example hyperspectral images from this study. Image processing was implemented in MATLAB (version 2015b, Mathwork®, Natick, MA) with the Image Processing Toolbox.

Principal component analysis was first implemented on spectral data to detect possible outliers in the spectral space. Partial least squares regression (PLSR) was used to model and predict water content, macronutrient, and micronutrient concentrations for maize and soybean plants from their reflectance spectra. We split the 120 plants into two groups: 60 plants (50%) for model calibration and the other 60 (50%) for independent model validation. The split was done such that the species (maize vs. soybean) and the treatment levels for both water stress and nutrient stress experiments were equally presented in the calibration and validation set. A check was done to ensure there was no significant difference in the response variables between the two sets (Supplementary Table 1), which was important for effective modeling. For model calibration, the scheme of leave-one-out cross validation was employed. Models having as many as 12 latent factors were considered, and the size of the models was determined by choosing the number of factors that gave the minimum root mean squared error of cross validation (RMSE_CV). The following statistics were calculated for model evaluation: Coefficient of Determination (R²) between lab-measured and model predicted values, RMSE (Equation 1), Ratio of Performance to Deviation (RPD, Equation 2), and Mean Absolute Percent Error (MAPE, Equation 3). These statistics were calculated for both cross validation (in model calibration) and validation. PLSR modeling was performed in R statistical environment (R Core Team, 2016) with “chemometrics” package.

N is the number of plants in the calibration or validation set (60); y_i and ŷ_i are the lab-measured and model-predicted values, respectively; SD and mean are the standard deviation and mean of the lab-measured values.

Effects of water and nutrient treatments on plant chemical concentrations

The experimental conditions described above created large variation in plant water content across water treatments. Plant water content ranged from 79.6 to 91.0% for maize and 68.2 to 81.9% for soybean (Figure 3). It can also be seen that plants under water limitation had significantly lower leaf water content than those under control.

Figure 4 shows the range of macronutrient and micronutrient levels observed in maize and soybean leaves under the high, medium, and low nutrient application rates. It can be seen that these traits responded to the nutrient treatment quite differently. For most nutrients, plants grown with higher nutrient rates tended to have higher concentrations of both macronutrients and micronutrients in leaves; even though in several cases, there were no statistically significant differences among all three nutrient application rates (such as Na for either maize or soybean). As mentioned above, our goal was to create wide ranges in macronutrients and micronutrients in plant leaf tissues, and Figure 4 indicates that this goal was successfully achieved for the majority of target nutrients. For example, the N concentration in maize and soybean plants combined ranged from 0.96 to 5.68%. This range was wider than what would be found for maize or soybean under agronomically relevant field conditions, and would benefit PLSR model calibration with hyperspectral imaging.

PLSR modeling of plant leaf chemical traits

Pairwise score plot of the first three principal components (PC1, PC2, and PC3) of the plant spectra from the water limitation experiment is shown in Figure 5A. The plot revealed one potential outlier in the spectral space. Further examination of this maize plant, however, indicated that its spectral and chemical data did not deviate substantially from other samples. This sample was therefore retained in the subsequent analysis. The plot also revealed that maize and soybean plants were largely separated from each other (in PC1 vs. PC2 and PC2 vs. PC3 plots). In addition, plants receiving the different water treatments was also separable (in PC1 vs. PC3 plot), although separation was not as pronounced as plant species. Similarly, Figure 5B shows the pairwise PC plot of plant spectra from the nutrient stress experiment. Still maize and soybean plants were largely separable (again PC1 vs. PC2 and PC2 vs. PC3 plots). However, different from the water treatments, no clear spectral separation was seen among the nutrient stress levels. Overall, it suggested that plant species contributed most to spectral differences, followed by the water treatments and then the nutrient treatments. This had important implications for PLSR modeling, which is discussed later with the results of PLSR modeling.

The results of PLSR analysis (Table 1) show that, among all the plant chemical traits studied, leaf water content can be modeled with the highest accuracy (R² = 0.93 and MAPE = 1.6%) for validation. For macronutrients, N, P, K and S were modeled quite satisfactorily, with their validation R² > 0.80 and MAPE < 15%. Mg and Ca were modeled with moderate accuracy, with their validation R² around 0.70 and MAPE around 15%. Compared to macronutrients, micronutrients were in general modeled with lower accuracy. Cu and Zn were modeled most successfully in this group, with validation R² of 0.86 and 0.73 (MAPE of 20.8 and 16.1%), respectively. Modeling of Fe and Mn showed somewhat lower accuracy, with validation R² slightly greater than 0.60 (MAPE of 13.7 and 17.3%). Models of Na and B showed the lowest prediction accuracy among all variables tested, with validation R² lower than 0.30. MAPE of Na was 49.5%, much higher than all other variables.

In analytical chemistry and chemometric modeling, RPD is widely used as a criterion to evaluate the usefulness of prediction models by normalizing model's RMSE with the standard deviation of the reference value. Models with an RPD > 2.0 are generally considered good models, with RPD > 3.0 indicative of quantitative analysis. Models with RPD between 1.5 and 2.0 are considered fair in a quantitative sense, but they can be useful for qualitative screening and have potential for improvement (Chang et al., 2001; Fearn, 2002). The majority of the models developed in this study had RPD between 1.5 and 4.0 (except for Na and B). Therefore, these models would be useful for the screening purpose of leaf biochemical traits. In many circumstances of plant breeding and phenotypic scoring, it is not necessary to precisely predict traits. Rather, the goal often is to rapidly screen out the superior lines from a population of hundreds or even thousands of lines. In this sense, the models developed from hyperspectral imaging in this study can be used as a practical tool for rapid screening of plant leaf biochemical traits.

Practical considerations to use hyperspectral imaging for high throughput plant phenotyping

Hyperspectral imaging is significantly different from some other imaging modalities such as RGB and thermal infrared (TIR) imaging. RGB is most widely used to extract green plant pixels which in turn are counted to estimate plant biomass. In this process, the intensities of the R G B components of plant pixels are not necessarily of interest to the users, as long as those plant pixels can be successfully segmented from background pixels. TIR imaging involves the detection of emitted electromagnetic energy (at much longer wavelengths of 8–14 μm) from plant leaves and therefore no illumination is needed. For these reasons, effort was not usually emphasized to radiometrically calibrate the RGB and TIR cameras.

On the other hand, the use of hyperspectral imaging to predict chemical properties of plants relies on the accurate measurement of reflectance of plant pixels at each waveband. This means that radiometric calibration is a key step in the collection of informative hyperspectral images. Radiometric calibration seeks to account for the short- and long-term variations in the spectral output intensity of light sources (halogen lamps) and imaging detectors' response and dark current. Ideally, a calibration after each plant measurement should be implemented to obtain the highest performance. This level of calibration has been implemented for hyperspectral imaging applications in some other fields including quality assessment of food products (Naganathan et al., 2008) and fruits and vegetables (Qin and Lu, 2008). However, more frequent calibrations create a tradeoff between image quality and analysis throughput, which are two important factors to consider in high throughput phenotyping.

Whereas, image acquisition by RGB is almost instantaneous, the time needed for hyperspectral image acquisition is substantially longer. With our system, acquiring one hyperspectral image cube takes ~2 min (due to line-by-line scanning mechanism of image formation). If a reference image is also acquired each time a plant is imaged, 4–5 min are therefore needed to scan one plant. This represents a significant rate limiting factor for high throughput phenotyping. With this speed, a work shift of 12 h continuous imaging can only scan about 180 plants. This is obviously not adequate for many large population studies which usually involve several hundreds of plants. Therefore, from an operational and logistics standpoint, system characterization is needed so that the researchers would know how often a calibration should be run to strike a balance between radiometric accuracy and imaging throughput.

Whole plant vs. multiple predictions per plant

In this study, we treated the plants as a whole. All leaves from the plant were harvested, dried and thoroughly homogenized and a portion of it was sent to the lab for analysis. At the image analysis side, all pixels of the plant were combined to obtain an average reflectance at each wavelength. Therefore, the prediction models developed in our study specified the correlations between spectra and the chemical traits of plants at the whole plant level (i.e., one prediction per plant per image time). However, it is possible to make spatially resolved predictions for the plant, because a complete spectrum can be extracted from every pixel of the plant. This would provide a tool to quantify the spatial distribution of these chemical properties within the plant. If such quantifications could be made at multiple time points along plants' life cycle, it would be possible to elucidate (1) how the macronutrient and micronutrient elements are taken up from the soils, and (2) the translocation of mobile nutrients (such as N, P, K) among plant tissues as plants develop. Currently, there is significant variation in image quality throughout the chamber as a result of non-uniform lighting (even after normalization), which makes accurate pixel by pixel prediction challenging. However, this is a solvable problem; and, in the future it should be possible to improve the lighting uniformity inside the chamber and retrieve the true reflectance of plant leaves from each plant pixel, which would enable spatially resolved predictions of plant leaf composition.

Finally, the positive results obtained in this study, indicate that it may also be feasible to use hyperspectral imaging data to phenotype for other plant chemical properties. For example, there is a wide interest in breeding improved accessions of energy crops for dedicated biofuel production (for example, biomass sorghum). One of the greatest challenges is to breed for optimized biomass compositions (in particular lignin content which simultaneously determines standability, resistance to fungi and diseases during growth, and to chemical and microbial pretreatments for biomass conversion) without compromising yield or abiotic or biotic stress tolerance (Rooney et al., 2007; Yuan et al., 2008). The lack of a non-destructive, in vivo screening tool that can measure biomass composition rapidly is a perceived bottleneck for the effective utilization of high throughput genomics-assisted breeding efforts in sorghum. Our results suggest that hyperspectral imaging could be an effective tool to fill in that technological gap in plant breeders' toolkit. It is quite possible that hyperspectral imaging can predict the chemical compositions of cell wall (such as lignin, cellulose, hemicellulose, and ash) from both plant leaves and stems.

Conflict of interest statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The reviewer TV and handling Editor declared their shared affiliation, and the handling Editor states that the process met the standards of a fair and objective review.