Towards Automated Analysis of Grain Spikes in Greenhouse Images Using Neural Network Approaches: A Comparative Investigation of Six Methods
<p>Example of spike detection and segmentation in greenhouse wheat images: (<b>a</b>) RGB visible light image of a matured wheat plant, (<b>b</b>) detection of spikes by rectangular bounding boxes, (<b>c</b>) pixel-wise segmentation of spikes.</p> "> Figure 2
<p>Examples of spike ROIs from the test image set: (<b>a</b>–<b>d</b>) emergent, partially visible spikes vs. (<b>e</b>,<b>f</b>) matured spikes (GSGC, YSYC).</p> "> Figure 3
<p>Comparison of performance of Faster-RCNN vs. YOLOv3: (<b>a</b>) Faster-RCNN in-training loss and average precision (AP) versus iterations of Faster-RCNN. At 6000 iterations, the binary cross entropy loss is minimized with high AP, and further training increases the loss and AP altogether. (<b>b</b>) YOLOv3 in-training binary cross entropy loss and average precision versus the epoch number.</p> "> Figure 4
<p>The detection of grain spikes using pre-trained Faster-RCNN (green bounding boxes) and YOLO (pink bounding boxes) DNN: (<b>a</b>–<b>d</b>) examples of detection of top wheat spikes, (<b>e</b>–<b>h</b>) examples of detection of emergent wheat spikes White bounding boxes indicate spikes that were not detected by the DNN classifier in this particular image.</p> "> Figure 5
<p>Confusion matrix corresponding to IoU = 0.75 for (<b>a</b>) Faster-RCNN and (<b>b</b>) YOLOv3.</p> "> Figure 6
<p>In-training accuracy of U-Net and DeepLabv3+ versus epochs: (<b>a</b>) Dice coefficient (red line) and binary cross-entropy (green line) reached pleateau around 35 epochs. The training was also validated by Dice coefficient (light sea-green line) and loss (purple line) to avoid overfitting. (<b>b</b>) Training of DeepLabv3+ is depicted as function of mean IoU and net loss. The loss converge around 1200 epochs.</p> "> Figure 7
<p>Examples of U-Net and DeepLabv3+ segmentation of spike images: (<b>a</b>) original test images, (<b>b</b>) ground truth binary segmentation of original images, and segmentation results predicted by (<b>c</b>) U-Net and (<b>d</b>) DeepLabv3+, respectively. The predominant inaccuracies in both NN models are associated with boundary pixels of spike followed false positive.</p> "> Figure 8
<p>Examples of application of Faster-RCNN trained on data set in <a href="#sensors-21-07441-t002" class="html-table">Table 2</a> for the detection of spikes of Central European wheat cultivars in images with different (white) background: (<b>a</b>) DNN failed to detect some spikes in the side view image, (<b>b</b>) early emergent spikes and some matured spike in the top view remained undetected.</p> "> Figure 9
<p>(<b>a</b>–<b>d</b>) shows detection examples of barley and rye spikes. White bounding boxes indicate spikes that were not detected by the DNN classifier in this particular image.</p> "> Figure 10
<p>Overlapping/occluding spikes in barley/rye dataset: (<b>a</b>–<b>c</b>) Faster-RCNN failed to detect overlapping spikes as separate objects in the majority of cases and (<b>d</b>–<b>f</b>) YOLOv3, sometimes managed to separate occluding spikes as in (<b>d</b>).</p> "> Figure 11
<p>Screenshot of the SpikeApp tool for demonstration of DNN/ANN performance on detection, segmentation and phenotyping of grain spikes. On the left-hand side of the tool, the control and parameter section can be found, while on the right, the output area located. On the right, below the images, a table with the extracted features for all images is provided to the user for quick feedback.</p> ">
Abstract
:1. Introduction
2. Methods
2.1. Image Acquisition
2.2. Data Set Preparation
2.3. Spike Detection DNN Models
2.3.1. Single Shot Multibox Detector
2.3.2. Faster-RCNN
2.3.3. YOLOv3 and YOLOv4
2.4. Spike Segmentation Models
2.4.1. Shallow Artificial Neural Network
2.4.2. U-Net
2.4.3. DeepLabv3+
2.5. Evaluation of Spike Detection Models
2.6. Evaluation of Spike Segmentation Models
3. Results
- First, the performance of neural network models for detection and segmentation of wheat spikes in side view greenhouse images was investigated.
- Then, the NN models trained on side view images of wheat plants were applied to other crops (barley and rye).
- Finally, spike detection and segmentation models trained on side view images of wheat plants were validated by application to side and top view images of other, more bushy wheat cultivars acquired from another greenhouse facility.
3.1. Spike Detection Experiments
3.1.1. Spike Detection Using SSD
3.1.2. Spike Detection Using Faster-RCNN
3.1.3. Spike Detection Using YOLOv3/v4
3.2. Spike Segmentation Experiments
3.2.1. Spike Segmentation Using ANN
3.2.2. Spike Segmentation Using U-Net
3.2.3. Spike Segmentation Using DeepLabv3+
3.3. Domain Adaptation Study
- Barley and rye side view images that were acquired with the optical setup, including blue background photo chamber, viewpoint and lighting conditions as used for wheat cultivars. This image set is given by 37 images (10 barley and 27 rye) RGB visible light images containing 111 spikes in total. The longitudinal lengths of spikes in barley and rye were greater than those of wheat by a few centimeters (based on visual inspection).
- Two bushy Central European wheat cultivars (42 images, 21 from each cultivar) imaged using LemnaTec-Scanalyzer3D (LemnaTec GmbH, Aachen, Germany) at the IPK Gatersleben in side view, having on average 3 spikes per plant Figure 8a, and top view Figure 8b comprising 15 spikes in 21 images. A particular challenge of this data set is that the color fingerprint of spikes is very much similar to the remaining plant structures.
3.3.1. Evaluation Tests with New Barley/Rye Images
3.3.2. Evaluation Tests with Images from Another Phenotyping Facility
3.4. SpikeApp Demo Tool
4. Discussion
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Guo, Z.; Zhao, Y.; Röder, M.S.; Reif, J.C.; Ganal, M.W.; Chen, D.; Schnurbusch, T. Manipulation and Prediction of Spike Morphology Traits for the Improvement of Grain Yield in Wheat. Sci. Rep. 2018, 8, 1–10. [Google Scholar] [CrossRef] [PubMed]
- Minervini, M.; Scharr, H.; Tsaftaris, S.A. Image Analysis: The New Bottleneck in Plant Phenotyping. IEEE Signal Proc. Mag. 2015, 32, 126–131. [Google Scholar] [CrossRef] [Green Version]
- Hasan, M.M.; Chopin, J.P.; Laga, H.; Miklavcic, S.J. Detection and Analysis of Wheat Spikes Using Convolutional Neural Networks. Plant Methods 2018, 14, 100. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Tan, C.; Zhang, P.; Zhang, Y.; Zhou, X.; Wang, Z.; Du, Y.; Mao, W.; Li, W.; Wang, D.; Guo, W. Rapid Recognition of Field-grown Wheat Spikes Based on a Superpixel Segmentation Algorithm Using Digital Images. Front. Plant Sci. 2020, 11, 259. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Alharbi, N.; Zhou, J.; Wang, W. Automatic Counting of Wheat Spikes From Wheat Growth Images. In Proceedings of the ICPRAM, Funchal, Portugal, 16–18 January 2018; pp. 346–355. [Google Scholar]
- Costa, C.; Schurr, U.; Loreto, F.; Menesatti, P.; Carpentier, S. Plant Phenotyping Research Trends, a Science Mapping Approach. Front. Plant Sci. 2019, 9, 1933. [Google Scholar] [CrossRef] [Green Version]
- Pieruschka, R.; Schurr, U. Plant Phenotyping: Past, Present, and Future. Plant Phenomics 2019, 2019, 7507131. [Google Scholar] [CrossRef]
- Grillo, O.; Blangiforti, S.; Venora, G. Wheat Landraces Identification through Glumes Image Analysis. Comput. Electron. Agric. 2017, 141, 223–231. [Google Scholar] [CrossRef]
- Bi, K.; Jiang, P.; Li, L.; Shi, B.; Wang, C. Non-destructive Measurement of Wheat Spike Characteristics Based on Morphological Image Processing. Trans. Chin. Soc. Agric. Eng. 2010, 26, 212–216. [Google Scholar]
- Misra, T.; Arora, A.; Marwaha, S.; Chinnusamy, V.; Rao, A.R.; Jain, R.; Sahoo, R.N.; Ray, M.; Kumar, S.; Raju, D. Spikesegnet—A Deep Learning Approach Utilizing Encoder-decoder Network with Hourglass for Spike Segmentation and Counting in Wheat Plant from Visual Imaging. Plant Methods 2020, 16, 1–20. [Google Scholar] [CrossRef]
- Pound, M.P.; Atkinson, J.A.; Wells, D.M.; Pridmore, T.P.; French, A.P. Deep Learning For Multi-task Plant Phenotyping. In Proceedings of the IEEE International Conference on Computer Vision Workshops, Venice, Italy, 22–29 October 2017. [Google Scholar]
- Qiongyan, L.; Cai, J.; Berger, B.; Okamoto, M.; Miklavcic, S.J. Detecting Spikes of Wheat Plants Using Neural Networks with Laws Texture Energy. Plant Methods 2017, 13, 83. [Google Scholar] [CrossRef] [Green Version]
- Narisetti, N.; Neumann, K.; Röder, M.S.; Gladilin, E. Automated Spike Detection In Diverse European Wheat Plants Using Textural Features And The Frangi Filter in 2D Greenhouse Images. Front. Plant Sci. 2020, 11, 666. [Google Scholar] [CrossRef]
- Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet Classification With Deep Convolutional Neural Networks. Adv. Neural Inf. Process. Syst. 2012, 25, 1097–1105. [Google Scholar] [CrossRef]
- Everingham, M.; Williams, C.K. The Pascal Visual Object Classes Challenge 2007 (Voc2007) Results. Available online: http://host.robots.ox.ac.uk/pascal/VOC/voc2007/ (accessed on 8 October 2021).
- Deng, J.; Dong, W.; Socher, R.; Li, L.J.; Li, K.; Fei-Fei, L. Imagenet: A Large-scale Hierarchical Image Database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 248–255. [Google Scholar]
- Tzutalin. LabelImg. Free Software: MIT License. 2015. Available online: https://github.com/tzutalin/labelImg (accessed on 1 December 2019).
- Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.Y.; Berg, A.C. SSD: Single Shot Multibox Detector. In European Conference on Computer Vision; Springer: Berlin/Heidelberg, Germany, 2016; pp. 21–37. [Google Scholar]
- Uijlings, J.R.; Van De Sande, K.E.; Gevers, T.; Smeulders, A.W. Selective Search For Object Recognition. Int. J. Comput. Vis. 2013, 104, 154–171. [Google Scholar] [CrossRef] [Green Version]
- Everingham, M.; Van Gool, L.; Williams, C.K.; Winn, J.; Zisserman, A. The Pascal Visual Object Classes (Voc) Challenge. Int. J. Comput. Vis. 2010, 88, 303–338. [Google Scholar] [CrossRef] [Green Version]
- Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-cnn: Towards Real-time Object Detection With Region Proposal Networks. Adv. Neural Inf. Process. Syst. 2015, 28, 91–99. [Google Scholar] [CrossRef] [Green Version]
- Redmon, J.; Farhadi, A. Yolov3: An Incremental Improvement. arXiv 2018, arXiv:1804.02767. [Google Scholar]
- Bochkovskiy, A.; Wang, C.Y.; Liao, H.Y.M. YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv 2020, arXiv:2004.10934. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
- Chang, T.; Kuo, C.C. Texture Analysis And Classification With Tree-structured Wavelet Transform. IEEE Trans. Image Process. 1993, 2, 429–441. [Google Scholar] [CrossRef] [Green Version]
- Christodoulou, C.I.; Pattichis, C.S.; Pantziaris, M.; Nicolaides, A. Texture-based Classification of Atherosclerotic Carotid Plaques. IEEE Trans. Med. Imaging 2003, 22, 902–912. [Google Scholar] [CrossRef]
- Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional Networks For Biomedical Image Segmentation. In Proceedings of the International Conference on Medical image computing and Computer-Assisted Intervention, Munich, Germany, 5–9 October 2015; Springer: Berlin/Heidelberg, Germany, 2015; pp. 234–241. [Google Scholar]
- Chen, L.C.; Zhu, Y.; Papandreou, G.; Schroff, F.; Adam, H. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. In Proceedings of the ECCV, Munich, Germany, 8–14 September 2018. [Google Scholar]
- Holschneider, M.; Kronland-Martinet, R.; Morlet, J.; Tchamitchian, P. A Real-time Algorithm for Signal Analysis with the Help of the Wavelet Transform. In Wavelets; Springer: Berlin/Heidelberg, Germany, 1990; pp. 286–297. [Google Scholar]
- Chen, L.C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A.L. Semantic Image Segmentation With Deep Convolutional Nets And Fully Connected CRFs. arXiv 2014, arXiv:1412.7062. [Google Scholar]
- Huang, J.; Rathod, V.; Sun, C.; Zhu, M.; Korattikara, A.; Fathi, A.; Fischer, I.; Wojna, Z.; Song, Y.; Guadarrama, S. Speed/Accuracy Trade-offs For Modern Convolutional Object Detectors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 7310–7311. [Google Scholar]
- Kingma, D.P.; Ba, J. Adam: A Method For Stochastic Optimization. arXiv 2014, arXiv:1412.6980. [Google Scholar]
- Salehi, S.S.M.; Erdogmus, D.; Gholipour, A. Tversky loss Function for Image Segmentation using 3D Fully Convolutional Deep Networks. In Proceedings of the International Workshop on Machine Learning in Medical Imaging, Quebec City, QC, Canada, 10 September 2017; pp. 379–387. [Google Scholar]
- Azad, R.; Asadi-Aghbolaghi, M.; Fathy, M.; Escalera, S. Attention deeplabv3+: Multi-level context attention mechanism for skin lesion segmentation. In Proceedings of the European Conference on Computer Vision, Glasgow, UK, 23–28 August 2020; pp. 251–266. [Google Scholar]
Publications | Methods | Measure | Value | Application |
---|---|---|---|---|
Tan et al. (2020) | Statistical analysis of morpho-colorimetric descriptors | AP | 0.90 | field |
Alharbi et al. (2018) | Transforming raw plant images using color index of vegetation extraction (CIVE) | AP | 0.90 | field |
Bi et al. (2010) | 3-layer neural network to extract spike traits | AP | 0.88 | field |
Misra et al. (2020) | Two cascaded feature networks | F1 | 0.97 | field |
Hasan et al. (2018) | R-CNN | AP/F1 | 0.93/0.95 | field |
Pound et al. (2017) | DNN | F1 | 0.89 | field |
Grillo et al. (2017) | Elliptic Fourier descriptors | AP | 0.89 | field |
Qiongyan et al. (2017) | Artificial (shallow) neural network (ANN) approach to spike segmentation | A | 0.92 | greenhouse |
Narisetti et al. (2020) | Extension of the ANN approach by Frangi line filter | A | 0.98 | greenhouse |
Genotypes | Cultivar | Images |
---|---|---|
1 | Avenue | 45 |
2 | Elly | 14 |
3 | IS Spirella | 32 |
4 | Amerigo | 13 |
5 | Manager | 23 |
6 | Pobeda | 24 |
7 | Jindra | 23 |
8 | Izvor | 29 |
9 | Timber | 12 |
10 | Ilona | 22 |
11 | Remaining cultivars | 55 |
Total | 292 |
Training Set | ||||||
---|---|---|---|---|---|---|
Total Images | GSGC | YSYC | Non-Spike | Test Set | View | Resolution |
292 | 203 | 27 | 4 | 58 | side | 2560 × 2976 |
Detection DNNs | Backbone | Training Set/Aug. | |||
---|---|---|---|---|---|
SSD | Inception resnet v2 | 234/none | 0.780 | 0.551 | 0.470 |
YOLOv3 | Darknet53 | 234/none | 0.941 | 0.680 | 0.604 |
YOLOv4 | CSPDarknet53 | 234/yes | 0.941 | 0.700 | 0.610 |
Faster-RCNN | Inception v2 | 234/none | 0.950 | 0.822 | 0.660 |
Top Spikes: 80 | Occluded/Emergent: 27 | Inner Spikes: 45 | |||||||
---|---|---|---|---|---|---|---|---|---|
Methods | Pr | Pr | Pr | ||||||
SSD | 0.81 | 0.910 | 0.620 | 0.59 | 0.650 | 0.301 | 0.70 | 0.650 | 0.320 |
YOLOv3 | 0.97 | 0.999 | 0.708 | 0.74 | 0.750 | 0.450 | 0.96 | 0.890 | 0.500 |
YOLOv4 | 0.99 | 0.999 | 0.700 | 0.74 | 0.805 | 0.550 | 0.96 | 0.880 | 0.520 |
Faster-RCNN | 0.99 | 0.999 | 0.750 | 0.79 | 0.800 | 0.532 | 0.96 | 0.910 | 0.570 |
Segmentation Model | Backbone | Training Set/Aug. | aDC/m.F1 | Jaccard Index |
---|---|---|---|---|
ANN | – | 234/none | 0.760 | 0.610 |
U-Net | VGG-16 | 384/150 | 0.906 | 0.840 |
DeepLabv3+ | ResNet101 | 298/43 | 0.935 | 0.922 |
Barley/Rye Dataset | Bushy Wheat Cultivars | |||||||||
---|---|---|---|---|---|---|---|---|---|---|
Methods | Backbone | aDC | F1barley | AP0.5:barley | F1rye | AP0.5:rye | F1top | APtop | F1side | APside |
YOLOv3 | Darknet53 | - | 0.91 | 0.850 | 0.99 | 0.870 | 0.15 | 0.100 | 0.25 | 0.233 |
YOLOv4 | CSPDarknet53 | - | 0.92 | 0.880 | 0.99 | 0.904 | 0.20 | 0.140 | 0.30 | 0.240 |
Faster-RCNN | Inception v2 | - | 0.80 | 0.690 | 0.79 | 0.650 | 0.28 | 0.205 | 0.55 | 0.410 |
U-Net | VGG-16 | 0.310 | - | - | - | - | - | - | - | - |
DeepLabv3+ | ResNet101 | 0.430 | - | - | - | - | - | - | - | - |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Ullah, S.; Henke, M.; Narisetti, N.; Panzarová, K.; Trtílek, M.; Hejatko, J.; Gladilin, E. Towards Automated Analysis of Grain Spikes in Greenhouse Images Using Neural Network Approaches: A Comparative Investigation of Six Methods. Sensors 2021, 21, 7441. https://doi.org/10.3390/s21227441
Ullah S, Henke M, Narisetti N, Panzarová K, Trtílek M, Hejatko J, Gladilin E. Towards Automated Analysis of Grain Spikes in Greenhouse Images Using Neural Network Approaches: A Comparative Investigation of Six Methods. Sensors. 2021; 21(22):7441. https://doi.org/10.3390/s21227441
Chicago/Turabian StyleUllah, Sajid, Michael Henke, Narendra Narisetti, Klára Panzarová, Martin Trtílek, Jan Hejatko, and Evgeny Gladilin. 2021. "Towards Automated Analysis of Grain Spikes in Greenhouse Images Using Neural Network Approaches: A Comparative Investigation of Six Methods" Sensors 21, no. 22: 7441. https://doi.org/10.3390/s21227441