Abstract
Fruit counting and tracking is a crucial component of fruit harvesting and yield forecasting applications within horticulture. We present a novel multi-object, multi-class fruit tracking system to count fruit from image sequences. We first train a recurrent neural network (RNN) comprised of a feature extractor stem and two heads for re-identification and maturity classification. We apply the network to detected fruits in image sequences and utilise the output of both network heads to maintain track consistency and reduce intra-class false positives between maturity stages. The counting-by-tracking system is evaluated by comparing with a popular detect-to-track architecture and against manually labelled tracks (counts). Our proposed system achieves a mean average percentage error (MAPE) of 3% (L1 loss = 7) improving on the baseline multi-object tracking approach which obtained an MAPE of 21% (L1 loss = 41). Validating this approach for use in horticulture.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bellocchio, E., Ciarfuglia, T.A., Costante, G., Valigi, P.: Weakly supervised fruit counting for yield estimation using spatial consistency. IEEE Robot. Autom. Lett. 4(3), 2348–2355 (2019). https://doi.org/10.1109/LRA.2019.2903260
Bewley, A., Ge, Z., Ott, L., Ramos, F., Upcroft, B.: Simple online and realtime tracking. CoRR abs/1602.00763 (2016). http://arxiv.org/abs/1602.00763
Bochinski, E., Eiselein, V., Sikora, T.: High-speed tracking-by-detection without using image information. In: International Workshop on Traffic and Street Surveillance for Safety and Security at IEEE AVSS 2017, Lecce, Italy, August 2017. http://elvera.nue.tu-berlin.de/files/1517Bochinski2017.pdf
Chen, S.W., et al.: Counting apples and oranges with deep learning: a data-driven approach. IEEE Robot. Autom. Lett. 2(2), 781–788 (2017). https://doi.org/10.1109/LRA.2017.2651944
Girshick, R.B., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. CoRR abs/1311.2524 (2013). http://arxiv.org/abs/1311.2524
Grimstad, L., From, P.J.: The Thorvald II agricultural robotic system. Robotics 6(4) (2017). https://doi.org/10.3390/robotics6040024. https://www.mdpi.com/2218-6581/6/4/24
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015). http://arxiv.org/abs/1512.03385
Kirk, R., Cielniak, G., Mangan, M.: L*a*b*fruits: a rapid and robust outdoor fruit detection system combining bio-inspired features with one-stage deep learning networks. Sensors 20(1) (2020). https://doi.org/10.3390/s20010275. https://www.mdpi.com/1424-8220/20/1/275
Leal-Taixé, L., Milan, A., Schindler, K., Cremers, D., Reid, I.D., Roth, S.: Tracking the trackers: an analysis of the state of the art in multiple object tracking. CoRR abs/1704.02781 (2017). http://arxiv.org/abs/1704.02781
Lin, T., Goyal, P., Girshick, R.B., He, K., Dollár, P.: Focal loss for dense object detection. CoRR abs/1708.02002 (2017). http://arxiv.org/abs/1708.02002
Liu, X., et al.: Monocular camera based fruit counting and mapping with semantic data association. IEEE Robot. Autom. Lett. 4(3), 2296–2303 (2019). https://doi.org/10.1109/LRA.2019.2901987
Liu, X., et al.: Robust fruit counting: combining deep learning, tracking, and structure from motion. CoRR abs/1804.00307 (2018). http://arxiv.org/abs/1804.00307
Mekhalfi, M.L., et al.: Vision system for automatic on-tree kiwifruit counting and yield estimation. Sensors 20(15) (2020). https://doi.org/10.3390/s20154214. https://www.mdpi.com/1424-8220/20/15/4214
Milan, A., Leal-Taixé, L., Reid, I., Roth, S., Schindler, K.: MOT16: a benchmark for multi-object tracking. arXiv:1603.00831 [cs], March 2016
Ning, G., Zhang, Z., Huang, C., He, Z., Ren, X., Wang, H.: Spatially supervised recurrent convolutional neural networks for visual object tracking. CoRR abs/1607.05781 (2016). http://arxiv.org/abs/1607.05781
Redmon, J., Farhadi, A.: YOLOv3: an incremental improvement. CoRR abs/1804.02767 (2018). http://arxiv.org/abs/1804.02767
Ren, S., He, K., Girshick, R.B., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. CoRR abs/1506.01497 (2015). http://arxiv.org/abs/1506.01497
Santos, T.T., de Souza, L.L., dos Santos, A.A., Avila, S.: Grape detection, segmentation, and tracking using deep neural networks and three-dimensional association. Comput. Electron. Agric. 170, 105247 (2020). https://doi.org/10.1016/j.compag.2020.105247. https://www.sciencedirect.com/science/article/pii/S0168169919315765
Wojke, N., Bewley, A., Paulus, D.: Simple online and realtime tracking with a deep association metric. CoRR abs/1703.07402 (2017). http://arxiv.org/abs/1703.07402
Acknowledgement
This work was partially funded by the RASberry project at the University of Lincoln in affiliation with the Collaborative Training Partnershipfor Fruit Crop Research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Kirk, R., Mangan, M., Cielniak, G. (2021). Robust Counting of Soft Fruit Through Occlusions with Re-identification. In: Vincze, M., Patten, T., Christensen, H.I., Nalpantidis, L., Liu, M. (eds) Computer Vision Systems. ICVS 2021. Lecture Notes in Computer Science(), vol 12899. Springer, Cham. https://doi.org/10.1007/978-3-030-87156-7_17
Download citation
DOI: https://doi.org/10.1007/978-3-030-87156-7_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87155-0
Online ISBN: 978-3-030-87156-7
eBook Packages: Computer ScienceComputer Science (R0)