Development and Analysis of a CNN- and Transfer-Learning-Based Classification Model for Automated Dairy Cow Feeding Behavior Recognition from Accelerometer Data
<p>Top-view image of the research barn acquired by one of the cameras.</p> "> Figure 2
<p>Component of the location and acceleration measuring system installed in a barn: RuuviTag inside a protecting plastic box (<b>a</b>), tag on the cow collar (<b>b</b>) and receiving station installed on a barn structure (<b>c</b>) marked by red circles.</p> "> Figure 3
<p>Illustration of the missing samples (value 0) in the recorded acceleration data (dots).</p> "> Figure 4
<p>CNN2 architecture.</p> "> Figure 5
<p>F1 score of the model CNN2 trained using randomly initialized model weights (CNN2) and by the transfer learning (CNN2 TL) for the window sizes of 60 s, depending on the training dataset size measured in training samples taken from the original dataset for the average F1 (<b>a</b>), feeding (<b>b</b>), rumination (<b>c</b>) and other behavior (<b>d</b>). The corresponding actual data after augmentation and balancing are: 336,…;3192, 13,878,…;132,762. The error bars represent the STD for 10-fold validation.</p> "> Figure 6
<p>Performance of the tested models, CNN2 and CNN4, trained using randomly initialized model weights and by transfer learning (CNN2 TL and CNN4 TL), depending on the window size for the average F1 (<b>a</b>), feeding (<b>b</b>), rumination (<b>c</b>) and other behavior (<b>d</b>). The error bars represent the STD for 10-fold validation.</p> ">
Abstract
:1. Introduction
2. Materials and Methods
2.1. Barn Study Area and Monitored Cows
2.2. System Design
2.3. Data Collection and Labeling
2.4. Data Processing
2.5. Tested Classifying Models
2.6. Analysis on the Effect of Training Dataset Size
2.7. Accuracy Evaluation
3. Results
4. Discussion
5. Conclusions
Supplementary Materials
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
Appendix A
Publication | Behavior Types | Interval Length, Sampling Rate | Number of Features | Method | Accuracy (F1) | Animals, Period, Barns |
---|---|---|---|---|---|---|
Achour, 2019 [33] | S, L, transition | 3–10 s, 1–4 Hz | 2 | DT | 99% | 8, 0.25, 1 |
Arcidiacono, 2017 [16] | F, S | 5 s, 4 Hz | 1 | DT | 93.3% | 5, 5 h, 1 |
Barwick, 2018 [34] | G, W, S, L | 10 s, 12 Hz | 12 | Quadratic discriminant analysis | 5, 2.5 h, 1 | |
Benaissa, 2017 [35] | F, Ru, other activity | 60 s, 10 Hz | 9 | DT, SVM | 94.4% | 10, 6 h, 1 |
Dutta, 2015 [36] | G, searching, W, Ru, Re, scratching | 5 s, 10 Hz | 9 | probabilistic principal components analysis, fuzzy C means, self-organizing map | 89% | |
Eerdekens, 2020 (horses) [37] | S, W, trot, canter, roll, paw, flank watching | 2.1 s, 25 Hz | CNN | 97.84% | ||
Kaler, 2019 (sheep) [38] | W, S, L | 7 s, 16 Hz | 16 | RF | 80% | 18, 1.6 |
Li, C., 2021 [12] | F, W, salting, Ru, Re | 10 s, 25 Hz | CNN | 94.4% | 6, 6 h, 1 | |
Pavlovic, 2021 [10] | F, Ru, Re | 10 Hz, 90 s | CNN | 82% | 18, 6–18 d, 1 | |
Pavlovic, 2022 [39] | F, Ru, Re | 10 Hz, 90 s | Hidden Markov model, LDA, partial least squares discriminant analysis | 83% | 18, 6–18 d, 1 | |
Peng, 2019 [11] | F, L, Ru, licking salt, moving, social licking and head butt | 3.2–12.8 s, 20 Hz | RNN with LSTM, CNN | 88.7% | 6, ? | |
Rahman, 2018 [40] | G, S, Re, Ru | 200 samples, 12 Hz | 6 | Majority voting, WEKA | ?, ? | |
Rayas-Amor, 2017 [3] | G, R | 30 s | 2 | Linear regression | 96.1(R2) | 7, 9 |
Riaboff, 2020 [15] | G, W, Ru, Re | 10 s, | Extreme boosting algorithm, Adaboost, SVM, RF | 98% accuracy | 86, 57 h, 4 | |
Shen, 2019 [41] | F, Ru, O | 256 samples, 5 Hz | 30 | K-nearest neighbor, SVM, PNN | 92.4% | 5, ? |
Simanungkalit, 2021 [42] | Licking, F, S, L | 10 s, 25 Hz | 8 | DT, RF, KNN, SVM | 95–99% accuracy | 4, 3.5 d |
Tian, 2021 [28] | F, Ru, running, Re, head-shaking, drinking, W | ?, 12.5 Hz | 9 | KNN, RF, KNN-RF fusion | 99.34% | 20, 3, 1 |
Vázquez Diosdado, 2015 [43] | F, S, L | 300 s, 50 Hz | DT, SVM | 91.7% | 6, 1.5 | |
Vázquez Diosdado, 2019 (sheep) [44] | W, S, L | 7 s, 16 Hz | 1 | k-means, KNN | 60.4% | 26, 39 |
Walton, 2019 (sheep) [45] | 5–7 s, 16–32 Hz | 44 | RF | 91–97% | ||
Wang, 2018 [46] | F, L, S, W | 5 s, 1 Hz | Adaptive boosting algorithm | 75% | 5, 25 h | |
Wang, 2020 [47] | Estrus | 0.5–1.5 h, 1 Hz | KNN, back-propagation neural network, LDA, classification and regression tree | 78.6–97.5% | 12, 12 d | |
Williams, 2019 [48] | G, Re and W | 13 ML algorithms | 93% | 40, 0.25 |
References
- García, R.; Aguilar, J.; Toro, M.; Pinto, A.; Rodríguez, P. A systematic literature review on the use of machine learning in precision livestock farming. Comput. Electron. Agric. 2020, 179, 105826. [Google Scholar] [CrossRef]
- Borchers, M.R.; Chang, Y.M.; Proudfoot, K.L.; Wadsworth, B.A.; Stone, A.E.; Bewley, J.M. Machine-learning-based calving prediction from activity, lying, and ruminating behaviors in dairy cattle. J. Dairy Sci. 2017, 100, 5664–5674. [Google Scholar] [CrossRef]
- Rayas-Amor, A.A.; Morales-Almaráz, E.; Licona-Velázquez, G.; Vieyra-Alberto, R.; García-Martínez, A.; Martínez-García, C.G.; Cruz-Monterrosa, R.G.; Miranda-de la Lama, G.C. Triaxial accelerometers for recording grazing and ruminating time in dairy cows: An alternative to visual observations. J. Vet. Behav. 2017, 20, 102–108. [Google Scholar] [CrossRef]
- Grinter, L.N.; Campler, M.R.; Costa, J.H.C. Technical note: Validation of a behavior-monitoring collar’s precision and accuracy to measure rumination, feeding, and resting time of lactating dairy cows. J. Dairy Sci. 2019, 102, 3487–3494. [Google Scholar] [CrossRef] [Green Version]
- Liu, L.S.; Ni, J.Q.; Zhao, R.Q.; Shen, M.X.; He, C.L.; Lu, M.Z. Design and test of a low-power acceleration sensor with Bluetooth Low Energy on ear tags for sow behaviour monitoring. Biosyst. Eng. 2018, 176, 162–171. [Google Scholar] [CrossRef]
- Arcidiacono, C.; Mancino, M.; Porto, S.M.C.; Bloch, V.; Pastell, M. IoT device-based data acquisition system with on-board computation of variables for cow behaviour recognition. Comput. Electron. Agric. 2021, 191, 106500. [Google Scholar] [CrossRef]
- Riaboff, L.; Shalloo, L.; Smeaton, A.F.; Couvreur, S.; Madouasse, A.; Keane, M.T. Predicting livestock behaviour using accelerometers: A systematic review of processing techniques for ruminant behaviour prediction from raw accelerometer data. Comput. Electron. Agric. 2022, 192, 106610. [Google Scholar] [CrossRef]
- Ferrari, A.; Micucci, D.; Mobilio, M.; Napoletano, P. Trends in human activity recognition using smartphones. J. Reliab. Intell. Env. 2021, 7, 189–213. [Google Scholar] [CrossRef]
- Wang, J.; Chen, Y.; Hao, S.; Peng, X.; Hu, L. Deep learning for sensor-based activity recognition: A survey. Pattern Recognit. Lett. 2019, 119, 3–11. [Google Scholar] [CrossRef] [Green Version]
- Pavlovic, D.; Davison, C.; Hamilton, A.; Marko, O.; Atkinson, R.; Michie, C.; Crnojević, V.; Andonovic, I.; Bellekens, X.; Tachtatzis, C. Classification of Cattle Behaviours Using Neck-Mounted Accelerometer-Equipped Collars and Convolutional Neural Networks. Sensors 2021, 21, 4050. [Google Scholar] [CrossRef]
- Peng, Y.; Kondo, N.; Fujiura, T.; Suzuki, T.; Wulandari Yoshioka, H.; Itoyama, E. Classification of multiple cattle behavior patterns using a recurrent neural network with long short-term memory and inertial measurement units. Comput. Electron. Agric. 2019, 157, 247–253. [Google Scholar] [CrossRef]
- Li, C.; Tokgoz, K.K.; Fukawa, M.; Bartels, J.; Ohashi, T.; Takeda, K.; Ito, H. Data Augmentation for Inertial Sensor Data in CNNs for Cattle Behavior Classification. IEEE Sens. Lett. 2021, 5, 1–4. [Google Scholar] [CrossRef]
- Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis. 2015, 115, 211–252. [Google Scholar] [CrossRef] [Green Version]
- Pastell, M.; Frondelius, L. A hidden Markov model to estimate the time dairy cows spend in feeder based on indoor positioning data. Comput. Electron. Agric. 2018, 152, 182–185. [Google Scholar] [CrossRef]
- Riaboff, L.; Poggi, S.; Madouasse, A.; Couvreur, S.; Aubin, S.; Bédère, N.; Goumand, E.; Chauvin, A.; Plantier, G. Development of a methodological framework for a robust prediction of the main behaviours of dairy cows using a combination of machine learning algorithms on accelerometer data. Comput. Electron. Agric. 2020, 169, 105179. [Google Scholar] [CrossRef]
- Arcidiacono, C.; Porto, S.M.; Mancino, M.; Cascone, G. Development of a threshold-based classifier for real-time recognition of cow feeding and standing behavioural activities from accelerometer data. Comput. Electron. Agric. 2017, 134, 124–134. [Google Scholar] [CrossRef]
- Shahriar, M.S.; Smith, D.V.; Rahman, A.; Freeman, M.; Hills, J.; Rawnsley, R.P.; Henry, D.; Bishop-Hurley, G. Detecting heat events in dairy cows using accelerometers and unsupervised learning. Comput. Electron. Agric. 2016, 128, 20–26. [Google Scholar] [CrossRef]
- WISDM HAR Dataset. Available online: https://www.cis.fordham.edu/wisdm/dataset.php (accessed on 2 January 2023).
- Hu, J.; Zou, W.; Wang, J.; Pang, L. Minimum training sample size requirements for achieving high prediction accuracy with the BN model: A case study regarding seismic liquefaction. Expert Syst. Appl. 2021, 185, 115702. [Google Scholar] [CrossRef]
- Kalouris, G.; Zacharaki, E.I.; Megalooikonomou, V. Improving CNN-based activity recognition by data augmentation and transfer learning. In Proceedings of the IEEE 17th International Conference on Industrial Informatics (INDIN), Helsinki, Finland, 22–25 July 2019. [Google Scholar] [CrossRef]
- Eyobu, S.O.; Han, D.S. Feature Representation and Data Augmentation for Human Activity Classification Based on Wearable IMU Sensor Data Using a Deep LSTM Neural Network. Sensors 2018, 18, 2892. [Google Scholar] [CrossRef] [Green Version]
- Oh, S.; Ashiquzzaman, A.; Lee, D.; Kim, Y.; Kim, J. Study on Human Activity Recognition Using Semi-Supervised Active Transfer Learning. Sensors 2021, 21, 2760. [Google Scholar] [CrossRef]
- Li, F.; Shirahama, K.; Nisar, M.A.; Huang, X.; Grzegorzek, M. Deep Transfer Learning for Time Series Data Based on Sensor Modality Classification. Sensors 2020, 20, 4271. [Google Scholar] [CrossRef] [PubMed]
- Bloch, V.; Pastell, M. Monitoring of Cow Location in a Barn by an Open-Source, Low-Cost, Low-Energy Bluetooth Tag System. Sensors 2020, 20, 3841. [Google Scholar] [CrossRef]
- Hossain, T.; Ahad, M.A.R.; Inoue, S. A Method for Sensor-Based Activity Recognition in Missing Data Scenario. Sensors 2020, 20, 3811. [Google Scholar] [CrossRef]
- Um, T.T.; Pfister, F.M.J.; Pichler, D.; Endo, S.; Lang, M.; Hirche, S.; Fietzek, U.; Kulić, D. Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks. In Proceedings of the 19th ACM International Conference on Multimodal Interaction, New York, NY, USA, 13–17 November 2017. [Google Scholar] [CrossRef] [Green Version]
- Brownlee, J. Deep Learning for Computer Vision: Image Classification, Object Detection, and Face Recognition in Python, 1st ed.; Machine Learning Mastery; 2019; Available online: https://books.google.com.hk/books/about/Deep_Learning_for_Computer_Vision.html?id=DOamDwAAQBAJ&redir_esc=y (accessed on 2 January 2023).
- Wang, Y.; Cang, S.; Yu, H. A survey on wearable sensor modality centred human activity recognition in health care. Expert Syst. Appl. 2019, 137, 167–190. [Google Scholar] [CrossRef]
- Tian, F.; Wang, J.; Xiong, B.; Jiang, L.; Song, Z.; Li, F. Real-Time Behavioral Recognition in Dairy Cows Based on Geomagnetism and Acceleration Information. IEEE Access 2021, 9, 109497–109509. [Google Scholar] [CrossRef]
- Weerakody, P.B.; Wong, K.W.; Wang, G.; Ela, W. A review of irregular time series data handling with gated recurrent neural networks. Neurocomputing 2021, 441, 161–178. [Google Scholar] [CrossRef]
- Tosi, J.; Taffoni, F.; Santacatterina, M.; Sannino, R.; Formica, D. Performance Evaluation of Bluetooth Low Energy: A Systematic Review. Sensors 2017, 17, 2898. [Google Scholar] [CrossRef] [Green Version]
- Wijekoon, A.; Wiratunga, N.; Sani, S.; Cooper, K. A knowledge-light approach to personalised and open-ended human activity recognition. Knowl. -Based Syst. 2020, 192, 105651. [Google Scholar] [CrossRef]
- Achour, B.; Belkadi, M.; Aoudjit, R.; Laghrouche, M. Unsupervised automated monitoring of dairy cows’ behavior based on Inertial Measurement Unit attached to their back. Comput. Electron. Agric. 2019, 167, 105068. [Google Scholar] [CrossRef]
- Barwick, J.; Lamb, D.W.; Dobos, R.; Welch, M.; Trotter, M. Categorising sheep activity using a tri-axial accelerometer. Comput. Electron. Agric. 2018, 145, 289–297. [Google Scholar] [CrossRef]
- Benaissa, S.; Tuyttens, F.A.M.; Plets, D.; de Pessemier, T.; Trogh, J.; Tanghe, E.; Martens, L.; Vandaele, L.; van Nuffel, A.; Joseph, W.; et al. On the use of on-cow accelerometers for the classification of behaviours in dairy barns. Res. Vet. Sci. 2019, 125, 425–433. [Google Scholar] [CrossRef] [Green Version]
- Dutta, R.; Smith, D.; Rawnsley, R.; Bishop-Hurley, G.; Hills, J.; Timms, G.; Henry, D. Dynamic cattle behavioural classification using supervised ensemble classifiers. Comput. Electron. Agric. 2015, 111, 18–28. [Google Scholar] [CrossRef]
- Eerdekens, A.; Deruyck, M.; Fontaine, J.; Martens, L.; De Poorter, E.; Joseph, W. Automatic equine activity detection by convolutional neural networks using accelerometer data. Comput. Electron. Agric. 2020, 168, 105139. [Google Scholar] [CrossRef]
- Kaler, J.; Mitsch, J.; Vázquez-Diosdado, J.A.; Bollard, N.; Dottorini, T.; Ellis, K.A. Automated detection of lameness in sheep using machine learning approaches: Novel insights into behavioural differences among lame and non-lame sheep. R. Soc. Open Sci. 2020, 7, 190824. [Google Scholar] [CrossRef] [Green Version]
- Pavlovic, D.; Czerkawski, M.; Davison, C.; Marko, O.; Michie, C.; Atkinson, R.; Crnojevic, V.; Andonovic, I.; Rajovic, V.; Kvascev, G.; et al. Behavioural Classification of Cattle Using Neck-Mounted Accelerometer-Equipped Collars. Sensors 2022, 22, 2323. [Google Scholar] [CrossRef]
- Rahman, A.; Smith, D.; Little, B.; Ingham, A.; Greenwood, P.; Bishop-Hurley, G.J. Cattle behaviour classification from collar, halter, and ear tag sensors. Inform. Process. Agric. 2018, 5, 124–133. [Google Scholar] [CrossRef]
- Shen, W.; Cheng, F.; Zhang, Y.; Wei, X.; Fu, Q.; Zhang, Y. Automatic recognition of ingestive-related behaviors of dairy cows based on triaxial acceleration. Inf. Process. Agric. 2020, 7, 427–443. [Google Scholar] [CrossRef]
- Simanungkalit, G.; Barwick, J.; Cowley, F.; Dobos, R.; Hegarty, R. A Pilot Study Using Accelerometers to Characterise the Licking Behaviour of Penned Cattle at a Mineral Block Supplement. Animals 2021, 11, 1153. [Google Scholar] [CrossRef]
- Vázquez Diosdado, J.A.; Barker, Z.E.; Hodges, H.R.; Amory, R.J.; Croft, D.P.; Bell, N.J.; Codling, E.A. Classification of behaviour in housed dairy cows using an accelerometer-based activity monitoring system. Animal Biotelemetry 2015, 3, 1–14. [Google Scholar] [CrossRef] [Green Version]
- Vázquez-Diosdado, J.A.; Paul, V.; Ellis, K.A.; Coates, D.; Loomba, R.; Kaler, J.A. Combined Offline and Online Algorithm for Real-Time and Long-Term Classification of Sheep Behaviour: Novel Approach for Precision Livestock Farming. Sensors 2019, 19, 3201. [Google Scholar] [CrossRef] [Green Version]
- Walton, E.; Casey, C.; Mitsch, J.; Vázquez-Diosdado, J.A.; Yan, J.; Dottorini, T.; Ellis, K.A.; Winterlich, A.; Kaler, J. Evaluation of sampling frequency, window size and sensor position for classification of sheep behaviour. R. Soc. Open Sci. 2018, 5, 171442. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Wang, J.; He, Z.; Zheng, G.; Gao, S.; Zhao, K. Development and validation of an ensemble classifier for real-time recognition of cow behavior patterns from accelerometer data and location data. PLoS ONE 2018, 13, e0203546. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Wang, J.; Bell, M.; Liu, X.; Liu, G. Machine-Learning Techniques Can Enhance Dairy Cow Estrus Detection Using Location and Acceleration Data. Animals 2020, 10, 1160. [Google Scholar] [CrossRef]
- Williams, M.L.; Wiliam, P.J.; Rose, M.T. Variable segmentation and ensemble classifiers for predicting dairy cow behaviour. Biosyst. Eng. 2019, 178, 156–167. [Google Scholar] [CrossRef]
N | Period, Days | Average Time, Hours | Total Time, Hours (Days) | Fe, % | Ru, % | Oth, % | |
---|---|---|---|---|---|---|---|
Collected data | 21 | 1–3 | 38.5 ± 12.4 | 809 (33.7) | 19.7 ± 5.7 | 36.9 ± 6.1 | 43.3 ± 6.9 |
Open-source data | 18 | 6–18 | 191.7 ± 87.5 | 3450.5 (143.7) | 17.6 ± 3.8 | 38.4 ± 3.5 | 43.9 ± 6.6 |
CNN2 | CNN4 | CNN2 TL | CNN4 TL | |
---|---|---|---|---|
Precision | 92.9 ± 2.5 | 93.3 ± 2.0 | 93.3 ± 2.5 | 93.3 ± 1.9 |
F1 | 93.3 ± 2.5 | 93.9 ± 1.9 | 93.6 ± 2.4 | 93.8 ± 1.8 |
Recall | 94.2 ± 1.7 | 94.3 ± 1.5 | 94.5 ± 2.5 | 94.4 ± 1.4 |
WS (s) | 60 | 90 | 90 | 120 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Bloch, V.; Frondelius, L.; Arcidiacono, C.; Mancino, M.; Pastell, M. Development and Analysis of a CNN- and Transfer-Learning-Based Classification Model for Automated Dairy Cow Feeding Behavior Recognition from Accelerometer Data. Sensors 2023, 23, 2611. https://doi.org/10.3390/s23052611
Bloch V, Frondelius L, Arcidiacono C, Mancino M, Pastell M. Development and Analysis of a CNN- and Transfer-Learning-Based Classification Model for Automated Dairy Cow Feeding Behavior Recognition from Accelerometer Data. Sensors. 2023; 23(5):2611. https://doi.org/10.3390/s23052611
Chicago/Turabian StyleBloch, Victor, Lilli Frondelius, Claudia Arcidiacono, Massimo Mancino, and Matti Pastell. 2023. "Development and Analysis of a CNN- and Transfer-Learning-Based Classification Model for Automated Dairy Cow Feeding Behavior Recognition from Accelerometer Data" Sensors 23, no. 5: 2611. https://doi.org/10.3390/s23052611
APA StyleBloch, V., Frondelius, L., Arcidiacono, C., Mancino, M., & Pastell, M. (2023). Development and Analysis of a CNN- and Transfer-Learning-Based Classification Model for Automated Dairy Cow Feeding Behavior Recognition from Accelerometer Data. Sensors, 23(5), 2611. https://doi.org/10.3390/s23052611