Abstract
In this paper, we propose a novel cooperative abnormal sound event detection framework for city surveillance in end-edge-cloud orchestrated systems. A novel offloading decision-making scheme that leverages hierarchical computational capabilities is proposed to speed up the detection process. The audio pre-processing (feature extraction) and post-processing (sound source localization and sound event classification) can be locally executed or offloaded to the edge or cloud based on the calculation of the so-called communication-to-computation ratio. Furthermore, considering the biased audio information due to source-sensor geometries, a cooperative decision-making algorithm is proposed to aggregate the sound event classification results with adaptive control and ensemble learning. In the audio pre-processing, the log-mel spectrogram and time of arrival information are first extracted from the audio waveform captured by the distributed acoustic sensors and then sent to the computation entity assigned by the offloading scheme. In the audio post-processing, the sound source is localized through least-square minimization. Guided by the localized sound source, the spectrograms are fed into the pre-trained neural networks and then the result aggregation algorithm for further classification. Extensive experiments regarding latency and detection accuracy show the superiority and robustness of the proposed scheme, avoiding the cumulative latency caused by the increased number of sensors while maintaining high detection accuracy.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Other sound source localization and sound event classification techniques are also applicable in the proposed framework.
The AudioSet ontology can be found at https://research.google.com/audioset/ontology.
References
Abu-El-Haija, S., Kothari, N., Lee, J., Natsev, P., Toderici, G., Varadarajan, B., Vijayanarasimhan, S.: Youtube-8M: a large-scale video classification benchmark.arXiv:1609.08675 (2016)
Adavanne, S., Politis, A., Nikunen, J., Virtanen, T.: Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE J. Sel. Top. Signal Process. 13(1), 34–48 (2018)
AudioSet download. https://research.google.com/audioset/download.html (2019)
Baucas, M.J., Spachos, P.: Using cloud and fog computing for large scale iot-based urban sound classification. Simul. Modell. Practice Theory 101, 102013 (2020)
Bauer, S., Clark, D.D., Lehr, W.: Understanding broadband speed measurements. In: Proc. of MIT TPRC (2010)
Cerutti, G., Prasad, R., Brutti, A., Farella, E.: Compact recurrent neural networks for acoustic event detection on low-energy low-complexity platforms. IEEE J. Sel. Top. Signal Process. (2020)
Cobos, M., Antonacci, F., Alexandridis, A., Mouchtaris, A., Lee, B.: A survey of sound source localization methods in wireless acoustic sensor networks. Wirel. Commun, Mobile Comput (2017)
Embleton, T.F.: Tutorial on sound propagation outdoors. J. Acoust. Soc. Am. 100(1), 31–48 (1996)
Fabregat, G., Belloch, J.A., Badía, J.M., Cobos, M.: Design and implementation of acoustic source localization on a low-cost IoT edge platform. Express Briefs IEEE Trans. Circ. Syst. II, (2020)
Goldsmith, A.: Wireless Communications. Cambridge University Press, Cambridge (2005)
Guo, Y., Zou, B., Ren, J., Liu, Q., Zhang, D., Zhang, Y.: Distributed and efficient object detection via interactions among devices, edge, and cloud. IEEE Trans. Multimedia 21(11), 2903–2915 (2019)
He, Y., Sainath, T. N., Prabhavalkar, R., McGraw, I., Alvarez, R., Zhao, D., Rybach, D., Kannan, A., Wu, Y., Pang, R., et al.: Streaming end-to-end speech recognition for mobile devices. In: Proc. of IEEE ICASSP, pp. 6381–6385 (2019)
Heittola, T., Mesaros, A., Eronen, A., Virtanen, T.: Context-dependent sound event detection. EURASIP J. Audio Speech Music Process. 1, 1–13 (2013)
Hershey, S., Chaudhuri, S., Ellis, D.P.W., Gemmeke, J.F., Jansen, A., Moore, R.C., Plakal, M., Platt, D., Saurous, R.A., Seybold, B., Slaney, M., Weiss, R.J., Wilson, K.: CNN architectures for large-scale audio classification. In: Proc. of IEEE ICASSP, pp. 131–135 (2017)
Hong, K., Lillethun, D., Ramachandran, U., Ottenwälder, B., Koldehofe, B.: Mobile fog: a programming model for large-scale applications on the internet of things. In: Proc. of ACM SIGCOMM workshop on Mobile cloud computing, pp. 15–20 (2013)
Huang, Y., Zhu, Y., Fan, X., Ma, X., Wang, F., Liu, J., Wang, Z., Cui, Y.: Task scheduling with optimized transmission time in collaborative cloud-edge learning. In: Proc. of ICCCN, pp. 1–9 (2018)
Junior, W., França, A., Dias, K., de Souza, J.N.: Supporting mobility-aware computational offloading in mobile cloud environment. J. Netw. Comput. Appl. 94, 93–108 (2017)
Kong, Q., Xu, Y., Wang, W., Plumbley, M.D.: Sound event detection of weakly labelled data with cnn-transformer and automatic threshold optimization. IEEE/ACM Trans. Audio Speech Lang. Process. 28, 2450–2460 (2020)
Li, Y., Liu, M., Drossos, K., Virtanen, T.: Sound event detection via dilated convolutional recurrent neural networks. In: Proc. of IEEE ICASSP, pp. 286–290 (2020)
Liu, K., Peng, J., Li, H., Zhang, X., Liu, W.: Multi-device task offloading with time-constraints for energy efficiency in mobile cloud computing. Future Gener. Comput. Syst. 64, 1–14 (2016)
Nguyen, T.N.T., Jones, D.L., Gan, W.-S.: A sequence matching network for polyphonic sound event localization and detection. In: Proc. of IEEE ICASSP, pp. 71–75 (2020)
Nirjon, S., Dickerson, R.F., Asare, P., Li, Q., Hong, D., Stankovic, J.A., Hu, P., Shen, G., Jiang, X.: Auditeur: a mobile-cloud service platform for acoustic event detection on smartphones. In: Proc. of MobiSys, pp. 403–416 (2013)
Niu, C., Li, Y., Hu, R.Q., Ye, F.: Fast and efficient radio resource allocation in dynamic ultra-dense heterogeneous networks. IEEE Access 5, 1911–1924 (2017)
Noble, B.D., Satyanarayanan, M., Narayanan, D., Tilton, J.E., Flinn, J., Walker, K.R.: Agile application-aware adaptation for mobility. Proc. ACM SIGOPS 31(5), 276–287 (1997)
Noghabi, S. A., Kolb, J., Bodik, P., Cuervo, E.: Steel: simplified development and deployment of edge-cloud applications. In: Proc. of USENIX HotCloud (2018)
Phan, H., Maaß, M., Mazur, R., Mertins, A.: Random regression forests for acoustic event detection and classification. IEEE/ACM Trans. Audio Speech Lang. Process. 23(1), 20–31 (2014)
Ran, X., Chen, H., Zhu, X., Liu, Z., Chen, J.: DeepDecision: a mobile deep learning framework for edge video analytics. In: Proc. of IEEE INFOCOM, pp. 1421–1429 (2018)
Ren, J., Zhang, D., He, S., Zhang, Y., Li, T.: A survey on end-edge-cloud orchestrated network computing paradigms: transparent computing, mobile edge computing, fog computing, and cloudlet. ACM Comput. Surv. 52(6), 1–36 (2019)
Shotspotter. http://www.shotspotter.com/technology (2019)
Sun, C., Li, H., Li, X., We, J., Xiong, Q., Wang, X., Leung, V.C.: Task offloading for end-edge-cloud orchestrated computing in mobile networks. In: Proc. of IEEE WCNC, pp. 1–6 (2020)
Tang, L., He, S.: Multi-user computation offloading in mobile edge computing: a behavioral perspective. IEEE Netw. 32(1), 48–53 (2018)
Tong, F., Pan, J.: Random-to-random nodal distance distributions in finite wireless networks. IEEE Trans. Vehic. Technol. 66(11), 10070–10083 (2017)
Trowitzsch, I., Schymura, C., Kolossa, D., Obermayer, K.: Joining sound event detection and localization through spatial segregation. IEEE/ACM Trans. Audio Speech Lang. Process. 28, 487–502 (2019)
Wang, J., Liu, K., Tzanetakis, G., Pan, J.: Learning-based cooperative sound event detection with edge computing. In: Proc. of IEEE IPCCC, pp. 1–8 (2018)
Wang, J., Ai, F., Sun, Q., Liu, T., Li, H., Yan, Z., Liu, D.: Diaphragm-based optical fiber sensor array for multipoint acoustic detection. Opt. Express 26(19), 25293–25304 (2018)
Yang, B., Cao, X., Bassey, J., Li, X., Qian, L.: Computation offloading in multi-access edge computing: a multi-task learning approach. IEEE Trans, Mobile Comput (2020)
Zhang, Y., Lan, X., Ren, J., Cai, L.: Efficient computing resource sharing for mobile edge-cloud computing networks. IEEE/ACM Trans. Netw. 28(3), 1227–1240 (2020)
Zhu, W., Luo, C., Wang, J., Li, S.: Multimedia cloud computing. IEEE Signal Process. Mag. 28(3), 59–69 (2011)
Funding
This work was supported in part by NSERC, CFI, BCKDF, and China Postdoctoral Science Foundation.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Wang, J., Liu, K., Tzanetakis, G. et al. Cooperative abnormal sound event detection in end-edge-cloud orchestrated systems. CCF Trans. Netw. 3, 158–170 (2020). https://doi.org/10.1007/s42045-020-00042-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42045-020-00042-x