Abstract
Ambient-aware applications need to know what objects are in the environment. Although video data contains this information, analyzing it is a challenge esp. on portable devices that are constrained in energy and storage. A naïve solution is to sample and stream video to the cloud, where advanced algorithms can be used for analysis. However, this increases communication energy costs, making this approach impractical. In this article, we show how to reduce energy in such systems by employing simple on-device computations. In particular, we use a low-complexity feature-based image classifier to filter out unnecessary frames from video. To lower the processing energy and sustain a high throughput, we propose a hierarchically pipelined hardware architecture for the image classifier. Based on synthesis results from an ASIC in a 45 nm SOI process, we demonstrate that the classifier can achieve minimum-energy operation at a frame rate of 12 fps, while consuming only 3 mJ of energy per frame. Using a prototype system, we estimate about 70 % reduction in communication energy when 5 % of frames are interesting in a video stream.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
D’Andrea, R.: Can drones deliver? IEEE Trans. Autom. Sci. Eng. 138–141 (2014)
Randell, C.: Wearable computing: a review. Technical Report Number CSTR-06-004. University of Bristol (2005)
Geronimo, D., Lopez, A.M., Sappa, A.D., Graf, T.: Survey of pedestrian detection for advanced driver assistance systems. IEEE Trans. Pattern Anal. Mach. Intell. 32(7), 1239–1258 (2009). doi:10.1109/TPAMI.2009.122
Baber, C., Smith, P., Cross, J., Zasikowski, D., Hunter, J.: Wearable technology for crime scene investigation. In: Proceedings of the IEEE International Symposium on Wearable Computers, pp. 138–141 (2005)
Mann, S.: WearCam (the wearable camera): personal imaging systems for long-term use in wearable tetherless computer-mediated reality and personal photo/videographic memory prosthesis. In: Proceedings of the IEEE International Symposium on Wearable Computers, pp. 124–131 (1998)
Kelly, P., Marshall, S.J., Badland, H., Kerr, J., Oliver, M., Doherty, A.R., Foster, C.: An ethical framework for automated, wearable cameras in health behavior research. Am. J. Prev. Med. 44(3), 314–319 (2013). doi:10.1016/j.amepre.2012.11.006
Navab, N.: Developing killer apps for industrial augmented reality. IEEE Comput. Graph. Appl. 24(3), 16–20 (2004)
Aleksya, M., Rissanenb, M.J., Maczeya, S., Dixa, M.: Wearable computing in industrial service applications. Int. Conf. Ambient Syst. Netw. Technol. 5, 394–400 (2011). doi:10.1016/j.procs.2011.07.051
Weinland, D., Ronfard, R., Boyer, E.: A survey of vision-based methods for action representation, segmentation and recognition. Elsevier Comput. Vis. Image Underst. 115(2), 224–241 (2011). doi:10.1016/j.cviu.2010.10.002
Poppe, R.: A survey on vision-based human action recognition. Elsevier Image Vis. Comput. 28(6), 976–990 (2010). doi:10.1016/j.imavis.2009.11.014
Crevier, D., Lepage, R.: Knowledge-based image understanding systems: a survey. Elsevier Comput. Vis. Image Underst. 67(2), 161–185 (1997). doi:10.1006/cviu.1996.0520
LiKamWa, R., Wang, Z., Carroll, A., Lin, X.F., Zong, L.: Draining our glass: an energy and heat characterization of Google Glass. In: Proceedings of Asia-Pacific Workshop on Systems, Article no. 10, (2014). doi:10.1145/2637166.2637230
Ha, K., Chen, Z., Hu, W., Richter, W., Pillai, P., Satyanarayanan, M.: Towards wearable cognitive assistance. In: Proceedings of the International Conference on Mobile Systems, Applications, and Services, pp. 68–81 (2014). doi:10.1145/2594368.2594383
Jia, Z., Balasuriya, A., Challa, S.: Vision based target tracking for autonomous land vehicle navigation: a brief survey. Recent Pat. Comput. Sci. 2(1), 32–42 (2009)
Soro, S., Heinzelman, W.: A survey of visual sensor networks. Hindawi Advances in Multimedia, Article no. 640386 (2009). doi:10.1155/2009/640386
Kyono, Y., Yonezawa, T., Nozaki, H., Keio, M.O., Keio, T.I., Keio, J.N., Takashio, K., Tokuda, H.: EverCopter: continuous and adaptive over-the-air sensing with detachable wired flying objects. In: Proceedings of the ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication, pp. 299–302 (2013). doi:10.1145/2494091.2494183
Kemp, R., Palmer, N., Kielmann, T., Bal, H.: Cuckoo: a computation offloading framework for smartphones. In: Mobile Computing, Applications, and Services. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecom. Engineering, vol. 76, pp. 59–79 (2012)
Ra, M-R., Sheth, A., Mummert, L., Pillai, P., Wetherall, D., Govindan, R.: Odessa: enabling interactive perception applications on mobile devices. In: Proceedings of the International Conference on Mobile Systems, Applications, and Services, pp. 43–56, (2011). doi:10.1145/1999995.2000000
Nishikawa, T., Takahashi, M., Hamada, M., Takayanagi, T., Arakida, H., Machiada, N., Yamamoto, H., Fujiyoshi, T., Matsumoto, Y., Yamagishi, O., Samata, T., Asano, A., Terazawa, T., Ohmori, K., Shirakura, J., Watanabe, Y., Nakamura, H., Minami, S., Kuroda, T., Furuyama, T.: A 60MHz 240mW MPEG-4 video-phone LSI with 16Mb embedded DRAM. IEEE J. Solid-State Circuits 35, 1713–1721 (2000)
Worlds most power-efficient 1080p/60 high definition imag sensor for front-facing camera applications. OV2740 1080p Product Brief. www.ovt.com (2014)
Halperin, D., Greenstein, B., Sheth, A., Wetherall, D.: Demystifying 802.11n power consumption. In: Proceedings of the International Conference on Power Aware Computing and Systems, Article no. 1 (2010)
Brostow, G., Shotton, J., Fauquer, J., Cipolla, R.: Segmentation and recognition using structure from motion point clouds. In: Proceedings of the European Conference on Computer Vision, pp. 44–57 (2008). doi:10.1007/978-3-540-88682-2_5
Venkataramani, S., Bahl, V., Hua, X.-S., Liu, J., Li, J., Phillipose, M., Priyantha, B., Shoaib, M.: SAPPHIRE: an always-on context-aware computer-vision system for portable devices. In: Proceedings of Conference on Design Automation and Test in Europe, to appear (2015)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Proceedings of Neural Information Processing Systems, pp. 1106–1114 (2012)
Jin, J., Gokhale, V., Dundar, A., Krishnamurthy, B., Martini, B., Culurciello, E.: An efficient implementation of deep convolutional neural networks on a mobile coprocessor. IEEE Int. Midwest Symp. Circuits Syst. 133–136 (2014). doi:10.1109/MWSCAS.2014.6908370
Gokhale, V., Jin, J., Dundar, A., Martini, B., Culurciello, E.: A 240 G-ops/s mobile coprocessor for deep neural networks. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops 696–701 (2014). doi:10.1109/CVPRW.2014.106
Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., Temam, O.: DianNao: a small-footprint high-throughput accelerator for ubiquitous machine-learning. In: Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 269–284 (2014). doi:10.1145/2541940.2541967
Perronnin, F., Sanchez, J., Mensink, T.: Improving the fisher kernel for large-scale image classification. In: Proceedings of the European Conference on Computer Vision, pp. 143–156 (2010)
Harris, C., Stephens, M.: A combined corner and edge detector. In: Proceedings of the Fourth Alvey Vision Conference, pp. 147–151 (1988)
Winder, S., Hua, G., Brown, M.: Picking the best daisy. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (2009)
Winder, S., Hua, G., Brown, M.: Discriminative learning of local image descriptors. IEEE Trans. Pattern Anal. Mach. Intell. 33(1), 43–57 (2009). doi:10.1109/TPAMI.2010.54
Shotton, J., Johnson, M., Cipolla, R.: Semantic texton forests for image categorization and segmentation. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008). doi:10.1109/CVPR.2008.4587503
He, X., Yan, S., Hu, Y., Niyogi, P., Zhang, H.-J.: Face recognition using Laplacian faces. IEEE Trans. Pattern Anal. Mach. Intell. 27(3), 328–340 (2005)
Chen, H.-T., Chang, H.-W., Liu, T.-L.: Local discriminant embedding and its variants. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition, pp. 846–853 (2005). doi:10.1109/CVPR.2005.216
Winder, S.A.J., Brown, M.: Learning local image descriptors. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)
Sanchez, J., Perronnin, F., Mensink, T., Jakob, V.: Image classification with the Fisher vector: theory and practice. Int. J. Comput. Vis. 105(3), 222–245 (2013). doi:10.1007/s11263-013-0636-x
Leijen, D., Schulte, W., Burchardt, S.: The design of a task parallel library. In: Proceedings of the Conference on Object Oriented Programming Systems Languages and Applications, pp. 227–242(2009). doi:10.1145/1640089.1640106
Griffin, G., Holum, A., Perona, P.: Caltech-256 object category dataset. Caltech Technical Report Number: CNS-TR-2007-001. http://authors.library.caltech.edu/7694 (2011)
LeCun, Y., Huang, F.J., Bottou, L.: Learning methods for generic object recognition with invariance to pose and lighting. Proc. Int. Conf. Comput. Vis. Pattern Recognit. (2004). doi:10.1109/CVPR.2004.1315150
Everingham, M., Ali, E.S.M., Luc, V.G., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes challenge: a retrospective. Int. J. Comput. Vis. 1–39 (2014). doi:10.1007/s11263-014-0733-5
OmniVision OV7735 Product Brief. www.ovt.com (2014)
Chen, S., Bermak, A., Wang, Y.: A CMOS image sensor with on-chip image compression based on predictive boundary adaptation and memoryless QTD algorithm. IEEE Trans. VLSI Syst. 19(4), 538–547 (2011)
Low Power Advantage of 802.11a/g vs. 802.11b. Whitepaper, Texas Instruments. www.ti.com (2003)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Shoaib, M., Venkataramani, S., Hua, XS., Liu, J., Li, J. (2015). Exploiting On-Device Image Classification for Energy Efficiency in Ambient-Aware Systems. In: Hua, G., Hua, XS. (eds) Mobile Cloud Visual Media Computing. Springer, Cham. https://doi.org/10.1007/978-3-319-24702-1_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-24702-1_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-24700-7
Online ISBN: 978-3-319-24702-1
eBook Packages: Computer ScienceComputer Science (R0)