research-article

Improving activity data collection with on-device personalization using fine-tuning

Authors:

Nattaya Mairittha,

Tittaya Mairittha,

Sozo InoueAuthors Info & Claims

UbiComp/ISWC '20 Adjunct: Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers

Pages 255 - 260

https://doi.org/10.1145/3410530.3414370

Published: 12 September 2020 Publication History

Abstract

One of the biggest challenges of activity data collection is the unavoidability of relying on users and keep them engaged to provide labels consistently. Recent breakthroughs in mobile platforms have proven effective in bringing deep neural networks powered intelligence into mobile devices. In this study, we propose on-device personalization using fine-tuning convolutional neural networks as a mechanism in optimizing human effort in data labeling. First, we transfer the knowledge gained by on-cloud pre-training based on crowdsourced data to mobile devices. Second, we incrementally fine-tune a personalized model on every individual device using its locally accumulated input. Then, we utilize estimated activities customized according to the on-device model inference as feedback to motivate participants to improve data labeling. We conducted a verification study and gathered activity labels with smartphone sensors. Our preliminary evaluation results indicate that the proposed method outperformed the baseline method by approximately 8% regarding accuracy recognition.

References

[1]

Shahriyar Amini and Yang Li. 2013. CrowdLearner: rapidly creating mobile recognizers using crowdsourcing. In Proceedings of the 26th annual ACM symposium on User interface software and technology. 163--172.

Digital Library

[2]

Michael Barz and Daniel Sonntag. 2016. Gaze-guided object classification using deep neural networks for attention-based computing. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. 253--256.

Digital Library

[3]

Tom Fawcett. 2006. An introduction to ROC analysis. Pattern recognition letters 27, 8 (2006), 861--874.

[4]

Nicholas D Lane, Petko Georgiev, and Lorena Qendro. 2015. DeepEar: robust smartphone audio sensing in unconstrained acoustic environments using deep learning. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 283--294.

Digital Library

[5]

Nattaya Mairittha and Sozo Inoue. 2019. Crowdsourcing System Management for Activity Data with Mobile Sensors. In 2019 Joint 8th International Conference on Informatics, Electronics & Vision (ICIEV) and 2019 3rd International Conference on Imaging, Vision & Pattern Recognition (icIVPR). IEEE, 85--90.

[6]

Nattaya Mairittha, Tittaya Mairittha, and Sozo Inoue. 2018. A Mobile App for Nursing Activity Recognition. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers. 400--403.

Digital Library

[7]

Nattaya Mairittha, Tittaya Mairittha, and Sozo Inoue. 2019. On-Device Deep Learning Inference for Efficient Activity Data Collection. Sensors 19, 15 (2019), 3434.

[8]

Tsuyoshi Okita and Sozo Inoue. 2018. Activity Recognition: Translation across Sensor Modalities Using Deep Learning. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers. 1462--1471.

Digital Library

[9]

Francisco Javier Ordóñez and Daniel Roggen. 2016. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors 16, 1 (2016), 115.

[10]

Valentin Radu and Maximilian Henne. 2019. Vision2sensor: Knowledge transfer across sensing modalities for human activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1--21.

Digital Library

[11]

Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S Emer. 2017. Efficient processing of deep neural networks: A tutorial and survey. Proc. IEEE 105, 12 (2017), 2295--2329.

[12]

Maja Vukovic, Rajarshi Das, and Soundar Kumara. 2013. From sensing to controlling: the state of the art in ubiquitous crowdsourcing. International Journal of Communication Networks and Distributed Systems 11, 1 (2013), 11--25.

Digital Library

[13]

Mengwei Xu, Feng Qian, Qiaozhu Mei, Kang Huang, and Xuanzhe Liu. 2018. Deeptype: On-device deep learning for input personalization service with minimal privacy concern. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 4 (2018), 1--26.

Digital Library

[14]

Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks?. In Advances in neural information processing systems. 3320--3328.

[15]

Ming Zeng, Le T Nguyen, Bo Yu, Ole J Mengshoel, Jiang Zhu, Pang Wu, and Joy Zhang. 2014. Convolutional neural networks for human activity recognition using mobile sensors. In 6th International Conference on Mobile Computing, Applications and Services. IEEE, 197--205.

Cited By

Huang YMillar JLong YZhao YHaddadi H(2024)Towards Low-Energy Adaptive Personalization for Resource-Constrained DevicesProceedings of the 4th Workshop on Machine Learning and Systems10.1145/3642970.3655826(73-80)Online publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1145/3642970.3655826
Long JYe GChen TWang YWang MYin HBaeza-Yates RBonchi F(2024)Diffusion-Based Cloud-Edge-Device Collaborative Learning for Next POI RecommendationsProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671743(2026-2036)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671743
Matthes PSpringer T(2022)Selecting Resource-Efficient ML Models for Transport Mode Detection on Mobile Devices2022 IEEE International Conference on Internet of Things and Intelligence Systems (IoTaIS)10.1109/IoTaIS56727.2022.9976004(135-141)Online publication date: 24-Nov-2022
https://doi.org/10.1109/IoTaIS56727.2022.9976004
Show More Cited By

Index Terms

Improving activity data collection with on-device personalization using fine-tuning
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing theory, concepts and paradigms
      1. Mobile computing

Recommendations

An FPGA-based Fine Tuning Accelerator for a Sparse CNN
FPGA '19: Proceedings of the 2019 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays

Fine-tuning learns abundant feature expression for a wide range of natural images by using a pre-trained CNN model. It can be applied to a wide range of the neural network (NN)based computer vision problems. This paper proposes an FPGA-based fine-tuning ...
Predicting Performance Improvement of Human Activity Recognition Model by Additional Data Collection

The development of a machine-learning-based human activity recognition (HAR) system using body-worn sensors is mainly composed of three phases: data collection, model training, and evaluation. During data collection, the HAR developer collects labeled ...
Boosting generalization of fine-tuning BERT for fake news detection
Abstract
Using deep neural networks to detect fake news has gradually become an extremely important field of research. Recent studies, which adopt fine-tuning BERT to detect fake news, have yielded promising results. Unfortunately, fine-tuning BERT is ...
Highlights
- Study the effects of freezing, initializing and iterations for generalization.
- Propose adversarial fine-tuning method with feature regularizer to detect fake news.
- Achieve better generalization than SOAT fake news detection ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UbiComp/ISWC '20 Adjunct: Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers

September 2020

732 pages

ISBN:9781450380768

DOI:10.1145/3410530

General Chairs:
Monica Tentori
CICESE, Mexico
,
Nadir Weibel
UC San Diego
,
Kristof Van Laerhoven
University of Siegen, Germany
,
Program Chairs:
Gregory Abowd
Georgia Tech
,
Flora Salim
RMIT, Australia

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 September 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

UbiComp/ISWC '20

Sponsor:

UbiComp/ISWC '20: 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and 2020 ACM International Symposium on Wearable Computers

September 12 - 17, 2020

Virtual Event, Mexico

Acceptance Rates

Overall Acceptance Rate 764 of 2,912 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
229
Total Downloads

Downloads (Last 12 months)54
Downloads (Last 6 weeks)9

Reflects downloads up to 02 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Huang YMillar JLong YZhao YHaddadi H(2024)Towards Low-Energy Adaptive Personalization for Resource-Constrained DevicesProceedings of the 4th Workshop on Machine Learning and Systems10.1145/3642970.3655826(73-80)Online publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1145/3642970.3655826
Long JYe GChen TWang YWang MYin HBaeza-Yates RBonchi F(2024)Diffusion-Based Cloud-Edge-Device Collaborative Learning for Next POI RecommendationsProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671743(2026-2036)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671743
Matthes PSpringer T(2022)Selecting Resource-Efficient ML Models for Transport Mode Detection on Mobile Devices2022 IEEE International Conference on Internet of Things and Intelligence Systems (IoTaIS)10.1109/IoTaIS56727.2022.9976004(135-141)Online publication date: 24-Nov-2022
https://doi.org/10.1109/IoTaIS56727.2022.9976004
Mairittha NMairittha TInoue S(2020)On-Device Deep Personalization for Robust Activity Data CollectionSensors10.3390/s2101004121:1(41)Online publication date: 23-Dec-2020
https://doi.org/10.3390/s21010041

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents