research-article

Public Access

Test time feature ordering with FOCUS: interactive predictions with minimal user burden

Authors:

Stephen E. Fienberg,

Jennifer MankoffAuthors Info & Claims

UbiComp '16: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing

Pages 992 - 1003

https://doi.org/10.1145/2971648.2971748

Published: 12 September 2016 Publication History

Abstract

Predictive algorithms are a critical part of the ubiquitous computing vision, enabling appropriate action on behalf of users. A common class of algorithms, which has seen uptake in ubiquitous computing, is supervised machine learning algorithms. Such algorithms are trained to make predictions based on a set of features (selected at training time). However, features needed at prediction time (such as mobile information that impacts battery life, or information collected from users via experience sampling) may be costly to collect. In addition, both cost and value of a feature may change dynamically based on real-world context (such as battery life or user location) and prediction context (what features are already known, and what their values are). We contribute a framework for dynamically trading off feature cost against prediction quality at prediction time. We demonstrate this work in the context of three prediction tasks: providing prospective tenants estimates for energy costs in potential homes, estimating momentary stress levels from both sensed and user-provided mobile data, and classifying images to facilitate opportunistic device interactions. Our results show that while our approach to cost-sensitive feature selection is up to 45% less costly than competing approaches, error rates are equivalent or better.

References

[1]

Miguel Ballesteros & Bernd Bohnet. (2014). Automatic feature selection for agenda-based dependency parsing. In COLING (pp. 794--805).

[2]

Nilanjan Banerjee, Ahmad Rahmati, Mark D. Corner, Sami Rollins, & Lin Zhong. (2007). Users and Batteries: Interactions and Adaptive Energy Management in Mobile Systems (pp. 217--234). Springer Berlin Heidelberg.

Digital Library

[3]

Michael Beigl. (1999). Point & click-interaction in smart environments. In Handheld and Ubiquitous Computing (pp. 311--313). Springer Berlin Heidelberg.

Digital Library

[4]

Leo Breiman, Jerome Friedman, Charles J. Stone, & R.A. Olshen. (1984). Classification and Regression Trees. CRC press.

[5]

Matthias Budde, Matthias Berning, Christopher Baumgärtner, Florian Kinn, Timo Kopf, Sven Ochs, Frederik Reiche, Till Riedel, & Michael Beigl. (2013). Point & control---interaction in smart environments: you only click twice. In Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication (pp. 303--306). ACM.

Digital Library

[6]

Tsung-Hsiang Chang & Yang Li. (2011). Deep shot: a framework for migrating tasks across devices using mobile phone cameras. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (pp. 2163--2172). ACM.

Digital Library

[7]

David A. Cohn, Zoubin Ghahramani, & Michael I. Jordan. (1996). Active learning with statistical models. Journal of Artificial Intelligence Research, 4, 129--145.

Digital Library

[8]

Thomas M. Cover & Peter E. Hart. (1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1), 21--27.

Digital Library

[9]

Adrian A. de Freitas, Michael Nebeling, Xiang "Anthony" Chen, Junrui Yang, Akshaye Ranithangam, & Anind K. Dey. (2016). Snap-To-It: a user-inspired platform for opportunistic device interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, (pp. 5909--5920).

Digital Library

[10]

Thomas Dietz, Gerald T. Gardner, Jonathan Gilligan, Paul C. Stern, & Michael P. Vandenbergh. (2009). Household actions can provide a behavioral wedge to rapidly reduce US carbon emissions. In Proceedings of the National Academy of Sciences, 106(44), 18452--18456.

[11]

Robin A. Douthitt. (1989). An economic analysis of the demand for residential space heating fuel in Canada. Energy, 14(4), 187--197.

[12]

Kirstin Early, Stephen E. Fienberg, & Jennifer Mankoff. (2016). Dynamic question ordering in online surveys. arXiv preprint arXiv: 1607.04209.

[13]

Denzil Ferreira, Eija Ferreira, Jorge Goncalves, Vassilis Kostakos, & Anind K. Dey. (2013). Revisiting human-battery interaction with an interactive battery interface. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (pp. 563--572). ACM.

Digital Library

[14]

Denzil Ferreira, Vassilis Kostakos, & Anind K. Dey. (2015). AWARE: Mobile context instrumentation framework. Frontiers in ICT, 2(6), 1--9.

[15]

Pedro Ferreira, Pedro Sanches, Kristina Höök, & Tove Jaensson. (2008). License to chill! How to empower users to cope with stress. In Proceedings of the Fifth Nordic Conference on Human-Computer Interaction: Building Bridges (pp. 123--132). ACM.

Digital Library

[16]

David Ferrucci, Anthony Levas, Sugato Bagchi, David Gondek, & Erik T. Mueller. (2013). Watson: Beyond Jeopardy!. Artificial Intelligence, 199, 93--105.

Digital Library

[17]

Jerome H. Friedman, Ron Kohavi, & Yeogirl Yun. (1996). Lazy decision trees. In Proceedings of the 1st International Conference on Artificial Intelligence (pp. 717--724). AAAI Press.

Digital Library

[18]

Nadav Golbandi, Yehuda Koren, & Ronny Lempel. (2011). Adaptive bootstrapping of recommender systems using decision trees. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining (pp. 595--604). ACM.

Digital Library

[19]

Isabelle Guyon & André Elisseeff. (2003). An introduction to variable and feature selection. The Journal of Machine Learning Research, 3, 1157--1182.

Digital Library

[20]

He He, Hal Daumé III, & Jason Eisner. (2012). Cost-sensitive dynamic feature selection. In ICML Inferning Workshop.

[21]

He He, Hal Daumé III, & Jason Eisner. (2013). Dynamic feature selection for dependency parsing. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (pp. 1455--1464).

[22]

Paul A. Hirschberg, Elliot Abrams, Andrea Bleistein, William Bua, Luca Delle Monache, Thomas W. Dulong, John E. Gaynor, Bob Glahn, Thomas M. Hamill, James A. Hansen, Douglas C. Hilderbrand, Ross N. Hoffman, Betty Hearn Morrow, Brenda Philips, John Sokich, & Neil Stuart. (2011). A weather and climate enterprise strategic implementation plan for generating and communicating forecast uncertainty information. Bulletin of the American Meteorological Society, 92(12), 1651--1666.

[23]

Suzanne S. Hudd, Jennifer Dumlao, Diane Erdmann-Sager, Daniel Murray, Emily Phan, Nicholas Soukas, & Nori Yokozuka. (2000). Stress at college: Effects on health habits, health status and self-esteem. College Student Journal, 34(2), 217--227.

[24]

Madhu Kalia. (2002). Assessing the economic impact of stress---the modern-day hidden epidemic. Metabolism, 51(6), 49--53.

[25]

Nikhil Kaza. (2010). Understanding the spectrum of residential energy consumption: a quantile regression approach. Energy Policy, 38(11), 6574--6585.

[26]

Blerina Lika, Kostas Kolomvatsos, & Stathes Hadjiefthymiades. (2014). Facing the cold start problem in recommender systems. Expert Systems with Applications, 41(4), 2065--2073.

Digital Library

[27]

Qiong Liu, Paul McEvoy, Don Kimber, Patrick Chiu, & Hanningn Zhou. (2006). On redirecting documents with a mobile camera. In 8th IEEE Workshop on Multimedia Signal Processing (pp. 467--470). IEEE.

[28]

David G. Lowe. (1999). Object recognition from local scale-invariant features. In Proceedings of the 7th IEEE International Conference on Computer Vision (pp. 1150--1157).

Digital Library

[29]

Hong Lu, Jun Yang, Zhigang Liu, Nicholas D. Lane, Tanzeem Choudhury, & Andrew T. Campbell. (2010). The Jigsaw continuous sensing engine for mobile phone applications. In Proceedings of the 8th ACM Conference on Embedded Networked Sensor Systems (pp. 71--84). ACM.

Digital Library

[30]

Chulhong Min, Chungkuk Yoo, Inseok Hwang, Seungwoo Kang, Youngki Lee, Seungchul Lee, Pillsoon Park, Changhun Lee, Seungpyo Choi, & Junehwa Song. (2015). Sandra helps you learn: The more you walk, the more battery your phone drains. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (pp. 421--432). ACM.

Digital Library

[31]

Ranjita Misra & Michelle McKean. (2000). College students' academic stress and its relation to their anxiety, time management, and leisure satisfaction. American Journal of Health Studies, 16(1), 41--51.

[32]

Gary P. Moberg. (2000). Biological response to stress: implications for animal welfare. In Gary P. Moberg & Joy A. Mench (Eds.), The Biology of Animal Stress: Basic Principles and Implications for Animal Welfare (1--21). New York: CABI Publishing.

[33]

Erman Pattuk, Murat Kantarcioglu, Huseyin Ulusoy, & Bradley Malin. (2015). Privacy-aware dynamic feature selection. In 31st IEEE International Conference on Data Engineering (pp. 78--88). IEEE.

[34]

Nishkam Ravi, James Scott, & Liviu Iftode. (2008). Context-aware battery management for mobile phones. In Sixth Annual IEEE International Conference on Pervasive Computing and Communications (pp. 224--233). IEEE.

Digital Library

[35]

John Seryak & Kelly Kissock. (2003). Occupancy and behavioral effects on residential energy use. In Proceedings of the Solar Conference (pp. 717--722).

[36]

Kirill Trapeznikov & Venkatesh Saligrama. (2013). Supervised sequential classification under budget constraints. In Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics (pp. 581--589).

[37]

Shannon E. Ross, Bradley C. Niebling, & Teresa M. Heckert. (1999). Sources of stress among college students. College Student Journal, 33(2), 312--317.

[38]

Maytal Saar-Tsechansky & Foster Provost. (2007). Handling missing values when applying classification models. The Journal of Machine Learning Research, 8, 1625--1657.

Digital Library

[39]

Mehdi Samadi, Partha Talukdar, Manuela Veloso, & Tom Mitchell. (2015). AskWorld: Budget-sensitive query evaluation for knowledge-on-demand. In Proceedings of the 24th International Conference on Artificial Intelligence (pp. 837--843). AAAI Press.

Digital Library

[40]

Nandita Sharma & Tom Gedeon. (2012). Objective measures, sensors and computational techniques for stress recognition and classification: A survey. Computer Methods and Programs in Biomedicine, 108(3), 1287--1301.

Digital Library

[41]

Tianlin Shi, Jacob Steinhardt, & Percy Liang. (2015). Learning where to sample in structured prediction. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics (AISTATS) (pp. 875--884).

[42]

Emma Strubell, Luke Vilnis, Kate Silverstein, & Andrew McCallum. (2015). Learning dynamic feature selection for fast sequential prediction. arXiv preprint arXiv:1505.06169.

[43]

Mingxuan Sun, Fuxin Li, Joonseok Lee, Ke Zhou, Guy Lebanon, & Hongyuan Zha. (2013). Learning multiple-question decision trees for cold-start recommendation. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining (pp. 445--454).

Digital Library

[44]

Lukas G. Swan & V. Ismet Ugursal. (2009). Modeling of end-use energy consumption in the residential sector: A review of modeling techniques. Renewable and Sustainable Energy Reviews, 13(8), 1819--1835.

[45]

Joel A. Tropp. (2004). Greed is good: algorithmic results for sparse approximation. IEEE Transactions on Information Theory, 50(10), 2231--2242.

Digital Library

[46]

U.S. Energy Information Administration. (2009). Residential Energy Consumption Survey 2009. www.eia.gov/consumption/residential/data/2009/.

[47]

Rui Wang, Fanglin Chen, Zhenyu Chen, Tianxing Li, Gabriella Harari, Stefanie Tignor, Xia Zhou, Dror BenZeev, & Andrew T. Campbell. (2014). StudentLife: Assessing mental health, academic performance, and behavioral trends of college students using smartphones. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (pp. 3--14).

Digital Library

[48]

Sanford Weisberg. (2014). Applied Linear Regression (4th ed.). New York: Wiley.

[49]

David J. Weiss & Ben Taskar. (2013). Learning adaptive value of information for structured prediction. In Advances in Neural Information Processing Systems (pp. 953--961).

Digital Library

[50]

Jiahui Wu, Gang Pan, Daqing Zhang, Shijian Li, & Zhaohui Wu. (2010). MagicPhone: pointing & interacting. In Proceedings of the 2010 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication (pp. 451--452). ACM.

Digital Library

[51]

Zhixiang (Eddie) Xu, Matt J. Kusner, Killian Q. Weinberger, Minmin Chen, & Olivier Chapelle. (2014). Classifier cascades and trees for minimizing feature evaluation cost. The Journal of Machine Learning Research, 15(1), 2113--2144.

Digital Library

Cited By

Shin HPark JYu JKim JKim HOh C(2024)Looping In: Exploring Feedback Strategies to Motivate Human Engagement in Interactive Machine LearningInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2413293(1-18)Online publication date: 14-Oct-2024
https://doi.org/10.1080/10447318.2024.2413293
Covert IQiu WLu MKim NWhite NLee SKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Learning to maximize mutual information for dynamic feature selectionProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3618665(6424-6447)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3618665
Mishra SRzeszotarski J(2023)Human Expectations and Perceptions of Learning in Machine TeachingProceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization10.1145/3565472.3595612(13-24)Online publication date: 18-Jun-2023
https://dl.acm.org/doi/10.1145/3565472.3595612
Show More Cited By

Index Terms

Test time feature ordering with FOCUS: interactive predictions with minimal user burden
1. Computing methodologies
  1. Machine learning

Recommendations

Leveraging the crowd to improve feature-sentiment analysis of user reviews
IUI '13: Proceedings of the 2013 international conference on Intelligent user interfaces

Crowdsourcing and machine learning are both useful techniques for solving difficult problems (e.g., computer vision and natural language processing). In this paper, we propose a novel method that harnesses and combines the strength of these two ...
A Human-in-the-Loop System for Sound Event Detection and Annotation
Special Issue on Human-Centered Machine Learning

Labeling of audio events is essential for many tasks. However, finding sound events and labeling them within a long audio file is tedious and time-consuming. In cases where there is very little labeled data (e.g., a single labeled example), it is often ...
OneLabeler: A Flexible System for Building Data Labeling Tools
CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

Labeled datasets are essential for supervised machine learning. Various data labeling tools have been built to collect labels in different usage scenarios. However, developing labeling tools is time-consuming, costly, and expertise-demanding on software ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UbiComp '16: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing

September 2016

1288 pages

ISBN:9781450344616

DOI:10.1145/2971648

General Chairs:
Paul Lukowicz
DFKI and University of Kaiserslautern
,
Antonio Krüger
DFKI and Saarland University
,
Program Chairs:
Andreas Bulling
Max Planck Institute for Informatics
,
Youn-Kyung Lim
KAIST
,
Shwetak N. Patel
University of Washington

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 September 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

UbiComp '16

Sponsor:

UbiComp '16: The 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing

September 12 - 16, 2016

Germany, Heidelberg

Acceptance Rates

UbiComp '16 Paper Acceptance Rate 101 of 389 submissions, 26%;

Overall Acceptance Rate 764 of 2,912 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
577
Total Downloads

Downloads (Last 12 months)73
Downloads (Last 6 weeks)7

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Shin HPark JYu JKim JKim HOh C(2024)Looping In: Exploring Feedback Strategies to Motivate Human Engagement in Interactive Machine LearningInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2413293(1-18)Online publication date: 14-Oct-2024
https://doi.org/10.1080/10447318.2024.2413293
Covert IQiu WLu MKim NWhite NLee SKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Learning to maximize mutual information for dynamic feature selectionProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3618665(6424-6447)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3618665
Mishra SRzeszotarski J(2023)Human Expectations and Perceptions of Learning in Machine TeachingProceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization10.1145/3565472.3595612(13-24)Online publication date: 18-Jun-2023
https://dl.acm.org/doi/10.1145/3565472.3595612
Cao ZGuo YChen YLin D(2023)Bayesian-Based Symptom Screening for Medical Dialogue Diagnosis2023 IEEE Symposium on Computers and Communications (ISCC)10.1109/ISCC58397.2023.10218219(1-6)Online publication date: 9-Jul-2023
https://doi.org/10.1109/ISCC58397.2023.10218219
Reščič NMayora OEccher CLuštrek M(2022)Food Frequency Questionnaire Personalisation Using Multi-Target RegressionNutrients10.3390/nu1419394314:19(3943)Online publication date: 23-Sep-2022
https://doi.org/10.3390/nu14193943
He WMao XMa CHuang YHernàndez-Lobato JChen T(2022)BSODA: A Bipartite Scalable Framework for Online Disease DiagnosisProceedings of the ACM Web Conference 202210.1145/3485447.3512123(2511-2521)Online publication date: 25-Apr-2022
https://dl.acm.org/doi/10.1145/3485447.3512123
Hong JTrubuil AIsenberg T(2022)LineageD: An Interactive Visual System for Plant Cell Lineage Assignments based on Correctable Machine LearningComputer Graphics Forum10.1111/cgf.1453341:3(195-207)Online publication date: 12-Aug-2022
https://doi.org/10.1111/cgf.14533
Erion GJanizek JHudelson CUtarnachitt RMcCoy ASayre MWhite NLee S(2022)A cost-aware framework for the development of AI models for healthcare applicationsNature Biomedical Engineering10.1038/s41551-022-00872-86:12(1384-1398)Online publication date: 7-Apr-2022
https://doi.org/10.1038/s41551-022-00872-8
Chikersal PDoryab ATumminia MVillalba DDutcher JLiu XCohen SCreswell KMankoff JCreswell JGoel MDey A(2021)Detecting Depression and Predicting its Onset Using Longitudinal Symptoms Captured by Passive SensingACM Transactions on Computer-Human Interaction10.1145/342282128:1(1-41)Online publication date: 20-Jan-2021
https://dl.acm.org/doi/10.1145/3422821
Liyanage YZois DChelmis C(2021)Dynamic Instance-wise Joint Feature Selection and ClassificationIEEE Transactions on Artificial Intelligence10.1109/TAI.2021.30772122:2(169-184)Online publication date: Apr-2021
https://doi.org/10.1109/TAI.2021.3077212
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents