research-article

A weakly supervised activity recognition framework for real-time synthetic biology laboratory assistance

Authors:

Chandrashekhar Lavania,

Sunil Thulasidasan,

Anthony LaMarca,

Jeffrey Scofield,

Jeff BilmesAuthors Info & Claims

UbiComp '16: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing

Pages 37 - 48

https://doi.org/10.1145/2971648.2971716

Published: 12 September 2016 Publication History

Abstract

We describe the design of a hybrid system -- a combination of a Dynamic Graphical Model (DGM) with a Deep Neural Network (DNN) -- to identify activities performed during synthetic biology experiments. The purpose is to provide real-time feedback to experimenters, thus helping to reduce human errors and improve experimental reproducibility. The data consists of unlabeled videos of recorded experiments and "weakly supervised" information (i.e., "theoretical" and asynchronous knowledge of sets of high level activity sequences in the experiment) used to train the system. Multiple activity sequences are modeled using a trellis, and deep features are extracted from video images. Model performance is accessed using real-time online statistical inference. The trellis incorporates variations during experiment execution, making our model very general and capable of high performance.

References

[1]

Hossein Azizpour, Ali Razavian, Josephine Sullivan, Atsuto Maki, and Stefan Carlsson. 2015. From generic to specific deep representations for visual recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 36--45.

[2]

John MS Bartlett and David Stirling. 2003. A short history of the polymerase chain reaction. In PCR protocols. Springer, 3--6.

[3]

Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. 2006. Surf: Speeded up robust features. In Computer vision--ECCV 2006. Springer, 404--417.

Digital Library

[4]

C Glenn Begley and Lee M Ellis. 2012. Drug development: Raise standards for preclinical cancer research. Nature 483, 7391 (2012), 531--533.

[5]

Steven A Benner and A Michael Sismour. 2005. Synthetic biology. Nature Reviews Genetics 6, 7 (2005), 533--543.

[6]

Jeff Bilmes. 2004. On virtual evidence and soft evidence in bayesian networks. (2004).

[7]

Jeff Bilmes and Geoffrey Zweig. 2002. The graphical models toolkit: An open source software system for speech and time-series processing. In Acoustics, Speech, and Signal Processing (ICASSP), 2002 IEEE International Conference on, Vol. 4. IV--3916.

[8]

Jeff A. Bilmes. 2014. Dynamic Graphical Models and the Graphical Models Toolkit (GMTK). University of Washington, Seattle.

[9]

Léon Bottou. 2012. Stochastic gradient descent tricks. In Neural Networks: Tricks of the Trade. Springer, 421--436.

[10]

H. Bourlard and N. Morgan. 1994. Connectionist Speech Recognition: A Hybrid Approach. Kluwer Academic Publishers.

Digital Library

[11]

Eugene Charniak and Robert P Goldman. 1993. A Bayesian model of plan recognition. Artificial Intelligence 64, 1 (1993), 53--79.

Digital Library

[12]

Ronald R Coifman and Stéphane Lafon. 2006. Diffusion maps. Applied and computational harmonic analysis 21, 1 (2006), 5--30.

[13]

Navneet Dalal and Bill Triggs. 2005. Histograms of oriented gradients for human detection. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, Vol. 1. IEEE, 886--893.

Digital Library

[14]

Arthur P Dempster, Nan M Laird, and Donald B Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the royal statistical society. Series B (methodological) (1977), 1--38.

[15]

Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2625--2634.

[16]

Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. 2014. DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. In Proceedings of The 31st International Conference on Machine Learning. 647--655.

Digital Library

[17]

John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research 12 (2011), 2121--2159.

Digital Library

[18]

Clément Farabet, Camille Couprie, Laurent Najman, and Yann LeCun. 2013. Learning hierarchical features for scene labeling. Pattern Analysis and Machine Intelligence, IEEE Transactions on 35, 8 (2013), 1915--1929.

Digital Library

[19]

Eileen Fung, Wilson W Wong, Jason K Suen, Thomas Bulter, Sun-gu Lee, and James C Liao. 2005. A synthetic gene--metabolic oscillator. Nature 435, 7038 (2005), 118--122.

[20]

Timothy S Gardner, Charles R Cantor, and James J Collins. 2000. Construction of a genetic toggle switch in Escherichia coli. Nature 403, 6767 (2000), 339--342.

[21]

Daniel G Gibson, Lei Young, Ray-Yuan Chuang, J Craig Venter, Clyde A Hutchison, and Hamilton O Smith. 2009. Enzymatic assembly of DNA molecules up to several hundred kilobases. Nature methods 6, 5 (2009), 343--345.

[22]

Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 6645--6649.

[23]

Erika Check Hayden. 2015. Synthetic biologists seek standards for nascent field. Nature 520, 7546 (2015), 141--142.

[24]

Harlan Hile, Jiwon Kim, and Gaetano Borriello. 2004. Microbiology tray and pipette tracking as a proactive tangible user interface. In Pervasive Computing. Springer, 323--339.

[25]

William C Hines, Ying Su, Irene Kuhn, Kornelia Polyak, and Mina J Bissell. 2014. Sorting out the FACS: a devil in the details. Cell reports 6, 5 (2014), 779--781.

[26]

Alan Julian Izenman. 2008. Linear discriminant analysis. In Modern Multivariate Statistical Techniques. Springer, 237--280.

[27]

Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 2013. 3D convolutional neural networks for human action recognition. Pattern Analysis and Machine Intelligence, IEEE Transactions on 35, 1 (2013), 221--231.

Digital Library

[28]

Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia. ACM, 675--678.

Digital Library

[29]

Ian Jolliffe. 2002. Principal component analysis. Wiley Online Library.

[30]

Henry A Kautz. 1987. A formal theory of plan recognition. Ph.D. Dissertation. Bell Laboratories.

Digital Library

[31]

Shian-Ru Ke, Hoang Le Uyen Thuc, Yong-Jin Lee, Jenq-Neng Hwang, Jang-Hee Yoo, and Kyoung-Ho Choi. 2013. A review on video-based human activity recognition. Computers 2, 2 (2013), 88--131.

[32]

Virginia C Klema and Alan J Laub. 1980. The singular value decomposition: Its computation and some applications. Automatic Control, IEEE Transactions on 25, 2 (1980), 164--176.

[33]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097--1105.

Digital Library

[34]

Neal Lesh and Oren Etzioni. 1995. A sound and fast goal recognizer. In IJCAI, Vol. 95. 1704--1710.

Digital Library

[35]

David G Lowe. 1999. Object recognition from local scale-invariant features. In Computer vision, 1999. The proceedings of the seventh IEEE international conference on, Vol. 2. Ieee, 1150--1157.

Digital Library

[36]

Sebastian Lühr, Hung H Bui, Svetha Venkatesh, and Geoff AW West. 2003. Recognition of human activity through hierarchical stochastic learning. In 2013 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE Computer Society, 416--416.

Digital Library

[37]

Aravindh Mahendran and Andrea Vedaldi. 2015. Understanding deep image representations by inverting them. In Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on. IEEE, 5188--5196.

[38]

Pedro J Moreno and Christopher Alberti. 2009. A factor automaton approach for the forced alignment of long speech recordings. In Acoustics, Speech and Signal Processing, 2009. ICASSP 2009. IEEE International Conference on. IEEE, 4869--4872.

Digital Library

[39]

Justin Muncaster and Yunqian Ma. 2007. Hierarchical model-based activity recognition with automatic low-level state discovery. Journal of Multimedia 2, 5 (2007), 66--76.

[40]

Judea Pearl. 1988. Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan Kaufmann.

Digital Library

[41]

Xianbiao Qi, Chun-Guang Li, Guoying Zhao, Xiaopeng Hong, and Matti Pietikäinen. 2016. Dynamic texture and scene classification by transferring deep image features. Neurocomputing 171 (2016), 1230--1241.

Digital Library

[42]

Philipp M Scholl, Tobias Schultes, and Kristof Van Laerhoven. 2015a. RFID-based compound identification in wet laboratories with google glass. In Proceedings of the 2nd international Workshop on Sensor-based Activity Recognition and Interaction. ACM, 13.

Digital Library

[43]

Philipp M Scholl, Matthias Wille, and Kristof Van Laerhoven. 2015b. Wearables in the wet lab: a laboratory system for capturing and guiding experiments. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 589--599.

Digital Library

[44]

Rafael Silva-Rocha and Víctor de Lorenzo. 2008. Mining logic gates in prokaryotic transcriptional regulation networks. FEBS letters 582, 8 (2008), 1237--1244.

[45]

Yi Sun, Xiaogang Wang, and Xiaoou Tang. 2014. Deep learning face representation from predicting 10,000 classes. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. IEEE, 1891--1898.

Digital Library

[46]

Joshua B Tenenbaum, Vin De Silva, and John C Langford. 2000. A global geometric framework for nonlinear dimensionality reduction. Science 290, 5500 (2000), 2319--2323.

[47]

Tim Van Kasteren, Athanasios Noulas, Gwenn Englebienne, and Ben Kröse. 2008. Accurate activity recognition in a home setting. In Proceedings of the 10th international conference on Ubiquitous computing. ACM, 1--9.

Digital Library

[48]

Tim LM van Kasteren, Gwenn Englebienne, and Ben JA Kröse. 2011. Hierarchical activity recognition using automatically clustered actions. In Ambient Intelligence. Springer, 82--91.

Digital Library

[49]

Daniel Weinland, Remi Ronfard, and Edmond Boyer. 2011. A survey of vision-based methods for action representation, segmentation and recognition. Computer Vision and Image Understanding 115, 2 (2011), 224--241.

Digital Library

Cited By

Rebmann ARehse JPinter MSchnaubelt MDaun KFettke P(2020)IoT-Based Activity Recognition for Process Assistance in Human-Robot Disaster ResponseBusiness Process Management Forum10.1007/978-3-030-58638-6_5(71-87)Online publication date: 2-Sep-2020
https://doi.org/10.1007/978-3-030-58638-6_5
Avrahami DPatel MYamaura YKratz SCooper M(2019)Unobtrusive Activity Recognition and Position Estimation for Work Surfaces Using RF-Radar SensingACM Transactions on Interactive Intelligent Systems10.1145/324138310:1(1-28)Online publication date: 9-Aug-2019
https://dl.acm.org/doi/10.1145/3241383
Avrahami DPatel MYamaura YKratz SBerkovsky SHijikata YRekimoto JBurnett MBillinghurst MQuigley A(2018)Below the SurfaceProceedings of the 23rd International Conference on Intelligent User Interfaces10.1145/3172944.3172962(439-451)Online publication date: 5-Mar-2018
https://dl.acm.org/doi/10.1145/3172944.3172962
Show More Cited By

Index Terms

A weakly supervised activity recognition framework for real-time synthetic biology laboratory assistance
1. Computing methodologies
  1. Machine learning
2. Information systems
  1. Information systems applications
    1. Decision support systems
      1. Expert systems

Recommendations

Time handling for real-time progressive activity recognition
SAGAware '11: Proceedings of the 2011 international workshop on Situation activity & goal awareness

In a dense sensor-based smart home (SH), a significant challenge is to segment the sensor data stream in real-time to continuously support progressive activity recognition (AR). In this paper, we evaluate an approach that supports the segmentation of ...
Weakly Supervised Recognition of Daily Life Activities with Wearable Sensors

This paper considers scalable and unobtrusive activity recognition using on-body sensing for context awareness in wearable computing. Common methods for activity recognition rely on supervised learning requiring substantial amounts of labeled training ...
Real-time crowd labeling for deployable activity recognition
CSCW '13: Proceedings of the 2013 conference on Computer supported cooperative work

Systems that automatically recognize human activities offer the potential of timely, task-relevant information and support. For example, prompting systems can help keep people with cognitive disabilities on track and surveillance systems can warn of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UbiComp '16: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing

September 2016

1288 pages

ISBN:9781450344616

DOI:10.1145/2971648

General Chairs:
Paul Lukowicz
DFKI and University of Kaiserslautern
,
Antonio Krüger
DFKI and Saarland University
,
Program Chairs:
Andreas Bulling
Max Planck Institute for Informatics
,
Youn-Kyung Lim
KAIST
,
Shwetak N. Patel
University of Washington

Copyright © 2016 ACM.

Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 September 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

UbiComp '16

Sponsor:

UbiComp '16: The 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing

September 12 - 16, 2016

Germany, Heidelberg

Acceptance Rates

UbiComp '16 Paper Acceptance Rate 101 of 389 submissions, 26%;

Overall Acceptance Rate 764 of 2,912 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
510
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)1

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Rebmann ARehse JPinter MSchnaubelt MDaun KFettke P(2020)IoT-Based Activity Recognition for Process Assistance in Human-Robot Disaster ResponseBusiness Process Management Forum10.1007/978-3-030-58638-6_5(71-87)Online publication date: 2-Sep-2020
https://doi.org/10.1007/978-3-030-58638-6_5
Avrahami DPatel MYamaura YKratz SCooper M(2019)Unobtrusive Activity Recognition and Position Estimation for Work Surfaces Using RF-Radar SensingACM Transactions on Interactive Intelligent Systems10.1145/324138310:1(1-28)Online publication date: 9-Aug-2019
https://dl.acm.org/doi/10.1145/3241383
Avrahami DPatel MYamaura YKratz SBerkovsky SHijikata YRekimoto JBurnett MBillinghurst MQuigley A(2018)Below the SurfaceProceedings of the 23rd International Conference on Intelligent User Interfaces10.1145/3172944.3172962(439-451)Online publication date: 5-Mar-2018
https://dl.acm.org/doi/10.1145/3172944.3172962
Zhang JZhang QLi DZhao RWang DGu TKotagiri RLiu H(2017)RFlow-IDProceedings of the 14th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services10.1145/3144457.3144463(333-342)Online publication date: 7-Nov-2017
https://dl.acm.org/doi/10.1145/3144457.3144463

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents