research-article

IoT Bugs and Development Challenges

Authors:

Amir Makhshari,

Ali MesbahAuthors Info & Claims

ICSE '21: Proceedings of the 43rd International Conference on Software Engineering

Pages 460 - 472

https://doi.org/10.1109/ICSE43902.2021.00051

Published: 05 November 2021 Publication History

Abstract

IoT systems are rapidly adopted in various domains, from embedded systems to smart homes. Despite their growing adoption and popularity, there has been no thorough study to understand IoT development challenges from the practitioners' point of view. We provide the first systematic study of bugs and challenges that IoT developers face in practice, through a large-scale empirical investigation. We collected 5,565 bug reports from 91 representative IoT project repositories and categorized a random sample of 323 based on the observed failures, root causes, and the locations of the faulty components. In addition, we conducted nine interviews with IoT experts to uncover more details about IoT bugs and to gain insight into IoT developers' challenges. Lastly, we surveyed 194 IoT developers to validate our findings and gain further insights. We propose the first bug taxonomy for IoT systems based on our results. We highlight frequent bug categories and their root causes, correlations between them, and common pitfalls and challenges that IoT developers face. We recommend future directions for IoT areas that require research and development attention.

References

[1]

R. Minerva, A. Biru, and D. Rotondi, "Towards a definition of the internet of things (IoT)," IEEE Internet Initiative, vol. 1, no. 1, pp. 1--86, 2015.

[2]

M. Hung, "Leading the IoT, Gartner insights on how to lead in a connected world," Gartner Research, pp. 1--29, 2017.

[3]

F. Schwandt, "Internet of things (IoT) connected devices installed base worldwide from 2015 to 2025 (in billions)," Statista, 2016.

[4]

F. Corno, L. De Russis, and J. P. Sáenz, "How is open source software development different in popular IoT projects?" IEEE Access, vol. 8, pp. 28 337-28 348, 2020.

[5]

T. W. Hnat, V. Srinivasan, J. Lu, T. I. Sookoor, R. Dawson, J. Stankovic, and K. Whitehouse, "The hitchhiker's guide to successful residential sensing deployments," in Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems, 2011, pp. 232--245.

Digital Library

[6]

F. Corno, L. De Russis, and J. P. Sáenz, "On the challenges novice programmers experience in developing IoT systems: A survey," Journal of Systems and Software, vol. 157, p. 110389, 2019.

[7]

B. L. R. Stojkoska and K. V. Trivodaliev, "A review of internet of things for smart home: Challenges and solutions," Journal of Cleaner Production, vol. 140, pp. 1454--1464, 2017.

[8]

Y. Chen, Z. Zhen, H. Yu, and J. Xu, "Application of fault tree analysis and fuzzy neural networks to fault diagnosis in the internet of things (IoT) for aquaculture," Sensors, vol. 17, no. 1, p. 153, 2017.

[9]

H. Liang, Q. Zhao, Y. Wang, and H. Liu, "Understanding and detecting performance and security bugs in IoT oses," in 2016 17th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD). IEEE, 2016, pp. 413--418.

[10]

G. Jahangirova, N. Humbatova, G. Bavota, V. Riccio, A. Stocco, and P. Tonella, "Taxonomy of real faults in deep learning systems," arXiv preprint arXiv:1910.11015, 2019.

[11]

Z. Wan, D. Lo, X. Xia, and L. Cai, "Bug characteristics in blockchain systems: a large-scale empirical study," in 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). IEEE, 2017, pp. 413--424.

Digital Library

[12]

M. E. Joorabchi, A. Mesbah, and P. Kruchten, "Real challenges in mobile app development," in 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE, 2013, pp. 15--24.

[13]

A. Čolaković and M. Hadžialić, "Internet of things (IoT): A review of enabling technologies, challenges, and open research issues," Computer Networks, vol. 144, pp. 17--39, 2018.

Digital Library

[14]

E. I. W. Group et al., "The three software stacks required for IoT architectures," 2016.

[15]

F. Javed, M. K. Afzal, M. Sharif, and B.-S. Kim, "Internet of things (IoT) operating systems support, networking technologies, applications, and challenges: A comparative review," IEEE Communications Surveys & Tutorials, vol. 20, no. 3, pp. 2062--2100, 2018.

[16]

H. Tschofenig, J. Arkko, and D. McPherson, "Architectural considerations in smart object networking, internet engineering task force, rfc-7452," Internet Engineering Task Force, Fremont, CA, USA, 2014.

[17]

W. Zhou, Y. Jia, Y. Yao, L. Zhu, L. Guan, Y. Mao, P. Liu, and Y. Zhang, "Discovering and understanding the security hazards in the interactions between IoT devices, mobile apps, and clouds on smart home platforms," in 28th USENIX Security Symposium (USENIX Security), 2019, pp. 1133--1150.

Digital Library

[18]

Github, "Lamps not identified as lamps with f/w 1.3.14," 2018, https://github.com/ggravlingen/pytradfri/issues/135.

[19]

A. Makhshari and A. Mesbah, IoT Bugs and Development Challenges Artifact Package, August 2020, https://github.com/IoTSEstudy/IoTbugschallenges.

Digital Library

[20]

Classifying your repository with topics, https://help.github.com/en/github/administering-a-repository/classifying-your-repository-with-topics.

[21]

H. Borges and M. T. Valente, "What's in a github star? understanding repository starring practices in a social coding platform," Journal of Systems and Software, vol. 146, pp. 112--129, 2018.

[22]

L. Tan, C. Liu, Z. Li, X. Wang, Y. Zhou, and C. Zhai, "Bug characteristics in open source software," Empirical software engineering, vol. 19, no. 6, pp. 1665--1705, 2014.

Digital Library

[23]

A. Avizienis, J.-C. Laprie, B. Randell, and C. Landwehr, "Basic concepts and taxonomy of dependable and secure computing," IEEE transactions on dependable and secure computing, vol. 1, no. 1, pp. 11--33, 2004.

Digital Library

[24]

O. Serrat, "The five whys technique," in Knowledge solutions. Springer, 2017, pp. 307--310.

[25]

C. B. Seaman, "Qualitative methods in empirical studies of software engineering," IEEE Transactions on software engineering, vol. 25, no. 4, pp. 557--572, 1999.

Digital Library

[26]

P. I. Fusch and L. R. Ness, "Are we there yet? data saturation in qualitative research," The qualitative report, vol. 20, no. 9, p. 1408, 2015.

[27]

M. D. C. Tongco, "Purposive sampling as a tool for informant selection," Ethnobotany Research and applications, vol. 5, pp. 147--158, 2007.

[28]

J. M. Morse, "Data were saturated... " 2015.

[29]

G. Guest, A. Bunce, and L. Johnson, "How many interviews are enough? an experiment with data saturation and variability," Field methods, vol. 18, no. 1, pp. 59--82, 2006.

[30]

L. Singer, F. Figueira Filho, and M.-A. Storey, "Software engineering at the speed of light: how developers stay current using twitter," in Proceedings of the 36th International Conference on Software Engineering, 2014, pp. 211--221.

Digital Library

[31]

M. Aniche, C. Treude, I. Steinmacher, I. Wiese, G. Pinto, M.-A. Storey, and M. A. Gerosa, "How modern news aggregators help development communities shape and share knowledge," in 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE). IEEE, 2018, pp. 499--510.

Digital Library

[32]

J. Henrich, S. J. Heine, and A. Norenzayan, "The weirdest people in the world?" Behavioral and brain sciences, vol. 33, no. 2-3, pp. 61--83, 2010.

[33]

G. Coleman and R. O'Connor, "Using grounded theory to understand software process improvement: A study of irish software product companies," Information and Software Technology, vol. 49, no. 6, pp. 654--667, 2007.

Digital Library

[34]

M. Usman, R. Britto, J. Börstler, and E. Mendes, "Taxonomies in software engineering: A systematic mapping study and a revised taxonomy development method," Information and Software Technology, vol. 85, pp. 43--59, 2017.

Digital Library

[35]

B. H. Kwasnik, "The role of classification in knowledge representation and discovery," Graduate School of Library and Information Science. University of Illinois ..., 1999.

[36]

vyshwanara, "Lack of hardware clock in raspberry pi - possible time lag issues," 2018. [Online]. Available: https://blog.pisignage.com/lack-of-hardware-clock-in-raspberry-pi-scheduling-issues/

[37]

M. Alhanahnah, C. Stevens, and H. Bagheri, "Scalable analysis of interaction threats in IoT systems," in Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2020, pp. 272--285.

Digital Library

[38]

M. Kamber, J. Pei et al., Data mining: Concepts and techniques. Morgan Kaufmann Publishers San Francisco, 2001, vol. 2.

[39]

J. P. Dias, F. Couto, A. C. Paiva, and H. S. Ferreira, "A brief overview of existing tools for testing the internet-of-things," in 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2018, pp. 104--109.

[40]

P. M. Pontes, B. Lima, and J. P. Faria, "Test patterns for IoT," in Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation, 2018, pp. 63--66.

Digital Library

[41]

IOTIFITY, "Advanced IoT system simulation engine and test automation for enterprise IoT apps." [Online]. Available: https://iotify.io/

[42]

V. Looga, Z. Ou, Y. Deng, and A. Ylä-Jääski, "Mammoth: A massive-scale emulation platform for internet of things," in 2012 IEEE 2nd International Conference on Cloud Computing and Intelligence Systems, vol. 3. IEEE, 2012, pp. 1235--1239.

[43]

M. Murdoch, "Arduinounit," 2013. [Online]. Available: https://github. com/mmurdoch/arduinounit

[44]

I. Kravets, "Platformio: An open source ecosystem for IoT development," PlatformIO.[En ligne]. Disponible sur: https://platformio.org.[Consulté le: 25-sept-2019], 2018.

[45]

C. Adjih, E. Baccelli, E. Fleury, G. Harter, N. Mitton, T. Noel, R. Pissard-Gibollet, F. Saint-Marcel, G. Schreiner, J. Vandaele et al., "Fit IoT-lab: A large scale open experimental IoT testbed," in 2015 IEEE 2nd World Forum on Internet of Things (WF-IoT). IEEE, 2015, pp. 459--464.

Digital Library

[46]

J. Voas, R. Kuhn, and P. Laplante, "Testing IoT systems," in 2018 IEEE Symposium on Service-Oriented System Engineering (SOSE). IEEE, 2018, pp. 48--52.

[47]

V. Song, "The never ending death of smart home gadgets," Mar 2020. [Online]. Available: https://gizmodo.com/the-never-ending-death-of-smart-home-gadgets-1842456125

[48]

P. Group, "Why can't my dishwasher program my tv? the problems with smart homes," February 2020. [Online]. Available: https://www.iotforall.com/smart-home-problems/

[49]

C. Towers-Clark, "Uk to introduce new law for IoT device security," May 2019. [Online]. Available: https://www.forbes.com/sites/charlestowersclark/2019/05/02/uk-to-introduce-new-law-for-iot-device-security/#72f4d763579d

[50]

A. Nguyen-Duc, R. Jabangwe, P. Paul, and P. Abrahamsson, "Security challenges in IoT development: a software engineering perspective." in XP Workshops, 2017, pp. 11--1.

[51]

Y. Xiao, Y. Jia, C. Liu, X. Cheng, J. Yu, and W. Lv, "Edge computing security: State of the art and challenges," Proceedings of the IEEE, vol. 107, no. 8, pp. 1608--1631, 2019.

[52]

G. Hernandez, O. Arias, D. Buentello, and Y. Jin, "Smart nest thermostat: A smart spy in your home," Black Hat USA, no. 2015, 2014.

[53]

Z. Ling, J. Luo, Y. Xu, C. Gao, K. Wu, and X. Fu, "Security vulnerabilities of internet of things: A case study of the smart plug system," IEEE Internet of Things Journal, vol. 4, no. 6, pp. 1899--1909, 2017.

[54]

S. Notra, M. Siddiqi, H. H. Gharakheili, V. Sivaraman, and R. Boreli, "An experimental study of security and privacy risks with emerging household appliances," in 2014 IEEE conference on communications and network security. IEEE, 2014, pp. 79--84.

[55]

E. Ronen, A. Shamir, A.-O. Weingarten, and C. O'Flynn, "IoT goes nuclear: Creating a zigbee chain reaction," in 2017 IEEE Symposium on Security and Privacy (SP). IEEE, 2017, pp. 195--212.

[56]

R. Goyal, N. Dragoni, and A. Spognardi, "Mind the tracker you wear: a security analysis of wearable health trackers," in Proceedings of the 31st Annual ACM Symposium on Applied Computing, 2016, pp. 131--136.

Digital Library

[57]

B. Fouladi and S. Ghanoun, "Honey, i'm home!!, hacking zwave home automation systems," Black Hat USA, 2013.

[58]

Z. B. Celik, G. Tan, and P. D. McDaniel, "IoTguard: Dynamic enforcement of security and safety policy in commodity IoT." in NDSS, 2019.

[59]

Z. B. Celik, P. McDaniel, and G. Tan, "Soteria: Automated IoT safety and security analysis," in Annual Technical Conference (USENIX ATC), 2018, pp. 147--158.

Digital Library

[60]

I. Alqassem and D. Svetinovic, "A taxonomy of security and privacy requirements for the internet of things (IoT)," in 2014 IEEE International Conference on Industrial Engineering and Engineering Management. IEEE, 2014, pp. 1244--1248.

[61]

K. Chen, S. Zhang, Z. Li, Y. Zhang, Q. Deng, S. Ray, and Y. Jin, "Internet-of-things security and vulnerabilities: Taxonomy, challenges, and practice," Journal of Hardware and Systems Security, vol. 2, no. 2, pp. 97--110, 2018.

[62]

A. Ahmad, F. Bouquet, E. Fourneret, F. Le Gall, and B. Legeard, "Model-based testing as a service for IoT platforms," in International Symposium on Leveraging Applications of Formal Methods. Springer, 2016, pp. 727--742.

[63]

P. Rosenkranz, M. Wählisch, E. Baccelli, and L. Ortmann, "A distributed test system architecture for open-source IoT software," in Proceedings of the 2015 Workshop on IoT challenges in Mobile and Industrial Systems, 2015, pp. 43--48.

Digital Library

[64]

L. Gutiérrez-Madroñal, A. García-Domínguez, and I. Medina-Bulo, "Evolutionary mutation testing for IoT with recorded and generated events," Software: Practice and Experience, vol. 49, no. 4, pp. 640--672, 2019.

[65]

L. Gutiérrez-Madroñal, I. Medina-Bulo, and J. J. Domínguez-Jiménez, "IoT-teg: Test event generator system," Journal of Systems and Software, vol. 137, pp. 784--803, 2018.

[66]

B. Morin, N. Harrand, and F. Fleurey, "Model-based software engineering to tame the IoT jungle," IEEE Software, vol. 34, no. 1, pp. 30--36, 2017.

Digital Library

[67]

A. Krishna, M. Le Pallec, R. Mateescu, L. Noirie, and G. Salaün, "IoT composer: Composition and deployment of IoT applications," in 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 2019, pp. 19--22.

Digital Library

[68]

F. Corno, L. De Russis, and J. P. Sáenz, "Towards computational notebooks for IoT development," in Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, 2019, pp. 1--6.

Digital Library

[69]

R. Zhang, W. Xiao, H. Zhang, Y. Liu, H. Lin, and M. Yang, "An empirical study on program failures of deep learning jobs," in 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). IEEE, 2020, pp. 1159--1170.

Digital Library

[70]

M. J. Islam, G. Nguyen, R. Pan, and H. Rajan, "A comprehensive study on deep learning bug characteristics," in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2019, pp. 510--520.

Digital Library

[71]

Y. Zhang, Y. Chen, S.-C. Cheung, Y. Xiong, and L. Zhang, "An empirical study on tensorflow program bugs," in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2018, pp. 129--140.

Digital Library

[72]

H. Zhou, J.-G. Lou, H. Zhang, H. Lin, H. Lin, and T. Qin, "An empirical study on quality issues of production big data platform," in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol. 2. IEEE, 2015, pp. 17--26.

Digital Library

[73]

F. S. Ocariza, K. Bajaj, K. Pattabiraman, and A. Mesbah, "A study of causes and consequences of client-side javascript bugs," IEEE Transactions on Software Engineering, vol. 43, no. 2, pp. 128--144, 2016.

Digital Library

[74]

K. M. Chan, J. Bishop, J. Steyn, L. Baresi, and S. Guinea, "A fault taxonomy for web service composition," in International Conference on Service-Oriented Computing. Springer, 2007, pp. 363--375.

[75]

W. Zou, D. Lo, P. S. Kochhar, X.-B. D. Le, X. Xia, Y. Feng, Z. Chen, and B. Xu, "Smart contract development: Challenges and opportunities," IEEE Transactions on Software Engineering, 2019.

Cited By

Rojas Castillo CMarra MGonzalez Boix EGonzalez Boix EScholliers C(2024)Language-Agnostic Debugging for MicrocontrollersProceedings of the 2nd ACM International Workshop on Future Debugging Techniques10.1145/3678720.3685317(22-27)Online publication date: 13-Sep-2024
https://dl.acm.org/doi/10.1145/3678720.3685317
Liang WXiao G(2024)An Exploratory Evaluation of Large Language Models Using Empirical Software Engineering TasksProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3674821(31-40)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3674821
Hassan MSalvador JSantu SRahman A(2024)State Reconciliation Defects in Infrastructure as CodeProceedings of the ACM on Software Engineering10.1145/36607901:FSE(1865-1888)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660790
Show More Cited By

Index Terms

IoT Bugs and Development Challenges
1. Human-centered computing
  1. Visualization

Index terms have been assigned to the content through auto-classification.

Recommendations

IoT development in the wild: bug taxonomy and developer challenges
ICSE '21: Proceedings of the 43rd International Conference on Software Engineering: Companion Proceedings

IoT systems are rapidly adopted in various domains, from embedded systems to smart homes. Despite their growing adoption and popularity, there has been no thorough study to understand IoT development challenges from the practitioners' point of view. We ...
Security challenges in IoT development: a software engineering perspective
XP '17: Proceedings of the XP2017 Scientific Workshops

The rapid growth of Internet-of-things (IoT) software applications has driven both practitioners and researchers' attention to methodological approaches for secure IoT development. Security issues for IoT is special in the way that they include not only ...
RFC 8576: Internet of Things (IoT) Security: State of the Art and Challenges

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '21: Proceedings of the 43rd International Conference on Software Engineering

May 2021

1768 pages

ISBN:9781450390859

Sponsors

Publisher

IEEE Press

Publication History

Published: 05 November 2021

Check for updates

Badges

Artifacts Evaluated & Functional / v1.1

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICSE '21

Sponsor:

SIGSOFT

ICSE '21: 43rd International Conference on Software Engineering

May 22 - 30, 2021

Madrid, Spain

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
111
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)5

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Rojas Castillo CMarra MGonzalez Boix EGonzalez Boix EScholliers C(2024)Language-Agnostic Debugging for MicrocontrollersProceedings of the 2nd ACM International Workshop on Future Debugging Techniques10.1145/3678720.3685317(22-27)Online publication date: 13-Sep-2024
https://dl.acm.org/doi/10.1145/3678720.3685317
Liang WXiao G(2024)An Exploratory Evaluation of Large Language Models Using Empirical Software Engineering TasksProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3674821(31-40)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3674821
Hassan MSalvador JSantu SRahman A(2024)State Reconciliation Defects in Infrastructure as CodeProceedings of the ACM on Software Engineering10.1145/36607901:FSE(1865-1888)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660790
Minani JSabir FEl Fellah YMoha N(2024)Practical Guidance for IoT Systems Testing: A TaxonomyProceedings of the ACM/IEEE 6th International Workshop on Software Engineering Research & Practices for the Internet of Things10.1145/3643794.3648349(57-64)Online publication date: 20-Apr-2024
https://dl.acm.org/doi/10.1145/3643794.3648349
De Santana TNeto PDe Almeida EAhmed I(2024)Bug Analysis in Jupyter Notebook Projects: An Empirical StudyACM Transactions on Software Engineering and Methodology10.1145/364153933:4(1-34)Online publication date: 18-Apr-2024
https://dl.acm.org/doi/10.1145/3641539
Rahman ABose DBarsha FPandita R(2023)Defect Categorization in Compilers: A Multi-vocal Literature ReviewACM Computing Surveys10.1145/362631356:4(1-42)Online publication date: 10-Nov-2023
https://dl.acm.org/doi/10.1145/3626313
Lauwaerts TRojas Castillo CGonzalez Boix EScholliers CGonzalez Boix EScholliers C(2023)Out-of-Place Debugging on Constraint Devices with the EDWARD Debugger (Demo)Proceedings of the 1st ACM International Workshop on Future Debugging Techniques10.1145/3605155.3605862(3-4)Online publication date: 17-Jul-2023
https://dl.acm.org/doi/10.1145/3605155.3605862
Lauwaerts TRojas Castillo CGonzalez Boix EScholliers CHui PAmiri Sani ANurmi PLiu Y(2023)Demo: Debugging Constraint Devices with EDWARDProceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services10.1145/3581791.3597293(602-603)Online publication date: 18-Jun-2023
https://dl.acm.org/doi/10.1145/3581791.3597293
Altarrazi SSzydlo TDustdar SSrirama SRanjan R(2023)Addressing the Faults Landscape in the Internet of Things: Toward Datacentric and System ResilienceIEEE Internet Computing10.1109/MIC.2023.330050827:6(43-51)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1109/MIC.2023.3300508
Yusuf IGrundy J(2023)Towards Automated Embedded Systems ProgrammingProceedings of the 45th International Conference on Software Engineering: Companion Proceedings10.1109/ICSE-Companion58688.2023.00061(224-226)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE-Companion58688.2023.00061
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents