Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3611643.3616264acmconferencesArticle/Chapter ViewAbstractPublication PagesfseConference Proceedingsconference-collections
research-article

Automatically Resolving Dependency-Conflict Building Failures via Behavior-Consistent Loosening of Library Version Constraints

Published: 30 November 2023 Publication History

Abstract

Python projects grow quickly by code reuse and building automation based on third-party libraries. However, the version constraints associated with these libraries are prone to mal-configuration, and this forms a major obstacle to correct project building (known as dependency-conflict (DC) building failure). Our empirical findings suggest that such mal-configured version constraints were mainly prepared manually, and could essentially be refined for better quality to improve the chance of successful project building. We propose a LooCo approach to refining Python projects’ library version constraints by automatically loosening them to maximize their solutions, while keeping the libraries to observe their original behaviors. Our experimental results with real-life Python projects report that LooCo could efficiently refine library version constraints (0.4s per version loosening) by effective loosening (5.5 new versions expanded on average) automatically, and transform 54.8% originally unsolvable cases into solvable ones (i.e., successful building) and significantly increase solutions (21 more on average) for originally solvable cases.

Supplementary Material

Video (fse23main-p191-p-video.mp4)
"Python projects grow quickly by code reuse and building automation based on third-party libraries. However, the version constraints associated with these libraries are prone to mal-configuration, and this forms a major obstacle to correct project building (known as as emph{dependency-conflict (DC) building failure}). Our empirical findings suggest that such mal-configured version constraints were mainly prepared manually, and could essentially be refined for better quality to improve the chance of successful project building. We propose a LooCo approach to refining Python projects’ library version constraints by automatically loosening them to maximize their solutions, while keeping the libraries to observe their original behaviors. Our experimental results with real-life Python projects report that LooCo could efficiently refine library version constraints (0.4s per version loosening) by effective loosening (5.5 new versions expanded on average) automatically, and transform 54.8% originally unsolvable cases into solvable ones (i.e., successful building) and significantly increase solutions (21 more on average) for originally solvable cases."

References

[1]
2023. Code2flow. https://code2flow.com/
[2]
2023. Conda. https://docs.conda.io/
[3]
2023. Dfflib. https://docs.python.org/3/library/difflib.html
[4]
2023. Filecmp. https://docs.python.org/3/library/filecmp.html
[5]
2023. Issue #131. https://github.com/garyd203/ssmash/issues/39
[6]
2023. Issue #272. https://github.com/chaoss/grimoirelab-manuscripts/issues/136
[7]
2023. Issue #56. https://github.com/ivanubi/api-indotel/issues/2
[8]
2023. Issue #99. https://github.com/KnowledgeLinks/rdfframework/issues/24
[9]
2023. LooCo website. https://agnes-u.github.io/LooCo/
[10]
2023. Maven. https://mvnrepository.com/repos
[11]
2023. PEP specifications. https://peps.python.org/
[12]
2023. PIP backtracking. https://pip.pypa.io/en/stable/topics/dependency-resolution/
[13]
2023. PIP freeze. https://pip.pypa.io/en/stable/cli/pip_freeze/
[14]
2023. PIP installer. https://pypi.org/project/pip/
[15]
2023. PIP legacy. https://pip.pypa.io/en/stable/user_guide/
[16]
2023. PyCG issues. https://github.com/vitsalis/PyCG/issues
[17]
2023. Python Package Index. https://pypi.org/
[18]
2023. Report #2135. https://github.com/andylokandy/rqalpha-mod-minute/issues/1
[19]
2023. Report #2688. https://github.com/gkeep/spotify-stats/issues/3
[20]
2023. Report #282. https://github.com/stratosphereips/StratosphereLinuxIPS/issues/163
[21]
2023. Report #700. https://github.com/yihong0618/running_page/issues/282
[22]
2023. SemVer. https://semver.org/
[23]
Pietro Abate, Roberto Di Cosmo, Georgios Gousios, and Stefano Zacchiroli. 2020. Dependency Solving Is Still Hard, but We Are Getting Better at It. In Proceedings of the 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER 2020). 547–551.
[24]
C. Artho, R. D. Cosmo, K. Suzaki, R. Treinen, and S. Zacchiroli. 2012. Why Do Software Packages Conflict? In Proceedings of the 2012 IEEE/ACM 9th International Conference on Mining Software Repositories (MSR 2012).
[25]
Aline Brito, Laerte Xavier, Andre Hora, and Marco Valente. 2018. APIDiff: Detecting API breaking changes. In Proceedings of the 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER 2018). 507–511.
[26]
Yulu Cao, Lin Chen, Wanwangying Ma, Yanhui Li, Yuming Zhou, and Linzhang Wang. 2023. Towards Better Dependency Management: A First Look at Dependency Smells in Python Projects. IEEE Transactions on Software Engineering, 49, 4 (2023), 1741–1765. https://doi.org/10.1109/TSE.2022.3191353
[27]
Wei Cheng, Xiangrong Zhu, and Wei Hu. 2022. Conflict-aware Inference of Python Compatible Runtime Environments with Domain Knowledge Graph.
[28]
Fraser D., Horner E., Jeronen J., and Massot P. [n. d.]. Pyan3: Offline call graph generator for Python 3. https://github.com/davidfraser/pyan
[29]
Jens Dietrich, David Pearce, Jacob Stringer, Amjed Tahir, and Kelly Blincoe. 2019. Dependency Versioning in the Wild. In Proceedings of the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR 2019). 349–359.
[30]
Darius Foo, Hendy Chua, Jason Yeo, Ming Yi Ang, and Asankhaya Sharma. 2018. Efficient Static Checking of Library Updates. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2018). ACM Press, Lake Buena Vista, FL, USA. 791–796.
[31]
Zhang G. and Wuxia J. 2023. Depends. https://github.com/multilang-depends/depends
[32]
G. Gharibi, R. Alanazi, and Y. Lee. 2018. Automatic Hierarchical Clustering of Static Call Graphs for Program Comprehension. In Proceedings of the 2018 IEEE International Conference on Big Data (Big Data). 4016–4025.
[33]
G. Gharibi, R. Tripathi, and Y. Lee. 2018. Code2graph: Automatic Generation of Static Call Graphs for Python Source Code. In Proceedings of the 2018 IEEE/ACM International Conference on Automated Software Engineering (ASE 2018). 880–883.
[34]
Joseph Hejderup, Arie van Deursen, and Georgios Gousios. 2018. Software Ecosystem Call Graph for Dependency Management. In Proceedings of the ACM/IEEE 40nd International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER 2018). ACM, New York, NY, USA. 101–104.
[35]
Johannes Henkel and Amer Diwan. 2005. CatchUp! Capturing and Replaying Refactorings to Support API Evolution. In Proceedings of the 27th International Conference on Software Engineering (ICSE 2005). Association for Computing Machinery, New York, NY, USA. 274–283.
[36]
Eric Horton and Chris Parnin. 2018. Gistable: Evaluating the Executability of Python Code Snippets on GitHub. In Proceedings of the 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME 2018). 217–227.
[37]
Eric Horton and Chris Parnin. 2019. DockerizeMe: Automatic Inference of Environment Dependencies for Python Code Snippets. In Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE 2019). IEEE, Montreal, QC, Canada. 328–338.
[38]
Eric Horton and Chris Parnin. 2019. V2: Fast Detection of Configuration Drift in Python. In Proceedings of the 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE 2019). IEEE, San Diego, CA, USA. 477–488.
[39]
Abbas Javan Jafari, Diego Elias Costa, Rabe Abdalkareem, Emad Shihab, and Nikolaos Tsantalis. 2022. Dependency Smells in JavaScript Projects. IEEE Trans. Softw. Eng., 48, 10 (2022), oct, 3790–3807. issn:0098-5589 https://doi.org/10.1109/TSE.2021.3106247
[40]
Zhouyang Jia, Shanshan Li, Tingting Yu, Chen Zeng, Erci Xu, Xiaodong Liu, Ji Wang, and Xiangke Liao. 2021. DepOwl: Detecting Dependency Bugs to Prevent Compatibility Failures. In Proceedings of the 43rd International Conference on Software Engineering (ICSE 2021). 86–98.
[41]
R. Kikas, G. Gousios, M. Dumas, and D. Pfahl. 2017. Structure and Evolution of Package Dependency Networks. In Proceedings of the 2017 IEEE/ACM International Conference on Mining Software Repositories (MSR 2017). 102–112.
[42]
Kula, R. Gaikovina, German, M. Daniel, Ouni, Ali, Ishio, Takashi, Inoue, and Katsuro. 2018. Do Developers Update Their Library Dependencies? An Empirical Study on the Impact of Security Advisories on Library Migration. Empirical Software Engineering, 23 (2018), 384–417.
[43]
Jasmine Latendresse, Suhaib Mujahid, Diego Elias Costa, and Emad Shihab. 2023. Not All Dependencies Are Equal: An Empirical Study on Production Dependencies in NPM. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE ’22). Association for Computing Machinery, New York, NY, USA. Article 73, 12 pages. isbn:9781450394758 https://doi.org/10.1145/3551349.3556896
[44]
Y. Li. 2019. Empirical Study of Python Call Graph. In Proceedings of the 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE 2019). 1274–1276.
[45]
Wanwangying Ma, Lin Chen, Xiangyu Zhang, Yang Feng, Zhaogui Xu, Zhifei Chen, Yuming Zhou, and Baowen Xu. 2020. Impact Analysis of Cross-project Bugs on Software Ecosystems. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE 2020). ACM, Seoul South Korea. 100–111.
[46]
Fabio Mancinelli, Jaap Boender, Roberto di Cosmo, Jerome Vouillon, Berke Durak, Xavier Leroy, and Ralf Treinen. 2006. Managing the Complexity of Large Free and Open Source Package-Based Software Distributions. In Proceedings of the 21st IEEE/ACM International Conference on Automated Software Engineering (ASE 2006). IEEE, Tokyo. 199–208.
[47]
Suchita Mukherjee, Abigail Almanza, and Cindy Rubio-González. 2021. Fixing Dependency Errors for Python Build Reproducibility. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2021), Cristian Cadar and Xiangyu Zhang (Eds.). ACM, Virtual Envent, Denmark. 439–451.
[48]
Yuen Tak YU Sau-Fun TANG Pak-Lok POON, Man Fai LAU. 2024. Spreadsheet quality assurance: a literature review. Frontiers of Computer Science (FCS), 18, 2 (2024), 182203.
[49]
Vitalis Salis, Thodoris Sotiropoulos, Panos Louridas, Diomidis Spinellis, and Dimitris Mitropoulos. 2021. PyCG: Practical Call Graph Generation in Python. In Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE 2021). IEEE, Madrid, ES. 1646–1657.
[50]
Chao Wang, Rongxin Wu, Haohao Song, Jiwu Shu Shu, and Guoqing Li. 2022. SmartPip: A Smart Approach to Resolving Python Dependency Conflict Issues. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE 2022). Oakland Center, Michigan, United States.
[51]
Jiawei Wang, Li Li, Kui Liu, and Haipeng Cai. 2020. Exploring How Deprecated Python Library APIs Are (not) Handled. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2020). ACM, Virtual Event USA. 233–244.
[52]
Ying Wang, Bihuan Chen, Kaifeng Huang, Bowen Shi, Congying Xu, Xin Peng, Yijian Wu, and Yang Liu. 2020. An Empirical Study of Usages, Updates and Risks of Third-Party Libraries in Java Projects. In Proceedings of the 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME 2020). 35–45.
[53]
Ying Wang, Ming Wen, Yepang Liu, Yibo Wang, Zhenming Li, Chao Wang, Hai Yu, Shing-Chi Cheung, Chang Xu, and Zhiliang Zhu. 2020. Watchman: Monitoring Dependency Conflicts for Python Library Ecosystem. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE 2020). ACM, Seoul South Korea. 125–135.
[54]
R. Widyasari, Q. S. Sheng, C. Lok, H. Qi, and E. L. Ouh. 2020. BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging studies. In Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2020). 1556–1560.
[55]
Yulun Wu, Zeliang Yu, Ming Wen, Qiang Li, Deqing Zou, and Hai Jin. 2023. Understanding the Threats of Upstream Vulnerabilities to Downstream Projects in the Maven Ecosystem.
[56]
Hongjie Ye, Wei Chen, Wensheng Dou, Guoquan Wu, and Jun Wei. 2022. Knowledge-Based Environment Dependency Inference for Python Programs. May, 12.
[57]
Zhaoxu Zhang, Hengcheng Zhu, Ming Wen, Yida Tao, Yepang Liu, and Yingfei Xiong. 2020. How Do Python Framework APIs Evolve? An Exploratory Study. In Proceedings of the IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER 2020). 81–92.
[58]
Ze-Lin Zhao, Di Huang, and Xiao-Xing Ma. 2022. TOAST:Automated Testing of Object Transformers in Dynamic Software Updates. Journal of Computer Science and Technology (JCST), 37, 1 (2022), 1, 50–66.
[59]
Zhenchang Xing and E. Stroulia. 2007. API-Evolution Support with Diff-CatchUp. IEEE Transactions on Software Engineering, 33, 12 (2007), Dec., 818–836.

Cited By

View all
  • (2024)How to Pet a Two-Headed Snake? Solving Cross-Repository Compatibility Issues with HeraProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695064(694-705)Online publication date: 27-Oct-2024
  • (2024)My Fuzzers Won’t Build: An Empirical Study of Fuzzing Build FailuresACM Transactions on Software Engineering and Methodology10.1145/3688842Online publication date: 21-Aug-2024
  • (2024)An Empirical Study on Python Library Dependency and Conflict Issues2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS62785.2024.00057(504-515)Online publication date: 1-Jul-2024

Index Terms

  1. Automatically Resolving Dependency-Conflict Building Failures via Behavior-Consistent Loosening of Library Version Constraints

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ESEC/FSE 2023: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
    November 2023
    2215 pages
    ISBN:9798400703270
    DOI:10.1145/3611643
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 30 November 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Dependency conflict
    2. loosening resolution
    3. version constraint

    Qualifiers

    • Research-article

    Funding Sources

    • the Natural Science Foundation of China under Grant
    • the Natural Science Foundation of Jiangsu Province under Grants

    Conference

    ESEC/FSE '23
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 112 of 543 submissions, 21%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)133
    • Downloads (Last 6 weeks)6
    Reflects downloads up to 23 Nov 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)How to Pet a Two-Headed Snake? Solving Cross-Repository Compatibility Issues with HeraProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695064(694-705)Online publication date: 27-Oct-2024
    • (2024)My Fuzzers Won’t Build: An Empirical Study of Fuzzing Build FailuresACM Transactions on Software Engineering and Methodology10.1145/3688842Online publication date: 21-Aug-2024
    • (2024)An Empirical Study on Python Library Dependency and Conflict Issues2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS62785.2024.00057(504-515)Online publication date: 1-Jul-2024

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media