Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3593434.3593449acmotherconferencesArticle/Chapter ViewAbstractPublication PageseaseConference Proceedingsconference-collections
short-paper
Open access

Improving the Reporting of Threats to Construct Validity

Published: 14 June 2023 Publication History

Abstract

Background: Construct validity concerns the use of indicators to measure a concept that is not directly measurable. Aim: This study intends to identify, categorize, assess and quantify discussions of threats to construct validity in empirical software engineering literature and use the findings to suggest ways to improve the reporting of construct validity issues. Method: We analyzed 83 articles that report human-centric experiments published in five top-tier software engineering journals from 2015 to 2019. The articles’ text concerning threats to construct validity was divided into segments (the unit of analysis) based on predefined categories. The segments were then evaluated regarding whether they clearly discussed a threat and a construct. Results: Three-fifths of the segments were associated with topics not related to construct validity. Two-thirds of the articles discussed construct validity without using the definition of construct validity given in the article. The threats were clearly described in more than four-fifths of the segments, but the construct in question was clearly described in only two-thirds of the segments. The construct was unclear when the discussion was not related to construct validity but to other types of validity. Conclusions: The results show potential for improving the understanding of construct validity in software engineering. Recommendations addressing the identified weaknesses are given to improve the awareness and reporting of CV.

References

[1]
Thomas D Cook and Donald T Campbell. 1979. The design and conduct of true experiments and quasi-experiments in field settings. In Reproduced in part in Research in Organizations: Issues and Controversies. Goodyear Publishing Company.
[2]
Lee J Cronbach and Paul E Meehl. 1955. Construct validity in psychological tests. Psychological bulletin 52, 4 (1955), 281–302.
[3]
Andreas Jedlitschka, Marcus Ciolkowski, and Dietmar Pfahl. 2008. Reporting experiments in software engineering. Guide to advanced empirical software engineering (2008), 201–228.
[4]
Barbara Kitchenham, Hiyam Al-Khilidar, Muhammed Ali Babar, Mike Berry, Karl Cox, Jacky Keung, Felicia Kurniawati, Mark Staples, He Zhang, and Liming Zhu. 2008. Evaluating guidelines for reporting empirical software engineering studies. Empirical Software Engineering 13 (2008), 97–121.
[5]
Barbara Kitchenham, Dag I. K. Sjøberg, Tore Dybå, O Pearl Brereton, David Budgen, Martin Höst, and Per Runeson. 2012. Trends in the Quality of Human-Centric Software Engineering Experiments—A Quasi-Experiment. IEEE Transactions on Software Engineering 39, 7 (2012), 1002–1017.
[6]
Samuel Messick. 1989. Validity. Educational measurement (3rd edn.) 1989 (1989), 13–103.
[7]
Paul Ralph and Ewan Tempero. 2018. Construct validity in software engineering research and software metrics. In Proc. 22nd Int’l Conference on Evaluation and Assessment in Software Engineering. 13–23.
[8]
Milton J Rosenberg. 1969. The Conditions and Consequences of Evaluation Apprehension. In Artifact in behavioral research, Robert Rosenthal and Ralph L Rosnow (Eds.). New York : Academic Press, 279–349.
[9]
Per Runeson and Martin Höst. 2009. Guidelines for conducting and reporting case study research in software engineering. Empirical software engineering 14, 2 (2009), 131–164.
[10]
William R Shadish, Thomas D Cook, and Donald Thomas Campbell. 2002. Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.
[11]
Mary Shaw. 2003. Writing good software engineering research papers. In 25th International Conference on Software Engineering, 2003. Proceedings. IEEE, 726–736.
[12]
Dag I.K. Sjøberg and Gunnar R. Bergersen. 2022. Construct Validity in Software Engineering. IEEE Transactions on Software Engineering (2022), early access. https://doi.org/10.1109/TSE.2022.3176725
[13]
Dag I. K. Sjøberg, Tore Dybå, and Magne Jørgensen. 2007. The future of empirical methods in software engineering research. In Future of Software Engineering (FOSE’07). IEEE, 358–378.
[14]
Robert K Yin. 2003. Case study research: Design and methods. (3 ed.). Sage Publications, Inc.

Cited By

View all
  • (2024)Increasing, not Diminishing: Investigating the Returns of Highly Maintainable CodeProceedings of the 7th ACM/IEEE International Conference on Technical Debt10.1145/3644384.3644471(21-30)Online publication date: 14-Apr-2024
  • (2023)Security Aspects of Social Robots in Public Spaces: A Systematic Mapping StudySensors10.3390/s2319805623:19(8056)Online publication date: 24-Sep-2023

Index Terms

  1. Improving the Reporting of Threats to Construct Validity

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    EASE '23: Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering
    June 2023
    544 pages
    ISBN:9798400700446
    DOI:10.1145/3593434
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 14 June 2023

    Check for updates

    Author Tags

    1. empirical research
    2. measurement
    3. research quality

    Qualifiers

    • Short-paper
    • Research
    • Refereed limited

    Conference

    EASE '23

    Acceptance Rates

    Overall Acceptance Rate 71 of 232 submissions, 31%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)303
    • Downloads (Last 6 weeks)28
    Reflects downloads up to 22 Nov 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Increasing, not Diminishing: Investigating the Returns of Highly Maintainable CodeProceedings of the 7th ACM/IEEE International Conference on Technical Debt10.1145/3644384.3644471(21-30)Online publication date: 14-Apr-2024
    • (2023)Security Aspects of Social Robots in Public Spaces: A Systematic Mapping StudySensors10.3390/s2319805623:19(8056)Online publication date: 24-Sep-2023

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media