The Role of Software Measurement in Assured LLM-Based Software Engineering

Assured Large Language Model Software Engineering (Assured LLMSE) addresses the twin challenges: 1. Ensuring LLM-generated code does not regress the properties of the original code 2. Quantifying the improvement over the original archived by the improve code in a verifiable and measurable way.

In so doing, the Assured LLMSE approach tackles the problem of LLMs’ tendency to hallucinate, as well as providing confidence that generated code improves an existing code base. Software testing and measurement play critical roles in this improvement process: testing is the guard against regression, while measurement provides the quantifiable assurance of improvement. Assured LLMSE takes its inspiration from previous work on genetic improvement, for which software measurement also plays a central role. In this keynote we outline the Assured LLMSE approach, highlighting the role of software measurement in the provision of quantifiable, verifiable assurances for code that originates from LLM–based inference. This paper is an outline of the content of the keynote by Mark Harman at the 28th International Conference on Evaluation and Assessment in Software Engineering.

This is joint work with Nadia Alshahwan, Andrea Aquino, Jubin Chheda, Anastasia Finegenova, Inna Harper, Mitya Lyubarskiy, Neil Maiden, Alexander Mols, Shubho Sengupta, Rotem Tal, Alexandru Marginean, and Eddy Wang.