Sobania, Dominik;
Geiger, Alina;
Callan, James;
Brownlee, Alexander;
Hanna, Carol;
Moussa, Rebecca;
Zamorano López, Mar;
... Sarro, Federica; + view all
(2023)
Evaluating Explanations for Software Patches Generated by Large Language Models.
In:
Search-Based Software Engineering. SSBSE 2023.
Springer
Preview |
Text
main.pdf - Accepted Version Download (299kB) | Preview |
Abstract
Large language models (LLMs) have recently been integrated in a variety of applications including software engineering tasks. In this work, we study the use of LLMs to enhance the explainability of software patches. In particular, we evaluate the performance of GPT 3.5 in explaining patches generated by the search-based automated program repair system ARJA-e for 30 bugs from the popular Defects4J benchmark. We also investigate the performance achieved when explaining the corresponding patches written by software developers. We find that on average 84% of the LLM explanations for machine-generated patches were correct and 54% were complete for the studied categories in at least 1 out of 3 runs. Furthermore, we find that the LLM generates more accurate explanations for machine-generated patches than for human-written ones.
| Type: | Proceedings paper |
|---|---|
| Title: | Evaluating Explanations for Software Patches Generated by Large Language Models |
| Event: | SSBSE 2023: Search-Based Software Engineering. 15th International Symposium |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1007/978-3-031-48796-5_12 |
| Publisher version: | https://doi.org/10.1007/978-3-031-48796-5_12 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. - For the purpose of open access, the author has applied a Creative Commons Attribution (CC BY) licence to any Author Accepted Manuscript version arising. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery-pp.ucl.ac.uk/id/eprint/10180564 |
Archive Staff Only
 |
View Item |
