Tawosi, V; Moussa, R; Sarro, F; (2024) Agile Effort Estimation: Have We Solved the Problem Yet? Insights from the Replication of the GPT2SP Study. In: 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). (pp. pp. 1034-1041). IEEE: Rovaniemi, Finland.

Preview

PDF GPT2SP_Replication_Report.pdf - Accepted Version Download (205kB) | Preview

Abstract

Replication studies in Software Engineering are indispensable for ensuring the reliability, generalizability, and transparency of research findings. They contribute to the cumulative growth of knowledge in the field and promote a scientific approach that benefits both researchers and practitioners. In this article, we report our experience replicating a recently published work proposing a Transformer-based approach for Agile Story Point Estimation' dubbed GPT2SP. GPT2SP was proposed with the intent of addressing the three limitations of a previous Deep Learning-based approach dubbed Deep-SE, and the results reported in the original study set GPT2SP as the new state-of-the-art. However, when we used the GPT2SP source code made publicly available by the authors of the original study, we found a bug in the computation of the evaluation measure and the reuse of erroneous results from previous work, which had unintentionally introduced biases in the GPT2SP's performance evaluation. In this study, we report on the results we obtained after fixing the issues present in the original study, which reveal that their results were in fact unintentionally inflated due to these issues and that despite advancements, challenges remain in providing accurate effort estimations for agile software projects.

Download activity - last month

Export as JSONCSVXML

Download activity - last 12 months

Export as XMLJSONCSV

Downloads by country - last 12 months

Export as JSONCSVXML

Archive Staff Only

View Item