Does Treatment Adherence Impact Experiment Results in TDD?
Karac, Itir; Panach, Jose Ignacio; Turhan, Burak; Juristo, Natalia (2024-11-15)
IEEE
15.11.2024
I. Karac, J. I. Panach, B. Turhan and N. Juristo, "Does Treatment Adherence Impact Experiment Results in TDD?," in IEEE Transactions on Software Engineering, vol. 51, no. 1, pp. 135-152, Jan. 2025, doi: 10.1109/TSE.2024.3497332
https://creativecommons.org/licenses/by/4.0/
© 2024 IEEE. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
https://creativecommons.org/licenses/by/4.0/
© 2024 IEEE. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
https://creativecommons.org/licenses/by/4.0/
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:oulu-202411226860
https://urn.fi/URN:NBN:fi:oulu-202411226860
Tiivistelmä
Abstract
Context:
In software engineering (SE) experiments, the way in which a treatment is applied could affect results. Different interpretations of how to apply the treatment and decisions on treatment adherence could lead to different results when data are analysed.
Objective:
This paper aims to study whether treatment adherence has an impact on the results of an SE experiment.
Method: The experiment used as test case for our research uses Test-Driven Development (TDD) and Incremental Test-Last Development, (ITLD) as treatments. We reported elsewhere the design and results of such an experiment where 24 participants were recruited from industry. Here, we compare experiment results depending on the use of data from adherent participants or data from all the participants irrespective of their adherence to treatments.
Results:
Only 40% of the participants adhere to both TDD protocol and to the ITLD protocol; 27% never followed TDD; 20% used TDD even in the control group; 13% are defiers (used TDD in ITLD session but not in TDD session). Considering that both TDD and ITLD are less complex than other SE methods, we can hypothesize that more complex SE techniques could get even lower adherence to the treatment.
Conclusion:
Both TDD and ITLD are applied differently across participants. Training participants could not be enough to ensure a medium to large adherence of experiment participants. Adherence to treatments impacts results and should not be taken for granted in SE experiments.
Context:
In software engineering (SE) experiments, the way in which a treatment is applied could affect results. Different interpretations of how to apply the treatment and decisions on treatment adherence could lead to different results when data are analysed.
Objective:
This paper aims to study whether treatment adherence has an impact on the results of an SE experiment.
Method: The experiment used as test case for our research uses Test-Driven Development (TDD) and Incremental Test-Last Development, (ITLD) as treatments. We reported elsewhere the design and results of such an experiment where 24 participants were recruited from industry. Here, we compare experiment results depending on the use of data from adherent participants or data from all the participants irrespective of their adherence to treatments.
Results:
Only 40% of the participants adhere to both TDD protocol and to the ITLD protocol; 27% never followed TDD; 20% used TDD even in the control group; 13% are defiers (used TDD in ITLD session but not in TDD session). Considering that both TDD and ITLD are less complex than other SE methods, we can hypothesize that more complex SE techniques could get even lower adherence to the treatment.
Conclusion:
Both TDD and ITLD are applied differently across participants. Training participants could not be enough to ensure a medium to large adherence of experiment participants. Adherence to treatments impacts results and should not be taken for granted in SE experiments.
Kokoelmat
- Avoin saatavuus [38840]
Ellei muuten mainita, aineiston lisenssi on https://creativecommons.org/licenses/by/4.0/