Exploring human-generative AI collaboration in software testing

Abstract

As the use of Generative AI grows within the software engineering domain, understanding how software practitioners engage with these technologies becomes increasingly important. Application of Generative AI is reshaping how professionals approach tasks across software engineering, including software testing. While organizations explore the integration of Generative AI in testing, the role of human–generative AI collaboration remains insufficiently examined. This study aims to explore software testers’ experiences with Generative AI tools in their testing activities. The primary objective is to understand how software testers collaborate with Generative AI in practice, with a focus on the benefits and challenges they encounter, as well as the opportunities and risks they perceive when integrating these tools into software testing workflows.

The study was conducted in two phases. The first phase was a scoping-style literature review. This process established a conceptual foundation by identifying key themes and sub-themes that informed the survey design in phase two. Statements extracted from the literature were initially grouped into 29 sub-themes and later consolidated into four main themes: benefits, challenges, opportunities, and risks, through thematic analysis and initial coding. In the second phase, a survey was designed based on the themes developed in phase one. It included both closed- and open-ended questions and was distributed through professional and academic networks. A total of 64 valid responses were collected and analyzed using descriptive analysis.

The results show that software testers are increasingly adopting Generative AI tools for tasks such as test case generation, automation, and debugging, typically through a human-in-the-loop model where outputs are reviewed and refined. Reported benefits include time savings, improved efficiency, reduced manual effort, better coverage, and learning support. However, challenges such as unreliable outputs, limited coverage, prompt engineering demands, and concerns about trust and maintainability remain. At the same time, opportunities are emerging through AI-assisted GUI and end-to-end testing, hybrid workflows, and creative uses, although concerns remain regarding accuracy, privacy, transparency, and the long-term quality of AI-generated outputs. Overall, testers view Generative AI as a supportive tool that complements their role rather than replacing it, with human oversight continuing to play a critical part.

Future research should include interviews or case studies to explore validation practices, team collaboration, and workflow adaptation. Broader participant diversity would improve generalizability, while further investigation is needed into supporting less-represented testing types and enabling ethical, trustworthy use.