Powered by
15th ACM International Workshop on Automating Test Case Design, Selection and Evaluation (A-TEST 2024),
September 19, 2024,
Vienna, Austria
15th ACM International Workshop on Automating Test Case Design, Selection and Evaluation (A-TEST 2024)
Frontmatter
Welcome from the Chairs
Welcome to the 15th ACM International Workshop on Automating Test Case Design, Selection and Evaluation (A-TEST), co-located with ECOOP and ISSTA 2024, in Vienna, on 19th of September, 2024.
This year's theme of the workshop is "Using DSLs for testing and the testing of DSLs", right at intersection of the topics of the two main conferences.
A-TEST'24 received six submissions, three of which were accepted for presentation. All papers were reviewed by three program committee members. Next to the regular session with papers the workshop features a hands-on session, titled "Testing DSLs with DSLs in Rascal and TESTAR", showcasing state-of-the-art automated testing techniques applied to a DSL implementation in the context of the Rascal language workbench. In particular, it highlights how a (formal) model of a DSL can function as an oracle for regular acceptance tests, and as a driver from scriptless UI testing, using tools like TESTAR.
Papers
GreeDDy: Accelerate Parallel DDMIN
Dániel Vince and
Ákos Kiss
(University of Szeged, Hungary)
One of the most important algorithms in the field of automated test case minimization is the minimizing Delta Debugging (DDMIN) algorithm. It is used with preference because it works on any kind of input without information about its structure. In this paper, we focus on the parallelization of DDMIN. We discuss its stability issues and outline a potential solution to it. Then, we discuss an idea to speed up parallel DDMIN without compromising the minimality guarantees of the algorithm. We evaluate this algorithm variant, named GreeDDy, on a publicly available test suite.
@InProceedings{A-TEST24p1,
author = {Dániel Vince and Ákos Kiss},
title = {GreeDDy: Accelerate Parallel DDMIN},
booktitle = {Proc.\ A-TEST},
publisher = {ACM},
pages = {1--4},
doi = {10.1145/3678719.3685690},
year = {2024},
}
Publisher's Version
Use of ChatGPT as an Assistant in the End-to-End Test Script Generation for Android Apps
Boni García,
Maurizio Leotta,
Filippo Ricca, and
Jim Whitehead
(Universidad Carlos III de Madrid, Spain; University of Genoa, Italy; University of California at Santa Cruz, USA)
Automated testing is crucial in software development to ensure that applications perform as intended. However, generating automated End-to-End (E2E) tests can be time-consuming and challenging, especially for junior developers. This study investigates the use of ChatGPT, a popular Generative Artificial Intelligence (GenAI) model, as an assistant in developing automated E2E test scripts for Android apps. We present an empirical study that compares the effort required to create E2E test scripts and the resulting reliability of these tests using two treatments: manually and assisted by ChatGPT. We used Gherkin, a domain-specific language that allows non-technical practitioners to define test scenarios using a human-readable syntax. Our findings indicate that using ChatGPT significantly reduces the time required to develop automated test scripts without compromising the reliability of the scripts. Statistical analysis shows a notable reduction in development time for the ChatGPT-assisted group compared to the manual group, with a large effect size. While the reliability of the tests did not show a significant difference between the two groups, the results suggest practical benefits in terms of efficiency.
@InProceedings{A-TEST24p5,
author = {Boni García and Maurizio Leotta and Filippo Ricca and Jim Whitehead},
title = {Use of ChatGPT as an Assistant in the End-to-End Test Script Generation for Android Apps},
booktitle = {Proc.\ A-TEST},
publisher = {ACM},
pages = {5--11},
doi = {10.1145/3678719.3685691},
year = {2024},
}
Publisher's Version
First Experiments on Automated Execution of Gherkin Test Specifications with Collaborating LLM Agents
Severin Bergsmann,
Alexander Schmidt,
Stefan Fischer, and
Rudolf Ramler
(Software Competence Center Hagenberg, Austria)
Gherkin is a domain-specific language for describing test scenarios in natural language, which are the basis for automated acceptance testing. The emergence of Large Language Models (LLMs) has opened up new possibilities for processing such test specifications and for generating executable test code. This paper investigates the feasibility of employing LLMs to execute Gherkin test specifications utilizing the AutoGen multi-agent framework. Our findings show that our LLM agent system is able to automatically run the given test scenarios by autonomously exploring the system under test, generating executable test code on the fly, and evaluating execution results. We observed high success rates for executing simple as well as more complex test scenarios, but we also identified difficulties regarding failure scenarios and fault detection.
@InProceedings{A-TEST24p12,
author = {Severin Bergsmann and Alexander Schmidt and Stefan Fischer and Rudolf Ramler},
title = {First Experiments on Automated Execution of Gherkin Test Specifications with Collaborating LLM Agents},
booktitle = {Proc.\ A-TEST},
publisher = {ACM},
pages = {12--15},
doi = {10.1145/3678719.3685692},
year = {2024},
}
Publisher's Version
Info
proc time: 1.14