Employing device inventory and failure data for test configuration discovery and device utilization

Abstract

Within network integration testing, device capacity is a key constraint for comprehensive testing. This thesis investigates how fault tickets and device inventory data could be utilized to automatically discover test configurations, increase device utilization, and aid decision making using a two-prong approach. Firstly, the research focuses on examining and developing methods for interpreting and analyzing natural language data found in fault tickets, with the goal of converting unstructured natural language into structured data. The system can then automatically identify the available test configurations and their compatibility based on the underlying devices. Secondly, the device inventory interface and the supporting infrastructure offer data from which utilization, failure rate, and conflicts can be calculated. This data will also be used to create a reporting system that can show reservation conflicts, devices, and their utilization. Resulting in more efficient configuration discovery and device utilization. The fine-tuned T5-base large language model (LLM) showed over a 10% increase in accuracy, compared to regular expressions (Regex), with a 94% accuracy on the test dataset. The developed application could effectively deduce configuration compatibility by utilizing the extraction process. Furthermore, a recommendation for organizing permissions and ownership within the device inventory was created.