Automated signal comparison using machine learning and statistical analysis
Ghaffar, Kinza (2025-06-18)
K. Ghaffar
18.06.2025
© 2025, Kinza Ghaffar. Tämä Kohde on tekijänoikeuden ja/tai lähioikeuksien suojaama. Voit käyttää Kohdetta käyttöösi sovellettavan tekijänoikeutta ja lähioikeuksia koskevan lainsäädännön sallimilla tavoilla. Muunlaista käyttöä varten tarvitset oikeudenhaltijoiden luvan.
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:oulu-202506194779
https://urn.fi/URN:NBN:fi:oulu-202506194779
Tiivistelmä
Background. In industrial embedded systems (ES), ensuring the integrity and reliability of digital signals such as I²C and SPI clock signals is critical for hardware (HW) validation and system performance. Traditional methods for comparing measured and simulated signals rely heavily on manual inspection, which is time-consuming and error prone. This creates a need for automated signal comparison systems using Machine Learning (ML) and Deep Learning (DL) models that can maintain accuracy while improving scalability.
Objective. The study aims to develop and evaluate an automated framework for comparing I²C and SPI signals by leveraging ML and DL models. The primary objective is to evaluate and compare the performance of these models in classifying measured and simulated clock signals using both binary and multi-class classification frameworks. The work also explores whether traditional ML models can outperform or match DL models in practical validation tasks.
Methods. We conducted experiments using 15 ML and 3 DL models to classify signal similarity. Both binary and multi-class classification tasks were performed using a 10×10-fold cross-validation strategy. Model performance was evaluated using five standard metrics: Accuracy, Precision, Recall, F1-score, and Matthews Correlation Coefficient (MCC). To validate the statistical significance, we applied the Anderson-Darling test for normality, followed by the Wilcoxon signed-rank test and Dunn's post-hoc test for pairwise model comparisons.
Results. The results showed that while DL models such as ResNet-18 achieved competitive performance, they did not significantly outperform top-performing ML models. In both classification tasks, CatBoost and Extra Trees consistently delivered high accuracy and stability. Statistical tests confirmed that all models significantly outperformed the DummyClassifier baseline, but differences between top ML and DL models were not statistically significant across all metrics.
Conclusions. The findings demonstrate that traditional ML models offer a computationally efficient and accurate alternative to DL models. The study supports the integration of ML-based frameworks into automated validation pipelines, reducing manual effort while ensuring consistency and scalability in HW testing workflows.
Objective. The study aims to develop and evaluate an automated framework for comparing I²C and SPI signals by leveraging ML and DL models. The primary objective is to evaluate and compare the performance of these models in classifying measured and simulated clock signals using both binary and multi-class classification frameworks. The work also explores whether traditional ML models can outperform or match DL models in practical validation tasks.
Methods. We conducted experiments using 15 ML and 3 DL models to classify signal similarity. Both binary and multi-class classification tasks were performed using a 10×10-fold cross-validation strategy. Model performance was evaluated using five standard metrics: Accuracy, Precision, Recall, F1-score, and Matthews Correlation Coefficient (MCC). To validate the statistical significance, we applied the Anderson-Darling test for normality, followed by the Wilcoxon signed-rank test and Dunn's post-hoc test for pairwise model comparisons.
Results. The results showed that while DL models such as ResNet-18 achieved competitive performance, they did not significantly outperform top-performing ML models. In both classification tasks, CatBoost and Extra Trees consistently delivered high accuracy and stability. Statistical tests confirmed that all models significantly outperformed the DummyClassifier baseline, but differences between top ML and DL models were not statistically significant across all metrics.
Conclusions. The findings demonstrate that traditional ML models offer a computationally efficient and accurate alternative to DL models. The study supports the integration of ML-based frameworks into automated validation pipelines, reducing manual effort while ensuring consistency and scalability in HW testing workflows.
Kokoelmat
- Avoin saatavuus [42420]