A comparative performance analysis of federated knowledge distillation for image classification under data heterogeneity

Abstract

Federated Learning (FL) enables decentralized training of machine learning models on private and distributed data which is critical for applications like medical imaging and intelligent edge systems. Standard FL approaches often struggle with data heterogeneity particularly non independent and identically distributed (non-IID) data across clients. It can be impacted by system heterogeneity arising from differences in client model architectures or computational resources. These factors can affect the performance of common aggregation methods like FedAvg and FedProx. Federated Knowledge Distillation (FKD) provides an alternative that can support heterogeneous models and dataset. It can potentially reduce communication overhead by exchanging knowledge through model outputs rather than direct parameter sharing. This thesis presents a systematic empirical evaluation of different learning strategies for image classification progressing from centralized to federated approach. The study begins with centralized supervised training, followed by an analysis of centralized knowledge distillation (KD). It then extends to federated settings evaluating standard parameter aggregation-based FL and subsequently FKD. Experiments are conducted on benchmark datasets including CIFAR-10, CIFAR-100 and the HAM10000 skin lesion dataset. It utilizes combinations of teacher and student models from the ResNet, EfficientNet and MobileNet families to simulate varying computational and architectural capabilities. The evaluations particularly consider the impact of non-IID data distributions on model performance. Key findings indicate the substantial benefit of transfer learning in centralized supervised settings and confirm that larger pretrained models generally achieve higher accuracy. In centralized knowledge distillation experiments under non-IID conditions, smaller models like MobileNetV3 Small when trained from scratch demonstrated significant performance improvements when guided by a teacher highlighting the value of KD in providing structured supervision for low capacity models in data scarce or imbalanced scenarios. The comparative analysis extends these insights to federated setup examining the performance trade offs of FL and FKD in handling data heterogeneity. The results contribute to a practical understanding of the benefits and drawbacks of these learning strategies in contexts relevant to edge intelligence systems. While this work primarily focuses on empirical evaluation, it also briefly discusses potential extensions, such as gated entropy approaches for knowledge aggregation in FKD, as avenues for future research to further enhance performance under pronounced heterogeneity.