Reducing annotation burden in computational pathology : a modular pipeline for scalable and efficient nuclei segmentation

Abstract

While deep learning continues to evolve rapidly and demand ever-larger, high-quality datasets, the creation of detailed annotations for nuclei segmentation remains a time-consuming bottleneck. Automating parts of this process, especially with human-in-the-loop strategies, can accelerate dataset creation, expanding the learning space for models and enabling rapid iteration. This also facilitates knowledge transfer across domains and the development of improved annotation tools, ultimately driving more scalable and efficient workflows.

This thesis presents a comprehensive pipeline where bounding box-level supervision is used to train detection models such as YOLOv11, RetinaNet, and Faster R-CNN, which are then used to prompt Segment Anything Model (SAM) for instance-level segmentation. The modularity of the pipeline enables rapid experimentation and model substitution. The SAM model is leveraged in a zero-shot setting without requiring any fine-tuning on medical data, relying solely on bounding boxes for guidance.

Experiments were conducted on the dataset using a 3-fold cross-validation setup. The results demonstrate that the proposed approach achieves strong performance in both detection (YOLOv11 achieving F1 score of 0.81, AP@50 of 0.84, and mAP@50–95 of 0.54) and segmentation (Dice score of 0.90, IoU of 0.82, and Hausdorff distance of 3.76 with oracle boxes). Robustness to bounding box perturbations further underscores the resilience of SAM. The qualitative and quantitative analyses confirm that accurate localization, rather than exact bounding box alignment, is sufficient for high-quality segmentation.

The experimental results underscore the pipeline’s ability to reduce annotation burden, improve generalizability, and facilitate modular experimentation. Error analysis identifies challenges with overlapping nuclei and detection accuracy affecting segmentation. Future work includes domain adaptation, enhanced prompting, and integration into clinical workflows, paving the way for scalable, human-in-the-loop nuclei segmentation systems that accelerate dataset growth and model development.