Training with Limited Labeled Data: Semi-Supervised Learning Methods

An educational article introduces the first installment of a series on machine learning with scarce labeled data, focusing on semi-supervised learning. The approach combines labeled and unlabeled data for model training, grounded in four theoretical assumptions: nearby samples in feature space should share similar labels, data naturally forms dense clusters with consistent labels, decision boundaries exist in sparse regions, and high-dimensional data resides on lower-dimensional manifolds. The piece reviews consistency regularization techniques such as Π-Model and Temporal Ensembling, which enforce stable predictions across multiple network passes. Mean Teacher enhances efficiency by maintaining an exponential moving average of model weights, demonstrating superior accuracy compared to Π-Model. Subsequent methods including Virtual Adversarial Training, Interpolation Consistency Training, and Unsupervised Data Augmentation extend these concepts through adversarial perturbations and advanced augmentation strategies, investigating how noise quality affects performance. Pseudo-labeling techniques leverage model predictions to label unlabeled data, functioning equivalently to entropy regularization. Label Propagation constructs similarity graphs to distribute labels across unlabeled samples. Recent self-training approaches, exemplified by Noisy Student, demonstrated extreme scalability by training EfficientNet on 300 million unlabeled images, combining noise in student training with clean teacher predictions.