Expanding training data: augmentation and synthesis strategies when data is scarce

When training datasets are limited, two complementary strategies address the problem: data augmentation, which modifies existing examples while preserving their meaning, and data synthesis, which generates new examples artificially. Augmentation operates through controlled transformations across images, text, audio, and model-internal representations, each modifying non-essential attributes while maintaining semantic integrity. Text augmentation employs simple yet effective operations: word replacement, insertion, swapping, and deletion, with the most pronounced gains on smaller datasets. More sophisticated approaches leverage pre-trained language models that replace words with semantically suitable synonyms, or create new texts by translating between languages. Audio augmentation masks temporal or frequency segments, applies pitch shifts, and blends multiple samples. Language model-based synthesis offers a scalable alternative. Research shows GPT-3 can annotate data at roughly one-tenth the cost of human labeling, functioning as a weak annotator within self-training frameworks. Fine-tuned models can similarly generate label-conditioned synthetic samples, a process that iterates over time. Generated data typically contains noise and inaccuracies, necessitating noise-robust training techniques. Standard regularization such as dropout and batch normalization help, as do specialized methods that filter compromised examples during training or adapt model predictions to match observed noise patterns. Final performance depends on two dimensions: affinity (relevance to the task) and diversity (coverage across the data space).