Streaming robotics learning pipeline built using NVIDIA cosmos3-droid dataset

Researchers have developed a streaming robotics learning pipeline using the NVIDIA Cosmos3-DROID dataset, enabling analysis and training without downloading the 707 GB repository locally. The approach leverages metadata-driven episode discovery, column- and row-group-level Par,quet projection, and seek-based video decoding to access only the required data for analysis and training. The pipeline converts individual episodes into state-action trajectories, analyzes joint motion, gripper events, and action-frequency spectra, and decodes only the necessary AV1 video windows. It also normalizes observations and actions using dataset statistics, constructs an ACT-style PyTorch dataset, and trains a multimodal behavior-cloning policy. The trained policy is evaluated through open-loop rollout with temporally ensembled action chunks, and results are visualized against ground-truth actions. The workflow provides a compact, extensible foundation for scaling across additional shards, failure demonstrations, and alternative action representations. The method supports both PyAV and FFmpeg decoding paths for efficient AV1 video processing and allows for optional caching of visual observations to manage computational resources. Training uses AdamW optimizer, OneCycle learning-rate scheduling, and mixed-precision execution. The final model includes normalization statistics and configuration metadata for reuse in future experiments. The project highlights the potential of metadata-driven data access in handling large-scale robotics datasets efficiently.