Cactus Whistle model challenges whisper base in speech recognition performance

A recent analysis compares the Cactus Whistle, a 16.9 MB speech-to-text model from Cactus Compute, with Whisper Base and Tiny on an Apple M4 CPU. The study finds Whistle achieves a 10.47% word error rate (WER) on the test-other dataset, nearly matching its published full-split score of 10.49%. It outperforms greedy Whisper Base by 2.19 points on a paired subset, though the gap is not significant against beam search. Whistle runs 5.7 times faster than greedy Whisper Base on a 10-second clip, though its speed advantage narrows with longer audio. The model also handles streaming with one-second chunks, committing words a median 1.68 seconds after they end. However, the 30-second limit causes the pinned engine to drop the middle of longer audio, highlighting the need for splitting or streaming. The study emphasizes the importance of pinning repository revisions and using a single normalizer for fair comparisons.