Meta releases Muse Glimmer, efficient 30B model with hybrid architecture

Meta released Muse Glimmer, a 30-billion-parameter multimodal language model using Gemma-like architecture. The model features a 131k token context window, dense design without mixture-of-experts, and hybrid attention combining grouped-query attention with sliding window attention in a 3:1 local-to-global ratio. It stands out for extreme KV-cache efficiency with 32 query heads to just 2 KV heads. The model uses SwiGLU instead of GeGLU for activations and applies gated attention in both GQA and sliding window attention components, offering efficiency advantages over similar architectures like Gemma 4.