Google introduces Gemma 4 with expanded context and improved architecture

Google released Gemma 4, a family of open-weight language models in multiple sizes, featuring a dense 31-billion-parameter variant and a 26B-A4B mixture-of-experts multimodal model. The model maintains familiar Gemma patterns with sliding window attention interspersed with full-attention layers in a 5:1 ratio, expanding the context window to 256k tokens, doubling the previous 128k limit of Gemma 3 27B. It includes grouped-query attention, QK-Norm, and unusual pre and post-RMSNorm arrangement, along with a dedicated draft model for multi-token prediction and speculative decoding. The model incorporates a vision encoder of approximately 550 million parameters, demonstrating significant improvement over Gemma 3 27B on benchmarks like MMLU-Pro.