Perplexity unveils lily: high-performance inference engine for Qwen3.6-35B-A3B on apple silicon

Perplexity has released Lily, a specialized inference engine designed for running the Qwen3.6-35B-A3B model on Apple Silicon hardware. Built with Rust and Metal, Lily operates as a single-process runtime, bypassing PyTorch and MLX. It features a Rust layer for checkpoint loading and generation, an OpenAI-compatible API for token streaming, and hand-written Metal kernels for model execution. The engine is optimized for Apple’s hardware, requiring a Mac with at least 32 GB of unified memory for deployment. Lily’s design focuses on performance through narrow specialization, with a 4-bit checkpoint of 19.4 GB. It employs groupwise affine quantization, compressing bfloat16 weights to reduce memory usage. The engine also integrates advanced techniques like grouped-query attention and GQA-aware attention, achieving significant speed improvements in both prefill and decode phases. Benchmarks show Lily outperforms MLX-LM in token throughput, with a 1.23x faster prefill rate and 1.35x faster decode speed on a 40-core M5 Max. The release includes a standalone demo and GitHub repository, with further details available for technical insights.