Perplexity reveals its gpu-based embedding infrastructure

Perplexity has detailed its GPU-based embedding infrastructure, explaining how its Ivy, Tulip, and ROSE systems support the pplx-embed model. The company emphasizes that embedding inference on mature Hopper and Blackwell hardware has largely converged, with performance gains stemming from runtime optimizations and harness management. Perplexity frames embedding serving as two distinct workloads: batch embedding for vector database construction and online embedding for query-time processing. The company opted not to build a separate embedding engine, instead reusing prefill and decode kernels from its LLM stack. Tulip manages sequences with a first-come, first-served approach, while ROSE supports multiple attention backends, including FlashInfer 3 and FlashAttention 4. Perplex, which benchmarks against vLLM v0.22.0, highlights the efficiency of lazy capture and LazyTensor for managing GPU workloads. The system also handles high-throughput and high-concurrency embeddings, with performance metrics covering various batch sizes and token counts.