Openai's Jalapeño chip demonstrates superior AI inference performance and efficiency

At the Hot Chips conference, OpenAI unveiled detailed insights into its Jalapeño chip, highlighting its performance on the SemiAnalysis InferenceX benchmark. Jalapeño achieved higher tokens per user and greater throughput per kilowatt compared to current state-of-the-art inference processors. Richard Ho, OpenAI’s head of hardware, emphasized that the results indicate a 'very, very significant performance advance over state of the art,' noting that Jalapeño can handle more AI work per unit of power while delivering faster responses. The chip is designed to be efficient for serving many customers and maintaining low latency. The comparison was made against an Nvidia Blackwell system, though Ho acknowledged that by the time Jalapeño is fully deployed in late 2026, the competition may have advanced significantly. He estimated that initial deployment would occur in very small volumes, with more substantial deployment planned for 2027. Jalapeño, first announced in October, was developed in collaboration with Broadcom, with OpenAI’s own models aiding in the development process. The company aims to make Jalapeño a multigenerational platform, integrating AI products, models, chips, and memory into a cohesive system. This full-stack approach allowed OpenAI to address specific challenges in the inference process, particularly minimizing delays during the prefill and communication phases, which are often bottlenecks. OpenAI explained in a blog post that Jalapeño was designed to reduce data movement and communication delays. By explicitly placing and keeping model state, including the KV cache used during response generation, local to the system, Jalapeño can activate the right combination of compute, memory, and networking for each inference phase. This design ensures efficient and rapid processing, enhancing both speed and power efficiency in AI inference tasks.