OpenAI on August 25 published its first disclosed benchmark results for Jalapeño, its custom chip for serving AI models. The company said the Broadcom-partnered inference ASIC delivered 1.5 to 1.9 times more AI work per watt at peak throughput and 1.7 to 3.6 times lower end-to-end latency than Nvidia GB200- and GB300-based systems in the selected comparisons. Those are OpenAI’s own measurements, not a full independently reproduced evaluation of the hardware. (openai.com)
What OpenAI tested
The results matter because model serving, or inference, is the recurring compute task behind every chatbot reply, generated token and multi-step agent action. OpenAI’s central claim is that Jalapeño can improve the usual balance operators must strike: batch requests aggressively to maximize throughput and power efficiency, or prioritize quick individual responses. Richard Ho, OpenAI’s hardware vice president, has framed the chip’s goal as reducing that latency-throughput trade-off for modern language-model workloads.
OpenAI tested Jalapeño with SemiAnalysis’ public InferenceX benchmark on three open models: GPT-OSS 120B, DeepSeek R1 670B and Kimi K2.5 1T. The company said it measured systems across operating points ranging from high-throughput serving to more interactive, low-latency use. It reported a 2.1 to 4.1 times performance advantage for highly interactive workloads, where an individual user’s token generation rate and the delay between generated tokens are especially important. (openai.com)
The disclosed model-by-model comparisons were not identical. For GPT-OSS 120B, OpenAI compared Jalapeño with a GB200 configuration; the appendix lists package power ratings of 700 watts for Jalapeño and 1,200 watts for the GB200. The DeepSeek R1 and Kimi K2.5 tests used GB300-based comparisons, listed at 1,400 watts. OpenAI said it normalized results using published accelerator power ratings, while Jalapeño’s measured sustained power stayed at or below 550 watts on the workloads tested. (openai.com)
What the figures do—and do not—show
That methodology makes the release more informative than a bare per-chip claim, but it also defines the limits of what can be concluded. The results apply to the named models, benchmark settings and comparison configurations—not to every inference workload, every software stack or future accelerator generation. The material supplied by OpenAI does not establish that an outside party executed the full suite, audited all system settings or reproduced the reported conclusions. Builders evaluating the claims will still want fuller visibility into topology, serving software, model settings and power treatment before generalizing the results to their own fleets.
OpenAI attributes Jalapeño’s reported gains to co-design across the chip, memory, networking, software and rack-scale system. In its description, processing an input prompt, known as prefill, is primarily compute-intensive, while generating an answer one token at a time, known as decode, is more constrained by memory bandwidth. Moving data and model state among resources can add further delay. OpenAI said Jalapeño is designed to keep state such as the KV cache close to where it is needed and to reduce those communication costs across the full request. (openai.com)
Why latency is central
That design focus helps explain why latency figures are central to the announcement. A small delay can become consequential in agentic workflows, where a system may call a model repeatedly to plan, use tools and revise an answer. High aggregate throughput remains essential for serving a large user base efficiently, but a system optimized solely around large batches can feel slow to an individual user. OpenAI is positioning Jalapeño as an attempt to maintain better efficiency at faster response settings, rather than treating interactive service as an unavoidable efficiency penalty.
The company said the chip reached tapeout—the point at which a design is sent for manufacturing—in nine months, with OpenAI models used in portions of the design and optimization process. It plans to begin deploying Jalapeño in its own compute infrastructure by the end of 2026, while continuing to use Nvidia and other partners’ accelerators for both training and inference. OpenAI also said a second Jalapeño generation is deep in development and a third is taking shape. (openai.com)
A broader push for specialized inference hardware
Jalapeño arrives amid a wider effort by large AI operators to tailor silicon to the economics of model serving. Meta, for example, announced in March a multigeneration MTIA roadmap that includes chips aimed at inference deployment in 2027. That does not validate OpenAI’s benchmark results, but it underscores the strategic appeal of specialized hardware: operators that control both workloads and infrastructure can try to optimize memory, networking, software and power consumption around their own demand patterns. (tomshardware.com)
For OpenAI, the immediate takeaway is not that Jalapeño has conclusively surpassed Nvidia hardware in general. Rather, the company has now published concrete, configuration-specific evidence for its first in-house inference system and a more detailed explanation of the performance targets it is pursuing. Its limited internal deployment later in 2026—and results after production qualification and operation at scale—will be the more consequential tests of whether those reported benchmark advantages translate into lower-cost, more responsive AI services.
