‘Tokens per Watt’ Becomes Key Metric in AI Chip Design, Analyst Says

The race for artificial intelligence dominance is shifting from a sprint for raw computing power to a marathon of efficiency, with chipmakers now focused on maximizing performance within fixed power constraints. According to a Barclays analyst, tokens per watt has become "one of the most critical yardsticks of success" as data center space is limited and demand for AI processing outstrips supply.

A New Benchmark for AI

Insights from the recent Hot Chips conference show a consensus among hardware providers, hyperscalers like Microsoft, and component suppliers on the need for greater power efficiency, analyst Tom O’Malley of Barclays noted. The industry is increasingly treating the entire server rack as "the unit of compute," moving beyond the performance of individual components to a more holistic, system-level view.

This change is a direct response to the physical and financial limitations of scaling data centers. As AI models grow more complex, simply adding more powerful chips is becoming unsustainable due to energy consumption and heat generation. Companies are now engineering solutions to extract the maximum number of AI tokens—the basic units of data processed by models—for every watt of power consumed.

Competing Strategies Emerge

Leading technology firms are pursuing different strategies to achieve this goal. According to the Barclays note, key approaches discussed at the conference include:

Ad

• Nvidia is exploring a "speculative decode" approach to improve efficiency.
• Cerebras, in partnership with AMD and AWS, is focused on "disaggregated inference," a technique O'Malley said "can have an order of magnitude improvement" on efficiency metrics.
• OpenAI presented its Jalapeño chip, which integrates prefill, draft, and decode functions on a single piece of silicon rather than splitting them across different hardware.
• Microsoft introduced a new architecture called Software Defined Local Access Dataflow, designed to lower the total cost of ownership through superior energy management.

On the custom chip front, Google is dual-sourcing, using MediaTek for one chip line while bringing in Marvell for an inference-focused variant. O'Malley added that the specialized SerDes expertise of Broadcom and Marvell keeps their market positions "defensible" in this evolving landscape.

Memory: The Next Frontier

Memory technology has emerged as another critical battleground, with capacity, bandwidth, and power efficiency being central considerations. Nvidia recently announced its NVHBM technology, which it claims could boost memory bandwidth by up to 30% while cutting HBM power consumption by 15%.

Looking further ahead, the industry is exploring 3D DRAM, with Samsung and Cerebras investigating stacked memory configurations. However, the Barclays analyst noted these efforts are in "very early days" due to significant thermal and power delivery challenges. While CXL memory expansion continues to draw interest, O'Malley stated he remains "wary until we see evidence of deployments."