AI Memory Demand Expands Beyond HBM to DRAM and Storage, Bernstein Says

The intense memory requirements for artificial intelligence are creating a bottleneck that extends far beyond high-bandwidth memory (HBM), driving demand for conventional DRAM, NAND flash, and storage, according to analysts at Bernstein. This diversification presents a broader investment opportunity across the memory sector, with specific needs depending on the AI workload.

Workload-Specific Demands

In a recent research note, Bernstein analysts explained that the investment opportunity in memory increasingly depends on the type of AI task being performed. While HBM remains central, different processes place distinct pressures on the entire memory stack.

• Training: This initial phase remains highly compute-intensive, with HBM's bandwidth limiting model size and speed. However, large-scale training also consumes significant system DRAM and requires local SSDs and networked storage for datasets and checkpointing.
• Inference: Running a model involves two stages. The initial "prefill" phase is compute-bound, relying on GPUs and HBM. The subsequent "decode" phase is memory-bound, limited by the size of the key-value (KV) cache, which stores the context of a conversation.
• Retrieval-Augmented Generation (RAG): This technique broadens demand to SSD or HDD capacity for databases and requires more system DRAM to perform searches on that data.
• Agentic AI: Autonomous AI systems could intensify memory needs further by retaining intermediate results, rapidly increasing demand for the KV cache and conventional server memory.

Market Impact and Emerging Tech

Ad

The growing importance of the KV cache is particularly notable. Bernstein said that in large-scale deployments, the memory required for the KV cache could exceed that needed for the model weights themselves, potentially limiting the number of concurrent users a service can support.

This pressure is fostering the development of new memory tiers designed to balance performance, capacity, and cost. The analysts pointed to emerging technologies like CXL memory and initiatives such as Nvidia’s "Storage Next" and CMX context storage as solutions being developed to address this challenge.

Memory manufacturers are also exploring technologies like high-bandwidth flash, which aims to provide HBM-like bandwidth with the higher capacity and lower cost of NAND. However, Bernstein cautioned that the technical hurdles for such technology remain high.

Based on this evolving landscape, Bernstein rated several memory and storage companies as Outperform, including Samsung Electronics, SK hynix, Micron, SanDisk, Seagate, and Western Digital. The firm rated Kioxia as Underperform.