AI inference is running into a memory bottleneck. Huawei wants to tackle it with OceanStor M900

What happened

At HUAWEI CONNECT 2026 in Shanghai, Huawei introduced OceanStor M900 Context Memory Storage. It is meant for AI inference (the step where a trained AI model answers questions) in large data centers and Huawei's Ascend SuperPoD systems. It adds a shared storage layer for the KV cache, which is the memory an AI model saves so it does not have to redo work on text it has already seen.

Why memory is a problem

AI chats are getting longer, and AI agents (programs that carry out tasks on their own) need to remember more. This makes the KV cache much bigger. GPU memory and system memory are fast but costly and limited in size. Normal storage has more room but is slower, because data has to pass through the CPU (the main processor), and this leaves expensive AI chips waiting.

Key numbers and claims

• Capacity: up to 64PB (petabytes) shared across a cluster, according to Huawei.
• Access latency (delay before data arrives): 60 microseconds, according to Huawei.
• Possible results: about twice the token throughput (words or word-pieces processed per second) and half the time to first token in typical AI coding work, according to Huawei.
• Endurance: up to 24 drive writes per day, which Huawei says is 16 times that of normal SSDs.

Why it matters

The article argues that AI speed now depends on memory and networking as much as on chip power. If Huawei's claims hold up, faster access to saved context could lower the cost of running AI at scale. Independent tests are not described in the article, so these numbers should be read as the company's own.

What to watch

• Whether independent tests confirm the speed and cost claims.
• How customers use the M900 with Huawei's Ascend chips and UnifiedBus network.
• Whether other storage and chip makers offer similar context memory layers.