AI Agents Need System RAM: 8 Sessions Can Take 515 GB of KV Cache

Eight AI agent sessions that each hold a million tokens of context need 515 GB of KV cache, the memory a model keeps for each conversation, at FP8 on MiniMax M3.1-Flash, Voltgrid calculates, more than five 96 GB workstation GPUs hold. Serving software already sends that overflow to system RAM, the computer's ordinary memory, and that is where the memory bottleneck for AI agents is moving.

Chip makers see the shift from their side. AMD told investors on Nov. 3 that agents are "quickly becoming the next killer app for CPUs in the cloud," and Arm said on Nov. 4 that persistent agents make efficient processors more critical in data centers.

An AI agent is a model that works through a task in steps. It reads, calls a tool such as a web search or a code runner, waits for the result, and reads again. Each time, the model keeps running notes on every token, the small chunks of text it processes, so it need not reread the whole conversation at the next step. Engineers call those notes the key-value (KV) cache. They grow with every token, and each conversation keeps its own.

A chatbot answers and is often done. An agent holds its conversation open while it waits on a tool, and a team or a cloud service runs many agents at once. The graphics processor's memory is fast but small. Serving software can therefore park the notes of paused conversations in main memory, the DDR5 or LPDDR5X attached to the CPU, and spill the rest to an SSD, much as an office moves papers from a small desk to a filing cabinet in the same room: slower to reach, far roomier.

For people buying a workstation to run models locally, system RAM now decides how many agents can work at once. For data centers, the ordinary server, a processor surrounded by memory, is back in demand while memory is scarce. Voltgrid tested no machine for this piece: every figure is a maker's rating, a named outlet's report or Voltgrid's arithmetic, labelled as such.

Why Every Agent Session Needs Its Own Cache

A language model has no memory of a conversation. To write each new word it consults notes it made about every earlier word as it read them. Those notes are the key-value (KV) cache, and without them the model would reread the whole conversation for every word.

The notes belong to one conversation, so each agent session keeps its own copy, and it grows with every token. Voltgrid calculates that one session of 1,048,576 tokens on MiniMax M3.1-Flash needs 64.4 GB at FP8, an 8-bit format, and eight sessions need 515 GB. One RTX PRO 6000, by NVIDIA's specification, holds 96 GB.

An agent pauses while a tool runs, and its notes must wait somewhere or be rebuilt. Graphics-card memory is fast but small, so software parks paused sessions in the computer's ordinary RAM, which is slower and far roomier.

DDR5, GDDR7 and HBM: The Memory Ladder

At the top sits high-bandwidth memory. NVIDIA's DGX Station puts 252 GB of HBM3e beside its GPU at 7.1 TB/s, according to NVIDIA. One rung down, the RTX PRO 6000 Blackwell carries 96 GB of GDDR7 at 1,792 GB/s, both in the 600 W Workstation Edition and in the 300 W Max-Q edition, which NVIDIA says scales to four cards per system. Apple rates the unified memory of the Mac Studio with M5 Ultra, shared by CPU and GPU, at 1.2 TB/s, with up to 512 GB on the top chip.

System memory comes next. AMD's WRX90 platform for Threadripper PRO runs eight DDR5 channels and up to 2 TB, and AMD says its Threadripper PRO 9000 WX-Series takes registered DDR5-6400 modules. Eight channels at 6,400 million transfers per second, 8 bytes each, give a theoretical peak of 409.6 GB/s by Voltgrid's calculation; it is a ceiling, since no source says a full 2 TB build runs at 6,400 MT/s, and real throughput is lower. The DGX Station's own CPU side, 496 GB of LPDDR5X on NVIDIA's Grace processor, is rated at 396 GB/s, almost the same on paper. At the bottom, a PCIe 5.0 SSD such as Samsung's 9100 PRO is rated at 14.8 GB/s for sequential reads, in sizes up to 8 TB.

By Voltgrid's arithmetic, eight-channel DDR5 is about 4.4 times slower than the RTX PRO 6000's GDDR7 and about 17 times slower than the Station's HBM3e, but about 28 times faster than that SSD, and the 2 TB ceiling holds about 21 times the 96 GB on one card. By the same arithmetic, reading a 64.4 GB session cache out of the Station's LPDDR5X takes at least 0.16 seconds at its rated 396 GB/s, and out of the SSD at least 4.4 seconds. Those floors leave out the hop to the GPU: the Station joins its memories at 900 GB/s over NVIDIA's NVLink-C2C, while a Threadripper workstation crosses PCIe, for which we have no sourced figure.

Price runs the other way. TrendForce's spot table, in its Oct. 26 module update, put a 32 GB DDR5 RDIMM (4800/5600) at a session average of $2,165, or $67.7 per gigabyte by Voltgrid's arithmetic. That is a spot price, not what large buyers pay under contract. Samsung lists the 8 TB 9100 PRO at $2,719.99, or $0.34 per gigabyte by the same arithmetic, about one two-hundredth of the RDIMM's spot price per gigabyte. No maker or analyst we read prices HBM3e, GDDR7 or LPDDR5X per gigabyte. One comparison shows how dear plain DRAM has become: by Voltgrid's arithmetic, eight of those modules, 256 GB to fill a WRX90 board's eight slots, would cost $17,320, more than the $16,000 NVIDIA listed for a 96 GB RTX PRO 6000 Workstation Edition on Aug. 10, when the card showed as out of stock.

How Many Agent Sessions Multiply KV Cache Memory

On Nov. 4, Voltgrid calculated the KV cache four open models need for one session of 1,048,576 tokens at FP8, an 8-bit format. MiniMax M3.1-Flash needs 64.4 GB, or up to 72.5 GB if its index keys are kept per token. Z.ai's GLM-5.3 needs 49.9 GB and Mistral Large 4 needs 43.4 GB. DeepSeek V4.1 Flash, whose design compresses the cache, needs 1.68 GB. That is the cache alone; the weights come on top.

Every extra session brings its own copy. Multiplying the same figures, eight full-length sessions need 515 GB for MiniMax, 399 GB for GLM-5.3, 347 GB for Mistral Large 4 and 13.4 GB for DeepSeek, and sixteen need 1,031 GB, 799 GB, 694 GB and 26.8 GB. Against those, one RTX PRO 6000 has 96 GB, two have 192 GB, the DGX Station has 252 GB of HBM3e and 748 GB in all, and a WRX90 board tops out at 2 TB.

A million tokens is a ceiling, and shorter sessions shrink the bill in proportion. In a scenario of 131,072 tokens (128K) each, an eighth of the ceiling, sixteen MiniMax sessions would still need 128.8 GB of cache at FP8 by Voltgrid's arithmetic, more than one 96 GB card, before the model's 252.4 GB of weights. On paper, by Voltgrid's arithmetic and ignoring everything else in memory, 2 TB of system RAM holds 31 full-length MiniMax caches at 64.4 GB each and the Station's 496 GB of CPU memory holds seven; if MiniMax keeps its index keys for every token, at 72.5 GB a session, those fall to 28 and six.

How KV Cache Offloading Works in vLLM and Dynamo

The main serving stacks already treat CPU memory as the second rung. NVIDIA's Dynamo documentation describes its KV Block Manager as a unified memory layer and write-through cache for vLLM and TensorRT-LLM, spanning GPU memory, host memory, SSD and remote storage. Two rules show where the load falls: a CPU cache smaller than the GPU's brings no benefit, the docs warn, and to spare SSDs from wear only blocks that score as frequently reused move on from CPU memory to disk.

vLLM, the open-source serving engine, gained an offloading connector in version 0.11.0, its developers at IBM Research wrote in January. It copies KV data to CPU DRAM in the background, for the case many parallel agents create: when concurrent requests fill the GPU, the engine may preempt one and discard its cache, and offloading it first saves the recomputation. Ray's serving guide names agents outright: a session that resumes after a pause may find its cache evicted from the GPU and have to recompute the whole prompt, unless a CPU or external tier kept it.

AMD, Arm, Intel and TrendForce Signals on Agent Demand

In prepared remarks for its third-quarter results on Nov. 3, it said agents "work continuously and in parallel," adding to accelerator demand while bringing what AMD called a substantial increase in CPU demand, and that cloud providers added significant EPYC capacity for agents. Data Center revenue was a record $8.5 billion, up 97% from a year earlier. Its all-CPU Helion rack holds up to 144 "Venice" processors, and AMD expects server CPU revenue to double in 2027, as Voltgrid reported.

Arm made the same case in its Nov. 4 shareholder letter: persistent agents run software, call tools and coordinate work between calls to the accelerator, which makes efficient CPUs more important to data-center performance. Arm said its data-center royalties doubled and, by its own count, Arm-based servers went from 15% of the top hyperscalers' server market in 2024 to more than 50% in 2025, with more than 60% expected by the end of 2026, as Voltgrid reported.

Intel's Oct. 29 results release said its immediate priority is more supply to meet what it called extraordinary demand for Xeon, and its Data Center and AI revenue rose 78% to $7.3 billion. The release ties Xeon to agents only through a Hot Chips 2026 showing of the coming "Diamond Rapids" Xeon for agentic orchestration.

TrendForce draws the memory line. On Sept. 30 it projected conventional DRAM contract prices up 10% to 15% quarter on quarter in the fourth quarter, with server DRAM undersupplied as cloud providers and server makers buy more bits for RDIMMs in general servers running agentic AI. It expects enterprise SSD bit demand to grow more than 80% in 2026. On Oct. 29 it said the move to inference and agents is driving a new wave of CPU demand, and that every CPU brings DDR5 with it.

How Much System RAM a Local AI Workstation Needs

Three designs answer the agent question differently. A Threadripper PRO tower with RTX PRO 6000 cards splits the job: up to four 96 GB Max-Q cards, 384 GB in all per NVIDIA, for speed, and up to 2 TB of DDR5 for capacity. ASUS's Pro WS WRX90E-SAGE SE board, for one, has eight slots for registered ECC DDR5. AMD is building the same shape: its Threadripper Halo Station pairs a 96-core Threadripper PRO 9995WX with two 144 GB Instinct MI350P cards and up to 2 TB of DDR5, ServeTheHome reported from AMD's Sept. 4 IFA keynote, about seven gigabytes of system memory per gigabyte of accelerator memory at the full 2 TB with two cards, by Voltgrid's arithmetic. AMD gave no price and said 2027.

NVIDIA's spec page lists one memory configuration for the DGX Station: 252 GB of HBM3e and 496 GB of LPDDR5X, about two gigabytes of CPU memory for every gigabyte on the GPU by Voltgrid's arithmetic, joined at 900 GB/s. The lowest US partner listing our desk found, Exxact's, starts at $98,450, as read on Nov. 4. Voltgrid's DGX Station review shows the cost of the lower rung: StorageReview measured 35 to 44 tokens a second for one user on GLM-5.2 and GLM-5.3 when part of the model sat in the Grace memory.

Apple's Mac Studio has one pool, up to 512 GB at 1.2 TB/s shared by CPU and GPU, so there is no trip between tiers, but Tom's Hardware notes the memory cannot be upgraded. MacRumors put the 512 GB model at a minimum of $22,299, of which $15,500 is the memory option. On a 256 GB unit, MacStories saw a long-context run fail for lack of memory, our review noted.

Voltgrid's reading: for one person and one model, GPU memory still decides the experience, because it sets how fast the answer arrives. Our DGX Station review said capacity decides which models run and HBM how fast. Agents add a third question, how many conversations stay warm while they wait, and that answer lives in system RAM. Only the Threadripper-class tower is built around standard slots a buyer can fill later; NVIDIA specifies one memory configuration for the Station, and the Mac's cannot be upgraded. Because Dynamo's documentation says a CPU cache smaller than the GPU's cache space brings no benefit, a buyer planning for several agents should treat system RAM at least equal to GPU memory as a rough floor, not a target, and price it at today's DRAM rates.

What Agents Mean for Data Center Memory and CPUs

In a data center the same arithmetic runs across fleets: agents that keep conversations alive between GPU calls and run tools on CPUs need more host memory per GPU server and more ordinary servers per fleet. Voltgrid has tracked those CPU tiers, from Oracle's EPYC-based OCI E7, which AMD says will run agent sandboxes and in-memory caches, to Dell's rack of 256 Vera CPUs.

TrendForce says memory makers cannot fully meet that demand this quarter, held back by component supply, packaging and test capacity and limited flexibility in DRAM production. Prices, not volume, carry the market: Counterpoint figures relayed by Korean press put third-quarter DRAM revenue at roughly $200 billion, up 369% from a year earlier, on bit shipments up only about 4%, as Voltgrid reported on Nov. 5.

Someone else pays: AMD said memory and component costs weighed on PC demand, and Arm said memory pricing hurt its smartphone royalties. Voltgrid's reading: agents turn the general-purpose server back into a large DRAM buyer just as suppliers steer capacity toward server parts, so a workstation owner adding DDR5 pays a price set largely by cloud demand.

The Case Against System RAM as the Bottleneck

Model design can shrink the problem. DeepSeek V4.1 Flash's cache, 1.68 GB for a full session, is about a 38th of MiniMax's 64.4 GB at FP8 in our Nov. 4 calculation, and 0.93 GB in DeepSeek's own FP4 format; if compressed attention spreads, the pressure on system RAM eases. Much of the CPU demand AMD and Arm describe comes from agents running tools and code, not from parked caches, which argues for more processors rather than more memory per processor. SSDs may also take part of the overflow, as TrendForce's enterprise SSD forecast and the disk tiers in Dynamo and the LMCache cache layer suggest. And for one user with one conversation, little of this applies.

What We Don't Know Yet

No source we read measures how many agents a machine can run against its system RAM; this piece rests on documented software design and arithmetic, not a benchmark. We have no sourced figure for the PCIe link between GPU and CPU memory on a Threadripper workstation, so the real cost of moving a cache back there is unknown. How long real agent sessions run, and how many run at once, is unmeasured here. The DDR5 price is a spot average for 32 GB 4800/5600 modules, not a contract price and not the larger modules a 2 TB build needs, and Apple's spec page does not name the Mac Studio's memory type. The Dynamo pages we read cover version 1.3.0; newer releases may tier differently.

What to watch: AMD says the first Helion systems ship this quarter, NVIDIA expects DGX Station for Windows from partners in December, Digital Trends reported that the first 512 GB Mac Studio orders arrive on Dec. 3, and TrendForce's next contract-price outlook will show whether agent servers keep server DRAM short into 2027.