How AI Model Size Drives Memory, and Where CXL and Optics Fit
Why model weights and KV cache outgrow the memory on each accelerator, and what CXL memory expansion, pooled memory, and optical links each solve.
For large AI models, memory is now as hard a limit as computing power.
A large model needs more memory than any single GPU holds, and every active user adds more. That spreads models across many GPUs while memory in other servers sits idle. CXL memory and optical links each solve part of the problem, with trade-offs in speed, cost, and power. Below are the questions we work through with AI infrastructure teams.
The questions we hear from AI infrastructure teams
Why won't our model fit on one GPU?
A model needs memory for every parameter. Llama 3.1 405B needs about 810 GB at 16-bit precision, and the largest GPUs today hold 288 GB. So it has to be split across at least three of them before it serves a single user.[1][2][7][8]
Why does memory run out as usage grows?
Each active conversation keeps its own working memory, called the KV cache. For that model, one long 128K-token conversation takes about 68 GB. Thirty-two of them take more than twice the memory of the model itself.[1][2]
Doesn't a mixture-of-experts model need less memory?
It needs less computation per answer, not less memory. Llama 4 Maverick uses 17B of its 400B parameters for each token, but all 400B must stay loaded, because the next token could need any of them.[4]
Can we just buy GPUs with more memory?
Only up to a point. Memory is fixed when the chip is built, and the largest today carry 288 GB. Past that you buy more GPUs, and the connections between them carry more of the work.[7][8]
Are we using the memory we already pay for?
Often not. Microsoft found that up to 25% of server memory sits unused, and about half of cloud VMs use less than half the memory they rent. Before newer techniques such as PagedAttention, serving software left most of its KV cache memory empty.[11][3]
Where does CXL memory fit?
CXL adds memory to a server beyond its slots, and lets several servers share one pool so idle memory gets used. The cost is speed: about 70 to 90 nanoseconds of extra delay for a pool shared by 8 to 16 processors. Some workloads absorb that and some don't.[11][12][13]
When do we need optical links?
When the memory pool spans more than one rack. Copper cables for PCIe and CXL reach about 2 m; past that the links go optical. PCI-SIG now standardizes PCIe over optics, and NVIDIA rates its co-packaged optics at 5 times the power efficiency of plug-in modules.[15][16][17]
A worked example: Llama 3.1 405B
126 layers, 8 KV heads of dimension 128, and a 128K-token context[1]. Weights are parameters times bytes per parameter; the KV cache is a key and a value per layer, KV head, and token at 2 bytes each[2].
| Item | Memory |
|---|---|
| Weights at BF16 | 810 GB |
| Weights at FP8 | 405 GB |
| Weights at FP4 | 203 GB |
| KV cache per token | 516 KB |
| KV cache, one 128K-token request | 67.6 GB |
| KV cache, 32 such requests | 2.16 TB |
FP4 formats also store scale factors, which add a little to the FP4 figure.
| Accelerator | Memory | Needed |
|---|---|---|
| NVIDIA H100 SXM[5] | 80 GB HBM3, 3.35 TB/s | 11 |
| NVIDIA H200[6] | 141 GB HBM3e, 4.8 TB/s | 6 |
| NVIDIA B300 (Blackwell Ultra)[7] | 288 GB HBM3e, 8 TB/s | 3 |
| AMD Instinct MI355X[8] | 288 GB HBM3E, 8 TB/s | 3 |
| Google TPU7x (Ironwood)[9] | 192 GiB HBM, 7.37 TB/s | 4 |
| AWS Trainium3[10] | 144 GB HBM3e, 4.9 TB/s | 6 |
Vendor specifications as of October 22, 2026. The count covers 810 GB of weights only, before KV cache and activations.
Where each memory tier fits
Latency is given relative to local server memory, because absolute figures depend on the processor and the CXL device.
HBM on the accelerator[7][8]
Reach
On the package
Cost
Highest bandwidth, fixed capacity
Fits
Weights and the active KV cache
Host DRAM
Reach
Same server
Cost
Baseline latency
Fits
Conversation memory moved off the GPU, training state
CXL memory expansion[11][14]
Reach
Same server, module or card
Cost
A little slower than local memory
Fits
More capacity per server than DIMM slots allow
Measure before moving data to a slower tier
In Pond's tests of 158 workloads with memory latency increased by 182%, 43% ran within 5% of their local-memory performance, while more than 21% slowed by over 25%. A remote memory tier helps some workloads and hurts others, so it has to be measured per workload.[11]
What a memory assessment covers
We size the memory your models and users really need, then decide which tier each part belongs in and what links connect them.
Plan a memory assessment- How much memory each model needs for its weights, its users, and their conversation lengths
- Which data can move to slower, cheaper CXL memory without hurting response times
- How big a shared memory pool should be, given the memory sitting idle today
- Whether copper is enough or the pool needs optical links
- Which vendors and standards fit, across CXL, PCIe, and optical interconnect
Selling to hyperscalers and AI data centers
Data center spending is expected to pass $1 trillion in 2026 and $3 trillion by 2030, according to Dell'Oro Group. If you make optics, CXL memory, networking, power, or rack components, a small number of engineering and sourcing teams decide whether you get a share of it. We help you reach those teams and get designed in.[18][24]
How these buyers decide
The hyperscaler chooses, the manufacturer orders
Hyperscalers and system makers often decide which components go into a design, while the contract manufacturers that build the systems place the orders. You have to win both.[19]
Design-in takes up to two years
Qualification can take several months or more, and volume shipments can start up to two years after a design win. Your launch plan and runway have to cover that wait.[19]
What we do for suppliers
Business development and sales support for companies selling optics, memory, networking, and rack components into AI data centers.
Talk about selling to hyperscalersAI Infrastructure Leaderboard- Find the teams that own your category at each hyperscaler, AI cloud, and system maker, and who signs off
- Introductions through the network our team built at AWS, Microsoft, and AI hardware companies
- An evaluation package buyers can approve: benchmarks, reference designs, reliability data, and a second-source plan
- Which standards bodies to join and what to contribute, so your part is written into the specification
- Partnerships with the system makers and contract manufacturers that build the racks
- Target accounts ranked by capital spending, from the Prospectory AI Infrastructure Leaderboard
Frequently Asked Questions
Every parameter has to stay in memory while the model answers. At 16-bit precision that is 2 bytes each, so a 405 billion parameter model needs about 810 GB before it serves anyone. That is more than any single GPU holds.
Each conversation keeps its own working memory, called the KV cache. For Llama 3.1 405B that is about 516 KB per token, so one long 128K-token conversation needs about 68 GB on top of the model.
When you run out of memory before you run out of speed: conversation memory moved off the GPU, large lookup tables, or memory sitting idle in individual servers. CXL memory is a little slower than memory in a server's own slots, so test each workload before moving it.
Copper cables for PCIe and CXL reach about 2 m, enough inside a rack. Pools that span racks need optical links, which PCI-SIG now standardizes and which use less power than plug-in optical modules when built into the switch package.
Yes. We find the engineering and sourcing teams that choose components at hyperscalers, AI clouds, and system makers, make introductions through our team's network from AWS and Microsoft, and build the evaluation package and standards plan those buyers expect. Design-in can take up to two years, so we plan for that from the start.
References
Figures checked against these sources on October 22, 2026.
- [1] The Llama 3 Herd of Models (Meta, 2024), Table 3. https://arxiv.org/abs/2407.21783
- [2] Mastering LLM Techniques: Inference Optimization (NVIDIA, 2023). https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/
- [3] Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023). https://arxiv.org/abs/2309.06180
- [4] The Llama 4 herd (Meta, 2025). https://ai.meta.com/blog/llama-4-multimodal-intelligence/
- [5] NVIDIA H100 Tensor Core GPU specifications. https://www.nvidia.com/en-us/data-center/h100/
- [6] NVIDIA H200 Tensor Core GPU specifications. https://www.nvidia.com/en-us/data-center/h200/
- [7] Inside NVIDIA Blackwell Ultra (NVIDIA, 2025). https://developer.nvidia.com/blog/inside-nvidia-blackwell-ultra-the-chip-powering-the-ai-factory-era/
- [8] AMD Instinct MI355X specifications. https://www.amd.com/en/products/accelerators/instinct/mi350/mi355x.html
- [9] Google Cloud TPU7x (Ironwood) documentation. https://docs.cloud.google.com/tpu/docs/tpu7x
- [10] AWS Trainium. https://aws.amazon.com/ai/machine-learning/trainium/
- [11] Pond: CXL-Based Memory Pooling Systems for Cloud Platforms (ASPLOS 2023). https://arxiv.org/abs/2203.00241
- [12] CXL 3.0 specification release (CXL Consortium, 2022). https://computeexpresslink.org/wp-content/uploads/2023/12/CXL_3.0-Specification-Release_FINAL.pdf
- [13] CXL 4.0 specification release (CXL Consortium, 2025). https://computeexpresslink.org/wp-content/uploads/2025/11/CXL_4.0-Specification-Release_FINAL_Website-Copy.pdf
- [14] Samsung CMM-D CXL memory module. https://semiconductor.samsung.com/cxl-memory/cmm-d/
- [15] PCI-SIG CopprLink cable specifications (2024). https://www.businesswire.com/news/home/20240501529875/en/
- [16] PCI-SIG optical interconnect solution for PCIe (2025). https://www.design-reuse.com/news/202528858-pci-sig-reg-announces-pcie-reg-optical-interconnect-solution/
- [17] NVIDIA silicon photonics networking. https://www.nvidia.com/en-us/networking/products/silicon-photonics/
- [18] AI infrastructure buildouts and memory cost inflation drove data center capex higher in 1Q 2026 (Dell'Oro Group, June 2026). https://www.delloro.com/news/ai-infrastructure-buildouts-and-memory-cost-inflation-drove-data-center-capex-higher-in-1q-2026/
- [19] Astera Labs annual report on Form 10-K for 2025 (SEC filing). https://www.sec.gov/Archives/edgar/data/1736297/000173629726000010/alab-20251231.htm
- [20] UALink 200G 1.0 specification release (UALink Consortium, 2025). https://ualinkconsortium.org/wp-content/uploads/2025/04/UALink-1.0-Specification-PR_FINAL.pdf
- [21] Ultra Ethernet Consortium specification 1.0 (2025). https://ultraethernet.org/ultra-ethernet-consortium-uec-launches-specification-1-0-transforming-ethernet-for-ai-and-hpc-at-scale/
- [22] Optical Scale-up Consortium established to create an open specification for AI infrastructure (TelecomTV, March 2026). https://www.telecomtv.com/content/next-gen-telco-infra/optical-scale-up-consortium-established-to-create-an-open-specification-for-ai-infrastructure-55073/
- [23] AWS Marketplace product types. https://docs.aws.amazon.com/marketplace/latest/buyerguide/buyer-product-types.html
- [24] AI buildout maintains momentum as data center capex surpasses $3 trillion by 2030 (Dell'Oro Group, August 2026). https://www.delloro.com/news/ai-buildout-maintains-momentum-as-data-center-capex-surpasses-3-trillion-by-2030/
Planning memory for a large model?
Bring one model, its context lengths, and your expected users. We work out the memory it needs and where each part should live.