Best memory for AI? The fastest one your model fits in. #shorts

2 Просмотры

Спасибо! Поделитесь с друзьями!

Вам не понравилось видео. Спасибо за то что поделились своим мнением!

Издатель Oct 2, 2026

"Which system memory is best for AI generation?" The short answer: the fastest memory your whole model fits in. Capacity decides whether a model runs at all; bandwidth decides how fast it generates. On a PC that usually means GPU VRAM, not system RAM: an RTX 5090's 32 GB of GDDR7 moves 1,792 GB/s, while desktop DDR5 on Intel's Core i9-14900K (two channels) tops out at 89.6 GB/s. Same 32 GB, twenty times the bandwidth.

RULE 1: THE MODEL MUST FIT. Hugging Face: "Loading the weights of a model having X billion parameters requires roughly 2 * X GB of VRAM in bfloat16/float16 precision". So an 8B model needs about 16 GB, and Llama-2-70b needs 140 GB, 4.4 times a 32 GB card.

RULE 2: BANDWIDTH SETS THE SPEED. NVIDIA: "The speed at which the data (weights, keys, values, activations) is transferred to the GPU from memory dominates the latency, not how fast the computation actually happens." Dividing bandwidth by model size gives a ceiling: a 16 GB model can generate at most about 5.6 tokens per second from DDR5 and 112 from GDDR7.

THE MEMORY LADDER. HBM3e on NVIDIA's H200: 141 GB at 4.8 TB/s. GDDR7 on the RTX 5090: 32 GB at 1,792 GB/s. Apple's M5 Ultra Mac Studio: "up to 512GB of unified memory and 1.2TB/s of memory bandwidth". DDR5 on a desktop: up to 192 GB at 89.6 GB/s.

WHEN VRAM RUNS OUT. NVIDIA's driver can spill into system memory: "This enabled applications which previously crashed when running out of GPU memory to continue to run, albeit at lower speeds." In llama.cpp, -ngl sets the "max. number of layers to store in VRAM"; leaving a quarter of a 16 GB model in DDR5 costs 87% of the time per token. Shrink the model first, and remember what Hugging Face says: "model quantization trades improved memory efficiency against accuracy and in some cases inference time."

Sources: https://www.nvidia.com/en-us/geforce/news/rtx-50-series-graphics-cards-gpu-laptop-announcements/ | https://www.intel.com/content/www/us/en/products/sku/236773/intel-core-i9-processor-14900k-36m-cache-up-to-6-00-ghz/specifications.html | https://huggingface.co/docs/transformers/llm_tutorial_optimization | https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/ | https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices | https://www.nvidia.com/en-us/data-center/h200/ | https://www.apple.com/newsroom/2026/08/apple-introduces-new-mac-studio-with-m5-max-and-m5-ultra/ | https://nvidia.custhelp.com/app/answers/detail/a_id/5490 | https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

#ai #llm #gpu #vram #ram #stablediffusion #localai #machinelearning #shorts