Parameter count is only the first term. A defensible answer requires weight precision, architecture, KV-cache geometry, context length, concurrency, runtime overhead and the memory topology of your machine.
This estimator uses transparent first-principles math. It is deliberately conservative: the result is a planning estimate, not a promise that a specific backend will allocate exactly the same amount.
Recommended installed memory for this scenario.
● Practical fitUnified memory lets CPU and GPU share the same pool, but the operating system and applications share it too. Do not plan to consume 100% of installed memory.
For autoregressive inference, the dominant persistent allocations are usually the model weights and the KV cache. Activations, scratch buffers, allocator fragmentation, multimodal components and the host runtime create additional headroom requirements.
A 30.5B-parameter model at an effective 4.5 bits/weight is about 16.0 GiB before format-specific and runtime effects. In an MoE model, active parameters primarily reduce compute per token; the full expert set still has to be stored unless the serving system explicitly offloads or sparsely loads experts.
The two is for keys + values. Grouped-query attention (GQA) cuts KV memory by using fewer KV heads than query heads. Context length T and concurrency B scale the cache roughly linearly for standard dense caches.
Then reserve capacity for the OS, other apps, framework buffers, tokenization, graph workspaces and transient peaks. The exact overhead is implementation-specific, so a planning margin is more honest than a universal fixed number.
Context length is not normally a second large memory object added independently after the KV cache. Its dominant persistent cost is expressed through the KV cache (plus comparatively smaller token/input/runtime buffers). Adding a generic “context GB” term on top of a KV calculation can double-count the same phenomenon.
The original is useful as an introductory mental model. The upgrade is to replace categorical RAM tiers with explicit assumptions and architecture-aware arithmetic.
Keep the intuition, refine the language. You need the stored weights plus sequence-state memory and runtime allocations. On CPU+GPU hybrid systems, all weights need not sit in GPU VRAM simultaneously.
Apple’s MLX uses a shared memory pool accessible by CPU and GPU. That is a genuine advantage for large local models, but installed unified memory is also consumed by macOS and other processes.
The tiers are not model-class laws. A 70B Q4 can fit in a 64 GiB envelope at moderate context, while 128K FP16 KV cache for a 70B GQA architecture can consume tens of GiB by itself.
Lower-bit weights are usually smaller, but speed depends on kernels, hardware and memory bandwidth. Mixed GGUF quantizers have real bits-per-weight that differ from their labels; for example, Hugging Face documents Q4_K at 4.5 bpw and Q6_K at 6.5625 bpw.
For memory sizing, total stored parameters matter more than active parameters. Qwen3-30B-A3B activates roughly 3.3B parameters per token but stores about 30.5B total. Llama 4 Scout similarly separates active and total parameter counts.
These are conservative, single-user inference guidelines for broadly supported 4–6 bit quantization. They assume reasonable system headroom and do not imply acceptable speed. A model can fit and still be painfully slow.
| Installed memory | Comfortable local envelope | Representative use | Watch-outs |
|---|---|---|---|
| 16 GiBentry local AI | 3B–8B models; some 12B/14B at aggressive or balanced quantization and short context | chatlight codinglearning | OS pressure arrives quickly; avoid assuming max advertised context. |
| 24 GiBcompact workstation | 8B–14B with generous context; many ~20B-class Q4 deployments | codingRAGsmall agents | 24B-class Q4 can be feasible but leaves less room for long context. |
| 32 GiBstrong mainstream tier | 24B dense Q4; ~30B total-parameter MoE Q4; 8B/14B with very large context | serious codingmultilinguallocal RAG | “Fits in 32 GB” often means moderate, not maximum, context. |
| 48 GiBupper-mid local tier | 30B-class at higher precision; 70B at Q3/Q4 with constrained context | researchagents | 70B Q4 is possible but operational headroom can be narrow on some stacks. |
| 64 GiBlarge-model sweet spot | 70B Q4 with short-to-moderate context; smaller models with long context or concurrency | 70B-classadvanced coding | Full 128K context can invalidate the fit depending on KV geometry and precision. |
| 96 GiBhigh-headroom local | 70B Q5/Q6; 70B Q4 with long context; 100B-class Q4 in selective cases | long contextmulti-agent | Memory bandwidth increasingly dominates usability. |
| 128 GiBpower workstation | 70B at high quantization fidelity; 100B+ Q4; larger MoE models if total weights fit | researchlarge RAG | MoE total weights still matter; “active B” is not a RAM number. |
| 192–256+ GiBspecialist / server class | 100B–400B-class quantized experiments, depending strongly on precision and architecture | labmulti-modelextreme context | Fit becomes a weak proxy for practicality; bandwidth, interconnect and backend support are decisive. |
GiB uses binary units (2³⁰ bytes). Model repositories often report GB/file sizes in decimal units. Always compare like with like.
Block scales, minima, importance matrices and mixed tensor types add overhead. The GGUF documentation therefore reports effective bits-per-weight (bpw) for several K-quants.
Reference precision. Roughly 1.86 GiB per billion parameters.
High-fidelity legacy GGUF block quantization; larger than naïve “8 bit” arithmetic.
High-quality K-quant; documented effective packing cost.
Middle-high precision for users with extra memory.
Common balanced local-inference class. Real files can vary with mixed tensors.
Importance-matrix-informed 4-bit family with lower packing cost.
Aggressive compression; quality sensitivity becomes model/task dependent.
Extreme memory saving. Validate quality for your workload rather than trusting a generic ranking.
GPTQ and AWQ established practical low-bit post-training quantization; newer importance-aware GGUF families push size lower. But the “best” format is backend- and workload-specific. Measure task accuracy, latency, throughput and memory together.
The examples use the standard dense GQA KV formula and publicly documented architecture values. They isolate KV cache to make the effect visible; real runtimes add further allocations.
Its Q4_K weight estimate is about 12.6 GiB. At the full 128K context, a conventional FP16 KV cache alone is about 20 GiB. So “fits on a 32 GB machine once quantized” does not mean “fits at 128K context with an FP16 KV cache and full headroom.”
Q4_K weights are roughly 16 GiB by parameter arithmetic. The MoE’s low active parameter count helps compute efficiency, but it does not turn the model into a 3.3B-memory model. GQA’s four KV heads keep cache growth relatively efficient.
Q4_K weights are roughly 36.7 GiB. At 128K context, standard FP16 KV pushes persistent weights + cache toward ~77 GiB before runtime overhead. This is why a 64 GiB machine can be a good 70B-Q4 box at moderate context yet fail at maximum context.
Use the standard equation as a baseline, then inspect what the architecture and backend change.
The exact same model can be memory-feasible on one machine and awkward on another because the accessible pool, memory bandwidth and CPU↔GPU interconnect are different.
CPU and GPU can operate on arrays in the same unified memory pool without explicit device copies in MLX. This makes large-memory Macs unusually straightforward for local inference. The trade-off is that the OS and everything else share the same pool.
For fully GPU-resident inference, VRAM is the hard fit constraint. CPU RAM does not magically become VRAM. Frameworks can offload weights/cache, but the interconnect and host memory path then become part of the latency model.
System RAM can hold very large GGUF models, and llama.cpp supports CPU+GPU hybrid inference. The practical ceiling is often memory bandwidth rather than capacity. A model that fits can still decode slowly.
Tensor/expert parallelism can aggregate capacity across devices, but effective fit depends on how weights, activations and KV cache are partitioned. Interconnect topology becomes a first-order performance variable.
For batch-1 autoregressive decoding, large quantized models are often memory-bandwidth-bound: much of the model must be streamed to produce each token. A simple roofline gives an optimistic ceiling before compute, cache traffic and kernel inefficiency.
This is intentionally a ceiling, not a benchmark. Actual throughput is lower and depends on kernel efficiency, cache traffic, compute, quantization support, batching and interconnects.
optimistic tokens/s bandwidth ceiling
This parser runs entirely in your browser. Paste a Hugging Face-style model config.json; no content is uploaded anywhere.
The best practice is to move from file facts → architecture facts → workload facts → platform facts → measured peak memory.
Prefer the actual GGUF/safetensors shard total over parameter-count arithmetic. Quantization labels are an approximation to file size, not a substitute for it.
Record layers, KV heads, head dimension, attention pattern and maximum context. Determine whether the standard GQA cache formula applies.
Choose the context you will actually allocate, plus batch/concurrency and multimodal requirements. Maximum advertised context is not a default target.
Decide what must fit in VRAM, unified memory or system RAM. If offloading is required, explicitly account for the performance cost.
Launch the exact backend, model, context and cache precision. Verify actual memory and token/s under representative prompts. Production sizing ends with measurement, not arithmetic.
Fine-tuning and training add gradients, optimizer states, master weights, activations and checkpointing behavior. QLoRA changes the economics, but it is a different memory model.
Research reviewed for this page includes official model cards/configs, framework documentation and peer-reviewed or primary academic work. Links open the original sources.