Hardware Requirements

Local inference is mostly a memory problem. The weights must live in fast memory the compute unit can reach — VRAM on a GPU, or unified memory on Apple Silicon. If they fit you get good speed; spill into system RAM and generation crawls.

Footprint is three things: parameters (trained weights in billions — the "7B" in a name), bytes per parameter (set by quantization — compressing 16-bit weights to 4-bit), and overhead (KV cache plus activations). A practical estimate:

memory_GB ≈ parameters_in_billions × bytes_per_param × 1.2

The 1.2 is rough 20% headroom for KV cache and activations. Bytes per parameter run ~2.0 at FP16 (full), ~1.0 at Q8 (near-lossless), and ~0.55 at Q4_K_M (the sweet spot). So a 7B at Q4_K_M needs roughly 7 × 0.55 × 1.2 ≈ 4.6 GB, versus ~17 GB at FP16. Quantization is the single biggest lever you have.

That gives a sizing map — Q4-class footprints including context, the practical minimum memory (VRAM, or usable unified memory):

Anchor on the size class, not a release — a "32B coding model" is a target you re-fill with the current best. With Ollama you pick a quant by tag (llama3.1:8b defaults to Q4_K_M; append -q8_0 for Q8). Q4_K_M is the default sweet spot, stepped up only with spare memory.

Context also claims memory: the KV cache grows with conversation length, adding ~1-3 GB at 32K tokens and several GB at 128K — for large documents, size up a tier.

NVIDIA is the path of least resistance: broadest tool support, mature CUDA drivers, most docs. What matters is VRAM (how big a model fits) and compute capability (whether the card is supported). Ollama needs compute capability 5.0+ and driver 531+ (legacy 5.0-6.2 cards want 570+). Run nvidia-smi to check driver and VRAM; most cards from the GTX 10-series on clear the 5.0 bar. What consumer cards run at Q4:

Data-centre cards (A100, H100) suit fine-tuning and multi-user serving — overkill for one person, usually rented.

AMD works, but the road is bumpier. Acceleration goes through ROCm, AMD's CUDA equivalent: Ollama needs the ROCm v7 driver stack on Linux (or ROCm v7 / HIP7 on Windows), covering recent Radeon RX, Radeon PRO, Ryzen AI, and Instinct cards. On Linux, confirm the GPU is visible:

rocminfo | grep -i "Marketing Name"

VRAM math matches NVIDIA — a 24 GB Radeon runs the same 32B Q4 model; the difference is driver maturity, not capacity. Acceleration paths, fastest first: native ROCm (supported cards), then Vulkan (unsupported AMD and Intel GPUs), then CPU.

Apple Silicon (M1 through M4, including Pro/Max/Ultra) is unusually good at local LLMs thanks to unified memory: the CPU and GPU share one high-bandwidth pool, so there is no separate VRAM and no PCIe copy. A 64 GB MacBook can give most of that to a model that needs a stack of GPUs on a PC.

Speed depends on unified memory and bandwidth, which climbs steeply up the line — the M4 Max supports up to 128 GB at up to 546 GB/s. Best of all, Ollama enables Metal acceleration automatically — no drivers or CUDA setup:

ollama run llama3.1:8b   # uses the Mac GPU, zero config

Mac sizing tracks unified RAM at Q4: 16 GB runs 7B-8B, 32 GB up to ~32B, 64 GB a 70B, 128 GB a 70B at higher quant or long context. The OS reserves some, so not all is usable.

Memory decides what runs; disk decides how many models you keep and how fast they load. A 7B Q4 model is ~4-5 GB, a 70B is ~40 GB — plan for 100 GB+ free and prefer an NVMe SSD (near-instant loads versus tens of seconds on disk). Manage the cache:

ollama list              # downloaded models and sizes
ollama rm llama3.1:70b   # remove a model to reclaim disk
du -sh ~/.ollama/models  # total cache size

To move the cache to a bigger drive, set OLLAMA_MODELS before starting Ollama (default differs by platform):

On a home connection a 40 GB model can take over an hour — pull large models before you need them.

Cannot run the models you want? You need not buy a GPU. Marketplaces rent instances by the hour — handy for trying a 70B, one-off fine-tuning, or serving.

Pricing moves constantly, but the durable picture: a consumer RTX 4090 rents in the low hundreds of cents per hour and data-centre A100/H100 cards in the low single digits of dollars per hour, with marketplaces cheapest on consumer GPUs and managed providers charging a premium for reliability. A few dollars an hour buys months of experiments before buying breaks even — rent for occasional jobs, own for private work.