Model Selection

"Open-weight" means the trained parameters are published for download — you run them offline, no API. A handful of families dominate what you run locally:

Run ollama list to see what you have, then browse the catalogue at ollama.com/library.

The number before the B is the parameter count in billions — a 7B model has roughly 7 billion weights. More parameters means more capability and more memory: footprint is roughly parameters x bytes-per-parameter, set by quantization (Step 3).

Sizing for dense models at a common 4-bit quant:

These are floors, not targets — leave headroom for the KV cache (the conversation's running memory, which grows with context length).

ollama show llama3.1:8b    # what a pulled model weighs
ollama ps                  # live: resident size + CPU/GPU split

Quantization stores each weight in fewer bits. Full precision is 16-bit (FP16); 4-bit cuts the footprint roughly 4x at a small, usually acceptable quality cost. The GGUF format used by llama.cpp and Ollama encodes this in the tag, e.g. Q4_K_M.

How to read a GGUF quant tag:

• The number is bits per weight: Q4 = 4-bit, Q5 = 5-bit, Q8 = 8-bit.
• _K is a "k-quant" — mixed-precision that spends more bits on the tensors that matter most.
• The trailing letter is the size class: _S (small), _M (medium), _L (large).

Approximate size for a Llama-3.1-8B model:

Heuristic: start at Q4_K_M; step up to Q5_K_M or Q6_K with headroom; go below Q4 only when forced.

ollama pull qwen3:8b              # default quant (typically a Q4 k-quant)
ollama pull qwen3:8b-q5_K_M       # request a higher-fidelity build

To compare models fairly, pull several side by side. Ollama keeps each tag as a separate entry, so you can hold several sizes and quants at once.

ollama pull llama3.1:8b
ollama pull qwen3:8b
ollama pull phi4:14b
ollama list                 # confirm what's resident, with sizes

Storage lives in one directory you can relocate via the OLLAMA_MODELS env var — handy when models are 4-8 GB each.

export OLLAMA_MODELS=/Volumes/FastSSD/ollama-models

A comparison sweep eats disk fast — clean up with ollama rm <tag> as you rule variants out.

Families have personalities — durable tendencies, not guarantees. Verify on your prompts, but this is a sane starting map:

Two more axes matter:

• Instruct vs base. A -instruct (or -it) build follows instructions; a base build just continues text and frustrates you in a chat loop. Always pick instruct for interactive use.
• License. Open weights still ship a license — some permit commercial use freely, some restrict it. Read the model card before building on it.

# Reasoning models "think" before answering — give them room
ollama run deepseek-r1:8b "A bat and ball cost 1.10; the bat costs 1.00 more. How much is the ball?"

Reputation gets you a shortlist; your own prompts pick the winner. Put your real prompts in prompts.txt and run every candidate against every one. The --verbose flag prints the eval rate (tokens/second), so you read quality and speed at once:

for m in llama3.1:8b qwen3:8b phi4:14b; do
  echo "===== $m ====="
  while IFS= read -r p; do
    ollama run "$m" "$p" --verbose
  done < prompts.txt
done

For structured scoring, hit Ollama's local API (port 11434) for the metrics:

curl -s http://localhost:11434/api/generate -d '{
  "model": "qwen3:8b",
  "prompt": "Write a haiku about local inference.",
  "stream": false
}' | jq '{eval_count, tps: (.eval_count / (.eval_duration/1e9))}'

Score on correctness, format, and tokens/second; re-run when a new release lands.