Troubleshooting

Problem/cause/fix reference for the issues you hit most running models locally, assuming Ollama or llama.cpp at the terminal. For background, see the model guide and hardware reference.

Vocabulary used throughout:

• VRAM — dedicated GPU memory. Models must fit here for fast inference.
• Quantization — compressing weights to fewer bits (Q4, Q5, Q8). Lower bits = smaller, less memory, lower quality.
• Context window — tokens (roughly word-pieces) the model reads at once, prompt plus reply.
• Offloading — splitting a model between GPU and CPU/RAM when it won't fully fit in VRAM.

Out-of-Memory Errors

Symptoms: CUDA out of memory, ggml_metal_init: failed to allocate, a Killed process, or model requires more system memory than is available.

Estimate before pulling: weights ≈ (billions of params) × (bits ÷ 8) GB, plus 1-3 GB KV cache. A 7B model at Q4 needs ~4-5 GB.

# Smaller context cuts KV-cache memory (set in an Ollama session)
ollama run llama3.1:8b
/set parameter num_ctx 4096

# llama.cpp: offload only 20 layers to GPU, rest on CPU
./llama-cli -m model-q4_k_m.gguf -ngl 20 -c 4096 -p "Hello"

Very Slow Inference

Symptoms: a few tokens per second or worse, long pauses before the first token, fans roaring.

# Measure real throughput (tokens/sec) and load time
ollama run llama3.1:8b --verbose "Write one sentence about the sea."

Rule of thumb: once any layers offload to CPU, speed can fall 5-20x. Getting the whole model into VRAM is the biggest lever.

GPU Not Detected

Symptoms: inference uses 100% CPU, nvidia-smi shows no process, or logs say no compatible GPUs were discovered / no usable GPU found.

# Is the NVIDIA GPU visible, and which driver/CUDA version?
nvidia-smi

# Build llama.cpp with CUDA (rebuild clean if you flip backends)
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release

# Run Ollama in Docker with GPU access
docker run --gpus all -d -v ollama:/root/.ollama -p 11434:11434 ollama/ollama

Model Download Failures

Symptoms: pull stalls, Error: max retries exceeded, checksum/digest mismatch, or EOF partway through.

# Resume a stalled pull (re-run); or remove + re-pull clean
ollama pull mistral:7b-instruct
ollama rm mistral:7b-instruct && ollama pull mistral:7b-instruct

# Move model storage to a bigger disk, then restart
export OLLAMA_MODELS=/mnt/big-disk/ollama-models
ollama serve

Garbled or Repetitive Output

Symptoms: nonsense tokens, infinite repetition, broken formatting, ignored template, or mixed languages.

# Tame repetition and randomness (Ollama, then llama.cpp)
/set parameter temperature 0.7
/set parameter repeat_penalty 1.15
./llama-cli -m model-q4_k_m.gguf --temp 0.7 --repeat-penalty 1.15 -p "..."

If output is fine at short prompts but degrades on long ones, it's almost always context length.

Context-Length Errors

Symptoms: context length exceeded, input is too long, the model "forgets" the start of a long document, or answers an earlier turn.

# Set context window explicitly (Ollama, then llama.cpp -c)
ollama run qwen2.5:7b
/set parameter num_ctx 8192
./llama-cli -m model-q4_k_m.gguf -c 8192 -p "Summarise: ..."

Token budgeting: roughly 0.75 words per token in English. A 4,000-word document is ~5,300 tokens — an 8K window has room for the reply, a 4K window truncates it.

Still Stuck?

Back to the Local LLMs course.