Production Deployment

"Serving" means exposing a model over HTTP so other processes — your app, a teammate's laptop, a CI job — can send a prompt and get tokens back. Three engines cover almost every case:

The key idea: all three can speak the OpenAI Chat Completions format, so client code stays identical no matter which engine sits behind it. Prototype on Ollama and swap in vLLM for production without rewriting the calling app. Ollama's OpenAI-compatible layer covers /v1/chat/completions, /v1/completions, /v1/embeddings, and /v1/models.

Ollama server mode. Ollama already runs a server; you just need it to bind beyond localhost. It binds 127.0.0.1:11434 by default — change the bind address with OLLAMA_HOST.

export OLLAMA_HOST=0.0.0.0:11434
ollama serve
ollama pull llama3.1:8b   # in another terminal

Smoke-test it with curl http://localhost:11434/v1/chat/completions -d '{"model":"llama3.1:8b","messages":[{"role":"user","content":"say ready"}]}' — the exact OpenAI shape your apps will use.

vLLM (GPU). One command launches an OpenAI-compatible server on port 8000 by default. Use --tensor-parallel-size to split a large model across multiple GPUs.

pip install vllm
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --port 8000 --tensor-parallel-size 1 --max-model-len 8192

TGI (GPU, Docker). Hugging Face ships TGI as a container. Map host port 8080 to the container's port 80 and pass a Hugging Face model ID.

docker run --gpus all --shm-size 1g -p 8080:80 -v $PWD/data:/data \
  ghcr.io/huggingface/text-generation-inference --model-id teknium/OpenHermes-2.5-Mistral-7B

Because the API is OpenAI-shaped, the official openai SDK works against any of the three engines — point base_url at your server (use a non-empty key; local engines ignore it). The switch from a cloud provider to your own box is literally one line:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="local-not-checked")
resp = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Summarise RAG in one sentence."}],
    stream=True,
)
for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

A bare ollama serve on someone's laptop is not a deployment — it dies when the laptop sleeps and nobody can reproduce its state. Docker fixes both. Use the official image and a docker-compose.yml so the whole stack comes up with one command.

# docker-compose.yml
services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama-models:/root/.ollama   # persist downloaded weights
    environment:
      - OLLAMA_HOST=0.0.0.0:11434
      - OLLAMA_NUM_PARALLEL=4          # concurrent requests per model
      - OLLAMA_MAX_LOADED_MODELS=2     # models resident at once
    restart: unless-stopped
    # NVIDIA GPUs: add a deploy.resources block reserving the nvidia driver
volumes:
  ollama-models:

OLLAMA_MAX_LOADED_MODELS defaults to 3 times the GPU count (3 for CPU), and OLLAMA_NUM_PARALLEL defaults to 1 parallel request per model. Tune both to your Lesson 9 numbers, then docker compose up -d and docker compose exec ollama ollama pull llama3.1:8b to warm a model so the first real request is not slow.

A single instance is a single point of contention. When concurrent requests exceed what one instance handles smoothly, run several and put a reverse proxy in front. Nginx round-robins (or least_conn-balances) requests across replicas.

# nginx.conf
upstream llm_backends {
    least_conn;                  # route to the instance with fewest active conns
    server ollama-1:11434 max_fails=3 fail_timeout=30s;
    server ollama-2:11434 max_fails=3 fail_timeout=30s;
}
server {
    listen 80;
    location / {
        proxy_pass http://llm_backends;
        proxy_read_timeout 300s; # long generations must not time out
        proxy_buffering off;     # required so streamed tokens flow immediately
    }
}

Run the model containers (ollama-1, ollama-2) and the proxy on one Docker network:

docker network create llm-net
docker run -d --name ollama-1 --network llm-net -v ollama-models:/root/.ollama ollama/ollama
# repeat the line above for ollama-2, then start the load balancer:
docker run -d --name lb --network llm-net -p 80:80 \
  -v "$PWD/nginx.conf:/etc/nginx/conf.d/default.conf:ro" nginx:alpine

You cannot operate what you cannot see. Three layers matter: alive (health), doing OK (metrics), and what happened (logs). Add a healthcheck to the compose service that curls http://localhost:11434/api/version so Docker restarts a wedged instance automatically.

vLLM and TGI both expose a Prometheus /metrics endpoint out of the box (scrape metrics_path: /metrics). Track the numbers that actually predict a bad user experience:

For logs, use structured logging shipped to a central store (Loki, CloudWatch) so you can search across replicas rather than tailing each container by hand.

Now control access. Local engines ship with no auth — anyone who reaches the port runs your GPU for free and reads every prompt. Extend the Step 4 server block with TLS, a bearer-token check, and a per-client rate limit:

limit_req_zone $binary_remote_addr zone=llm:10m rate=10r/s;  # ~10 req/s per IP

# inside server { listen 443 ssl; ... } location / { ... }
if ($http_authorization != "Bearer CHANGE_ME") { return 401; }  # reject others
limit_req zone=llm burst=20 nodelay;

Access checklist for a self-hosted endpoint:

The decision is rarely about raw price per token; it is about utilisation. Self-hosting is a fixed cost (you pay for the GPU busy or idle); cloud APIs are variable (you pay per token). The crossover depends on how busy the hardware stays. Estimate it from your Lesson 9 numbers:

monthly_tokens = 500_000_000               # tokens generated per month
self_host = 2000 / 24 + 30                  # amortised GPU + power per month
cloud = (monthly_tokens / 1_000_000) * 0.50 # per 1M tokens, your provider's rate
print("Self-host wins" if self_host < cloud else "Cloud wins")