Deploying Local AI Cluster with Ollama & Docker GPU Passthrough
In this companion guide to our @TechBlogTutorial video, we construct a scalable compute cluster for hosting LLMs (Llama 3, DeepSeek, Mistral) completely offline.
1. Verify NVIDIA GPU Container Runtime
Execute telemetry verification to confirm container GPU access:
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi2. Multi-Node Cluster Architecture
Below is our primary reverse proxy routing architecture diagram:
3. Deploying Ollama Service Stack
Run the containerized inference cluster daemon with automated restart policies:
docker compose -f docker-compose.ai.yml up -d --scale worker=2\\\yaml
# docker-compose.ai.yml
version: '3.8'
services:
ollama-primary:
image: ollama/ollama:latest
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
ports:
- "11434:11434"
volumes:
- /mnt/fast-nvme/models:/root/.ollama
\\\
Now test prompt generation via curl:
curl http://192.168.1.120:11434/api/generate -d '{\