>_TECH BLOG TUTORIAL

CLUSTER: ONLINE

BACK TO LOCAL AI & GPU TRACK

LOCAL AI CLUSTER•2026-10-14•1 MIN READ

Deploying Local AI Cluster with Ollama & Docker GPU Passthrough

High-performance inference clustering with NVIDIA GPU pass-through on Ubuntu 24.04 LTS.

Deploying Local AI Cluster with Ollama & Docker GPU Passthrough

In this companion guide to our **@TechBlogTutorial** video, we construct a scalable compute cluster for hosting LLMs (Llama 3, DeepSeek, Mistral) completely offline.

Hardware Requisite

Ensure NVIDIA CUDA drivers (v550+) and the NVIDIA Container Toolkit are installed prior to starting the Docker swarm.

1. Verify NVIDIA GPU Container Runtime

Execute telemetry verification to confirm container GPU access:

NVIDIA Core Driver Verification

Ubuntu 24.04

$docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

2. Multi-Node Cluster Architecture

Below is our primary reverse proxy routing architecture diagram:

Inference Routing Mesh

SVG // HIGH-AVAILABILITY TOPOLOGY

HAProxy VIP10.10.10.100

── 10GbE ──▶

Ollama-01RTX 3090 #1

Ollama-02RTX 3090 #2

── NVMe-oF ──▶

ZFS PoolRAIDZ2 8TB

3. Deploying Ollama Service Stack

Run the containerized inference cluster daemon with automated restart policies:

Cluster Startup

Debian/Ubuntu

$docker compose -f docker-compose.ai.yml up -d --scale worker=2

\`\`\`yaml # docker-compose.ai.yml version: '3.8' services: ollama-primary: image: ollama/ollama:latest deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] ports: - "11434:11434" volumes: - /mnt/fast-nvme/models:/root/.ollama \`\`\`

Now test prompt generation via curl:

Local Inference Test

bash

$curl http://192.168.1.120:11434/api/generate -d '{\

Deploying Local AI Cluster with Ollama & Docker GPU Passthrough | TechNHIT Local AI | TechNHIT