Artificial intelligence

The Complete Self-Hosted AI Stack for 2026

Building a local AI server is more than installing Ollama. This guide builds a full production AI stack with Open WebUI, LiteLLM, Qdrant, Traefik, and Langfuse — what each component does, why it's needed, and three complete Docker Compose configurations for beginners, power users, and small teams.

Traven

31 Jul 2026 • 20 min read

Most guides to self-hosted AI stop at ollama run llama3. You get the model running, you have a terminal prompt, and the guide says congratulations. Then you try to use it seriously — share it with a teammate, hook it into your editor, run two models at once, or figure out why responses started getting weird — and you discover that Ollama alone is not a stack. It's one layer of a stack, and you've been building on bare concrete.

A real local AI setup looks like the diagram above. Eight components working together, each solving a specific problem that no other component solves. Remove any one and you either lose a capability or you're running something that will fail quietly in ways that are hard to debug.

This guide builds the full stack in three tiers. Tier 1 is the beginner setup: Ollama, Open WebUI, and Traefik. Thirty minutes, three containers, works immediately. Tier 2 adds LiteLLM as a proxy layer, Qdrant as a vector store for RAG, and Langfuse for observability. Tier 3 adds monitoring, multi-user authentication, and the configuration choices that make a shared team deployment trustworthy. Every tier ships with a complete, copy-pasteable Docker Compose file.

Before you start: this guide assumes Docker and Docker Compose are installed. If they're not, the What is Docker primer covers the concepts, and the install section of the AI agent stack guide has the full installation steps for Ubuntu and Debian.

What Does Each Component Actually Do?

The architecture diagram shows the flow. Here's why every component exists and what breaks without it.

What Is Ollama and Why Is It the Foundation?

Ollama wraps llama.cpp — the high-performance C++ inference library — behind a clean REST API and a CLI. It handles GPU memory allocation, model layer distribution across devices, automatic quantization selection, and model storage as content-addressed layers. You pull a model the same way you pull a Docker image. Switch models in one command. The API is OpenAI-compatible, which means anything built for the OpenAI API works with Ollama without modification.

What Ollama does not handle: multi-user auth, spend tracking, request routing across multiple model instances, fallback to cloud providers when the local model is slow or unavailable, and observability. Those gaps are why the rest of the stack exists.

The local LLM complete guide covers Ollama installation, hardware requirements, model selection, and quantization in depth. Read it first if you're new to running local models. This guide treats Ollama as a configured dependency and focuses on everything that sits on top of it.

What Is Open WebUI and Why Not Just Use the Terminal?

Open WebUI is a self-hosted chat interface that connects to Ollama (and any other OpenAI-compatible backend). It gives you persistent conversation history, model switching without restarting anything, file uploads, image generation integration, Retrieval Augmented Generation (RAG) from your own documents, multi-user accounts with role-based access, and a web interface that works from any browser on your network.

The terminal interface disappears when you close it. Open WebUI persists. Every conversation is stored. You can search your chat history. You can share a link to a specific chat with a teammate. You can upload a PDF and ask questions about it. The terminal is for testing. Open WebUI is for working.

What Is LiteLLM and Why Does It Go Between Open WebUI and Ollama?

LiteLLM exposes a unified OpenAI-compatible API in front of 100+ LLM providers, handles load balancing, fallback routing, virtual key management, rate limiting, and spend tracking — all in a single proxy container.

Without LiteLLM, Open WebUI talks directly to Ollama. That works for a single user with one model. It breaks the moment you want to:

Route slow or complex requests to a cloud model while fast/simple requests stay local
Add a second Ollama instance (second GPU, second machine) and split load between them
Set per-user or per-team spending limits across all models
Give different users different model access without giving them all direct API access
Log every request centrally for debugging and cost analysis
Switch from Ollama to vLLM or a different inference backend without changing Open WebUI's config
LiteLLM handles all of this. Load balancing is built in: you configure round-robin, least-busy, usage-based, or latency-based routing across multiple API keys or providers. Fallbacks are declarative: if the primary model fails, LiteLLM automatically retries the next provider in the list with no changes to your application code.

What Is a Vector Database and Why Does RAG Need One?

RAG (Retrieval Augmented Generation) is the technique that lets an LLM answer questions about documents it wasn't trained on — your internal docs, your codebase, your notes, your company's knowledge base. The process: ingest documents, chunk them into pieces, generate embedding vectors for each chunk, store those vectors in a vector database, then at query time embed the user's question and search for the most semantically similar chunks to inject as context.

A vector database stores these embedding vectors and makes similarity search fast. Qdrant is the right choice for this stack: purpose-built for vector search, has a REST API, supports payload filtering (search by semantic similarity AND filter by metadata), and handles collections that grow to millions of vectors without restructuring. Open WebUI's built-in RAG uses Qdrant when configured. So does every other serious RAG pipeline.

What Does Traefik Do and Why Not Just Expose Ports?

Traefik is a reverse proxy that sits in front of all your services. It handles TLS termination (HTTPS certificates from Let's Encrypt, automatically renewed), routes incoming requests to the right container, and provides authentication middleware. Without it, you'd either expose Open WebUI and LiteLLM on raw HTTP, manage certificates manually, or set up Nginx with static configuration files for each service.

Traefik reads Docker labels. You add four labels to a service definition and it appears on a subdomain with a valid HTTPS certificate. No Nginx config files. No manual certbot renew cron jobs. No static upstream blocks that break when container IPs change. Adding a new service is four label lines.

What Does Langfuse Do and Why Do You Need Observability?

Langfuse records every LLM interaction as a structured trace: the full prompt, the model and parameters, every tool call and its result, the response, token counts, and latency at each step. Without it, when your agent gives a wrong answer, you have no systematic way to find out why.

LiteLLM integrates with Langfuse natively — set two environment variables and every request flowing through the proxy is automatically traced. You get dashboards showing which models are being used, which queries are failing, where latency is coming from, and what each model run actually cost. This is the difference between running an LLM stack and operating one.

How Do You Choose Between the Three Tiers?

Before writing any configuration, match your situation to the right tier.

Tier 1: Beginner — Solo developer or single user. You want local inference with a proper web UI instead of a terminal. You don't need multi-user access, cost tracking, or production-grade routing. You have one machine with one GPU. Setup time: 30 minutes.

Tier 2: Power User — You've outgrown Tier 1 because you want to access the stack from multiple devices, you want RAG on your own documents, you want LiteLLM's routing so you can fall back to a cloud model when the local model is too slow for a task, or you want to understand what your stack is actually doing. Setup time: 90 minutes.

Tier 3: Small Team — Two to fifteen people sharing a stack. You need user accounts, shared document collections, per-user spending limits, monitoring that tells you when something breaks, and the security configuration that makes it safe to put this behind a domain name and share it with people who aren't developers. Setup time: half a day.

Tier 1: The Beginner Stack

Three containers. Ollama for inference, Open WebUI for the chat interface, Traefik for HTTPS routing. This is the minimum setup that's actually comfortable to use daily.

Prerequisites

A server or desktop with Docker installed
A domain name with DNS access (for Traefik's Let's Encrypt certificates)
Ollama installed on the host with at least one model pulled
If you're running on the same machine where Ollama is already installed (not in Docker), Traefik and Open WebUI can connect to it at http://host.docker.internal:11434.

Directory Setup

mkdir -p /opt/ai-stack/traefik/certs
touch /opt/ai-stack/traefik/certs/acme.json
chmod 600 /opt/ai-stack/traefik/certs/acme.json
 
docker network create ai-proxy

Tier 1 Docker Compose

# /opt/ai-stack/docker-compose.yml
version: "3.9"
 
networks:
  ai-proxy:
    external: true
 
services:
 
  traefik:
    image: traefik:v3.1
    container_name: ai-traefik
    restart: unless-stopped
    networks:
      - ai-proxy
    ports:
      - "80:80"
      - "443:443"
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=ai-proxy"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      - "[email protected]"
      - "--certificatesresolvers.le.acme.storage=/certs/acme.json"
      - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web"
      - "--log.level=WARN"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /opt/ai-stack/traefik/certs:/certs
 
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: ai-open-webui
    restart: unless-stopped
    networks:
      - ai-proxy
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - open_webui_data:/app/backend/data
    environment:
      OLLAMA_BASE_URL: http://host.docker.internal:11434
      WEBUI_SECRET_KEY: "CHANGE_THIS_generate_with_openssl_rand_hex_32"
      # Disable signup after creating your first account
      ENABLE_SIGNUP: "true"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.open-webui.rule=Host(`ai.yourdomain.com`)"
      - "traefik.http.routers.open-webui.entrypoints=websecure"
      - "traefik.http.routers.open-webui.tls.certresolver=le"
      - "traefik.http.services.open-webui.loadbalancer.server.port=8080"
 
volumes:
  open_webui_data:

Start it:

cd /opt/ai-stack
docker compose up -d
 
# Watch for certificate issuance
docker logs ai-traefik -f
# Look for: "Retrieving certificate" and "Certificate obtained successfully"

Navigate to https://ai.yourdomain.com. Create your account on first load. Set ENABLE_SIGNUP: "false" in the compose file immediately after and restart the container — otherwise anyone who can reach your domain can create an account.

# Disable signups
sed -i 's/ENABLE_SIGNUP: "true"/ENABLE_SIGNUP: "false"/' docker-compose.yml
docker compose up -d

We recommend reading How to Run a Local LLM on Your PC in 2026 (Complete Guide) to continue reading our selection of content. The Tier 1 stack you just built needs Ollama running on the host with the right model pulled and the right GPU configuration. That guide covers every hardware tier, model selection, quantization settings, and the Ollama setup that powers everything above it.

https://coderoasis.com/run-local-llm-on-your-pc-complete-guide-2026/

Tier 2: The Power User Stack

Tier 2 adds LiteLLM as the proxy layer between Open WebUI and your inference backends, Qdrant as the vector store for RAG, and Langfuse for tracing every request. This is where the stack becomes genuinely powerful.

The Environment File

Create /opt/ai-stack/.env. Never commit this to version control.

# Generate secrets:
# openssl rand -hex 32  (for most fields)
# openssl rand -base64 32  (for Langfuse fields)
 
# ── Open WebUI ──────────────────────────────────────────────
WEBUI_SECRET_KEY=generate_with_openssl_rand_hex_32
 
# ── LiteLLM ─────────────────────────────────────────────────
LITELLM_MASTER_KEY=sk-generate_with_openssl_rand_hex_32
LITELLM_DB_PASSWORD=generate_strong_password
 
# Cloud providers (optional -- leave empty to use local only)
OPENAI_API_KEY=
ANTHROPIC_API_KEY=
 
# ── Qdrant ──────────────────────────────────────────────────
QDRANT_API_KEY=generate_with_openssl_rand_hex_32
 
# ── Langfuse ────────────────────────────────────────────────
LANGFUSE_SECRET_KEY=generate_with_openssl_rand_base64_32
LANGFUSE_NEXTAUTH_SECRET=generate_with_openssl_rand_base64_32
LANGFUSE_SALT=generate_with_openssl_rand_hex_64
LANGFUSE_ENCRYPTION_KEY=generate_with_openssl_rand_hex_64
LANGFUSE_DB_PASSWORD=generate_strong_password
CLICKHOUSE_PASSWORD=generate_strong_password

Tier 2 Docker Compose

# /opt/ai-stack/docker-compose.yml
version: "3.9"
 
networks:
  ai-proxy:
    external: true
  ai-internal:
    driver: bridge
 
volumes:
  open_webui_data:
  qdrant_data:
  litellm_postgres_data:
  langfuse_postgres_data:
  clickhouse_data:
  clickhouse_logs:
 
services:
 
  # ── Traefik (same as Tier 1) ─────────────────────────────
  traefik:
    image: traefik:v3.1
    container_name: ai-traefik
    restart: unless-stopped
    networks:
      - ai-proxy
    ports:
      - "80:80"
      - "443:443"
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=ai-proxy"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.websecure.address=:443"
      - "[email protected]"
      - "--certificatesresolvers.le.acme.storage=/certs/acme.json"
      - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web"
      - "--log.level=WARN"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /opt/ai-stack/traefik/certs:/certs
 
  # ── PostgreSQL for LiteLLM ────────────────────────────────
  litellm-postgres:
    image: postgres:16-alpine
    container_name: ai-litellm-postgres
    restart: unless-stopped
    networks:
      - ai-internal
    environment:
      POSTGRES_DB: litellm
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: ${LITELLM_DB_PASSWORD}
    volumes:
      - litellm_postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm -d litellm"]
      interval: 10s
      retries: 5
 
  # ── Redis for LiteLLM caching ────────────────────────────
  redis:
    image: redis:7-alpine
    container_name: ai-redis
    restart: unless-stopped
    networks:
      - ai-internal
    command: redis-server --maxmemory 512mb --maxmemory-policy allkeys-lru
 
  # ── LiteLLM Proxy ─────────────────────────────────────────
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    container_name: ai-litellm
    restart: unless-stopped
    networks:
      - ai-internal
      - ai-proxy
    extra_hosts:
      - "host.docker.internal:host-gateway"
    depends_on:
      litellm-postgres:
        condition: service_healthy
    volumes:
      - /opt/ai-stack/litellm/config.yaml:/app/config.yaml:ro
    environment:
      DATABASE_URL: postgresql://litellm:${LITELLM_DB_PASSWORD}@litellm-postgres:5432/litellm
      LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
      REDIS_HOST: redis
      REDIS_PORT: 6379
      # Cloud provider keys (passed through to backends when needed)
      OPENAI_API_KEY: ${OPENAI_API_KEY}
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
      # Langfuse tracing
      LANGFUSE_PUBLIC_KEY: "pk-lf-..."        # Fill after Langfuse is running
      LANGFUSE_SECRET_KEY: ${LANGFUSE_SECRET_KEY}
      LANGFUSE_HOST: http://langfuse-web:3000
    command: ["--config", "/app/config.yaml", "--port", "4000", "--detailed_debug"]
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.litellm.rule=Host(`llm.yourdomain.com`)"
      - "traefik.http.routers.litellm.entrypoints=websecure"
      - "traefik.http.routers.litellm.tls.certresolver=le"
      - "traefik.http.services.litellm.loadbalancer.server.port=4000"
 
  # ── Open WebUI ────────────────────────────────────────────
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: ai-open-webui
    restart: unless-stopped
    networks:
      - ai-internal
      - ai-proxy
    depends_on:
      - litellm
      - qdrant
    volumes:
      - open_webui_data:/app/backend/data
    environment:
      # Point at LiteLLM, not Ollama directly
      OPENAI_API_KEY: ${LITELLM_MASTER_KEY}
      OPENAI_API_BASE_URL: http://litellm:4000
      WEBUI_SECRET_KEY: ${WEBUI_SECRET_KEY}
      ENABLE_SIGNUP: "false"
      # RAG configuration
      RAG_EMBEDDING_ENGINE: ollama
      RAG_OLLAMA_BASE_URL: http://host.docker.internal:11434
      QDRANT_URI: http://qdrant:6333
      QDRANT_API_KEY: ${QDRANT_API_KEY}
      RAG_EMBEDDING_MODEL: nomic-embed-text
      CHUNK_SIZE: 1000
      CHUNK_OVERLAP: 100
    extra_hosts:
      - "host.docker.internal:host-gateway"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.open-webui.rule=Host(`ai.yourdomain.com`)"
      - "traefik.http.routers.open-webui.entrypoints=websecure"
      - "traefik.http.routers.open-webui.tls.certresolver=le"
      - "traefik.http.services.open-webui.loadbalancer.server.port=8080"
 
  # ── Qdrant ────────────────────────────────────────────────
  qdrant:
    image: qdrant/qdrant:latest
    container_name: ai-qdrant
    restart: unless-stopped
    networks:
      - ai-internal
    environment:
      QDRANT__SERVICE__API_KEY: ${QDRANT_API_KEY}
    volumes:
      - qdrant_data:/qdrant/storage
    # Internal only -- no external exposure
 
  # ── Langfuse PostgreSQL ───────────────────────────────────
  langfuse-postgres:
    image: postgres:16-alpine
    container_name: ai-langfuse-postgres
    restart: unless-stopped
    networks:
      - ai-internal
    environment:
      POSTGRES_DB: langfuse
      POSTGRES_USER: langfuse
      POSTGRES_PASSWORD: ${LANGFUSE_DB_PASSWORD}
    volumes:
      - langfuse_postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U langfuse -d langfuse"]
      interval: 10s
      retries: 5
 
  # ── ClickHouse (Langfuse trace storage) ──────────────────
  clickhouse:
    image: clickhouse/clickhouse-server:24.8-alpine
    container_name: ai-clickhouse
    restart: unless-stopped
    networks:
      - ai-internal
    environment:
      CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD}
    volumes:
      - clickhouse_data:/var/lib/clickhouse
      - clickhouse_logs:/var/log/clickhouse-server
    ulimits:
      nofile: { soft: 262144, hard: 262144 }
    healthcheck:
      test: ["CMD", "clickhouse-client", "--password", "${CLICKHOUSE_PASSWORD}", "--query", "SELECT 1"]
      interval: 10s
      retries: 5
 
  # ── Langfuse Web ──────────────────────────────────────────
  langfuse-web:
    image: langfuse/langfuse:3
    container_name: ai-langfuse-web
    restart: unless-stopped
    networks:
      - ai-internal
      - ai-proxy
    depends_on:
      langfuse-postgres:
        condition: service_healthy
      clickhouse:
        condition: service_healthy
    environment:
      DATABASE_URL: postgresql://langfuse:${LANGFUSE_DB_PASSWORD}@langfuse-postgres:5432/langfuse
      CLICKHOUSE_URL: http://default:${CLICKHOUSE_PASSWORD}@clickhouse:8123
      NEXTAUTH_URL: https://traces.yourdomain.com
      NEXTAUTH_SECRET: ${LANGFUSE_NEXTAUTH_SECRET}
      SALT: ${LANGFUSE_SALT}
      ENCRYPTION_KEY: ${LANGFUSE_ENCRYPTION_KEY}
      TELEMETRY_ENABLED: "false"
      AUTH_DISABLE_SIGNUP: "false"  # Disable after first user
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.langfuse.rule=Host(`traces.yourdomain.com`)"
      - "traefik.http.routers.langfuse.entrypoints=websecure"
      - "traefik.http.routers.langfuse.tls.certresolver=le"
      - "traefik.http.services.langfuse.loadbalancer.server.port=3000"
 
  # ── Langfuse Worker ───────────────────────────────────────
  langfuse-worker:
    image: langfuse/langfuse-worker:3
    container_name: ai-langfuse-worker
    restart: unless-stopped
    networks:
      - ai-internal
    depends_on:
      langfuse-postgres:
        condition: service_healthy
      clickhouse:
        condition: service_healthy
    environment:
      DATABASE_URL: postgresql://langfuse:${LANGFUSE_DB_PASSWORD}@langfuse-postgres:5432/langfuse
      CLICKHOUSE_URL: http://default:${CLICKHOUSE_PASSWORD}@clickhouse:8123
      SALT: ${LANGFUSE_SALT}
      ENCRYPTION_KEY: ${LANGFUSE_ENCRYPTION_KEY}
      TELEMETRY_ENABLED: "false"

LiteLLM Configuration File

The config.yaml drives LiteLLM's routing logic. Create /opt/ai-stack/litellm/config.yaml:

# /opt/ai-stack/litellm/config.yaml
 
model_list:
  # Primary: local Ollama models
  - model_name: qwen2.5:14b
    litellm_params:
      model: ollama/qwen2.5:14b
      api_base: http://host.docker.internal:11434
      stream: true
 
  - model_name: qwen2.5-coder:14b
    litellm_params:
      model: ollama/qwen2.5-coder:14b
      api_base: http://host.docker.internal:11434
 
  - model_name: deepseek-r1:14b
    litellm_params:
      model: ollama/deepseek-r1:14b
      api_base: http://host.docker.internal:11434
 
  # Embedding model (for RAG)
  - model_name: nomic-embed-text
    litellm_params:
      model: ollama/nomic-embed-text
      api_base: http://host.docker.internal:11434
 
  # Cloud fallbacks (only active when API keys are set)
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
 
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
 
  # Router group: smart auto-routing
  - model_name: auto
    model_info:
      mode: embeddings
    litellm_params:
      model: ollama/qwen2.5:14b
      api_base: http://host.docker.internal:11434
 
router_settings:
  # Route fast requests local, slow/complex requests to cloud
  routing_strategy: latency-based-routing
  fallbacks:
    - {"qwen2.5:14b": ["gpt-4o"]}
    - {"qwen2.5-coder:14b": ["claude-sonnet"]}
 
litellm_settings:
  # Semantic caching -- never pay twice for the same prompt
  cache: true
  cache_params:
    type: redis
    host: redis
    port: 6379
    ttl: 600  # Cache for 10 minutes
 
  # Send all traces to Langfuse automatically
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]
 
  # Drop params that local models don't support
  drop_params: true
 
  # Retry on failure before fallback
  num_retries: 2
  request_timeout: 120
 
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
 
  # Virtual key management (requires PostgreSQL)
  store_model_in_db: true
 
  # UI for the LiteLLM admin panel
  ui_access_mode: all
 
  allow_user_auth: true

What the LiteLLM Config Actually Does

routing_strategy: latency-based-routing tracks response time per model and routes to the fastest available option. If your local Qwen2.5 14B is crunching a previous request and latency spikes, LiteLLM shifts the next request to a different instance or cloud fallback. Invisible to the user.

fallbacks defines what happens when a model fails entirely (not just slow — actually errors). If qwen2.5:14b returns a 503, the request automatically retries against gpt-4o. This is the config that makes your stack resilient to inference failures. The user sees a slightly slower response; they never see an error.

cache: true with Redis stores semantic embeddings of recent queries. If two users ask the same question within ten minutes, the second answer comes from the cache in milliseconds. The model never gets called for the second request. For common questions in a team context, this can cut your inference load by 30-50%.

drop_params: true silently removes parameters that local Ollama models don't support (like response_format: {"type": "json_schema"} which is an OpenAI-only feature). Without this, requests that work fine with cloud models fail with local ones because Open WebUI sometimes sends cloud-specific parameters.

Setting Up Langfuse After First Boot

Navigate to https://traces.yourdomain.com and create the first admin account. Then:

# Get your API keys from Langfuse's UI: Settings → API Keys
# Copy the public key (pk-lf-...) and secret key (sk-lf-...)
 
# Update your .env
echo "LANGFUSE_PUBLIC_KEY=pk-lf-yourkeyhere" >> /opt/ai-stack/.env
 
# Update the LiteLLM service environment in docker-compose.yml
# Replace "pk-lf-..." with your actual public key
 
# Restart LiteLLM to pick up the new config
docker compose restart litellm

Within minutes, every request flowing through LiteLLM appears as a trace in Langfuse. Click any trace to see the full prompt, the model response, token counts, and latency breakdown. If a response was wrong, you'll see exactly what the model received and returned.

We recommend reading Build Your Own AI Agent Stack: The Complete Self-Hosted Guide for 2026 to continue reading our selection of content. The Tier 2 stack you have now is the inference and interface layer. That guide builds the automation and orchestration layer on top of it — n8n workflows, Flowise agent builders, and multi-step RAG pipelines that turn your AI stack into something that actually does work for you.

Tier 3: The Small Team Stack

Tier 3 adds three things: infrastructure monitoring with Prometheus and Grafana, production-grade access control with Open WebUI's organization and group permissions, and the security hardening that makes this safe to run for a team. The Docker Compose additions below build on the Tier 2 file.

Adding Prometheus and Grafana

LiteLLM exposes Prometheus metrics at /metrics when configured. Open WebUI doesn't have native metrics, but Node Exporter covers the host.

# Add to docker-compose.yml
 
  prometheus:
    image: prom/prometheus:v2.52.0
    container_name: ai-prometheus
    restart: unless-stopped
    networks:
      - ai-internal
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=30d"
      - "--web.enable-lifecycle"
    volumes:
      - /opt/ai-stack/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    # Internal only -- access via Grafana
 
  grafana:
    image: grafana/grafana:11.0.0
    container_name: ai-grafana
    restart: unless-stopped
    networks:
      - ai-internal
      - ai-proxy
    depends_on:
      - prometheus
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD}
      GF_ANALYTICS_REPORTING_ENABLED: "false"
      GF_SERVER_ROOT_URL: "https://grafana.yourdomain.com"
    volumes:
      - grafana_data:/var/lib/grafana
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.grafana.rule=Host(`grafana.yourdomain.com`)"
      - "traefik.http.routers.grafana.entrypoints=websecure"
      - "traefik.http.routers.grafana.tls.certresolver=le"
      - "traefik.http.services.grafana.loadbalancer.server.port=3000"

Create /opt/ai-stack/prometheus/prometheus.yml:

global:
  scrape_interval: 15s
 
scrape_configs:
  - job_name: 'litellm'
    static_configs:
      - targets: ['litellm:4000']
    metrics_path: /metrics
 
  - job_name: 'qdrant'
    static_configs:
      - targets: ['qdrant:6333']
    metrics_path: /metrics
 
  - job_name: 'node'
    static_configs:
      - targets: ['host.docker.internal:9100']

The full Prometheus and Grafana setup — alert rules, dashboard configuration, and PromQL for the queries that actually matter — is in the Prometheus and Grafana complete guide. The setup above wires them in. That guide fills them with content.

Team Access Control in Open WebUI

Open WebUI's permission model works at three levels: global defaults, workspaces, and individual user roles. For a small team:

Global settings (Admin Panel → Settings):

Default user role: User (not Admin)
Enable model file uploads: Admin only
Enable community sharing: Disabled
Chat export: Enabled for all
Document upload for RAG: Enabled for all

User roles:

Admin: Full access including model management, user management, and system settings. One or two people.
User: Can chat, use RAG, access models permitted by the admin. The default for everyone else.
Pending: New signups before approval. Use this if you want to vet users before they get access.
Model filtering: In Admin Panel → Models, you can restrict which models are visible to non-admin users. If you have cloud fallback models configured in LiteLLM but don't want users to accidentally run expensive GPT-4o queries, hide those models from the UI for regular users.

Workspaces (Admin Panel → Workspaces): Group users and share document collections within a workspace without exposing those documents to everyone. A team of ten with two separate projects can have separate document namespaces without separate Open WebUI instances.

Securing Remote Access

For team access from outside the local network, you have two options. The simpler one: Tailscale. The more powerful one: Teleport.

Tailscale: Every team member installs Tailscale, joins the same tailnet, and connects to Open WebUI via the internal Tailscale IP. No public-facing domain needed. Works in minutes. Fine for small teams with technical members.

Teleport: Puts every service behind hardware-MFA-required, zero-trust access control. Every access attempt is logged, audited, and tied to a specific user identity. Short-lived certificates instead of passwords. The Teleport self-hosted guide covers the full setup. At CoderOasis, we proxy Open WebUI through Teleport rather than exposing it directly — the access log alone is worth the setup time.

How Do You Tune Performance Across the Stack?

Ollama: VRAM Allocation and Context Windows

The single biggest performance lever is keeping your model entirely in VRAM. A model that spills from VRAM into system RAM runs at 20-30% of its VRAM-only speed.

# Check what's loaded and where
ollama ps
 
# Output:
# NAME               ID        SIZE    PROCESSOR    UNTIL
# qwen2.5:14b        ...       9.1 GB  100% GPU     4 min
#                                       ^^^^^^^^^^^
# "100% GPU" = entirely in VRAM. Good.
# "76%/24% CPU" = spilling. Load a smaller model.

Context window size directly affects VRAM usage. The default is 4096 tokens. Setting it to 32768 uses roughly 50% more VRAM for the same model. Set context size explicitly rather than relying on the default:

# Set context size for a specific model via Ollama's REST API
curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5:14b",
  "options": {"num_ctx": 8192}
}'

Or configure it in Open WebUI's model settings per-model.

LiteLLM: Connection Pooling and Worker Counts

LiteLLM defaults are conservative. For a team deployment under regular load:

# In your litellm config.yaml
litellm_settings:
  # Increase concurrent request handling
  max_parallel_requests: 100
 
  # Aggressive timeout to fail fast to fallback
  request_timeout: 60
 
  # Keep connections to Ollama warm
  num_retries: 1        # Try once, fall back quickly

# In docker-compose.yml, LiteLLM service
command: >
  --config /app/config.yaml
  --port 4000
  --workers 4           # One worker per CPU core available to the container

Qdrant: Collection Optimization

Default Qdrant settings work for small collections. For a knowledge base with 100,000+ chunks, enable HNSW indexing optimization:

# Update the collection configuration after initial creation
curl -X PATCH "http://localhost:6333/collections/your_collection" \
  -H "Content-Type: application/json" \
  -H "api-key: ${QDRANT_API_KEY}" \
  -d '{
    "optimizers_config": {
      "default_segment_number": 4,
      "indexing_threshold": 20000
    },
    "hnsw_config": {
      "m": 16,
      "ef_construct": 100,
      "full_scan_threshold": 10000
    }
  }'

The hnsw_config settings trade index build time for query speed. Higher m and ef_construct values give better recall accuracy at the cost of more memory and slower index builds. For a static knowledge base, build the index overnight and query it fast during the day. For a frequently updated collection, use the defaults.

Open WebUI: RAG Chunk Tuning

The default chunk size (1000 tokens) and overlap (100 tokens) work for most prose documents. For technical documentation with lots of code:

CHUNK_SIZE: 1500
CHUNK_OVERLAP: 200

For short-form documents (emails, Slack messages, notes):

CHUNK_SIZE: 500
CHUNK_OVERLAP: 50

The right chunk size is the one where the model's retrieved context contains the complete answer to a typical query without including irrelevant surrounding text. Test by asking questions you know the answer to and checking whether the retrieved chunks include the right information.

What Security Mistakes Does Every Stack Make at First?

Exposing Services That Should Be Internal

The most common mistake: adding Traefik labels to services that have no reason to be publicly accessible. Qdrant, Redis, LiteLLM's internal metrics endpoint, and Langfuse's worker process do not need Traefik routing. If they have a label, they're reachable.

Correct pattern: services get Traefik labels only if humans need direct browser access. Everything else stays on ai-internal with no external routing.

# Audit which services are exposed
docker inspect $(docker compose ps -q) | \
  jq '.[] | select(.Labels."traefik.enable"=="true") | .Name'
 
# Should only show: open-webui, langfuse-web, grafana
# Should NOT show: qdrant, redis, litellm-postgres, clickhouse

LiteLLM Master Key Exposure

The LITELLM_MASTER_KEY gives full admin access to LiteLLM's API. Never put it in the OPENAI_API_KEY field of your IDE or code. Create virtual keys for different use cases:

# Create a virtual key with budget limit for a specific user
curl -X POST "https://llm.yourdomain.com/key/generate" \
  -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["qwen2.5:14b", "qwen2.5-coder:14b"],
    "max_budget": 10.0,
    "budget_duration": "30d",
    "metadata": {"user": "alice", "team": "engineering"},
    "aliases": {"qwen2.5:14b": "local-main"}
  }'

This returns a virtual key (sk-...) that Alice can use in her IDE. It can only access the two listed models, has a $10 monthly budget cap, and her usage is tracked separately in the Langfuse dashboard. The master key stays in your .env and never touches a developer's machine.

Open WebUI's session cookies default to non-secure settings in some configurations. Verify in the admin panel that WEBUI_SESSION_COOKIE_SECURE=true is set when serving over HTTPS. This prevents session cookie theft over non-HTTPS connections.

Qdrant Without Authentication

Qdrant's default configuration runs with no authentication. The QDRANT__SERVICE__API_KEY environment variable enables API key validation. Without it, anyone who can reach the Qdrant port can read, write, and delete every collection. The ai-internal network isolation provides one layer of protection. The API key provides the second.

How Do You Upgrade Components Without Downtime?

The stack separates concerns cleanly, which means most components can be upgraded independently.

Open WebUI (ships frequently, usually safe to update):

docker compose pull open-webui
docker compose up -d --no-deps open-webui
# --no-deps prevents restarting unrelated services

LiteLLM (check release notes before upgrading -- config format changes occasionally):

# Test the new config format in a separate container first
docker run --rm \
  -v /opt/ai-stack/litellm/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.91.0-stable \
  --config /app/config.yaml --test_connection
 
# If it passes, upgrade
docker compose pull litellm
docker compose up -d --no-deps litellm

Langfuse (requires coordinated update of web + worker):

docker compose pull langfuse-web langfuse-worker
docker compose up -d --no-deps langfuse-web langfuse-worker
# Both use the same image version -- always update together

Qdrant (data format changes happen between major versions -- read the changelog):

# Back up first
docker exec ai-qdrant tar -czf /tmp/qdrant-backup.tar.gz /qdrant/storage
docker cp ai-qdrant:/tmp/qdrant-backup.tar.gz /opt/ai-stack/backups/
 
# Then upgrade
docker compose pull qdrant
docker compose up -d --no-deps qdrant

How Do You Back Up the Stack?

Each component has data that matters:

Component	Data Location	Backup Priority
Open WebUI	`open_webui_data` Docker volume	High (chat history, user accounts)
LiteLLM PostgreSQL	`litellm_postgres_data` volume	Medium (virtual keys, spend data)
Qdrant	`qdrant_data` volume	High if RAG knowledge base has custom documents
Langfuse PostgreSQL	`langfuse_postgres_data` volume	Medium (project config, user accounts)
ClickHouse	`clickhouse_data` volume	Low (traces are re-generated; can accept loss)

Here is the backup script we created.

#!/bin/bash
# /opt/ai-stack/backup.sh
 
BACKUP_DATE=$(date +%Y%m%d-%H%M)
BACKUP_DIR="/opt/backups/ai-stack/${BACKUP_DATE}"
mkdir -p "${BACKUP_DIR}"
 
# Dump Open WebUI SQLite database
docker exec ai-open-webui sqlite3 /app/backend/data/webui.db \
  ".backup /tmp/webui-backup.db"
docker cp ai-open-webui:/tmp/webui-backup.db "${BACKUP_DIR}/webui.db"
 
# Dump LiteLLM PostgreSQL
docker exec ai-litellm-postgres pg_dump -U litellm litellm | \
  gzip > "${BACKUP_DIR}/litellm-postgres.sql.gz"
 
# Dump Langfuse PostgreSQL
docker exec ai-langfuse-postgres pg_dump -U langfuse langfuse | \
  gzip > "${BACKUP_DIR}/langfuse-postgres.sql.gz"
 
# Snapshot Qdrant collections
docker exec ai-qdrant curl -s \
  -X POST "http://localhost:6333/collections/knowledge_base/snapshots" \
  -H "api-key: ${QDRANT_API_KEY}"
 
# Compress everything
tar -czf "/opt/backups/ai-stack-${BACKUP_DATE}.tar.gz" "${BACKUP_DIR}/"
rm -rf "${BACKUP_DIR}"
 
# Sync offsite
rsync -avz "/opt/backups/ai-stack-${BACKUP_DATE}.tar.gz" \
  "backup-server:/backups/ai-stack/"
 
# Clean up local backups older than 14 days
find /opt/backups -name "ai-stack-*.tar.gz" -mtime +14 -delete
 
echo "Backup complete: ai-stack-${BACKUP_DATE}.tar.gz"

The stack described here is the one CoderOasis runs internally for research and content work. Tier 1 took thirty minutes. Tier 2 took an afternoon. Tier 3 accumulated over several months as the team grew and monitoring gaps became visible.

The key principle: each tier adds components that solve real problems the previous tier has. Add LiteLLM when you want routing and cost visibility, not before. Add Langfuse when you want to understand why responses are wrong. Add monitoring when you want to know before a user tells you that something broke.

For the self-hosting ecosystem beyond the AI stack — the file storage, password manager, email, and media server that round out a complete self-hosted setup — the CoderOasis self-hosted productivity guide covers that full picture. For every sysadmin and AI article across the site, the SysAdmin section and the AI section are the places to start.