Self-Hosting Prometheus and Grafana: Infrastructure Monitoring That Actually Works

Monitoring is the thing most self-hosters and small teams do wrong for longer than any other part of their infrastructure. Not because it's hard — it's not — but because it feels non-urgent until it is. Until you're staring at an alert from your hosting provider at 2AM saying your server ran out of disk space three hours ago, and you have no idea why or when it started filling up, because you have no historical data. Until a database query starts performing badly and you can't tell if it's been slow for a day or a week or since a deploy last Tuesday.

I've been in that position. Before CoderOasis ran Prometheus and Grafana, our "monitoring" was a cron job that emailed us if the web server stopped responding. That's not monitoring. That's knowing the server is down after it's already down. Real monitoring tells you that disk usage crossed 70% eleven days ago and has been climbing 3% per day — giving you over a week of warning before the problem becomes an incident.

This guide builds a complete, production-ready monitoring stack. Prometheus for metrics collection and storage, Grafana for visualization and dashboards, Alertmanager for intelligent alert routing, and Node Exporter plus additional exporters to instrument your actual infrastructure. By the end, you'll have the same setup we run at CoderOasis: per-host metrics, per-container metrics, database performance, Traefik request rates, automated alerts to wherever your team communicates, and dashboards that let you answer infrastructure questions at a glance.

This pairs with the self-hosted productivity stack guide — Grafana appears there as one of the services we proxy through Teleport. It also integrates with the Teleport setup — we access Grafana through Teleport's application proxy, so it's never directly exposed on the internet.

What You're Actually Building

Prometheus's Data Model

Prometheus works by pulling (scraping) metrics from targets on a regular interval rather than receiving pushed metrics. Every target you monitor exposes an HTTP endpoint (typically /metrics) that returns the current value of all its metrics in a text format. Prometheus fetches that endpoint every 15 seconds (or whatever you configure), stores the time series, and makes it queryable.

This pull model has important operational properties. Prometheus knows when a target goes down (it can't scrape it). Targets don't need to know where Prometheus is. Adding new targets doesn't require restarting or reconfiguring anything that's already being monitored.

Prometheus stores data as time series: streams of timestamped values identified by a metric name and a set of key-value labels.

node_disk_read_bytes_total{device="sda", instance="web-01:9100", job="node"} 1.234567e+09
node_disk_read_bytes_total{device="sdb", instance="web-01:9100", job="node"} 2.345678e+08

Same metric name (node_disk_read_bytes_total), different label sets (device="sda" vs device="sdb"), different time series. PromQL — the query language — lets you filter, aggregate, and compute over these time series.

The Four Components

Prometheus Server: The core. Scrapes targets, stores time series in a local TSDB (time series database), evaluates alerting rules, and exposes a query API. Everything else talks to or from the Prometheus server.

Exporters: Adapters that translate existing metrics from applications and systems into Prometheus's format. Node Exporter exposes Linux host metrics (CPU, memory, disk, network). cAdvisor exposes Docker container metrics. PostgreSQL Exporter exposes Postgres statistics. The exporter ecosystem covers essentially every technology you'd want to monitor.

Alertmanager: Handles the alerting pipeline. Prometheus evaluates alert rules and fires alerts to Alertmanager. Alertmanager deduplicates, groups, silences, and routes alerts to the right destination (Slack, email, PagerDuty, etc.). Keeping alerting in a separate component means you can maintain alert routing and silencing even when Prometheus itself is being restarted.

Grafana: Visualization layer. Connects to Prometheus as a data source and renders dashboards, graphs, and alerts based on PromQL queries. Grafana is where your team actually spends time — it's the interface for asking infrastructure questions.

Installation using Docker Compose for the Stack

We run the entire monitoring stack in Docker Compose alongside our other services. This gives us easy updates, volume management, and the same operational model as the rest of our infrastructure.

Directory Structure

mkdir -p /opt/monitoring/{prometheus,grafana,alertmanager}
mkdir -p /opt/monitoring/prometheus/{rules,targets}
mkdir -p /opt/monitoring/grafana/{provisioning/{datasources,dashboards},dashboards}
 
# Set permissions -- Grafana runs as UID 472
chown -R 472:472 /opt/monitoring/grafana

The Core Docker Compose File

# /opt/monitoring/docker-compose.yml
version: "3.9"
 
networks:
  monitoring:
    driver: bridge
  proxy:
    external: true   # Shared with Traefik
 
volumes:
  prometheus_data:
  grafana_data:
  alertmanager_data:
 
services:
  prometheus:
    image: prom/prometheus:v2.52.0
    container_name: prometheus
    restart: unless-stopped
    user: "nobody"
    networks:
      - monitoring
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=90d"       # Keep 90 days of metrics
      - "--storage.tsdb.retention.size=50GB"       # Hard cap at 50GB
      - "--web.enable-lifecycle"                   # Allow config reload via HTTP POST
      - "--web.enable-admin-api"                   # Enable admin API for snapshot/delete
      - "--query.max-samples=50000000"             # Prevent runaway queries
    volumes:
      - /opt/monitoring/prometheus:/etc/prometheus:ro
      - prometheus_data:/prometheus
    ports:
      - "127.0.0.1:9090:9090"   # Localhost only -- access via Traefik or Teleport
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.prometheus.rule=Host(`prometheus.teleport.yourdomain.com`)"
      - "traefik.http.routers.prometheus.middlewares=teleport-auth"   # Auth via Teleport
 
  grafana:
    image: grafana/grafana:11.0.0
    container_name: grafana
    restart: unless-stopped
    networks:
      - monitoring
      - proxy
    volumes:
      - grafana_data:/var/lib/grafana
      - /opt/monitoring/grafana/provisioning:/etc/grafana/provisioning:ro
      - /opt/monitoring/grafana/dashboards:/var/lib/grafana/dashboards:ro
    environment:
      # Security
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: "${GRAFANA_ADMIN_PASSWORD}"   # In .env file
      GF_SECURITY_SECRET_KEY: "${GRAFANA_SECRET_KEY}"
      GF_SECURITY_DISABLE_GRAVATAR: "true"
      GF_SECURITY_COOKIE_SECURE: "true"
      GF_SECURITY_COOKIE_SAMESITE: "strict"
 
      # Authentication -- disable anonymous, require login
      GF_AUTH_ANONYMOUS_ENABLED: "false"
      GF_AUTH_BASIC_ENABLED: "true"
 
      # Disable telemetry
      GF_ANALYTICS_REPORTING_ENABLED: "false"
      GF_ANALYTICS_CHECK_FOR_UPDATES: "false"
 
      # Server
      GF_SERVER_DOMAIN: "grafana.teleport.yourdomain.com"
      GF_SERVER_ROOT_URL: "https://grafana.teleport.yourdomain.com"
 
      # Paths
      GF_PATHS_DATA: /var/lib/grafana
      GF_PATHS_LOGS: /var/log/grafana
      GF_PATHS_PLUGINS: /var/lib/grafana/plugins
      GF_PATHS_PROVISIONING: /etc/grafana/provisioning
 
      # Install useful plugins on first start
      GF_INSTALL_PLUGINS: >
        grafana-clock-panel,
        grafana-piechart-panel,
        grafana-worldmap-panel,
        vonage-status-panel
 
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.grafana.rule=Host(`grafana.teleport.yourdomain.com`)"
      - "traefik.http.routers.grafana.entrypoints=websecure"
      - "traefik.http.routers.grafana.tls.certresolver=letsencrypt"
      - "traefik.http.services.grafana.loadbalancer.server.port=3000"
 
  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: alertmanager
    restart: unless-stopped
    networks:
      - monitoring
    command:
      - "--config.file=/etc/alertmanager/alertmanager.yml"
      - "--storage.path=/alertmanager"
      - "--web.external-url=https://alertmanager.teleport.yourdomain.com"
    volumes:
      - /opt/monitoring/alertmanager:/etc/alertmanager:ro
      - alertmanager_data:/alertmanager
    ports:
      - "127.0.0.1:9093:9093"
 
  # Node Exporter -- on every host you want to monitor
  # For the monitoring host itself:
  node-exporter:
    image: prom/node-exporter:v1.8.0
    container_name: node-exporter
    restart: unless-stopped
    network_mode: host    # Must be host mode to see all network interfaces
    pid: host             # Must see host PIDs for process metrics
    command:
      - "--path.rootfs=/host"
      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
      - "--collector.processes"
      - "--collector.systemd"
    volumes:
      - /:/host:ro,rslave
    # Note: no port mapping needed with network_mode: host
    # Exposed on :9100 on the host automatically
 
  # cAdvisor -- Docker container metrics
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.49.1
    container_name: cadvisor
    restart: unless-stopped
    privileged: true   # Required for container metric collection
    networks:
      - monitoring
    devices:
      - /dev/kmsg:/dev/kmsg
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker:/var/lib/docker:ro
      - /cgroup:/cgroup:ro
    ports:
      - "127.0.0.1:8080:8080"

Create the .env file (never commit this to Git):

# /opt/monitoring/.env
GRAFANA_ADMIN_PASSWORD=generate-a-strong-password-here
GRAFANA_SECRET_KEY=generate-a-64-char-random-string-here
 
# Generate these:
# GRAFANA_ADMIN_PASSWORD: openssl rand -base64 24
# GRAFANA_SECRET_KEY: openssl rand -hex 32

Prometheus Configuration

Main Configuration File

# /opt/monitoring/prometheus/prometheus.yml
global:
  scrape_interval: 15s          # Default scrape interval
  evaluation_interval: 15s      # Rule evaluation interval
  scrape_timeout: 10s           # Timeout per scrape
  external_labels:
    cluster: 'coderoasis-production'
    environment: 'production'
 
# Reference to Alertmanager
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093
      timeout: 10s
 
# Load alerting rules from separate files
rule_files:
  - "rules/*.yml"
 
# Scrape configurations
scrape_configs:
  # Prometheus monitoring itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
    metrics_path: /metrics
 
  # The monitoring host itself (running node exporter in host network mode)
  - job_name: 'node-monitoring'
    static_configs:
      - targets: ['host.docker.internal:9100']
        labels:
          instance: 'monitoring-host'
          role: 'monitoring'
 
  # Docker containers (cAdvisor on the monitoring host)
  - job_name: 'cadvisor-monitoring'
    static_configs:
      - targets: ['cadvisor:8080']
        labels:
          instance: 'monitoring-host'
 
  # Production servers -- Node Exporter on each host
  - job_name: 'node'
    static_configs:
      - targets:
          - 'web-01.internal:9100'
          - 'web-02.internal:9100'
          - 'db-01.internal:9100'
        labels:
          env: production
    relabel_configs:
      # Use the target address as the instance label
      - source_labels: [__address__]
        regex: '([^:]+):.*'
        target_label: instance
 
  # File-based service discovery -- drop new targets in here
  # Useful for dynamic infrastructure
  - job_name: 'file-sd-nodes'
    file_sd_configs:
      - files:
          - '/etc/prometheus/targets/*.yml'
        refresh_interval: 30s
 
  # cAdvisor on production hosts
  - job_name: 'cadvisor'
    static_configs:
      - targets:
          - 'web-01.internal:8080'
          - 'web-02.internal:8080'
    metrics_path: /metrics
 
  # PostgreSQL exporter
  - job_name: 'postgres'
    static_configs:
      - targets:
          - 'db-01.internal:9187'
        labels:
          env: production
          db: postgresql
 
  # Nginx/Traefik metrics
  - job_name: 'traefik'
    static_configs:
      - targets:
          - 'traefik:8082'   # Traefik metrics port (internal Docker network)
        labels:
          component: traefik
 
  # Blackbox Exporter -- external endpoint monitoring (HTTP, TCP, ICMP)
  - job_name: 'blackbox-http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://coderoasis.com
          - https://coderoasis.com/ghost/api/admin/
          - https://grafana.teleport.yourdomain.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

File-Based Service Discovery

Rather than restarting Prometheus every time you add a server, use file SD:

# /opt/monitoring/prometheus/targets/production-nodes.yml
- targets:
    - 'web-01.internal:9100'
    - 'web-02.internal:9100'
  labels:
    env: production
    role: webserver
    team: platform
 
- targets:
    - 'db-01.internal:9100'
  labels:
    env: production
    role: database
    team: platform

Prometheus watches this directory and hot-reloads when files change. Adding a new server: create or edit the YAML file. No Prometheus restart. New targets appear in the next scrape interval.

Alerting Rules

This is where Prometheus earns its keep for operations. Rules evaluate PromQL expressions on the scrape interval and fire alerts when conditions are met.

# /opt/monitoring/prometheus/rules/host-alerts.yml
groups:
  - name: host.rules
    interval: 30s    # Evaluate these every 30 seconds
    rules:
 
      # Disk space -- the one that bites you if you don't have monitoring
      - alert: DiskSpaceWarning
        expr: |
          (
            node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
            / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}
          ) * 100 < 25
        for: 10m     # Must be true for 10 consecutive minutes before alerting
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "Disk space below 25% on {{ $labels.instance }}"
          description: >
            {{ $labels.instance }} device {{ $labels.device }} has
            {{ $value | printf "%.1f" }}% free disk space remaining.
          runbook: "https://wiki.internal/runbooks/disk-space"
 
      - alert: DiskSpaceCritical
        expr: |
          (
            node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
            / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}
          ) * 100 < 10
        for: 5m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "CRITICAL: Disk space below 10% on {{ $labels.instance }}"
          description: >
            {{ $labels.instance }} device {{ $labels.device }} has only
            {{ $value | printf "%.1f" }}% free. Immediate action required.
 
      # Disk filling rate -- predict when you'll run out
      - alert: DiskFillingFast
        expr: |
          predict_linear(
            node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}[6h],
            4 * 3600   # predict 4 hours ahead
          ) < 0
        for: 30m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "Disk predicted to fill in under 4 hours on {{ $labels.instance }}"
          description: >
            Based on the last 6 hours of data, {{ $labels.instance }}
            device {{ $labels.device }} will run out of disk space within 4 hours.
 
      # Memory
      - alert: MemoryUsageHigh
        expr: |
          (
            1 - (
              node_memory_MemAvailable_bytes
              / node_memory_MemTotal_bytes
            )
          ) * 100 > 90
        for: 15m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "Memory usage above 90% on {{ $labels.instance }}"
          description: >
            {{ $labels.instance }} is using {{ $value | printf "%.1f" }}%
            of available memory for 15+ minutes.
 
      # CPU
      - alert: CPUUsageHigh
        expr: |
          100 - (avg by (instance) (
            rate(node_cpu_seconds_total{mode="idle"}[5m])
          ) * 100) > 85
        for: 20m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "CPU usage above 85% on {{ $labels.instance }}"
          description: >
            {{ $labels.instance }} has had {{ $value | printf "%.1f" }}%
            CPU utilization for over 20 minutes.
 
      # Load average
      - alert: HighLoadAverage
        expr: |
          node_load1 / on(instance) count by (instance) (
            node_cpu_seconds_total{mode="idle"}
          ) > 2
        for: 15m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "Load average above 2x CPU count on {{ $labels.instance }}"
          description: >
            {{ $labels.instance }} 1-minute load average is
            {{ $value | printf "%.2f" }}x the number of CPU cores.
 
      # Target down -- the most important alert
      - alert: TargetDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "Target {{ $labels.instance }} is down"
          description: >
            Prometheus cannot reach {{ $labels.job }}/{{ $labels.instance }}.
            It has been unreachable for 5+ minutes.
 
      # System reboot detection
      - alert: SystemRebooted
        expr: node_time_seconds - node_boot_time_seconds < 600
        for: 0m    # Fire immediately
        labels:
          severity: info
          team: platform
        annotations:
          summary: "{{ $labels.instance }} rebooted recently"
          description: >
            {{ $labels.instance }} has been up for less than 10 minutes,
            indicating a recent reboot.
 
---
# /opt/monitoring/prometheus/rules/database-alerts.yml
groups:
  - name: postgres.rules
    rules:
 
      - alert: PostgresDown
        expr: pg_up == 0
        for: 1m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "PostgreSQL is down on {{ $labels.instance }}"
 
      - alert: PostgresReplicationLag
        expr: pg_replication_lag > 30
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "PostgreSQL replication lag > 30s on {{ $labels.instance }}"
          description: "Replication lag is {{ $value | printf \"%.1f\" }} seconds."
 
      - alert: PostgresTooManyConnections
        expr: |
          pg_stat_activity_count / pg_settings_max_connections * 100 > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "PostgreSQL connection count above 80% of max"
          description: >
            {{ $value | printf "%.1f" }}% of max_connections are in use.
 
      - alert: PostgresSlowQueries
        expr: |
          rate(pg_stat_statements_total_time_seconds[5m]) /
          rate(pg_stat_statements_calls_total[5m]) > 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "PostgreSQL average query time > 1s"
 
---
# /opt/monitoring/prometheus/rules/http-alerts.yml
groups:
  - name: http.rules
    rules:
 
      - alert: EndpointDown
        expr: probe_success == 0
        for: 3m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "HTTP probe failing for {{ $labels.instance }}"
          description: >
            The endpoint {{ $labels.instance }} has been failing
            HTTP health checks for 3+ minutes.
 
      - alert: SSLCertExpiringSoon
        expr: probe_ssl_earliest_cert_expiry - time() < 30 * 24 * 3600
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "SSL cert expiring within 30 days for {{ $labels.instance }}"
          description: >
            Certificate expires at {{ $value | humanizeTimestamp }}.
 
      - alert: SSLCertExpiryCritical
        expr: probe_ssl_earliest_cert_expiry - time() < 7 * 24 * 3600
        for: 30m
        labels:
          severity: critical
        annotations:
          summary: "SSL cert expiring within 7 DAYS for {{ $labels.instance }}"
 
      - alert: HighHTTPErrorRate
        expr: |
          sum by (instance) (rate(traefik_service_requests_total{code=~"5.."}[5m]))
          /
          sum by (instance) (rate(traefik_service_requests_total[5m]))
          * 100 > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "HTTP 5xx error rate above 5% on {{ $labels.instance }}"
          description: "Error rate is {{ $value | printf \"%.2f\" }}%."

Pre-Compute Expensive Queries

Recording rules pre-compute expensive PromQL expressions and store the results as new time series. Use them for any query you run frequently in dashboards:

# /opt/monitoring/prometheus/rules/recording-rules.yml
groups:
  - name: aggregations
    interval: 1m
    rules:
      # Pre-compute CPU usage per instance -- used in every dashboard
      - record: instance:cpu_usage:rate5m
        expr: |
          100 - (avg by (instance) (
            rate(node_cpu_seconds_total{mode="idle"}[5m])
          ) * 100)
 
      # Pre-compute memory usage percentage
      - record: instance:memory_usage_percent
        expr: |
          (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
 
      # Pre-compute disk usage per device
      - record: instance:disk_usage_percent
        expr: |
          (1 - (
            node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
            / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}
          )) * 100
 
      # HTTP request rate by job
      - record: job:http_requests:rate5m
        expr: sum by (job, code) (rate(traefik_service_requests_total[5m]))

Alertmanager Configuration

# /opt/monitoring/alertmanager/alertmanager.yml
global:
  # Default time an alert stays resolved before it's no longer considered active
  resolve_timeout: 5m
  # SMTP for email alerts
  smtp_smarthost: 'mail.yourdomain.com:587'
  smtp_from: '[email protected]'
  smtp_auth_username: '[email protected]'
  smtp_auth_password: "${SMTP_PASSWORD}"
  smtp_require_tls: true
 
# Alert routing tree
route:
  # Default receiver
  receiver: 'slack-platform'
  # Group alerts with same job+alertname together
  group_by: ['alertname', 'job', 'instance']
  # Wait before sending first notification for a new group
  group_wait: 30s
  # Interval for sending notifications when alerts in group haven't changed
  group_interval: 5m
  # Minimum interval before re-sending a firing alert
  repeat_interval: 4h
 
  routes:
    # Critical alerts go to Slack AND email immediately
    - match:
        severity: critical
      receiver: 'critical-all'
      group_wait: 10s        # Faster notification for critical
      repeat_interval: 1h    # Re-alert every hour if still firing
 
    # Database alerts go to DBA channel
    - match:
        team: dba
      receiver: 'slack-dba'
 
    # Info alerts go to low-noise channel
    - match:
        severity: info
      receiver: 'slack-info'
      group_wait: 2m
      repeat_interval: 24h   # Don't spam info alerts
 
receivers:
  - name: 'slack-platform'
    slack_configs:
      - api_url: "${SLACK_WEBHOOK_URL}"
        channel: '#platform-alerts'
        send_resolved: true
        title: |-
          [{{ .Status | toUpper }}{{ if eq .Status "firing" }}: {{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}
        text: >-
          {{ range .Alerts -}}
          *Alert:* {{ .Annotations.summary }}
          *Severity:* `{{ .Labels.severity }}`
          *Instance:* `{{ .Labels.instance }}`
          *Description:* {{ .Annotations.description }}
          {{ end }}
        color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
 
  - name: 'critical-all'
    slack_configs:
      - api_url: "${SLACK_WEBHOOK_URL}"
        channel: '#platform-critical'
        send_resolved: true
        title: "🚨 [CRITICAL] {{ .CommonLabels.alertname }}"
        text: >-
          {{ range .Alerts -}}
          *{{ .Annotations.summary }}*
          {{ .Annotations.description }}
          {{ end }}
    email_configs:
      - to: '[email protected]'
        subject: '[CRITICAL] {{ .CommonLabels.alertname }} on {{ .CommonLabels.instance }}'
        body: |
          {{ range .Alerts }}
          Alert: {{ .Annotations.summary }}
          Description: {{ .Annotations.description }}
          Severity: {{ .Labels.severity }}
          Instance: {{ .Labels.instance }}
          Starts At: {{ .StartsAt }}
          {{ end }}
 
  - name: 'slack-info'
    slack_configs:
      - api_url: "${SLACK_WEBHOOK_URL}"
        channel: '#platform-info'
        send_resolved: false   # Don't notify when info alerts resolve
 
  - name: 'slack-dba'
    slack_configs:
      - api_url: "${SLACK_DBA_WEBHOOK_URL}"
        channel: '#database-alerts'
        send_resolved: true
 
# Inhibition rules -- suppress certain alerts when others are firing
inhibit_rules:
  # If an instance is completely down, suppress other alerts for that instance
  - source_match:
      alertname: TargetDown
    target_match_re:
      alertname: '.+'
    equal:
      - instance
 
  # If disk is critically low, suppress the warning
  - source_match:
      alertname: DiskSpaceCritical
    target_match:
      alertname: DiskSpaceWarning
    equal:
      - instance
      - device

Installing Node Exporter on Every Host

On each server you want to monitor, Node Exporter needs to run. Two options: Docker (simpler, slightly less access) or system service (more access, especially for systemd metrics).

System Service Installation (Recommended)

# Download and install Node Exporter on the target host
NODE_EXPORTER_VERSION="1.8.0"
cd /tmp
curl -LO "https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXPORTER_VERSION}/node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz"
tar xf "node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz"
cp "node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64/node_exporter" /usr/local/bin/
chmod +x /usr/local/bin/node_exporter
 
# Create system user
useradd --system --no-create-home --shell /bin/false node_exporter
 
# Create systemd service
cat > /etc/systemd/system/node_exporter.service << 'EOF'
[Unit]
Description=Prometheus Node Exporter
Documentation=https://github.com/prometheus/node_exporter
After=network-online.target
 
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
    --collector.filesystem.mount-points-exclude='^/(sys|proc|dev|host|etc)($$|/)' \
    --collector.processes \
    --collector.systemd \
    --web.listen-address=:9100
Restart=on-failure
RestartSec=5s
 
# Security hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
 
[Install]
WantedBy=multi-user.target
EOF
 
systemctl daemon-reload
systemctl enable node_exporter
systemctl start node_exporter
 
# Verify it's running and exposing metrics
curl http://localhost:9100/metrics | head -20

Firewall Rules

Node Exporter should only be accessible from your Prometheus server:

# Allow Prometheus to scrape Node Exporter
# Replace with your Prometheus server's actual IP
ufw allow from PROMETHEUS_SERVER_IP to any port 9100 proto tcp
ufw reload

Never expose port 9100 to the public internet. Node Exporter exposes detailed system information that's valuable for an attacker.

PostgreSQL Exporter Setup

# On the database server (or a host with access to the database)
POSTGRES_EXPORTER_VERSION="0.15.0"
curl -LO "https://github.com/prometheus-community/postgres_exporter/releases/download/v${POSTGRES_EXPORTER_VERSION}/postgres_exporter-${POSTGRES_EXPORTER_VERSION}.linux-amd64.tar.gz"
tar xf "postgres_exporter-${POSTGRES_EXPORTER_VERSION}.linux-amd64.tar.gz"
cp postgres_exporter /usr/local/bin/
 
# Create monitoring user in PostgreSQL
# Run as postgres user:
psql -c "CREATE USER prometheus_exporter WITH PASSWORD 'strong-random-password';"
psql -c "GRANT pg_monitor TO prometheus_exporter;"
psql -c "GRANT SELECT ON pg_stat_statements TO prometheus_exporter;" 2>/dev/null || true

# /etc/default/postgres_exporter
DATA_SOURCE_NAME="postgresql://prometheus_exporter:strong-random-password@localhost:5432/postgres?sslmode=disable"
PG_EXPORTER_DISABLE_SETTINGS_METRICS="false"
PG_EXPORTER_AUTO_DISCOVER_DATABASES="true"

# /etc/systemd/system/postgres_exporter.service
[Unit]
Description=Prometheus PostgreSQL Exporter
After=network.target
 
[Service]
User=prometheus_exporter
EnvironmentFile=/etc/default/postgres_exporter
ExecStart=/usr/local/bin/postgres_exporter \
    --web.listen-address=:9187 \
    --log.level=info
Restart=on-failure
 
[Install]
WantedBy=multi-user.target

Grafana Configuration and Dashboards

Provisioning: Datasources

Provision data sources through code so they're automatically available on fresh deploys:

# /opt/monitoring/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
 
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    version: 1
    editable: false
    jsonData:
      httpMethod: POST
      exemplarTraceIdDestinations: []
      # Enable Prometheus features
      prometheusType: Prometheus
      prometheusVersion: "2.52.0"
      cacheLevel: "Low"
      incrementalQuerying: true
      incrementalQueryOverlapWindow: 10m
      queryTimeout: "60s"

Provisioning: Dashboards

Auto-provision dashboard folders:

# /opt/monitoring/grafana/provisioning/dashboards/default.yaml
apiVersion: 1
 
providers:
  - name: 'Default'
    orgId: 1
    folder: 'Infrastructure'
    folderUid: 'infrastructure'
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

The Most Useful Pre-Built Dashboards

Import these from Grafana's dashboard repository by ID (Dashboard > Import > Enter ID):

Building a Custom Overview Dashboard

Import by ID gets you started fast. For your team's actual day-to-day needs, build a custom overview dashboard. Here's the PromQL for the panels we use most:

# CPU Usage - Gauge or stat panel, one per server
instance:cpu_usage:rate5m{instance="web-01.internal:9100"}
 
# Memory Usage %
instance:memory_usage_percent{instance="web-01.internal:9100"}
 
# Disk Usage % - for all mounts on a host
instance:disk_usage_percent{instance="web-01.internal:9100"}
 
# Network receive rate (bytes/sec)
rate(node_network_receive_bytes_total{instance="web-01.internal:9100", device!~"lo|docker.*"}[5m])
 
# Network transmit rate
rate(node_network_transmit_bytes_total{instance="web-01.internal:9100", device!~"lo|docker.*"}[5m])
 
# HTTP requests per second (from Traefik)
sum(rate(traefik_service_requests_total{code!~"4..|5.."}[5m]))
 
# HTTP error rate %
sum(rate(traefik_service_requests_total{code=~"5.."}[5m])) /
sum(rate(traefik_service_requests_total[5m])) * 100
 
# PostgreSQL active connections
pg_stat_activity_count{datname="appdb", state="active"}
 
# System load (normalized by CPU count)
node_load1 / count by (instance)(node_cpu_seconds_total{mode="idle"})
 
# Time since last scrape per target (shows which targets are going stale)
time() - prometheus_target_scrape_pool_last_scrape_timestamp_seconds

The Queries That Actually Matter

PromQL is Prometheus's query language. It's powerful and somewhat unintuitive until you understand its data model. Here are the patterns you'll use constantly:

# ── RATES AND AVERAGES ───────────────────────────────────────────────
 
# Rate of increase over 5 minutes (for counters that only go up)
rate(http_requests_total[5m])
 
# Per-second average rate from irate (more responsive to spikes, use for alerts)
irate(http_requests_total[5m])
 
# Average value over time range
avg_over_time(node_memory_MemAvailable_bytes[1h])
 
# ── FILTERING ────────────────────────────────────────────────────────
 
# Exact match
node_cpu_seconds_total{mode="idle"}
 
# Regex match (all non-idle CPU modes)
node_cpu_seconds_total{mode!~"idle|iowait"}
 
# Multiple label match
node_filesystem_avail_bytes{instance="web-01:9100", fstype!="tmpfs"}
 
# ── AGGREGATION ──────────────────────────────────────────────────────
 
# Sum across all instances
sum(rate(http_requests_total[5m]))
 
# Sum, grouped by instance
sum by (instance) (rate(http_requests_total[5m]))
 
# Average across all instances
avg(node_memory_MemAvailable_bytes)
 
# Maximum value, grouped by job
max by (job) (up)
 
# Count of running instances per job
count by (job) (up == 1)
 
# ── MATH AND CALCULATIONS ────────────────────────────────────────────
 
# Percentage calculation
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
 
# Rate per minute (rate() gives per-second by default)
rate(requests_total[5m]) * 60
 
# Bytes to MB
node_filesystem_avail_bytes / 1024 / 1024
 
# ── PREDICTION ───────────────────────────────────────────────────────
 
# Predict disk usage 24 hours from now using 6 hours of history
predict_linear(node_filesystem_avail_bytes[6h], 24 * 3600)
 
# Days until disk runs out
(
  node_filesystem_avail_bytes{fstype!="tmpfs"}
  / -rate(node_filesystem_avail_bytes{fstype!="tmpfs"}[6h])
) / 86400
 
# ── JOINS AND COMPARISONS ────────────────────────────────────────────
 
# Load average normalized by CPU count (multi-label join)
node_load1 / on(instance) group_left count by (instance)(
  node_cpu_seconds_total{mode="idle"}
)
 
# Alert when memory is low AND CPU is high (correlate conditions)
(
  instance:memory_usage_percent > 80
) and (
  instance:cpu_usage:rate5m > 70
)
 
# ── HISTOGRAM QUERIES ────────────────────────────────────────────────
 
# 95th percentile HTTP request duration
histogram_quantile(0.95,
  rate(http_request_duration_seconds_bucket[5m])
)
 
# 99th percentile grouped by endpoint
histogram_quantile(0.99,
  sum by (le, endpoint) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

Setting Up the Blackbox Exporter

The Blackbox Exporter probes endpoints from outside -- HTTP checks, SSL certificate validation, TCP connectivity. Install it alongside your monitoring stack:

# Add to docker-compose.yml
  blackbox-exporter:
    image: prom/blackbox-exporter:v0.25.0
    container_name: blackbox-exporter
    restart: unless-stopped
    networks:
      - monitoring
    command:
      - "--config.file=/config/blackbox.yml"
    volumes:
      - /opt/monitoring/blackbox:/config:ro
    ports:
      - "127.0.0.1:9115:9115"

# /opt/monitoring/blackbox/blackbox.yml
modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: []    # Default to 2xx
      method: GET
      follow_redirects: true
      tls_config:
        insecure_skip_verify: false
      preferred_ip_protocol: "ip4"
 
  http_post_2xx:
    prober: http
    timeout: 10s
    http:
      method: POST
      headers:
        Content-Type: application/json
 
  tcp_connect:
    prober: tcp
    timeout: 10s
 
  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: "ip4"

Add endpoints to monitor in your Prometheus scrape config (already shown above). Grafana dashboard ID 7587 gives you a complete view of all probed endpoints.

Operational Tasks

Reload Configuration Without Restart

# Reload Prometheus config (requires --web.enable-lifecycle flag)
curl -X POST http://localhost:9090/-/reload
 
# Reload Alertmanager config
curl -X POST http://localhost:9093/-/reload
 
# Grafana picks up provisioned dashboard changes automatically
# For settings changes: docker compose restart grafana

Taking Prometheus Snapshots for Backup

# Create a snapshot (requires --web.enable-admin-api)
curl -X POST http://localhost:9090/api/v1/admin/tsdb/snapshot
 
# Returns: {"status":"success","data":{"name":"20260501T030000Z-abc123"}}
# Snapshot is at /var/lib/prometheus/snapshots/20260501T030000Z-abc123/
 
# Back up the snapshot directory
rsync -av /var/lib/prometheus/snapshots/ backup-server:/backups/prometheus/

Pruning Old Metrics

# Delete metrics for a decommissioned instance
# (requires --web.enable-admin-api)
curl -X POST \
  -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="decommissioned-server:9100"}' \
  
# Clean up tombstones after deletion
curl -X POST http://localhost:9090/api/v1/admin/tsdb/clean_tombstones

Checking What's Consuming Storage

# Largest time series by cardinality
topk(10,
  count by (__name__)(
    {__name__!=""}
  )
)
 
# Storage used per job
# Check Prometheus's own TSDB metrics
prometheus_tsdb_head_series
 
# Series count by job
count by (job) ({__name__=~".+"})

Frequently Asked Questions

How much storage does Prometheus need?

A typical production server (single host, all exporters) generates approximately 1-5 million samples per day across all metrics. Prometheus compresses time series to approximately 1-2 bytes per sample. For 5 hosts, all exporters, 90-day retention: roughly 5 hosts × 5M samples/day × 1.5 bytes × 90 days ≈ 3.4GB. Add a safety factor and plan for 20-50GB for a small infrastructure. The --storage.tsdb.retention.size flag in the compose file caps it.

Should I expose Prometheus and Grafana directly with HTTPS?

No. Keep Prometheus on localhost and proxy Grafana through Traefik with authentication, or proxy both through Teleport for zero-trust access control. Prometheus exposes detailed infrastructure information and its admin API can delete data. Grafana has had security vulnerabilities. Neither should be directly internet-accessible.

What's the difference between rate() and irate()?

rate() computes the per-second average rate over the entire time window. irate() computes it using only the last two data points, making it much more responsive to sudden spikes. Use rate() for smooth graphs and recording rules. Use irate() for alerting where you want to catch sudden spikes. Use irate() with a window like [5m] to allow for missing scrapes.

Can Prometheus monitor external services I don't control?

Yes, via the Blackbox Exporter. You can probe any HTTP endpoint, check SSL certificate expiry, and verify TCP connectivity. You won't have internal metrics, but you'll know when things are down and when certificates are about to expire.

Prometheus and Grafana give you the observability layer that makes everything else manageable. The Teleport audit logs we ship to Grafana Loki. The self-hosted services from the productivity stack guide all expose metrics via Node Exporter on their hosts. The infrastructure at CoderOasis has had a monitoring blind spot period of about eighteen months total — before we set this up. We've never had a disk-full incident since. That's a good enough return on one afternoon of setup.