Monitoring MariaDB Galera with Prometheus and Grafana

A database cluster nobody watches is a cluster that goes down quietly and lets customers find out first. This is the exact stack I run on MariaDB Galera, from node-exporter up to a dashboard that answers "are we okay?" in five seconds.

The pieces

• Prometheus stores time series and evaluates alert rules
• node-exporter collects host metrics (CPU, disk, network)
• mysqld_exporter reads database and Galera status
• Grafana renders dashboards the ops team actually shares
• Alertmanager routes to Slack or LINE Notify

A compose file that works

services:
  prometheus:
    image: prom/prometheus:v2.54.1
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.retention.time=30d
    volumes: ['./prometheus.yml:/etc/prometheus/prometheus.yml:ro', 'prom-data:/prometheus']
    ports: ['9090:9090']

  grafana:
    image: grafana/grafana:11.1.0
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD}
    volumes: ['grafana-data:/var/lib/grafana']
    ports: ['3000:3000']

Expose MySQL metrics through a dedicated least-privilege user:

CREATE USER 'exporter'@'10.%' IDENTIFIED BY 'strong-secret' WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'10.%';

The metrics that actually matter

There are thousands of series, but only a handful ever got me out of bed successfully:

Alerts that respect sleep

Rule of thumb: a 3am page must be something a human can fix at 3am. Capacity issues belong in a Slack channel during business hours.

groups:
  - name: galera
    rules:
      - alert: GaleraClusterDegraded
        expr: wsrep_cluster_size < 3
        for: 2m
        labels: { severity: critical }
      - alert: DiskAlmostFull
        expr: node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.15
        for: 30m
        labels: { severity: warning }

Two years of hindsight

Prometheus disk is the classic blind spot. Thirty days of millions of series consumes real storage — set an explicit retention and mount a separate volume.

Keep labels disciplined. If instance is sometimes an IP and sometimes a hostname, every query becomes a archaeology project later. Use the hostname as the source of truth and keep the IP as a secondary label.

And a good dashboard is not the one filling the screen with graphs — it is the one that says whether the system is healthy within the first five seconds.