Auto vs predictive scaling: scale before the load, not after it

HPA works from what just happened. That makes it structurally slower than step-shaped traffic — and the gap is exactly when users feel the slowdown.

Why HPA always arrives late

HPA reads CPU or custom metrics from the pods that exist, then adds replicas once a threshold is crossed. Suppose traffic triples in thirty seconds:

1. the metric waits for a scrape interval (15–30s)
2. HPA decides to scale
3. new pods pull the image, start, and warm caches (20–60s)
4. excess traffic queues in the meantime

Two minutes of degraded latency before capacity catches up.

Three layers worth having

Layer one — reactive: HPA on the metric closest to the user, such as request rate per pod, not CPU. CPU rises after people press reload.

Layer two — scheduled: if Monday 09:00 always brings a crowd, pre-scale fifteen minutes early with a CronJob or a KEDA cron trigger:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: api-morning-ramp
spec:
  scaleTargetRef:
    name: api
  minReplicaCount: 4
  maxReplicaCount: 40
  triggers:
    - type: cron
      metadata:
        timezone: Asia/Bangkok
        start: 0 45 8 * * *
        end: 0 30 11 * * *
        desiredReplicas: '18'
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: sum(rate(http_requests_total{job="api"}[2m])) / sum(kube_deployment_status_replicas{deployment="api"})
        threshold: '45'

Layer three — predictive: feed a seasonal model (Prophet, or a moving average with a seasonal term) from historical data and publish the forecast as a metric HPA can read.

Warning: predictive scaling without guardrails will scale the wrong way on a public holiday. Always cap the maximum.

Numbers from a real tuning session

On a recent e-commerce engagement, peak traffic starts at 20:00. Adding a scheduled pre-scale cut p95 latency from 1.8s to 420ms at peak and removed 96% of timeout errors.

Check these first

• At least three months of history, with outages excluded from training.
• Cost of idle replicas versus the cost of slow responses.
• A reactive fallback whenever the model is wrong.

Good scaling is not great guessing. It is knowing that being wrong will not break the system.