You can’t watch a dashboard 24/7. Alerting flips the model: instead of you checking the system, the system tells you when something’s wrong. Prometheus evaluates alert rules; Alertmanager decides who gets notified and how.

🎯 Learning Objectives

• Write a Prometheus alerting rule
• Understand how Alertmanager routes and groups alerts
• Send notifications to a receiver (e.g. Slack)
• Apply best practices so alerts are actionable, not noise

How Alerting Works

Alerting is split into two jobs on purpose:

Prometheus                         Alertmanager
──────────                         ────────────
evaluates alert rules  ──fires──▶  groups, deduplicates, silences
against metrics                    routes to the right receiver
                                 ──▶ Slack / email / PagerDuty

• Prometheus decides whether an alert is firing (based on PromQL).
• Alertmanager decides what to do about firing alerts — grouping, silencing, and routing.

Writing an Alert Rule

Alert rules live in a rules file that Prometheus loads. Each rule is a PromQL expression plus a duration and metadata:

groups:
- name: service-alerts
  rules:
    - alert: HighErrorRate
      expr: |
        sum(rate(http_requests_total{status=~"5.."}[5m]))
        /
        sum(rate(http_requests_total[5m])) > 0.05
      for: 10m
      labels:
        severity: critical
      annotations:
        summary: "High 5xx error rate on {{ $labels.service }}"
        description: "Error rate is above 5% for 10 minutes."

Key parts:

• expr — the condition. Here: 5xx errors are more than 5% of traffic.
• for — how long it must stay true before firing. This prevents flapping on a one-second blip.
• labels — attached to the alert (e.g. severity) and used for routing.
• annotations — human-readable context sent in the notification.

Routing with Alertmanager

Alertmanager’s config decides where alerts go and how they’re grouped:

route:
group_by: ["alertname", "service"]   # collapse related alerts into one message
group_wait: 30s
repeat_interval: 4h
receiver: "slack-default"
routes:
  - match:
      severity: critical
    receiver: "pagerduty"            # critical → page a human

receivers:
- name: "slack-default"
  slack_configs:
    - channel: "#alerts"
- name: "pagerduty"
  pagerduty_configs:
    - routing_key: "<your-key>"

group_by is powerful: if 50 pods all fail at once, you get one grouped notification, not 50.

Best Practices: Alert on Symptoms, Not Causes

Bad alerting trains people to ignore alerts. Good alerting is rare and always actionable.

• Alert on symptoms users feel (high error rate, high latency, site down) rather than every internal cause (one CPU spike).
• Every alert must be actionable — if there’s nothing to do, it shouldn’t page.
• Tier by severity: critical pages a human now; warning goes to a channel to review later.
• Include a runbook link in annotations so the responder knows the first steps.
• Tune for and thresholds based on real incidents — alert fatigue is a real outage risk.

🧪 Hands-on Lab

groups:
- name: availability
  rules:
    - alert: InstanceDown
      expr: up == 0
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "Instance {{ $labels.instance }} is down"
        description: "{{ $labels.job }} target has been unreachable for 2 minutes."

🧠 Knowledge Check

💼 Interview Preparation

Summary

You’ve completed the monitoring track: you understand metrics, Prometheus and PromQL, Grafana dashboards, and alerting with Alertmanager. You can now see your systems and be told when they break. Next up is Argo CD and the GitOps approach to deploying all of this.