Alerts & Alarms

Overview

The CDN Manager ships a set of pre-configured alerting rules evaluated by vmalert against VictoriaMetrics. When a rule fires, the alert is routed to Alertmanager, which handles deduplication, grouping, silencing, and delivery to configured notification channels.

This page documents every built-in alert rule, what it means, its severity, and the recommended operator action.

Alert Severity Levels

Alert Groups

Alerts are organised into the following groups, each evaluated on a 15-second interval.

• infra-disk — Disk space and I/O
• infra-compute — CPU and memory
• infra-network — Network errors and traffic anomalies
• longhorn — Persistent storage health

infra-disk

Monitors disk space utilisation and I/O latency on cluster nodes.

StorageFillingUp

What it means: A node’s root filesystem is running low on space. If left unchecked this will progress to a full disk, which can cause pod evictions, write failures, and potential data loss.

Recommended actions:

1. Identify the node from the host label in the alert.
2. Log into the node and check disk usage:

   df -h /
   du -sh /var/log/* | sort -rh | head -20
   

3. Clear old log files, unused container images, or temporary files:

   # On the node
   journalctl --vacuum-size=500M
   crictl rmi --prune
   

4. If disk usage is due to application data growth, consider expanding the volume or adjusting retention settings. See Metrics Retention.

HighDiskLatency

What it means: Disk write operations are taking longer than 100 ms on average. High disk latency can degrade database performance (PostgreSQL, VictoriaMetrics) and cause timeouts in write-heavy components.

Recommended actions:

1. Identify the affected disk from the name label in the alert.
2. Check for I/O-intensive processes on the node:

   iostat -x 2 5
   iotop -o
   

3. Check for Longhorn replica rebuilds or rebalancing activity, which can saturate disk I/O.
4. If the issue persists on a production node, review whether the storage hardware meets the System Requirements.

infra-compute

Monitors CPU and memory utilisation on cluster nodes.

CpuSaturation

What it means: A node is running at near-full CPU capacity. Sustained CPU saturation causes request latency increases across all workloads on that node and may result in pod throttling.

Recommended actions:

1. Identify the saturated node from the host label in the alert.
2. Check which pods are consuming CPU:

   kubectl top pods --sort-by=cpu -A
   

3. Check for runaway processes on the node:

   top -b -n 1 | head -20
   

4. If saturation is caused by a legitimate workload spike (e.g. CDN traffic burst), consider scaling the deployment or redistributing load across nodes.

MemoryCriticallyLow

What it means: The node has very little free memory remaining. The Linux OOM killer may begin terminating processes, which can cause abrupt pod restarts, data corruption in in-memory caches, and service unavailability.

Recommended actions:

1. Identify the affected node from the host label in the alert.
2. Immediately check for memory-leaking or oversized pods:

   kubectl top pods --sort-by=memory -A
   

3. Identify and restart any pods showing abnormal memory consumption:

   kubectl rollout restart deployment/<name>
   

4. Check kernel OOM kill log for any processes already killed:

   dmesg | grep -i "oom\|killed"
   

5. Review memory resource limits and requests for affected deployments and adjust if necessary.

SwapUsageDetected

What it means: The node is swapping memory to disk. Swap usage in a Kubernetes cluster is a strong indicator of memory pressure. It degrades performance significantly and may mask an underlying memory shortage that could escalate to a MemoryCriticallyLow event.

Recommended actions:

1. Treat this as an early warning for the same conditions as MemoryCriticallyLow.
2. Identify memory-intensive pods and investigate whether resource limits are configured appropriately.
3. Swap should ideally never be active on a production Kubernetes node. If it persists, escalate to a memory capacity review.

infra-network

Monitors network interface errors and traffic anomalies on cluster nodes.

NetworkInterfaceErrors

What it means: A network interface is dropping or corrupting packets. Even a low error rate can cause TCP retransmissions, increased latency, and connection failures — directly impacting CDN Director communication and external traffic delivery.

Recommended actions:

1. Identify the affected host and interface from the host and interface labels in the alert.
2. Check interface error counters on the node:

   ip -s link show <interface>
   ethtool -S <interface> | grep -i error
   

3. Check for duplex/speed mismatches between the node NIC and the upstream switch:

   ethtool <interface> | grep -E "Speed|Duplex"
   

4. Escalate to network/hardware team if errors are persistent and cannot be attributed to a software configuration issue.

SuddenNetworkEgressDrop

What it means: A significant, sudden reduction in outbound traffic has been detected. This typically indicates a upstream network failure, link fault, or a routing issue. A CDN node that stops transmitting traffic is effectively out of service.

Recommended actions:

1. Identify the affected node and interface from the alert labels.
2. Verify the node’s network connectivity:

   ping <gateway-ip>
   traceroute <upstream-endpoint>
   

3. Check for interface errors or link-down events:

   ip link show
   dmesg | grep -i "link\|eth\|nic"
   

4. Verify that upstream routing and firewall rules have not changed.
5. If the node is healthy and traffic has legitimately dropped (e.g. a CDN traffic shift), the alert can be silenced if the traffic reduction is expected and understood.

SuddenNetworkIngressSpike

What it means: A sudden surge of inbound traffic has been detected. This may indicate a legitimate traffic event (e.g. a large stream audience spike), a DDoS attempt, or a misconfigured client sending unexpected volume.

Recommended actions:

1. Identify the affected node and interface from the alert labels.
2. Review active connections and top talkers:

   ss -s
   netstat -an | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head
   

3. Correlate with CDN Director metrics in Grafana to determine whether the spike is legitimate CDN traffic.
4. If the spike is unexpected and sustained, consider rate-limiting or blocking the source at the network edge.

longhorn

Monitors the health of Longhorn distributed block storage, which backs persistent volumes for PostgreSQL, VictoriaMetrics, and other stateful components.

Note: Longhorn alert rules are always present in the alert configuration, but will not fire in environments where Longhorn is not installed (e.g. cloud deployments using external storage).

LonghornVolumeDegraded

What it means: A Longhorn volume has fewer healthy replicas than its configured replication factor. Data is not at immediate risk, but the volume has reduced redundancy. A single additional node or disk failure could result in data loss or volume unavailability.

Recommended actions:

1. Identify the affected volume from the volume label in the alert.
2. Open the Longhorn UI and inspect the volume’s replica status.
3. Check whether a replica is in the process of rebuilding (this is normal after a node restart). Rebuilding may take several minutes depending on volume size.
4. If a replica has failed and is not rebuilding, attempt to evict and re-schedule it via the Longhorn UI.
5. Investigate the health of the node that hosted the failed replica:

   kubectl get nodes
   kubectl describe node <node-name>
   

LonghornVolumeFaulted

What it means: A Longhorn volume has lost all healthy replicas and is no longer accessible. Any workload that depends on this volume (e.g. PostgreSQL, VictoriaMetrics) will be unable to write and may crash. Data may be at risk.

Recommended actions:

1. Identify the affected volume from the volume label.
2. Immediately check which pods are using the volume:

   kubectl get pods -A -o wide | grep -i <volume-name>
   

3. Open the Longhorn UI. Check whether any replicas are still present and whether they can be recovered.
4. Do not delete faulted replicas without first attempting recovery — they may contain the only copy of the data.
5. Contact AgileTV support if the volume cannot be recovered, providing Longhorn UI screenshots and node logs.

LonghornNodeDown

What it means: A storage node is unreachable or unhealthy from Longhorn’s perspective. All volumes with replicas on this node are at reduced redundancy. If more than one node goes down simultaneously, faulted volumes and data loss become a risk.

Recommended actions:

1. Identify the affected node from the node label in the alert.
2. Check the node’s status in Kubernetes:

   kubectl get nodes
   kubectl describe node <node-name>
   

3. Attempt to SSH to the node and check system health:

   ssh root@<node-ip>
   systemctl status k3s
   

4. If the node has crashed and cannot be recovered quickly, consider evicting its Longhorn replicas to allow rebuilding on healthy nodes — but only if the remaining healthy nodes have sufficient capacity.

LonghornDiskSpaceLow

What it means: A node’s Longhorn-managed disk is running low on storage. When Longhorn disk space is exhausted, it cannot schedule new replicas or accommodate volume growth, which can lead to LonghornVolumeDegraded or LonghornVolumeFaulted conditions.

Recommended actions:

1. Identify the affected node and disk from the node and disk labels in the alert.
2. Open the Longhorn UI and check which volumes have replicas on this disk.
3. Check for snapshots or backups that can be cleaned up to reclaim space.
4. If space cannot be reclaimed, consider adding a disk to the node or expanding the underlying block device.
5. Review Metrics Retention settings — reducing VictoriaMetrics retention is often the fastest way to reclaim Longhorn disk space in a monitoring-heavy deployment.

Adding Custom Alert Rules

Additional alert rules can be defined by extending the victoria_metrics_alert.server.config.alerts.groups list in your values.yaml. Rules follow the Prometheus alerting rule format.

Example: Adding a Custom Alert

The following example adds an alert group that fires when a Kafka consumer lag exceeds a threshold:

victoria_metrics_alert:
  server:
    config:
      alerts:
        groups:
          # ... existing groups are preserved alongside your additions ...
          - name: kafka
            interval: 15s
            rules:
              - alert: KafkaConsumerLagHigh
                expr: kafka_consumer_group_lag > 10000
                for: 5m
                labels:
                  severity: warning
                annotations:
                  summary: "High consumer lag on {{ $labels.topic }}"
                  description: "Consumer group {{ $labels.group }} is {{ $value }} messages behind on topic {{ $labels.topic }}."

Apply the change using the standard upgrade procedure in the Configuration Guide.

Rule Fields Reference

Tip: Use the Alertmanager UI (https://<manager-host>/alertmanager) to verify that fired alerts are being received and routed correctly after adding new rules.

Configuring Alert Routes

By default, all alerts are routed to the built-in null receiver, which silently discards them. To receive alerts, configure one or more receivers and update the routing rules — all within the alertmanager.config section of your values.yaml.

Route Structure

The top-level route defines the default behaviour. Child routes under routes match alerts by label and direct them to specific receivers:

alertmanager:
  config:
    route:
      receiver: 'null'          # Default: discard unmatched alerts
      group_by: ['alertname']
      group_wait: 10s           # Wait before sending first notification for a new group
      group_interval: 10s       # Wait before sending updated notifications for a group
      repeat_interval: 1h       # Re-notify if an alert is still firing after this period
      routes:
        - matchers:
            - severity="critical"
          receiver: 'slack'
        - matchers:
            - severity="warning"
          receiver: 'email-warning'

Routes are evaluated top-to-bottom. The first matching route wins unless continue: true is set on the route.

Notification Channels

Email

Email requires an SMTP server to be configured globally. Both a critical and warning receiver can be defined independently.

alertmanager:
  config:
    global:
      smtp_smarthost: 'smtp.example.com:587'
      smtp_from: 'alertmanager@example.com'
      smtp_require_tls: true
    route:
      routes:
        - matchers:
            - severity="critical"
          receiver: 'email-critical'
        - matchers:
            - severity="warning"
          receiver: 'email-warning'
    receivers:
      - name: 'null'
      - name: 'email-critical'
        email_configs:
          - to: 'oncall@example.com'
            send_resolved: true
      - name: 'email-warning'
        email_configs:
          - to: 'alerts@example.com'
            send_resolved: true

Slack

Requires an incoming webhook URL created in your Slack workspace.

alertmanager:
  config:
    route:
      routes:
        - matchers:
            - severity="critical"
          receiver: 'slack'
    receivers:
      - name: 'null'
      - name: 'slack'
        slack_configs:
          - api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
            channel: '#alerts'
            send_resolved: true
            title: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}'
            text: |
              *Severity:* {{ .CommonLabels.severity }}
              *Host:* {{ .CommonLabels.host }}
              {{ range .Alerts }}{{ .Annotations.description }}{{ end }}

Telegram

Requires a Telegram bot token and a channel/group chat ID. Create a bot via @BotFather and add it to your alert channel before configuring.

alertmanager:
  config:
    route:
      routes:
        - matchers:
            - severity="critical"
          receiver: 'telegram'
    receivers:
      - name: 'null'
      - name: 'telegram'
        telegram_configs:
          - bot_token: 'your-bot-token'
            chat_id: -1234567890
            parse_mode: 'Markdown'
            send_resolved: true
            message: |
              *Alert:* {{ .CommonLabels.alertname }}
              *Severity:* {{ .CommonLabels.severity }}
              *Host:* {{ .CommonLabels.host }}
              {{ range .Alerts }}
                {{ .Annotations.description }}
              {{ end }}

Finding your chat ID: Add your bot to the channel or group, send a message, then call https://api.telegram.org/bot<token>/getUpdates and read the chat.id from the response. Note that group and channel chat IDs are negative numbers.

Combining Multiple Receivers

Routes and receivers can be combined to send different alert severities to different channels simultaneously. For example, critical alerts to PagerDuty and Slack, warnings to email only:

alertmanager:
  config:
    route:
      receiver: 'null'
      routes:
        - matchers:
            - severity="critical"
          receiver: 'slack'
          continue: true        # Continue matching so the next route also fires
        - matchers:
            - severity="critical"
          receiver: 'email-critical'
        - matchers:
            - severity="warning"
          receiver: 'email-warning'
    receivers:
      - name: 'null'
      - name: 'slack'
        slack_configs:
          - api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
            channel: '#critical-alerts'
            send_resolved: true
      - name: 'email-critical'
        email_configs:
          - to: 'oncall@example.com'
            send_resolved: true
      - name: 'email-warning'
        email_configs:
          - to: 'alerts@example.com'
            send_resolved: true

Apply any receiver or routing changes using the standard upgrade procedure in the Configuration Guide.

Silencing Alerts

Silences suppress alert notifications for a defined time window without disabling the underlying alert rule. They are useful during planned maintenance, known incidents, or when investigating a non-urgent condition.

Silences are managed via the Alertmanager UI, accessible at:

https://<manager-host>/alertmanager

Creating a Silence

1. Navigate to the Alertmanager UI and click Silences in the top navigation.
2. Click Create Silence.
3. Set the Start and End times for the silence window.
4. Add one or more matchers to scope which alerts are suppressed. For example:
   • alertname = StorageFillingUp — silence a specific alert
   • severity = warning — silence all warnings
   • host = node-01 — silence all alerts from a specific host
5. Add a Comment describing the reason for the silence (e.g. Planned disk expansion on node-01).
6. Click Create. The silence takes effect immediately.

Expiring a Silence

Silences expire automatically at the configured end time. To remove a silence early, navigate to Silences in the Alertmanager UI, locate the silence, and click Expire.

Next Steps

1. Operations Guide - Day-to-day operational procedures
2. Troubleshooting Guide - Resolve underlying issues surfaced by alerts
3. Metrics & Monitoring Overview - Return to the monitoring overview