Migrating Monitors

The Audit

Before migrating anything, we exported the full vendor inventory and categorized every monitor by signal source. The question: “if this alert stopped firing, would anyone notice?”

A significant chunk of monitors were dropped - decommissioned services, duplicates, informational messages nobody read, thresholds so loose they never triggered.

Phase 1: Infrastructure (Day 1)

Already working from day one - Prometheus scrapes node-exporter and kube-state-metrics.

Host: instance down, CPU > 85%, memory > 85%, disk > 85%, I/O wait > 25%.

K8s: CrashLoopBackOff, ImagePullBackOff, pending pods, restart storms, node NotReady, DaemonSet health, CronJob failures.

Standard Prometheus rules. Battle-tested community templates.

Example Alert Rules

# Service error rate > 5% for 10 minutes
- alert: HighErrorRate
  expr: |
    sum(rate(traces_spanmetrics_calls_total{
      status_code="STATUS_CODE_ERROR",
      deployment_environment="prod"
    }[5m])) by (service)
    /
    sum(rate(traces_spanmetrics_calls_total{
      deployment_environment="prod"
    }[5m])) by (service)
    > 0.05
  for: 10m
  labels:
    severity: critical

# P90 latency > 2 seconds
- alert: HighP90Latency
  expr: |
    histogram_quantile(0.9,
      sum(rate(traces_spanmetrics_duration_milliseconds_bucket{
        deployment_environment="prod"
      }[5m])) by (service, le)
    ) > 2000
  for: 5m
  labels:
    severity: warning

The important detail:

these metrics (traces_spanmetrics_calls_total, traces_spanmetrics_duration_milliseconds_bucket) are generated by the trace backend’s metrics-generator, not emitted by the application. The tracing pipeline receives spans and automatically computes RED metrics (Rate, Errors, Duration) from them. This means:

No application changes needed for alerting. Instrument once with OTel tracing, get both traces and RED metrics. Traces become the source of truth for service health. The same span that appears in a trace viewer also increments the counters that fire alerts. Consistency. The error rate on the dashboard, the alert threshold, and the trace search all agree because they derive from the same data.

The for: 10m duration on the error rate alert is deliberate - it filters out deploy-time spikes and transient errors. The P90 latency alert uses for: 5m because sustained high latency is usually a real problem, not a blip.

Phase 2: Service Health (1st half of the week)

Span-metrics from Tempo’s metrics-generator power per-service alerting. We calibrated thresholds using baseline data from the previous vendor - critical services got tight windows, batch processing got loose thresholds.

Plus latency alerts (avg, P90, P99) and consumer/cron error rates.

Phase 3: AWS Resources (2st half of the week)

YACE (Yet Another CloudWatch Exporter) scrapes CloudWatch metrics and exposes them as Prometheus metrics. Monitors across managed databases, load balancers, message queues, and serverless functions.

The cost difference is significant - YACE polls every few minutes, which is fine for infrastructure monitoring, and the API costs are a fraction of what the vendor charged.

discovery:
  jobs:
    - type: AWS/RDS
      regions: [us-east-2]
      metrics:
        - name: CPUUtilization
          statistics: [Average]
          period: 300
        - name: DatabaseConnections
          statistics: [Sum]
          period: 300

Each discovery.job tells YACE to auto-discover all resources of that AWS type and scrape the specified CloudWatch metrics. The period: 300 means 5-minute granularity which is the CloudWatch default resolution for most managed services and aligns with Prometheus scrape intervals.

The economics: each CloudWatch GetMetricData API call costs $0.01 per 1,000 metrics requested. At a 5-minute scrape interval, monitoring 50 metrics across RDS, SQS, ELB, and Lambda costs roughly a few dollars per month in API calls. The previous vendor’s CloudWatch integration was orders of magnitude more expensive for the same data. YACE exposes these as standard Prometheus metrics, so they slot into the same dashboards and alert rules as everything else.

Phase 4: Synthetics (3rd half of the week - weekend)

External probe nodes in multiple AWS regions, each running blackbox_exporter. HTTPS health checks every 30-60 seconds. Multiple regions distinguish “service is down” from “regional network issue.”

PagerDuty

Added Grafana Alerting integration alongside existing vendor integration - both ran in parallel during transition. Notification policy routes by service label. Integration keys in Secrets Manager.

Grafana Alerting vs Alertmanager

We use both, and they serve different purposes.

Alertmanager runs on the trace/metrics nodes alongside Prometheus. It evaluates Prometheus recording rules and alert rules, infrastructure-level alerts like node down, disk full, high CPU, etc. These are pure PromQL evaluated locally. Alertmanager handles deduplication, grouping, and silencing for these infrastructure alerts.

Grafana Alerting handles service-level alerts because it can evaluate PromQL against the metrics backend AND LogQL against the log backend in the same rule group. A single alert rule can check both “error rate > 5% (PromQL metric query)” and “specific error pattern appearing in logs (LogQL query).” This cross-signal capability is something Alertmanager alone can’t do.

During the migration, both the previous vendor and Grafana Alerting paged in parallel. Once we confirmed parity (same alerts, same timing, same routing), we cut over by disabling the vendor’s PagerDuty integration. The parallel period lasted about one week.

What We Gained and Lost

Gained: PromQL expressiveness, unified alerting across all signals, alerts as code in git, dramatically cheaper CloudWatch monitoring (cut from ~$1000 to ~$10).

Lost: Anomaly detection, forecasting, composite monitors, zero-maintenance SaaS experience.

Next: Session 9 - The Cost Breakdown