DŠ

The Observability Migration · 3 of 10

Logs: From AWS CloudWatch to Loki

Moving logs off CloudWatch onto Loki: label design, chunk storage on S3, and what queries cost once you own them.

Why CloudWatch Wasn’t Enough

CloudWatch Logs works. But “works” and “works well at scale” are different things. Cross-service log correlation required multiple tabs and manual timeline stitching. Query costs scaled with volume. And retention options were “keep forever and pay” or “delete.”

We needed logs from all services in one place, queryable with one language, with retention that didn’t require constant cost management.

Loki’s Model

Traditional log systems index every field at write time, which gives powerful queries and expensive storage. Loki flips this: index only labels, store compressed chunks in S3, brute-force grep at query time. For operational use, say “show me errors from service X in the last hour”, it’s fast enough. That trade-off is correct 95% of the time.

Loki Configuration

schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: loki_index_
        period: 24h

limits_config:
  ingestion_rate_mb: 80
  per_stream_rate_limit: 20MB
  max_streams_per_user: 50000
  max_query_length: 168h  # 7 days
  max_entries_limit_per_query: 10000
  max_query_bytes_read: 40GB
  required_labels:
    - service_name
  allow_structured_metadata: true
  retention_period: 2555d  # ~7 years default

A few things worth explaining:

Schema v13 with TSDB is Loki’s latest storage format. It supports structured metadata, which means you can attach key-value pairs to log entries that are queryable but don’t create new streams. This is the escape hatch for when you need something between “label” (indexed, creates streams) and “log line content” (requires parsing at query time).

required_labels: [service_name] is the enforcement mechanism for label discipline. Loki rejects any push request that doesn’t include service_name as a label. This isn’t a convention or a best practice. It’s a hard gate. If your OTel Collector pipeline doesn’t set this label, your logs are dropped with a 400 error. Catching misconfiguration at ingestion time is cheaper than debugging missing logs later.

ingestion_rate_mb: 80 is per-tenant (we run single-tenant). This caps the total ingestion rate per node. If a runaway service starts dumping massive logs, this limit prevents it from overwhelming the ingester. The 80MB/s ceiling is generous for normal operation but provides a safety valve.

max_query_length: 168h caps queries at 7 days. Without this, a user could accidentally query a year of logs and OOM the querier. Seven days covers most operational debugging; anything older usually means you’re doing forensics and should be more targeted about time ranges.

Data Lifecycle

App → OTel/Alloy → Loki Push API (:3100)

→ WAL on EBS (immediate durability, 5-min checkpoints)

→ In-memory chunk (accumulating)

→ Flush to S3: chunk full (256KB), idle 15min, or max age 30min

→ S3 lifecycle tiering

RF=3 means every log goes to all 3 ingesters. Lose one node: zero data loss, NLB routes around it. Lose the EBS volume: wipe the WAL, restart, re-sync from S3 and peers.

Label Discipline

The #1 mistake migrating to Loki is treating labels like Elasticsearch fields.

In CloudWatch or Elasticsearch, everything is searchable by default. In Loki, labels create streams. level (5) x service (30) x env (3) x component (20) = 9,000 streams. Add method = 45,000. Add path = game over.

Three labels. service_name (enforced, so Loki rejects logs without it), deployment_environment, cluster. Everything else is | json at query time.

LogQL Examples

# Errors from a specific service in the last hour
{service_name="my-api", deployment_environment="prod"} | json | level="error"

# Slow requests over 2 seconds, reformatted for readability
{service_name="my-api"} | json | duration > 2000 | line_format "{{.method}} {{.path}} {{.duration}}ms"

# Count errors per service over time (metric query)
sum by (service_name) (count_over_time(
  {deployment_environment="prod"} | json | level="error" [5m]
))

The first two are “filter” queries; stream selector narrows by label, then | json parses the log line and filters on extracted fields. Fast when the stream selector is selective. Slow when it isn’t and {deployment_environment=“prod”} without a service_name hits every stream in the environment.

The third is a “metric” query count_over_time with | json scans every log line in every matching stream, parses it as JSON, filters on level, and counts matches in 5-minute windows. This is expensive on high-volume services because it’s a full scan. For dashboards that show aggregate error rates, use span-metrics from the tracing pipeline instead, they’re pre-computed counters that Prometheus can query in microseconds. Save LogQL metric queries for ad-hoc investigation, not continuous dashboard panels.

This connects to a broader principle: the right tool for aggregate numbers is metrics, not logs. Logs are for “show me what happened.” Metrics are for “show me how often.”

Per-Stream Retention

Not everything needs the same retention. The Loki compactor applies overrides: dev/staging get shorter retention of 30 days.

High-volume device logs and edge logs get even shorter. Production application logs get the full retention via S3 Glacier tiering.

The Glacier Gotcha

Loki can’t read Glacier objects directly. Archived logs need an S3 restore request first (hours, not seconds). We’ve done this a couple of times and it works fine. For daily ops: queries hit recent data, always instant.

How Logs Get to Loki

Applications don’t push to Loki directly. The OTel Collector runs as a DaemonSet on every application cluster node; one collector instance per host. Application pods emit OTLP-formatted logs to the local collector over localhost (no network hop). The collector batches, compresses, and forwards to Loki’s push API (`:3100`) over the VPC peering connection.

The critical detail: the collector buffers to local disk. If Loki is unreachable due to maintenance, network blip, all nodes down then the collector queues data in a persistent file-based buffer and retries with exponential backoff. The retry is infinite. The collector never drops data; it applies backpressure to senders only when disk buffer is exhausted.

This architecture means the DR picture is better than Loki’s RF alone would suggest. Loki going down doesn’t mean log loss, it means log delay. The buffer bridge covers maintenance windows, restarts, and even multi-node failures.

Disaster Recovery

There’s an important layer that changes the DR picture: the OTel Collector Gateway runs as a DaemonSet on every EKS node (where apps are running), buffering logs and traces to local disk. If Loki (or Tempo) goes down, data spools on the gateway and drains automatically when the backend recovers. Retry is infinite, so the gateway never drops data.

This means the DR matrix is better than it looks:

→ 1 node down: zero data loss (RF=3), NLB routes around automatically

→ 2 nodes down: logs lost during outage, fix easiest node first

→ All 3 down: up to 30 min unflushed data, sequential restart 1 → 2 → 3

  • EBS corrupt: only unflushed WAL lost, wipe and rebuild from S3

S3 is the source of truth. Everything else is a buffer.

Next: Session 4. Traces: Span Filtering

All writing · Get in touch