The Observability Migration · 10 of 10
Lessons Learned
Ten sessions in: what I would do the same, what I would do differently, and whether it was worth it.
What Worked
S3 as the source of truth. When a Loki node lost its EBS volume, we wiped the WAL directory, restarted, and it re-synced from S3 and peers. Minutes. Zero data loss. This only works because all historical data is in S3, and local disk is treated as expendable.
The GitOps sync/restart split. Three incidents prevented: a YAML parse error, an invalid PromQL rule, a non-existent image tag. In every case, the sync pulled the bad config but the manual restart step caught it before it reached all nodes. Blast radius: zero or one node.
Span filtering. Significant memory and compaction improvement. All error paths preserved. The filter runs in the OTel Collector - per-span, not tail sampling. There’s minimal memory overhead and no trace incompleteness.
Dedicated node pools. Simple capacity planning. Isolated failure domains. No guessing about cross-service contention.
Signal correlation. Click a trace, see the logs from that request. Click a log line, jump to the trace. See the metrics that spiked at the same time. Logs, traces, and metrics linked by trace ID and time - all in one Grafana UI. Can you get this from a commercial vendor? Sure. But you’re paying for each signal separately, and correlation is the premium feature that justifies the premium price. With the LGTM stack, it’s just how the system works - Grafana links Loki, Tempo, and Mimir natively. No extra tier. No per-feature licensing.
What Hurt
The Mimir Data Gap
After moving Mimir to dedicated nodes, recent metric queries returned empty. query_store_after (default: 12h) tells the querier “ingesters have recent data, don’t check S3.” After migration, new ingesters didn’t have it. ignore_blocks_within (default: 10h) compounded it, the store-gateway skipped recent S3 blocks too.
Fix: both to 0s. Always check both paths.
Tempo Target Mode
target: all uses in-memory hash rings. Each node thinks it’s alone and it owns 100% of the ring. Duplicate trace storage, inconsistent queries. target: scalable-single-binary uses memberlist for proper ring coordination.
The documentation says “single-binary mode for small deployments.” Three nodes seemed small. Wrong mode.
Cardinality Explosion
Auto-instrumentation created span names per device endpoint - thousands of unique names. Tempo stored them fine, but the metrics-generator created a Prometheus series per unique name. The metric store started rejecting writes.
Transform processor upstream: collapse device-specific names into generic ones. Problem solved.
The Forgotten Remote-Write
Moved the metric store. Updated Prometheus remote-write. Updated K8s agent configs. Forgot Tempo’s metrics-generator remote-write. Tracing dashboards went flat - service maps, request rates, latency, all empty. Infrastructure and pod metrics worked fine. Just the trace-derived panels.
“No data” doesn’t look like an error.
Five Things We’d Do Differently
- Dedicated nodes from day one. We wasted time on shared nodes before WAL replay contention forced the split.
- Ingest data to dev environment first and run load test. We built development and production first and tested in production because development environment load is small.
- Pin every image tag. A patch release at 3am changed behavior. Now everything is pinned to exact versions.
- Map remote-write dependencies before migration. Multiple configs across multiple repos. Write the checklist before the first move.
- Monitor the monitor from day one. Remote-write success rate, ring health, cardinality growth, WAL replay duration. Every alert was added after the incident that proved we needed it.
Kill Your Bad Habits
Something we didn’t expect: the hardest part of migration wasn’t the infrastructure. It was changing how engineers use observability.
On CloudWatch, nobody thought about query cost. On a self-hosted stack, that’s a compaction storm and a disk I/O spike that affects everyone. We kept finding dashboards that derived metrics from logs - scanning thousands of lines to calculate error rates. That’s the wrong signal. Error rate is a metric. Emit it as a metric.
The rule is simple: metrics for aggregate numbers, logs for context and detail, traces for request-level debugging. When you own the stack, you feel the cost of doing it wrong and that makes you a better engineer.
Self-hosting puts someone on the team in the position of preventing cardinality explosions, managing retention, watching query patterns. That responsibility changes how the whole team thinks about observability.
Full Circle
Self-hosting observability isn’t just about saving money. It’s about having the depth to operate what you build as a team.
The Summary
Cost: vendor contract + overages → predictable monthly fixed cost Log retention: CloudWatch (limited) → multi-year (S3 lifecycle) Trace retention: limited by vendor → extended (S3) Metric retention: fragmented → multi-year (unified in Mimir) Vendor lock-in: high → none (OTel standard) Predictability: low → high Operational cost: zero → ~ well, we will find out
It was the right trade. Predictable cost, better retention, full control, and the satisfaction of operating a system we actually understand.
Thanks for following all 10 sessions. If you’re building something similar, I hope the scars are as useful as the wins.