The Observability Migration · 9 of 10
The Cost Breakdown
What the self-hosted stack costs per month, line by line, against the bill it replaced.
Why Predictability Matters
Usage-based pricing means the bill lags behind consumption. By the time changes show up, overages have accumulated. You can’t budget for observability when the cost is coupled to activity.
What We Pay Now
The self-hosted infrastructure is a fixed monthly cost:
Compute: dedicated nodes per signal type (Loki, Tempo, Mimir), Grafana, NAT instances, synthetic probes Storage: EBS for WAL and scratch, S3 for long-term retention with lifecycle tiering Network: VPC peering from application environments, cross-AZ transfer Security: AWS KMS encryption keys, AWS Config, CloudTrail
Similar number every month. No overages.
With Reserved Instances on the larger nodes, the compute cost drops further.
The Engineering Investment
One-time build: ~4 weeks, 1 engineer.
Week 1: Terraform/infrastructure Week 2: Service configuration + testing Week 3: OTel instrumentation Week 4: Dashboards + alert migration
What’s NOT in the Infrastructure Cost
Being transparent: engineering time, opportunity cost (that time could go to features), and the learning curve (Loki/Mimir/Tempo internals are concentrated knowledge that needs documentation).
Cost Depends on Criticality
This is the part most blog posts skip: our setup is sized for a distributed system that other systems depend on. Replication, multiple AZs, dedicated node pools, disk-buffered gateways, it is all because we can’t afford to lose telemetry during an incident.
Not everyone needs that.
If you’re monitoring a small business app with office hours 9-17 and acceptable downtime, the entire LGTM stack runs on a single EC2 instance. Loki, Tempo, Mimir, Grafana - one box. Grafana even ships a Docker image for exactly this (grafana/otel-lgtm). You could run production observability for under $100/month and accept the trade-off: if the instance dies, you lose recent data and fly blind until it’s back.
That’s fine for some use cases. The stack is the same. The architecture scales with how much you care about losing telemetry:
Don’t care much: single instance, no replication, accept gaps - under $100/month Care somewhat: 3 nodes per signal, RF=2 or RF=3, S3 backend, recover from node failures - mid hundreds/month Can’t afford to lose anything: dedicated nodes, disk-buffered gateways, multi-AZ, NLB health checks is what we built
The LGTM stack doesn’t force you into one tier. You choose the reliability that matches your system’s criticality.
When Self-Hosting Doesn’t Make Sense
If you don’t have an engineer who understands distributed systems internals - the WAL replay, the ring coordination, the compaction lifecycle - you’re buying a product without the team to maintain it. The infrastructure cost doesn’t include the expertise.
This worked because I have the background and the operational discipline.
Next: Session 10 - Lessons Learned