Writing
Notes from the platform.
Mostly observability and infrastructure — what we built, what it cost, and what I would do differently.
Series · 10 parts · 40 min
The Observability Migration
We replaced a commercial APM and an AWS CloudWatch log bill with a self-hosted OpenTelemetry and Grafana stack. This is the whole account, in order: the architecture, each signal in turn, the migration, the bill, and the parts that went wrong.
- OpenTelemetry Changed the GameWhy vendor-neutral instrumentation turned the monitoring backend from a marriage into a decision we can revisit.
- The ArchitectureWhy we ran the LGTM stack on EC2 with dedicated node pools per signal, and how we designed out the single points of failure.
- Logs: From AWS CloudWatch to LokiMoving logs off CloudWatch onto Loki: label design, chunk storage on S3, and what queries cost once you own them.
- Traces and Span FilteringGetting trace volume under control with tail sampling and span filtering, without losing the traces worth keeping.
- Metrics: Mimir Over PrometheusPrometheus scrapes well but stores poorly. Why Mimir's S3 block storage replaced a fleet of unreplicated Prometheus instances.
- OTel InstrumentationWiring the OpenTelemetry SDK into real services: resource attributes, batch processors, and the settings that actually matter.
- GitOps Deployment ModelTerraform modules, configuration in Git, automated sync - running an observability stack you can rebuild from scratch.
- Migrating MonitorsPorting hundreds of alerts into a new query language, and the remote-write we forgot to move.
- The Cost BreakdownWhat the self-hosted stack costs per month, line by line, against the bill it replaced.
- Lessons LearnedTen sessions in: what I would do the same, what I would do differently, and whether it was worth it.