The Architecture

Right Tool for the Job

The workload is a handful of long-lived services with persistent local storage and S3 backends. That’s not a Kubernetes use case. There’s no horizontal pod autoscaling, no rolling deployment of 50 replicas, no service mesh routing.

EC2 + Docker Compose + systemd timers. Simple to reason about, simple to operate, simple to price. SSH in, docker logs, restart a container. The operational model matches the workload, and the cost is predictable because EC2 pricing is straightforward.

Dedicated AWS account. Separate IAM, billing, blast radius. VPC peering to application environments, so all traffic stays private.

Network Topology

• :4317 for gRPC OTLP (traces from application OTel SDKs)
• :4318 for HTTP OTLP (traces from environments that can’t do gRPC)
• :3100 for the Loki push API (logs from OTel Collector gateways)
• :9009 for Mimir remote-write (metrics from Prometheus agents and Tempo’s metrics-generator)
• :9090 for the Prometheus query API (for Grafana to query infrastructure metrics)

Each port routes to a different set of backend nodes: traces go to the trace nodes, logs to the log nodes, metrics to the metric nodes. One NLB, multiple target groups, clean separation.

Why NLB instead of ALB? gRPC. gRPC multiplexes many logical streams over a single HTTP/2 connection. An ALB terminates HTTP/2 at the load balancer and opens separate HTTP/1.1 connections to backends, which breaks gRPC stream semantics entirely. The connection pooling that makes ALB efficient for HTTP REST traffic is exactly what destroys gRPC performance. NLB operates at L4, passing TCP connections through transparently. The gRPC client’s HTTP/2 connection lands directly on the backend node. No protocol translation, no broken streams.

VPC peering connects each application environment to the observability VPC. Security groups open ports 4317, 4318, 3100, and 9009 from the peered CIDRs. All telemetry traffic stays on private IPs, so nothing touches the public internet. The peering connections are simple point-to-point links, no transit gateway needed for this topology.

Why Not One Big Instance?

You can run the entire LGTM stack on a single instance. Grafana even ships a grafana/otel-lgtm Docker image that does exactly this. It works. For dev, for testing, for demos, it’s great.

But for production, a single instance means a single failure domain. One bad deploy, one OOM, one disk issue, and your entire observability goes dark at the exact moment you need it most. The tool you rely on to debug problems can’t have the same availability characteristics as a toy. If it’s not reliable, it’s not a tool. It’s a liability.

The Split: Why Dedicated Node Pools

We started with shared “observability nodes”: Loki, Tempo and Prometheus all on the same instances. Two problems:

WAL replay contention. Both Tempo and Loki replay Write-Ahead Logs on restart. Both spike CPU and memory. On the same box, they fight for resources. Startup times tripled. We got OOM kills.

Disk I/O contention. Both compactors run periodic disk-heavy operations. On shared EBS, one service’s compaction storm causes latency spikes in the other.

The fix was obvious in retrospect: each signal gets its own hardware. Tempo shares with Prometheus and OTel Collector (complementary profiles, since one is memory-heavy and the others are CPU/network-heavy). Loki and Mimir get dedicated nodes.

S3 as the Source of Truth

Every LGTM component uses the same storage pattern:

1. Write to WAL on EBS for immediate durability
2. Flush to S3 when chunks close, blocks compact and data ships
3. Query from S3, since all historical reads come from the object store
4. EBS is scratch. Lose it, wipe it, restart. Node rebuilds.

S3 with 11 nines of durability is the backend. Local disk is a buffer. Node recovery is “wipe and restart,” not “restore from backup.”

Replication

Logs (Loki): RF=3 → tolerates 1 node down, zero data loss Traces (Tempo): RF=2 → tolerates 1 node down Metrics (Mimir): RF=1 + HA dedup → write redundancy via multiple Prometheus agents

S3 Gateway Endpoint

A critical cost optimization that’s easy to overlook. Without a VPC Gateway Endpoint for S3, every byte Loki, Tempo, and Mimir read from or write to S3 would route through the NAT instances. NAT data processing charges add up fast, and an observability stack moves terabytes of data to and from S3 every month (chunk flushes, compaction reads, query fetches, block uploads).

The S3 Gateway Endpoint is a VPC-level route table entry that sends S3 traffic directly to the S3 service endpoint within the AWS network. Zero data transfer charges. Zero NAT bandwidth consumption. Zero additional latency. It’s a single Terraform resource (aws_vpc_endpoint with type = “Gateway”) that must exist before any other resource in the stack, which is why it’s the first item in the Terragrunt dependency chain.

For an observability platform, this isn’t optional. It’s foundational. Without it, your NAT bill alone could approach what you’re trying to save by self-hosting.

Replication

• Logs (Loki): RF=3, tolerates 1 node down with zero data loss
• Traces (Tempo): RF=2, tolerates 1 node down
• Metrics (Mimir): RF=1 plus HA dedup, write redundancy via multiple Prometheus agents

The replication factors aren’t arbitrary. They reflect the economics and criticality of each signal type.

RF=3 for Loki means every log entry is written to all 3 ingesters. The write quorum is 2 of 3, so one node can fail completely and writes continue without interruption. The trade-off is 3x write amplification, because every log line consumes 3x the ingestion bandwidth and 3x the WAL disk. This is acceptable for logs because durability matters most here: logs are the forensic record, often the only way to reconstruct what happened during an incident, and they may need to satisfy compliance retention for years.

RF=2 for Tempo means each span goes to 2 of 3 nodes. Lower write amplification because trace volume is typically higher than log volume (a single request generates dozens of spans) and individual spans are less critical than individual log lines. What matters for traces is the trace parent, the root span that ties the whole request together. Losing one leaf span in a 40-span trace doesn’t meaningfully degrade the debugging experience. RF=2 strikes the right balance between durability and write cost for this signal type.

RF=1 for Mimir with HA dedup takes a different approach entirely. Instead of replicating at the storage level, redundancy comes from the write path: multiple Prometheus agents (one per cluster, plus the metrics-generator) independently scrape and remote-write the same metrics. Mimir deduplicates using cluster and __replica__ labels and it identifies samples from different agents as duplicates and keeps only one copy. This gives write-path redundancy without storage-level replication. It’s simpler and cheaper than RF=3 for the metric volume involved, and it means losing a single Prometheus agent doesn’t cause data gaps because the other agents are writing the same samples.

Network

NLB (not ALB, since we needed L4 for gRPC) health-checks each backend’s /ready endpoint. Node goes down, NLB routes around it in 2 seconds.

Infrastructure as Code

Terraform modules, Terragrunt for dependency ordering and DRY config. A full run-all apply provisions everything from zero.

The infrastructure is the easy part. The interesting decisions are in the service configuration, which is what the next 8 posts cover.

*Next: Session 3. Logs: From CloudWatch to Loki