DŠ

The Observability Migration · 4 of 10

Traces and Span Filtering

Getting trace volume under control with tail sampling and span filtering, without losing the traces worth keeping.

Thinking Like an APM Engineer

At Instana, I saw how trace pipelines work from the inside: ingestion, indexing, storage, query. Every span has a cost at every stage. The art is keeping the spans that matter and dropping the ones that don’t, without breaking trace continuity.

With the LGTM stack, we have full control over this pipeline. The OTel Collector sits between applications and Tempo, and spans can be processed however we want.

The Pipeline

Apps → OTLP gRPC → NLB → OTel Collector

Pipeline 1 (ingress):

→ Normalize high-cardinality span names

→ Load-balance by traceID to correct Tempo node

Pipeline 2 (routed):

→ Filter noisy leaf spans

→ Batch → Export to local Tempo

Tempo → WAL → S3

Two pipelines: normalize before routing (applies globally), filter after routing (close to Tempo). Order matters.

Here's the actual filter processor configuration:

filter/drop_noisy_spans:
  error_mode: ignore
  traces:
    span:
      - 'name == "GET /heartbeat/"'
      - 'attributes["db.system"] == "mysql" and status.code != STATUS_CODE_ERROR and duration < 150000000'
      - 'attributes["db.system"] == "redis" and status.code != STATUS_CODE_ERROR and duration < 20000000'

A few things that aren’t obvious:

Duration is in nanoseconds. OTTL (OpenTelemetry Transformation Language) uses nanosecond precision. 150ms = 150,000,000ns. 20ms = 20,000,000ns. Easy to get wrong by a factor of 1000 if you think it’s microseconds. error_mode: ignore is critical. If a span doesn’t have the db.system attribute (say, an HTTP span), the rule referencing that attribute would normally error. ignore means “skip the rule for this span” rather than “reject the span” or “crash the pipeline.” Without this, non-DB spans get dropped or the processor panics. Each condition is AND-ed within a rule, OR-ed across rules. A span matching any rule gets dropped. Within the MySQL rule, all three conditions (db.system, status, duration) must be true.

The Filter

Health checks (specific endpoints): ~5% dropped Fast DB queries (success + < 150ms): ~30% dropped Fast cache (success + < 20ms): ~25% dropped Total: ~60% of spans dropped

Always kept: all errors, slow queries, HTTP parents, message queue spans, application logic. This is a per-span filter, not tail sampling, so there is minimal memory overhead, no latency penalty and no incomplete traces.

Why These Thresholds?

150ms for MySQL and 20ms for Redis aren’t magic numbers. They came from P90 analysis of production traffic.

For MySQL: the vast majority of successful queries complete under 150ms. Anything above that is worth investigating. It could be a missing index, a full table scan, or lock contention. By keeping all slow queries regardless of status, you get a natural “slow query log” built into your traces without configuring MySQL’s own slow query logging.

For Redis: it’s an in-memory store. Sub-millisecond responses are normal. When a Redis command takes more than 20ms, something is wrong: network issues, large payloads (`MGET` with thousands of keys), or a blocking command on the server. 20ms is generous; in a healthy cluster, P99 is usually under 5ms.

The key insight: profile your own workload before choosing thresholds. Run a query against your trace backend for duration distributions by db.system and pick the inflection point where “normal” ends and “interesting” begins. Our numbers reflect our query patterns, and yours will differ.

The Cardinality Trap

Auto-instrumentation creates span names from API calls. Most are fine. But one instrumentor created names per device endpoint, thousands of unique names.

Tempo stored them fine. But Tempo’s metrics-generator creates a Prometheus time series per {service, span_name, status}. Thousands of span names = thousands of series = the metric store exceeds cardinality limits and rejects writes.

We caught this by tracing the full chain: span → metrics-generator → Prometheus series → Mimir cardinality. A transform processor collapses device-specific names into 2 generic names. Fixed.

Trace-ID Routing

The OTel Collector’s load balancing exporter hashes by trace ID, routing all spans from one trace to the same Tempo node. Better assembly, fewer cross-node queries.

Metrics from Traces

Tempo’s metrics-generator creates RED metrics (rate, errors, duration) from traces and remote-writes them to Mimir. This powers service maps, request rate panels, and latency dashboards.

If this remote-write breaks, tracing dashboards go empty, not because traces are lost, but because the derived metrics stop flowing. We learned this the hard way during a migration (Session 10).

Tempo Config

scalable-single-binarymode (NOT all, which forces in-memory rings and silently breaks multi-node). RF=2, S3 retention. This mode distinction cost us days of debugging, also covered in Session 10.

Key storage settings worth understanding:

compactor:
  compaction:
    compaction_window: 4h
    max_block_bytes: 536870912  # 512 MB
    max_compaction_objects: 6000000
    block_retention: 0s  # handled by S3 lifecycle
storage:
  trace:
    blocklist_poll: 5m
    wal:
      encoding: snappy
    block:
      encoding: zstd
      bloom_filter_false_positive: 0.01

compaction_window: 4h controls how aggressively Tempo merges blocks. Tempo looks at blocks within this window and merges overlapping ones. Too small = excessive S3 API calls from constant small merges. Too large = large merge jobs that take a long time and hold memory. 4h is a balance, small enough for reasonable query performance over recent data, large enough to avoid API cost spiraling. max_block_bytes: 512MB caps individual block size. Larger blocks mean fewer S3 objects (cheaper) but slower queries when you only need a slice. 512MB keeps per-block query time reasonable. max_block_duration: 30m (set at the ingester level) limits how much time a single block covers before being flushed. This prevents blocks from spanning too wide a time range, which would force the querier to read entire blocks for narrow time queries. block_retention: 0s means Tempo itself doesn’t delete blocks, since we handle retention via S3 lifecycle policies. This separation of concerns means storage tiering (Standard -> IA -> Glacier) happens at the infrastructure layer, not the application layer. WAL encoding: snappy for speed (WAL is write-heavy). Block encoding: zstd for compression ratio (blocks are read-heavy from S3, smaller is cheaper and faster).

Sampling Strategy

We run 100% collection, with no head-based sampling, no probabilistic sampling at the SDK level. Every trace from every request reaches the collector.

This is a deliberate choice. The span filter in the collector is not sampling. The distinction matters:

Sampling drops random traces. You lose a percentage of everything uniformly: errors, slow requests and normal requests. You can reason about rates statistically, but you can’t investigate any individual request unless it was sampled. Filtering drops predictable, low-value leaf spans. All errors are kept. All slow operations are kept. All trace parents (the HTTP request spans that form the trace backbone) are kept. You lose nothing you’d investigate.

The result: any request that caused an error or was slow has a complete trace. Any request that succeeded quickly has a trace with the parent spans intact and the fast DB/cache leaf spans trimmed. You can still see the request happened, how long it took, and which services it touched. You just don’t see every 2ms Redis GET.

If your volume requires sampling, do it at the SDK level with ParentBasedTraceIdRatio so child spans follow the parent’s decision. But try filtering first, because you might not need sampling at all.

*Next: Session 5. Metrics: Mimir Over Prometheus

All writing · Get in touch