Docker Tempo container fails due to OOM (Out of memory) when searching traces in Grafana, returns 502 Bad Gateway

[nikhil-ht]

I’m running a Tempo container for distributed tracing, and I'm encountering an issue where the container fails due to an out-of-memory (OOM) error. This happens when someone searches for a trace in Grafana, and if the parent trace is not received(Root span not received yet), the Tempo container stops and returns the following error:

Failed to get trace with ID: 4e6db4e64ba01c2f192b044b3a540cad. Status: 502 Bad Gateway. Body: 502 Bad

Tempo version: 2.4

current memory allocated for tempo is 4GB

Below is my tempo config file

server:
  http_listen_port: 3200

distributor:
  receivers:
      otlp:
        protocols:
          http:
          grpc:

compactor:
  compaction:
    block_retention: 48h

storage:
  trace:
    backend: s3
    s3:
      region: ${AWS_REGION}
      bucket: ${AWS_BUCKET_NAME}
      endpoint: ${AWS_S3_ENDPOINT}
    wal:
      path: /opt/tempo/wal
    local:
      path: /opt/tempo/blocks
      
overrides:
  defaults:
    ingestion:
      max_traces_per_user: 0
      max_global_traces_per_user: 0

I have review the existed container by inspecting docker logs getting below error in log

level=error ts=2024-09-06T08:55:20.716948657Z caller=retention.go:71 msg="failed to mark block compacted during retention" blockID=5a343b1e-b9e2-4aa1-9f68-4d81b34f55a1 tenantID=single-tenant err="error copying obj meta to compacted obj meta: The specified key does not exist."

[joe-elliott]

This is likely unrelated to the root span not yet received and simply due to the size of the trace.

We have a trace summary cli command that will print data about a trace id to determine roughly how large it is and how many blocks it spans.