Storage both locally and in object storage

[bentqc]

Hi,

I did noticed in the documentation that it is currently not possible to use local storage and object storage in the same Tempo deployment.

What i would be interested is to store locally for a certain period of time, lets say two days, then move what is older to long term storage so that i can retain 30 or 60 days of data.

Is there any plans to support such thing and/or is there any recommendation on how i can achieve the use case above mentioned ?

thanks.

[joe-elliott]

This is technically possible with Tempo as its written today. However, you may see search latency issues in your ingesters with 2 full days of traces in them depending on volume. If you would like to do this:

This will tell the ingesters to hold blocks for 48h after being flushed to the backend.

ingester:
  complete_block_timeout: 48h

Something like these settings can be used to tell the queriers to rely on the ingesters for the first 48 hours of data and not look at the backend.

query_frontend:
  search:
    query_backend_after: 47h30m
    query_ingesters_until: 48h

I'm not sure this makes much sense. It will require quite a bit more block storage in your ingesters and I think, overall, search performance will be worse due to the amount of data kept there.

[bentqc]

thank you, i did realized after posting this that our search on long term storate were pretty quick and did see the performance degradation you mentioned when using large retention on local storage.