Skip to content

Commit

Permalink
Update details about logs retention
Browse files Browse the repository at this point in the history
  • Loading branch information
sandeepsukhani committed May 30, 2019
1 parent 18392ea commit cff6124
Showing 1 changed file with 6 additions and 2 deletions.
8 changes: 6 additions & 2 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,11 +59,14 @@ For more information about mixins, take a look at the [mixins project docs](http

## Retention/Deleting old data

A retention policy and API to delete ingested logs is still under development.
Retention in Loki can be done by configuring Table Manager. You need to set a retention period and enable deletes for retention using yaml config as seen [here](https://github.com/grafana/loki/blob/39bbd733be4a0d430986d9513476a91334485e9f/production/ksonnet/loki/config.libsonnet#L128-L129) or using `table-manager.retention-period` and `table-manager.retention-deletes-enabled` command line args. Retention period needs to be a duration in string format that can be parsed using [time.Duration](https://golang.org/pkg/time/#ParseDuration).

In the case of chunks retention when using S3 or GCS, you need to set the expiry policy on the bucket that is configured for storing chunks. For more details check [this](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html) for S3 and [this](https://cloud.google.com/storage/docs/managing-lifecycles) for GCS.

Currently we only support global retention policy. A per user retention policy and API to delete ingested logs is still under development.
Feel free to add your use case to this [GitHub issue](https://github.com/grafana/loki/issues/162).

A design goal of Loki is that storing logs should be cheap, hence a volume-based deletion API was deprioritized.
But we realize that time-based retention could be a compliance issue.

Until this feature is released: If you suddenly must delete ingested logs, you can delete old chunks in your object store.
Note that this will only delete the log content while keeping the label index intact.
Expand Down Expand Up @@ -185,3 +188,4 @@ create the table manually you cannot easily erase old data and your index just g
If you set your DynamoDB table manually, ensure you set the primary index key to `h`
(string) and use `r` (binary) as the sort key. Also set the "period" attribute in the yaml to zero.
Make sure adjust your throughput base on your usage.

0 comments on commit cff6124

Please sign in to comment.