Add support for OpenStack Swift storage

[tlvenn]

It would be great if object_store could use OpenStack Swift containers.

[adityacs]

@tlvenn Loki supports any object storage with S3 compatible APIs. Swift supports S3 APIs as mentioned here.

Also, please refer S3 section in this doc

[tlvenn]

Yes I know but this middleware is optional and some cloud providers choose not to enable it, OVH for example as far as I know.

[adityacs]

@tlvenn Loki is dependent on Cortex for storage implementation. Could you please raise an issue in cortexproject/cortex ?

[tlvenn]

Sure will do. Quick question related with Cortex. Is it on the roadmap to be able to have 2 stores, one for recent data only that could be co located with the service and one long term to store all the log that would leverage S3 and the like ?

[tlvenn]

Or does Cortex support both Short & Long term storages ? On the surface, it seemed to be targeted at the latter.

[tlvenn]

@tomwilkie, it would be great when you have the time to shed some light on this, thx in advance !

[sh0rez]

@tlvenn at least regarding Loki, you have this behaviour (kind of):

The Ingester component, which is responsible for storing data, is a stateful service which keeps the last ~12h of working data in memory, to run optimizations and similar things.

This data is continuously written in batches to the chunk store (bigtable, cassandra, s3 etc.) and a reference to it is stored in the index store (another bigtable, cassandra etc).

See the cortex docs for more details on this:

• Ingester: https://github.com/cortexproject/cortex/blob/master/docs/architecture.md#ingester
• Chunk Store: https://github.com/cortexproject/cortex/blob/master/docs/architecture.md#chunk-store

They basically apply for Loki as well, just keep in mind Loki is dealing with a much bigger data volume than Cortex.

Does that help?

[adityacs]

@sh0rez Any plans to support Swift storage? If yes, I can work on this. Pl let me know.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[Madko]

any news on this ?

[cyriltovena]

Started something here https://github.com/cyriltovena/cortex/tree/swift-openstack

[cyriltovena]

PR open cortexproject/cortex#2440 there is still some work but the biggest is done.

[Madko]

ah great news ! Thank you Cyril

[Morishiri]

@cyriltovena is it possible to add support for it to the helm chart as well? Currently I don't see the option to configure that. Shall I open an issue/PR?