Thanos receive component umbrella issue

[brancz]

This is an umbrella issue to track the development of the accepted proposal for the Thanos receive component.

• remote-write protocol accepting API *: Add initial remote-write-receive component #811
• upload blocks produced by local tsdb receive: Add block shipping #1011 receive: Add retention flag for local tsdb storage #1046
• hashring for sharded storage (as a start to not be blocked by the next item, we expect that all requests are sent to all receive nodes and they persist only the time-series assigned to them) pkg/receive: basic hashring support #1217
• split and forward requests for time-series to receive nodes according to hashring pkg/receive: enable forwarding metrics #1243
• create mechanism to force flush and upload, while not accepting writes (necessary for hashring resizing)
• announce multiple sets of external labels through StoreAPI info response *: Allow exposing multiple label-sets #1284 (this does not only influence the receive component, but will also allow sharding distinct sets of blocks by the store component as well, in addition to time sharding as introduced by Store: Add time & duration based partitioning #1077)
• add management of multiple tsdb instances based on write request header input
• dynamo style replication and read/write quorum. This was only part of the "Open Questions" in the design doc, but it is becoming increasingly clear that this will need to be necessary, but likely still the last thing to implement. receive: Replication #1270
• component documentation

Let me know if any of this is unclear or needs more detail, should the design doc not have enough.

@bwplotka @domgreen @povilasv @metalmatze @squat @mxinden @paulfantom @s-urbaniak

[bwplotka]

I think it would be nice to also document the receive like we do for other commands here: https://thanos.io/getting-started.md/

[dawidmalina]

@bwplotka what is actual state of the receive implementation? I mean how far we are from having this ready for wider audience (at least for test purpose)?

[brancz]

@dawidmalina the issue is up to date. When everything is implemented I think it's ready for a wider audience to test. There will still be a number of features we are going to want to implement, but I think then all the things featured in the original design doc will be done.

[dawidmalina]

Good to hear that 👍 but I'm still suspicious about definition of the word "everything". According to the number of todo tasks attached to this ticket we have only one topic related to management of multiple tsdb instances. Is this the only one to consider receive implementation "ready for a wider audience" or we still rather stay away and wait until other features will be implemented?

[brancz]

Sorry I meant everything on this list. “Other features” refers to things that we may only discover as we use this more intensely :)

[bwplotka]

I think we should slowly consider making the receiver a non-experimental component.

I know multi-tsdb support is still in progress, plus docs (edited the description to add that point), but we are having feedback that receiver is quite stable for users already: https://cloud-native.slack.com/archives/CK5RSSC10/p1575471712121900 so worth to either reconsider making in stable now with some docs or to helpe with multi-tsdb work (:

[brancz]

I've been working on the multi-tsdb work for some time, and I don't think there is a way to introduce it completely disruptionless, so I would prefer to not remove the experimental naming/labeling until we have the multi-tsdb work and have it sink in for a little bit. That said, we do already use this in prod, so while there may be manual migrations necessary, it's unlikely to break very badly at this point.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kakkoyun]

Still in-progress.

[stale]

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

[bwplotka]

Stale bot, go away for now (:

…

On Thu, 27 Feb 2020 at 11:14, stale[bot] ***@***.***> wrote: This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1093?email_source=notifications&email_token=ABVA3O7RHS4EHABHMWDUIADRE6OCTA5CNFSM4HJEDGSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEND65QA#issuecomment-591916736>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVA3OZORR6NG3QAN5GTP63RE6OCTANCNFSM4HJEDGSA> .

[stale]

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

[brancz]

Actively working on finishing this :)

[dinesh4747]

@brancz - When can we expect Thanos Receiver docs and General availability just curious

[brancz]

The last feature feels fairly close to completion: #2012

After that we need to write comprehensive docs and I am expecting one configuration change in the hashring configuration, but I think it can be done in a backward compatible way.

No promises, but I don't think it will be too long.

[stale]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[bwplotka]

I guess last part is missing only: Docs (:

[brancz]

We also know of some limitations on the way that series are routed through the hashring, causing every request to translate into N internal requests, which makes a hashring only scale to the number of requests a single node can handle (in terms of parsing, so CPU bound). We need to come up with a mechanism to split up requests across multiple hashrings other than just by tenant ID. I think we might be able to do that in a backward compatible way but we should think about it.

[kakkoyun]

For the docs #1093

[kakkoyun]

Initial component documentation is done. Also it's no longer marked as experimental. Shall we close this one?

[brancz]

Initial implementation is done. This component is being used a lot in various production environments already, so it's been battle tested in various ways already. That said there are two main things that I would like to get agreement on before taking it our of experimental state:

• WIP: docs/proposals/receive: Improved routing and distribution config #2675
• proposal: Add scalable rule storage proposal #2661

That said, I think this umbrella issue can be closed as all the initial features have been implemented. Thanks to everyone involved in this, it's been ~1.5years worth of work.