Add data infra exporter and data infra events topic #405
Conversation
martskins
force-pushed
the
add-data-infra-exporter
branch
2 times, most recently
from
b599627
to
22db3fe
Compare
martskins
force-pushed
the
add-data-infra-exporter
branch
from
22db3fe
to
a9c64f8
Compare
|
|
||
| config = { | ||
| "remote.storage.enable" = "true" | ||
| "retention.bytes" = "-1" |
There was a problem hiding this comment.
Just to confirm this config... you'll need infinite retention on it, right?
There was a problem hiding this comment.
to be honest, this one is a new topic and it's for the data infra integration. Maybe it could do with less than infinite. I default to infinite because I'm concerned we'll later find out that actually we need to reconsume everything for whatever reason, but maybe there's a saner default for this topic specifically, considering it's specifically for this data infra use-case.
There was a problem hiding this comment.
Is it possible to find out for how long is UW required to keep this data?
We could also go with infinite and revisit later, but it would be good to settle this from the get go.
There was a problem hiding this comment.
the data will be kept in braze anyways, so how long it lives in the topic will not necessarily match that I suppose. I've changed it to a month so that if something breaks we have plenty of time to fix it without risk of losing messages. I've asked data infra whether they have any recommendations for this value though, so won't merge just yet, but I guess a month is a saner value than forever for this topic.
There was a problem hiding this comment.
I haven't received an answer from data-infra yet, but given this is dev I'm happy to just stick with 1 month for it. We won't even have much data going through this one in dev anyways.
No description provided.