Add support for ES index aliases / rollover to the dependency store (Resolves #2143) #2144

frittentheke · 2020-03-30T14:30:00Z

Which problem is this PR solving?

Currently the dependency store has no support for index aliases / rollover indices.
Resolves #2143

Short description of the changes

Use the already existing and used config variable UseReadWriteAliases to switch between the current behavior and using non-dated "-read" and "-write" index names. This aligns with the behavior that is already in place for spans and services.

pavolloffay · 2020-03-30T16:31:56Z

You have cracked an issue that will require some more work :)

Remove dependency mapping from go code and move it to JSON in mappings directory. Then install index template at Jaeger startup like we do for main indices.
Change rollover script to work with dependency index - we probably want to handle dependency indices only when users want it - so we should introduce enable flag for it in the script.
Change index cleaner to remove dependency index when rollover is enabled. There are a couple of things to look for e.g. do not remove active writer index

jaeger/plugin/storage/es/esCleaner.py

Line 62 in 47d2029

def filter_main_indices_rollover(ilo, prefix):
Add test for index cleaner.

jaeger/plugin/storage/integration/es_index_cleaner_test.go

Line 17 in 4bf5e9b

package integration

I would recommend submitting a separate PR for the first step initially.

plugin/storage/es/dependencystore/storage.go

frittentheke · 2020-04-01T21:30:58Z

You have cracked an issue that will require some more work :)

* [ ]  Change rollover script to work with dependency index - we probably want to handle dependency indices only when users want it - so we should introduce enable flag for it in the script.

* [ ]  Change index cleaner to remove dependency index when rollover is enabled. There are a couple of things to look for e.g. do not remove active writer index https://github.com/jaegertracing/jaeger/blob/47d20296982d2e01432be46daef1dfc2c365872b/plugin/storage/es/esCleaner.py#L62

* [ ]  Add test for index cleaner.  https://github.com/jaegertracing/jaeger/blob/4bf5e9b440b0d95025248182a4b2caff8b0036fc/plugin/storage/integration/es_index_cleaner_test.go#L17

If I already cracked it and you thankfully already looked a little deeper into the other dependencies / breaking changes I take the liberty to go even another step further:

As I suggested in #2143 (comment) how about simply switching to using / providing a CronJob using the ElasticSearch curator (including some basic config for it). It's also a Python based tool, but very flexible and advertised by Elastic themselves to actually be the one stop housekeeper - It can do rules with time and name filters and then apply all sorts of actions, including rollover and alias. It also has a dry-run mode for people to first check what will be done (https://github.com/elastic/curator/blob/master/docs/asciidoc/examples.asciidoc).

I mean this is Jaeger tracing and likely you do not want to maintain and provide housekeeping tooling for all sorts of storages.

I would recommend submitting a separate PR for the first step initially.

I shall get started on that shortly.

pavolloffay · 2020-04-02T14:34:35Z

As I suggested in #2143 (comment) how about simply switching to using / providing a CronJob using the ElasticSearch curator (including some basic config for it). It's also a Python based tool, but very flexible and advertised by Elastic themselves to actually be the one stop housekeeper

See the response in #2143 (comment). If you manage to simplify those scripts we will be happy to accept the patch.

frittentheke · 2020-04-02T21:24:43Z

@pavolloffay I pushed some more commits - thanks for your patience.
Please kindly TAL.

frittentheke · 2020-04-02T23:02:14Z

@pavolloffay

As for the index templates - I just created another PR: #2149
If you feel this should just be one PR "adding ES rollover" to dependencies please let me know, I shall then add these commits this PR.

pavolloffay · 2020-04-03T08:28:22Z

#2149 looks good just remove there any other changes that are not related to creating index templates.

frittentheke · 2020-04-03T08:41:42Z

#2149 looks good just remove there any other changes that are not related to creating index templates.

Could you elaborate a bit please @pavolloffay . The PR #2149 is based on this one that is why those commits are in there as well. Should I reorder the changes of the two PRs then? Otherwise after merging this one, the other one should be rebased and then only contain the index template bits.

pavolloffay · 2020-04-03T08:55:09Z

#2149 should contain only refactoring to move the index mappings to json file and submitting those mappings as index templates at Jaeger startup.

frittentheke · 2020-04-03T18:11:57Z

#2149 should contain only refactoring to move the index mappings to json file and submitting those mappings as index templates at Jaeger startup.

@pavolloffay I did rearrange the two PRs now. So first step is the removal of the mappings in #2149 and this PR here then follows with adding the support for index aliases and the Params struct to configure things.

I noticed that there is not writer for dependencies initialized, that is why I kept the creation of templates with the reader.

frittentheke · 2020-06-11T14:49:29Z

@pavolloffay with the merge of #2285 I now did a rebase of the PR here to allow using an rollover alias for the dependency store.

FTAL

pavolloffay

Apart from this PR we will need changes to rollover and clean indices scripts.

plugin/storage/es/dependencystore/storage.go

plugin/storage/es/factory.go

Leinad0033 · 2020-08-25T18:57:37Z

Any chance of this getting updated to a state where is can be merged into master?

frittentheke · 2020-10-10T14:53:09Z

Any chance of this getting updated to a state where is can be merged into master?

@pavolloffay does it make sense to pick up on this then?
Should I rebase the PR and work on your remarks?

jpkrohling · 2021-08-23T08:19:59Z

Closing due to inactivity.

frittentheke · 2021-08-23T09:31:50Z

Closing due to inactivity.

@jpkrohling it's unfortunate that this, being such a rather small change, was not pursued.
Was there anything I could have done different?

jpkrohling · 2021-08-23T10:30:36Z

Thank you for asking! Sometimes, things fall off the cracks and this might have been one of them. Perhaps it would have helped to ping @pavolloffay more frequently (or even other maintainers).

If you are still interested in contributing this feature, let me know and I'll reopen this one!

frittentheke · 2021-08-23T20:53:38Z

Thank you for asking! Sometimes, things fall off the cracks and this might have been one of them. Perhaps it would have helped to ping @pavolloffay more frequently (or even other maintainers).

If you are still interested in contributing this feature, let me know and I'll reopen this one!

I certainly would still like to finish this PR.
Another reminder might have helped, but that honestly is a fine line. I neither want to get on @pavolloffay's nerves nor force something onto a project that does not want a certain contribution.

Maybe going through open PRs from time to time so see what is keeping them from moving would be sensible and a good sign to contributors. Kinda like what you did, just before hitting the close button :-)

As said, I gladly rebase the PR and would love to hear your feedback on what needs to be adjusted.

jpkrohling · 2021-08-24T11:25:00Z

I neither want to get on @pavolloffay's nerves nor force something onto a project that does not want a certain contribution.

I think it's fine to ping people from time to time. If it's a feature we don't want for some reason, we'll say so :-)

Would you prefer to open a new PR, or should I reopen this one?

@pavolloffay, would you be able to take another look at this PR and share what you think it's missing?

frittentheke · 2021-08-24T18:20:04Z

Would you prefer to open a new PR, or should I reopen this one?

if you don't mind please reopen this one.

frittentheke · 2021-12-09T18:27:46Z

@pavolloffay @jpkrohling could you kindly take a look at this?

wiardvanrij · 2021-12-10T14:33:36Z

+1 let's move this forward please. :) @frittentheke thanks for your work!

pavolloffay · 2021-12-13T14:13:17Z

@frittentheke / @wiardvanrij please add support for dependency index to rollover and index cleaner components. I see some changes in rollover but the init command does not create any dependency indices/aliases.

yellow open jaeger-span-000001            Cz0EQoQ1R4SXYY-Yv1fqZQ 5 1   0   0    810b    810b
yellow open jaeger-service-000001         S5vE45HfRkq_zXuELooc_g 5 1   0   0    810b    810b
yellow open .triggered_watches            EV2sv3m7SkegBcCXdvkJuQ 1 1   0   0    162b    162b
yellow open .monitoring-alerts-6          qOhRKYxbRe6KY4JumeoERw 1 1   1   0   6.2kb   6.2kb
yellow open .watches                      DW-akf7oQcKORu6TLJTztQ 1 1   4   0  19.7kb  19.7kb
yellow open .watcher-history-6-2021.12.13 -xI_8_EWTkaOf4Krk24G_g 1 1  25   0  67.2kb  67.2kb
yellow open .monitoring-es-6-2021.12.13   GyB2V6haRSCK1d0LstBTyg 1 1 356 170 614.2kb 614.2kb

vvdaal · 2022-01-12T16:04:54Z

@frittentheke possible any update on this or require help?

frittentheke · 2022-01-14T10:59:12Z

@frittentheke possible any update on this or require help?

Sorry for the delay. I shall look into this soon and push an update.

vvdaal · 2022-01-14T11:15:33Z

Sorry for the delay. I shall look into this soon and push an update.

No problem at all! It's all on a volunteer after all :) Really appreciate your contributions as this feature will help us in a use case we have!

frittentheke · 2022-01-24T22:56:24Z

@vvdaal @wiardvanrij sorry about the huge delay ⌛

@frittentheke / @wiardvanrij please add support for dependency index to rollover and index cleaner components. I see some changes in rollover but the init command does not create any dependency indices/aliases.

yellow open jaeger-span-000001            Cz0EQoQ1R4SXYY-Yv1fqZQ 5 1   0   0    810b    810b
yellow open jaeger-service-000001         S5vE45HfRkq_zXuELooc_g 5 1   0   0    810b    810b
yellow open .triggered_watches            EV2sv3m7SkegBcCXdvkJuQ 1 1   0   0    162b    162b
yellow open .monitoring-alerts-6          qOhRKYxbRe6KY4JumeoERw 1 1   1   0   6.2kb   6.2kb
yellow open .watches                      DW-akf7oQcKORu6TLJTztQ 1 1   4   0  19.7kb  19.7kb
yellow open .watcher-history-6-2021.12.13 -xI_8_EWTkaOf4Krk24G_g 1 1  25   0  67.2kb  67.2kb
yellow open .monitoring-es-6-2021.12.13   GyB2V6haRSCK1d0LstBTyg 1 1 356 170 614.2kb 614.2kb

@pavolloffay @jpkrohling Could you kindly check out my PR again, as I don't actually see what is not working correctly.
To test I just started a single node ES (via Docker/podman) and ran the following commands:

cmd/es-rollover/es-rollover-linux-amd64 init http://127.0.0.1:9200 which resulted in:

$ curl localhost:9200/_cat/indices
green  open .geoip_databases           vhPaxwXNTaiiuDCvAwtfbQ 1 0 42 0 40.4mb 40.4mb
yellow open jaeger-dependencies-000001 BxPTInxIQEe7dpyld_x0DA 5 1  0 0  1.1kb  1.1kb
yellow open jaeger-span-000001         684B2KtOQwSMxEgHzcF6UQ 5 1  0 0  1.1kb  1.1kb
yellow open jaeger-service-000001      rh0Ut4T6Tb-_2r3FqJQUjA 5 1  0 0  1.1kb  1.1kb

$ curl localhost:9200/_cat/aliases
jaeger-span-read          jaeger-span-000001         - - - -
jaeger-span-write         jaeger-span-000001         - - - -
jaeger-dependencies-read  jaeger-dependencies-000001 - - - -
jaeger-dependencies-write jaeger-dependencies-000001 - - - -
jaeger-service-read       jaeger-service-000001      - - - -
jaeger-service-write      jaeger-service-000001      - - - -

I then tried the rollover via cmd/es-rollover/es-rollover-linux-amd64 rollover http://127.0.0.1:9200 --conditions "{\"max_age\": \"1m\"}" (the condition is just to force a rollover, did not see any other option to achieve this):

$ curl localhost:9200/_cat/indices                                                                           
green  open .geoip_databases           vhPaxwXNTaiiuDCvAwtfbQ 1 0 42 0 40.4mb 40.4mb
yellow open jaeger-dependencies-000001 BxPTInxIQEe7dpyld_x0DA 5 1  0 0  1.1kb  1.1kb
yellow open jaeger-dependencies-000002 VwCy5gf4QvSxfHK6XHmcFQ 5 1  0 0   641b   641b
yellow open jaeger-span-000001         684B2KtOQwSMxEgHzcF6UQ 5 1  0 0  1.1kb  1.1kb
yellow open jaeger-span-000002         hohYZJ4aQiCbhXwjzhSAbA 5 1  0 0   641b   641b
yellow open jaeger-service-000002      SFWvaMULSkyHeeoMoISZog 5 1  0 0   641b   641b
yellow open jaeger-service-000001      rh0Ut4T6Tb-_2r3FqJQUjA 5 1  0 0  1.1kb  1.1kb

$ curl localhost:9200/_cat/aliases                                                                           
jaeger-span-read          jaeger-span-000001         - - - -
jaeger-dependencies-read  jaeger-dependencies-000001 - - - -
jaeger-service-read       jaeger-service-000002      - - - -
jaeger-service-write      jaeger-service-000002      - - - -
jaeger-dependencies-read  jaeger-dependencies-000002 - - - -
jaeger-dependencies-write jaeger-dependencies-000002 - - - -
jaeger-span-read          jaeger-span-000002         - - - -
jaeger-span-write         jaeger-span-000002         - - - -
jaeger-service-read       jaeger-service-000001      - - - -

is that not exactly what should happen?

albertteoh · 2022-01-25T11:03:23Z

@frittentheke, the integration tests will need updating to include the newly added jaeger-dependencies indices. For example: https://github.com/jaegertracing/jaeger/runs/4935280899?check_suite_focus=true#step:7:633

albertteoh · 2022-01-25T11:31:09Z

is that not exactly what should happen?

@frittentheke Thanks for the test details, the rollover results look good to me.

I think we might need to support dependencies for the index cleaner as well.

I've basically continued on from your test above. After executing the index cleaner, I expect all indices (span, service and dependencies) except the latest to be deleted. I can see span and service indices being deleted but not dependencies.

$ ./cmd/es-index-cleaner/es-index-cleaner-linux-amd64 0 http://127.0.0.1:9200  --rollover
{"level":"info","ts":1643109670.3972557,"caller":"./main.go:86","msg":"Indices before this date will be deleted","date":"2022-01-26T00:00:00Z"}
{"level":"info","ts":1643109670.397428,"caller":"./main.go:95","msg":"Queried indices","indices":[{"Index":"jaeger-dependencies-000001","CreationTime":"2022-01-25T11:18:28.229Z","Aliases":{"jaeger-dependencies-read":true}},{"Index":"jaeger-dependencies-000002","CreationTime":"2022-01-25T11:20:51.639Z","Aliases":{"jaeger-dependencies-read":true,"jaeger-dependencies-write":true}},{"Index":"jaeger-service-000001","CreationTime":"2022-01-25T11:18:28.039Z","Aliases":{"jaeger-service-read":true}},{"Index":"jaeger-service-000002","CreationTime":"2022-01-25T11:20:51.477Z","Aliases":{"jaeger-service-read":true,"jaeger-service-write":true}},{"Index":"jaeger-span-000001","CreationTime":"2022-01-25T11:18:27.834Z","Aliases":{"jaeger-span-read":true}},{"Index":"jaeger-span-000002","CreationTime":"2022-01-25T11:20:51.281Z","Aliases":{"jaeger-span-read":true,"jaeger-span-write":true}}]}
{"level":"info","ts":1643109670.3976908,"caller":"./main.go:102","msg":"Deleting indices","indices":[{"Index":"jaeger-service-000001","CreationTime":"2022-01-25T11:18:28.039Z","Aliases":{"jaeger-service-read":true}},{"Index":"jaeger-span-000001","CreationTime":"2022-01-25T11:18:27.834Z","Aliases":{"jaeger-span-read":true}}]}

... and this is evident with:

$ curl localhost:9200/_cat/aliases
jaeger-span-read          jaeger-span-000002         - - - -
jaeger-span-write         jaeger-span-000002         - - - -
jaeger-dependencies-read  jaeger-dependencies-000001 - - - -
jaeger-dependencies-read  jaeger-dependencies-000002 - - - -
jaeger-dependencies-write jaeger-dependencies-000002 - - - -
jaeger-service-read       jaeger-service-000002      - - - -
jaeger-service-write      jaeger-service-000002      - - - -

frittentheke · 2022-01-26T19:43:12Z

@albertteoh @pavolloffay @jpkrohling I pushed a new revision adding rollover to the es-index-cleaner and the missing tests.

PTAL

albertteoh · 2022-01-28T09:56:17Z

Thanks @frittentheke. It looks like some ES integration tests are failing.

You can run these tests locally:

make index-rollover-integration-test
make index-cleaner-integration-test

frittentheke · 2022-01-28T23:09:01Z

Thanks @frittentheke. It looks like some ES integration tests are failing.

You can run these tests locally:
* `make index-rollover-integration-test`

* `make index-cleaner-integration-test`

@albertteoh sorry for the delay. Tool be a minute more to figure that the optional ilm parts where missing in plugin/storage/es/mappings/jaeger-dependencies-7.json.

It's all green on my machine now - let's see what CI says then.

albertteoh · 2022-01-29T05:28:03Z

The TestMappingBuilder_GetMapping/jaeger-dependencies unit test is failing.

I think it may be related to the optional ILM part in jaeger-dependencies-7.json you discovered earlier.

By the way, the tests can be executed locally as well with: make test.

frittentheke · 2022-01-29T13:17:46Z

The TestMappingBuilder_GetMapping/jaeger-dependencies unit test is failing.
I think it may be related to the optional ILM part in jaeger-dependencies-7.json you discovered earlier.

Yep, the fixtures needed fixing (additions). make test shows a PASS now.
@albertteoh PTAL

albertteoh

LGTM, just a couple of nits.

cmd/es-index-cleaner/app/index_filter.go

albertteoh · 2022-02-01T12:01:12Z

plugin/storage/es/factory.go

-	reader := esDepStore.NewDependencyStore(f.primaryClient, f.logger, f.primaryConfig.GetIndexPrefix(),
-		f.primaryConfig.GetIndexDateLayoutDependencies(), f.primaryConfig.GetMaxDocCount())
-	return reader, nil
+	return createDependencyReader(f.logger, f.primaryClient, f.primaryConfig)


nit: I think we can just return esDepStore.NewDependencyStore(...).

I can see the intention here is to maintain consistency, though I think CreateSpanReader/Writer extract the function for reuse with its Archive equivalent. The DependencyStore doesn't support archiving AFAIK, so there doesn't seem to be much benefit of extracting the function for the cost of the additional indirection, unless you had other reasons?

What do you think?

While I understand your point and while I could argue there might arise the need to add archiving, I actually just would favor the consistency. As for performance, the compiler will optimize the additional call away and it's not like this is a hot path by any means.

In short, if you insist I will certainly change this, but I believe the code is just more approachable if similar things, as in creating readers for three types of stores, are not handled any differently.

We can leave this as is, I'm not strongly attached to my suggestion. 😄

* Give DependencyStore a params struct like the SpanStore to carry its configuration parameters * Adapt and extend the tests accordingly Signed-off-by: Christian Rohmann <christian.rohmann@inovex.de>

…es indices Signed-off-by: Christian Rohmann <christian.rohmann@inovex.de>

albertteoh · 2022-02-02T10:20:25Z

plugin/storage/es/factory.go

-	reader := esDepStore.NewDependencyStore(f.primaryClient, f.logger, f.primaryConfig.GetIndexPrefix(),
-		f.primaryConfig.GetIndexDateLayoutDependencies(), f.primaryConfig.GetMaxDocCount())
-	return reader, nil
+	return createDependencyReader(f.logger, f.primaryClient, f.primaryConfig)


We can leave this as is, I'm not strongly attached to my suggestion. 😄

frittentheke requested a review from a team as a code owner March 30, 2020 14:30

frittentheke mentioned this pull request Mar 30, 2020

Use single ElasticSearch index to store dependencies #2143

Closed

frittentheke force-pushed the depAliases branch from a5ed9f9 to c7eb8c8 Compare March 30, 2020 14:50

pavolloffay reviewed Mar 30, 2020

View reviewed changes

plugin/storage/es/dependencystore/storage.go Outdated Show resolved Hide resolved

plugin/storage/es/dependencystore/storage.go Outdated Show resolved Hide resolved

frittentheke mentioned this pull request Apr 2, 2020

Move ES dependencies index mapping to JSON template file #2149

Closed

frittentheke force-pushed the depAliases branch from 18810f9 to b68ab3d Compare April 3, 2020 18:09

frittentheke mentioned this pull request May 6, 2020

Add support for ElasticSearch alias to read spans from (Resolves #68) jaegertracing/spark-dependencies#86

Merged

frittentheke force-pushed the depAliases branch from b68ab3d to 1822874 Compare June 11, 2020 14:45

frittentheke force-pushed the depAliases branch 2 times, most recently from 1f314b1 to cc2a52d Compare June 11, 2020 20:11

pavolloffay reviewed Jun 15, 2020

View reviewed changes

plugin/storage/es/dependencystore/storage.go Outdated Show resolved Hide resolved

plugin/storage/es/dependencystore/storage.go Show resolved Hide resolved

plugin/storage/es/factory.go Outdated Show resolved Hide resolved

jpkrohling closed this Aug 23, 2021

pavolloffay added the storage/elasticsearch label Dec 13, 2021

frittentheke force-pushed the depAliases branch from 7a0da46 to c1be651 Compare January 24, 2022 22:48

frittentheke force-pushed the depAliases branch 2 times, most recently from 4a51446 to d11cddd Compare January 26, 2022 19:41

frittentheke force-pushed the depAliases branch from d11cddd to f3efb63 Compare January 28, 2022 23:05

frittentheke force-pushed the depAliases branch from f3efb63 to d3f037f Compare January 29, 2022 13:16

albertteoh previously approved these changes Feb 1, 2022

View reviewed changes

frittentheke added 2 commits February 1, 2022 22:12

Add support for ES index aliases / rollover to the dependency store

82a1d09

* Give DependencyStore a params struct like the SpanStore to carry its configuration parameters * Adapt and extend the tests accordingly Signed-off-by: Christian Rohmann <christian.rohmann@inovex.de>

Extend es-rollover and es-index-cleaner to support rolling dependenci…

a3cb643

…es indices Signed-off-by: Christian Rohmann <christian.rohmann@inovex.de>

frittentheke dismissed albertteoh’s stale review via a3cb643 February 1, 2022 21:12

frittentheke force-pushed the depAliases branch from d84b0bc to a3cb643 Compare February 1, 2022 21:12

Merge branch 'main' into depAliases

6288680

albertteoh approved these changes Feb 2, 2022

View reviewed changes

albertteoh enabled auto-merge (squash) February 2, 2022 10:21

albertteoh disabled auto-merge February 2, 2022 10:21

albertteoh merged commit 18e9d5b into jaegertracing:main Feb 3, 2022

Add support for ES index aliases / rollover to the dependency store (Resolves #2143) #2144

Add support for ES index aliases / rollover to the dependency store (Resolves #2143) #2144

Conversation

frittentheke commented Mar 30, 2020 • edited by pavolloffay Loading

Which problem is this PR solving?

Short description of the changes

pavolloffay commented Mar 30, 2020 • edited Loading

frittentheke commented Apr 1, 2020

pavolloffay commented Apr 2, 2020

frittentheke commented Apr 2, 2020

frittentheke commented Apr 2, 2020

pavolloffay commented Apr 3, 2020

frittentheke commented Apr 3, 2020 • edited Loading

pavolloffay commented Apr 3, 2020

frittentheke commented Apr 3, 2020

frittentheke commented Jun 11, 2020

pavolloffay left a comment

Choose a reason for hiding this comment

Leinad0033 commented Aug 25, 2020

frittentheke commented Oct 10, 2020

jpkrohling commented Aug 23, 2021

frittentheke commented Aug 23, 2021

jpkrohling commented Aug 23, 2021

frittentheke commented Aug 23, 2021 • edited Loading

jpkrohling commented Aug 24, 2021

frittentheke commented Aug 24, 2021

frittentheke commented Dec 9, 2021

wiardvanrij commented Dec 10, 2021

pavolloffay commented Dec 13, 2021

vvdaal commented Jan 12, 2022

frittentheke commented Jan 14, 2022

vvdaal commented Jan 14, 2022

frittentheke commented Jan 24, 2022

albertteoh commented Jan 25, 2022

albertteoh commented Jan 25, 2022

frittentheke commented Jan 26, 2022

albertteoh commented Jan 28, 2022

frittentheke commented Jan 28, 2022

albertteoh commented Jan 29, 2022 • edited Loading

frittentheke commented Jan 29, 2022

albertteoh left a comment

Choose a reason for hiding this comment

albertteoh Feb 1, 2022

Choose a reason for hiding this comment

frittentheke Feb 1, 2022 • edited Loading

Choose a reason for hiding this comment

albertteoh Feb 2, 2022

Choose a reason for hiding this comment

albertteoh Feb 2, 2022

Choose a reason for hiding this comment

frittentheke commented Mar 30, 2020 •

edited by pavolloffay

Loading

pavolloffay commented Mar 30, 2020 •

edited

Loading

frittentheke commented Apr 3, 2020 •

edited

Loading

frittentheke commented Aug 23, 2021 •

edited

Loading

albertteoh commented Jan 29, 2022 •

edited

Loading

frittentheke Feb 1, 2022 •

edited

Loading