You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It will help us store notifications for users and collections. Currently, we use Elasticsearch to store notifications.
We can use it as an audit log to be able to able to audit
the actions of a user
the actions in a collection
the actions on an entity
ftm-store is kind of an event store already where we store entity fragments and combine them to generate full entities.
Features we need
The ability to store events permanently
We need these events to persist until they are explicitly flushed
The ability to query, sort these events.
ftm-store needs to be able to sort fragments based on entity id, origin etc
for audit, we may need to query for events from a particular user within a certain time frame
The ability possibly run this locally without a complex set up.
This is helpful if we are generating FtM locally outside Aleph
It will also be helpful if we implement a way to ingest files offline to generate ftm-bundles (See Run ingestors on the CLI to generate "ftm-bundles" ingest-file#223). We'll need a way to store ingest logs locally so that we can aggregate them and print out a report at the end of the process.
Possible implementation methods
The industry seems to like Kafka as an event store.
Kafka can store events permanently.
But I couldn't find a easy way to query or sort the stored data without replaying it from the beginning. One possible way to do it is by using something like KSQL on top of Kafka.
Running Kafka for small local jobs seems like a overkill.
The other option is use SQL
We already use PostgreSQL/SQLite for ftm-store. It can of course store the data permanently.
It's easy to query and sort the stored data
For local jobs, sqlite is a great option.
The text was updated successfully, but these errors were encountered:
Problems that an event store can solve
With an event store we can tackle a few problems:
Features we need
Possible implementation methods
The text was updated successfully, but these errors were encountered: