-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Epic: Realtime Ingestion Improvements #1642
Comments
https://groups.google.com/forum/#!searchin/druid-development/tier/druid-development/1I3CmxlOipM/e3-SpWqG170J could fit here also? That directly applies to 5, 2 (and maybe 6?) on the list. |
@drcrallen added |
I have created a document at https://docs.google.com/document/d/1PUG3crI2jiPa_u926R0KrkZVM7t706rXp1IuUxVXB5E/edit?usp=sharing to capture various design details of kafka/tranquility ingestion work . This has been created with inputs from @gianm and still under active development. feel free to discuss here. |
I updated the doc with some thoughts and preliminary code around push-based/tranquility ingestion. |
A couple of tangentially related things. #1881 - Restorable indexing tasks (PR) - so middleManagers can be restarted similarly to realtime nodes |
Updated the google doc with the current state of kafka ingestion stuff. |
Work based off this proposal was released a couple releases ago. Circling back and closing this. http://druid.io/docs/latest/development/extensions-core/kafka-ingestion.html |
This issue tracks all the "related" efforts targeted towards making realtime ingestion better in one way or the other.
Here is a wishlist of items we should try to solve for. I understand that some of this is already solved by tranquility (probably not in kafka based ingestion), but we should ensure that those are not broken as we make changes and are supported by kafka based ingestion as well.
(Realtime Delta Ingestion: ability to ingest late events as they come would probably happen as a side effect of 1st)
Related Refs:
https://groups.google.com/forum/#!msg/druid-development/kHgHTgqKFlQ/fXvtsNxWzlMJ (No window period proposal)
https://groups.google.com/forum/#!msg/druid-development/9HB9hCcqvuI/L59RgsloZfoJ (FirehoseV2 proposal)
https://docs.google.com/document/d/1PUG3crI2jiPa_u926R0KrkZVM7t706rXp1IuUxVXB5E/edit?usp=sharing (doc covering design details for both above)
https://groups.google.com/forum/#!searchin/druid-development/tier/druid-development/1I3CmxlOipM/e3-SpWqG170J (Task Tiering proposal)
Related PRs:
#1609 (kafka simple consumer based firehose and initial FirehoseV2 updates)
#1639 (new plumber)
Related Issues:
#401 (log management for long-running tasks)
#1513 (preemption for indexing service locks)
#1514 (aggregatorFactories in segment metadata)
#1515 (AllocateSegmentAction)
#1516 (ElasticShardSpec)
#1517 (user-friendly Hadoop-based re-indexing/compaction)
The text was updated successfully, but these errors were encountered: