-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
0.8.3 backports #2022
0.8.3 backports #2022
Conversation
* Adds a special PropertyChecker interface which is ONLY for setting string properties at the very start of psvm
…ful exception when the same inner aggregator is referenced by multiple types of outer aggregators
few fixes delete intermediate file early better exception handling use static pattern instead of compiling it every time Add retry for transient exceptions remove usage of deprecated method. Add test fix imports fix javadoc review comment. review comment: handle crazy snapshot naming review comments remove default retry count in favour of already present constant review comment make random intermediate and final paths. review comment, use temporaryFolder where possible
In apache#933 the ForkingTaskRunner's logging was changed to buffered from unbuffered. This means that the last few KB of the logs are generally not visible while a task is running, which makes debugging running tasks difficult.
…are. This is done by killing and respawning the jvms rather than reconnecting to existing jvms, for a couple reasons. One is that it lets you restore tasks after server reboots too, and another is that it lets you upgrade all the software on a box at once by just restarting everything. The main changes are, 1) Add "canRestore" and "stopGracefully" methods to Tasks that say if a task can stop gracefully, and actually do a graceful stop. RealtimeIndexTask is the only one that currently implements this. 2) Add "stop" method to TaskRunners that attempts to do an orderly shutdown. ThreadPoolTaskRunner- call stopGracefully on restorable tasks, wait for exit ForkingTaskRunner- close output stream to restorable tasks, wait for exit RemoteTaskRunner- do nothing special, we actually don't want to shutdown 3) Add "restore" method to TaskRunners that attempts to bootstrap tasks from last run. Only ForkingTaskRunner does anything here. It maintains a "restore.json" file with a list of restorable tasks. 4) Have the CliPeon's ExecutorLifecycle lock the task base directory to avoid a restored task and a zombie old task from stomping on each other.
old names are still valid though so as to be backwards compatible for now
Otherwise we could end up with a metric and a dimension with the same column name.
can you please include #1991 as well? |
adds support for "cardinality" aggregator. do not create sketch per event at ingestion time to make realtime ingestion faster
@himanshug ok, added |
@nishantmonu51 added #1957, but there are a lot of merge conflicts with #1943 so maybe we can do that in a separate PR |
#1832 could be included too. |
can #1929 be added as well ? |
@guobingkun added #1832 |
401c907
to
e59b7bd
Compare
added #1791 |
@nishantmonu51 ok, sounds good. I think this PR is ready to review then. |
👍 for me once the build passes, not sure if we can review all the changes in this PR. |
@himanshug I mostly just meant the list of patches included :) |
mmx#10 is just #1960 |
This PR adds adds the ability to skip incremental index when querying results from realtime nodes. default behaviour is to include incrementalIndex in queries. review comment
add an EventReceiverFirehoseMonitor so that we can monitor how many events have been queued in the EventReceiverFirehose and get a sense about whether the firehose is under too much pressure.
e59b7bd
to
57c52a9
Compare
57c52a9
to
85cb4dc
Compare
👍 |
Backport:
#1387 Add special handler to allow logger messages during shutdown [Ease of Use]
#1791 EventReceiverFirehoseMonitor
#1799 Support multiple outer aggregators of same type and provide more help…
#1815 Fix Race in jar upload during hadoop indexing
#1832 add examples for duration and period granularities
#1843 "druid.manager.segment" should be "druid.manager.segments
#1854 Fix documentation about lookup
#1867 fixing hadoop test scope dependencies in indexing-hadoop
#1881 Restorable indexing tasks
#1888 forward cancellation request to all brokers, fixes #1802
#1897 complex aggregator based on http://datasketches.github.io
#1900 fix doc - correct default value for
#1917 RemoteTaskActionClient: Fix statusCode check.
#1920 Update curator to 2.9.1
#1929 separate ingestion and query thread pool
#1932 DataSchema: Exclude metric names from dimension list.
#1935 ForkingTaskRunner: Log without buffering.
#1943 Enable caching on intermediate realtime persists
#1954 EC2 autoscaler: avoid hitting aws filter limits
#1957 Ability to skip Incremental Index during query using query context
#1960 optimize index merge
#1991 datasketches module updates
Additional Release Notes updates:
min/max aggregators are deprecated and will be removed in druid-0.9.0 release. please use [long/double]Min/Max aggregators.