-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DO NOT MERGE - 27.0.0 WIP release notes #14600
Conversation
|
||
[14281](https://github.com/apache/druid/pull/14281) | ||
|
||
## Dependency updates |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Similar to the 26.0 release, should we include a "Bug fixes" section with a pointer to the list:
https://github.com/apache/druid/issues?q=is%3Aclosed+milestone%3A27.0+-label%3ABackport+label%3ABug+
For the 27.0.0 release, though, the bugs list looks relatively small - from a cursory scan of the merged PRs in 27.0, it seems like some of them weren't tagged "Bug".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, several of them might have been bug fixes but we might have just forgotten to label them appropriately.
docs/do-not-merge.md
Outdated
|
||
[13197](https://github.com/apache/druid/pull/13197) | ||
|
||
## Hadoop 2 support dropped |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is not being dropped in this release, but we did decide to deprecate it.
Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>
docs/do-not-merge.md
Outdated
| `segment/loadQueue/success` |Number of segment assignments that completed successfully.|`dataSource`, `server`|Varies| | ||
| `segment/loadQueue/cancelled` |Number of segment assignments that were canceled before completion. |`dataSource`,`server` | 0 | | ||
| `segment/loadQueue/failed` |Number of segment assignments that failed to complete.|`dataSource`, `server`|0| | ||
| `segment/assigned/broadcast` | | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kfaraz It looks like the 2 broadcast metrics from 13197 didn't get added to https://github.com/apache/druid/blob/master/docs/operations/metrics.md
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, @317brian , these metrics were originally a part of the PR but were later removed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added some items that were left out because I had forgotten to update/add release notes in respective PRs.
|
||
[14281](https://github.com/apache/druid/pull/14281) | ||
|
||
## Dependency updates |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, several of them might have been bug fixes but we might have just forgotten to label them appropriately.
Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>
docs/do-not-merge.md
Outdated
@@ -53,20 +53,21 @@ For more information about this feature, see the [26.0.0 release notes for Schem | |||
|
|||
## Smart segment loading | |||
|
|||
Revamped Coordinator to make it more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities. | |||
The Coordinator is now more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"The Coordinator" seems to have been repeated several times in this section, can we reduce it?
Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>
docs/do-not-merge.md
Outdated
## Smart segment loading | ||
|
||
INSTEAD OF PRIORITIZING THINGS EQUALLY; WE PRIORITIZE NEW SEGMENTS THAT ARE UNDERREPLICATED |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Suggested content for this section:
The Coordinator is now much more stable and user-friendly. In the new smartSegmentLoading
mode, it dynamically computes values for several configs which maximize performance.
The Coordinator can now prioritize load of more recent segments and segments that are completely unavailable over load of segments that already have some replicas loaded in the cluster. It can also re-evaluate decisions taken in previous runs and cancel operations that are not needed anymore. Moreoever, move operations started by segment balancing do not compete with the load of unavailable segments thus reducing the reaction time for changes in the cluster and speeding up segment assignment decisions.
docs/do-not-merge.md
Outdated
* What to do about input source security. There are no docs for it. | ||
* New query filters | ||
* Confirm the correct naming in https://github.com/apache/druid/pull/14359 (OSHI sys mon change). The changed files say one thing and the PR description says another | ||
* Community extensions. What do we want to say about iceberg and the other one (what was the other one?) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also SQL compatability mdoe mention somewhere?
docs/do-not-merge.md
Outdated
* Caching cost for upgrade notes - is it this? [#14484](https://github.com/apache/druid/pull/14484) | ||
* User impact of https://github.com/apache/druid/pull/14048 for highlights | ||
* https://github.com/apache/druid/pull/14476 seems to only be documented here in release notes and the PR? | ||
* Vad's input for web console section |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about:
Web console
Console now uses the new statements API for all MSQ interaction
The console uses the new async statements API for all sql-msq-task
engine queries.
While this has relatively few impacts on the UX of the query view you are invited to peek under the hood and check out the new network requests being sent as a working example of the new API.
There are some fun UX benefits: you can now specify durableStorage
as the result destination for SELECT
queries (when durable storage is configured).
After running a SELECT
query that wrote its resuls to durableStorage
you can download the full, unlimited, result set directly from the broker.
Added UI around data source with zero replica segments
This release of Druid supports having datasources with segments that are not replicated on any historicals, the console now had special handling for datasources like that.
[https://github.com//pull/14540]
Added a Explore view
The Explore view is a simple, stateless, SQL backed, data exploration view to the web console. It lets users explore data in Druid with point-and-click interaction and visualizations (instead of writing SQL and looking at a table). This can provide faster time-to-value for a user new to Druid and can allow a Druid veteran to quickly chart some data that they care about.
The Explore view is accessible from the more button in the header.
Added a dialog for viewing and setting the dynamic compaction config
[https://github.com//pull/14540]
Other changes
- Replaced the Ingestion view with two views: Supervisors and Tasks #14395
- Added 'Start time' column to the Services tab [https://github.com/Add start_time column to sys.servers #13358]
- Added 'Replication factor' column to the Segments view #14403
- Added stateful filter URLs for all views [https://github.com/Web console: split the Ingestion view into two views: Supervisors and Tasks #14395]
docs/do-not-merge.md
Outdated
|`segment/serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled. This metric can be used in conjunction with `segment/serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise| | ||
|`segment/serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled.|`server`, `tier`|Not emitted for synced servers.| | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
|`segment/serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled. This metric can be used in conjunction with `segment/serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise| | |
|`segment/serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled.|`server`, `tier`|Not emitted for synced servers.| | |
|`serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled. This metric can be used in conjunction with `serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise| | |
|`serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled.|`server`, `tier`|Not emitted for synced servers.| | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Manually updated in 0959b28
I've cleaned up all my notes and included more information as discussed. @abhishekagarwal87, @AmatyaAvadhanula , and @2bethere can y'all double check the deprecation list to make sure I have everything. I ran the spellchecker locally, and the errors I received back were expected. (The RAT test will continue to fail since we intentionally did not add a license snippet to this file) |
docs/do-not-merge.md
Outdated
@@ -293,8 +293,8 @@ TBD for name change | |||
|
|||
|Metric|Description|Dimensions|Normal value| | |||
|------|-----------|-----------|----------| | |||
|`segment/serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled. This metric can be used in conjunction with `segment/serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise| | |||
|`segment/serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled.|`server`, `tier`|Not emitted for synced servers.| | |||
|`serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled. This metric can be used in conjunction with `segment/serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise| |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
segment/serverview/sync/unstableTime -> serverview/sync/unstableTime
Draft PR used to collaborate on the release notes so that we can use the comment/suggestion system.