-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[bugfix] Run cold schema refresh thread periodically #16873
Conversation
ServiceMetricEvent.Builder metricBuilder = | ||
new ServiceMetricEvent.Builder().setDimension(DruidMetrics.DATASOURCE, dataSourceName); | ||
|
||
emitter.emit(metricBuilder.setMetric("metadatacache/cold/segment/count", coldSegments)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: Use a prefix for metadatacache/cold/segment/
docs/operations/metrics.md
Outdated
@@ -382,6 +382,9 @@ These metrics are emitted by the Druid Coordinator in every run of the correspon | |||
|`metadatacache/finalizedSchemaPayload/count`|Number of finalized segment schema cached.||Depends on the number of distinct schema in the cluster.| | |||
|`metadatacache/temporaryMetadataQueryResults/count`|Number of segments for which schema was fetched by executing segment metadata query.||Eventually it should be 0.| | |||
|`metadatacache/temporaryPublishedMetadataQueryResults/count`|Number of segments for which schema is cached after back filling in the database.||This value gets reset after each database poll. Eventually it should be 0.| | |||
|`metadatacache/cold/segment/count`|Number of cold segments.|`dataSource`|| | |||
|`metadatacache/cold/refresh/count`|Number of cold segments with cached schema.|`dataSource`|| |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"cold segment" is not a term that exists anywhere outside of CoordinatorSegmentMetadataCache
.
It would be better to use a term that is more easily relatable.
Also, the metadatacache metrics seems all over the place with their prefixes. It is unavoidable for metrics across multiple features to not have a consistent naming scheme. But metrics within the same feature should be consistent. Since this feature is still nascent, I would advise we revisit all the metric names for this feature and categorize them nicely with proper prefixes. (not necessarily in this PR)
cc: @cryptoe
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be better to use a term that is more easily relatable.
I don't find much references for such segments, do you have any suggestions?
Since this feature is still nascent, I would advise we revisit all the metric names for this feature and categorize them nicely with proper prefixes.
Are you suggesting to use a different prefix than metadatacache
? How about cds
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes lets change the wording from cold
to deepstorageonly
.
Issue
Parent issue: #14989
Fixes a bug in #16676.
Schema from cold segments wasn't getting reflected in the datasource schema.
The problem turned out to be in the refresh thread, it ran only once instead of running periodically.
This PR also adds some metrics to monitor cold schema refresh process.
Testing
cold
datasource in the attached screen shot with 1 segment.partially_cold
datasource in the attached screen shot with 2 segments. The hot segment has columns c3, c4, c5 and cold segment has columns c1, c2, c3.Release Notes
Added followings metrics to monitor cold schema refresh process.
This PR has: