-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Filter out tombstone segments from metadata cache #16890
Conversation
@@ -2214,6 +2215,77 @@ protected void coldDatasourceSchemaExec() | |||
Assert.assertEquals(0, latch.getCount()); | |||
} | |||
|
|||
@Test | |||
public void testTombstoneSegmentIsNotAdded() throws InterruptedException |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: This test case is very similar to BSMC#testTombstoneSegmentIsNotAdded(). Can we do a bit of refactor to remove redundancy.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The test is trying ensure that tombstone segments are not added to the Broker and Coordinator cache instances.
Refactoring the test wouldn't be very beneficial since,
- Timeline used in the two caches is different.
org.apache.druid.segment.metadata.AbstractSegmentMetadataCache#getSegmentMetadataSnapshot
implementation differs between the two instances.
* Fix build * Support segment metadata queries for tombstones * Filter out tombstone segments from metadata cache * Revert some changes * checkstyle * Update docs
This PR #16890 introduced a change to skip adding tombstone segments to the cache. It turns out that as a side effect tombstone segments appear unavailable in the console. This happens because availability of a segment in Broker is determined from the metadata cache. The fix is to keep the segment in the metadata cache but skip them from refresh. This doesn't affect any functionality as metadata query for tombstone returns empty causing continuous refresh of those segments.
This PR apache#16890 introduced a change to skip adding tombstone segments to the cache. It turns out that as a side effect tombstone segments appear unavailable in the console. This happens because availability of a segment in Broker is determined from the metadata cache. The fix is to keep the segment in the metadata cache but skip them from refresh. This doesn't affect any functionality as metadata query for tombstone returns empty causing continuous refresh of those segments.
This PR apache#16890 introduced a change to skip adding tombstone segments to the cache. It turns out that as a side effect tombstone segments appear unavailable in the console. This happens because availability of a segment in Broker is determined from the metadata cache. The fix is to keep the segment in the metadata cache but skip them from refresh. This doesn't affect any functionality as metadata query for tombstone returns empty causing continuous refresh of those segments.
This PR apache#16890 introduced a change to skip adding tombstone segments to the cache. It turns out that as a side effect tombstone segments appear unavailable in the console. This happens because availability of a segment in Broker is determined from the metadata cache. The fix is to keep the segment in the metadata cache but skip them from refresh. This doesn't affect any functionality as metadata query for tombstone returns empty causing continuous refresh of those segments.
This PR apache#16890 introduced a change to skip adding tombstone segments to the cache. It turns out that as a side effect tombstone segments appear unavailable in the console. This happens because availability of a segment in Broker is determined from the metadata cache. The fix is to keep the segment in the metadata cache but skip them from refresh. This doesn't affect any functionality as metadata query for tombstone returns empty causing continuous refresh of those segments.
This PR #16890 introduced a change to skip adding tombstone segments to the cache. It turns out that as a side effect tombstone segments appear unavailable in the console. This happens because availability of a segment in Broker is determined from the metadata cache. The fix is to keep the segment in the metadata cache but skip them from refresh. This doesn't affect any functionality as metadata query for tombstone returns empty causing continuous refresh of those segments.
Broker attempts to continuously refresh tombstone segments due to missing metadata (due to a limitation with segment metadata queries, see: #12137).
This PR resolves the issue by preventing tombstone segments from being added to the metadata cache.
Related to #14989
This PR has: