-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable caching on intermediate realtime persists #1943
Conversation
@@ -26,6 +26,7 @@ | |||
import com.google.common.base.Throwables; | |||
import com.google.common.collect.ImmutableMap; | |||
import com.google.common.collect.ImmutableSet; | |||
import com.google.common.collect.Iterables; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
unused import ?
3f38dfc
to
ac66b4e
Compare
cache.close(input.getSegment().getIdentifier() + "_" + input.getCount()); | ||
return null; | ||
} | ||
}); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we need to iterate for the close to be called ? or may be simply replace it with a for loop.
Also, can we abstract the the namespace creation method.
The spills are not consistent across nodes so this would only work if the cache is local. That should probably be called out or enforced or something. |
@xvrl is it possible to get real world performance numbers? |
5cd6404
to
7bea48e
Compare
@gianm addressed comments, will look into getting some real-word numbers. |
7bea48e
to
e22d8eb
Compare
e22d8eb
to
1339f28
Compare
1339f28
to
5b27f53
Compare
@gianm roughly 20% speedup on a typical top-n query 10-15min into the hour of a 1-hour realtime index task, with 10 persists completed (~ 1.4M rows to scan)
|
👍 since old behavior is preserved by default and this gets into action only if user explicitly provides the configuration. |
@himanshug yes, will add docs, was waiting for comments before doing so |
9841a7f
to
d7eb2f7
Compare
Enable caching on intermediate realtime persists
No description provided.