-
Notifications
You must be signed in to change notification settings - Fork 24.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DIsk based allocator should take into account the amount of disk space which can be reused #10103
Comments
@dakrone any thoughts about this? |
@clintongormley hmm.. to implement this we would have to send a request for each shard to all nodes to see what files and checksums they have in common. This would be quite heavy, especially for an |
@clintongormley no, it hasn't be resolved still, I'm not sure if the seqids work is far enough along to work on this, maybe @bleskes knows? |
I don't think we can expect this in the first iteration. The problem is very similar to the current file based copy - we don't know how much will be reused (or if seq no base recovery will succeed) until we start the work. The master needs to know this in advance somehow (as Lee already said). |
Pinging @elastic/es-distributed |
As I understand it, the situation described here is a transient problem around a node restart in the sense that the cluster sorts itself out eventually once it works out what data it no longer needs. As such, since its opening we have not seen enough feedback that it is something we should work on. We prefer to close this issue as a clear indication that we are not going to work on this at this time. We are always open to reconsidering this in the future based on compelling feedback; despite this issue being closed please feel free to leave feedback (including +1s). |
Assume we have a node where the disk usage is above the low watermark. If we restart this node then no shards will be allocated on it since the disk usage is above the low watermark.
It would be better if the disk based allocator could realize that it can throw away or at least reuse the space which is currently used for outdated shard copies on the node.
The text was updated successfully, but these errors were encountered: