Use IndexWriter.getFlushingBytes() rather than tracking it ourselves

[romseygeek]

Currently we keep track of how many bytes are currently being written to disk
in an AtomicLong within InternalEngine, updating it on refresh. The IndexWriter
has its own accounting for this, and exposes it via a getFlushingBytes method
in the latest lucene 8 snapshot. This commit removes the InternalEngine tracking
in favour of just using the IndexWriter method.

[elasticmachine]

Pinging @elastic/es-distributed

[s1monw]

I think we need to keep the version map in the picture but it's straight forward I think:

diff --git a/server/src/main/java/org/elasticsearch/index/engine/LiveVersionMap.java b/server/src/main/java/org/elasticsearch/index/engine/LiveVersionMap.java
index 18d3cedb37e..d1d96c76372 100644
--- a/server/src/main/java/org/elasticsearch/index/engine/LiveVersionMap.java
+++ b/server/src/main/java/org/elasticsearch/index/engine/LiveVersionMap.java
@@ -434,6 +434,12 @@ final class LiveVersionMap implements ReferenceManager.RefreshListener, Accounta
         return maps.current.ramBytesUsed.get();
     }
 
+    /**
+     * Returns how much RAM is currently being freed up by refreshing. This is {@link #ramBytesUsed} except does not include tombstones
+     * because they don't clear on refresh.
+     */
+    long getRefreshingBytes() { return maps.old.ramBytesUsed.get(); }
+
     @Override
     public Collection<Accountable> getChildResources() {
         // TODO: useful to break down RAM usage here?

I also think the getRefreshingBytes needs a test though

[s1monw]

left one comment

[romseygeek]

Maybe it should be called LiveVersionMap#getFlushingBytes() as well, to keep in line with the IndexWriter method name?

[s1monw]

Maybe it should be called LiveVersionMap#getFlushingBytes() as well, to keep in line with the IndexWriter method name?

I am ok either way

[romseygeek]

Thinking more about it, I'll keep the names as they are. The IndexWriter is flushing, but the LiveVersionMap is refreshing, and the total is what is being written. So it all makes sense :)