Delayed Write: calculate delay based on current state

[Yuval-Ariel]

Owner: yuval

as can be seen in the analysis in #154 (comment) , the current mechanism which calculates the delayed write rate needs improvement.
the current delay rate is calculated in SetupDelay in db/column_family.cc. (called from RecalculateWriteStallConditions)
the problem with the current implementation is that the delayed write rate (DWR from now) is calculated based on the changes in compaction pending bytes between the current and previous call to RecalculateWriteStallConditions (which is called on every InstallSuperVersion ). If compaction pending bytes is less than the prev then it increases the current DWR by 25% and if equal or greater than the prev then it reduces the DWR by 20%.
so no matter by how much the state has changed, the DWR is always changed by the same amount.
furthermore, these changes are bigger the closer we are to the original (user) DWR, e.g. reduce by 20Mb/s from 100Mb/s versus reduce by 2 Mb/s from 10MB/s. so the closer we are to the stop limitation, the less we change the amount by which we delay which is not what we want. we'd like the delay to be small at small violations of the limitations (L0 files, compaction pending bytes, memory) and increase the delay by bigger quantities as we get closer to the stop limitations.

We propose to calculate the DWR based on by how much we're currently in a violation of the limitation and calculate according to the user DWR and not the previously calculated DWR. e.g. if the user DWR is 100MB/s, L0 slowdown is set to 10 and stop is set to 20 files, and there are 14 L0 files:
there are 10 steps until stop. so each step toward the stop is 10MB/s reduction from the delayed rate the user specified and the other way when increasing the write rate.
10 L0 files - 100 Mb/s
11 L0 files - 90 Mb/s
until total stop at 20 L0 files.
same solution for compaction pending bytes and memtables.

[Yuval-Ariel]

consider adding the option to the c- API and to the java

[Yuval-Ariel]

example of the benefits of this approach: red - this branch. green - main branch

as can be seen, the rate stays mostly stable.

[Yuval-Ariel]

QA passed on ffe9b9c

[erez-speedb]

The Performance tests show no improvements, no degradation on average.
The stability above was shown on the fillup only (single thread), and not with overwrite (multithreads).

This behaviour is not existing with small obj.

[Yuval-Ariel]

@erez-speedb , regarding the 25% deterioration in the fillup of the small obj test:
i've looked at the LOGs of the small obj results and the new code isnt reached at all (there are no cases when Speedb is in a delayed state) so the difference in performance is because of something else. can you guess why?