WC: fix for stop while shutting down

[Yuval-Ariel]

as part of facebook/rocksdb#10765 , a check of whether the waiting db was shutting_down_ was intoduced.
The write thread then stops waiting when shutting_down_ is turned on which is the case when CancelAllBackgroundWork is called. But the rebased code now waits forever since #346 replaced the waiting on bg_cv_ to a wait on a write controller condition variable inside WriteController::WaitOnCV.
The condition variable inside WaitOnCV (stop_cv_) is only notified through the detor of a StopWriteToken.

The write_controller_token_ in each CF is being reset in any case after the call to CancelAllBackgroundWork from DBImpl::CloseHelper() (detor of DBImpl) so if the test would issue a db Close and then wait for the thread there would be no problem.

also consider fixing by instead of replacing CancelAllBackgroundWork with Close, to add a function right after the call to CancelAllBackgroundWork which notifies the stop_cv_ . this will make it clear to further unit test writers that calling CancelAllBackgroundWork is not sufficient in order to remove the stop condition

[Yuval-Ariel]

@udi-speedb , plz wait with the review since changing CancelAllBackgroundWork to Close seemed to reveal a new issue which is handled in #500 .
setting this PR to on hold until the above issue is fixed

[Yuval-Ariel]

also need to include a notify on the write controllers condition variable when there a BG error (SetBGError)

[Yuval-Ariel]

@udi-speedb , plz review. issue #500 is still unresolved as it requires further discussion so we can proceed with this PR IMO.

[udi-speedb]

@Yuval-Ariel - Approving.
Just a reminder - Did you go over all the places where CancelAllBackgroundWork() is called to verify that no need to call ResetWriteControllerTokens()?

[Yuval-Ariel]

Just a reminder - Did you go over all the places where CancelAllBackgroundWork() is called to verify that no need to call ResetWriteControllerTokens()?

yes i did

[Yuval-Ariel]

@erez-speedb , plz check that theres no degradation. thanks

[Yuval-Ariel]

performance - no degradation.
still need to update history file