-
Notifications
You must be signed in to change notification settings - Fork 6.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failed to resume DB after "no space left on device" error. #11643
Comments
we hit this too, any update? |
I don't think we can simply reset error on the last WAL. In this repro, nothing is written to memtable and WAL has a partial write batch record. After error is cleared, new writes will append to this WAL after the corrupted record, and may not be recovered after reopening. |
where in the code do we insert the partial record? maybe we should undo the inserting when encountering |
Sorry for the late reply. When writing a write batch to WAL, the write batch can be broken down into several WAL records and several writes. This happens in Line 91 in b6c3495
I think for auto-recovery, we should make sure not to write to this WAL anymore and make sure this WAL is not needed for this DB anymore. For the latter, we do flush during auto-recovery, but in this case memtable could be empty and flush is not done. |
It looks like RocksDB failed to resume after hitting "no space left on device".
"no space left on device" error is marked as an
HardError
, which according to Background-Error-Handling should be recoverable back to read-write mode after the issue has been addressed.Unfortunately, @siying has introduced 10489 which adds a flag in
WritableFileWriter
that remembers an error has occurred.Resuming the DB after no space error doesn't clear the flag, causing successive write failures, even though the recovery was completed successfully.
Expected behavior
RocksDB should be able to recover after not enough disk space error, either by calling
Resume
manually or using auto recovery.Actual behavior
Both auto recovery and calling resume manually succeed but any further writes fails.
Steps to reproduce the behavior
Consider the following code:
In the above code example, I expect the DB to be writeable after a disk space has been freed, but it keeps failing due to the
seen_error_
flag inWritableFileWriter
.Notes:
PessimisticTransactionDB
.logs_.back().writer->file()->reset_seen_error();
inResumeImpl
, but I'm not familiar enough with rocks error handling, and I'm not sure what are the consequences of such change.The text was updated successfully, but these errors were encountered: