-
Notifications
You must be signed in to change notification settings - Fork 6.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SIGSEGV calling EventListener's onCompactionBegin() callback method with rocksdbjni 7.4.5 #10686
Comments
I met the same problem, besides, I found NoClassDefFoundError in the dump output.
|
Hey there, looks like this has been out standing for quite a while. But I recently met the same issue - Adding Wondering if there's any ongoing work addressing this issue? Thanks! |
As a workaround, I tried implementing the constructor of my subclass of rocksdb/java/src/main/java/org/rocksdb/AbstractEventListener.java Lines 73 to 89 in 9d37408
only calling super(ON_BACKGROUND_ERROR, ON_ERROR_RECOVERY_BEGIN, ON_ERROR_RECOVERY_COMPLETED) for what I need.
At least now I'm not seeing the application exiting nor getting the Hope that narrows the problem down a little, perhaps. |
Hello @dparrella @PuGuanyu @zz-x404 Thank you for reporting this issue and sorry it took us so long to pick it up. Do you think you will be able to recompile RocksDB in debug mode with If you are not able to compile source code by yourself, can you please provide Maven project with JUnit test where you are able to reproduce this error. Radek |
Hello Hello @dparrella @PuGuanyu @zz-x404, I'm able to replicate your issue. It's really problem with class loader. Looks simmilar like #691 . I will submit fix soon. Radek |
Hello RocksDB Team!
My team has found a serious issue in production where RocksDB crashed and took the JVM down with it. This started when we introduced an
EventListener
that we were hoping would give us more targeted logging of events as they occur. In fact we added this to help with investigating #10258. The issue occurs specifically when background compaction starts and RocksDB attempts to call ourEventListener
'sonCompactionBegin()
method.Any insights as to what we may be doing wrong or help to resolve the issue would be much appreciated!
Expected behavior
Background compaction event callbacks should not crash the JVM.
Actual behavior
JVM exits with SIGSEGV while calling
onCompactionBegin()
callback method onEventListener
.Partial dump from crash:
Steps to reproduce the behavior
We defined a simple
EventListener
implementation that subscribes to the following events:Then in our
onCompactionBegin()
implementation we log (perhaps a bit too verbosely) using SLF4J:What's strange is that we saw this for the first time in one of our production environments. The same code made its way through several stages of test environments before going to production.
Also worth noting is that the data being loaded into RocksDB is identical to the previously deployed version of our application. The RocksDB version is also the same. The only difference is that we added the
EventListener
.One thing that is different with this production system is the size of the database. We store a "snapshot" of the database in a
tar.gz
file that is downloaded when the application bootstraps. The compressed archive is about 2.5 GB - much larger than what we have in our test systems.The listener is added like this:
The text was updated successfully, but these errors were encountered: