Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Qiqqa crashing - out of memory error when Qiqqa (v.81s/v82 on Windows) crashes #281

Open
GerHobbelt opened this issue Dec 29, 2020 · 6 comments
Open

Qiqqa crashing - out of memory error when Qiqqa (v.81s/v82 on Windows) crashes #281

GerHobbelt opened this issue Dec 29, 2020 · 6 comments
Labels
🐛bug Something isn't working
Milestone
v82

Comments

@GerHobbelt
Copy link
Collaborator

I am also interested in this issue. In the log files, I have an out of memory error which I believe occurs when Qiqqa (v.81s on Windows) crashes.
This is the most relevant part from the log:

20201209.073654 DEBUG [Daemon.Maintainable:BackgroundWorkerDaemon.DoMaintenance_Infrequent] [274.918444] DoMaintenance: Breaking out of outer processing loop due to no more files to process (count = 0)
20201209.073654 DEBUG [Daemon.Maintainable:BackgroundWorkerDaemon.DoMaintenance_Infrequent] [274.918444] DoMaintenance_Infrequent LINE: 196
20201209.073654 DEBUG [Daemon.Maintainable:BackgroundWorkerDaemon.DoMaintenance_Infrequent] [274.918444] DoMaintenance_Infrequent LINE: 205
20201209.073654 DEBUG [Daemon.Maintainable:BackgroundWorkerDaemon.DoMaintenance_Infrequent] [274.918444] DoMaintenance_Infrequent END @ LINE: 207
20201209.073655 DEBUG [Daemon.Maintainable:BackgroundWorkerDaemon.DoMaintenance_Infrequent] [274.918444] DoMaintenance_Infrequent START @ LINE: 162
20201209.073658 INFO  [Main] [274.918444] Saving configuration
20201209.073658 INFO  [Main] [274.934772] Saved configuration
20201209.073658 INFO  [Main] [274.942964] +Setting user agent
20201209.073658 INFO  [Main] [274.942964] -Setting user agent
20201209.073658 INFO  [Main] [274.942964] Setting DEFAULT for GeckoFX
20201209.073658 INFO  [Main] [274.942964] Screen position stored as 333|199|1024|700
20201209.073658 ERROR [Main] [275.155956] RemarkOnException.....

System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
   at System.Windows.Media.Composition.DUCE.Channel.SyncFlush()
   at System.Windows.Interop.HwndTarget.UpdateWindowSettings(Boolean enableRenderTarget, Nullable`1 channelSet)
   at System.Windows.Interop.HwndTarget.UpdateWindowPos(IntPtr lParam)
   at System.Windows.Interop.HwndTarget.HandleMessage(WindowMessage msg, IntPtr wparam, IntPtr lparam)
   at System.Windows.Interop.HwndSource.HwndTargetFilterMessage(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
20201209.073658 ERROR [Main] [275.188724] RemarkOnException_GUI_THREAD...

System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
   at System.Windows.Media.Composition.DUCE.Channel.SyncFlush()
   at System.Windows.Interop.HwndTarget.UpdateWindowSettings(Boolean enableRenderTarget, Nullable`1 channelSet)
   at System.Windows.Interop.HwndTarget.UpdateWindowPos(IntPtr lParam)
   at System.Windows.Interop.HwndTarget.HandleMessage(WindowMessage msg, IntPtr wparam, IntPtr lparam)
   at System.Windows.Interop.HwndSource.HwndTargetFilterMessage(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)

If needed, I can also send an email with the Log folder as zip file.

Originally posted by @danieleboaretti in #264 (comment)
@GerHobbelt GerHobbelt added the 🐛bug Something isn't working label Dec 29, 2020
@GerHobbelt GerHobbelt added this to the v82 milestone Dec 29, 2020
@GerHobbelt
Copy link
Collaborator Author

@danieleboaretti Moved your comment to a new issue to keep the original issue very focused.

You can email me your Qiqqa logfiles (zipped, please) to ger at hobbelt.com with a email subject line mentioning both Qiqqa and this issue number at least so the spam filters will be able to decide properly. 😉

For guidance about collecting the logfiles, the comments in #264 are hopefully sufficient to make that a doable task.

I'm still swamped with RL stuff, so please do not expect an answer before the new year has arrived (this weekend); ring the 🛎️ here in the issue tracker if I'm even slower than that.

Oh, and please mention in this issue that you've sent me an email with log files, thanks! 👍

@danieleboaretti
Copy link

@GerHobbelt I have just sent you the zip file, please let me know if you have received the email, it should be compliant with your instructions 😄

@GerHobbelt
Copy link
Collaborator Author

GerHobbelt commented Dec 29, 2020 via email

@danieleboaretti
Copy link

I apologize for that, The email app did not work as expected.

I have just sent you the email. I checked and it has been delivered.

@GerHobbelt GerHobbelt changed the title Qiqqa crashing - out of memory error when Qiqqa (v.81s on Windows) crashes Qiqqa crashing - out of memory error when Qiqqa (v.81s/v82 on Windows) crashes Dec 31, 2020
GerHobbelt added a commit to GerHobbelt/qiqqa-open-source that referenced this issue Jan 1, 2021
@GerHobbelt
- worked on jimmejardine#281 / jimmejardine#280: SORAX is again one o…
e3da81e 
…f the culprits it seems.

  Added memory status report code to dump into the logging as the GC output is not reliable when we want to know how much memory was actually used at the peak, when the OutOfMemory exception happened.
- lots of CultureInfo.CurrentCulture params are not necessary and only make the code harder to read. Removed where MSVC2019 doesn't yak about these any more.
- added *reason* message for logging lines which report when Qiqqa is being shut down. This should help diagnose jimmejardine#280 / jimmejardine#281 as well.
- tweaked the exit process to track and report the total number of threads running as well as only the SafeThreadPool
- document comments for several routines re LDA and AutoTag creation. Finally understanding how it's done. The LDA is *not* used for creating the AutoTags but only to find the *commonly shared* tags when running in Expedition.
  The BuzzwordGenerator is the AutoTag generator (in combination with some other code) and is using some plain basic heuristics. autoTags are extracted from the *document title* only, which explains the lousy AutoTag performance I got with the latest lib, as that one has very few PDFs which pass through OCR unscathed.
@GerHobbelt
Copy link
Collaborator Author

@danieleboaretti : Quick heads up: new release to try: https://github.com/GerHobbelt/qiqqa-open-source/releases/tag/v83.0.7655.37537

Please report anything you observe with the new release. Thanks!

@GerHobbelt GerHobbelt mentioned this issue Jan 16, 2021
Closed
@GerHobbelt
Copy link
Collaborator Author

@danieleboaretti : Quick heads up: hotfix release to try: https://github.com/GerHobbelt/qiqqa-open-source/releases/tag/v83.0.7656.6401 (which fixes known issue in previous release https://github.com/GerHobbelt/qiqqa-open-source/releases/tag/v83.0.7655.37537)

Please report anything you observe with the new release. Thanks!

GerHobbelt added a commit to GerHobbelt/qiqqa-open-source that referenced this issue Jan 17, 2021
@GerHobbelt
- fixing SORAX-related crashes by moving all SORAX actions into the U…
a3292bc 
…I thread; the lib uses COM under the hood, which requires a working and accessible Windows message pipe, something which only the UI thread can provide.

- littered the code with WPFDoEvents UI/not-UI assertions -- which caught the above scenario in a Dispose() for a page image render. And that was the hint the needed to progress a little further towards stibility: it was SORAX which caused a *lot* of the out-of-memory failures due to crazy COM/WPF/UI failures, even for smaller libraries under test.
- fix bit of an odd crash in the Lucene flush/cleanup during shutdown, where Lucene kept busy with 'optimizing the index' while a quick application termination was happening in the background, resulting in lockup and then a crash.
- this MAY be a fix for the reported "number of documents reported not matching reality": added update/refresh code to update the library list panel when PDF documents are added in the background via FolderWatcher or other means (async library loading). WARNING: this code is still incomplete/buggy!
- most UI assertions have been covered now. Keeping them anyway as this is hairy stuff and should be tested more.

Addresses (but is not guaranteed to fix) jimmejardine#290, jimmejardine#283, jimmejardine#281, jimmejardine#280, jimmejardine#243
@GerHobbelt GerHobbelt mentioned this issue Feb 27, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
🐛bug Something isn't working
Projects
None yet
Milestone
v82
Development

No branches or pull requests
2 participants
@GerHobbelt @danieleboaretti