-
-
Notifications
You must be signed in to change notification settings - Fork 60
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Qiqqa crashing #264
Comments
What would help are the Qiqqa logfiles, which are stored at
for example: Qiqqa uses a rolling log system, which means that it cycles through logfiles with every run, deleting (very) old ones on application start, so that the log collection will not swamp your disk. Step 1: collecting logfilesAnyway, what would be handy to have is a ZIP (or RAR or 7zip) archive of those log files [of the day when the crash happened, but if you send more, that's fine, I can filter]. Be aware that 7zip (or ZIP or RAR) should compress the logfiles in best compression mode to achieve a small 7z/zip bundle file: that is useful for the next step: emailing this to me: email works fine but does not accept large attachments (beyond about 20-50 MBytes -- have to look up the recent gmail limits, but that's the ballpark figure) Step 2: emailing with subject line containing
|
Dear Ger,
Thanks for getting back to me. The compressed log files are 32Mb, which is too big for e-mail. I will send them to you using the University’s dropoff service. Please would you let me know what e-mail you would like me to send them to (you will get an e-mail with a download link).
Thanks again,
Chris.
|
Hi Chris, 👍 Saw the message above. Download in progress. It's near 1 AM here, so will look at it later tomorrow -- expect a late reply as I'll be off-net most of the day. |
Dear Ger,
Great that it came through OK. Thanks for your help.
Chris,
|
OK,
Now Qiqqa uses hashes like that to uniquely identify PDFs (the key is calculated from the actual file content). Since you're at 800+MB memory usage already when this happens (at least the .NET environment reports this usage) it MAY be a rather huge or otherwise "possibly odd" PDF that's being indexed there: it's not so much that the PDF itself is loaded into RAM, but the extracted text file, produced by a custom-tweaked Next step in digging this one outYou could do me a solid and check your Qiqqa library to see if you can find a PDF file named like that and then send it to me the same way you did the log files bundle. Not a guarantee that that's the culprit, but at least it's top suspect in the current investigation! Name of the PDF would be
If your Qiqqa libraries on your local machine are, for example, located at ... note that path in bold right there! Then you will find your PDF files which are stored in the libraries in a subdirectory such as Might be handy to ZIP that one as well before sending it off to me. Thanks for your assistance! Ger P.S. I'll go through the logs a bit more later, as I saw a few lines I don't recognize as 'usual stuff' right away, but that's not related to this crash/out-of-memory issue, just some stuff that made me go 'hmmmm' 🤔 while I was scanning the logs in paranoid mode. 😉 |
Dear Ger,
Thank you again for looking at this for me. I attach the pdf, which looks OK!
Best wishes,
Chris.
From: Ger Hobbelt <notifications@github.com>
Sent: 28 October 2020 20:01
To: jimmejardine/qiqqa-open-source <qiqqa-open-source@noreply.github.com>
Cc: Chris Hicks <chris.hicks@newcastle.ac.uk>; Author <author@noreply.github.com>
Subject: Re: [jimmejardine/qiqqa-open-source] Qiqqa crashing (#264)
⚠ External sender. Take care when opening links or attachments. Do not provide your login details.
OK,
Had a first look at the logs. First sign of something going bad in a fatal way is this line:
20201027.185126 [Q] ERROR [Daemon.Maintainable:BackgroundWorkerDaemon.DoMaintenance_Infrequent] [817.235M] There was a problem while indexing document 8B84D2BDD7D72C1663B1317DFD67D32EDBB921
System.OutOfMemoryException
Message: Exception of type 'System.OutOfMemoryException' was thrown.
...
Now Qiqqa uses hashes like that to uniquely identify PDFs (the key is calculated from the actual file content).
Since you're at 800+MB memory usage already when this happens (at least the .NET environment reports this usage) it MAY be a rather huge or otherwise "possibly odd" PDF that's being indexed there: it's not so much that the PDF itself is loaded into RAM, but the extracted text file, produced by a custom-tweaked pdfdraw -tt tool, is and that may be the cause, though do note that I am guessing as I haven't seen this type of 'out-of-memory' often enough to remember them. 🤔
Next step in digging this one out
You could do me a solid and check your Qiqqa library to see if you can find a PDF file named like that and then send it to me the same way you did the log files bundle. Not a guarantee that that's the culprit, but at least it's top suspect in the current investigation!
Name of the PDF would be
8B84D2BDD7D72C1663B1317DFD67D32EDBB921.pdf
If your Qiqqa libraries on your local machine are, for example, located at G:\Qiqqa\base, then you can find out quickly when you don't know the storage location off the top of your head by looking at the path reported when Qiqqa starts up:
[image]<https://user-images.githubusercontent.com/402462/97488874-1e8bd480-195f-11eb-99a3-ba027804925e.png>
... note that path in bold right there!
Then you will find your PDF files which are stored in the libraries in a subdirectory such as G:\Qiqqa\base\Guest\documents\8 -- that path depends on the library the PDF is in, so please do a search inside your equivalent of the Qiqqa 'base' directory, which was shown in the screenshot above to dig up the sought 8B84D2BDD7D72C1663B1317DFD67D32EDBB921.pdf.
Might be handy to ZIP that one as well before sending it off to me.
Thanks for your assistance!
Ger
P.S. I'll go through the logs a bit more later, as I saw a few lines I don't recognize as 'usual stuff' right away, but that's not related to this crash/out-of-memory issue, just some stuff that made me go 'hmmmm' 🤔 while I was scanning the logs in paranoid mode. 😉
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#264 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AGVCBDMBOQDGDZCSLNKCSODSNBZ6RANCNFSM4TBFRY3Q>.
|
@quissicks : whoops. Looks like you replied via email to github: that path drops attachments like that as github is very picky. 😢 Better would be to send me an email directly at |
(Maybe same approach as you did for the logfiles as those did arrive fine?) |
Quick prelim release for you to try out: https://github.com/GerHobbelt/qiqqa-open-source/releases/tag/v82.0.7578.6369 I'm curious to know if the crash recurs with this one. A few items in your log files hinted at issues which should be fixed in this version, but the explanation for the main issue remains unsolved, alas. Anyway, please let me know how this one fares. P.S. had a look at the PDF and nothing nasty came up. I must say, though, that now that I've done some testing with a set of large Qiqqa databases, including ones with a bit of their own corruption, that memory use got quite similar to what your logging was saying, so I might not be entirely unsurprised when the new version kicks up a MessageBox about data corruption on application start. |
…ces in the WebLibraryDetails->Library->Document and the library managers, which all reference back to library. This crap has been hackily "resolved" by me in the past using WeakReferences, but it's crap either way, with lots of runtime bugs waiting to happen, such as the ones in jimmejardine#264 So now everyone if refering back to the WebLibraryDetails, which have the longest lifetime of the stuff: the Library takes a long time to load and is alive later. Of course, the current refactored code still does not check the validate the validity of the Library forward references so null references galore. This is to be fixed in subsequent commit(s).
Here's a new version for you to test: https://github.com/GerHobbelt/qiqqa-open-source/releases/tag/v82.0.7579.33985 As it says in the notes: I cannot guarantee that you won't see a 'object null reference' reported, but I am very interested to hear about anything you find, both good and bad. See the release notes at that link: bottom line is that this version does more rigorous checking and reporting of any issues it finds. It also attempts to recover some damaged metadata records it would have discarded before -- though that bit is more about my own botched v76 commercial databases than your actual problem. Another behaviour to watch: how is this one on application exit? Note that it MAY take up to 15 seconds of extra time after you terminate the application while Qiqqa attempts to close itself down in an orderly fashion before pulling the stops, hard like. Enjoy. |
I am also interested in this issue. In the log files, I have an out of memory error which I believe occurs when Qiqqa (v.81s on Windows) crashes.
If needed, I can also send an email with the Log folder as zip file. |
Quick heads up: new release to try: https://github.com/GerHobbelt/qiqqa-open-source/releases/tag/v83.0.7655.37537 Please report anything you observe with the new release. Thanks! |
…ces in the WebLibraryDetails->Library->Document and the library managers, which all reference back to library. This crap has been hackily "resolved" by me in the past using WeakReferences, but it's crap either way, with lots of runtime bugs waiting to happen, such as the ones in jimmejardine/qiqqa-open-source#264 So now everyone if refering back to the WebLibraryDetails, which have the longest lifetime of the stuff: the Library takes a long time to load and is alive later. Of course, the current refactored code still does not check the validate the validity of the Library forward references so null references galore. This is to be fixed in subsequent commit(s).
…ces in the WebLibraryDetails->Library->Document and the library managers, which all reference back to library. This crap has been hackily "resolved" by me in the past using WeakReferences, but it's crap either way, with lots of runtime bugs waiting to happen, such as the ones in jimmejardine/qiqqa-open-source#264 So now everyone if refering back to the WebLibraryDetails, which have the longest lifetime of the stuff: the Library takes a long time to load and is alive later. Of course, the current refactored code still does not check the validate the validity of the Library forward references so null references galore. This is to be fixed in subsequent commit(s).
Qiqqa has crashed a couple of times and then I have been unable to launch qiqqa again. The dialogue says a version of qiqqa is running, but it isn't showing up in the task manager. When I have rebooted the machine I can run qiqqa again and my previous work is there. Please will you let me know if it would be helpful for me to run an support functions so that the issue can be investigated. Many thanks for your help. Chris.
The text was updated successfully, but these errors were encountered: