Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Qiqqa lagging when terminating: FolderWatcher takes forever on large Watch dirtree #50

Closed
GerHobbelt opened this issue Aug 14, 2019 · 2 comments
Closed

Qiqqa lagging when terminating: FolderWatcher takes forever on large Watch dirtree #50

GerHobbelt opened this issue Aug 14, 2019 · 2 comments
Labels
🐛bug Something isn't working ⛷performance Anything that's related to UX: speed of response; I/O speed, etc.
Milestone
v82

Comments

@GerHobbelt
Copy link
Collaborator

GerHobbelt commented Aug 14, 2019 •
edited

HTR:

  • have a directory tree with many files and directories, which is watched by Qiqqa for new PDFs to occur.
  • Quit Qiqqa

Logging will report something like this:

20190814.130928 [Q] DEBUG [Daemon.Maintainable:GeneralTaskDaemon.DoMaintenance] [187.176M] 
      Directory.GetFiles took 300433 ms

(augmented Qiqqa @ commit XXX)

Note the 300 seconds it took GetFiles() to complete.
GerHobbelt added a commit to GerHobbelt/qiqqa-open-source that referenced this issue Aug 14, 2019
@GerHobbelt
- fixed jimmejardine#50 using EnumerateFiles API instead of GetFiles.…
08af03e 
… (TODO: inspect other sites in code where GetFiles is invoked)

- introduced `Utilities.Shutdownable.ShutdownableManager.Instance.IsShuttingDown` as a global flag to check whether the app is terminating: this works the same way as `!thread.IsRunning` but without the need to memoize the thread reference or pass it via extra function arguments.
- Addd a couple of extra `IsShuttingDown` checks to improve 'exit' termination speed of the application.
@GerHobbelt
Copy link
Collaborator Author

Fixed in commit SHA-1: 08af03e

using EnumerateFiles() API instead of GetFiles().

TODO: inspect other sites in code where GetFiles is invoked.

@GerHobbelt GerHobbelt changed the title Qiqqa lagging when terminating: FolderWatcher takes forever on large Watch dirtree ✅Qiqqa lagging when terminating: FolderWatcher takes forever on large Watch dirtree Aug 14, 2019
GerHobbelt added a commit to GerHobbelt/qiqqa-open-source that referenced this issue Aug 15, 2019
@GerHobbelt
refactored: Wasn't happy with the code flow in the FolderWatcher: now…
ac59217 
… the long recursive directory scan (using `EnumerateFiles()`) is only aborted whn the app is terminated or when it has run its course (or when there are more or less dire circumstances); otherwise the dirtreescan is periodically paused to give the machine a bit of air to cope with the results and/or other pending work, while an app exit is very quickly discovered still (just like before, it is also detected inside the `daemon.Sleep(N)` calls in there, so we're good re that one. Tested it and works nicely. jimmejardine#50
GerHobbelt added a commit to GerHobbelt/qiqqa-open-source that referenced this issue Aug 16, 2019
@GerHobbelt
fix for bug introduced by me while working on jimmejardine#50 : on la…
937dc03 
…rge dirtrees being watched, the time limit can be reached and is never reset ==> no more PDFs are added! **Whoops!** **BUG**

Also: when importing PDFs from a watched directory the proper check to see if a PDF is already existing in the library is by checking its HASH, rather than the DownloadSource location, which can be **different** for identical files, or even **identical** when files have been patched in-line using tools such as QPDF (https://github.com/GerHobbelt/qiqqa-revengin)
@GerHobbelt
Copy link
Collaborator Author

Closing and decluttering the issue list so it stays workable for me: fixed in https://github.com/GerHobbelt/qiqqa-open-source mainline=master branch, pending #15 / any maintainer rights/actions.

This was referenced Sep 3, 2019
Closed
Closed
GerHobbelt added a commit to GerHobbelt/qiqqa-open-source that referenced this issue Oct 2, 2019
@GerHobbelt
- fixed jimmejardine#50 using EnumerateFiles API instead of GetFiles.…
2f7779d 
… (TODO: inspect other sites in code where GetFiles is invoked)

- introduced `Utilities.Shutdownable.ShutdownableManager.Instance.IsShuttingDown` as a global flag to check whether the app is terminating: this works the same way as `!thread.IsRunning` but without the need to memoize the thread reference or pass it via extra function arguments.
- Addd a couple of extra `IsShuttingDown` checks to improve 'exit' termination speed of the application.
GerHobbelt added a commit to GerHobbelt/qiqqa-open-source that referenced this issue Oct 2, 2019
@GerHobbelt
refactored: Wasn't happy with the code flow in the FolderWatcher: now…
de329e2 
… the long recursive directory scan (using `EnumerateFiles()`) is only aborted whn the app is terminated or when it has run its course (or when there are more or less dire circumstances); otherwise the dirtreescan is periodically paused to give the machine a bit of air to cope with the results and/or other pending work, while an app exit is very quickly discovered still (just like before, it is also detected inside the `daemon.Sleep(N)` calls in there, so we're good re that one. Tested it and works nicely. jimmejardine#50
GerHobbelt added a commit to GerHobbelt/qiqqa-open-source that referenced this issue Oct 2, 2019
@GerHobbelt
fix for bug introduced by me while working on jimmejardine#50 : on la…
3a7c58f 
…rge dirtrees being watched, the time limit can be reached and is never reset ==> no more PDFs are added! **Whoops!** **BUG**

Also: when importing PDFs from a watched directory the proper check to see if a PDF is already existing in the library is by checking its HASH, rather than the DownloadSource location, which can be **different** for identical files, or even **identical** when files have been patched in-line using tools such as QPDF (https://github.com/GerHobbelt/qiqqa-revengin)
@GerHobbelt GerHobbelt added ⛷performance Anything that's related to UX: speed of response; I/O speed, etc. 🐛bug Something isn't working labels Oct 4, 2019
@GerHobbelt GerHobbelt added this to the v82 milestone Oct 4, 2019
@GerHobbelt GerHobbelt changed the title ✅Qiqqa lagging when terminating: FolderWatcher takes forever on large Watch dirtree Qiqqa lagging when terminating: FolderWatcher takes forever on large Watch dirtree Oct 4, 2019
GerHobbelt added a commit that referenced this issue Nov 5, 2019
@GerHobbelt
- fixed #50 using EnumerateFiles API instead of GetFiles. (TODO: insp…
1053fce 
…ect other sites in code where GetFiles is invoked)

- introduced `Utilities.Shutdownable.ShutdownableManager.Instance.IsShuttingDown` as a global flag to check whether the app is terminating: this works the same way as `!thread.IsRunning` but without the need to memoize the thread reference or pass it via extra function arguments.
- Addd a couple of extra `IsShuttingDown` checks to improve 'exit' termination speed of the application.
GerHobbelt added a commit that referenced this issue Nov 5, 2019
@GerHobbelt
refactored: Wasn't happy with the code flow in the FolderWatcher: now…
2eb1380 
… the long recursive directory scan (using `EnumerateFiles()`) is only aborted whn the app is terminated or when it has run its course (or when there are more or less dire circumstances); otherwise the dirtreescan is periodically paused to give the machine a bit of air to cope with the results and/or other pending work, while an app exit is very quickly discovered still (just like before, it is also detected inside the `daemon.Sleep(N)` calls in there, so we're good re that one. Tested it and works nicely. #50
GerHobbelt added a commit that referenced this issue Nov 5, 2019
@GerHobbelt
fix for bug introduced by me while working on #50 : on large dirtrees…
ec06fdc 
… being watched, the time limit can be reached and is never reset ==> no more PDFs are added! **Whoops!** **BUG**

Also: when importing PDFs from a watched directory the proper check to see if a PDF is already existing in the library is by checking its HASH, rather than the DownloadSource location, which can be **different** for identical files, or even **identical** when files have been patched in-line using tools such as QPDF (https://github.com/GerHobbelt/qiqqa-revengin)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
🐛bug Something isn't working ⛷performance Anything that's related to UX: speed of response; I/O speed, etc.
Projects
None yet
Milestone
v82
Development

No branches or pull requests
1 participant
@GerHobbelt