-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
On CI, sort files & check modified w/ digest intead of mtime #1522
Conversation
Thanks, @swrobel. Perhaps you could document that env variable in case folks have to manually supply them (if not available).
Please do 👍 |
Hello. Thanks for working on this caching issue. This solution seems to work for me in Travis, but not in Circle. On both services, I am caching the relevant directories (the packs directory and tmp/cache/webpacker) and using the latest version of webpacker (3.5.5). Have you been able to get this working in Circle? |
Circle is exactly where I'm using this. Thanks for reminding me, I've been meaning to write a blog post about this. Here are the relevant sections of my config: - restore_cache:
name: Restore webpacker cache
key: myapp-webpacker-v5
- restore_cache:
name: Restore compiled packs
key: myapp-packs-{{ checksum "tmp/cache/webpacker/.last-compilation-digest-test" }}
- save_cache:
name: Store webpacker cache
key: myapp-webpacker-v5-{{ epoch }}
paths:
- tmp/cache/webpacker
when: always
- save_cache:
name: Store compiled packs
key: myapp-packs-{{ checksum "tmp/cache/webpacker/.last-compilation-digest-test" }}
paths:
- public/packs-test
when: always I'm making use of the fact that |
This change more than doubled our build time on Semaphore from 6 minutes up to 14 minutes. We don't cache public assets between builds, rather we let them get compiled every time. Maybe our use-case is the outlier here, but I'm concerned this is a dangerous change to introduce on a patch release because of the assumptions it makes about your CI and/or build process. |
@dwightwatson I just can't comprehend how many files it would require to cause that large a jump. In my case, the compilation step of my build takes 3 seconds when no files have changed. I know SHA1 is slower than grabbing mtimes, but that just doesn't seem to add up... |
We've got about 349 files - we use Webpacker to crunch our stylesheets and images as well. Having looked through the code too it seemed odd to cause such a dramatic change, but we were able to narrow it down to updating Webpacker from 3.5.3 to 3.5.5. Will keep digging to see what's going on, but thought it was worth dropping a comment here in case other people run into a similar issue. |
In my quest to speed up our CI builds I noticed that, despite caching & restoring compiled packs &
tmp/cache/webpacker
, they were still being recompiled on every test run. It seems this is because the mtimes for all of the files will always be the time that the repo was cloned. As a result, I needed a different strategy for determining whether files were changed, so I'm computing SHA1 digests on a sorted list of files (I was also seeing inconsistency with file ordering between CI runs, so this ensures that the same list of files will be in the same order). To avoid killing performance outside of CI, I left the existing logic in place.I'm open to suggestions about how to flag this, but I chose the CI env var because it seems that all major CI platforms set it.
It seems a further optimization could be made by caching the output of
watched_files_digest
because it seems to run a number of times on the same set of files during a CI build, which seems unnecessary. However, I know that aWebpacker.instance
exists globally for a running Rails app, so I didn't want to implement that and have it failing to check for modified files when it should be. Any thoughts there? I can open a new issue to discuss separately.