On CI, sort files & check modified w/ digest intead of mtime

[swrobel]

In my quest to speed up our CI builds I noticed that, despite caching & restoring compiled packs & tmp/cache/webpacker, they were still being recompiled on every test run. It seems this is because the mtimes for all of the files will always be the time that the repo was cloned. As a result, I needed a different strategy for determining whether files were changed, so I'm computing SHA1 digests on a sorted list of files (I was also seeing inconsistency with file ordering between CI runs, so this ensures that the same list of files will be in the same order). To avoid killing performance outside of CI, I left the existing logic in place.

I'm open to suggestions about how to flag this, but I chose the CI env var because it seems that all major CI platforms set it.

It seems a further optimization could be made by caching the output of watched_files_digest because it seems to run a number of times on the same set of files during a CI build, which seems unnecessary. However, I know that a Webpacker.instance exists globally for a running Rails app, so I didn't want to implement that and have it failing to check for modified files when it should be. Any thoughts there? I can open a new issue to discuss separately.

[gauravtiwari]

Thanks, @swrobel. Perhaps you could document that env variable in case folks have to manually supply them (if not available).

However, I know that a Webpacker.instance exists globally for a running Rails app, so I didn't want to implement that and have it failing to check for modified files when it should be. Any thoughts there?

Please do 👍

[joelmichael]

Hello. Thanks for working on this caching issue. This solution seems to work for me in Travis, but not in Circle. On both services, I am caching the relevant directories (the packs directory and tmp/cache/webpacker) and using the latest version of webpacker (3.5.5). Have you been able to get this working in Circle?

[swrobel]

Circle is exactly where I'm using this. Thanks for reminding me, I've been meaning to write a blog post about this. Here are the relevant sections of my config:

      - restore_cache:
          name: Restore webpacker cache
          key: myapp-webpacker-v5

      - restore_cache:
          name: Restore compiled packs
          key: myapp-packs-{{ checksum "tmp/cache/webpacker/.last-compilation-digest-test" }}

      - save_cache:
          name: Store webpacker cache
          key: myapp-webpacker-v5-{{ epoch }}
          paths:
            - tmp/cache/webpacker
          when: always

      - save_cache:
          name: Store compiled packs
          key: myapp-packs-{{ checksum "tmp/cache/webpacker/.last-compilation-digest-test" }}
          paths:
            - public/packs-test
          when: always

I'm making use of the fact that epoch is a timestamp and when restoring, it will take the closest match to the requested filename with the newest epoch (or something like that ... but it works!)

[dwightwatson]

This change more than doubled our build time on Semaphore from 6 minutes up to 14 minutes.

We don't cache public assets between builds, rather we let them get compiled every time.

Maybe our use-case is the outlier here, but I'm concerned this is a dangerous change to introduce on a patch release because of the assumptions it makes about your CI and/or build process.

[swrobel]

@dwightwatson I just can't comprehend how many files it would require to cause that large a jump. In my case, the compilation step of my build takes 3 seconds when no files have changed. I know SHA1 is slower than grabbing mtimes, but that just doesn't seem to add up...

[dwightwatson]

We've got about 349 files - we use Webpacker to crunch our stylesheets and images as well. Having looked through the code too it seemed odd to cause such a dramatic change, but we were able to narrow it down to updating Webpacker from 3.5.3 to 3.5.5. Will keep digging to see what's going on, but thought it was worth dropping a comment here in case other people run into a similar issue.