Extremely slow in jest testing env

[ms88privat]

Why is it so slow in jest? You can also see it in the examples, which had to increase the testing timeout to be able to finish the test. Totally unusable :(

I tried to pinpoint it and it is the fs.readStream it looks like. I will investigate further later on. Just opened this issue as a reference.

[NimaSoroush]

Hi @ms88privat,

Would you please tell me how long it takes to run your test?

Yeah, being a bit slow is totally expected behavior, as it needs to launch browser, load the page, go through steps, capture screenshot, store it and compare it. All these processes take a significant amount of time. I've already tried to optimised this process by storing the captured image in memory instead of hard disk and using the fastest image comparison library.
On my local machine, I have 36 tests that run in 126 seconds. I managed to run same tests on docker which took around 92 seconds to complete (3 seconds per test) which is quite acceptable for me.
You should also consider your test scenario as well. what are your steps and how long it takes to reach the point that you would want to capture the screenshot. You should always optimize that as well.

I am happy to help more when I know more details about your issue.

Nima

[NimaSoroush]

@ms88privat : Please let me know node and jest version you are running your test on

[ms88privat]

Are you using jest as the test runner? I'm on node 7.6.0 and jest 20.0.4.
https://github.com/NimaSoroush/differencify/blob/master/examples/jest-example.test.js#L22
Here in your example, the one test has 40 seconds timeout. Why did you choose this? At my place it nearly takes 20+ seconds for this test, thats why this option is necessary.

But I did some research and it is just about the file system access. I didn't use your library for it and I didn't use puppeteer or image compare. It is just loading the images into the buffer which takes all the time. If I start the same script without jest environment (just via node CLI), it is super fast.

Let me try upgrading jest. 3 sec per test would be fine, but 20+ sec is not :)

[ms88privat]

Upgrading jest didn't solve it.

[ms88privat]

So it isn't related to your library. I'm just curios if it is working fast enough on your side. But I didn't believed so because of the extreme timeout option you did set in your example. That's why I opened this issue. But if it is working on your side, feel free to close this one

[NimaSoroush]

Hi @ms88privat:

Thanks for providing more info. I am wondering why on jest env should cause this issue for you. As I said I am not seeing this issue in my local env and docker. I will close this issue but lets keep posting here till we understand what is the root cause as it might be the case for other peoples.

I will also update the extreme timeout on jest example. That was my arbitrary number when I was developing this package.

Nima

[ms88privat]

I see, it looks like the resolution makes the difference. Please try using "1920x1080" instead of your default 800x600 or so.
But this shouldn't be an issue anyway because the same code via node CLI is super fast.

[NimaSoroush]

What is the image size you getting by "1920x1080"?

I am actually trying { width: 1600, height: 1200 } and it is pretty fast.

Can you try it with .capture({ omitBackground: true }) it will remove the useless background and makes the image a bit lighter

[NimaSoroush]

@ms88privat : Would you please run you test with debug enabled? Basically:

 const differencify = new Differencify({ debug: true });

this should give you the timestamp of each step execution. Please post the outcome here

[ms88privat]

So it looks like the test itself is "fast" but the whole test case did take 30s. It is really slow after the tab closes and before the image comparison (confirms my suspicions about the file reading step). The image itself is 26KB.

I created a fresh app with "create-react-app". Node and everything else is up2date.

My test setup looks like this:

import Differencify from 'differencify';
const differencify = new Differencify({ debug: true });

describe('tests differencify', () => {
  it('validate github page appear correctly', async () => {
    await differencify
      .init()
      .resize({ width: 1600, height: 1200 })
      .goto('http://localhost:3000/')
      .capture()
      .toMatchSnapshot()
      .close()
      .end();
  });
});

Here the debug test output:
The timestamps looking normal, but it did stop the time at least for 15 seconds in between..

[ms88privat]

Just tested the same code on coworker's computer. Same result :(
It really get's stuck in between. Which OS are you on? We are working on macOS.

But yeah, its a problem with Jest somehow... Just reading an image files does also not work, without using your library at all. Thanks anyway! 👍

[ms88privat]

I just saw the same pattern in your test-fail.png in the readme? Only one test, which takes 17s, but the whole test suite took almost 40s?

[NimaSoroush]

Ok, Now I see where does the problem come from. It is actually the image comparison part that takes a long time. Just tried to make all my tests fail locally and yeah, it takes a long time for every single test to be compared with the original screenshot.

I already have some ideas around this. I am using Jimp as the image processing lib which uses pixelmatch for image comparison. Jimp also does perceptual hash comparison of generated images which I guess will be the most time-consuming bit.

I will do a benchmark on this and will update the result here.

[NimaSoroush]

Ok, here is my the result on running first test of jest-example.test.js

Original run

Disabling perceptual hash comparison

Disabling perceptual hash comparison and no writing diff image to disk

No writing diff image to disk

So basically it is writing to disk that takes a long time in case of failure. As you might already know if you pass saveDifferencifiedImage: false to differencify then it wont write the diff image to disk and we are doing this as we are running out tests in CI and Docker. we only enable that flag when there is a failure on tests. you might want to do the same

[NimaSoroush]

As an improvement, I am happy to put perceptual hash comparison behind a config flag. Maybe:

 const differencify = new Differencify({ compareDistance: true });

[NimaSoroush]

Meantime we can look into how to improve image write to disk functionality. I just played with some Jimp config and I managed to reduce Original run's time from 46.537s to 14.794s by sacrificing image quality.
I can put this under config flag too. I will put a PR for this

[ms88privat]

Thanks for further investigating this. But for me, if I just import Jimp and do Jimp.read inside jest, it also takes so much time. But when I execute the same line of code Jimp.read or the whole stuff and invoke it via node directly without jest, it is super fast, even in best quality!

[NimaSoroush]

@ms88privat : Interesting! Would you be able to verify the same behavior with pixelmatch using with/without Jest?

[ms88privat]

pixelmatch is fast on both. But for pixelmatch to work I need to read the images to compare, which is slow again on jest...

[NimaSoroush]

@ms88privat : Ok then you confirm that it is actually reading image from the disk is the slow path! This is pretty odd to me. Image size on disk is around 27kb which I expect fs.readSync should be quite fast on this.
I need to spend more time on this and doing some benchmarks to understand what is going on

[ms88privat]

maybe related to this? jestjs/jest#2925