Add visual tests support

[geriux]

Related PRs:

• [Mobile] - Support Visual Regression testing WordPress/gutenberg#47845

Description

This PR adds the setup to be able to have visual tests, the idea is to start migrating the manual test we currently do each release and to also start expanding our test suites, like testing themes and important components like RichText.

We will be using two new libraries, jest-image-snapshot which enables us to compare two images using Jest. It compares the baseline image against the new one generated on CI, if the difference is over the threshold it will mark the test as failed and it will return the difference output as base64 in the console, this will be handy to check quickly what's breaking.

The other library is jimp which will be used to edit the screenshots taken using Appium, mostly to remove the status bar, resize and reduce quality.

As you'll see in the files changed and in the new tests, we just need to write them as we were doing for the other E2E tests, to match a visual change we'll just need to do:

const screenshot = await takeScreenshot();
expect( screenshot ).toMatchImageSnapshot();

To test CI checks should pass.

PR submission checklist:

• I have considered adding unit tests where possible.
• I have considered if this change warrants user-facing release notes more info and have added them to RELEASE-NOTES.txt if necessary.

[fluiddot]

@geriux Seems we'd need to tweak the CI job configuration to address the following error in Test Android on Device - Full:

EACCES: permission denied, mkdir '/home/circleci/project/__device-tests__/__image_snapshots__/__diff_output__'

[peril-wordpress-mobile]

Wanna run full suite of Android and iOS UI tests? Click here and 'Approve' CI job!

[geriux]

@geriux Seems we'd need to tweak the CI job configuration to address the following error in Test Android on Device - Full:

Yup we needed to create the directory for the snapshot difference output, I included it in a2ab57c

[dcalhoun]

This looks great!

[geriux]

Hey there 👋 I've just updated this PR as there were some conflicts. I think I've addressed all comments but please let me know if there are any other questions, thank you!

[dcalhoun]

Looking forward to merging this. I left one question related to test clean up.

Additionally, what is the command used to run these tests locally?

[dcalhoun]

I noticed the previous test has what appears to be clean up code removing the block. Should all tests including this one have that as well? This question applies to __device-tests__/gutenberg-editor-group-visual.test.js as well.

Suggested change

	expect( screenshot ).toMatchImageSnapshot();
	expect( screenshot ).toMatchImageSnapshot();
	await editorPage.removeBlockAtPosition( blockNames.gallery );

[geriux]

It will depend on the test, for this case, to start with a "blank" canvas in the second test you need to remove the blocks you created before, as the other tests in the file will run in the same session.

[dcalhoun]

Additionally, what is the command used to run these tests locally?

Never mind. I was running the correct command locally: npm run device-tests:local -- gutenberg-editor-gallery-visual.test.js.

The errors I encountered locally related to the e2e tests' requirement of Xcode 13. I only had Xcode 14 installed. Installing Xcode 13 and sudo xcode-select -s /Applications/Xcode\ 13.app allowed me to run the new visual tests successfully. 🎉

[geriux]

Additionally, what is the command used to run these tests locally?

Never mind. I was running the correct command locally: npm run device-tests:local -- gutenberg-editor-gallery-visual.test.js.

The errors I encountered locally related to the e2e tests' requirement of Xcode 13. I only had Xcode 14 installed. Installing Xcode 13 and sudo xcode-select -s /Applications/Xcode\ 13.app allowed me to run the new visual tests successfully. 🎉

Glad you got it running! Yeah until we upgrade to Appium 2, we are stuck with the Xcode 13 issue, becuase SauceLabs currently doesn't support Appium 2 for simulators on Android.