Add automated visual regression testing

[schlessera]

Feature description

We're close to a point where we practically have feature parity with non-AMP in most use cases.

Because of that, there are a few scenarios in the plugin by now that could highly benefit from automated visual regression testing:

• Non-AMP to AMP conversion
• AMP to optimized AMP conversion
• Theme conversion

This can be done more or less efficiently with any of the popular visual regression testing tools like BackstopJS or AyeSpy with the current process:

1. Use a WordPress reference site and run snapshots of test URLs that are then saved as reference images. This means we don't have to maintain a set of reference images, we just consider WordPress without the AMP plugin to be the canonical source.
2. Install and activate AMP plugin.
3. Add filter to disable optimizer.
4. Create snapshots of test URLs, compare with reference images and report on results.
5. Make snapshots from 4. new reference images. This way, we can compare the optimizer separately.
6. Remove filter to disable optimizer.
7. Create snapshots of test URLs, compare with reference images and report on results.

Benefits:

• Easy to detect regressions when changing the logic to process the markup or the styling, saving peer review time.
• Easy to pinpoint issues where we still fail to provide feature parity, increasing overall robustness of our turnkey solutions.
• Quantitative assertion of how close we are to making AMP fully transparent for WordPress users, allowing use to market these qualities with confidence.

---

Do not alter or remove anything below. The following sections will be managed by moderators only.

Acceptance criteria

Implementation brief

QA testing instructions

Demo

Changelog entry

[amedina]

Should we expand this to having both visual and performance regression testing? The former to track AMP Content generation (and the visual impact of the optimized content), and the latter to track core web vitals and other metrics. We can also split it into separate issues.

[westonruter]

For generating a site with unit test data and creating a monster post, here's what I've used in the past: https://github.com/westonruter/amp-wp-theme-compat-analysis/blob/913d6f7f18682b092db7657cae970da8e0d1c5c5/start.sh#L19-L40

[schlessera]

Playing around with the preload hero image transformer makes it obvious we need better ways of testing sanitizers and optimizers.

I'm prioritising this so we can build transformers like the preload hero images one with confidence.

[schlessera]

Note: we have a Percy license for the AMP plugin that we can use for this, which is also what Web Stories use.