Test is marked as successful, but Could not take DOM snapshot error is present.

[tommy-anderson]

The problem

We're running Cypress in parallel on a couple of machines, with DEBUG=* logs enabled and are running tests which use the Percy-cypress plugin.

We have noticed that a couple of our runs report a "New Snapshot" from time to time, even if we haven't added another snapshot in our tests, so decided to investigate to figure out what's going on.

In the Cypress debug logs, we've noticed the following lines

cypress:server:request received an error making http request { url: 'http://localhost:5338/percy/snapshot',

[percy] Could not take DOM snapshot "Onboarding Step 7" [percy] [object Object]

the [object Object] output seems like a minor bug :) should be a stringified version of the json so that useful info would be displayed in logs.

The more serious issue is that in case of such errors (network requests) I would expect a couple of retry attempts or something to tackle flakyness, because there was an obvious non-success ( snapshot was not taken ) but the tests don't indicate failure, so it seems like nothing was wrong.

Environment

• Node.js v14.19.1.
• "@percy/cli": "1.1.0",
• "@percy/cypress": "3.1.1",
• Ubuntu VERSION="18.04.4 LTS (Bionic Beaver)"
• bash

[Robdel12]

Hey @tommy-anderson! It's a specific design decision to not fail tests on Percy fails like this since it would be shown on your Percy build that there are missing snapshots. I do agree we need to expand the product to actually mark those missing snapshot builds as failed in the SCM (or needing approval). That's on our roadmap and hopefully will get picked up in our next OKR planning meeting.

I noticed your CLI and SDK version are a bit out of date. I'd try upgrading to the latest CLI (1.5.1) and Cypress SDK (3.1.2) to see if that helps aid these errors. If not, we'll want to figure out why the POST of the DOM snapshot fails for you (full verbose logs would be helpful here).

[tommy-anderson]

Hey there Robdel, thanks for the answer. We'll upgrade our packages and add verbose logs before getting back to you, thank you.

[esetnik]

@Robdel12 I am facing the same issue. About 10% of our percy builds are missing at least one snapshot. In all cases the error handler here is getting triggered. I am using the latest CLI and SDK.

[esetnik]

This might also be the same issue as #451 (reply in thread). It looks like if there's a networking error in the cypress backend then percy fails to upload the snapshots and does not attempt retry. Since missing snapshot builds do not fail on percy it causes all subsequent builds to consider the snapshot as reintroduced.