-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Cypress Studio for Cypress 10 #23544
Merged
Merged
Changes from 18 commits
Commits
Show all changes
30 commits
Select commit
Hold shift + click to select a range
c38e019
chore: wire up Cypress Studio (#23413)
lmiller1990 5089992
Merge remote-tracking branch 'origin/develop' into feat/studio-cypres…
lmiller1990 60dc66c
fix a bug in the assertion API
lmiller1990 451e382
fix bug in assertions [skip ci]
lmiller1990 268d5dc
wip - bugs [skip ci]
lmiller1990 8d0e0c8
feat: add experimentalStudio flag back (#23506)
lmiller1990 abf8f07
Merge remote-tracking branch 'origin/develop' into feat/studio-cypres…
lmiller1990 e90cca7
Merge remote-tracking branch 'origin/develop' into feat/studio-cypres…
lmiller1990 849409c
chore: Add Studio UI to Cypress 10 (#23537)
astone123 cef5a5c
test: studio e2e tests (#23546)
lmiller1990 973f93e
chore: UI feedback
astone123 821179d
fix race condition
lmiller1990 f3728b1
update tests
lmiller1990 3086106
rename test
lmiller1990 3923cc1
improve types in reporter
lmiller1990 507f3b3
remove dead code
lmiller1990 5ffeae7
improve tests
lmiller1990 ffc10d4
merge tests into one spec
lmiller1990 13fde97
chore: Cap instruction modal width; exit studio mode when new spec is…
astone123 21d213f
chore: Only render studio error when test has failed; add test for st…
astone123 5370f8b
Merge branch 'develop' into feat/studio-cypress-10
astone123 d52d637
correctly check if command is studio or not
lmiller1990 24d8718
improve specs and hopefully reduce flake
lmiller1990 c6eab49
communicate studio state from app->reporter
lmiller1990 827d4f5
receive studio save state validity from app
lmiller1990 d1db435
fix test
lmiller1990 81ef719
Merge branch 'develop' into feat/studio-cypress-10
astone123 c48827f
improve test coverage
lmiller1990 14dc87f
Merge remote-tracking branch 'origin/develop' into feat/studio-cypres…
lmiller1990 06debea
fix external link
lmiller1990 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
## Cypress Studio Tests | ||
|
||
These are the tests for the Cypress Studio feature. [Learn more here](https://docs.cypress.io/guides/references/cypress-studio). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
export function launchStudio () { | ||
cy.scaffoldProject('experimental-studio') | ||
cy.openProject('experimental-studio') | ||
cy.startAppServer('e2e') | ||
cy.visitApp() | ||
cy.get(`[data-cy-row="spec.cy.js"]`).click() | ||
cy.visit(`http://localhost:4455/__/#/specs/runner?file=cypress/e2e/spec.cy.js`) | ||
|
||
cy.waitForSpecToFinish() | ||
|
||
// Should not show "Studio Commands" until we've started a new Studio session. | ||
cy.get('[data-cy="hook-name-studio commands"]').should('not.exist') | ||
|
||
cy | ||
.contains('visits a basic html page') | ||
.closest('.runnable-wrapper') | ||
.realHover() | ||
.findByTestId('launch-studio') | ||
.click() | ||
|
||
// Studio re-executes spec before waiting for commands - wait for the spec to finish executing. | ||
cy.waitForSpecToFinish() | ||
|
||
cy.get('[data-cy="hook-name-studio commands"]').should('exist') | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,242 @@ | ||
import { launchStudio } from './helper' | ||
|
||
describe('Cypress Studio', () => { | ||
it('updates an existing test with a click action', () => { | ||
launchStudio() | ||
|
||
cy.getAutIframe().within(() => { | ||
cy.get('p').contains('Count is 0') | ||
|
||
// (1) First Studio action - get | ||
cy.get('#increment') | ||
|
||
// (2) Second Studio action - click | ||
.realClick().then(() => { | ||
cy.get('p').contains('Count is 1') | ||
}) | ||
}) | ||
|
||
cy.get('[data-cy="hook-name-studio commands"]').closest('.hook-studio').within(() => { | ||
cy.get('.command').should('have.length', 2) | ||
// (1) Get Command | ||
cy.get('.command-name-get').should('contain.text', '#increment') | ||
|
||
// (2) Click Command | ||
cy.get('.command-name-click').should('contain.text', 'click') | ||
}) | ||
|
||
cy.get('button').contains('Save Commands').click() | ||
|
||
cy.withCtx(async (ctx) => { | ||
const spec = await ctx.actions.file.readFileInProject('cypress/e2e/spec.cy.js') | ||
|
||
expect(spec.trim()).to.eq(` | ||
astone123 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
it('visits a basic html page', () => { | ||
cy.visit('cypress/e2e/index.html') | ||
/* ==== Generated with Cypress Studio ==== */ | ||
cy.get('#increment').click(); | ||
/* ==== End Cypress Studio ==== */ | ||
})`.trim()) | ||
}) | ||
|
||
// Studio re-executes the test after writing it file. | ||
// It should pass | ||
cy.waitForSpecToFinish({ passCount: 1 }) | ||
|
||
// Assert the commands we input via Studio are executed. | ||
cy.get('.command-name-visit').within(() => { | ||
cy.contains('visit') | ||
cy.contains('cypress/e2e/index.html') | ||
}) | ||
|
||
cy.get('.command-name-get').within(() => { | ||
cy.contains('get') | ||
cy.contains('#increment') | ||
}) | ||
|
||
cy.get('.command-name-click').within(() => { | ||
cy.contains('click') | ||
}) | ||
}) | ||
|
||
it('writes a test with all kinds of assertions', () => { | ||
function assertStudioHookCount (num: number) { | ||
cy.get('[data-cy="hook-name-studio commands"]').closest('.hook-studio').within(() => { | ||
cy.get('.command').should('have.length', num) | ||
}) | ||
} | ||
|
||
launchStudio() | ||
|
||
cy.getAutIframe().within(() => { | ||
cy.get('#increment').rightclick().then(() => { | ||
cy.get('.__cypress-studio-assertions-menu').shadow().contains('be enabled').realClick() | ||
}) | ||
}) | ||
|
||
assertStudioHookCount(2) | ||
|
||
cy.getAutIframe().within(() => { | ||
cy.get('#increment').rightclick().then(() => { | ||
cy.get('.__cypress-studio-assertions-menu').shadow().contains('be visible').realClick() | ||
}) | ||
}) | ||
|
||
assertStudioHookCount(4) | ||
|
||
cy.getAutIframe().within(() => { | ||
cy.get('#increment').rightclick().then(() => { | ||
cy.get('.__cypress-studio-assertions-menu').shadow().contains('have text').realHover() | ||
cy.get('.__cypress-studio-assertions-menu').shadow().contains('Increment').realClick() | ||
}) | ||
}) | ||
|
||
assertStudioHookCount(6) | ||
|
||
cy.getAutIframe().within(() => { | ||
cy.get('#increment').rightclick().then(() => { | ||
cy.get('.__cypress-studio-assertions-menu').shadow().contains('have id').realHover() | ||
cy.get('.__cypress-studio-assertions-menu').shadow().contains('increment').realClick() | ||
}) | ||
}) | ||
|
||
assertStudioHookCount(8) | ||
|
||
cy.getAutIframe().within(() => { | ||
cy.get('#increment').rightclick().then(() => { | ||
cy.get('.__cypress-studio-assertions-menu').shadow().contains('have attr').realHover() | ||
cy.get('.__cypress-studio-assertions-menu').shadow().contains('onclick').realClick() | ||
}) | ||
}) | ||
|
||
assertStudioHookCount(10) | ||
|
||
cy.get('[data-cy="hook-name-studio commands"]').closest('.hook-studio').within(() => { | ||
// 10 Commands - 5 assertions, each is a child of the subject's `cy.get` | ||
cy.get('.command').should('have.length', 10) | ||
|
||
// 5x cy.get Commands | ||
cy.get('.command-name-get').should('have.length', 5) | ||
|
||
// 5x Assertion Commands | ||
cy.get('.command-name-assert').should('have.length', 5) | ||
|
||
// (1) Assert Enabled | ||
cy.get('.command-name-assert').should('contain.text', 'expect <button#increment> to be enabled') | ||
|
||
// (2) Assert Visible | ||
cy.get('.command-name-assert').should('contain.text', 'expect <button#increment> to be visible') | ||
|
||
// (3) Assert Text | ||
cy.get('.command-name-assert').should('contain.text', 'expect <button#increment> to have text Increment') | ||
|
||
// (4) Assert Id | ||
cy.get('.command-name-assert').should('contain.text', 'expect <button#increment> to have id increment') | ||
|
||
// (5) Assert Attr | ||
cy.get('.command-name-assert').should('contain.text', 'expect <button#increment> to have attr onclick with the value increment()') | ||
}) | ||
|
||
cy.get('button').contains('Save Commands').click() | ||
|
||
cy.withCtx(async (ctx) => { | ||
const spec = await ctx.actions.file.readFileInProject('cypress/e2e/spec.cy.js') | ||
|
||
expect(spec.trim()).to.eq(` | ||
it('visits a basic html page', () => { | ||
cy.visit('cypress/e2e/index.html') | ||
/* ==== Generated with Cypress Studio ==== */ | ||
cy.get('#increment').should('be.enabled'); | ||
cy.get('#increment').should('be.visible'); | ||
cy.get('#increment').should('have.text', 'Increment'); | ||
cy.get('#increment').should('have.id', 'increment'); | ||
cy.get('#increment').should('have.attr', 'onclick', 'increment()'); | ||
/* ==== End Cypress Studio ==== */ | ||
})` | ||
.trim()) | ||
}) | ||
}) | ||
|
||
it('creates a test using Studio, but cancels and does not write to file', () => { | ||
launchStudio() | ||
|
||
cy.getAutIframe().within(() => { | ||
cy.get('p').contains('Count is 0') | ||
|
||
// (1) First Studio action - get | ||
cy.get('#increment') | ||
|
||
// (2) Second Studio action - click | ||
.realClick().then(() => { | ||
cy.get('p').contains('Count is 1') | ||
}) | ||
}) | ||
|
||
cy.get('[data-cy="hook-name-studio commands"]').closest('.hook-studio').within(() => { | ||
cy.get('.command').should('have.length', 2) | ||
// (1) Get Command | ||
cy.get('.command-name-get').should('contain.text', '#increment') | ||
|
||
// (2) Click Command | ||
cy.get('.command-name-click').should('contain.text', 'click') | ||
}) | ||
|
||
cy.get('[data-cy="hook-name-studio commands"]').should('exist') | ||
|
||
cy.get('a').contains('Cancel').click() | ||
|
||
// Cyprss re-runs after you cancel Studio. | ||
// Original spec should pass | ||
cy.waitForSpecToFinish({ passCount: 1 }) | ||
|
||
cy.get('.command').should('have.length', 1) | ||
|
||
// Assert the spec was executed without any new commands. | ||
cy.get('.command-name-visit').within(() => { | ||
cy.contains('visit') | ||
cy.contains('cypress/e2e/index.html') | ||
}) | ||
|
||
cy.get('[data-cy="hook-name-studio commands"]').should('not.exist') | ||
|
||
cy.withCtx(async (ctx) => { | ||
const spec = await ctx.actions.file.readFileInProject('cypress/e2e/spec.cy.js') | ||
|
||
// No change, since we cancelled. | ||
expect(spec.trim()).to.eq(` | ||
it('visits a basic html page', () => { | ||
cy.visit('cypress/e2e/index.html') | ||
})`.trim()) | ||
}) | ||
}) | ||
|
||
// TODO: Can we somehow do the "Create Test" workflow within Cypress in Cypress? | ||
it('creates a brand new test', () => { | ||
cy.scaffoldProject('experimental-studio') | ||
cy.openProject('experimental-studio') | ||
cy.startAppServer('e2e') | ||
cy.visitApp() | ||
cy.visit(`http://localhost:4455/__/#/specs/runner?file=cypress/e2e/empty.cy.js`) | ||
|
||
cy.waitForSpecToFinish() | ||
|
||
cy.contains('Create test with Cypress Studio').click() | ||
cy.get('[data-cy="aut-url"]').as('urlPrompt') | ||
|
||
cy.get('@urlPrompt').within(() => { | ||
cy.contains('Continue ➜').should('be.disabled') | ||
}) | ||
|
||
cy.get('@urlPrompt').type('http://localhost:4455/cypress/e2e/index.html') | ||
|
||
cy.get('@urlPrompt').within(() => { | ||
cy.contains('Continue ➜').should('not.be.disabled') | ||
cy.contains('Cancel').click() | ||
}) | ||
|
||
// TODO: Can we somehow do the "Create Test" workflow within Cypress in Cypress? | ||
// If we hit "Continue" here, it updates the domain (as expected) but since we are | ||
// Cypress in Cypress, it redirects us the the spec page, which is not what normally | ||
// would happen in production. | ||
}) | ||
}) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These test are failing when I run them locally, but are passing in CI? Is there something I'm missing when running these tests?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking into this... this test is also failing for me but for a different reason
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah the tests in this file are really flakey for me... I'm still looking into it. Seems like there's some situations where after
launchStudio
runs, we're not in studio mode. Probably a race condition of sortsThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Couldn't get this exact error. I tried a little commit 24d8718 to reduce flake. These run reliably for me, and on CI 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I'm still seeing them flake a bunch in
studio.cy.ts
. I've run them several times and can't get them all to pass at once 🤔 Must just be an issue with my machine. If they're running well in CI then I'd say this isn't a blocker