Check tests on PR

[DavertMik]

Summary

This PR adds a GitHub action that analyzes current cypress.io tests and shows a nice overview of them. When a test is added / skipped / removed this analyzer shows it in a comment.

When a PR is created with a test a new label "has tests" is added

Genrally, it will add a comment with full information on current tests:

• Tests added by this Pull Request
• Tests removed by this Pull Request
• Tests skipped by this Pull Request
• Adds label 'has tests' if PR contains tests
• Fails when found .only exclusive tests

Design Decisions

This works as GitHub action on each PR, and compares changes with base branch. Then cypress/integration tests are statically analyzed for changes and a comment is added to PR.

Read more about it here: https://github.com/testomatio/check-tests
Demo: DavertMik/vue-cli#1 (comment)

Purpose

I hope this improves the visibility of e2e tests and helps to manage and review tests.

Demo

mermaid was already analized in my fork and here is the report I got: DavertMik#1 (comment)

I also copy pasted comment to show it here 👇

---

🌀 Tests overview by Testomatio

Found 92 cypress tests in 14 files
No new tests added or removed

📑 List all tests

---

📝 cypress/integration/other/configuration.spec.js

• 📎 Configuration
  • 📎 arrowMarkerAbsolute
    • should handle default value false of arrowMarkerAbsolute
    • should handle default value false of arrowMarkerAbsolute
    • should handle arrowMarkerAbsolute excplicitly set to false
    • should handle arrowMarkerAbsolute excplicitly set to "false" as false
    • should handle arrowMarkerAbsolute set to true

📝 cypress/integration/other/interaction.spec.js

• 📎 Interaction
  • 📎 Interaction - security level loose
    • should handle a click on a node with a bound function
    • should handle a click on a node with a bound function where the node starts with a number
    • should handle a click on a node with a bound url
    • should handle a click on a node with a bound url where the node starts with a number
    • should handle a click on a task with a bound URL clicking on the rect
    • should handle a click on a task with a bound URL clicking on the text
    • should handle a click on a task with a bound function
    • should handle a click on a task with a bound function
  • 📎 Interaction - security level tight
    • should handle a click on a node without a bound function
    • should handle a click on a node with a bound function where the node starts with a number
    • should handle a click on a node with a bound url
    • should handle a click on a node with a bound url where the node starts with a number
    • should handle a click on a task with a bound URL clicking on the rect
    • should handle a click on a task with a bound URL clicking on the text
    • should handle a click on a task with a bound function
    • should handle a click on a task with a bound function
  • 📎 Interaction - security level other, missspelling
    • should handle a click on a node with a bound function
    • should handle a click on a node with a bound function where the node starts with a number
    • should handle a click on a node with a bound url
    • should handle a click on a task with a bound function
    • should handle a click on a task with a bound function

📝 cypress/integration/other/rerender.spec.js

• 📎 Rerendering
  • should be able to render and rerender a graph via API

📝 cypress/integration/other/webpackUsage.spec.js

• 📎 Sequencediagram
  • should render a simple sequence diagrams

📝 cypress/integration/other/xss.spec.js

• 📎 XSS
  • should handle xss in tags
  • should handle xss in tags in non-html mode

📝 cypress/integration/rendering/classDiagram.spec.js

• 📎 Class diagram
  • 1: should render a simple class diagram
  • 2: should render a simple class diagrams with cardinality
  • should render a simple class diagram with different visibilities
  • should render multiple class diagrams
  • 4: should render a simple class diagram with comments
  • 5: should render a simple class diagram with abstract method
  • 6: should render a simple class diagram with static method
  • 7: should render a simple class diagram with Generic class
  • 8: should render a simple class diagram with Generic class and relations

📝 cypress/integration/rendering/current.spec.js

• 📎 State diagram
  • should render a state with states in it

📝 cypress/integration/rendering/flowchart.spec.js

• 📎 Flowcart
  • 1: should render a simple flowchart no htmlLabels
  • 2: should render a simple flowchart with htmlLabels
  • 3: should render a simple flowchart with line breaks
  • 4: should render a simple flowchart with trapezoid and inverse trapezoid vertex options.
  • 4: should style nodes via a class.
  • 5: should render a flowchart full of circles
  • 6: should render a flowchart full of icons
  • 7: should render labels with numbers at the start
  • 8: should render subgraphs
  • 9: should render subgraphs with a title starting with a digit
  • 10: should render styled subgraphs
  • 11: should render a flowchart with long names and class definitions
  • 12: should render color of styled nodes
  • 13: should render hexagons
  • 14: should render a simple flowchart with comments
  • 15: Render Stadium shape
  • 16: Render Stadium shape
  • 17: Chaining of nodes
  • 18: Multiple nodes and chaining in one statement
  • 19: Multiple nodes and chaining in one statement

📝 cypress/integration/rendering/gantt.spec.js

• 📎 Sequencediagram
  • should render a gantt chart
  • Multiple dependencies syntax

📝 cypress/integration/rendering/info.spec.js

• 📎 Sequencediagram
  • should render a simple info diagrams

📝 cypress/integration/rendering/pie.spec.js

• 📎 Pie Chart
  • should render a simple pie diagram
  • should render a simple pie diagram with long labels
  • should render a simple pie diagram with capital letters for labels

📝 cypress/integration/rendering/sequencediagram.spec.js

• 📎 Sequence diagram
  • should render a simple sequence diagram
  • should handle different line breaks
  • 📎 background rects
    • should render a single and nested rects
    • should render rect around and inside loops
    • should render rect around and inside alts
    • should render rect around and inside opts

📝 cypress/integration/rendering/stateDiagram.spec.js

• 📎 State diagram
  • should render a simple state diagrams
  • should render a long descriptions instead of id when available
  • should render a long descriptions with additional descriptions
  • should render a single state with short descr
  • should render a transition descrions with new lines
  • should render a state with a note
  • should render a state with on the left side when so specified
  • should render a state with a note together with another state
  • should render a note with multiple lines in it
  • should render a states with descriptions including multi-line descriptions
  • should render a simple state diagrams
  • should render a simple state diagrams with labels
  • should render state descriptions
  • should render composit states
  • should render multiple composit states
  • should render forks in composit states
  • should render forks and joins
  • should render conurrency states
  • should render a state with states in it
  • Simplest compone state

[klemmchr]

Looks good. Gonna look deeper into it the next days. They also support jest, which we should integrate too. We had coveralls but it seems to be gone for some time. Need to have a look at that.

[knsv]

Much appreciated! We will have good use of this action!

[DavertMik]

Much appreciated! We will have good use of this action!

Thanks! I am the maintainer of this action so ping me if you have issues.

. They also support jest, which we should integrate too.

yeah, however, I am not sure how to list both setups in one action
maybe this should be taken as the next feature to roadmap

[gene1wood]

Looks like these new GitHub actions tests are failing

[klemmchr]

That's already tracked: #1164