-
Notifications
You must be signed in to change notification settings - Fork 63
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement a junit reporter #241
Comments
Hi @pgrond, At the moment trying to use pa11y reporters with pa11y-ci results in no output (details are captured in #174). Just my opinion, but it is definitely possible, although with some challenges. As as example, here's a (very) quick adaptation of 'use strict';
// Converted to a pa11y-ci reporter from pa11y-reporter-junit at
// https://github.com/macieklewkowicz/pa11y-reporter-junit/blob/master/lib/reporter.js.
const builder = require('junit-report-builder');
// The `results` method is called once for each page that is tested and
// includes the URL and test results. Only accessibility issues are
// included, not all checks.
const results = (results) => {
const suite = builder.testSuite().name(results.pageUrl);
results.issues.forEach((issue) => {
suite
.testCase()
.className(issue.code)
.name(`[${results.pageUrl}] ${issue.selector}`)
.failure(`${issue.message}\n\nContext: ${issue.context}`);
});
};
// The `error` method is called when an error occurs executing pa11y
// against a page.
const error = (message) => {
console.error(message);
};
// The `afterAll` method is called after all pages have been tested.
const afterAll = () => {
const junitReport = builder.build();
console.log(junitReport);
};
module.exports = {
results,
error,
afterAll
}; To run you'd need to pa11y-ci --reporter ./pa11y-ci-reporter-junit.js https://pa11y.org https://github.com/pa11y/pa11y > junit.xml One challenge with these results is that pa11y-ci only reports accessibility issues that are identified, not all checks that are performed, so the results are something like: <?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="https://pa11y.org/" tests="0" failures="0" errors="0" skipped="0"/>
<testsuite name="https://github.com/pa11y/pa11y" tests="127" failures="127" errors="0" skipped="0">
<testcase classname="WCAG2AA.Principle3.Guideline3_2.3_2_2.H32.2" name="[https://github.com/pa11y/pa11y] #query-builder-test-form">
<failure message="This form does not contain a submit button, which creates issues for those who cannot submit the form using the keyboard. Submit buttons are INPUT elements with type attribute "submit" or "image", or BUTTON elements with type "submit" or omitted/invalid.

Context: <form id="query-builder-test-form" action="" accept-charset="UTF-8" method="get">
 <query-builder data-target="...</form>"/>
</testcase>
<testcase classname="WCAG2AA.Principle4.Guideline4_1.4_1_2.H91.Textarea.Name" name="[https://github.com/pa11y/pa11y] #feedback">
<failure message="This textarea element does not have a name available to an accessibility API. Valid names are: label element, title undefined, aria-label undefined, aria-labelledby undefined.

Context: <textarea name="feedback" class="form-control width-full mb-2" style="height: 120px" id="feedback"></textarea>"/>
</testcase>
... (plus 125 more)
</testsuite>
</testsuites> So, it appears as no tests run if all pa11y tests pass. There have been requests to offer an option to list all tests in the results, including those that pass (pa11y/pa11y#687, pa11y/pa11y#700). It seems like having that first would be a more representative JUnit report. |
We are using Bitbucket for running pipelines, and test reporting in Bitbucket only supports Junit format output. There is a pa11y Junit reporter but not one for pa11y-ci.
Is it possible to implement this? Or use the pa11y one? I could not find any documentation on how to use pa11y reporters for pa11y-ci.
The text was updated successfully, but these errors were encountered: