Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lost JUnit support for V8 tests #236

Closed
Tracked by #44650
targos opened this issue Jul 20, 2022 · 7 comments · Fixed by nodejs/node#44049
Closed
Tracked by #44650

Lost JUnit support for V8 tests #236

targos opened this issue Jul 20, 2022 · 7 comments · Fixed by nodejs/node#44049

Comments

@targos
Copy link
Member

targos commented Jul 20, 2022

We've been floating an old patch for some time (nodejs/node@d8b4a7e) but unfortunately it doesn't apply anymore since V8 did a big refactor of its test runner.

This breaks V8 CI: https://ci.nodejs.org/job/node-test-commit-v8-linux/4781/
@targos
Copy link
Member Author

targos commented Jul 20, 2022

@nodejs/python @nodejs/build

@targos targos mentioned this issue Jul 20, 2022
Closed
@bnoordhuis
Copy link
Member

It's possible to fix up the patch but it's arguably a better idea to just use the built-in --json-test-results flag and transform the JSON file to JUnit, or teach Jenkins the JSON format (possibly harder, I don't know how much harder.)

@kvakil
Copy link

kvakil commented Jul 25, 2022

Here's a script which does the JSON -> JUnit conversion:
https://gist.github.com/kvakil/8ed172559f5bc3c8391ec34065682599

It seems to work on all the cases I tried locally (on Linux: passed,
timeout, and failed test cases). One weird thing is that the JSON output
doesn't include all the successful test cases, only the failing ones and
the slowest slow_tests_cutoff ones. Of course, you can set
slow_tests_cutoff arbitrary high in order to get all the output, which
is what that script assumes:

$ python3 tools/run-tests.py --gn --arch=x64 --json-test-results $(pwd)/v8-tap.json --slow-tests-cutoff=1000000 cctest
$ python3 json_to_xml_tap.py < v8-tap.json > v8-tap.xml

@bnoordhuis
Copy link
Member

Seems like a good forward. Can I suggest you PR it to nodejs/node? I don't know how to hook it up to the CI but someone else no doubt does.

One style nit: the project's python style is two space indent, not four.

@cclauss
Copy link

cclauss commented Jul 29, 2022

Would this make sense to add to https://github.com/nodejs/tap2junit ?

  • A question, not a suggestion unless others agree.

kvakil added a commit to kvakil/node that referenced this issue Jul 30, 2022
@kvakil
build: convert V8 test JSON to JUnit XML
1673df7 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: nodejs/node-v8#236
@kvakil kvakil mentioned this issue Jul 30, 2022
Merged
@kvakil
Copy link

kvakil commented Jul 30, 2022

Seems like a good forward. Can I suggest you PR it to nodejs/node? I don't know how to hook it up to the CI but someone else no doubt does.

One style nit: the project's python style is two space indent, not four.

PR is now at nodejs/node#44049 (with your style nit addressed).

Would this make sense to add to https://github.com/nodejs/tap2junit ?

I'm not opposed to this. I can't find tap2unit anywhere in the nodejs repo with git grep tap2junit. Is it just hanging out in the Jenkins job config somewhere?

@cclauss
Copy link

cclauss commented Jul 30, 2022

https://github.com/nodejs/tap2junit is a URL that you can click on.

@targos targos mentioned this issue Sep 15, 2022
Closed
5 tasks
bnoordhuis pushed a commit to kvakil/node that referenced this issue Sep 18, 2022
@kvakil @bnoordhuis
build: convert V8 test JSON to JUnit XML
a8bf726 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: nodejs/node-v8#236
targos pushed a commit to targos/node that referenced this issue Sep 21, 2022
@kvakil @targos
[hack] build: convert V8 test JSON to JUnit XML
290d7a0 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: nodejs/node-v8#236
targos pushed a commit to targos/node that referenced this issue Sep 25, 2022
@kvakil @targos
[hack] build: convert V8 test JSON to JUnit XML
e6d8a3a 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: nodejs/node-v8#236
targos pushed a commit to nodejs/node that referenced this issue Sep 25, 2022
@kvakil @targos
[hack] build: convert V8 test JSON to JUnit XML
f4afab1 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: nodejs/node-v8#236
nodejs-github-bot pushed a commit that referenced this issue Sep 26, 2022
@kvakil @nodejs-github-bot
[hack] build: convert V8 test JSON to JUnit XML
d3cfc11 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: #236
nodejs-github-bot pushed a commit that referenced this issue Sep 27, 2022
@kvakil @nodejs-github-bot
[hack] build: convert V8 test JSON to JUnit XML
e2ddca5 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: #236
targos pushed a commit to targos/node that referenced this issue Sep 28, 2022
@kvakil @targos
[hack] build: convert V8 test JSON to JUnit XML
37376f3 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: nodejs/node-v8#236
nodejs-github-bot pushed a commit that referenced this issue Sep 28, 2022
@kvakil @nodejs-github-bot
[hack] build: convert V8 test JSON to JUnit XML
f616c96 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: #236
targos pushed a commit to targos/node that referenced this issue Sep 29, 2022
@kvakil @targos
[hack] build: convert V8 test JSON to JUnit XML
a1c73f0 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: nodejs/node-v8#236
nodejs-github-bot pushed a commit that referenced this issue Sep 29, 2022
@kvakil @nodejs-github-bot
[hack] build: convert V8 test JSON to JUnit XML
de6c887 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: #236
nodejs-github-bot pushed a commit that referenced this issue Sep 30, 2022
@kvakil @nodejs-github-bot
[hack] build: convert V8 test JSON to JUnit XML
2b0bde8 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: #236
nodejs-github-bot pushed a commit that referenced this issue Oct 1, 2022
@kvakil @nodejs-github-bot
[hack] build: convert V8 test JSON to JUnit XML
29e647f 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: #236
nodejs-github-bot pushed a commit that referenced this issue Oct 2, 2022
@kvakil @nodejs-github-bot
[hack] build: convert V8 test JSON to JUnit XML
05fcd4b 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: #236
nodejs-github-bot pushed a commit that referenced this issue Oct 3, 2022
@kvakil @nodejs-github-bot
[hack] build: convert V8 test JSON to JUnit XML
7db5605 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: #236
nodejs-github-bot pushed a commit to nodejs/node that referenced this issue Oct 3, 2022
@kvakil
build: convert V8 test JSON to JUnit XML
691f8d1 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: nodejs/node-v8#236
PR-URL: #44049
Fixes: nodejs/node-v8#236
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
panva pushed a commit to panva/node that referenced this issue Oct 4, 2022
@kvakil @panva
build: convert V8 test JSON to JUnit XML
5832c33 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: nodejs/node-v8#236
PR-URL: nodejs#44049
Fixes: nodejs/node-v8#236
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
danielleadams pushed a commit to nodejs/node that referenced this issue Oct 11, 2022
@kvakil @danielleadams
build: convert V8 test JSON to JUnit XML
37808b3 
This introduces some code to convert from V8's test JSON output to JUnit
XML. We need this because V8's latest refactor of their test runner has
made it difficult to float our JUnit reporter patch on top (see the
referenced issue).

I also think that there needs to be the same changes to vcbuild.bat, but
I don't know how to do test those yet. I can create a Windows VM and
test it if we decide to go with this approach.

Refs: nodejs/node-v8#236
PR-URL: #44049
Fixes: nodejs/node-v8#236
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

build: convert V8 test JSON to JUnit XML kvakil/node
4 participants
@bnoordhuis @targos @cclauss @kvakil