Not waiting for reporters to finish in synchronous mode

[keitharobertson]

The problem

It seems like the test runner is not waiting for the reporters to finish before exiting, which results in the last events not being handled.

Environment

• WebdriverIO version: 4.4.0
• Node.js version: v6.9.1
• Standalone mode or wdio testrunner: wdio testrunner
• if wdio testrunner, running synchronous or asynchronous tests: synchronous
• Additional wdio packages used (if applicable): wdio-allure-reporter, wdio-mocha-framework

Details

I am using the wdio-allure-reporter. I kept getting 'Could not find any allure results' when trying to generate the allure report. When I look in the allure-results directory, there is no xml file for the test suite. If I add an after hook with a browser.pause(5000), it works as expected. This seems to be more prevalent when running a long test and/or running on a system under load.

I modified the allure reporter to log out messages when events are being handled. I notice that the events for the last few commands are not handled before the process exits. If I put in the pause(5000), I do see the final events getting processed. For example, without the pause, the suite:end event is not processed and as a result no xml file is output to allure-results. With the pause, it is processed and an xml file is output.

[keitharobertson]

When it fails (it is intermittent), the global.browser.on events are executed in runner.js which sends the events back to the parent process in launcher.js, but the event is not handled there (messageHandler() is never called). I added console logs to the events. In the case where it works as it should, there are logs from messageHandler in launcher.js that correspond to the logs from the events in runner.js. When it does not work correctly, there are many unmatched logs from runner.js at the end.

[keitharobertson]

process.send is async nodejs/node#760. The problem is if you are doing large things (like screenshots), it is possible for the data to not be sent back before the process.exit. This can be fixed by moving the process.exit into the process.send callback on the runner:end event runner.js line 190. If this is changed from

process.send({
    event: 'runner:end',
    failures: failures,
    cid: this.cid,
    specs: this.specs
})
process.exit(failures === 0 ? 0 : 1)

to be this (process.exit in process.send callback), the problem is fixed

process.send({
    event: 'runner:end',
    failures: failures,
    cid: this.cid,
    specs: this.specs
}, null, {}, () => process.exit(failures === 0 ? 0 : 1))

[christian-bromann]

@krobertson92 can you provide a minimal reproducible example?

[christian-bromann]

@krobertson92 thanks for filing the issue. It got fixed and will be released with the next framework adapter releases