Filtering out all scenarios in the story file causes NPE

[jsedlace]

We use Jbehave with serenity, we also quite often use filtering of stories/scenarios with metafilters.

What is not working now:
one story file, 2 scenarios inside, each scenario is tagged with some tag (doesn't matter if same for all or different)
If metafilter is written so that all scenarios are filtered out, but the story itself is not filtered out, it fails with NPE:

java.lang.NullPointerException
        at com.github.valfirst.jbehave.junit.monitoring.JUnitScenarioReporter.afterStory(JUnitScenarioReporter.java:114)
        at org.jbehave.core.reporters.DelegatingStoryReporter.afterStory(DelegatingStoryReporter.java:51)
        at org.jbehave.core.reporters.ConcurrentStoryReporter.afterStory(ConcurrentStoryReporter.java:133)
        at org.jbehave.core.embedder.PerformableTree$PerformableStory.perform(PerformableTree.java:841)
        at org.jbehave.core.embedder.PerformableTree.performCancellable(PerformableTree.java:418)
        at org.jbehave.core.embedder.PerformableTree.perform(PerformableTree.java:389)
        at org.jbehave.core.embedder.StoryManager$EnqueuedStory.call(StoryManager.java:292)
        at org.jbehave.core.embedder.StoryManager$EnqueuedStory.call(StoryManager.java:266)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

It's quite unpleasant for us, because even if all scenarios pass (real tests), this fails the maven build.

If you can point me to some specific class where to focus with fixing, I'll be happy to provide PR, I have some ideas where to go, but the code is quite complex and even if I fix it somewhere, it does not need to be the right place (I may solve symptoms, not the root cause).

Reproducer is attached.
jbehave-reproducer.zip

[jsedlace]

@valfirst I created test for the issue in the project and tried to fix it.

[valfirst]

@jsedlace , thank you for your contribution, I'll take a deep look today, but at first glance you're fixing symptom, anyway I'll use your test story to check behavior

[jsedlace]

One more finding - according to JBehave grammar, there cannot be story without any scenario, if story contains just one scenario, the 'Scenario:' keyword is optional, but at least one step is mandatory. So if I tried story file without any scenario or step, it failed as well somewhere in JUnitScenarioReporter, but because it was invalid story file, it is acceptable (it created rootDescription with 3 stories

• BeforeStories
• [Empty Story without steps - invalid one]
• AfterStories

If all scenarios are filtered out, rootDescription contains just

• BeforeStories
• AfterStories

Generation of descriptions happens in com.github.valfirst.jbehave.junit.monitoring.JUnitDescriptionGenerator#createDescriptionFrom(org.jbehave.core.embedder.PerformableTree) - storyDescription is added to list of all description only if it contains at least one allowed scenario - but in my case, all are filtered out, so no story is added.

It looks that there is a conflict in behaviour of jbehave filtering - story is allowed, even if all scenarios are not allowed, and behavior of jbehave-junit-runner - story is not put to descriptions if at least one scenario is not allowed.

[valfirst]

@jsedlace thank you for investigation, I've looked deeper, and from my point of view the root cause is in JBehave core, I think story having only filtered out scenarios shouldn't be executed. I see 2 options:

1. Report JBehave issue, provide a fix (I think I can do it), wait for review/approvals.
2. Option 1 + at the same time apply your fix to workaround the issue (it should be removed after JBehave release).

What's impact of this issue? Does it have a high priority for you?

[jsedlace]

@valfirst I already looked into jbehave-core little bit, because it seemed to me like the right place - filtering stories/scenarios (is/is not allowed to be run) is done on one place, so fix should be quite simple, I cannot think out all consequences now, but unit tests can lead me.

Regarding impact/priority - I'll ask the team, but we have this issue for few weeks already, it actually started when we filtered out more tags because of some flakyness of tests/incompatible tests. I'll let you now, we can use our snapshot build for a while probably

[valfirst]

@jsedlace,

• I filed a new JBehave issue: JBEHAVE-1301
• and provided PRs:
  • https://github.com/jbehave/jbehave-core/pull/195
  • Fix #136: handle case when all scenarios in story are filtered out #140

[jsedlace]

@valfirst Thanks, good job. jbehave-core 4.3.4 was released, do you plan to release also jbehave-junit-runner with this fix?

[valfirst]

@jsedlace , yes, I plan to release jbehave-junit-runner tomorrow

[valfirst]

@jsedlace, release 2.2.0 is published