Test Metrics Count Incorrect in ReportPortal

[Priyanshu19992002]

Hi Team,

We have a serenity bdd cucumber framework in which we are using ReportPortal v23.1 with below dependency version:
Serenity - v3.3.10
Serenity-ReportPortal-Integration - v1.5.7

We are facing problem with respect to test counts in ReportPortal. We have executed the tests in parallel. The test metrics in ReportPortal is not matching with the actual test case executed. The error logs which we are seeing is as below:

07:01:54.055 [rp-io-39] ERROR c.e.r.utils.SubscriptionUtils - Start test item completed with error
2024-04-05T07:01:54.0902133Z com.epam.reportportal.exception.ReportPortalException: Report Portal returned error
2024-04-05T07:01:54.0903045Z Status code: 406
2024-04-05T07:01:54.0903627Z Status message: Not Acceptable
2024-04-05T07:01:54.0905013Z Error Message: Start time of child ['Fri Apr 05 06:52:56 UTC 2024'] item should be same or later than start time ['2024-04-05T06:52:58.183'] of the parent item/launch '9828'
2024-04-05T07:01:54.0906165Z Error Type: CHILD_START_TIME_EARLIER_THAN_PARENT
2024-04-05T07:01:54.0906596Z
2024-04-05T07:01:54.0907847Z at com.epam.reportportal.service.ReportPortalErrorHandler.handleError(ReportPortalErrorHandler.java:85)
2024-04-05T07:01:54.0909470Z at com.epam.reportportal.service.ReportPortalErrorHandler.handle(ReportPortalErrorHandler.java:61)
2024-04-05T07:01:54.0911132Z at com.epam.reportportal.restendpoint.http.HttpClientRestEndpoint$1.subscribe(HttpClientRestEndpoint.java:526)
2024-04-05T07:01:54.0912856Z at io.reactivex.internal.operators.maybe.MaybeCreate.subscribeActual(MaybeCreate.java:45)
2024-04-05T07:01:54.0914102Z at io.reactivex.Maybe.subscribe(Maybe.java:4290)
2024-04-05T07:01:54.0915264Z at io.reactivex.internal.operators.maybe.MaybeSubscribeOn$SubscribeTask.run(MaybeSubscribeOn.java:54)
2024-04-05T07:01:54.0916941Z at io.reactivex.internal.schedulers.ScheduledDirectTask.call(ScheduledDirectTask.java:38)
2024-04-05T07:01:54.0918395Z at io.reactivex.internal.schedulers.ScheduledDirectTask.call(ScheduledDirectTask.java:26)
2024-04-05T07:01:54.0919259Z at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2024-04-05T07:01:54.0919922Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2024-04-05T07:01:54.0920644Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2024-04-05T07:01:54.0921187Z at java.lang.Thread.run(Thread.java:750)

What can be the possible reason behind this?

[grey-rain]

hi @Priyanshu19992002 !
Could you please let me know how are your tests paralleled? There are multiple ways to do it in Cucumber. It would be great if you could provide test sample which helps to replicate the issue.
BTW, is there a chance to try the same with the latest integration version 1.6.0?

[Priyanshu19992002]

Hi @grey-rain ,

1. We are using Serenity v3.3.10 in which we are running our test parallely at feature level.
2. We have serenity bdd framework in place where we test multiple applications (each application test are running in parallel at feature level). We are observing the above mentioned issue with respect to one particular application in which a single test takes longer time to execute in comparison to other application. Could this be the reason for this particular issue pls clarify and let us know for further questions.

Note- We haven't tried with v1.6.0 since it is not compatible with serenity v3.3.10.

[Priyanshu19992002]

Hi @grey-rain,

This fix #176 seems to be similar issue fix. I wanted to check this solution. Is it released? If not then how can i verify this fix with my serenity bdd code. Can u pls help me on this asap as this issue is growing with more usage of ReportPortal in our team?

[grey-rain]

@Priyanshu19992002 I do not think it was released. Let me push an update, so that you could check. Fingers crossed it helps 🤞🏻

[grey-rain]

@Priyanshu19992002 1.6.1 has been released. Please let me know if it resolves the issue.

[Priyanshu19992002]

Hi @grey-rain ,

Thanks for the update. Can u pls answer my below query also before i test this new release with my application?

Is the latest release 1.6.1 compatible with Serenity v3.3.10? If no then how can we test with this v3.3.10 of Serenity as it will take sometime for us to move to Serenity v4.x. Any help or suggestion would be appreciated.

[Priyanshu19992002]

Hi @grey-rain,

I do not see the release in the Maven Repository as shown below. Generally, how much time it will take to reflect in Maven repository?

[Priyanshu19992002]

Hi @grey-rain ,

Any update when the new release v1.6.1 will be available in maven central repository? I am not able to proceed with the latest v1.6.1 since it is not available in maven repository yet.

[grey-rain]

@Priyanshu19992002 hm, something went wrong. I'll recheck publish job once again.

[grey-rain]

@Priyanshu19992002 unfortunately, Serenity 3.3 is not being fully supported after 1.6+ release.
If you know that particular change made in #176 potentially can help you, you can make a fork of version 1.5.7 which you have and check if it fixes the issue.
If it does, you can live on share that fork with others on your team, or propose a PR into current (original) repo for merging. We can merge and deploy it under some version labeled like 1.5.8.

[Priyanshu19992002]

Hi @grey-rain ,

Thanks for the update. Pls let me know when the new release v1.6.1 is published to maven central. We have started working on upgrading our application to Serenity v4 so that we can use this new release v1.6.1

[Priyanshu19992002]

Hi @grey-rain ,

Any update on v1.6.1 release? I see the previous release of v1.6.1 is reverted back.

[Priyanshu19992002]

Hi @grey-rain ,

As per the recommendation we have upgraded our Serenity version of the application to v4.0.0. We are looking forward to the new release v1.6.1 of the serenity-reportportal-integration plugin with #176 fix. Kindly let us know when the new version 1.6.1 is released and available in the maven central.

[grey-rain]

@Priyanshu19992002 we're facing an issue with publishing releases. I am working on fixing it.
In a meanwhile, could you please try version 1.6.2-SNAPSHOT ? It should have the fix you're waiting for.
I will let you know when 1.6.1 release is available (once I fix the publisher in #183).

I am sorry about delays. :(

[Priyanshu19992002]

Hi @grey-rain ,

Thanks for the update. Let me check the snapshot version.

[Priyanshu19992002]

Hi @grey-rain ,

I tried testing the snapshot version but testing a snapshot version is restricted in our organization. I would wait for the v1.6.1 to get released into maven central so pls let me know once the new version v1.6.1 is released.

[grey-rain]

ok, @Priyanshu19992002 , I will to make it ASAP

[grey-rain]

1.6.1 is available on Maven Central, please check.

[Priyanshu19992002]

Hi @grey-rain ,

Thank you for the update and for addressing the above issue which i have raised. I appreciate your prompt response and effort in releasing the new version.

I tested the new version with my application although the above reported error logs is no longer coming but still the test count is not matching. I did further debugging and found exactly same issue recently reported by a user #185.

Once one of the example test in a test scenario fails the other failed examples in the same scenario is not reported in ReportPortal. This is the main bug behind the test count mismatch.

Can you please check this issue and let me know if there is any quick fix for it? Let me know if there's any additional information needed from my side to assist you in troubleshooting this issue.

[grey-rain]

@Priyanshu19992002 thank you for kind words! ❤️
I am currently working on #185, I plan to rise a PR in the nearest time.

[grey-rain]

@Priyanshu19992002 1.6.2 is released, it contains qty fix for #185.