Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky tests in IndexLabelTest due to non deterministic JSON key ordering #398

Merged
merged 1 commit into from
Dec 13, 2022

Conversation

aditya-kumbhar
Copy link
Contributor

Motivation

IndexLabelTest has the following tests that are flaky, detected using the NonDex tool:
org.apache.hugegraph.unit.IndexLabelTest#testIndexLabel
org.apache.hugegraph.unit.IndexLabelTest#testIndexLabelV49
org.apache.hugegraph.unit.IndexLabelTest#testIndexLabelV56

The tests fail due to non-deterministic ordering of the properties in the JSON string created by the JsonUtil.toJson() method.

Command to reproduce

mvn -pl hugegraph-client edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.apache.hugegraph.unit.IndexLabelTest

Log

[ERROR]   IndexLabelTest.testIndexLabel:47 expected:<...ame":"personByAge","[id":0,"check_exist":true,"user_data":{},"base_type":"VERTEX_LABEL","base_value":"person","index_type":"SECONDARY","fields":["age"],"rebuild":true]}> but was:<...ame":"personByAge","[check_exist":true,"user_data":{},"id":0,"fields":["age"],"base_value":"person","rebuild":true,"index_type":"SECONDARY","base_type":"VERTEX_LABEL"]}>

[ERROR]   IndexLabelTest.testIndexLabelV49:66 expected:<{"id":0,"[name":"personByAge","check_exist":true],"base_type":"VERTEX...> but was:<{"id":0,"[check_exist":true,"name":"personByAge"],"base_type":"VERTEX...>

[ERROR]   IndexLabelTest.testIndexLabelV56:88 expected:<{"[id":0,"name":"personByAge","check_exist":true,"user_data":{},"base_type":"VERTEX_LABEL","base_value":"person","index_type":"SECONDARY]","fields":["age"]}> but was:<{"[name":"personByAge","id":0,"check_exist":true,"user_data":{},"index_type":"SECONDARY","base_type":"VERTEX_LABEL","base_value":"person]","fields":["age"]}>

Changes

Using the jackson ObjectMapper, comparing the JSONNodes instead of raw json strings would make the test deterministic, as the order of the properties will not be compared.

@imbajin
Copy link
Member

imbajin commented Dec 11, 2022

thanks,we will take a look for it soon

and what's the influence in production env?

@codecov
Copy link

codecov bot commented Dec 11, 2022

Codecov Report

Merging #398 (2e88ff7) into master (1c47673) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #398   +/-   ##
=========================================
  Coverage     62.55%   62.55%           
  Complexity     1866     1866           
=========================================
  Files           260      260           
  Lines          9405     9405           
  Branches        872      872           
=========================================
  Hits           5883     5883           
  Misses         3140     3140           
  Partials        382      382           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@aditya-kumbhar
Copy link
Contributor Author

aditya-kumbhar commented Dec 11, 2022

A flaky test may have no direct impact on production environment, but the tests themselves may fail if moved/run in different java environments due to assumptions made about the determinism of the Java APIs. In this case the order of the elements of the json returned by JsonUtil.toJson() is assumed to be same as the predefined order defined in the expected string json.

The Nondex tool helps in identifying such assumptions by exploring different behaviors of non-deterministic Java APIs and reports the flaky tests.

Fixing the flaky tests would make the behavior of the tests consistent across java environments.

Copy link
Contributor

@javeme javeme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@javeme javeme merged commit e02b7d1 into apache:master Dec 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants