Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Account for service being triggered twice in tests #80000

Merged
merged 1 commit into from
Oct 28, 2021

Conversation

davidkyle
Copy link
Member

There is a race in MlDailyMaintenanceServiceTests where the service may be scheduled twice before the main thread waiting on the latch catches up as described in this code comment

// This could happen if the main test thread happens to be waiting for a CPU for the whole 100ms after the

Unfortunately the main thread sometimes takes longer than 100ms as we can see in the log messages

[2021-10-27T18:44:34,942] ... [MlDailyMaintenanceServiceTests] triggering scheduled [ML] maintenance tasks
...
[2021-10-27T18:44:35,112] ... [MlDailyMaintenanceServiceTests] Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask |  
[2021-10-27T18:44:35,267] ... [MlDailyMaintenanceServiceTests] triggering scheduled [ML] maintenance tasks

The second trigger causes the failures. The fix is to change the assertions to atLeast as the test only wants to prove that the tasks are all triggered in the event of a failure it does not matter if they are triggered twice.

Closes #79971

@davidkyle davidkyle added >test Issues or PRs that are addressing/adding tests :ml Machine learning v7.16.1 v8.1.0 v8.0.0-beta1 labels Oct 28, 2021
@elasticmachine elasticmachine added the Team:ML Meta label for the ML team label Oct 28, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (Team:ML)

@davidkyle davidkyle added the auto-backport-and-merge Automatically create backport pull requests and merge when ready label Oct 28, 2021
Copy link
Contributor

@droberts195 droberts195 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@davidkyle davidkyle merged commit fea9217 into elastic:master Oct 28, 2021
@davidkyle davidkyle deleted the fix-nightly-service-test branch October 28, 2021 11:08
@elasticsearchmachine
Copy link
Collaborator

💚 Backport successful

Status Branch Result
7.16

davidkyle added a commit to davidkyle/elasticsearch that referenced this pull request Oct 28, 2021
A race can cause the service under test to be triggered twice.
As the test is asserting that the trigger occurs at least once multiple
invocations should not be considered a failure
elasticsearchmachine pushed a commit that referenced this pull request Oct 28, 2021
)

A race can cause the service under test to be triggered twice.
As the test is asserting that the trigger occurs at least once multiple
invocations should not be considered a failure
elasticsearchmachine pushed a commit that referenced this pull request Oct 28, 2021
)

A race can cause the service under test to be triggered twice.
As the test is asserting that the trigger occurs at least once multiple
invocations should not be considered a failure
weizijun added a commit to weizijun/elasticsearch that referenced this pull request Oct 28, 2021
…formance

* upstream/master: (153 commits)
  [ML] update truncation default & adding field output when input is truncated (elastic#79942)
  [ML] stop using isAllowedByLicense for model license checks (elastic#79908)
  [ML] Retain built-in ML roles granting Kibana privileges (elastic#80014)
  [Transform] remove old mixed cluster BWC layers, not required for 8x (elastic#79927)
  Increase test timeout for CoordinatorTests testAllSearchesExecuted
  [Transform] add rolling upgrade tests for upgrade endpoint (elastic#79721)
  [ML] Update trained model docs for truncate parameter for bert tokenization (elastic#79652)
  `CoordinatorTests` sometimes needs three term bumps (elastic#79574)
  [ML] Account for service being triggered twice in tests (elastic#80000)
  SearchContext: remove unused variable (elastic#79917)
  Revert "Deprecate resolution loss on date field (elastic#78921)" (elastic#79914)
  Re-enable GeoIpDownloaderIT#testStartWithNoDatabases() (elastic#79907)
  Fix SnapshotBasedIndexRecoveryIT#testSeqNoBasedRecoveryIsUsedAfterPrimaryFailOver (elastic#79469)
  Fix RecoverySourceHandlerTests (elastic#79546)
  SQL: stabilize SqlSearchPageTimeoutIT (elastic#79928)
  Wait 3 seconds for the server to reload trust (elastic#79778)
  Skip automatically preserved request headers when rewriting (elastic#79973)
  Check whether stdout is a real console (elastic#79882)
  Convert remote license checker to use LicensedFeature (elastic#79876)
  Miscellaneous fixes for LDAP SDK v6 upgrade (elastic#79891)
  ...

# Conflicts:
#	libs/x-content/src/main/java/org/elasticsearch/xcontent/support/filtering/FilterPath.java
#	libs/x-content/src/test/java/org/elasticsearch/xcontent/support/filtering/FilterPathGeneratorFilteringTests.java
#	libs/x-content/src/test/java/org/elasticsearch/xcontent/support/filtering/FilterPathTests.java
@danhermann danhermann added v7.16.0 and removed v7.16.1 labels Nov 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport-and-merge Automatically create backport pull requests and merge when ready :ml Machine learning Team:ML Meta label for the ML team >test Issues or PRs that are addressing/adding tests v7.16.0 v8.0.0-beta1 v8.1.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[CI] MlDailyMaintenanceServiceTests testBothTasksAreTriggered_DeleteJobsTaskFails failing
5 participants