[ML] Delete unused data frame analytics state #50243
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Pinging @elastic/ml-core (:ml) |
This commit adds removal of unused data frame analytics state from the _delete_expired_data API (and in extend th ML daily maintenance task). At the moment the potential state docs include the progress document and state for regression and classification analyses.
dimitris-athanasiou
force-pushed
the
delete-unused-data-frame-analytics-state
branch
from
92dd2e7
to
8d2b5ec
Compare
davidkyle
reviewed
Dec 17, 2019
| @@ -39,6 +39,8 @@ | |||
| public static final ParseField TRAINING_PERCENT = new ParseField("training_percent"); | |||
| public static final ParseField RANDOMIZE_SEED = new ParseField("randomize_seed"); | |||
|
|
|||
| private static final String STATE_DOC_ID_SUFFIX = "_classification_state#1"; | |||
There was a problem hiding this comment.
Should the suffix have the #1 attached, it feels like thats a number that will change
There was a problem hiding this comment.
I also thought about this. We currently handle just a single doc. But we can fix this if and when we switch to handling state split over multiple docs.
| assertInferenceModelPersisted(jobId); | ||
|
|
||
| // Delete the config straight from the config index | ||
| DeleteResponse deleteResponse = client().prepareDelete(".ml-config", DataFrameAnalyticsConfig.documentId(jobId)) |
There was a problem hiding this comment.
Why delete the config? Does DeleteExpiredDataAction require the config to be deleted
There was a problem hiding this comment.
Yes, for the state to become "unused", ie. there is no job owning those state docs.
dimitris-athanasiou
deleted the
delete-unused-data-frame-analytics-state
branch
dimitris-athanasiou
added a commit
to dimitris-athanasiou/elasticsearch
that referenced
this pull request
Dec 17, 2019
This commit adds removal of unused data frame analytics state from the _delete_expired_data API (and in extend th ML daily maintenance task). At the moment the potential state docs include the progress document and state for regression and classification analyses. Backport of elastic#50243
dimitris-athanasiou
added a commit
that referenced
this pull request
Dec 18, 2019
This commit adds removal of unused data frame analytics state from the _delete_expired_data API (and in extend th ML daily maintenance task). At the moment the potential state docs include the progress document and state for regression and classification analyses. Backport of #50243
SivagurunathanV
pushed a commit
to SivagurunathanV/elasticsearch
that referenced
this pull request
Jan 23, 2020
This commit adds removal of unused data frame analytics state from the _delete_expired_data API (and in extend th ML daily maintenance task). At the moment the potential state docs include the progress document and state for regression and classification analyses.
This was referenced Feb 3, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit adds removal of unused data frame analytics state
from the _delete_expired_data API (and in extend th ML daily
maintenance task). At the moment the potential state docs
include the progress document and state for regression and
classification analyses.