You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If we're going to introduce completely new job types in the future, we need to change the way unknown job/datafeed cluster state is validated.
While trying to add categorizer jobs, which are quite similar to anomaly_detector jobs, I ran into the following problem:
Logically, a categorizer job should have no detectors
But the AnalysisConfig class requires detectors
There are two possible solutions that seem reasonable at first glance:
Have categorizer jobs have a categorization_configinstead of analysis_config
Change analysis_config so that detectors is not required if the job_type is categorizer
Unfortunately neither of these works:
Old nodes will ignore categorization_config when parsing metadata, but then error because Job requires an analysis_config
Old nodes will not tolerate an analysis_config with no detectors
This results in the messy solution that categorizer jobs will have to have an analysis_config that includes unnecessary fields - new nodes will ignore these fields and mask them when printing the config in REST responses, but old nodes will show the unnecessary bits
I think the only long term solution that allows the necessary degree of extensibility is to hold Jobs as arbitrary Map<String, Object> or BytesReference when parsing from cluster state, and only interpret what's in the Map or BytesReference if the job_type is understood. This is pretty much how index settings work.
The text was updated successfully, but these errors were encountered:
Original comment by @droberts195:
If we're going to introduce completely new job types in the future, we need to change the way unknown job/datafeed cluster state is validated.
While trying to add
categorizer
jobs, which are quite similar toanomaly_detector
jobs, I ran into the following problem:categorizer
job should have nodetectors
AnalysisConfig
class requiresdetectors
categorizer
jobs have acategorization_config
instead ofanalysis_config
analysis_config
so thatdetectors
is not required if thejob_type
iscategorizer
categorization_config
when parsing metadata, but then error becauseJob
requires ananalysis_config
analysis_config
with no detectorscategorizer
jobs will have to have ananalysis_config
that includes unnecessary fields - new nodes will ignore these fields and mask them when printing the config in REST responses, but old nodes will show the unnecessary bitsI think the only long term solution that allows the necessary degree of extensibility is to hold
Job
s as arbitraryMap<String, Object>
orBytesReference
when parsing from cluster state, and only interpret what's in theMap
orBytesReference
if thejob_type
is understood. This is pretty much how index settings work.The text was updated successfully, but these errors were encountered: