Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Meta] support xpack delete and recreate pipeline with same pipeline ID and config string #12387

Closed
2 of 3 tasks
kaisecheng opened this issue Oct 27, 2020 · 0 comments
Closed
2 of 3 tasks
Labels

Comments

@kaisecheng
Copy link
Contributor

kaisecheng commented Oct 27, 2020

When users delete a pipeline and recreate the same pipeline with the same pipeline ID and config string, Logstash is unable to pick up again because the hash value is the same and the hash value does not consider metadata.

{
    "pipeline": "input { generator { count => 100000 }} output { stdout { codec => \"rubydebug\"}} ",
    "last_modified": "2020-10-01T15:42:30.229Z",
    "pipeline_metadata": { "version": "1"},
    "username": "log.stash",
}

If users delete the above pipeline and recreate it with the same config string with a different metadata {"version": "2"}, Logstash won't recreate the pipeline because the hash function only considers "input { generator { count => 100 }} output { stdout { codec => \"rubydebug\"}} " which is the same as before.
The proposal is to change the hash function to take the metadata into account. Logstash will be able to recompile the pipeline.

Version: 8.0.0-SNAPSHOT
Operating System: Mac
Config File (if you have sensitive info, please remove it): xpack.management.pipeline.id: ["pipeline_id"]
Steps to Reproduce:

  1. enable xpack

  2. config xpack.management.pipeline.id with a valid pipeline id

  3. create a pipeline in elasticsearch version 7.10+
    curl -X PUT -v "localhost:9200/_logstash/pipeline/heartbeat" -H 'Content-Type: application/json' -d '{
    "pipeline": "input { generator { count => 100000 }} output { stdout { codec => "rubydebug"}} ",
    "last_modified": "2020-10-01T15:42:30.229Z",
    "pipeline_metadata": { "version": "1", "type": "logstash_pipeline"},
    "username": "log.stash",
    "pipeline_settings": {"pipeline.batch.delay": "50"}
    }'

  4. delete the same pipeline in elasticsearch
    curl -X DELETE "localhost:9200/_logstash/pipeline/heartbeat"

  5. recreate the pipeline with the same content but "version" update to "2"
    curl -X PUT -v "localhost:9200/_logstash/pipeline/heartbeat" -H 'Content-Type: application/json' -d '{
    "pipeline": "input { generator { count => 100000 }} output { stdout { codec => "rubydebug"}} ",
    "last_modified": "2020-10-01T15:42:30.229Z",
    "pipeline_metadata": { "version": "2", "type": "logstash_pipeline"},
    "username": "log.stash",
    "pipeline_settings": {"pipeline.batch.delay": "50"}
    }'

  6. Logstash unable to output count

Action item:

@kaisecheng kaisecheng changed the title support xpack delete and recreate pipeline with same pipeline ID and config string [Meta] support xpack delete and recreate pipeline with same pipeline ID and config string Oct 28, 2020
kaisecheng added a commit to kaisecheng/logstash that referenced this issue Nov 9, 2020
kaisecheng added a commit to kaisecheng/logstash that referenced this issue Nov 9, 2020
kaisecheng added a commit that referenced this issue Nov 9, 2020
@kaisecheng kaisecheng reopened this Nov 11, 2020
kaisecheng added a commit to kaisecheng/logstash that referenced this issue Nov 11, 2020
elasticsearch-bot pushed a commit that referenced this issue Nov 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant