Enable require_alias for Bulk requests for all actions when target is a write alias #29879

kvch · 2022-01-17T16:52:19Z

What does this PR do?

This PR adds support for requiring alias when using ILM. From now on a Selector can tell Elasticsearch client if the target we are shipping events to is an alias or an index. By default, we consider everything an index, and only consider a target an alias when ILM is enabled.

The feature is only supported since ES 7.10, so if the user tries to connect to an older version, we cannot help them with this parameter.

Why is it important?

We see issues around ILM sometimes where users have deleted their write alias causing running beats instances to auto-create an index where the write alias should (with auto-mappings to boot, since the template won't be applied). To a

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Closes #27874

elasticmachine · 2022-01-17T16:55:59Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

elasticmachine · 2022-01-17T17:12:55Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-01-18T08:02:45.787+0000
Duration: 147 min 41 sec
Commit: 3f61d77

Test stats 🧪

Test	Results
Failed	0
Passed	55172
Skipped	5322
Total	60494

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

rdner · 2022-01-17T17:38:35Z

libbeat/esleg/eslegclient/bulkapi.go

+	DocType      string `json:"_type,omitempty" struct:"_type,omitempty"`
+	Pipeline     string `json:"pipeline,omitempty" struct:"pipeline,omitempty"`
+	ID           string `json:"_id,omitempty" struct:"_id,omitempty"`
+	RequireAlias bool   `json:"require_alias,omitempty" struct:"require_alias"`


Is there any reason that it is omitempty for JSON but not for struct?

cmacknz · 2022-01-17T18:56:27Z

libbeat/outputs/elasticsearch/client.go

@@ -333,6 +337,10 @@ func (client *Client) createEventBulkMeta(version common.Version, event *beat.Ev
 	return eslegclient.BulkIndexAction{Index: meta}, nil
 }

+func isRequireAliasSupported(version common.Version) bool {
+	return !version.LessThan(common.MustNewVersion("7.9.0"))


The PR description mentions ES version 7.10. I'd expect either a <= 7.9.0 check or a < 7.10.0 check from that. This reads as not < v7.9.0 to me which allows v7.9.0.

…arget is a write alias (elastic#29879)" This reverts commit 84bf434.

…arget is a write alias (#29879)" (#30058) This reverts commit 84bf434.

* upstream/7.17: (30 commits) [7.17](backport #29966) Add the Elastic product origin header when talking to Elasticsearch or Kibana. (#30000) [Heartbeat] Change size of data on ICMP packet (#29948) (#29978) Add clarification about enableing dashboard loading (#29985) (#29989) Improve aws-s3 gzip file detection to avoid false negatives (#29969) (#29974) ci: docker login step for pulling then pushing (#29960) (#29963) x-pack/auditbeat/module/system/socket: get full length path and arg from /proc when not available from kprobe (#29410) (#29958) [Automation] Update elastic stack version to 7.17.0-ab4975a2 for testing (#29956) [Automation] Update elastic stack version to 7.17.0-1bd58b32 for testing (#29938) [7.17](backport #29913) [Metricbeat] gcp.gke: fix overview dashboard (#29914) [7.17](backport #29605) Fix annotation enrichment (#29834) [Automation] Update elastic stack version to 7.17.0-e1efbe3a for testing (#29922) [Automation] Update elastic stack version to 7.17.0-68da5d12 for testing (#29904) [7.17][Heartbeat] Defer monitor / ICMP errors to monitor runtime / ES (backport #29413) (#29896) Merge pull request from GHSA-rj4h-hqvq-cc6q [7.17](backport #29681) Change docker image from CentOS 7 to Ubuntu 20.04 (#29817) Fix YAML indentation in `parsers` examples (#29663) (#29894) [Automation] Update elastic stack version to 7.17.0-079761a0 for testing (#29864) Fix Filebeat dissect processor field tokenization in documentation (#29680) (#29883) Enable require_alias for Bulk requests for all actions when target is a write alias (#29879) Update Index template loading guide to use the correct endpoint (#29869) (#29877) ...

* upstream/7.17: (30 commits) [7.17](backport elastic#29966) Add the Elastic product origin header when talking to Elasticsearch or Kibana. (elastic#30000) [Heartbeat] Change size of data on ICMP packet (elastic#29948) (elastic#29978) Add clarification about enableing dashboard loading (elastic#29985) (elastic#29989) Improve aws-s3 gzip file detection to avoid false negatives (elastic#29969) (elastic#29974) ci: docker login step for pulling then pushing (elastic#29960) (elastic#29963) x-pack/auditbeat/module/system/socket: get full length path and arg from /proc when not available from kprobe (elastic#29410) (elastic#29958) [Automation] Update elastic stack version to 7.17.0-ab4975a2 for testing (elastic#29956) [Automation] Update elastic stack version to 7.17.0-1bd58b32 for testing (elastic#29938) [7.17](backport elastic#29913) [Metricbeat] gcp.gke: fix overview dashboard (elastic#29914) [7.17](backport elastic#29605) Fix annotation enrichment (elastic#29834) [Automation] Update elastic stack version to 7.17.0-e1efbe3a for testing (elastic#29922) [Automation] Update elastic stack version to 7.17.0-68da5d12 for testing (elastic#29904) [7.17][Heartbeat] Defer monitor / ICMP errors to monitor runtime / ES (backport elastic#29413) (elastic#29896) Merge pull request from GHSA-rj4h-hqvq-cc6q [7.17](backport elastic#29681) Change docker image from CentOS 7 to Ubuntu 20.04 (elastic#29817) Fix YAML indentation in `parsers` examples (elastic#29663) (elastic#29894) [Automation] Update elastic stack version to 7.17.0-079761a0 for testing (elastic#29864) Fix Filebeat dissect processor field tokenization in documentation (elastic#29680) (elastic#29883) Enable require_alias for Bulk requests for all actions when target is a write alias (elastic#29879) Update Index template loading guide to use the correct endpoint (elastic#29869) (elastic#29877) ...

Reinit

d81c39f

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 17, 2022

kvch mentioned this pull request Jan 17, 2022

[libbeat] ILM writes should use require_alias=true #27874

Closed

jlind23 added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Jan 17, 2022

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 17, 2022

kvch added 2 commits January 17, 2022 18:08

add tests

c99ebc6

add changelog entry

fe38070

kvch requested a review from andrewvc January 17, 2022 17:12

feature is only supported since 7.10

1c9ecb1

rdner reviewed Jan 17, 2022

View reviewed changes

cmacknz reviewed Jan 17, 2022

View reviewed changes

kvch added 2 commits January 18, 2022 09:02

address review notes

40aad49

add more tests and follow up in integration tests

3f61d77

rdner approved these changes Jan 18, 2022

View reviewed changes

kvch merged commit 84bf434 into elastic:7.17 Jan 18, 2022

ruflin mentioned this pull request Jan 26, 2022

metricbeat monitoring of Elasticsearch and Kibana broke on recent 7.17.0-SNAPSHOT rpm #29920

Closed

This was referenced Jan 27, 2022

Monitoring data is not shown for the last 15 mins in the cloud cluster. elastic/kibana#123880

Closed

Metricbeat fails to publish elasticsearch metricbeat documents due to require_alias error #30044

Closed

kvch added a commit to kvch/beats that referenced this pull request Jan 27, 2022

Revert "Enable require_alias for Bulk requests for all actions when t…

60a757e

…arget is a write alias (elastic#29879)" This reverts commit 84bf434.

kvch mentioned this pull request Jan 27, 2022

Revert "Enable require_alias for Bulk requests for all actions when t… #30058

Merged

kvch added a commit that referenced this pull request Jan 27, 2022

Revert "Enable require_alias for Bulk requests for all actions when t…

93708bd

…arget is a write alias (#29879)" (#30058) This reverts commit 84bf434.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable require_alias for Bulk requests for all actions when target is a write alias #29879

Enable require_alias for Bulk requests for all actions when target is a write alias #29879

kvch commented Jan 17, 2022 •

edited

Loading

elasticmachine commented Jan 17, 2022

elasticmachine commented Jan 17, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

rdner Jan 17, 2022

cmacknz Jan 17, 2022 •

edited

Loading

Enable require_alias for Bulk requests for all actions when target is a write alias #29879

Enable require_alias for Bulk requests for all actions when target is a write alias #29879

Conversation

kvch commented Jan 17, 2022 • edited Loading

What does this PR do?

Why is it important?

Checklist

Related issues

elasticmachine commented Jan 17, 2022

elasticmachine commented Jan 17, 2022 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

rdner Jan 17, 2022

Choose a reason for hiding this comment

cmacknz Jan 17, 2022 • edited Loading

Choose a reason for hiding this comment

kvch commented Jan 17, 2022 •

edited

Loading

elasticmachine commented Jan 17, 2022 •

edited by jenkins-beats-ci bot

Loading

cmacknz Jan 17, 2022 •

edited

Loading