Elastic agent on k8s #19727

ChrsMark · 2020-07-08T09:33:27Z

What does this PR do?

Add manifests and docs regarding running Agent on k8s.

Why is it important?

It's a suggested way to run Agent on k8s environments in oder to deploy Kubernetes Package (elastic/integrations#70) efficiently.

cc: @exekias

Signed-off-by: chrismark <chrismarkou92@gmail.com>

elasticmachine · 2020-07-08T09:33:29Z

Pinging @elastic/integrations-platforms (Team:Platforms)

elasticmachine · 2020-07-08T09:33:31Z

Pinging @elastic/ingest-management (Team:Ingest Management)

Signed-off-by: chrismark <chrismarkou92@gmail.com>

elasticmachine · 2020-07-08T09:40:33Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #19727 updated]
Start Time: 2020-07-23T13:32:07.387+0000
Duration: 57 min 16 sec

sayden · 2020-07-14T14:30:15Z

x-pack/elastic-agent/docs/running-on-kubernetes.asciidoc

+
+ifeval::["{release-state}"=="unreleased"]
+
+However, version {version} of {agent} has not yet been


I'd delete this paragraph or we may risk to forget about it in the near future. In other words, docs shouldn't "warn" or "announce" future updates or changes

I think we are covered by the ifeval::["{release-state}"=="unreleased"] :)
The same we do also in manifests for x-Beats too: https://github.com/elastic/beats/blame/master/metricbeat/docs/running-on-kubernetes.asciidoc#L9

sayden

LGTM but I'd remove that paragraph I mentioned

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ruflin · 2020-07-23T10:02:58Z

Note: The points below should not block this PR.

@exekias Is this just a "recommendation" or our "supported" way or running the agent on k8s?
@ph We should probably also have a follow up discussion on who owns updating these manifests?

exekias · 2020-07-23T10:09:13Z

deploy/kubernetes/elastic-agent-kubernetes.yaml

+          image: docker.elastic.co/beats/elastic-agent:8.0.0-SNAPSHOT
+          command: ["/bin/sh", "-c"]
+          args: [
+            "yes | ./elastic-agent enroll $KIBANA_HOST $FLEET_ENROLLMENT_TOKEN && ./elastic-agent run"


uhm I wonder if this should be handled by the agent image itself. What will happen here when we are already enrolled but get a pod recreation?

exekias

I saw the last change in the run command, let's discuss that before merging

exekias · 2020-07-23T10:12:11Z

Note: The points below should not block this PR.

@exekias Is this just a "recommendation" or our "supported" way or running the agent on k8s?

This should be the supported way of running the agent, in many cases you can expect some user tweaks. In any case, as of now It feels more like a first stab at it, we should keep iterating over this story and make it sound for GA

ChrsMark · 2020-07-23T10:53:38Z

I saw the last change in the run command, let's discuss that before merging

This change performs the same commands with what docker-entrypoint performs: https://github.com/elastic/beats/blob/master/dev-tools/packaging/templates/docker/docker-entrypoint.elastic-agent.tmpl

enroll:

beats/dev-tools/packaging/templates/docker/docker-entrypoint.elastic-agent.tmpl

Line 56 in 7579d51

./{{ .BeatName }} enroll ${KIBANA_HOST:-http://localhost:5601} $apikey -f
start:

beats/dev-tools/packaging/templates/docker/docker-entrypoint.elastic-agent.tmpl

Line 62 in 7579d51

exec {{ .BeatName }} run "$@"

This was used to work with master versions because of https://github.com/elastic/beats/pull/18445/files#diff-4847a986cf06ab4d26f26f5212720eedR52. However the respective PR was not backported to 7.9 (see

beats/dev-tools/packaging/templates/docker/docker-entrypoint.elastic-agent.tmpl

Line 53 in ab41986

apikey=$(echo $enrollResp | jq -r '.item.api_key')

) and hence the startup script was failing cause the apikey was always overwritten.

I agree we should not patch the startup command of the container with custom one. I will open a PR with this small change for 7.9.

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ChrsMark · 2020-07-28T10:00:34Z

Tested the manifests e2e with a fresh 7.9 docker image. Previous issue resolved, merging this one.

(cherry picked from commit 9f99091)

* upstream/7.9: (32 commits) feat(ci): support storing artifacts for PRs in separate dirs (elastic#20282) (elastic#20301) Cisco ASA: Fix message 106100 (elastic#20245) (elastic#20277) [CI] Change upstream reference (elastic#20296) (elastic#20297) [docs] Fix Windows download link for agent (elastic#20258) (elastic#20290) Cherry-pick to 7.9: [docs] Rename release highlights to what's new (elastic#20255) (elastic#20285) Elastic agent on k8s (elastic#19727) (elastic#20262) [Filebeat Module] Defender ATP - Adding dashboard (elastic#20058) (elastic#20093) fix: use a fixed worker type for tests (elastic#20130) (elastic#20247) [Elastic Agent] Fix Windows powershell install service script (elastic#20203) (elastic#20252) [Ingest Manager] Fixed unzip on older windows (elastic#20088) (elastic#20109) adding possibility to override content-type checks, it was breaking certain webhooks that is not able to set content-headers at all. Still defaults to application/json (elastic#20232) (elastic#20237) [Filebeat][Gsuite] Make GSuite docs more clear (elastic#19981) (elastic#20067) Increase index.max_docvalue_fields_search to 200 (elastic#20218) (elastic#20221) Call host parser only once when building light metricsets (elastic#20149) (elastic#20190) [Metricbeat] Use MySQL Host Parser in Query metricset (elastic#20191) (elastic#20212) [Filebeat] Ignore cylance.protect timestamps while testing (elastic#20207) (elastic#20217) [libbeat] Fix write error in ensureWriter.Write (elastic#20112) (elastic#20145) Cherry-pick elastic#20127 to 7.9: Fix failing unit tests on windows (elastic#20180) Remove f5/firepass rsa2elk fileset (elastic#20160) (elastic#20206) Cherry-pick elastic#20138 to 7.9: [Filebeat] Update crowdstrike module (elastic#20177) ...

(cherry picked from commit b891b34)

ChrsMark added 2 commits July 8, 2020 12:03

Add k8s manifests

5bee148

Signed-off-by: chrismark <chrismarkou92@gmail.com>

Add docs

38114a5

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ChrsMark added Team:Platforms Label for the Integrations - Platforms team Team:Ingest Management labels Jul 8, 2020

ChrsMark self-assigned this Jul 8, 2020

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 8, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 8, 2020

fix

800d663

Signed-off-by: chrismark <chrismarkou92@gmail.com>

sayden reviewed Jul 14, 2020

View reviewed changes

sayden approved these changes Jul 14, 2020

View reviewed changes

exekias approved these changes Jul 23, 2020

View reviewed changes

ChrsMark added 2 commits July 23, 2020 12:36

Fix start commands

f96ed0b

Signed-off-by: chrismark <chrismarkou92@gmail.com>

minor fix

e08e319

Signed-off-by: chrismark <chrismarkou92@gmail.com>

exekias reviewed Jul 23, 2020

View reviewed changes

exekias suggested changes Jul 23, 2020

View reviewed changes

ChrsMark mentioned this pull request Jul 23, 2020

Agent entrypoint fix #20187

Merged

Move back to default startup command

ab8780b

Signed-off-by: chrismark <chrismarkou92@gmail.com>

exekias approved these changes Jul 23, 2020

View reviewed changes

ChrsMark merged commit 9f99091 into elastic:master Jul 28, 2020

ChrsMark mentioned this pull request Jul 28, 2020

Cherry-pick #19727 to 7.9: Elastic agent on k8s #20262

Merged

ChrsMark added a commit to ChrsMark/beats that referenced this pull request Jul 28, 2020

Elastic agent on k8s (elastic#19727)

083c313

(cherry picked from commit 9f99091)

ChrsMark added the v7.9.0 label Jul 28, 2020

ChrsMark added a commit to ChrsMark/beats that referenced this pull request Jul 28, 2020

Elastic agent on k8s (elastic#19727)

3814edb

(cherry picked from commit 9f99091)

ChrsMark mentioned this pull request Jul 28, 2020

Cherry-pick #19727 to 7.x: Elastic agent on k8s #20263

Merged

ChrsMark added the v7.10.0 label Jul 28, 2020

ChrsMark added a commit that referenced this pull request Jul 28, 2020

Elastic agent on k8s (#19727) (#20262)

0a10562

(cherry picked from commit 9f99091)

ChrsMark added a commit that referenced this pull request Jul 28, 2020

Elastic agent on k8s (#19727) (#20263)

c280d56

(cherry picked from commit 9f99091)

melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020

Elastic agent on k8s (elastic#19727)

c3bf700

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

Elastic agent on k8s (elastic#19727) (elastic#20262)

1e1d56f

(cherry picked from commit b891b34)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elastic agent on k8s #19727

Elastic agent on k8s #19727

ChrsMark commented Jul 8, 2020 •

edited

Loading

elasticmachine commented Jul 8, 2020

elasticmachine commented Jul 8, 2020

elasticmachine commented Jul 8, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

sayden Jul 14, 2020

ChrsMark Jul 23, 2020

sayden left a comment

ruflin commented Jul 23, 2020

exekias Jul 23, 2020

exekias left a comment

exekias commented Jul 23, 2020

ChrsMark commented Jul 23, 2020

ChrsMark commented Jul 28, 2020


		ifeval::["{release-state}"=="unreleased"]

		However, version {version} of {agent} has not yet been

Elastic agent on k8s #19727

Elastic agent on k8s #19727

Conversation

ChrsMark commented Jul 8, 2020 • edited Loading

What does this PR do?

Why is it important?

elasticmachine commented Jul 8, 2020

elasticmachine commented Jul 8, 2020

elasticmachine commented Jul 8, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

sayden Jul 14, 2020

Choose a reason for hiding this comment

ChrsMark Jul 23, 2020

Choose a reason for hiding this comment

sayden left a comment

Choose a reason for hiding this comment

ruflin commented Jul 23, 2020

exekias Jul 23, 2020

Choose a reason for hiding this comment

exekias left a comment

Choose a reason for hiding this comment

exekias commented Jul 23, 2020

ChrsMark commented Jul 23, 2020

ChrsMark commented Jul 28, 2020

ChrsMark commented Jul 8, 2020 •

edited

Loading

elasticmachine commented Jul 8, 2020 •

edited by jenkins-beats-ci bot

Loading