[Elastic Agent] Support the install, control, and uninstall of Endpoint #19248

blakerouse · 2020-06-17T14:09:08Z

What does this PR do?

Adds the ability to install and uninstall Endpoint.

Adds new exec_file to the transpiler.StepList so file execution can be defined in a spec file. This allows the endpoint.spec file to encode all the actions required to install, check if installed, and uninstall Endpoint.

pre_uninstall steps added to the spec file and new uninstall-operation was added to perform uninstallation of applications.

check_install steps added to the spec file along with a new InstallerChecker that handles checking if the installation steps need to be ran.

Breakdown of the installation:

Download endpoint (doesn't work yet, must be present in data/downloads, in a custom format until the required changes are released on Endpoint side)
Verify endpoint
Check if endpoint install under data/install exists, if it does call ./endpoint-security verify which will verify if its installed. If it returns 0 then it will skip install, as its already installed. (Currently this command doesn't exist, so it will always return non-zero, so installation is always performed. Once added it will just work!)
If needs install, extract endpoint to data/install, then exec ./endpoint-security install --upgrade --resources endpoint-security-resources.zip
Register endpoint with the GRPC server and start the v0 connection information socket.
Endpoint is running!

Breakdown of the uninstallation:

Send stop over GRPC to endpoint (Not currently connected, so times out after 30 seconds).
Run the ./endpoint-security uninstall command from the PreUninstallSteps.
Endpoint is uninstalled!

Why is it important?

To support Endpoint.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

…fferent app types.

…unning.

elasticmachine · 2020-06-17T14:11:11Z

Pinging @elastic/ingest-management (Team:Ingest Management)

elasticmachine · 2020-06-17T14:16:48Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #19248 updated]
Start Time: 2020-06-26T20:00:36.715+0000
Duration: 34 min 12 sec

Test stats 🧪

Test	Results
Failed	0
Passed	540
Skipped	127
Total	667

michalpristas · 2020-06-18T16:07:50Z

x-pack/elastic-agent/pkg/agent/operation/operator.go

@@ -175,6 +179,7 @@ func (o *Operator) start(p Descriptor, cfg map[string]interface{}) (err error) {
 func (o *Operator) stop(p Descriptor) (err error) {
 	flow := []operation{
 		newOperationStop(o.logger, o.config),
+		newOperationUninstall(o.logger, p, o.uninstaller),


what in case of multiple outputs running, you have metricbeat A and B A is stopped and uninstalled? it is the same binary as metricbeat for output B

Is that even the case today? When looping with detectPrograms it only loops through the Supported programs once, creating an array. So only 1 metricbeat will be running.

The only case where that is not true with the sidecars for metrics and logs of the running metricbeat and filebeat.

At the moment uninstall doesn't do anything, other than call the pre_uninstall hooks. So this will not cause an issues currently.

x-pack/elastic-agent/pkg/agent/transpiler/steps.go

x-pack/elastic-agent/pkg/artifact/install/hooks/hooks_installer.go

michalpristas · 2020-06-29T07:06:14Z

can you provide a guide for testing this?

michalpristas

it makes sense to me, code looks good, but i havent tested it locally

michalpristas · 2020-06-29T07:08:20Z

x-pack/elastic-agent/pkg/agent/application/info/agent_id.go

@@ -21,13 +21,13 @@ import (

 // defaultAgentConfigFile is a name of file used to store agent information
 const defaultAgentConfigFile = "fleet.yml"
-const agentInfoKey = "agent_info"
+const agentInfoKey = "agent"


note: this is breaking between 7.8 and 7.9 so maybe we should put a note that agent might change ID in between versions when updating from 7.8

should we just handle it? support reading it with either agent_info or agent?

Lets document it, breaking changes are ok from 7.8->7.9

Created #19488 that handles it transparently, or we can just document. I prefer to just handle it, but okay with breaking.

michalpristas · 2020-06-29T07:15:13Z

x-pack/elastic-agent/pkg/agent/transpiler/steps.go

+	}
+
+	// timeout defaults to 60 seconds
+	if r.Timeout == 0 {


can we take that to a comment?

comment where?

blakerouse · 2020-06-29T12:53:11Z

Went ahead and merged this, wanting to get it into 7.x ASAP so we can get some real testing from the snapshot builds, which should allow Agent to download Endpoint directly from the artifact store.

Working on a follow up for the agent_info I think we can support that out of the box, instead of breaking upgrade.

…nt (elastic#19248) * Initial spec parsing for endpoint. * Update comment. * Fix spec test. * Update code so it copies the entire input. * Fix ast test. * Merge agent-improve-restart-loop * Merge agent-endpoint-spec * Refactor core/plugin/app into mostly core/ and use core/plugin for different app types. * Work on endpoint service application. * More fixes. * Fix format and tests. * Fix some imports. * More cleanups. * Fix export comment. * Pass the program.Spec into the descriptor. * Run endpoint verify, install, and uninstall when endpoint should be running. * Fix install and uninstall of Endpoint * Fix some small issues with service app. * Add changelog entry. * Fix lint and tests. * Fix lint. * Remove the code no longer needed because of newer config format. * Fix rules and review. * Update to Endpoint Security. * Fix issues so endpoint security runs. * Add comments. * Update docstring. * Some more fixes. * Delete the extra endpoint testdata files. * Add timeout to exec_file step. * Fix supported map. * Fix getting support programs by cmd. * Improve app started checks. * Fix buildspec. (cherry picked from commit 0fe1554)

…nt (#19248) (#19497) * Initial spec parsing for endpoint. * Update comment. * Fix spec test. * Update code so it copies the entire input. * Fix ast test. * Merge agent-improve-restart-loop * Merge agent-endpoint-spec * Refactor core/plugin/app into mostly core/ and use core/plugin for different app types. * Work on endpoint service application. * More fixes. * Fix format and tests. * Fix some imports. * More cleanups. * Fix export comment. * Pass the program.Spec into the descriptor. * Run endpoint verify, install, and uninstall when endpoint should be running. * Fix install and uninstall of Endpoint * Fix some small issues with service app. * Add changelog entry. * Fix lint and tests. * Fix lint. * Remove the code no longer needed because of newer config format. * Fix rules and review. * Update to Endpoint Security. * Fix issues so endpoint security runs. * Add comments. * Update docstring. * Some more fixes. * Delete the extra endpoint testdata files. * Add timeout to exec_file step. * Fix supported map. * Fix getting support programs by cmd. * Improve app started checks. * Fix buildspec. (cherry picked from commit 0fe1554)

…ne-beats * upstream/master: (105 commits) ci: enable packaging job (elastic#19536) ci: disable upstream trigger on PRs for the packaging job (elastic#19490) Implement memlog on-disk handling (elastic#19408) fix go.mod for PR elastic#19423 (elastic#19521) [MetricBeat] add param `aws_partition` to support aws-cn, aws-us-gov regions (elastic#19423) Input v2 stateless manager (elastic#19406) Input v2 compatibility layer (elastic#19401) [Elastic Agent] Fix artifact downloading to allow endpoint-security to be downloaded (elastic#19503) fix: ignore target changes on scans (elastic#19510) Add more helpers to pipeline/testing package (elastic#19405) Report dependencies in CSV format (elastic#19506) [Filebeat] Fix reference leak in TCP and Unix socket inputs (elastic#19459) Cursor input skeleton (elastic#19378) Add changelog. (elastic#19495) [DOC] Typo in Kerberos (elastic#19265) Remove accidentally commited unused NOTICE template (elastic#19485) [Elastic Agent] Support the install, control, and uninstall of Endpoint (elastic#19248) [Filebeat][httpjson] Add split_events_by config setting (elastic#19246) ci: disabling packaging job until we fix it (elastic#19481) Fix golang.org/x/tools to release1.13 (elastic#19478) ...

…nt (elastic#19248) * Initial spec parsing for endpoint. * Update comment. * Fix spec test. * Update code so it copies the entire input. * Fix ast test. * Merge agent-improve-restart-loop * Merge agent-endpoint-spec * Refactor core/plugin/app into mostly core/ and use core/plugin for different app types. * Work on endpoint service application. * More fixes. * Fix format and tests. * Fix some imports. * More cleanups. * Fix export comment. * Pass the program.Spec into the descriptor. * Run endpoint verify, install, and uninstall when endpoint should be running. * Fix install and uninstall of Endpoint * Fix some small issues with service app. * Add changelog entry. * Fix lint and tests. * Fix lint. * Remove the code no longer needed because of newer config format. * Fix rules and review. * Update to Endpoint Security. * Fix issues so endpoint security runs. * Add comments. * Update docstring. * Some more fixes. * Delete the extra endpoint testdata files. * Add timeout to exec_file step. * Fix supported map. * Fix getting support programs by cmd. * Improve app started checks. * Fix buildspec.

blakerouse added 20 commits June 12, 2020 09:13

Initial spec parsing for endpoint.

a64aeba

Update comment.

a79064b

Fix spec test.

f6a19fe

Update code so it copies the entire input.

0f93b15

Fix ast test.

ce54a3f

Merge agent-improve-restart-loop

f5da411

Merge agent-endpoint-spec

977f586

Refactor core/plugin/app into mostly core/ and use core/plugin for di…

eb630d7

…fferent app types.

Work on endpoint service application.

7de8c67

More fixes.

12ce078

Fix format and tests.

51408bf

Fix some imports.

4ee0754

Merge branch 'master' into agent-endpoint-app

157d0bd

More cleanups.

bb110fa

Fix export comment.

fd98ec4

Pass the program.Spec into the descriptor.

26cd0bf

Run endpoint verify, install, and uninstall when endpoint should be r…

e4a5aca

…unning.

Fix install and uninstall of Endpoint

f29dead

Fix some small issues with service app.

a497625

Merge branch 'agent-endpoint-app' into agent-endpoint-install

e51d529

blakerouse added the Team:Ingest Management label Jun 17, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 17, 2020

Add changelog entry.

73b4fa7

blakerouse marked this pull request as ready for review June 17, 2020 14:11

blakerouse added 3 commits June 17, 2020 11:14

Fix lint and tests.

0a3ce3d

Merge branch 'agent-endpoint-app' into agent-endpoint-install

515e664

Fix lint.

1d1c674

blakerouse added 7 commits June 18, 2020 09:55

Merge branch 'agent-endpoint-app' into agent-endpoint-install

2d60852

Update to Endpoint Security.

13dbbda

Fix issues so endpoint security runs.

1a3054d

Add comments.

6c6df5e

Update docstring.

1146b2d

Merge branch 'master' into agent-endpoint-install

c9d9b4e

Some more fixes.

1f32e78

michalpristas reviewed Jun 18, 2020

View reviewed changes

x-pack/elastic-agent/pkg/agent/transpiler/steps.go Show resolved Hide resolved

michalpristas reviewed Jun 18, 2020

View reviewed changes

x-pack/elastic-agent/pkg/artifact/install/hooks/hooks_installer.go Outdated Show resolved Hide resolved

blakerouse added 7 commits June 26, 2020 09:58

Delete the extra endpoint testdata files.

d97336c

Merge branch 'master' into agent-endpoint-install

6bf6cde

Add timeout to exec_file step.

c22b475

Fix supported map.

d383c5a

Fix getting support programs by cmd.

0283238

Improve app started checks.

81ef3d5

Fix buildspec.

6ccfcb1

michalpristas approved these changes Jun 29, 2020

View reviewed changes

blakerouse merged commit 0fe1554 into elastic:master Jun 29, 2020

This was referenced Jun 29, 2020

[Elastic Agent] Handle migration of agent_info to agent in Fleet configuration #19488

Closed

Elastic Agent install fails on Win 2012 Server as: filebeat[xyz] operation-install: exit status 1 #18792

Closed

blakerouse deleted the agent-endpoint-install branch June 29, 2020 17:58

blakerouse mentioned this pull request Jun 29, 2020

Cherry-pick #19248 to 7.x: [Elastic Agent] Support the install, control, and uninstall of Endpoint #19497

Merged

4 tasks

blakerouse added the v7.9.0 label Jun 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Elastic Agent] Support the install, control, and uninstall of Endpoint #19248

[Elastic Agent] Support the install, control, and uninstall of Endpoint #19248

blakerouse commented Jun 17, 2020 •

edited

Loading

elasticmachine commented Jun 17, 2020

elasticmachine commented Jun 17, 2020 •

edited

Loading

Build stats

Test stats 🧪

michalpristas Jun 18, 2020

blakerouse Jun 26, 2020

michalpristas commented Jun 29, 2020

michalpristas left a comment

michalpristas Jun 29, 2020

blakerouse Jun 29, 2020

ph Jun 29, 2020

blakerouse Jun 29, 2020

michalpristas Jun 29, 2020

blakerouse Jun 29, 2020

blakerouse commented Jun 29, 2020

[Elastic Agent] Support the install, control, and uninstall of Endpoint #19248

[Elastic Agent] Support the install, control, and uninstall of Endpoint #19248

Conversation

blakerouse commented Jun 17, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

elasticmachine commented Jun 17, 2020

elasticmachine commented Jun 17, 2020 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michalpristas commented Jun 29, 2020

michalpristas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blakerouse commented Jun 29, 2020

blakerouse commented Jun 17, 2020 •

edited

Loading

elasticmachine commented Jun 17, 2020 •

edited

Loading