Cherry-pick #23865 to 7.x: [Elastic Agent] Enroll with Fleet Server

[blakerouse]

Cherry-pick of PR #23865 to 7.x branch. Original message:

What does this PR do?

This adds the ability to enroll the Elastic Agent with Fleet Server executed locally on the same machine. To get this work a few things needed to be added to Elastic Agent.

• Wire in the status.Controller to the socket control protocol.
• Add new bootstrap operating mode that just runs a Fleet Server.
• Add a new bootstrap of Fleet Server before the start of the Fleet Gateway (in managed mode w/ Fleet Server)

Note: This has a breaking change in the parameters to enroll. kibana_url and enrollment_token move to being parameters instead of positional arguments. This makes install and enroll take the same parameters, and closes #21897.

Why is it important?

So that Fleet Server can be bootstrapped on a machine with Fleet Server also running on that same machine.

How does it work?

The enroll command handles the coordination of controlling the running Elastic Agent daemon. The install command proxies to the enroll command so this can be ran from the install or from the DEB/RPM.

Breakdown of the steps that are completed to handle the bootstrap:

1. Enroll must be executed with --fleet-server parameter. This parameter is a connection string for Fleet Server to communicate to elasticsearch. (Example: --fleet-server http://elastic:changeme@localhost:9200)
2. Enroll ensures that it can communicate with a running Elastic Agent. (This requires that Elastic Agent to be running).
3. Enroll writes the fleet.yml with fleet.server configuration, with fleet.server.bootstrap: true.
4. Enroll triggers the Elastic Agent to restart (causing re-execution)
5. Elastic Agent is re-executed into the Fleet Server bootstrap mode.
6. Elastic Agent starts the Fleet Server passing it the configuration.
7. Enroll polls the status GRPC of the Elastic Agent until Fleet Server is started and is in degraded state (should be degraded, because the Elastic Agent is not enrolled yet).
8. Enroll performs the enrollment against newly running Fleet Server.
9. Enroll writes a new fleet.yml with enrollment information and the fleet.server information. The fleet.server.bootstrap is removed (aka. False).
10. Enroll triggers the Elastic Agent to restart (causing re-execution)
11. Elastic Agent is re-executed into the Fleet mode.
12. Elastic Agent starts up the Fleet Server before starting the Fleet Gateway communication (because fleet.server is set in the fleet.yml).
13. Elastic Agent then starts the Fleet Gateway communication
14. Elastic Agent is now executing and being controlled through the locally running Fleet Server.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• Normall enrollment without Fleet Server works as expected
• Enrollment with --fleet-server works.

How to test this PR locally

Run the latest 8.0.0-SNAPSHOT of elasticsearch and Kibana. Start Kibana with the xpack.fleet.agents.fleetServerEnabled: true.

Add the Fleet Server integration to a policy.

Look up the policy ID (as this is currently needed until a default policy for Fleet Server is added to Kibana).

Start Elastic Agent.

Run the following command to bootstrap and enroll the Elastic Agent.

./elastic-agent enroll --insecure --url http://localhost:8000 --enrollment-token {token} --fleet-server http://elastic:changeme@localhost:9200 --fleet-server-policy {policy_id}

Related issues

• Closes [Elastic Agent] Inconsistency for enroll and install command #21897

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #24064 opened

• Start Time: 2021-02-16T15:05:00.161+0000

• Duration: 59 min 0 sec

• Commit: 37d2fa6

Test stats 🧪

Test	Results
Failed	0
Passed	6486
Skipped	24
Total	6510

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	6486
Skipped	24
Total	6510

[michalpristas]

Backport looks good