[Fleet] Support Fleet server system indices #89372

nchaulet · 2021-01-26T22:07:27Z

Summary

Related to #87372
Follow #86179

To support Fleet Server we need to use .fleet* system indices instead of saved object for agents enrollment keys and agents, this PR add this.

What to expect/test

with the feature flag `xpack.fleet.agents.fleetServerEnabled: false`

Kibana should use the SO and all the features should work as expected, you should be able to enroll an agent through Kibana, endpoint should work.

with the feature flag `xpack.fleet.agents.fleetServerEnabled: true` without Fleet Server

Kibana should use the .fleet indices and all the features should work as expected, you should be able to enroll an agent through Kibana, endpoint should work.

with the feature flag `xpack.fleet.agents.fleetServerEnabled: true` and Fleet Server

Kibana should use the .fleet indices and all the features should work as expected,

you should be able to enroll an agent through Kibana, endpoint should work, if you change the kibana url in the fleet settings to fleet server the agent should continue to work but talk to Fleet server instead of kibana
You should be able to enroll an agent directly throught Fleet server (need to change the kibana url to Fleet server http://localhost:8000 by default)

What is not yet working

The suggest bar in the UI is still using the saved object for autocompletion.
with Fleet server the endpoint download manifest is not working

…roll

…-server-agent-routes

elasticmachine · 2021-01-28T18:23:14Z

Pinging @elastic/ingest-management (Team:Ingest Management)

…-server-agent-routes

jen-huang

@nchaulet Did you get a chance to review my notes in this comment? The ES errors still occur: #89372 (comment) If this is just a known limitation right now that Fleet server needs to be started after Fleet setup is done, then that's fine, but let's make sure we confirm that.

I'm now seeing this Kibana error being logged when Fleet server is enabled, this is preventing the agent list from appearing:

server    log   [17:35:09.610] [error][fleet][plugins] TypeError: kuery.replace is not a function
    at removeSOAttributes (/Users/jen/Projects/kibana/x-pack/plugins/fleet/server/services/agents/crud_fleet_server.ts:35:16)
    at Object.listAgents (/Users/jen/Projects/kibana/x-pack/plugins/fleet/server/services/agents/crud_fleet_server.ts:61:18)
    at listAgents (/Users/jen/Projects/kibana/x-pack/plugins/fleet/server/services/agents/crud.ts:29:30)
    at concurrency (/Users/jen/Projects/kibana/x-pack/plugins/fleet/server/services/agents/status.ts:67:7)
    at /Users/jen/Projects/kibana/node_modules/p-map/index.js:57:28
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
server   error  [17:35:09.583]  Error: Internal Server Error
    at HapiResponseAdapter.toError (/Users/jen/Projects/kibana/src/core/server/http/router/response_adapter.ts:122:19)
    at HapiResponseAdapter.toHapiResponse (/Users/jen/Projects/kibana/src/core/server/http/router/response_adapter.ts:72:19)
    at HapiResponseAdapter.handle (/Users/jen/Projects/kibana/src/core/server/http/router/response_adapter.ts:67:17)
    at Router.handle (/Users/jen/Projects/kibana/src/core/server/http/router/router.ts:273:34)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at handler (/Users/jen/Projects/kibana/src/core/server/http/router/router.ts:227:11)
    at exports.Manager.execute (/Users/jen/Projects/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/Users/jen/Projects/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/Users/jen/Projects/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/Users/jen/Projects/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)
    at Request._execute (/Users/jen/Projects/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)

I did some testing on when Fleet server is not enabled, and things look good. I know we don't expect Fleet server integration to be perfect yet, but the above error is blocking for me for this PR. I'm happy to give a 👍🏻 once that's resolved and I get another round of successful non-Fleet server testing.

jen-huang · 2021-02-05T00:22:56Z

x-pack/plugins/fleet/server/services/agents/actions.ts

@@ -78,6 +80,7 @@ async function createAction(
  ) {
    const body: FleetServerAgentAction = {
      '@timestamp': new Date().toISOString(),
+      expiration: new Date(Date.now() + ONE_MONTH_IN_MS).toISOString(),


wow that's a long expiration! 😄 what does tell fleet server to do?

It's mandatory for Fleet server actions to have an expiration, it just say if the agent connect after more than 1 month to fleet server the agent will not get the action.

…-server-agent-routes

nchaulet · 2021-02-05T15:09:00Z

@jen-huang thanks for testing this again and again.

I missed the kuery.replace error when I merged master, it should be fixed.

For the error in the logs, the way Fleet server check for the indice to be present, is by trying to search this indices so the error in the logs is "okay" and should not affect how Fleet Server works we could probably improve that by checking if the index is present and ready.

nchaulet · 2021-02-05T17:19:20Z

@kevinlog I would love a review from endpoint on that one, I tested with the flagged turn off and nothing looks broken on your end.

…-server-agent-routes

jen-huang

Tested locally again today and am not seeing any issues when feature flag is disabled. I tested with feature flag enabled too (which also did a light test of the SO->indices migrations) and found that the kuery error is resolved and I didn't see any major issues apart from known ones while we continue to work on supporting Fleet server.

Thanks for iterating on this with my testing @nchaulet and great work!!

kevinlog · 2021-02-05T21:18:57Z

@nchaulet my testing steps:

Add Agent with Endpoint
Update policy
Remove Policy, see Endpoint gets uninstalled

The above worked with the feature flag both ON and OFF.
Artifacts not working with Fleet server is a known issue - I just created this to keep track of it. We'll fix it in 7.13.
#90513

Other than that, looks good!

kevinlog · 2021-02-05T21:21:16Z

x-pack/plugins/security_solution/server/endpoint/routes/artifacts/download_exception_list.ts

@@ -54,7 +54,11 @@ export function registerDownloadExceptionListRoute(
      // The ApiKey must be associated with an enrolled Fleet agent
      try {
        scopedSOClient = endpointContext.service.getScopedSavedObjectsClient(req);
-        await authenticateAgentWithAccessToken(scopedSOClient, req);
+        await authenticateAgentWithAccessToken(


@paul-tavares FYI

with the feature flag OFF artifacts were still being downloaded

patrykkopycinski · 2021-02-06T11:33:42Z

Hi @nchaulet @jen-huang

I've found one more issue I think, I've had a case where my agents were assigned to the policy that was removed and when I tried to reassign agents to the new policy I was getting an error that saved object with old policy id doesn't exist
https://github.com/elastic/kibana/pull/89372/files#diff-11bcc5bb1eaf5ddae753f9ee690233bef988d13ddbf6baf71c4c8edd0d6a4933L186-L189

I've had to wrap this line into try/catch block to make sure that saved object error is not being thrown

let agentPolicySO;

try {
  agentPolicySO = await soClient.get<AgentPolicySOAttributes>(SAVED_OBJECT_TYPE, id);
  // eslint-disable-next-line no-empty
} catch (e) {}

if (!agentPolicySO) {
  return null;
}

nchaulet · 2021-02-07T20:10:34Z

@elasticmachine merge upstream

nchaulet · 2021-02-07T20:14:20Z

I've found one more issue I think, I've had a case where my agents were assigned to the policy that was removed and when I tried to reassign agents to the new policy I was getting an error that saved object with old policy id doesn't exist

@patrykkopycinski I think the bug existed before that PR, I am wondering how did you remove the policy? normally we cannot delete a policy with active agents, maybe there is a bug here.

I will merge that PR as it is to avoid more conflict, but we should still iterate on that

kibanamachine · 2021-02-07T22:17:32Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 34af5f7

Metrics [docs]

✅ unchanged

History

💚 Build #104360 succeeded 915cea6
💛 Build #104285 was flaky 288c861
💔 Build #104258 failed 8dd23bf
💚 Build #103931 succeeded 9fa440b
💚 Build #103668 succeeded 8e03379

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* master: (55 commits) [APM-UI][E2E] use githubNotify step (elastic#90514) [APM] Export ProcessorEvent type (elastic#90540) [Lens] Retain column config (elastic#90048) [Data Table] Add unit tests (elastic#90173) Migrate most plugins to synchronous lifecycle (elastic#89562) skip flaky suite (elastic#90555) skip flaky suite (elastic#64473) [actions] improve email action doc (elastic#90020) [Fleet] Support Fleet server system indices (elastic#89372) skip flaky suite (elastic#90552) Bump immer dependencies (elastic#90267) Unrevert "Migrations v2: don't auto-create indices + FTR/esArchiver support (elastic#85778)" (elastic#89992) [Search Sessions] Use sync config (elastic#90138) chore(NA): add safe guard to remove bazelisk from yarn global at bootstrap (elastic#90538) [test] Await retry.waitFor (elastic#90456) chore(NA): integrate build buddy with our bazel setup and remote cache for ci (elastic#90116) Skip failing suite (elastic#90526) [Fleet] Fix incorrect conversion of string to numeric values in agent YAML (elastic#90371) [Docs] Update reporting troubleshooting for arm rhel/centos (elastic#90385) chore(NA): build bazel projects all at once in the distributable build process (elastic#90328) ...

jen-huang · 2021-02-08T15:25:13Z

Thanks for reporting @patrykkopycinski, agree with @nchaulet that this is existing behavior. I've opened an issue for that #90625. To be honest I am surprised at the behavior, I thought we had some conditions to not be able to delete a policy if active agents are enrolled into it, but maybe not 🤔

[Fleet] Install the Fleet Server package during setup

007690f

nchaulet self-assigned this Jan 26, 2021

nchaulet mentioned this pull request Jan 27, 2021

Acking a POLICY_CHANGE action do not update agent.policy_revision elastic/fleet-server#72

Closed

nchaulet force-pushed the feature-fleet-server-agent-routes branch 2 times, most recently from f41c0bc to 599fe99 Compare January 27, 2021 18:48

nchaulet mentioned this pull request Jan 27, 2021

[Fleet] Migrate Agents from saved object to fleet server indices #87372

Closed

16 tasks

[Fleet] Support fleet server indices in agent routes ack, checkin, en…

9e777e5

…roll

nchaulet force-pushed the feature-fleet-server-agent-routes branch from 599fe99 to 9e777e5 Compare January 27, 2021 20:36

nchaulet added 2 commits January 27, 2021 16:19

Merge branch 'master' of github.com:elastic/kibana into feature-fleet…

11c7d5f

…-server-agent-routes

Fix tests

71b1942

nchaulet force-pushed the feature-fleet-server-agent-routes branch from 331435f to 71b1942 Compare January 28, 2021 01:18

nchaulet added 3 commits January 28, 2021 08:49

Fix tests

c59780a

Merge branch 'master' of github.com:elastic/kibana into feature-fleet…

59f9feb

…-server-agent-routes

Merge branch 'master' of github.com:elastic/kibana into feature-fleet…

9cf6cdd

…-server-agent-routes

nchaulet marked this pull request as ready for review January 28, 2021 18:22

nchaulet requested a review from a team as a code owner January 28, 2021 18:22

nchaulet requested a review from a team January 28, 2021 18:22

nchaulet requested a review from a team as a code owner January 28, 2021 18:22

nchaulet added release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.12.0 v8.0.0 labels Jan 28, 2021

nchaulet mentioned this pull request Jan 28, 2021

[Fleet] Install the Fleet Server package during setup #89224

Merged

Merge branch 'master' of github.com:elastic/kibana into feature-fleet…

8544acc

…-server-agent-routes

botelastic bot added the Team:Ingest Management label Jan 29, 2021

jen-huang removed the Team:Ingest Management label Jan 29, 2021

Fix merge conflic

ef9c9b6

botelastic bot added the Team:Ingest Management label Jan 29, 2021

ph removed the Team:Ingest Management label Jan 29, 2021

Merge branch 'master' of github.com:elastic/kibana into feature-fleet…

9fa440b

…-server-agent-routes

jen-huang self-requested a review February 5, 2021 00:20

jen-huang reviewed Feb 5, 2021

View reviewed changes

nchaulet added 2 commits February 5, 2021 09:38

Fix kuery issues after mergin master

3c839bb

Merge branch 'master' of github.com:elastic/kibana into feature-fleet…

8dd23bf

…-server-agent-routes

Fix tests after merge

288c861

Merge branch 'master' of github.com:elastic/kibana into feature-fleet…

915cea6

…-server-agent-routes

jen-huang self-requested a review February 5, 2021 20:22

jen-huang approved these changes Feb 5, 2021

View reviewed changes

kevinlog reviewed Feb 5, 2021

View reviewed changes

nchaulet requested a review from paul-tavares February 5, 2021 21:27

kevinlog approved these changes Feb 5, 2021

View reviewed changes

Merge branch 'master' into feature-fleet-server-agent-routes

34af5f7

nchaulet merged commit 91ffe73 into elastic:master Feb 7, 2021

nchaulet deleted the feature-fleet-server-agent-routes branch February 7, 2021 23:57

jen-huang mentioned this pull request Feb 8, 2021

[Fleet] Unable to reassign agents to another policy when current policy is deleted #90625

Closed

nchaulet mentioned this pull request Feb 8, 2021

[7.x] [Fleet] Support Fleet server system indices (#89372) #90640

Merged

nchaulet added a commit to nchaulet/kibana that referenced this pull request Feb 8, 2021

[Fleet] Support Fleet server system indices (elastic#89372)

59dce3e

nchaulet added a commit that referenced this pull request Feb 8, 2021

[Fleet] Support Fleet server system indices (#89372) (#90640)

89920ee

nchaulet mentioned this pull request Feb 15, 2021

[Fleet] Fleet server migrations SavedObjectsClient#delete race-condition #89252

Closed

spalger mentioned this pull request Apr 7, 2021

Failing ES Promotion: Fleet Endpoints EPM Endpoints EPM - list "after all" hook in "EPM - list #96515

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Support Fleet server system indices #89372

[Fleet] Support Fleet server system indices #89372

nchaulet commented Jan 26, 2021 •

edited

Loading

elasticmachine commented Jan 28, 2021

jen-huang left a comment

jen-huang Feb 5, 2021

nchaulet Feb 5, 2021

nchaulet commented Feb 5, 2021

nchaulet commented Feb 5, 2021

jen-huang left a comment

kevinlog commented Feb 5, 2021

kevinlog Feb 5, 2021

patrykkopycinski commented Feb 6, 2021 •

edited

Loading

nchaulet commented Feb 7, 2021

nchaulet commented Feb 7, 2021

kibanamachine commented Feb 7, 2021

jen-huang commented Feb 8, 2021

[Fleet] Support Fleet server system indices #89372

[Fleet] Support Fleet server system indices #89372

Conversation

nchaulet commented Jan 26, 2021 • edited Loading

Summary

What to expect/test

with the feature flag xpack.fleet.agents.fleetServerEnabled: false

with the feature flag xpack.fleet.agents.fleetServerEnabled: true without Fleet Server

with the feature flag xpack.fleet.agents.fleetServerEnabled: true and Fleet Server

What is not yet working

elasticmachine commented Jan 28, 2021

jen-huang left a comment

Choose a reason for hiding this comment

jen-huang Feb 5, 2021

Choose a reason for hiding this comment

nchaulet Feb 5, 2021

Choose a reason for hiding this comment

nchaulet commented Feb 5, 2021

nchaulet commented Feb 5, 2021

jen-huang left a comment

Choose a reason for hiding this comment

kevinlog commented Feb 5, 2021

kevinlog Feb 5, 2021

Choose a reason for hiding this comment

patrykkopycinski commented Feb 6, 2021 • edited Loading

nchaulet commented Feb 7, 2021

nchaulet commented Feb 7, 2021

kibanamachine commented Feb 7, 2021

💚 Build Succeeded

Metrics [docs]

History

jen-huang commented Feb 8, 2021

nchaulet commented Jan 26, 2021 •

edited

Loading

with the feature flag `xpack.fleet.agents.fleetServerEnabled: false`

with the feature flag `xpack.fleet.agents.fleetServerEnabled: true` without Fleet Server

with the feature flag `xpack.fleet.agents.fleetServerEnabled: true` and Fleet Server

patrykkopycinski commented Feb 6, 2021 •

edited

Loading