Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fleet] Allow integration data to also be sent to Remote Elasticsearch #173237

Closed
nimarezainia opened this issue Dec 13, 2023 · 5 comments · Fixed by #173353
Closed

[Fleet] Allow integration data to also be sent to Remote Elasticsearch #173237

nimarezainia opened this issue Dec 13, 2023 · 5 comments · Fixed by #173353
Assignees
@juliaElastic
Labels
Team:Fleet Team label for Observability Data Collection Fleet team

Comments

@nimarezainia
Copy link
Contributor

Currently with the support of remote Elasticsearch added to an Agent output, it is not possible to configure it to send user integration data to the remote. this restriction needs to be removed. there are many use cases that we need to have enabled with this particular option.

image
@nimarezainia nimarezainia added the Team:Fleet Team label for Observability Data Collection Fleet team label Dec 13, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@juliaElastic juliaElastic self-assigned this Dec 13, 2023
@juliaElastic
Copy link
Contributor

juliaElastic commented Dec 13, 2023 •
edited
Loading

Docs should also be updated with this feature: https://www.elastic.co/guide/en/fleet/master/monitor-elastic-agent.html#external-elasticsearch-monitoring cc @kilfoyle

@kilfoyle kilfoyle mentioned this issue Dec 13, 2023
Closed
@kilfoyle
Copy link
Contributor

Thanks Julia! Docs issue is here.

@juliaElastic juliaElastic mentioned this issue Dec 14, 2023
Merged
1 task
@juliaElastic
Copy link
Contributor

Raised a pr in kibana to remove the integration data output restriction: #173353
I tested locally with system and nginx integration, and the data is ingested correctly in the remote es, so I think no further changes are needed.

@juliaElastic juliaElastic mentioned this issue Dec 14, 2023
Merged
juliaElastic added a commit that referenced this issue Dec 18, 2023
@juliaElastic
[Fleet] removed restriction to use remote es as integration data (#17…
4d35abe 
…3353)

## Summary

Closes #173237

Removed restriction to allow using remote es output as integration data
output.

### Steps to verify:

#### Send system integration data to remote es
- Create a remote es output, verify that the output is allowed to be set
as default for agent integrations
- Create an agent policy with system integration and set the remote es
output as integration data output
- Enroll an agent to the agent policy
- Check the remote kibana - Discover, verify that system metrics are
coming in from the agent
- Install system package on the remote cluster to see dashboards,
mappings, etc.
#### Send nginx integration data to remote es
- Add nginx integration to the agent policy
- Create a dummy nginx log file in `/var/tmp/nginx/access.log` and add
some dummy data to it
- Verify that the data from the nginx log file appears in the remote
kibana Discover in `logs-*` data view.
#### Back to default output
- Change the agent policy integration output back to default
- Verify that the system integration data is ingested in the main
cluster.
- Verify that the API key is invalidated in the remote cluster

<img width="1742" alt="image"
src="https://github.com/elastic/kibana/assets/90178898/47ed4e89-e761-4f24-90c3-bf3a49a6b4f1">

<img width="937" alt="image"
src="https://github.com/elastic/kibana/assets/90178898/2e41f37e-a4ef-4f18-aed0-d4160efe306a">

<img width="2162" alt="image"
src="https://github.com/elastic/kibana/assets/90178898/bf5d900b-f3bd-493e-b61a-4554224a97fc">

<img width="2150" alt="image"
src="https://github.com/elastic/kibana/assets/90178898/31e4f32e-8751-4b02-855f-dc7fbd5e64a9">

System dashboard on remote cluster populated:
<img width="2157" alt="image"
src="https://github.com/elastic/kibana/assets/90178898/dfb10791-ab15-4058-9170-7cad51935493">


### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
juliaElastic added a commit to juliaElastic/kibana that referenced this issue Dec 18, 2023
@juliaElastic
[Fleet] removed restriction to use remote es as integration data (ela…
f6d1b17 
…stic#173353)

Closes elastic#173237

Removed restriction to allow using remote es output as integration data
output.

- Create a remote es output, verify that the output is allowed to be set
as default for agent integrations
- Create an agent policy with system integration and set the remote es
output as integration data output
- Enroll an agent to the agent policy
- Check the remote kibana - Discover, verify that system metrics are
coming in from the agent
- Install system package on the remote cluster to see dashboards,
mappings, etc.
- Add nginx integration to the agent policy
- Create a dummy nginx log file in `/var/tmp/nginx/access.log` and add
some dummy data to it
- Verify that the data from the nginx log file appears in the remote
kibana Discover in `logs-*` data view.
- Change the agent policy integration output back to default
- Verify that the system integration data is ingested in the main
cluster.
- Verify that the API key is invalidated in the remote cluster

<img width="1742" alt="image"
src="https://github.com/elastic/kibana/assets/90178898/47ed4e89-e761-4f24-90c3-bf3a49a6b4f1">

<img width="937" alt="image"
src="https://github.com/elastic/kibana/assets/90178898/2e41f37e-a4ef-4f18-aed0-d4160efe306a">

<img width="2162" alt="image"
src="https://github.com/elastic/kibana/assets/90178898/bf5d900b-f3bd-493e-b61a-4554224a97fc">

<img width="2150" alt="image"
src="https://github.com/elastic/kibana/assets/90178898/31e4f32e-8751-4b02-855f-dc7fbd5e64a9">

System dashboard on remote cluster populated:
<img width="2157" alt="image"
src="https://github.com/elastic/kibana/assets/90178898/dfb10791-ab15-4058-9170-7cad51935493">

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
@juliaElastic
Copy link
Contributor

The elasticsearch pr is still open to add traces-* privileges to fleet-server-remote service account. It's not going to be needed until we add support to use APM with remote ES output (AFAIK it's the only package using traces).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@juliaElastic juliaElastic

Labels
Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Fleet] removed restriction to use remote es as integration data juliaElastic/kibana
4 participants
@nimarezainia @elasticmachine @kilfoyle @juliaElastic