[Fleet] Testing over Agent collection of Agent & Endpoint log files #22549

EricDavisX · 2020-11-11T17:25:33Z

This will be the test ticket to track remaining items for
[Elastic Agent] Allow the agent to collect and send the log of endpoint.

screenshot:

Questions / tasks / to-do:

Outline minimal AC for the feature

API user can consume this using the Logs & Metrics API ?UI Jen H confirmed the component re-use and that it is should covered by the Logs team
User can see stream of Agent logs in Fleet (see pr: [Fleet] Agent logs UI kibana#83356)
User sees the logs display updated by clicking refresh browser, or by pressing 'apply' in the date picker
User sees the display include Endpoint (if configured) and Agent, Metricbeat, & Filebeat interleaved together
User sees the display when non-default namespace is used for Agent & Endpoint
User can filter the display by which 'Dataset' exist, by standard 'Log Level' grouping, and by Time (default is last 24 hours)
User can do a KQL search of all data, as filtered
User can click a button to go to Logging UI
User can see the logs for all Agent / Endpoint supported OSes (will test macOS, Windows, Linux Tarball, Linux .rpm, Linux .deb, Agent Docker image)

Let's cite what Unit / API / e2e tests we have with what is merged
@jen-huang do we have any of the above covered in API / unit tests so far? Whichever parts we intend to cover but are not done yet can we log a separate issue for it please?

@blakerouse do we have any of the above covered in API & e2e / unit tests so far? Whichever parts we intend to cover but are not done yet can we log a separate issue for it please?

Based on the above answers we can assess what other automation we desire?

Stand-alone Agent, in Docker (Agent logs only) - basic test can be added in e2e
Fleet mode w/ different OSes - test for Debian & Centos can be added in e2e

Until more automation is confirmed, I assess the risk and test need for manual validation as per below:

Stand-alone Agent, in Docker (Agent logs only) (basic test can be added in e2e, with Logs API)
Fleet mode w/ different OSes, see above (basic test can be added in e2e)
No need for tests relating to reboot / persistence
No need for Browsers related testing, not a new component.
anything else?

@gogochan I saw you assessed the Endpoint side support, was there anything more we should coordinate on testing (beyond the above)?

elasticmachine · 2020-11-11T17:27:51Z

Pinging @elastic/ingest-management (Team:Ingest Management)

EricDavisX · 2020-11-11T17:28:52Z

@rahulgupta-qasource we have a new feature for 7.11 coming in. Please plan on making at least a simple test case for it, and we can mark it as automated if it ends up being covered fully there.

dikshachauhan-qasource · 2020-11-17T12:57:19Z

Hi @EricDavisX,

We have created 06 testcases for above feature on endpoint log files location for different OSs. Testcases links are as follows:
https://elastic.testrail.io/index.php?/cases/view/35075
https://elastic.testrail.io/index.php?/cases/view/35076
https://elastic.testrail.io/index.php?/cases/view/35077
https://elastic.testrail.io/index.php?/cases/view/35078
https://elastic.testrail.io/index.php?/cases/view/35079
https://elastic.testrail.io/index.php?/cases/view/35080

Query: Could you please confirm if we need to include scenarios for .deb and .rpm packages as well for logs generation at linux machines.

Also, we could not validate this due to issue reported on ticket #69006. As, we were unable to deploy 7.11.0-snapshot Kibana build through staging cloud platform.

Please let us know if we are missing anything and is required to be covered.

EricDavisX · 2020-11-17T16:13:14Z

I have posted the work-around to get 7.11 cloud working, but this can be tested on 8.0 as well (as most features can be as they are merged first to 8.0 / master line, by specific process).

I will review the manual test need with Jen later and we can report back and assign specific tests, and confirm more expectations / AC

EricDavisX · 2020-11-17T16:37:04Z

I'm tagging this pr here: elastic/kibana#83356

it is not testable in snapshot builds yet, fyi - its a draft PR and isn't merged. :)

EricDavisX · 2020-11-17T23:10:50Z

@blakerouse as part of the review for this feature, can you cite if you think the code paths or log paths are different enough that we need to test on different install types (macOS, Linux Tarball, Linux RPM, Windows)? If so, we can do it manually and can automate some in the e2e-testing repo, for us to work. Thanks for the input.

blakerouse · 2020-11-18T13:24:15Z

@EricDavisX Yes the log path is unique per OS. So it should be ensured that it works on all.

dikshachauhan-qasource · 2020-11-18T13:29:17Z

Hi @EricDavisX

Thanks for the update.
We have created 10 testcases for pr here: elastic/kibana#83356 covering Agent logs with a new UI. Testcases links are as follows:

Please let us know if we are missing anything and is required to be covered.

gogochan · 2020-11-18T16:28:46Z

Hi @EricDavisX, nothing much from my end.

jen-huang · 2020-11-18T18:32:02Z

API user can consume this (e2e-testing or es-archive Kibana FTR?) @jen-huang what Fleet API is used to support this?

We don't use a Fleet API for this. The logs component used for this is written by the Logs & Metrics UI team and uses its own API. I'm not sure what coverage they have for their component. We also populate query suggestions and filter options (dataset and log level) from data plugin's API (a core Kibana plugin). IMO it's out of scope for us to test other plugins' APIs.

@jen-huang do we have any of the above covered in API / unit tests so far? Whichever parts we intend to cover but are not done yet can we log a separate issue for it please?

There are no API or unit tests for reasons above. Similar to what we discussed yesterday, UI tests would be best to test this kind of feature to make sure all the pieces work together but we don't have that framework set up yet for Fleet..

EricDavisX · 2020-11-18T18:51:51Z

Thank you - I agree it is beyond our need to test the Logs api, this is the benefit of component re-use. We can review the FunctionalUI test framework separately, it may not be hard to set that up for Fleet team if we wanted to make use of what Kibana team has in place already. I'll update the above AC notes.

jen-huang · 2020-11-18T22:48:23Z

Hi again @EricDavisX , regarding this AC:

User sees the logs display updated in real time with no need to refresh (Refresh button is gone)

Due to current limitations with the logs component, updating the log entries in real time is a poor UX right now (a loading screen takes over, and user loses their scroll position). I added this to my PR description:

The log entries shown does not update automatically with new data, the user will need to change their queries/filters/date range to update the entries, or click Apply in the Quick select interface of the date picker

EricDavisX · 2020-11-18T23:23:25Z

@dikshachauhan-qasource @rahulgupta-qasource hi - I think we have pretty good AC and test requirements cited now, please do update the tests we've started based on the latest you see in the description AC. Hopefully it will be merged and available in an 8.0 snapshot a few days or so.

dikshachauhan-qasource · 2020-11-19T12:55:03Z

Hi @EricDavisX

Thanks for the update.

We have created 09 testcases for as per new AC update in #22549 for covering Agent logs with a new UI. Testcases links are as follows:

Testcases added under Integrate with endpoint
https://elastic.testrail.io/index.php?/cases/view/35099
https://elastic.testrail.io/index.php?/cases/view/35098

Testcases added under section Activity logs
https://elastic.testrail.io/index.php?/cases/view/35100
https://elastic.testrail.io/index.php?/cases/view/35101
https://elastic.testrail.io/index.php?/cases/view/35102
https://elastic.testrail.io/index.php?/cases/view/35103
https://elastic.testrail.io/index.php?/cases/view/35104
https://elastic.testrail.io/index.php?/cases/view/35105
https://elastic.testrail.io/index.php?/cases/view/35106

However, we have 03 queries and are as follows:

Query 1.
User sees the display when non-default namespace is used for Agent & Endpoint

Could you please confirm if we need to validate non-default namespace used for Agent & Endpoint in logs, being generated on agent Logs page.

Query 2.
User can see the logs for all Agent / Endpoint supported OSes (will test macOS, Windows, Linux Tarball, Linux .rpm, Linux .deb, Agent Docker image)

Could you please if logs are going to generate at '/opt/Elastic/Endpoint/state/log/endpoint-XXXXXX.log' location in Linux for .rpm and .deb too.

Query 3.
Stand-alone Agent, in Docker (Agent logs only) (basic test can be added in e2e, with Logs API)

Could you please confirm if we need to create testcase for this scenario as well with all OSs.

Please let us know if we are missing anything.

EricDavisX · 2020-11-19T15:19:57Z

Hi - my reply on testing queries:
Query 1.
Could you please confirm if we need to validate non-default namespace used for Agent & Endpoint in logs, being generated on agent Logs page.
A: I suggest we check the index the data is going into for the final test, if it appears in the UI we can validate it there as well, but I'm not sure if the logs themselves contain that detail.

Query 2.
Could you please if logs are going to generate at '/opt/Elastic/Endpoint/state/log/endpoint-XXXXXX.log' location in Linux for .rpm and .deb too.
A: Logs for Endpoint location for .rpm/.deb have not changed since 7.10 that I know of, so same location should have them.

Query 3.
Stand-alone Agent, in Docker (Agent logs only) (basic test can be added in e2e, with Logs API)
Eric will manage the Docker image (stand-alone agent) test, and hopes to automate it. If you wish to create one test case for it, that is fine, to remind us that we covered it.

dikshachauhan-qasource · 2020-11-20T13:02:34Z

Hi @EricDavisX

Thanks for the feedback.

We will update testcases for queries as follows:
For query 1 : Whenever the build is available, we will verifying the functionality on the the feature build and update testcases accordingly.
For query 2: We will updated testcases as per the Logs location for .rpm/.deb in 7.10 versions.
For query 3: we will skip testcase creation as it is a automation test.

EricDavisX · 2020-11-23T17:33:55Z

Hi - note the minor refinement in the query bar for this feature: elastic/kibana#84108

in case anyone notices. :)
Also - note the above feature test ticket to actually change the log-level in the UI, we should test those together

EricDavisX · 2020-11-23T21:50:43Z

The initial check-in is exciting to review - I am using 8.0 and have noticed / discussed a few items that are being reviewed.

logs are clipped, no horizontal scroll being put in, but line wrap is. per [Logs UI] wrap lines inside <LogStream /> kibana#83449
no refresh button in UI, but ^ pr has one, so maybe it is coming too.
refresh button in browser will lose saved state, Jen says she is doing a PR for that
the available log levels to pick from are dynamic, based on the datasets ingested. But we know we'll have a core-4 we want to prioritize and organize for the users (debug, info, warning, error) and we decided to show those first and then subsequent / other log levels will come after in the list. Its a compromise - and will be done as part of the other 'set log levels' ticket. kibana#84119

Still will be interesting to see what else will come with it.

dikshachauhan-qasource · 2020-11-24T13:39:56Z

Hi @EricDavisX

Thanks for the update.

We are performed testing on validating agent logs collection in kibana under Agent and Endpoint logs collection in Kibana Test Run on 7.11.0-SNAPSHOT Kibana cloud build. Build details are as follows:

Artifacts link: https://artifacts-api.elastic.co/v1/search/7.11.0-SNAPSHOT
BUILD 36717
COMMIT 968b6e5104013e02b55fefaa27793f878764361f

Also, we have updated 08 testcases and created a new test case: https://elastic.testrail.io/index.php?/cases/view/35132
as per specification shared in #22549 (comment) for point discussion

the available log levels to pick from are dynamic, based on the datasets ingested. But we know we'll have a core-4 we want to prioritize and organize for the users (debug, info, warning, error) and we decided to show those first and then subsequent / other log levels will come after in the list. Its a compromise - and will be done as part of the other 'set log levels' ticket. kibana#84119

Further, we will create/update testcases once new pr is available as mentioned in above comment.

However, we faced a issue while working upon rpm and deb agents so they remain untested for this feature. Issue is logged as: #22744

EricDavisX · 2020-11-24T15:24:37Z

@rahulgupta-qasource @dikshachauhan-qasource Can you transfer this to a new issue please? And while researching, I think we still have an RPM / Deb relating issue logged, you can make additional notes there.

EricDavisX · 2020-11-24T20:27:26Z

@dikshachauhan-qasource as you can see above, I logged the issue for us and have edited its citation in your post for clarity. We will hopefully have a fix to allow us to continue testing... but I doubt it will be for a week perhaps due to vacation and priority. For now, and until fixed, I would continue testing around all things .deb/.rpm in 7.11 / 8.0 as possible.

dikshachauhan-qasource · 2020-12-08T13:15:30Z

Hi @EricDavisX

We have created a new testcase as per the update available comment linked to ticket: #83449 summary. Testcase link is as follows: https://elastic.testrail.io/index.php?/cases/view/35182

Please let us know if anything else is required.

dikshachauhan-qasource · 2021-01-05T13:34:16Z

Hi @EricDavisX

We have executed 22 testcases under test plan Collection of Agent & Endpoint log files Test Run

Build details are as follows:

BUILD 37399
COMMIT f3abc08ac648f8b302733c5c22a39048314a027c
Artifact link: https://staging.elastic.co/7.11.0-710164a0/downloads/beats/elastic-agent/elastic-agent-7.11.0-windows-x86_64.zip

Please let us know if anything is required from our side.

Thanks
QAS

dikshachauhan-qasource · 2021-02-08T12:09:50Z

Hi @EricDavisX

We have performed exploratory testing today around this feature on 7.11 BC7 and found it working fine. Hence, closing this out.

Build details are as follows:

Build: 37897
Commit: 3f71ce7177a41e067ddb1e670ec4ace5f6d4f5fe
Artifact page: https://staging.elastic.co/7.11.0-1f38d552/summary-7.11.0.html

Thanks
QAS

EricDavisX assigned blakerouse and EricDavisX Nov 11, 2020

EricDavisX added the v7.11.0 label Nov 11, 2020

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Nov 11, 2020

EricDavisX mentioned this issue Nov 11, 2020

[Elastic Agent] Enable log shipping of endpoint-security by Elastic Agent #22526

Merged

4 tasks

EricDavisX added the Team:Ingest Management label Nov 11, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 11, 2020

EricDavisX changed the title ~~[Fleet] Testing over Agent collection of endpoint log files~~ [Fleet] Testing over Agent collection of Agent & Endpoint log files Nov 17, 2020

EricDavisX mentioned this issue Nov 23, 2020

[Fleet] Testing over Ability to Set Agent Log Level in Kibana Fleet UI elastic/kibana#84119

Closed

12 tasks

EricDavisX mentioned this issue Nov 23, 2020

[Fleet] Filtering Log level display order can be organized better in Agent Details Fleet UI elastic/kibana#84143

Closed

EricDavisX mentioned this issue Nov 24, 2020

[Elastic Agent] - RPM / DEB not working, Agent can't enroll to fleet - shows message: "Usage: /etc/init.d/elastic-agent {start|stop|status|restart|force-reload}" #22744

Closed

dikshachauhan-qasource mentioned this issue Jan 4, 2021

[Ingest Manager]: 'elastic-agent.endpoint-security' option not added to Dataset filter when endpoint-security integration is added to policy after agent-installation. elastic/kibana#87124

Closed

dikshachauhan-qasource closed this as completed Feb 8, 2021

dikshachauhan-qasource mentioned this issue May 8, 2023

Debug Logging level option is displayed under Logging level filter on Agent logs tab, when no entries for debug level is present in logs. elastic/kibana#156954

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Testing over Agent collection of Agent & Endpoint log files #22549

[Fleet] Testing over Agent collection of Agent & Endpoint log files #22549

EricDavisX commented Nov 11, 2020 •

edited

Loading

elasticmachine commented Nov 11, 2020

EricDavisX commented Nov 11, 2020

dikshachauhan-qasource commented Nov 17, 2020

EricDavisX commented Nov 17, 2020

EricDavisX commented Nov 17, 2020

EricDavisX commented Nov 17, 2020

blakerouse commented Nov 18, 2020

dikshachauhan-qasource commented Nov 18, 2020

gogochan commented Nov 18, 2020

jen-huang commented Nov 18, 2020

EricDavisX commented Nov 18, 2020

jen-huang commented Nov 18, 2020

EricDavisX commented Nov 18, 2020

dikshachauhan-qasource commented Nov 19, 2020

EricDavisX commented Nov 19, 2020

dikshachauhan-qasource commented Nov 20, 2020

EricDavisX commented Nov 23, 2020

EricDavisX commented Nov 23, 2020

dikshachauhan-qasource commented Nov 24, 2020 •

edited by EricDavisX

Loading

EricDavisX commented Nov 24, 2020

EricDavisX commented Nov 24, 2020

dikshachauhan-qasource commented Dec 8, 2020

dikshachauhan-qasource commented Jan 5, 2021 •

edited

Loading

dikshachauhan-qasource commented Feb 8, 2021

[Fleet] Testing over Agent collection of Agent & Endpoint log files #22549

[Fleet] Testing over Agent collection of Agent & Endpoint log files #22549

Comments

EricDavisX commented Nov 11, 2020 • edited Loading

elasticmachine commented Nov 11, 2020

EricDavisX commented Nov 11, 2020

dikshachauhan-qasource commented Nov 17, 2020

EricDavisX commented Nov 17, 2020

EricDavisX commented Nov 17, 2020

EricDavisX commented Nov 17, 2020

blakerouse commented Nov 18, 2020

dikshachauhan-qasource commented Nov 18, 2020

gogochan commented Nov 18, 2020

jen-huang commented Nov 18, 2020

EricDavisX commented Nov 18, 2020

jen-huang commented Nov 18, 2020

EricDavisX commented Nov 18, 2020

dikshachauhan-qasource commented Nov 19, 2020

EricDavisX commented Nov 19, 2020

dikshachauhan-qasource commented Nov 20, 2020

EricDavisX commented Nov 23, 2020

EricDavisX commented Nov 23, 2020

dikshachauhan-qasource commented Nov 24, 2020 • edited by EricDavisX Loading

EricDavisX commented Nov 24, 2020

EricDavisX commented Nov 24, 2020

dikshachauhan-qasource commented Dec 8, 2020

dikshachauhan-qasource commented Jan 5, 2021 • edited Loading

dikshachauhan-qasource commented Feb 8, 2021

EricDavisX commented Nov 11, 2020 •

edited

Loading

dikshachauhan-qasource commented Nov 24, 2020 •

edited by EricDavisX

Loading

dikshachauhan-qasource commented Jan 5, 2021 •

edited

Loading