[Fleet] Agent activity enhancements

[juliaElastic]

Continuation of #140267
Moved out stretch goals to this issue.

Figma designs

Requirements (helps supportability):

1. View agents button that navigates to a filtered list of agents included in the selected action @criamico

• Create a new API that returns the query that captures the agents that are included in the action

  • For this, take the actionId from the /action_status API, query .fleet-actions index, take the agent ids from the agents field of the matching documents. The query can be created by using the agent ids e.g. "id in (agent1, agent2, ...)"
  • Alternatively the original action's kuery could be saved in the action document to be used by the View agents feature.

• Implement the View agents link that navigates to the agents list and applies the query filter (ideally pass the query in the url, or in UI state).

  Design

  

2. Review error log button that navigates to Discover app to show relevant error logs @juliaElastic

• Create a link to Discover or Logs UI that searches on the actionId (or any other information that would capture the corresponding errors). Review the existing error logs to see if they have to be tweaked to help discoverability.
• Alternatively Review errors can be implemented by querying the .fleet-actions-results data stream and searching for the actionId to find those agents that failed the given action. For errors there will be an error field in the documents.

3. Include Agent policy update in Agent activity.

• Create action document for Agent policy updates to make visible in Agent activity
• Is there already an action document created for this?
• If not, the update action can be enhanced to have an action document with action results created, similarly to force unenroll
• Is there a simpler way to make policy update visible other than creating an action document?

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jlind23]

@jen-huang @kpollich pulling this in current sprint after discussing with @criamico

[juliaElastic]

As discussed with Cristina, I am picking up the improvement to Review error log.

Here is what I'm planning (extending a bit on the design):

• Review error log link: similar to Open in Logs link in Agent Details, with a relevant filter e.g. elastic_agent.id:(agent1 or agent2) and (data_stream.dataset:elastic_agent) - decided to move the buttons to specific agents errors, see below
• To find the relevant logs, the agentIds from the .fleet-action-results error records can be used. Note: there might be many agents with errors (up to 100k) so we have to limit the filter to avoid too long queries. To start with, showing the last 3 agent errors.
• It would be useful to also show the error message from .fleet-action-results, that can give a quick glance before diving into the logs (which might not be available). To achieve this, the /action_status API can be extended to return top error hits.
• We could even have a Review error log button for each agent error in activity, to focus the search on one agent, as it might get confusing to look at the logs from different agents at the same time.

@nimarezainia Hey, I would love to get some feedback from product side on the suggested changes, you can find a screenshot on how the error messages look like here.

Note about bulk update tags action:

• .fleet-actions docs for bulk update tags are not going to contain real agentIds, but generated UUIDs. This is because the action count had to be synchronized with the results counts. In case of ES update conflicts, we don't have information on the individual agents which conflicted, that's why a generated id is used. So View agents is not going to work for bulk update tags action, only if we do a different implementation e.g. filter agents by the tag that was added/removed in the action.

[criamico]

Also adding here some considerations about View agents:

I'm currently adding an endpoint that, given an array of actionIds, returns a list of agentIds. It would look like this:

POST kbn:/api/fleet/agents 
{
	actionIds: [
    	'action1',
        'action2'
    ]
}

This endpoint internally will query .fleet-actions, then get actions.agents to retrieve the list of agentIds associated with those actions. The agentIds will then be used to filter the agents list.

@juliaElastic this is basically the implementation that you described above. Do you think that there could be issues with performance?

[juliaElastic]

@criamico Performance is probably not an issue, there are at most 10 documents per action with max 10k agentIds each. We might hit a limit of query size though if we try to filter Agents by all the agentIds (we support actioning max 100k at a time). I think for bulk actions, it could make sense to store the actioned query with the .fleet-actions document (e.g. policy_id:policy1, and use that query to View agents.

We might want to split this feature into 2, and first focus on most common use cases (e.g. up to 20 agents), and implement the bulk View agents later with query.

[nimarezainia]

• .fleet-actions docs for bulk update tags are not going to contain real agentIds, but generated UUIDs. This is because the action count had to be synchronized with the results counts. In case of ES update conflicts, we don't have information on the individual agents which conflicted, that's why a generated id is used. So View agents is not going to work for bulk update tags action, only if we do a different implementation e.g. filter agents by the tag that was added/removed in the action.

@juliaElastic How would the user know which real agent this is referring to? Is host.name available here? i would agent.id is meaningless to the user (it also changes during the life cycle of the agent).

[nimarezainia]

@juliaElastic I made some comments but generally looks good to me. If possible we could perhaps walk through the completed/final look. For the most I believe the goal is to follow the option(b) in the Figma designs that were done sometime ago. Correct?

[juliaElastic]

How would the user know which real agent this is referring to? Is host.name available here? i would agent.id is meaningless to the user (it also changes during the life cycle of the agent).

This note only referred to update tags, where there is no way to correlate the error back to the agent. That shouldn't be a big problem though as there won't be errors on the agents side with update tags, the only likely error is a version conflict, which is auto retried.

For the other actions we have the real agent id (and can query the host name to display on the UI). The Review error logs feature is similar to the Figma designs, the difference I'm proposing is to go to error logs per agent, not try to show all agent logs at once for an action.

[nimarezainia]

For the other actions we have the real agent id (and can query the host name to display on the UI).

thanks for the explanation. I would suggest we use the hostname in the display rather than the agent-id. Hopefully the user has readable host names and easier to identify which host at a glance.

[criamico]

About the "view agents" work, I realised that once the agents are filtered there is currently no way of removing the filter. This is a simple example of how it looks with a single agent selected:

I couldn't find much in the figma designs as they are focused around the activity flyout.
I could add a "clear all filters" link on top of the table. We already have it when the there are no agents selected, but maybe we should display it also in a case like this.

However, there is also no visual indication of the applied filter, which is a bit confusing for the user.

@nimarezainia @juliaElastic what do you think?

[nimarezainia]

@criamico yes I think a clear all filters would be useful. Avoids the user from going into each one which can be tedious.

[juliaElastic]

@criamico Would it be possible to show the applied filter in the filter box? It would be clearer to confirm what the user sees. A Clear all filters link could be also useful.

[criamico]

@juliaElastic Do you mean here?

I could add a visual indication of the filter action (something like "Unenrolled", "Upgraded"...) but where should I add it? i don't think we already have a filter like this one, but correct me if I'm wrong.

[juliaElastic]

@criamico I mean the Filter text field like below. Though it might get long if we add a long filter with many agent ids.

[criamico]

@juliaElastic oh I see. I wouldn't add the list of all the ids, as there could be even hundreds of them. I would rather display the action, but at that point it wouldn't be the correct query to show in the filter box, so I don't know if that's the right place.

@nimarezainia what do you think?

[nimarezainia]

@juliaElastic oh I see. I wouldn't add the list of all the ids, as there could be even hundreds of them. I would rather display the action, but at that point it wouldn't be the correct query to show in the filter box, so I don't know if that's the right place.

@nimarezainia what do you think?

yes I agree that would not be readable at all once we have more than one agent. Why do we need to add this in the filter box? perhaps i'm misunderstanding the workflow (sorry)

[criamico]

@nimarezainia I was suggesting to add some kind of indication of what sublist of agents we are showing in the table. After the user clicks on view agents from the flyout, it gets to a filtered list of agents but we don't show what these agents are.

However, I think that I can finish off my PR without this part and if it proves necessary we can add it as an enhancement later.

[nimarezainia]

i think "view agents" implies that it's a subset already. if we have the "clear all" option that was suggested the user can get back to the full list.

[jlind23]

@criamico @juliaElastic is the current description accurate with the unchecked boxes?

[juliaElastic]

@jlind23 updated, the first 2 enhancements are done

[jlind23]

Thanks @juliaElastic - Is there anyone about to work on point 4 to 7? Or should we keep these for later and consider this issue as done?

[criamico]

@jlind23 At the moment I took a bugfix ticket as it was requested to do it this week, I might come back to this one after that. We can also move the rest of the enhancements to a new ticket if you prefer.

[juliaElastic]

I have some ideas on 3. Include Agent policy update in Agent activity., I can pick it up this week.
Came across in performance tests that we don't really have an easy way to track agent policy updates, so it would be useful to add that to Agent activity.

So currently there is no corresponding agent doc for policy updates, Fleet Server polls the policies table and detects new versions. The agents are then updated with the new policy revision when they received the new policy.

In Agent Activity we can do something similar to show the Policy update actions: query the policy revisions, and check how many agents use the new policy. This works well with in progress or recent policy updates, but can't provide all the info historically.
For example, if the agents are reassigned to a new policy, we don't have a way to know what was the previous policy's revision in the agent.
Another example, if the agent is unenrolled, it won't apply any more policy changes.
For these cases we could display "unknown" status on the UI.
Alternatively we could hide the status and agent count completely and only state that there was a policy update.
cc @nimarezainia

I think it is a good start though, rather than introducing the overhead of storing action document (especially as other actions like upgrade is planned to be implemented more as a state convergence, rather than an action https://github.com/elastic/ingest-dev/issues/1621)

Latest screenshot: instead of unknown action, just showing the fact that the policy was change (with the policy name and revision number). When we don't find any agents assigned to the policy, we can't provide more information, so hiding the agent count.

[jlind23]

@criamico Cleaned up the description and left only #3 in as Julia might be working on it fairly soon.
Created this as a follow up issue.

[juliaElastic]

Recorded a demo of improvements of policy change and displaying errors: https://www.loom.com/share/25d0a9427f50405186ee7f35ee195aff

[criamico]

I also recorded a (very short) demo of the "view agents" functionality:
https://www.loom.com/share/8b0b6c216ffd47d0bfd4de30698c5567

[amolnater-qasource]

Hi Team,

We have created 05 testcases for this feature under Fleet test suite at links:

• Validate view agents button under Agent activity, filters the agents on which action is performed.
• Validate if agent upgrade fails Show errors is shown for that action under Agent Activity.
• Validate on clicking Show Errors detailed errors and Review error logs button is shown for each agent.
• Validate on clicking Review error logs button user is navigated to Observability>Logs>Stream with agent and log level error selected.
• Validate View agents button is available under Agent activity when agents are scheduled to upgrade and on clicking the agents are visible.

Please let us know if we are missing any scenario to be covered here.

Thanks!

[amolnater-qasource]

Hi Team,
We have executed 05 testcases under the Feature test run for the 8.8.0 release at the link:

• Agent activity enhancements

Status:

• PASS: 05

Build details:
VERSION: 8.8 BC3
BUILD: 62994
COMMIT: 85b22d3

As the testing is completed on this feature, we are marking this as QA:Validated.

Please let us know if anything else is required from our end.
Thanks