[Ingest Manager] Upgrade Package APIs

[jonathan-buttner]

Background

For an external app to upgrade a package it is interested in, it has to perform the following actions:

1. GET /packages to list the available packages, then filter the response done to the one they're interested in
2. Check if the package of interest is marked as installed
3. If not installed, GET /packages/<package key> to get the full information about the package including the latestVersion field to determine if an update exists
4. If an update exists, install it

This could probably be simplified by adjusting the /packages/endpoint-0.1.0 to:

POST /packages/endpoint - Installs the latest available endpoint package
GET /packages/endpoint - Returns all information about the endpoint package (current version installed, and latest version available from the registry)
POST /packages/endpoint/0.2.0 - Installs endpoint package version 0.2.0
GET /packages/endpoint/0.2.0 - Gets the information about endpoint package version 0.2.0 (whether it is installed, and registry information)

Having a bulk upgrade api could also be useful for the Ingest manager to handle automatically upgrading the default packages:

POST /packages-bulk <-- thoughts?
{
  upgrade: ["endpoint", "system"]
}

This would upgrade both the endpoint and system packages.

POST /packages is already used for bulk install of uploaded packages:
https://github.com/elastic/kibana/blob/master/x-pack/plugins/ingest_manager/server/routes/epm/index.ts#L87
https://github.com/elastic/kibana/blob/master/x-pack/plugins/ingest_manager/common/constants/routes.ts#L25

Related issues I could find:

#76438
#62479
#66777 (comment)

Phase 1 - Implement Bulk API

Implementation

The easiest way forward with the least sweeping code changes is probably to implement the bulk API. It would have the following implementation:

For each package in the upgrade array from the request body

1. Get the saved object information for the package
2. Get the registry's package information
3. If the package is not installed, install the latest available
4. If the package is installed and the latest package from the registry's version is greater than the installed version, install the latest

Doing a greater than version check will avoid unnecessary installations of a package that already has the latest version installed. This is useful because the bulk upgrade API will be called each time a user navigates to a different package within the ingest manager UI and each time a user navigates to a different page in the Security Solution UI.

Handling Errors

If installing a package failed, the bulk upgrade API would record the error and move on to the next package to upgrade.

Response

The bulk API response would be:

{
  response: {
    package1: {
      version: "version number installed"
      assets: AssetReference[];
    },
    package2: {
      version: "version number installed"
      assets: AssetReference[];
    }
  }
}

Or maybe:

{
  response: [
    {
      name: package1
      version: "version number installed"
      assets: AssetReference[];
    },
    {
      name: package2
      version: "version number installed"
      assets: AssetReference[];
    }
  ]
}

Later Phases

Refactor the single package APIs to something like:

POST /packages/endpoint - Installs the latest available endpoint package
GET /packages/endpoint - Returns all information about the endpoint package (current version installed, and latest version available from the registry)
POST /packages/endpoint/0.2.0 - Installs endpoint package version 0.2.0
GET /packages/endpoint/0.2.0 - Gets the information about endpoint package version 0.2.0 (whether it is installed, and registry information)

This would allow the Security Solution and other external apps that are only interested in upgrading a single package to leverage POST /packages/<package name> instead of using the bulk upgrade API.

[elasticmachine]

Pinging @elastic/ingest-management (Feature:EPM)

[jonathan-buttner]

PR is here: #77827

[EricDavisX]

Hi, Jon - is there anything else to be done on this ticket or is it completed during the current cycle? thanks. @jonathan-buttner @rahulgupta-qasource

[jonathan-buttner]

Hi, Jon - is there anything else to be done on this ticket or is it completed during the current cycle? thanks. @jonathan-buttner @rahulgupta-qasource

Oh sorry, yeah should have closed this. It's been completed. Should I close it?

[jfsiii]

closed by #77827