Better support for installing plugins in ephemeral environments

[pugnascotia]

In environments where an Elasticsearch node is ephemeral, e.g. containerisation with Docker, it isn't always desirable or even possible to reach out to the internet to install plugins when the container starts. We would like to have some concept or support for "bundling" plugins, or otherwise making it easier to install plugins on what could be a short-lived ES node.

An additional complication is support for air-gapped environments, where an ES node cannot simply reach out to the internet to download a plugin.

[elasticmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[mark-vieira]

We discussed this at length with folks from @elastic/cloud-delivery. There are basically two general solutions here w/o getting too much into details:

1. Bundle the elasticsearch plugin artifacts into the Docker image itself and the performing an "install" simply leverages those artifacts already existent in the file system.
2. Introduce an "artifact server" component into Cloud which serves up these artifacts from a container deployed in the cluster.

The general sentiment of the Elasticsearch team is to avoid option (1). Plugins are packaged and distributed separately from Elasticsearch for a reason, there are many of them (20) and they support very specific and often times mutually exclusive use cases. There is quite simply never a need to use most of these plugins. Eagerly including them in our generic Docker image would increase it's size by 118MB and provide little or no benefit the vast majority of the time.

The preference would be some variation of (2) where Elasticsearch would produce an artifact server as another Docker image. ES installs in cloud would then be configured to install plugins from that local server. There are still some details to figure out here such as would such a thing be per Cloud environment or per cluster, how do we describe such an "internal" component in stack packs, does this align well with our plans for cloud-first testing, etc.

In any case, we need to make a decision here because obviously (2) is probably going to mean some additional effort from both ES and Cloud delivery teams.

cc @bleskes

[pugnascotia]

What changes are we imagining on the ES side? I ask because we already support installing plugins via a proxy. If ESS could make a suitably-configured proxy available and call elasticsearch-plugin install with the appropriate proxy params set, that would work, no?

[alpar-t]

@pugnascotia we would prefer a more declarative approach for cloud, where we can specify the plugins we want to have install rather imperatively adding and removing plugins. The other consideration is that we always install at least some plugins in cloud ( e.g. repository-<> ) and we are sensitive to how long it takes as it delays the time it takes to create a deployment. We discussed implementing this as a remove + add operation in the call that would work from a cloud perspective as long as it doesn't delay the time it takes to create the deployment. Calling elasticsearch-plugin twice would incur a potentially significant JVM startup overhead twice.

We looked at the potential of using the download service too and do on-line installation of plugins (air-gaped installations would be required to provide their own proxy/mirror, similar to how they provide their own repository). In the testing we did on a staging allocator, it took 3.986s to install the repostiroy-s3 plugin over the network, but a download of the plugin on the same allocator only took 0.048s, so the network transfer doesn't play a signification role in the plugin installation time. The most dominating factor seems to be the JVM startup time. I'm adding this as a data point, not as a proposal, we did not discuss if this is acceptable for air gaped installations.

[alpar-t]

I ran into this Today:

docker run -i -t --user 2120:4190  \
 --volume $PWD/es/config:/usr/share/elasticsearch/config \
 --volume $PWD/es/data:/usr/share/elasticsearch/data \
 --volume $PWD/es/logs:/usr/share/elasticsearch/logs \
 --volume $PWD/es/plugins:/usr/share/elasticsearch/plugins \
 docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT ./bin/elasticsearch-plugin install repository-s3
-> Installing repository-s3
-> Failed installing repository-s3
-> Rolling back repository-s3
-> Rolled back repository-s3
ERROR: attempted to install release build of official plugin on snapshot build of Elasticsearch

We need to take pre-releases into account too in the context of cloud first testing.

[mark-vieira]

Right, that won't work because you're attempting to install the latest released version of repository-s3 (7.11.1) on 8.0.0-SNAPSHOT.

I think if the intention is to install these plugins from some proxy/mirror/etc, then for several reasons like what Alpar ran into above, and cloud-first testing, that said proxy should be produced by Elasticsearch itself so that pre-release plugins can also be distributed/installed in ESS.

[rjernst]

you're attempting to install the latest released version of repository-s3 (7.11.1) on 8.0.0-SNAPSHOT.

I don't think that is the case, though the error message is poor here. The plugin installer knows that ES is a snapshot, and in that case tries to form an appropriate staging url to find the plugin at. Yet there is no staging hash passed in, so we bail. All that is required is passing in the staging hash.

However, in this case, it looks like this is not a staged released, but just a daily snapshot. We don't currently support that since snapshots have no unique identifier for one day vs another (each snapshot overwrites the previous day in the artifacts backend). If we had snapshots prefixed by hash (which was the plan long ago for the artifacts service, but I don't think ever got implemented), then simply passing in that hash would make this all work, no need for an additional proxy.

Having such a proxy, especially with the expectation that ES owns it, seems very heavyweight to me for something just for testing snapshots. It's not something we want users to ever do, hence why we guarded it with the hash so that there is no possibility of trying to install a newer snapshot of a plugin that is incompatible and can fail in mysterious ways.

[mark-vieira]

Ah, thanks for the clarification on that, Ryan.

[mark-vieira]

I feel like we've stalled a bit on this conversation. @alpar-t I think we need to escalate this within the cloud team to arrive at a consensus on which direction to take here.

[pugnascotia]

I think we can close this in favour of #70219.

[mgreau]

Hi

Just for your information, we do publish daily snapshots with unique identifiers. The Docker images are available through our Docker UI .

For example, by using the latest snapshot version and adding the environment variable ES_JAVA_OPTS="-Des.plugins.staging=3f9e9c2d", I can install the plugin like @alpar-t was tying to do:

docker run -i -t --user 2120:4190  \
 --volume $PWD/es/config:/usr/share/elasticsearch/config \
 --volume $PWD/es/data:/usr/share/elasticsearch/data \
 --volume $PWD/es/logs:/usr/share/elasticsearch/logs \
 --volume $PWD/es/plugins:/usr/share/elasticsearch/plugins \
 --env ES_JAVA_OPTS="-Des.plugins.staging=3f9e9c2d" \
 docker.elastic.co/elasticsearch/elasticsearch:8.0.0-3f9e9c2d-SNAPSHOT ./bin/elasticsearch-plugin install repository-s3
-> Installing repository-s3
-> Downloading repository-s3 from elastic
[=================================================] 100%??
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.SocketPermission * connect,resolve
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed repository-s3
-> Please restart Elasticsearch to activate any plugins installed

ls $PWD/es/plugins/repository-s3
LICENSE.txt                      commons-codec-1.11.jar           jackson-annotations-2.10.4.jar   log4j-1.2-api-2.11.1.jar
NOTICE.txt                       commons-logging-1.1.3.jar        jackson-databind-2.10.4.jar      plugin-descriptor.properties
aws-java-sdk-core-1.11.749.jar   httpclient-4.5.10.jar            jaxb-api-2.2.2.jar               plugin-security.policy
aws-java-sdk-s3-1.11.749.jar     httpcore-4.4.12.jar              jmespath-java-1.11.749.jar       repository-s3-8.0.0-SNAPSHOT.jar

I hope that helps.

[pugnascotia]

Actually, I was forgetting that this issue also covers installing snapshot plugins.

[Conky5]

We don't currently support that since snapshots have no unique identifier for one day vs another (each snapshot overwrites the previous day in the artifacts backend). If we had snapshots prefixed by hash (which was the plan long ago for the artifacts service, but I don't think ever got implemented), then simply passing in that hash would make this all work, no need for an additional proxy.

Unless I am misunderstanding the context here, I believe every snapshot does have a unique identifier as a prefix in the form of build-id: <bucket>/<build-id>/<artifacts>.

[rjernst]

@Conky5 @mgreau I'm glad to hear this! My knowledge is outdated. It looks like this may be possible now.

Looking at a recent snapshot as an example, the analysis-stempel plugin is available at:

https://snapshots.elastic.co/7.13.0-27b0dbc2/downloads/elasticsearch-plugins/analysis-stempel/analysis-stempel-7.13.0-SNAPSHOT.zip

So, I think we can update the plugin installer to work in this case. We will still need the hash, but it will work very similarly to the build candidate support, just with a different base.

[alpar-t]

Is this issue still needed to complement #70219 and have the proposed declarative way of installing plugins also honor build Ids ?

[pugnascotia]

Closing in favour of #70219.