Merge branch 'master' into ccr

* master:
  Mute ML upgrade test (#30458)
  Stop forking javac (#30462)
  Client: Deprecate many argument performRequest (#30315)
  Docs: Use task_id in examples of tasks (#30436)
  Security: Rename IndexLifecycleManager to SecurityIndexManager (#30442)
  [Docs] Fix typo in cardinality-aggregation.asciidoc (#30434)
  Avoid NPE in `more_like_this` when field has zero tokens (#30365)
  Build: Switch to building javadoc with html5 (#30440)
  Add a quick tour of the project to CONTRIBUTING (#30187)
  Reindex: Use request flavored methods (#30317)
  Silence SplitIndexIT.testSplitIndexPrimaryTerm test failure. (#30432)
  Auto-expand replicas when adding or removing nodes (#30423)
  Docs: fix changelog merge
  Fix line length violation in cache tests
  Add stricter geohash parsing (#30376)
  Add failing test for core cache deadlock
  [DOCS] convert forcemerge snippet
  Update forcemerge.asciidoc (#30113)
  Added zentity to the list of API extension plugins (#29143)
  Fix the search request default operation behavior doc (#29302) (#29405)

CONTRIBUTING.md

@@ -41,7 +41,7 @@ We enjoy working with contributors to get their code accepted. There are many ap
 Note that it is unlikely the project will merge refactors for the sake of refactoring. These
 types of pull requests have a high cost to maintainers in reviewing and testing with little
 to no tangible benefit. This especially includes changes generated by tools. For example,
-converting all generic interface instances to use the diamond operator. 
+converting all generic interface instances to use the diamond operator.
 
 The process for contributing to any of the [Elastic repositories](https://github.com/elastic/) is similar. Details for individual projects can be found below.
 
@@ -209,6 +209,95 @@ Before submitting your changes, run the test suite to make sure that nothing is
 ./gradlew check
 ```
 
+### Project layout
+
+This repository is split into many top level directories. The most important
+ones are:
+
+#### `docs`
+Documentation for the project.
+
+#### `distribution`
+Builds our tar and zip archives and our rpm and deb packages.
+
+#### `libs`
+Libraries used to build other parts of the project. These are meant to be
+internal rather than general purpose. We have no plans to
+[semver](https://semver.org/) their APIs or accept feature requests for them.
+We publish them to maven central because they are dependencies of our plugin
+test framework, high level rest client, and jdbc driver but they really aren't
+general purpose enough to *belong* in maven central. We're still working out
+what to do here.
+
+#### `modules`
+Features that are shipped with Elasticsearch by default but are not built in to
+the server. We typically separate features from the server because they require
+permissions that we don't believe *all* of Elasticsearch should have or because
+they depend on libraries that we don't believe *all* of Elasticsearch should
+depend on.
+
+For example, reindex requires the `connect` permission so it can perform
+reindex-from-remote but we don't believe that the *all* of Elasticsearch should
+have the "connect". For another example, Painless is implemented using antlr4
+and asm and we don't believe that *all* of Elasticsearch should have access to
+them.
+
+#### `plugins`
+Officially supported plugins to Elasticsearch. We decide that a feature should
+be a plugin rather than shipped as a module because we feel that it is only
+important to a subset of users, especially if it requires extra dependencies.
+
+The canonical example of this is the ICU analysis plugin. It is important for
+folks who want the fairly language neutral ICU analyzer but the library to
+implement the analyzer is 11MB so we don't ship it with Elasticsearch by
+default.
+
+Another example is the `discovery-gce` plugin. It is *vital* to folks running
+in [GCP](https://cloud.google.com/) but useless otherwise and it depends on a
+dozen extra jars.
+
+#### `qa`
+Honestly this is kind of in flux and we're not 100% sure where we'll end up.
+Right now the directory contains
+* Tests that require multiple modules or plugins to work
+* Tests that form a cluster made up of multiple versions of Elasticsearch like
+full cluster restart, rolling restarts, and mixed version tests
+* Tests that test the Elasticsearch clients in "interesting" places like the
+`wildfly` project.
+* Tests that test Elasticsearch in funny configurations like with ingest
+disabled
+* Tests that need to do strange things like install plugins that thrown
+uncaught `Throwable`s or add a shutdown hook
+But we're not convinced that all of these things *belong* in the qa directory.
+We're fairly sure that tests that require multiple modules or plugins to work
+should just pick a "home" plugin. We're fairly sure that the multi-version
+tests *do* belong in qa. Beyond that, we're not sure. If you want to add a new
+qa project, open a PR and be ready to discuss options.
+
+#### `server`
+The server component of Elasticsearch that contains all of the modules and
+plugins. Right now things like the high level rest client depend on the server
+but we'd like to fix that in the future.
+
+#### `test`
+Our test framework and test fixtures. We use the test framework for testing the
+server, the plugins, and modules, and pretty much everything else. We publish
+the test framework so folks who develop Elasticsearch plugins can use it to
+test the plugins. The test fixtures are external processes that we start before
+running specific tests that rely on them.
+
+For example, we have an hdfs test that uses mini-hdfs to test our
+repository-hdfs plugin.
+
+#### `x-pack`
+Commercially licensed code that integrates with the rest of Elasticsearch. The
+`docs` subdirectory functions just like the top level `docs` subdirectory and
+the `qa` subdirectory functions just like the top level `qa` subdirectory. The
+`plugin` subdirectory contains the x-pack module which runs inside the
+Elasticsearch process. The `transport-client` subdirectory contains extensions
+to Elasticsearch's standard transport client to work properly with x-pack.
+
+
 Contributing as part of a class
 -------------------------------
 In general Elasticsearch is happy to accept contributions that were created as

buildSrc/src/main/groovy/org/elasticsearch/gradle/BuildPlugin.groovy

@@ -497,10 +497,15 @@ class BuildPlugin implements Plugin<Project> {
         project.afterEvaluate {
             project.tasks.withType(JavaCompile) {
                 final JavaVersion targetCompatibilityVersion = JavaVersion.toVersion(it.targetCompatibility)
-                // we fork because compiling lots of different classes in a shared jvm can eventually trigger GC overhead limitations
-                options.fork = true
-                options.forkOptions.javaHome = new File(project.compilerJavaHome)
-                options.forkOptions.memoryMaximumSize = "512m"
+                final compilerJavaHomeFile = new File(project.compilerJavaHome)
+                // we only fork if the Gradle JDK is not the same as the compiler JDK
+                if (compilerJavaHomeFile.canonicalPath == Jvm.current().javaHome.canonicalPath) {
+                    options.fork = false
+                } else {
+                    options.fork = true
+                    options.forkOptions.javaHome = compilerJavaHomeFile
+                    options.forkOptions.memoryMaximumSize = "512m"
+                }
                 if (targetCompatibilityVersion == JavaVersion.VERSION_1_8) {
                     // compile with compact 3 profile by default
                     // NOTE: this is just a compile time check: does not replace testing with a compact3 JRE
@@ -549,6 +554,11 @@ class BuildPlugin implements Plugin<Project> {
             javadoc.classpath = javadoc.getClasspath().filter { f ->
                 return classes.contains(f) == false
             }
+            /*
+             * Generate docs using html5 to suppress a warning from `javadoc`
+             * that the default will change to html5 in the future.
+             */
+            javadoc.options.addBooleanOption('html5', true)
         }
         configureJavadocJar(project)
     }

client/rest/src/main/java/org/elasticsearch/client/RestClient.java

@@ -210,7 +210,9 @@ public void performRequestAsync(Request request, ResponseListener responseListen
      * @throws IOException in case of a problem or the connection was aborted
      * @throws ClientProtocolException in case of an http protocol error
      * @throws ResponseException in case Elasticsearch responded with a status code that indicated an error
+     * @deprecated prefer {@link #performRequest(Request)}
      */
+    @Deprecated
     public Response performRequest(String method, String endpoint, Header... headers) throws IOException {
         Request request = new Request(method, endpoint);
         request.setHeaders(headers);
@@ -229,7 +231,9 @@ public Response performRequest(String method, String endpoint, Header... headers
      * @throws IOException in case of a problem or the connection was aborted
      * @throws ClientProtocolException in case of an http protocol error
      * @throws ResponseException in case Elasticsearch responded with a status code that indicated an error
+     * @deprecated prefer {@link #performRequest(Request)}
      */
+    @Deprecated
     public Response performRequest(String method, String endpoint, Map<String, String> params, Header... headers) throws IOException {
         Request request = new Request(method, endpoint);
         addParameters(request, params);
@@ -252,7 +256,9 @@ public Response performRequest(String method, String endpoint, Map<String, Strin
      * @throws IOException in case of a problem or the connection was aborted
      * @throws ClientProtocolException in case of an http protocol error
      * @throws ResponseException in case Elasticsearch responded with a status code that indicated an error
+     * @deprecated prefer {@link #performRequest(Request)}
      */
+    @Deprecated
     public Response performRequest(String method, String endpoint, Map<String, String> params,
                                    HttpEntity entity, Header... headers) throws IOException {
         Request request = new Request(method, endpoint);
@@ -289,7 +295,9 @@ public Response performRequest(String method, String endpoint, Map<String, Strin
      * @throws IOException in case of a problem or the connection was aborted
      * @throws ClientProtocolException in case of an http protocol error
      * @throws ResponseException in case Elasticsearch responded with a status code that indicated an error
+     * @deprecated prefer {@link #performRequest(Request)}
      */
+    @Deprecated
     public Response performRequest(String method, String endpoint, Map<String, String> params,
                                    HttpEntity entity, HttpAsyncResponseConsumerFactory httpAsyncResponseConsumerFactory,
                                    Header... headers) throws IOException {
@@ -310,7 +318,9 @@ public Response performRequest(String method, String endpoint, Map<String, Strin
      * @param endpoint the path of the request (without host and port)
      * @param responseListener the {@link ResponseListener} to notify when the request is completed or fails
      * @param headers the optional request headers
+     * @deprecated prefer {@link #performRequestAsync(Request, ResponseListener)}
      */
+    @Deprecated
     public void performRequestAsync(String method, String endpoint, ResponseListener responseListener, Header... headers) {
         Request request;
         try {
@@ -333,7 +343,9 @@ public void performRequestAsync(String method, String endpoint, ResponseListener
      * @param params the query_string parameters
      * @param responseListener the {@link ResponseListener} to notify when the request is completed or fails
      * @param headers the optional request headers
+     * @deprecated prefer {@link #performRequestAsync(Request, ResponseListener)}
      */
+    @Deprecated
     public void performRequestAsync(String method, String endpoint, Map<String, String> params,
                                     ResponseListener responseListener, Header... headers) {
         Request request;
@@ -361,7 +373,9 @@ public void performRequestAsync(String method, String endpoint, Map<String, Stri
      * @param entity the body of the request, null if not applicable
      * @param responseListener the {@link ResponseListener} to notify when the request is completed or fails
      * @param headers the optional request headers
+     * @deprecated prefer {@link #performRequestAsync(Request, ResponseListener)}
      */
+    @Deprecated
     public void performRequestAsync(String method, String endpoint, Map<String, String> params,
                                     HttpEntity entity, ResponseListener responseListener, Header... headers) {
         Request request;
@@ -394,7 +408,9 @@ public void performRequestAsync(String method, String endpoint, Map<String, Stri
      * connection on the client side.
      * @param responseListener the {@link ResponseListener} to notify when the request is completed or fails
      * @param headers the optional request headers
+     * @deprecated prefer {@link #performRequestAsync(Request, ResponseListener)}
      */
+    @Deprecated
     public void performRequestAsync(String method, String endpoint, Map<String, String> params,
                                     HttpEntity entity, HttpAsyncResponseConsumerFactory httpAsyncResponseConsumerFactory,
                                     ResponseListener responseListener, Header... headers) {

docs/CHANGELOG.asciidoc

@@ -3,7 +3,7 @@
 
 [partintro]
 --
-// To add a release, copy and paste the template text 
+// To add a release, copy and paste the template text
 // and add a link to the new section. Note that release subheads must
 // be floated and sections cannot be empty.
 
@@ -104,6 +104,8 @@ ones that the user is authorized to access in case field level security is enabl
 [float]
 === Bug Fixes
 
+Fix NPE in 'more_like_this' when field has zero tokens ({pull}30365[#30365])
+
 Fixed prerelease version of elasticsearch in the `deb` package to sort before GA versions
 ({pull}29000[#29000])
 
@@ -137,8 +139,11 @@ coming[6.4.0]
 //[float]
 //=== Breaking Java Changes
 
-//[float]
-//=== Deprecations
+[float]
+=== Deprecations
+
+Deprecated multi-argument versions of the request methods in the RestClient.
+Prefer the "Request" object flavored methods. ({pull}30315[#30315])
 
 [float]
 === New Features
@@ -155,8 +160,8 @@ analysis module.  ({pull}30397[#30397])
 
 {ref-64}/breaking_64_api_changes.html#copy-source-settings-on-resize[Allow copying source settings on index resize operations] ({pull}30255[#30255])
 
-Added new "Request" object flavored request methods. Prefer these instead of the
-multi-argument versions. ({pull}29623[#29623])
+Added new "Request" object flavored request methods in the RestClient. Prefer
+these instead of the multi-argument versions. ({pull}29623[#29623])
 
 The cluster state listener to decide if watcher should be
 stopped/started/paused now runs far less code in an executor but is more
@@ -169,6 +174,8 @@ Added put index template API to the high level rest client ({pull}30400[#30400])
 [float]
 === Bug Fixes
 
+Fix NPE in 'more_like_this' when field has zero tokens ({pull}30365[#30365])
+
 Do not ignore request analysis/similarity settings on index resize operations when the source index already contains such settings ({pull}30216[#30216])
 
 Fix NPE when CumulativeSum agg encounters null value/empty bucket ({pull}29641[#29641])
@@ -177,10 +184,18 @@ Machine Learning::
 
 * Account for gaps in data counts after job is reopened ({pull}30294[#30294])
 
+Add validation that geohashes are not empty and don't contain unsupported characters ({pull}30376[#30376])
+
 Rollup::
 * Validate timezone in range queries to ensure they match the selected job when
 searching ({pull}30338[#30338])
 
+
+Allocation::
+
+Auto-expand replicas when adding or removing nodes to prevent shard copies from
+being dropped and resynced when a data node rejoins the cluster ({pull}30423[#30423])
+
 //[float]
 //=== Regressions
 

docs/plugins/api.asciidoc

@@ -19,6 +19,9 @@ A number of plugins have been contributed by our community:
 
 * https://github.com/YannBrrd/elasticsearch-entity-resolution[Entity Resolution Plugin]:
   Uses http://github.com/larsga/Duke[Duke] for duplication detection (by Yann Barraud)
+
+* https://github.com/zentity-io/zentity[Entity Resolution Plugin] (https://zentity.io[zentity]):
+  Real-time entity resolution with pure Elasticsearch (by Dave Moore)
 
 * https://github.com/NLPchina/elasticsearch-sql/[SQL language Plugin]:
   Allows Elasticsearch to be queried with SQL (by nlpcn)

docs/reference/aggregations/metrics/cardinality-aggregation.asciidoc

@@ -63,7 +63,7 @@ POST /sales/_search?size=0
 defines a unique count below which counts are expected to be close to
 accurate. Above this value, counts might become a bit more fuzzy. The maximum
 supported value is 40000, thresholds above this number will have the same
-effect as a threshold of 40000. The default values is +3000+.
+effect as a threshold of 40000. The default value is +3000+.
 
 ==== Counts are approximate
 

docs/reference/cluster/tasks.asciidoc

@@ -64,9 +64,10 @@ It is also possible to retrieve information for a particular task:
 
 [source,js]
 --------------------------------------------------
-GET _tasks/task_id:1 <1>
+GET _tasks/task_id <1>
 --------------------------------------------------
 // CONSOLE
+// TEST[s/task_id/node_id:1/]
 // TEST[catch:missing]
 
 <1> This will return a 404 if the task isn't found.
@@ -75,9 +76,10 @@ Or to retrieve all children of a particular task:
 
 [source,js]
 --------------------------------------------------
-GET _tasks?parent_task_id=parentTaskId:1 <1>
+GET _tasks?parent_task_id=parent_task_id <1>
 --------------------------------------------------
 // CONSOLE
+// TEST[s/=parent_task_id/=node_id:1/]
 
 <1> This won't return a 404 if the parent isn't found.
 

docs/reference/docs/delete-by-query.asciidoc

@@ -357,9 +357,10 @@ With the task id you can look up the task directly:
 
 [source,js]
 --------------------------------------------------
-GET /_tasks/taskId:1
+GET /_tasks/task_id
 --------------------------------------------------
 // CONSOLE
+// TEST[s/task_id/node_id:1/]
 // TEST[catch:missing]
 
 The advantage of this API is that it integrates with `wait_for_completion=false`
@@ -378,8 +379,9 @@ Any Delete By Query can be canceled using the <<tasks,Task Cancel API>>:
 
 [source,js]
 --------------------------------------------------
-POST _tasks/task_id:1/_cancel
+POST _tasks/task_id/_cancel
 --------------------------------------------------
+// TEST[s/task_id/node_id:1/]
 // CONSOLE
 
 The `task_id` can be found using the tasks API above.
@@ -397,8 +399,9 @@ using the `_rethrottle` API:
 
 [source,js]
 --------------------------------------------------
-POST _delete_by_query/task_id:1/_rethrottle?requests_per_second=-1
+POST _delete_by_query/task_id/_rethrottle?requests_per_second=-1
 --------------------------------------------------
+// TEST[s/task_id/node_id:1/]
 // CONSOLE
 
 The `task_id` can be found using the tasks API above.