Merge branch 'master' into translog-generation

* master:
  Fix typo in allocation explain API docs
  Add unit tests for ReverseNestedAggregator (elastic#23651)
  Revert "Revert "Build: Upgrade min gradle to 3.3 (elastic#23544)""
  Revert "Build: Upgrade min gradle to 3.3 (elastic#23544)"
  Build: Upgrade min gradle to 3.3 (elastic#23544)
  Fix took assertion in response filter test
  Search took time should use a relative clock
  Adds toString() to snapshot operations in progress
  Docs: fix a typo in transport client's put-mapping.asciidoc (elastic#23607)
  Use include-tagged macro for high level client docs (elastic#23438)
  Update fill-column in .dir-locals.el to 100 characters
  Setup keystore during integration tests (elastic#22966)
  Fix typo 'Elastisearch' -> 'Elasticsearch' (elastic#23633)
  Comment and blank line cleanups (elastic#23647)
  docs: guidelines for students and teachers (elastic#23648)
  Fix MapperService StackOverflowError (elastic#23605)

.dir-locals.el

@@ -83,6 +83,6 @@
       ))
    (c-basic-offset . 4)
    (c-comment-only-line-offset . (0 . 0))
-   (fill-column . 140)
-   (fci-rule-column . 140)
+   (fill-column . 100)
+   (fci-rule-column . 100)
    (compile-command . "gradle compileTestJava"))))

.github/PULL_REQUEST_TEMPLATE.md

@@ -11,3 +11,4 @@ attention.
 - If submitting code, have you built your formula locally prior to submission with `gradle check`?
 - If submitting code, is your pull request against master? Unless there is a good reason otherwise, we prefer pull requests against master and will backport as needed.
 - If submitting code, have you checked that your submission is for an [OS that we support](https://www.elastic.co/support/matrix#show_os)?
+- If you are submitting this code for a class then read our [policy](https://github.com/elastic/elasticsearch/blob/master/CONTRIBUTING.md#contributing-as-part-of-a-class) for that.

CONTRIBUTING.md

@@ -88,8 +88,8 @@ Contributing to the Elasticsearch codebase
 **Repository:** [https://github.com/elastic/elasticsearch](https://github.com/elastic/elasticsearch)
 
 Make sure you have [Gradle](http://gradle.org) installed, as
-Elasticsearch uses it as its build system. Gradle must be version 2.13 _exactly_ in
-order to build successfully.
+Elasticsearch uses it as its build system. Gradle must be at least
+version 3.3 in order to build successfully.
 
 Eclipse users can automatically configure their IDE: `gradle eclipse`
 then `File: Import: Existing Projects into Workspace`. Select the
@@ -139,3 +139,32 @@ Before submitting your changes, run the test suite to make sure that nothing is
 ```sh
 gradle check
 ```
+
+Contributing as part of a class
+-------------------------------
+In general Elasticsearch is happy to accept contributions that were created as
+part of a class but strongly advise against making the contribution as part of
+the class. So if you have code you wrote for a class feel free to submit it.
+
+Please, please, please do not assign contributing to Elasticsearch as part of a
+class. If you really want to assign writing code for Elasticsearch as an
+assignment then the code contributions should be made to your private clone and
+opening PRs against the primary Elasticsearch clone must be optional, fully
+voluntary, not for a grade, and without any deadlines.
+
+Because:
+
+* While the code review process is likely very educational, it can take wildly
+varying amounts of time depending on who is available, where the change is, and
+how deep the change is. There is no way to predict how long it will take unless
+we rush.
+* We do not rush reviews without a very, very good reason. Class deadlines
+aren't a good enough reason for us to rush reviews.
+* We deeply discourage opening a PR you don't intend to work through the entire
+code review process because it wastes our time.
+* We don't have the capacity to absorb an entire class full of new contributors,
+especially when they are unlikely to become long time contributors.
+
+Finally, we require that you run `gradle check` before submitting a
+non-documentation contribution. This is mentioned above, but it is worth
+repeating in this section because it has come up in this context.

README.textile

@@ -200,7 +200,7 @@ We have just covered a very small portion of what Elasticsearch is all about. Fo
 
 h3. Building from Source
 
-Elasticsearch uses "Gradle":https://gradle.org for its build system. You'll need to have version 2.13 of Gradle installed.
+Elasticsearch uses "Gradle":https://gradle.org for its build system. You'll need to have at least version 3.3 of Gradle installed.
 
 In order to create a distribution, simply run the @gradle assemble@ command in the cloned directory.
 

buildSrc/build.gradle

@@ -23,8 +23,8 @@ apply plugin: 'groovy'
 
 group = 'org.elasticsearch.gradle'
 
-if (GradleVersion.current() < GradleVersion.version('2.13')) {
-  throw new GradleException('Gradle 2.13+ is required to build elasticsearch')
+if (GradleVersion.current() < GradleVersion.version('3.3')) {
+  throw new GradleException('Gradle 3.3+ is required to build elasticsearch')
 }
 
 if (JavaVersion.current() < JavaVersion.VERSION_1_8) {

buildSrc/src/main/groovy/org/elasticsearch/gradle/BuildPlugin.groovy

@@ -122,7 +122,7 @@ class BuildPlugin implements Plugin<Project> {
             }
 
             // enforce gradle version
-            GradleVersion minGradle = GradleVersion.version('2.13')
+            GradleVersion minGradle = GradleVersion.version('3.3')
             if (GradleVersion.current() < minGradle) {
                 throw new GradleException("${minGradle} or above is required to build elasticsearch")
             }

buildSrc/src/main/groovy/org/elasticsearch/gradle/test/ClusterConfiguration.groovy

@@ -125,6 +125,8 @@ class ClusterConfiguration {
 
     Map<String, Object> settings = new HashMap<>()
 
+    Map<String, String> keystoreSettings = new HashMap<>()
+
     // map from destination path, to source file
     Map<String, Object> extraConfigFiles = new HashMap<>()
 
@@ -144,6 +146,11 @@ class ClusterConfiguration {
         settings.put(name, value)
     }
 
+    @Input
+    void keystoreSetting(String name, String value) {
+        keystoreSettings.put(name, value)
+    }
+
     @Input
     void plugin(String path) {
         Project pluginProject = project.project(path)

buildSrc/src/main/groovy/org/elasticsearch/gradle/test/ClusterFormationTasks.groovy

@@ -38,6 +38,7 @@ import org.gradle.api.tasks.Copy
 import org.gradle.api.tasks.Delete
 import org.gradle.api.tasks.Exec
 
+import java.nio.charset.StandardCharsets
 import java.nio.file.Paths
 import java.util.concurrent.TimeUnit
 
@@ -157,10 +158,14 @@ class ClusterFormationTasks {
                 node.cwd.mkdirs()
             }
         }
+
         setup = configureCheckPreviousTask(taskName(prefix, node, 'checkPrevious'), project, setup, node)
         setup = configureStopTask(taskName(prefix, node, 'stopPrevious'), project, setup, node)
         setup = configureExtractTask(taskName(prefix, node, 'extract'), project, setup, node, configuration)
         setup = configureWriteConfigTask(taskName(prefix, node, 'configure'), project, setup, node, seedNode)
+        setup = configureCreateKeystoreTask(taskName(prefix, node, 'createKeystore'), project, setup, node)
+        setup = configureAddKeystoreSettingTasks(prefix, project, setup, node)
+
         if (node.config.plugins.isEmpty() == false) {
             if (node.nodeVersion == VersionProperties.elasticsearch) {
                 setup = configureCopyPluginsTask(taskName(prefix, node, 'copyPlugins'), project, setup, node)
@@ -303,6 +308,33 @@ class ClusterFormationTasks {
         }
     }
 
+    /** Adds a task to create keystore */
+    static Task configureCreateKeystoreTask(String name, Project project, Task setup, NodeInfo node) {
+        if (node.config.keystoreSettings.isEmpty()) {
+            return setup
+        } else {
+            File esKeystoreUtil = Paths.get(node.homeDir.toString(), "bin/" + "elasticsearch-keystore").toFile()
+            return configureExecTask(name, project, setup, node, esKeystoreUtil, 'create')
+        }
+    }
+
+    /** Adds tasks to add settings to the keystore */
+    static Task configureAddKeystoreSettingTasks(String parent, Project project, Task setup, NodeInfo node) {
+        Map kvs = node.config.keystoreSettings
+        File esKeystoreUtil = Paths.get(node.homeDir.toString(), "bin/" + "elasticsearch-keystore").toFile()
+        Task parentTask = setup
+        for (Map.Entry<String, String> entry in kvs) {
+            String key = entry.getKey()
+            String name = taskName(parent, node, 'addToKeystore#' + key)
+            Task t = configureExecTask(name, project, parentTask, node, esKeystoreUtil, 'add', key, '-x')
+            t.doFirst {
+                standardInput = new ByteArrayInputStream(entry.getValue().getBytes(StandardCharsets.UTF_8))
+            }
+            parentTask = t
+        }
+        return parentTask
+    }
+
     static Task configureExtraConfigFilesTask(String name, Project project, Task setup, NodeInfo node) {
         if (node.config.extraConfigFiles.isEmpty()) {
             return setup

buildSrc/src/main/resources/checkstyle_suppressions.xml

@@ -491,12 +491,10 @@
   <suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]action[/\\]search[/\\]ReduceSearchPhaseException.java" checks="LineLength" />
   <suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]action[/\\]search[/\\]RemoteClusterConnection.java" checks="LineLength" />
   <suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]action[/\\]search[/\\]RemoteClusterService.java" checks="LineLength" />
-  <suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]action[/\\]search[/\\]SearchDfsQueryThenFetchAsyncAction.java" checks="LineLength" />
   <suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]action[/\\]search[/\\]SearchPhase.java" checks="LineLength" />
   <suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]action[/\\]search[/\\]SearchPhaseContext.java" checks="LineLength" />
   <suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]action[/\\]search[/\\]SearchPhaseController.java" checks="LineLength" />
   <suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]action[/\\]search[/\\]SearchPhaseExecutionException.java" checks="LineLength" />
-  <suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]action[/\\]search[/\\]SearchQueryThenFetchAsyncAction.java" checks="LineLength" />
   <suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]action[/\\]search[/\\]SearchRequest.java" checks="LineLength" />
   <suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]action[/\\]search[/\\]SearchRequestBuilder.java" checks="LineLength" />
   <suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]action[/\\]search[/\\]SearchResponse.java" checks="LineLength" />

client/rest-high-level/src/test/java/org/elasticsearch/client/documentation/DeleteDocumentationIT.java

@@ -56,35 +56,35 @@ public class DeleteDocumentationIT extends ESRestHighLevelClientTestCase {
     public void testDelete() throws IOException {
         RestHighLevelClient client = highLevelClient();
 
-        // tag::delete-request[]
+        // tag::delete-request
         DeleteRequest request = new DeleteRequest(
             "index",    // <1>
             "type",     // <2>
             "id");      // <3>
-        // end::delete-request[]
+        // end::delete-request
 
-        // tag::delete-request-props[]
+        // tag::delete-request-props
         request.timeout(TimeValue.timeValueSeconds(1));                     // <1>
         request.timeout("1s");                                              // <2>
         request.setRefreshPolicy(WriteRequest.RefreshPolicy.WAIT_UNTIL);    // <3>
         request.setRefreshPolicy("wait_for");                               // <4>
         request.version(2);                                                 // <5>
         request.versionType(VersionType.EXTERNAL);                          // <6>
-        // end::delete-request-props[]
+        // end::delete-request-props
 
-        // tag::delete-execute[]
+        // tag::delete-execute
         DeleteResponse response = client.delete(request);
-        // end::delete-execute[]
+        // end::delete-execute
 
         try {
-            // tag::delete-notfound[]
+            // tag::delete-notfound
             if (response.getResult().equals(DocWriteResponse.Result.NOT_FOUND)) {
                 throw new Exception("Can't find document to be removed"); // <1>
             }
-            // end::delete-notfound[]
+            // end::delete-notfound
         } catch (Exception ignored) { }
 
-        // tag::delete-execute-async[]
+        // tag::delete-execute-async
         client.deleteAsync(request, new ActionListener<DeleteResponse>() {
             @Override
             public void onResponse(DeleteResponse deleteResponse) {
@@ -96,17 +96,17 @@ public void onFailure(Exception e) {
                 // <2>
             }
         });
-        // end::delete-execute-async[]
+        // end::delete-execute-async
 
-        // tag::delete-conflict[]
+        // tag::delete-conflict
         try {
             client.delete(request);
         } catch (ElasticsearchException exception) {
             if (exception.status().equals(RestStatus.CONFLICT)) {
                 // <1>
             }
         }
-        // end::delete-conflict[]
+        // end::delete-conflict
 
     }
 }

core/src/main/java/org/elasticsearch/action/search/AbstractSearchAsyncAction.java

@@ -43,6 +43,7 @@
 import java.util.List;
 import java.util.Map;
 import java.util.concurrent.Executor;
+import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicInteger;
 import java.util.function.Function;
 import java.util.stream.Collectors;
@@ -67,17 +68,17 @@ abstract class AbstractSearchAsyncAction<Result extends SearchPhaseResult> exten
     private final SetOnce<AtomicArray<ShardSearchFailure>> shardFailures = new SetOnce<>();
     private final Object shardFailuresMutex = new Object();
     private final AtomicInteger successfulOps = new AtomicInteger();
-    private final long startTime;
+    private final TransportSearchAction.SearchTimeProvider timeProvider;
 
 
     protected AbstractSearchAsyncAction(String name, Logger logger, SearchTransportService searchTransportService,
                                         Function<String, Transport.Connection> nodeIdToConnection,
                                         Map<String, AliasFilter> aliasFilter, Map<String, Float> concreteIndexBoosts,
                                         Executor executor, SearchRequest request,
-                                        ActionListener<SearchResponse> listener, GroupShardsIterator shardsIts, long startTime,
+                                        ActionListener<SearchResponse> listener, GroupShardsIterator shardsIts, TransportSearchAction.SearchTimeProvider timeProvider,
                                         long clusterStateVersion, SearchTask task, SearchPhaseResults<Result> resultConsumer) {
         super(name, request, shardsIts, logger);
-        this.startTime = startTime;
+        this.timeProvider = timeProvider;
         this.logger = logger;
         this.searchTransportService = searchTransportService;
         this.executor = executor;
@@ -94,10 +95,9 @@ protected AbstractSearchAsyncAction(String name, Logger logger, SearchTransportS
     /**
      * Builds how long it took to execute the search.
      */
-    private long buildTookInMillis() {
-        // protect ourselves against time going backwards
-        // negative values don't make sense and we want to be able to serialize that thing as a vLong
-        return Math.max(1, System.currentTimeMillis() - startTime);
+    long buildTookInMillis() {
+        return TimeUnit.NANOSECONDS.toMillis(
+                timeProvider.getRelativeCurrentNanos() - timeProvider.getRelativeStartNanos());
     }
 
     /**
@@ -300,7 +300,7 @@ public final ShardSearchTransportRequest buildShardSearchRequest(ShardIterator s
         assert filter != null;
         float indexBoost = concreteIndexBoosts.getOrDefault(shard.index().getUUID(), DEFAULT_INDEX_BOOST);
         return new ShardSearchTransportRequest(request, shardIt.shardId(), getNumShards(),
-            filter, indexBoost, startTime);
+            filter, indexBoost, timeProvider.getAbsoluteStartMillis());
     }
 
     /**

core/src/main/java/org/elasticsearch/action/search/SearchDfsQueryThenFetchAsyncAction.java

@@ -33,28 +33,59 @@
 import java.util.function.Function;
 
 final class SearchDfsQueryThenFetchAsyncAction extends AbstractSearchAsyncAction<DfsSearchResult> {
+
     private final SearchPhaseController searchPhaseController;
 
-    SearchDfsQueryThenFetchAsyncAction(Logger logger, SearchTransportService searchTransportService,
-                                       Function<String, Transport.Connection> nodeIdToConnection,
-                                       Map<String, AliasFilter> aliasFilter, Map<String, Float> concreteIndexBoosts,
-                                       SearchPhaseController searchPhaseController, Executor executor, SearchRequest request,
-                                       ActionListener<SearchResponse> listener, GroupShardsIterator shardsIts, long startTime,
-                                       long clusterStateVersion, SearchTask task) {
-        super("dfs", logger, searchTransportService, nodeIdToConnection, aliasFilter, concreteIndexBoosts, executor,
-                request, listener, shardsIts, startTime, clusterStateVersion, task, new SearchPhaseResults<>(shardsIts.size()));
+    SearchDfsQueryThenFetchAsyncAction(
+            final Logger logger,
+            final SearchTransportService searchTransportService,
+            final Function<String, Transport.Connection> nodeIdToConnection,
+            final Map<String, AliasFilter> aliasFilter,
+            final Map<String, Float> concreteIndexBoosts,
+            final SearchPhaseController searchPhaseController,
+            final Executor executor,
+            final SearchRequest request,
+            final ActionListener<SearchResponse> listener,
+            final GroupShardsIterator shardsIts,
+            final TransportSearchAction.SearchTimeProvider timeProvider,
+            final long clusterStateVersion,
+            final SearchTask task) {
+        super(
+                "dfs",
+                logger,
+                searchTransportService,
+                nodeIdToConnection,
+                aliasFilter,
+                concreteIndexBoosts,
+                executor,
+                request,
+                listener,
+                shardsIts,
+                timeProvider,
+                clusterStateVersion,
+                task,
+                new SearchPhaseResults<>(shardsIts.size()));
         this.searchPhaseController = searchPhaseController;
     }
 
     @Override
-    protected void executePhaseOnShard(ShardIterator shardIt, ShardRouting shard, ActionListener listener) {
+    protected void executePhaseOnShard(
+            final ShardIterator shardIt,
+            final ShardRouting shard,
+            final ActionListener<DfsSearchResult> listener) {
         getSearchTransport().sendExecuteDfs(getConnection(shard.currentNodeId()),
             buildShardSearchRequest(shardIt, shard) , getTask(), listener);
     }
 
     @Override
-    protected SearchPhase getNextPhase(SearchPhaseResults<DfsSearchResult> results, SearchPhaseContext context) {
-        return new DfsQueryPhase(results.results, searchPhaseController,
-            (queryResults) -> new FetchSearchPhase(queryResults, searchPhaseController, context), context);
+    protected SearchPhase getNextPhase(
+            final SearchPhaseResults<DfsSearchResult> results, final SearchPhaseContext context) {
+        return new DfsQueryPhase(
+                results.results,
+                searchPhaseController,
+                (queryResults) ->
+                        new FetchSearchPhase(queryResults, searchPhaseController, context),
+                context);
     }
+
 }