From c6d1da4cb378d5102d4f92a5df0da02021c50836 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Wed, 4 Sep 2019 10:21:50 -0400 Subject: [PATCH 1/5] [DOCS] Reformat index stats API docs --- docs/reference/cluster/nodes-stats.asciidoc | 25 +-- docs/reference/indices/stats.asciidoc | 175 +++++++++++------- docs/reference/rest-api/common-parms.asciidoc | 141 +++++++++++++- 3 files changed, 250 insertions(+), 91 deletions(-) diff --git a/docs/reference/cluster/nodes-stats.asciidoc b/docs/reference/cluster/nodes-stats.asciidoc index b5f3c3bb2d4e6..521b3f18329db 100644 --- a/docs/reference/cluster/nodes-stats.asciidoc +++ b/docs/reference/cluster/nodes-stats.asciidoc @@ -113,25 +113,15 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=node-id] [[cluster-nodes-stats-api-query-params]] ==== {api-query-parms-title} -`completion_fields`:: - (Optional, string) A comma-separated list of fields for `fielddata` and - `suggest` index metric (supports wildcards). +include::{docdir}/rest-api/common-parms.asciidoc[tag=completion-fields] -`fielddata_fields`:: - (Optional, string) A comma-separated list of fields for `fielddata` index - metric (supports wildcards). +include::{docdir}/rest-api/common-parms.asciidoc[tag=fielddata-fields] -`fields`:: - (Optional, string) A comma-separated list of fields for `fielddata` and - `completion` index metric (supports wildcards). +include::{docdir}/rest-api/common-parms.asciidoc[tag=fields] -`groups`:: - (Optional, string) A comma-separated list of search groups for `search` - index metric. +include::{docdir}/rest-api/common-parms.asciidoc[tag=groups] -`level`:: - (Optional, string) Returns indices stats aggregated at index, node or shard - level. Supported options: `indices`, `node`, `shards`. +include::{docdir}/rest-api/common-parms.asciidoc[tag=level] `types`:: (Optional, string) A comma-separated list of document types for the @@ -139,10 +129,7 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=node-id] include::{docdir}/rest-api/common-parms.asciidoc[tag=timeoutparms] -`include_segment_file_sizes`:: - (Optional, boolean) If `true`, the call reports the aggregated disk usage of - each one of the Lucene index files (only applies if segment stats are - requested). Defaults to `false`. +include::{docdir}/rest-api/common-parms.asciidoc[tag=include-segment-file-sizes] [[cluster-nodes-stats-api-response-body]] diff --git a/docs/reference/indices/stats.asciidoc b/docs/reference/indices/stats.asciidoc index c85d63c8e579e..50cf24945dccc 100644 --- a/docs/reference/indices/stats.asciidoc +++ b/docs/reference/indices/stats.asciidoc @@ -1,98 +1,143 @@ [[indices-stats]] -=== Indices Stats +=== Index stats API +++++ +Index stats +++++ -Indices level stats provide statistics on different operations happening -on an index. The API provides statistics on the index level scope -(though most stats can also be retrieved using node level scope). - -The following returns high level aggregation and index level stats for -all indices: +Returns statistics for an index. [source,js] --------------------------------------------------- -GET /_stats --------------------------------------------------- +---- +GET /twitter/_stats +---- // CONSOLE +// TEST[setup:twitter] -Specific index stats can be retrieved using: -[source,js] --------------------------------------------------- -GET /index1,index2/_stats --------------------------------------------------- -// CONSOLE -// TEST[s/^/PUT index1\nPUT index2\n/] +[[index-stats-api-request]] +==== {api-request-title} + +`GET //_stats` + +`GET //_stats/` + +`GET /_stats` + + +[[index-stats-api-desc]] +==== {api-description-title} + +Use the index stats API to get high-level aggregation and statistics for an index. + +By default, +the returned statistics are index-level +with `primaries` and `total` aggregations. +`primaries` are the values for only the primary shards. +`total` are the accumulated values for both primary and replica shards. + +To get shard-level statistics, +set the `level` parameter to `shards`. + +[NOTE] +==== +When moving to another node, +the shard-level statistics for a shard are cleared. +Although the shard +is no longer part of the node, +that node retains any node-level statistics +to which the shard contributed. +==== + + +[[index-stats-api-path-params]] +==== {api-path-parms-title} + +include::{docdir}/rest-api/common-parms.asciidoc[tag=index] ++ +To retrieve statistics for all indices, +use a value of `_all` or `*` or omit this parameter. + +include::{docdir}/rest-api/common-parms.asciidoc[tag=index-metric] -By default, all stats are returned, returning only specific stats can be -specified as well in the URI. Those stats can be any of: -[horizontal] -`docs`:: The number of docs / deleted docs (docs not yet merged out). - Note, affected by refreshing the index. +[[index-stats-api-query-params]] +==== {api-query-parms-title} -`store`:: The size of the index. +include::{docdir}/rest-api/common-parms.asciidoc[tag=expand-wildcards] ++ +Defaults to `open`. -`indexing`:: Indexing statistics, can be combined with a comma - separated list of `types` to provide document type level stats. +include::{docdir}/rest-api/common-parms.asciidoc[tag=fields] -`get`:: Get statistics, including missing stats. +include::{docdir}/rest-api/common-parms.asciidoc[tag=completion-fields] -`search`:: Search statistics including suggest statistics. - You can include statistics for custom groups by adding - an extra `groups` parameter (search operations can be associated with one or more - groups). The `groups` parameter accepts a comma separated list of group names. - Use `_all` to return statistics for all groups. +include::{docdir}/rest-api/common-parms.asciidoc[tag=fielddata-fields] -`segments`:: Retrieve the memory use of the open segments. Optionally, setting the `include_segment_file_sizes` flag, report the aggregated disk usage of each one of the Lucene index files. +`forbid_closed_indices`:: +(Optional, boolean) +If `true`, statistics are *not* collected from closed indices. +Defaults to `true`. -`completion`:: Completion suggest statistics. -`fielddata`:: Fielddata statistics. -`flush`:: Flush statistics. -`merge`:: Merge statistics. -`request_cache`:: <> statistics. -`refresh`:: Refresh statistics. -`warmer`:: Warmer statistics. -`translog`:: Translog statistics. +include::{docdir}/rest-api/common-parms.asciidoc[tag=groups] -Some statistics allow per field granularity which accepts a list -comma-separated list of included fields. By default all fields are included: +include::{docdir}/rest-api/common-parms.asciidoc[tag=level] -[horizontal] -`fields`:: +include::{docdir}/rest-api/common-parms.asciidoc[tag=include-segment-file-sizes] - List of fields to be included in the statistics. This is used as the - default list unless a more specific field list is provided (see below). +include::{docdir}/rest-api/common-parms.asciidoc[tag=include-unloaded-segments] -`completion_fields`:: - List of fields to be included in the Completion Suggest statistics. +[[index-stats-api-example]] +==== {api-examples-title} -`fielddata_fields`:: - List of fields to be included in the Fielddata statistics. +[[index-stats-api-multiple-ex]] +===== Get statistics for multiple indices +[source,js] +-------------------------------------------------- +GET /index1,index2/_stats +-------------------------------------------------- +// CONSOLE +// TEST[s/^/PUT index1\nPUT index2\n/] -Here are some samples: + +[[index-stats-api-all-ex]] +===== Get statistics for all indices + +[source,js] +-------------------------------------------------- +GET /_stats +-------------------------------------------------- +// CONSOLE +// TEST[setup:twitter] + + +[[index-stats-api-specific-stats-ex]] +===== Get specific statistics + +The following request returns +only the `merge` and `refresh` statistics +for all indices. [source,js] -------------------------------------------------- -# Get back stats for merge and refresh only for all indices GET /_stats/merge,refresh -# Get back stats for type1 and type2 documents for the my_index index -GET /my_index/_stats/indexing?types=type1,type2 -# Get back just search stats for group1 and group2 -GET /_stats/search?groups=group1,group2 -------------------------------------------------- // CONSOLE -// TEST[s/^/PUT my_index\n/] +// TEST[setup:twitter] + -The stats returned are aggregated on the index level, with -`primaries` and `total` aggregations, where `primaries` are the values for only the -primary shards, and `total` are the accumulated values for both primary and replica shards. +[[index-stats-api-specific-groups-ex]] +===== Get statistics for specific search groups -In order to get back shard level stats, set the `level` parameter to `shards`. +The following request returns +only search statistics +for the `group1` and `group2` search groups. -Note, as shards move around the cluster, their stats will be cleared as -they are created on other nodes. On the other hand, even though a shard -"left" a node, that node will still retain the stats that shard -contributed to. +[source,js] +-------------------------------------------------- +GET /_stats/search?groups=group1,group2 +-------------------------------------------------- +// CONSOLE +// TEST[setup:twitter] diff --git a/docs/reference/rest-api/common-parms.asciidoc b/docs/reference/rest-api/common-parms.asciidoc index f281c2112fecf..43acc9c2d4d8b 100644 --- a/docs/reference/rest-api/common-parms.asciidoc +++ b/docs/reference/rest-api/common-parms.asciidoc @@ -28,6 +28,13 @@ tag::bytes[] (Optional, <>) Unit used to display byte values. end::bytes[] +tag::completion-fields[] +`completion_fields`:: +(Optional, string) +Comma-separated list or wildcard expressions of fields +to include in `fielddata` and `suggest` statistics. +end::completion-fields[] + tag::default_operator[] `default_operator`:: (Optional, string) The default operator for query string query: AND or OR. @@ -61,11 +68,26 @@ Wildcard expressions are not accepted. -- end::expand-wildcards[] -tag::flat-settings[] -`flat_settings`:: -(Optional, boolean) If `true`, returns settings in flat format. Defaults to -`false`. -end::flat-settings[] +tag::fielddata-fields[] +`fielddata_fields`:: +(Optional, string) +Comma-separated list or wildcard expressions of fields +to include in `fielddata` statistics. +end::fielddata-fields[] + +tag::fields[] +`fields`:: ++ +-- +(Optional, string) +Comma-separated list or wildcard expressions of fields +to include in the statistics. + +Used as the default list +unless a specific field list is provided +in the `completion_fields` or `fielddata_fields` parameters. +-- +end::fields[] tag::index-alias-filter[] <> @@ -75,6 +97,12 @@ If specified, the index alias only applies to documents returned by the filter. end::index-alias-filter[] +tag::flat-settings[] +`flat_settings`:: +(Optional, boolean) If `true`, returns settings in flat format. Defaults to +`false`. +end::flat-settings[] + tag::http-format[] `format`:: (Optional, string) Short version of the @@ -87,6 +115,13 @@ tag::from[] (Optional, integer) Starting document offset. Defaults to `0`. end::from[] +tag::groups[] +`groups`:: +(Optional, string) +Comma-separated list of search groups +to include in the `search` statistics. +end::groups[] + tag::cat-h[] `h`:: (Optional, string) Comma-separated list of column names to display. @@ -129,6 +164,14 @@ tag::index-ignore-unavailable[] response. Defaults to `false`. end::index-ignore-unavailable[] +tag::include-segment-file-sizes[] +`include_segment_file_sizes`:: +(Optional, boolean) +If `true`, the call reports the aggregated disk usage of +each one of the Lucene index files (only applies if segment stats are +requested). Defaults to `false`. +end::include-segment-file-sizes[] + tag::include-unloaded-segments[] `include_unloaded_segments`:: (Optional, boolean) If `true`, the response includes information from segments @@ -141,6 +184,74 @@ tag::index[] used to limit the request. end::index[] +tag::index-metric[] +``:: ++ +-- +(Optional, string) +Comma-separated list of metrics used to limit the request. +Supported metrics are: + +`_all`:: +Return all statistics + +`completion`:: +<> statistics. + +`docs`:: +Number of documents and deleted docs, which have not yet merged out. +<> can affect this statistic. + +`fielddata`:: +<> statistics. + +`flush`:: +<> statistics. + +`get`:: +Get statistics, +including missing stats. + +`indexing`:: +<> statistics. + +`merge`:: +<> statistics. + +`query_cache`:: +<> statistics. + +`refresh`:: +<> statistics. + +`request_cache`:: +<> statistics. + +`search`:: +Search statistics including suggest statistics. +You can include statistics for custom groups +by adding an extra `groups` parameter +(search operations can be associated with one or more groups). +The `groups` parameter accepts a comma separated list of group names. +Use `_all` to return statistics for all groups. + +`segments`:: +Retrieve the memory use of the open segments. Optionally, setting the + +`store`:: +Size of the index in <>. + +`suggest`:: +<> statistics. + +`translog`:: +<> statistics. + +`warmer`:: +<> statistics. +-- +end::index-metric[] + tag::index-template[] ``:: (Required, string) @@ -154,6 +265,22 @@ tag::lenient[] providing text to a numeric field) will be ignored. Defaults to `false`. end::lenient[] +tag::level[] +`level`:: ++ +-- +(Optional, string) +Indicates whether statistics are aggregated +at the cluster, index, or shard level. + +Valid values are: + +* `cluster` +* `indices` +* `shards` +-- +end::level[] + tag::local[] `local`:: (Optional, boolean) If `true`, the request retrieves information from the local @@ -210,8 +337,8 @@ end::request_cache[] tag::requests_per_second[] `requests_per_second`:: - (Optional, integer) The throttle for this request in sub-requests per second. - -1 means no throttle. Defaults to 0. +(Optional, integer) The throttle for this request in sub-requests per second. +-1 means no throttle. Defaults to 0. end::requests_per_second[] tag::routing[] From f3f46c27eb2fd002b390c0637328f11cc85a0ec7 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Wed, 4 Sep 2019 16:05:43 -0400 Subject: [PATCH 2/5] change // CONSOLE comments to [source,console] --- docs/reference/indices/stats.asciidoc | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/docs/reference/indices/stats.asciidoc b/docs/reference/indices/stats.asciidoc index 50cf24945dccc..089704c96618a 100644 --- a/docs/reference/indices/stats.asciidoc +++ b/docs/reference/indices/stats.asciidoc @@ -6,21 +6,20 @@ Returns statistics for an index. -[source,js] +[source,console] ---- GET /twitter/_stats ---- -// CONSOLE // TEST[setup:twitter] [[index-stats-api-request]] ==== {api-request-title} -`GET //_stats` - `GET //_stats/` +`GET //_stats` + `GET /_stats` @@ -94,22 +93,20 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=include-unloaded-segments] [[index-stats-api-multiple-ex]] ===== Get statistics for multiple indices -[source,js] +[source,console] -------------------------------------------------- GET /index1,index2/_stats -------------------------------------------------- -// CONSOLE // TEST[s/^/PUT index1\nPUT index2\n/] [[index-stats-api-all-ex]] ===== Get statistics for all indices -[source,js] +[source,console] -------------------------------------------------- GET /_stats -------------------------------------------------- -// CONSOLE // TEST[setup:twitter] @@ -120,11 +117,10 @@ The following request returns only the `merge` and `refresh` statistics for all indices. -[source,js] +[source,console] -------------------------------------------------- GET /_stats/merge,refresh -------------------------------------------------- -// CONSOLE // TEST[setup:twitter] @@ -135,9 +131,8 @@ The following request returns only search statistics for the `group1` and `group2` search groups. -[source,js] +[source,console] -------------------------------------------------- GET /_stats/search?groups=group1,group2 -------------------------------------------------- -// CONSOLE // TEST[setup:twitter] From da54465234a48f9bdff06b75e2b946ef6f4fdd06 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Thu, 5 Sep 2019 08:27:14 -0400 Subject: [PATCH 3/5] Update docs/reference/rest-api/common-parms.asciidoc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: István Zoltán Szabó --- docs/reference/rest-api/common-parms.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/rest-api/common-parms.asciidoc b/docs/reference/rest-api/common-parms.asciidoc index 43acc9c2d4d8b..e759591aad976 100644 --- a/docs/reference/rest-api/common-parms.asciidoc +++ b/docs/reference/rest-api/common-parms.asciidoc @@ -193,7 +193,7 @@ Comma-separated list of metrics used to limit the request. Supported metrics are: `_all`:: -Return all statistics +Return all statistics. `completion`:: <> statistics. From 2e4d91ef4c1ec6fb4122949d2e618ca7f5e6992f Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Thu, 5 Sep 2019 08:31:27 -0400 Subject: [PATCH 4/5] fix copy/paste error --- docs/reference/rest-api/common-parms.asciidoc | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/reference/rest-api/common-parms.asciidoc b/docs/reference/rest-api/common-parms.asciidoc index e759591aad976..d687050f0ce4d 100644 --- a/docs/reference/rest-api/common-parms.asciidoc +++ b/docs/reference/rest-api/common-parms.asciidoc @@ -236,7 +236,11 @@ The `groups` parameter accepts a comma separated list of group names. Use `_all` to return statistics for all groups. `segments`:: -Retrieve the memory use of the open segments. Optionally, setting the +Memory use of all open segments. ++ +If the `include_segment_file_sizes` parameter is `true`, +this statistic includes the aggregated disk usage +of each Lucene index file. `store`:: Size of the index in <>. From 730f68f683d3cd86f157b1795497bbb7ab77400a Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Thu, 5 Sep 2019 08:32:22 -0400 Subject: [PATCH 5/5] fix wording --- docs/reference/rest-api/common-parms.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/rest-api/common-parms.asciidoc b/docs/reference/rest-api/common-parms.asciidoc index d687050f0ce4d..ff0ae9bf56bfe 100644 --- a/docs/reference/rest-api/common-parms.asciidoc +++ b/docs/reference/rest-api/common-parms.asciidoc @@ -239,7 +239,7 @@ Use `_all` to return statistics for all groups. Memory use of all open segments. + If the `include_segment_file_sizes` parameter is `true`, -this statistic includes the aggregated disk usage +this metric includes the aggregated disk usage of each Lucene index file. `store`::