Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Align docs etc with new discovery setting names #38492

Merged
merged 5 commits into from
Feb 6, 2019
Merged

Align docs etc with new discovery setting names #38492

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c014dcf
Align docs etc with new discovery setting names
DaveCTurner Feb 6, 2019
e468397
Reorder settings and lose duplicate definitions
DaveCTurner Feb 6, 2019
448f8a1
More reordering
DaveCTurner Feb 6, 2019
c1f25d0
Review feedback
DaveCTurner Feb 6, 2019
f07a2a8
Fix list syntax
DaveCTurner Feb 6, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions docs/plugins/discovery-azure-classic.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
[[discovery-azure-classic]]
=== Azure Classic Discovery Plugin

The Azure Classic Discovery plugin uses the Azure Classic API for unicast discovery.
The Azure Classic Discovery plugin uses the Azure Classic API to identify the
addresses of seed hosts.

// TODO: Link to ARM plugin when ready
// See issue https://github.com/elastic/elasticsearch/issues/19146
Expand All @@ -14,8 +15,8 @@ include::install_remove.asciidoc[]
[[discovery-azure-classic-usage]]
==== Azure Virtual Machine Discovery

Azure VM discovery allows to use the azure APIs to perform automatic discovery (similar to multicast in non hostile
multicast environments). Here is a simple sample configuration:
Azure VM discovery allows to use the azure APIs to perform automatic discovery.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/azure APIs/Azure APIs/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep c1f25d0

Here is a simple sample configuration:

[source,yaml]
----
Expand Down
39 changes: 21 additions & 18 deletions docs/plugins/discovery-ec2.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
[[discovery-ec2]]
=== EC2 Discovery Plugin

The EC2 discovery plugin uses the https://github.com/aws/aws-sdk-java[AWS API] for unicast discovery.
The EC2 discovery plugin uses the https://github.com/aws/aws-sdk-java[AWS API]
to identify the addresses of seed hosts.

*If you are looking for a hosted solution of Elasticsearch on AWS, please visit http://www.elastic.co/cloud.*

Expand All @@ -11,12 +12,12 @@ include::install_remove.asciidoc[]
[[discovery-ec2-usage]]
==== Getting started with AWS

The plugin provides a hosts provider for zen discovery named `ec2`. This hosts
provider finds other Elasticsearch instances in EC2 through AWS metadata.
Authentication is done using
The plugin adds a seed hosts provider named `ec2`. This seed hosts provider
finds other Elasticsearch instances in EC2 by querying the AWS metadata
service. Authentication is done using
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html[IAM
Role] credentials by default. To enable the plugin, set the unicast host
provider for Zen discovery to `ec2`:
Role] credentials by default. To enable the plugin, configure {es} to use the
`ec2` seed hosts provider:

[source,yaml]
----
Expand All @@ -25,9 +26,9 @@ discovery.seed_providers: ec2

==== Settings

EC2 host discovery supports a number of settings.
Some settings are sensitive and must be stored in the {ref}/secure-settings.html[elasticsearch keystore].
For example, to use explicit AWS access keys:
EC2 discovery supports a number of settings. Some settings are sensitive and
must be stored in the {ref}/secure-settings.html[elasticsearch keystore]. For
example, to use explicit AWS access keys:

[source,sh]
----
Expand Down Expand Up @@ -184,15 +185,18 @@ Management Console. It should look similar to this.
[[discovery-ec2-filtering]]
===== Filtering by Tags

The ec2 discovery can also filter machines to include in the cluster based on tags (and not just groups). The settings
to use include the `discovery.ec2.tag.` prefix. For example, if you defined a tag `stage` in EC2 and set it to `dev`,
setting `discovery.ec2.tag.stage` to `dev` will only filter instances with a tag key set to `stage`, and a value
of `dev`. Adding multiple `discovery.ec2.tag` settings will require all of those tags to be set for the instance to be included.
The ec2 discovery plugin can also filter machines to include in the cluster
based on tags (and not just groups). The settings to use include the
`discovery.ec2.tag.` prefix. For example, if you defined a tag `stage` in EC2
and set it to `dev`, setting `discovery.ec2.tag.stage` to `dev` will only
filter instances with a tag key set to `stage`, and a value of `dev`. Adding
multiple `discovery.ec2.tag` settings will require all of those tags to be set
for the instance to be included.

One practical use for tag filtering is when an ec2 cluster contains many nodes that are not running Elasticsearch. In
this case (particularly with high `discovery.zen.ping_timeout` values) there is a risk that a new node's discovery phase
will end before it has found the cluster (which will result in it declaring itself master of a new cluster with the same
name - highly undesirable). Tagging Elasticsearch ec2 nodes and then filtering by that tag will resolve this issue.
One practical use for tag filtering is when an ec2 cluster contains many nodes
that are not master-eligible {es} nodes. In this case, tagging the ec2
instances that _are_ running the master-eligible {es} nodes, and then filtering
by that tag, will help discovery to run more efficiently.

[[discovery-ec2-attributes]]
===== Automatic Node Attributes
Expand Down Expand Up @@ -229,7 +233,6 @@ Prefer https://aws.amazon.com/amazon-linux-ami/[Amazon Linux AMIs] as since Elas

===== Networking
* Networking throttling takes place on smaller instance types in both the form of https://lab.getbase.com/how-we-discovered-limitations-on-the-aws-tcp-stack/[bandwidth and number of connections]. Therefore if large number of connections are needed and networking is becoming a bottleneck, avoid https://aws.amazon.com/ec2/instance-types/[instance types] with networking labeled as `Moderate` or `Low`.
* Multicast is not supported, even when in an VPC; the aws cloud plugin which joins by performing a security group lookup.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we removing it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although it's technically correct that multicast is not supported in EC2 discovery, it's confusing to call it out specifically here: multicast isn't supported anywhere any more. We've been unicast-only for so long that it's getting kinda confusing even to mention it, so this PR removes many instances of the words "unicast" and "multicast".

Sadly not all of them: the file that file-based discovery uses is called unicast_hosts.txt.

* When running in multiple http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html[availability zones] be sure to leverage {ref}/allocation-awareness.html[shard allocation awareness] so that not all copies of shard data reside in the same availability zone.
* Do not span a cluster across regions. If necessary, use a cross cluster search.

Expand Down
16 changes: 8 additions & 8 deletions docs/plugins/discovery-gce.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,17 @@
[[discovery-gce]]
=== GCE Discovery Plugin

The Google Compute Engine Discovery plugin uses the GCE API for unicast discovery.
The Google Compute Engine Discovery plugin uses the GCE API to identify the
addresses of seed hosts.

:plugin_name: discovery-gce
include::install_remove.asciidoc[]

[[discovery-gce-usage]]
==== GCE Virtual Machine Discovery

Google Compute Engine VM discovery allows to use the google APIs to perform automatic discovery (similar to multicast
in non hostile multicast environments). Here is a simple sample configuration:
Google Compute Engine VM discovery allows to use the google APIs to perform
automatic discovery of seed hosts. Here is a simple sample configuration:

[source,yaml]
--------------------------------------------------
Expand Down Expand Up @@ -358,11 +359,10 @@ The GCE discovery can also filter machines to include in the cluster based on ta
For example, setting `discovery.gce.tags` to `dev` will only filter instances having a tag set to `dev`. Several tags
set will require all of those tags to be set for the instance to be included.

One practical use for tag filtering is when an GCE cluster contains many nodes that are not running
Elasticsearch. In this case (particularly with high `discovery.zen.ping_timeout` values) there is a risk that a new
node's discovery phase will end before it has found the cluster (which will result in it declaring itself master of a
new cluster with the same name - highly undesirable). Adding tag on Elasticsearch GCE nodes and then filtering by that
tag will resolve this issue.
One practical use for tag filtering is when a GCE cluster contains many nodes
that are not master-eligible {es} nodes. In this case, tagging the GCE
instances that _are_ running the master-eligible {es} nodes, and then filtering
by that tag, will help discovery to run more efficiently.

Add your tag when building the new instance:

Expand Down
17 changes: 9 additions & 8 deletions docs/plugins/discovery.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
[[discovery]]
== Discovery Plugins

Discovery plugins extend Elasticsearch by adding new hosts providers that can be
used to extend the {ref}/modules-discovery.html[cluster formation module].
Discovery plugins extend Elasticsearch by adding new seed hosts providers that
can be used to extend the {ref}/modules-discovery.html[cluster formation
module].

[float]
==== Core discovery plugins
Expand All @@ -12,22 +13,22 @@ The core discovery plugins are:
<<discovery-ec2,EC2 discovery>>::

The EC2 discovery plugin uses the https://github.com/aws/aws-sdk-java[AWS API]
for unicast discovery.
to identify the addresses of seed hosts.

<<discovery-azure-classic,Azure Classic discovery>>::

The Azure Classic discovery plugin uses the Azure Classic API for unicast
discovery.
The Azure Classic discovery plugin uses the Azure Classic API to identify the
addresses of seed hosts.

<<discovery-gce,GCE discovery>>::

The Google Compute Engine discovery plugin uses the GCE API for unicast
discovery.
The Google Compute Engine discovery plugin uses the GCE API to identify the
addresses of seed hosts.

[float]
==== Community contributed discovery plugins

A number of discovery plugins have been contributed by our community:
The following discovery plugins have been contributed by our community:

* https://github.com/fabric8io/elasticsearch-cloud-kubernetes[Kubernetes Discovery Plugin] (by Jimmi Dyson, http://fabric8.io[fabric8])

Expand Down
2 changes: 1 addition & 1 deletion docs/reference/migration/migrate_7_0/plugins.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ renamed to `requiresAnalysisSettings`
* This plugin has been removed since its functionality is now part of
Elasticsearch and requires no plugin. The location of the hosts file has moved
from `$ES_PATH_CONF/file-discovery/unicast_hosts.txt` to
`$ES_PATH_CONF/unicast_hosts.txt`. See <<file-based-hosts-provider, the
`$ES_PATH_CONF/unicast_hosts.txt`. See <<file-based-hosts-provider,the
file-based hosts provider documentation>> for further information.

[float]
Expand Down
137 changes: 77 additions & 60 deletions docs/reference/modules/discovery/discovery-settings.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,73 @@

Discovery and cluster formation are affected by the following settings:

`discovery.seed_hosts`::

Provides a list of master-eligible nodes in the cluster. The list contains
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would just remove the The list contains either an array of hosts or a comma-delimited string sentence. I think it causes more confusion and distracts from the main message, namely what this setting is about. Linking to an example usage of this setting with the array syntax (which is preferred) is probably more useful.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, sentence deleted and there is a link to the "important settings" page with a full example.

either an array of hosts or a comma-delimited string. Each value has the
format `host:port` or `host`, where `port` defaults to the setting
`transport.profiles.default.port`. Note that IPv6 hosts must be bracketed.
The default value is `127.0.0.1, [::1]`. See <<unicast.hosts>>.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

<<unicast.hosts>> -> should we rename that as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This renders correctly as See discovery.seed_hosts so it's only the name of the internal cross-ref that's referring to the old name. I'm not sure what extra hoops we need to jump through to rename these cross-refs so I've left them alone.

This setting was previously known as `discovery.zen.ping.unicast.hosts`.
Its old name is deprecated but continues to work in order to preserve
backwards compatibility. Support for the old name will be removed in a
future version.

`discovery.seed_providers`::
Specifies which types of <<built-in-hosts-providers,seed hosts provider>>
to use to obtain the addresses of the seed nodes used to start the
discovery process. By default, it is the
<<settings-based-hosts-provider,settings-based seed hosts provider>>.
This setting was previously known as `discovery.zen.hosts_provider`. Its
old name is deprecated but continues to work in order to preserve backwards
compatibility. Support for the old name will be removed in a future
version.

`discovery.cluster_formation_warning_timeout`::

Sets how long a node will try to form a cluster before logging a warning
that the cluster did not form. Defaults to `10s`. If a cluster has not
formed after `discovery.cluster_formation_warning_timeout` has elapsed then
the node will log a warning message that starts with the phrase `master not
discovered` which describes the current state of the discovery process.

`discovery.find_peers_interval`::

Sets how long a node will wait before attempting another discovery round.
Defaults to `1s`.

`discovery.probe.connect_timeout`::

Sets how long to wait when attempting to connect to each address. Defaults
to `3s`.

`discovery.probe.handshake_timeout`::

Sets how long to wait when attempting to identify the remote node via a
handshake. Defaults to `1s`.

`discovery.request_peers_timeout`::
Sets how long a node will wait after asking its peers again before
considering the request to have failed. Defaults to `3s`.

`discovery.seed_resolver.max_concurrent_resolvers`::

Specifies how many concurrent DNS lookups to perform when resolving the
addresses of seed nodes. Defaults to `10`. This setting was previously
known as `discovery.zen.ping.unicast.concurrent_connects`. Its old name is
deprecated but continues to work in order to preserve backwards
compatibility. Support for the old name will be removed in a future
version.

`discovery.seed_resolver.timeout`::

Specifies how long to wait for each DNS lookup performed when resolving the
addresses of seed nodes. Defaults to `5s`. This setting was previously
known as `discovery.zen.ping.unicast.hosts.resolve_timeout`. Its old name
is deprecated but continues to work in order to preserve backwards
compatibility. Support for the old name will be removed in a future
version.

`cluster.auto_shrink_voting_configuration`::

Controls whether the <<modules-discovery-voting,voting configuration>>
Expand All @@ -22,20 +89,20 @@ Discovery and cluster formation are affected by the following settings:

Sets how long each election is allowed to take before a node considers it to
have failed and schedules a retry. This defaults to `500ms`.

`cluster.election.initial_timeout`::

Sets the upper bound on how long a node will wait initially, or after the
elected master fails, before attempting its first election. This defaults
to `100ms`.


`cluster.election.max_timeout`::

Sets the maximum upper bound on how long a node will wait before attempting
an first election, so that an network partition that lasts for a long time
does not result in excessively sparse elections. This defaults to `10s`

[[fault-detection-settings]]`cluster.fault_detection.follower_check.interval`::

Sets how long the elected master waits between follower checks to each
Expand Down Expand Up @@ -74,7 +141,7 @@ Discovery and cluster formation are affected by the following settings:
state updates from lagging nodes. The default value is `90s`. If a node does
not successfully apply the cluster state update within this period of time,
it is considered to have failed and is removed from the cluster. See
<<cluster-state-publishing>>.
<<cluster-state-publishing>>.

`cluster.initial_master_nodes`::

Expand All @@ -88,51 +155,19 @@ Discovery and cluster formation are affected by the following settings:
Sets how long a node will wait after sending a request to join a cluster
before it considers the request to have failed and retries. Defaults to
`60s`.

`cluster.max_voting_config_exclusions`::

Sets a limit on the number of voting configuration exclusions at any one
time. The default value is `10`. See
<<modules-discovery-adding-removing-nodes>>.
`cluster.publish.timeout`::

`cluster.publish.timeout`::

Sets how long the master node waits for each cluster state update to be
completely published to all nodes. The default value is `30s`. If this
period of time elapses, the cluster state change is rejected. See
<<cluster-state-publishing>>.

`discovery.cluster_formation_warning_timeout`::

Sets how long a node will try to form a cluster before logging a warning
that the cluster did not form. Defaults to `10s`. If a cluster has not
formed after `discovery.cluster_formation_warning_timeout` has elapsed then
the node will log a warning message that starts with the phrase `master not
discovered` which describes the current state of the discovery process.

`discovery.find_peers_interval`::

Sets how long a node will wait before attempting another discovery round.
Defaults to `1s`.

`discovery.probe.connect_timeout`::

Sets how long to wait when attempting to connect to each address. Defaults
to `3s`.

`discovery.probe.handshake_timeout`::

Sets how long to wait when attempting to identify the remote node via a
handshake. Defaults to `1s`.

`discovery.request_peers_timeout`::
Sets how long a node will wait after asking its peers again before
considering the request to have failed. Defaults to `3s`.

`discovery.seed_providers`::
Specifies which types of <<built-in-hosts-providers,seed provider>> provide
the addresses of seed nodes. By default, it is the
<<settings-based-hosts-provider,settings-based hosts provider>>.
<<cluster-state-publishing>>.

[[no-master-block]]`cluster.no_master_block`::
Specifies which operations are rejected when there is no active master in a
Expand All @@ -153,29 +188,11 @@ cluster.
* The `cluster.no_master_block` setting doesn't apply to nodes-based APIs
(for example, cluster stats, node info, and node stats APIs). Requests to these
APIs are not be blocked and can run on any available node.

* For the cluster to be fully operational, it must have an active master.
===============================

WARNING: This setting replaces the `discovery.zen.no_master_block` setting in
earlier versions. The `discovery.zen.no_master_block` setting is ignored.

--

`discovery.seed_hosts`::

Provides a list of master-eligible nodes in the cluster. The list contains
either an array of hosts or a comma-delimited string. Each value has the
format `host:port` or `host`, where `port` defaults to the setting
`transport.profiles.default.port`. Note that IPv6 hosts must be bracketed.
The default value is `127.0.0.1, [::1]`. See <<unicast.hosts>>.

`discovery.seed_resolver.timeout`::

Sets the amount of time to wait for DNS lookups on each round of discovery.
This is specified as a <<time-units, time unit>> and defaults to `5s`.

`discovery.seed_resolver.max_concurrent_resolvers`::

Sets the number of threads with which to perform DNS lookups for seed nodes.
This defaults to `10`.
Loading