Introduce cluster level dynamic default and minimum refresh interval setting

[ashking94]

Is your feature request related to a problem? Please describe.
This proposes to introduce 2 dynamic NodeScope settings i.e. cluster default refresh interval setting and cluster level minimum refresh interval setting. Currently the refresh interval for an index is read from an index scoped settings i.e. index.refresh_interval. The default value taken is 1s which is read from the hardcoded default value in the above setting.

Code reference -

OpenSearch/server/src/main/java/org/opensearch/index/IndexSettings.java

Lines 301 to 308 in 5c283e0

	public static final TimeValue DEFAULT_REFRESH_INTERVAL = new TimeValue(1, TimeUnit.SECONDS);
	public static final Setting<TimeValue> INDEX_REFRESH_INTERVAL_SETTING = Setting.timeSetting(
	"index.refresh_interval",
	DEFAULT_REFRESH_INTERVAL,
	new TimeValue(-1, TimeUnit.MILLISECONDS),
	Property.Dynamic,
	Property.IndexScope
	);

Currently, if the customer wishes to override the index.refresh_interval setting, it also leads to Search Idle feature not being honoured due to explicit refresh interval check.

New settings -

cluster.default.index.refresh_interval

This setting is used to set a default refresh index for all indexes created on the cluster unless the index.refresh_interval is present during index creation. If an index is stripped off the index.refresh_interval, then the index would use the cluster default index refresh interval. If an index is using the cluster default setting, then search idle feature continue to get honoured.

cluster.minimum.index.refresh_interval

This setting is used to set a minimum refresh index which is used during index creation & update validation. If the specified index.refresh_interval value is less than the cluster minimum index refresh interval, then the creation/updation operation fails.

The cluster.default.index.refresh_interval and cluster.minimum.index.refresh_interval are dynamic with Node scope property. The default is always equal or greater than the minimum value.

Expected behaviour

• Index creation without refresh interval index setting uses cluster default.
• Stripping an index of the index.refresh_interval index setting leads to the usage of the cluster default.
• Shard idle feature will work for all use cases except where the index setting index.refresh_interval is passed during index creation or index update operation.
• When user is providing index setting index.refresh_interval during index creation or update, the provided value is validated against the cluster minimum settings.
• If the index setting index.refresh_interval is set to null in update index settings call, the current latest cluster default settings will be used for setting the refresh interval.
• If the cluster default changes, then all indexes using cluster default settings will have the refresh interval updated with shard Idle working.

Evaluation for different modes

• Single mode:
  • Consider cases where a user is operating cluster in single replication type mode, in such cases the user has option to set a cluster default that is different than 1s and still achieve Shard Idle feature. This is not possible currently as on explicit refresh setting, the refresh happens periodically even though there are no searches happening on a cluster.
  • For Remote Store feature, there is an option to restrict the cluster with only remote store enabled indexes. Honouring the current default 1s refresh interval is bit tricky as the entire round of replication involves segments upload to remote store and download of segments from remote store onto replicas. Having a cluster default will allow the indexes created without explicit refresh interval to use the cluster default. User also does not need to mention the index setting index.refresh_interval. Having a cluster minimum will disallow the creation of index with an unreasonable (or unachievable) index refresh interval.
• Multi mode:
  • In the current state, it is possible to have a cluster where both docrep and segrep indexes exists. In such a cluster, the cluster default and minimum needs to be most lenient to allow the user to pass in index setting index.refresh_interval that fits the purpose.
• Most importantly, this setting on default, allows backward compatibility for background async refreshes.

Describe the solution you'd like
Introduce the 2 settings mentioned above.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.