Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Default max concurrent search req. numNodes * 5 #31171

Merged
merged 4 commits into from
Jun 8, 2018
Merged

Default max concurrent search req. numNodes * 5 #31171

merged 4 commits into from
Jun 8, 2018

Conversation

s1monw
Copy link
Contributor

@s1monw s1monw commented Jun 7, 2018 •
edited
Loading

We moved to 1 shard by default which caused some issues in how many
concurrent shard requests we allow by default. For instance searching
a 5 shard index on a single node will now be executed serially per shard
while we want these cases to have a good concurrency out of the box. This
change moves to numNodes * 5 which corresponds to the default we used to
have in the previous version.

Relates to #30783
Closes #30994

@s1monw
Default max concurrent search req. to the avg shards size per index
1c47301 
We moved to 1 shard by default which caused some issues in how many
concurrent shard requests we allow by default. For instance searching
a 5 shard index on a single node will now be executed serially per shard
while we want these cases to have a good concurrency out of the box. This
change moves to defaults based on the avg. number of shards per index in
the current search request to provide a good out of the box concurrency.

Relates to elastic#30783
Closes elastic#30994
@s1monw s1monw added >enhancement :Search/Search Search-related issues that do not fall into other categories v7.0.0 labels Jun 7, 2018
@s1monw s1monw requested a review from jasontedor June 7, 2018 13:51
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search-aggs

@s1monw s1monw requested a review from jpountz June 7, 2018 13:53
jasontedor
jasontedor approved these changes Jun 7, 2018
View reviewed changes
Copy link
Member

@jasontedor jasontedor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

jpountz
jpountz requested changes Jun 7, 2018
View reviewed changes
Copy link
Contributor

@jpountz jpountz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit unhappy that it breaks the rule that searching N indices with 1 shard performs the same as searching 1 index with N shards. Since the right value for this parameter doesn't really have anything to do with the number of shards per index, I'd rather like the limit to be the node count times some constant, possibly 5 to avoid changing too much the runtime behavior compared to previous versions.

@s1monw
Copy link
Contributor Author

s1monw commented Jun 7, 2018

I'm a bit unhappy that it breaks the rule that searching N indices with 1 shard performs the same as searching 1 index with N shards.

this is a very good reason I agree we shouldn't do it. I will open a PR to move it to 5 to not change any thing compared to 6.x and we can discuss if we wanna change it after that.

@s1monw s1monw changed the title Default max concurrent search req. to the avg shards size per index Default max concurrent search req. numNodes * 5 Jun 7, 2018
jpountz
jpountz approved these changes Jun 7, 2018
View reviewed changes
Copy link
Contributor

@jpountz jpountz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @s1monw.

@s1monw
Copy link
Contributor Author

s1monw commented Jun 7, 2018

@jasontedor FYI

jasontedor
jasontedor requested changes Jun 7, 2018
View reviewed changes
Copy link
Member

@jasontedor jasontedor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels odd to me. We are using five as the multiplicative factor because we used five in the past because we used five as the default number of shards because a long time ago someone decided that five a good number of default shards. It feels weird to be stuck to that number now.

@jpountz
Copy link
Contributor

jpountz commented Jun 7, 2018

I like num_nodes*5 because it means that we only start trading latency if you query more than 5 shards per node on average. It feels like a good default number to me, which I see as completely independent of the number of shards per index. Any value in the [4,8] range sounds like a good default to me. I suggested 5 because it minimizes the change in runtime behavior compared to previous versions but I don't see keeping the exact same value as a requirement.

@s1monw
Copy link
Contributor Author

s1monw commented Jun 8, 2018

While I do agree with @jpountz I still think it has issues. For instance it doesn't take into account the nodes in a remote cluster which can cause issues in CCS. I think we should look into finding a better default but I see this particular PR rather as rolling back a regression and get the benchmarks back to where they used to be. I suggest the following:

  • lets merge this PR as it its. I will do that in a bit
  • I will open a new issue and make it a blocker for 6.4 and 7.0 since we also should fix this in the 6.x series if we find a better default.

@s1monw s1monw mentioned this pull request Jun 8, 2018
Closed
@s1monw s1monw merged commit 435a825 into elastic:master Jun 8, 2018
@s1monw
Copy link
Contributor Author

s1monw commented Jun 8, 2018

@jpountz @jasontedor see #31192 I detached the regression from finding a better default which is crucial.

@jasontedor
Copy link
Member

That works for me, thanks @s1monw.

dnhatn added a commit that referenced this pull request Jun 10, 2018
@dnhatn
Merge branch 'master' into ccr
afe02a3 
* master:
  Move default location of dependencies report (#31228)
  Remove dependencies report task dependencies (#31227)
  Add recognition of MPL 2.0 (#31226)
  Fix unknown licenses (#31223)
  Remove version from license file name for GCS SDK (#31221)
  Fully encapsulate LocalCheckpointTracker inside of the engine (#31213)
  [DOCS] Added 'fail_on_unsupported_field' param to MLT. Closes #28008 (#31160)
  Add licenses for transport-nio (#31218)
  Remove DocumentFieldMappers#simpleMatchToFullName. (#31041)
  Allow to trim all ops above a certain seq# with a term lower than X, post backport fix (#31211)
  Compliant SAML Response destination check (#31175)
  Remove DocumentFieldMappers#smartNameFieldMapper, as it is no longer needed. (#31018)
  Remove extraneous references to 'tokenized' in the mapper code. (#31010)
  Allow to trim all ops above a certain seq# with a term lower than X (#30176)
  SQL: Make a single JDBC driver jar (#31012)
  Enhance license detection for various licenses (#31198)
  [DOCS] Add note about long-lived idle connections (#30990)
  Move number of language analyzers to analysis-common module (#31143)
  Default max concurrent search req. numNodes * 5 (#31171)
  flush job to ensure all results have been written (#31187)
jasontedor added a commit to rjernst/elasticsearch that referenced this pull request Jun 10, 2018
@jasontedor
Merge remote-tracking branch 'elastic/master' into isolate_version_ch…
e332600 
…ecker

* elastic/master: (309 commits)
  [test] add fix for rare virtualbox error (elastic#31212)
  Move default location of dependencies report (elastic#31228)
  Remove dependencies report task dependencies (elastic#31227)
  Add recognition of MPL 2.0 (elastic#31226)
  Fix unknown licenses (elastic#31223)
  Remove version from license file name for GCS SDK (elastic#31221)
  Fully encapsulate LocalCheckpointTracker inside of the engine (elastic#31213)
  [DOCS] Added 'fail_on_unsupported_field' param to MLT. Closes elastic#28008 (elastic#31160)
  Add licenses for transport-nio (elastic#31218)
  Remove DocumentFieldMappers#simpleMatchToFullName. (elastic#31041)
  Allow to trim all ops above a certain seq# with a term lower than X, post backport fix (elastic#31211)
  Compliant SAML Response destination check (elastic#31175)
  Remove DocumentFieldMappers#smartNameFieldMapper, as it is no longer needed. (elastic#31018)
  Remove extraneous references to 'tokenized' in the mapper code. (elastic#31010)
  Allow to trim all ops above a certain seq# with a term lower than X (elastic#30176)
  SQL: Make a single JDBC driver jar (elastic#31012)
  Enhance license detection for various licenses (elastic#31198)
  [DOCS] Add note about long-lived idle connections (elastic#30990)
  Move number of language analyzers to analysis-common module (elastic#31143)
  Default max concurrent search req. numNodes * 5 (elastic#31171)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jasontedor jasontedor jasontedor requested changes

@jpountz jpountz jpountz approved these changes

Assignees
No one assigned
Labels
>enhancement >regression :Search/Search Search-related issues that do not fall into other categories v7.0.0-beta1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@s1monw @elasticmachine @jpountz @jasontedor @colings86