Search - enable low_level_cancellation by default.

[markharwood]

Benchmarking on worst-case queries (max agg on match_all or popular-term query with large index) was not noticeably slower.

Closes #26258

[elasticmachine]

Pinging @elastic/es-search

[markharwood]

Benchmarks were conducted using Rally with the following settings:

• nyc_taxis dataset used as a larger-sized index
• dedicated test hardware was used and reset between each run using
  • cpufreq-set to ensure clock rate wasn't varying
  • usual rally . best-practice for clearing any OS-level caches
  • elasticsearch server restarted
• Index was pre-built and benchmarks were tested on the same physical index (ensuring same segments etc)
• Used worst-case scenario queries to stress-test the loops containing low_level_cancellation (LLC) checks (a max agg on the trip_distance field for both match_all queries and match on high-popularity term passenger_count:1). Each of the queries were expensive to run over so much data so we had a warm-up of 20 iterations followed by timed loop for 100 iterations

Despite these settings, the expected slow-down in LLC:on versus LLC:off failed to show up in test results. If anything the LLC:on results were observed to be faster (?!).

It should be noted that even within runs using the same settings there were variable result timings which suggests some environmental issues running repeated tests. Despite this, the trend seems to show that LLC:on is faster than LLC:off which is counter-intuitive.

Conclusion

Ideally we would have clearly demonstrated our expectations that LLC:on added a small but acceptable overhead to search performance. Instead we seem to see a variety of timings which if anything suggest LLC:on makes things faster. This is a little baffling but my assumption is that we should enable it by default and keep an eye on nightly benchmarks.

[markharwood]

test this please

[jpountz]

The diff is simple, but this required lots of testing. Thanks @markharwood for looking into this!

[danielmitterdorfer]

Thanks for all the testing! LGTM