Allow realtime get to read from translog

[ywelsch]

The realtime GET API currently has erratic performance in case where a document is accessed that has just been indexed but not refreshed yet, as the implementation will currently force an internal refresh in that case. Refreshing can be an expensive operation, and also will block the thread that executes the GET operation, blocking other GETs to be processed. In case of frequent access of recently indexed documents, this can lead to a refresh storm and terrible GET performance.

While older versions of Elasticsearch (2.x and older) did not trigger refreshes and instead opted to read from the translog in case of realtime GET API or update API, this was removed in 5.0 (#20102) to avoid inconsistencies between values that were returned from the translog and those returned by the index. This was partially reverted in 6.3 (#29264) to allow _update and upsert to read from the translog again as it was easier to guarantee consistency for these, and also brought back more predictable performance characteristics of this API. Calls to the realtime GET API, however, would still always do a refresh if necessary to return consistent results. This means that users that were calling realtime GET APIs to coordinate updates on client side (realtime GET + CAS for conditional index of updated doc) would still see very erratic performance.

This PR (together with #48707) resolves the inconsistencies between reading from translog and index. In particular it fixes the inconsistencies that happen when requesting stored fields, which were not available when reading from translog. In case where stored fields are requested, this PR will reparse the _source from the translog and derive the stored fields to be returned. With this, it changes the realtime GET API to allow reading from the translog again, avoid refresh storms and blocking the GET threadpool, and provide overall much better and predictable performance for this API.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Engine)

[original-brownbear]

LGTM (probably best to wait for one other reviewer though ... I understand this one just fine now but side-effects I'd probably miss :))

[jpountz]

Given the number of performance issues I've seen over the past years that were related to refreshes because of GETs, I agree with you that stopping to handle GET requests via the translog was probably a bad idea. We already acknowledged this when re-introducing reads from the translog for updates, I'm good with making all realtime reads use the translog again.

I tried to think about potential sources of inconsistencies and couldn't find any besides of the ones that you already covered.

[henningandersen]

Drive-by comment: I wonder if we should attempt to also improve the actual get from translog to always succeed, see:

https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/index/engine/InternalEngine.java#L670

I think that this PR together with the PRRL changes to no longer retain additional translog increases the risk of not being able to lookup the operation in translog.

Can certainly be done in a follow-up rather than here.

[ywelsch]

@elasticmachine run elasticsearch-ci/packaging-sample-matrix

I wonder if we should attempt to also improve the actual get from translog to always succeed

I thought a bit about this before pushing this change, and came to the conclusion that it will not matter in practice. Even with peer recovery retention leases now allowing quicker clean-up of translog, this is very unlikely to happen:

• Retention leases are synced every 30 seconds.
• Enough translog is kept around to allow restoring from safe commit (typically at least 10s of seconds old).

Triggering an internal refresh in the remaining edge cases does not sound too bad to me.

I can look into what it would take to never fall back to a refresh, but the added complexity might not be worth adding more code to deal with this.