Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scripting: Cache script results if deterministic #50106

Merged

Conversation

stu-elastic
Copy link
Contributor

Cache results from queries that use scripts if they use only
deterministic API calls. Nondeterministic API calls are marked in the
whitelist with the @nondeterministic annotation. Examples are
Math.random() and new Date().

Refs: #49466

@stu-elastic stu-elastic added the :Core/Infra/Scripting Scripting abstractions, Painless, and Mustache label Dec 11, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (:Core/Infra/Scripting)

Copy link
Contributor

@jdconrad jdconrad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM once we address the couple of comments I left that require further discussion.

Copy link
Member

@rjernst rjernst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Left a few suggestions.

@stu-elastic stu-elastic removed the WIP label Dec 12, 2019
@stu-elastic
Copy link
Contributor Author

Currently debugging NPE when using scripts for weighted avg, fieldName coming in as null.

@stu-elastic
Copy link
Contributor Author

Ok, now that scripts are cachable, here's what's causing the NPE

SearchService.loadOrExecuteQueryPhase

calls into

IndicesService.loadIntoContext (instead of queryPhase.execute)

which calls

ShardSearchRequest.cacheKey

which eventually calls

MultiValuesSourceFieldConfig.writeTo

which nulls out because there is no fieldName.

Here's the series of calls to reproduce:

curl -H 'Content-Type: application/json' -XPUT --user 'elastic-admin:elastic-password' localhost:9200/exams -d '{ "mappings": { "properties": { "grade": { "type": "byte" } } } }'

curl -H 'Content-Type: application/json' -XPOST --user 'elastic-admin:elastic-password' localhost:9200/exams/_doc -d '{"grade": 100, "weight": 2}'

curl -H 'Content-Type: application/json' -XPOST --user 'elastic-admin:elastic-password' localhost:9200/exams/_doc -d '{"grade": 50, "weight": 3}'

curl -H 'Content-Type: application/json' -XPOST --user 'elastic-admin:elastic-password' localhost:9200/exams/_search -d '{
    "size": 0,
    "aggs" : {
        "weighted_grade": {
            "weighted_avg": {
                "value": {
                    "script": "doc.grade.value + 1"
                },
                "weight": {
                    "script": "doc.weight.value + 1"
                }
            }
        }
    }
}'

@stu-elastic
Copy link
Contributor Author

This just happened to be caught by a test. Are there other places where cacheKey calls writeTo with fields that are not available if a script is used?

@stu-elastic
Copy link
Contributor Author

@elasticmachine run elasticsearch-ci/1

@jdconrad
Copy link
Contributor

@stu-elastic Thanks again for all the work getting this done. The tests look especially good.

@stu-elastic stu-elastic merged commit 7794625 into elastic:master Dec 18, 2019
stu-elastic added a commit to stu-elastic/elasticsearch that referenced this pull request Dec 18, 2019
Cache results from queries that use scripts if they use only
deterministic API calls.  Nondeterministic API calls are marked in the
whitelist with the `@nondeterministic` annotation.  Examples are
`Math.random()` and `new Date()`.

Refs: elastic#49466
stu-elastic added a commit to stu-elastic/elasticsearch that referenced this pull request Dec 18, 2019
stu-elastic added a commit that referenced this pull request Dec 18, 2019
Cache results from queries that use scripts if they use only
deterministic API calls.  Nondeterministic API calls are marked in the
whitelist with the `@nondeterministic` annotation.  Examples are
`Math.random()` and `new Date()`.

Refs: #49466
SivagurunathanV pushed a commit to SivagurunathanV/elasticsearch that referenced this pull request Jan 23, 2020
Cache results from queries that use scripts if they use only
deterministic API calls.  Nondeterministic API calls are marked in the
whitelist with the `@nondeterministic` annotation.  Examples are
`Math.random()` and `new Date()`.

Refs: elastic#49466
SivagurunathanV pushed a commit to SivagurunathanV/elasticsearch that referenced this pull request Jan 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Core/Infra/Scripting Scripting abstractions, Painless, and Mustache >enhancement v7.6.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants