Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Painless compilation threshold breach with correct error status #1066

Merged

Conversation

shiv0408
Copy link
Member

@shiv0408 shiv0408 commented Aug 9, 2021

Description

When a circuit breaker occurs due to compilation threshold breach, then OpenSearchRejectedExecutionException is called.

Issues Resolved

Resolves #1064

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@shiv0408 shiv0408 marked this pull request as draft August 9, 2021 13:56
@shiv0408 shiv0408 marked this pull request as ready for review August 9, 2021 13:56
@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Wrapper Validation success f8ceccf

@opensearch-ci-bot
Copy link
Collaborator

✅   DCO Check Passed f8ceccf

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Precommit success f8ceccf

Comment on lines 134 to 136
} else if (cause instanceof CircuitBreakingException) {
throw new OpenSearchRejectedExecutionException(
"Failed to compile " + type + " script [" + id + "] using lang [" + lang + "]", cause);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CircuitBreakingException already returns 429?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CircuitBreakingException was wrapped in GeneralScriptException which return 500 error. You can have a look at reproduced error.

Copy link
Collaborator

@Bukhtawar Bukhtawar Aug 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally the status is determined by the underlying cause https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/OpenSearchException.java#L253-L260. . In this case since CircuitBreakingException is the underlying cause, the error code should have been 429

Can you please confirm if the below is not being invoked.

public static RestStatus status(Throwable t) {
        if (t != null) {
            if (t instanceof OpenSearchException) {
                return ((OpenSearchException) t).status();
            } else if (t instanceof IllegalArgumentException) {
                return RestStatus.BAD_REQUEST;
            } else if (t instanceof JsonParseException) {
                return RestStatus.BAD_REQUEST;
            } else if (t instanceof OpenSearchRejectedExecutionException) {
                return RestStatus.TOO_MANY_REQUESTS;
            } 
        }
        return RestStatus.INTERNAL_SERVER_ERROR;
    }

Also please update the steps to reproduce

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Wrapper Validation success f1973da

@opensearch-ci-bot
Copy link
Collaborator

✅   DCO Check Passed f1973da

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Precommit success f1973da

Copy link
Collaborator

@Bukhtawar Bukhtawar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@Bukhtawar
Copy link
Collaborator

Pinging @dblock @adnapibar for help with approval and merge

@dblock
Copy link
Member

dblock commented Aug 10, 2021

start gradle check

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success f1973da
Log 379

Reports 379

@dblock
Copy link
Member

dblock commented Aug 10, 2021

Whatever the external behavior change here, it's not captured. I think this needs a test.

@shiv0408
Copy link
Member Author

Updated the linked issue with better reproduction steps.

@shiv0408
Copy link
Member Author

Earlier when CircuitBreakingException was thrown if compilation limit exceeded, then new GeneralScriptException was thrown with CircuitBreakingException as cause. But then status of call was return 500. Here, I made GeneralScriptException an implementation of OpenSearchWrapperException and now status of GeneralScriptException is same as that of CircuitBreakingException because it is now able to unwrap the exception and find root cause.

@dblock do you want me to create a unit test to check this behaviour change?

@dblock
Copy link
Member

dblock commented Aug 12, 2021

@dblock do you want me to create a unit test to check this behaviour change?

Yes, a test that shows that the response is now a 429, instead of a 500.

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Wrapper Validation success bfce557

@opensearch-ci-bot
Copy link
Collaborator

✅   DCO Check Passed bfce557

@shiv0408
Copy link
Member Author

@Bukhtawar @dblock created a test to check the status. Please review the changes. Any feedback is welcome 😄 .

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Precommit failure bfce557
Log 924

@dblock
Copy link
Member

dblock commented Aug 13, 2021

@Bukhtawar @dblock created a test to check the status. Please review the changes. Any feedback is welcome 😄 .

The unit test is great, it happens at the lowest compilation level, keep it!

Because the original issue and the repro is at REST level, try also adding a REST test? One that makes an HTTP request, and gets back a 429 whereas before this fix it would cause a 500?

Check the pre-commit failure, too.

@shiv0408
Copy link
Member Author

Because the original issue and the repro is at REST level, try also adding a REST test? One that makes an HTTP request, and gets back a 429 whereas before this fix it would cause a 500?

I manually tested code by running the commands. Is the there any example in the codebase to run automated REST test?

@opensearch-ci-bot
Copy link
Collaborator

✅   DCO Check Passed 9376e28

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Wrapper Validation success 9376e28

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Precommit success 9376e28

@shiv0408 shiv0408 changed the title Painless compilation threshold breach throws OpenSearchRejectedExecutionException Painless compilation threshold breach with correct error status Aug 16, 2021
@adnapibar
Copy link
Contributor

start gradle check

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success 9376e28
Log 416

Reports 416

@shiv0408
Copy link
Member Author

@dblock Can you please go through the integTest I created?

@opensearch-ci-bot
Copy link
Collaborator

✅   DCO Check Passed e7732078552e92d984c63c8d39e8a77b87af1788

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Wrapper Validation success e7732078552e92d984c63c8d39e8a77b87af1788

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Precommit failure e7732078552e92d984c63c8d39e8a77b87af1788
Log 1146

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Wrapper Validation success fe8414e8778f8c1b8fb98ad0710af6a2fd9f563a

@opensearch-ci-bot
Copy link
Collaborator

✅   DCO Check Passed fe8414e8778f8c1b8fb98ad0710af6a2fd9f563a

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Precommit success fe8414e8778f8c1b8fb98ad0710af6a2fd9f563a

@dblock
Copy link
Member

dblock commented Sep 16, 2021

start gradle check

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure fe8414e8778f8c1b8fb98ad0710af6a2fd9f563a
Log 534

Reports 534

@shiv0408
Copy link
Member Author

Opened a new issue for unrelated test failures. @dblock Can please start gradle check again?

@dblock
Copy link
Member

dblock commented Sep 28, 2021

start gradle check

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure fe8414e8778f8c1b8fb98ad0710af6a2fd9f563a
Log 580

Reports 580

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Wrapper Validation success eae8a31348323f0197fd536ec293ef7139a98470

@opensearch-ci-bot
Copy link
Collaborator

❌   DCO Check Failed eae8a31348323f0197fd536ec293ef7139a98470
Run ./dev-tools/signoff-check.sh remotes/origin/main eae8a31348323f0197fd536ec293ef7139a98470 to check locally
Use git commit with -s to add 'Signed-of-by: {EMAIL}' on impacted commits

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Precommit success eae8a31348323f0197fd536ec293ef7139a98470

…xception

Signed-off-by: Shivansh Arora <hishiv@amazon.com>
@dblock
Copy link
Member

dblock commented Sep 28, 2021

start gradle check

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Wrapper Validation success aef5077

@opensearch-ci-bot
Copy link
Collaborator

✅   DCO Check Passed aef5077

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Precommit success aef5077

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success aef5077
Log 582

Reports 582

@dblock dblock merged commit 416220f into opensearch-project:main Sep 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Wrong error code for Painless scripts compilation thresholds
5 participants