Make action retries configurable

[ersin-erdal]

Resolves: #146222

This PR makes maximum number of retries of an action configurable.

Follows the same pattern we used in alerting plugin.
xpack.actions.run.maxAttempts as a global settings and
xpack.actions.run.connectorTypeOverrides to override the global settings for specific connector types.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[doakalexi]

LGTM! Verified locally, and works as expected

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: eb1988f

Metrics [docs]

Unknown metric groups

ESLint disabled in files

id	before	after	diff
osquery	1	2	+1

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
fleet	61	67	+6
osquery	109	115	+6
securitySolution	439	445	+6
total			+20

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
fleet	70	76	+6
osquery	110	117	+7
securitySolution	515	521	+6
total			+21

History

• 💔 Build #96200 failed 89e43dd
• 💚 Build #96182 succeeded cb1970a
• 💔 Build #96093 failed 971c936

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[ymao1]

This says that maxAttempts indicates the number of retries, but when verifying it seems that it means the number of tries. When I set maxAttempts to 1, I only see the action run once and then not retried, when I would expect it to see it retried 1 time.

[ymao1]

I set maxAttempts: 1 and force an email error. After the first action run, I see this log:

[2022-12-21T13:22:25.464-05:00][ERROR][plugins.actions.email] Error: email action error
    at Object.executor (/Users/ying/Code/kibana/x-pack/plugins/stack_connectors/server/connector_types/stack/email/index.ts:230:9)
    at Object.wrapper [as executor] (/Users/ying/Code/kibana/node_modules/lodash/lodash.js:5255:19)
    at /Users/ying/Code/kibana/x-pack/plugins/actions/server/lib/action_executor.ts:154:38
[2022-12-21T13:22:25.464-05:00][WARN ][plugins.actions.email] action execution failure: .email:gmail: email: my gmail: an error occurred while running the action: email action error; retry: true

which says retry: true. Since maxAttempts is meant to be max retries, I would expect this to be retried once and the logs indicate it will be retried. However, the action task is deleted after this and the action is never retried.

[mikecote]

Good catch, @ymao1! We should fix our docs to mention "attempt" instead of "retries". Basically this value confirms how many times an action will run (attempt) before aborting it.

[mikecote]

Agreed that the getRetry / retry: true stuff can be confusing 🙈 +1 to a follow up if we didn't break it in this PR.

[ymao1]

Yea, I don't think this is caused in this PR. A followup issue is fine

[ersin-erdal]

Replaced retry with attempt :)
And checked the retry: true in the response, action executor returns it when there is an error other than a validation error. I think it means retryable not "the last execution was a retry".

[ymao1]

LGTM