[Response Ops][Task Manager] Emitting error metric when task update fails

[ymao1]

Resolves #184173

Summary

Catches errors updating the task from the taskStore.bulkUpdate function and emitting an error count so these errors are reflected in the metrics.

To Verify

1. Add the following to force an error when running an example rule:

--- a/x-pack/plugins/task_manager/server/task_store.ts
+++ b/x-pack/plugins/task_manager/server/task_store.ts
@@ -24,6 +24,7 @@ import {
   ISavedObjectsRepository,
   SavedObjectsUpdateResponse,
   ElasticsearchClient,
+  SavedObjectsErrorHelpers,
 } from '@kbn/core/server';

 import { RequestTimeoutsConfig } from './config';
@@ -309,6 +310,16 @@ export class TaskStore {
       this.logger.warn(`Skipping validation for bulk update because excludeLargeFields=true.`);
     }

+    const isProcessResult = docs.some(
+      (doc) =>
+        doc.taskType === 'alerting:example.always-firing' &&
+        doc.status === 'idle' &&
+        doc.retryAt === null
+    );
+    if (isProcessResult) {
+      throw SavedObjectsErrorHelpers.decorateEsUnavailableError(new Error('test'));
+    }
+
     const attributesByDocId = docs.reduce((attrsById, doc) => {

2. Create an example.always-firing rule and let it run. You should see an error in the logs:

[2024-08-26T14:44:07.065-04:00][ERROR][plugins.taskManager] Task alerting:example.always-firing "80b8481d-7bfc-4d38-a31b-7a559fbe846b" failed: Error: test

3. Navigate to https://localhost:5601/api/task_manager/metrics?reset=false and you should see a framework error underneath the overall metrics and the alerting metrics.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[ymao1]

@elasticmachine merge upstream

[js-jankisalvi]

Verified locally, works as expected 👍

[mikecote]

Changes LGTM! Tested locally and works as expected

[ymao1]

@elasticmachine merge upstream

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 8f53c5f

Failed CI Steps

Metrics [docs]

✅ unchanged

History

• 💛 Build #229939 was flaky aa5233b
• 💛 Build #229905 was flaky 6e2adcc

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ymao1