[FTR][Serverless] Err handling needed in search_source_alert.ts

[wayneseymour]

After investigating failures in our serverless ftr test runs (real mki), it seems prudent to add some error handling and perhaps some retry logic around some of the calls within:

x-pack/test_serverless/functional/test_suites/common/discover_ml_uptime/discover/search_source_alert.ts
&&
x-pack/test/functional_with_es_ssl/apps/discover_ml_uptime/discover/search_source_alert.ts
...duplicates

One of the failed calls is createSourceIndex, with an err msg of:

Error message
TimeoutError: Request timed out
    at SniffingTransport._request (node_modules/@elastic/transport/src/Transport.ts:630:36)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at /root/.qaf/data/git/kibana/node_modules/@elastic/transport/src/Transport.ts:704:22
    at SniffingTransport.request (node_modules/@elastic/transport/src/Transport.ts:701:14)
    at Client.IndexApi [as index] (node_modules/@elastic/elasticsearch/src/api/api/index.ts:83:10)
    at Context. (search_source_alert.ts:374:7)
    at Object.apply (wrap_function.js:74:16) {
  options: { redaction: { type: 'replace', additionalKeys: [] } },
  meta: {
    body: undefined,
    statusCode: 0,
    headers: {},
    meta: {
      context: null,
      request: [Object],
      name: 'elasticsearch-js',
      connection: [Object],
      attempts: 0,
      aborted: false
    },
    warnings: null
  }
}

Although it is nice that the info is redacted, it doesn't tell me much as a maintainer and not the author of this.

One option is to add this to a re-useable service.
Even if we don't move it to a service, a try / catch with an err msg such as the following would be more helpful for debugging

log.error(`Failed to create source index ${SOURCE_DATA_VIEW}, TimeoutError: Request timed out`)

\cc @davismcphee

Continuation

This morning at midnight we've has more failures in this file: x-pack/test_serverless/functional/test_suites/common/discover_ml_uptime/discover/search_source_alert.ts

await checkUpdatedDataViewState('search-s*');

Err Msg

Error: expected 'search-source-alert' to equal 'search-s*'
    at Assertion.assert (expect.js:100:11)
    at Assertion.equal (expect.js:227:8)
    at checkUpdatedDataViewState (search_source_alert.ts:365:54)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Context. (search_source_alert.ts:553:7)
    at Object.apply (wrap_function.js:74:16)

Screen Shot

await testSubjects.click('confirmModalConfirmButton');

Err Msg

Error: retry.try reached timeout 120000 ms
TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="confirmModalConfirmButton"])
Wait timed out after 10017ms
    at /root/.qaf/data/git/kibana/node_modules/selenium-webdriver/lib/webdriver.js:915:22
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at onFailure (retry_for_success.ts:18:9)
    at retryForSuccess (retry_for_success.ts:85:7)
    at RetryService.try (retry.ts:52:12)
    at Proxy.clickByCssSelector (find.ts:420:5)
    at TestSubjects.click (test_subjects.ts:169:5)
    at Context. (search_source_alert.ts:541:7)
    at Object.apply (wrap_function.js:74:16)

Screen Shot

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[wayneseymour]

Based on a real mki run failure from early this morning.
I recalled this test having problems and remembered this issue.

After looking around, I found some related items:
#190519, fixed by #190090

• In this case, the fix was applied to the test file itself, while this issue originally pointed out the service that is used, is most likely the culprit