[ftr] pass password for UI login + improve login/logout steps with pr…

…oper wait logic (elastic#166936)

## Summary

Hopefully

closes elastic#167104
closes elastic#167130
closes elastic#167100 
closes elastic#167013
closes elastic#166964

Fixing a few issues with login/logout:

1. Failed to login in "before" hook
<img width="1336" alt="Screenshot 2023-09-25 at 12 37 45"
src="https://github.com/elastic/kibana/assets/10977896/e3b2830e-7b0d-4467-9b90-261b385bf71e">

My theory is that we are loading `/login` route too soon while log out
was not completed yet.
When we navigate to `https://localhost:5620/logout` there are multiple
url re-directions with final page being Cloud login form. This PR makes
sure we wait for this form to be displayed + 2500 ms extra to avoid
"immediate" /login navigation

2. Failed login on MKI:
Updating login via UI for serverless to pass password valid for
deployment: currently FTR uses `changeme` for both Kibana CI & MKI.

3. ES activate user profile call returning 500
We saw some login failures that are preceded with the following logs:
```
[00:03:27]           │ debg Find.clickByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:03:27]           │ debg Find.findByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:03:27]           │ debg Find.waitForDeletedByCssSelector('.kibanaWelcomeLogo') with timeout=10000
[00:03:27]           │ proc [kibana] [2023-09-19T07:08:26.126+00:00][INFO ][plugins.security.routes] Logging in with provider "basic" (basic)
[00:03:27]           │ info [o.e.x.s.s.SecurityIndexManager] [ftr] security index does not exist, creating [.security-profile-8] with alias [.security-profile]
[00:03:27]           │ proc [kibana] [2023-09-19T07:08:26.140+00:00][ERROR][plugins.security.user-profile] Failed to activate user profile: {"error":{"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [1] shards, but this cluster currently has [27]/[27] maximum normal shards open;"}],"type":"validation_exception","reason":"Validation Failed: 1: this action would add [1] shards, but this cluster currently has [27]/[27] maximum normal shards open;"},"status":400}.
[00:03:27]           │ proc [kibana] [2023-09-19T07:08:26.140+00:00][ERROR][http] 500 Server Error
[00:03:27]           │ warn browser[SEVERE] http://localhost:5620/internal/security/login - Failed to load resource: the server responded with a status of 500 (Internal Server Error)
```

User activation happens during `POST internal/security/login` call to
Kibana server. ~~The only improvement that we can do from FTR
perspective is to call this end-point via API to makes sure user is
activated and only after proceed with UI login.~~
While working on issue #4 and talking to @jeramysoucy I believe retrying
login via UI will work here as well. We are checking if we are still on
login page (similar to incorrect password login), waiting 2500 ms and
pressing login button again.

4. Failed to login with Kibana reporting UNEXPECTED_SESSION_ERROR and
been re-directed to Cloud login page
```
proc [kibana] [2023-09-25T11:35:12.794+00:00][INFO ][plugins.security.authentication] Authentication attempt failed: UNEXPECTED_SESSION_ERROR
```

Temporary solution is to retry login from scratch (navigation to Kibana
login page & re-login )



Flaky-test-runner for functional obtl tests 50x
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3215


This PR is not fixing random 401 response when user navigates to some
apps with direct url

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

x-pack/test/functional/page_objects/security_page.ts

@@ -168,7 +168,7 @@ export class SecurityPageObject extends FtrService {
     );
   }
 
-  private async isLoginFormVisible() {
+  public async isLoginFormVisible() {
     return await this.testSubjects.exists('loginForm');
   }
 
@@ -323,7 +323,14 @@ export class SecurityPageObject extends FtrService {
         if (alert?.accept) {
           await alert.accept();
         }
-        return !(await this.browser.getCurrentUrl()).includes('/logout');
+
+        if (this.config.get('serverless')) {
+          // Logout might trigger multiple redirects, but in the end we expect the Cloud login page
+          this.log.debug('Wait 5 sec for Cloud login page to be displayed');
+          return await this.find.existsByDisplayedByCssSelector('.login-form-password', 5000);
+        } else {
+          return !(await this.browser.getCurrentUrl()).includes('/logout');
+        }
       });
     }
   }

x-pack/test_serverless/functional/page_objects/svl_common_page.ts

@@ -10,16 +10,90 @@ import { FtrProviderContext } from '../ftr_provider_context';
 export function SvlCommonPageProvider({ getService, getPageObjects }: FtrProviderContext) {
   const testSubjects = getService('testSubjects');
   const config = getService('config');
-  const pageObjects = getPageObjects(['security']);
+  const pageObjects = getPageObjects(['security', 'common']);
+  const retry = getService('retry');
+  const deployment = getService('deployment');
+  const log = getService('log');
+  const browser = getService('browser');
+
+  const delay = (ms: number) =>
+    new Promise((resolve) => {
+      setTimeout(resolve, ms);
+    });
 
   return {
+    async navigateToLoginForm() {
+      const url = deployment.getHostPort() + '/login';
+      await browser.get(url);
+      // ensure welcome screen won't be shown. This is relevant for environments which don't allow
+      // to use the yml setting, e.g. cloud
+      await browser.setLocalStorageItem('home:welcome:show', 'false');
+
+      log.debug('Waiting for Login Form to appear.');
+      await retry.waitForWithTimeout('login form', 10_000, async () => {
+        return await pageObjects.security.isLoginFormVisible();
+      });
+    },
+
     async login() {
       await pageObjects.security.forceLogout({ waitForLoginPage: false });
-      return await pageObjects.security.login(config.get('servers.kibana.username'));
+
+      // adding sleep to settle down logout
+      await pageObjects.common.sleep(2500);
+
+      await retry.waitForWithTimeout(
+        'Waiting for successful authentication',
+        90_000,
+        async () => {
+          if (!(await testSubjects.exists('loginUsername', { timeout: 1000 }))) {
+            await this.navigateToLoginForm();
+
+            await testSubjects.setValue('loginUsername', config.get('servers.kibana.username'));
+            await testSubjects.setValue('loginPassword', config.get('servers.kibana.password'));
+            await testSubjects.click('loginSubmit');
+          }
+
+          if (await testSubjects.exists('userMenuButton', { timeout: 10_000 })) {
+            log.debug('userMenuButton is found, logged in passed');
+            return true;
+          } else {
+            throw new Error(`Failed to login to Kibana via UI`);
+          }
+        },
+        async () => {
+          // Sometimes authentication fails and user is redirected to Cloud login page
+          // [plugins.security.authentication] Authentication attempt failed: UNEXPECTED_SESSION_ERROR
+          const currentUrl = await browser.getCurrentUrl();
+          if (currentUrl.startsWith('https://cloud.elastic.co')) {
+            log.debug(
+              'Probably authentication attempt failed, we are at Cloud login page. Retrying from scratch'
+            );
+          } else {
+            const authError = await testSubjects.exists('promptPage', { timeout: 2500 });
+            if (authError) {
+              log.debug('Probably SAML callback page, doing logout again');
+              await pageObjects.security.forceLogout({ waitForLoginPage: false });
+            } else {
+              const isOnLoginPage = await testSubjects.exists('loginUsername', { timeout: 1000 });
+              if (isOnLoginPage) {
+                log.debug(
+                  'Probably ES user profile activation failed, waiting 2 seconds and pressing Login button again'
+                );
+                await delay(2000);
+                await testSubjects.click('loginSubmit');
+              } else {
+                log.debug('New behaviour, trying to navigate and login again');
+              }
+            }
+          }
+        }
+      );
+      log.debug('Logged in successfully');
     },
 
     async forceLogout() {
       await pageObjects.security.forceLogout({ waitForLoginPage: false });
+      log.debug('Logged out successfully');
     },
 
     async assertProjectHeaderExists() {