[Endpoint] Unifying the test index name for resolver and alerts #59073

jonathan-buttner · 2020-03-02T19:44:24Z

This PR unifies the alerts and events index name. These names need to be the same for retrieving events and alerts since they will be stored in the same index. The name I used is more inline with what the package will have: elastic/package-registry#223

After this change goes in, you'll want to use events-endpoint-1 to store the alerts and events for testing.

elasticmachine · 2020-03-02T19:45:07Z

Pinging @elastic/endpoint-app-team (Feature:Endpoint)

oatkiller · 2020-03-02T19:48:44Z

x-pack/plugins/endpoint/common/types.ts

  static ENDPOINT_INDEX_NAME = 'endpoint-agent*';
-  static EVENT_INDEX_NAME = 'endpoint-events-*';
+  static ALERT_INDEX_NAME = 'events-endpoint-1';


so alerts always go to '-1' and events can go to anywhere starting with 'events-endpoint-'?

I have the same question... where do the other events go? Why "-1"?

No, ultimately the alerts will go to events-endpoint-<some rollover number> same as other events. I had to hard code -1 because I ran into an issue getting this working with the alert details API.

@madirey @peluja1012 @marshallmain
The alert details API use the elasticsearch's get API which requires a specific index: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-get.html the problem with that is I don't think the alert list api returns the specific index that a document is in with the response and the event index will roll over based on the ILM policy. So it'll change from events-endpoint-000001 to events-endpoint-000002 etc. I tried having the alert details request take a pattern like events-endpoint-* but that causes a 500 from elasticsearch because the get API needs a single index.

I think we need to encode the index of the document in the _id or something so when the frontend passes it back, the backend can determine the specific index since we won't know that ahead of time.

Another example:

The index is returned by ES for each hit when searching... we can return that in the response to the FE and/or cache it... will need some logic to look it up if it's not available, or we could just change it to use the search API...

Is this a case where we should use aliases for the alert indexes to unify them into a single index endpoint?

@achuguy hmm I don't think we could use an alias to abstract the search/get api. The endpoint will use an alias for all events in general so it doesn't know the specific rollover index either. We could use a pipeline to break them out but we already made the decision to write the alerts and events to a single index.

@jonathan-buttner Yeah, that sounds like a good approach...

@jonathan-buttner If the endpoint is using an alias to write to, can you use a get with the same alias or are you specifically trying to isolate you get to a single index?

@achuguy so I believe you can have an alias that points to multiple indices but it doesn't look like you can do a GET on an alias with multiple indices:

You can execute searches I believe though but I think we should avoid doing a search since we have all the info we need to just run a GET.

Ah, thanks for clearing that up for me

jonathan-buttner · 2020-03-02T22:32:51Z

x-pack/plugins/endpoint/server/routes/resolver/utils/normalize.ts

@@ -7,7 +7,7 @@
 import { ResolverEvent, LegacyEndpointEvent } from '../../../../common/types';

 function isLegacyData(data: ResolverEvent): data is LegacyEndpointEvent {
-  return data.agent.type === 'endgame';
+  return data.agent?.type === 'endgame';


The endpoint isn't sending the agent.type field yet so guard against a crash here

EricDavisX · 2020-03-03T01:26:24Z

I recall a test file for alerts had my-index, does this need updating:
https://github.com/elastic/kibana/blob/master/x-pack/test/functional/es_archives/endpoint/alerts/api_feature/mappings.json#L6 ? @jonathan-buttner

jonathan-buttner · 2020-03-03T13:42:49Z

Thanks @EricDavisX yeah it was updated

jonathan-buttner · 2020-03-04T19:45:14Z

@elasticmachine merge upstream

jonathan-buttner · 2020-03-04T22:50:26Z

@elasticmachine merge upstream

jonathan-buttner · 2020-03-05T14:13:34Z

@elasticmachine merge upstream

jonathan-buttner · 2020-03-05T21:42:00Z

@elasticmachine merge upstream

kibanamachine · 2020-03-05T23:44:05Z

💛 Build succeeded, but was flaky

continuous-integration/kibana-ci/pull-request
Commit: 9b8e984
Flaky suites:
- xpack-kibana-ciGroup6

Test Failures

Kibana Pipeline / kibana-xpack-agent / X-Pack OpenID Connect API Integration Tests.x-pack/test/oidc_api_integration/apis/authorization_code_flow/oidc_auth·js.apis OpenID Connect authentication finishing handshake should succeed if both the OpenID Connect response and the cookie are provided

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 3 times on tracked branches: https://github.com/elastic/kibana/issues/43736

[00:00:00]       │
[00:00:00]         └-: apis
[00:00:00]           └-> "before all" hook
[00:00:00]           └-: OpenID Connect authentication
[00:00:00]             └-> "before all" hook
[00:00:00]             └-> should reject API requests if client is not authenticated
[00:00:00]               └-> "before each" hook: global before each
[00:00:00]               └- ✓ pass  (162ms) "apis OpenID Connect authentication should reject API requests if client is not authenticated"
[00:00:00]             └-: initiating handshake
[00:00:00]               └-> "before all" hook
[00:00:00]             └-: finishing handshake
[00:00:00]               └-> "before all" hook
[00:00:00]               └-> should fail if OpenID Connect response is not complemented with handshake cookie
[00:00:00]                 └-> "before each" hook: global before each
[00:00:00]                 └-> "before each" hook
[00:00:00]                 └- ✓ pass  (93ms) "apis OpenID Connect authentication finishing handshake should fail if OpenID Connect response is not complemented with handshake cookie"
[00:00:00]               └-> should fail if state is not matching
[00:00:00]                 └-> "before each" hook: global before each
[00:00:00]                 └-> "before each" hook
[00:00:00]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1583444604837070989] Authentication to realm oidc1 failed - Failed to authenticate user with OpenID Connect (Caused by org.elasticsearch.ElasticsearchSecurityException: Invalid state parameter [someothervalue], while [iis_zrnZDmOj3_j-N1kiQM2DRh9wQTmyVE88MmznqAg] was expected)
[00:00:00]                 └- ✓ pass  (182ms) "apis OpenID Connect authentication finishing handshake should fail if state is not matching"
[00:00:00]               └-> should succeed if both the OpenID Connect response and the cookie are provided
[00:00:00]                 └-> "before each" hook: global before each
[00:00:00]                 └-> "before each" hook
[00:00:00]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1583444604837070989] Authentication to realm oidc1 failed - Failed to authenticate user with OpenID Connect (Caused by org.elasticsearch.ElasticsearchSecurityException: Failed to exchange code for Id Token using the Token Endpoint.)
[00:00:01]                 └- ✖ fail: "apis OpenID Connect authentication finishing handshake should succeed if both the OpenID Connect response and the cookie are provided"
[00:00:01]                 │

Stack Trace

Error: expected 302 "Found", got 401 "Unauthorized"
    at Test._assertStatus (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:268:12)
    at Test._assertFunction (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:283:11)
    at Test.assert (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:173:18)
    at assert (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:131:12)
    at /dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:128:5
    at Test.Request.callback (/dev/shm/workspace/kibana/node_modules/superagent/lib/node/index.js:718:3)
    at parser (/dev/shm/workspace/kibana/node_modules/superagent/lib/node/index.js:906:18)
    at IncomingMessage.res.on (/dev/shm/workspace/kibana/node_modules/superagent/lib/node/parsers/json.js:19:7)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)