Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SecuritySolution][Detections] Adds UI for new Rule Fields: Related Integrations, Required Fields, and Setup #131475

Merged
merged 10 commits into from
May 23, 2022
Merged

[SecuritySolution][Detections] Adds UI for new Rule Fields: Related Integrations, Required Fields, and Setup #131475

merged 10 commits into from
May 23, 2022

Conversation

spong
Copy link
Member

@spong spong commented May 4, 2022 •
edited
Loading

Summary

Adds UI for new Rule Fields Related Integrations, Required Fields, and Setup to both the Rules Table and Rule Details pages. On the Rules Table a new column is added that shows the number of related integrations, and upon clicking will show you details about those integrations and links off to the integration page in fleet. On the Rule Details page Setup is added as a tab pill within the About section (if provided), and Related Integrations and Required Fields are displayed in the Definition section.

Once package/integration install data is added in #132667, the UI will show the installed status of an integration, and whether or not the installed version satisfies the related integration dependency.

NOTE: Until then, please follow the test instructions below for how to add a custom rule and return mock data to test the installed/uninstalled UI.

Related Issues
Related Links
Steps to test

In this initial implementation these new fields are only visible with Prebuilt Rules, and so there is limited API support and currently no UI for editing them. If a Prebuilt Rule is duplicated, these fields are emptied (set to '' or []). When a Rule is exported these fields are included (as empty values), and it is possible to edit the ndjson and re-import and then see these fields for the Custom Rule (but still not editable in the UI). This is expected behavior, and is actually a nice and easy way to test.

Here is a sample export you can paste into an test.ndjson file and import to test this feature. You can modify the package/version fields to test corner cases like if a package is installed but it's the wrong version.

{"id":"6cc39c80-da3a-11ec-9fce-65c1a0bee904","updated_at":"2022-05-23T01:48:23.422Z","updated_by":"elastic","created_at":"2022-05-23T01:48:20.940Z","created_by":"elastic","name":"Testing #131475, don't mind me...","tags":["Elastic","Endpoint Security"],"interval":"5m","enabled":false,"description":"Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.","risk_score":47,"severity":"medium","license":"Elastic License v2","output_index":".siem-signals-default","meta":{"from":"5m"},"rule_name_override":"message","timestamp_override":"event.ingested","author":["Elastic"],"false_positives":[],"from":"now-600s","rule_id":"2c66bf23-6ae9-4eb2-859e-446bea181ae9","max_signals":10000,"risk_score_mapping":[{"field":"event.risk_score","operator":"equals","value":""}],"severity_mapping":[{"field":"event.severity","operator":"equals","severity":"low","value":"21"},{"field":"event.severity","operator":"equals","severity":"medium","value":"47"},{"field":"event.severity","operator":"equals","severity":"high","value":"73"},{"field":"event.severity","operator":"equals","severity":"critical","value":"99"}],"threat":[],"to":"now","references":[],"version":7,"exceptions_list":[{"id":"endpoint_list","list_id":"endpoint_list","namespace_type":"agnostic","type":"endpoint"}],"immutable":false,"related_integrations":[{"package":"system","version":"1.6.4"},{"package":"aws","integration":"cloudtrail","version":"1.11.0"}],"required_fields":[{"ecs":true,"name":"event.code","type":"keyword"},{"ecs":true,"name":"message","type":"match_only_text"},{"ecs":false,"name":"winlog.event_data.AttributeLDAPDisplayName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.AttributeValue","type":"keyword"},{"ecs":false,"name":"winlog.event_data.ShareName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.RelativeTargetName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.AccessList","type":"keyword"}],"setup":"## Config\\n\\nThe 'Audit Detailed File Share' audit policy must be configured (Success Failure).\\nSteps to implement the logging policy with with Advanced Audit Configuration:\\n\\n```\\nComputer Configuration > \\nPolicies > \\nWindows Settings > \\nSecurity Settings > \\nAdvanced Audit Policies Configuration > \\nAudit Policies > \\nObject Access > \\nAudit Detailed File Share (Success,Failure)\\n```\\n\\nThe 'Audit Directory Service Changes' audit policy must be configured (Success Failure).\\nSteps to implement the logging policy with with Advanced Audit Configuration:\\n\\n```\\nComputer Configuration > \\nPolicies > \\nWindows Settings > \\nSecurity Settings > \\nAdvanced Audit Policies Configuration > \\nAudit Policies > \\nDS Access > \\nAudit Directory Service Changes (Success,Failure)\\n```\\n","type":"query","language":"kuery","index":["logs-endpoint.alerts-*"],"query":"event.kind:alert and event.module:(endpoint and not endgame)\\n","filters":[],"throttle":"no_actions","actions":[]}
{"exported_count":1,"exported_rules_count":1,"missing_rules":[],"missing_rules_count":0,"exported_exception_list_count":0,"exported_exception_list_item_count":0,"missing_exception_list_item_count":0,"missing_exception_list_items":[],"missing_exception_lists":[],"missing_exception_lists_count":0}
Existing plumbing for showing integration install state

This PR includes a useInstalledIntegrations hook wired up to the DETECTION_ENGINE_INSTALLED_INTEGRATIONS_URL route to be added in #132667. I plumbed the initial logic as if that API returned an array of integrations in the same format stored by the rule (i.e. the RelatedIntegrationArray type), so this will need to be adapted when integrating this feature. There's also a packages[] that can be provided to useInstalledIntegrations to constrain the search against all installed integrations, but this may not be used in the initial API.

To test the Installed Integrations UI, just uncomment the mock data return in use_installed_integrations.tsx.

Additional Notes/Todo:
  • Probably want to revisit the copy on the Rules Table integrations popover -- it reads a little off if we don't have have integration install data. Will work with docs folks on this.
  • No additional overflow logic was added for Related Integrations/Required Fields. We don't really have an overflow pattern for these description list items, so instead of just adding support for these two fields (as like another description list item that's a popover), would like to solve this for generically for all items.
  • TODO: Increase test coverage
Rule Details

Rule Details without Installed Integrations API changes

Rules Table

Rules Table without Installed Integrations API changes

Version mismatch

In cases where the related package/integration is installed, but the version is not satisfied, a warning icon/tooltip will display next to the integration link letting the user know the installed vs targeted version. I just wanted to make sure this case was handled so copy/UI isn't final -- any feedback welcome here! 🙂

Checklist

Delete any items that are not applicable to this PR.

@spong spong added enhancement New value added to drive a business result Feature:Detection Rules Anything related to Security Solution's Detection Rules Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Feature:Rule Management Security Solution Detection Rule Management Team:Detection Rule Management Security Detection Rule Management Team Theme: simp_prot_mgmt Security Solution Simplified Protection Management Theme v8.3.0 labels May 4, 2022
@spong spong self-assigned this May 4, 2022
@spong spong force-pushed the adds-related-integrations-ui branch from 9736848 to 4e7ecaf Compare May 20, 2022 03:09
@banderror banderror mentioned this pull request May 20, 2022
Merged
2 tasks
@spong spong force-pushed the adds-related-integrations-ui branch from 0b7e8f9 to 9c62a92 Compare May 20, 2022 16:50
spong pushed a commit that referenced this pull request May 20, 2022
@banderror
[Security Solution][Detections] Add new fields to the rule model: Rel…
fb1eeb0 
…ated Integrations, Required Fields, and Setup (#132409)

**Addresses partially:** elastic/security-team#2083, elastic/security-team#558, elastic/security-team#2856, elastic/security-team#1801 (internal tickets)

## Summary

**TL;DR:** With this PR, it's now possible to specify `related_integrations`, `required_fields`, and `setup` fields in prebuilt rules in https://github.com/elastic/detection-rules. They are returned within rules in the API responses.

This PR:

- Adds 3 new fields to the model of Security detection rules. These fields are common to all of the rule types we have.
  - **Related Integrations**. It's a list of Fleet integrations associated with a given rule. It's assumed that if the user installs them, the rule might start to work properly because it will start receiving source events potentially matching the rule's query.
  - **Required Fields**. It's a list of event fields that must be present in the source indices of a given rule.
  - **Setup Guide**. It's any instructions for the user for setting up their environment in order to start receiving source events for a given rule. It's a text. Markdown is supported. It's similar to the Investigation Guide that we show on the Details page.
- Adjusts API endpoints accordingly:
  - These fields are for prebuilt rules only and are supposed to be read-only in the UI.
  - Specifying these fields in the request parameters of the create/update/patch rule API endpoints is not supported.
  - These fields are returned in all responses that contain rules. If they are missing in a rule, default values are returned (empty array, empty string).
  - When duplicating a prebuilt rule, these fields are being reset to their default value (empty array, empty string).
  - Export/Import is supported. Edge case / supported hack: it's possible to specify these fields manually in a ndjson doc and import with a rule.
  - The fields are being copied to `kibana.alert.rule.parameters` field of an alert document, which is mapped as a flattened field type. No special handling for the new fields was needed there.
- Adjusts tests accordingly.

## Related Integrations

Example (part of a rule returned from the API):

```json
{
  "related_integrations": [
    {
      "package": "windows",
      "version": "1.5.x"
    },
    {
      "package": "azure",
      "integration": "activitylogs",
      "version": "~1.1.6"
    }
  ],
}
```

Schema:

```ts
/**
 * Related integration is a potential dependency of a rule. It's assumed that if the user installs
 * one of the related integrations of a rule, the rule might start to work properly because it will
 * have source events (generated by this integration) potentially matching the rule's query.
 *
 *   NOTE: Proper work is not guaranteed, because a related integration, if installed, can be
 *   configured differently or generate data that is not necessarily relevant for this rule.
 *
 * Related integration is a combination of a Fleet package and (optionally) one of the
 * package's "integrations" that this package contains. It is represented by 3 properties:
 *
 *   - `package`: name of the package (required, unique id)
 *   - `version`: version of the package (required, semver-compatible)
 *   - `integration`: name of the integration of this package (optional, id within the package)
 *
 * There are Fleet packages like `windows` that contain only one integration; in this case,
 * `integration` should be unspecified. There are also packages like `aws` and `azure` that contain
 * several integrations; in this case, `integration` should be specified.
 *
 * @example
 * const x: RelatedIntegration = {
 *   package: 'windows',
 *   version: '1.5.x',
 * };
 *
 * @example
 * const x: RelatedIntegration = {
 *   package: 'azure',
 *   version: '~1.1.6',
 *   integration: 'activitylogs',
 * };
 */
export type RelatedIntegration = t.TypeOf<typeof RelatedIntegration>;
export const RelatedIntegration = t.exact(
  t.intersection([
    t.type({
      package: NonEmptyString,
      version: NonEmptyString,
    }),
    t.partial({
      integration: NonEmptyString,
    }),
  ])
);
```

## Required Fields

Example (part of a rule returned from the API):

```json
{
  "required_fields": [
    {
      "name": "event.action",
      "type": "keyword",
      "ecs": true
    },
    {
      "name": "event.code",
      "type": "keyword",
      "ecs": true
    },
    {
      "name": "winlog.event_data.AttributeLDAPDisplayName",
      "type": "keyword",
      "ecs": false
    }
  ],
}
```

Schema:

```ts
/**
 * Almost all types of Security rules check source event documents for a match to some kind of
 * query or filter. If a document has certain field with certain values, then it's a match and
 * the rule will generate an alert.
 *
 * Required field is an event field that must be present in the source indices of a given rule.
 *
 * @example
 * const standardEcsField: RequiredField = {
 *   name: 'event.action',
 *   type: 'keyword',
 *   ecs: true,
 * };
 *
 * @example
 * const nonEcsField: RequiredField = {
 *   name: 'winlog.event_data.AttributeLDAPDisplayName',
 *   type: 'keyword',
 *   ecs: false,
 * };
 */
export type RequiredField = t.TypeOf<typeof RequiredField>;
export const RequiredField = t.exact(
  t.type({
    name: NonEmptyString,
    type: NonEmptyString,
    ecs: t.boolean,
  })
);
```

## Setup Guide

Example (part of a rule returned from the API):

```json
{
  "setup": "## Config\n\nThe 'PowerShell Script Block Logging' logging policy must be enabled.\nSteps to implement the logging policy with with Advanced Audit Configuration:\n\n```\nComputer Configuration > \nAdministrative Templates > \nWindows PowerShell > \nTurn on PowerShell Script Block Logging (Enable)\n```\n\nSteps to implement the logging policy via registry:\n\n```\nreg add \"hklm\\SOFTWARE\\Policies\\Microsoft\\Windows\\PowerShell\\ScriptBlockLogging\" /v EnableScriptBlockLogging /t REG_DWORD /d 1\n```\n",
}
```

Schema:

```ts
/**
 * Any instructions for the user for setting up their environment in order to start receiving
 * source events for a given rule.
 *
 * It's a multiline text. Markdown is supported.
 */
export type SetupGuide = t.TypeOf<typeof SetupGuide>;
export const SetupGuide = t.string;
```

## Details on the schema

This PR adjusts all the 6 rule schemas we have:

1. Alerting Framework rule `params` schema:
    - `security_solution/server/lib/detection_engine/schemas/rule_schemas.ts`
    - `security_solution/server/lib/detection_engine/schemas/rule_converters.ts`
2. HTTP API main old schema:
    - `security_solution/common/detection_engine/schemas/response/rules_schema.ts`
3. HTTP API main new schema:
    - `security_solution/common/detection_engine/schemas/request/rule_schemas.ts`
4. Prebuilt rule schema:
    - `security_solution/common/detection_engine/schemas/request/add_prepackaged_rules_schema.ts`
5. Import rule schema:
    - `security_solution/common/detection_engine/schemas/request/import_rules_schema.ts`
6. Rule schema used on the frontend side:
    - `security_solution/public/detections/containers/detection_engine/rules/types.ts`

Names of the fields on the HTTP API level:

- `related_integrations`
- `required_fields`
- `setup`

Names of the fields on the Alerting Framework level:

- `params.relatedIntegrations`
- `params.requiredFields`
- `params.setup`

## Next steps

- Create a new endpoint for returning installed Fleet integrations (gonna be a separate PR).
- Rebase #131475 on top of this PR after merge.
- Cover the new fields with dedicated tests (gonna be a separate PR).
- Update API docs (gonna be a separate PR).
- Address the tech debt of having 6 different schemas (gonna create a ticket for that).

### Checklist

- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
@spong spong force-pushed the adds-related-integrations-ui branch from 9c62a92 to 0b48577 Compare May 20, 2022 22:30
spong
spong commented May 21, 2022
View reviewed changes
x-pack/plugins/security_solution/common/constants.ts Outdated
@@ -237,6 +237,8 @@ export const DETECTION_ENGINE_PREPACKAGED_URL =
`${DETECTION_ENGINE_RULES_URL}/prepackaged` as const;
export const DETECTION_ENGINE_PRIVILEGES_URL = `${DETECTION_ENGINE_URL}/privileges` as const;
export const DETECTION_ENGINE_INDEX_URL = `${DETECTION_ENGINE_URL}/index` as const;
export const DETECTION_ENGINE_INSTALLED_INTEGRATIONS_URL =
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@banderror -- this is as far as I went with plumbing for the installed_integrations API. Created this route, and also wired it up to to the useInstalledIntegrations hook, so should just start working if you return a RelatedIntegrationArray from this route 🙂

@spong spong mentioned this pull request May 23, 2022
Closed
7 tasks
@spong spong marked this pull request as ready for review May 23, 2022 04:59
@spong spong requested review from a team as code owners May 23, 2022 04:59
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@spong
Copy link
Member Author

spong commented May 23, 2022

Files by Code Owner

elastic/kibana-app-services

  • packages/kbn-react-field/src/field_icon/snapshots/field_icon.test.tsx.snap
  • packages/kbn-react-field/src/field_icon/field_icon.tsx

elastic/security-detections-response-rules

  • x-pack/plugins/security_solution/public/common/components/ml_popover/jobs_table/snapshots/jobs_table.test.tsx.snap
  • x-pack/plugins/security_solution/public/common/components/ml_popover/jobs_table/jobs_table.tsx
  • x-pack/plugins/security_solution/public/detections/components/rules/description_step/helpers.tsx
  • x-pack/plugins/security_solution/public/detections/components/rules/description_step/index.tsx
  • x-pack/plugins/security_solution/public/detections/components/rules/description_step/required_integrations_description.tsx
  • x-pack/plugins/security_solution/public/detections/components/rules/description_step/translations.tsx
  • x-pack/plugins/security_solution/public/detections/components/rules/step_about_rule_details/index.test.tsx
  • x-pack/plugins/security_solution/public/detections/components/rules/step_about_rule_details/index.tsx
  • x-pack/plugins/security_solution/public/detections/components/rules/step_about_rule_details/translations.ts
  • x-pack/plugins/security_solution/public/detections/components/rules/step_define_rule/index.tsx
  • x-pack/plugins/security_solution/public/detections/components/rules/step_define_rule/schema.tsx
  • x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/api.ts
  • x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/translations.ts
  • x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/use_installed_integrations.tsx
  • x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/all/mocks/mock.ts
  • x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/all/use_columns.tsx
  • x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/helpers.test.tsx
  • x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/helpers.tsx
  • x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/translations.ts
  • x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/types.ts

elastic/security-solution

  • x-pack/plugins/security_solution/common/constants.ts
  • x-pack/plugins/security_solution/public/common/components/integrations_popover/helpers.tsx
  • x-pack/plugins/security_solution/public/common/components/integrations_popover/index.tsx
  • x-pack/plugins/security_solution/public/common/components/ml_popover/jobs_table/snapshots/jobs_table.test.tsx.snap
  • x-pack/plugins/security_solution/public/common/components/ml_popover/jobs_table/jobs_table.tsx
  • x-pack/plugins/security_solution/public/common/components/popover_items/index.tsx
  • x-pack/plugins/security_solution/public/detections/components/rules/description_step/helpers.tsx
  • x-pack/plugins/security_solution/public/detections/components/rules/description_step/index.tsx
  • x-pack/plugins/security_solution/public/detections/components/rules/description_step/required_integrations_description.tsx
  • x-pack/plugins/security_solution/public/detections/components/rules/description_step/translations.tsx
  • x-pack/plugins/security_solution/public/detections/components/rules/step_about_rule_details/index.test.tsx
  • x-pack/plugins/security_solution/public/detections/components/rules/step_about_rule_details/index.tsx
  • x-pack/plugins/security_solution/public/detections/components/rules/step_about_rule_details/translations.ts
  • x-pack/plugins/security_solution/public/detections/components/rules/step_define_rule/index.tsx
  • x-pack/plugins/security_solution/public/detections/components/rules/step_define_rule/schema.tsx
  • x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/api.ts
  • x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/translations.ts
  • x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/use_installed_integrations.tsx
  • x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/all/mocks/mock.ts
  • x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/all/use_columns.tsx
  • x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/helpers.test.tsx
  • x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/helpers.tsx
  • x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/translations.ts
  • x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/types.ts

spong
spong commented May 23, 2022
View reviewed changes
x-pack/plugins/security_solution/public/common/components/ml_popover/jobs_table/jobs_table.tsx
Comment on lines +92 to +98
<PopoverItems
items={groups}
numberOfItemsToDisplay={0}
popoverButtonTitle={`${groups.length} Groups`}
renderItem={renderItem}
dataTestPrefix="groups"
/>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't mind this... 😅

I was testing #131166 the other day and had it with the badge overflow. Since in doing this PR I learned of the PopoverItems component I figured there was a nice quick fix here:

cc @randomuserid

spong
spong commented May 23, 2022
View reviewed changes
x-pack/plugins/security_solution/public/common/components/popover_items/index.tsx
Comment on lines +79 to +83
{numberOfItemsToDisplay !== 0 && (
<EuiFlexItem grow={1} className="eui-textTruncate">
<OverflowList items={items.slice(0, numberOfItemsToDisplay)} />
</EuiFlexItem>
)}
Copy link
Member Author

@spong spong May 23, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spacing fix from leftover element when displaying empty items. Current implementation/use on Rules Table for Tags didn't exercise this path.

@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #21 / Actions and Triggers app Rule status filter should allow rule statuses to be filtered
  • [job] [logs] FTR Configs #44 / Home page feature controls security global all privileges shows all available solutions
  • [job] [logs] Security Solution Tests #4 / Timeline search and filters Update kqlMode for timeline "before all" hook for "should be able to update timeline kqlMode with filter"

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
securitySolution 2931 2971 +40

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
dataVisualizer 535.8KB 535.8KB +41.0B
discover 529.0KB 529.0KB +82.0B
graph 474.8KB 474.8KB +41.0B
lens 1.2MB 1.2MB +41.0B
maps 2.6MB 2.6MB +41.0B
presentationUtil 127.6KB 127.6KB +41.0B
securitySolution 5.0MB 5.0MB +30.7KB
stackAlerts 202.5KB 202.5KB +41.0B
total +31.0KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
osquery 16.2KB 16.3KB +41.0B
securitySolution 249.4KB 249.5KB +97.0B
total +138.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @spong

klacabane added a commit to klacabane/kibana that referenced this pull request May 23, 2022
@klacabane
Squashed commit of the following:
3f9cba4 
commit bdb4966
Author: Angela Chuang <6295984+angorayc@users.noreply.github.com>
Date:   Mon May 23 13:13:23 2022 +0100

    styling (elastic#132539)

    Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

commit a807c90
Author: Esteban Beltran <academo@users.noreply.github.com>
Date:   Mon May 23 13:36:00 2022 +0200

    [Cases] Add a key to userActionMarkdown to prevent stale state (elastic#132681)

commit ba84602
Author: Tomasz Ciecierski <ciecierskitomek@gmail.com>
Date:   Mon May 23 13:33:20 2022 +0200

    [Osquery] Change prebuilt saved queries to include prebuilt flag (elastic#132651)

commit 6b846af
Author: Faisal Kanout <faisal.kanout@elastic.co>
Date:   Mon May 23 14:11:04 2022 +0300

    [Actionable Observability] Update the Rule details design and clean up (elastic#132616)

    * Add rule status in the rule summary

    * Match design

    * Remove unused imports

    * code review

commit c993ff2
Author: Byron Hulcher <byron.hulcher@elastic.co>
Date:   Mon May 23 06:25:17 2022 -0400

    [Workplace Search] Add categories to source data for internal connectors (elastic#132671)

commit b59fb97
Author: Pablo Machado <pablo.nevesmachado@elastic.co>
Date:   Mon May 23 12:02:43 2022 +0200

    [Security Solution] Update use_url_state to work with new side nav (elastic#132518)

    * Fix landing pages browser tab title

    * Fix new navigation url state

    * Fix unit tests

    Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

commit 693b3e8
Author: Tomasz Ciecierski <ciecierskitomek@gmail.com>
Date:   Mon May 23 11:54:29 2022 +0200

    [Osquery] Add Osquery to Alert context menu (elastic#131790)

commit 2cddced
Author: Jordan <51442161+JordanSh@users.noreply.github.com>
Date:   Mon May 23 12:50:55 2022 +0300

    [Cloud Posture] Trendline query changes (elastic#132680)

commit 7591fb6
Author: Giorgos Bamparopoulos <georgios.bamparopoulos@elastic.co>
Date:   Mon May 23 10:37:03 2022 +0100

    Fix agent config indicator when applied through fleet integration (elastic#131820)

    * Fix agent config indicator when applied through fleet integration

    * Add synthrace scenario

    Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

commit 37d40d7
Author: Dominique Clarke <dominique.clarke@elastic.co>
Date:   Mon May 23 04:56:34 2022 -0400

    [Synthetics] fix browser type as default in monitor management (elastic#132572)

    Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

commit ae8b6c8
Author: Lucas F. da Costa <lucas.costa@elastic.co>
Date:   Mon May 23 09:29:11 2022 +0100

    [Uptime] Fix bug causing all monitors to be saved to all locations [solves elastic#132314] (elastic#132325)

    Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

commit e0944d1
Author: Nodir Latipov <nodir.latypov@gmail.com>
Date:   Mon May 23 13:27:24 2022 +0500

    [Unified search] Use the DataViews service (elastic#130008)

    * feat: cleanup deprecated service and type

    * fix: rollback test

    * refact: replace deprecated type

    * refact: changed deprecation type

    * feat: added comments to deprecated imports that can't be cleaned up in this PR

    * refact: rollback query_string_input.test file

commit a3646eb
Author: Pablo Machado <machadoum@gmail.com>
Date:   Mon May 23 10:17:12 2022 +0200

    [Security Solutions] Refactor breadcrumbs to support new menu structure (elastic#131624)

    * Refactor breadcrumbs to support new structure

    * Fix code style

    * Fix more code style

    * Fix unit test

    Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

commit 9649307
Author: István Zoltán Szabó <istvan.szabo@elastic.co>
Date:   Mon May 23 10:12:54 2022 +0200

    [DOCS] Updates alerting authorization docs with info on retaining API keys (elastic#132402)

    Co-authored-by: Lisa Cawley <lcawley@elastic.co>

commit 40df1f3
Author: Tomasz Ciecierski <ciecierskitomek@gmail.com>
Date:   Mon May 23 08:45:50 2022 +0200

    [Osquery] Add labels, move osquery schema link (elastic#132584)

commit fbaf058
Author: Jiawei Wu <74562234+JiaweiWu@users.noreply.github.com>
Date:   Sun May 22 17:14:23 2022 -0700

    [RAM] Add shareable rules list (elastic#132437)

    * Shareable rules list

    * Hide snooze panel in rules list

    * Address comments and added tests

    * Fix tests

    * Fix tests

    * Fix lint

    * Address design comments and fix tests

    Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

commit 383239e
Author: Kfir Peled <61654899+kfirpeled@users.noreply.github.com>
Date:   Sun May 22 13:18:42 2022 +0300

    [Cloud Posture] Findings - Group by resource - Fixed bug not showing results (elastic#132529)

commit fb1eeb0
Author: Georgii Gorbachev <georgii.gorbachev@elastic.co>
Date:   Sat May 21 00:21:53 2022 +0200

    [Security Solution][Detections] Add new fields to the rule model: Related Integrations, Required Fields, and Setup (elastic#132409)

    **Addresses partially:** elastic/security-team#2083, elastic/security-team#558, elastic/security-team#2856, elastic/security-team#1801 (internal tickets)

    ## Summary

    **TL;DR:** With this PR, it's now possible to specify `related_integrations`, `required_fields`, and `setup` fields in prebuilt rules in https://github.com/elastic/detection-rules. They are returned within rules in the API responses.

    This PR:

    - Adds 3 new fields to the model of Security detection rules. These fields are common to all of the rule types we have.
      - **Related Integrations**. It's a list of Fleet integrations associated with a given rule. It's assumed that if the user installs them, the rule might start to work properly because it will start receiving source events potentially matching the rule's query.
      - **Required Fields**. It's a list of event fields that must be present in the source indices of a given rule.
      - **Setup Guide**. It's any instructions for the user for setting up their environment in order to start receiving source events for a given rule. It's a text. Markdown is supported. It's similar to the Investigation Guide that we show on the Details page.
    - Adjusts API endpoints accordingly:
      - These fields are for prebuilt rules only and are supposed to be read-only in the UI.
      - Specifying these fields in the request parameters of the create/update/patch rule API endpoints is not supported.
      - These fields are returned in all responses that contain rules. If they are missing in a rule, default values are returned (empty array, empty string).
      - When duplicating a prebuilt rule, these fields are being reset to their default value (empty array, empty string).
      - Export/Import is supported. Edge case / supported hack: it's possible to specify these fields manually in a ndjson doc and import with a rule.
      - The fields are being copied to `kibana.alert.rule.parameters` field of an alert document, which is mapped as a flattened field type. No special handling for the new fields was needed there.
    - Adjusts tests accordingly.

    ## Related Integrations

    Example (part of a rule returned from the API):

    ```json
    {
      "related_integrations": [
        {
          "package": "windows",
          "version": "1.5.x"
        },
        {
          "package": "azure",
          "integration": "activitylogs",
          "version": "~1.1.6"
        }
      ],
    }
    ```

    Schema:

    ```ts
    /**
     * Related integration is a potential dependency of a rule. It's assumed that if the user installs
     * one of the related integrations of a rule, the rule might start to work properly because it will
     * have source events (generated by this integration) potentially matching the rule's query.
     *
     *   NOTE: Proper work is not guaranteed, because a related integration, if installed, can be
     *   configured differently or generate data that is not necessarily relevant for this rule.
     *
     * Related integration is a combination of a Fleet package and (optionally) one of the
     * package's "integrations" that this package contains. It is represented by 3 properties:
     *
     *   - `package`: name of the package (required, unique id)
     *   - `version`: version of the package (required, semver-compatible)
     *   - `integration`: name of the integration of this package (optional, id within the package)
     *
     * There are Fleet packages like `windows` that contain only one integration; in this case,
     * `integration` should be unspecified. There are also packages like `aws` and `azure` that contain
     * several integrations; in this case, `integration` should be specified.
     *
     * @example
     * const x: RelatedIntegration = {
     *   package: 'windows',
     *   version: '1.5.x',
     * };
     *
     * @example
     * const x: RelatedIntegration = {
     *   package: 'azure',
     *   version: '~1.1.6',
     *   integration: 'activitylogs',
     * };
     */
    export type RelatedIntegration = t.TypeOf<typeof RelatedIntegration>;
    export const RelatedIntegration = t.exact(
      t.intersection([
        t.type({
          package: NonEmptyString,
          version: NonEmptyString,
        }),
        t.partial({
          integration: NonEmptyString,
        }),
      ])
    );
    ```

    ## Required Fields

    Example (part of a rule returned from the API):

    ```json
    {
      "required_fields": [
        {
          "name": "event.action",
          "type": "keyword",
          "ecs": true
        },
        {
          "name": "event.code",
          "type": "keyword",
          "ecs": true
        },
        {
          "name": "winlog.event_data.AttributeLDAPDisplayName",
          "type": "keyword",
          "ecs": false
        }
      ],
    }
    ```

    Schema:

    ```ts
    /**
     * Almost all types of Security rules check source event documents for a match to some kind of
     * query or filter. If a document has certain field with certain values, then it's a match and
     * the rule will generate an alert.
     *
     * Required field is an event field that must be present in the source indices of a given rule.
     *
     * @example
     * const standardEcsField: RequiredField = {
     *   name: 'event.action',
     *   type: 'keyword',
     *   ecs: true,
     * };
     *
     * @example
     * const nonEcsField: RequiredField = {
     *   name: 'winlog.event_data.AttributeLDAPDisplayName',
     *   type: 'keyword',
     *   ecs: false,
     * };
     */
    export type RequiredField = t.TypeOf<typeof RequiredField>;
    export const RequiredField = t.exact(
      t.type({
        name: NonEmptyString,
        type: NonEmptyString,
        ecs: t.boolean,
      })
    );
    ```

    ## Setup Guide

    Example (part of a rule returned from the API):

    ```json
    {
      "setup": "## Config\n\nThe 'PowerShell Script Block Logging' logging policy must be enabled.\nSteps to implement the logging policy with with Advanced Audit Configuration:\n\n```\nComputer Configuration > \nAdministrative Templates > \nWindows PowerShell > \nTurn on PowerShell Script Block Logging (Enable)\n```\n\nSteps to implement the logging policy via registry:\n\n```\nreg add \"hklm\\SOFTWARE\\Policies\\Microsoft\\Windows\\PowerShell\\ScriptBlockLogging\" /v EnableScriptBlockLogging /t REG_DWORD /d 1\n```\n",
    }
    ```

    Schema:

    ```ts
    /**
     * Any instructions for the user for setting up their environment in order to start receiving
     * source events for a given rule.
     *
     * It's a multiline text. Markdown is supported.
     */
    export type SetupGuide = t.TypeOf<typeof SetupGuide>;
    export const SetupGuide = t.string;
    ```

    ## Details on the schema

    This PR adjusts all the 6 rule schemas we have:

    1. Alerting Framework rule `params` schema:
        - `security_solution/server/lib/detection_engine/schemas/rule_schemas.ts`
        - `security_solution/server/lib/detection_engine/schemas/rule_converters.ts`
    2. HTTP API main old schema:
        - `security_solution/common/detection_engine/schemas/response/rules_schema.ts`
    3. HTTP API main new schema:
        - `security_solution/common/detection_engine/schemas/request/rule_schemas.ts`
    4. Prebuilt rule schema:
        - `security_solution/common/detection_engine/schemas/request/add_prepackaged_rules_schema.ts`
    5. Import rule schema:
        - `security_solution/common/detection_engine/schemas/request/import_rules_schema.ts`
    6. Rule schema used on the frontend side:
        - `security_solution/public/detections/containers/detection_engine/rules/types.ts`

    Names of the fields on the HTTP API level:

    - `related_integrations`
    - `required_fields`
    - `setup`

    Names of the fields on the Alerting Framework level:

    - `params.relatedIntegrations`
    - `params.requiredFields`
    - `params.setup`

    ## Next steps

    - Create a new endpoint for returning installed Fleet integrations (gonna be a separate PR).
    - Rebase elastic#131475 on top of this PR after merge.
    - Cover the new fields with dedicated tests (gonna be a separate PR).
    - Update API docs (gonna be a separate PR).
    - Address the tech debt of having 6 different schemas (gonna create a ticket for that).

    ### Checklist

    - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
    - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

commit 788dd2e
Author: Andrew Goldstein <andrew-goldstein@users.noreply.github.com>
Date:   Fri May 20 16:02:05 2022 -0600

    [Security Solution] Fixes sorting and tooltips on columns for non-ECS fields that are only one level deep (elastic#132570)

    ## [Security Solution] Fixes sorting and tooltips on columns for non-ECS fields that are only one level deep

    This PR fixes <elastic#132490>, an issue where Timeline columns for non-ECS fields that are only one level deep couldn't be sorted, and displayed incomplete metadata in the column's tooltip.

    ### Before

    ![test_field_1_actual_tooltip](https://user-images.githubusercontent.com/4459398/169208299-51d9296a-15e1-4eb0-bc31-a0df6a63f0c5.png)

    _Before: The column is **not** sortable, and the tooltip displays incomplete metadata_

    ### After

    ![after](https://user-images.githubusercontent.com/4459398/169414767-7274a795-015f-4805-8c3f-b233ead994ea.png)

    _After: The column is sortable, and the tooltip displays the expected metadata_

    ### Desk testing

    See the _Steps to reproduce_ section of <elastic#132490> for testing details.

commit 51ae020
Author: Constance <constancecchen@users.noreply.github.com>
Date:   Fri May 20 14:30:36 2022 -0700

    Upgrade EUI to v55.1.3 (elastic#132451)

    * Upgrade EUI to 55.1.3 backport

    * [Deprecation] Remove `watchedItemProps` from EuiContextMenu usage - should no longer be necessary

    * Update snapshots with new data-popover attr

    * Fix failing FTR test

    - Now that EuiContextMenu focus is restored correctly, there is a tooltip around the popover toggle that's blocking an above item that the test wants to click

    - swapping the order so that the tooltip does not block the clicked item should work

    * Fix 2nd maps FTR test with blocking tooltip

    Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

commit 642290b
Author: Nathan Reese <reese.nathan@elastic.co>
Date:   Fri May 20 15:11:15 2022 -0600

    [maps] convert ESPewPewSource to typescript (elastic#132656)

    * [maps] convert ESPewPewSource to typescript

    * move @ts-expect-error moved by fix

commit eb6a061
Author: Brian Seeders <brian.seeders@elastic.co>
Date:   Fri May 20 16:57:49 2022 -0400

    [docs] Add 'yarn dev-docs' for managing and starting dev docs (elastic#132647)

commit e0ea600
Author: Hannah Mudge <Heenawter@users.noreply.github.com>
Date:   Fri May 20 14:55:31 2022 -0600

    Add group 6 to FTR config (elastic#132655)

commit 41635e2
Author: Karl Godard <karl.godard@elastic.co>
Date:   Fri May 20 13:35:30 2022 -0700

    fixed search highlighting. was only showing highlighted text w/o context (elastic#132650)

    Co-authored-by: mitodrummer <karlgodard@elastic.co>

commit 791ebfa
Author: debadair <debadair@elastic.co>
Date:   Fri May 20 13:34:04 2022 -0700

    [DOCS] Remove obsolete license expiration info (elastic#131474)

    * [DOCS] Remove obsolete license expiration info

    As of elastic/elasticsearch#79671, Elasticsearch does a more stringent license check
    rather than operating in a semi-degraded mode.

    Closes elastic#127845
    Closes elastic#125702

    * Update docs/management/managing-licenses.asciidoc

    Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

commit e55bf40
Author: Nathan Reese <reese.nathan@elastic.co>
Date:   Fri May 20 14:15:00 2022 -0600

    [Maps] create MVT_VECTOR when using choropleth wizard (elastic#132648)

commit 46cd729
Author: Jan Monschke <jan.monschke@elastic.co>
Date:   Fri May 20 22:02:00 2022 +0200

    [SecuritySolution] Disable agent status filters and timeline interaction (elastic#132586)

    * fix: disable drag-ability and hover actions for agent statuses

    The agent fields cannot be queried with ECS and therefore should not provide Filter In/Out functionality nor should users be able to add their representative fields to timeline investigations. Therefore users should not be able to add them to a timeline query by dragging them.

    * chore: make code more readable

commit e857b30
Author: Vadim Kibana <82822460+vadimkibana@users.noreply.github.com>
Date:   Fri May 20 20:36:59 2022 +0200

    remove human-readable automatic slug generation (elastic#132593)

    * remove human-readable automatic slug generation

    * make change non-breaking

    * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

    * remove test

    Co-authored-by: streamich <streamich@gmail.com>
    Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

commit 6fc2fff
Author: Lisa Cawley <lcawley@elastic.co>
Date:   Fri May 20 10:48:15 2022 -0700

    [ML] Minor edits in prebuilt job descriptions (elastic#132633)

commit ecca231
Author: Felix Stürmer <weltenwort@users.noreply.github.com>
Date:   Fri May 20 19:37:03 2022 +0200

    [Stack Monitoring] Convert setup routes to TypeScript (elastic#131265)

commit 065ea3e
Author: Byron Hulcher <byron.hulcher@elastic.co>
Date:   Fri May 20 13:12:49 2022 -0400

    [Workplace Search] Remove Custom API Source Integration tile (elastic#132538)

commit 583d2b7
Author: Byron Hulcher <byron.hulcher@elastic.co>
Date:   Fri May 20 13:12:32 2022 -0400

    [Workplace Search] Add documentation links for v8.3.0 connectors (elastic#132547)

commit c244883
Author: Nathan Reese <reese.nathan@elastic.co>
Date:   Fri May 20 10:35:00 2022 -0600

    [maps] show marker size in legend (elastic#132549)

    * [Maps] size legend

    * clean-up

    * refine spacing

    * clean up

    * more cleanup

    * use euiTheme for colors

    * fix jest test

    * do not show marker sizes for icons

    * remove lodash

    Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

commit d70ae0f
Author: Quynh Nguyen <43350163+qn895@users.noreply.github.com>
Date:   Fri May 20 11:34:35 2022 -0500

    [ILM] Add warnings for managed system policies  (elastic#132269)

    * Add warnings to system/managed policies

    * Fix translations, policies

    * Add jest tests

    * Add jest tests to assert new toggle behavior

    * Add jest tests for edit policy callout

    * Fix snapshot

    * [ML] Update jest tests with helper, rename helper for clarity

    * [ML] Add hook for local storage to remember toggle setting

    * [ML] Fix naming

    Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

commit f70b4af
Author: Nicolas Chaulet <nicolas.chaulet@elastic.co>
Date:   Fri May 20 12:22:08 2022 -0400

    [Fleet] Fix rolling upgrade CANCEL and UI fixes (elastic#132625)

commit d9f141a
Author: Kevin Logan <56395104+kevinlog@users.noreply.github.com>
Date:   Fri May 20 11:37:35 2022 -0400

    [Security Solution] Telemetry for Event Filters counts on both user and global entries (elastic#132542)

commit 1b4ac7d
Author: Yaroslav Kuznietsov <kuznetsov.yaroslav.yk@gmail.com>
Date:   Fri May 20 17:54:13 2022 +0300

    [XY] Reference lines overlay fix. (elastic#132607)

commit 759f13f
Author: Nicolas Chaulet <nicolas.chaulet@elastic.co>
Date:   Fri May 20 10:39:09 2022 -0400

    [Fleet] Remove reference to non removable package feature (elastic#132458)

commit 7e15097
Author: Lisa Cawley <lcawley@elastic.co>
Date:   Fri May 20 07:32:27 2022 -0700

    [ML] Adds placeholder text for testing NLP models (elastic#132486)

commit bc31053
Author: Dmitry Tomashevich <39378793+Dmitriynj@users.noreply.github.com>
Date:   Fri May 20 17:09:20 2022 +0300

    [Discover][Alerting] Implement editing of dataView, query & filters (elastic#131688)

    * [Discover] introduce params editing using unified search

    * [Discover] fix unit tests

    * [Discover] fix functional tests

    * [Discover] fix unit tests

    * [Discover] return test subject name

    * [Discover] fix alert functional test

    * Update x-pack/plugins/stack_alerts/public/alert_types/es_query/expression/search_source_expression_form.tsx

    Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>

    * Update x-pack/plugins/stack_alerts/public/alert_types/es_query/expression/search_source_expression_form.tsx

    Co-authored-by: Matthias Wilhelm <ankertal@gmail.com>

    * [Discover] hide filter panel options

    * [Discover] improve functional test

    * [Discover] apply suggestions

    * [Discover] change data view selector

    * [Discover] fix tests

    * [Discover] apply suggestions, fix lang mode toggler

    * [Discover] mote interface to types file, clean up diff

    * [Discover] fix saved query issue

    * Update x-pack/plugins/stack_alerts/server/alert_types/es_query/alert_type.ts

    Co-authored-by: Matthias Wilhelm <ankertal@gmail.com>

    * [Discover] remove zIndex

    * [Discover] omit null searchType from esQuery completely, add isEsQueryAlert check for useSavedObjectReferences hook

    * [Discover] set searchType to esQuery when needed

    * [Discover] fix unit tests

    * Update x-pack/plugins/stack_alerts/server/alert_types/es_query/alert_type_params.ts

    Co-authored-by: Matthias Wilhelm <ankertal@gmail.com>

    * Update x-pack/plugins/stack_alerts/server/alert_types/es_query/alert_type.ts

    Co-authored-by: Matthias Wilhelm <ankertal@gmail.com>

    Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>
    Co-authored-by: Matthias Wilhelm <ankertal@gmail.com>

commit d344088
Author: Nathan Reese <reese.nathan@elastic.co>
Date:   Fri May 20 08:06:25 2022 -0600

    [maps] Use label features from ES vector tile search API to fix multiple labels (elastic#132080)

    * [maps] mvt labels

    * eslint

    * only request labels when needed

    * update vector tile integration tests for hasLabels parameter

    * [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'

    * fix tests

    * fix test

    * only add _mvt_label_position filter when vector tiles are from ES vector tile search API

    * review feedback

    * include hasLabels in source data

    * fix jest test

    Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

commit 1d8bc7e
Author: Shivindera Singh <shivindera@gmail.com>
Date:   Fri May 20 15:53:00 2022 +0200

    hasData service - hit search api in case of an error with resolve api (elastic#132618)

commit 7c37eda
Author: Tomasz Ciecierski <ciecierskitomek@gmail.com>
Date:   Fri May 20 15:42:28 2022 +0200

    [Osquery] Fix pagination issue on Alert's Osquery Flyout (elastic#132611)

commit 2e51140
Author: Katerina Patticha <aikaterini.patticha@elastic.co>
Date:   Fri May 20 15:34:29 2022 +0200

    Show service group icon only when there are service groups (elastic#131138)

    * Show service group icon when there are service groups

    * Fix fix errors

    * Remove additional request and display  icon only for the service groups

    * Revert "Remove additional request and display  icon only for the service groups"

    This reverts commit 7ff2bc9.

    * Add dependencies

    Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

commit 24cbb32
Author: Yaroslav Kuznietsov <kuznetsov.yaroslav.yk@gmail.com>
Date:   Fri May 20 16:27:14 2022 +0300

    [XY] `pointsRadius`, `showPoints` and `lineWidth`. (elastic#130391)

    * Added pointsRadius, showPoints and lineWidth.

    * Added tests.

    Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

commit 1c2eb9f
Author: Sergi Massaneda <sergi.massaneda@elastic.co>
Date:   Fri May 20 13:59:56 2022 +0100

    [Security Solution] New Side nav integrating links config  (elastic#132210)

    * Update navigation landing pages to use appLinks config

    * align app links changes

    * link configs refactor to use updater$

    * navigation panel categories

    * test and type fixes

    * [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'

    * types changes

    * shared style change moved to a separate PR

    * use old deep links

    * minor changes after ux meeting

    * add links filtering

    * remove duplicated categories

    * temporary increase of plugin size limit

    * swap management links order

    * improve performance closing nav panel

    * test updated

    * host isolation page filterd and some improvements

    * remove async from plugin start

    * move links register from start to mount

    * restore size limits

    * Fix use_show_timeline unit tests

    Co-authored-by: Pablo Neves Machado <pablo.nevesmachado@elastic.co>
    Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

commit 92ac7f9
Author: Katrin Freihofner <katrin.freihofner@elastic.co>
Date:   Fri May 20 13:51:51 2022 +0200

    adds small styling updates to header panels (elastic#132596)
emilioalvap pushed a commit to emilioalvap/kibana that referenced this pull request May 23, 2022
@banderror @emilioalvap
[Security Solution][Detections] Add new fields to the rule model: Rel…
85112ca 
…ated Integrations, Required Fields, and Setup (elastic#132409)

**Addresses partially:** elastic/security-team#2083, elastic/security-team#558, elastic/security-team#2856, elastic/security-team#1801 (internal tickets)

## Summary

**TL;DR:** With this PR, it's now possible to specify `related_integrations`, `required_fields`, and `setup` fields in prebuilt rules in https://github.com/elastic/detection-rules. They are returned within rules in the API responses.

This PR:

- Adds 3 new fields to the model of Security detection rules. These fields are common to all of the rule types we have.
  - **Related Integrations**. It's a list of Fleet integrations associated with a given rule. It's assumed that if the user installs them, the rule might start to work properly because it will start receiving source events potentially matching the rule's query.
  - **Required Fields**. It's a list of event fields that must be present in the source indices of a given rule.
  - **Setup Guide**. It's any instructions for the user for setting up their environment in order to start receiving source events for a given rule. It's a text. Markdown is supported. It's similar to the Investigation Guide that we show on the Details page.
- Adjusts API endpoints accordingly:
  - These fields are for prebuilt rules only and are supposed to be read-only in the UI.
  - Specifying these fields in the request parameters of the create/update/patch rule API endpoints is not supported.
  - These fields are returned in all responses that contain rules. If they are missing in a rule, default values are returned (empty array, empty string).
  - When duplicating a prebuilt rule, these fields are being reset to their default value (empty array, empty string).
  - Export/Import is supported. Edge case / supported hack: it's possible to specify these fields manually in a ndjson doc and import with a rule.
  - The fields are being copied to `kibana.alert.rule.parameters` field of an alert document, which is mapped as a flattened field type. No special handling for the new fields was needed there.
- Adjusts tests accordingly.

## Related Integrations

Example (part of a rule returned from the API):

```json
{
  "related_integrations": [
    {
      "package": "windows",
      "version": "1.5.x"
    },
    {
      "package": "azure",
      "integration": "activitylogs",
      "version": "~1.1.6"
    }
  ],
}
```

Schema:

```ts
/**
 * Related integration is a potential dependency of a rule. It's assumed that if the user installs
 * one of the related integrations of a rule, the rule might start to work properly because it will
 * have source events (generated by this integration) potentially matching the rule's query.
 *
 *   NOTE: Proper work is not guaranteed, because a related integration, if installed, can be
 *   configured differently or generate data that is not necessarily relevant for this rule.
 *
 * Related integration is a combination of a Fleet package and (optionally) one of the
 * package's "integrations" that this package contains. It is represented by 3 properties:
 *
 *   - `package`: name of the package (required, unique id)
 *   - `version`: version of the package (required, semver-compatible)
 *   - `integration`: name of the integration of this package (optional, id within the package)
 *
 * There are Fleet packages like `windows` that contain only one integration; in this case,
 * `integration` should be unspecified. There are also packages like `aws` and `azure` that contain
 * several integrations; in this case, `integration` should be specified.
 *
 * @example
 * const x: RelatedIntegration = {
 *   package: 'windows',
 *   version: '1.5.x',
 * };
 *
 * @example
 * const x: RelatedIntegration = {
 *   package: 'azure',
 *   version: '~1.1.6',
 *   integration: 'activitylogs',
 * };
 */
export type RelatedIntegration = t.TypeOf<typeof RelatedIntegration>;
export const RelatedIntegration = t.exact(
  t.intersection([
    t.type({
      package: NonEmptyString,
      version: NonEmptyString,
    }),
    t.partial({
      integration: NonEmptyString,
    }),
  ])
);
```

## Required Fields

Example (part of a rule returned from the API):

```json
{
  "required_fields": [
    {
      "name": "event.action",
      "type": "keyword",
      "ecs": true
    },
    {
      "name": "event.code",
      "type": "keyword",
      "ecs": true
    },
    {
      "name": "winlog.event_data.AttributeLDAPDisplayName",
      "type": "keyword",
      "ecs": false
    }
  ],
}
```

Schema:

```ts
/**
 * Almost all types of Security rules check source event documents for a match to some kind of
 * query or filter. If a document has certain field with certain values, then it's a match and
 * the rule will generate an alert.
 *
 * Required field is an event field that must be present in the source indices of a given rule.
 *
 * @example
 * const standardEcsField: RequiredField = {
 *   name: 'event.action',
 *   type: 'keyword',
 *   ecs: true,
 * };
 *
 * @example
 * const nonEcsField: RequiredField = {
 *   name: 'winlog.event_data.AttributeLDAPDisplayName',
 *   type: 'keyword',
 *   ecs: false,
 * };
 */
export type RequiredField = t.TypeOf<typeof RequiredField>;
export const RequiredField = t.exact(
  t.type({
    name: NonEmptyString,
    type: NonEmptyString,
    ecs: t.boolean,
  })
);
```

## Setup Guide

Example (part of a rule returned from the API):

```json
{
  "setup": "## Config\n\nThe 'PowerShell Script Block Logging' logging policy must be enabled.\nSteps to implement the logging policy with with Advanced Audit Configuration:\n\n```\nComputer Configuration > \nAdministrative Templates > \nWindows PowerShell > \nTurn on PowerShell Script Block Logging (Enable)\n```\n\nSteps to implement the logging policy via registry:\n\n```\nreg add \"hklm\\SOFTWARE\\Policies\\Microsoft\\Windows\\PowerShell\\ScriptBlockLogging\" /v EnableScriptBlockLogging /t REG_DWORD /d 1\n```\n",
}
```

Schema:

```ts
/**
 * Any instructions for the user for setting up their environment in order to start receiving
 * source events for a given rule.
 *
 * It's a multiline text. Markdown is supported.
 */
export type SetupGuide = t.TypeOf<typeof SetupGuide>;
export const SetupGuide = t.string;
```

## Details on the schema

This PR adjusts all the 6 rule schemas we have:

1. Alerting Framework rule `params` schema:
    - `security_solution/server/lib/detection_engine/schemas/rule_schemas.ts`
    - `security_solution/server/lib/detection_engine/schemas/rule_converters.ts`
2. HTTP API main old schema:
    - `security_solution/common/detection_engine/schemas/response/rules_schema.ts`
3. HTTP API main new schema:
    - `security_solution/common/detection_engine/schemas/request/rule_schemas.ts`
4. Prebuilt rule schema:
    - `security_solution/common/detection_engine/schemas/request/add_prepackaged_rules_schema.ts`
5. Import rule schema:
    - `security_solution/common/detection_engine/schemas/request/import_rules_schema.ts`
6. Rule schema used on the frontend side:
    - `security_solution/public/detections/containers/detection_engine/rules/types.ts`

Names of the fields on the HTTP API level:

- `related_integrations`
- `required_fields`
- `setup`

Names of the fields on the Alerting Framework level:

- `params.relatedIntegrations`
- `params.requiredFields`
- `params.setup`

## Next steps

- Create a new endpoint for returning installed Fleet integrations (gonna be a separate PR).
- Rebase elastic#131475 on top of this PR after merge.
- Cover the new fields with dedicated tests (gonna be a separate PR).
- Update API docs (gonna be a separate PR).
- Address the tech debt of having 6 different schemas (gonna create a ticket for that).

### Checklist

- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
@spong spong enabled auto-merge (squash) May 23, 2022 20:09
banderror
banderror approved these changes May 23, 2022
View reviewed changes
Copy link
Contributor

@banderror banderror left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Peer-reviewed with @spong and code changes LGTM 👍
🚀 🚀 🚀

@spong spong merged commit 76e77c2 into elastic:main May 23, 2022
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label May 23, 2022
@spong spong deleted the adds-related-integrations-ui branch May 23, 2022 21:01
@banderror banderror mentioned this pull request May 23, 2022
Merged
1 task
banderror added a commit that referenced this pull request May 23, 2022
@banderror
[Security Solution][Detections] Implement endpoint for fetching insta…
dcf83f9 
…lled Fleet integrations (#132667)

**Addresses partially:** elastic/security-team#2856, elastic/security-team#3624 (internal tickets)

## Summary

Adds a new detections endpoint that returns installed Fleet integrations. It is to be used on the Rule Management and Rule Details pages (see #131475 for context and screenshots). This endpoint is `internal` - no need to document it.

```
GET /internal/detection_engine/fleet/integrations/installed
```

```json
{
  "installed_integrations": [
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "billing",
      "integration_title": "AWS Billing",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "cloudtrail",
      "integration_title": "AWS CloudTrail",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "cloudwatch",
      "integration_title": "AWS CloudWatch",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "dynamodb",
      "integration_title": "Amazon DynamoDB",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "ebs",
      "integration_title": "Amazon EBS",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "ec2",
      "integration_title": "Amazon EC2",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "elb",
      "integration_title": "AWS ELB",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "lambda",
      "integration_title": "AWS Lambda",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "natgateway",
      "integration_title": "Amazon NAT Gateway",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "firewall",
      "integration_title": "AWS Network Firewall",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "rds",
      "integration_title": "Amazon RDS",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "s3",
      "integration_title": "Amazon S3",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "s3_storage_lens",
      "integration_title": "Amazon S3 Storage Lens",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "sns",
      "integration_title": "Amazon SNS",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "sqs",
      "integration_title": "Amazon SQS",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "transitgateway",
      "integration_title": "AWS Transit Gateway",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "usage",
      "integration_title": "AWS Usage",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "vpcflow",
      "integration_title": "Amazon VPC",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "vpn",
      "integration_title": "Amazon VPN",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "waf",
      "integration_title": "AWS WAF",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "route53",
      "integration_title": "AWS Route 53",
      "is_enabled": false
    },
    {
      "package_name": "aws",
      "package_title": "AWS",
      "package_version": "1.16.1",
      "integration_name": "cloudfront",
      "integration_title": "Amazon CloudFront",
      "is_enabled": true
    },
    {
      "package_name": "system",
      "package_title": "System",
      "package_version": "1.13.0",
      "is_enabled": true
    }
  ]
}
```

## Next steps

- Test with users with different privileges (non-superusers). Fleet privileges: none, read, all. Security Solution privileges. SO privileges.
- Add filtering by `package_name` and `is_enabled` (will be done in a separate PR).
- Add test coverage (will be done in a separate PR).


### Checklist

Delete any items that are not applicable to this PR.

- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
@spong spong mentioned this pull request May 24, 2022
Merged
4 tasks
spong added a commit that referenced this pull request May 25, 2022
@spong
[Security Solution][Detections] Integrates installed integrations int…
3cacbe7 
…o interface (#132847)

## Summary

Wires up the new Installed Integrations API added in #132667 to the new Related Integrations UI added in #131475.

#### Additional changes include (though not all necessary for this specific PR):
- [X] Updates integrations badge icon to `package` on Rules Table
- [ ] Add Kibana Advanced Setting for disabling integrations badge on Rules Table
- [ ] Add loaders where necessary since there can now be API delay
- [ ] Separate description step components to specific files

Please see #131475 for screenshots and additional details.


#### Steps to test
In this initial implementation these new fields are only visible with Prebuilt Rules, and so there is limited API support and currently no UI for editing them. If a Prebuilt Rule is duplicated, these fields are emptied (set to `''` or `[]`). When a Rule is exported these fields are included (as empty values), and it is possible to edit the `ndjson` and re-import and then see these fields for the Custom Rule (but still not editable in the UI). This is expected behavior, and is actually a nice and easy way to test. 

Here is a sample export you can paste into a `test.ndjson` file and import to test this feature. You can modify the `package`/`version` fields to test corner cases like if a package is installed but it's the wrong version.

```
{"id":"6cc39c80-da3a-11ec-9fce-65c1a0bee904","updated_at":"2022-05-23T01:48:23.422Z","updated_by":"elastic","created_at":"2022-05-23T01:48:20.940Z","created_by":"elastic","name":"Testing #131475, don't mind me...","tags":["Elastic","Endpoint Security"],"interval":"5m","enabled":false,"description":"Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.","risk_score":47,"severity":"medium","license":"Elastic License v2","output_index":".siem-signals-default","meta":{"from":"5m"},"rule_name_override":"message","timestamp_override":"event.ingested","author":["Elastic"],"false_positives":[],"from":"now-600s","rule_id":"2c66bf23-6ae9-4eb2-859e-446bea181ae9","max_signals":10000,"risk_score_mapping":[{"field":"event.risk_score","operator":"equals","value":""}],"severity_mapping":[{"field":"event.severity","operator":"equals","severity":"low","value":"21"},{"field":"event.severity","operator":"equals","severity":"medium","value":"47"},{"field":"event.severity","operator":"equals","severity":"high","value":"73"},{"field":"event.severity","operator":"equals","severity":"critical","value":"99"}],"threat":[],"to":"now","references":[],"version":7,"exceptions_list":[{"id":"endpoint_list","list_id":"endpoint_list","namespace_type":"agnostic","type":"endpoint"}],"immutable":false,"related_integrations":[{"package":"system","version":"1.6.4"},{"package":"aws","integration":"cloudtrail","version":"1.11.0"}],"required_fields":[{"ecs":true,"name":"event.code","type":"keyword"},{"ecs":true,"name":"message","type":"match_only_text"},{"ecs":false,"name":"winlog.event_data.AttributeLDAPDisplayName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.AttributeValue","type":"keyword"},{"ecs":false,"name":"winlog.event_data.ShareName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.RelativeTargetName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.AccessList","type":"keyword"}],"setup":"## Config\\n\\nThe 'Audit Detailed File Share' audit policy must be configured (Success Failure).\\nSteps to implement the logging policy with with Advanced Audit Configuration:\\n\\n```\\nComputer Configuration > \\nPolicies > \\nWindows Settings > \\nSecurity Settings > \\nAdvanced Audit Policies Configuration > \\nAudit Policies > \\nObject Access > \\nAudit Detailed File Share (Success,Failure)\\n```\\n\\nThe 'Audit Directory Service Changes' audit policy must be configured (Success Failure).\\nSteps to implement the logging policy with with Advanced Audit Configuration:\\n\\n```\\nComputer Configuration > \\nPolicies > \\nWindows Settings > \\nSecurity Settings > \\nAdvanced Audit Policies Configuration > \\nAudit Policies > \\nDS Access > \\nAudit Directory Service Changes (Success,Failure)\\n```\\n","type":"query","language":"kuery","index":["logs-endpoint.alerts-*"],"query":"event.kind:alert and event.module:(endpoint and not endgame)\\n","filters":[],"throttle":"no_actions","actions":[]}
{"exported_count":1,"exported_rules_count":1,"missing_rules":[],"missing_rules_count":0,"exported_exception_list_count":0,"exported_exception_list_item_count":0,"missing_exception_list_item_count":0,"missing_exception_list_items":[],"missing_exception_lists":[],"missing_exception_lists_count":0}
```
j-bennet pushed a commit to j-bennet/kibana that referenced this pull request Jun 2, 2022
@banderror @j-bennet
[Security Solution][Detections] Add new fields to the rule model: Rel…
b838e8b 
…ated Integrations, Required Fields, and Setup (elastic#132409)

**Addresses partially:** https://github.com/elastic/security-team/issues/2083, https://github.com/elastic/security-team/issues/558, https://github.com/elastic/security-team/issues/2856, https://github.com/elastic/security-team/issues/1801 (internal tickets)

## Summary

**TL;DR:** With this PR, it's now possible to specify `related_integrations`, `required_fields`, and `setup` fields in prebuilt rules in https://github.com/elastic/detection-rules. They are returned within rules in the API responses.

This PR:

- Adds 3 new fields to the model of Security detection rules. These fields are common to all of the rule types we have.
  - **Related Integrations**. It's a list of Fleet integrations associated with a given rule. It's assumed that if the user installs them, the rule might start to work properly because it will start receiving source events potentially matching the rule's query.
  - **Required Fields**. It's a list of event fields that must be present in the source indices of a given rule.
  - **Setup Guide**. It's any instructions for the user for setting up their environment in order to start receiving source events for a given rule. It's a text. Markdown is supported. It's similar to the Investigation Guide that we show on the Details page.
- Adjusts API endpoints accordingly:
  - These fields are for prebuilt rules only and are supposed to be read-only in the UI.
  - Specifying these fields in the request parameters of the create/update/patch rule API endpoints is not supported.
  - These fields are returned in all responses that contain rules. If they are missing in a rule, default values are returned (empty array, empty string).
  - When duplicating a prebuilt rule, these fields are being reset to their default value (empty array, empty string).
  - Export/Import is supported. Edge case / supported hack: it's possible to specify these fields manually in a ndjson doc and import with a rule.
  - The fields are being copied to `kibana.alert.rule.parameters` field of an alert document, which is mapped as a flattened field type. No special handling for the new fields was needed there.
- Adjusts tests accordingly.

## Related Integrations

Example (part of a rule returned from the API):

```json
{
  "related_integrations": [
    {
      "package": "windows",
      "version": "1.5.x"
    },
    {
      "package": "azure",
      "integration": "activitylogs",
      "version": "~1.1.6"
    }
  ],
}
```

Schema:

```ts
/**
 * Related integration is a potential dependency of a rule. It's assumed that if the user installs
 * one of the related integrations of a rule, the rule might start to work properly because it will
 * have source events (generated by this integration) potentially matching the rule's query.
 *
 *   NOTE: Proper work is not guaranteed, because a related integration, if installed, can be
 *   configured differently or generate data that is not necessarily relevant for this rule.
 *
 * Related integration is a combination of a Fleet package and (optionally) one of the
 * package's "integrations" that this package contains. It is represented by 3 properties:
 *
 *   - `package`: name of the package (required, unique id)
 *   - `version`: version of the package (required, semver-compatible)
 *   - `integration`: name of the integration of this package (optional, id within the package)
 *
 * There are Fleet packages like `windows` that contain only one integration; in this case,
 * `integration` should be unspecified. There are also packages like `aws` and `azure` that contain
 * several integrations; in this case, `integration` should be specified.
 *
 * @example
 * const x: RelatedIntegration = {
 *   package: 'windows',
 *   version: '1.5.x',
 * };
 *
 * @example
 * const x: RelatedIntegration = {
 *   package: 'azure',
 *   version: '~1.1.6',
 *   integration: 'activitylogs',
 * };
 */
export type RelatedIntegration = t.TypeOf<typeof RelatedIntegration>;
export const RelatedIntegration = t.exact(
  t.intersection([
    t.type({
      package: NonEmptyString,
      version: NonEmptyString,
    }),
    t.partial({
      integration: NonEmptyString,
    }),
  ])
);
```

## Required Fields

Example (part of a rule returned from the API):

```json
{
  "required_fields": [
    {
      "name": "event.action",
      "type": "keyword",
      "ecs": true
    },
    {
      "name": "event.code",
      "type": "keyword",
      "ecs": true
    },
    {
      "name": "winlog.event_data.AttributeLDAPDisplayName",
      "type": "keyword",
      "ecs": false
    }
  ],
}
```

Schema:

```ts
/**
 * Almost all types of Security rules check source event documents for a match to some kind of
 * query or filter. If a document has certain field with certain values, then it's a match and
 * the rule will generate an alert.
 *
 * Required field is an event field that must be present in the source indices of a given rule.
 *
 * @example
 * const standardEcsField: RequiredField = {
 *   name: 'event.action',
 *   type: 'keyword',
 *   ecs: true,
 * };
 *
 * @example
 * const nonEcsField: RequiredField = {
 *   name: 'winlog.event_data.AttributeLDAPDisplayName',
 *   type: 'keyword',
 *   ecs: false,
 * };
 */
export type RequiredField = t.TypeOf<typeof RequiredField>;
export const RequiredField = t.exact(
  t.type({
    name: NonEmptyString,
    type: NonEmptyString,
    ecs: t.boolean,
  })
);
```

## Setup Guide

Example (part of a rule returned from the API):

```json
{
  "setup": "## Config\n\nThe 'PowerShell Script Block Logging' logging policy must be enabled.\nSteps to implement the logging policy with with Advanced Audit Configuration:\n\n```\nComputer Configuration > \nAdministrative Templates > \nWindows PowerShell > \nTurn on PowerShell Script Block Logging (Enable)\n```\n\nSteps to implement the logging policy via registry:\n\n```\nreg add \"hklm\\SOFTWARE\\Policies\\Microsoft\\Windows\\PowerShell\\ScriptBlockLogging\" /v EnableScriptBlockLogging /t REG_DWORD /d 1\n```\n",
}
```

Schema:

```ts
/**
 * Any instructions for the user for setting up their environment in order to start receiving
 * source events for a given rule.
 *
 * It's a multiline text. Markdown is supported.
 */
export type SetupGuide = t.TypeOf<typeof SetupGuide>;
export const SetupGuide = t.string;
```

## Details on the schema

This PR adjusts all the 6 rule schemas we have:

1. Alerting Framework rule `params` schema:
    - `security_solution/server/lib/detection_engine/schemas/rule_schemas.ts`
    - `security_solution/server/lib/detection_engine/schemas/rule_converters.ts`
2. HTTP API main old schema:
    - `security_solution/common/detection_engine/schemas/response/rules_schema.ts`
3. HTTP API main new schema:
    - `security_solution/common/detection_engine/schemas/request/rule_schemas.ts`
4. Prebuilt rule schema:
    - `security_solution/common/detection_engine/schemas/request/add_prepackaged_rules_schema.ts`
5. Import rule schema:
    - `security_solution/common/detection_engine/schemas/request/import_rules_schema.ts`
6. Rule schema used on the frontend side:
    - `security_solution/public/detections/containers/detection_engine/rules/types.ts`

Names of the fields on the HTTP API level:

- `related_integrations`
- `required_fields`
- `setup`

Names of the fields on the Alerting Framework level:

- `params.relatedIntegrations`
- `params.requiredFields`
- `params.setup`

## Next steps

- Create a new endpoint for returning installed Fleet integrations (gonna be a separate PR).
- Rebase elastic#131475 on top of this PR after merge.
- Cover the new fields with dedicated tests (gonna be a separate PR).
- Update API docs (gonna be a separate PR).
- Address the tech debt of having 6 different schemas (gonna create a ticket for that).

### Checklist

- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
@spong spong mentioned this pull request Jun 6, 2022
Merged
spong added a commit that referenced this pull request Jun 7, 2022
@spong
Adds match_only_text to ES_FIELD_TYPES (#133690)
de3410e 
## Summary

A new Security Solution feature (#131475) was added in `8.3` that displays a field name and icon token using the reusable `FieldIcon` component. In testing an issue was reported (#133291) that the wrong icon token was being displayed. 

I had [previously updated](https://github.com/elastic/kibana/pull/131475/files#diff-d79a8297783f3177da25dd13fe807425d9136a0e235fe170f7c0a61f2448dacaR23) `FieldIcon` to support `match_only_text` however this new issue was with the `float` type not displaying correctly. After some searching I found the `castEsToKbnFieldTypeName` utility which solved the issue with `float` fields not displaying, but then `match_only_text` field types would not show since it is missing from `ES_FIELD_TYPES` and so would resolve as `unknown`. 

This PR adds the `match_only_text` [ES type](https://www.elastic.co/guide/en/elasticsearch/reference/current/text.html#match-only-text-field-type) to `ES_FIELD_TYPES` to resolve this missing icon token issue so that `castEsToKbnFieldTypeName` can be used in conjunction with the resuable `FieldIcon` component. 

I imagine this is fine as it's a sibling type to `text`, but am curious here since `sortable: true,` is set for the KbnFieldType even though it includes `ES_FIELD_TYPES.TEXT` (which is not sortable) as well?
kibanamachine pushed a commit that referenced this pull request Jun 7, 2022
@spong
Adds match_only_text to ES_FIELD_TYPES (#133690)
938f9ee 
## Summary

A new Security Solution feature (#131475) was added in `8.3` that displays a field name and icon token using the reusable `FieldIcon` component. In testing an issue was reported (#133291) that the wrong icon token was being displayed.

I had [previously updated](https://github.com/elastic/kibana/pull/131475/files#diff-d79a8297783f3177da25dd13fe807425d9136a0e235fe170f7c0a61f2448dacaR23) `FieldIcon` to support `match_only_text` however this new issue was with the `float` type not displaying correctly. After some searching I found the `castEsToKbnFieldTypeName` utility which solved the issue with `float` fields not displaying, but then `match_only_text` field types would not show since it is missing from `ES_FIELD_TYPES` and so would resolve as `unknown`.

This PR adds the `match_only_text` [ES type](https://www.elastic.co/guide/en/elasticsearch/reference/current/text.html#match-only-text-field-type) to `ES_FIELD_TYPES` to resolve this missing icon token issue so that `castEsToKbnFieldTypeName` can be used in conjunction with the resuable `FieldIcon` component.

I imagine this is fine as it's a sibling type to `text`, but am curious here since `sortable: true,` is set for the KbnFieldType even though it includes `ES_FIELD_TYPES.TEXT` (which is not sortable) as well?

(cherry picked from commit de3410e)
kibanamachine added a commit that referenced this pull request Jun 7, 2022
@kibanamachine @spong
Adds match_only_text to ES_FIELD_TYPES (#133690) (#133787)
3c870c4 
## Summary

A new Security Solution feature (#131475) was added in `8.3` that displays a field name and icon token using the reusable `FieldIcon` component. In testing an issue was reported (#133291) that the wrong icon token was being displayed.

I had [previously updated](https://github.com/elastic/kibana/pull/131475/files#diff-d79a8297783f3177da25dd13fe807425d9136a0e235fe170f7c0a61f2448dacaR23) `FieldIcon` to support `match_only_text` however this new issue was with the `float` type not displaying correctly. After some searching I found the `castEsToKbnFieldTypeName` utility which solved the issue with `float` fields not displaying, but then `match_only_text` field types would not show since it is missing from `ES_FIELD_TYPES` and so would resolve as `unknown`.

This PR adds the `match_only_text` [ES type](https://www.elastic.co/guide/en/elasticsearch/reference/current/text.html#match-only-text-field-type) to `ES_FIELD_TYPES` to resolve this missing icon token issue so that `castEsToKbnFieldTypeName` can be used in conjunction with the resuable `FieldIcon` component.

I imagine this is fine as it's a sibling type to `text`, but am curious here since `sortable: true,` is set for the KbnFieldType even though it includes `ES_FIELD_TYPES.TEXT` (which is not sortable) as well?

(cherry picked from commit de3410e)

Co-authored-by: Garrett Spong <spong@users.noreply.github.com>
@banderror banderror added release_note:feature Makes this part of the condensed release notes and removed release_note:enhancement labels Jun 14, 2022
@banderror
Copy link
Contributor

@nastasha-solomon I changed the label here to align with our decision to treat Related Integrations, Required Fields, and Setup Guide as new features rather than enhancements in the release notes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@banderror banderror banderror approved these changes

@jloleysens jloleysens jloleysens approved these changes

Assignees

@spong spong

Labels
backport:skip This commit does not require backporting enhancement New value added to drive a business result Feature:Detection Rules Anything related to Security Solution's Detection Rules Feature:Rule Management Security Solution Detection Rule Management release_note:feature Makes this part of the condensed release notes Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Theme: simp_prot_mgmt Security Solution Simplified Protection Management Theme v8.3.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@spong @elasticmachine @kibana-ci @banderror @jloleysens @kibanamachine