Introduce alias support for index pattern #10075

ruflin · 2019-01-15T12:43:25Z

The goal of this PR is that in case an alias is detected in the fields.yml, for the index pattern the fields from the original are copied over.

houndci-bot · 2019-01-15T12:43:51Z

libbeat/common/field.go

@@ -142,6 +142,12 @@ func (f Fields) HasKey(key string) bool {
 	return f.hasKey(keys)
 }

+func (f Fields) GetField(key string) *Field {


exported method Fields.GetField should have comment or be unexported

ruflin · 2019-01-15T12:43:51Z

libbeat/kibana/testdata/beat-6.json

@@ -2,8 +2,8 @@
  "objects": [
    {
      "attributes": {
-        "fieldFormatMap": "{\"long\":{\"id\":\"url\",\"params\":{\"inputFormat\":\"string\",\"labelTemplate\":\"long template\",\"outputFormat\":\"float\",\"outputPrecision\":5,\"urlTemplate\":\"_a=(query:(language:lucene,query:'context.app.name:\\\"{{value}}\\\"'))\"}}}",
-        "fields": "[{\"aggregatable\":true,\"analyzed\":false,\"count\":0,\"doc_values\":true,\"indexed\":true,\"name\":\"long\",\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":false,\"analyzed\":false,\"count\":0,\"doc_values\":true,\"indexed\":true,\"name\":\"multifield_field\",\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"analyzed\":false,\"count\":0,\"doc_values\":true,\"indexed\":true,\"name\":\"multifield_field.keyword\",\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":false,\"analyzed\":false,\"count\":0,\"doc_values\":false,\"indexed\":false,\"name\":\"_id\",\"scripted\":false,\"searchable\":false,\"type\":\"string\"},{\"aggregatable\":true,\"analyzed\":false,\"count\":0,\"doc_values\":false,\"indexed\":false,\"name\":\"_type\",\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":false,\"analyzed\":false,\"count\":0,\"doc_values\":false,\"indexed\":false,\"name\":\"_index\",\"scripted\":false,\"searchable\":false,\"type\":\"string\"},{\"aggregatable\":false,\"analyzed\":false,\"count\":0,\"doc_values\":false,\"indexed\":false,\"name\":\"_score\",\"scripted\":false,\"searchable\":false,\"type\":\"number\"},{\"aggregatable\":false,\"analyzed\":false,\"count\":0,\"doc_values\":false,\"indexed\":false,\"name\":\"blob\",\"scripted\":false,\"searchable\":false,\"type\":\"binary\"}]",
+        "fieldFormatMap": "{\"long\":{\"id\":\"url\",\"params\":{\"inputFormat\":\"string\",\"labelTemplate\":\"long template\",\"outputFormat\":\"float\",\"outputPrecision\":5,\"urlTemplate\":\"_a=(query:(language:lucene,query:'context.app.name:\\\"{{value}}\\\"'))\"}},\"alias\":{\"id\":\"url\",\"params\":{\"inputFormat\":\"string\",\"labelTemplate\":\"long template\",\"outputFormat\":\"float\",\"outputPrecision\":5,\"urlTemplate\":\"_a=(query:(language:lucene,query:'context.app.name:\\\"{{value}}\\\"'))\"}}}",


@simitt Is there a good way to generate these files? I'm fighting to get the tests passing.

ruflin · 2019-01-15T12:43:56Z

libbeat/kibana/testdata/beat-6.json

@@ -2,8 +2,8 @@
  "objects": [
    {
      "attributes": {
-        "fieldFormatMap": "{\"long\":{\"id\":\"url\",\"params\":{\"inputFormat\":\"string\",\"labelTemplate\":\"long template\",\"outputFormat\":\"float\",\"outputPrecision\":5,\"urlTemplate\":\"_a=(query:(language:lucene,query:'context.app.name:\\\"{{value}}\\\"'))\"}}}",
-        "fields": "[{\"aggregatable\":true,\"analyzed\":false,\"count\":0,\"doc_values\":true,\"indexed\":true,\"name\":\"long\",\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":false,\"analyzed\":false,\"count\":0,\"doc_values\":true,\"indexed\":true,\"name\":\"multifield_field\",\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"analyzed\":false,\"count\":0,\"doc_values\":true,\"indexed\":true,\"name\":\"multifield_field.keyword\",\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":false,\"analyzed\":false,\"count\":0,\"doc_values\":false,\"indexed\":false,\"name\":\"_id\",\"scripted\":false,\"searchable\":false,\"type\":\"string\"},{\"aggregatable\":true,\"analyzed\":false,\"count\":0,\"doc_values\":false,\"indexed\":false,\"name\":\"_type\",\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":false,\"analyzed\":false,\"count\":0,\"doc_values\":false,\"indexed\":false,\"name\":\"_index\",\"scripted\":false,\"searchable\":false,\"type\":\"string\"},{\"aggregatable\":false,\"analyzed\":false,\"count\":0,\"doc_values\":false,\"indexed\":false,\"name\":\"_score\",\"scripted\":false,\"searchable\":false,\"type\":\"number\"},{\"aggregatable\":false,\"analyzed\":false,\"count\":0,\"doc_values\":false,\"indexed\":false,\"name\":\"blob\",\"scripted\":false,\"searchable\":false,\"type\":\"binary\"}]",
+        "fieldFormatMap": "{\"long\":{\"id\":\"url\",\"params\":{\"inputFormat\":\"string\",\"labelTemplate\":\"long template\",\"outputFormat\":\"float\",\"outputPrecision\":5,\"urlTemplate\":\"_a=(query:(language:lucene,query:'context.app.name:\\\"{{value}}\\\"'))\"}},\"alias\":{\"id\":\"url\",\"params\":{\"inputFormat\":\"string\",\"labelTemplate\":\"long template\",\"outputFormat\":\"float\",\"outputPrecision\":5,\"urlTemplate\":\"_a=(query:(language:lucene,query:'context.app.name:\\\"{{value}}\\\"'))\"}}}",


@simitt Is there a good way to generate these files? I'm fighting to get the tests passing.

simitt · 2019-01-15T17:37:54Z

libbeat/kibana/testdata/beat-6.json

@@ -2,8 +2,8 @@
  "objects": [
    {
      "attributes": {
-        "fieldFormatMap": "{\"long\":{\"id\":\"url\",\"params\":{\"inputFormat\":\"string\",\"labelTemplate\":\"long template\",\"outputFormat\":\"float\",\"outputPrecision\":5,\"urlTemplate\":\"_a=(query:(language:lucene,query:'context.app.name:\\\"{{value}}\\\"'))\"}}}",
-        "fields": "[{\"aggregatable\":true,\"analyzed\":false,\"count\":0,\"doc_values\":true,\"indexed\":true,\"name\":\"long\",\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":false,\"analyzed\":false,\"count\":0,\"doc_values\":true,\"indexed\":true,\"name\":\"multifield_field\",\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"analyzed\":false,\"count\":0,\"doc_values\":true,\"indexed\":true,\"name\":\"multifield_field.keyword\",\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":false,\"analyzed\":false,\"count\":0,\"doc_values\":false,\"indexed\":false,\"name\":\"_id\",\"scripted\":false,\"searchable\":false,\"type\":\"string\"},{\"aggregatable\":true,\"analyzed\":false,\"count\":0,\"doc_values\":false,\"indexed\":false,\"name\":\"_type\",\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":false,\"analyzed\":false,\"count\":0,\"doc_values\":false,\"indexed\":false,\"name\":\"_index\",\"scripted\":false,\"searchable\":false,\"type\":\"string\"},{\"aggregatable\":false,\"analyzed\":false,\"count\":0,\"doc_values\":false,\"indexed\":false,\"name\":\"_score\",\"scripted\":false,\"searchable\":false,\"type\":\"number\"},{\"aggregatable\":false,\"analyzed\":false,\"count\":0,\"doc_values\":false,\"indexed\":false,\"name\":\"blob\",\"scripted\":false,\"searchable\":false,\"type\":\"binary\"}]",
+        "fieldFormatMap": "{\"long\":{\"id\":\"url\",\"params\":{\"inputFormat\":\"string\",\"labelTemplate\":\"long template\",\"outputFormat\":\"float\",\"outputPrecision\":5,\"urlTemplate\":\"_a=(query:(language:lucene,query:'context.app.name:\\\"{{value}}\\\"'))\"}},\"alias\":{\"id\":\"url\",\"params\":{\"inputFormat\":\"string\",\"labelTemplate\":\"long template\",\"outputFormat\":\"float\",\"outputPrecision\":5,\"urlTemplate\":\"_a=(query:(language:lucene,query:'context.app.name:\\\"{{value}}\\\"'))\"}}}",


@ruflin you were asking how you could make the alias show up in the fieldFormat section of the Kibana index pattern. This is the entry point for handling them.

This is the way I make it show up fieldsFormat: https://github.com/elastic/beats/pull/10075/files#diff-086ab841dd8317d2aeec5107211984c2R87 Running it against Kibana this trick seems to work. My problem is with the tests and fieldFormatMap

Actually I don't recall a good way for creating the tests automatically.

exekias · 2019-01-16T16:55:13Z

I pushed a commit that I think fixes this, please take a look, I had to change part of the test data as I think something was missing

ruflin

LGTM. Unfortunately I can't approve it with a checkbox as Github tells me it's my own :-) We should probably also add a changelog entry?

ruflin · 2019-01-17T12:50:43Z

libbeat/kibana/fields_transformer.go

@@ -24,6 +24,8 @@ import (
 	"github.com/elastic/beats/libbeat/common"
 )

+var v640 = common.MustNewVersion("6.4.0")


Almost forgot that part.

The goal of this PR is that in case an alias is detected in the fields.yml, for the index pattern the fields from the original are copied over.

ruflin added in progress Pull request is currently in progress. libbeat :Dashboards ecs labels Jan 15, 2019

ruflin requested a review from a team as a code owner January 15, 2019 12:43

houndci-bot reviewed Jan 15, 2019

View reviewed changes

ruflin commented Jan 15, 2019

View reviewed changes

simitt reviewed Jan 15, 2019

View reviewed changes

This was referenced Jan 16, 2019

Populate more ECS fields in the Suricata module #10006

Merged

Add migration.enabled config option #9805

Merged

ruflin mentioned this pull request Jan 17, 2019

Remove 5.0 tests for index pattern generation #10077

Merged

ruflin commented Jan 17, 2019

View reviewed changes

exekias approved these changes Jan 18, 2019

View reviewed changes

ruflin added review and removed in progress Pull request is currently in progress. labels Jan 18, 2019

ruflin and others added 4 commits January 18, 2019 15:37

Introduce alias support for index pattern

bcc9877

The goal of this PR is that in case an alias is detected in the fields.yml, for the index pattern the fields from the original are copied over.

push partial fix

00f3eea

Copy alias info to fieldFormatMap too

598d1ca

add changelog

e170ba6

ruflin force-pushed the index-pattern-alias-support branch from 58b0b63 to e170ba6 Compare January 18, 2019 14:38

make hound happy

0403c04

ruflin merged commit b292ba2 into elastic:master Jan 21, 2019

ruflin deleted the index-pattern-alias-support branch January 21, 2019 07:47

ruflin self-assigned this Jan 21, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce alias support for index pattern #10075

Introduce alias support for index pattern #10075

ruflin commented Jan 15, 2019

houndci-bot Jan 15, 2019

ruflin Jan 15, 2019

ruflin Jan 15, 2019

simitt Jan 15, 2019 •

edited

Loading

ruflin Jan 16, 2019

simitt Jan 16, 2019

exekias commented Jan 16, 2019

ruflin left a comment

ruflin Jan 17, 2019

Introduce alias support for index pattern #10075

Introduce alias support for index pattern #10075

Conversation

ruflin commented Jan 15, 2019

houndci-bot Jan 15, 2019

Choose a reason for hiding this comment

ruflin Jan 15, 2019

Choose a reason for hiding this comment

ruflin Jan 15, 2019

Choose a reason for hiding this comment

simitt Jan 15, 2019 • edited Loading

Choose a reason for hiding this comment

ruflin Jan 16, 2019

Choose a reason for hiding this comment

simitt Jan 16, 2019

Choose a reason for hiding this comment

exekias commented Jan 16, 2019

ruflin left a comment

Choose a reason for hiding this comment

ruflin Jan 17, 2019

Choose a reason for hiding this comment

simitt Jan 15, 2019 •

edited

Loading