Add the option to generate the template into a file #4323

tsg · 2017-05-16T09:57:58Z

Part of #3654.

This adds two settings: setup.template.output_to_file.path and setup.template.output_to_file.version. The version refers to the ES version and is optional, we'll use the Beats version if not specified. I put it under output_to_file to make it clear that it only applies when outputting to a file:

To generate a config, one can do:

./metricbeat -e -E setup.template.output_to_file.path=template.json

In the current version, the Beat automatically stops after generating the template, but the output might be slightly confusing:

2017/05/16 09:55:02.043671 load.go:116: INFO Template for Elasticsearch 6.0.0-alpha2 written to: template.json
2017/05/16 09:55:02.043717 beat.go:538: CRIT Exiting: Stopping after successfully writing the template to the file.
Exiting: Stopping after successfully writing the template to the file.

IMO this is better than the alternative of leaving it running.

To generate the template for the 2.x version, one can do:

./metricbeat -e -E setup.template.output_to_file.path=template.json -E setup.template.output_to_file.version=2.4.0

Remaining TODOs:

System test
Docs
Changelog

Part of elastic#3654.

ruflin · 2017-05-16T10:31:33Z

Perhaps we should discuss again with after --setup the beat should stop? Because then one could do:

./metricbeat --setup -e -E setup.template.output_to_file.path=template.json

Or assuming -S is implies --setup and prefixes -E with the setup namespace. Then the following could be used:

./metricbeat -S template.output_to_file.path=template.json

elasticmachine · 2017-05-16T10:32:49Z

libbeat/beat/beat.go


+			// XXX: Should we kill the Beat here or just continue?
+			return fmt.Errorf("Stopping after successfully writing the template to the file.")


[golint] _{reported by reviewdog 🐶}
error strings should not be capitalized or end with punctuation or a newline

In this case, dear reviewdog, we know that the error is final, so the punctuation looks better to the user.

elasticmachine · 2017-05-16T10:32:49Z

libbeat/template/load.go

@@ -80,6 +82,41 @@ func (l *Loader) Load() error {
 	return nil
 }

+func (l *Loader) Generate() error {


[golint] _{reported by reviewdog 🐶}
exported method Loader.Generate should have comment or be unexported

tsg · 2017-05-16T12:03:04Z

@ruflin Yeah, I guess we should review the whole -setup mechanism, maybe we can unify the handling in such a way that it makes sense. The beats-tools is tempting, but we'll need to build it for all supported platforms, which makes me a bit worried about the packaging overhead (especially since it's not actually a beat).

I have the feeling we can't get a nice solution as part of this PR, so the question is what do we want for the time being? IMO stopping is much nicer in this case, but it's inconsistent with -setup, that's true :(. I'm still leaning towards stopping.

ruflin · 2017-05-16T12:12:09Z

Agree that we can't find a good solution as part of this PR. +1 on stopping as if you want to manually load the template, you don't want to already send data.

ruflin

I left a few comments which we should handle in a follow up. Will merge it for now as this is a big step forward.

ruflin · 2017-05-18T07:42:54Z

libbeat/beat/beat.go

@@ -469,8 +469,30 @@ func (b *Beat) registerTemplateLoading() error {
 		}
 	}

-	esConfig := b.Config.Output["elasticsearch"]
+	// Check if outputting to file is enabled, and output to file if it is


This comment confused me at first because the check for the file only happens later, but I see you meant it for the "complete" block

ruflin · 2017-05-18T07:43:40Z

libbeat/template/config.go

 	Settings     templateSettings `config:"settings"`
+	OutputToFile OutputToFile     `config:"output_to_file"`


Not too much of a fan of the config name, but well :-)

ruflin · 2017-05-18T07:45:25Z

libbeat/template/load.go

@@ -80,6 +82,43 @@ func (l *Loader) Load() error {
 	return nil
 }

+// Generate generates the template and writes it to a file based on the configuration
+// from `output_to_file`.
+func (l *Loader) Generate() error {


We could probably now use this function also in https://github.com/elastic/beats/blob/master/dev-tools/cmd/index_template/index_template.go

Oh, I didn't realize we have that utility already.

ruflin · 2017-05-18T07:46:29Z

libbeat/tests/system/base.py

@@ -9,4 +9,11 @@ def setUpClass(self):
        self.beat_name = "mockbeat"
        self.beat_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "../../"))
        self.test_binary = self.beat_path + "/libbeat.test"
+        self.beats = [
+            "filebeat",


Looks like a beats specifics dependency in libbeat.

Yeah, only needed for those skipped tets for now, I can delete them.

ruflin · 2017-05-18T07:46:59Z

libbeat/tests/system/test_dashboard.py

-        beats = ["metricbeat", "packetbeat", "filebeat", "winlogbeat"]
-
-        for beat in beats:
+        for beat in self.beats:


be aware that these tests are currently skipped.

Add the option to generate the template into a file

dabc10c

Part of elastic#3654.

tsg added :Templates review in progress Pull request is currently in progress. libbeat labels May 16, 2017

ruflin mentioned this pull request May 16, 2017

Dynamically generate and load index templates #3654

Closed

14 tasks

elasticmachine reviewed May 16, 2017

View reviewed changes

Tudor Golubenco added 2 commits May 17, 2017 12:24

Added system test that generates all templates

2dd1e96

changelog and docs

8b4daa2

tsg removed the in progress Pull request is currently in progress. label May 17, 2017

Test on fields.yml from libbeat only, to not depend on other beats

54ff4c6

ruflin approved these changes May 18, 2017

View reviewed changes

ruflin merged commit f9c0af6 into elastic:master May 18, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the option to generate the template into a file #4323

Add the option to generate the template into a file #4323

tsg commented May 16, 2017 •

edited

Loading

ruflin commented May 16, 2017

elasticmachine May 16, 2017

tsg May 16, 2017

elasticmachine May 16, 2017

tsg commented May 16, 2017

ruflin commented May 16, 2017

ruflin left a comment

ruflin May 18, 2017

ruflin May 18, 2017

ruflin May 18, 2017

tsg May 23, 2017

ruflin May 18, 2017

tsg May 23, 2017

ruflin May 18, 2017


		// XXX: Should we kill the Beat here or just continue?
		return fmt.Errorf("Stopping after successfully writing the template to the file.")

		Settings templateSettings `config:"settings"`
		OutputToFile OutputToFile `config:"output_to_file"`

Add the option to generate the template into a file #4323

Add the option to generate the template into a file #4323

Conversation

tsg commented May 16, 2017 • edited Loading

ruflin commented May 16, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsg commented May 16, 2017

ruflin commented May 16, 2017

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsg commented May 16, 2017 •

edited

Loading