Convert all projects to use Mage #9842

andrewkroh · 2019-01-02T09:57:51Z

Convert all projects to use Mage https://magefile.org for building.

Each Beat is buildable from both the OSS directory and the X-Pack directory.
- This will make it simpler to add x-pack specific content in the future if desired.
beat-dashboards.zip will include the all dashboards (include x-pack).
Use the same logic to generate include/list.go in each Beat.
Generate a fields.go for each Beat/Module/Input/Monitor.
Generate the Travis config based on what the root magefile.go specifies to build/test.
Remove some unused scripts and files.
Update jenkins scripts to execute mage for building/testing.
Add a mage docs target that generates docs and opens the browser when PREVIEW is set.

TODO:

Update all projects to use mage.BeatProjectType.
Do we want to have the all the various config files checked in?
Fix metricbeat's system module docs to remove Go template stuff from the config.
Add docs to the root magefile.go.
Test APM
Fix the generators so they work on any OS.
Add a fields.go file to libbeat and remove libbeat content from each individual beat's include/fields.go.
Add something to aggregate code coverage reports.
Remove all the lingering requirements.txt files and use a single one.
Remove code that existed only to transition between make/mage.
Continue to remove duplication.
Find out what targets we are missing from users.
Add some basic docs for the build system (how to do X).

houndci-bot · 2019-01-02T09:58:23Z

magefile.go

+	})
+}
+
+func (Test) Integ() error {


exported method Test.Integ should have comment or be unexported

houndci-bot · 2019-01-02T09:58:24Z

magefile.go

+	})
+}
+
+func (Test) Unit() error {


exported method Test.Unit should have comment or be unexported

houndci-bot · 2019-01-02T09:58:24Z

magefile.go

+
+type Test mg.Namespace
+
+func (Test) All() error {


exported method Test.All should have comment or be unexported

houndci-bot · 2019-01-02T09:58:24Z

magefile.go

+	})
+}
+
+type Test mg.Namespace


exported type Test should have comment or be unexported

houndci-bot · 2019-01-02T09:58:25Z

magefile.go

+	Stage string
+}
+
+type Package mg.Namespace


exported type Package should have comment or be unexported

houndci-bot · 2019-01-02T09:58:26Z

magefile.go

+	"update:fields",
+}
+
+func (Check) Targets() error {


exported method Check.Targets should have comment or be unexported

houndci-bot · 2019-01-02T09:58:26Z

magefile.go

@@ -94,7 +172,372 @@ func addLicenseHeaders() error {
 	)
 }

+func (Check) Vet() error {


exported method Check.Vet should have comment or be unexported

houndci-bot · 2019-01-02T09:58:26Z

magefile.go

+
+type Check mg.Namespace
+
+// Check checks that code is formatted and generated files are up-to-date.


comment on exported method Check.All should be of the form "All ..."

houndci-bot · 2019-01-02T09:58:26Z

magefile.go

+	return mage.Clean(paths)
+}
+
+type Check mg.Namespace


exported type Check should have comment or be unexported

houndci-bot · 2019-01-02T09:58:26Z

magefile.go

-		OutputFile: "build/distributions/dashboards/{{.Name}}-{{.Version}}{{if .Snapshot}}-SNAPSHOT{{end}}",
+// --- Targets ---
+
+func Clean() error {


exported function Clean should have comment or be unexported

Convert all projects to using Mage https://magefile.org for building. Generate the travis config based on what the root magefile specifies. Remove some unused scripts and files. Update jenkins scripts to use mage for testing. Add a mage docs target that generates docs and opens the browser when PREVIEW is set.

houndci-bot · 2019-01-02T10:02:36Z

dev-tools/mage/target/build/build.go

+}
+
+// BuildGoDaemon builds the go-daemon binary (use crossBuildGoDaemon).
+func BuildGoDaemon() error {


func name will be used as build.BuildGoDaemon by other packages, and that stutters; consider calling this GoDaemon

houndci-bot · 2019-01-02T10:02:37Z

magefile.go

+		// TODO: Add generators.
+	}
+
+	Aliases = map[string]interface{}{


exported var Aliases should have comment or be unexported

ph · 2019-01-02T13:29:34Z

x-pack/functionbeat/make.bat

@@ -6,6 +6,6 @@ REM
 REM After running this once you may invoke mage.exe directly.

 WHERE mage
-IF %ERRORLEVEL% NEQ 0 go install github.com/ph/functionbeat/vendor/github.com/magefile/mage
+IF %ERRORLEVEL% NEQ 0 go install github.com/elastic/beats/vendor/github.com/magefile/mage


🤦‍♂️

ruflin

This change is great and I can't wait to get all this in. As usual to I will ask if we could split this up into multiple smaller PR's? I think there are quite a few changes in there that could happen separately to get it in quickly and make it more reviewable.

I hope most magefile commands could already be introduce by calling mage from the make commands, for example for the docs and this could happen in separate PR's?

ruflin · 2019-01-07T09:48:07Z

.travis.yml

+  - name: check
+  - name: test
+  - name: crosscompile
+    if: type != pull_request


What is the reason you do not want to have this on each PR?

ruflin · 2019-01-07T09:53:51Z

deploy/kubernetes/filebeat/filebeat-service-account.yaml

@@ -1,3 +1,4 @@
+---


Could all the changes to k8s go into a separate PR?

ruflin · 2019-01-07T09:55:08Z

dev-tools/ci/templates/travis.yml.tmpl

@@ -0,0 +1,146 @@
+# DO NOT EDIT - AUTO-GENERATED


Very nice. I hope we can also use this later on to generate the Jenkins Pipeline file or that the Pipeline file actually contains all this logic.

Could this change also happen without mage?

ruflin · 2019-01-07T09:59:18Z

heartbeat/monitors/active/dialchain/fields.go

@@ -0,0 +1,35 @@
+// Licensed to Elasticsearch B.V. under one or more contributor


Separate PR for changes in heartbeat?

urso · 2020-02-28T10:30:42Z

Hi @andrewkroh, this PR seems to be stalled and quite out of date. Can we close this one?

andrewkroh added the in progress Pull request is currently in progress. label Jan 2, 2019

andrewkroh requested a review from a team as a code owner January 2, 2019 09:57

houndci-bot reviewed Jan 2, 2019

View reviewed changes

andrewkroh added 12 commits January 2, 2019 05:01

Add magefile to deploy/kubernetes

8f75c8b

Refactor Auditbeat build logic

9b0c54a

Refactor filebeat build logic

a579379

Refactor heartbeat build logic

4026e43

Refactor journalbeat build logic

d4b55c7

Refactor libbeat build logic

7dbddca

Refactor metricbeat build logic

e6e3a11

Refactor packetbeat build logic

442c241

Refactor winlogbeat build logic

bb2cdc6

Refactor functionbeat build logic

c26e70c

Fix haproxy module expected log

c3a6acb

andrewkroh force-pushed the feature/mage-refactor branch from 8e72c60 to c3a6acb Compare January 2, 2019 10:02

houndci-bot reviewed Jan 2, 2019

View reviewed changes

ph reviewed Jan 2, 2019

View reviewed changes

andrewkroh added 6 commits January 2, 2019 11:24

Fix targets in jenkins_ci.ps1

4092ddd

Debug Filebeat on Jenkins

35a95fd

Fix journalbeat package target deps

56a0cda

CreateDir for the Docker image file

ad377ca

Enable debug logging for the metricbeat build

fe83244

Fix metricbeat packaging module glob

e3ded46

cwurm mentioned this pull request Jan 4, 2019

Remove _meta/kibana.generated symlink #9892

Merged

ruflin reviewed Jan 7, 2019

View reviewed changes

This was referenced Jan 7, 2019

Cherry-pick #9892 to 6.x: Remove _meta/kibana.generated symlink #9917

Merged

Cherry-pick #9892 to 6.6: Remove _meta/kibana.generated symlink #9918

Merged

andrewkroh mentioned this pull request Jan 23, 2019

Metricbeat Technical Debt meta issue #10238

Closed

10 tasks

andrewkroh mentioned this pull request Apr 8, 2019

Setup an independent build for x-pack/winlogbeat #11679

Merged

andrewkroh mentioned this pull request May 1, 2019

[WIP] Introduce local fields generation #9507

Closed

ycombinator mentioned this pull request Jan 29, 2020

Re-enabling golang integration tests #15819

Merged

urso added the Stalled label Feb 28, 2020

andrewkroh closed this Mar 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert all projects to use Mage #9842

Convert all projects to use Mage #9842

andrewkroh commented Jan 2, 2019 •

edited

Loading

houndci-bot Jan 2, 2019

houndci-bot Jan 2, 2019

houndci-bot Jan 2, 2019

houndci-bot Jan 2, 2019

houndci-bot Jan 2, 2019

houndci-bot Jan 2, 2019

houndci-bot Jan 2, 2019

houndci-bot Jan 2, 2019

houndci-bot Jan 2, 2019

houndci-bot Jan 2, 2019

houndci-bot Jan 2, 2019

houndci-bot Jan 2, 2019

ph Jan 2, 2019

ruflin left a comment

ruflin Jan 7, 2019

ruflin Jan 7, 2019

ruflin Jan 7, 2019

ruflin Jan 7, 2019

ruflin Jan 7, 2019

urso commented Feb 28, 2020


		type Check mg.Namespace

		// Check checks that code is formatted and generated files are up-to-date.

		@@ -0,0 +1,35 @@
		// Licensed to Elasticsearch B.V. under one or more contributor

Convert all projects to use Mage #9842

Convert all projects to use Mage #9842

Conversation

andrewkroh commented Jan 2, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

urso commented Feb 28, 2020

andrewkroh commented Jan 2, 2019 •

edited

Loading