Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[libbeat] Add image.id and account.id to EC2 add_cloud_metadata #10914

Closed
wants to merge 1 commit into from

Conversation

jbagel2
Copy link

@jbagel2 jbagel2 commented Feb 22, 2019

Some addition data inside the same instance meta-data document is very helpful with node spanning many accounts.

@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

@elasticcla
Copy link

Hi @jbagel2, we have found your signature in our records, but it seems like you have signed with a different e-mail than the one used in yout Git commit. Can you please add both of these e-mails into your Github profile (they can be hidden), so we can match your e-mails to your Github profile?

@jbagel2
Copy link
Author

jbagel2 commented Feb 27, 2019

@elasticcla Done and done

Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I anticipate that we will see more requests similar to this over time. If we continue to add fields to the processor it will get bloated over time. At some point this processor will need to allow additional fields to be made configurable. Like

- add_cloud_metadata:
    extra_fields:
      ec2: {privateIp: private.ip}
      digitalocean: {interfaces.private.ipv4.ip_address: private.ip}

With that being said, I think the fields being added in this PR are generally useful to everyone. The account ID is specified in Elastic Common Schema as cloud.account.id. The image ID isn't part of the schema, but we might want to add it.

  • Can you please look at the unit tests for the other providers to see if their data includes an image ID or account ID and note it in this PR or update their implementations to populate the fields.

  • The documentation needs to be updated to show the new fields for EC2. It's in libbeat/docs/processors-using.asciidoc.

  • The EC2 unit test needs updated.

  • The instance.id field needs added to add_cloud_metadata/_meta/fields.yml. Then make update must be run.

@@ -32,6 +32,8 @@ func newEc2MetadataFetcher(config *common.Config) (*metadataFetcher, error) {
"machine_type": c.Str("instanceType"),
"region": c.Str("region"),
"availability_zone": c.Str("availabilityZone"),
"accountId": c.Str("accountId"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should use account.id to align with ECS.

Copy link
Contributor

@odacremolbap odacremolbap May 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andrewkroh
do you mean cloud.account.id?

I guess some other fields in there could also fit the cloud ecs schema
https://www.elastic.co/guide/en/beats/filebeat/current/add-cloud-metadata.html

(looks that this task is also needed for most other cloud metadata processors)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cloud. part gets added later. So changing the left side accoundId to account.id should make this match with the ECS cloud.account.id field.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant there are some other fields that looks not ECS compliant, like machine_type (should be machine.type), instance_id(should be instance.id)

am I missing something there?
I'll go through all cloud metadata processors, since I think there are a number of non ECS compliant schemas

@@ -32,6 +32,8 @@ func newEc2MetadataFetcher(config *common.Config) (*metadataFetcher, error) {
"machine_type": c.Str("instanceType"),
"region": c.Str("region"),
"availability_zone": c.Str("availabilityZone"),
"accountId": c.Str("accountId"),
"ami": c.Str("imageId"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ami is not a vendor neutral term so I wouldn't want to put it into the schema shared by all of the provider implementations. image.id would be better in my opinion. This field will need to be added to the _meta/fields.yml that's in this package.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andrewkroh @webmat sounds like cloud.image.id would be a nice ECS candidate.
WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@odacremolbap Interesting. I'll note it down as a thing to look into.

Volume snapshots & images are available on other technology platforms than cloud providers, however. So not sure the images will end up nested under cloud necessarily.

But you can still feel free to proceed with this for now. Beats has many non-ECS fields peppered inside other ECS field sets. A bit of a risk, but not the end of the world :-)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@odacremolbap Best directly open an issue here to discuss: https://github.com/elastic/ecs

@andrewkroh andrewkroh added the ecs label Feb 28, 2019
@andrewkroh andrewkroh changed the title Just a quick addition of aws account and imageId to cloud-meta-data [libbeat] Add image.id and account.id to EC2 add_cloud_metadata Feb 28, 2019
@andrewkroh
Copy link
Member

andrewkroh commented Feb 28, 2019

I just noticed this PR is opened against 6.6. It would be better for us if you could change the target to be master (you can edit the target from this page). Then we can backport it to the other branches that need it.

@jbagel2
Copy link
Author

jbagel2 commented Apr 3, 2019

Update, This is still high priority for us. I have just not had cycles to get the requested changes made. You may get wind of this from elastic internal, just FYI.

@ruflin ruflin added Team:Integrations Label for the Integrations team [zube]: In Review labels Apr 4, 2019
@odacremolbap
Copy link
Contributor

@jbagel2 I'll take this one over if you don't mind

SHA3_512 HashType = "sha3_512"
SHA512 HashType = "sha512"
SHA512_224 HashType = "sha512_224"
SHA512_256 HashType = "sha512_256"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use ALL_CAPS in Go names; use CamelCase

SHA3_384 HashType = "sha3_384"
SHA3_512 HashType = "sha3_512"
SHA512 HashType = "sha512"
SHA512_224 HashType = "sha512_224"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use ALL_CAPS in Go names; use CamelCase

SHA3_224 HashType = "sha3_224"
SHA3_256 HashType = "sha3_256"
SHA3_384 HashType = "sha3_384"
SHA3_512 HashType = "sha3_512"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use ALL_CAPS in Go names; use CamelCase

SHA384 HashType = "sha384"
SHA3_224 HashType = "sha3_224"
SHA3_256 HashType = "sha3_256"
SHA3_384 HashType = "sha3_384"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use ALL_CAPS in Go names; use CamelCase

SHA256 HashType = "sha256"
SHA384 HashType = "sha384"
SHA3_224 HashType = "sha3_224"
SHA3_256 HashType = "sha3_256"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use ALL_CAPS in Go names; use CamelCase

SHA224 HashType = "sha224"
SHA256 HashType = "sha256"
SHA384 HashType = "sha384"
SHA3_224 HashType = "sha3_224"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use ALL_CAPS in Go names; use CamelCase

const (
BLAKE2B_256 HashType = "blake2b_256"
BLAKE2B_384 HashType = "blake2b_384"
BLAKE2B_512 HashType = "blake2b_512"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use ALL_CAPS in Go names; use CamelCase

// Enum of hash types.
const (
BLAKE2B_256 HashType = "blake2b_256"
BLAKE2B_384 HashType = "blake2b_384"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use ALL_CAPS in Go names; use CamelCase


// Enum of hash types.
const (
BLAKE2B_256 HashType = "blake2b_256"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use ALL_CAPS in Go names; use CamelCase

}

// BuildGoDaemon builds the go-daemon binary (use crossBuildGoDaemon).
func BuildGoDaemon() error {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

func name will be used as build.BuildGoDaemon by other packages, and that stutters; consider calling this GoDaemon

OverwritePipelines bool `config:"overwrite_pipelines"`
}

type Registry struct {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported type Registry should have comment or be unexported

func (fs *Fileset) GetPipeline(esVersion common.Version) (pipelineID string, content map[string]interface{}, err error) {
path, err := applyTemplate(fs.vars, fs.manifest.IngestPipeline, false)
// GetPipelines returns the JSON content of the Ingest Node pipeline that parses the logs.
func (fs *Fileset) GetPipelines(esVersion common.Version) (pipelines []pipeline, err error) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported method GetPipelines returns unexported type []fileset.pipeline, which can be annoying to use

OverwritePipelines bool `config:"overwrite_pipelines"`
}

type Registry struct {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported type Registry should have comment or be unexported

LogType = "log"
StdinType = "stdin"
RedisType = "redis"
UdpType = "udp"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const UdpType should be UDPType

LogType = "log"
StdinType = "stdin"
RedisType = "redis"
UdpType = "udp"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const UdpType should be UDPType


type File struct {
*os.File
}

func (File) Continuable() bool { return true }
func (File) HasState() bool { return true }
func (f File) Removed() bool { return file.IsRemoved(f.File) }

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported method File.Removed should have comment or be unexported

LogType = "log"
StdinType = "stdin"
RedisType = "redis"
UdpType = "udp"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const UdpType should be UDPType

@@ -30,6 +31,8 @@ import (
var Version = "9.9.9"
var Name = "mockbeat"

var Settings = instance.Settings{Name: Name, Version: Version}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported var Settings should have comment or be unexported

return false
}

// Recursively generates the correct key based on the dots

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment on exported function GenerateKey should be of the form "GenerateKey ..."

return nil
}

func LoadFieldsYaml(path string) (Fields, error) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported function LoadFieldsYaml should have comment or be unexported


type DynamicType struct{ Value interface{} }

func (d *DynamicType) Unpack(s string) error {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported method DynamicType.Unpack should have comment or be unexported

Value string `config:"value"`
}

type DynamicType struct{ Value interface{} }

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported type DynamicType should have comment or be unexported


retError = extractError(result)
return resp.StatusCode, result, retError
// Implements RoundTrip interface

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment on exported method Connection.RoundTrip should be of the form "RoundTrip ..."

return resp.StatusCode, result, retError
}

// Sends an application/json request to Kibana with appropriate kbn headers

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment on exported method Connection.Send should be of the form "Send ..."

}

var (
ErrESVersionNotSupported = errors.New("ILM is not supported by the Elasticsearch version in use")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported var ErrESVersionNotSupported should have comment or be unexported

} else {
importVia = useKibana
}
return errors.New("kibana configuration missing for loading dashboards.")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

error strings should not be capitalized or end with punctuation or a newline

})
}

func ScopedIsUnique() UniqScopeTracker {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported function ScopedIsUnique should have comment or be unexported

return c.access().Remove(name, idx, configOpts...)
}

func (c *Config) Has(name string, idx int) (bool, error) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported method Config.Has should have comment or be unexported

@@ -188,6 +188,14 @@ func (c *Config) PathOf(field string) string {
return c.access().PathOf(field, ".")
}

func (c *Config) Remove(name string, idx int) (bool, error) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported method Config.Remove should have comment or be unexported

err error
}

// New returns a new timeout reader from an input line reader.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment on exported function NewTimeoutReader should be of the form "NewTimeoutReader ..."

errTimeout = errors.New("timeout")
)

// timeoutProcessor will signal some configurable timeout error if no

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment on exported type TimeoutReader should be of the form "TimeoutReader ..." (with optional leading article)

lineEndingFunc func(*StripNewline, []byte) int
}

// New creates a new line reader stripping the last tailing newline.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment on exported function NewStripNewline should be of the form "NewStripNewline ..."

Terminator LineTerminator
}

// New creates a new Encode reader from input reader by applying

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment on exported function NewEncodeReader should be of the form "NewEncodeReader ..."

"github.com/elastic/beats/libbeat/reader/readfile/encoding"
)

// Reader produces lines by reading lines from an io.Reader

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment on exported type EncoderReader should be of the form "EncoderReader ..." (with optional leading article)

type matcher func(last, current []byte) bool

var (
sigMultilineTimeout = errors.New("multiline timeout")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

error var sigMultilineTimeout should have name of the form errFoo

"github.com/elastic/beats/libbeat/reader/readfile"
)

// MultiLine reader combining multiple line events into one multi-line event.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment on exported type Reader should be of the form "Reader ..." (with optional leading article)


// WithBeatMeta adds beat meta information as builtin fields to a processing pipeline.
// The `key` parameter defines the field to be used.
func WithBeatMeta(key string) modifier {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported func WithBeatMeta returns unexported type processing.modifier, which can be annoying to use

}, nil
}

func (m *MetricSet) Fetch() ([]common.MapStr, error) {
func (m *MetricSet) Fetch(reporter mb.ReporterV2) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported method MetricSet.Fetch should have comment or be unexported


// +build !linux

package socket_summary

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use an underscore in package name


// +build linux

package socket_summary

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use an underscore in package name


// +build linux

package socket_summary

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use an underscore in package name

}

// KeyPattern contains the information required to query keys
type KeyPattern struct {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

type name will be used as key.KeyPattern by other packages, and that stutters; consider calling this Pattern

"github.com/elastic/beats/libbeat/outputs"
"github.com/elastic/beats/libbeat/testing"
)

func GenTestOutputCmd(name, beatVersion string) *cobra.Command {
func GenTestOutputCmd(settings instance.Settings) *cobra.Command {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported function GenTestOutputCmd should have comment or be unexported

@@ -27,18 +27,18 @@ import (
"github.com/elastic/beats/libbeat/cmd/instance"
)

func GenTestConfigCmd(name, version string, beatCreator beat.Creator) *cobra.Command {
func GenTestConfigCmd(settings instance.Settings, beatCreator beat.Creator) *cobra.Command {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported function GenTestConfigCmd should have comment or be unexported

@@ -30,14 +30,18 @@ import (
"github.com/elastic/beats/libbeat/common/cfgwarn"
"github.com/elastic/beats/libbeat/logp"

conf "github.com/elastic/beats/journalbeat/config"
"github.com/elastic/beats/journalbeat/config"
_ "github.com/elastic/beats/journalbeat/include"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a blank import should be only in a main or test package, or have a comment justifying it

"github.com/elastic/beats/heartbeat/beater"
_ "github.com/elastic/beats/heartbeat/monitors/defaults"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a blank import should be only in a main or test package, or have a comment justifying it

@@ -44,3 +44,4 @@ func (p Pipe) Name() string { return p.File.Name() }
func (p Pipe) Stat() (os.FileInfo, error) { return p.File.Stat() }
func (p Pipe) Continuable() bool { return false }
func (p Pipe) HasState() bool { return false }
func (p Pipe) Removed() bool { return false }

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported method Pipe.Removed should have comment or be unexported

// specific language governing permissions and limitations
// under the License.

package add_observer_metadata

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use an underscore in package name

// specific language governing permissions and limitations
// under the License.

package add_observer_metadata

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use an underscore in package name

// specific language governing permissions and limitations
// under the License.

package add_observer_metadata

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use an underscore in package name

// specific language governing permissions and limitations
// under the License.

package add_observer_metadata

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use an underscore in package name

// specific language governing permissions and limitations
// under the License.

package add_observer_metadata

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use an underscore in package name

// specific language governing permissions and limitations
// under the License.

package add_observer_metadata

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use an underscore in package name

// specific language governing permissions and limitations
// under the License.

package add_observer_metadata

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use an underscore in package name

// specific language governing permissions and limitations
// under the License.

package add_observer_metadata

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use an underscore in package name

// specific language governing permissions and limitations
// under the License.

package add_observer_metadata

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use an underscore in package name

@odacremolbap odacremolbap force-pushed the aws_metadata_additional_fields branch from 59b8e05 to 25ccad7 Compare May 27, 2019 11:20
@odacremolbap
Copy link
Contributor

just noticed that you forked from elastic:6.6
I'll open a new branch with this change at master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants