[Filebeat]Azure module - activity logs #13776

narph · 2019-09-24T10:45:51Z

PR is a POC for the azure module in filebeat.
The azure module will contain:

the activitylogs fileset which will retrieve the Azure activity logs.
Requirements are exporting these logs to an event hub (with Kafka feature enabled), steps here
the signinlogs fileset which will retrieve the Sign in logs
the auditlogs fileset which will retrieve the AD audit logs
Requirements are exporting these 2 AD reports to an event hub (with Kafka feature enabled), steps here

GH issue #13385

filebeat/input/kafka/config.go

filebeat/input/kafka/azure_log_patterns.go

filebeat/input/kafka/config.go

filebeat/input/kafka/azure_log_patterns.go

faec

This looks mostly good and I will do a more thorough review after I return from vacation on Tuesday... a question in addition to the other comments: what specifically is restricted to x-pack here? It looks like ~all the code is in OSS and only module config is in x-pack. This would still leave the functionality in OSS, right? If there hasn't been a conversation about where the division lies, we should probably have one

faec · 2019-10-04T14:06:40Z

filebeat/input/kafka/config.go

@@ -50,6 +50,7 @@ type kafkaInputConfig struct {
 	TLS            *tlscommon.Config `config:"ssl"`
 	Username       string            `config:"username"`
 	Password       string            `config:"password"`
+	AzureLogs      string            `config:"azure_logs"`


This field should be documented and validated -- consider following the pattern used for enums like kafka.Version, which is validated by its Unpack function (see libbeat/common/kafka/version.go) and can thereafter be used as an enum constant rather than a string match.

@faec , I have replaced the AzureLogs with YieldEventsFromField property. This property value, if entered will have the json object name that will contain multiple events.
Ex for azure:
{ "records": [ {event1}, {event2}]}
The YieldEventsFromField property value will be "records".
I have also removed any check for the azure type logs and added some validation inside the actual pipeline.
Let me know if you think this property should be handled the same way as "kafka.Version".

faec · 2019-10-04T14:08:37Z

filebeat/input/kafka/input.go

 	handler := &groupHandler{
 		version: input.config.Version,
 		outlet:  input.outlet,
+		logType: input.config.AzureLogs,


I think I follow what's happening here (in generic kafka config there is no AzureLogs field so this gets set to the empty string -> no special processing), but a comment would be nice to clarify that this is a no-op if there's no Azure-specific configuration

AzureLogs have been replaced with a more generic option "YieldEventsFromField" and added some comments there

faec · 2019-10-04T14:10:45Z

filebeat/input/kafka/input.go

+		event := beat.Event{
+			Timestamp: timestamp,
+			Fields: common.MapStr{
+				"message": msg,


Does this work ok? My recollection is that the explicit string conversion (string(message.Value) in the old code) was necessary for a lot of things to work, since otherwise it got interpreted as raw bytes by the backend, which messed up the logs and the indexing.

I had no issues with it so far, let me know if you know a more secure option, happy to change it

exekias · 2019-10-07T15:15:17Z

filebeat/input/kafka/input.go

+	// if azure input, then a check for the message is done regarding the list of events
+
+	var events []beat.Event
+	messages := h.parseMultipleMessages(message.Value)


I understand that you need azure specific code in order to parse a message into multiple events. The rest could be done with a json processor in a pipeline, isn't it?

If that's the case, perhaps it would be better to have a generic way to say: spawn events from list under this JSON field: "records"

Unfortunately that is not possible, I could separate the json elements but they will still be generated as one event. The workaround would have been to create a new processor and a new interface for processors that can return multiple events.

andrewkroh · 2019-10-07T16:45:54Z

x-pack/filebeat/module/azure/activitylogs/test/activity_log_expected.json

+            "module" : "azure",
+            "dataset" : "azure.activitylogs"
+        },
+        "resultType" : "Failure",


I think some of these fields could be mapped into ECS like

resultType -> event.result

callerIpAddress -> source.ip

I'm sure they're are more and maybe you are already planning for some of these mappings.

@andrewkroh , thanks, in the course of identifying the ecs fields. I could not find event.result in https://www.elastic.co/guide/en/beats/filebeat/master/exported-fields-ecs.html. Is it event.outcome or I am not looking in the right list.

You're right. It's event.outcome. I was working from memory on that example.

libbeat/common/kafka/logtype.go

narph · 2019-10-08T12:20:09Z

hi @faec , regarding:

This looks mostly good and I will do a more thorough review after I return from vacation on Tuesday... a question in addition to the other comments: what specifically is restricted to x-pack here? It looks like ~all the code is in OSS and only module config is in x-pack. This would still leave the functionality in OSS, right? If there hasn't been a conversation about where the division lies, we should probably have one

I assumed the reason for adding the kafka input inside the oss is to be reused by other non x-pack modules as well.
I have tried to remove any azure specific logic from the input as I think the input should not know/care about it.
The azure module is in the same location (x-pack) as the rest as it's "type" , aws etc.
Was there any other reason the kafka input is inside the oss?

filebeat/input/kafka/input.go

exekias

Did a first pass, I like how this is looking!

CHANGELOG.next.asciidoc

x-pack/filebeat/module/azure/_meta/docs.asciidoc

x-pack/filebeat/module/azure/_meta/fields.yml

x-pack/filebeat/module/azure/activitylogs/_meta/fields.yml

x-pack/filebeat/module/azure/auditlogs/_meta/fields.yml

x-pack/filebeat/module/azure/module.yml

x-pack/filebeat/module/cisco/shared/gen-ecs-mapping-docs.go

x-pack/filebeat/module/azure/signinlogs/_meta/fields.yml

x-pack/filebeat/module/azure/_meta/config.yml

…filebeat

exekias

I think this is good to go if CI is happy. Thank you for implementing the changes!

Something that would be great to see here (in a different PR) is cloud.* metadata matching the Metricbeat module. That should improve correlation opportunities

narph · 2019-10-10T11:10:23Z

jenkins test this

narph added 6 commits September 19, 2019 12:17

First commit

736a73b

Creating the filebeat module

d99085f

Work on module

6f6b6c5

Work on the azure module

43992b5

Temp commit

9932a5b

Adding azure filebeat input

1bfadb2

houndci-bot reviewed Sep 24, 2019

View reviewed changes

filebeat/input/kafka/config.go Outdated Show resolved Hide resolved

filebeat/input/kafka/config.go Outdated Show resolved Hide resolved

narph self-assigned this Sep 24, 2019

narph added Team:Integrations Label for the Integrations team x-pack Issues and pull requests for X-Pack features. Filebeat Filebeat [zube]: In Progress labels Sep 24, 2019

Modify kafka input

14feac4

houndci-bot reviewed Sep 26, 2019

View reviewed changes

Adding audit filesets

14ea02f

houndci-bot reviewed Oct 4, 2019

View reviewed changes

work on kafka input

75ec856

narph requested a review from faec October 4, 2019 13:19

faec reviewed Oct 4, 2019

View reviewed changes

Fix config

8f156ea

exekias reviewed Oct 7, 2019

View reviewed changes

Work on defining fields and examples

c04a476

andrewkroh reviewed Oct 7, 2019

View reviewed changes

Work on kafka logtype

766026e

houndci-bot reviewed Oct 8, 2019

View reviewed changes

libbeat/common/kafka/logtype.go Outdated Show resolved Hide resolved

houndci-bot reviewed Oct 8, 2019

View reviewed changes

libbeat/common/kafka/logtype.go Outdated Show resolved Hide resolved

Move azure log validation in pipelines

cb3ce2f

exekias reviewed Oct 8, 2019

View reviewed changes

filebeat/input/kafka/input.go Outdated Show resolved Hide resolved

Work on event format

f2a5518

narph mentioned this pull request Oct 8, 2019

Modify kafka input to include parsing multiple messages #13965

Merged

narph added 3 commits October 8, 2019 18:33

Work on activitylogs format

ff26f04

Work on event format

9a2f77b

Merge branch 'master' into azure-filebeat

3faf839

narph marked this pull request as ready for review October 9, 2019 09:37

narph requested a review from a team as a code owner October 9, 2019 09:37

narph requested review from andrewkroh and exekias October 9, 2019 09:38

exekias reviewed Oct 9, 2019

View reviewed changes

narph added 4 commits October 9, 2019 15:38

Work on filesets

f18c856

Merge branch 'master' of https://github.com/elastic/beats into azure-…

948db25

…filebeat

Revert local changes

3d7452a

Revert change

a5b3534

narph requested a review from exekias October 9, 2019 15:19

narph added 3 commits October 9, 2019 23:30

Progress on defining azure module fields

8fb4581

Work on auditlog event format

617b3e5

Merge branch 'master' into azure-filebeat

f01a8e3

exekias approved these changes Oct 10, 2019

View reviewed changes

narph merged commit 0aea418 into elastic:master Oct 10, 2019

zube bot added [zube]: Done and removed [zube]: In Progress labels Oct 10, 2019

narph deleted the azure-filebeat branch October 10, 2019 13:31

narph added the v7.5.0 label Oct 11, 2019

andresrc removed the [zube]: Done label Oct 15, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat]Azure module - activity logs #13776

[Filebeat]Azure module - activity logs #13776

narph commented Sep 24, 2019 •

edited

Loading

faec left a comment

faec Oct 4, 2019

narph Oct 8, 2019

faec Oct 4, 2019

narph Oct 8, 2019

faec Oct 4, 2019

narph Oct 8, 2019

exekias Oct 7, 2019

narph Oct 7, 2019

andrewkroh Oct 7, 2019

narph Oct 8, 2019

andrewkroh Oct 8, 2019

narph commented Oct 8, 2019

exekias left a comment

exekias left a comment

narph commented Oct 10, 2019

[Filebeat]Azure module - activity logs #13776

[Filebeat]Azure module - activity logs #13776

Conversation

narph commented Sep 24, 2019 • edited Loading

faec left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

narph commented Oct 8, 2019

exekias left a comment

Choose a reason for hiding this comment

exekias left a comment

Choose a reason for hiding this comment

narph commented Oct 10, 2019

narph commented Sep 24, 2019 •

edited

Loading