Add mapping network event guidance doc

[ebeahan]

Add a new documentation page, Mapping Network Events, which provides guidance and best practices for mapping network-related events to ECS.

Goals:

• Provide better guidance on the usage of source/destination and when client/server should also be populated.
• Provide example application of the network.* & related.* field sets.
• Document that network.protocol is always populated when an event is categorized as: event.category:network + event.type:protocol.

Relates to #948

Docs Preview

[dainperkins]

I don't quite understand the source/dest vs client/server differentiation you are shooting for (assuming complete session visibility for the purposes of the discussion)

• From a strict TCP standpoint they should be the same (assuming you can determine which side is the syn, and which side is the ack, or which station sent the first packet - which from e.g. a netflow or packetbeat perspective is usually not a problem - aws vpcflow as the exception unless things have changed. w/o syn/ack or first packet its a miserable process to determine which side is the initiator (aside from e.g. well known ports - but get into mongodb and its literally a toss up as everything is high random ports)

• From a udp standpoint its also going to depend on where the first packet came from, the difference being the 5 tuple isn't correlated with a higher level session, but theres still a dns request (randomPortA to 53) and response (53 -> randomPortA)

Its been a really long time since I looked at netflow5 unidirectional flows, but, the observer (assuming no asymmetric routing) should still know the source and destination and track the same across src->dst & dst->src connection (both flows should use the same src/dst identifiers)

Is there some use case / corner case I'm missing (or maybe just some esoteric blind spot I can't get past?)

[webmat]

@dainperkins what we're trying to do here is simply to more clearly establish that we want source+destination to always be populated. Then when it makes sense / is desired, folks can also use client+server.

So this is just to help make sure people can reliably depend on source+destination to be populated across the hundreds of data sources they may have to deal with.

Now if you see a problem with the current wording, we can always improve on that.

[dainperkins]

maybe call out the specific ips to include in the related.ip field in the example?

The related.ip field captures all the IPs present in the event in a single array, in this case including source.ip, destination.ip, as well as any IPs included in the dns.answer field:

Should we include e.g. pipeline code to show how to determine if a dns.answer is an IP?

[webmat]

I don't think we should provide example code here. This would be fine in a blog post, but I think this will be too much of a maintenance burden.

[ebeahan]

maybe call out the specific ips to include in the related.ip field in the example?

Good call out. I can definitely add a little more to the explanation.

Should we include e.g. pipeline code to show how to determine if a dns.answer is an IP?

I approached this as an implementation-neutral reference of sorts. I've kept this focused on what the final mappings should be and avoided being too prescriptive about the how.

[dainperkins]

Would it make sense to use an ECS version of the source data?

[ebeahan]

I chose an example event that wasn't already mapped to ECS trying to capture each steps in the exercise of mapping the event's network-related values.

Would breaking down an event that's already mapped to ECS (e.g. a packetbeat DNS event) for the example be more useful?

[dainperkins]

Actually I think the original makes more sense now that I've thought about it. I think the example data threw me for a loop as any request/response protocol analysis will have clear source/destination linking to client/server..

basically for anything besides unidirectional flows (meraki netflow, aws vpcflow) & broadcast requests (dhcp) copy source/destination to client/server.

for the corner cases just use source/dest.

I think I may be getting wrapped around an axle that is as much about base semantics, as it is about using the data e.g. the network siem page displays...

[dainperkins]

@ebeahan & @webmat
Ok I think I get it - the basis for this is the idea that a given network observer will NOT be indicating the session role of a given unidirectional session? that is to say we are not treating source and destination pairs as e.g. unidirectional flow indicators with no concept of transport or application session (i,e, tcp: syn|syn-ack or udp: e.g. dns request/response)

for example (stripped of irrelevant data, and no transport or app session - aws vpc flows used to look like this, unsure if they still do)

TCP Example

Event 1:
source: 192.168.1.10:42113 -> destination: 192.168.1.50:443. 1 packet, 10 bytes (syn)

Event 2:
source: 192.168.1.50:443 -> destination: 192168.1.10:42113 1 packet 10 bytes (syn-ack)

Event 3:
source: 192.168.1.10:42113 -> destination: 192.168.1.50:443. 1 packet, 10 bytes (ack)

or for UDP

Event 1:
source: 192.168.1.10:42113 -> destination: 192.168.1.50:53. 1 packet, 10 bytes (dns request)

Event 2:
source: 192.168.1.50:53 -> destination: 192168.1.10:42113 1 packet 10 bytes (dns response)

[webmat]

@ebeahan I think we will need to revisit a lot of the other static asciidoc pages. For example there's overlap between some old pages and newer ones. We'll also need to consider the different personas that consume ECS, and help guide users to the different sections in a more useful manner, right from the top.

But for now, I think we should move this page under "Using ECS", perhaps as the last sub-page. It's a bit weird to move it to this section, but the "Migrating" and "Additional" sections are too far down IMO, and not as up to date. So adding this new page down there runs the risk of not being noticed.

[ebeahan]

Ok I think I get it - the basis for this is the idea that a given network observer will NOT be indicating the session role of a given unidirectional session?

The aim is to provide a reference where we can refer users for how network events should be mapped. @webmat captured a list in the second checklist item from #948 which served as the foundation.

[webmat]

Great content, this will be helpful :-)

I have a few comments below on content suggestions, let me know what you think.

[webmat]

I think we should name this section "Categorizing the event" or something. The act of hardcoding these values per type of event is kind of a different thing than the mapping of values to different fields that have been discussed so far.

We could also adjust the wording in the first paragraph of this section to also link to the categorization section.

[webmat]

Perhaps we could also have the network.* section and event.* section grouped together somehow, as the former is also related to categorization, as you explain wrt event.category: protocol.

[ebeahan]

I merged the two sections by dovetailing the network.* section onto the event categorization portion.

I also realized there could be additional event.* fields mapped that this example doesn't cover, so I also made some adjustments to focus on event categorization over the event.* field set in general.

[webmat]

Great, thanks for the adjustments

[webmat]

I think we should conclude with a section that shows the full event, that includes all snippets discussed so far.

I agree with you not getting in the dns.* stuff as mentioned above, but the section could be present in the JSON event, with an ellipsis.

So something like this:

{
  "event": {
    "category": [
      "network"
    ],
    "type": [
      "connection",
      "protocol"
    ],
    "kind": "event"
  },
  "network": {
    "protocol": "dns",
    "type": "ipv4",
    "transport": "udp"
  },
  "source": {
    "ip": "192.168.86.222",
    "port": 54162
  },
  "destination": { full section as above },
  "client": { full section as above },
  "server": { full section as above },
  "dns": { ... },   <= actual ellipsis, we're not getting into DNS
  "related": { "ip": [ "192.168.86.222", "192.168.86.1", "93.184.216.34" ] },
  "zeek": { "ts":1599775747.53056, ... } <= original fields can optionally be kept around as custom fields

[webmat]

I think this is useful. Not sure if it gets all the way to demonstrate the subtleties, though.

It's good we're using DNS for this, because it lets us get into subtleties between client/server vs source/destination.

Here's looking at DNS traffic with source/destination vs looking at it with client/server.

source/destination

source.ip	destination.ip	event
192.168.86.222	192.168.86.1	DNS question 1
192.168.86.1	192.168.86.222	DNS answer 1
192.168.86.42	192.168.86.1	DNS question 2
192.168.86.1	192.168.86.42	DNS answer 2

client/server

client.ip	server.ip	event
192.168.86.222	192.168.86.1	DNS question 1
192.168.86.222	192.168.86.1	DNS answer 1
192.168.86.42	192.168.86.1	DNS question 2
192.168.86.42	192.168.86.1	DNS answer 2

The latter is more helpful, when the client & server roles flip around on query and response events.

But it also implies that in these cases the "copy source and destination" values needs to adjust where stuff is copied, depending on the event:

DNS question: source.ip => client.ip, destination.ip => server.ip
DNS answer: source.ip => server.ip, destination.ip => client.ip

[dainperkins]

I figured out where I'm getting bogged down...

I strongly dislike mixing unidirectional and bidirectional data in one bucket. I know it completely skews ML analysis (unidirectional features don't work in bidirectional, and vice versa), and I think it plays havoc with things like the network tab on the SIEM if filtering / bidirectional normalization isn't being done.

How to deal with that I'm not really sure...

[ebeahan]

@webmat yes it's good idea to call out that while source/destination will flip, client/server assignments will apply throughout the transaction.

@dainperkins I don't think we're introducing any new concepts or guidance on unidirectional vs. bidirectional data here, but trying to better state what the existing guidance already is in a write-up that can live in the ECS docs.

Are these issues something we can improve through better ECS guidance here? Or would that topic be better as a separate discussion?

[webmat]

LGTM thanks for the adjustments. This is looking great!