Add fileset for AWS ELB logs

[jsoriano]

Add fileset for AWS ELB logs that uses the S3 input to
collect ELB access logs.

It supports the three kinds of Load Balancers offered by
AWS: ELB, Application LB and Network LB.

Includes Terraform configuration to deploy example
scenarios.

TCP logs are only tested with examples from the
documentation, I didn't manage to collect logs using
the Terraform scenario.

Fixes #12893

[jsoriano]

This fileset will need gzip support in the s3 input, in progress in #13980

[jsoriano]

Failure in Jenkins is not related. @kaiyan-sheng would you like to take another look before merging?

[kaiyan-sheng]

@jsoriano LGTM, one thing I found after finally getting some logs from alb:

{
  "_index": "filebeat-8.0.0-2019.10.15-000001",
  "_type": "_doc",
  "_id": "1ee1b4d775-000000000626",
  "_version": 1,
  "_score": null,
  "_source": {
    "container": {
      "id": "elasticloadbalancing"
    },
    "agent": {
      "hostname": "KaiyanMacBookPro",
      "id": "7578d49c-6588-4843-85cc-ad3859f99ed1",
      "type": "filebeat",
      "ephemeral_id": "2e0a020c-0362-4b23-a1ce-be6263a5186b",
      "version": "8.0.0"
    },
    "log": {
      "file.path": "https://test-alb-log-s3.s3-us-east-1.amazonaws.com/AWSLogs/627959692251/elasticloadbalancing/us-east-1/2019/10/15/627959692251_elasticloadbalancing_us-east-1_app.andrewvc-elbv2-test.8e858b04373118d3_20191015T1405Z_3.224.132.43_39xza92o.log.gz",
      "offset": 626
    },
    "fileset": {
      "name": "elb"
    },
    "message": "http 2019-10-15T14:01:24.711297Z app/andrewvc-elbv2-test/8e858b04373118d3 89.248.169.17:52370 - -1 -1 -1 400 - 0 0 \"- http://andrewvc-elbv2-test-1670032515.us-east-1.elb.amazonaws.com:80- -\" \"-\" - - - \"-\" \"-\" \"-\" - 2019-10-15T14:01:24.611000Z \"-\" \"-\" \"-\"\n",
    "error": {
      "message": "Provided Grok expressions do not match field value: [http 2019-10-15T14:01:24.711297Z app/andrewvc-elbv2-test/8e858b04373118d3 89.248.169.17:52370 - -1 -1 -1 400 - 0 0 \\\"- http://andrewvc-elbv2-test-1670032515.us-east-1.elb.amazonaws.com:80- -\\\" \\\"-\\\" - - - \\\"-\\\" \\\"-\\\" \\\"-\\\" - 2019-10-15T14:01:24.611000Z \\\"-\\\" \\\"-\\\" \\\"-\\\"\\n]"
    },
    "cloud": {
      "provider": "aws",
      "region": "us-east-1"
    },
    "input": {
      "type": "s3"
    },
    "@timestamp": "2019-10-15T14:05:50.756Z",
    "ecs": {
      "version": "1.1.0"
    },
    "service": {
      "type": "aws"
    },
    "host": {
      "hostname": "KaiyanMacBookPro",
      "os": {
        "build": "17G7024",
        "kernel": "17.7.0",
        "name": "Mac OS X",
        "family": "darwin",
        "version": "10.13.6",
        "platform": "darwin"
      },
      "name": "KaiyanMacBookPro",
      "id": "9C7FAB7B-29D1-5926-8E84-158A9CA3E25D",
      "architecture": "x86_64"
    },
    "aws": {
      "s3": {
        "bucket": {
          "name": "test-alb-log-s3",
          "arn": "arn:aws:s3:::test-alb-log-s3"
        },
        "object.key": "AWSLogs/627959692251/elasticloadbalancing/us-east-1/2019/10/15/627959692251_elasticloadbalancing_us-east-1_app.andrewvc-elbv2-test.8e858b04373118d3_20191015T1405Z_3.224.132.43_39xza92o.log.gz"
      }
    },
    "event": {
      "module": "aws",
      "dataset": "aws.elb"
    }
  },
  "fields": {
    "suricata.eve.timestamp": [
      "2019-10-15T14:05:50.756Z"
    ],
    "@timestamp": [
      "2019-10-15T14:05:50.756Z"
    ]
  },
  "sort": [
    1571148350756
  ]
}

This can be fixed in a separate PR. The rest of the logs got parsed just fine besides this one special case.

[jsoriano]

Failing test is not related, merging.

[kaiyan-sheng]

Do we have any plans to add a dashboard for elb logs?

[jsoriano]

Do we have any plans to add a dashboard for elb logs?

Yes, and I also want to investigate why tcp load balancers don't log anything on the terraform scenario.

[tbragin]

@jsoriano I'm writing the 7.5 release blog - have we added a dashboard for ELB logs in 7.5?

[jsoriano]

@tbragin no, there is no dashboard yet