Log structured monitoring metrics

[andrewkroh]

• Use structured logging for the metrics that are periodically logged.

• Add beat.info.uptime.ms to the list of gauges so that the total value is always reported rather than a difference.

• Made a change to ensure that only non-zero counter values are reported at shutdown (this was bug introduced in my last refactoring). Note that zero-value gauges are reported which kind of makes the "Total non-zero metrics" message misleading.

Log samples:

2017-12-19T11:57:21.086-0500	INFO	[monitoring]	log/log.go:79	Starting metrics logging every 3s
2017-12-19T11:57:24.087-0500	INFO	[monitoring]	log/log.go:106	Non-zero metrics in the last 3s	{"monitoring": {"metrics": {"beat":{"info":{"uptime":{"ms":3004}},"memstats":{"gc_next":4194304,"memory_alloc":2298976,"memory_total":12648056}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":1,"published":1,"total":1}}},"metricbeat":{"file_integrity":{"file":{"events":1,"success":1}}}}}}
2017-12-19T11:57:25.305-0500	INFO	[monitoring]	log/log.go:114	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"info":{"uptime":{"ms":4222}},"memstats":{"gc_next":4194304,"memory_alloc":3336376,"memory_total":20548168}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":1,"published":1,"total":1}}}}}}
2017-12-19T11:57:25.305-0500	INFO	[monitoring]	log/log.go:115	Uptime: 4.226912514s
2017-12-19T11:57:25.305-0500	INFO	[monitoring]	log/log.go:92	Stopping metrics logging.

To show the key structure more clearly here is a pretty-printed version of the logged JSON metrics.

{
  "monitoring": {
    "metrics": {
      "beat": {
        "info": {
          "uptime": {
            "ms": 4222
          }
        },
        "memstats": {
          "gc_next": 4194304,
          "memory_alloc": 3336376,
          "memory_total": 20548168
        }
      },
      "libbeat": {
        "config": {
          "module": {
            "running": 0
          }
        },
        "output": {
          "type": "elasticsearch"
        },
        "pipeline": {
          "clients": 0,
          "events": {
            "active": 1,
            "published": 1,
            "total": 1
          }
        }
      }
    }
  }
}

[andrewkroh]

@ruflin I'm going back on what I said and have "unflattened" the key names.

[ruflin]

@andrewkroh I rebased and merged as I assume that is what you had in mind.

One thing I don't like about having multiple commits in 1 PR is that CI only tests the last commit and shows the others as just green. So in case the second commit is reverted later it's not clear if the first commit without the second would work.