Remove nil-zero metrics and linux-exclusive metrics from Metricbeat

[fearful-symmetry]

What does this PR do?

This PR has two main parts:

• If a metric is currently reporting a "null zero" (that is, the metric doesn't exist for an OS, but we report a 0 instead of nothing) for a given metric on a given OS, add some conditional logic so we don't report it.
• If we're reporting a group of metrics that is highly OS-specific, add that metric group to the linux module, and don't report it if we're running under fleet. This is because we want to move some of these metrics, but don't want to break metricbeat. As Fleet is beta, we want to introduce some needed changes while we can.

Also note, I'm actively working on this PR, so it might be a tad rough right now, but I want eyes on it.

Where do we go from here?

• A large part of this PR is essentially a holdover until we can make breaking changes to reported fields in metricbeat. Once we hit 8.0, we can remove the IsAgent logic we have to prevent breaking field changes in metricbeat.
• Once this is merged, the additional linux metricsets will be added to the Fleet integration.

Open Questions

• We don't want to pull metrics out from under user's feet. To accomplish this, any linux-exclusive metrics we're reporting in system now have their own module in linux. After this PR, we'll add these new linux modules to the Linux integration within fleet. From the perspective of metricbeat, we're now reporting identical data in two places: in the default system module, and in new metricsets within the linux integration. Is this the best route? Is there a better way to handle metrics we're trying to relocate?

Why is it important?

This is the result of a lot of work between @mukeshelastic and I, with two goals:

• Create a leaner system Integration for fleet that contains only cross-compatible datasets
• Clean up the metricsets so we don't report "null zeros" on operating systems where a given metric doesn't exist.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• Find a way to run this under Agent
• Test with the rest of the elastic stack
• Test under Windows

How to test this PR locally

• Pull PR and build via mage package to generate a system-specific build object. You can use the PLATFORMS environment variable to constrain to a given OS/arch pair like linux/amd64.
• Running as metricbeat (no Fleet stack), test a few things depending on your OS:
  • Linux:
    • Make sure the cpu, memory, diskio, process, filesystem, fsstat fields haven't changed relative to data.json
    • Test the new memory and iostat linux metricsets for all the expected fields.
  • Windows:
    • Using the spreadsheet linked above, check the results of the cpu, memory, diskio, process, filesystem, andfsstat metricsets to make sure all the fields reported as Incorrect 0 value? on the spreadsheet are no longer reported.
• If you can somehow connect your custom build to Agent/Fleet, check to make sure the IsAgent constraints are working and we're not reporting additional data.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #21457 updated]

• Start Time: 2020-10-06T16:42:47.631+0000

• Duration: 94 min 57 sec

Test stats 🧪

Test	Results
Failed	0
Passed	16294
Skipped	1347
Total	17641

[fearful-symmetry]

Alright, looks like I need to fix the python tests...

[faec]

I don't have the broader context to evaluate the migration approach so I hope other reviewers chime in, but taking the business logic as a given, the content of the change looks good to me

[kaiyan-sheng]

Thanks for adding the part to make sure host.cpu.pct exists.

[ph]

@fearful-symmetry I've looked a core logic seems OK to me, but I am not the most familiar with the metric modules. @exekias or @andrewkroh might be good to ping.

[fearful-symmetry]

New diskio document with Agent:

    "system": {
      "diskio": {
        "name": "sda2",
        "read": {
          "time": 27139320,
          "bytes": 109501018112,
          "count": 2889700
        },
        "write": {
          "count": 74365389,
          "time": 272761008,
          "bytes": 1864660139008
        },
        "io": {
          "time": 48258392
        }
      }

New memory document with Agent:

  "system": {
      "memory": {
        "swap": {
          "total": 7897870336,
          "used": {
            "pct": 0.0085,
            "bytes": 67371008
          },
          "free": 7830499328,
          "in": {
            "pages": 11339
          },
          "out": {
            "pages": 43449
          },
          "readahead": {
            "pages": 6855,
            "cached": 4852
          }
        },
        "total": 15620788224,
        "used": {
          "pct": 0.9865,
          "bytes": 15409446912
        },
        "free": 211341312,
        "actual": {
          "free": 9764274176,
          "used": {
            "pct": 0.3749,
            "bytes": 5856514048
          }
        }
      }
    },