Add mongobeat to list of community beats

[scottcrespo]

mongobeat

Mongobeat discovers instances in a mongo cluster and can be configured to ship multiple document types - from the commands db.stats() and db.serverStatus()

Still an early work in progress as I'm learning more about the beats platform - so far I really enjoy it! I've done my best to follow the contributor guide and conventions. Please let me know if you have any feedback or would like me to make some changes.

[elasticsearch-release]

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

[elasticmachine]

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

[ruflin]

@scottcrespo Thanks a lot for creating a beat. I'm curious how it compares to our mongodb module in Metricbeat? It seems you have some additional db stats? Perhaps it could make sense to merge the two? https://github.com/elastic/beats/tree/master/metricbeat/module/mongodb

[scottcrespo]

Hey @ruflin,

Thanks for merging!

I think the most sensible things to merge from mongobeat are the additional documents returned by MongoDB. For starters, I'd be happy to open a new issue/proposal and submit code to metricbeat to collect db stats.

There are also some other differences between the two, which I'll briefly describe. If any are of interest to you, please let me know, and I can submit more detailed proposals for you to consider.

---

DB Stats

As you've mentioned, the data difference (at this time) is that mongobeat also collects db stats.

For each database in a mongo instance, an event is published after calling db.stats(). Below is an example document.

    "db_stats": {
      "avg_obj_size": 0,
      "collections": 4,
      "data_size": 1168,
      "db": "admin",
      "file_size": 67108864,
      "index_size": 24528,
      "indexes": 3,
      "num_extents": 4,
      "objects": 11,
      "ok": 1,
      "storage_size": 28672
    },
    "type": "mongobeat.db_stats"
}

Additional Mongo Document Types

I plan to add some additional metrics that can be optionally collected, such as collStats and shardConnPoolStats.

So, I can propose and submit for the other mongo document types as they're implemented.

Concurrency Patterns

I haven't looked too deep into the concurrency patterns used by metric beat. But essentially mongobeat monitors each document type (i.e. status and dbstats) on separate goroutines, so one call can't block the other.

Centralized Mode

mongobeat can optionally run as a centralized monitoring service. It connects to "seed instances," discovers additional nodes in the cluster, and establishes direct connections with each instance. Then, each node in the cluster is monitored on its own goroutine so a non-responsive/latent instance doesn't block the reporting of others.

Soon, I'm going into implement periodic discovery, so every n seconds mongobeat will search for additional nodes in the cluster.

Minor-Version Compatibility

mongobeat is attempting to support minor-version compatibility across Mongo 3.x (i.e. 3.0, 3.2, 3.4). Thus, each minor version has the same document schema in Elasticsearch, and Kibana users can compare data from different mongo versions without data discrepancies. It will also allow for easier sharing of visualizations and dashboards.

Mongo 2.0 Support

I'll probably add support for Mongo2.x data schemas at some point. It would be an additional benefit if 2.x data can be transformed into a common format with 3.x documents for complete backwards compatibility.

Major-Version Compatibility

I've created an explicit modeling layer with Golang structs instead using the schema package. My intuition is that once 3.x and 2.x are fully implemented, we may be able to use struct composition to create a cross-major-version compatible data schema.

[ruflin]

@scottcrespo Thanks for the detailed elaboration of your thoughts. Here some answers:

• DB Stats: It would be awesome if you could contribute the db_stats module. We would be more then happy to have it as part of Metricbeat.
• collStats and shardConnPoolStats seems definitively worth 2 additional Metricsets for the MongoDB module.
• Concurrency: There is one go routing per metricset + host so they will not conflict with each other: https://github.com/elastic/beats/blob/master/metricbeat/beater/module.go#L131
• Centralized mode: We recommend to install metricbeat on the edge nodes directly with for example mongodb. So each instance of a mongodb cluster is monitored by an elasticsearch instance. Centralised monitoring would be possible through definining multiple hosts, but it does not do a dynamic lookup.
• Version compatibility: We also have the intention of making the schema compatible across versions. One tricky thing we encountered so far in MongoDB is that the schema not only changes based on the version but also on the storage system used: Metricbeat MongoDB module improvements #2999
• 2.x Support: I'm not aware of the user bases of the different mongodb versions, but as usual I'm quite sure there are lots of people still on 2.x. So having support for it would be nice.
• Major-version compatibility: Can you elaborate / link to the way you used the modeling? Sounds very interesting.

[scottcrespo]

@ruflin

Thanks for all the feedback!

DB Stats

I'll open an issue and start working on DB Stats for metricbeat.

collStats and sharedConnPoolStats

Will follow later after DbStats

Concurrency

Sweet! We implemented the same pattern! Great validation of our approach.

Centralized Mode:

Completely agree the proper way to use any agent is to collocate the agent with the thing you want to monitor. The challenge comes with managed hosting services where you may not have direct access to the host(s) to install a beat, nor know what nodes are in the cluster without discovery. Especially for databases, more deployments are moving to managed platforms.

But, whether Beats wants to go after this user segment is another question. I'm also new to the community and not really up to speed on the overall strategy.

Worth exploring further? I haven't used a lot of Mongo as a Service platforms, but here are a few:

MongoDB Atlas
Rackspace/Object Rocket

Version Compatibility

According to the docs, there are 3 storage engines. Not sure if the in-memory storage is reflected in serverStatus output, but worth looking into. I think output associated with the different storage engines will result in inevitable data discrepancies, so not much you can do there....

2.x

I still have to use 2.x for some projects, sadly. So I'm one of those users.

Major Version Compatibility

Mongobeat Model Layer

It's just intuition at this point, but as we try to transform documents into a standardized schema for elasticsearch, having an explicit modeling layer might be able to help make the transformations via a interfaces, struct composition, and instance methods. These may or may not be mutually exclusive with the schema package.

For example, I could have interface Transformer, parent struct ServerStatusV3 with an anonymous struct member AssertsV3. The actual beat "controller" would marshal mongo's output to either ServerStatusV3 for mongo3.x instances, and ServerStatusV2 for mongo2.x instances.

Model Layer

type Tranformer interface {
  GetStandardSchema() map[string]interface{}
}

// ServerStatusV3 implements transformer interface
type ServerStatusV3 struct {
  // AssertsV3 is an anonymous struct
  AssertsV3
}

func (s *ServerStatusV3) GetStandardSchema() map[string]interface{} {
  
  result := map[string]interface{}
    "asserts": s.AssertsV3.GetSandardSchema(),

    ... same for additional fields
  }

  return result  
}

// AssertsV3 implements transformer interface
type AssertsV3 struct {
  field1 uint
  field2 uint
  ...
}

func (ac *AssertsV3) GetStandardSchema() map[string]interface{} {
  transformation logic goes here
}

"Controller" Layer

func (bt *Mongobeat) Run(b *beat.Beat) error {
     ...
    mongoOutputRaw := makeDBCall()

    if versionMajor == "2" {
      results = marshalData to ServerStatusV2
    } else {
      results = marshal data to ServerStatusV3

    resultsMapStrIface := results.GetStandardSchema()

    sendEvent()
}

[ruflin]

• Contributions: 🎉 🎈
• Hosted solutions: Metricbeat does already support it, it is just not the recommended way. If someone provides a hosted solution, I would hope they also provide the monitoring with it, so no need for the client to have it normally, but sometimes still good to have it ;-)
• Schema: Our schema implementation applies in the generic cases. There are still cases it can't handle. We will have to add logic for versions pretty soon, but we did not add it in the first version as it adds complexity. Good to see you are also thinking about this problem and already have some solutions.

[scottcrespo]

Contributions

Forum: https://discuss.elastic.co/t/metricbeat-mongodb-report-db-stats/69218

Issue: #3205

=)

Hosted Solutions

I agree with what you're saying: 3rd party Hosted solutions typically provide analytics for MongoDB. And, an agent should be collocated with the system it monitors.

Schema

Agreed. It makes a lot of sense for an early version not to take on too much complexity.

To implement db.stats() for Metricbeat's MongoDB module, I plan on using the schema package and focus on the generic case (to maintain consistency). Backwards compatibility (i.e. Mongo 2.x) could follow in a subsequent release.

Conclusion

Looking forward to speaking with you in the new issues I've opened for db.stats !!!