Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Metricbeat] Add FailOnRequired when applying schema #14143

Merged
merged 6 commits into from
Oct 22, 2019
Merged

[Metricbeat] Add FailOnRequired when applying schema #14143

merged 6 commits into from
Oct 22, 2019

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Oct 18, 2019

Original problem reported is when running metricbeat mongodb module, metrics metricset failled with:

Error fetching data for metricset mongodb.metrics: failed to apply schema: 5 errors: key `metrics.repl.apply.attemptsToBecomeSecondary` not found; key `metrics.repl.initialSync` not found; key `metrics.repl.executor` not found; key `metrics.commands` not found; key `metrics.operation.writeConflicts` not found

I was able to reproduce it and turned out this error message changes with different keys not found when I reproduced it.
This error should be fixed by adding schema.FailOnRequired when applying the schema.

Second problem is after adding schema.FailOnRequired, a warning showed up when trying to run metrics metricset:

...
"reason":"failed to parse field [mongodb.metrics.query_executor.scanned_indexes] of type [long] in document...

I believe some metrics in metrics metricset are missing .count suffix, like mongodb.metrics.query_executor.scanned_indexes.

How to test this:

Please start mongodb locally and start Metricbeat with mongodb module enabled. Make sure metrics are flowing with no error above. You might see some error/warning message at INFO level indicating permission error but that's not related to this.

@kaiyan-sheng kaiyan-sheng marked this pull request as ready for review October 20, 2019 19:01
@kaiyan-sheng kaiyan-sheng requested a review from a team as a code owner October 20, 2019 19:01
@kaiyan-sheng kaiyan-sheng self-assigned this Oct 20, 2019
@kaiyan-sheng kaiyan-sheng added Team:Integrations Label for the Integrations team bug Metricbeat Metricbeat needs_backport PR is waiting to be backported to other branches. labels Oct 20, 2019
Copy link
Contributor

@fearful-symmetry fearful-symmetry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Glad to see the optional Apply types getting used, there's probably lots of other places where they could be useful.

@kaiyan-sheng
Copy link
Contributor Author

LGTM. Glad to see the optional Apply types getting used, there's probably lots of other places where they could be useful.

Yeah I see in mongodb module, we are not using FailOnRequired in other metricsets. But in these other metricsets, some fields are labeled with c.DictOptional or s.Optional and I haven't seen any error message related to other metricsets. That's why I didn't change all metricsets to use FailOnRequired in this PR.

@kaiyan-sheng kaiyan-sheng added the test-plan Add this PR to be manual test plan label Oct 22, 2019
@kaiyan-sheng kaiyan-sheng merged commit 09bc35a into elastic:master Oct 22, 2019
@kaiyan-sheng kaiyan-sheng deleted the mongodb_metrics branch October 22, 2019 20:25
@kaiyan-sheng kaiyan-sheng added v7.5.0 and removed needs_backport PR is waiting to be backported to other branches. labels Oct 22, 2019
kaiyan-sheng added a commit that referenced this pull request Oct 23, 2019
…ing schema (#14202)

* [Metricbeat] Add FailOnRequired when applying schema (#14143)

* Add FailOnRequired when applying schema

(cherry picked from commit 09bc35a)
@digz6666
Copy link

Thanks for the fix. Is it released in metricbeat 7.4.1?

@kaiyan-sheng
Copy link
Contributor Author

@digz6666 I don't think this is in 7.4.1 unfortunately. But it will definitely be in 7.5 release in near future!

jorgemarey pushed a commit to jorgemarey/beats that referenced this pull request Jun 8, 2020
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
…n applying schema (elastic#14202)

* [Metricbeat] Add FailOnRequired when applying schema (elastic#14143)

* Add FailOnRequired when applying schema

(cherry picked from commit 972900f)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Metricbeat Metricbeat review Team:Integrations Label for the Integrations team test-plan Add this PR to be manual test plan v7.5.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants