Automatic merge from master to 6.x branch #7688
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The prefix for the stats metrics was metrics but renamed to `stats` by accident as the name is now auto generated. This reverts this change. Closes elastic#7562
Take `http.request.method` from ECS and apply it to the Kibana fileset. Additional logs are added to the example log files.
Instead of the from field the to field was logged.
We recently started to move fields.yml into the Golang binary to be used internally. To make sure the loading important and loading of all the data into the binary works as expected for Metricbeat, this adds some basic tests. Related to elastic#7605.
GC log format for JVM9 is more detailed than for JVM8. Differences and possible improvements: * To get cpu_times.* a corellation between log lines is required. * Some GC metrics are available in jvm8 are not in jvm9 (class_unload_time_sec, weak_refs_processing_time_sec, ...) * heap.used_kb is empty, but it can be calculated as young_gen.used_kb + old_gen.size_kb * GC phase times are logged in miliseconds vs seconds in jvm8
From now on when a user provides a type hint in an Ingest pipeline, it's added to the generated `fields.yml` instead of guessing. Closes elastic#7472
Filebeat introduces a meta field to registry entries in 6.3.1. The meta field is used to distuingish different log streams in docker files. For other input types the meta field must be null. Unfortunately the input loader did initialize the meta field with an empty dictionary. This leads to failing matches of old and new registry entries. Due to the match failing, old entries will not be removed, and filebeat will handle all files as new files on startup (old logs are send again).
Users will observe duplicate entries in the reigstry file. One entry with "meta": null and one entry with "meta": {}. The entry with "meta": {} will be used by filebeat. The null-entry will not be used by filebeat, but is kept in the registry file, cause it has now active owner (yet).
Improvements provided by this PR:
* when matching states consider an empty map and a null-map to be equivalent
* update input loader to create a null map for old state -> registry entries will be compatible on upgrade
* Add checks in critical places replacing an empty map with a null-map
* Add support to fix registry entries on load. states from corrupted 6.3.1 files will be merged into one single state on load
* introduce unit tests for loading different registry formats
* introduce system tests validating output and registry when upgrading filebeat from an older version
Closes: elastic#7634
This commit seeks to establish a pattern for testing heartbeat jobs. It currently tests the HTTP and TCP jobs. It also required some minor refactors of those tasks for HTTP/TCP. To do this, it made sense to validate event maps with a sort of schema library. I couldn't find one that did exactly what I wanted here, so I wrote one called mapval. That turned out to be a large undertaking, and is now the majority of this commit. Further tests need to be written, but this commit is large enough as is. One of the nicest things about the heartbeat architecture is the dialer chain behavior. It should be the case that any validated protocol using TCP (e.g. HTTP, TCP, Redis, etc.) has the exact same tcp metadata. To help make testing these properties easy mapval lets users compose portions of a schema into a bigger one. In other words, you can say "An HTTP response should be a TCP response, with the standard monitor data added in, and also the special HTTP fields". Even having only written a handful of tests this has uncovered some inconsistencies there, where TCP jobs have a hostname, but HTTP ones do not.
This PR makes it so that the `elasticsearch/shard` metricset only fetches information from the Elasticsearch node if that node is the master node.
…#7525) This PR takes the `stats` metricset of the `kibana` Metricbeat module and makes it ship documents to `.monitoring-kibana-6-mb-%{YYYY.MM.DD}` indices, while preserving the current format/mapping expected by docs in these indices. This will ensure that current consumers of the data in these indices, viz. the X-Pack Monitoring UI and the Telemetry shipping module in Kibana, will continue to work as-is.
Release the rename processor as GA
From copy paste Kafka was in the log message instead of Kibana.
Sometimes the old logging mechanism was used. If all use the new one it is easier to find all the entries. In addition some messages were inconsistent.
This PR marks the `elasticsearch` module and all its 8 existing metricsets all as `beta`. Previously only 2 metricsets were marked as `beta` with the remaining 6 marked as `experimental`.
Add `locks`, `global_locks`, `oplatencies` and `process` fields to `status` metricset of MongoDB module.
In elastic/elasticsearch#31669 the field `region_iso_code` was added to the geoip processor. Because of this test broke with the most recent release of Elasticsearch as the events contain an undocumented field.
* Fix duplicated module headers Closes elastic#7643 * fix metricset titles for munin and kvm * fix imssing kubernetes apiserver metricset doc * remove headers from modules / metricset generator and clean up traefik title
* Release munin and treafik module as beta. * fixes to munin module
Instead of doing it from the `state_container`. Problem with the previous approach is that `state_container` metricset is not run in all nodes, but from a single point. Making performance metrics not available in all cases. With this new approach, the enriching process will also collect performance metrics, so they should be available everywhere where the module is run.
Running `make misspell`.
- Update Sarama to 1.17. The Sarama testsuite tests kafka versions between 0.11 and 1.1.0. - Update compatible versions in output docs - Add compression_level setting for gzip compression
Make will now delegate to mage for generating fields.yml. Make will check if the mage command exists and go install it if not. The FIELDS_FILE_PATH make variable is not longer used because the path(s) are specified in magefile.go. This allows fields.yml to be generated on Windows using Mage. The CI scripts for Windows have been updated so that fields.yml is generated for all Beats during testing. This also adds a make.bat in each directory where building occurs to give Windows users a starting point. Some fixes were made to the generators because: - rsync was excluding important source files contained in a directory named "build" - the generated project needed to be `git init` before running certain magefile targets that detect project's root dir and import path.
Update fixes config unpacking if users overwrite settings from CLI, with
missing values. When using `-E key=` (e.g. in scripts defining potential
empty defaults via env variables like `-E key=${MYVALUE}`), an untyped
`nil`-values was inserted into the config. This untyped value will make
Unpack fail for most typed settings.
For APM Server the recommended way of loading dashboards and Kibana index pattern will be through the Kibana UI from 6.4 on. Since the docs are based on the libbeat docs we need to add a deprecation flag for dashboard and index pattern related documentation. relates to elastic/apm-server#1142
ruflin
force-pushed
the
automatic_merge_from_master_to_6.x_branch
branch
from
aee89d5
to
b11c10c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.