-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Splunk Metrics serializer #4339
Changes from 9 commits
135799b
5dc75f4
864ba51
49a3c28
b53b2c3
8c4535f
30cad33
cc29b29
ac56a95
a3c8374
9c01483
1bb4e2e
67072ba
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,135 @@ | ||
# Splunk Metrics serialzier | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. spelling |
||
|
||
This serializer formats and outputs the metric data in a format that can be consumed by a Splunk metrics index. It can be used to write to a file using the file output, or for sending metrics to a HEC using the standard telegraf HTTP output. If you're using the HTTP output, this serializer knows how to batch the metrics so you don't end up with an HTTP POST per metric. | ||
|
||
Th data is output in a format that conforms to the specified Splunk HEC JSON format as found here: [Send metrics in JSON format](http://dev.splunk.com/view/event-collector/SP-CAAAFDN). | ||
|
||
An example event looks like: | ||
```javascript | ||
{ | ||
"time": 1529708430, | ||
"event": "metric", | ||
"host": "patas-mbp", | ||
"fields": { | ||
"_value": 0.6, | ||
"cpu": "cpu0", | ||
"dc": "mobile", | ||
"metric_name": "cpu.usage_user", | ||
"user": "ronnocol" | ||
} | ||
} | ||
``` | ||
In the above snippet, the following keys are dimensions: | ||
* cpu | ||
* dc | ||
* user | ||
|
||
## Using with the HTTP output | ||
|
||
To send this data to a Splunk HEC, you can use the HTTP output, there are some custom headers that you need to add | ||
to manage the HEC authorization, here's a sample config for an HTTP output: | ||
|
||
```toml | ||
[[outputs.http]] | ||
# ## URL is the address to send metrics to | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. again, leading |
||
url = "https://localhost:8088/services/collector" | ||
# | ||
# ## Timeout for HTTP message | ||
# # timeout = "5s" | ||
# | ||
# ## HTTP method, one of: "POST" or "PUT" | ||
# # method = "POST" | ||
# | ||
# ## HTTP Basic Auth credentials | ||
# # username = "username" | ||
# # password = "pa$$word" | ||
# | ||
# ## Optional TLS Config | ||
# # tls_ca = "/etc/telegraf/ca.pem" | ||
# # tls_cert = "/etc/telegraf/cert.pem" | ||
# # tls_key = "/etc/telegraf/key.pem" | ||
# ## Use TLS but skip chain & host verification | ||
# # insecure_skip_verify = false | ||
# | ||
# ## Data format to output. | ||
# ## Each data format has it's own unique set of configuration options, read | ||
# ## more about them here: | ||
# ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md | ||
data_format = "splunkmetric" | ||
## Provides time, index, source overrides for the HEC | ||
hec_routing = true | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you rename this variable There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. will do There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just to keep things consistent, lets call it There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ok, will do. |
||
# | ||
# ## Additional HTTP headers | ||
[outputs.http.headers] | ||
# # Should be set manually to "application/json" for json data_format | ||
Content-Type = "application/json" | ||
Authorization = "Splunk xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" | ||
X-Splunk-Request-Channel = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" | ||
``` | ||
|
||
## Overrides | ||
You can override the default values for the HEC token you are using by adding additional tags to the config file. | ||
|
||
The following aspects of the token can be overriden with tags: | ||
* index | ||
* source | ||
|
||
You can either use `[global_tags]` or using a more advanced configuration as documented [here](https://github.com/influxdata/telegraf/blob/master/docs/CONFIGURATION.md). | ||
|
||
Such as this example which overrides the index just on the cpu metric: | ||
```toml | ||
[[inputs.cpu]] | ||
percpu = false | ||
totalcpu = true | ||
[inputs.cpu.tags] | ||
index = "cpu_metrics" | ||
``` | ||
|
||
## Using with the File output | ||
|
||
You can use the file output when running telegraf on a machine with a Splunk forwarder. | ||
|
||
A sample event when `hec_routing` is false (or unset) looks like: | ||
```javascript | ||
{ | ||
"_value": 0.6, | ||
"cpu": "cpu0", | ||
"dc": "mobile", | ||
"metric_name": "cpu.usage_user", | ||
"user": "ronnocol", | ||
"time": 1529708430 | ||
} | ||
``` | ||
Data formatted in this manner can be ingested with a simple `props.conf` file that | ||
looks like this: | ||
|
||
```ini | ||
[telegraf] | ||
category = Metrics | ||
description = Telegraf Metrics | ||
pulldown_type = 1 | ||
DATETIME_CONFIG = | ||
NO_BINARY_CHECK = true | ||
SHOULD_LINEMERGE = true | ||
disabled = false | ||
INDEXED_EXTRACTIONS = json | ||
KV_MODE = none | ||
TIMESTAMP_FIELDS = time | ||
TIME_FORMAT = %s.%3N | ||
``` | ||
|
||
An example configuration of a file based output is: | ||
|
||
```toml | ||
# # Send telegraf metrics to file(s) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. same |
||
[[outputs.file]] | ||
# ## Files to write to, "stdout" is a specially handled file. | ||
files = ["/tmp/metrics.out"] | ||
# | ||
# ## Data format to output. | ||
# ## Each data format has its own unique set of configuration options, read | ||
# ## more about them here: | ||
# ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md | ||
data_format = "splunkmetric" | ||
hec_routing = false | ||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no need for the initial
#
through line 252