Add ceph osd_df to metricbeat #5606
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| //// | ||
| This file is generated! See scripts/docs_collector.py | ||
| //// | ||
|
|
||
| [[metricbeat-metricset-ceph-osd_df]] | ||
| === Ceph osd_df metricset | ||
|
|
||
| experimental | ||
|
|
||
| include::../../../module/ceph/osd_df/_meta/docs.asciidoc[] | ||
|
|
||
|
|
||
| ==== Fields | ||
|
|
||
| For a description of each field in the metricset, see the | ||
| <<exported-fields-ceph,exported fields>> section. | ||
|
|
||
| Here is an example document generated by this metricset: | ||
|
|
||
| [source,json] | ||
| ---- | ||
| include::../../../module/ceph/osd_df/_meta/data.json[] | ||
| ---- |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| { | ||
| "output": { | ||
| "nodes": [ | ||
| { | ||
| "crush_weight": 0.048691, | ||
| "depth": 2, | ||
| "device_class": "hdd", | ||
| "id": 0, | ||
| "kb": 52325356, | ||
| "kb_avail": 51245860, | ||
| "kb_used": 1079496, | ||
| "name": "osd.0", | ||
| "pgs": 0, | ||
| "pool_weights": {}, | ||
| "reweight": 1.0, | ||
| "type": "osd", | ||
| "type_id": 0, | ||
| "utilization": 2.063046, | ||
| "var": 1.0 | ||
| }, | ||
| { | ||
| "crush_weight": 0.048691, | ||
| "depth": 2, | ||
| "device_class": "hdd", | ||
| "id": 1, | ||
| "kb": 52325356, | ||
| "kb_avail": 51245860, | ||
| "kb_used": 1079496, | ||
| "name": "osd.1", | ||
| "pgs": 0, | ||
| "pool_weights": {}, | ||
| "reweight": 1.0, | ||
| "type": "osd", | ||
| "type_id": 0, | ||
| "utilization": 2.063046, | ||
| "var": 1.0 | ||
| } | ||
| ], | ||
| "stray": [], | ||
| "summary": { | ||
| "average_utilization": 2.063046, | ||
| "dev": 0.0, | ||
| "max_var": 1.0, | ||
| "min_var": 1.0, | ||
| "total_kb": 104650712, | ||
| "total_kb_avail": 102491720, | ||
| "total_kb_used": 2158992 | ||
| } | ||
| }, | ||
| "status": "OK" | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| { | ||
| "@timestamp": "2017-11-16T02:30:32.779Z", | ||
| "@metadata": { | ||
| "beat": "metricbeat", | ||
| "type": "doc", | ||
| "version": "7.0.0-alpha1" | ||
| }, | ||
| "ceph": { | ||
| "osd_df": { | ||
| "device_class": "hdd", | ||
| "name": "osd.1", | ||
| "total_byte": 52325356, | ||
| "used_byte": 1079496, | ||
| "avail_byte": 51245860, | ||
| "pg_num": 0, | ||
| "used_pct": 0.020630456866839092, | ||
| "id": 1 | ||
| } | ||
| }, | ||
| "metricset": { | ||
| "module": "ceph", | ||
| "name": "osd_df", | ||
| "host": "192.168.56.131:5000", | ||
| "rtt": 5452 | ||
| }, | ||
| "beat": { | ||
| "name": "centos7gui", | ||
| "hostname": "centos7gui", | ||
| "version": "7.0.0-alpha1" | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| === Ceph cluster_health metricset | ||
|
There was a problem hiding this comment. Also this title can now be removed with our new script. Sorry missed that previously but realised it now that i checked out the branch locally. |
||
|
|
||
| This is the `osd_df` metricset of the Ceph module. | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| - name: osd_df | ||
|
There was a problem hiding this comment. Left the comments in the fields.asciidoc, but they should obviously go here. |
||
| type: group | ||
|
There was a problem hiding this comment. It's a new feature I just added recently. Could you add |
||
| description: > | ||
| ceph osd disk usage information | ||
| fields: | ||
| - name: id | ||
| type: long | ||
| description: > | ||
| osd node id | ||
| - name: name | ||
| type: text | ||
| description: > | ||
| osd node name | ||
| - name: device_class | ||
| type: keyword | ||
| description: > | ||
| osd node type, illegal type include hdd, ssd etc. | ||
| - name: total.byte | ||
| type: long | ||
| format: bytes | ||
| description: > | ||
| osd disk total volume | ||
| - name: used.byte | ||
| type: long | ||
| format: bytes | ||
| description: > | ||
| osd disk usage volume | ||
| - name: available.bytes | ||
| type: long | ||
| format: bytes | ||
| description: > | ||
| osd disk available volume | ||
| - name: pg_num | ||
| type: long | ||
| description: > | ||
| shows how many pg located on this osd | ||
| - name: used.pct | ||
| type: scaled_float | ||
| format: percent | ||
| description: > | ||
| osd disk usage percentage | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| package osd_df | ||
|
There was a problem hiding this comment. don't use an underscore in package name |
||
|
|
||
| import ( | ||
| "encoding/json" | ||
|
|
||
| "github.com/elastic/beats/libbeat/common" | ||
| "github.com/elastic/beats/libbeat/logp" | ||
| ) | ||
|
|
||
| type Node struct { | ||
|
There was a problem hiding this comment. exported type Node should have comment or be unexported |
||
| ID int64 `json:"id"` | ||
| Name string `json:"name"` | ||
| Used int64 `json:"kb_used"` | ||
| Available int64 `json:"kb_avail"` | ||
| Total int64 `json:"kb"` | ||
| PgNum int64 `json:"pgs"` | ||
| DeviceClass string `json:"device_class"` | ||
| } | ||
|
|
||
| type Output struct { | ||
|
There was a problem hiding this comment. exported type Output should have comment or be unexported |
||
| Nodes []Node `json:"nodes"` | ||
| } | ||
|
|
||
| type OsdDfRequest struct { | ||
|
There was a problem hiding this comment. exported type OsdDfRequest should have comment or be unexported |
||
| Status string `json:"status"` | ||
| Output Output `json:"output"` | ||
| } | ||
|
|
||
| func eventsMapping(content []byte) ([]common.MapStr, error) { | ||
| var d OsdDfRequest | ||
| err := json.Unmarshal(content, &d) | ||
| if err != nil { | ||
| logp.Err("Error: ", err) | ||
| return nil, err | ||
| } | ||
|
|
||
| nodeList := d.Output.Nodes | ||
|
|
||
| //osd node list | ||
| events := []common.MapStr{} | ||
| for _, node := range nodeList { | ||
| nodeInfo := common.MapStr{ | ||
| "id": node.ID, | ||
| "name": node.Name, | ||
| "total.byte": node.Total, | ||
| "used.byte": node.Used, | ||
| "available.byte": node.Available, | ||
| "device_class": node.DeviceClass, | ||
| "pg_num": node.PgNum, | ||
| } | ||
|
|
||
| if 0 != node.Total { | ||
| var usedPct float64 | ||
| usedPct = float64(node.Used) / float64(node.Total) | ||
| nodeInfo["used.pct"] = usedPct | ||
| } | ||
|
|
||
| events = append(events, nodeInfo) | ||
| } | ||
|
|
||
| return events, nil | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| package osd_df | ||
|
There was a problem hiding this comment. don't use an underscore in package name |
||
|
|
||
| import ( | ||
| "github.com/elastic/beats/libbeat/common" | ||
| "github.com/elastic/beats/libbeat/common/cfgwarn" | ||
| "github.com/elastic/beats/metricbeat/helper" | ||
| "github.com/elastic/beats/metricbeat/mb" | ||
| "github.com/elastic/beats/metricbeat/mb/parse" | ||
| ) | ||
|
|
||
| const ( | ||
| defaultScheme = "http" | ||
| defaultPath = "/api/v0.1/osd/df" | ||
| ) | ||
|
|
||
| var ( | ||
| hostParser = parse.URLHostParserBuilder{ | ||
| DefaultScheme: defaultScheme, | ||
| DefaultPath: defaultPath, | ||
| }.Build() | ||
| ) | ||
|
|
||
| func init() { | ||
| if err := mb.Registry.AddMetricSet("ceph", "osd_df", New, hostParser); err != nil { | ||
| panic(err) | ||
| } | ||
| } | ||
|
|
||
| type MetricSet struct { | ||
|
There was a problem hiding this comment. exported type MetricSet should have comment or be unexported |
||
| mb.BaseMetricSet | ||
| *helper.HTTP | ||
| } | ||
|
|
||
| func New(base mb.BaseMetricSet) (mb.MetricSet, error) { | ||
|
There was a problem hiding this comment. exported function New should have comment or be unexported |
||
| cfgwarn.Beta("The ceph osd_df metricset is experimental") | ||
|
|
||
| http := helper.NewHTTP(base) | ||
| http.SetHeader("Accept", "application/json") | ||
|
|
||
| return &MetricSet{ | ||
| base, | ||
| http, | ||
| }, nil | ||
| } | ||
|
|
||
| func (m *MetricSet) Fetch() ([]common.MapStr, error) { | ||
|
There was a problem hiding this comment. exported method MetricSet.Fetch should have comment or be unexported |
||
| content, err := m.HTTP.FetchContent() | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
|
|
||
| events, err := eventsMapping(content) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
|
|
||
| return events, nil | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One thing that popped up in my head is that we have here
dfand we havesystem.filesystemwith similar info. Don't know ifosd_filesystemwould make sense here (even though pretty long).There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually the function of osd df in ceph is very similar with *system.filesystem * in linux. It shows osds' weight, reweight, size, used, avai and used pct information. Naming this new metricbet as osd_df is because it's a standard command in Ceph, and I consider that it will be much more familer to ceph users. For more about the command's detials:
http://docs.ceph.com/docs/master/man/8/ceph/
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We had a similar discussion with the system module. Most sysadmins will be used to run the
dfcommand. But someone consuming the data is more looking for information about the filesystem which could be a different person. TBH don't know which one is better. I'm fine moving forward withdf.