[Metricbeat] Add performance metricset to Oracle module #12547

sayden · 2019-06-14T15:54:44Z

In the performance Metricset of Oracle module a list of key performance metrics for Oracle must be included.

Cursor and cache data are included in this first PR.

houndci-bot · 2019-06-14T15:54:53Z

x-pack/metricbeat/module/oracle/testing.go

+	return port
+}
+
+// GetOracleEnvUsername returns the port of the Oracle server or the value of the environment variable ORACLE_PASSWORD if not empty


comment on exported function GetOracleEnvPassword should be of the form "GetOracleEnvPassword ..."

sayden · 2019-07-08T11:06:46Z

@odbaeu pinging you in case you want to contribute with something or add some comments 😉

houndci-bot · 2019-07-09T14:39:03Z

x-pack/metricbeat/module/oracle/sql.go

+	return s.Valid
+}
+
+func (s *StringValue) Value() interface{} {


exported method StringValue.Value should have comment or be unexported

houndci-bot · 2019-07-09T14:39:03Z

x-pack/metricbeat/module/oracle/sql.go

+	return i.Int64
+}
+
+type StringValue struct {


exported type StringValue should have comment or be unexported

houndci-bot · 2019-07-09T14:39:03Z

x-pack/metricbeat/module/oracle/sql.go

+	return i.Valid
+}
+
+func (i *Int64Value) Value() interface{} {


exported method Int64Value.Value should have comment or be unexported

houndci-bot · 2019-07-09T14:39:04Z

x-pack/metricbeat/module/oracle/sql.go

+	return i.Float64
+}
+
+type Int64Value struct {


exported type Int64Value should have comment or be unexported

houndci-bot · 2019-07-09T14:39:04Z

x-pack/metricbeat/module/oracle/sql.go

+	return i.Valid
+}
+
+func (i *Float64Value) Value() interface{} {


exported method Float64Value.Value should have comment or be unexported

houndci-bot · 2019-07-09T14:39:05Z

x-pack/metricbeat/module/oracle/sql.go

+
+type SqlValueWrapper struct {
+	Field    string
+	SqlValue SqlValue


struct field SqlValue should be SQLValue

houndci-bot · 2019-07-09T14:39:05Z

x-pack/metricbeat/module/oracle/sql.go

+	}
+}
+
+type SqlValueWrapper struct {


exported type SqlValueWrapper should have comment or be unexported
type SqlValueWrapper should be SQLValueWrapper

houndci-bot · 2019-07-09T14:39:05Z

x-pack/metricbeat/module/oracle/sql.go

+	return
+}
+
+func NewSqlWrapper(fieldName string, value SqlValue) *SqlValueWrapper {


exported function NewSqlWrapper should have comment or be unexported
func NewSqlWrapper should be NewSQLWrapper

houndci-bot · 2019-07-09T14:39:06Z

x-pack/metricbeat/module/oracle/sql.go

+)
+
+// checkNullString avoid setting an invalid empty string value on Metricbeat event
+func CheckNullSqlValue(logger *logp.Logger, output map[string]common.MapStr, parentTargetFieldName, targetFieldName string, v SqlValue) {


func CheckNullSqlValue should be CheckNullSQLValue

houndci-bot · 2019-07-09T14:39:06Z

x-pack/metricbeat/module/oracle/sql.go

+	"github.com/elastic/beats/libbeat/logp"
+)
+
+// checkNullString avoid setting an invalid empty string value on Metricbeat event


comment on exported function CheckNullSqlValue should be of the form "CheckNullSqlValue ..."

odbaeu · 2019-07-10T23:00:40Z

Hey @sayden,

thank you for pinging me! I very much appreciate that you continue to develop the oracle mb.

For me as an experienced Oracle DBA, it's more important to focus monitoring things which cause an outage that performance. But I know that performance monitoring sells much better than availability monitoring :-).

When monitoring performance in Oracle we have to be very careful to not query views which belong to the Enterprise Edition Option "Diagnostic" or "Tuning". For example: All view starting with DBA_HIST*, DBA_ADVISOR*, v$active_session_history
If we use these views, we should warn the users that they need an Oracle license. But I don't think wee need these views for now.

I was thinking about the metrics above... here are some thoughts:

Cache hit ratios
In the past (5+ years ago) the cache hit ratio was an important value for me but this changed. Today I prefer to look at the view v$db_cache_advice. The reason is that some databases will never reach a good hit ratio. With the information from v$db_cache_advice I can tell, if adding memory would help or not (actually it's more tricky than just looking at the view because the view would very often tell "not more mem is needed" but that's actually wrong. It depends on HOW the view tells me the information :-) ). Maybe we can add v$db_cache_advice in a later release.
Wait times
(y) Top 10 (foreground) wait events is the place I look at first when analysing the performance of a database.
Response times
What do you mean by "response time"? Average response time of all SQLs? Or response time of select * from dual;? Hmm...
I would prefer to monitor the following:
The top 10 SQLs of the following categories: CPU consumption, number of executions, elapsed time, I/O, memory consumption (optional)
Shared pool data
Nice to have... Would be low priority for me...
PGA performance data
I almost never look at PGA performance data. I must admit that I never took care of a database with a very large number (10k+) of active connections. Most DBs have a connection pool of 20-50 connections from an application server.
Would be low priority for me...
I/O response time
It's very interesting how the response time is distributed. Let's say: 40% of all I/O requests 1ms - 10ms response time, 60% of all I/O requests 11ms - 50ms. This gives much more intel on how good the I/O performance is then just the average.
For most of the foreground wait events it's very interesting to look at the wait time distribution as described in "I/O response time". Especially all wait events related to I/O, concurrency and redo management.
For example "checkpoint not complete" can be very costly if there is a large percentage above 1 second (or even more than a minute!).

It's quite late... I hope there are not too many grammar or spelling mistakes... If you find one, you can keep it ;-).

x-pack/metricbeat/module/oracle/sql.go

jsoriano

This is looking pretty good, you did a good job looking for the queries and adapting the results.

I think it'd be good to add some additional explanations on the queries, to help understand what they do, and on the docs to explain the events that are being generated by the metricset.

x-pack/metricbeat/module/oracle/performance/extractor.go

x-pack/metricbeat/module/oracle/performance/cursors.go

x-pack/metricbeat/module/oracle/performance/library.go

x-pack/metricbeat/module/oracle/performance/data.go

jsoriano

Thanks for addressing the comments, thanks to having the example data files I have found a possible problem with mapping conflicts. This would be the only important pending thing.

It would be also nice to automate the collection of data files as suggested, but this can be left for a future change.

x-pack/metricbeat/module/oracle/performance/_meta/cursor_by_username_and_machine_data.json

jsoriano · 2019-08-23T09:50:53Z

x-pack/metricbeat/module/oracle/performance/buffer_cache_hit_ratio.go

+	hitRatio       sql.NullFloat64
+}
+
+/*


Nit. Not important because they are not exported method, but it'd be nice to follow godocs and start with something like this:

Suggested change

/*

/*

* bufferCacheHitRatio collects information about the cache hit ratio of an Oracle database server

x-pack/metricbeat/module/oracle/performance/cursors.go

x-pack/metricbeat/module/oracle/performance/metricset_test.go

x-pack/metricbeat/module/oracle/performance/buffer_cache_hit_ratio.go

x-pack/metricbeat/module/oracle/performance/_meta/fields.yml

sayden · 2019-08-23T18:45:18Z

Error is unrelated. Merging

sorantis · 2019-09-11T09:41:36Z

Partially addresses #1935

sayden added enhancement in progress Pull request is currently in progress. Metricbeat Metricbeat Team:Integrations Label for the Integrations team labels Jun 14, 2019

sayden self-assigned this Jun 14, 2019

houndci-bot reviewed Jun 14, 2019

View reviewed changes

andresrc added the [zube]: In Progress label Jun 17, 2019

sayden force-pushed the feature/xp/mb/oracle/performance branch 2 times, most recently from d90fa1c to d1ee956 Compare July 8, 2019 11:10

houndci-bot reviewed Jul 9, 2019

View reviewed changes

sayden added 6 commits August 20, 2019 11:45

Atomic commit

263e2c3

Refactor

05ea805

Cursor queries

44401f5

Fix test

2afb4ef

Pre-review commit

17013a1

Skip integration test until image is ready

9fcc315

sayden force-pushed the feature/xp/mb/oracle/performance branch from ff24d8d to 9fcc315 Compare August 21, 2019 12:39

houndci-bot reviewed Aug 21, 2019

View reviewed changes

x-pack/metricbeat/module/oracle/sql.go Show resolved Hide resolved

x-pack/metricbeat/module/oracle/sql.go Show resolved Hide resolved

sayden added review and removed in progress Pull request is currently in progress. labels Aug 21, 2019

sayden marked this pull request as ready for review August 21, 2019 12:41

sayden requested a review from a team as a code owner August 21, 2019 12:41

sayden added 2 commits August 21, 2019 18:04

Go vet

9e90f05

Add Changelog entry

8216f4a

jsoriano reviewed Aug 22, 2019

View reviewed changes

sayden added 3 commits August 22, 2019 22:43

Improve code with comments

c9803f2

Fix indentation

864e1c1

Skip integration test

7038c2d

jsoriano reviewed Aug 23, 2019

View reviewed changes

Improve testing

27984ab

jsoriano approved these changes Aug 23, 2019

View reviewed changes

Reorder functions and add some comments

d51b60b

jsoriano reviewed Aug 23, 2019

View reviewed changes

x-pack/metricbeat/module/oracle/performance/buffer_cache_hit_ratio.go Show resolved Hide resolved

sayden added 2 commits August 23, 2019 15:11

Reorder imports

9a9fa3c

Fix order once more

58e2f1d

kaiyan-sheng reviewed Aug 23, 2019

View reviewed changes

x-pack/metricbeat/module/oracle/performance/_meta/fields.yml Outdated Show resolved Hide resolved

Field fixed

14d8ee8

kaiyan-sheng approved these changes Aug 23, 2019

View reviewed changes

sayden merged commit dbced15 into elastic:master Aug 23, 2019

zube bot added [zube]: Done and removed [zube]: In Progress labels Aug 23, 2019

andresrc removed the [zube]: Done label Aug 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metricbeat] Add performance metricset to Oracle module #12547

[Metricbeat] Add performance metricset to Oracle module #12547

sayden commented Jun 14, 2019 •

edited

Loading

houndci-bot Jun 14, 2019

sayden commented Jul 8, 2019 •

edited

Loading

houndci-bot Jul 9, 2019

houndci-bot Jul 9, 2019

houndci-bot Jul 9, 2019

houndci-bot Jul 9, 2019

houndci-bot Jul 9, 2019

houndci-bot Jul 9, 2019

houndci-bot Jul 9, 2019

houndci-bot Jul 9, 2019

houndci-bot Jul 9, 2019

houndci-bot Jul 9, 2019

odbaeu commented Jul 10, 2019

jsoriano left a comment

jsoriano left a comment

jsoriano Aug 23, 2019

sayden commented Aug 23, 2019

sorantis commented Sep 11, 2019

	/*
	/*
	* bufferCacheHitRatio collects information about the cache hit ratio of an Oracle database server

[Metricbeat] Add performance metricset to Oracle module #12547

[Metricbeat] Add performance metricset to Oracle module #12547

Conversation

sayden commented Jun 14, 2019 • edited Loading

Choose a reason for hiding this comment

sayden commented Jul 8, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

odbaeu commented Jul 10, 2019

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sayden commented Aug 23, 2019

sorantis commented Sep 11, 2019

sayden commented Jun 14, 2019 •

edited

Loading

sayden commented Jul 8, 2019 •

edited

Loading