Support user databases for transaction log #4869

ritalwar · 2022-12-19T15:54:23Z

Enhancement

What does this PR do?

The MSSQL Integration currently loads "transaction_log metrics" from system dbs.
This PR has updated the integration to load "transaction_log metrics" from the user dbs by providing the list of databases for which the metrics is to be collected.

Also, this PR fixes the bug in the query for fetching metrics from sys.dm_db_log_space_usage.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Add the microsoft_sql_server integration, add server config, enable transaction_log metrics and verify the data.
Also, for the metrics from sys.dm_db_log_space_usage table, specify the list of databases.

Related issues

Screenshots

elasticmachine · 2022-12-19T16:11:42Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-12-22T11:48:26.783+0000
Duration: 17 min 30 sec

Test stats 🧪

Test	Results
Failed	0
Passed	19
Skipped	0
Total	19

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-12-19T16:11:58Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`2/2`)	💚
Files	100.0% (`2/2`)	💚
Classes	100.0% (`2/2`)	💚
Methods	93.333% (`28/30`)	❕
Lines	100.0% (`1238/1238`)	💚
Conditionals	100.0% (`0/0`)	💚

lalit-satapathy

Let's not close #4108 rather keep it open if we find a future solution to automatically find all user DBs.

lalit-satapathy · 2022-12-20T13:14:53Z

packages/microsoft_sqlserver/_dev/build/docs/README.md

@@ -75,7 +75,8 @@ See: [Instructions about each performance counter metrics](https://docs.microsof

 ### transaction_log metrics

-Collects system level `transaction_log` metrics information for SQL Server instance.
+Collects `transaction_log` metrics information for all the system and user Dbs of SQL Server instance.


Nit: let's be consistent with Dbs or dbs

lalit-satapathy · 2022-12-20T13:16:02Z

packages/microsoft_sqlserver/data_stream/transaction_log/manifest.yml

@@ -17,7 +17,7 @@ streams:
        title: Databases
        multi: true
        required: true
-        show_user: false
+        show_user: true
        description: Default system databases are preloaded. For custom database please add additional rows and enter the database name.


Change from "custom database" to "user defined databases"?

muthu-mps · 2022-12-20T16:18:01Z

packages/microsoft_sqlserver/data_stream/transaction_log/agent/stream/stream.yml.hbs

 # Collect the transaction logs from the system database
 sql_queries:
+  - query: "SELECT @@servername AS server_name, @@servicename AS instance_name, name As 'database_name', database_id FROM sys.databases;"
+    response_format: table
+  - query: "SELECT @@servername AS server_name, @@servicename AS instance_name, name As 'database_name', s.database_id, total_log_size_mb, active_log_size_mb,log_backup_time,log_since_last_log_backup_mb,log_since_last_checkpoint_mb,log_recovery_size_mb from sys.databases As s CROSS APPLY sys.dm_db_log_stats(s.database_id);"


When you show all db data for log_stats and only customer provided db data for log_space_usage, there is a possibility that customer will not provided the custom database names then Kibana will only represent data for log_stats and not the log_space_usage for the selected custom database. I would recommend keeping both the stats rendered in similar way.

As per discussion with Lalit, we are providing blend of long and short term solution, where we are giving all databases metrics for log_stats and for log_space_usage we are providing short term solution i.e user can input dbs and will get the metrics for same.
This has also been updated in docs.
Additional benefit is that user will have list of all the dbs as result of 1st query and they can give the names from there to get metrics of log_space_usage.

muthu-mps · 2022-12-20T16:28:27Z

packages/microsoft_sqlserver/data_stream/transaction_log/agent/stream/stream.yml.hbs

@@ -5,15 +5,19 @@ hosts:
 period: {{period}}
 driver: mssql
 raw_data.enabled: true
+databases:
+{{#each databases}}


Do we need this database iteration here ?

Yes, this is needed to get the list of databases to fetch user dbs metrics, upon which the query iterator will work.

ShourieG

LGTM

muthu-mps

LGTM!

elasticmachine · 2022-12-22T16:01:19Z

Package microsoft_sqlserver - 1.11.0 containing this change is available at https://epr.elastic.co/search?package=microsoft_sqlserver

Support user databases for transaction log

6e49623

ritalwar self-assigned this Dec 19, 2022

Update changelog.yml

04cd8a3

ritalwar added 2 commits December 20, 2022 13:15

Update sample_event.json

61afa21

Update README.md

386e0ce

ritalwar marked this pull request as ready for review December 20, 2022 08:16

ritalwar requested review from a team as code owners December 20, 2022 08:16

lalit-satapathy reviewed Dec 20, 2022

View reviewed changes

muthu-mps reviewed Dec 20, 2022

View reviewed changes

ritalwar added 2 commits December 21, 2022 09:54

Address review comments

50ce900

Chages for short term solution

3d9368a

ShourieG approved these changes Dec 22, 2022

View reviewed changes

muthu-mps self-requested a review December 22, 2022 15:03

muthu-mps approved these changes Dec 22, 2022

View reviewed changes

ritalwar merged commit 91adc40 into elastic:main Dec 22, 2022

lalit-satapathy mentioned this pull request Dec 26, 2022

MSSQL integration updates needed #4238

Closed

9 tasks

ritalwar mentioned this pull request Dec 27, 2022

MSSQL enhancements: support user databases #4108

Closed

shmsr mentioned this pull request Jul 16, 2023

x-pack/metricbeat/module/sql: Add option to execute SQL queries for all databases elastic/beats#35688

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support user databases for transaction log #4869

Support user databases for transaction log #4869

ritalwar commented Dec 19, 2022 •

edited

Loading

elasticmachine commented Dec 19, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Dec 19, 2022 •

edited

Loading

lalit-satapathy left a comment

lalit-satapathy Dec 20, 2022

lalit-satapathy Dec 20, 2022

muthu-mps Dec 20, 2022

ritalwar Dec 21, 2022

muthu-mps Dec 20, 2022

ritalwar Dec 21, 2022

ShourieG left a comment

muthu-mps left a comment

elasticmachine commented Dec 22, 2022

Support user databases for transaction log #4869

Support user databases for transaction log #4869

Conversation

ritalwar commented Dec 19, 2022 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Dec 19, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Dec 19, 2022 • edited Loading

🌐 Coverage report

lalit-satapathy left a comment

Choose a reason for hiding this comment

lalit-satapathy Dec 20, 2022

Choose a reason for hiding this comment

lalit-satapathy Dec 20, 2022

Choose a reason for hiding this comment

muthu-mps Dec 20, 2022

Choose a reason for hiding this comment

ritalwar Dec 21, 2022

Choose a reason for hiding this comment

muthu-mps Dec 20, 2022

Choose a reason for hiding this comment

ritalwar Dec 21, 2022

Choose a reason for hiding this comment

ShourieG left a comment

Choose a reason for hiding this comment

muthu-mps left a comment

Choose a reason for hiding this comment

elasticmachine commented Dec 22, 2022

ritalwar commented Dec 19, 2022 •

edited

Loading

elasticmachine commented Dec 19, 2022 •

edited

Loading

elasticmachine commented Dec 19, 2022 •

edited

Loading