Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(DENG-4577): add monitoring bigeye_usage view to allow us to get more insight into bq usage and costs from bigeye #6260

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
+37 −0
Draft

feat(DENG-4577): add monitoring bigeye_usage view to allow us to get more insight into bq usage and costs from bigeye #6260

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
24650d4
feat: add monitoring bigeye_usage view to allow us to get more insigh…
kik-kik Sep 25, 2024
477a752
feat: add extract of bigeye table id checked in the bq information_sc…
kik-kik Sep 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions sql/moz-fx-data-shared-prod/monitoring/bigeye_usage/view.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
SELECT
submission_date,
reference_project_id,
reference_dataset_id,
reference_table_id,
creation_date,
task_duration,
total_terabytes_processed,
total_terabytes_billed,
total_slot_ms,
cost,
job_id,
user_email AS service_account,
bigeye_metric_id,
bigeye_monitored_table_id,
FROM
`moz-fx-data-shared-prod.monitoring.bigquery_usage`
WHERE
DATE(submission_date) = "2024-09-24"
AND user_type = "bigeye"
AND total_slot_ms IS NOT NULL
2 changes: 2 additions & 0 deletions sql/moz-fx-data-shared-prod/monitoring/bigquery_usage/view.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ SELECT
THEN "search-terms"
WHEN user_email LIKE "%airflow%"
THEN "airflow"
WHEN user_email LIKE "%bigeye%"
THEN "bigeye"
WHEN ENDS_WITH(user_email, "mozilla.com")
THEN "individual"
WHEN ENDS_WITH(user_email, "mozillafoundation.org")
Expand Down
4 changes: 4 additions & 0 deletions sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_usage_v2/query.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,8 @@ jobs_by_project AS (
user_email,
REGEXP_EXTRACT(query, r'Username: (.*?),') AS username,
REGEXP_EXTRACT(query, r'Query ID: (\w+), ') AS query_id,
REGEXP_EXTRACT(query, r'\'metric_id:(\d+)\'') AS bigeye_metric_id,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Marlene-M-Hirose @wwyc was there a specific reason why we didn't just pass the query through here? There is another bit of information I could extract here for BigEye, but wasn't sure if I should be polluting this query with BigEye specific details? If I had query available here I could just do it downstream in my bigeye monitoring view.

Copy link
Contributor

@Marlene-M-Hirose Marlene-M-Hirose Sep 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kik-kik , I do not think adding this specific information here is good. This table is supposed to be focused on BigQuery usage, not for monitoring bigeye. I would prefer to keep bigeye separate from this table.

We didn't add the query because we didn't need the query. We only needed the query_id and the username from the query.

REPLACE(REGEXP_EXTRACT(LOWER(query), r'`__grain_start__` from `([\w\d_\-\.`]+) as'), "`", "") AS bigeye_monitored_table_id,
FROM
`{{project}}.region-us.INFORMATION_SCHEMA.JOBS_BY_PROJECT` AS jp
LEFT JOIN
Expand Down Expand Up @@ -103,6 +105,8 @@ SELECT DISTINCT
jo.error_reason,
jo.error_message,
jo.resource_warning,
jo.bigeye_metric_id,
jo.bigeye_monitored_table_id,
@submission_date AS submission_date,
FROM
jobs_by_org AS jo
Expand Down
10 changes: 10 additions & 0 deletions sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_usage_v2/schema.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,16 @@ fields:
type: STRING
description: The warning message that appears if the resource usage is above the internal threshold of the system

- mode: NULLABLE
name: bigeye_metric_id
type: STRING
description: Metric ID corresponding to the metric executed by BigEye (Data Observability Platform).

- mode: NULLABLE
name: bigeye_monitored_table_id
type: STRING
description: Fully qualified table id of the table BigEye run a monitor against (volume and freshness).

- mode: NULLABLE
name: submission_date
type: DATE
Expand Down