-
Notifications
You must be signed in to change notification settings - Fork 39
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add new macros for diff calculation, and unit tests (#99)
* Add macro for new hash-based comparison strategy * split out SF-focused version of macro * Fix change to complex object * Fix overuse of star * switch from compare rels to compare queries * provide wrapping parens * switch to array of columns for PK * split unit tests into own files, change unit tests to array pk * tidy up get_comp_bounds * fix arg rename * add quick_are_queries_identical and unit tests * Move data tests into own directory * Add test for multiple PKs * fix incorrect unit test configs * make data types for id and id_2 big enough nums * Mock event_time response * fix hardcoded value in quick_are_qs_identical * Add unit tests for null handling (still broken) * Rename columsn to be more unique * Steal surrogate key macro from utils * Use generated surrogate key across the board in place of PK * rm my profile reference * Update quick_are_queries_identical.sql * Add diagram explaining comparison bounds * Add comments explaining warehouse-specific optimisations * cross-db support * subq * no postgres or redshift for a sec * add default var values for compare wrappers * avoid lateral alias reference for BQ * BQ doesn't support count(arg1, arg2) * re-enable redshift * Alias subq for redshift * remove extra comma * add row status of nonunique_pk * remove redundant test and wrapper model * Create json-y tests for snowflake * Add workaround for redshift to support count num rows in status * skip incompatible tests * Fix redshift lack of bool_or support in window funcs * add skip exclusions for everything else * fix incorrect skip tag application * Move user configs to project.yml from profiles * Temporarily disable unpassable redshift tests * add temp skip to circle's config.yml * forgot tag: method * Temporarily skip reworked_compare_all_statuses_different_column_set * Skip another test redshift * disable unsupported tests BQ * postgres too? * Fixes for postgres * namespace macros * It's a postgres problem, not a redshift problem * Handle postgres 63 char limit * Add databricks * Rename tests to data_tests * Found a better workaround for missing count distinct window * actually call the macro * disable syntax-failing tests on dbx
- Loading branch information
Showing
45 changed files
with
997 additions
and
36 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,5 @@ | ||
target/ | ||
dbt_packages/ | ||
logs/ | ||
logfile | ||
logfile | ||
.DS_Store |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
{ | ||
"yaml.schemas": { | ||
"https://github.com/raw/dbt-labs/dbt-jsonschema/main/schemas/latest/dbt_yml_files-latest.json": [ | ||
"/**/*.yml", | ||
"!profiles.yml", | ||
"!dbt_project.yml", | ||
"!packages.yml", | ||
"!selectors.yml", | ||
"!profile_template.yml" | ||
], | ||
"https://github.com/raw/dbt-labs/dbt-jsonschema/main/schemas/latest/dbt_project-latest.json": [ | ||
"dbt_project.yml" | ||
], | ||
"https://github.com/raw/dbt-labs/dbt-jsonschema/main/schemas/latest/selectors-latest.json": [ | ||
"selectors.yml" | ||
], | ||
"https://github.com/raw/dbt-labs/dbt-jsonschema/main/schemas/latest/packages-latest.json": [ | ||
"packages.yml" | ||
] | ||
}, | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 change: 1 addition & 0 deletions
1
integration_tests/models/unit_test_placeholder_models/unit_test_model_a.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
select 12 as id, 22 as id_2, 'xyz' as col1, 'tuv' as col2, 123 as col3, {{ dbt.current_timestamp() }} as created_at |
1 change: 1 addition & 0 deletions
1
integration_tests/models/unit_test_placeholder_models/unit_test_model_b.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
select 12 as id, 22 as id_2, 'xyz' as col1, 'tuv' as col2, 123 as col3, {{ dbt.current_timestamp() }} as created_at |
3 changes: 3 additions & 0 deletions
3
integration_tests/models/unit_test_placeholder_models/unit_test_struct_model_a.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
{{ config(tags=['skip' if (target.type in ['redshift', 'bigquery', 'postgres', 'databricks']) else 'runnable']) }} | ||
|
||
select 1 as id, 'John Doe' as col1, object_construct('street', '123 Main St', 'city', 'Anytown', 'state', 'CA') as col2 |
3 changes: 3 additions & 0 deletions
3
integration_tests/models/unit_test_placeholder_models/unit_test_struct_model_b.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
{{ config(tags=['skip' if (target.type in ['redshift', 'bigquery', 'postgres', 'databricks']) else 'runnable']) }} | ||
|
||
select 1 as id, 'John Doe' as col1, object_construct('street', '123 Main St', 'city', 'Anytown', 'state', 'CA') as col2 |
8 changes: 8 additions & 0 deletions
8
integration_tests/models/unit_test_wrappers/unit_compare_queries.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
|
||
{{ | ||
audit_helper.compare_queries( | ||
"select * from " ~ ref('unit_test_model_a'), | ||
"select * from " ~ ref('unit_test_model_b'), | ||
summarize = var('compare_queries_summarize') | ||
) | ||
}} |
47 changes: 47 additions & 0 deletions
47
integration_tests/models/unit_test_wrappers/unit_compare_queries.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
unit_tests: | ||
- name: identical_records_compare_queries | ||
model: unit_compare_queries | ||
description: The world's most basic unit test. | ||
|
||
given: | ||
- input: ref('unit_test_model_a') | ||
rows: | ||
- { "id": 1, "col1": "abc", "col2": "def" } | ||
- { "id": 2, "col1": "hij", "col2": "klm" } | ||
- { "id": 3, "col1": "nop", "col2": "qrs" } | ||
- input: ref('unit_test_model_b') | ||
rows: | ||
- { "id": 1, "col1": "abc", "col2": "def" } | ||
- { "id": 2, "col1": "hij", "col2": "klm" } | ||
- { "id": 3, "col1": "nop", "col2": "qrs" } | ||
|
||
expect: | ||
rows: | ||
- {"in_a": true, "in_b": true} | ||
|
||
overrides: | ||
vars: | ||
compare_queries_summarize: true | ||
|
||
- name: identical_records_compare_queries_no_summarize | ||
model: unit_compare_queries | ||
description: The world's second most basic unit test. | ||
|
||
given: | ||
- input: ref('unit_test_model_a') | ||
rows: | ||
- { "id": 1, "col1": "abc", "col2": "def" } | ||
- { "id": 2, "col1": "hij", "col2": "klm" } | ||
- { "id": 3, "col1": "nop", "col2": "qrs" } | ||
- input: ref('unit_test_model_b') | ||
rows: | ||
- { "id": 1, "col1": "abc", "col2": "def" } | ||
- { "id": 2, "col1": "hij", "col2": "klm" } | ||
- { "id": 3, "col1": "nop", "col2": "qrs" } | ||
|
||
expect: | ||
rows: [] | ||
|
||
overrides: | ||
vars: | ||
compare_queries_summarize: false |
10 changes: 10 additions & 0 deletions
10
integration_tests/models/unit_test_wrappers/unit_quick_are_queries_identical.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
{{ config(tags=['skip' if (target.type in ['redshift', 'bigquery', 'postgres', 'databricks']) else 'runnable']) }} | ||
|
||
{{ | ||
audit_helper.quick_are_queries_identical( | ||
"select * from " ~ ref('unit_test_model_a'), | ||
"select * from " ~ ref('unit_test_model_b'), | ||
columns=var('quick_are_queries_identical_cols'), | ||
event_time=var('quick_are_queries_identical_event_time') | ||
) | ||
}} |
Oops, something went wrong.