[0.2.0] Metadata Implementation #81

MooooCat · 2023-12-19T15:26:45Z

Description

The metadata component for single table and multi table.

Motivation and Context

This metadata is used to describe:

the base attribute (data type, pii, format, etc) of each column in table;
the relationship between columns;
the relationship between tables (parent table, child table);
etc.

Also, metadata will also provide methods to:

auto inspections;
add/change attributes;
add/change/remove relationship;
check the validation of relationship;
etc.

How has this been tested?

Types of changes

Maintenance (no change in code, maintain the project's CI, docs, etc.)
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.

for more information, see https://pre-commit.ci

MooooCat · 2023-12-19T15:27:18Z

Currently, metadata is in the design stage.

I will probably complete the specific design tomorrow, I plan to initiate a discussion when completing the design.

sweep-ai · 2023-12-19T15:27:51Z

Apply Sweep Rules to your PR?

Apply: All new business logic should have corresponding unit tests.
Apply: Refactor large functions to be more modular.
Apply: Add docstrings to all functions and file headers.

Wh1isper · 2023-12-20T01:17:28Z

We can ignore the multi-table scenario for now and get the single table fields done. The table-to-table relationship (A.a vs. B.a) can be considered separately from the field (a,b,c,d) case within the table.

I'd like to know what is necessarily contained in a single table Metadata, such as the currently implemented discrete columns: https://github.com/hitsz-ids/synthetic-data-generator/blob/main/sdgx/data_models/inspectors/discrete.py

MooooCat · 2023-12-20T04:36:53Z

We can ignore the multi-table scenario for now and get the single table fields done. The table-to-table relationship (A.a vs. B.a) can be considered separately from the field (a,b,c,d) case within the table.

Yes, we will first update the metadata of a single table.

Currently, the metadata of a single table has the following new information (please help me check if there are any gaps):

Primary key: primary key information, here we need to be compatible with composite primary keys;
Data type of each column: Use dict description, key is the column name, and value is the DataType object of each column. Different attributes/methods can be provided according to the different DataType.

MooooCat · 2023-12-20T04:38:27Z

I'd like to know what is necessarily contained in a single table Metadata, such as the currently implemented discrete columns: https://github.com/hitsz-ids/synthetic-data-generator/blob/main/sdgx/data_models/inspectors/discrete.py

I think this is very necessary. We may start from support 5 basic types:

ID
Numeric value
Discrete value
DataTime
Bool value

MooooCat · 2023-12-20T04:42:27Z

However, the multi-table scenario is also the focus of our work.

We have faced the need for multi tables in some actual work, and existing solutions cannot solve their needs.

After finishing single table metadata, I will implement metadata for multi tables based on SingleTableMetadata.

Metadata will develop based on 0.1.0.

for more information, see https://pre-commit.ci

Wh1isper · 2023-12-21T03:04:55Z

Seems no need to intro a submodule as metadata

metadata.py for single table and relationship.py + The-Multi-Table-Combiner.py for multi table is enough.

MooooCat · 2023-12-21T04:56:26Z

Seems no need to intro a submodule as metadata

metadata.py for single table and relationship.py + The-Multi-Table-Combiner.py for multi table is enough.

It’s okay, there aren’t many new modules in metadata.

for more information, see https://pre-commit.ci

codecov-commenter · 2023-12-23T08:39:50Z

Codecov Report

Attention: 52 lines in your changes are missing coverage. Please review.

Comparison is base (897e252) 76.29% compared to head (ad312c8) 75.87%.

Files	Patch %	Lines
sdgx/data_models/multi_table_combiner.py	0.00%	28 Missing ⚠️
sdgx/data_models/relationship.py	0.00%	12 Missing ⚠️
sdgx/data_models/metadata.py	76.19%	10 Missing ⚠️
sdgx/data_models/inspectors/datetime.py	93.75%	2 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #81      +/-   ##
==========================================
- Coverage   76.29%   75.87%   -0.42%     
==========================================
  Files          52       58       +6     
  Lines        2054     2230     +176     
==========================================
+ Hits         1567     1692     +125     
- Misses        487      538      +51

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

for more information, see https://pre-commit.ci

Use a single list for single or composite primary key. Co-authored-by: Zhongsheng Ji <9573586@qq.com>

Wh1isper · 2023-12-25T03:41:19Z

sdgx/data_models/multi_table_combiner.py

+        relationship_cnt = len(self.relationships)
+        metadata_cnt = len(self.metadata_dict.keys())
+        if metadata_cnt != relationship_cnt + 1:
+            raise ValueError("Number of tables should corresponds to relationships.")


Can we add a new exception in sdgx/exceptions.py? This will help us differentiate between what are expected exceptions.

This is something I didn't consider, I'll add new exceptions MultiTableCombineError and MetadataError in sdgx.exceptions.

For better unit testing and bug fixing, I'd suggest synchronizing the main branch, which also includes Exceptions with error codes and exit codes

sdgx/data_models/relationship.py

MooooCat · 2023-12-25T03:44:37Z

is the current design sufficient for this?

I think the current design of relationship is sufficient to describe existing multi-table relationships.

In synthetic data, the multi-table relationships we currently implement are limited to “one-to-one” or “one-to-many”, that is, a parent table is associated with different child tables through foreign keys. In the current design, relationship is like a line in this correspondence, and each relationship only describes the relationship between two tables. We use multi-table combiner to combine table's meatadata and relationships.

Wh1isper · 2023-12-25T03:45:49Z

Sounds fine, looking forward to more commits!

Key are described using list, which is compatible with single or composite foreign key. Co-authored-by: Zhongsheng Ji <9573586@qq.com>

for more information, see https://pre-commit.ci

Unit testing also needs to be implemented.

for more information, see https://pre-commit.ci

sdgx/data_models/metadata.py

Wh1isper · 2023-12-25T09:50:45Z

sdgx/data_models/metadata.py

+        # check missing columns
+        for each_column in self.column_list:
+            if each_column not in all_dtype_columns:
+                raise MetadaError(f"Undefined data type for column {each_column}.")
+
+        # check unfamiliar columns in dtypes
+        for each_dtype_column in all_dtype_columns:
+            if each_dtype_column not in self.column_list:
+                raise MetadaError(f"Found undefined column: {each_dtype_column}.")
+


Maybe we can use set(self.column_list) - set(all_dtype_columns) here, which we can indicate all invalid keys once.

And maybe we can rename MetadaError to MetadataInvalidError

Maybe we can use set(self.column_list) - set(all_dtype_columns) here, which we can indicate all invalid keys once.

This is a more concise way of writing.

The current implementation is because the error information raised in these two cases is different, which may make it easier for users to locate the two error types.

The current implementation is because the error information raised in these two cases is different, which may make it easier for users to locate the two error types.

There is no conflict, just use two if-expressions instead of two for-ifs.

Co-authored-by: Zhongsheng Ji <9573586@qq.com>

Unit testing also needs to be implemented.

for more information, see https://pre-commit.ci

still some cases not completed yet.

for more information, see https://pre-commit.ci

Wh1isper · 2023-12-27T08:50:35Z

Could you please add some tests on Metadata.save and Metadata.load? Although they still seem to be working fine.

I think this PR can be merged at the moment, and one or more PRs may be needed next to intro Relationship and multi-table model code.

MooooCat · 2023-12-27T08:57:01Z

Could you please add some tests on Metadata.save and Metadata.load? Although they still seem to be working fine.

No problem, I will merge after completing the tests of Metadata.save and Metadata.load.

for more information, see https://pre-commit.ci

Update file structure

de083b8

MooooCat added this to the 0.1.0 milestone Dec 19, 2023

[pre-commit.ci] auto fixes from pre-commit.com hooks

7834da0

for more information, see https://pre-commit.ci

Wh1isper mentioned this pull request Dec 20, 2023

[Good First Issue | Feature] Implement Data Processor #57

Closed

1 task

Update single_table.py

dd87ee1

Wh1isper changed the title ~~[0.1.0] Metadata Implementation~~ [0.2.0] Metadata Implementation Dec 20, 2023

Wh1isper modified the milestones: 0.1.0, 0.2.0 Dec 20, 2023

MooooCat and others added 3 commits December 21, 2023 09:45

Merge branch 'main' into feature-metadata

14e83e6

Metadata will develop based on 0.1.0.

Update metadata (still draft)

06a14f2

[pre-commit.ci] auto fixes from pre-commit.com hooks

ac89e8e

for more information, see https://pre-commit.ci

MooooCat and others added 5 commits December 23, 2023 15:11

Update metadata and reset file structure

1c322d0

[pre-commit.ci] auto fixes from pre-commit.com hooks

462b3cb

for more information, see https://pre-commit.ci

add numeric inspector

fb1565c

[pre-commit.ci] auto fixes from pre-commit.com hooks

f6da1d3

for more information, see https://pre-commit.ci

fix metadata initialization

7e65c83

MooooCat and others added 5 commits December 23, 2023 17:00

fix type hits error in py38

329755b

add inspectors

f2d50f5

[pre-commit.ci] auto fixes from pre-commit.com hooks

1f4ff08

for more information, see https://pre-commit.ci

Update relationship.py

10f0175

[pre-commit.ci] auto fixes from pre-commit.com hooks

c027feb

for more information, see https://pre-commit.ci

MooooCat and others added 3 commits December 25, 2023 11:14

add composite key list in relationship

c768a24

[pre-commit.ci] auto fixes from pre-commit.com hooks

8af1e1d

for more information, see https://pre-commit.ci

Apply suggestions from code review

ed30dad

Use a single list for single or composite primary key. Co-authored-by: Zhongsheng Ji <9573586@qq.com>

Wh1isper requested changes Dec 25, 2023

View reviewed changes

MooooCat and others added 7 commits December 25, 2023 12:02

Apply suggestions from code review

f2e4954

Key are described using list, which is compatible with single or composite foreign key. Co-authored-by: Zhongsheng Ji <9573586@qq.com>

use a single list for primary key(s)

44d0c61

[pre-commit.ci] auto fixes from pre-commit.com hooks

75985e0

for more information, see https://pre-commit.ci

Sync with main.

cfd2a11

Update expections

644568f

Update check functions

c1b239c

Unit testing also needs to be implemented.

[pre-commit.ci] auto fixes from pre-commit.com hooks

d6077d6

for more information, see https://pre-commit.ci

Wh1isper reviewed Dec 25, 2023

View reviewed changes

MooooCat and others added 8 commits December 25, 2023 17:57

Apply suggestions from code review

89e4d1f

Co-authored-by: Zhongsheng Ji <9573586@qq.com>

Apply suggestions from code review

143eeb9

Unit testing also needs to be implemented.

[pre-commit.ci] auto fixes from pre-commit.com hooks

49e02c4

for more information, see https://pre-commit.ci

update some test case

b2889da

still some cases not completed yet.

[pre-commit.ci] auto fixes from pre-commit.com hooks

8faa6e2

for more information, see https://pre-commit.ci

update test cases

4cf8db0

[pre-commit.ci] auto fixes from pre-commit.com hooks

98f4596

for more information, see https://pre-commit.ci

Merge branch 'main' into feature-metadata

b50e41b

Wh1isper marked this pull request as ready for review December 27, 2023 08:50

Wh1isper approved these changes Dec 27, 2023

View reviewed changes

MooooCat and others added 2 commits December 27, 2023 17:18

update metadata save and load test cases.

7687f77

[pre-commit.ci] auto fixes from pre-commit.com hooks

ad312c8

for more information, see https://pre-commit.ci

MooooCat merged commit 9b4c683 into main Dec 27, 2023
11 checks passed

MooooCat deleted the feature-metadata branch December 27, 2023 09:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[0.2.0] Metadata Implementation #81

[0.2.0] Metadata Implementation #81

MooooCat commented Dec 19, 2023 •

edited by Wh1isper

Loading

MooooCat commented Dec 19, 2023

sweep-ai bot commented Dec 19, 2023

Wh1isper commented Dec 20, 2023

MooooCat commented Dec 20, 2023

MooooCat commented Dec 20, 2023

MooooCat commented Dec 20, 2023

Wh1isper commented Dec 21, 2023

MooooCat commented Dec 21, 2023

codecov-commenter commented Dec 23, 2023 •

edited

Loading

Wh1isper Dec 25, 2023

MooooCat Dec 25, 2023

Wh1isper Dec 25, 2023

MooooCat commented Dec 25, 2023

Wh1isper commented Dec 25, 2023

Wh1isper Dec 25, 2023

Wh1isper Dec 25, 2023

MooooCat Dec 25, 2023

Wh1isper Dec 25, 2023 •

edited

Loading

Wh1isper commented Dec 27, 2023

MooooCat commented Dec 27, 2023

[0.2.0] Metadata Implementation #81

[0.2.0] Metadata Implementation #81

Conversation

MooooCat commented Dec 19, 2023 • edited by Wh1isper Loading

Description

Motivation and Context

How has this been tested?

Types of changes

Checklist:

MooooCat commented Dec 19, 2023

sweep-ai bot commented Dec 19, 2023

Apply Sweep Rules to your PR?

Wh1isper commented Dec 20, 2023

MooooCat commented Dec 20, 2023

MooooCat commented Dec 20, 2023

MooooCat commented Dec 20, 2023

Wh1isper commented Dec 21, 2023

MooooCat commented Dec 21, 2023

codecov-commenter commented Dec 23, 2023 • edited Loading

Codecov Report

Wh1isper Dec 25, 2023

Choose a reason for hiding this comment

MooooCat Dec 25, 2023

Choose a reason for hiding this comment

Wh1isper Dec 25, 2023

Choose a reason for hiding this comment

MooooCat commented Dec 25, 2023

Wh1isper commented Dec 25, 2023

Wh1isper Dec 25, 2023

Choose a reason for hiding this comment

Wh1isper Dec 25, 2023

Choose a reason for hiding this comment

MooooCat Dec 25, 2023

Choose a reason for hiding this comment

Wh1isper Dec 25, 2023 • edited Loading

Choose a reason for hiding this comment

Wh1isper commented Dec 27, 2023

MooooCat commented Dec 27, 2023

MooooCat commented Dec 19, 2023 •

edited by Wh1isper

Loading

codecov-commenter commented Dec 23, 2023 •

edited

Loading

Wh1isper Dec 25, 2023 •

edited

Loading