Introduce inspect_level in inspector and metadata

[MooooCat]

Description

Inspected level is a concept newly introduced in version 0.1.5.

Since a single column in the table may be marked by different inspectors at the same time, which may cause confuse.

Thus, a inspect_level is introduced when determining the specific type of a column.

Motivation and Context

A single column, can be labeled by different multiple inspector.

For example, A email column may be recognized as email, but it may also be recognized as the id column, and it may also be recognized by different inspectors at the same time identified as a discrete column, which will cause confusion in subsequent processing.

Also, you can see from our multi-table demo dataset, child table "train", the Date column is marked twice, as discrete type and date type.

We will preset different inspector levels for different inspectors, usually more specific inspectors will get higher levels, and general inspectors (like discrete) will have inspect_level. In baseclass, the inspect_level is set to 1.

How has this been tested?

Types of changes

• Maintenance (no change in code, maintain the project's CI, docs, etc.)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.

[sweep-ai]

Apply Sweep Rules to your PR?

• Apply: All new business logic should have corresponding unit tests.
• Apply: Refactor large functions to be more modular.
• Apply: Add docstrings to all functions and file headers.

This is an automated message generated by Sweep AI.

[codecov-commenter]

Codecov Report

Attention: 11 lines in your changes are missing coverage. Please review.

Comparison is base (1bbf3f5) 78.94% compared to head (4e43bdb) 79.13%.

Files	Patch %	Lines
sdgx/data_models/metadata.py	79.54%	9 Missing ⚠️
sdgx/data_models/inspectors/base.py	87.50%	2 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #113      +/-   ##
==========================================
+ Coverage   78.94%   79.13%   +0.19%     
==========================================
  Files          64       64              
  Lines        2764     2823      +59     
==========================================
+ Hits         2182     2234      +52     
- Misses        582      589       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Wh1isper]

I suggest limiting the priority to a range such as (1-100), the larger the priority, the sooner the execution.

Our built-in inspector should be presented in a hierarchy (10,20,30...), so that users can insert a custom inspector from the middle.

[Guo-Yunzhe]

Nice suggestion! I will plan it later. 從我的iPhone傳送 Zhongsheng Ji ***@***.***>於2024年1月17日 16:53寫道：  I suggest limiting the priority to a range such as (1-100), the larger the priority, the sooner the execution. Our built-in inspector should be presented in a hierarchy (10,20,30...), so that users can insert a custom inspector from the middle.

[Wh1isper]

Request some teses on inspect_level

[MooooCat]

It seems that all our package build test are failed because the SSL error of datahub.io., causing the fail of some unit test (downloading the demo dataset).

[Wh1isper]

Can a more stable data source be found? Or hitz-ids could provide a public download source like s3?

[Guo-Yunzhe]

Kangda Wu will work on this 從我的iPhone傳送 Zhongsheng Ji ***@***.***>於2024年1月19日 23:42寫道：  Can a more stable data source be found? Or hitz-ids could provide a public download source similar to s3?

[Wh1isper]

LGTM, just one more question, should we add some tests in test_metadata.py for column_inspect_level? We can leave it in next PR.

[MooooCat]

LGTM, just one more question, should we add some tests in test_metadata.py for column_inspect_level? We can leave it in next PR.

Sure, currently some test are in each inspectors, we can add test cases in test_metadata.py