Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pull: Support NPZ format in current transformer #167

Open
wants to merge 92 commits into
base: feature-npz-support
Choose a base branch
from
Open

pull: Support NPZ format in current transformer #167

wants to merge 92 commits into from

Commits on Mar 21, 2024

  1. added main.py with demo code

    @sam3179
    sam3179 committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    69fe003 View commit details
    Browse the repository at this point in the history

  2. renamed demo file from "main.py" to "demo.py"

    @bdiverbridgewater
    bdiverbridgewater committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    ebf8b3a View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2024

  1. added theoretic method calls for issue 33 and commented them out

    @sam3179
    sam3179 committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    4a5d1e9 View commit details
    Browse the repository at this point in the history

  2. added output file in npz format for column_data_list data in _synchro… 

    …nous_transform method
    @sam3179
    sam3179 committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    0afbe83 View commit details
    Browse the repository at this point in the history

Commits on Mar 26, 2024

  1. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    9ed4f55 View commit details
    Browse the repository at this point in the history

  2. commit

    @SirisNeupane
    SirisNeupane committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    f2c3e13 View commit details
    Browse the repository at this point in the history

  3. added input parameter for _synchronous_transform method and added npz… 

    … format where the method is called
    @sam3179
    sam3179 committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    3c6d232 View commit details
    Browse the repository at this point in the history

  4. Test commit

    @c3kimball
    c3kimball committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    d947053 View commit details
    Browse the repository at this point in the history

  5. added print statements for debugging. create raw_data and DataTransfo… 

    …rmer variables
    @sam3179
    sam3179 committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    c79506e View commit details
    Browse the repository at this point in the history

  6. moved output column data into its own method

    @sam3179
    sam3179 committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    ff6384a View commit details
    Browse the repository at this point in the history

  7. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    a97b1a3 View commit details
    Browse the repository at this point in the history

  8. Adding trimmed data set to Git

    @c3kimball
    c3kimball committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    4c1f3cc View commit details
    Browse the repository at this point in the history

  9. Adding parameters to _synchronous_transform() and transform()

    @c3kimball
    c3kimball committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    d94ae6d View commit details
    Browse the repository at this point in the history

  10. Using trimmed dataset

    @c3kimball
    c3kimball committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    84f27dc View commit details
    Browse the repository at this point in the history

  11. Starting a test function for saving an npz file

    @c3kimball
    c3kimball committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    6f9acb3 View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2024

  1. Create data_transformer_test.py 

    Adding tests for DataTransformer._output_column_data method
-Making sure it correctly processes 'npz' output type by calling numpy.savez with expected arguments.
- Verify appropriate error handling and messaging for unsupported output types.
    @sirisneupane123
    sirisneupane123 committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    d4ebdfb View commit details
    Browse the repository at this point in the history

  2. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    f8dc42d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' of https://github.com/sam3179/synthetic-data-gene… 

    …rator into Craig

# Conflicts:
#	demo.py
    @c3kimball
    c3kimball committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    e3b744a View commit details
    Browse the repository at this point in the history

  4. Creating a hardcoded example of saving the transformed data into an n… 

    …pz file
    @c3kimball
    c3kimball committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    7f37ad7 View commit details
    Browse the repository at this point in the history

  5. Modifying demo to print out the transformed data from the npz file

    @c3kimball
    c3kimball committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    be30f58 View commit details
    Browse the repository at this point in the history

  6. Adding print statements as feedback to the user of what the demo is d… 

    …oing
    @c3kimball
    c3kimball committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    b54381c View commit details
    Browse the repository at this point in the history

  7. Getting rid of hardcoded paths, and now asks the user for a path

    @c3kimball
    c3kimball committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    350335c View commit details
    Browse the repository at this point in the history

  8. updating gitignore

    @c3kimball
    c3kimball committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    e661ff9 View commit details
    Browse the repository at this point in the history

  9. removed unneeded code and commented out unnecessary code

    @sam3179
    sam3179 committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    40595ce View commit details
    Browse the repository at this point in the history

  10. moved _output_column_data() to its proper file location, created _dem… 

    …onstrate_issue_33() to demonstrate solved issue 33
    @sam3179
    sam3179 committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    0d8ffdd View commit details
    Browse the repository at this point in the history

  11. Merge remote-tracking branch 'origin/main'

    @sam3179
    sam3179 committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    30991e1 View commit details
    Browse the repository at this point in the history

  12. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    5488de1 View commit details
    Browse the repository at this point in the history

  13. moved pytest to proper file location

    @sam3179
    sam3179 committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    90e3f51 View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2024

  1. Merge branch 'main' of https://github.com/sam3179/synthetic-data-gene… 

    …rator
    @SirisNeupane
    SirisNeupane committed Mar 28, 2024
    Configuration menu
    Copy the full SHA
    983f3f3 View commit details
    Browse the repository at this point in the history

  2. added output_path input parameter like issue asks for, for _output_co… 

    …lumn_data
    @sam3179
    sam3179 committed Mar 28, 2024
    Configuration menu
    Copy the full SHA
    a087165 View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2024

  1. Beginning work on Issue hitsz-ids#136

    @c3kimball
    c3kimball committed Apr 21, 2024
    Configuration menu
    Copy the full SHA
    6f5bc80 View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. Merge remote-tracking branch 'origin/main' 

    # Conflicts:
#	demo.py
#	sdgx/models/components/optimize/sdv_ctgan/data_transformer.py
    @c3kimball
    c3kimball committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    1eeddca View commit details
    Browse the repository at this point in the history

  2. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    890f066 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2024

  1. created personal demo file

    @sam3179
    sam3179 committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    b59f6d7 View commit details
    Browse the repository at this point in the history

  2. personal issue 142: creates a list to hold a string with column detai… 

    …ls for each column
    @sam3179
    sam3179 committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    e78e43a View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/main'

    @sam3179
    sam3179 committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    28009ff View commit details
    Browse the repository at this point in the history

  4. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    cf3e614 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2024

  1. decreased the number of example elements in demonstrate npz issue

    @sam3179
    sam3179 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    83d00c5 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2024

  1. personal issue 142: added column details for int64 data type, column … 

    …details min, max, mean and standard deviation
    @sam3179
    sam3179 committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    06ef889 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/main' 

    # Conflicts:
#	sam_demo.py
    @sam3179
    sam3179 committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    e745f83 View commit details
    Browse the repository at this point in the history

  3. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    ddfd673 View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2024

  1. removed unneeded imports

    @sam3179
    sam3179 committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    3766368 View commit details
    Browse the repository at this point in the history

  2. personal issue 142: added column details for datetime64 data type. co… 

    …lumn details start date and end date
    @sam3179
    sam3179 committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    135ef4c View commit details
    Browse the repository at this point in the history

  3. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    c9558b8 View commit details
    Browse the repository at this point in the history

  4. personal issue 142: added column details for category data type. colu… 

    …mn details number of all categories and number of unique categories
    @sam3179
    sam3179 committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    a924417 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/main' 

    # Conflicts:
#	add_column_descriptions.py
    @sam3179
    sam3179 committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    06f27c0 View commit details
    Browse the repository at this point in the history

  6. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    baa831f View commit details
    Browse the repository at this point in the history

  7. personal issue 142: added column details for object data type. column… 

    … details number of all objects and number of unique objects
    @sam3179
    sam3179 committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    e6e4565 View commit details
    Browse the repository at this point in the history

  8. group issue 137: removed print statements where loggers were added

    @sam3179
    sam3179 committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    78ef872 View commit details
    Browse the repository at this point in the history

  9. group issue 137: added new logger indicating CSVConnector initialization

    @sam3179
    sam3179 committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    300d4e8 View commit details
    Browse the repository at this point in the history

  10. group issue 137: added new logger indicating Synthesizer initialization

    @sam3179
    sam3179 committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    c1508ec View commit details
    Browse the repository at this point in the history

  11. group issue 137: added new logger indicating demo data downloading

    @sam3179
    sam3179 committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    842866d View commit details
    Browse the repository at this point in the history

  12. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    01abeb4 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2024

  1. group issue 137: added new logger indicating a cleanup of the dataloa… 

    …der cache
    @c3kimball
    c3kimball committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    b6c7502 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/main' 

    # Conflicts:
#	extend_bug_demo.py
    @c3kimball
    c3kimball committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    8836d45 View commit details
    Browse the repository at this point in the history

  3. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    49d354d View commit details
    Browse the repository at this point in the history

  4. group issue 137: added new logger indicating the use of synchronous_t… 

    …ransform() from issue: 33
    @c3kimball
    c3kimball committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    d538a84 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/main'

    @c3kimball
    c3kimball committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    be5efb7 View commit details
    Browse the repository at this point in the history

  6. Updating logger message, both now say "new logger"

    @c3kimball
    c3kimball committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    82cc7df View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2024

  1. Add log for confirming data connector initialization 

    This log entry is placed after initializing the data connector to confirm which connector (file type, database, etc.) is being used. It is useful for ensuring that the correct data source setup is in place, especially in a configuration where multiple connectors could be employed.
    @sirisneupane123
    sirisneupane123 committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    a491aab View commit details
    Browse the repository at this point in the history

  2. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    8556b0b View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' of https://github.com/sam3179/synthetic-data-gene… 

    …rator
    @SirisNeupane
    SirisNeupane committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    7afe0bb View commit details
    Browse the repository at this point in the history

  4. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    e32fc6d View commit details
    Browse the repository at this point in the history

  5. Added a log that specifies the path from which a model is loaded, hel… 

    …ping in the debugging of issues related to incorrect model versions being used or misconfigured paths.
    @SirisNeupane
    SirisNeupane committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    8b05930 View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'origin/main'

    @SirisNeupane
    SirisNeupane committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    4b5d1b2 View commit details
    Browse the repository at this point in the history

  7. This log entry helps in tracking whether the metadata is being loaded… 

    … from a file or directly passed as an object. It's particularly useful when errors may be associated with metadata formatting or source, or maybe in complex systems where multiple metadata configurations might exist.
    @SirisNeupane
    SirisNeupane committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    9f37efc View commit details
    Browse the repository at this point in the history

  8. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    2dfefa6 View commit details
    Browse the repository at this point in the history

  9. group issue 137: added new logger indicating when we are fitting the … 

    …synthesizer with metadata and data processors
    @sam3179
    sam3179 committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    65f366b View commit details
    Browse the repository at this point in the history

  10. personal issue 142: made method return a string like issue asked for

    @sam3179
    sam3179 committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    5381b37 View commit details
    Browse the repository at this point in the history

  11. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    ce5fa44 View commit details
    Browse the repository at this point in the history

  12. Adding the FAQ file 

    FAQ: by Siris Neupane
    @sirisneupane123
    sirisneupane123 committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    e2722e9 View commit details
    Browse the repository at this point in the history

  13. Delete Github Issue hitsz-ids#155.docx

    @sirisneupane123
    sirisneupane123 committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    9bf3baa View commit details
    Browse the repository at this point in the history

  14. Create GitHubIssueNumber#155.md 

    FAQ by Siris Neupane
    @sirisneupane123
    sirisneupane123 committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    8bbe491 View commit details
    Browse the repository at this point in the history

  15. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    cd4047f View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. Implementing save_extend() that successfully saves any objects added … 

    …to a metadata object to a json file
    @c3kimball
    c3kimball committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    7cddb24 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/main'

    @c3kimball
    c3kimball committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    485d886 View commit details
    Browse the repository at this point in the history

  3. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    feee9ad View commit details
    Browse the repository at this point in the history

  4. Adding some error handling for save_extend()

    @c3kimball
    c3kimball committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    601edb5 View commit details
    Browse the repository at this point in the history

  5. Cleaning up and removing tests prints from the demo

    @c3kimball
    c3kimball committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    550c9bc View commit details
    Browse the repository at this point in the history

  6. Cleaning up merge conflicts

    @c3kimball
    c3kimball committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    e3b5c5c View commit details
    Browse the repository at this point in the history

  7. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    202770e View commit details
    Browse the repository at this point in the history

  8. personal issue 142: adjusted demo file

    @sam3179
    sam3179 committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    377404b View commit details
    Browse the repository at this point in the history

  9. personal issue 142: moved method contents to file where issue asks, i… 

    …nstead of production file
    @sam3179
    sam3179 committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    10a74bd View commit details
    Browse the repository at this point in the history

Commits on May 1, 2024

  1. Adding data for the automated tests

    @c3kimball
    c3kimball committed May 1, 2024
    Configuration menu
    Copy the full SHA
    dc48293 View commit details
    Browse the repository at this point in the history

  2. Completing tests for save_extend()

    @c3kimball
    c3kimball committed May 1, 2024
    Configuration menu
    Copy the full SHA
    28d6b35 View commit details
    Browse the repository at this point in the history

  3. Updating the doc string and clarifying the comments a little more

    @c3kimball
    c3kimball committed May 1, 2024
    Configuration menu
    Copy the full SHA
    6200bec View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/main' 

    # Conflicts:
#	sdgx/data_models/metadata.py
    @c3kimball
    c3kimball committed May 1, 2024
    Configuration menu
    Copy the full SHA
    604ff74 View commit details
    Browse the repository at this point in the history

  5. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed May 1, 2024
    Configuration menu
    Copy the full SHA
    2c90a86 View commit details
    Browse the repository at this point in the history

  6. personal issue 142: updated method

    @sam3179
    sam3179 committed May 1, 2024
    Configuration menu
    Copy the full SHA
    2703ec6 View commit details
    Browse the repository at this point in the history

  7. personal issue 142: added demonstration of my issue to main demo file

    @sam3179
    sam3179 committed May 1, 2024
    Configuration menu
    Copy the full SHA
    cbb85ac View commit details
    Browse the repository at this point in the history

  8. personal issue 142: added pytest, implmented tests for integer column… 

    … type and object column type
    @sam3179
    sam3179 committed May 1, 2024
    Configuration menu
    Copy the full SHA
    9d284ee View commit details
    Browse the repository at this point in the history

  9. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed May 1, 2024
    Configuration menu
    Copy the full SHA
    83bc02c View commit details
    Browse the repository at this point in the history