Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor catalog write #587

Merged
merged 6 commits into from
Jun 7, 2024
Merged

Refactor catalog write #587

merged 6 commits into from
Jun 7, 2024

Commits on Jun 5, 2024

  1. start refactoring write_pp_catalog to populate output catalog info fr… 

    …om existing catalog
    @wrongkindofdoctor
    wrongkindofdoctor committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    0161e4d View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2024

  1. fix temp_dir_root definition in filesystem.py

    @wrongkindofdoctor
    wrongkindofdoctor committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    5787321 View commit details
    Browse the repository at this point in the history

  2. add realm back to DataSourceBase query attribute 

    remove deprecated data source classes and functions from data_sources module
    @wrongkindofdoctor
    wrongkindofdoctor committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    1082ea5 View commit details
    Browse the repository at this point in the history

  3. remove filename parsers from catalog.py 

    add chunk_freq and path to catalog_define_pp_catalog_assets columns
    @wrongkindofdoctor
    wrongkindofdoctor committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    bd1fda4 View commit details
    Browse the repository at this point in the history

  4. add realm back to query_catalog and a check for modeling realm if it … 

    …is used instead of realm

refactor the write_pp_catalog method to populate output catalog information from ingested catalog and varlist attributes instead of inferring information from output file name
    @wrongkindofdoctor
    wrongkindofdoctor committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    0fcdc02 View commit details
    Browse the repository at this point in the history

  5. explicity define query attributes to avoid issues with retaining info… 

    …rmation from prior

variable query in preprocessor
    @wrongkindofdoctor
    wrongkindofdoctor committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    7f74cfb View commit details
    Browse the repository at this point in the history