Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reading data from KNMI Data Platform #166

Merged
merged 34 commits into from
May 26, 2023
Merged

Reading data from KNMI Data Platform #166

merged 34 commits into from
May 26, 2023

Commits on Apr 14, 2023

  1. Add first methods to get data from knmi data platform

    @rubencalje
    rubencalje committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    bcddf30 View commit details
    Browse the repository at this point in the history

  2. Improve knmi_data_platform

    @rubencalje
    rubencalje committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    48fe72f View commit details
    Browse the repository at this point in the history

  3. from this day onward, I pledge to use type hints when making new func… 

    …tions
    @rubencalje
    rubencalje committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    a1d65a2 View commit details
    Browse the repository at this point in the history

  4. minor change to make os.path.isdir work

    @rubencalje
    rubencalje committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    41015d8 View commit details
    Browse the repository at this point in the history

  5. Read multiple files at once and add tests

    @rubencalje
    rubencalje committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    46bd163 View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2023

  1. create method to get api_key

    @martinvonk
    martinvonk committed Apr 15, 2023
    Configuration menu
    Copy the full SHA
    4ec8bc7 View commit details
    Browse the repository at this point in the history

  2. Update knmi_data_platform.py 

    add two type hints and move some arguments
    @martinvonk
    martinvonk committed Apr 15, 2023
    Configuration menu
    Copy the full SHA
    4dd96e0 View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2023

  1. add h5netcdf as dependency 

    needed for reading knmi_data_platform files
    @martinvonk
    martinvonk committed Apr 16, 2023
    Configuration menu
    Copy the full SHA
    d135281 View commit details
    Browse the repository at this point in the history

  2. add functions to read hdf5 files 

    - also reformats the read_dataset_from_zip
- add a function to read_nc_knmi
    @martinvonk
    martinvonk committed Apr 16, 2023
    Configuration menu
    Copy the full SHA
    5c804d2 View commit details
    Browse the repository at this point in the history

  3. update read_h5 and read_nc functions

    @martinvonk
    martinvonk committed Apr 16, 2023
    Configuration menu
    Copy the full SHA
    49de03a View commit details
    Browse the repository at this point in the history

Commits on Apr 17, 2023

  1. add read_grib_knmi function

    @martinvonk
    martinvonk committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    ddcf8e2 View commit details
    Browse the repository at this point in the history

  2. Update knmi_data_platform.py 

    fix some handling with ZipFile
    @martinvonk
    martinvonk committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    7ae146f View commit details
    Browse the repository at this point in the history

  3. update reading h5 file 

    only open h5file when reading contents, also fix bug with getting h5Dataset
    @martinvonk
    martinvonk committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    dbc642e View commit details
    Browse the repository at this point in the history

  4. isort

    @martinvonk
    martinvonk committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    6e05e44 View commit details
    Browse the repository at this point in the history

  5. Update knmi_data_platform.py 

    add check_hour function that handles hour=24
    @martinvonk
    martinvonk committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    93f8872 View commit details
    Browse the repository at this point in the history

  6. don't provide engine when reading netcdf

    @martinvonk
    martinvonk committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    8d007be View commit details
    Browse the repository at this point in the history

  7. Update knmi_data_platform.py 

    remove reading from download function
    @martinvonk
    martinvonk committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    bbdb42d View commit details
    Browse the repository at this point in the history

  8. Update knmi_data_platform.py 

    some renaming
    @martinvonk
    martinvonk committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    ef634fe View commit details
    Browse the repository at this point in the history

  9. Delete HA40_N55_202304130000_00600_GB

    @martinvonk
    martinvonk committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    7645657 View commit details
    Browse the repository at this point in the history

  10. Update knmi_data_platform.py 

    remove _knmi from function names
read dataset now also supports a list of fnames
    @martinvonk
    martinvonk committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    7285001 View commit details
    Browse the repository at this point in the history

  11. Update test_018_knmi_data_platform.py

    @martinvonk
    martinvonk committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    aa6f8c7 View commit details
    Browse the repository at this point in the history

  12. Create KNMI_Data_Platform_GRIB.tar

    @martinvonk
    martinvonk committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    b7501d2 View commit details
    Browse the repository at this point in the history

  13. Create KNMI_Data_Platform_H5.zip

    @martinvonk
    martinvonk committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    84f552e View commit details
    Browse the repository at this point in the history

  14. move hour check to read_dataset

    @martinvonk
    martinvonk committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    b74d51d View commit details
    Browse the repository at this point in the history

  15. Update knmi_data_platform.py 

    fix download_files function
    @martinvonk
    martinvonk committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    dfd8e8d View commit details
    Browse the repository at this point in the history

Commits on May 2, 2023

  1. Merge branch 'dev' into knmi_data_platform

    @martinvonk
    martinvonk committed May 2, 2023
    Configuration menu
    Copy the full SHA
    82791b6 View commit details
    Browse the repository at this point in the history

Commits on May 25, 2023

  1. full install in ci

    @martinvonk
    martinvonk committed May 25, 2023
    Configuration menu
    Copy the full SHA
    d884737 View commit details
    Browse the repository at this point in the history

  2. resolve merge conflicts setup.py?

    @martinvonk
    martinvonk committed May 25, 2023
    Configuration menu
    Copy the full SHA
    ebecab1 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'dev' into knmi_data_platform

    @martinvonk
    martinvonk committed May 25, 2023
    Configuration menu
    Copy the full SHA
    ca163e3 View commit details
    Browse the repository at this point in the history

  4. add dependencies for knmi_data_platform

    @martinvonk
    martinvonk committed May 25, 2023
    Configuration menu
    Copy the full SHA
    f62db61 View commit details
    Browse the repository at this point in the history

  5. seperate download and file reading 

    file reading took too long
    @martinvonk
    martinvonk committed May 25, 2023
    Configuration menu
    Copy the full SHA
    62adbf2 View commit details
    Browse the repository at this point in the history

  6. upload correct dataset

    @martinvonk
    martinvonk committed May 25, 2023
    Configuration menu
    Copy the full SHA
    5d9c7e1 View commit details
    Browse the repository at this point in the history

  7. fix codacy stuff and add simple docstring 

    - add timeout to request.get
- better exceptions
- add docstring
- {} instead of dict()?
- nog wat ander gejank
    @martinvonk
    martinvonk committed May 25, 2023
    Configuration menu
    Copy the full SHA
    caaa084 View commit details
    Browse the repository at this point in the history

  8. fix meta update bug

    @martinvonk
    martinvonk committed May 25, 2023
    Configuration menu
    Copy the full SHA
    1588b0c View commit details
    Browse the repository at this point in the history