Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

merge all new developments to main #27

Merged
merged 77 commits into from
Aug 12, 2024
Merged

merge all new developments to main #27

merged 77 commits into from
Aug 12, 2024

Commits on Mar 8, 2024

  1. - Added support for separate data directories for separate datasets. 

    - Fixed two smaller issues (issues #8 and #9)
    Christian Lessig committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    ed65f39 View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2024

  1. - Switched to training data in zarr 

      - Not fully functional yet, e.g. support for surface fields is missing or global forecast and similar things.
- Added support for mixed precision training.
    Christian Lessig committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    38ab84b View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2024

  1. normalization and tokenization

    @iluise
    iluise committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    29dd3e6 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2024

  1. working bert.py

    @iluise
    iluise committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    dbe1bbc View commit details
    Browse the repository at this point in the history

  2. first working version with sfc data + source info

    @iluise
    iluise committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    967013d View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2024

  1. simplify log_validate

    @iluise
    iluise committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    490df64 View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2024

  1. restructure data_writer

    @iluise
    iluise committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    2f7ecc3 View commit details
    Browse the repository at this point in the history

  2. add one comment

    @iluise
    iluise committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    36cf3e9 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2024

  1. Removed duplicate fields_tokens_masked_idx.

    Christian Lessig committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    16e160a View commit details
    Browse the repository at this point in the history

  2. modify log_attention

    @iluise
    iluise committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    27e725b View commit details
    Browse the repository at this point in the history

  3. Fixed tokenize for 2D input (but not tested or used).

    Christian Lessig committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    5886b3c View commit details
    Browse the repository at this point in the history

  4. Cleaned up some details in log_validate_BERT().

    Christian Lessig committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    1c65c7a View commit details
    Browse the repository at this point in the history

  5. merge

    @iluise
    iluise committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    fde838b View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2024

  1. fix loss

    @iluise
    iluise committed Mar 28, 2024
    Configuration menu
    Copy the full SHA
    67e9a7e View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2024

  1. Re-enabled standard data loaders.

    Christian Lessig committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    092478c View commit details
    Browse the repository at this point in the history

  2. Removed stale code.

    Christian Lessig committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    5b6641e View commit details
    Browse the repository at this point in the history

  3. Adapted forecast to only having tokens_mask_idx_list

    Christian Lessig committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    6a2c25c View commit details
    Browse the repository at this point in the history

  4. Various fixes and changing, in particular enabling again global forec… 

    …ast.
    Christian Lessig committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    b7f380a View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2024

  1. commit temporary code

    @iluise
    iluise committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    9be35b3 View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2024

  1. running global forecast evaluation

    @iluise
    iluise committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    2d86650 View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2024

  1. Changed set_global to numpy arrays for consistency.

    Christian Lessig committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    4c1f8ca View commit details
    Browse the repository at this point in the history

  2. Fixed bug in returned idx_masked for BERT.

    Christian Lessig committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    1b9e20d View commit details
    Browse the repository at this point in the history

  3. - Fixed issues in log_BERT due to bug fixing fo masked_idx. 

    - Minor clean up here and there.
    Christian Lessig committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    7c6413f View commit details
    Browse the repository at this point in the history

Commits on Apr 9, 2024

  1. validated version against main

    @iluise
    iluise committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    e0f72d1 View commit details
    Browse the repository at this point in the history

  2. - Fixed handling of shuffle() 

    - Cleaned up code in various places (e.g. removed load_data from atmorep_model which was no longer needed).
    Christian Lessig committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    6ba6521 View commit details
    Browse the repository at this point in the history

  3. fix bert_strategy

    @iluise
    iluise committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    296595a View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. fix temporal interpol, fix overlap, remove evaluate

    @iluise
    iluise committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    f28e1c9 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2024

  1. Fixed hard coded path.

    @clessig
    clessig committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    f2b0646 View commit details
    Browse the repository at this point in the history

  2. Removed load_data for training.

    Christian Lessig committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    008031b View commit details
    Browse the repository at this point in the history

  3. Merge branch 'iluise-sfc-fields' of https://github.com/clessig/atmorep … 

    …into iluise-sfc-fields
    Christian Lessig committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    6d523be View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2024

  1. new normalization from zarr

    @iluise
    iluise committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    f9b609a View commit details
    Browse the repository at this point in the history

  2. validated status

    @iluise
    iluise committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    ecc6621 View commit details
    Browse the repository at this point in the history

  3. validated new normalization

    @iluise
    iluise committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    45d4fdc View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2024

  1. temporal interpolation for multiple time steps

    @iluise
    iluise committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    e56952f View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. Adding sample/sec to console output.

    Christian Lessig committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    afbdf04 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'iluise-sfc-fields' of https://github.com/clessig/atmorep … 

    …into iluise-sfc-fields
    Christian Lessig committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    5f5efa5 View commit details
    Browse the repository at this point in the history

  3. Removed variable length batch size.

    Christian Lessig committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    77a2aac View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2024

  1. Fixes for new/old ordering of fields.

    Christian Lessig committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    ffe7e18 View commit details
    Browse the repository at this point in the history

Commits on May 21, 2024

  1. Implemented efficient fused flash-attention (i.e. flash attention wit… 

    …h heads fused in a common projection matrix). Minor fixes around.
    Christian Lessig committed May 21, 2024
    Configuration menu
    Copy the full SHA
    2c73df2 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2024

  1. Fixed bug with multi-year training data ranges.

    Christian Lessig committed May 23, 2024
    Configuration menu
    Copy the full SHA
    92b097b View commit details
    Browse the repository at this point in the history

Commits on May 27, 2024

  1. delete unused files

    @iluise
    iluise committed May 27, 2024
    Configuration menu
    Copy the full SHA
    6831ffc View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2024

  1. first implem of weight_translate + time bug fix

    @iluise
    iluise committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    e3a7faa View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2024

  1. validate MultiCrossAttentionHead

    @iluise
    iluise committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    ecee5b5 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. prepare full config example for Epicure

    Ilaria Luise committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    705f03e View commit details
    Browse the repository at this point in the history

  2. comment out requirements for wheel installation

    Ilaria Luise committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    f211001 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. increase n workers evaluate

    Ilaria Luise committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    4fcff35 View commit details
    Browse the repository at this point in the history

  2. fix path in evaluate

    Ilaria Luise committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    88ee5c0 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. new wip weighted area

    Ilaria Luise committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    e286c7e View commit details
    Browse the repository at this point in the history

  2. final weighted average LOSS

    Ilaria Luise committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    9c49ded View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. clean code version. time still off.

    Ilaria Luise committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    a9a85c5 View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2024

  1. add test

    Ilaria Luise committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    46b22b8 View commit details
    Browse the repository at this point in the history

  2. add test

    Ilaria Luise committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    fe05e28 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2024

  1. include validation tests within evaluate.py

    Ilaria Luise committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    cf3a165 View commit details
    Browse the repository at this point in the history

  2. tests now working in BERT, forecast and global_forecast mode

    Ilaria Luise committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    1f9489e View commit details
    Browse the repository at this point in the history

  3. restore values in evaluate

    Ilaria Luise committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    3868ab4 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2024

  1. Reenabled packages.

    @clessig
    clessig committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    7863acc View commit details
    Browse the repository at this point in the history

  2. Merge branch 'atmorep-dev' of github.com:clessig/atmorep into atmorep… 

    …-dev
    @clessig
    clessig committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    104a23d View commit details
    Browse the repository at this point in the history

  3. Added version requirement for torch.

    @clessig
    clessig committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    4c6111c View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2024

  1. Factored MLP into a separate file.

    @clessig
    clessig committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    56a6a39 View commit details
    Browse the repository at this point in the history

  2. Adding logging code but still needs to be used.

    @clessig
    clessig committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    7d8e24c View commit details
    Browse the repository at this point in the history

  3. Removing used file and functionality (which has moved to multifield_d… 

    …ata_sampler.py).
    @clessig
    clessig committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    9f69a5d View commit details
    Browse the repository at this point in the history

  4. - Added optimization suggested by BSC (although no performance improv… 

    …ement can be seen on ECMWF-ATOS)

- Minor clenaup.
    @clessig
    clessig committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    c8b180d View commit details
    Browse the repository at this point in the history

  5. Cleaned up unused parameters and dependencies.

    @clessig
    clessig committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    0dd99a7 View commit details
    Browse the repository at this point in the history

  6. Fixed partial merge

    Christian Lessig committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    3dd417c View commit details
    Browse the repository at this point in the history

  7. Cleaned up code.

    Christian Lessig committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    179b95f View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2024

  1. validated code

    Ilaria Luise committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    679adb3 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. Fixed bug when number of samples is larger than available samples.

    @clessig
    clessig committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    224bf91 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2024

  1. restore settings for batch size 96

    Ilaria Luise committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    5c77311 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'develop' of https://github.com/clessig/atmorep into dev… 

    …elop
    Ilaria Luise committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    debb1e8 View commit details
    Browse the repository at this point in the history

  3. restore paths

    Ilaria Luise committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    11f5d7e View commit details
    Browse the repository at this point in the history

  4. add BSC Epicure team suggestions

    Ilaria Luise committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    2afc62b View commit details
    Browse the repository at this point in the history

  5. fix local norm

    Ilaria Luise committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    df7b23c View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2024

  1. fix evaluate.py for years_val

    Ilaria Luise committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    a73be45 View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2024

  1. answer review comments

    Ilaria Luise committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    ba60b71 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #24 from clessig/iluise/head 

    Iluise/head
    @iluise
    iluise committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    db4d77f View commit details
    Browse the repository at this point in the history

  3. resolve conflicts with develop

    Ilaria Luise committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    f38ddc5 View commit details
    Browse the repository at this point in the history

  4. Merge pull request #28 from clessig/iluise/head 

    main and develop are protected. this MR is needed to solve a merging conflicts between the two
    @clessig
    clessig committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    e3412a6 View commit details
    Browse the repository at this point in the history