Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

merge all new developments to main #27

Merged
merged 77 commits into from
Aug 12, 2024
Merged

merge all new developments to main #27

merged 77 commits into from
Aug 12, 2024

Conversation

iluise
Copy link
Collaborator

@iluise iluise commented Aug 12, 2024

Main updates:

  • move to zarr files
  • include normalisations within zarr preprocessed files
  • use flash attention
  • partially rewrite I/O and clean up the unused files
  • fix time shift with respect to ERA5. Pre-print runs made with the old version of the code are still supported

Christian Lessig and others added 30 commits March 8, 2024 19:27
- Added support for separate data directories for separate datasets.
ed65f39 
- Fixed two smaller issues (issues #8 and #9)
- Switched to training data in zarr
38ab84b 
  - Not fully functional yet, e.g. support for surface fields is missing or global forecast and similar things.
- Added support for mixed precision training.
@iluise
normalization and tokenization
29dd3e6
@iluise
working bert.py
dbe1bbc
@iluise
first working version with sfc data + source info
967013d
@iluise
simplify log_validate
490df64
@iluise
restructure data_writer
2f7ecc3
@iluise
add one comment
36cf3e9
Removed duplicate fields_tokens_masked_idx.
16e160a
@iluise
modify log_attention
27e725b
Fixed tokenize for 2D input (but not tested or used).
5886b3c
Cleaned up some details in log_validate_BERT().
1c65c7a
@iluise
merge
fde838b
@iluise
fix loss
67e9a7e
Re-enabled standard data loaders.
092478c
Removed stale code.
5b6641e
Adapted forecast to only having tokens_mask_idx_list
6a2c25c
Various fixes and changing, in particular enabling again global forec…
b7f380a 
…ast.
@iluise
commit temporary code
9be35b3
@iluise
running global forecast evaluation
2d86650
Changed set_global to numpy arrays for consistency.
4c1f8ca
Fixed bug in returned idx_masked for BERT.
1b9e20d
- Fixed issues in log_BERT due to bug fixing fo masked_idx.
7c6413f 
- Minor clean up here and there.
@iluise
validated version against main
e0f72d1
- Fixed handling of shuffle()
6ba6521 
- Cleaned up code in various places (e.g. removed load_data from atmorep_model which was no longer needed).
@iluise
fix bert_strategy
296595a
@iluise
fix temporal interpol, fix overlap, remove evaluate
f28e1c9
@clessig
Fixed hard coded path.
f2b0646
Removed load_data for training.
008031b
Merge branch 'iluise-sfc-fields' of https://github.com/clessig/atmorep …
6d523be 
…into iluise-sfc-fields
Ilaria Luise and others added 27 commits July 24, 2024 18:26
add test
46b22b8
add test
fe05e28
include validation tests within evaluate.py
cf3a165
tests now working in BERT, forecast and global_forecast mode
1f9489e
restore values in evaluate
3868ab4
@clessig
Reenabled packages.
7863acc
@clessig
Merge branch 'atmorep-dev' of github.com:clessig/atmorep into atmorep…
104a23d 
…-dev
@clessig
Added version requirement for torch.
4c6111c
@clessig
Factored MLP into a separate file.
56a6a39
@clessig
Adding logging code but still needs to be used.
7d8e24c
@clessig
Removing used file and functionality (which has moved to multifield_d…
9f69a5d 
…ata_sampler.py).
@clessig
- Added optimization suggested by BSC (although no performance improv…
c8b180d 
…ement can be seen on ECMWF-ATOS)

- Minor clenaup.
@clessig
Cleaned up unused parameters and dependencies.
0dd99a7
Fixed partial merge
3dd417c
Cleaned up code.
179b95f
validated code
679adb3
@clessig
Fixed bug when number of samples is larger than available samples.
224bf91
restore settings for batch size 96
5c77311
Merge branch 'develop' of https://github.com/clessig/atmorep into dev…
debb1e8 
…elop
restore paths
11f5d7e
add BSC Epicure team suggestions
2afc62b
fix local norm
df7b23c
fix evaluate.py for years_val
a73be45
answer review comments
ba60b71
@iluise
Merge pull request #24 from clessig/iluise/head
db4d77f 
Iluise/head
resolve conflicts with develop
f38ddc5
@clessig
Merge pull request #28 from clessig/iluise/head
e3412a6 
main and develop are protected. this MR is needed to solve a merging conflicts between the two
@clessig clessig self-assigned this Aug 12, 2024
@iluise iluise merged commit ac3c433 into main Aug 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@clessig clessig clessig approved these changes

Assignees

@clessig clessig

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@iluise @clessig