Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Need to implement the ability to update archives #69

Closed
DavidT3 opened this issue Jan 25, 2023 · 5 comments
Closed

Need to implement the ability to update archives #69

DavidT3 opened this issue Jan 25, 2023 · 5 comments
Assignees
@DavidT3
Labels
enhancement New feature or request

Comments

@DavidT3
Copy link
Owner

DavidT3 commented Jan 25, 2023

Even if it doesn't change a version number or anything just yet, it needs to go in so I can resume interrupted processing without regenerating expensive files!

More context will be added to this issue soon.
@DavidT3 DavidT3 added the enhancement New feature or request label Jan 25, 2023
@DavidT3 DavidT3 self-assigned this Jan 25, 2023
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Didn't change anything that matters but having this commit so I can r…
1a660f6 
…eference to the issue I should have been pointing these commits at from the start of this branch. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
@DavidT3
Copy link
Owner Author

DavidT3 commented Sep 2, 2024

The missions can now be updated very easily, using the update() method thanks to the infrastructure for storing filtering operations that I added a while back.

Updating should be easy to add to the archives now that this method exists

DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Added internal attributes to BaseMission to store the saved usable st…
d53c2c3 
…ates of selected observations when loading back in from disk, also added the read in step in '_load_state' - this is for issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Changed the loading in of usable flags from save state to be in dicti…
c5a1415 
…onaries, with ObsIDs as keys - just a little more convenient. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Started to add checking for any data selection changes to MissionBase…
f0cb9d9 
… update() method. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Forgot the .all() call when checking to see if all previously selecte…
b7b0300 
…d obs are in the current selection, and vice versa. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Testing out whether the finding of ObsIDs that have been removed is w…
11ef784 
…orking in the update() method of BaseMission. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Testing out whether the finding of ObsIDs that have been removed is w…
c8f6b3a 
…orking in the update() method of BaseMission. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
The arrays of new and removed ObsIDs was inverted from what it should…
a5d1a18 
… be. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Made sure that I wouldn't be comparing dictionaries that include an O…
24bc07d 
…bsID that has been removed (uncommon but does happen with live missions). For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Added a bunch of comments to the update() method, as well an update m…
8d7197d 
…eta data attribute and property (to store info about what changed during an update). For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Made the update() method of BaseMission record which ObsIDs have had …
0c554c1 
…changed usable states, and included all meta information in the update_meta_info attribute. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Wasn't comparing proprietary usable state to the correct dictionary w…
526f8bb 
…hen setting up which_pr_us_ch. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Started the update() method of Archive. For issue #69
0ba7ab9 
Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Should have altered the config-capture code so that any kwargs that a…
5c268e8 
…re different from default will be captured, and every default value is just read from the function signature. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Added a process_configurations property getter (and accompanying attr…
fbe28ca 
…ibute) - this returns the configuration parameters used for each processing function that has been run. In aid of issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Hopefully XMM processing should now store the processing configuratio…
3cab4f3 
…ns in the archive they are working on. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Misunderstood what was being captured by the sas_call decorator again…
3838eb6 
… - hopefully now the arg capturing for processing configuration storage should work as intended. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
I am silly and was not writing process confs to the archive class bec…
8f3855a 
…ause I was accidentally checking if there was already an entry in extra info (which there always is as it is set before the process confs). Should be sorted now. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Made process run configurations be saved with the rest of the informa…
8f3f0bd 
…tion about an archive - should also ensure they are loaded back in. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Converting the process run configuations dictionary so that quantitie…
a3a121b 
…s can be stored as strings in the archive save file. For issue #69. Now need to make sure they are read back in correctly.

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Hopefully made sure that any quantities are loaded back in (when load…
9011725 
…ing saved archives that have had processing run) as quantities rather than just strings (which is how they were stored). For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 2, 2024
@DavidT3
Changed how process configurations are stored, and we now retain the …
a39834e 
…order in which they were executed. This will be crucial for re-running the same processing on new data. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 3, 2024
@DavidT3
Started adding the proc_lookup dictionary so that processing step nam…
5264262 
…es for particular missions can be easily and uniquely linked back to an actual function. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 3, 2024
@DavidT3
Started adding the proc_lookup dictionary so that processing step nam…
5387ff7 
…es for particular missions can be easily and uniquely linked back to an actual function. ALSO added a check to the sas_call decorator to make sure no new processes are added without a PROC_LOOKUP entry For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
@DavidT3
Copy link
Owner Author

DavidT3 commented Sep 3, 2024

Alrighty, much progress has been made today. Missions now can be updated, as can archives. Archives also save processing steps and configuration, and when they are updated the processing steps are run again, in the right order, with identical configuration. As far as I know so far that is working fine.

Need to do tomorrow:

  • Make sure the processing steps only process the ObsIDs that haven't already been processed at each step, and stop all these awful warnings I put in
  • Do a bunch more testing with multiple missions etc.
  • Add process configuration code to eROSITA call decorator as well
  • Finish populating the PROC_FUNC constant

DavidT3 added a commit that referenced this issue Sep 3, 2024
@DavidT3
Added the eROSITA mappings of function name to actual function in PRO…
17f0443 
…C_LOOKUP. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 3, 2024
@DavidT3
Started adding checks for if a process has already been run for any o…
fb578d6 
…f the relevant data to sas_call. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
@DavidT3
Copy link
Owner Author

DavidT3 commented Sep 3, 2024

I was starting to add a check for data IDs that had been run for a particular process in the sas_call decorator, which would have been neat, but it doesn't handle the problem of temporary directories that have already been created for an anticipated run. As such the processing functions (particularly the 'if not os.path.exists(final_path):' bits) need to be altered to simply not include relevant IDs that have been run already (whether they failed or not).

DavidT3 added a commit that referenced this issue Sep 3, 2024
@DavidT3
Added better checks for if processes have been run before - to all XM…
8e608e2 
…M processes in the assemble section. Indirectly for issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 3, 2024
@DavidT3
Added previous process run checking to emanom. Indirectly for issue #69
7bb46af 
Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 3, 2024
@DavidT3
Ensured that espfilt will not run on ObsID-inst-subexp combos that it…
5798ce5 
… has already run on. Indirectly for issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 3, 2024
@DavidT3
Ensured that espfilt will not run on ObsID-inst-subexp combos that it…
f951de0 
… has already run on. Indirectly for issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 3, 2024
@DavidT3
Made sure that cif build and odf ingest shouldn't run on observations…
30049de 
… that they have already been run on. Indirectly for issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 3, 2024
@DavidT3
Changed how the process_success setter adds info to the storage dicti…
58a461e 
…onary, and under what circumstances it throws a warning out to the user. Need to make the same changes on all of the similar setters for Archive class. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 3, 2024
@DavidT3
The mission update method WASN'T downloading the new data - this may …
258356e 
…be fixed now. For issue #69 sort of

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 3, 2024
@DavidT3
The mission update method WASN'T downloading the new data - this may …
04f480b 
…be fixed now. For issue #69 sort of

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 3, 2024
@DavidT3
Maybe ironed out the kinks in updating and successfully running the p…
5008569 
…rocessing steps again? For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
@DavidT3
Copy link
Owner Author

DavidT3 commented Sep 3, 2024

This feature may now be fully implemented!

DavidT3 added a commit that referenced this issue Sep 3, 2024
@DavidT3
Added a save command to the update() method of Archive. For issue #69
4588e3e 
Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 3, 2024
@DavidT3
I am a fool and was double copying the failed ObsIDs to the failed da…
402f771 
…ta part of the archive. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
@DavidT3
Copy link
Owner Author

DavidT3 commented Sep 3, 2024

As far as I can tell, everything is working as intended with XMM. I do need to test the eROSITA implementation though.

DavidT3 added a commit that referenced this issue Sep 5, 2024
@DavidT3
Added an 'Updating an existing Archive' section to the Archives tutor…
0cc5d87 
…ial. For issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 5, 2024
@DavidT3
Added _version and _last_version attributes to the Archive class, as …
3c86dde 
…well as ensuring that _version gets written out to the save json. Indirectly for issue #69

Signed-off-by: David Turner <turne540@msu.edu>
DavidT3 added a commit that referenced this issue Sep 5, 2024
@DavidT3
Added a version property to Archive. Indirectly for issue #69
7f3a082 
Signed-off-by: David Turner <turne540@msu.edu>
@DavidT3 DavidT3 closed this as completed Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DavidT3 DavidT3

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@DavidT3