Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retain data loaded from disk for multiple Calcs in a CalcSuite, rather than reloading each time #4

Open
spencerahill opened this issue Jun 24, 2015 · 2 comments
Labels
Milestone

Comments

@spencerahill
Copy link
Owner

  • Determines what the needed arrays and time subsets are for each computation, and then holds on to a copy whenever a subsequent Calc needs one identical to (or a subset of) a pre-loaded one
  • So would keep the loading functions within the Calc class, but would have the Calc check to see if the CalcSuite has the data first. OR that check is done within CalcSuite -- if it doesn't have the data already, it calls on the Calc's loading functions to get the data.

Currently, the netCDF data is loaded and freed with each Calc, even if all of them require the same data, which slows things down a lot, especially for larger computations.

@spencerahill
Copy link
Owner Author

A nascent form of this functionality has been implemented in the aospy-obj-lib repo, but it is buggy and messy.

Also, important that the location-specific file access methods (e.g. the dmget or hsmget calls required to free files saved on the /archive of NOAA GFDL) be factored out of the other code to the fullest extent possible. Eventually want to support users at other institutions, and enable them to make their necessary system calls as they deem necessary.

@spencerahill
Copy link
Owner Author

This is partially being implemented in #155.

However, the aspect of holding onto data across calculations, which would definitely be useful, is not addressed there. So leaving this open.

@spencerahill spencerahill modified the milestones: before v1.0, v0.2 Aug 31, 2017
@spencerahill spencerahill changed the title CalcSuite class that contains all Calcs specified by main Retain data loaded from disk for multiple Calcs in a CalcSuite, rather than reloading each time Aug 31, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant