You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If RASTER_USE_CELERY = True and (RASTER_PARSE_SINGLE_TASK = False or not set) then
a temp file is created multiple times in def open_raster_file in RasterLayerParser in parser.py
Also when not in the right reprojection, the projection is done multiple times.
Why problem
Big raster files are copied in my case 4 times, processed by GDAL 4 times . and sometimes (when not in the right projection) 4 times reprojected.
How tested
by adding self.log to print out tmp file creation resulting in:
How to mitigate
put RASTER_PARSE_SINGLE_TASK = True in settings , but meaning will not use concurrency to process raster file
Possible solution to process parallel and not duplicate work
check that only 1 tmp folder is created :
so this line in parser.py should change self.tmpdir = tempfile.mkdtemp(dir=raster_workdir (as always unique)
self.dataset in parser.py (in class RasterLayerParser) should be shared by all parallel tasks for same raster file
The text was updated successfully, but these errors were encountered:
Problem
If RASTER_USE_CELERY = True and (RASTER_PARSE_SINGLE_TASK = False or not set) then
a temp file is created multiple times in def open_raster_file in RasterLayerParser in parser.py
Also when not in the right reprojection, the projection is done multiple times.
Why problem
Big raster files are copied in my case 4 times, processed by GDAL 4 times . and sometimes (when not in the right projection) 4 times reprojected.
How tested
by adding self.log to print out tmp file creation resulting in:
![image](https://user-images.githubusercontent.com/13424954/149962226-1f0e08ee-ddb8-46c1-82e6-2003c9f550dd.png)
How to mitigate
put RASTER_PARSE_SINGLE_TASK = True in settings , but meaning will not use concurrency to process raster file
Possible solution to process parallel and not duplicate work
so this line in parser.py should change self.tmpdir = tempfile.mkdtemp(dir=raster_workdir (as always unique)
The text was updated successfully, but these errors were encountered: