You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I put it here as an issue , as not sure If you would like my pull request.
Description:
Problem statement
Big raster files in AWS issues
/tmp storage gets full with 20GB docker image limit in AWS
it takes 7 /8 hours to process a 10GB Sentinel 2 tif
Query in parser.py to be written to db , is to big
in process_quadrant bulk_create a result which is a to big query to be fired off by postgres, query string is just to big and results in postgres firing off an out of memory error.
Can not remove raster layers from django admin
Changes
GDAL vsimem for creating tiles in memory (and not in tmp files) in parser.py dest_file_name = os.path.join('/vsimem/', '{}.tif'.format(uuid.uuid4()))
added to .bulk_create second parameter with with default 50 , so query to be written to db is not to big, 2nd parameter(for
write bulk in batches of 2nd param) is new since Django (X?).
Hi I made some changes in here:
https://github.com/justRishi/django-raster/tree/improve-working-with-big-raster-files-needs-lot-of-ram
I put it here as an issue , as not sure If you would like my pull request.
Description:
Problem statement
Big raster files in AWS issues
Query in parser.py to be written to db , is to big
in process_quadrant bulk_create a result which is a to big query to be fired off by postgres, query string is just to big and results in postgres firing off an out of memory error.
Can not remove raster layers from django admin
Changes
dest_file_name = os.path.join('/vsimem/', '{}.tif'.format(uuid.uuid4()))
write bulk in batches of 2nd param) is new since Django (X?).
Solves
Drawbacks using vsimem:
Drawback using max batch-size parameter in bulkcreate:
will not work with old Django versions .
The text was updated successfully, but these errors were encountered: