Memory exhaustion and crash on malformed input

[Shnatsel]

Decoding any of the attached files triggers a crash with the following error message: memory allocation of 136902082592 bytes failedAborted

tiff-oom-crashes.zip

The exact reproduction code can be found in #28. Found via AFL.rs, tested on image-tiff version 0.2.2

Most decoding libraries face this issue at some point. This is usually solved by limiting the amount of allocated memory to some sane default, and letting people override it if they're really dealing with enormous amounts of data. In Rust we can easily allow the API user to override these limits via the builder pattern.

See https://libpng.sourceforge.io/decompression_bombs.html for more info on how a similar issue was solved in libpng. See also the Limits struct from flif crate.

[birktj]

Thanks for the bug report! I think that using a Limits struct is a good idea, but instead of limiting image width and height like in image-png, we should probably limit the size of a DecodingBuffer. This is so we can use limits in a good way together with #26. When I was working on image-rs/image#860 I had to remove the limits completely so that it would be possible to decode large images, now I probably have to go back and take another look at that.

[Shnatsel]

I'm pretty sure image-png will accept a PR to set limits via buffer size instead of image dimensions, especially if you have a good rationale. The current solution was a result of a push to get something done after months and months of inactivity.