Consider switching to libflate for decoding

[Shnatsel]

inflate crate is only used by png, with no other high-profile users. Everybody else uses either libflate or flate2 crates, since they're much faster and have better compatibility.

Since version 0.1.25 libflate carries just one trivial unsafe block (I've killed off everything else), so its safety story is as good as inflate. There seems to be no reason not to switch.

[oyvindln]

It would be fine for decoding. For encoding however, libflate doesn't have many options for compression levels, and can't compress as well as the currently used deflate and flate2. We may want to keep deflate for encoding for now (has no unsafe use in the latest release), or alternatively swap to using flate2 for everything once the unsafe bits of the encoder and api bridge in miniz_oxide have been removed or thoroughly checked for soundness.

[HeroicKatora]

Thanks for the mention, I certainly was not aware of libflate and it seems to be progressing rather speedily. Its development seems (from commit log) focussed on decoding right now, not necessarily a bad thing, and results look genuinely interesting. Imo, having upsides and downsides in both crates is a good result of competing implementations. Using both has a tradeoff of build time and should be an implementation details (with no public features) to not be confusing. Also, if anyone spot ways to improve libflate maybe we get the best of both worlds?

[Shnatsel]

I concur with keeping deflate for encoding, at least for now. However, for decoding inflate seems to be strictly inferior than the alternatives at this point. I've updated the title to make it more clear.

[Shnatsel]

miniz_oxide (the Rust backend for flate2) has been making great strides in terms of safety lately, and it is much faster than both inflate and libflate. It's now a viable contender. However, some of the (upcoming) safety improvements are going to require Rust 1.34

[HeroicKatora]

Given that image has switched to 1.34 as well that should be no problem.

[Shnatsel]

miniz_oxide (the Rust backend for flate2) is now 100% safe, forbids unsafe code, and is 2.5x faster than libflate. There is an even bigger gap compared to inflate. It also does both encoding and decoding.

There is still some unsafety in flate2 even when using miniz_oxide backend, and potentially devastating one too, so I believe the best bet is to use miniz_oxide directly.

[HeroicKatora]

I'm having trouble integrating miniz_oxide as stream decoding is awkward in the current interface (and lacks some documentation). It's not as simple as switching the inflater class as inflate does all the necessary internal buffering. It also needs no explicit indication for EOS (although I'm not sure how it does that) which is a problem insofar as PNG allows an arbitrary number of consecutive IDAT chunks containing compressed data, sliced at arbitrary positions within the compressed byte stream. Properly detecting the end of the image data requires some work on the internal state machine.

Integration work in this branch: https://github.com/image-rs/image-png/tree/miniz-oxide

[HeroicKatora]

Note: It's currently failing tests due to panic! within miniz_oxide:

tests/pngsuite/PngSuite.png: thread 'render_images' panicked at 'index out of bounds: the len is 65536 but the index is 65536', /home/andreas/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.3/src/inflate/core.rs:761:34
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.34/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.34/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:47
   3: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:36
   4: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:200
   5: std::panicking::default_hook
             at src/libstd/panicking.rs:211
   6: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:477
   7: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:384
   8: rust_begin_unwind
             at src/libstd/panicking.rs:311
   9: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
  10: core::panicking::panic_bounds_check
             at src/libcore/panicking.rs:61
  11: miniz_oxide::inflate::core::transfer
             at /home/andreas/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.3/src/inflate/core.rs:761
  12: miniz_oxide::inflate::core::decompress_inner
             at /home/andreas/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.3/src/inflate/core.rs:1545
  13: miniz_oxide::inflate::core::decompress
             at /home/andreas/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.3/src/inflate/core.rs:1030
  14: png::decoder::stream::ZlibStream::decompress
             at src/decoder/stream.rs:705
  15: png::decoder::stream::StreamingDecoder::next_state
             at src/decoder/stream.rs:395
  16: png::decoder::stream::StreamingDecoder::update
             at src/decoder/stream.rs:200
  17: png::decoder::ReadDecoder::decode_next
             at ./src/decoder/mod.rs:146
  18: png::decoder::Reader::next_raw_interlaced_row
             at ./src/decoder/mod.rs:471
  19: png::decoder::Reader::next_interlaced_row
             at ./src/decoder/mod.rs:285
  20: png::decoder::Reader::next_row
             at ./src/decoder/mod.rs:273
  21: png::decoder::Reader::next_frame
             at ./src/decoder/mod.rs:264
  22: check_testimages::render_images::{{closure}}
             at tests/check_testimages.rs:72
  23: check_testimages::process_images
             at tests/check_testimages.rs:29
  24: check_testimages::render_images
             at tests/check_testimages.rs:68
  25: check_testimages::render_images::{{closure}}
             at tests/check_testimages.rs:66
  26: core::ops::function::FnOnce::call_once
             at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/libcore/ops/function.rs:235
  27:  as core::ops::function::FnOnce>::call_once
             at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/liballoc/boxed.rs:787
  28: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:80
  29: std::panicking::try
             at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/libstd/panicking.rs:275
  30: std::panic::catch_unwind
             at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/libstd/panic.rs:394
  31: test::run_test::run_test_inner::{{closure}}
             at src/libtest/lib.rs:1408
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[Shnatsel]

flate2 provides a higher-level wrapper on top of miniz_oxide that allows more convenient stream decoding. However, flate2 creates buffer of uninitialized memory as the output, and not overwriting that buffer fully could be a devastating security vulnerability for an image decoder. Perhaps flate2 could optionally zero-initialize the buffer for high-assurance use cases?

[Shnatsel]

Also that panic is very cool because I've fuzzed miniz_oxide and couldn't find any panics. Please report it on the issue tracker!

[HeroicKatora]

The use of the wrappers from libflate likely requires some larger reworks. It consumes a generic BufRead (which png would provide) and consequently interprets an empty input buffer as eof. We would need to keep reading more PNG chunks to check if there is more data to refill the input buffer. Communicating via an io::* interface also means io::Error everywhere instead of differentiating between corrupt stream and real io errors.

Also that panic is very cool because I've fuzzed miniz_oxide and couldn't find any panics. Please report it on the issue tracker!

The integration above uses the inflate::core interface directly instead of relying on decompress_to_vec. I think this was not covered by any fuzzing yet.