Crash on fuzzing encode

[tmatth]

Testcase attached.

tmatth@bellini:/big-repos/rav1e$ RUST_LOG=debug fuzz/target/x86_64-unknown-linux-gnu/debug/encode fuzz/artifacts/encode/crash-f0c957104bb1b80c9d125d9c8cbb3f06fbf2ab1a 
INFO: Seed: 3796335402
INFO: Loaded 1 modules   (296716 guards): 296716 [0x5573c0073430, 0x5573c0195060), 
fuzz/target/x86_64-unknown-linux-gnu/debug/encode: Running 1 inputs 1 time(s) each.
Running: fuzz/artifacts/encode/crash-f0c957104bb1b80c9d125d9c8cbb3f06fbf2ab1a
 DEBUG rav1e::fuzzing > config = Config {
    enc: EncoderConfig {
        width: 1,
        height: 1,
        bit_depth: 8,
        chroma_sampling: Cs420,
        chroma_sample_position: Unknown,
        pixel_range: Limited,
        color_description: None,
        mastering_display: None,
        content_light: None,
        still_picture: false,
        time_base: Rational {
            num: 17179869188,
            den: 17179869188,
        },
        min_key_frame_interval: 0,
        max_key_frame_interval: 1,
        reservoir_frame_delay: None,
        low_latency: false,
        quantizer: 4398046512128,
        min_quantizer: 0,
        bitrate: 4,
        tune: Psychovisual,
        tile_cols: 0,
        tile_rows: 0,
        tiles: 0,
        rdo_lookahead_frames: 17179869188,
        speed_settings: SpeedSettings {
            min_block_size: BLOCK_64X64,
            multiref: false,
            fast_deblock: true,
            reduced_tx_set: true,
            tx_domain_distortion: true,
            tx_domain_rate: false,
            encode_bottomup: false,
            rdo_tx_decision: false,
            prediction_modes: Simple,
            include_near_mvs: false,
            no_scene_detection: true,
            diamond_me: true,
            cdef: true,
            quantizer_rdo: false,
            use_satd_subpel: false,
        },
        show_psnr: false,
        train_rdo: false,
    },
    threads: 1,
}
 INFO  rav1e::api     > CPU Feature Level: AVX2
 DEBUG rav1e::fuzzing > ctx.receive_packet() = Err(
    NeedMoreData,
)
 DEBUG rav1e::fuzzing > ctx.receive_packet() = Err(
    NeedMoreData,
)
 DEBUG rav1e::fuzzing > ctx.receive_packet() = Err(
    NeedMoreData,
)
==4236== ERROR: libFuzzer: deadly signal
    #0 0x5573bd7f5341 in __sanitizer_print_stack_trace /rustc/0b36e9dea3f2ff25b1d0df2669836c33cce89ae5/src/llvm-project/compiler-rt/lib/asan/asan_stack.cc:86:3
    #1 0x5573bfa54489 in fuzzer::PrintStackTrace() /home/tmatth/.cargo/git/checkouts/libfuzzer-sys-e07fde05820d7bc6/4a41319/libfuzzer/FuzzerUtil.cpp:206:38
    #2 0x5573bfa6b822 in fuzzer::Fuzzer::CrashCallback() /home/tmatth/.cargo/git/checkouts/libfuzzer-sys-e07fde05820d7bc6/4a41319/libfuzzer/FuzzerLoop.cpp:237:18
    #3 0x5573bfa6b6d7 in fuzzer::Fuzzer::StaticCrashSignalCallback() /home/tmatth/.cargo/git/checkouts/libfuzzer-sys-e07fde05820d7bc6/4a41319/libfuzzer/FuzzerLoop.cpp:209:19
    #4 0x5573bfa42398 in fuzzer::CrashHandler(int, siginfo_t*, void*) /home/tmatth/.cargo/git/checkouts/libfuzzer-sys-e07fde05820d7bc6/4a41319/libfuzzer/FuzzerUtilPosix.cpp:36:36
    #5 0x7fba54ab6f3f  (/lib/x86_64-linux-gnu/libpthread.so.0+0x13f3f)
    #6 0x7fba548e1ed6 in __libc_signal_restore_set /build/glibc-KRRWSm/glibc-2.29/signal/../sysdeps/unix/sysv/linux/internal-signals.h:84:10
    #7 0x7fba548e1ed6 in gsignal /build/glibc-KRRWSm/glibc-2.29/signal/../sysdeps/unix/sysv/linux/raise.c:48:3
    #8 0x7fba548c3534 in abort /build/glibc-KRRWSm/glibc-2.29/stdlib/abort.c:79:7
    #9 0x5573bfa87bc6 in std::sys::unix::abort_internal::he0202c5b60f82d93 /rustc/0b36e9dea3f2ff25b1d0df2669836c33cce89ae5/src/libstd/sys/unix/mod.rs:156:4
    #10 0x5573bfa82e95 in std::process::abort::h20dcc6ac8a6c7c53 /rustc/0b36e9dea3f2ff25b1d0df2669836c33cce89ae5/src/libstd/process.rs:1575:13
    #11 0x5573bfa2866b in libfuzzer_sys::initialize::_$u7b$$u7b$closure$u7d$$u7d$::h06bab99fc123acb5 /home/tmatth/.cargo/git/checkouts/libfuzzer-sys-e07fde05820d7bc6/4a41319/src/lib.rs:32:12
    #12 0x5573bfa862db in std::panicking::rust_panic_with_hook::haae23c48d2056f90 /rustc/0b36e9dea3f2ff25b1d0df2669836c33cce89ae5/src/libstd/panicking.rs:481:16
    #13 0x5573bf84bbc3 in std::panicking::begin_panic::hda2b17b7ae1ab72a /rustc/0b36e9dea3f2ff25b1d0df2669836c33cce89ae5/src/libstd/panicking.rs:411:4
    #14 0x5573bef5fcce in rav1e::api::ContextInner$LT$T$GT$::guess_frame_subtypes::hc1a8218015073823 /big-repos/rav1e/src/api/mod.rs:2390:10
    #15 0x5573bf172f39 in rav1e::rate::RCState::select_qi::hc3a0113849dc279b /big-repos/rav1e/src/rate.rs:917:12
    #16 0x5573bef5a6e8 in rav1e::api::ContextInner$LT$T$GT$::receive_packet::he5515072bf2f4ae0 /big-repos/rav1e/src/api/mod.rs:2115:20
    #17 0x5573bef29241 in rav1e::api::Context$LT$T$GT$::receive_packet::_$u7b$$u7b$closure$u7d$$u7d$::h14a1b946d94409cf /big-repos/rav1e/src/api/mod.rs:1166:20
    #18 0x5573bf0b05ba in rayon_core::thread_pool::ThreadPool::install::_$u7b$$u7b$closure$u7d$$u7d$::haba058acad71452c /home/tmatth/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/thread_pool/mod.rs:132:39
    #19 0x5573bf03d36b in rayon_core::registry::Registry::in_worker_cold::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::he6957e77b8550035 /home/tmatth/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:501:20
    #20 0x5573bf5ddd7c in _$LT$rayon_core..job..StackJob$LT$L$C$F$C$R$GT$$u20$as$u20$rayon_core..job..Job$GT$::execute::call::_$u7b$$u7b$closure$u7d$$u7d$::hac0d17bbf0ff3fcc /home/tmatth/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/job.rs:113:20
    #21 0x5573bf5e314e in _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hfeedbb3d599a4bb3 /rustc/0b36e9dea3f2ff25b1d0df2669836c33cce89ae5/src/libstd/panic.rs:315:8
    #22 0x5573bf628136 in std::panicking::try::do_call::hd837f770807b0bc6 /rustc/0b36e9dea3f2ff25b1d0df2669836c33cce89ae5/src/libstd/panicking.rs:296:39
    #23 0x5573bfa87f38 in __rust_maybe_catch_panic /rustc/0b36e9dea3f2ff25b1d0df2669836c33cce89ae5/src/libpanic_abort/lib.rs:28:4

NOTE: libFuzzer has rudimentary signal handlers.
      Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal

crash.zip

[YaLTeR]

Fixed by #1630