Consider integrating with OSS-Fuzz?

[Valloric]

(Similar to my issue for regex: rust-lang/regex#650)

OSS-Fuzz is a Google-provided service that can automatically and continuously fuzz OSS code and report bugs for discovered issues.

The requirement for being accepted in OSS-Fuzz is "project must have a significant user base and/or be critical to the global IT infrastructure" which I personally believe includes the rustc since it's at the core of so much stuff. Also, OSS-Fuzz supports fuzzing Rust libraries, so IMO that signals the project views Rust as important enough. List of projects that are currently in OSS-Fuzz. Note that LLVM and Go are on the list, so compilers have been accepted before.

Can't hurt to file a form and ask.

OSS-Fuzz offers monetary rewards for integration, up to $20k. It can be donated to a charity and Google will double the reward. (Has the much-talked-about Rust Foundation been created? If so, it would be a great fit.)

$40k to a charity for a few days of work doesn't sound like a bad deal...

(Disclaimer: I work for Google, but not on anything fuzzing or security-related. I do not speak for my employer. This suggestion comes from "personal Val", not "business Val". 😃)

cc @Shnatsel @rust-fuzz

[Mark-Simulacrum]

I can't find an issue for it, but I know we've discussed this on the t-compiler Zulip at least.

My personal opinion is that writing good fuzzing tests for rustc is likely to be quite hard, and quite limited in scope. I suspect that the "ideal" would be to target unsoundness bugs in the compiler (borrow check, trait system, etc.) by way of generating rust code and then checking that the compiler errors/passes equivalently to some (very) simple reference implementation. We don't currently have a reference implementation though, so that's hard :)

One area where a fuzzer could be useful today is testing incremental, as I know that we've struggled to come up with reproductions for bugs there in the past (as they often depend on a relatively complex setup to get the state "just right" AIUI). I don't know how difficult it would be to get fuzzing to simply generate "random" Rust programs with slight mutations and try to test incremental that way -- i.e., we'd have a run with and without incremental and expect at least the overall status code of compilation to match. cc @michaelwoerister @Zoxc on this, as you might have thoughts on how viable/useful something like this would be.

With regards to the more general bugs, e.g., ICEs, I don't personally think that we have the developer capacity for a fuzzer to be useful at this point. We have hundreds of ICE issues open, and for the most part they're just not super important at this point in rustc's lifecycle. It's relatively rare in practice that code will generate an ICE.

[Shnatsel]

There is already a way to get slightly more sophisticated programs than just random input, see rust-fuzz/trophy-case#36

But so far the fuzzing harness has only found ICEs, and I tend to agree with Mark that they're not terribly important at this point when triggered by contrived fuzzer-generated examples.

[bdfshongjie]

test On Tue, Mar 3, 2020 at 11:42 <notifications@github.com> wrote: There is already a way to get slightly more sophisticated programs than just random input, see rust-fuzz/trophy-case#36 <rust-fuzz/trophy-case#36> But so far the fuzzing harness has only found ICEs, and I tend to agree with Mark that they're not terribly important at this point when triggered by contrived fuzzer-generated examples. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#69655?email_source=notifications&email_token=ANJSMYJIRF32DQXNRBEUTPTRFVMR7A5CNFSM4LAAUN7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENU3VAY#issuecomment-594131587>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANJSMYO2WYOCB3QITHBCFPDRFVMR7ANCNFSM4LAAUN7A> . <script type="application/ld+json">[ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": " #69655?email_source=notifications\u0026email_token=ANJSMYJIRF32DQXNRBEUTPTRFVMR7A5CNFSM4LAAUN7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENU3VAY#issuecomment-594131587", "url": " #69655?email_source=notifications\u0026email_token=ANJSMYJIRF32DQXNRBEUTPTRFVMR7A5CNFSM4LAAUN7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENU3VAY#issuecomment-594131587", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": " https://github.com" } } ]</script>