Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fuzz] Configure the differential target #4773

Merged
merged 5 commits into from
Aug 24, 2022

Conversation

abrown
Copy link
Contributor

@abrown abrown commented Aug 24, 2022

This change is a follow-on from #4515 to add the ability to configure
the differential fuzz target by limiting which engines and modules are
used for fuzzing. This is incredibly useful when troubleshooting, e.g.,
when an engine is more prone to failure, we can target that engine
exclusively. The effect of this configuration is visible in the
statistics now printed out from #4739.

Engines are configured using the ALLOWED_ENGINES environment variable.
We can either subtract from the set of allowed engines (e.g.,
ALLOWED_ENGINES=-v8) or build up a set of allowed engines (e.g.,
ALLOWED_ENGINES=wasmi,spec), but not both at the same time.
ALLOWED_ENGINES only configures the left-hand side engine; the
right-hand side is always Wasmtime. When omitted, ALLOWED_ENGINES
defaults to [wasmtime, wasmi, spec, v8].

The generated WebAssembly modules are configured using
ALLOWED_MODULES. This environment variables works the same as above
but the available options are: [wasm-smith, single-inst].

This change is a follow-on from bytecodealliance#4515 to add the ability to configure
the `differential` fuzz target by limiting which engines and modules are
used for fuzzing. This is incredibly useful when troubleshooting, e.g.,
when an engine is more prone to failure, we can target that engine
exclusively. The effect of this configuration is visible in the
statistics now printed out from bytecodealliance#4739.

Engines are configured using the `ALLOWED_ENGINES` environment variable.
We can either subtract from the set of allowed engines (e.g.,
`ALLOWED_ENGINES=-v8`) or build up a set of allowed engines (e.g.,
`ALLOWED_ENGINES=wasmi,spec`), but not both at the same time.
`ALLOWED_ENGINES` only configures the left-hand side engine; the
right-hand side is always Wasmtime. When omitted, `ALLOWED_ENGINES`
defaults to [`wasmtime`, `wasmi`, `spec`, `v8`].

The generated WebAssembly modules are configured using
`ALLOWED_MODULES`. This environment variables works the same as above
but the available options are: [`wasm-smith`, `single-inst`].
@abrown abrown requested a review from jameysharp August 24, 2022 17:55
@abrown
Copy link
Contributor Author

abrown commented Aug 24, 2022

Here is an example from running the target locally:

$ ALLOWED_MODULES=-wasm-smith cargo +nightly fuzz run differentia
...
=== Execution rate (384 successes / 2000 attempted modules): 19.20% ===
        wasmi: 0.96%, spec: 8.85%, wasmtime: 83.97%, v8: 6.22%
        wasm-smith: 0.00%, single-inst: 100.00%

@abrown
Copy link
Contributor Author

abrown commented Aug 24, 2022

@jameysharp, you've had some opinions on this in the past so I tagged you for review (?): I really would have preferred to use &[&str] everywhere (in the static variables, function parameters, etc.) but once env gave me back Strings this all started sliding toward Vec<String>. This makes for some awkward match and if statements. Do you see an easy way to convert to slices and &str or even just one of those?

Copy link
Contributor

@jameysharp jameysharp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's true: I have, at times, had opinions! 😆

I don't feel very strongly about any of these review comments. This PR is fine as-is, so I'm marking this approved. But if you want to take a little more time on it I think these suggestions are improvements.

crates/fuzzing/src/oracles/engine.rs Show resolved Hide resolved
crates/fuzzing/src/oracles/engine.rs Outdated Show resolved Hide resolved
crates/fuzzing/src/oracles/engine.rs Show resolved Hide resolved
fuzz/fuzz_targets/differential.rs Outdated Show resolved Hide resolved
fuzz/fuzz_targets/differential.rs Outdated Show resolved Hide resolved
crates/fuzzing/src/oracles/engine.rs Show resolved Hide resolved
@github-actions github-actions bot added the fuzzing Issues related to our fuzzing infrastructure label Aug 24, 2022
@github-actions
Copy link

Subscribe to Label Action

cc @fitzgen

This issue or pull request has been labeled: "fuzzing"

Thus the following users have been cc'd because of the following labels:

  • fitzgen: fuzzing

To subscribe or unsubscribe from this label, edit the .github/subscribe-to-label.json configuration file.

Learn more.

Copy link
Contributor

@jameysharp jameysharp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you've addressed all my suggestions. Thanks! I hope you found these suggestions helpful.

@abrown
Copy link
Contributor Author

abrown commented Aug 24, 2022

I think you've addressed all my suggestions. Thanks! I hope you found these suggestions helpful.

Yeah, they were great!

@abrown abrown merged commit d3c463a into bytecodealliance:main Aug 24, 2022
@abrown abrown deleted the fuzz-lists branch August 24, 2022 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fuzzing Issues related to our fuzzing infrastructure
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants