Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(all): Fuzzing #709

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

feat(all): Fuzzing #709

wants to merge 2 commits into from

Conversation

marioevz
Copy link
Member

πŸ—’οΈ Description

First iteration at implementation of idea in #591

Implements the fuzz command that allows testers to mark a test as fuzzable, and then listing parameters that can be randomized:

@pytest.mark.parametrize(
    "key,value",
    [(1, 1)],
)
@pytest.mark.fuzzable("key", "value")
def test_sstore(
    state_test: StateTestFiller,
    fork: Fork,
    pre: Alloc,
    key: Annotated[int, lambda: random.randint(0, 2**64)],
    value: int,
):

In this simple example, a test function is marked as fuzzable by adding the @pytest.mark.fuzzable decorator and as its parameters it lists two parameter names: "key" and "value".

When the fuzz command is executed and pointed to this file, the test is prepared by first removing its existing pytest.mark.parametrize that already contains "key" and "value" parameters, and then the test is re-parametrized to fuzzed values for both parameters.

The fuzzer reads the annotation of the test function signature and then proceeds to attempt to generate a fuzzing function for each one.

In the case of "value", the annotation is simply int so the default fuzzer for integers is used (random.randint(0, 2**256)).

In the case of "key", the type is annotated and the second parameter of the annotation is a callable lambda function which specifies how the value should be randomized, in this case, a random integer from 0 to 2**64.

We can implement many fuzzers for each type, including e.g., random opcodes, random code, random valid or invalid EOF code.

πŸ”— Related Issues

βœ… Checklist

  • All: Set appropriate labels for the changes.
  • All: Considered squashing commits to improve commit history.
  • All: Added an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • Tests: All converted JSON/YML tests from ethereum/tests have been added to converted-ethereum-tests.txt.
  • Tests: A PR with removal of converted JSON/YML tests from ethereum/tests have been opened.
  • Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant