Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add scripts and tests to reproduce fuzzer errors in pytest #587

Merged
merged 5 commits into from
Jan 9, 2024

Conversation

niccokunzmann
Copy link
Member

  • move fuzzing tests in to test folder
  • create a script that runs the fuzzer and extracts the test case if the test fails

- move fuzzing tests in to test folder
- create a script that runs the fuzzer and extracts the test case if the test fails
@coveralls
Copy link

Pull Request Test Coverage Report for Build 6945563133

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 94.011%

Totals Coverage Status
Change from base Build 6919080280: 0.0%
Covered Lines: 1240
Relevant Lines: 1319

💛 - Coveralls

@niccokunzmann
Copy link
Member Author

niccokunzmann commented Nov 21, 2023

My intention is to create small pull requests like this one: niccokunzmann#2
Most OSS FUZZ errors depend on ICS files. These errors get reproduced though these pull requests. Then, we can fix, review and reproduce them locally with pytest. Probably, there is more to do about this. This is my current state of work.

https://github.com/niccokunzmann/icalendar/actions/runs/6945596913/job/18895335145?pr=2 shows how the errors are reproduced.

See also #578

@coveralls
Copy link

coveralls commented Nov 21, 2023

Pull Request Test Coverage Report for Build 6945838229

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 94.011%

Totals Coverage Status
Change from base Build 6919080280: 0.0%
Covered Lines: 1240
Relevant Lines: 1319

💛 - Coveralls

@niccokunzmann
Copy link
Member Author

My idea is that I create the infrastructure to reproduce the tests easily. From there on, each time I get an email about a new error, I will create a PR with the error so it can be reproduced.

@capuanob
Copy link
Contributor

@niccokunzmann This is very cool! I just want to make sure of a few things

  1. Did you test this locally with a checked out OSSFuzz (instructions here) to make sure that it doesn't break the OSSFuzz integration?
  2. Also, I wonder if there is any performance impact to the fuzz-testing with the addition of stdout prints? I'd imagine its minimal, but worth confirming the reported exec/s before and after the changes when running locally

@niccokunzmann
Copy link
Member Author

niccokunzmann commented Jan 2, 2024

id you test this locally with a checked out OSSFuzz (instructions here) to make sure that it doesn't break the OSSFuzz integration?

Yes, I did that. This allowed me to extract the ICS file used.

Also, I wonder if there is any performance impact to the fuzz-testing with the addition of stdout prints?

That is minimal (a few milliseconds if I look at it). The corpus has a few files in it. Printing these does not make is slower. Running the fuzzer and preparing the fuzzing tests take most of the time.

Thanks for your reply!

@ennamarie19
Copy link
Collaborator

LGTM!

@ennamarie19 ennamarie19 closed this Jan 9, 2024
@ennamarie19 ennamarie19 merged commit bf8ad7a into collective:master Jan 9, 2024
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants