idea: treat a major coverage drop an issue!

[bagder]

The curl code base has been fuzzed by oss-fuzz for many years. These days we rarely get any reports. It chugs away in the background and while it certainly gives us confidence and helps us "stay honest", it also falls off the top of our minds. We don't think about it too much in our daily work. I think that's good. It still tells us when it finds something. Which these days are many months between each time.

With this in mind, when we changed a config detail a year ago, the fuzz coverage for curl dropped considerably but no one noticed because no one checked. We are used to oss-fuzz telling us when there's an issue. Thus, oss-fuzz has the last year been fuzzing a much smaller part of the code base, which then of course could have made us miss problems.

The other day we fixed this issue and the coverage is again back up to levels similar to those a year ago, and nothing new has been reported (yet) so I figure in a way we were lucky and dodged one this time. But it brings up this topic on my agenda: how do I detect this problem the next time?

And perhaps even more broadly: how do we all (all oss-fuzz users) detect this because I presume that even though it happened to us this time, it might happen to others every once in a while. And maybe it has already happened to 22 users who have not even noticed yet and now the fuzzer is missing out?

How do others do? Is there a best practice I have missed? Can oss-fuzz perhaps itself detect huge drops and file an issue about that?

[evverx]

How do others do? Is there a best practice I have missed?

I opened #7190 so that CIFuzz could catch coverage drops as soon as possible but it kind of stalled. The only way to keep track of drops like that I'm aware of is the Fuzz-Introspector dashboard but it works when there are less than ten fuzz targets. With more than 40 fuzz targets it gets blurry.

[DavidKorczynski]

but it works when there are less than ten fuzz targets.

Could you clarify this? It works when there are 10 or more as well?

FWIF it's visible that there was a sudden jump in Curl's coverage: https://introspector.oss-fuzz.com/project-profile?project=curl so I think the drop would have been easy to spot but it looks like it happened at a point before the app was started (April 20 2023)

[evverx]

Could you clarify this? It works when there are 10 or more as well?

The global dashboard itself works with a lot of fuzz targets. The problem is that sometimes coverage drops affect certain fuzz targets and it's hard to spot them because they are hidden behind the global coverage where several fuzz targets can cover the same code a bit differently. Basically I have to go over specific coverage reports one by one.