-
Notifications
You must be signed in to change notification settings - Fork 11.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[fuzzer] Relax the FileCheck string for test fuzzer-custommutator.test #66343
[fuzzer] Relax the FileCheck string for test fuzzer-custommutator.test #66343
Conversation
@llvm/pr-subscribers-compiler-rt-sanitizer ChangesThe FileCheck string `LLVMFuzzerCustomMutatorLongSequence: {{.*}} MS: {{[0-9]*}} {{(([a-zA-Z]*-){11,})}} {{.*}}` is too restrictive and may fail the test in some case.If we look at the commit that added this check(66df989), This check is for printing out the long mutation sequence, such as this one
But if we look at the code doing the printing void MutationDispatcher::PrintMutationSequence(bool Verbose) {
Printf("MS: %zd ", CurrentMutatorSequence.size());
size_t EntriesToPrint =
Verbose ? CurrentMutatorSequence.size()
: std::min(kMaxMutationsToPrint, CurrentMutatorSequence.size());
for (size_t i = 0; i < EntriesToPrint; i++)
Printf("%s-", CurrentMutatorSequence[i].Name);
if (!CurrentDictionaryEntrySequence.empty()) {
Printf(" DE: ");
EntriesToPrint = Verbose ? CurrentDictionaryEntrySequence.size()
: std::min(kMaxMutationsToPrint,
CurrentDictionaryEntrySequence.size());
for (size_t i = 0; i < EntriesToPrint; i++) {
Printf("\"");
PrintASCII(CurrentDictionaryEntrySequence[i]->GetW(), "\"-");
}
}
} We can see that the
This output is totally legit and will fail that check. So I remove the check for the following strings, I think -- 1 Files Affected:
diff --git a/compiler-rt/test/fuzzer/fuzzer-custommutator.test b/compiler-rt/test/fuzzer/fuzzer-custommutator.test index 7d94ae064bf9698..314530415746c3d 100644 --- a/compiler-rt/test/fuzzer/fuzzer-custommutator.test +++ b/compiler-rt/test/fuzzer/fuzzer-custommutator.test @@ -16,7 +16,7 @@ LLVMFuzzerCustomMutatorWithLenControl: BINGO RUN: %cpp_compiler %S/CustomMutatorWithLongSequencesTest.cpp -o %t-CustomMutatorWithLongSequencesTest RUN: not %run %t-CustomMutatorWithLongSequencesTest -verbosity=2 2>&1 | FileCheck %s --check-prefix=LLVMFuzzerCustomMutatorLongSequence LLVMFuzzerCustomMutatorLongSequence: Flag: verbosity 2 -LLVMFuzzerCustomMutatorLongSequence: {{.*}} MS: {{[0-9]*}} {{(([a-zA-Z]*-){11,})}} {{.*}} +LLVMFuzzerCustomMutatorLongSequence: {{.*}} MS: {{[0-9]*}} {{(([a-zA-Z]*-){11,})}} LLVMFuzzerCustomMutatorLongSequence: BINGO # check a target that prints long mutation sequences and verifies the printed |
Ping @vitalybuka @MaskRay . Could you please review this PR, I cannot add reviewers like Phabricator. |
The FileCheck string
LLVMFuzzerCustomMutatorLongSequence: {{.*}} MS: {{[0-9]*}} {{(([a-zA-Z]*-){11,})}} {{.*}}
is too restrictive and may fail the test in some case.If we look at the commit that added this check(66df989), This check is for printing out the long mutation sequence, such as this one
But if we look at the code doing the printing
We can see that the
DE: XXX
is not always printed. So the following output is possible(and is from real-life failure), notince the missing ofDE: XXX
.This output is totally legit and will fail that check.
So I remove the check for the following strings, I think
MS: {{[0-9]*}} {{(([a-zA-Z]*-){11,})}}
is sufficient for checking the long mutation sequence. This should help resolve the flaky failure of fuzzer-custommutator.test.