Skip to content

Commit

Permalink
[fuzzer] Relax the FileCheck string for test fuzzer-custommutator.test (
Browse files Browse the repository at this point in the history
#66343)

The FileCheck string `LLVMFuzzerCustomMutatorLongSequence: {{.*}} MS:
{{[0-9]*}} {{(([a-zA-Z]*-){11,})}} {{.*}}` is too restrictive and may
fail the test in some case.

If we look at the commit that added this
check(66df989),
This check is for printing out the long mutation sequence, such as this
one
```
#53552  REDUCE cov: 6 ft: 6 corp: 5/9b lim: 4096 exec/s: 0 rss: 37Mb L: 2/3 MS: 54 ChangeByte-PersAutoDict-ChangeBit-ChangeBinInt-ChangeBit-ChangeBit-ChangeByte-CMP-EraseBytes-EraseBytes-CrossOver-InsertRepeatedBytes-ChangeByte-EraseBytes-InsertRepeatedBytes-ShuffleBytes-ChangeByte-ShuffleBytes-ChangeBit-CrossOver-ChangeBit-ShuffleBytes-ChangeBinInt-ShuffleBytes-EraseBytes-InsertByte-Custom-ShuffleBytes-CopyPart-InsertRepeatedBytes-PersAutoDict-InsertRepeatedBytes-ChangeByte-CrossOver-CrossOver-PersAutoDict-PersAutoDict-EraseBytes-ChangeBit-CopyPart-ChangeByte-CopyPart-InsertRepeatedBytes-CrossOver-CrossOver-CrossOver-CrossOver-ShuffleBytes-EraseBytes-InsertByte-InsertRepeatedBytes-CrossOver-EraseBytes-Custom- DE: "\377\377"-"\001\000"-"\001\000"-"\000\000\000\000\000\000\000\000"-"\001\000\000\000"-
```

But if we look at the code doing the printing
```cpp
void MutationDispatcher::PrintMutationSequence(bool Verbose) {
  Printf("MS: %zd ", CurrentMutatorSequence.size());
  size_t EntriesToPrint =
      Verbose ? CurrentMutatorSequence.size()
              : std::min(kMaxMutationsToPrint, CurrentMutatorSequence.size());
  for (size_t i = 0; i < EntriesToPrint; i++)
    Printf("%s-", CurrentMutatorSequence[i].Name);
  if (!CurrentDictionaryEntrySequence.empty()) {
    Printf(" DE: ");
    EntriesToPrint = Verbose ? CurrentDictionaryEntrySequence.size()
                             : std::min(kMaxMutationsToPrint,
                                        CurrentDictionaryEntrySequence.size());
    for (size_t i = 0; i < EntriesToPrint; i++) {
      Printf("\"");
      PrintASCII(CurrentDictionaryEntrySequence[i]->GetW(), "\"-");
    }
  }
}
```

We can see that the `DE: XXX` is not always printed. So the following
output is possible(and is from real-life failure), notince the missing
of `DE: XXX`.
```
#13613  NEW    cov: 5 ft: 5 corp: 4/6b lim: 4096 exec/s: 0 rss: 32Mb L: 2/2 MS: 27 InsertByte-ChangeBinInt-ChangeBinInt-CrossOver-ShuffleBytes-ChangeBit-EraseBytes-ShuffleBytes-InsertByte-InsertRepeatedBytes-CopyPart-InsertByte-ChangeByte-ChangeBit-InsertByte-CrossOver-EraseBytes-CopyPart-ShuffleBytes-EraseBytes-InsertByte-InsertRepeatedBytes-CrossOver-CrossOver-ShuffleBytes-ChangeBit-Custom-
#13765  ......
```
This output is totally legit and will fail that check.

So I remove the check for the following strings, I think `MS: {{[0-9]*}}
{{(([a-zA-Z]*-){11,})}}` is sufficient for checking the long mutation
sequence. This should help resolve the flaky failure of
fuzzer-custommutator.test.
  • Loading branch information
yingcong-wu committed Sep 20, 2023
1 parent c414e58 commit db4ba21
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion compiler-rt/test/fuzzer/fuzzer-custommutator.test
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ LLVMFuzzerCustomMutatorWithLenControl: BINGO
RUN: %cpp_compiler %S/CustomMutatorWithLongSequencesTest.cpp -o %t-CustomMutatorWithLongSequencesTest
RUN: not %run %t-CustomMutatorWithLongSequencesTest -verbosity=2 2>&1 | FileCheck %s --check-prefix=LLVMFuzzerCustomMutatorLongSequence
LLVMFuzzerCustomMutatorLongSequence: Flag: verbosity 2
LLVMFuzzerCustomMutatorLongSequence: {{.*}} MS: {{[0-9]*}} {{(([a-zA-Z]*-){11,})}} {{.*}}
LLVMFuzzerCustomMutatorLongSequence: {{.*}} MS: {{[0-9]*}} {{(([a-zA-Z]*-){11,})}}
LLVMFuzzerCustomMutatorLongSequence: BINGO

# check a target that prints long mutation sequences and verifies the printed
Expand Down

0 comments on commit db4ba21

Please sign in to comment.