Spelling corrections for multiple locations #564
Conversation
jmartin-tech
commented
Mar 21, 2024
jmartin-tech
commented
- Correct encoding in xss probe
- Consistent spelling in glitch probe
- Adjust README.md for consistency
- Adjust misp_descriptions for consistency
* Correct encoding in xss probe * Consistent spelling in glitch probe * Adjust README.md for consistency * Adjust misp_descriptions for consistency
There was a problem hiding this comment.
Good catches overall -- I think there are a few things here that are a matter of UK vs. US spelling conventions e.g. behaviour that are fine to spell either way. Will defer to @leondz to accept the American way of life.
| @@ -98,7 +98,7 @@ python3 -m garak --model_type huggingface --model_name gpt2 --probes dan.Dan_11_ | |||
|
|
|||
| ## Reading the results | |||
|
|
|||
| For each probe loaded, garak will print a progress bar as it generates. Once generation is complete, a row evaluating that probe's results on each detector is given. If any of the prompt attempts yielded an undesirable behaviour, the response will be marked as FAIL, and the failure rate given. | |||
There was a problem hiding this comment.
British spelling is to be expected from @leondz :P
|
|
||
| ## Logging | ||
|
|
||
| `garak` generates multiple kinds of log: | ||
| * A log file, `garak.log`. This includes debugging information from `garak` and its plugins, and is continued across runs. | ||
| * A report of the current run, structured as JSONL. A new report file is created every time `garak` runs. The name of this file is output at the beginning and, if successful, also the end of the run. In the report, an entry is made for each probing attempt both as the generations are received, and again when they are evaluated; the entry's `status` attribute takes a constant from `garak.attempts` to describe what stage it was made at. | ||
| * A hitlog, detailing attempts that yielded a vulnerability (a 'hit') |
There was a problem hiding this comment.
We safe the files as {prefix}.hitlog.jsonl, so I'm inclined to keep this spelling as a term of art, but you are technically correct -- the best kind of correct.
There was a problem hiding this comment.
Consider then expanding this similar to the log file entry above as:
* A hit log, `{run_id}.hitlog.jsonl`, detailing attempts that yielded a vulnerability (a 'hit')
| @@ -1,7 +1,7 @@ | |||
| #!/usr/bin/env python3 | |||
| """Glitch keyword probes | |||
|
|
|||
| Testing for tokens that lead to unusual model behaviour | |||
There was a problem hiding this comment.
UK/US spelling again.
There was a problem hiding this comment.
Just for consistency as both spellings are in this file.
| quality:Behavioral:ContentSafety:HarmfulViolent Harmful or violent Harmful or violent | ||
| quality:Behavioral:ContentSafety:Sexualized Sexualised Sexualised |
There was a problem hiding this comment.
Again, UK/US spelling
There was a problem hiding this comment.
Consistency as alternate spelling is used in in the same entry.
There was a problem hiding this comment.
I only know English English and haven't put learning variants on my roadmap - go with whatever works for you :)
|
Note I made allowances for US vs UK spelling and and only made changes where consistency would be helpful. |
|
@erickgalinkin out of fear of being hanged for treason I'll defer to you to review/merge this |
