Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

skip verbose flag in secondary parser #576

Merged
merged 1 commit into from
Mar 26, 2024

Conversation

jmartin-tech
Copy link
Collaborator

When parsing the command line options on the CLI the verbose flag count is already treated uniquely. This change removes verbose from the aux_parser to avoid inconsistent action expectations.

Prior to this change when walking the options verbose would be registered or aux_parser with the default action store causing the Argparser to expect an value to be provided during parsing.

Using the exampled gpt-2 command can validate this change:
Existing:

% python3 -m garak -v --model_type huggingface --model_name gpt2 --probes dan.Dan_11_0
garak LLM security probe v0.9.0.12.post1 ( https://github.com/leondz/garak ) at 2024-03-25T14:29:12.154243
usage: __main__.py [-h] [--verbose VERBOSE] [--report_prefix REPORT_PREFIX] [--narrow_output] [--parallel_requests] [--parallel_attempts] [--seed SEED] [--deprefix]
                   [--eval_threshold EVAL_THRESHOLD] [--generations GENERATIONS] [--config CONFIG] [--model_type MODEL_TYPE] [--model_name MODEL_NAME]
                   [--generator_option_file GENERATOR_OPTION_FILE] [--generator_options GENERATOR_OPTIONS] [--probes PROBES] [--probe_option_file PROBE_OPTION_FILE]
                   [--probe_options PROBE_OPTIONS] [--probe_tags PROBE_TAGS] [--detectors DETECTORS] [--extended_detectors] [--buffs BUFFS] [--taxonomy TAXONOMY]
                   [--plugin_info PLUGIN_INFO] [--list_probes] [--list_detectors] [--list_generators] [--list_buffs] [--list_config] [--version] [--report REPORT]
                   [--interactive] [--generate_autodan] [--interactive.py]
__main__.py: error: argument --verbose/-v: expected one argument

Patched:

% python3 -m garak -v --model_type huggingface --model_name gpt2 --probes dan.Dan_11_0
garak LLM security probe v0.9.0.12.post1 ( https://github.com/leondz/garak ) at 2024-03-25T14:28:43.975393
📜 reporting to garak_runs/garak.074fe0ab-e113-42de-ab29-180e2547a0a4.report.jsonl
🦜 loading generator: Hugging Face 🤗 pipeline: gpt2
🕵️  queue of probes: dan.Dan_11_0
loading probe: garak.probes.dan.Dan_11_0
loading detector: garak.detectors.dan.DAN
loading detector: garak.detectors.mitigation.MitigationBypass
probes.dan.Dan_11_0:   0%|                                                                                                                           | 0/1 [00:00<?, ?it/s]This is a friendly reminder - the current text generation call will exceed the model's predefined maximum length (1024). Depending on the model, you may observe exceptions, performance degradation, or nothing at all.
dan.Dan_11_0                                                                                 dan.DAN: SKIP  ok on    0/   0
dan.Dan_11_0                                                             mitigation.MitigationBypass: SKIP  ok on    0/   0
📜 report closed :) garak_runs/garak.074fe0ab-e113-42de-ab29-180e2547a0a4.report.jsonl
📜 report html summary being written to garak_runs/garak.074fe0ab-e113-42de-ab29-180e2547a0a4.report.html
loading probe: garak.probes.dan.Dan_11_0
loading detector: garak.detectors.dan.DAN
loading detector: garak.detectors.mitigation.MitigationBypass
✔️  garak run complete in 11.93s

When parsing the command line options on the CLI the `verbose` flag
count is already treated uniquely. This change removes `verbose` from
the `aux_parser` to avoid inconsistent `action` expectations.

Signed-off-by: Jeffrey Martin <jemartin@nvidia.com>
@leondz
Copy link
Owner

leondz commented Mar 26, 2024

thanks!

things that rely on nb verbosity levels are now deprecated (were there ever any, is another question)

@leondz leondz merged commit 8f5bf3d into leondz:main Mar 26, 2024
1 check passed
@github-actions github-actions bot locked and limited conversation to collaborators Mar 26, 2024
@jmartin-tech jmartin-tech deleted the fix/enable-verbose branch March 26, 2024 19:55
@jmartin-tech
Copy link
Collaborator Author

jmartin-tech commented Mar 26, 2024

things that rely on nb verbosity levels are now deprecated

Thanks, I may open an issue around removing the count here or at least converting to just > 0 for all locations that still check for a level:

garak/detectors/base.py:        if hasattr(_config.system, "verbose") and _config.system.verbose > 0:
garak/detectors/packagehallucination.py:                        and _config.system.verbose >= 2
garak/evaluators/base.py:        if _config.system.verbose > 0 and messages:
garak/evaluators/base.py:        if _config.system.verbose > 0 and messages:
garak/generators/ggml.py:        if _config.system.verbose > 1:
garak/harnesses/base.py:            if hasattr(_config.system, "verbose") and _config.system.verbose >= 2:
garak/harnesses/base.py:            if hasattr(_config.system, "verbose") and _config.system.verbose >= 2:
garak/harnesses/probewise.py:            if hasattr(_config.system, "verbose") and _config.system.verbose >= 2:
garak/probes/atkgen.py:        output_is_conversation = _config.system.verbose >= 2
garak/probes/base.py:        if hasattr(_config.system, "verbose") and _config.system.verbose > 0:

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants