Fixing Exception Cause By Type Error When Scanning LLMs Via Replicate

[DavidLee528]

Hi professor,

Recently I launched a garak scan using debug mode of vscode with the following configuration:

    {
        "name": "replicate+meta/llama-2-70b-chat",
        "type": "python",
        "request": "launch",
        "program": "${workspaceFolder}/garak/__main__.py",
        "console": "integratedTerminal",
        "args": [
            "--model_type",
            "replicate",
            "--model_name",
            "meta/llama-2-70b-chat",
            "--probes",
            "malwaregen"
        ],
        "justMyCode": false,
        "env": {
            "REPLICATE_API_TOKEN": "****************************************************************" 
        }
    },

But I got exception raised by ....../python3.11/site-packages/backoff/_sync.py:106:

input.seed: Invalid type. Expected: integer, given: null

Finally I found the reason of the exception:

The value assignment operation of self.seed = 9 in garak/generators/replicate.py:38 has been proved invalid according to the collision described below:

1. garak/resources/garak.core.yaml:9: the attribute seed has a default value "empty", which is regarded as None in Python syntax;
2. The above None value could assign to self.seed of class ReplicateGenerator by self.seed = _config.run.seed in garak/generators/replicate.py:40 and generator.seed = _config.run.seed in
   garak/cli.py:445;
3. garak/generators/replicate.py:63 needs self.seed to be int type but it is None type.

To address this problem, I made some modifications to avoid value assignment operation when self.seed appears to None type. I believe those modifications could enhance the robustness of the framework.

Please review the code comparison for detail. Any problems, feel free to point it out. 😊

[leondz]

Thanks. This is a fine workaround - I'm sure we'll see this in more than one place. The ultimate fix is to use a better data structure for _config, but this fix looks good to me.