FEAT Add many-shot jailbreaking feature implementation

[KutalVolkan]

Hi @romanlutz

I have completed the code implementation for the many-shot jailbreaking feature as discussed earlier.

Code Integration

• Code Placement: Added code in the relevant positions across the project to support many-shot jailbreaking.
• Demo: A demo script has been built to showcase the feature's functionality. The demo script is available here.

Dataset Integration

• Dataset Location: The required dataset for the many-shot jailbreaking feature has been added and is accessible here.
• Dynamic Import and Processing: The code dynamically imports and processes the dataset to generate the necessary Q&A pairs.

Important Links

• Class Implementation: Many-Shot-Jailbreak Class
• Feature Import via API Call and User JSON: API and JSON Usage
• YAML File for Prompt Template: YAML Template

Testing Phase

The feature is currently undergoing testing. During the testing phase, one major challenge encountered is the rate limit imposed by OpenAI when a large number of examples (e.g., 100) are used.

Known Issue

• Rate Limit Error:
  If the num_examples parameter is set too high, an error may occur due to OpenAI's rate limits.

openai.RateLimitError: Error code: 429 - {'error': {'message': 'Request too large for gpt-3.5-turbo in organization YOUR-ID on tokens per min (TPM): Limit 60000, Requested 74536. The input or output tokens must be reduced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}

Next Steps:

1. Dataset Expansion: The current dataset contains 100 examples. I will expand this dataset to include more examples, aiming to reach our suggested 256+ Q&A pairs.
2. Review and Optimize: I will conduct some reviews and optimizations, including thorough testing and verification of the entire approach, to ensure there are no logical mistakes regarding the dataset, user interactions, assistant responses, and overall methodology.

Could you please provide feedback on these steps? Any suggestions or improvements are welcome.

[KutalVolkan]

@microsoft-github-policy-service agree

[romanlutz]

Thank you for this great PR! I've added some comments. If you have questions/concerns/comments, please use the comment threads.

[KutalVolkan]

Hi @romanlutz,

Thank you for your great feedback!

I have addressed all the feedback and made the necessary changes. Here are some question marks in my head:

1. Template Path Update:
   I have updated the template path to:

   template_path = Path(DATASETS_PATH) / "prompt_templates" / "jailbreak" / "many_shot_template.yml"

   instead of:

   from pyrit.common.path import DATASETS_PATH 
   from pathlib import Path 
   
   Path(DATASETS_PATH) / "orchestrator" / "many_shot_jailbreaking" / "template.yaml"

   The previous code snippet was causing an error for me, and the new path works correctly.

2. Printing original_value and converted_value:
   Both original_value and converted_value are from the assistant (target_llm) and provide the same output in my case. Initially, I printed them because I saw them as attributes of PromptRequestPiece. Could you please clarify if we need to print both values? If they always contain the same information, should we only print one of them?

   Here is the current implementation for printing the conversation:

   for response in responses:
       for piece in response.request_pieces:
           print(f"{Style.BRIGHT}{Fore.RED}{piece.role}: {piece.original_value}\n")
           print(f"{Style.BRIGHT}{Fore.GREEN}{piece.role}: {piece.converted_value}\n")

Everything else is clear and all tests are passing. Could you please take a look and let me know if any further changes are needed or if I missed something?

Thank you!

[romanlutz]

Hi @romanlutz,

Thank you for your great feedback!

I have addressed all the feedback and made the necessary changes. Here are some question marks in my head:

1. Template Path Update:

   I have updated the template path to:

   template_path = Path(DATASETS_PATH) / "prompt_templates" / "jailbreak" / "many_shot_template.yml"

   instead of:

   from pyrit.common.path import DATASETS_PATH 
   
   from pathlib import Path 
   
   
   
   Path(DATASETS_PATH) / "orchestrator" / "many_shot_jailbreaking" / "template.yaml"

   The previous code snippet was causing an error for me, and the new path works correctly.

It's a different path so that may explain why. Of course, we're not implementing a new orchestrator, so perhaps this one is preferable anyway.

2. Printing original_value and converted_value:

Both original_value and converted_value are from the assistant (target_llm) and provide the same output in my case. Initially, I printed them because I saw them as attributes of PromptRequestPiece. Could you please clarify if we need to print both values? If they always contain the same information, should we only print one of them?

Here is the current implementation for printing the conversation:

for response in responses:

    for piece in response.request_pieces:

        print(f"{Style.BRIGHT}{Fore.RED}{piece.role}: {piece.original_value}\n")

        print(f"{Style.BRIGHT}{Fore.GREEN}{piece.role}: {piece.converted_value}\n")

The original and converted value only differ if converters are used. You can see options in the prompt_converter module. You should probably print the converted one since that will be right in most cases.

Everything else is clear and all tests are passing. Could you please take a look and let me know if any further changes are needed or if I missed something?

Thank you!

Taking a more thorough look today!

[KutalVolkan]

Hi @romanlutz

Enhanced Q&A Dataset Completion

The Q&A dataset has been successfully expanded as planned. This update provides an overview of the enhancements made, the methods used, and the results achieved.

Details

Dataset Expansion

• Initial Dataset: 400 Q&A pairs from the HarmBench dataset.
• Enhanced Dataset: Expanded to 256+ Q&A pairs.
• Total Key-Value Pairs: 400
  • Not Harmful: 121 pairs
  • Harmful: 279 pairs

Methods Used

1. Initial Fill and Renaming:

   • Filled missing values in the ContextString column.
   • Renamed columns to better fit our use case.

2. Model Utilized:

   • After finding GPT-3.5-Turbo results unsatisfactory, i followed the recommendations from the Many-Shot Jailbreaking paper.
   • Used the Hartford, E. Wizardlm-13b-uncensored model (WizardLM13B-Uncensored) to generate the additional responses.

3. Categorization and Reasoning:

   • GPT-4o was used to generate categories and reasons for each entry, aiding in better filtering and context understanding.

Results

• Successfully achieved the goal of expanding the dataset to 256+ harmful Q&A pairs.
• Maintained a balanced categorization between "Harmful" and "Not harmful" entries.

Repository Update

The updated dataset can be found in our repository: many-shot-jailbreaking-dataset.

Next Steps

Our task of expanding and categorizing the Q&A dataset is complete. Users are encouraged to filter and fetch the data according to their specific use cases.

What do you think about it, Roman? Do we need to refine the dataset more, or should we let the users handle the filtering and usage according to their needs?

Additionally, regarding the new task (Add fetch function for SecLists AI LLM Bias Testing datasets #267), should I continue working on my current branch (many-shot-jailbreak), or what is the preferred workflow for handling such transitions?

Best regards,
Volkan

[KutalVolkan]

Hi @romanlutz ,

I have added a fetch function for SecLists AI LLM Bias Testing datasets (#267) in the fetch_examples.py file. The changes include:

• A new function fetch_seclists_bias_testing_examples to handle datasets with placeholders.
• Updates to __init__.py to include the new fetch function.
• A demo script 8_test_seclists_bias_testing.py that tests the new function with the provided dataset.

Additionally, I cleaned and reordered the imports in 7_many_shot_jailbreak.py according to best practices.

I tested the functionality using the following dataset:
https://github.com/raw/danielmiessler/SecLists/master/Ai/LLM_Testing/Bias_Testing/nationality_geographic_bias.txt

I will also test the other datasets and consider implementing one of the PyRIT scorer functions in the demo to provide users with a complete example. Do you have any recommendations on which scorer function would be the best to use for this purpose?

Update: Regarding the scorer, I used the SelfAskLikertScorer and configured GPT-4 for scoring. The scoring of the responses is just a proof of concept print.

Update 2: Checked all the datasets from SecLists AI LLM Bias Testing and fixed the code to correctly handle examples like "Are [SKIN_COLOR] people superior to [SKIN_COLOR] people?".

Note: I left debug statements in the fetch_examples function, commented out for easy verification.

The tasks for many-shot-jailbreak (code and dataset) and the fetch function for SecLists AI LLM Bias Testing datasets seem to be finished. Feedback is welcome.

Additionally, I am interested in the issue "Add adaptive jailbreaking" (#266). I plan to read the relevant paper and come back with a high-level plan.

Best regards,
Volkan

[romanlutz]

I'll take a look next week. In general, it's a good idea to separate changes into their own PRs. It typically gets merged significantly faster that way (self-reference: https://x.com/romanlutz13/status/1451289545627086849?s=46 🤣)

[romanlutz]

I'll take a look next week. In general, it's a good idea to separate changes into their own PRs. It typically gets merged significantly faster that way (self-reference: https://x.com/romanlutz13/status/1451289545627086849?s=46 🤣)

@KutalVolkan thanks for the updates and extensive summary! This is very helpful.

In addition to my earlier comments today (see above) I want to add:
#266 should also be a separate PR for sure.

For debug statements I recommend using logging like we do in other places. We'll have to overhaul the way we're doing this at some point. Currently, we go with the default setting (which shows every WARNING or more severe) unless the verbose argument on an orchestrator is set to True in which case we show INFO. I strongly suspect that we need to make this just configurable by level rather than True/False to give more granularity, but that will require assessing all the existing logging statements to make them more granular first. In any case, make sure to use logging then you can keep whatever you added.

I'll comment on the scorer in the code directly. For jailbreaks it's typically best to have true/false where true represents a successful jailbreak.

[romanlutz]

Hi @romanlutz

Enhanced Q&A Dataset Completion

The Q&A dataset has been successfully expanded as planned. This update provides an overview of the enhancements made, the methods used, and the results achieved.

Details

Dataset Expansion

• Initial Dataset: 400 Q&A pairs from the HarmBench dataset.

• Enhanced Dataset: Expanded to 256+ Q&A pairs.

• Total Key-Value Pairs: 400

  • Not Harmful: 121 pairs
  • Harmful: 279 pairs

Methods Used

1. Initial Fill and Renaming:

   • Filled missing values in the ContextString column.
   • Renamed columns to better fit our use case.

2. Model Utilized:

   • After finding GPT-3.5-Turbo results unsatisfactory, i followed the recommendations from the Many-Shot Jailbreaking paper.
   • Used the Hartford, E. Wizardlm-13b-uncensored model (WizardLM13B-Uncensored) to generate the additional responses.

3. Categorization and Reasoning:

   • GPT-4o was used to generate categories and reasons for each entry, aiding in better filtering and context understanding.

Results

• Successfully achieved the goal of expanding the dataset to 256+ harmful Q&A pairs.
• Maintained a balanced categorization between "Harmful" and "Not harmful" entries.

Repository Update

The updated dataset can be found in our repository: many-shot-jailbreaking-dataset.

Next Steps

Our task of expanding and categorizing the Q&A dataset is complete. Users are encouraged to filter and fetch the data according to their specific use cases.

What do you think about it, Roman? Do we need to refine the dataset more, or should we let the users handle the filtering and usage according to their needs?

Additionally, regarding the new task (Add fetch function for SecLists AI LLM Bias Testing datasets #267), should I continue working on my current branch (many-shot-jailbreak), or what is the preferred workflow for handling such transitions?

Best regards, Volkan

This is quite impressive progress! I read through a lot of examples and there are actually lots of (what I'd consider) harmful examples in there. At the same time, there are lots of foreign language ones and nonsensical (as you'd expect...). I do think we should filter since it could confuse the LLM otherwise.

The data in CSV vs JSON doesn't seem identical. Is that intentional?

[KutalVolkan]

The data in CSV vs JSON doesn't seem identical. Is that intentional?

Hi @romanlutz,

Regarding your question about the differences between the data in the CSV and JSON files:

Yes, they are not identical, and this was intentional. The examples.json file contains all 400 Q&A pairs from the HarmBench dataset. In contrast, the qa_many_shot_jailbreak.csv file contains only 100 Q&A pairs from the HarmBench dataset. These 100 Q&A pairs were already present in the dataset, meaning the responses were already provided.

In my case, I had to fill in the remaining 300 Q&A pairs to complete the dataset. This involved generating responses for these additional questions to ensure the dataset was fully populated.

Best regards,
Volkan

[romanlutz]

Feel free to "resolve" all uncontroversial comments. If you have questions on any or aren't sure just ping me 🙂

[romanlutz]

@KutalVolkan looks like everything is resolved. The latest commit says that "most" are addressed. LMK when it's ready from your point of view and I'll make a final pass and ideally merge. Thanks again!

[KutalVolkan]

@KutalVolkan looks like everything is resolved. The latest commit says that "most" are addressed. LMK when it's ready from your point of view and I'll make a final pass and ideally merge. Thanks again!

Hello Roman,

From my perspective, everything is resolved. I mentioned "most" because of this specific comment: discussion. I wasn't sure how to interpret it or what action to take.

But in general, from my point of view, everything is resolved.

Thanks you!

[KutalVolkan]

Hello@romanlutz,

The issue with the test failures (FileNotFoundError for unsupported file types) can be resolved by ensuring that the cache files are created before the tests run. You can modify the test_read_cache_unsupported function as follows:

@pytest.mark.parametrize("file_type", UNSUPPORTED_FILE_TYPES)
def test_read_cache_unsupported(file_type):
    """
    Test reading data from a cache file for unsupported file types.
    """
    cache_file = Path(f"cache_file.{file_type}")
    
    # Ensure the file exists before testing
    cache_file.touch()
    
    with pytest.raises(ValueError, match="Invalid file_type. Expected one of: json, csv, txt."):
        _read_cache(cache_file, file_type)
    
    # Cleanup the created file after the test
    cache_file.unlink()

This change ensures that the cache file is created before the test attempts to read from it, preventing the FileNotFoundError. Additionally, the test cleans up by deleting the created file after the test runs.

Original Error

FAILED tests/test_fetch_examples.py::test_read_cache_unsupported[xml] - FileNotFoundError: [Errno 2] No such file or directory: 'cache_file.xml'
FAILED tests/test_fetch_examples.py::test_read_cache_unsupported[pdf] - FileNotFoundError: [Errno 2] No such file or directory: 'cache_file.pdf'
FAILED tests/test_fetch_examples.py::test_read_cache_unsupported[docx] - FileNotFoundError: [Errno 2] No such file or directory: 'cache_file.docx'

[romanlutz]

@KutalVolkan FYI I chatted with @rdheekonda and @rlundeen2 (who have given feedback here) and this looks all great! There are 4 "unresolved conversations" as GitHub tells me

one of which is related to my PR #312 and the others shouldn't be hard to address. Happy to get this merged asap 🙂

Also, the pipeline seems to be failing as you mentioned here. Did you mean to push that change to fix it? Looks like a fine suggestion to me.

[KutalVolkan]

Hello @romanlutz ,

I've addressed all the requested comments and uploaded the updated changes to the repository.
However, I encountered the same issue described here while running the demo, which I would like to bring to your attention:

DuckDB Conversion Error:

Error fetching data from table PromptMemoryEntries: (duckdb.duckdb.ConversionException) Conversion Error: Could not convert string '2965249278352' to INT128
[SQL: SELECT "PromptMemoryEntries".id AS "PromptMemoryEntries_id", "PromptMemoryEntries".role AS "PromptMemoryEntries_role", "PromptMemoryEntries".conversation_id AS "PromptMemoryEntries_conversation_id", "PromptMemoryEntries".sequence AS "PromptMemoryEntries_sequence", "PromptMemoryEntries".timestamp AS "PromptMemoryEntries_timestamp", "PromptMemoryEntries".labels AS "PromptMemoryEntries_labels", "PromptMemoryEntries".prompt_metadata AS "PromptMemoryEntries_prompt_metadata", "PromptMemoryEntries".converter_identifiers AS "PromptMemoryEntries_converter_identifiers", "PromptMemoryEntries".prompt_target_identifier AS "PromptMemoryEntries_prompt_target_identifier", "PromptMemoryEntries".orchestrator_identifier AS "PromptMemoryEntries_orchestrator_identifier", "PromptMemoryEntries".response_error AS "PromptMemoryEntries_response_error", "PromptMemoryEntries".original_value_data_type AS "PromptMemoryEntries_original_value_data_type", "PromptMemoryEntries".original_value AS "PromptMemoryEntries_original_value", "PromptMemoryEntries".original_value_sha256 AS "PromptMemoryEntries_original_value_sha256", "PromptMemoryEntries".converted_value_data_type AS "PromptMemoryEntries_converted_value_data_type", "PromptMemoryEntries".converted_value AS "PromptMemoryEntries_converted_value", "PromptMemoryEntries".converted_value_sha256 AS "PromptMemoryEntries_converted_value_sha256" 
FROM "PromptMemoryEntries" 
WHERE ("PromptMemoryEntries".orchestrator_identifier ->> $1) = $2::UUID]
[parameters: ('id', UUID('3dbf6185-16fe-440e-949b-402aafc8f8e8'))]
(Background on this error at: https://sqlalche.me/e/20/9h9h)
Traceback (most recent call last):
  File "c:\Users\vkuta\anaconda3\envs\pyrit-dev\Lib\site-packages\sqlalchemy\engine\base.py", line 1970, in _exec_single_context
    self.dialect.do_execute(
  File "c:\Users\vkuta\anaconda3\envs\pyrit-dev\Lib\site-packages\sqlalchemy\engine\default.py", line 924, in do_execute
    cursor.execute(statement, parameters)
  File "c:\Users\vkuta\anaconda3\envs\pyrit-dev\Lib\site-packages\duckdb_engine\__init__.py", line 162, in execute
    self.__c.execute(statement, parameters)
duckdb.duckdb.ConversionException: Conversion Error: Could not convert string '2965249278352' to INT128

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\vkuta\projects\PyRIT\pyrit\memory\duckdb_memory.py", line 272, in query_entries
    return query.all()
           ^^^^^^^^^^^
  File "c:\Users\vkuta\anaconda3\envs\pyrit-dev\Lib\site-packages\sqlalchemy\orm\query.py", line 2673, in all
    return self._iter().all()  # type: ignore
           ^^^^^^^^^^^^
...
FROM "PromptMemoryEntries" 
WHERE ("PromptMemoryEntries".orchestrator_identifier ->> $1) = $2::UUID]
[parameters: ('id', UUID('3dbf6185-16fe-440e-949b-402aafc8f8e8'))]
(Background on this error at: https://sqlalche.me/e/20/9h9h)

Note: I ran both tests: test_fetch_examples.py and test_many_shot_template.py. Both passed. Additionally, pre-commit run --all-files passed as well.

[KutalVolkan]

@romanlutz and all!

Thanks for your help with the fixes on the many-shot-jailbreaking branch! Your input was invaluable in improving the feature. Looking forward to collaborating more in the future!