[FEAT] Implement PAIR #255

dlmgary · 2024-06-23T15:58:59Z

Description

This PR:

Implements the PAIR algorithm as a PyRIT orchestrator. This algorithm is described in Chao, Patrick, et al. "Jailbreaking black box large language models in twenty queries." arXiv preprint arXiv:2310.08419 (2023).
Adds tests.
Adds Jupyter notebook to show example PAIR Orchestrator in action.
Improves MemoryInterface.duplicate_conversation_for_new_orchestrator() to return the conversation_id of the new conversation and removes edge cases related to UUID collisions. This caused some tests to be removed and another one updated.
New function added to utils to generate UUID without collisions.
Fixes bug in tests/test_attack_strategy.py.

Tests and Documentation

Tests added for PAIR Orchestrator
Some tests removed for MemoryInterface.duplicate_conversation_for_new_orchestrator() and one updated.

…urn new conversation and fully remove edge cases with UUID collisions.

pyrit/orchestrator/pair_orchestrator.py

pyrit/common/default_values.py

pyrit/datasets/orchestrators/pair/attacker_system_prompt.yaml

doc/code/orchestrators/pair_orchestrator.py

pyrit/orchestrator/pair_orchestrator.py

tests/orchestrator/test_pair_orchestrator.py

pyrit/orchestrator/pair_orchestrator.py

doc/code/orchestrators/pair_orchestrator.py

…ation starts when `start_new_conversation` is set to true. Previously, it always evaluated to True.

pyrit/datasets/orchestrators/pair/judge_system_prompt.yaml

pyrit/orchestrator/__init__.py

pyrit/orchestrator/pair_orchestrator.py

refactor: improve readability of `author` field. Co-authored-by: Roman Lutz <romanlutz13@gmail.com>

doc/code/orchestrators/pair_orchestrator.ipynb

pyrit/memory/memory_interface.py

pyrit/orchestrator/pair_orchestrator.py

pyrit/orchestrator/red_teaming_orchestrator.py

tests/orchestrator/test_pair_orchestrator.py

pyrit/orchestrator/pair_orchestrator.py

doc/code/orchestrators/pair_orchestrator.ipynb

tests/orchestrator/test_pair_orchestrator.py

…ing_chat, fix tests

dlmgary added 8 commits June 23, 2024 11:13

refactor: update duplicate_conversation_for_new_orchestrator to ret…

0079ac7

…urn new conversation and fully remove edge cases with UUID collisions.

bugfix: fixed issue with relative test file reference.

ca01dd9

feat: implement PAIR algorithm as PyRIT orchestrator.

6eefc57

refactor: remove unnecessary comments

9cc4a1e

refactor: fix mypy issues.

28d6940

refactor: undo accidental file changes.

9631fd9

Merge branch 'main' into pair

f2d06f1

refactor: undo accidental file changes.

8da3d44

dlmgary requested review from romanlutz, rlundeen2 and rdheekonda June 24, 2024 17:54

dlmgary added 2 commits June 24, 2024 14:29

docs: add more documentation to pair orchestrator

0081982

refactor: fixes mypy

70c8f2b

romanlutz reviewed Jun 24, 2024

View reviewed changes

pyrit/orchestrator/pair_orchestrator.py Show resolved Hide resolved

romanlutz reviewed Jun 24, 2024

View reviewed changes

pyrit/orchestrator/pair_orchestrator.py Outdated Show resolved Hide resolved

romanlutz reviewed Jun 24, 2024

View reviewed changes

dlmgary added 6 commits June 25, 2024 08:52

refactor: updates code based on PR suggestions.

19fc085

docs: add .py version of PAIR jupyter notebook.

044d180

refactor: fixes mypy

2d0e0b8

refactor: linter

9de53b9

refactor: update to use PromptNormalizer

cf245e4

Merge branch 'main' into pair

69745ae

romanlutz reviewed Jun 27, 2024

View reviewed changes

bugfix: update _get_attacker_response_and_store so that new convers…

89ad6bb

…ation starts when `start_new_conversation` is set to true. Previously, it always evaluated to True.

rlundeen2 reviewed Jun 27, 2024

View reviewed changes

pyrit/datasets/orchestrators/pair/judge_system_prompt.yaml Outdated Show resolved Hide resolved

rlundeen2 reviewed Jun 27, 2024

View reviewed changes

pyrit/orchestrator/__init__.py Outdated Show resolved Hide resolved

rlundeen2 reviewed Jun 27, 2024

View reviewed changes

pyrit/orchestrator/pair_orchestrator.py Outdated Show resolved Hide resolved

dlmgary and others added 3 commits June 28, 2024 09:16

refactor: fix import order name.

8e1b97c

Update pyrit/datasets/orchestrators/pair/attacker_system_prompt.yaml

8a8d101

refactor: improve readability of `author` field. Co-authored-by: Roman Lutz <romanlutz13@gmail.com>

refactor: update orchestrator variables to be private.

2ec26f4

dlmgary added 6 commits July 23, 2024 07:06

Merge branch 'main' into pair

0753550

refactor: remove SelfAskPAIRScorer and refactor PAIR Orchestrator

bd5ffaa

refactor: update notebook

0e803ea

bugfix: fixes invalid exception

98d9270

refactor: update test

41a71d5

Merge branch 'main' into pair

7254abd

romanlutz mentioned this pull request Jul 24, 2024

FEAT return ID in conversation duplication code #296

Merged

romanlutz reviewed Jul 24, 2024

View reviewed changes

dlmgary added 13 commits July 24, 2024 12:45

refactor: update jupyter notebook to use AzureOpenAIChatTarget

155fcbb

refactor: fix variable names

9dccd2b

refactor: undo memory updates

241da3a

refactor: undo memory changes

4d3da2f

refactor: add more test + rename PAIROrchestrator

f41df89

Merge branch 'main' into pair

2aae93f

refactor: bugfix.

f442974

refactor: reformat code

6b6e114

Merge branch 'main' into pair

ba8a1d4

Merge branch 'main' into pair

e931f0f

Merge branch 'main' into pair

3a9add5

refactor: undo PR changes to red_teaming_orchestrator.py

64bc685

refactor: update internal variable names.

6d5bc5c

romanlutz reviewed Jul 26, 2024

View reviewed changes

doc/code/orchestrators/pair_orchestrator.ipynb Show resolved Hide resolved

tests/orchestrator/test_pair_orchestrator.py Outdated Show resolved Hide resolved

romanlutz and others added 5 commits July 26, 2024 12:35

Merge branch 'main' of https://github.com/Azure/PyRIT into pair

226e3ee

Merge branch 'main' into pair

3ddf8db

various fixes to address comments on the PR, fix retries for red_team…

ff08178

…ing_chat, fix tests

Merge branch 'main' of https://github.com/Azure/PyRIT into pair

d0cfa18

Merge branch 'pair' of https://github.com/dlmgary/PyRIT into pair

1768833

romanlutz approved these changes Jul 27, 2024

View reviewed changes

romanlutz merged commit 6b92903 into Azure:main Jul 27, 2024
4 checks passed

dlmgary deleted the pair branch July 29, 2024 17:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEAT] Implement PAIR #255

[FEAT] Implement PAIR #255

dlmgary commented Jun 23, 2024 •

edited

Loading

[FEAT] Implement PAIR #255

[FEAT] Implement PAIR #255

Conversation

dlmgary commented Jun 23, 2024 • edited Loading

Description

Tests and Documentation

dlmgary commented Jun 23, 2024 •

edited

Loading