FEAT Adding Image Converter: add text on image

doc/code/converters.ipynb

@@ -5,12 +5,32 @@
    "id": "d0e03432",
    "metadata": {},
    "source": [
-    "### Converters\n",
-    "\n",
+    "## Converters"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "abbe7811",
+   "metadata": {},
+   "source": [
     "Converters are used to transform prompts before sending them to the target.\n",
     "\n",
-    "This can be useful for a variety of reasons, such as encoding the prompt in a different format, or adding additional information to the prompt. For example, you might want to convert a prompt to base64 before sending it to the target, or add a prefix to the prompt to indicate that it is a question.\n",
-    "\n",
+    "This can be useful for a variety of reasons, such as encoding the prompt in a different format, or adding additional information to the prompt. For example, you might want to convert a prompt to base64 before sending it to the target, or add a prefix to the prompt to indicate that it is a question."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c0d04756",
+   "metadata": {},
+   "source": [
+    "### Simple Converters"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "06d7cb11",
+   "metadata": {},
+   "source": [
     "Converters can be used to perform these types of transformations. Here is a simple program that uses Rot13Converter converter, RandomCapitalLettersConverter, and AsciiArtConverter"
    ]
   },
@@ -77,10 +97,39 @@
     "print(AsciiArtConverter().convert(prompt=prompt))"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "a422e1b0",
+   "metadata": {},
+   "source": [
+    "### Orchestrators"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f9247b88",
+   "metadata": {},
+   "source": [
+    "Converters should be thought of as a piece in the pipeine.\n",
+    "\n",
+    "An orchestrator will typically initialize these requests, and they are sent to a target.\n",
+    "Converters can also stack, so a converter is used one after another.\n",
+    "\n",
+    "See [demo3](../demo/3_send_all_prompts.ipynb) and [demo4](../demo/4_prompt_variation.ipynb) for an example of how to use a converter in the pipeline."
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "9bdd65e9",
    "metadata": {},
+   "source": [
+    "### Converters with LLMs"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9d7e8e24",
+   "metadata": {},
    "source": [
     "Some converters use external infrastructure like attacker LLMs. `VariationConverter` is a converter that does this.\n",
     "However, converters like this are significantly slower to run."
@@ -147,12 +196,7 @@
    "id": "346c727b",
    "metadata": {},
    "source": [
-    "Converters should be thought of as a piece in the pipeine.\n",
-    "\n",
-    "An orchestrator will typically initialize these requests, and they are sent to a target.\n",
-    "Converters can also stack, so a converter is used one after another.\n",
-    "\n",
-    "See [demo3](../demo/3_send_all_prompts.ipynb) and [demo4](../demo/4_prompt_variation.ipynb) for an example of how to use a converter in the pipeline."
+    "### Audio Converters"
    ]
   },
   {
@@ -196,6 +240,66 @@
     "print (audio_convert_result)\n",
     "assert os.path.exists(audio_convert_result.output_text)"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e031a104",
+   "metadata": {},
+   "source": [
+    "### Image Converters"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d7495ced",
+   "metadata": {},
+   "source": [
+    "Text can be added to images by using the `AddTextImageConverter`. \n",
+    "The converted image file will be saved in the db/results/images folder. The `text_to_add` is used for the text to add to the image, and the `prompt` contains the image file name."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "367ba859",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pyrit.prompt_converter import AddTextImageConverter\n",
+    "from pyrit.common.path import HOME_PATH\n",
+    "import pathlib\n",
+    "\n",
+    "image_converter = AddTextImageConverter(\n",
+    "    font_size=0.03, \n",
+    "    color=(0,0,0), \n",
+    "    text_to_add=[\"We can add text into this image now!\"])\n",
+    "output_image_file = image_converter.convert(\n",
+    "    prompt=str(pathlib.Path(HOME_PATH) / \"assets\" / \"pyrit_architecture.png\"),\n",
+    ")\n",
+    "\n",
+    "print(output_image_file)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7de9cc54",
+   "metadata": {},
+   "source": [
+    "To view the resulting image, run the code below"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fae3d0c3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from PIL import Image\n",
+    "\n",
+    "output_image = Image.open(output_image_file.output_text)\n",
+    "output_image.show()"
+   ]
   }
  ],
  "metadata": {

doc/code/converters.py

@@ -1,13 +1,29 @@
-# %% [markdown]
-# ### Converters
-#
+# ---
+# jupyter:
+#   jupytext:
+#     cell_metadata_filter: -all
+#     text_representation:
+#       extension: .py
+#       format_name: light
+#       format_version: '1.5'
+#       jupytext_version: 1.16.1
+#   kernelspec:
+#     display_name: pyrit-311
+#     language: python
+#     name: pyrit-311
+# ---
+
+# ## Converters
+
 # Converters are used to transform prompts before sending them to the target.
 #
 # This can be useful for a variety of reasons, such as encoding the prompt in a different format, or adding additional information to the prompt. For example, you might want to convert a prompt to base64 before sending it to the target, or add a prefix to the prompt to indicate that it is a question.
-#
+
+# ### Simple Converters
+
 # Converters can be used to perform these types of transformations. Here is a simple program that uses Rot13Converter converter, RandomCapitalLettersConverter, and AsciiArtConverter
 
-# %%
+# +
 # Copyright (c) Microsoft Corporation.
 # Licensed under the MIT license.
 
@@ -28,13 +44,24 @@
 print(RandomCapitalLettersConverter(percentage=25.0).convert(prompt=prompt))
 
 print(AsciiArtConverter().convert(prompt=prompt))
+# -
+
 
+# ### Orchestrators
+
+# Converters should be thought of as a piece in the pipeine.
+#
+# An orchestrator will typically initialize these requests, and they are sent to a target.
+# Converters can also stack, so a converter is used one after another.
+#
+# See [demo3](../demo/3_send_all_prompts.ipynb) and [demo4](../demo/4_prompt_variation.ipynb) for an example of how to use a converter in the pipeline.
+
+# ### Converters with LLMs
 
-# %% [markdown]
 # Some converters use external infrastructure like attacker LLMs. `VariationConverter` is a converter that does this.
 # However, converters like this are significantly slower to run.
 
-# %%
+# +
 import os
 import pathlib
 
@@ -59,20 +86,14 @@
     )
     variation_converter = VariationConverter(converter_target=attack_llm, prompt_template=variation_converter_strategy)
     print(variation_converter.convert(prompt=prompt))
+# -
 
-# %% [markdown]
-# Converters should be thought of as a piece in the pipeine.
-#
-# An orchestrator will typically initialize these requests, and they are sent to a target.
-# Converters can also stack, so a converter is used one after another.
-#
-# See [demo3](../demo/3_send_all_prompts.ipynb) and [demo4](../demo/4_prompt_variation.ipynb) for an example of how to use a converter in the pipeline.
+# ### Audio Converters
 
-# %% [markdown]
 #
 # Converters can also be multi-modal. Because it's an abstract function used interchangeably on a single `PromptRequestPiece`, it can only deal with one input value and type per time, and have one output value and type per time. Below is an example of using `AzureSpeechTextToAudioConverter`, which has an input type of `text` and an output type of `audio_path`.
 
-# %%
+# +
 
 from pyrit.prompt_converter import AzureSpeechTextToAudioConverter
 
@@ -82,4 +103,32 @@
 
 print(audio_convert_result)
 assert os.path.exists(audio_convert_result.output_text)
-# %%
+# -
+
+# ### Image Converters
+
+# Text can be added to images by using the `AddTextImageConverter`.
+# The converted image file will be saved in the db/results/images folder. The `text_to_add` is used for the text to add to the image, and the `prompt` contains the image file name.
+
+# +
+from pyrit.prompt_converter import AddTextImageConverter
+from pyrit.common.path import HOME_PATH
+import pathlib
+
+image_converter = AddTextImageConverter(
+    font_size=0.03, color=(0, 0, 0), text_to_add=["We can add text into this image now!"]
+)
+output_image_file = image_converter.convert(
+    prompt=str(pathlib.Path(HOME_PATH) / "assets" / "pyrit_architecture.png"),
+)
+
+print(output_image_file)
+# -
+
+# To view the resulting image, run the code below
+
+# +
+from PIL import Image
+
+output_image = Image.open(output_image_file.output_text)
+output_image.show()

pyproject.toml

@@ -35,6 +35,7 @@ requires-python = ">=3.10, <3.12"
 dependencies = [
     "appdirs>=1.4.0",
     "art==6.1.0",
+    "augly==1.0.0",
     "azure-cognitiveservices-speech>=1.36.0",
     "azure-core>=1.26.1",
     "azure-identity>=1.12.0",

pyrit/models/data_type_serializer.py

@@ -62,11 +62,17 @@ def save_data(self, data: bytes) -> None:
         with open(self.value, "wb") as file:
             file.write(data)
 
-    def save_b64_image(self, data: str) -> None:
+    def save_b64_image(self, data: str, output_filename: str = None) -> None:
         """
         Saves the base64 encoded image to disk.
+        Arguments:
+            data: string with base64 data
+            output_filename (optional, str): filename to store image as. Defaults to UUID if not provided
         """
-        self.value = str(self.get_data_filename())
+        if output_filename:
+            self.value = output_filename
+        else:
+            self.value = str(self.get_data_filename())
         with open(self.value, "wb") as file:
             image_bytes = base64.b64decode(data)
             file.write(image_bytes)

pyrit/prompt_converter/__init__.py

@@ -3,6 +3,7 @@
 
 from pyrit.prompt_converter.prompt_converter import ConverterResult, PromptConverter
 
+from pyrit.prompt_converter.add_text_image_converter import AddTextImageConverter
 from pyrit.prompt_converter.ascii_art_converter import AsciiArtConverter
 from pyrit.prompt_converter.base64_converter import Base64Converter
 from pyrit.prompt_converter.search_replace_converter import SearchReplaceConverter
@@ -18,6 +19,7 @@
 
 
 __all__ = [
+    "AddTextImageConverter",
     "AsciiArtConverter",
     "Base64Converter",
     "ConverterResult",