Release v1.1.0

What's new in Counterfit v1.1.0

                              __            _____ __
      _________  __  ______  / /____  _____/ __(_) /_
     / ___/ __ \/ / / / __ \/ __/ _ \/ ___/ /_/ / __/
    / /__/ /_/ / /_/ / / / / /_/  __/ /  / __/ / /
    \___/\____/\__,_/_/ /_/\__/\___/_/  /_/ /_/\__/

                    Version: 1.1.0
        

We're thrilled to announce a new release version for Counterfit! 🚀

1. Counterfit as a package

• Counterfit can now be used from within a Python script.
• Access to all of the targets and attacks that the Counterfit CLI supports.

import counterfit
import counterfit.targets as cf_targets


digits_target = cf_targets.Digits()
digits_target.load()
cf_attack = counterfit.Counterfit.build_attack(digits_target, 'hop_skip_jump')
results = counterfit.Counterfit.run_attack(cf_attack)

2. Updates to Counterfit CLI

• Counterfit CLI now automatically loads attacks and frameworks.
• Two new sample targets that show you how to attack reinforcement learning. i.e. Cart Pole targets.

$ counterfit
                          __            _____ __
  _________  __  ______  / /____  _____/ __(_) /_
 / ___/ __ \/ / / / __ \/ __/ _ \/ ___/ /_/ / __/
/ /__/ /_/ / /_/ / / / / /_/  __/ /  / __/ / /
\___/\____/\__,_/_/ /_/\__/\___/_/  /_/ /_/\__/

                Version: 1.1.0

counterfit> list targets 
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name                ┃ Model Type ┃ Data Type ┃ Input Shape   ┃ # Samples ┃ Endpoint                                             ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ cart_pole           │ closed-box │ tabular   │ (1080000,)    │ 0         │ cartpole_dqn_10000.pt.gz                             │
│ cart_pole_initstate │ closed-box │ tabular   │ (4,)          │ 0         │ cartpole_dqn_10000.pt.gz                             │
│ creditfraud         │ closed-box │ tabular   │ (30,)         │ 0         │ creditfraud/creditfraud_sklearn_pipeline.pkl         │
│ digits_keras        │ closed-box │ image     │ (28, 28, 1)   │ 0.        │ digits_keras/mnist_model.h5                          │
│ digits_mlp          │ closed-box │ image     │ (1, 28, 28)   │ 0         │ digits_mlp/mnist_sklearn_pipeline.pkl                │
│ movie_reviews       │ closed-box │ text      │ (1,)          │ 0.        │ movie_reviews/movie_reviews_sentiment_analysis.pt    │
│ satellite           │ closed-box │ image     │ (3, 256, 256) │ 0         │ satellite/satellite-image-params-airplane-stadium.h5 │
└─────────────────────┴────────────┴───────────┴───────────────┴───────────┴──────────────────────────────────────────────────────┘

counterfit> set_target satellite

satellite> set_attack hop_skip_jump 
[+] success:  Using fb58020f

satellite>HopSkipJump:fb58020f> show info
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Attack Field ┃ Description                                                                                                                                               ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Name         │ hop_skip_jump                                                                                                                                             │
│ Type         │ closed-box                                                                                                                                                │
│ Category     │ evasion                                                                                                                                                   │
│ Tags         │ image, tabular                                                                                                                                            │
│ Framework    │ art                                                                                                                                                       │
│ Docs         │ Implementation of the HopSkipJump attack from Jianbo et al. (2019). This is a powerful closed-box attack that only requires final class prediction, and   │
│              │ is an advanced version of the boundary attack. | Paper link: https://arxiv.org/abs/1904.02144                                                             │
└──────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

satellite>HopSkipJump:fb58020f> run
HopSkipJump:   0%|                                                                                                                                    | 0/1 [00:00<?, ?it/sFailed to draw a random image that is adversarial, attack failed.
HopSkipJump: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:04<00:00,  4.88s/it]
[+] success:  Attack completed fb58020f

3. Update to internal architecture

• Internal Counterfit repository is updated to be more extensible and easier to understand.
• Updated targets and attacks naming convention.
• Adds framework for integration tests before pull-requests can be merged to main.

4. Support for Reinforcement Learning attacks

• Counterfit leverages OpenAI gym to use different attacks on Reinforcement Learning (RL) targets such as Cart Pole.
• Allow Windows and Linux users with a GUI to see the live attack on the Cart Pole target.
• Counterfit can run in headless mode to attack Cart Pole target when no GUI available.
• See demo example on the Hands-on Hacking of Reinforcement Learning Systems demo in our Counterfit Examples document.

The not so fine print: Please read!

Release v.1.1.0 of Counterfit is not backwards compatible with Counterfit v1.0.0. This is due to major changes in the architecture of the software and how Counterfit targets and attacks are loaded. The good news is that Counterfit v1.1.0 has fully ported all the previous attacks and targets.

To improve the readability of the Counterfit command line interface (CLI), Counterfit v1.1.0 updates the naming convention of some of the commands while it deprecates others.

• The interact command has been replaced with the more idiomatic set_target command.
• The use command has been replaced with the more idiomatic set_attack command.
• The set command has been replaced with the more idiomatic set_params command.
• The load command has been deprecated. Frameworks and attacks are now automatically loaded by the CLI.

Release v1.0.0

                          __            _____ __
  _________  __  ______  / /____  _____/ __(_) /_
 / ___/ __ \/ / / / __ \/ __/ _ \/ ___/ /_/ / __/
/ /__/ /_/ / /_/ / / / / /_/  __/ /  / __/ / /
\___/\____/\__,_/_/ /_/\__/\___/_/  /_/ /_/\__/

                                v1.0    #ATML

---------------------------------------------------

Description

First and foremost, the ATML team would like the thank everyone for their support over the last few months. Counterfit recieved a very warm welcome from the community. What started as some simple red team tooling has become a place for collaboration, experiementatation, and of course security assessments. While verson 0.1 was useful, unless a user was familiar with the code, it was admitedly difficult to use beyond it's basic functionality. Users of Counterfit should know that their frustrations with the tool were also our frustrations. While our internal version may have different targets, custom algos, reporting, the public version of Counterfit is ultimately the base of our internal version. For those unfamiliar with infosec, this is a common practice that creates a shared experience. These shared experiences will allow us to communicate and come to a common understanding of risk in the ML space.

Let's checkout the new digs. We will cover the changes at a high-level and get into details later,

• Frameworks are a first-class concept.
• New logging capabilities
• Options structure
• New attacks from art, textattack
• New attacks via Augly
• Various command functionality
• Running via run_pyscript
• New reporting structure
• Python Rich integration
• docs and tests

Frameworks are a first-class concept

Frameworks are the drivers behind Counterfit and they provide the functionality for Counterfit. Counterfit now takes a back seat and offloads the majority of work to the framework responsible for an attack. Frameworks are not loaded on start, rather by using the load command Like other objects in Counterfit, frameworks are built around their folder structure within the project. Each framework has its own folder under counterfit/frameworks.In order to be loaded by Counterfit, a framework should inherit from counterfit.core.frameworks.Framework. A framework should also define a number of core functions. These include load(), build(), run(), check_success(), pre_attack_proccessing(), post_attack_processing(). Everything begins and ends with a framework and so in order to add a new framework it is important to be familiar with some Counterfit internals.

Python Rich integration

Thanks to Python Rich, Counterfit has a lot more colors and is generally better looking. Rich requires that everything is string or a "renderable". Be aware of this when using the logging module.

Notes

A new version of Counterfit will be released in late October, 2022. To get the latest version of Counterfit, see release v1.1.0.