Mixtral.py V0.0.1

Installation

$ pip/pip3 install mixtral.py

Usage

Run the model

import mixtral
mixtral = Mixtral()
print("Running the model:")
print(mixtral.generate_text("..."))

In half-precision

import mixtral
print("Running the model in half precision:")
mixtral_half_precision = Mixtral(use_half_precision=True)
print(mixtral_half_precision.generate_text("..."))

Lower precision using (8-bit & 4-bit) using bitsandbytes

import mixtral
print("Running the model in lower precision using (8-bit & 4-bit) using bitsandbytes:")
mixtral_4bit = Mixtral(load_in_4bit=True)
print(mixtral_4bit.generate_text("..."))

Load the model with Flash Attention 2

import mixtral
print("Load the model with Flash Attention 2:")
mixtral_flash_attention_2 = Mixtral(use_flash_attention_2=True)
print(mixtral_flash_attention_2.generate_text("..."))

Hardware Requirements

• minimum 100GB GPU RAM (Mistral AI)
• Hardware requirements after fine-tuning can be found in the discussion here

Licenses

• mixtral.py is under the GNU General Public License v3
• Mixtral 8x7b is under the Apache 2.0 License

Verifying Signatures

Import PGP Key into Keyring

gpg --keyserver 185.125.188.27 --recv-keys 20247EC023F2769E66181C0F581B4A2A862BBADE

or

wget https://github.com/ibnaleem/ibnaleem/blob/main/public_key.asc

Download Signature

The signatures (.asc and .sig) can be found in the /sig directory. Download either of them. Open an issue with the title "invalid signature/wrong signature/corrupt signature" for any issues regarding my signatures.

Sign My Key

gpg --sign-key 20247EC023F2769E66181C0F581B4A2A862BBADE
gpg --send-keys 20247EC023F2769E66181C0F581B4A2A862BBADE

If you cannot upload your signature, see "gpg: keyserver receive failed: No route to host"

Verify

gpg --verify mixtral.py.asc mixtral.py

Desired output:

gpg: Signature made Tue  6 Feb 10:27:34 2024 GMT
gpg:                using RSA key 20247EC023F2769E66181C0F581B4A2A862BBADE
gpg: Good signature from "Ibn Aleem <ibnaleem@outlook.com>" [ultimate]