v2.7.0 (#350)

Changes:
- New multi-choice benchmarking task;
- Refactor changes (.env config-based, decoupled parts of the code);
- Poetry setup;
- Only 5 tasks are included: QA, DateQA, Summary, MultiChoice, Organic.

.env.miner.example

@@ -0,0 +1,20 @@
+# The network UID [1, 61]
+NETUID=61
+
+# The network name [test, finney, local]
+SUBTENSOR_NETWORK = "test"
+
+# The chain endpoint, only required to be set if you run local.
+SUBTENSOR_CHAIN_ENDPOINT = None
+
+# The name of your wallet.
+WALLET_NAME="miner"
+
+# The name of the hotkey associated with the validator wallet.
+HOTKEY="default"
+
+# Open port which can be used to connect to the network.
+AXON_PORT=22116
+
+# The OpenAI API key (only needed for the OpenAI test miner).
+OPENAI_API_KEY="YOUR_API_HERE"

.env.validator.example

@@ -0,0 +1,20 @@
+# The network UID. If 1 for main, 61 for testnet.
+NETUID=61
+
+# The network name [test, finney, local].
+SUBTENSOR_NETWORK = "test"
+
+# The chain endpoint, only required to be set if you run local.
+SUBTENSOR_CHAIN_ENDPOINT = None
+
+# The name of your wallet.
+WALLET_NAME="validator"
+
+# The name of the hotkey associated with the validator wallet.
+HOTKEY="default"
+
+# Open port which can be used to connect to the network.
+AXON_PORT=22116
+
+# HuggingFace Access Token.
+HF_TOKEN=""

.github/workflows/python-package.yml

@@ -5,10 +5,8 @@ name: Prompting CI/CD
 
 on:
   push:
-
 jobs:
   build:
-
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
@@ -24,19 +22,16 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        python -m pip install flake8 pytest black
         bash install.sh
-
-    - name: Lint with flake8
-      run: |
-        # stop the build if there are Python syntax errors or undefined names
-        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
-        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
-        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
-    - name: Black
+        poetry install --all-extras
+        poetry run pip uninstall -y uvloop
+    - name: Debug Environment
       run: |
-        black .
+        poetry run python -V
+        poetry run which python
+        poetry env info
+        poetry run pip list
     - name: Test with pytest
       run: |
         # run tests in tests/ dir and only fail if there are failures or errors
-        pytest tests/ --verbose --failed-first --exitfirst --disable-warnings
+        poetry run pytest tests/ --verbose --failed-first --exitfirst --disable-warnings

.gitignore

@@ -5,6 +5,8 @@ __pycache__/
 .DS_Store
 **/.DS_Store
 
+*.npz
+prompting/storage/
 
 # C extensions
 *.so
@@ -124,6 +126,8 @@ celerybeat.pid
 
 # Environments
 .env
+.env.miner
+.env.validator
 .venv
 env/
 venv/
@@ -165,8 +169,7 @@ cython_debug/
 testing/
 data/*
 plots/*
-notebooks/*
 core
 app.config.js
 wandb
-.vscode
+.vscode

.pre-commit-config.yaml

@@ -0,0 +1,16 @@
+repos:
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v2.3.0
+    hooks:
+    -   id: check-yaml
+    -   id: end-of-file-fixer
+    -   id: trailing-whitespace
+- repo: https://github.com/astral-sh/ruff-pre-commit
+  # Ruff version.
+  rev: v0.5.2
+  hooks:
+    # Run the linter.
+    - id: ruff
+      args: [ --fix ]
+    # Run the formatter.
+    - id: ruff-format

Makefile

@@ -1,4 +1,4 @@
 SHELL:=/bin/bash
 
 promote-changes:
-	./scripts/promote_changes.sh
+	./scripts/promote_changes.sh

README.md

@@ -8,7 +8,7 @@
 
 # **Bittensor SN1** <!-- omit in toc -->
 [![Discord Chat](https://img.shields.io/discord/308323056592486420.svg)](https://discord.gg/bittensor)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) 
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
 ---
 
@@ -24,98 +24,18 @@ This repository is the **official codebase for Bittensor Subnet 1 (SN1) v1.0.0+,
 
 # Introduction
 
-This repo defines an incentive mechanism to create a distributed conversational AI for Subnet 1 (SN1). 
+This repo defines an incentive mechanism to create a distributed conversational AI for Subnet 1 (SN1).
 
-Validators and miners are based on large language models (LLM). The validation process uses **internet-scale datasets and goal-driven behaviour to drive human-like conversations**. 
+Validators and miners are based on large language models (LLM). The validation process uses **internet-scale datasets and goal-driven behaviour to drive human-like conversations**.
 
 
 </div>
 
-# Installation
-This repository requires python3.9 or higher. To install it, simply clone this repository and run the [install.sh](./install.sh) script.
-```bash
-git clone https://github.com/opentensor/prompting.git
-cd prompting
-bash install.sh
-```
-If you are running a miner, you will also need to uninstall uvloop.
-```bash
-pip uninstall uvloop -y
-```
-
-If you are running a validator, logging in to Hugging Face is required:
-```shell
-huggingface-cli login
-```
-You also need to accept the License Agreement for the LMSYS-Chat-1M dataset: https://huggingface.co/datasets/lmsys/lmsys-chat-1m
+# Usage
 
-</div>
+<div align="center">
 
-# Compute Requirements
+**[For Validators](./assets/validator.md)** · **[For Miners](./assets/miner.md)**
 
-1. To run a **validator**, you will need at least 62GB of VRAM. 
-2. To run the default huggingface **miner**, you will need at least 62GB of VRAM.
 
-
-**It is important to note that the baseminers are not recommended for main, and exist purely as an example. Running a base miner on main will result in no emissions and a loss in your registration fee.**
-If you have any questions please reach out in the SN1 channel in the Bittensor Discord.
 </div>
-
-# How to Run
-You can use the following command to run a miner or a validator. 
-
-```bash
-python <SCRIPT_PATH>
-    --netuid 1
-    --subtensor.network <finney/local/test>
-    --neuron.device cuda
-    --wallet.name <your wallet> # Must be created using the bittensor-cli
-    --wallet.hotkey <your hotkey> # Must be created using the bittensor-cli
-    --logging.debug # Run in debug mode, alternatively --logging.trace for trace mode
-    --axon.port # VERY IMPORTANT: set the port to be one of the open TCP ports on your machine
-```
-
-where `SCRIPT_PATH` is either: 
-1. neurons/miners/openai/miner.py
-2. neurons/validator.py
-
-For ease of use, you can run the scripts as well with PM2. Installation of PM2 is: 
-**On Linux**:
-```bash
-sudo apt update && sudo apt install jq && sudo apt install npm && sudo npm install pm2 -g && pm2 update
-``` 
-
-Example of running an Openai miner on Main:
-
-```bash
-pm2 start neurons/miners/openai/miner.py --interpreter python --name openai_miner -- --netuid 1  --subtensor.network finney --wallet.name my_wallet --wallet.hotkey my_hotkey --neuron.model_id gpt-3.5-turbo-1106 --axon.port 8091 
-```
-
-## Running with autoupdate
-
-You can run the validator in auto-update mode by using pm2 along with the `run.sh` bash script. This command will initiate two pm2 processes: one for auto-update monitoring, named **s1_validator_update**, and another for running the validator itself, named **s1_validator_main_process**.
-```bash
-pm2 start run.sh --name s1_validator_autoupdate -- --wallet.name <your-wallet-name> --wallet.hotkey <your-wallet-hot-key>
-```
-
-> Note: this is not an end solution, major releases or changes in requirements will still require you to manually restart the processes. Regularly monitor the health of your validator to ensure optimal performance.
-
-# Testnet 
-We highly recommend that you run your miners on testnet before deploying on main. This is give you an opportunity to debug your systems, and ensure that you will not lose valuable immunity time. The SN1 testnet is **netuid 61**. 
-
-In order to run on testnet, you will need to go through the same hotkey registration proceure as on main, but using **testtao**. You will need to ask for some in the community discord if you do not have any. 
-
-To run:
-
-```bash
-pm2 start neurons/miners/openai/miner.py --interpreter python3 --name openai_miner -- --netuid 61 --subtensor.network test --wallet.name my_test_wallet --wallet.hotkey my_test_hotkey --neuron.model_id gpt-3.5-turbo-1106 --axon.port 8091
-```
-
-# Limitations
-> Important: vLLM currently faces a [notable limitation](https://github.com/vllm-project/vllm/issues/3012) in designating a specific GPU for model execution via code. Consequently, to employ a particular CUDA device for your model's operations, it's necessary to manually adjust your environment variable `CUDA_VISIBLE_DEVICES`. For instance, setting `export CUDA_VISIBLE_DEVICES=1,2` will explicitly define the CUDA devices available for use.
-
-# Resources
-The archiecture and methodology of SN1 is complex, and as such we have created a comprehensive resource to outline our design. Furthermore, we have strict requirements for how miners should interact with the network. Below are the currently available resources for additional information: 
-
-1. [SN1 architecture details](docs/SN1_validation.md)
-2. [StreamMiner requirements](docs/stream_miner_template.md)

assets/miner.md

@@ -0,0 +1,59 @@
+# **Miners**
+
+# ⚠️ **DISCLAIMER** ⚠️ **DO NOT RUN THIS MINER ON MAINNET!**
+
+> **The openai miner provided in this repo is _not intended_ to be run on mainnet!**
+> 
+> **If you run the base miner on mainnet, you will not earn anything!**
+> It is provided as an example to help you build your own custom mining operation!
+> 
+## Compute Requirements
+
+| Resource      | Requirement       |
+|---------------|-------------------|
+| **VRAM**      | None              |
+| **vCPU**      | 8 vCPU            |
+| **RAM**       | 8 GB              |
+| **Storage**   | 80 GB             |
+
+## Installation
+
+Clone this repository and run the [install.sh](./install.sh) script.
+
+```bash
+git clone https://github.com/opentensor/prompting.git
+cd prompting
+bash install.sh
+```
+
+## Configuration
+⚠️ **Reminder! Do not run this miner on main!** ⚠️
+Before running a miner, you will need to create a .env.miner environment file. It is necessary for you to provide the following 
+
+```text
+NETUID= #[1, 61, 102]
+SUBTENSOR_NETWORK= #The network name [test, main, local]
+SUBTENSOR_CHAIN_ENDPOINT= #The chain endpoint [test if running on test, main if running on main, custom endpoint if running on local] 
+WALLET_NAME= #Name of your wallet(coldkey) 
+HOTKEY= #Name of your hotkey associated with above wallet
+AXON_PORT= #Number of the open tcp port
+OPENAI_API_KEY= #The openai key that you would like to mine with
+```
+## Testnet - RECOMMENDED
+We highly recommend that you run your miners on testnet before deploying on main. This is give you an opportunity to debug your systems, and ensure that you will not lose valuable immunity time. The SN1 testnet is **netuid 61**.
+
+In order to run on testnet, you will need to go through the same hotkey registration proceure as on main, but using **testtao**. You will need to ask for some in the community discord if you do not have any.
+
+Then, simply set test=True in your .env file and execute all other steps as before.
+
+Then post in the Subnet 1 channel on discord so we can activate a validator for your miner to respond to.
+
+You can use wandb to see how successful your miner would be on mainnet, an example notebook is pinned in the channel.
+
+## Running
+
+After creating the above environment file, run 
+
+```bash
+pm2 start "poetry run python neurons/miners/openai/miner.py"
+```

assets/validator.md

@@ -0,0 +1,50 @@
+# **VALIDATORS**
+
+## Compute Requirements
+
+| Resource      | Requirement       |
+|---------------|-------------------|
+| **VRAM**      | 62 GB             |
+| **vCPU**      | 24 vCPU           |
+| **RAM**       | 60 GB             |
+| **Storage**   | 150 GB            |
+
+## Installation
+
+Clone this repository and run the [install.sh](./install.sh) script.
+
+```bash
+git clone https://github.com/opentensor/prompting.git
+cd prompting
+bash install.sh
+```
+
+You will also need to log into huggingface and accept the License Agreement for the LMSYS-Chat-1M dataset: https://huggingface.co/datasets/lmsys/lmsys-chat-1m :
+```shell
+huggingface-cli login
+```
+
+## Configuration
+
+Before running a validator, you will need to create a .env.validator environment file. It is necessary for you to provide the following 
+
+```text
+NETUID= #[1, 61, 102]
+SUBTENSOR_NETWORK= #The network name [test, main, local]
+SUBTENSOR_CHAIN_ENDPOINT= #The chain endpoint [test if running on test, main if running on main, custom endpoint if running on local] 
+WALLET_NAME= #Name of your wallet(coldkey) 
+HOTKEY= #Name of your hotkey associated with above wallet
+AXON_PORT= #Number of the open tcp port
+HF_TOKEN= #Access token from Hugging Face where you accepted the License Agreement for lmsys
+```
+
+## Running
+
+After creating the above environment file, run 
+
+```bash
+bash run.sh
+```
+It will spawn 2 pm2 processes, one to run the validator and one to autoupdate. 
+
+> Note: this is not an end solution, major releases or changes in requirements will still require you to manually restart the processes. Regularly monitor the health of your validator to ensure optimal performance.