Skip to content

Commit

Permalink
Bring rao branch to up to date with staging (#2241)
Browse files Browse the repository at this point in the history
* chore:lint

* Fix LA test

* Skip faucet test

* Skip faucet test

* Prevent E2E tests run on drafts and non_prs.

* Prevent E2E tests run on drafts and non_prs.

* init commit

* Add child singular cli done

* Set Child Hotkey

* Set children multiple command.

* Remove merge conflict artifacts

* Add GetChildrenHotkeysCommand

* Add RevokeChildHotkey

* lint

* Revoke children multiple command

* lint

* CLI revoke multiple

* lint

* Imports

* lint

* lint

* imports in init

* fix e2e ci

* fix e2e ci - update

* update if statement

* update if statement

* Fix Faucet and fastblocks

* Refactor to follow new pattern

* U64 for all proportion values

* lint

* debug

* Clean up a bit.

* Clean up a bit.

* u16 to u64

* un-refactor

* APY + other updates

* lint

* lint

* Extract Method, and params

* Pr comments

* feat: enhances dendrite error messages

* add timeout and change pattern

* feat: enhances dendrite error messages

* Merge pull request #2012 from opentensor/feature/gus/liquid-alpha-params

Liquid alpha

* ci: adds release steps

* ci: replace CircleCi with GH workflow

* Re-adds check_coldkey_swap command (#2126)

* Inital commit: Readds check-coldkey-swap

* Adds check_coldkey_swap command

* ci: github workflow release

* Bumps version to 7.3.0

* Fixes leaked semaphores (#2125)

Fixes leaked semaphores in POW solving by correctly handling the shutdown/joining of multiprocessing objects.

* Adds check_pr_status

* Make check_pr_status.sh executable

* Fixes ruff checks

* Updated e2e yaml file

* ci: add branch regex

* ci: add hotfix regex

* chore: update changelog

* update e2e test

* ruff

* ci: rename release workflow file

* ci: adding release.yml

* chore: add doc string

* Merge master into staging

* fix: coldkeypub usage instead of coldkey for arbitration_stats

* test: fix mocksubtensor query previous blocks

* Removes extra no_prompts in commands

* Adds timeout for e2e tests

* fix: updates test_axon verify body async tests

* Adds e2e for metagraph command + subtensor.metagraph

* Adds E2E for wallet creations

* Added doc strings and temp variable for testing

* Temp: Added capsys before each command

* Changed pattern to splitting output before matching regex

* ci: removes deprecated release scripts and adds docker release workflow

* Adds test for wallet regenerations + fixes input bug for regen hotkey

* Bumps setuptools~=70.0.0

* bump pysub to 1.7.9+

* bump scale-codec to 1.2.10

* bump scale-codec to 1.2.11

* Removed monkey-patch of py-scale-codec which hasn't been necessary since v1.2.9. Removed comments regarding verification of the package upgrades because of this patch.

* ruff

* fix streaming synapse regression

* Fixed type annotations and doc strings

* Adds docstring to test_wallet_regen test

* Fix naming convention of swap hotkey test

* set streaming response headers

* Removes wait_epoch

* Merge pull request #2156 from opentensor/feat/ledger-integration

[Ledger Integration] [Feature] bump pysub to 1.7.9+

* Revert "fix streaming synapse regression"

* Merge pull request #2159 from backend-developers-ltd/fix_streaming_synapse

fix streaming synapse regression

* ci: auto assigns cortex to opened PRs

* no op

* tests/e2e_tests/utils.py: logging and epoch logic fix

- Log commands as executed, so that CI errors can be pinpointed to their
  originating command.
- Dropped wait_epoch() as it is not used.
- Improved wait_interval(), explanation below:

For tempo T, the epoch interval is T+1, and the offset depends on the
netuid. See subtensor/src/block_step.rs, blocks_until_next_epoch(), with
the comment stating:

https://github.com/opentensor/subtensor/blob/1332d077ea73bc7bf40f551c7f1adea3370df2bd/pallets/subtensor/src/block_step.rs#L33
"Networks run their epoch when (block_number + netuid + 1 ) % (tempo + 1) = 0"

This comment from the subtensor code is not correct, the algorithm
actually tests:

(block_number + netuid + 1 ) % (tempo + 1) == tempo

This is because what is tested, is whether

    blocks_until_next_epoch() == 0

and defining:

    A = (block_number + netuid + 1)%(tempo + 1)

we can say that, looking at https://github.com/opentensor/subtensor/blob/1332d077ea73bc7bf40f551c7f1adea3370df2bd/pallets/subtensor/src/block_step.rs#L47:

    blocks_until_next_epoch() = tempo - A

And so it is easy to see that we need A == tempo to run the epoch.

Then, to find the last epoch, calculating mod M = tempo+1:

    (block_number + netuid + 1)%M = tempo%M
    (block_number + netuid + 1)%M = (-1)%M
    (block_number + netuid + 1)%M + 1 = 0

So the last epoch is at:

    last_epoch = block_number - 1 - (block_number + netuid + 1) % interval

It is easily seen that this is in the range:

    [block_number-interval, block_number-1]

And so the current block, if it were epoch, is not seen as the last
epoch.

The next epoch is then:

    last_epoch + interval

Which is in the range:

    [block_number, block_number+interval-1]

And so if the current block is epoch, wait_for_interval() will return
immediately.

It is suspected that CI tests fail because the wait_epoch() waits for
the wrong block. And if this is not an issue at the present time, it may
become an issue later.

Note that difficulty adjustments follow a different schedule.

In the reworked function, blocks passing while waiting are reported
after passing 10 blocks, not when exactly hitting block N*10. This
ensures that logging is seen, even if the N*10 block passes during
time.sleep(1).

TODO: check if time.sleep() should be replaced by asyncio.sleep(),
as time.sleep() halts the entire process.

* e2e_tests/multistep/test_axon.py: replace magic constant 1 by netuid

* e2e_tests/multistep/test_dendrite.py: replace magic constant 1 by netuid

* e2e_tests/multistep/test_emissions.py: replace magic constant 1 by netuid

* e2e_tests/multistep/test_incentive.py: replace magic constant 1 by netuid

* e2e_tests: improvements

- Replace wait_for_interval(360) calls with wait_epoch(), reducing the
  number of magic constants and preparing for optionally changing the
  tempo used in tests.
- Automatically determine tempo from on-chain data.
- Make wait functions async, eliminating time.sleep() which basically
  halts everything, which is not beneficial to other coroutines.

* replaced bittensor.btlogging.error with bittensor.logging.error as bittensor.btlogging is a module, and thus does not contain the `error` function.

* First terminates the worker process before joining.

* Fixes mock of correct object.

* Ruff.

* Fix typo

Co-authored-by: gus-opentensor <158077861+gus-opentensor@users.noreply.github.com>

* ci: update reviews

* Init commit

* 2nd commit: enhancing e2e suite

* Reverts conftest env

* Clean up

* Fix ruff

* Skips emissions, fixes senate vote assertion, fixes exec call in swap_hotkey

* Adds check for participation of a neuron

* Adds updated type in timeouts dendrite

* Bump black from 23.7.0 to 24.3.0 in /requirements

Bumps [black](https://github.com/psf/black) from 23.7.0 to 24.3.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](psf/black@23.7.0...24.3.0)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>

* btlogging/loggingmachine.py: Fix bw compat API.

- add stacklevel=2 so that the filename of the originating call is
  logged (requires Python >= 3.8)
- move *args before prefix and suffix kwargs so regular logging calls
  using args, such as in retry, still work as intended

To elaborate on the last point, retry does the following:

    logger.warning('%s, retrying in %s seconds...', e, _delay)

and the args e and _delay would land on kwargs prefix and suffix without
this patch.

Any code relying on prefix and suffix also being positional arguments,
is now broken.

Note that logger.exception() calls through to logger.error() without
compensating for this additional internal call when determining the
calling file, on CPython < 3.11. This code works around this issue.

* btlogging/loggingmachine.py: Improve bw compat API.

In case prefix and suffix are not used, messages are still prefixed and
suffixed with " - ".

This patch addresses that and only injects separators between non-empty
strings.

* Add unit test for logging output

Add a unit test for:
- correct filename in loglines
- prefix= and suffix= kwargs
- format string functionality

* test: subnet list e2e

* Ensures that each element of _concat_msg is a string

* updates to use local_chain fixture

* fixes test

* test: adds wallet list command e2e test

* rm local_chain

* update path

* Fixes tests depending on explicit line numbers

* Fixes ruff

* Adds workflow for multiple bittensor version tests

* Init: changes name

* Changes name

* Bumps ansible and certifi based on security vulnerabilities

* Fixes version number

* fix Synapse performance regression

* Update child hotkeys with new subtensor calls and remove ChildInfo object.

* ruff

* update

* update

* ruff

* mypy error

* Improve child hotkeys to not double prompt and display table in appropriate places.

* lint

* # Add documentation for retrieve_children and render_table methods

Added detailed docstrings to the retrieve_children and render_table methods in stake.py to specify their purpose, parameters, and return values. This enhances code readability and assists other developers in understanding the methods' functionalities better.

* Refactor normalization to use floor function and handle excess.

In staking.py, introduced the floor function for normalized proportions and added logic to handle excess sums. Updated formatting.py to use floor for float-to-u64 conversion, ensuring values stay within bounds.

* fix DelegateInfoLite name

* add ruff to dev.txt

* ruff formatter

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Gus <gus@opentensor.dev>
Co-authored-by: opendansor <daniel@opentensor.dev>
Co-authored-by: ibraheem-opentensor <165814940+ibraheem-opentensor@users.noreply.github.com>
Co-authored-by: Maciej Urbański <122983254+mjurbanski-reef@users.noreply.github.com>
Co-authored-by: Samuel Dare <sam@tunnelvisionlabs.xyz>
Co-authored-by: open-junius <zhou@opentensor.dev>
Co-authored-by: open-junius <165236073+open-junius@users.noreply.github.com>
Co-authored-by: gus-opentensor <158077861+gus-opentensor@users.noreply.github.com>
Co-authored-by: ibraheem-opentensor <ibraheem@opentensor.dev>
Co-authored-by: Benjamin Himes <37844818+thewhaleking@users.noreply.github.com>
Co-authored-by: Benjamin Himes <benhimes@opentensor.dev>
Co-authored-by: Roman <167799377+RomanCh-OT@users.noreply.github.com>
Co-authored-by: Rapiiidooo <vincent.ljeune@gmail.com>
Co-authored-by: Tamerlan <tamerlan.abilov95@gmail.com>
Co-authored-by: Cameron Fairchild <cameron@opentensor.dev>
Co-authored-by: Maciej Urbanski <maciej.rooter.urbanski@reef.pl>
Co-authored-by: μ <maarten@coldint.io>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Loading branch information
19 people committed Aug 16, 2024
1 parent e7ab819 commit 4016a10
Show file tree
Hide file tree
Showing 125 changed files with 11,294 additions and 4,408 deletions.
26 changes: 26 additions & 0 deletions .circleci/check_pr_status.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/bin/bash

# Extract the repository owner
REPO_OWNER=$(echo $CIRCLE_PULL_REQUEST | awk -F'/' '{print $(NF-3)}')

# Extract the repository name
REPO_NAME=$(echo $CIRCLE_PULL_REQUEST | awk -F'/' '{print $(NF-2)}')

# Extract the pull request number
PR_NUMBER=$(echo $CIRCLE_PULL_REQUEST | awk -F'/' '{print $NF}')


PR_DETAILS=$(curl -s \
"https://github.com/gitapi/repos/$REPO_OWNER/$REPO_NAME/pulls/$PR_NUMBER")


IS_DRAFT=$(echo "$PR_DETAILS" | jq -r .draft)
echo $IS_DRAFT

if [ "$IS_DRAFT" == "true" ]; then
echo "This PR is a draft. Skipping the workflow."
exit 1
else
echo "This PR is not a draft. Proceeding with the workflow."
exit 0
fi
78 changes: 48 additions & 30 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,21 @@ version: 2.1
orbs:
python: circleci/python@2.1.1
python-lib: dialogue/python-lib@0.1.55
# coveralls: coveralls/coveralls@1.0.6

jobs:
black:
check-if-pr-is-draft:
docker:
- image: cimg/python:3.10
steps:
- checkout
- run:
name: Install jq
command: sudo apt-get update && sudo apt-get install -y jq
- run:
name: Check if PR is a draft
command: .circleci/check_pr_status.sh

ruff:
resource_class: small
parameters:
python-version:
Expand All @@ -18,29 +29,29 @@ jobs:
- checkout

- restore_cache:
name: Restore cached black venv
name: Restore cached ruff venv
keys:
- v2-pypi-py-black-<< parameters.python-version >>
- v2-pypi-py-ruff-<< parameters.python-version >>

- run:
name: Update & Activate black venv
name: Update & Activate ruff venv
command: |
python -m venv env/
. env/bin/activate
python -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
pip install black==23.7.0
pip install ruff -c requirements/dev.txt
- save_cache:
name: Save cached black venv
name: Save cached ruff venv
paths:
- "env/"
key: v2-pypi-py-black-<< parameters.python-version >>
- ".venv/"
key: v2-pypi-py-ruff-<< parameters.python-version >>

- run:
name: Black format check
name: Ruff format check
command: |
. env/bin/activate
python -m black --exclude '(env|venv|.eggs)' --check .
. .venv/bin/activate
ruff format --diff .
check_compatibility:
parameters:
Expand Down Expand Up @@ -85,8 +96,8 @@ jobs:
- run:
name: Update & Activate venv
command: |
python -m venv env/
. env/bin/activate
python -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install '.[dev]'
Expand All @@ -99,20 +110,20 @@ jobs:
- run:
name: Install Bittensor
command: |
. env/bin/activate
. .venv/bin/activate
pip install -e '.[dev]'
- run:
name: Instantiate Mock Wallet
command: |
. env/bin/activate
. .venv/bin/activate
./scripts/create_wallet.sh
# TODO: Update test durations on different runs
- run:
name: Unit Tests
no_output_timeout: 20m
command: |
. env/bin/activate
. .venv/bin/activate
export PYTHONUNBUFFERED=1
pytest -n2 --reruns 3 --durations=0 --verbose --junitxml=test-results/unit_tests.xml \
--cov=. --cov-append --cov-config .coveragerc \
Expand All @@ -122,8 +133,9 @@ jobs:
- run:
name: Integration Tests
no_output_timeout: 30m
command: |
. env/bin/activate
. .venv/bin/activate
export PYTHONUNBUFFERED=1
pytest -n2 --reruns 3 --reruns-delay 15 --durations=0 --verbose --junitxml=test-results/integration_tests.xml \
--cov=. --cov-append --cov-config .coveragerc \
Expand All @@ -136,14 +148,15 @@ jobs:
- store_artifacts:
path: test-results


#- when:
#condition:
#equal: ["3.10.5", << parameters.python-version >> ]
#steps:
#- run:
#name: Upload Coverage
#command: |
#. env/bin/activate && coveralls
#. .venv/bin/activate && coveralls
#env:
#CI_NAME: circleci
#CI_BUILD_NUMBER: $CIRCLE_BUILD_NUM
Expand All @@ -152,6 +165,7 @@ jobs:
#CI_JOB_ID: $CIRCLE_NODE_INDEX
#COVERALLS_PARALLEL: true


lint-and-type-check:
resource_class: medium
parallelism: 2
Expand All @@ -173,8 +187,8 @@ jobs:
- run:
name: Update & Activate venv
command: |
python -m venv env/
. env/bin/activate
python -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install '.[dev]'
pip install flake8
Expand All @@ -188,19 +202,19 @@ jobs:
- run:
name: Install Bittensor
command: |
. env/bin/activate
. .venv/bin/activate
pip install -e '.[dev]'
- run:
name: Lint with flake8
command: |
. env/bin/activate
. .venv/bin/activate
python -m flake8 bittensor/ --count
- run:
name: Type check with mypy
command: |
. env/bin/activate
. .venv/bin/activate
python -m mypy --ignore-missing-imports bittensor/
unit-tests-all-python-versions:
Expand Down Expand Up @@ -274,7 +288,6 @@ jobs:
command: |
./scripts/release/release.sh --github-token ${GH_API_ACCESS_TOKEN}
workflows:
compatibility_checks:
jobs:
Expand All @@ -290,12 +303,17 @@ workflows:

pr-requirements:
jobs:
- black:
- check-if-pr-is-draft
- ruff:
python-version: "3.9.13"
requires:
- check-if-pr-is-draft
- build-and-test:
matrix:
parameters:
python-version: ["3.9.13", "3.10.6", "3.11.4"]
requires:
- check-if-pr-is-draft
- unit-tests-all-python-versions:
requires:
- build-and-test
Expand All @@ -304,7 +322,7 @@ workflows:
parameters:
python-version: ["3.9.13", "3.10.6", "3.11.4"]
requires:
- build-and-test
- check-if-pr-is-draft
#- coveralls:
#requires:
#- build-and-test
Expand Down
51 changes: 51 additions & 0 deletions .github/workflows/docker_release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
name: Build and Push Docker Image
# https://github.com/sigstore/cosign
on:
workflow_dispatch:
inputs:
tag:
description: 'Docker image tag'
required: true
default: 'latest'

jobs:
build-and-push:
runs-on: ubuntu-latest

permissions:
contents: read
id-token: write

steps:
- name: Check out code
uses: actions/checkout@v3

- name: Install cosign
uses: sigstore/cosign-installer@v3

- name: Log in to Docker Hub
uses: docker/login-action@v2
with:
registry: docker.io
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

- name: Build and push Docker image
uses: docker/build-push-action@v4
with:
context: .
push: true
tags: |
opentensorfdn/bittensor:${{ github.event.inputs.tag }}
opentensorfdn/bittensor:latest
provenance: false

- name: Sign the images with GitHub OIDC Token
env:
DIGEST: ${{ steps.build.outputs.digest }}
TAGS: ${{ steps.build.outputs.tags }}
run: |
echo "${TAGS}" | xargs -I {} cosign sign --yes {}@${DIGEST}
Loading

0 comments on commit 4016a10

Please sign in to comment.