Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chain bloat #541

Closed
wants to merge 68 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
0281700
initial commit
Jun 15, 2024
c7a114b
initial commit
Jun 15, 2024
5ccb2bc
add comments
Jun 16, 2024
46b5522
add comments
Jun 16, 2024
2ca9071
add comments
Jun 16, 2024
7ff9b11
clean coinbase
Jun 16, 2024
d5680c3
clean chain bloat further
Jun 16, 2024
5d240fa
clean chain bloat further
Jun 16, 2024
8aca0bb
clean chain bloat further
Jun 16, 2024
aad5478
clean build no warnings
Jun 16, 2024
9bc4dca
fix unsafe math
open-junius Jun 17, 2024
45ea7f6
use safe math
open-junius Jun 17, 2024
d47590a
fix more unsafe math
open-junius Jun 17, 2024
231cc88
fix all unsafe math
open-junius Jun 17, 2024
70ec3c6
add todo
Jun 17, 2024
5b8141d
add todo
Jun 17, 2024
1062fb6
testable now
open-junius Jun 17, 2024
7fa36e4
tests
Jun 17, 2024
9d4de66
merge remote
Jun 17, 2024
15cc368
add child tests
Jun 17, 2024
ea9aa97
add children tests
Jun 18, 2024
3e97db1
fix compile errors
Jun 18, 2024
f1f1dc0
sd: children tests plus getters
Jun 18, 2024
2b040c8
chore: remove comment
Jun 18, 2024
083b5cd
add comments
Jun 18, 2024
7fb0cf6
Merge branch 'chain_bloat' of https://github.com/opentensor/subtensor…
Jun 19, 2024
ca41be6
add coinbase test
Jun 19, 2024
f6e2b8a
add coinbase test
Jun 19, 2024
784a28a
fix tests add new ones for chain bloat
Jun 19, 2024
d673a57
add owner cut
Jun 19, 2024
4802671
add safe math
Jun 19, 2024
25fae32
feat: revoke childkey , childkey extrinsics , tests , intial benchmar…
Jun 19, 2024
46eea19
feat: lints , tests , gettter , setters
Jun 20, 2024
deb4908
feat: benchmarking set_hotkey_emission_tempo, clippy
Jun 22, 2024
26908c3
feat: set children multiple + tests
Jun 23, 2024
b33f55a
feat: revoke multiple children + tests
Jun 23, 2024
03b1131
feat: started test_comprehensive_coinbase
Jun 23, 2024
576b20f
feat: check for self loops via dfs in children
Jun 24, 2024
7d26751
feat: full coinbase test
Jun 25, 2024
0d371bc
feat: caps , cap tests
Jun 26, 2024
b480518
feat: child_info rpc
Jun 27, 2024
2c5f6e6
chore: fix rpcs , tests
Jun 27, 2024
2368e61
chore: lint
Jun 27, 2024
6dceea5
feat: add Parents to child info
Jun 27, 2024
d85d11b
fix: sat maths
Jun 28, 2024
09ebd25
feat: enforce proportions to 1
Jul 1, 2024
71ca55e
add do_set_delegate_takes
open-junius Jul 1, 2024
8a56833
fix conflict
open-junius Jul 1, 2024
fb8231a
fix fmt
open-junius Jul 1, 2024
8535dcf
optimize the impl
open-junius Jul 1, 2024
e0f2c2b
fix clippy
open-junius Jul 1, 2024
17889ec
fix unit test
open-junius Jul 1, 2024
2de784a
Merge branch 'main' into chain_bloat
distributedstatemachine Jul 1, 2024
42763ae
fix compilation error
open-junius Jul 2, 2024
d5c5a77
fix test_get_children_info
open-junius Jul 2, 2024
ae628b8
add duplicate child check
open-junius Jul 2, 2024
bd0acfd
merge with main
open-junius Jul 5, 2024
feaa4b2
merge with main
open-junius Jul 5, 2024
122c021
fix clippy
open-junius Jul 5, 2024
85fafe6
fix clippy
open-junius Jul 5, 2024
b761d37
fix clippy
open-junius Jul 5, 2024
7641f15
fix unit test
open-junius Jul 5, 2024
feb1a42
fix justfile
open-junius Jul 5, 2024
7d68716
rebase code
open-junius Jul 10, 2024
8aaac52
trigger ci
open-junius Jul 11, 2024
2a94385
merge with main
open-junius Jul 16, 2024
aeeb039
remove commented code
open-junius Jul 16, 2024
3430b08
fix unit test
open-junius Jul 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
183 changes: 107 additions & 76 deletions pallets/subtensor/src/coinbase.rs
Original file line number Diff line number Diff line change
@@ -1,35 +1,34 @@
use super::*;
use frame_support::IterableStorageDoubleMap;
// use sp_runtime::Saturating;
use substrate_fixed::types::{I64F64, I96F32};

impl<T: Config> Pallet<T> {

/// The `coinbase` function performs a four-part emission distribution process involving
/// subnets, epochs, hotkeys, and nominators.
// It is divided into several steps, each handling a specific part of the distribution:

// Step 1: Compute the block-wise emission for each subnet.
// This involves calculating how much (TAO) should be emitted into each subnet using the
// root epoch function.
// This involves calculating how much (TAO) should be emitted into each subnet using the
// root epoch function.

// Step 2: Accumulate the subnet block emission.
// After calculating the block-wise emission, these values are accumulated to keep track
// of how much each subnet should emit before the next distribution phase. This accumulation
// is a running total that gets updated each block.

// Step 3: Distribute the accumulated emissions through epochs.
// Subnets periodically distribute their accumulated emissions to hotkeys (active validators/miners)
// in the network on a `tempo` --- the time between epochs. This step runs Yuma consensus to
// in the network on a `tempo` --- the time between epochs. This step runs Yuma consensus to
// determine how emissions are split among hotkeys based on their contributions and roles.
// The accumulation of hotkey emissions is done through the `accumulate_hotkey_emission` function.
// The function splits the rewards for a hotkey amongst itself and its `parents`. The parents are
// the hotkeys that are delegating their stake to the hotkey.
// The function splits the rewards for a hotkey amongst itself and its `parents`. The parents are
// the hotkeys that are delegating their stake to the hotkey.

// Step 4: Further distribute emissions from hotkeys to nominators.
// Finally, the emissions received by hotkeys are further distributed to their nominators,
// who are stakeholders that support the hotkeys.
// who are stakeholders that support the hotkeys.
pub fn run_coinbase() {

// --- 0. Get current block.
let current_block: u64 = Self::get_current_block_as_u64();

Expand All @@ -38,61 +37,66 @@ impl<T: Config> Pallet<T> {

// --- 2. Run the root epoch function which computes the block emission for each subnet.
// coinbase --> root() --> subnet_block_emission
match Self::root_epoch( current_block ) { Ok(_) => (), Err(e) => {log::trace!("Error while running root epoch: {:?}", e);}}
match Self::root_epoch(current_block) {
Ok(_) => (),
Err(e) => {
log::trace!("Error while running root epoch: {:?}", e);
}
}

// --- 3. Drain the subnet block emission and accumulate it as subnet emission, which increases until the tempo is reached in #4.
// subnet_blockwise_emission -> subnet_pending_emission
for netuid in subnets.clone().iter() {

// --- 3.1 Get the network's block-wise emission amount.
// This value is newly minted TAO which has not reached staking accounts yet.
let subnet_blockwise_emission: u64 = EmissionValues::<T>::get( *netuid );
let subnet_blockwise_emission: u64 = EmissionValues::<T>::get(*netuid);

// --- 3.2 Accumulate the subnet emission on the subnet.
PendingEmission::<T>::mutate( *netuid, |subnet_emission| *subnet_emission += subnet_blockwise_emission);
PendingEmission::<T>::mutate(*netuid, |subnet_emission| {
*subnet_emission = subnet_emission.saturating_add(subnet_blockwise_emission);
});
}

// --- 4. Drain the accumulated subnet emissions, pass them through the epoch().
// --- 4. Drain the accumulated subnet emissions, pass them through the epoch().
// Before accumulating on the hotkeys the function redistributes the emission towards hotkey parents.
// subnet_emission --> epoch() --> hotkey_emission --> (hotkey + parent hotkeys)
for netuid in subnets.clone().iter() {

// 4.1 Check to see if the subnet should run its epoch.
if Self::should_run_epoch( *netuid, current_block ) {

if Self::should_run_epoch(*netuid, current_block) {
// 4.2 Drain the subnet emission.
let subnet_emission: u64 = PendingEmission::<T>::get( *netuid );
PendingEmission::<T>::insert( *netuid, 0 );
let subnet_emission: u64 = PendingEmission::<T>::get(*netuid);
PendingEmission::<T>::insert(*netuid, 0);

// 4.3 Pass emission through epoch() --> hotkey emission.
let hotkey_emission: Vec<(T::AccountId, u64, u64)> = Self::epoch( *netuid, subnet_emission );
let hotkey_emission: Vec<(T::AccountId, u64, u64)> =
Self::epoch(*netuid, subnet_emission);

// 4.3 Drain the subnet emission through the epoch()
for (hotkey, mining_emission, validator_emission) in hotkey_emission {

// 4.4 Accumulate the emission on the hotkey and parent hotkeys.
Self::accumulate_hotkey_emission( &hotkey, *netuid, mining_emission + validator_emission );
Self::accumulate_hotkey_emission(
&hotkey,
*netuid,
mining_emission + validator_emission,
distributedstatemachine marked this conversation as resolved.
Show resolved Hide resolved
);
}
}
}

// --- 5. Drain the accumulated hotkey emissions through to the nominators.
// --- 5. Drain the accumulated hotkey emissions through to the nominators.
// The hotkey takes a proportion of the emission, the remainder is drained through to the nominators.
// We keep track of the last stake increase event for accounting purposes.
// hotkeys --> nominators.
for (index, ( hotkey, hotkey_emission )) in PendingdHotkeyEmission::<T>::iter().enumerate() {

for (index, (hotkey, hotkey_emission)) in PendingdHotkeyEmission::<T>::iter().enumerate() {
// --- 5.1 Check if we should drain the hotkey emission on this block.
if Self::should_drain_hotkey( index as u64 , current_block ) {

if Self::should_drain_hotkey(index as u64, current_block) {
// --- 5.2 Drain the hotkey emission and distribute it to nominators.
Self::drain_hotkey_emission( &hotkey, hotkey_emission, current_block );
Self::drain_hotkey_emission(&hotkey, hotkey_emission, current_block);

// --- 5.3 Increase total issuance
TotalIssuance::<T>::put( TotalIssuance::<T>::get().saturating_add( hotkey_emission ) );
TotalIssuance::<T>::put(TotalIssuance::<T>::get().saturating_add(hotkey_emission));
}
}

}

/// Accumulates the mining and validator emissions on a hotkey and distributes the validator emission among its parents.
Expand All @@ -108,51 +112,59 @@ impl<T: Config> Pallet<T> {
/// * `mining_emission` - The amount of mining emission allocated to the hotkey.
/// * `validator_emission` - The amount of validator emission allocated to the hotkey.
///
pub fn accumulate_hotkey_emission( hotkey: &T::AccountId, netuid: u16, emission: u64 ) {

pub fn accumulate_hotkey_emission(hotkey: &T::AccountId, netuid: u16, emission: u64) {
// --- 1. First, calculate the hotkey's share of the emission.
let take_proportion: I64F64 = I64F64::from_num( Delegates::<T>::get( hotkey ) ) / I64F64::from_num( u16::MAX );
let hotkey_take: u64 = ( take_proportion * I64F64::from_num( emission ) ).to_num::<u64>();
let take_proportion: I64F64 = I64F64::from_num(Delegates::<T>::get(hotkey))
.saturating_div(I64F64::from_num(u16::MAX));
let hotkey_take: u64 = (take_proportion * I64F64::from_num(emission)).to_num::<u64>();

// --- 2. Compute the remaining emission after the hotkey's share is deducted.
let emission_minus_take: u64 = emission - hotkey_take;
let emission_minus_take: u64 = emission.saturating_sub(hotkey_take);

// --- 3. Track the remaining emission for accounting purposes.
let mut remaining_emission: u64 = emission_minus_take;
// --- 4. Calculate the total stake of the hotkey, adjusted by the stakes of parents and children.

// --- 4. Calculate the total stake of the hotkey, adjusted by the stakes of parents and children.
// Parents contribute to the stake, while children reduce it.
// If this value is zero, no distribution to anyone is necessary.
let total_hotkey_stake: u64 = Self::get_stake_with_children_and_parents( hotkey, netuid );
if total_hotkey_stake != 0 {

let total_hotkey_stake: u64 = Self::get_stake_with_children_and_parents(hotkey, netuid);
if total_hotkey_stake != 0 {
// --- 5. If the total stake is not zero, iterate over each parent to determine their contribution to the hotkey's stake,
// and calculate their share of the emission accordingly.
for (proportion, parent) in ParentKeys::<T>::get( hotkey, netuid ) {

for (proportion, parent) in ParentKeys::<T>::get(hotkey, netuid) {
// --- 5.1 Retrieve the parent's stake. This is the raw stake value including nominators.
let parent_stake: u64 = Self::get_total_stake_for_hotkey( &parent );
let parent_stake: u64 = Self::get_total_stake_for_hotkey(&parent);

// --- 5.2 Calculate the portion of the hotkey's total stake contributed by this parent.
// Then, determine the parent's share of the remaining emission.
let stake_from_parent: I96F32 = I96F32::from_num( parent_stake ) * ( I96F32::from_num( proportion ) / I96F32::from_num( u64::MAX ) );
let proportion_from_parent: I96F32 = stake_from_parent / I96F32::from_num( total_hotkey_stake );
let parent_emission_take: u64 = ( proportion_from_parent * I96F32::from_num( emission_minus_take ) ).to_num::<u64>();
let stake_from_parent: I96F32 = I96F32::from_num(parent_stake).saturating_mul(
I96F32::from_num(proportion).saturating_div(I96F32::from_num(u64::MAX)),
);
let proportion_from_parent: I96F32 =
stake_from_parent.saturating_div(I96F32::from_num(total_hotkey_stake));
let parent_emission_take: u64 = proportion_from_parent
.saturating_mul(I96F32::from_num(emission_minus_take))
.to_num::<u64>();

// --- 5.5. Accumulate emissions for the parent hotkey.
PendingdHotkeyEmission::<T>::mutate( parent, |parent_accumulated| *parent_accumulated += parent_emission_take );
PendingdHotkeyEmission::<T>::mutate(parent, |parent_accumulated| {
*parent_accumulated = parent_accumulated.saturating_add(parent_emission_take)
});

// --- 5.6. Subtract the parent's share from the remaining emission for this hotkey.
remaining_emission -= parent_emission_take;
remaining_emission = remaining_emission.saturating_sub(parent_emission_take);
}
}

// --- 6. Add the remaining emission plus the hotkey's initial take to the pending emission for this hotkey.
PendingdHotkeyEmission::<T>::mutate( hotkey, |hotkey_accumulated| *hotkey_accumulated += remaining_emission + hotkey_take );
PendingdHotkeyEmission::<T>::mutate(hotkey, |hotkey_accumulated| {
*hotkey_accumulated =
hotkey_accumulated.saturating_add(remaining_emission.saturating_add(hotkey_take))
});
}

//. --- 4. Drains the accumulated hotkey emission through to the nominators. The hotkey takes a proportion of the emission.
/// The remainder is drained through to the nominators keeping track of the last stake increase event to ensure that the hotkey does not
/// The remainder is drained through to the nominators keeping track of the last stake increase event to ensure that the hotkey does not
/// gain more emission than it's stake since the last drain.
/// hotkeys --> nominators.
///
Expand All @@ -163,49 +175,63 @@ impl<T: Config> Pallet<T> {
/// 7. Finally, the hotkey's own take and any undistributed emissions are added to the hotkey's total stake.
///
/// This function ensures that emissions are fairly distributed according to stake proportions and delegation agreements, and it updates the necessary records to reflect these changes.
pub fn drain_hotkey_emission( hotkey: &T::AccountId, emission: u64, block_number: u64 ) {

pub fn drain_hotkey_emission(hotkey: &T::AccountId, emission: u64, block_number: u64) {
// --- 1.0 Drain the hotkey emission.
PendingdHotkeyEmission::<T>::insert( hotkey, 0 );
PendingdHotkeyEmission::<T>::insert(hotkey, 0);

// --- 1.1 Retrieve the last time this hotkey's emissions were drained.
let last_hotkey_emission_drain: u64 = LastHotkeyEmissionDrain::<T>::get( hotkey );
let last_hotkey_emission_drain: u64 = LastHotkeyEmissionDrain::<T>::get(hotkey);

// --- 1.2 Update the block value to the current block number.
LastHotkeyEmissionDrain::<T>::insert( hotkey, block_number );
LastHotkeyEmissionDrain::<T>::insert(hotkey, block_number);

// --- 1.3 Retrieve the total stake for the hotkey from all nominations.
let total_hotkey_stake: u64 = Self::get_total_stake_for_hotkey( hotkey );
let total_hotkey_stake: u64 = Self::get_total_stake_for_hotkey(hotkey);

// --- 1.4 Calculate the emission take for the hotkey.
let take_proportion: I64F64 = I64F64::from_num( Delegates::<T>::get( hotkey ) ) / I64F64::from_num( u16::MAX );
let hotkey_take: u64 = ( take_proportion * I64F64::from_num( emission ) ).to_num::<u64>();
let take_proportion: I64F64 = I64F64::from_num(Delegates::<T>::get(hotkey))
.saturating_div(I64F64::from_num(u16::MAX));
let hotkey_take: u64 =
(take_proportion.saturating_mul(I64F64::from_num(emission))).to_num::<u64>();

// --- 1.5 Compute the remaining emission after deducting the hotkey's take.
let emission_minus_take: u64 = emission - hotkey_take;
let emission_minus_take: u64 = emission.saturating_sub(hotkey_take);

// --- 1.6 Calculate the remaining emission after the hotkey's take.
let mut remainder: u64 = emission_minus_take;

// --- 1.7 Iterate over each nominator.
for ( nominator, nominator_stake ) in <Stake<T> as IterableStorageDoubleMap<T::AccountId, T::AccountId, u64>>::iter_prefix( hotkey ) {

for (nominator, nominator_stake) in
<Stake<T> as IterableStorageDoubleMap<T::AccountId, T::AccountId, u64>>::iter_prefix(
hotkey,
)
{
// --- 1.7.0 Check if the stake was manually increased by the user since the last emission drain for this hotkey.
// If it was, skip this nominator as they will not receive their proportion of the emission.
if LastAddStakeIncrease::<T>::get( hotkey, nominator.clone() ) > last_hotkey_emission_drain { continue; }
if LastAddStakeIncrease::<T>::get(hotkey, nominator.clone())
> last_hotkey_emission_drain
{
continue;
}

// --- 1.7.2 Calculate this nominator's share of the emission.
let nominator_emission: I64F64 = I64F64::from_num( emission_minus_take ) * ( I64F64::from_num( nominator_stake ) / I64F64::from_num( total_hotkey_stake ) );
let nominator_emission: I64F64 = I64F64::from_num(emission_minus_take)
.saturating_mul(I64F64::from_num(nominator_stake))
.saturating_div(I64F64::from_num(total_hotkey_stake));

// --- 1.7.2 Increase the stake for the nominator.
Self::increase_stake_on_coldkey_hotkey_account( &nominator, hotkey, nominator_emission.to_num::<u64>() );
Self::increase_stake_on_coldkey_hotkey_account(
&nominator,
hotkey,
nominator_emission.to_num::<u64>(),
);

// --- 1.7.4 Subtract the nominator's emission from the remainder.
remainder -= nominator_emission.to_num::<u64>();
remainder = remainder.saturating_sub(nominator_emission.to_num::<u64>());
}

// --- 1.8. Finally, add the stake to the hotkey itself, including its take and the remaining emission.
Self::increase_stake_on_hotkey_account( hotkey, hotkey_take + remainder );
Self::increase_stake_on_hotkey_account(hotkey, hotkey_take + remainder);
}

///////////////
Expand All @@ -221,19 +247,19 @@ impl<T: Config> Pallet<T> {
///
/// # Returns
/// * `bool` - True if the hotkey emission should be drained, false otherwise.
pub fn should_drain_hotkey( index: u64, block: u64 ) -> bool {
return block % 7200 == index % 7200 // True once per day for each index assuming we run this every block.
pub fn should_drain_hotkey(index: u64, block: u64) -> bool {
return block % 7200 == index % 7200; // True once per day for each index assuming we run this every block.
}

/// Checks if the epoch should run for a given subnet based on the current block.
///
/// # Arguments
/// * `netuid` - The unique identifier of the subnet.
///
/// # Returns
/// # Returns
/// * `bool` - True if the epoch should run, false otherwise.
pub fn should_run_epoch( netuid: u16, current_block: u64) -> bool {
return Self::blocks_until_next_epoch( netuid, Self::get_tempo( netuid ), current_block ) == 0
pub fn should_run_epoch(netuid: u16, current_block: u64) -> bool {
return Self::blocks_until_next_epoch(netuid, Self::get_tempo(netuid), current_block) == 0;
}

/// Helper function which returns the number of blocks remaining before we will run the epoch on this
Expand All @@ -248,7 +274,12 @@ impl<T: Config> Pallet<T> {
/// Special case: tempo = 0, the network never runs.
///
pub fn blocks_until_next_epoch(netuid: u16, tempo: u16, block_number: u64) -> u64 {
if tempo == 0 { return u64::MAX; }
tempo as u64 - (block_number + netuid as u64 + 1) % (tempo as u64 + 1)
if tempo == 0 {
return u64::MAX;
}
(tempo as u64).saturating_sub(
(block_number.saturating_add((netuid as u64).saturating_add(1)))
% (tempo as u64).saturating_add(1),
)
}
}
}