spike: investigate best code practices and tests from other subnets

[mudler]

Summary for Implementing Tests for Our Subnet

Overview

We are enhancing our testing strategy to include tests that require the miner/validator to be running on the testnet. These tests will interact with the various endpoints of the nodes to ensure comprehensive coverage and reliability of our subnet.

Current Testing Strategy

Our current tests, as seen in the tests folder, primarily focus on mocking various components and verifying their behavior in isolation. For example:

Mock Subtensor Tests: Validate the behavior of MockSubtensor and its interactions with neurons.

@pytest.mark.parametrize("netuid", [1, 2, 3])
@pytest.mark.parametrize("n", [2, 4, 8, 16, 32, 64])
@pytest.mark.parametrize("wallet", [bt.MockWallet(), None])
def test_mock_subtensor(netuid, n, wallet):
    subtensor = MockSubtensor(netuid=netuid, n=n, wallet=wallet)
    neurons = subtensor.neurons(netuid=netuid)
    # Check netuid
    assert subtensor.subnet_exists(netuid)
    # Check network
    assert subtensor.network == "mock"
    assert subtensor.chain_endpoint == "mock_endpoint"
    # Check number of neurons
    assert len(neurons) == (n + 1 if wallet is not None else n)
    # Check wallet
    if wallet is not None:
        assert subtensor.is_hotkey_registered(
            netuid=netuid, hotkey_ss58=wallet.hotkey.ss58_address
        )

    for neuron in neurons:
        assert isinstance(neuron, bt.NeuronInfo)
        assert subtensor.is_hotkey_registered(netuid=netuid, hotkey_ss58=neuron.hotkey)

Mock Metagraph Tests: Check the properties and interactions of MockMetagraph.

@pytest.mark.parametrize("n", [16, 32, 64])
def test_mock_metagraph(n):
    mock_subtensor = MockSubtensor(netuid=1, n=n)
    mock_metagraph = MockMetagraph(subtensor=mock_subtensor)
    # Check axons
    axons = mock_metagraph.axons
    assert len(axons) == n
    # Check ip and port
    for axon in axons:
        assert isinstance(axon, bt.AxonInfo)
        assert axon.ip == mock_metagraph.default_ip
        assert axon.port == mock_metagraph.default_port
        ```
        
### Mock Dendrite Timings: Ensure that MockDendrite respects timing constraints and handles timeouts correctly.
```python
@pytest.mark.parametrize("timeout", [0.1, 0.2])
@pytest.mark.parametrize("min_time", [0, 0.05, 0.1])
@pytest.mark.parametrize("max_time", [0.1, 0.15, 0.2])
@pytest.mark.parametrize("n", [4, 16, 64])
def test_mock_dendrite_timings(timeout, min_time, max_time, n):
    mock_wallet = None
    mock_dendrite = MockDendrite(mock_wallet)
    mock_dendrite.min_time = min_time
    mock_dendrite.max_time = max_time
    mock_subtensor = MockSubtensor(netuid=1, n=n)
    mock_metagraph = MockMetagraph(subtensor=mock_subtensor)
    axons = mock_metagraph.axons

    async def run():
        return await mock_dendrite(
            axons,
            synapse=PromptingSynapse(
                roles=["user"], messages=["What is the capital of France?"]
            ),
            timeout=timeout,
        )

    responses = asyncio.run(run())
    for synapse in responses:
        assert hasattr(synapse, "dendrite") and isinstance(
            synapse.dendrite, bt.TerminalInfo
        )

        dendrite = synapse.dendrite
        # check synapse.dendrite has (process_time, status_code, status_message)
        for field in ("process_time", "status_code", "status_message"):
            assert hasattr(dendrite, field) and getattr(dendrite, field) is not None

        # check that the dendrite take between min_time and max_time
        assert min_time <= dendrite.process_time
        assert dendrite.process_time <= max_time + 0.1
        # check that responses which take longer than timeout have 408 status code
        if dendrite.process_time >= timeout + 0.1:
            assert dendrite.status_code == 408
            assert dendrite.status_message == "Timeout"
            assert synapse.dummy_output == synapse.dummy_input
        # check that responses which take less than timeout have 200 status code
        elif dendrite.process_time < timeout:
            assert dendrite.status_code == 200
            assert dendrite.status_message == "OK"
            # check that outputs are not empty for successful responses
            assert synapse.dummy_output == synapse.dummy_input * 2
        # dont check for responses which take between timeout and max_time because they are not guaranteed to have a status code of 200 or 408

New Testing Strategy

To ensure our subnet's robustness, we will introduce tests that require the miner/validator to be running on the testnet. These tests will:

1. Hit Different Endpoints: Interact with various endpoints of the nodes to verify their responses and behavior under real network conditions.

2. Validate Network Interactions: Ensure that the nodes correctly handle requests, process transactions, and maintain network integrity.

3. Performance and Reliability: Measure the performance and reliability of the nodes under different scenarios, including stress tests and edge cases.

Implementation Plan

1. Setup Testnet Environment: Ensure that the testnet environment is properly configured and that the miner/validator nodes are running.

2. Write Integration Tests: Develop tests that interact with the testnet endpoints. These tests will be more comprehensive and cover scenarios that are not possible with mocked components.

3. Automate Test Execution: Integrate these tests into our CI/CD pipeline to ensure they are run automatically and consistently.

Example Test

Here is a conceptual example of how a test might look:

import requests

def test_node_endpoint():
    response = requests.get("http://testnet-node-endpoint/api/status")
    assert response.status_code == 200
    assert response.json()["status"] == "running"

This test checks the status endpoint of a node running on the testnet, ensuring it is up and running.

Conclusion

By incorporating tests that interact with the testnet, we will achieve a higher level of confidence in the reliability and performance of our subnet. This approach will help us identify and address issues that may not be apparent in isolated, mocked tests.

[mudler]

@Luka-Loncar / @hide-on-bush-x what's the result of the spike?

[Luka-Loncar]

We agreed on Friday that @hide-on-bush-x will add the description.

[hide-on-bush-x]

@mudler @Luka-Loncar updated!