P2P Architecture

Package - p2p

Description

p2p package exposes minimal functions as a service to other components in Spacemesh. any part of Spacemesh can send a p2p message, broadcast or register a protocol to handle incoming messages of a specified type.

Public API

• interface Service - An interface that represents the methods required for a p2p service.
• interface Message - An interface that represents a message that will be sent or received by Service's methods.
• New(p2p.config) - Creates a new swarm, swarm satisfies the Service interface.

swarm

Description

When a message is received in the swarm it unpacks the message from binary to a general format which holds basic data about the message as Timestamp, ClientVersion, AuthorSign and Protocol, Swarm uses this metadata to check the validity of the message, after the message passed this check, it sends the payload to the protocol. Any message payload that is sent to swarm is packed with this format exactly and sent through a net.Connection.

Protocols are registered to Swarm in order to receive messages directed to them.

Public API

• SendMessage(nodeID string, protocol string, payload []byte) error

  sends a direct P2P message to a specific nodeID. to do so it creates or acquires a secured (encrypted) connection and sends the message with the protocol name attached as metadata. NOTE : this function will block and might take time to lookup the node or create the connection.

• Broadcast(protocol string, payload []byte) NOTE: to implement with gossip.

  Disseminates a message to the node's neighbors and starting a gossip protocol

• Narrowcast() - TODO: Consider narrowcast.

• RegisterProtocol(Protocol string) chan Message -

  Registering a protocol name to a channel that is sending messages as the Message interface. the received channel should be listening to handle incoming messages from this protocol type. (See Protocols)

• Bootstrap() error - Bootstrap starts by registering the pre-configurated nodes, query them FindNode protocol about ourselves, then do the same to the results they return until we don't see any new results or we filled out routing table with more then our minimum random connections. NOTE: Bootstrap() blocks until finished or returns an error, but it should be used as a background process

Under the hood

When a message is sent using Swarm.SendMessage Swarm will find a peer in the dht and ask ConnectionPool for an active or new connection with that peer. once the connection is achieved it will wrap the message with a general protocol structure and attach metadata to it so the receiving Swarm would be able to open it, read this metadata, validate signatures and route the message one of the registered protocols.

• identity.LocalNode - our p2p identity
• dht - generate a dht for p2p, bootstrap it, manage it and issue Kademlia operations to lookup nodes.

Message Metadata structure

message Metadata {
  string protocol = 1;      // Protocol id string
  bytes reqId = 2;          // Unique request id. Generated by caller. Returned in responses.
  string clientVersion = 3; // Author client version
  int64 timestamp = 4;      // Unix time - authoring time (not sending time)
  bool gossip = 5;          // True to have receiver peer gossip the message to its neighbors
  bytes authPubKey = 6;     // Authoring node Secp256k1 public key (32bytes) - may not be sender
  string authorSign = 7;    // Signature of message data by author + method specific data by message creator node. format: hex-encoded bytes
}

Dependencies

• crypto - Generating a key-pair identity and signing/veryfing messages.
• log - for logging purposes.
• p2p/net - to listen on net events and use Connection to send messages.
• p2p/timesync - time related operations such as checking a message time drift and our own drift against NTP
• p2p/identity - to create our own and handle other nodes in the network.
• p2p/dht - all dht operations used for looking up peers and saving a quick healthy "goto" peer list.

package p2p/connectionpool

Description

ConnectionPool stores all net.Connections and make them available to all users of net.Connection. Outgoing and incoming connections will be created inside connectinopool and will be stores in a way that any client could use ConnectionPool to initiate or request a connection with a peer.

Public API

• NewConnectionPool(network networker, lPub crypto.PublicKey) *ConnectionPool - constructs a connection pool from a net interface and a local public key. returns the created *ConnectionPool. this starts listening to new connection from the consumed networker interface.
• GetConnection(address string, remotePub crypto.PublicKey) (net.Connection, error) - Tries to get a connection from the pool, if the connection doesn't exist we dial to this connection to initiate a new connection. this is a blocking function, it will block until a net.Connection or an error is returned.
• Shutdown() - gracefully shutsdown the connection pool by closing all the connection and returning all pending GetConnection queries.

Dependencies

• crypto
• p2p/net - Work with connections form net.

package p2p/node

Description

Package node implements containers that hold basic information about nodes in the network. the most basic struct is Node, it also implements LocalNode and DhtID.

LocalNode includes a Node.

Public API

• New(key crypto.PublicKey, address string) Node - creates a basic node identity from a PublicKey and an IP address.
• NewNodeFromString(data string) (Node, error) - Tries to break a node string descriptor into a Node. example string: 126.0.0.1:3572/r9gJRWVB9JVPap2HKnduoFySvHtVTfJdQ4WG8DriUD82
• StringFromNode(node Node) string - Creates the above format string from a Node
• Union(list1 []Node, list2 []Node) []Node - combine two Node slices without duplicates.
• SortByDhtID(nodes []Node, id DhtID) []Node - sort a node slice in relation to a DhtID.
• NewLocalNode(Address string, config nodeconfig, persist bool) (*LocalNode, error) - tries to create a node from an existing file, if it can't be find generates a new node with NewNodeIdentity, persist boolean - save the node to the disk or not.
• NewNodeIdentity(config config.Config, address string, persist bool) (*LocalNode, error) - generates a private-public key pair and creates a LocalNodefrom it.persist` boolean - save the node to the disk or not.
• NewDhtID(key []byte) DhtID - Creates a DhtID from a PublicKey byte array
• NewDhtIDFromBase58(s string) DhtID - Creates a DhtID from a base58 encoded PublicKey
• NewDhtIDFromHex(s string) (DhtID, error) - Creates a DhtID from an hex encoded string.

Node

Description

A basic identity node struct that is included in more complicated structs to implement basic identity features.

Public API -

• ID() - byte array of the node's public key
• DhtID() - A DhtID created from the node's ID
• String() - An hex-encoded string representation of the node ID. Implements go's Stringer interface.
• Address() - An IP address

LocalNode

Description

The LocalNode is the struct that represent a our identity in the network. Every p2p operation happens with a LocalNode identity at its source. It holds a Private and Public key. LocalNode can initialize from scratch or from existing persistent keys and files. it needs a local address and port. PublicKey is the node's identifier in the network.

Public API

• Node Methods.
• PublicKey() - The node's public key as crypto.PublicKey also used as the node's identity in the network.
• PrivateKey() - The node's Private Key as crypto.PrivateKey used for signing.
• NetworkID() - The NetworkID this nodes belongs to
• Log - LocalNode embeds a log which lets it act as a logger.

DhtID

DhtID is implemented in node to arrange all identity entities in one place and make dht agnostic of its DhtID implementation. for more explanations about our current DhtID implementation please see The XOR Space in dht.

Dependencies

• crypto - used for public and private keys.
• filesystem - Used for persisting the node's info to files
• log - to create a dedicated log for LocalNode

Package - p2p/net

Description

package p2p/net is the package that handles all network communications, net is a general purpose connection creator and message listener. net is basically a connection factory. it is used to initiate and to accept all network operations including connections and messages. Each connection that established with a net.Connection struct that holds a go net.Conn TCP connection that is getting piped through the passed wire formatter.

net is wire format agnostic, at this moment we use binary delimited protobuf messages, we do it using delimited delimited wraps the connections and reads/writes length delimited messages over it. All connections created with net are secured (encrypted) connections.

Public API

• Dial(address string, remotePublicKey crypto.PublicKey) (Connection, error) - Initiate a secured connection with another peer , returns error or nil. blocks until session is created.
• SubscribeOnNewRemoteConnections() chan Connection - subscribe to get update on new established remote connections.
• HandlePreSessionIncomingMessage(c Connection, message []byte) error - handles an incoming message which received before a session was created (must be a session message, usually the first message). returns error when session creation failed.
• GetClosingConnections() - A channel of connections that are closing because of an error or requested closing.
• IncomingMessages() chan IncomingMessageEvent - A channel that bridges incoming messages from all the connections in a fan-in fashion.

Session message format

// The initiator creates a HandshakeData object
// with a random IV based on the requested node Public Key
// and our own Private Key.
// The request also contains basic details
// about the initiator to decide if we can can create a session
// Handshake protocol data used for both request and response - sent unencrypted over the wire
message HandshakeData {
  bytes sessionId = 1;    // for req - same as iv. for response - set to req id
  bytes payload = 2; // empty for now
  int64 timestamp = 3; // sending time
  string clientVersion = 4; // client version of the sender
  int32 networkID = 5; // network id of sending node
  string protocol = 6; // 'handshake/req' || 'handshake/resp'
  bytes nodePubKey = 7; // 65 bytes uncompressed
  bytes iv = 8; // 16 bytes - AES-256-CBC IV
  bytes pubKey = 9; // 65 bytes (uncompressed) ephemeral public key
  bytes hmac = 10; // HMAC-SHA-256 32 bytes
  string sign = 11; // hex encoded string 32 bytes sign of all above data by node public key (verifies he has the priv key and he wrote the data
} ```

#### `Connection`

##### Public API -

- `ID()` - The connection `UUID`
- `Send(message) error` - Sends a message on the connection, returns an error if failed.
- `Session() NetworkSession` - this connection's session. returns nil when no session.
- `Close()` - Closes the connection.
- `RemoteAddr()` - The remote connection address.
- `RemotePublicKey() crypto.PublicKey` - The remote connection `crypto.PublicKey`

#####  Dependencies
 - `crypto` - used to create a `UUID` for a connection

###  Dependencies -
 - `log`
 - `nodeconfig`

#### `NetworkSession`

##### Description 
`NetworkSession` is an authenticated network session between 2 peers.
Sessions may be used between 'connections' until they expire.
Session provides the encryptor/decryptor for all messages exchanged between 2 peers.
enc/dec is using an ephemeral symmetric key exchanged securely between the peers via the secured connection handshake protocol.

##### Public API

  - `Decrypt(in []byte) ([]byte, error)` - decrypt data using session dec key
  - `Encrypt(in []byte) ([]byte, error)` - encrypt data using session enc key

#### `p2p/net/delimited`

##### Description

We are using our own simple length-prefix binary format:

<32 bits big-endian data-length><message binary data (protobufs-bin-encoded)>

We need to use length prefixed protobufs messages because `protobuf`s data doesn't include length and we need this to allow multiple messages on the same tcp/ip connection . see p2p2.conn.go.

##### Public API
  - `NewReader(r io.Reader) *delimited.Reader` - Creates a `delimited` wrapped reader. not used directly.
  - `NewWriter(r io.Writer) *delimited.Writer` - Creates a `delimited` wrapped writer. not used directly.
  - `Pipe(io.ReadWriteCloser)` - Pipe an `io.ReadWriteCloser` through the `delimited` format.
  - `In() chan []byte` - The incoming messages channel
  - `Out(message []byte) error` - sends out a message on the wire, it is a blocking operation.
  - `Close()` - closes the internal connection and the channels.

## Package - `p2p/dht`

### Description

`dht` is an integral part of the `p2p` package, it is used to implement the `Kademlia` DHT. A [DHT](https://en.wikipedia.org/wiki/Distributed_hash_table) is mainly used in p2p networks to lookup and probe an ID into an IP address or a file hash to the node it's stored on.

we broke `Kademlia` into 2 parts the node store known as the `RoutingTable` and the protocols, we skip most of `Kademlia` protocol and implement only the `FindNode` protocol which is used to probe node's in the network by IDs.

The `RoutingTable` is a table that holds all the nodes that we know from communications or lookups in the network, it is in charge of keeping that list of nodes healthy and ready for when the node needs to preform a lookup or get information from the network.

For `Kademlia` lookups we're hashing the original nodeIDs to SHA256 hashes. every node ID has its deterministic `dht.ID`.

Using those hashes we calculate and arrange nodes inside the `RoutingTable` according to their "Distance" from us in `Kademlia` "Distance" is determined by a space called the `XOR` Keyspace

#### The `XOR` Keyspace

The `dht.ID`s exist inside the `XOR` keyspace, `dht` also exposes sorting and functionality in this space which should help determine the distance to other nodes in this space.

"Distance" to a node in the XOR space is the result of XORing the IDs of both nodes.
XOR is symmetric and will always produce the same result for the same set of IDs.

When XORing the `dht.ID`s together we use the given result and check the amount of leading zero's in the binary representation of the result.
this is called the `Common Prefix Length` of thses two `dht.ID`s.

The higher the `Common Prefix Length` the closer the node to us.

Example:

// For this example and readability we assume the keyspace is 24bits (3 bytes) long instead of 256bits "123" as dht.ID = a665a4 binary - 10100110110010110100100

"567" as dht.ID = 97a6d2 binary - 100101111010011011010010

XORing : { "123"(a665a4) XOR "567"(97a6d2) = 31c376 binary - 110001110000111110110 }

The zero prefix length of the xor result which is the Common Prefix Length : 2 (100101111010011011010010)

A third ID : "789" "789" as dht.ID = 35a9e3 binary - 1101011010100111100011 XORing "123" with "789" { "123"(a665a4) XOR "789"(35a9e3) = 93cc47 binary - 110001110000111110110 }

The zero prefix length of the xor result which is the Common Prefix Length : 0

567 is closer in xor space to 123 than 789

### `DHT`

#### Public API

  - `New` - creates a new `DHT`


  - `DHT.Bootstrap(nodes)` - bootstrap the routing table using the given nodes
  - `DHT.Update(node)` - Update a node in the routing table
  if we know this peer we move it to the top of the `Bucket`
   If we never heard of this peer, we insert it to the appropriate bucket, if this bucket is full then we ping the last contacted peer (which should be the last in the bucket) and see if its alive. we then compare latency metrics and choose the best of the two.
  - `DHT.FindNode(id)` - Issue a `Kademlia` find node operation in the network

`dht` implements `RoutingTable` which is a table that holds all the nodes that we know from communications or lookups in the network.

The `RoutingTable` is in charge of keeping that list of nodes healthy and ready for when the node needs to preform a lookup or get information from the network.

The nodes in the `RoutingTable` are arranged inside `Bucket`s in `Kademlia` they are called `KBuckets`. every bucket represent a `Common Prefix Length` we store 20 `Buckets` which hold nodes according to our `BucketSize` parameter which is currently 20.
The higher the `Bucket` number, the closer the nodes in the list to us.
Most of the nodes are going to be in `Bucket` 0, half of the network according to `Kademlia`, half of the rest of the nodes will be in `Bucket` 1, and so forth.
this is why we hold only 20 buckets.

[Extra notes about DHT](https://github.com/spacemeshos/go-spacemesh/blob/develop/p2p/dht/readme.md)

#### Dependencies

 -  `net` - issue operations through a secure/unsecured connections

##### Bootstrapping

The bootstrap process is what lets us join the p2p network.

When the Spacemesh node start it has to fill it's `RoutingTable` with fresh and active nodes, to do this process the node is equipped with a list of Bootstrap Nodes. every Spacemesh node can act as Bootstrap Node so they can be replaced with a detailed acquired from any other source.

To bootstrapping process is just a matter of issuing the `FindNode` `Kademlia` protocol (refer to `P2P Protocols`) with the node own `dht.ID` as a parameter. this will return a list of nodes close to our own `dht.ID`, we'll query those
nodes (Concurrently) for the same ID until we don't get new results and we queried all nodes.

This will make every node that see our request to update its `RoutingTable` with our node's `dht.ID` and also fill our `dht.ID` with these nodes.

## package `p2p/nodeconfig`

### Description

`nodeconfig` loads config parameters from the default config file, parameters inside `nodeconfig` are related to `p2p`.

### Public API

 - `DefaultConfig()` - A default config for the node