P2P Architecture

Package p2p

Description

p2p package exposes minimal peer-to-peer functionality as a service to other components in Spacemesh. Any protocol in Spacemesh can send a direct p2p message, broadcast to all peers or register a callback to handle incoming messages of a specified type.

Public API

• interface Service - An interface that represents the methods required for a p2p service.
• interface Message - An interface that represents a message that will be sent or received by Service's methods.
• New(p2p.config) - Creates a new swarm, swarm satisfies the Service interface.

NOTE : Service and Message are declared in p2p/service but re-declared and exposed in p2p to expose them out of the p2p package easily.

type Service interface {
   SendMessage(nodeID string, protocol string, payload []byte) error
   RegisterProtocol(Protocol string) chan Message
   Shutdown()
}

type Message interface {
   Data() []byte
   Sender() node.Node
}

Struct Swarm

Description

When a message is received, Swarm unpack the message from its binary form to a structured format which holds basic data about the message as Timestamp, ClientVersion, AuthorSign and Protocol. Swarm uses this metadata to check the validity of the message, after the message passed this check, it sends the payload to the registered protocol. Any message payload that is sent by Swarm is packed with this format exactly and sent using a net.Connection.

Protocols are registered to a Swarm in order to receive messages holding their protocol name.

Public API

• SendMessage(nodeID string, protocol string, payload []byte) error

  Sends a direct P2P message to a specific nodeID. When a message is sent using Swarm.SendMessage, Swarm will find the target peer in DHT and ask ConnectionPool for a secured connection with that peer. Once the connection is achieved it will add a header to the payload and send the message to the remote Swarm would be able to open it, read this metadata, validate signatures and route the message to the registered protocols. NOTE : this function will block and might take time to lookup the node or create the connection.

• Broadcast(protocol string, payload []byte) -

NOTE TODO: implement with gossip. Disseminates a message to the node's neighbours via the gossip protocol

• RegisterProtocol(Protocol string) chan Message -

  Associating a protocol with a channel, all messages carrying the protocol name in the header will be delivered on the returned channel and will be structured according to the Message interface.

• Bootstrap() error -

Bootstrap starts by registering the pre-configurated bootstrap nodes, query them according to the FindNode protocol for the local node and continue doing the same with the results nodes until there are no new results or until the routing table is full. NOTE: Bootstrap() blocks until finished or returns an error, but it should be used as a background process

Message Metadata

message Metadata {
  string protocol = 1;      // Protocol id string
  bytes reqId = 2;          // Unique request id. Generated by caller. Returned in responses.
  string clientVersion = 3; // Author client version
  int64 timestamp = 4;      // Unix time - authoring time (not sending time)
  bool gossip = 5;          // True to have receiver peer gossip the message to its neighbors
  bytes authPubKey = 6;     // Authoring node Secp256k1 public key (32bytes) - may not be sender
  string authorSign = 7;    // Signature of message data by author + method specific data by message creator node. format: hex-encoded bytes
}

Dependencies

• crypto
• log
• p2p/net
• p2p/timesync
• p2p/identity
• p2p/dht

Package p2p/connectionpool

Struct ConnectionPool

Description

ConnectionPool stores all net.Connections and make them available to all users of net.Connection. All local connections are created by ConnectionPool, using Net. Also ConnectionPool is in charge of listening to new remote connections. Local and remote connections are treated the same by ConnectionPool and at any given point there is at most one connection with any given peer.

Public API

• NewConnectionPool(network networker, lPub crypto.PublicKey) *ConnectionPool - Constructs a new connection pool. Once created, the ConnectionPool will start listening to new remote connections using the networker interface.
• GetConnection(address string, remotePub crypto.PublicKey) (net.Connection, error) - Tries to fetch a connection from the pool, if the connection doesn't exist it will establish one using the networker interface. This is a blocking function, it will block until a net.Connection or an error is returned.
• Shutdown() - Gracefully shuts down the connection pool by closing all the connections in the pool and releasing all pending GetConnection queries.

Dependencies

• crypto
• p2p/net

Package p2p/node

Description

Package node implements containers that hold basic information about nodes in the network. the most basic struct is Node, it also implements LocalNode and DhtID.

LocalNode includes a Node.

Public API

• New(key crypto.PublicKey, address string) Node - creates a basic node identity from a PublicKey and an IP address.
• NewNodeFromString(data string) (Node, error) - Tries to break a node string descriptor into a Node. example string: 126.0.0.1:3572/r9gJRWVB9JVPap2HKnduoFySvHtVTfJdQ4WG8DriUD82
• StringFromNode(node Node) string - Creates the above format string from a Node
• Union(list1 []Node, list2 []Node) []Node - combine two Node slices without duplicates.
• SortByDhtID(nodes []Node, id DhtID) []Node - sort a node slice in relation to a DhtID.
• NewLocalNode(Address string, config nodeconfig, persist bool) (*LocalNode, error) - tries to create a node from an existing file, if it can't be find generates a new node with NewNodeIdentity, persist boolean - save the node to the disk or not.
• NewNodeIdentity(config config.Config, address string, persist bool) (*LocalNode, error) - generates a private-public key pair and creates a LocalNodefrom it.persist` boolean - save the node to the disk or not.
• NewDhtID(key []byte) DhtID - Creates a DhtID from a PublicKey byte array
• NewDhtIDFromBase58(s string) DhtID - Creates a DhtID from a base58 encoded PublicKey
• NewDhtIDFromHex(s string) (DhtID, error) - Creates a DhtID from an hex encoded string.

Node

Description

A basic identity node struct that is included in more complicated structs to implement basic identity features.

Public API -

• ID() - byte array of the node's public key
• DhtID() - A DhtID created from the node's ID
• String() - An hex-encoded string representation of the node ID. Implements go's Stringer interface.
• Address() - An IP address

LocalNode

Description

The LocalNode is the struct that represent a our identity in the network. Every p2p operation happens with a LocalNode identity at its source. It holds a Private and Public key. LocalNode can initialize from scratch or from existing persistent keys and files. it needs a local address and port. PublicKey is the node's identifier in the network.

Public API

• Node Methods.
• PublicKey() - The node's public key as crypto.PublicKey also used as the node's identity in the network.
• PrivateKey() - The node's Private Key as crypto.PrivateKey used for signing.
• NetworkID() - The NetworkID this nodes belongs to
• Log - LocalNode embeds a log which lets it act as a logger.

DhtID

NOTE: DhtID is implemented in node to arrange all identity entities in one place and make dht agnostic of its DhtID implementation. for further explanatio about DhtID's implementation please see The XOR Space in dht.

Dependencies

• crypto - used for public and private keys.
• filesystem - Used for persisting the node's info to files
• log - to create a dedicated log for LocalNode

Package - p2p/net

Description

package p2p/net is the package that handles all network communications, net is a general purpose connection creator and message listener. net is basically a connection factory. it is used to initiate and to accept all network operations including connections and messages. Each connection that established with a net.Connection struct that holds a go net.Conn TCP connection that is getting piped through the passed wire formatter.

net is wire format agnostic, at this moment we use binary delimited protobuf messages, we do it using delimited delimited wraps the connections and reads/writes length delimited messages over it. All connections created with net are secured (encrypted) connections.

Public API

• Dial(address string, remotePublicKey crypto.PublicKey) (Connection, error) - Initiate a secured connection with another peer , returns error or nil. blocks until session is created.
• SubscribeOnNewRemoteConnections() chan Connection - subscribe to get update on new established remote connections.
• HandlePreSessionIncomingMessage(c Connection, message []byte) error - handles an incoming message which received before a session was created (must be a session message, usually the first message). returns error when session creation failed.
• GetClosingConnections() - A channel of connections that are closing because of an error or requested closing.
• IncomingMessages() chan IncomingMessageEvent - A channel that bridges incoming messages from all the connections in a fan-in fashion.

Session message format

// The initiator creates a HandshakeData object
// with a random IV based on the requested node Public Key
// and our own Private Key.
// The request also contains basic details
// about the initiator to decide if we can can create a session
// Handshake protocol data used for both request and response - sent unencrypted over the wire
message HandshakeData {
  bytes sessionId = 1;    // for req - same as iv. for response - set to req id
  bytes payload = 2; // empty for now
  int64 timestamp = 3; // sending time
  string clientVersion = 4; // client version of the sender
  int32 networkID = 5; // network id of sending node
  string protocol = 6; // 'handshake/req' || 'handshake/resp'
  bytes nodePubKey = 7; // 65 bytes uncompressed
  bytes iv = 8; // 16 bytes - AES-256-CBC IV
  bytes pubKey = 9; // 65 bytes (uncompressed) ephemeral public key
  bytes hmac = 10; // HMAC-SHA-256 32 bytes
  string sign = 11; // hex encoded string 32 bytes sign of all above data by node public key (verifies he has the priv key and he wrote the data
} 

Connection

Public API -

• ID() - The connection UUID
• Send(message) error - Sends a message on the connection, returns an error if failed.
• Session() NetworkSession - this connection's session. returns nil when no session.
• Close() - Closes the connection.
• RemoteAddr() - The remote connection address.
• RemotePublicKey() crypto.PublicKey - The remote connection crypto.PublicKey

Dependencies

• crypto - used to create a UUID for a connection

Dependencies -

• log
• nodeconfig

NetworkSession

Description

NetworkSession is an authenticated network session between 2 peers. Sessions may be used between 'connections' until they expire. Session provides the encryptor/decryptor for all messages exchanged between 2 peers. enc/dec is using an ephemeral symmetric key exchanged securely between the peers via the secured connection handshake protocol.

Public API

- `Decrypt(in []byte) ([]byte, error)` - decrypt data using session dec key
- `Encrypt(in []byte) ([]byte, error)` - encrypt data using session enc key

p2p/net/delimited

Description

We are using our own simple length-prefix binary format:

<32 bits big-endian data-length><message binary data (protobufs-bin-encoded)>

We need to use length prefixed protobufs messages because protobufs data doesn't include length and we need this to allow multiple messages on the same tcp/ip connection . see p2p2.conn.go.

Public API

• NewReader(r io.Reader) *delimited.Reader - Creates a delimited wrapped reader. not used directly.
• NewWriter(r io.Writer) *delimited.Writer - Creates a delimited wrapped writer. not used directly.
• Pipe(io.ReadWriteCloser) - Pipe an io.ReadWriteCloser through the delimited format.
• In() chan []byte - The incoming messages channel
• Out(message []byte) error - sends out a message on the wire, it is a blocking operation.
• Close() - closes the internal connection and the channels.

Package - p2p/dht

Description

dht is an integral part of the p2p package, it is used to implement the Kademlia DHT. A DHT is mainly used in p2p networks to lookup and probe an ID into an IP address or a file hash to the node it's stored on.

we broke Kademlia into 2 parts the node store known as the RoutingTable and the protocols, we skip most of the Kademlia protocols and implement only the FindNode protocol which is used to probe nodes in the network by IDs.

The RoutingTable is a table that holds all the nodes that we know from communications or lookups in the network, it is in charge of keeping that list of nodes healthy and ready for when the node needs to preform a lookup or get information from the network.

For Kademlia lookups we're hashing the original Node Public Keys to SHA256 hashes. every node ID has its deterministic DhtID.

Using those hashes we calculate and arrange nodes inside the RoutingTable according to their "Distance" from us in Kademlia "Distance" is determined by a space called the XOR Keyspace

The XOR Keyspace

The dht.IDs exist inside the XOR keyspace, dht also exposes sorting and functionality in this space which should help determine the distance to other nodes in this space.

"Distance" to a node in the XOR space is the result of XORing the IDs of both nodes. XOR is symmetric and will always produce the same result for the same set of IDs.

When XORing the dht.IDs together we use the given result and check the amount of leading zero's in the binary representation of the result. this is called the Common Prefix Length of thses two dht.IDs.

The higher the Common Prefix Length the closer the node to us.

Example:

// For this example and readability we assume the keyspace is 24bits (3 bytes) long instead of 256bits
"123" as DhtID = a665a4
binary - 10100110110010110100100

"567" as DhtID = 97a6d2
binary - 100101111010011011010010

XORing : {
"123"(a665a4) XOR "567"(97a6d2) = 31c376
binary - 110001110000111110110
}

The zero prefix length of the xor result
which is the Common Prefix Length : 2
(100101111010011011010010)

A third ID : "789"
"789" as DhtID = 35a9e3
binary - 1101011010100111100011
XORing "123" with "789" {
  "123"(a665a4) XOR "789"(35a9e3) = 93cc47
  binary - 110001110000111110110
}

The zero prefix length of the xor result
which is the Common Prefix Length : 0

567 is closer in xor space to 123 than 789

DHT

Public API

• New(p2p.Config) DHT - creates a new DHT with the config parameters passed.
• Bootstrap() error - bootstrap the routing table with the bootnodes configured.
• Update(node node.Node) - Update a node in the routing table if we know this peer we move it to the top of the Bucket If we never heard of this peer, we insert it to the appropriate bucket, if this bucket is full then we ping the last contacted peer (which should be the last in the bucket) and see if its alive. we then compare latency metrics and choose the best of the two.
• Lookup(id string) (node.Node, error) - Tries to lookup the node in the local RoutingTable if it wasn't found, issues a network FindNode operation to find it..

dht implements RoutingTable which is a table that holds all the nodes that we know from communications or lookups in the network.

The RoutingTable is in charge of keeping that list of nodes healthy and ready for when the node needs to preform a lookup or get information from the network.

The nodes in the RoutingTable are arranged inside Buckets, in Kademlia they are called KBuckets. every bucket is represented by a Common Prefix Length, we store 20 Buckets which hold nodes according to our BucketSize parameter which is currently 20. The higher the Bucket's Common Prefix Length, the closer the nodes in the list to us. Most of the nodes are going to be in Bucket 0, half of the network according to Kademlia, half of the rest of the nodes will be in Bucket 1, and so forth. this is why we hold only 20 buckets.

Extra notes about DHT

Dependencies

• net - issue operations through a secure/unsecured connections
• node - usage of node identities and DhtIDs.

FindNode

Our KadDHT implementation holds a FindNode protocol. this protocol is the actual Kademlia - FindNode implementation. When it receives a FindNode request it looks up the local table for matching or close nodes to the requested ID. it then sends them back to the requester as a sorted array.

The protocol is created by passing a p2p.Service and a RoutingTable the constructing function is only exposed to the dht package.

Public API

• FindNode(serverNode node.Node, target string) ([]node.Node, error) - the only method this protocol exposes, it asks the serverNode about the target ID. it returns a sorted slice with the closest nodes to this ID with the requested node ID as the first one if found. this function is blocking while sending the message and waiting for response.

Bootstrapping

The Bootstrap process is what let's the node join the p2p network.

The operation will run in the background and fill our RoutingTable with fresh and active nodes, to do this the node is pre-configured with a list of Bootstrap Nodes. every node can act as Bootstrap Node so they can be replaced with node information acquired from any other source.

The bootstrapping process is just a matter of issuing the FindNode Kademlia protocol with the node's own identity as a parameter. this will return a list of nodes close to our own DhtID, we'll query those nodes (Concurrently) for the same ID until we don't get new results and we queried all nodes.

This will make every node that see our request to update its RoutingTable with our node and also fill our RoutingTable with these nodes. soon peers from all around the network will know that we join and start querying us in the future.

p2p/sim - the p2p simulator

Description

The p2p simulator is used for testing and simulating a p2p network of nodes without actually listening to ports and sending network messages. it creates nodes that satisfy the p2p.Service interface.

Public API

• New() *Simulator - Creates a simulator that acts as services factory and bridge
• Simulator.NewNode() *Node - Creates a node on the receiving Simulator network.
• Simulator.NewNodeFrom(node.Node) *Node - Creates a node on the receiving Simulator network from an existing node data.

Dependencies

• node - to implement the Service interface
• service - to implement the Service interface