FIL nodes update

[yiannisbot]

This PR is an update to section 2.1 of the spec on Filecoin Nodes. In the first instance it will cover all other parts of the section, apart from Node Types, which will come later or in a different PR.

[yiannisbot]

In the end there is an update on the node types in this PR, so this includes updates for the whole of the Filecoin Nodes subsection (currently 2.1).

[schomatis]

I'm not correctly following the node hierarchy here (it may be that this in fact exist somewhere in the code and I'm not aware of it), I only recognize two nodes, full and miner.

[schomatis]

Ok, now I think I follow. What you're calling nodes here are what the old spec refers to as services or subsystems. There are only two nodes, full and miner, the rest are services of which the nodes are composed. In fact, a node can be seen as nothing more than the API (services) it exposes, there is nothing intrinsically different in the full node apart from the fact that it's running the sync manager. (Maybe I'm misreading this and the retrieval/storage miner nodes are conceptual abstractions used here in the spec.)

[yiannisbot]

I've rephrased the text leading up to this list of "nodes". They are indeed services provided on top of the two nodes. Let me know if that clarifies things from the beginning.

I wouldn't mind renaming these as "services" or "subsystems" and taking the term "node" out of these completely, but they're referred as "nodes" in lots of other documentation I've seen. Most importantly, there's nothing to prevent an implementation from having all those as totally separate node types, right?

[schomatis]

Most importantly, there's nothing to prevent an implementation from having all those as totally separate node types, right?

Theoretically I think no, not sure if it has been tried out.

[hannahhoward]

@yiannisbot the distinction I would make is this:

• services can be provided by a node. some services include:

  • chain verification
  • storage market client
  • storage market provider
  • retrieval market client
  • retrieval market provider
  • storage mining

• a node is combination of these services that can participate as a node in the network. I think it's important that a node is ALWAYS someone who participates in the network, and MUST implement the chain verification service as a baseline in order to be considered a node.

• a process is just an OS process, and in our context, it packages a set of services together (you can call it a subsystem, though I find that term confusing) . A process may be a node on its own, or not -- it has to be on the network as a chain verifier to be considered a node.

• a "type of node" describes a common combination of services to make a filecoin node that plays a given role on the network. It can be a single OS process or multiple OS processes. I think the above descriptions (verifier, client, retrieval miner, storage miner) are accurate and should be left as is. These are all the intended types of nodes that will likely exist on the network at some point.

• Where I think you are actually getting lost it trying to map this on to lotus processes.

• When we say "full node" in Lotus, we're referring to the lotus daemon process. You're saying a "full node" is a chain verification node. However, running the Lotus daemon process on its own is actually a client node:

  • it has the chain verification system + message publishing functionality
  • it has the client services (storage market client & retrieval market client)
  • it has data transfer

• I find the term Full Node confusing and I do not think it should be in the spec

• the lotus-storage-miner process is:

  • the storage mining subsystem
  • the storage market provider
  • the retrieval market provider
  • data transfer for the provider

• the lotus-storage-miner process, on it's own, is not a node. In fact, it won't do much run on it's own.

• Running BOTH the lotus daemon and lotus-storage-miner process, absent customization, is both a Storage Mining Node, a Retrieval Mining Node, and a Client Node

  • it has the chain verification system + message publishing functionality
  • it has the client services as well (though they're not often used)
  • it has the markets provider services (both retrieval & storage) and the storage mining subsystem to seal pieces and min blocks
  • it has data transfer for client side and provider side

FWIW, there are other nodes coming:

• The initial Forrest implementation is a "verifier node" -- it has no client services at all, just chain verification
• however, the forrest process can be connected (or will be able to be connected) to lotus-storage-miner to make a Storage Miner Node and a Retrieval Miner Node
• the forrest team is making a go process which runs just the client subsystems -- that will together with Forrest make a Client Node
• I assume at some point someone is going to extract only the retrieval parts of the lotus-storage-miner process and back it with a purely unsealed data storage mechanism to make a process that when combined with lotus or Forrest will make a true "Retrieval Miner Node" who is not also a "Storage Miner Node"

[hannahhoward]

like @schomatis I think the node stuff is still pretty confusing. I've written out my thoughts in a way that is hopefully helpful.

the main thing I'd like to see is clarity that:

• a node is someone who participates on the network. to be a node, you have to verify the chain
• the types of nodes listed are different roles nodes can play on the network. to play this role, they have to implement a specific set of services on top of chain verification (unless they are just a verifier node)
• I would lose the description of "Full Node / Miner Node" -- you're trying to map Lotus OS processes on to nodes, and in fact, the lotus process has all the services to be a Client Node, while lotus-storage-miner on its own is not a node at all.

[hannahhoward]

@yiannisbot the distinction I would make is this:

• services can be provided by a node. some services include:

  • chain verification
  • storage market client
  • storage market provider
  • retrieval market client
  • retrieval market provider
  • storage mining

• a node is combination of these services that can participate as a node in the network. I think it's important that a node is ALWAYS someone who participates in the network, and MUST implement the chain verification service as a baseline in order to be considered a node.

• a process is just an OS process, and in our context, it packages a set of services together (you can call it a subsystem, though I find that term confusing) . A process may be a node on its own, or not -- it has to be on the network as a chain verifier to be considered a node.

• a "type of node" describes a common combination of services to make a filecoin node that plays a given role on the network. It can be a single OS process or multiple OS processes. I think the above descriptions (verifier, client, retrieval miner, storage miner) are accurate and should be left as is. These are all the intended types of nodes that will likely exist on the network at some point.

• Where I think you are actually getting lost it trying to map this on to lotus processes.

• When we say "full node" in Lotus, we're referring to the lotus daemon process. You're saying a "full node" is a chain verification node. However, running the Lotus daemon process on its own is actually a client node:

  • it has the chain verification system + message publishing functionality
  • it has the client services (storage market client & retrieval market client)
  • it has data transfer

• I find the term Full Node confusing and I do not think it should be in the spec

• the lotus-storage-miner process is:

  • the storage mining subsystem
  • the storage market provider
  • the retrieval market provider
  • data transfer for the provider

• the lotus-storage-miner process, on it's own, is not a node. In fact, it won't do much run on it's own.

• Running BOTH the lotus daemon and lotus-storage-miner process, absent customization, is both a Storage Mining Node, a Retrieval Mining Node, and a Client Node

  • it has the chain verification system + message publishing functionality
  • it has the client services as well (though they're not often used)
  • it has the markets provider services (both retrieval & storage) and the storage mining subsystem to seal pieces and min blocks
  • it has data transfer for client side and provider side

FWIW, there are other nodes coming:

• The initial Forrest implementation is a "verifier node" -- it has no client services at all, just chain verification
• however, the forrest process can be connected (or will be able to be connected) to lotus-storage-miner to make a Storage Miner Node and a Retrieval Miner Node
• the forrest team is making a go process which runs just the client subsystems -- that will together with Forrest make a Client Node
• I assume at some point someone is going to extract only the retrieval parts of the lotus-storage-miner process and back it with a purely unsealed data storage mechanism to make a process that when combined with lotus or Forrest will make a true "Retrieval Miner Node" who is not also a "Storage Miner Node"

[schomatis]

I'm not sure I would want to talk about OS processes in the spec, but other than that I'd extract what Hannah wrote there.

[yiannisbot]

@hannahhoward good points raised. I have done a major revision of the section (in commit: a019686) along the lines of: there are services and combinations of services form nodes. Please have another look when you get a chance.

[hannahhoward]

I don't want to hold up this section cause I'm not authoritative. The nodes section does make a lot of more sense now and is good.

I would like to get clarification on what we mean by Markets Order Book or if we can't, let's just delete it.

I can confirm the markets software has no Markets Order Book -- I believe it may just be a holdover from some earlier design of FC.

maybe @whyrusleeping has thoughts.

The base branch was changed.