cmd: Consider having multiple binaries per Node Type

[Wondertan]

[adlerjohn]

Is there a reason to make them different?

[liamsi]

Good question. I'd recommend editing the opening comment with a brief rationale instead of replying.

[Wondertan]

The reason is simple. In case you run them both on one machine they would not conflict.

[liamsi]

Shouldn't the case of running a light node and a full node on the same machine be accompanied by a script (or e.g. simply docker compose), or simply passing a flag/config for that use-case?
If the only reason is that running on the same machine use-case/edge-case, I don't think it's a good idea to change the default behavior really.

[Wondertan]

We agreed with @renaynay that making each node type a different binary starts to make sense already. If we agree on the former, using different default ports for different apps starts to make even more sense.

[liamsi]

Interesting, so that whole idea of being able to turn a running light node into full node and vice versa is off the table (it was a rather vague long-term goal anyways)?

[renaynay]

What? I didn't agree to this @Wondertan . I am in favour of everything being the same binary. It doesn't make sense at all yet to make them different. People shouldn't be running full and light nodes on the same machine - this is not the point of "different node types". And if they do choose to run both nodes on their machine, then they should change the p2p port by passing a flag --p2p.port < port >. This is a super small use case that you are tailoring for, and I am against this.

[Wondertan]

We synced up on the concern above with @renaynay and figured out that this fleeting discussion was somewhere during offsite without a clear decision and we decided to start a conversation about multiple binaries again.

[Wondertan]

I am promoting this to a discussion about multiple binaries, as this is still a concern and we don't have an agreement on this side.

Dumping informative pieces from the private conversation with some irrelevant msgs omitted:

Wondertan: what should happen if there is one type running already and we start an another type on the same machine?

Rene: Should fail due to port overlap
Rene: It should fail.

Wondertan: And do you think this is normal?

Rene: This is 100% normal, there is no reason that a user should run two different nodes on the same machine without knowledge of node infra. It’s not the intended use of the software, so it should throw this error.

Wondertan: And this is the reason why those things should as different binaries

Rene: They don’t have to be different binaries. We could make the process more explicit.

Rene: I understand the problem you’re trying to address here, but, the reality is:

• most people who are unfamiliar with running node software will follow detailed documentation any way by copying and pasting commands, which will prevent them from running into this issue
• most people familiar with running node infra will understand port in use error and will take a look at the commands they are actually running
  ./celestia light init then ./celestia full start will fail as no ~/.celestia-full repo is initialised (assuming this person hasn’t init’d a full node)
  Rene: by making it more explicit, I mean we could make the commands less confusing (less steps). We could bundle initialisation and startup for those who just want to run a node with default settings.
  Rene: (without start or init) and we could leave start and init as additional configurability options
  Rene: There are ways around this that don’t require making separate binaries.

Wondertan: 1000%, but what's the point of making them one binary???

Rene: What’s the point of separating them? Other than this nebulous issue?

Wondertan: Look. There is standard practice in the whole Unix world. Keep different processes and daemons in different binaries. You won't give me an example of an application that would do that. Therefore, the should be a real reason not to follow standards and not vice versa.

Rene: There are several reasons:

1. if we want to work towards the runtime switchability vision, we should maintain one binary - light nodes and full nodes are basically the same thing with an additional service involved which IMO does not qualify as a “separate process” nor will they ever run simultaneously on the same machine
2. it’s additional work for us to separate them (which I don’t see as important at the moment)
3. Tendermint does not follow those standards - we can choose to break cohesion with core or we can maintain the same CLI style
   tendermint init [full|validator|seed]

Wondertan:

1. The bridge node is not the same, and at least should be separated.
2. This is not a reason, but a consequence of originally wrong choice.
3. Init != Start. The Tendermint case is a different story. Their init variations are basically configuration profiles and not node types with the different component layouts.

Rene:

1. bridge node is a bit different and I’d be more open to creating a separate binary for that. But full and light stay the same.
2. Yes it’s an additional reason, it’s called timesink + opportunity cost.
3. This doesn’t matter in the perspective of the user. They don’t know what’s happening under the hood, they just see 3 different options for init

Wondertan:

1. Finally, some movements from a dead point
2. I am not telling you we should fix this now. I am telling our current approach is unreasonable.
3. I can't say for tendermint, but in our case, it matters from the perspective of the user. Accidentally running the wrong type may lead to undefined behavior. If our binary allows running light node and then full node one after another, then we should test that those things would not break each other. It's like testing if clicking different buttons would not break things.
   Also, if we want to have dynamic switchability, we should have only one repository/store on init and not different dirs(celestia-light , celestia-full), so start for different types/modes uses the same folder. So my last point here, is that unless we don't do one common repo/store(.celestia) per light and full , there is no point to have one binary.

[Wondertan]

Status quo

Even though we have a single binary, we:

• Have different repository/store for each Type(.celestia-light and .celestia-full)
• Allow running multiple daemons from binary

Unless we start using one shared .celestia repo/store, there is no point in keeping the binaries not logically separated. If we go that direction, then we would have to test that running different node types one after another does not break anything. Because if one binary allows you to do so, then eventually some user will do it and break things.

My main point here is to keep things logically separated until we don't have some kind of runtime switchability of services. I hope at least it makes sense to separate the Bridge node from others, but it would look deeper it starts to make sense to segregate Full from Light as well.

The question to everyone, if keep one binary for everything. How do imagine commands being part of it?
Getting all the peers - celestia full net peers or celestia net peers.:

• If the first case, then there is no need to have one binary, again, all the commands are just duplicated.
• if the second, how we should understand what node type to contact?
  Also, how we would process commands related to some node type specifically?

Special note for @liamsi. Having multiple binaries per node Type is not the same as having multiple binaries that need to be run at the same time for things to work, e.g. app/node/dalc/optimint stack.

[renaynay]

I don't want to spend too much time thinking about this issue during this iteration as it's not really relevant to what we're working on right now, but I want to make one thing clear:

The CLI should only be used to configure and start a node, not necessarily to access any kind of runtime data -- this is specifically left up RPC.

It is incorrect to start building CLI architecture for the idea that someone is going to be running multiple node types on their machines -- if they choose to do so, that is their problem and they will have to configure their nodes (and repositories) such that they don't conflict. That is a devops issue that we do not have to design for, other than making the nodes highly configurable.

[Wondertan]

It is incorrect to start building CLI architecture for the idea that someone is going to be running multiple node types on their machines.

@renaynay, Totally true and this is why we should split different types into multiple binaries, so you download/build only one that is relevant to you. In case you have one binary allowing you to start multiple things it is implied that you can start multiple things and we should avoid that!

[Bidon15]

Grooming 01/11/2022:

We need to revisit this discussion post-mainnet