Make the Unified web console available on Coordinator

[leventov]

An issue to track the task of making the Unified web console available on Coordinator. See #6832 (comment) and #6923 (comment).

FYI @b-slim @vogievetsky

[gianm]

I think this came up in the discussion, but it might also make sense on the Broker.

One thing I was thinking of proposing along those lines is folding the Router functionality into the Broker in general, even beyond the unified console. The rationale would be that the Router is a really thin process type, and its functionality would fit in well on the Broker and would make the Broker able to serve all your interactions with Druid. (In addition to serving queries it could also serve the web console and proxy to coordinator/overlord like the Router does today.)

[leventov]

A reason to make it available on Coordinator (rather than Router or Broker) is to separate Druid query interfaces and "management" interfaces on different nodes for some security purposes.

[vogievetsky]

@leventov are you picturing the coordinator version of the unified console would be a 'lite' version of the console without reliance on the DruidSQL route?

[leventov]

What do you mean by "reliance on the DruidSQL route"?

[vogievetsky]

Right now the console primarily uses the /druid/v2/sql route to query the sys.* tables via DruidSQL. Also that what the query view uses. The console has a mode where it runs with no SQL but it would still have the Query view that would let you run the native queries. Are you picturing a light mode of the console that never issues Druid SQL / native queries or are you thinking that the coordinator will route those queries to the broker?

[clintropolis]

Are you picturing a light mode of the console that never issues Druid SQL / native queries

I can't speak for @leventov, but that is what I am hoping for in the long run; basically replacements for the legacy coordinator and overlord consoles using views from the unified console. Then we can

• remove all of the old consoles from the codebase
• have more opportunity for visibility when the cluster is in a degraded state

I think the former is important to me from a development/code hygiene/maintenance standpoint, and Druid itself would seem much more coherent if it didn't have a bunch of random old web interfaces that look nothing like the new one scattered everywhere.

The latter feels important for large clusters, as the current console is sort of a happy path console. It does have some graceful degradation, but it still requires multiple nodes to be functioning for it to work. I think there is value in being able to still load the traditionally coordinator views directly from the coordinator, or the indexing views from the overlord, in cases where other parts of the cluster are down.

[leventov]

Are you picturing a light mode of the console that never issues Druid SQL / native queries or are you thinking that the coordinator will route those queries to the broker?

Light mode, I would say.

[leventov]

@gianm @vogievetsky does Imply has any plans to implement part 2 of #6923?

[gianm]

@gianm @vogievetsky does Imply has any plans to implement part 2 of #6923?

I would definitely like to make it available outside of just the Router. The tricky bit is that the unified console uses Coordinator APIs, Overlord APIs, and Broker APIs. It currently runs on the Router because that is the only process type that can make all three kinds of APIs available (it proxies to them all).

I think one of the following plans makes the most sense,

1. Do it on the Coordinator, perhaps by falling back to a 'light' mode that only uses Coordinator and Overlord APIs. Not all functionality would be available. In particular, query functionality would not be available, as well as anything that relies on system tables.
2. Do it on the Broker, perhaps by folding Router functionality into the Broker (druid.broker.asRouter = true?). This would allow the Broker to proxy to the Coordinator and Overlord APIs.

When I think about this right now I lean to the second idea (Broker) since it promotes the idea of a single server type that you use to interact with your Druid cluster. It feels like a simpler experience for especially new users.

[gianm]

Do it on the Broker, perhaps by folding Router functionality into the Broker (druid.broker.asRouter = true?). This would allow the Broker to proxy to the Coordinator and Overlord APIs.

This could also involve adding a management proxy to the Broker, rather than full-on Router functionality.

[gianm]

It may also be valuable to add the console to… both? I think they serve different goals.

Coordinator: serves the goal of deprecating / removing the old consoles.

Broker: serves the goal of having a single server type that you use to interact with the Druid cluster.