Adding a Unified web console. #6923

vogievetsky · 2019-01-28T03:07:18Z

This PR implements the #6832 proposal.

From a technical point of view this creates a single unified console that leverages system tables via SQL. This console queries both the overlord and the coordinator meaning that is has to run on a node that can route requests to both. In this PR it is expected to run on the router node (part 1) of the proposal. In part (2) this console will also be placed on the coordinator.

This console utilises TypeScript and React and is built in such a way as to be maximally simple and maintainable by non web developers while still being slick and powerful.

In an effort to consolidate this PR also pulls into itself the legacy consoles. Their functionality is still fully available for anyone to use.

There is also a handy tab for running SQL (and native) queries without using curl:

nishantmonu51 · 2019-01-28T12:54:02Z

@vogievetsky: Any additional steps needed to deploy it or just running the druid cluster will deploy it ?
What is the URL to access the new console ?

gianm · 2019-01-28T13:31:53Z

@vogievetsky Something went wrong during the CI build: https://travis-ci.org/apache/incubator-druid/jobs/485215413. Any idea what's going on there? It looks like it logs Webpacking everything... and then the job fails.

dclim · 2019-01-28T15:51:29Z

web-console/src/console-application.tsx

+          It appears that the SQL endpoint is disabled. Either <a
+          href="http://druid.io/docs/latest/querying/sql.html">enable the SQL endpoint</a> or use the old <a
+          href="/legacy-coordinator-console.html">coordinator</a> and <a
+          href="/legacy-overlord-console.html">overlord</a> consoles that do not reply on the SQL endpoint.


typo: reply -> rely

vogievetsky · 2019-01-28T18:49:45Z

Looks like this is the issue messing up the CI:

I'm looking into this

jihoonson · 2019-01-29T02:42:19Z

The Tasks page shows both supervisors and tasks. How about calling Indexing Jobs instead of Tasks? I'm not sure it's the best term though.

jon-wei

Left some initial comments, will try out the UI and continue review

jon-wei · 2019-01-30T01:34:30Z

pom.xml

+                                    <licenseFamilyName>Font Awesome</licenseFamilyName>
+                                    <notes></notes>
+                                    <patterns>
+                                        <pattern>Font Awesome 4.2.0 by @davegandy - http://fontawesome.io - @fontawesome</pattern>


Should we use the newer 4.7.0 version or a 5.x version instead?

No, Font Awesome is being used by the legacy coordinator console. It was copied here without changes. It is out of scope for this project to make any kind of modification to the code of the original coordinator console

jon-wei · 2019-01-30T01:35:10Z

pom.xml

+                                    <notes></notes>
+                                    <patterns>
+                                        <pattern>Font Awesome 4.2.0 by @davegandy - http://fontawesome.io - @fontawesome</pattern>
+                                        <pattern>License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License)</pattern>


This link looks like the pro license, maybe better to use https://github.com/FortAwesome/Font-Awesome/blob/master/LICENSE.txt for the free license

I am not sure how that works, this is matching what was in the font awesome file. I think this is an old version of FA, the link might have changed.

Hmmmm, let's use the github link with the free version license then

jon-wei · 2019-01-30T01:42:40Z

web-console/README.md

+1. Install the modules with `npm install`
+2. Run `npm start` will start in development mode and will proxy druid requests to `localhost:8888`
+
+Note: you can provide and environment variable to proxy to a different Druid host like so: `druid_host=1.2.3.4:8888 npm start` 


and environment -> an environment

jon-wei · 2019-01-30T01:48:08Z

web-console/README.md

+
+## How to watch and run for development
+
+1. Install the modules with `npm install`


let's note here that you need to be in the web-console directory when running npm install

jon-wei · 2019-01-30T01:58:08Z

web-console/src/dialogs/coordinator-dynamic-config.tsx

+    let newState: any = this.state.config;
+    const whiteList: any[] = newState["killDataSourceWhitelist"];
+    const skipList: any[] = newState["killPendingSegmentsSkipList"];
+    //newState["killDataSourceWhitelist"] = newState["killDataSourceWhitelist"].join(",");


let's remove the unused code

jon-wei · 2019-01-30T06:44:32Z

In the Supervisors view, can you add a per-supervisor link to /druid/indexer/v1/supervisor/<supervisor-id>/stats

In the Tasks view, can you add a per-task link to /druid/indexer/v1/task/<task-id>/reports?

These endpoints expose some information about ingestion row counts and errors: http://druid.io/docs/latest/ingestion/reports.html

clintropolis

A lot of web stuff so hard to review it individually so I checked out this branch and just ran everything. I can think of lots of things missing, but it seems like nice bones to build on top of, and looks slick.

I do think some sort of graceful degradation mode would probably be useful, since if your brokers go out you are basically blind with this new UI, but I don't think belongs in this PR, and may also be better filled or at least augmented by adding some additional web apps that replace the old ui's and just contain stuff specific to that service type, re-using some of the controls that are laid out here where possible.

Old ui's appear to work, but there may be some direct links broken since some of the static resources have different names, not sure how big of deal that is.

Consider adding this snippet to .idea/misc.xml

  <component name="JavaScriptSettings">
    <option name="languageLevel" value="ES6" />
  </component>

to make intellij not complain about javascript syntax.

clintropolis · 2019-01-30T10:22:12Z

server/pom.xml

@@ -47,6 +47,11 @@
            <artifactId>druid-console</artifactId>


Is the old io.druid dependency still needed?

jihoonson · 2019-01-31T18:34:33Z

@gianm I changed those file names because they are at the same directory level with the new router UI. It's not easy to figure out what file's for overlord, coordinator, or router. Also, the coordinator console file name was index.html which now conflicts with the router one. Do you think we should keep the coordinator console file name and rename the router one?

jihoonson · 2019-01-31T18:35:16Z

@clintropolis thanks for checking it. I also checked it, but wasn't sure it's worth to add a new dependency..

gianm · 2019-01-31T18:37:50Z

@jihoonson The new names on the router make sense, but the old ones should continue to work on the overlord/coordinator so bookmarks are still valid. IMO it could make sense to either put both the old/new everywhere, or to put the new names on the router only, and keep the old names the same on the overlord/coordinator for now.

jihoonson · 2019-01-31T20:04:14Z

There was a bug for the coordinator: http://{COORDINATOR_ADDRESS} shows the old coordinator console instead of the current one. I fixed it in the latest change.

- Add ExitOnOutOfMemoryError to jvm.config examples. It was added a pretty long time ago (8u92) and is helpful since it prevents zombie processes from hanging around. (OOMEs tend to bork things) - Disable Broker caching and enable it on Historicals in example configs. This config tends to scale better since it enables the Historicals to merge results rather than sending everything by-segment to the Broker. Also switch to "caffeine" cache from "local". - Increase concurrency a bit for Broker example config. - Enable SQL in the example config, a baby step towards making SQL more of a thing. (It's still off by default in the code.) - Reduce memory use a bit for the quickstart configs. - Add example Router configs, in case someone wants to use that. One reason might be to get the fancy new console (apache#6923).

jihoonson · 2019-01-31T20:27:24Z

I also renamed coordinator-console.html to index.html and removed the term legacy from docs.

jon-wei

LGTM

gianm

Thanks for fixing up the current URLs. The rest 👍

gianm · 2019-02-01T01:04:30Z

server/pom.xml

@@ -43,9 +43,9 @@
            <version>${project.parent.version}</version>
        </dependency>
        <dependency>
-            <groupId>io.druid</groupId>
+            <groupId>org.apache.druid</groupId>


* Some adjustments to config examples. - Add ExitOnOutOfMemoryError to jvm.config examples. It was added a pretty long time ago (8u92) and is helpful since it prevents zombie processes from hanging around. (OOMEs tend to bork things) - Disable Broker caching and enable it on Historicals in example configs. This config tends to scale better since it enables the Historicals to merge results rather than sending everything by-segment to the Broker. Also switch to "caffeine" cache from "local". - Increase concurrency a bit for Broker example config. - Enable SQL in the example config, a baby step towards making SQL more of a thing. (It's still off by default in the code.) - Reduce memory use a bit for the quickstart configs. - Add example Router configs, in case someone wants to use that. One reason might be to get the fancy new console (#6923). * Add example Router configs. * Fix up router example properties. * Add router to quickstart supervise conf.

* Adding new web console. * fixed css * fix form height * fix typo * do import custom react-table css * added repo field so npm does not complain * ask travis for node 10 * move indexing-service/src/main/resources/indexer_static into web-console * fix resource names and paths * add licenses * fix exclude file * add licenses to misc files and tidy up * remove rebase marker * fix link * updated env variable name * tidy up licenses and surface errors * cleanup * remove unused code, fix missing await * TeamCity does not like the name aux * add more links to tasks view * rm pages * update gitignore * update readme to be accurate * make clean script * removed old console dependancy * update Jetty routes * add a comment for welcome files for coordinator * do not show inital notifaction for now * renamed overlord console back to console.html * fix coordinator console * rename coordinator-console.html to index.html

* Some adjustments to config examples. - Add ExitOnOutOfMemoryError to jvm.config examples. It was added a pretty long time ago (8u92) and is helpful since it prevents zombie processes from hanging around. (OOMEs tend to bork things) - Disable Broker caching and enable it on Historicals in example configs. This config tends to scale better since it enables the Historicals to merge results rather than sending everything by-segment to the Broker. Also switch to "caffeine" cache from "local". - Increase concurrency a bit for Broker example config. - Enable SQL in the example config, a baby step towards making SQL more of a thing. (It's still off by default in the code.) - Reduce memory use a bit for the quickstart configs. - Add example Router configs, in case someone wants to use that. One reason might be to get the fancy new console (apache#6923). * Add example Router configs. * Fix up router example properties. * Add router to quickstart supervise conf.

aleksi75 · 2019-03-08T13:47:17Z

I am looking forward to unified web console 👍

Is there a possibility to set credentials, eg. login and pwd? At the moment, everybody how knows the url ist able to stop/suspend even kill eg. kafka supervisors which is a little bit painful.

gianm · 2019-03-12T01:01:17Z

@aleksi75 The underlying APIs the console uses can be protected, and if you do so, then valid credentials will be needed in order to do anything useful with the console.

leventov · 2019-07-24T17:22:19Z

@vogievetsky do you plan to work on part (2) - allowing to access the unified console on Coordinator?

leventov · 2019-07-24T17:27:17Z

Opened #8150 to track this.

Adding new web console.

172d160

fjy added this to the 0.14.0 milestone Jan 28, 2019

fixed css

9b5e04e

dclim reviewed Jan 28, 2019

View reviewed changes

vogievetsky added 4 commits January 28, 2019 09:29

fix form height

436abbd

fix typo

6ec7817

do import custom react-table css

6f01b79

added repo field so npm does not complain

d72979e

vogievetsky and others added 3 commits January 28, 2019 10:58

ask travis for node 10

0f56a26

move indexing-service/src/main/resources/indexer_static into web-console

44b718c

fix resource names and paths

2179192

jihoonson and others added 5 commits January 28, 2019 19:12

add licenses

05e03fe

fix exclude file

f613aab

add licenses to misc files and tidy up

be4e173

remove rebase marker

ee01390

fix link

4cdc8e3

jihoonson mentioned this pull request Jan 29, 2019

Print files with unapproved licenses in travis #6947

Merged

eva and others added 4 commits January 29, 2019 13:16

updated env variable name

aae5c00

tidy up licenses and surface errors

0c46198

cleanup

d767849

Merge branch 'master' of github.com:druid-io/druid into console

a735d21

jon-wei reviewed Jan 30, 2019

View reviewed changes

vogievetsky added 2 commits January 29, 2019 20:18

remove unused code, fix missing await

e26225d

TeamCity does not like the name aux

240fbbc

clintropolis reviewed Jan 30, 2019

View reviewed changes

server/pom.xml Outdated

@@ -47,6 +47,11 @@

<artifactId>druid-console</artifactId>

Copy link

Member

clintropolis Jan 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the old io.druid dependency still needed?

fix coordinator console

bf0564d

gianm mentioned this pull request Jan 31, 2019

Some adjustments to config examples. #6973

Merged

rename coordinator-console.html to index.html

ba84dd4

jon-wei approved these changes Feb 1, 2019

View reviewed changes

gianm approved these changes Feb 1, 2019

View reviewed changes

gianm merged commit 7f1b19b into apache:master Feb 1, 2019

vogievetsky mentioned this pull request Feb 4, 2019

Fixing little issues and CSS bugs in the new web console. #6995

Merged

gianm mentioned this pull request Feb 19, 2019

[Proposal] "Unified" web console #6832

Closed

jon-wei mentioned this pull request Feb 22, 2019

0.14.0-incubating release notes #7126

Closed

vogievetsky mentioned this pull request Feb 25, 2019

Downgrade blueprintjs version in the web console to one with a vanilla Apache 2.0 license #7139

Merged

jon-wei mentioned this pull request Feb 27, 2019

Add web consoles doc page #7123

Merged

clintropolis added the Area - Web Console label Mar 18, 2019

vogievetsky mentioned this pull request Apr 1, 2019

Migrate the web console back to Blueprint v3 #7398

Merged

vogievetsky mentioned this pull request Apr 18, 2019

[Proposal] Data loader GUI #7502

Closed

leventov mentioned this pull request Jul 24, 2019

Make the Unified web console available on Coordinator #8150

Closed

vogievetsky deleted the console branch September 22, 2019 22:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding a Unified web console. #6923

Adding a Unified web console. #6923

vogievetsky commented Jan 28, 2019

nishantmonu51 commented Jan 28, 2019

gianm commented Jan 28, 2019

dclim Jan 28, 2019

vogievetsky commented Jan 28, 2019

jihoonson commented Jan 29, 2019

jon-wei left a comment

jon-wei Jan 30, 2019

vogievetsky Jan 30, 2019

jon-wei Jan 30, 2019

jon-wei Jan 30, 2019

vogievetsky Jan 30, 2019

jon-wei Jan 30, 2019

jon-wei Jan 30, 2019

jon-wei Jan 30, 2019

jon-wei Jan 30, 2019

jon-wei commented Jan 30, 2019

clintropolis left a comment

clintropolis Jan 30, 2019

jihoonson commented Jan 31, 2019 •

edited

Loading

jihoonson commented Jan 31, 2019

gianm commented Jan 31, 2019

jihoonson commented Jan 31, 2019

jihoonson commented Jan 31, 2019

jon-wei left a comment

gianm left a comment

gianm Feb 1, 2019

aleksi75 commented Mar 8, 2019

gianm commented Mar 12, 2019

leventov commented Jul 24, 2019

leventov commented Jul 24, 2019


		## How to watch and run for development

		1. Install the modules with `npm install`

Adding a Unified web console. #6923

Adding a Unified web console. #6923

Conversation

vogievetsky commented Jan 28, 2019

nishantmonu51 commented Jan 28, 2019

gianm commented Jan 28, 2019

Choose a reason for hiding this comment

vogievetsky commented Jan 28, 2019

jihoonson commented Jan 29, 2019

jon-wei left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jon-wei commented Jan 30, 2019

clintropolis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jihoonson commented Jan 31, 2019 • edited Loading

jihoonson commented Jan 31, 2019

gianm commented Jan 31, 2019

jihoonson commented Jan 31, 2019

jihoonson commented Jan 31, 2019

jon-wei left a comment

Choose a reason for hiding this comment

gianm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aleksi75 commented Mar 8, 2019

gianm commented Mar 12, 2019

leventov commented Jul 24, 2019

leventov commented Jul 24, 2019

jihoonson commented Jan 31, 2019 •

edited

Loading