-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding a Unified web console. #6923
Conversation
@vogievetsky: Any additional steps needed to deploy it or just running the druid cluster will deploy it ? |
@vogievetsky Something went wrong during the CI build: https://travis-ci.org/apache/incubator-druid/jobs/485215413. Any idea what's going on there? It looks like it logs |
It appears that the SQL endpoint is disabled. Either <a | ||
href="http://druid.io/docs/latest/querying/sql.html">enable the SQL endpoint</a> or use the old <a | ||
href="/legacy-coordinator-console.html">coordinator</a> and <a | ||
href="/legacy-overlord-console.html">overlord</a> consoles that do not reply on the SQL endpoint. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo: reply -> rely
The |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left some initial comments, will try out the UI and continue review
pom.xml
Outdated
<licenseFamilyName>Font Awesome</licenseFamilyName> | ||
<notes></notes> | ||
<patterns> | ||
<pattern>Font Awesome 4.2.0 by @davegandy - http://fontawesome.io - @fontawesome</pattern> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we use the newer 4.7.0 version or a 5.x version instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, Font Awesome is being used by the legacy coordinator console. It was copied here without changes. It is out of scope for this project to make any kind of modification to the code of the original coordinator console
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, got it
pom.xml
Outdated
<notes></notes> | ||
<patterns> | ||
<pattern>Font Awesome 4.2.0 by @davegandy - http://fontawesome.io - @fontawesome</pattern> | ||
<pattern>License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License)</pattern> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This link looks like the pro license, maybe better to use https://github.com/FortAwesome/Font-Awesome/blob/master/LICENSE.txt for the free license
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure how that works, this is matching what was in the font awesome file. I think this is an old version of FA, the link might have changed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmmmm, let's use the github link with the free version license then
web-console/README.md
Outdated
1. Install the modules with `npm install` | ||
2. Run `npm start` will start in development mode and will proxy druid requests to `localhost:8888` | ||
|
||
Note: you can provide and environment variable to proxy to a different Druid host like so: `druid_host=1.2.3.4:8888 npm start` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and environment -> an environment
web-console/README.md
Outdated
|
||
## How to watch and run for development | ||
|
||
1. Install the modules with `npm install` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's note here that you need to be in the web-console
directory when running npm install
let newState: any = this.state.config; | ||
const whiteList: any[] = newState["killDataSourceWhitelist"]; | ||
const skipList: any[] = newState["killPendingSegmentsSkipList"]; | ||
//newState["killDataSourceWhitelist"] = newState["killDataSourceWhitelist"].join(","); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's remove the unused code
In the Supervisors view, can you add a per-supervisor link to In the Tasks view, can you add a per-task link to These endpoints expose some information about ingestion row counts and errors: http://druid.io/docs/latest/ingestion/reports.html |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A lot of web stuff so hard to review it individually so I checked out this branch and just ran everything. I can think of lots of things missing, but it seems like nice bones to build on top of, and looks slick.
I do think some sort of graceful degradation mode would probably be useful, since if your brokers go out you are basically blind with this new UI, but I don't think belongs in this PR, and may also be better filled or at least augmented by adding some additional web apps that replace the old ui's and just contain stuff specific to that service type, re-using some of the controls that are laid out here where possible.
Old ui's appear to work, but there may be some direct links broken since some of the static resources have different names, not sure how big of deal that is.
Consider adding this snippet to .idea/misc.xml
<component name="JavaScriptSettings">
<option name="languageLevel" value="ES6" />
</component>
to make intellij not complain about javascript syntax.
server/pom.xml
Outdated
@@ -47,6 +47,11 @@ | |||
<artifactId>druid-console</artifactId> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the old io.druid
dependency still needed?
@gianm I changed those file names because they are at the same directory level with the new router UI. It's not easy to figure out what file's for overlord, coordinator, or router. Also, the coordinator console file name was |
@clintropolis thanks for checking it. I also checked it, but wasn't sure it's worth to add a new dependency.. |
@jihoonson The new names on the router make sense, but the old ones should continue to work on the overlord/coordinator so bookmarks are still valid. IMO it could make sense to either put both the old/new everywhere, or to put the new names on the router only, and keep the old names the same on the overlord/coordinator for now. |
There was a bug for the coordinator: |
- Add ExitOnOutOfMemoryError to jvm.config examples. It was added a pretty long time ago (8u92) and is helpful since it prevents zombie processes from hanging around. (OOMEs tend to bork things) - Disable Broker caching and enable it on Historicals in example configs. This config tends to scale better since it enables the Historicals to merge results rather than sending everything by-segment to the Broker. Also switch to "caffeine" cache from "local". - Increase concurrency a bit for Broker example config. - Enable SQL in the example config, a baby step towards making SQL more of a thing. (It's still off by default in the code.) - Reduce memory use a bit for the quickstart configs. - Add example Router configs, in case someone wants to use that. One reason might be to get the fancy new console (apache#6923).
I also renamed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for fixing up the current URLs. The rest 👍
@@ -43,9 +43,9 @@ | |||
<version>${project.parent.version}</version> | |||
</dependency> | |||
<dependency> | |||
<groupId>io.druid</groupId> | |||
<groupId>org.apache.druid</groupId> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice.
* Some adjustments to config examples. - Add ExitOnOutOfMemoryError to jvm.config examples. It was added a pretty long time ago (8u92) and is helpful since it prevents zombie processes from hanging around. (OOMEs tend to bork things) - Disable Broker caching and enable it on Historicals in example configs. This config tends to scale better since it enables the Historicals to merge results rather than sending everything by-segment to the Broker. Also switch to "caffeine" cache from "local". - Increase concurrency a bit for Broker example config. - Enable SQL in the example config, a baby step towards making SQL more of a thing. (It's still off by default in the code.) - Reduce memory use a bit for the quickstart configs. - Add example Router configs, in case someone wants to use that. One reason might be to get the fancy new console (#6923). * Add example Router configs. * Fix up router example properties. * Add router to quickstart supervise conf.
* Adding new web console. * fixed css * fix form height * fix typo * do import custom react-table css * added repo field so npm does not complain * ask travis for node 10 * move indexing-service/src/main/resources/indexer_static into web-console * fix resource names and paths * add licenses * fix exclude file * add licenses to misc files and tidy up * remove rebase marker * fix link * updated env variable name * tidy up licenses and surface errors * cleanup * remove unused code, fix missing await * TeamCity does not like the name aux * add more links to tasks view * rm pages * update gitignore * update readme to be accurate * make clean script * removed old console dependancy * update Jetty routes * add a comment for welcome files for coordinator * do not show inital notifaction for now * renamed overlord console back to console.html * fix coordinator console * rename coordinator-console.html to index.html
* Some adjustments to config examples. - Add ExitOnOutOfMemoryError to jvm.config examples. It was added a pretty long time ago (8u92) and is helpful since it prevents zombie processes from hanging around. (OOMEs tend to bork things) - Disable Broker caching and enable it on Historicals in example configs. This config tends to scale better since it enables the Historicals to merge results rather than sending everything by-segment to the Broker. Also switch to "caffeine" cache from "local". - Increase concurrency a bit for Broker example config. - Enable SQL in the example config, a baby step towards making SQL more of a thing. (It's still off by default in the code.) - Reduce memory use a bit for the quickstart configs. - Add example Router configs, in case someone wants to use that. One reason might be to get the fancy new console (apache#6923). * Add example Router configs. * Fix up router example properties. * Add router to quickstart supervise conf.
I am looking forward to unified web console 👍 Is there a possibility to set credentials, eg. login and pwd? At the moment, everybody how knows the url ist able to stop/suspend even kill eg. kafka supervisors which is a little bit painful. |
@aleksi75 The underlying APIs the console uses can be protected, and if you do so, then valid credentials will be needed in order to do anything useful with the console. |
@vogievetsky do you plan to work on part (2) - allowing to access the unified console on Coordinator? |
Opened #8150 to track this. |
This PR implements the #6832 proposal.
From a technical point of view this creates a single unified console that leverages system tables via SQL. This console queries both the overlord and the coordinator meaning that is has to run on a node that can route requests to both. In this PR it is expected to run on the router node (part 1) of the proposal. In part (2) this console will also be placed on the coordinator.
This console utilises TypeScript and React and is built in such a way as to be maximally simple and maintainable by non web developers while still being slick and powerful.
In an effort to consolidate this PR also pulls into itself the legacy consoles. Their functionality is still fully available for anyone to use.
There is also a handy tab for running SQL (and native) queries without using curl: