k8s-int-test-build: zk-less druid cluster and http based segment/task managment #10686
Conversation
himanshug
changed the title
| ### Gotchas | ||
|
|
||
| - Label/Annotation path in each pod spec MUST EXIST, which is easily satisfied if there is at least one label/annotation in the pod spec already. This limitation may be removed in future. | ||
| - Druid Pods need permissions to be able to add labels to self-pod, List and Watch other Pods, create ConfigMap for leader election. Assuming, "default" service account is used by Druid pods, you might need to add following or something similar Kubernetes Role and Role Binding. |
There was a problem hiding this comment.
Just curious. Which kind of druid server needs these add labels to self-pod, List and Watch other Pods, create ConfigMap permissions? MiddleManager or all the druid servers.
There was a problem hiding this comment.
everyone needs pod list/update .
create/update ConfigMap is needed only by coordinators/overlords which participate in leader election ... but read ConfigMap is needed by all so as to know who the leader is. That said, I haven't tested such granularity and would just give same permission to all pods.
|
|
||
| ## Debug And FastFail | ||
|
|
||
| sudo /usr/local/bin/kubectl get pod |
There was a problem hiding this comment.
nit: /usr/local/bin/kubectl could be replaced by $KUBECTL. The same as sudo /usr/local/bin/kubectl get svc
| export MINIKUBE_WANTREPORTERRORPROMPT=false | ||
| export MINIKUBE_HOME=$HOME | ||
| export KUBECONFIG=$HOME/.kube/config | ||
| export KUBECTL="sudo /usr/local/bin/kubectl" |
There was a problem hiding this comment.
nit: It seems that export KUBECTL="sudo /usr/local/bin/kubectl" is never used in setup_k8s_cluster.sh . Maybe can be removed?
|
@zhangyue19921010 thanks for taking a look |
|
LTGM :) |
There was a problem hiding this comment.
🎉
As a follow-up we should try to get some or all of https://github.com/apache/druid/blob/master/integration-tests/src/test/java/org/apache/druid/tests/leadership/ITHighAvailabilityTest.java running on this configuration.
Hi @clintropolis and @himanshug Are there any progress being made on that? If no, I'd like to take the chance to implement it after this PR have been merged. :) |
|
@clintropolis thanks for approving, I will do one more round of checks of things in this PR tomorrow and merge.
@zhangyue19921010 I did some work there and will finish it, I am adding few more related things in that PR . that work currently lives in my fork at https://github.com/himanshug/druid/tree/k8s_build2 |
… managment (apache#10686) * zk-less druid cluster in k8s build * attempt to fix build and use http based remote task management * mm/router logs for debugging * add default account k8s role and binding for pod, configMap access * fix issue * change router port to 8088 for common readinessProbe * break build_run_k8s_cluster.sh into separate scripts * revert changes to K8sDruidNodeAnnouncer.java * k8s extension doc update * add license to new file * address review comments * do not try to load lookups at startup to improve cluster startup time
Linked #10542
Description
Follow up on #10669
Updates K8S based integration test build to deploy zk-less Druid cluster along with http based segment and remote task management to sanity test kubernetes based discovery druid extension to run Druid on K8S without Zookeeper #10544
Splits https://github.com/apache/druid/blob/1884c356984695cf31ef63dd0075602b5087ec4b/integration-tests/script/build_run_k8s_cluster.sh into 3 separate scripts , "setup_k8s_cluster.sh", "setup_druid_operator_on_k8s.sh" and "setup_druid_on_k8s.sh" for easily running specific parts while debugging setup or test failures locally.
Updates to k8s extension documentation for few gotchas.