Do Integrate test for Druid base on K8s cluster #10669

zhangyue19921010 · 2020-12-11T08:31:32Z

Description

As the relationship between Druid and K8s gets closer, maybe it is good to have a IT job to check the running status of Druid on K8s like #10544

This PR add a new travis job to do simple integrate test on K8s, using Minikube + druid-operator + single-server
What I have done are as follows :

Launched/Destroyed a Druid Cluster on Minikube locally with a script.
2.Tested ingest -> load segment -> query. And it works fine.

Although It is in WIP
There are some bugs need to be fixed, such as

Permissions issue
Integration with travis: It seems that I can't set hostNetWork in statefuleset using druid-operator. So I have to use NodePort Service to expose Druid service like router, middleManager and so on but only can use ports 30000-32767.
So I perform integration tests in int-tests-config-file mentioned https://github.com/apache/druid/blob/master/integration-tests/README.md
Pod resource adjust including disks.

Any help is greatly appreciated.

Updated:

This PR add a new IT Job 71:

Deploy a K8s cluster based on minikube.
Deploy Druid cluster on K8s: single Druid, use local disk as Deep Storage and use derby as MetaStorage.
Run single IT test ITNestedQueryPushDownTest which includes data ingestion, Historical loading and query action.

Now job 71 is successful which means this PR is not WIP anymore!

This PR has:

been self-reviewed.
- using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
added documentation for new or modified features or behaviors.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added or updated version, license, or notice information in licenses.yaml
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
added integration tests.
been tested in a test Druid cluster.

Key changed/added classes in this PR

.travis.yml

zhangyue19921010 · 2020-12-15T04:17:04Z

re-start to trigger CI again.

zhangyue19921010 · 2020-12-15T05:45:10Z

This PR add a new IT Job 71:

Deploy a K8s cluster based on minikube.
Deploy Druid cluster on K8s: single Druid, use local disk as Deep Storage and use derby as MetaStorage
Run single IT test ITNestedQueryPushDownTest which includes data ingestion, Historical loading and query action.
Now job 71 is successful which means this PR is not WIP anymore!

himanshug

Thanks, this is great start for testing k8s stuff.

himanshug · 2020-12-15T22:11:20Z

integration-tests/script/build_run_k8s_cluster.sh

+cp integration-tests/tiny-cluster.yaml druid-operator/examples/
+cp integration-tests/tiny-cluster-zk.yaml druid-operator/examples/


nit: not sure why we need to copy, can't we directly use those two files ?

There are some changes in integration-tests/tiny-cluster.yaml and integration-tests/tiny-cluster-zk.yaml, such as:

New NodeType Service.

Configs changing.

Add MiddleManager configs.

So the original files in druid-operator can't be used directly here.

sorry, I wasn't clear, in this context "original" meant files in "integration-tests/" i.e. you have

cp integration-tests/tiny-cluster.yaml druid-operator/examples/ cp integration-tests/tiny-cluster-zk.yaml druid-operator/examples/ ... sed -i "s|REPLACE_VOLUMES|`pwd`|g" druid-operator/examples/tiny-cluster.yaml .. sudo /usr/local/bin/kubectl apply -f druid-operator/examples/tiny-cluster-zk.yaml sudo /usr/local/bin/kubectl apply -f druid-operator/examples/tiny-cluster.yaml

as opposed to

... sed -i "s|REPLACE_VOLUMES|`pwd`|g" integration-tests/tiny-cluster.yaml .. sudo /usr/local/bin/kubectl apply -f integration-tests/tiny-cluster-zk.yaml sudo /usr/local/bin/kubectl apply -f integration-tests/tiny-cluster.yaml

regardless, it is a nit and doesn't really matter much.

himanshug · 2020-12-15T22:12:21Z

integration-tests/script/build_run_k8s_cluster.sh

+# Prepare For Druid-Operator
+git clone https://github.com/druid-io/druid-operator.git
+cd druid-operator
+git checkout -b druid-operator-0.0.3 druid-operator-0.0.3


nit: maybe define DRUID_OPERATOR_VERSION=0.0.3 ?

Sure thing. Done!

himanshug · 2020-12-15T22:25:42Z

distribution/docker/DockerfileBuildTarAdvanced

@@ -0,0 +1,62 @@
+#


nit: this looks mostly a copy of other Dockerfile , it would be nice to reuse same if possible.

Also. There are some changes between DockerfileBuildTarAdvanced and original Dockerfile, such as :

Prepare test data.

Move mvn package out of Dockerfile.

Mkdir necessary for IT.

So that we can't use the original Dockerfile here.

zhangyue19921010 · 2020-12-16T03:03:23Z

@himanshug Thanks for your review!
Job 71 is successful after changing as you suggested, but 48, 55, 65 are failed unexpected. I am pretty sure these failures are not this PR related.
Will trigger the CI again soon.

Updated:

CI is successful. PTAL :)

zhangyue19921010 · 2020-12-17T09:53:44Z

@himanshug Thanks for your review and merge!

* add a travls job to do integrate test on K8s * revert build_run_cluster.sh * revert msic * run IT test * ready to test * modify before/after script * done * change mod for script * done * add env DRUID_OPERATOR_VERSION=0.0.3 * change version Co-authored-by: yuezhang <yuezhang@freewheel.tv>

yuezhang added 9 commits December 11, 2020 15:49

add a travls job to do integrate test on K8s

9b3a2ad

revert build_run_cluster.sh

3ca937f

revert msic

1183b70

Merge branch 'master' into intergrate-test-base-on-K8s

f215f79

run IT test

29f1d40

ready to test

2aef19d

modify before/after script

34901b2

done

3a57cf2

Merge branch 'master' into intergrate-test-base-on-K8s

dec7c64

zhangyue19921010 force-pushed the intergrate-test-base-on-K8s branch from 7a9fa17 to dec7c64 Compare December 14, 2020 13:36

change mod for script

389d890

zhangyue19921010 closed this Dec 15, 2020

zhangyue19921010 reopened this Dec 15, 2020

Merge branch 'master' into intergrate-test-base-on-K8s

5d82d88

zhangyue19921010 changed the title ~~[WIP] Do Integrate test base on K8s~~ Do Integrate test for Druid base on K8s cluster Dec 15, 2020

zhangyue19921010 mentioned this pull request Dec 15, 2020

integration test for coordinator and overlord leadership client #10680

Merged

9 tasks

clintropolis added the Area - Testing label Dec 15, 2020

zhangyue19921010 mentioned this pull request Dec 15, 2020

Create a travis build that runs Druid on a K8S Cluster and runs the integration tests #10542

Open

yuezhang added 2 commits December 15, 2020 16:10

Merge branch 'master' into intergrate-test-base-on-K8s

adf69ab

done

b7443be

himanshug self-requested a review December 15, 2020 16:32

himanshug approved these changes Dec 15, 2020

View reviewed changes

add env DRUID_OPERATOR_VERSION=0.0.3

375c253

change version

1a42794

himanshug merged commit 1884c35 into apache:master Dec 17, 2020

himanshug mentioned this pull request Dec 17, 2020

k8s-int-test-build: zk-less druid cluster and http based segment/task managment #10686

Merged

jihoonson added this to the 0.21.0 milestone Jan 4, 2021

zhangyue19921010 deleted the intergrate-test-base-on-K8s branch February 9, 2021 08:42

himanshug mentioned this pull request May 5, 2021

kubernetes-druid-discovery-extension: stress-test overlord/coordinator leadership changes #11205

Closed

9 tasks

gianm mentioned this pull request Aug 15, 2022

Promote druid-kubernetes-extensions out of experimental status #12904

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Do Integrate test for Druid base on K8s cluster #10669

Do Integrate test for Druid base on K8s cluster #10669

zhangyue19921010 commented Dec 11, 2020 •

edited

Loading

zhangyue19921010 commented Dec 15, 2020

zhangyue19921010 commented Dec 15, 2020

himanshug left a comment

himanshug Dec 15, 2020

zhangyue19921010 Dec 16, 2020

himanshug Dec 16, 2020

himanshug Dec 15, 2020

zhangyue19921010 Dec 16, 2020

himanshug Dec 15, 2020

zhangyue19921010 Dec 16, 2020 •

edited

Loading

zhangyue19921010 commented Dec 16, 2020 •

edited

Loading

zhangyue19921010 commented Dec 17, 2020

		cp integration-tests/tiny-cluster.yaml druid-operator/examples/
		cp integration-tests/tiny-cluster-zk.yaml druid-operator/examples/

Do Integrate test for Druid base on K8s cluster #10669

Do Integrate test for Druid base on K8s cluster #10669

Conversation

zhangyue19921010 commented Dec 11, 2020 • edited Loading

Description

Updated:

Key changed/added classes in this PR

zhangyue19921010 commented Dec 15, 2020

zhangyue19921010 commented Dec 15, 2020

himanshug left a comment

Choose a reason for hiding this comment

himanshug Dec 15, 2020

Choose a reason for hiding this comment

zhangyue19921010 Dec 16, 2020

Choose a reason for hiding this comment

himanshug Dec 16, 2020

Choose a reason for hiding this comment

himanshug Dec 15, 2020

Choose a reason for hiding this comment

zhangyue19921010 Dec 16, 2020

Choose a reason for hiding this comment

himanshug Dec 15, 2020

Choose a reason for hiding this comment

zhangyue19921010 Dec 16, 2020 • edited Loading

Choose a reason for hiding this comment

zhangyue19921010 commented Dec 16, 2020 • edited Loading

zhangyue19921010 commented Dec 17, 2020

zhangyue19921010 commented Dec 11, 2020 •

edited

Loading

zhangyue19921010 Dec 16, 2020 •

edited

Loading

zhangyue19921010 commented Dec 16, 2020 •

edited

Loading