feat: support docker for loader #530

aroundabout · 2023-11-06T08:01:57Z

Purpose of the PR

support docker for hugegraph-loader

Main Changes

Dockerfile for loader
an example docker-compose for loader

usage

(add the usage to README and doc later)

current test image is dandelionivy/loader

cd hugegraph-loader/docker/example/
docker-compose up -d 
docker exec -it loader bash

sh bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -h graph -p 8080

It should be like:

user@LAPTOP-MRGBSP9S:~/incubator-hugegraph-toolchain/hugegraph-loader/docker/example$ sudo docker-compose up -d
Creating graph  ... done
Creating loader ... done
Creating hubble ... done
user@LAPTOP-MRGBSP9S:~/incubator-hugegraph-toolchain/hugegraph-loader/docker/example$ sudo docker ps
CONTAINER ID   IMAGE                 COMMAND                  CREATED              STATUS              PORTS                                         NAMES
59f431e622d0   hugegraph/hubble      "./bin/start-hubble.…"   About a minute ago   Up About a minute   0.0.0.0:8088->8088/tcp, :::8088->8088/tcp     hubble
55360d18f7d7   dandelionivy/loader   "/usr/bin/dumb-init …"   About a minute ago   Up About a minute                                                 loader
35b3feefc8bf   hugegraph/hugegraph   "/usr/bin/dumb-init …"   About a minute ago   Up About a minute   0.0.0.0:18081->8080/tcp, :::18081->8080/tcp   graph
user@LAPTOP-MRGBSP9S:~/incubator-hugegraph-toolchain/hugegraph-loader/docker/example$ sudo docker exec -it loader bash
root@55360d18f7d7:/loader# sh bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -h graph -p 8080
...
vertices/edges loaded this time : 8/6
--------------------------------------------------
count metrics
    input read success            : 14                  
    input read failure            : 0                   
    vertex parse success          : 8                   
    vertex parse failure          : 0                   
    vertex insert success         : 8                   
    vertex insert failure         : 0                   
    edge parse success            : 6                   
    edge parse failure            : 0                   
    edge insert success           : 6                   
    edge insert failure           : 0                   
--------------------------------------------------
meter metrics
    total time                    : 0.199s              
    read time                     : 0.046s              
    load time                     : 0.153s              
    vertex load time              : 0.077s              
    vertex load rate(vertices/s)  : 103                 
    edge load time                : 0.112s              
    edge load rate(edges/s)       : 53                  
root@55360d18f7d7:/loader#

Then we can see the result in hubble:

Verifying these changes

Trivial rework / code cleanup without any test coverage. (No Need)
Already covered by existing tests, such as (please modify tests here).
Need tests and can be verified as follows:
- my test docker image is here dandelionivy/loader

Does this PR potentially affect the following parts?

Documentation Status

Doc - TODO
Doc - Done
Doc - No Need

codecov · 2023-11-06T08:17:10Z

Codecov Report

Merging #530 (56ac508) into master (71e623d) will decrease coverage by 0.03%.
Report is 1 commits behind head on master.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master     #530      +/-   ##
============================================
- Coverage     62.49%   62.47%   -0.03%     
+ Complexity     1903      930     -973     
============================================
  Files           262       93     -169     
  Lines          9541     4509    -5032     
  Branches        886      529     -357     
============================================
- Hits           5963     2817    -3146     
+ Misses         3190     1483    -1707     
+ Partials        388      209     -179

see 169 files with indirect coverage changes

📣 Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in Chrome or Firefox today!

aroundabout · 2023-11-06T08:54:05Z

hugegraph-loader/Dockerfile

+WORKDIR /pkg
+
+RUN set -x \
+    && mvn install -pl hugegraph-client,hugegraph-loader -am -Dmaven.javadoc.skip=true -DskipTests -ntp


It seems that in the center maven repo, the client-1.0.0 does not have some methods like istext(). Hence, we should install the jar locally.

It seems that in the center maven repo, the client-1.0.0 does not have some methods like istext(). Hence, we should install the jar locally.

seems we don't rely on fixed version? we use $revision in the project, so install client first is necessary (if we want build the latest image)

hugegraph-loader/Dockerfile

imbajin · 2023-11-07T02:18:13Z

@liuxiaocs7 could also check the PR when free

imbajin · 2023-11-07T02:21:12Z

hugegraph-loader/Dockerfile

+WORKDIR /pkg
+
+RUN set -x \
+    && mvn install -pl hugegraph-client,hugegraph-loader -am -Dmaven.javadoc.skip=true -DskipTests -ntp


It seems that in the center maven repo, the client-1.0.0 does not have some methods like istext(). Hence, we should install the jar locally.

seems we don't rely on fixed version? we use $revision in the project, so install client first is necessary (if we want build the latest image)

imbajin · 2023-11-07T02:22:28Z

hugegraph-loader/Dockerfile

+ENTRYPOINT ["/usr/bin/dumb-init", "--"]
+CMD [ "/bin/bash" ]


maybe check other usage way for dumb-init ?

It seems there is no special usage. We only need to add dumb-init in entrypoint. Maybe tail -f /dev/null is better than /bin/bash.

imbajin · 2023-11-07T02:23:03Z

hugegraph-loader/README.md

-## Building
+## 2. Usage for Docker(Recommand)
+
+- deploy `loader` with Docker


Suggested change

- deploy `loader` with Docker

- run `loader` with Docker

imbajin · 2023-11-07T02:25:08Z

hugegraph-loader/README.md

+#### 2.2.1 enter the docker container
+
+```bash
+docker exec -it loader bash


remember update here when doc update apache/incubator-hugegraph-doc#299 (comment)

imbajin · 2023-11-07T02:28:41Z

hugegraph-loader/README.md

consider also add a new section (run with docker) in root (README) https://github.com/apache/incubator-hugegraph-toolchain/blob/master/README.md#modules

and keep the module section clear

liuxiaocs7 · 2023-11-10T08:39:14Z

Hi, @aroundabout, thanks for your outstanding efforts on this issue, i test locally and find some issues and hope your help.

Firstly, i try to build the image locally but fails.

(base) kirin@kirinbiotech:~/lx/incubator-hugegraph-toolchain$ docker build -t hg-loader:1.0 hugegraph-loader/

Then, i use loader image from here: https://hub.docker.com/r/dandelionivy/loader/tags where server and hubble deployed separately.

kirin@kirinbiotech:~/lx/incubator-hugegraph-toolchain/hugegraph-loader/assembly/static$ docker exec -it loader bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -p 8080 -h 192.168.110.158

but still fails to load data.

CC: @imbajin

aroundabout · 2023-11-10T08:44:09Z

Hi, @aroundabout, thanks for your outstanding efforts on this issue, i test locally and find some issues.

Firstly, i try to build the image locally but fails.
(base) kirin@kirinbiotech:~/lx/incubator-hugegraph-toolchain$ docker build -t hg-loader:1.0 hugegraph-loader/
Then, i use loader image from here: https://hub.docker.com/r/dandelionivy/loader/tags
kirin@kirinbiotech:~/lx/incubator-hugegraph-toolchain/hugegraph-loader/assembly/static$ docker exec -it loader bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -p 8080 -h 192.168.110.158
but still fails to load data.

How to build locally
try docker build -f hugegraph-loader/Dockerfile hg-loader .
Because loader depends on client and the client-1.0.0 in maven repo does not have some methods like 'istext()', so we should build client locally in dockerfile. Hence, the docker build context should be root path
The question is caused by server [Bug] Unexpected stream error with gzip & need enhance filter logic incubator-hugegraph#2346. The PR to fix it has been opened fix(api): refactor/downgrade record logic for slow log incubator-hugegraph#2347

liuxiaocs7 · 2023-11-10T08:52:22Z

How to build locally
try docker build -f hugegraph-loader/Dockerfile hg-loader .
Because loader depends on client and the client-1.0.0 in maven repo does not have some methods like 'istext()', so we should build client locally in dockerfile. Hence, the docker build context should be root path

Thanks for your details explain, i see.

docker build -f hugegraph-loader/Dockerfile hg-loader .

seems fails too? And does it need to be added to the readme?

aroundabout · 2023-11-10T08:59:27Z

How to build locally
try docker build -f hugegraph-loader/Dockerfile hg-loader .
Because loader depends on client and the client-1.0.0 in maven repo does not have some methods like 'istext()', so we should build client locally in dockerfile. Hence, the docker build context should be root path

Thanks for your details explain, i see.
docker build -f hugegraph-loader/Dockerfile hg-loader .
seems fails too? And does it need to be added to the readme?

Sorry, I forgot an argument. XD

docker build -f hugegraph-loader/Dockerfile -t hg-loader .

It can works now.

liuxiaocs7 · 2023-11-10T09:03:54Z

How to build locally
try docker build -f hugegraph-loader/Dockerfile hg-loader .
Because loader depends on client and the client-1.0.0 in maven repo does not have some methods like 'istext()', so we should build client locally in dockerfile. Hence, the docker build context should be root path

Thanks for your details explain, i see.
docker build -f hugegraph-loader/Dockerfile hg-loader .
seems fails too? And does it need to be added to the readme?
Sorry, I forgot an argument. XD

docker build -f hugegraph-loader/Dockerfile -t hg-loader .

It can works now.

works!

liuxiaocs7

LGTM~

imbajin

THX

feat: support docker for loader

d6aff34

github-actions bot added the loader hugegraph-loader label Nov 6, 2023

add empty line at end of file

2d70c48

supplement license

0ae8ced

aroundabout commented Nov 6, 2023

View reviewed changes

imbajin reviewed Nov 6, 2023

View reviewed changes

hugegraph-loader/Dockerfile Show resolved Hide resolved

supplement readme

9fa0614

github-actions bot added the hubble hugegraph-hubble label Nov 6, 2023

supplement readme

705b66f

imbajin reviewed Nov 7, 2023

View reviewed changes

aroundabout added 2 commits November 8, 2023 12:57

change dockerfile cmd and readme

acc9b54

opt doc and docker compose

56ac508

liuxiaocs7 approved these changes Nov 13, 2023

View reviewed changes

imbajin approved these changes Nov 13, 2023

View reviewed changes

simon824 approved these changes Nov 13, 2023

View reviewed changes

simon824 merged commit e8fcdd3 into apache:master Nov 13, 2023
11 of 12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support docker for loader #530

feat: support docker for loader #530

aroundabout commented Nov 6, 2023 •

edited

Loading

codecov bot commented Nov 6, 2023 •

edited

Loading

aroundabout Nov 6, 2023

imbajin Nov 7, 2023

imbajin commented Nov 7, 2023

imbajin Nov 7, 2023

imbajin Nov 7, 2023

aroundabout Nov 7, 2023

imbajin Nov 7, 2023

imbajin Nov 7, 2023

imbajin Nov 7, 2023

liuxiaocs7 commented Nov 10, 2023 •

edited

Loading

aroundabout commented Nov 10, 2023 •

edited

Loading

liuxiaocs7 commented Nov 10, 2023

aroundabout commented Nov 10, 2023

liuxiaocs7 commented Nov 10, 2023

liuxiaocs7 left a comment

imbajin left a comment

feat: support docker for loader #530

feat: support docker for loader #530

Conversation

aroundabout commented Nov 6, 2023 • edited Loading

Purpose of the PR

Main Changes

usage

Verifying these changes

Does this PR potentially affect the following parts?

Documentation Status

codecov bot commented Nov 6, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imbajin commented Nov 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liuxiaocs7 commented Nov 10, 2023 • edited Loading

aroundabout commented Nov 10, 2023 • edited Loading

liuxiaocs7 commented Nov 10, 2023

aroundabout commented Nov 10, 2023

liuxiaocs7 commented Nov 10, 2023

liuxiaocs7 left a comment

Choose a reason for hiding this comment

imbajin left a comment

Choose a reason for hiding this comment

aroundabout commented Nov 6, 2023 •

edited

Loading

codecov bot commented Nov 6, 2023 •

edited

Loading

liuxiaocs7 commented Nov 10, 2023 •

edited

Loading

aroundabout commented Nov 10, 2023 •

edited

Loading