feat: support docker for loader (#530)

* feat: support docker for loader * add empty line at end of file * supplement license * supplement readme * supplement readme * change dockerfile cmd and readme * opt doc and docker compose
apache · Nov 13, 2023 · e8fcdd3 · e8fcdd3
1 parent 4346269
commit e8fcdd3
Show file tree

Hide file tree

Showing 5 changed files with 216 additions and 7 deletions.
diff --git a/README.md b/README.md
@@ -12,10 +12,16 @@
 ## Modules
 
 - [hugegraph-loader](./hugegraph-loader): Loading datasets into the HugeGraph from multiple data sources.
-- [hugegraph-hubble](./hugegraph-hubble): Online HugeGraph management and analysis dashboard (Include: data loading, schema management, graph traverser and display). We can use `docker run -itd --name=hubble -p 8088:8088 hugegraph/hubble` to quickly start [hubble](https://hub.docker.com/r/hugegraph/hubble) or we can follow [this](hugegraph-hubble/README.md#quick-start) to use docker-compose to start `hubble` with `server`.
+- [hugegraph-hubble](./hugegraph-hubble): Online HugeGraph management and analysis dashboard (Include: data loading, schema management, graph traverser and display).
 - [hugegraph-tools](./hugegraph-tools): Command line tool for deploying, managing and backing-up/restoring graphs from HugeGraph.
 - [hugegraph-client](./hugegraph-client): A Java-written client for HugeGraph, providing `RESTful` APIs for accessing graph vertex/edge/schema/gremlin/variables and traversals etc.
 
+## Usage
+
+- [hugegraph-loader](./hugegraph-loader): We can use `docker run -itd --name loader hugegraph/loader` to quickly start [loader](https://hub.docker.com/r/hugegraph/loader) or we can follow [this](./hugegraph-loader/README.md#212-docker-compose) to use docker-compose to start `loader` with `server`. And we can find more details in the [doc](https://hugegraph.apache.org/docs/quickstart/hugegraph-loader/).
+- [hugegraph-hubble](./hugegraph-hubble): We can use `docker run -itd --name=hubble -p 8088:8088 hugegraph/hubble` to quickly start [hubble](https://hub.docker.com/r/hugegraph/hubble) or we can follow [this](hugegraph-hubble/README.md#quick-start) to use docker-compose to start `hubble` with `server`. And we can find more details in the [doc](https://hugegraph.apache.org/docs/quickstart/hugegraph-hubble/).
+- [hugegraph-client](./hugegraph-client): We can follow the [doc](https://hugegraph.apache.org/docs/quickstart/hugegraph-client/) to learn how to quick start with `client`.
+
 ## Maven Dependencies
 
 You could use import the dependencies in `maven` like this:

diff --git a/hugegraph-hubble/README.md b/hugegraph-hubble/README.md
@@ -20,8 +20,7 @@ We can quickly start `hubble` in two ways:
 1. We can use `docker run -itd --name=hubble -p 8088:8088 hugegraph/hubble` to quickly start [hubble](https://hub.docker.com/r/hugegraph/hubble).
 2. Or we can use the `docker-compose.yml` to start `hubble` with `hugegraph-server`. If we set `PRELOAD=true`, we can preload the example graph when starting `hugegraph-server`:
 
-
-    ```
+    ```yaml
     version: '3'
     services:
         server:

diff --git a/hugegraph-loader/Dockerfile b/hugegraph-loader/Dockerfile
@@ -0,0 +1,51 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM maven:3.9.0-eclipse-temurin-11 AS build
+
+COPY . /pkg
+WORKDIR /pkg
+
+RUN set -x \
+    && mvn install -pl hugegraph-client,hugegraph-loader -am -Dmaven.javadoc.skip=true -DskipTests -ntp
+
+
+RUN set -x \
+    && cd /pkg/hugegraph-loader/ \
+    && echo "$(ls)" \
+    && mvn clean package -DskipTests
+
+
+FROM openjdk:11-slim
+
+COPY --from=build /pkg/hugegraph-loader/apache-hugegraph-loader-incubating-*/ /loader
+WORKDIR /loader/
+
+RUN set -x \
+    && apt-get -q update \
+    && apt-get -q install -y --no-install-recommends --no-install-suggests \
+       dumb-init \
+       procps \
+       curl \
+       lsof \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+
+VOLUME /loader
+
+ENTRYPOINT ["/usr/bin/dumb-init", "--"]
+CMD ["tail","-f","/dev/null"]
diff --git a/hugegraph-loader/README.md b/hugegraph-loader/README.md
@@ -7,15 +7,131 @@
 
 hugegraph-loader is a customizable command line utility for loading small to medium size graph datasets into the HugeGraph database from multiple data sources with various input formats.
 
-## Features
+## 1. Features
 
 - Multiple data sources, such as local file(path), HDFS file(path), MySQL
 - Various input formats, such as json, csv, and text with any delimiters.
 - Diverse options, with which users can manage the data loading intuitively.
 - Detecting schema from data automatically, reduce the complex work of schema management.
 - Advanced customized operations with groovy script, users can configure how to construct vertices and edges by themselves.
 
-## Building
+## 2. Usage for Docker(Recommand)
+
+- Run `loader` with Docker
+    - Docker run
+    - Docker-compose
+- Load data in docker container `loader`
+
+### 2.1 Start with Docker
+
+#### 2.1.1 Docker run
+
+Use the command `docker run -itd --name loader hugegraph/loader` to start loader.
+
+If you want to load your data, you can mount the data folder like `-v /path/to/data/file:/loader/file`
+
+
+#### 2.1.2 Docker-compose
+
+The example `docker-compose.yml` is [here](./docker/example/docker-compose.yml)
+
+If you want to load your data, you can mount the data folder like:
+```yaml
+volumes:
+    - /path/to/data/file:/loader/file
+```
+
+Use the command `docker-compose up -d` to deploy `loader` with `server` and `hubble`.
+
+### 2.2 Load data with docker container
+
+#### 2.2.1 load data with docker
+
+> If the `loader` and `server` is in the same docker network (for example, you deploy `loader` and `server` with `docker-compose`), we can set `-h {server_container_name}`. In our example, the container name of `server` is `graph`
+>
+> If `loader` is deployed alone, the `-h` should be set to the ip of the host of `server`. Other parameter description is [here](https://hugegraph.apache.org/docs/quickstart/hugegraph-loader/#341-parameter-description)
+
+```bash
+docker exec -it loader bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -h graph -p 8080
+```
+
+Then we can see the result.
+
+```bash
+HugeGraphLoader worked in NORMAL MODE
+vertices/edges loaded this time : 8/6
+--------------------------------------------------
+count metrics
+    input read success            : 14                  
+    input read failure            : 0                   
+    vertex parse success          : 8                   
+    vertex parse failure          : 0                   
+    vertex insert success         : 8                   
+    vertex insert failure         : 0                   
+    edge parse success            : 6                   
+    edge parse failure            : 0                   
+    edge insert success           : 6                   
+    edge insert failure           : 0                   
+--------------------------------------------------
+meter metrics
+    total time                    : 0.199s              
+    read time                     : 0.046s              
+    load time                     : 0.153s              
+    vertex load time              : 0.077s              
+    vertex load rate(vertices/s)  : 103                 
+    edge load time                : 0.112s              
+    edge load rate(edges/s)       : 53   
+```
+
+Then you can use `curl` or `hubble` to see the result.
+
+```bash
+> curl "http://localhost:8080/graphs/hugegraph/graph/vertices" | gunzip
+{"vertices":[{"id":1,"label":"software","type":"vertex","properties":{"name":"lop","lang":"java","price":328.0}},{"id":2,"label":"software","type":"vertex","properties":{"name":"ripple","lang":"java","price":199.0}},{"id":"1:tom","label":"person","type":"vertex","properties":{"name":"tom"}},{"id":"1:josh","label":"person","type":"vertex","properties":{"name":"josh","age":32,"city":"Beijing"}},{"id":"1:marko","label":"person","type":"vertex","properties":{"name":"marko","age":29,"city":"Beijing"}},{"id":"1:peter","label":"person","type":"vertex","properties":{"name":"peter","age":35,"city":"Shanghai"}},{"id":"1:vadas","label":"person","type":"vertex","properties":{"name":"vadas","age":27,"city":"Hongkong"}},{"id":"1:li,nary","label":"person","type":"vertex","properties":{"name":"li,nary","age":26,"city":"Wu,han"}}]}
+```
+
+If you want to check the edges, use `curl "http://localhost:8080/graphs/hugegraph/graph/edges" | gunzip`
+
+#### 2.2.2 enter the docker container to load data
+
+If you want to do some additional operation in the container, you can enter the container as follows:
+
+```bash
+docker exec -it loader bash
+```
+
+Then, you can load data as follows:
+
+```bash
+sh bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -h graph -p 8080
+```
+
+The result is as same as above.
+
+## 3. Use loader directly
+
+> notice: currently, version is `1.0.0`
+
+Download and unzip the compiled archive
+
+```bash
+wget https://downloads.apache.org/incubator/hugegraph/{version}/apache-hugegraph-toolchain-incubating-{version}.tar.gz
+tar zxf *hugegraph*.tar.gz
+```
+
+Then, load data with example file:
+
+```bash
+cd apache-hugegraph-toolchain-incubating-{version}
+cd apache-hugegraph-loader-incubating-{version}
+sh bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy
+```
+
+More details is in the [doc](https://hugegraph.apache.org/docs/quickstart/hugegraph-loader/)
+
+## 4. Building
+
+You can also build the `loader` by yourself.
 
 Required:
 
@@ -34,10 +150,10 @@ To build with default tests:
 mvn clean install
 ```
 
-## Doc
+## 5. Doc
 
 The [loader homepage](https://hugegraph.apache.org/docs/quickstart/hugegraph-loader/) contains more information about it. 
 
-## License
+## 6. License
 
 hugegraph-loader is licensed under Apache 2.0 License.
diff --git a/hugegraph-loader/docker/example/docker-compose.yml b/hugegraph-loader/docker/example/docker-compose.yml
@@ -0,0 +1,37 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+version: '3'
+services:
+  server:
+    image: hugegraph/hugegraph
+    container_name: graph
+    ports:
+      - 8080:8080
+
+  hubble:
+    image: hugegraph/hubble
+    container_name: hubble
+    ports:
+      - 8088:8088
+
+  loader:
+    image: hugegraph/loader
+    container_name: loader
+    # mount your own data here
+    # volumes:
+      # - /path/to/data/file:/loader/file