feat: added the OpenTelemetry trace support (#2477)

TODO: we need enhance our shell experience
---------

Co-authored-by: imbajin <jin@apache.org>
Co-authored-by: VGalaxies <vgalaxies@apache.org>

.gitignore

@@ -83,3 +83,6 @@ hs_err_pid*
 .mtj.tmp/
 # blueJ files
 *.ctxt
+
+# docker volumes ignore
+hugegraph-server/hugegraph-dist/docker/data/

README.md

@@ -16,7 +16,9 @@
 ## What is Apache HugeGraph?
 
 [HugeGraph](https://hugegraph.apache.org/) is a fast-speed and highly-scalable [graph database](https://en.wikipedia.org/wiki/Graph_database). 
-Billions of vertices and edges can be easily stored into and queried from HugeGraph due to its excellent OLTP ability. As compliance to [Apache TinkerPop 3](https://tinkerpop.apache.org/) framework, various complicated graph queries can be accomplished through [Gremlin](https://tinkerpop.apache.org/gremlin.html)(a powerful graph traversal language).
+Billions of vertices and edges can be easily stored into and queried from HugeGraph due to its excellent OLTP ability. 
+As compliance to [Apache TinkerPop 3](https://tinkerpop.apache.org/) framework, various complicated graph queries can be 
+achieved through [Gremlin](https://tinkerpop.apache.org/gremlin.html)(a powerful graph traversal language).
 
 
 ## Features
@@ -40,7 +42,7 @@ You can visit [doc page](https://hugegraph.apache.org/docs/quickstart/hugegraph-
 >  
 > 1. The docker image of hugegraph is a convenience release, but not **official distribution** artifacts. You can find more details from [ASF Release Distribution Policy](https://infra.apache.org/release-distribution.html#dockerhub).
 > 
-> 2. Recommand to use `release tag`(like `1.2.0`) for the stable version. Use `latest` tag to experience the newest functions in development.
+> 2. Recommend to use `release tag`(like `1.2.0`) for the stable version. Use `latest` tag to experience the newest functions in development.
 
 ### 2. Download Way
 

hugegraph-server/hugegraph-dist/docker/README.md

@@ -4,11 +4,11 @@
 > 
 > 1. The docker image of hugegraph is a convenience release, not official distribution artifacts from ASF. You can find more details from [ASF Release Distribution Policy](https://infra.apache.org/release-distribution.html#dockerhub).
 > 
-> 2. Recommand to use `release tag`(like `1.2.0`) for the stable version. Use `latest` tag to experience the newest functions in development.
+> 2. Recommend to use `release tag`(like `1.2.0`) for the stable version. Use `latest` tag to experience the newest functions in development.
 
 ## 1. Deploy
 
-We can use docker to quickly start an inner HugeGraph server with RocksDB in background.
+We can use docker to quickly start an inner HugeGraph server with RocksDB in the background.
 
 1. Using docker run
 
@@ -29,9 +29,9 @@ We can use docker to quickly start an inner HugeGraph server with RocksDB in bac
 
 ## 2. Create Sample Graph on Server Startup
 
-If you want to **pre-load** some (test) data or graphs in container(by default), you can set the env `PRELOAD=ture`
+If you want to **preload** some (test) data or graphs in container(by default), you can set the env `PRELOAD=ture`
 
-If you want to customize the pre-loaded data, please mount the the groovy scripts (not necessary).
+If you want to customize the preloaded data, please mount the groovy scripts (not necessary).
 
 1. Using docker run
 
@@ -67,7 +67,7 @@ If you want to customize the pre-loaded data, please mount the the groovy script
 
 2. Using docker compose
 
-    Similarly, we can set the envionment variables in the docker-compose.yaml:
+    Similarly, we can set the environment variables in the docker-compose.yaml:
 
     ```yaml
     version: '3'
@@ -81,3 +81,34 @@ If you want to customize the pre-loaded data, please mount the the groovy script
           - AUTH=true
           - PASSWORD=123456
     ```
+## 4. Running Open-Telemetry-Collector
+
+> CAUTION:
+> 
+> The `docker-compose-trace.yaml` utilizes `Grafana` and `Grafana-Tempo`, both of them are licensed under [AGPL-3.0](https://www.gnu.org/licenses/agpl-3.0.en.html), you should be aware of and use them with caution. Currently, we mainly provide this template for everyone to **test**
+>
+1. Start Open-Telemetry-Collector
+
+    ```bash
+    cd hugegraph-server/hugegraph-dist/docker/example
+    docker-compose -f docker-compose-trace.yaml -p hugegraph-trace up -d
+    ```
+
+2. Active Open-Telemetry-Agent
+
+    ```bash
+    ./start-hugegraph.sh -y true
+    ```
+
+3. Stop Open-Telemetry-Collector
+
+    ```bash
+    cd hugegraph-server/hugegraph-dist/docker/example
+    docker-compose -f docker-compose-trace.yaml -p hugegraph-trace stop
+    ```
+
+4. References
+
+   - [What is OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/)
+
+   - [Tempo in Grafana](https://grafana.com/docs/tempo/latest/getting-started/tempo-in-grafana/)

hugegraph-server/hugegraph-dist/docker/conf/grafana-datasources.yaml

@@ -0,0 +1,47 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: 1
+
+datasources:
+- name: Prometheus
+  type: prometheus
+  uid: prometheus
+  access: proxy
+  orgId: 1
+  url: http://prometheus:9090
+  basicAuth: false
+  isDefault: false
+  version: 1
+  editable: false
+  jsonData:
+    httpMethod: GET
+- name: Tempo
+  type: tempo
+  access: proxy
+  orgId: 1
+  url: http://tempo:3200
+  basicAuth: false
+  isDefault: true
+  version: 1
+  editable: false
+  apiVersion: 1
+  uid: tempo
+  jsonData:
+    httpMethod: GET
+    serviceMap:
+      datasourceUid: prometheus

hugegraph-server/hugegraph-dist/docker/conf/prometheus.yaml

@@ -0,0 +1,28 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+global:
+  scrape_interval:     15s
+  evaluation_interval: 15s
+
+scrape_configs:
+  - job_name: 'prometheus'
+    static_configs:
+      - targets: [ 'localhost:9090' ]
+  - job_name: 'tempo'
+    static_configs:
+      - targets: [ 'tempo:3200' ]

hugegraph-server/hugegraph-dist/docker/conf/tempo.yaml

@@ -0,0 +1,56 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+stream_over_http_enabled: true
+server:
+  http_listen_port: 3200
+  log_level: info
+
+distributor:
+  receivers:
+    otlp:
+      protocols:
+        http:
+        grpc:
+
+compactor:
+  compaction:
+    block_retention: 48h # configure total trace retention here
+
+metrics_generator:
+  registry:
+    external_labels:
+      source: tempo
+
+  storage:
+    path: /tmp/tempo/generator/wal
+    remote_write:
+      - url: http://prometheus:9090/api/v1/write
+        send_exemplars: true
+
+storage:
+  trace:
+    backend: local                     # backend configuration to use
+    wal:
+      path: /tmp/tempo/wal             # where to store the wal locally
+    local:
+      path: /tmp/tempo/blocks
+
+overrides:
+  defaults:
+    metrics_generator:
+      processors: [ service-graphs, span-metrics ] # enables metrics generator

hugegraph-server/hugegraph-dist/docker/example/docker-compose-trace.yaml

@@ -0,0 +1,56 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+version: "3"
+
+services:
+  tempo:
+    image: grafana/tempo:latest
+    command: ["-config.file=/etc/tempo.yaml"]
+    volumes:
+      - ../conf/tempo.yaml:/etc/tempo.yaml
+      - ../data/tempo:/tmp/tempo
+    ports:
+      - "3200:3200"  # tempo
+      - "9095:9095"  # tempo grpc
+      - "4317:4317"  # otlp grpc
+      - "4318:4318"  # otlp http
+
+  prometheus:
+    image: prom/prometheus:latest
+    command:
+      - --config.file=/etc/prometheus.yaml
+      - --web.enable-remote-write-receiver
+      - --enable-feature=exemplar-storage
+      - --storage.tsdb.path=/tmp/data
+    volumes:
+      - ../conf/prometheus.yaml:/etc/prometheus.yaml
+      - ../data/prometheus:/tmp/data
+    ports:
+      - "9090:9090"
+
+  grafana:
+    image: grafana/grafana:10.2.2
+    volumes:
+      - ../conf/grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
+    environment:
+      - GF_AUTH_ANONYMOUS_ENABLED=true
+      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
+      - GF_AUTH_DISABLE_LOGIN_FORM=true
+      - GF_FEATURE_TOGGLES_ENABLE=traceqlEditor traceQLStreaming metricsSummary
+    ports:
+      - "3000:3000"

hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh

@@ -15,6 +15,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+
 function abs_path() {
     SOURCE="${BASH_SOURCE[0]}"
     while [[ -h "$SOURCE" ]]; do
@@ -39,36 +40,33 @@ EXT="$TOP/ext"
 PLUGINS="$TOP/plugins"
 LOGS="$TOP/logs"
 OUTPUT=${LOGS}/hugegraph-server.log
+GITHUB="https://github.com"
 
 export HUGEGRAPH_HOME="$TOP"
 . "${BIN}"/util.sh
 
-GREMLIN_SERVER_CONF="$1"
-REST_SERVER_CONF="$2"
-OPEN_SECURITY_CHECK="$3"
-
-if [[ $# -eq 3 ]]; then
-    USER_OPTION=""
-    GC_OPTION=""
-elif [[ $# -eq 4 ]]; then
-    USER_OPTION="$4"
-    GC_OPTION=""
-elif [[ $# -eq 5 ]]; then
-    USER_OPTION="$4"
-    GC_OPTION="$5"
-fi
+# Parse the server arguments in array way
+SERVER_ARGS=("$@")
+GREMLIN_SERVER_CONF="${SERVER_ARGS[0]:-}"
+REST_SERVER_CONF="${SERVER_ARGS[1]:-}"
+OPEN_SECURITY_CHECK="${SERVER_ARGS[2]:-}"
+# Param will be empty str("") if not set
+USER_OPTION="${SERVER_ARGS[3]:-}"
+GC_OPTION="${SERVER_ARGS[4]:-}"
+OPEN_TELEMETRY="${SERVER_ARGS[5]:-}"
 
 ensure_path_writable "$LOGS"
 ensure_path_writable "$PLUGINS"
 
 # The maximum and minimum heap memory that service can use
 MAX_MEM=$((32 * 1024))
 MIN_MEM=$((1 * 512))
+# TODO: upgrade to Java 11 in 1.5.0
 MIN_JAVA_VERSION=8
 
-# download binary file
+# Note: Download for HTTPS, could comment out if you don't need it
 if [[ ! -e "${CONF}/hugegraph-server.keystore" ]]; then
-    download "${CONF}" "https://github.com/apache/hugegraph-doc/raw/binary-1.0/dist/server/hugegraph-server.keystore"
+    download "${CONF}" "${GITHUB}/apache/hugegraph-doc/raw/binary-1.0/dist/server/hugegraph-server.keystore"
 fi
 
 # Add the slf4j-log4j12 binding
@@ -81,7 +79,7 @@ CP="$CP":$(find -L $LIB -name '*.jar' \
     \! -name 'log4j-slf4j-impl*.jar' | sort | tr '\n' ':')
 # Add the jars in ext (at any subdirectory depth)
 CP="$CP":$(find -L $EXT -name '*.jar' | sort | tr '\n' ':')
-# Add the jars in plugins (at any subdirectory depth)
+# Add the jars in plugins (at any subdirectory depth), check "javaagent" related jars carefully
 CP="$CP":$(find -L $PLUGINS -name '*.jar' | sort | tr '\n' ':')
 
 # (Cygwin only) Use ; classpath separator and reformat paths for Windows ("C:\foo")
@@ -153,6 +151,44 @@ if [[ ${OPEN_SECURITY_CHECK} == "true" ]]; then
     JVM_OPTIONS="${JVM_OPTIONS} -Djava.security.manager=org.apache.hugegraph.security.HugeSecurityManager"
 fi
 
+if [ "${OPEN_TELEMETRY}" == "true" ]; then
+    OT_JAR="opentelemetry-javaagent.jar"
+    OT_JAR_PATH="${PLUGINS}/${OT_JAR}"
+
+    if [[ ! -e "${OT_JAR_PATH}" ]]; then
+        echo "## Downloading ${OT_JAR}..."
+        download "${PLUGINS}" \
+            "${GITHUB}/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.1.0/${OT_JAR}"
+
+        if [[ ! -e "${OT_JAR_PATH}" ]]; then
+            echo "## Error: Failed to download ${OT_JAR}." >>${OUTPUT}
+            exit 1
+        fi
+    fi
+
+    # Note: remember update it if we change the jar 
+    expected_md5="e3bcbbe8ed9b6d840fa4c333b36f369f"
+    actual_md5=$(md5sum "${OT_JAR_PATH}" | awk '{print $1}')
+
+    if [[ "${expected_md5}" != "${actual_md5}" ]]; then
+        echo "## Error: MD5 checksum verification failed for ${OT_JAR_PATH}." >>${OUTPUT}
+        echo "## Tips: Remove the file and try again." >>${OUTPUT}
+        exit 1
+    fi
+
+    # Note: check carefully if multi "javeagent" params are set
+    export JAVA_TOOL_OPTIONS="-javaagent:${PLUGINS}/${OT_JAR}"
+    export OTEL_TRACES_EXPORTER=otlp
+    export OTEL_METRICS_EXPORTER=none
+    export OTEL_LOGS_EXPORTER=none
+    export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc
+    # 127.0.0.1:4317 is the port of otel-collector running in Docker located in
+    # 'hugegraph-server/hugegraph-dist/docker/example/docker-compose-trace.yaml'.
+    # Make sure the otel-collector is running before starting HugeGraphServer.
+    export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:4317
+    export OTEL_RESOURCE_ATTRIBUTES=service.name=server
+fi
+
 # Turn on security check
 exec ${JAVA} -Dname="HugeGraphServer" ${JVM_OPTIONS} ${JAVA_OPTIONS} -cp ${CLASSPATH}: \
     org.apache.hugegraph.dist.HugeGraphServer ${GREMLIN_SERVER_CONF} ${REST_SERVER_CONF} \