Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hugegraph benchmark能否以hbase作为存储后端测试 #134

Closed
fisherinbox opened this issue Oct 26, 2018 · 2 comments
Closed

hugegraph benchmark能否以hbase作为存储后端测试 #134

fisherinbox opened this issue Oct 26, 2018 · 2 comments
Labels

Comments

@fisherinbox
Copy link

Expected behavior 期望表现

用hugegraph benchmark 源码进行测试,是否能修改配置文件以hbase作为存储后端

Actual behavior 实际表现

修改了hugegraph.properties,增加了 hbase相关配置,出错
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/client/TableDescriptorBuilder
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.baidu.hugegraph.backend.store.hbase.HbaseSessions.createTable(HbaseSessions.java:152)
at com.baidu.hugegraph.backend.store.hbase.HbaseStore.init(HbaseStore.java:210)
at com.baidu.hugegraph.backend.store.AbstractBackendStoreProvider.init(AbstractBackendStoreProvider.java:86)
at com.baidu.hugegraph.HugeGraph.initBackend(HugeGraph.java:189)
at eu.socialsensor.graphdatabases.HugeGraphCoreDatabase.loadGraph(HugeGraphCoreDatabase.java:492)
at eu.socialsensor.graphdatabases.HugeGraphCoreDatabase.buildGraphEnv(HugeGraphCoreDatabase.java:201)
at eu.socialsensor.graphdatabases.HugeGraphCoreDatabase.createGraphForMassiveLoad(HugeGraphCoreDatabase.java:156)
at eu.socialsensor.benchmarks.MassiveInsertionBenchmark.benchmarkOne(MassiveInsertionBenchmark.java:41)
at eu.socialsensor.benchmarks.PermutingBenchmarkBase.startBenchmarkInternalOnePermutation(PermutingBenchmarkBase.java:68)
at eu.socialsensor.benchmarks.PermutingBenchmarkBase.startBenchmarkInternal(PermutingBenchmarkBase.java:57)
at eu.socialsensor.benchmarks.BenchmarkBase.startBenchmark(BenchmarkBase.java:35)
at eu.socialsensor.main.GraphDatabaseBenchmark.runBenchmark(GraphDatabaseBenchmark.java:136)
at eu.socialsensor.main.GraphDatabaseBenchmark.run(GraphDatabaseBenchmark.java:101)
at eu.socialsensor.main.GraphDatabaseBenchmarkTest.testGraphDatabaseBenchmark(GraphDatabaseBenchmarkTest.java:14)

Results :

Tests in error:
GraphDatabaseBenchmarkTest.testGraphDatabaseBenchmark:14 » NoClassDefFound org...

Steps to reproduce the problem 复现步骤

  1. {step 1}
  2. {step 2}
  3. {step 3}

Status of loaded data 数据状态

Vertex/Edge summary 数据量

  • loaded vertices amount: {like 10 million}
  • loaded edges amount: {like 20 million}
  • loaded time: {like 200s}

Vertex/Edge example 数据示例

{type something here...}

Schema(VertexLabel, EdgeLabel, IndexLabel) 元数据结构

{type something here...}

Specifications of environment 环境信息

  • hugegraph version: {like v0.7.4}
  • operating system: {like centos 7.4, 32 CPUs, 64G RAM}
  • hugegraph backend: {like cassandra 3.10, cluster with 20 nodes, 3 x 1TB HDD disk each node}
@zhoney
Copy link
Contributor

zhoney commented Oct 29, 2018

@fisherinbox 您好,graphdb-benchmark适配了hugegraph-core,原则上各种后端都是可以的。

上面的错误是因为包冲突。因为graphdb-benchmark中的titan依赖于hbase-client的1.2版本,HugeGraph则依赖于hbase-client的2.0.0版本。所以想要运行hbase后端存储需要修改一下pom.xml,并做后端适配

patch如下:

diff --git a/pom.xml b/pom.xml
index 2bc10c0..7844f1a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -54,7 +54,7 @@
         <orientdb.version>2.2.5</orientdb.version>
         <titan.version>0.5.4</titan.version>
         <!--<hbase.version>0.98.8-hadoop2</hbase.version>-->
-        <hbase.version>1.2.0</hbase.version>
+        <hbase.version>2.0.0</hbase.version>
         <neo4j.version>2.0.1</neo4j.version>
         <dynamodb.titan.version>1.0.0</dynamodb.titan.version>
         <log4j2.version>2.1</log4j2.version>
@@ -259,6 +259,24 @@
             <groupId>org.apache.hbase</groupId>
             <artifactId>hbase-client</artifactId>
             <version>${hbase.version}</version>
+            <exclusions>
+                <exclusion>
+                    <groupId>com.fasterxml.jackson.core</groupId>
+                    <artifactId>jackson-databind</artifactId>
+                </exclusion>
+                <exclusion>
+                    <groupId>org.slf4j</groupId>
+                    <artifactId>slf4j-api</artifactId>
+                </exclusion>
+                <exclusion>
+                    <groupId>org.slf4j</groupId>
+                    <artifactId>slf4j-log4j12</artifactId>
+                </exclusion>
+                <exclusion>
+                    <groupId>log4j</groupId>
+                    <artifactId>log4j</artifactId>
+                </exclusion>
+            </exclusions>
         </dependency>
         <dependency>
             <groupId>com.amazonaws</groupId>
diff --git a/src/test/resources/META-INF/input.properties b/src/test/resources/META-INF/input.properties
index 3001d1b..18de93e 100644
--- a/src/test/resources/META-INF/input.properties
+++ b/src/test/resources/META-INF/input.properties
@@ -19,12 +19,12 @@ eu.socialsensor.metrics.csv.directory=./metrics
 # Available dbs are:
 #eu.socialsensor.databases=tbdb
 #eu.socialsensor.databases=tddb
-eu.socialsensor.databases=tc
+#eu.socialsensor.databases=tc
 #eu.socialsensor.databases=thb
 #eu.socialsensor.databases=tce
 #eu.socialsensor.databases=tp
 #eu.socialsensor.databases=orient
-eu.socialsensor.databases=neo4j
+#eu.socialsensor.databases=neo4j
 #eu.socialsensor.databases=sparksee
 #eu.socialsensor.databases=hugegraphclient
 eu.socialsensor.databases=hugegraphcore
diff --git a/src/test/resources/hugegraph.properties b/src/test/resources/hugegraph.properties
index f53abd0..73bee0c 100644
--- a/src/test/resources/hugegraph.properties
+++ b/src/test/resources/hugegraph.properties
@@ -5,6 +5,7 @@ gremlin.graph=com.baidu.hugegraph.HugeFactory
 
 backend=rocksdb
 serializer=binary
+
 rocksdb.data_path=./storage/hugegraphcore
 rocksdb.wal_path=./storage/hugegraphcore
 
@@ -15,3 +16,8 @@ cassandra.host=localhost
 cassandra.port=9042
 cassandra.username=
 cassandra.password=
+
+#backend=hbase
+#serializer=hbase
+#hbase.hosts=localhost
+#hbase.port=2181

@github-actions
Copy link

Due to the lack of activity, the current issue is marked as stale and will be closed after 20 days, any update will remove the stale label

VGalaxies pushed a commit that referenced this issue Aug 3, 2024
* add: dependency-review

* tiny improve

* fix

Co-authored-by: imbajin <jin@apache.org>

---------

Co-authored-by: imbajin <jin@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants