fix: Spark with loader has dependency conflicts #480

haohao0103 · 2023-06-14T05:25:56Z

No description provided.

合并原仓库代码

[Bug] Spark loader meet Exception: Class is not registered

Spark loader has dependency conflicts

codecov · 2023-06-14T05:37:36Z

Codecov Report

Merging #480 (841e774) into master (549ca82) will decrease coverage by 0.07%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master     #480      +/-   ##
============================================
- Coverage     62.48%   62.42%   -0.07%     
+ Complexity     1868      895     -973     
============================================
  Files           260       91     -169     
  Lines          9432     4404    -5028     
  Branches        875      519     -356     
============================================
- Hits           5894     2749    -3145     
+ Misses         3154     1450    -1704     
+ Partials        384      205     -179

Files Changed	Coverage Δ
...e/hugegraph/loader/spark/HugeGraphSparkLoader.java	`0.00% <0.00%> (ø)`

... and 169 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Spark loader has dependency conflicts

haohao0103 · 2023-06-14T11:06:48Z

@imbajin @simon824 @JackyYangPassion Could you give me some advice? thanks.The error unittest I can pass in the local .

javeme

Thanks for your contribution~
Could you share some detailed context information? such as conflicting error messages.

hugegraph-loader/pom.xml

hugegraph-loader/src/main/java/org/apache/hugegraph/loader/spark/HugeGraphSparkLoader.java

hugegraph-loader/pom.xml

imbajin · 2023-06-14T16:57:56Z

@imbajin @simon824 @JackyYangPassion Could you give me some advice? thanks.The error unittest I can pass in the local .

could rebase/merge #478 first (will also fix the dependency-review problem)

imbajin · 2023-06-18T13:09:29Z

@haohao0103 please update the branch with the latest code & fix the conflicts & then test the bug again

# Conflicts: # hugegraph-dist/scripts/dependency/known-dependencies.txt # hugegraph-loader/pom.xml

Spark loader has dependency conflicts

hugegraph-loader/src/main/resources/META-INF/services/shaded.jakarta.ws.rs.client.ClientBuilder

hugegraph-loader/assembly/static/bin/hugegraph-spark-loader.sh

Spark loader has dependency conflicts

bypass server for hbase bulk delete massive vertices and edges

hugegraph-loader/src/main/java/org/apache/hugegraph/loader/executor/LoadOptions.java

hugegraph-loader/src/main/java/org/apache/hugegraph/loader/direct/loader/HBaseDirectLoader.java

hugegraph-loader/src/main/java/org/apache/hugegraph/loader/executor/LoadOptions.java

hugegraph-loader/src/main/java/org/apache/hugegraph/loader/constant/Constants.java

bypass server for hbase bulk delete massive vertices and edges

hugegraph-loader/src/main/java/org/apache/hugegraph/loader/executor/LoadOptions.java

hugegraph-loader/src/main/java/org/apache/hugegraph/loader/constant/Constants.java

bypass server for hbase bulk delete massive vertices and edges

liuxiaocs7 · 2023-08-01T07:37:49Z

Hi, @haohao0103, Wouldn it be better to separate the two things in different PRs?

…rtices and edges" This reverts commit 053d27b

…rtices and edges" This reverts commit d8b31fc

haohao0103 · 2023-08-02T02:47:12Z

你好，@haohao0103，将两个东西分开放在不同的 PR 中会更好吗？

@liuxiaocs7 hi ,I have revert the last few commits and will deal with it after this pr merge.

remove incubating

liuxiaocs7 · 2023-08-03T16:01:33Z

Hi, @haohao0103, thanks for your work, but -1 for me.

I tested locally use ./bin/hugegraph-spark-loader.sh --master local[*] --name spark-hugegraph-loader --file ./example/spark/struct.json --host 192.168.34.164 --port 18080 --graph hugegraph

but got:

java.lang.NullPointerException
        at java.lang.String.split(String.java:2337)
        at java.lang.String.split(String.java:2422)

If I remember correctly, delimieter has a default comma, we need to check this?

elements = builder.build(fileSource.header(),
                         row.mkString(delimiter).split(delimiter));

liuxiaocs7 · 2023-08-03T16:33:18Z

hugegraph-loader/src/main/resources/META-INF/services/shaded.jakarta.ws.rs.client.ClientBuilder

+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+shaded.org.glassfish.jersey.client.JerseyClientBuilder


I have a question, why do I need to add SPI to these two classes (this one and below), it seems to be able to import normally without adding these two?

hello, I understand that it is because SPI added these two classes in jersey-client jar and other jars. When we do shade, we are actually changing the path information of these two classes, so we need to add SPI to these two shade classes.

haohao0103 · 2023-08-04T09:59:43Z

Hi, @haohao0103, thanks for your work, but -1 for me.

I tested locally use ./bin/hugegraph-spark-loader.sh --master local[*] --name spark-hugegraph-loader --file ./example/spark/struct.json --host 192.168.34.164 --port 18080 --graph hugegraph

but got:
java.lang.NullPointerException
        at java.lang.String.split(String.java:2337)
        at java.lang.String.split(String.java:2422)
If I remember correctly, delimieter has a default comma, we need to check this?
elements = builder.build(fileSource.header(),
                         row.mkString(delimiter).split(delimiter));

hi,i have check it .when file format is json the default delimiter is null。

fix delimiter is null

imbajin · 2023-08-05T15:51:15Z

hugegraph-loader/src/main/java/org/apache/hugegraph/loader/spark/HugeGraphSparkLoader.java

+                if (Optional.ofNullable(delimiter).isPresent()) {
+                    elements = builder.build(fileSource.header(),
+                               row.mkString(delimiter).split(delimiter));


@liuxiaocs7 could test it again?

liuxiaocs7

LGTM

imbajin

we lack the test for spark-ci indeed, so next time could add some basic UT for it?

haohao0103 and others added 18 commits September 16, 2022 17:11

Merge pull request #1 from apache/master

0a87143

合并原仓库代码

Merge branch 'apache:master' into master

cf86be5

Merge branch 'apache:master' into master

2550ab8

Merge branch 'apache:master' into master

41e14d9

Merge branch 'apache:master' into master

6f57a3d

Merge branch 'apache:master' into master

7eca671

Merge branch 'apache:master' into master

1e7cdc8

Merge branch 'apache:master' into master

674e944

Merge branch 'apache:master' into master

6ebd8a5

Merge branch 'apache:master' into master

934cb96

Merge branch 'apache:master' into master

2eb26c7

Merge branch 'apache:master' into master

c06aa61

fix:apache#404

ff99f1e

[Bug] Spark loader meet Exception: Class is not registered

fix:apache#404

b12af0a

[Bug] Spark loader meet Exception: Class is not registered

Merge branch 'master' into schemaCache-optimize

0d7e26e

Merge branch 'apache:master' into master

60691de

Merge branch 'apache:master' into schemaCache-optimize

6993d9a

fix:apache#464

bc2c8e9

Spark loader has dependency conflicts

github-actions bot added the loader hugegraph-loader label Jun 14, 2023

alanzhao added 4 commits June 14, 2023 14:07

fix:apache#464

de42751

Spark loader has dependency conflicts

fix:apache#464

7a3a0a1

Spark loader has dependency conflicts

fix:apache#464

eac9b18

Spark loader has dependency conflicts

fix:apache#464

48faf3c

Spark loader has dependency conflicts

imbajin requested review from simon824, javeme and JackyYangPassion June 14, 2023 12:53

javeme reviewed Jun 14, 2023

View reviewed changes

hugegraph-loader/pom.xml Outdated Show resolved Hide resolved

hugegraph-loader/src/main/java/org/apache/hugegraph/loader/spark/HugeGraphSparkLoader.java Outdated Show resolved Hide resolved

hugegraph-loader/pom.xml Show resolved Hide resolved

haohao0103 and others added 3 commits June 19, 2023 12:25

Merge branch 'apache:master' into master

a415bab

Merge remote-tracking branch 'origin/master' into schemaCache-optimize

ddb5e0b

# Conflicts: # hugegraph-dist/scripts/dependency/known-dependencies.txt # hugegraph-loader/pom.xml

fix:apache#464

8f3b528

Spark loader has dependency conflicts

imbajin reviewed Jun 19, 2023

View reviewed changes

hugegraph-loader/src/main/resources/META-INF/services/shaded.jakarta.ws.rs.client.ClientBuilder Outdated Show resolved Hide resolved

hugegraph-loader/assembly/static/bin/hugegraph-spark-loader.sh Outdated Show resolved Hide resolved

imbajin requested a review from javeme June 19, 2023 07:35

alanzhao added 2 commits June 19, 2023 15:44

fix:apache#464

88963c1

Spark loader has dependency conflicts

fix:apache#488

d8b31fc

bypass server for hbase bulk delete massive vertices and edges

javeme reviewed Jul 10, 2023

View reviewed changes

fix:apache#488

053d27b

bypass server for hbase bulk delete massive vertices and edges

javeme reviewed Jul 12, 2023

View reviewed changes

hugegraph-loader/src/main/java/org/apache/hugegraph/loader/executor/LoadOptions.java Outdated Show resolved Hide resolved

hugegraph-loader/src/main/java/org/apache/hugegraph/loader/constant/Constants.java Outdated Show resolved Hide resolved

fix:apache#488

f817163

bypass server for hbase bulk delete massive vertices and edges

alanzhao added 2 commits August 2, 2023 10:29

Revert "fix:apache#488 bypass server for hbase bulk delete massive ve…

e610fb7

…rtices and edges" This reverts commit 053d27b

Revert "fix:apache#488 bypass server for hbase bulk delete massive ve…

b0918d5

…rtices and edges" This reverts commit d8b31fc

fix:apache#464

5658a87

remove incubating

imbajin previously approved these changes Aug 2, 2023

View reviewed changes

liuxiaocs7 mentioned this pull request Aug 3, 2023

[Bug] Spark Loader example schema and struct mismatch #501

Closed

1 task

liuxiaocs7 reviewed Aug 3, 2023

View reviewed changes

fix:apache#464

841e774

fix delimiter is null

haohao0103 dismissed imbajin’s stale review via 841e774 August 4, 2023 10:09

imbajin reviewed Aug 5, 2023

View reviewed changes

liuxiaocs7 approved these changes Aug 6, 2023

View reviewed changes

imbajin approved these changes Aug 7, 2023

View reviewed changes

simon824 approved these changes Aug 7, 2023

View reviewed changes

simon824 merged commit 0ea28fd into apache:master Aug 7, 2023
10 of 12 checks passed

liuxiaocs7 mentioned this pull request Dec 18, 2023

[Improve] Update loader package shade plugin #562

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Spark with loader has dependency conflicts #480

fix: Spark with loader has dependency conflicts #480

haohao0103 commented Jun 14, 2023

codecov bot commented Jun 14, 2023 •

edited

Loading

haohao0103 commented Jun 14, 2023 •

edited

Loading

javeme left a comment

imbajin commented Jun 14, 2023

imbajin commented Jun 18, 2023

liuxiaocs7 commented Aug 1, 2023 •

edited

Loading

haohao0103 commented Aug 2, 2023

liuxiaocs7 commented Aug 3, 2023

liuxiaocs7 Aug 3, 2023

haohao0103 Aug 4, 2023

haohao0103 commented Aug 4, 2023

imbajin Aug 5, 2023

liuxiaocs7 left a comment

imbajin left a comment

fix: Spark with loader has dependency conflicts #480

fix: Spark with loader has dependency conflicts #480

Conversation

haohao0103 commented Jun 14, 2023

codecov bot commented Jun 14, 2023 • edited Loading

Codecov Report

haohao0103 commented Jun 14, 2023 • edited Loading

javeme left a comment

Choose a reason for hiding this comment

imbajin commented Jun 14, 2023

imbajin commented Jun 18, 2023

liuxiaocs7 commented Aug 1, 2023 • edited Loading

haohao0103 commented Aug 2, 2023

liuxiaocs7 commented Aug 3, 2023

liuxiaocs7 Aug 3, 2023

Choose a reason for hiding this comment

haohao0103 Aug 4, 2023

Choose a reason for hiding this comment

haohao0103 commented Aug 4, 2023

imbajin Aug 5, 2023

Choose a reason for hiding this comment

liuxiaocs7 left a comment

Choose a reason for hiding this comment

imbajin left a comment

Choose a reason for hiding this comment

codecov bot commented Jun 14, 2023 •

edited

Loading

haohao0103 commented Jun 14, 2023 •

edited

Loading

liuxiaocs7 commented Aug 1, 2023 •

edited

Loading