support: invalid cache through rpc #1357

javeme · 2021-02-11T19:10:23Z

Change-Id: I3fe18cd8c893a84a56fb66df651b78c5e0fe5f4b

Change-Id: I65b2da4e8dd3b89440e7c4d715a87ea95d6c14ac

Change-Id: I2f8edb2c70b1ad185f66da527af0e34557c6c889

Change-Id: I4e4abf3b71a6182e8f16af1cd17f60c2690c0767

Change-Id: I8723f401ee430519bc488554c31da16e4fdffdb4

Change-Id: Iefde073a4f07b5bad4cc08f065a3ee0f034114d8

Change-Id: I9c6c5f5e069d2e6f4aad9ec7adbb3dba26de2460

Change-Id: Id45b11844ed5c0a7e6cfb672057c42b455119ba3

javeme · 2021-02-13T04:28:49Z

Cassandra/Scylla ci error:

HBase ci error:

codecov · 2021-02-13T05:00:58Z

Codecov Report

Merging #1357 (8da4366) into master (b4c767b) will decrease coverage by 7.11%.
The diff coverage is 40.03%.

@@             Coverage Diff              @@
##             master    #1357      +/-   ##
============================================
- Coverage     69.19%   62.08%   -7.12%     
- Complexity     5401     5835     +434     
============================================
  Files           328      386      +58     
  Lines         26334    32169    +5835     
  Branches       3750     4492     +742     
============================================
+ Hits          18222    19972    +1750     
- Misses         6333    10168    +3835     
- Partials       1779     2029     +250

Impacted Files	Coverage Δ	Complexity Δ
...api/src/main/java/com/baidu/hugegraph/api/API.java	`69.04% <ø> (+3.49%)`	`0.00 <0.00> (ø)`
...n/java/com/baidu/hugegraph/api/auth/AccessAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...n/java/com/baidu/hugegraph/api/auth/BelongAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...in/java/com/baidu/hugegraph/api/auth/GroupAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...n/java/com/baidu/hugegraph/api/auth/TargetAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...ain/java/com/baidu/hugegraph/api/auth/UserAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...com/baidu/hugegraph/api/gremlin/GremlinClient.java	`90.90% <ø> (ø)`	`0.00 <0.00> (ø)`
.../java/com/baidu/hugegraph/api/job/ComputerAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...va/com/baidu/hugegraph/api/metrics/MetricsAPI.java	`65.78% <ø> (ø)`	`0.00 <0.00> (ø)`
...ain/java/com/baidu/hugegraph/api/raft/RaftAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
... and 373 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a50c4c8...8da4366. Read the comment docs.

Linary · 2021-02-24T11:04:27Z

hugegraph-api/src/main/java/com/baidu/hugegraph/auth/ConfigAuthenticator.java

-        this.tokens.put(USER_ADMIN, config.get(ServerOptions.ADMIN_TOKEN));
-        this.tokens.putAll(config.getMap(ServerOptions.USER_TOKENS));
+        this.tokens.put(USER_ADMIN, config.get(ServerOptions.AUTH_ADMIN_TOKEN));
+        this.tokens.putAll(config.getMap(ServerOptions.AUTH_USER_TOKENS));


need check AUTH_USER_TOKENS doesn't contains user named 'admin'?

we can change their order to ensure the admin stays in the map

Linary · 2021-02-24T11:14:54Z

hugegraph-api/src/main/java/com/baidu/hugegraph/rpc/RpcClientProvider.java

+        String rpcUrl = conf.get(ServerOptions.RPC_REMOTE_URL);
+        String selfUrl = conf.get(ServerOptions.RPC_SERVER_HOST) + ":" +
+                         conf.get(ServerOptions.RPC_SERVER_PORT);
+        rpcUrl = execludeSelfUrl(rpcUrl, selfUrl);


Linary · 2021-02-24T11:57:35Z

hugegraph-api/src/main/java/com/baidu/hugegraph/rpc/RpcConsumerConfig.java

+            }
+
+            if (responses.size() > 0) {
+                // Just choose one


Linary · 2021-02-24T11:58:37Z

hugegraph-api/src/main/java/com/baidu/hugegraph/rpc/RpcConsumerConfig.java

+        }
+
+        private static String methodName(SofaRequest request) {
+            String method = request.getInterfaceName() + "." +


just return is enough

hugegraph-api/src/main/java/com/baidu/hugegraph/rpc/RpcServer.java

Linary · 2021-02-24T12:09:13Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/cache/CacheNotifier.java

+
+    public void invalid(HugeType type, Id id);
+
+    public void invalid2(HugeType type, Object[] ids);


any friendship name?

sofa-rpc don't allowed to use the same method name in a service, may be names like invalidBatch() is more friendship, here just to keep the name short.

Change-Id: I23878e7991774dbb3dcc7723137bac314502cc68

Change-Id: Ia671437ae51f88f8eebb3fcdac20321af65d10a9

zhoney · 2021-03-03T12:13:10Z

hugegraph-api/src/main/java/com/baidu/hugegraph/rpc/RpcClientProvider.java

-        return config.refer();
+    private static String excludeSelfUrl(String rpcUrl, String selfUrl) {
+        String[] urls = StringUtils.splitWithCommaOrSemicolon(rpcUrl);
+        Set<String> urlsSet = new LinkedHashSet<>(Arrays.asList(urls));


zhoney · 2021-03-03T12:39:48Z

hugegraph-api/src/main/java/com/baidu/hugegraph/rpc/RpcConsumerConfig.java

+            } else if (excepts.size() > 0) {
+                throw excepts.get(0);
+            } else {
+                assert providers.isEmpty();


check providers.isEmpty() in line 135

empty provider must come here

Change-Id: Ic24ef99b74edf8e95b701bb6509f127a5d4eee91

Change-Id: I8a8266d7d618410dcff2e882dcbc71c9d71f66e5

javeme added 8 commits February 12, 2021 03:03

support: invalid cache through rpc

fc7e50c

Change-Id: I3fe18cd8c893a84a56fb66df651b78c5e0fe5f4b

fix test

4a7ee70

Change-Id: I65b2da4e8dd3b89440e7c4d715a87ea95d6c14ac

add RpcServiceConfig* interfaces

f3f8dd2

Change-Id: I2f8edb2c70b1ad185f66da527af0e34557c6c889

add FanoutCluster to broadcast reuqest

b44c873

Change-Id: I4e4abf3b71a6182e8f16af1cd17f60c2690c0767

fix GraphManager.close() is not called by HttpServer.DESTROY_FINISHED

3d57156

Change-Id: I8723f401ee430519bc488554c31da16e4fdffdb4

notify cache with Id[] arg to avoid too many events

883b7f3

Change-Id: Iefde073a4f07b5bad4cc08f065a3ee0f034114d8

improve perf of raft notify-cache

cbb45f7

Change-Id: I9c6c5f5e069d2e6f4aad9ec7adbb3dba26de2460

only enable cache-rpc service for shared storage

578a16c

Change-Id: Id45b11844ed5c0a7e6cfb672057c42b455119ba3

javeme force-pushed the rpc-invalid-cache branch from 46f1cc6 to 578a16c Compare February 12, 2021 18:34

Linary requested changes Feb 24, 2021

View reviewed changes

javeme added 2 commits February 25, 2021 14:10

tiny improve

cad6dc4

Change-Id: I23878e7991774dbb3dcc7723137bac314502cc68

check raft mode can only be enabled if non-shared store

ace11ad

Change-Id: Ia671437ae51f88f8eebb3fcdac20321af65d10a9

Linary previously approved these changes Mar 2, 2021

View reviewed changes

Merge branch 'master' into rpc-invalid-cache

011b118

javeme dismissed Linary’s stale review via 011b118 March 3, 2021 08:07

Linary previously approved these changes Mar 3, 2021

View reviewed changes

javeme dismissed Linary’s stale review via ec012eb March 3, 2021 12:43

zhoney requested changes Mar 3, 2021

View reviewed changes

Merge branch 'master' into rpc-invalid-cache

988480e

Change-Id: Ic24ef99b74edf8e95b701bb6509f127a5d4eee91

javeme force-pushed the rpc-invalid-cache branch from ec012eb to 988480e Compare March 3, 2021 13:37

tiny improve

8da4366

Change-Id: I8a8266d7d618410dcff2e882dcbc71c9d71f66e5

javeme force-pushed the rpc-invalid-cache branch from 3039823 to 8da4366 Compare March 4, 2021 10:36

Linary approved these changes Mar 10, 2021

View reviewed changes

zhoney approved these changes Mar 11, 2021

View reviewed changes

zhoney merged commit 314fd8a into master Mar 11, 2021

zhoney deleted the rpc-invalid-cache branch March 11, 2021 13:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support: invalid cache through rpc #1357

support: invalid cache through rpc #1357

javeme commented Feb 11, 2021

javeme commented Feb 13, 2021

codecov bot commented Feb 13, 2021 •

edited

Loading

Linary Feb 24, 2021

javeme Feb 25, 2021

Linary Feb 24, 2021

Linary Feb 24, 2021

Linary Feb 24, 2021

Linary Feb 24, 2021

javeme Feb 25, 2021

zhoney Mar 3, 2021

zhoney Mar 3, 2021

javeme Mar 3, 2021 •

edited

Loading


		public void invalid(HugeType type, Id id);

		public void invalid2(HugeType type, Object[] ids);

support: invalid cache through rpc #1357

support: invalid cache through rpc #1357

Conversation

javeme commented Feb 11, 2021

javeme commented Feb 13, 2021

codecov bot commented Feb 13, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javeme Mar 3, 2021 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Feb 13, 2021 •

edited

Loading

javeme Mar 3, 2021 •

edited

Loading