Add a callback 'onBusy' used to adaptive rate limit #1401

Linary · 2021-03-22T07:59:39Z

Change-Id: I03859dbc9dc2ae71a2a47f3072d65276565ebba0

hugegraph-dist/src/assembly/static/conf/rest-server.properties

javeme · 2021-03-23T11:38:59Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/StoreStateMachine.java

@@ -112,6 +113,7 @@ public void onApply(Iterator iter) {
        LOG.debug("Node role: {}", this.node().selfIsLeader() ?
                                   "leader" : "follower");
        StoreClosure closure = null;
+        List<Future<?>> futures = new ArrayList<>();


move to line 138

line 156 need use it

javeme · 2021-03-23T11:40:13Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/StoreStateMachine.java

@@ -133,7 +135,7 @@ public void onApply(Iterator iter) {
                    byte[] bytes = iter.getData().array();
                    // Follower seems no way to wait future
                    // Let the backend thread do it directly
-                    this.context.backendExecutor().submit(() -> {
+                    futures.add(this.context.backendExecutor().submit(() -> {


how many tasks in general?

about 10 ~ 20

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftNode.java

hugegraph-core/src/main/java/com/baidu/hugegraph/StandardHugeGraph.java

javeme · 2021-03-23T11:47:04Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftNode.java

@@ -285,6 +286,13 @@ public void onError(PeerId peer, Status status) {
            }
        }

+        @Override
+        public void onBusy(PeerId peer, Status status) {


seperate into 2 commits:

raft commit for: raft/* and raft-tools.sh

task commit for: TaskManager and rest-server.properties

javeme · 2021-04-02T06:48:05Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/StoreStateMachine.java

                }
                iter.next();
            }
+            System.out.println("futures size: " + count);


javeme · 2021-04-02T06:52:08Z

hugegraph-dist/src/assembly/static/bin/raft-tools.sh

+    print_usage
+    ;;
+    # list-peers
+    --list-peers|-l)


update to list_peers and seperate the commit

codecov · 2021-04-02T08:33:42Z

Codecov Report

Merging #1401 (eed2800) into master (39b9474) will decrease coverage by 0.04%.
The diff coverage is 28.57%.

@@             Coverage Diff              @@
##             master    #1401      +/-   ##
============================================
- Coverage     62.09%   62.05%   -0.05%     
+ Complexity     5854     5853       -1     
============================================
  Files           386      386              
  Lines         32290    32297       +7     
  Branches       4506     4507       +1     
============================================
- Hits          20052    20042      -10     
- Misses        10201    10216      +15     
- Partials       2037     2039       +2

Impacted Files	Coverage Δ	Complexity Δ
.../baidu/hugegraph/license/LicenseVerifyManager.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...m/baidu/hugegraph/backend/store/raft/RaftNode.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...ugegraph/backend/store/raft/RaftSharedContext.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...ugegraph/backend/store/raft/StoreStateMachine.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
.../main/java/com/baidu/hugegraph/util/Consumers.java	`44.21% <25.00%> (-0.38%)`	`18.00 <0.00> (ø)`
...com/baidu/hugegraph/auth/ContextGremlinServer.java	`53.84% <100.00%> (ø)`	`0.00 <0.00> (ø)`
...java/com/baidu/hugegraph/config/ServerOptions.java	`98.91% <100.00%> (ø)`	`0.00 <0.00> (ø)`
...n/java/com/baidu/hugegraph/config/CoreOptions.java	`99.45% <100.00%> (ø)`	`2.00 <1.00> (ø)`
...raph/backend/store/rocksdb/RocksDBStdSessions.java	`64.45% <100.00%> (ø)`	`37.00 <0.00> (ø)`
...va/com/baidu/hugegraph/task/ServerInfoManager.java	`70.78% <0.00%> (-2.81%)`	`51.00% <0.00%> (-1.00%)`
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 39b9474...eed2800. Read the comment docs.

javeme · 2021-04-02T11:45:30Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/StoreStateMachine.java

                }
                iter.next();
            }
+            System.out.println("futures size: " + count);


remove the println

javeme · 2021-04-02T11:45:50Z

hugegraph-dist/src/assembly/static/bin/raft-tools.sh

+    print_usage
+    ;;
+    # list-peers
+    --list-peers|-l)


update to list_peers and seperate the commit

Change-Id: I2a1139f6e436744ee6b20557ec18f2181b2e63be

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftNode.java

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/StoreStateMachine.java

Change-Id: Ifc12c9b408ee24583e42fcd4842915ec2a71e554

javeme · 2021-04-13T07:32:44Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftSharedContext.java

@@ -362,7 +362,7 @@ private ExecutorService createBackendExecutor(int threads) {
    private static ExecutorService newPool(int coreThreads, int maxThreads,
                                           String name,
                                           RejectedExecutionHandler handler) {
-        BlockingQueue<Runnable> workQueue = new LinkedBlockingQueue<>();
+        BlockingQueue<Runnable> workQueue = new SynchronousQueue<>();


userArrayBlockingQueue with capacity

Change-Id: Ib74c3df54ed2f7cbbe27a76d6f344173792bf654

Change-Id: I2a6a8b39f7066008483fde2c8fb66fa7db32ca0e

javeme · 2021-04-13T12:37:03Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftSharedContext.java

@@ -73,6 +74,7 @@
    public static final int WAIT_RPC_TIMEOUT = 30 * 60 * 1000;
    // compress block size
    public static final int BLOCK_SIZE = 8192;
+    public static final int QUEUE_SIZE = 10;


coderzc previously approved these changes Mar 22, 2021

View reviewed changes

javeme requested changes Mar 23, 2021

View reviewed changes

Linary dismissed coderzc’s stale review via 35cd7de April 2, 2021 05:10

Linary force-pushed the raft-improve branch from ad2cc92 to 35cd7de Compare April 2, 2021 05:10

Linary changed the title ~~Some improve for raft backend~~ Add a callback 'onBusy' used to adaptive rate limit Apr 2, 2021

javeme reviewed Apr 2, 2021

View reviewed changes

Add a callback 'onBusy' used to adaptive rate limit

2bc2ab8

Change-Id: I2a1139f6e436744ee6b20557ec18f2181b2e63be

Linary force-pushed the raft-improve branch from 74f5209 to 2bc2ab8 Compare April 8, 2021 10:03

javeme previously approved these changes Apr 9, 2021

View reviewed changes

zhoney requested changes Apr 13, 2021

View reviewed changes

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftNode.java Outdated Show resolved Hide resolved

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/StoreStateMachine.java Outdated Show resolved Hide resolved

tiny improve

3b7837f

Change-Id: Ifc12c9b408ee24583e42fcd4842915ec2a71e554

Linary dismissed javeme’s stale review via 3b7837f April 13, 2021 07:16

zhoney previously approved these changes Apr 13, 2021

View reviewed changes

javeme requested changes Apr 13, 2021

View reviewed changes

Linary dismissed zhoney’s stale review via fdce052 April 13, 2021 07:45

improve some comment

706c316

Change-Id: Ib74c3df54ed2f7cbbe27a76d6f344173792bf654

Linary force-pushed the raft-improve branch from fdce052 to 706c316 Compare April 13, 2021 07:50

tiny improve

eed2800

Change-Id: I2a6a8b39f7066008483fde2c8fb66fa7db32ca0e

javeme reviewed Apr 13, 2021

View reviewed changes

javeme approved these changes Apr 13, 2021

View reviewed changes

zhoney approved these changes Apr 14, 2021

View reviewed changes

zhoney merged commit ec974d0 into master Apr 14, 2021

zhoney deleted the raft-improve branch April 14, 2021 02:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a callback 'onBusy' used to adaptive rate limit #1401

Add a callback 'onBusy' used to adaptive rate limit #1401

Linary commented Mar 22, 2021 •

edited

Loading

javeme Mar 23, 2021

Linary Apr 2, 2021

javeme Mar 23, 2021

Linary Apr 2, 2021 •

edited

Loading

javeme Mar 23, 2021

Linary Apr 2, 2021

javeme Apr 2, 2021

javeme Apr 2, 2021

codecov bot commented Apr 2, 2021 •

edited

Loading

javeme Apr 2, 2021

javeme Apr 2, 2021

javeme Apr 13, 2021

javeme Apr 13, 2021

Add a callback 'onBusy' used to adaptive rate limit #1401

Add a callback 'onBusy' used to adaptive rate limit #1401

Conversation

Linary commented Mar 22, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Linary Apr 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Apr 2, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Linary commented Mar 22, 2021 •

edited

Loading

Linary Apr 2, 2021 •

edited

Loading

codecov bot commented Apr 2, 2021 •

edited

Loading