[release/microbenchmarks] Please take a look at the results (and possibly sign off) #15247

krfricke · 2021-04-12T11:50:36Z

single client get calls per second 45451.85 +- 260.67
single client put calls per second 34434.73 +- 1169.37
multi client put calls per second 147515.09 +- 4571.9
single client get calls (Plasma Store) per second 8851.22 +- 70.09
single client put calls (Plasma Store) per second 5454.56 +- 8.64
multi client put calls (Plasma Store) per second 9306.63 +- 120.43
single client put gigabytes per second 16.03 +- 11.37
multi client put gigabytes per second 37.95 +- 1.8
single client tasks sync per second 1658.41 +- 47.78
single client tasks async per second 15046.33 +- 516.59
multi client tasks async per second 43697.03 +- 5338.69
1:1 actor calls sync per second 2710.45 +- 68.32
1:1 actor calls async per second 7021.72 +- 163.53
1:1 actor calls concurrent per second 7201.4 +- 198.41
1:n actor calls async per second 16829.7 +- 349.17
n:n actor calls async per second 46951.12 +- 651.15
n:n actor calls with arg async per second 13085.9 +- 171.39
1:1 async-actor calls sync per second 1754.34 +- 46.87
1:1 async-actor calls async per second 3870.6 +- 168.3
1:1 async-actor calls with args async per second 2637.22 +- 77.39
1:n async-actor calls async per second 14599.06 +- 515.27
n:n async-actor calls async per second 37827.88 +- 860.12
client: get calls per second 2155.77 +- 80.11
client: put calls per second 1190.19 +- 9.34
client: remote put calls per second 9119.79 +- 83.24
client: 1:1 actor calls sync per second 649.32 +- 15.85
client: 1:1 actor calls async per second 712.39 +- 2.36
client: 1:1 actor calls concurrent per second 715.0 +- 5.37

Results from 1.2.0:

single client get calls per second 48106.48 +- 847.52
single client put calls per second 42709.1 +- 84.85
multi client put calls per second 172608.71 +- 3071.81
single client get calls (Plasma Store) per second 10669.26 +- 286.63
single client put calls (Plasma Store) per second 6622.51 +- 47.03
multi client put calls (Plasma Store) per second 9804.51 +- 462.32
single client put gigabytes per second 11.45 +- 10.79
multi client put gigabytes per second 35.06 +- 0.26
single client tasks sync per second 1899.11 +- 87.63
single client tasks async per second 18599.58 +- 124.02
multi client tasks async per second 50388.88 +- 2585.47
1:1 actor calls sync per second 3053.21 +- 60.37
1:1 actor calls async per second 7768.59 +- 268.78
1:1 actor calls concurrent per second 7106.24 +- 219.87
1:n actor calls async per second 17132.11 +- 881.8
n:n actor calls async per second 51037.11 +- 1732.95
n:n actor calls with arg async per second 13746.19 +- 171.94
1:1 async-actor calls sync per second 2103.39 +- 52.51
1:1 async-actor calls async per second 4100.13 +- 53.6
1:1 async-actor calls with args async per second 3085.78 +- 165.8
1:n async-actor calls async per second 13906.28 +- 363.9
n:n async-actor calls async per second 40269.65 +- 1113.55
client: get calls per second 2414.77 +- 43.07
client: put calls per second 1346.13 +- 8.2
client: remote put calls per second 58855.54 +- 849.21
client: 1:1 actor calls sync per second 730.58 +- 11.66
client: 1:1 actor calls async per second 774.79 +- 14.1
client: 1:1 actor calls concurrent per second 805.73 +- 11.46

It seems there are a few regressions here, e.g.:

multi client put calls per second 147515.09 +- 4571.9
vs.
multi client put calls per second 172608.71 +- 3071.81

and notably

client: remote put calls per second 9119.79 +- 83.24
vs.
client: remote put calls per second 58855.54 +- 849.21

I also got this

2021-04-12 04:45:15,897 WARNING services.py:1726 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 53772902400 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=55.09gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM.

is this the reason for the regressions?

cc @wuisawesome @rkooo567 @ericl

The text was updated successfully, but these errors were encountered:

krfricke · 2021-04-12T11:52:20Z

Note: For these benchmarks the cluster was stopped as it would otherwise complain about passing a _system_config argument to ray.init(). So for each of the separate benchmarks, ray.init() would start the cluster anew.

wuisawesome · 2021-04-12T18:37:17Z

@AmeerHajAli is the client stable enough for these benchmarks to be meaningful?

@krfricke do you have the output of release/util/microbenchmark_analysis.py?

krfricke · 2021-04-12T18:39:07Z

I'll run this shortly

rkooo567 · 2021-04-12T19:48:05Z

It seems like the single client put call also has regression actually.

rkooo567 · 2021-04-12T19:48:46Z

So, it looks like put calls have 20%~ ish regression in general.

rkooo567 · 2021-04-12T19:48:54Z

Is it possible to bisect this?

rkooo567 · 2021-04-12T20:03:46Z

@krfricke Ignore my comment above. Is this error message appearing in every test?

2021-04-12 04:45:15,897 WARNING services.py:1726 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 53772902400 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=55.09gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM.

krfricke · 2021-04-12T20:05:28Z

New fresh run:

single client get calls per second 47682.41 +- 676.48
single client put calls per second 34652.4 +- 270.37
multi client put calls per second 145246.14 +- 8285.84
single client get calls (Plasma Store) per second 10131.4 +- 86.98
single client put calls (Plasma Store) per second 5944.66 +- 102.8
multi client put calls (Plasma Store) per second 9783.34 +- 148.88
single client put gigabytes per second 15.54 +- 10.72
multi client put gigabytes per second 34.76 +- 0.39
single client tasks sync per second 1713.59 +- 46.24
single client tasks async per second 16620.92 +- 365.9
multi client tasks async per second 45571.81 +- 2395.59
1:1 actor calls sync per second 2763.12 +- 72.25
1:1 actor calls async per second 6548.38 +- 92.55
1:1 actor calls concurrent per second 7233.52 +- 346.27
1:n actor calls async per second 16056.12 +- 380.82
n:n actor calls async per second 49234.5 +- 850.7
n:n actor calls with arg async per second 14163.01 +- 39.25
1:1 async-actor calls sync per second 1940.7 +- 76.88
1:1 async-actor calls async per second 4277.24 +- 64.95
1:1 async-actor calls with args async per second 2914.13 +- 50.13
1:n async-actor calls async per second 15354.11 +- 723.7
n:n async-actor calls async per second 38910.18 +- 486.24
client: get calls per second 2282.7 +- 78.73
client: put calls per second 1285.64 +- 9.3
client: remote put calls per second 9593.84 +- 162.54
client: 1:1 actor calls sync per second 716.77 +- 4.13
client: 1:1 actor calls async per second 781.81 +- 8.67
client: 1:1 actor calls concurrent per second 785.81 +- 10.37

Comparison with 1.2.0:

kai@Kais-MacBook-Pro:~/coding/ray/release/ > python util/microbenchmark_analysis.py
single client get calls (Plasma Store)  performance change rate: -5.04%
single client put calls (Plasma Store)  performance change rate: -10.24%
single client put gigabytes  performance change rate: 35.72%
multi client put calls (Plasma Store)  performance change rate: -0.22%
multi client put gigabytes  performance change rate: -0.86%
single client tasks sync  performance change rate: -9.77%
single client tasks async  performance change rate: -10.64%
multi client tasks async  performance change rate: -9.56%
1:1 actor calls sync  performance change rate: -9.5%
1:1 actor calls async  performance change rate: -15.71%
n:n actor calls async  performance change rate: -3.53%
1:1 actor calls concurrent  performance change rate: 1.79%
1:n actor calls async  performance change rate: -6.28%
n:n actor calls with arg async  performance change rate: 3.03%
1:1 async-actor calls sync  performance change rate: -7.73%
1:1 async-actor calls async  performance change rate: 4.32%
1:1 async-actor calls with args async  performance change rate: -5.56%
1:n async-actor calls async  performance change rate: 10.41%
n:n async-actor calls async  performance change rate: -3.38%
single client get calls  performance change rate: -0.88%
single client put calls  performance change rate: -18.86%
multi client put calls  performance change rate: -15.85%
client: get calls  performance change rate: -5.47%
client: put calls  performance change rate: -4.49%
client: remote put calls  performance change rate: -83.7%
client: 1:1 actor calls sync  performance change rate: -1.89%
client: 1:1 actor calls async  performance change rate: 0.91%
client: 1:1 actor calls concurrent  performance change rate: -2.47%

krfricke · 2021-04-12T20:05:42Z

The error message did not appear this time (and previously only once)

ericl · 2021-04-12T21:24:20Z

Sang can you look into why that message might be appearing? We should be always trying to use devshm when possible, the error message makes it seem like there is some rounding error where we detected 50GB but it only has 49GB or something.

…

On Mon, Apr 12, 2021, 1:05 PM Kai Fricke ***@***.***> wrote: The error message did not appear this time — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#15247 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADUSRXFWUITXSGQYUXFBDTINHCJANCNFSM42ZEIYEA> .

rkooo567 · 2021-04-12T21:36:23Z

cc @wuisawesome ^

wuisawesome · 2021-04-12T22:42:58Z

@krfricke it looks like most of these tests have performance regressions. Can we bisect to figure out when the task/actor submission performance dropped? And also when the ray.put performance dropped?

wuisawesome · 2021-04-14T01:45:02Z

Did some bisecting.

After 6a0b3062214c7fcec3aff67981eca9e606dd7ee9:

Tip: set TESTS_TO_RUN='pattern' to run a subset of benchmarks
2021-04-13 18:12:13,793	INFO services.py:1195 -- View the Ray dashboard at �[1m�[32mhttp://127.0.0.1:8266�[39m�[22m
single client get calls per second 46521.78 +- 816.73
single client put calls per second 36183.25 +- 42.45
multi client put calls per second 152263.31 +- 2028.55
2021-04-13 18:12:45,513	INFO services.py:1195 -- View the Ray dashboard at �[1m�[32mhttp://127.0.0.1:8266�[39m�[22m
single client get calls (Plasma Store) per second 9234.92 +- 168.31
single client put calls (Plasma Store) per second 5580.05 +- 89.68
multi client put calls (Plasma Store) per second 9422.23 +- 253.22
single client put gigabytes per second 14.44 +- 10.79
multi client put gigabytes per second 38.03 +- 2.2
single client tasks sync per second 1731.38 +- 42.0
single client tasks async per second 15884.15 +- 1004.75
multi client tasks async per second 49343.22 +- 1560.82
1:1 actor calls sync per second 2901.35 +- 75.86
1:1 actor calls async per second 7042.07 +- 450.14
1:1 actor calls concurrent per second 6775.94 +- 182.52
1:n actor calls async per second 15792.74 +- 460.5
n:n actor calls async per second 44905.41 +- 1414.97
n:n actor calls with arg async per second 12814.23 +- 184.17
1:1 async-actor calls sync per second 1858.18 +- 91.31
1:1 async-actor calls async per second 4412.72 +- 137.75
1:1 async-actor calls with args async per second 2880.37 +- 137.65
1:n async-actor calls async per second 13850.25 +- 738.41
n:n async-actor calls async per second 37433.47 +- 868.49
2021-04-13 18:17:35,071	WARNING services.py:1662 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 50417352704 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=81.51gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM.
client: get calls per second 2227.32 +- 60.8
2021-04-13 18:17:48,596	INFO logservicer.py:85 -- New logs connection
2021-04-13 18:17:49,004	INFO services.py:1195 -- View the Ray dashboard at �[1m�[32mhttp://127.0.0.1:8266�[39m�[22m
client: put calls per second 1056.08 +- 60.39
2021-04-13 18:17:59,652	INFO dataservicer.py:70 -- Lost data connection from client 15c5bd7a6b4e442bb3d783fa5b58ec74
2021-04-13 18:18:02,306	INFO dataservicer.py:34 -- New data connection from client feb181191b2748eb94967942816eabd4
2021-04-13 18:18:02,307	INFO logservicer.py:85 -- New logs connection
2021-04-13 18:18:02,824	INFO services.py:1195 -- View the Ray dashboard at �[1m�[32mhttp://127.0.0.1:8266�[39m�[22m
client: remote put calls per second 8568.32 +- 140.53
2021-04-13 18:18:13,839	INFO dataservicer.py:70 -- Lost data connection from client feb181191b2748eb94967942816eabd4
2021-04-13 18:18:17,122	INFO dataservicer.py:34 -- New data connection from client 165c7c89d8fa40d8bdc5ac0358af3ee3
2021-04-13 18:18:17,123	INFO logservicer.py:85 -- New logs connection
2021-04-13 18:18:17,657	INFO services.py:1195 -- View the Ray dashboard at �[1m�[32mhttp://127.0.0.1:8266�[39m�[22m
client: 1:1 actor calls sync per second 668.51 +- 17.67
client: 1:1 actor calls async per second 758.64 +- 28.13
client: 1:1 actor calls concurrent per second 781.57 +- 4.78
2021-04-13 18:18:51,612	INFO dataservicer.py:70 -- Lost data connection from client 165c7c89d8fa40d8bdc5ac0358af3ee3
�[0m�[1m�[7m#�[27m�[1m�[0m

After 3ffe375a095f1fc2547c2ca6f12663f52c001c53:

Tip: set TESTS_TO_RUN='pattern' to run a subset of benchmarks
2021-04-13 18:22:21,312	INFO services.py:1195 -- View the Ray dashboard at �[1m�[32mhttp://127.0.0.1:8266�[39m�[22m
single client get calls per second 46730.87 +- 423.22
single client put calls per second 36110.28 +- 132.13
multi client put calls per second 169110.22 +- 2119.74
2021-04-13 18:22:52,818	INFO services.py:1195 -- View the Ray dashboard at �[1m�[32mhttp://127.0.0.1:8266�[39m�[22m
single client get calls (Plasma Store) per second 10181.12 +- 151.06
single client put calls (Plasma Store) per second 6640.91 +- 52.16
multi client put calls (Plasma Store) per second 10659.44 +- 219.8
single client put gigabytes per second 12.65 +- 12.32
multi client put gigabytes per second 41.32 +- 1.25
single client tasks sync per second 1959.22 +- 39.29
single client tasks async per second 18204.43 +- 719.7
multi client tasks async per second 50705.1 +- 2021.22
1:1 actor calls sync per second 3215.29 +- 47.85
1:1 actor calls async per second 8293.83 +- 99.5
1:1 actor calls concurrent per second 7707.23 +- 319.14
1:n actor calls async per second 18850.19 +- 393.12
n:n actor calls async per second 49570.41 +- 585.37
n:n actor calls with arg async per second 14773.22 +- 64.67
1:1 async-actor calls sync per second 2133.95 +- 8.53
1:1 async-actor calls async per second 4518.24 +- 29.45
1:1 async-actor calls with args async per second 3185.96 +- 48.48
1:n async-actor calls async per second 16261.3 +- 718.6
n:n async-actor calls async per second 40460.7 +- 946.18
2021-04-13 18:27:21,791	WARNING services.py:1662 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 49483083776 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=80.64gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM.
client: get calls per second 2394.5 +- 41.88
2021-04-13 18:27:35,386	INFO logservicer.py:85 -- New logs connection
2021-04-13 18:27:35,788	INFO services.py:1195 -- View the Ray dashboard at �[1m�[32mhttp://127.0.0.1:8266�[39m�[22m
client: put calls per second 1291.03 +- 15.35
2021-04-13 18:27:46,428	INFO dataservicer.py:70 -- Lost data connection from client 70221302345a4ca4bd4ba913fc8d0cea
2021-04-13 18:27:48,883	INFO dataservicer.py:34 -- New data connection from client b72896ae7ef94e3e8f42fde76c96d71f
2021-04-13 18:27:48,884	INFO logservicer.py:85 -- New logs connection
2021-04-13 18:27:49,290	INFO services.py:1195 -- View the Ray dashboard at �[1m�[32mhttp://127.0.0.1:8266�[39m�[22m
client: remote put calls per second 9723.93 +- 177.26
2021-04-13 18:28:00,131	INFO dataservicer.py:70 -- Lost data connection from client b72896ae7ef94e3e8f42fde76c96d71f
2021-04-13 18:28:03,357	INFO dataservicer.py:34 -- New data connection from client 19b9f49c3ba44bfb9298344d05260596
2021-04-13 18:28:03,358	INFO logservicer.py:85 -- New logs connection
2021-04-13 18:28:03,764	INFO services.py:1195 -- View the Ray dashboard at �[1m�[32mhttp://127.0.0.1:8266�[39m�[22m
client: 1:1 actor calls sync per second 733.34 +- 5.9
client: 1:1 actor calls async per second 854.29 +- 7.12
client: 1:1 actor calls concurrent per second 830.45 +- 17.46
2021-04-13 18:28:35,820	INFO dataservicer.py:70 -- Lost data connection from client 19b9f49c3ba44bfb9298344d05260596
�[0m�[1m�[7m#�[27m�[1m�[0m

Conclusion: #14192 probably caused the multi client put regression.

krfricke · 2021-04-14T10:04:31Z

Thanks Alex for bisecting this. I'll try to find the client remote put calls regression.

AmeerHajAli · 2021-04-14T10:08:09Z

I wonder why client: remote put calls performance change rate: performance dropped -83.7%? @ijrsvt This looks fishy to me. Can you please help investigate this?

krfricke · 2021-04-14T11:16:48Z

Found the regression.

4b4941435d42e5b7329388de137a590f376d18bb:

client: remote put calls per second 56596.72 +- 665.3

3a230fa1a439a7c6b56099d450faf5702ac5b4ae

client: remote put calls per second 8775.76 +- 172.0

Relevant PR: #13919

cc @richardliaw

rkooo567 · 2021-04-14T17:00:31Z

Hey @richardliaw Do you take a look at this issue now?

richardliaw · 2021-04-14T17:02:30Z

I am taking a look right now, yes

wuisawesome · 2021-04-15T01:19:53Z

@richardliaw assigning you for the ray client regressions

@rkooo567 assigning you since you own #15287 (the multi client put regression).

amogkam · 2021-04-15T08:29:06Z

I tried this again on the most up to date release branch and am getting these results

single client get calls per second 48601.85 +- 1117.72
single client put calls per second 40772.64 +- 212.07
multi client put calls per second 152117.52 +- 7964.62
single client get calls (Plasma Store) per second 10729.88 +- 172.84
single client put calls (Plasma Store) per second 6228.85 +- 92.13
multi client put calls (Plasma Store) per second 10129.35 +- 142.65
single client put gigabytes per second 14.47 +- 12.33
multi client put gigabytes per second 35.34 +- 0.23
single client tasks sync per second 1769.26 +- 32.3
single client tasks async per second 18126.94 +- 437.77
multi client tasks async per second 44223.88 +- 1900.22
1:1 actor calls sync per second 2916.96 +- 37.23
1:1 actor calls async per second 6567.69 +- 355.57
1:1 actor calls concurrent per second 6670.2 +- 333.41
1:n actor calls async per second 17785.68 +- 670.6
n:n actor calls async per second 45672.27 +- 1636.68
n:n actor calls with arg async per second 13847.58 +- 348.7
1:1 async-actor calls sync per second 1989.68 +- 16.89
1:1 async-actor calls async per second 4230.18 +- 159.26
1:1 async-actor calls with args async per second 2899.82 +- 132.57
1:n async-actor calls async per second 14541.01 +- 1992.84
n:n async-actor calls async per second 37923.7 +- 1284.69
client: get calls per second 2469.61 +- 29.88
client: put calls per second 1283.07 +- 10.1
client: remote put calls per second 9451.17 +- 60.11
client: 1:1 actor calls sync per second 705.2 +- 6.06
client: 1:1 actor calls async per second 760.94 +- 9.3
client: 1:1 actor calls concurrent per second 765.69 +- 4.79

single client get calls (Plasma Store)  performance change rate: 0.57%
single client put calls (Plasma Store)  performance change rate: -5.94%
single client put gigabytes  performance change rate: 26.38%
multi client put calls (Plasma Store)  performance change rate: 3.31%
multi client put gigabytes  performance change rate: 0.8%
single client tasks sync  performance change rate: -6.84%
single client tasks async  performance change rate: -2.54%
multi client tasks async  performance change rate: -12.23%
1:1 actor calls sync  performance change rate: -4.46%
1:1 actor calls async  performance change rate: -15.46%
n:n actor calls async  performance change rate: -10.51%
1:1 actor calls concurrent  performance change rate: -6.14%
1:n actor calls async  performance change rate: 3.81%
n:n actor calls with arg async  performance change rate: 0.74%
1:1 async-actor calls sync  performance change rate: -5.41%
1:1 async-actor calls async  performance change rate: 3.17%
1:1 async-actor calls with args async  performance change rate: -6.03%
1:n async-actor calls async  performance change rate: 4.56%
n:n async-actor calls async  performance change rate: -5.83%
single client get calls  performance change rate: 1.03%
single client put calls  performance change rate: -4.53%
multi client put calls  performance change rate: -11.87%
client: get calls  performance change rate: 2.27%
client: put calls  performance change rate: -4.68%
client: remote put calls  performance change rate: -83.94%
client: 1:1 actor calls sync  performance change rate: -3.47%
client: 1:1 actor calls async  performance change rate: -1.79%
client: 1:1 actor calls concurrent  performance change rate: -4.97%

@rkooo567 looks like merging #15287 did not do the trick. multi client put still has ~12% performance regression.

amogkam · 2021-04-15T08:30:22Z

Actually @wuisawesome I'm not able to replicate your results. I tried pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/master/3ffe375a095f1fc2547c2ca6f12663f52c001c53/ray-2.0.0.dev0-cp37-cp37m-manylinux2014_x86_64.whl

and am still getting

multi client put calls per second 155838.94 +- 3684.28

amogkam · 2021-04-15T08:35:14Z

Also shouldn't 1:1 actor calls async and multi client tasks async also be a concern?

rkooo567 · 2021-04-15T16:29:50Z

Hmm @wuisawesome @amogkam can you guys try bisecting again to isolate the commit?

amogkam · 2021-04-15T16:54:49Z

Yep I can bisect this

krfricke · 2021-04-16T12:02:31Z

Running bisect now

krfricke · 2021-04-16T16:12:01Z

Here seems to be a regression:



6f151ad510e5f545a74475ab25e4229c72688c8d

Tip: set TESTS_TO_RUN='pattern' to run a subset of benchmarks
1:1 actor calls async per second 7650.62 +- 253.85
client: 1:1 actor calls async per second 771.6 +- 43.09

Tip: set TESTS_TO_RUN='pattern' to run a subset of benchmarks
multi client tasks async per second 48900.06 +- 3476.46

$

26907b7708f44a2c07603d0ac04b30c094e7afa3

Tip: set TESTS_TO_RUN='pattern' to run a subset of benchmarks
1:1 actor calls async per second 6807.07 +- 93.14
client: 1:1 actor calls async per second 754.67 +- 23.86

Tip: set TESTS_TO_RUN='pattern' to run a subset of benchmarks
multi client tasks async per second 43283.88 +- 959.73

Relevant PR: #14345

Hm wait, it seems this PR actually improved performance...

ijrsvt · 2021-04-16T16:45:20Z

@krfricke That PR was reverted because it broke memory detection for Docker & K8s clusters. (See #14556)

rkooo567 · 2021-04-16T21:02:44Z

I think 26907b7 should be a noise. It only changes Ray Java code.

krfricke · 2021-04-19T16:54:57Z

Could this be a regression here?


72d87093b9712f16ca3060de27cb6020b131c106

Tip: set TESTS_TO_RUN='pattern' to run a subset of benchmarks
1:1 actor calls async per second 6564.06 +- 187.44
client: 1:1 actor calls async per second 814.89 +- 12.84

Tip: set TESTS_TO_RUN='pattern' to run a subset of benchmarks
multi client tasks async per second 47831.05 +- 2611.02

$

534846a1d215605b1a61528eaf8e9c2210316a92

Tip: set TESTS_TO_RUN='pattern' to run a subset of benchmarks
1:1 actor calls async per second 7732.28 +- 261.41
client: 1:1 actor calls async per second 791.3 +- 38.09

Tip: set TESTS_TO_RUN='pattern' to run a subset of benchmarks
multi client tasks async per second 49348.01 +- 1032.94

It seems like the commits after 72d8709 are also around ~6500 1:1 actor calls

PR: #14211

amogkam · 2021-04-20T16:23:43Z

Here are the final results that have been signed off by core team:

single client get calls (Plasma Store)  performance change rate: 2.04%
single client put calls (Plasma Store)  performance change rate: -3.97%
single client put gigabytes  performance change rate: 35.46%
multi client put calls (Plasma Store)  performance change rate: 6.32%
multi client put gigabytes  performance change rate: 1.48%
single client tasks sync  performance change rate: -4.01%
single client tasks async  performance change rate: 1.03%
multi client tasks async  performance change rate: -5.35%
1:1 actor calls sync  performance change rate: -13.42%
1:1 actor calls async  performance change rate: -10.78%
n:n actor calls async  performance change rate: -5.67%
1:1 actor calls concurrent  performance change rate: 4.56%
1:n actor calls async  performance change rate: 3.81%
n:n actor calls with arg async  performance change rate: -0.51%
1:1 async-actor calls sync  performance change rate: -6.55%
1:1 async-actor calls async  performance change rate: 8.0%
1:1 async-actor calls with args async  performance change rate: -0.44%
1:n async-actor calls async  performance change rate: 3.23%
n:n async-actor calls async  performance change rate: -3.24%
single client get calls  performance change rate: -2.45%
single client put calls  performance change rate: -7.75%
multi client put calls  performance change rate: -8.9%
client: get calls  performance change rate: 1.3%
client: put calls  performance change rate: -5.28%
client: remote put calls  performance change rate: 6.73%
client: 1:1 actor calls sync  performance change rate: -3.94%
client: 1:1 actor calls async  performance change rate: -1.81%
client: 1:1 actor calls concurrent  performance change rate: -5.83%

krfricke · 2021-04-20T19:39:29Z

Can you quickly post the absolute (rather than relative) performance or alternatively add them to the release log branch?

amogkam · 2021-04-20T19:41:19Z

I pushed it to the branch already, but here they are

single client get calls per second 46927.37 +- 505.73
single client put calls per second 39399.54 +- 259.56
multi client put calls per second 157242.98 +- 9062.57
single client get calls (Plasma Store) per second 10887.08 +- 162.62
single client put calls (Plasma Store) per second 6359.41 +- 50.91
multi client put calls (Plasma Store) per second 10423.84 +- 160.85
single client put gigabytes per second 15.51 +- 12.57
multi client put gigabytes per second 35.58 +- 0.4
single client tasks sync per second 1823.02 +- 39.05
single client tasks async per second 18790.51 +- 165.17
multi client tasks async per second 47691.3 +- 3460.68
1:1 actor calls sync per second 2875.89 +- 99.88
1:1 actor calls async per second 8206.1 +- 397.58
1:1 actor calls concurrent per second 7430.24 +- 416.53
1:n actor calls async per second 17784.41 +- 680.76
n:n actor calls async per second 48141.31 +- 1807.31
n:n actor calls with arg async per second 13675.78 +- 71.04
1:1 async-actor calls sync per second 1965.66 +- 59.41
1:1 async-actor calls async per second 4427.98 +- 93.71
1:1 async-actor calls with args async per second 3072.24 +- 44.44
1:n async-actor calls async per second 14355.97 +- 1529.8
n:n async-actor calls async per second 38964.81 +- 1125.15
client: get calls per second 2446.19 +- 30.1
client: put calls per second 1275.03 +- 4.84
client: remote put calls per second 62818.07 +- 1390.38
client: 1:1 actor calls sync per second 701.76 +- 15.77
client: 1:1 actor calls async per second 760.75 +- 23.3
client: 1:1 actor calls concurrent per second 758.73 +- 28.31

krfricke added the release-blocker P0 Issue that blocks the release label Apr 12, 2021

krfricke assigned ericl, wuisawesome and rkooo567 Apr 12, 2021

rkooo567 added this to the Core Bugs milestone Apr 12, 2021

rkooo567 unassigned ericl and rkooo567 Apr 12, 2021

wuisawesome assigned richardliaw and rkooo567 and unassigned wuisawesome Apr 15, 2021

richardliaw mentioned this issue Apr 15, 2021

Fix release test -- client remote put #15325

Merged

6 tasks

amogkam closed this as completed Apr 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[release/microbenchmarks] Please take a look at the results (and possibly sign off) #15247

[release/microbenchmarks] Please take a look at the results (and possibly sign off) #15247

krfricke commented Apr 12, 2021

krfricke commented Apr 12, 2021

wuisawesome commented Apr 12, 2021

krfricke commented Apr 12, 2021

rkooo567 commented Apr 12, 2021

rkooo567 commented Apr 12, 2021

rkooo567 commented Apr 12, 2021

rkooo567 commented Apr 12, 2021

krfricke commented Apr 12, 2021

krfricke commented Apr 12, 2021 •

edited

Loading

ericl commented Apr 12, 2021 via email

rkooo567 commented Apr 12, 2021

wuisawesome commented Apr 12, 2021

wuisawesome commented Apr 14, 2021

krfricke commented Apr 14, 2021

AmeerHajAli commented Apr 14, 2021 •

edited

Loading

krfricke commented Apr 14, 2021

rkooo567 commented Apr 14, 2021

richardliaw commented Apr 14, 2021

wuisawesome commented Apr 15, 2021 •

edited

Loading

amogkam commented Apr 15, 2021

amogkam commented Apr 15, 2021

amogkam commented Apr 15, 2021

rkooo567 commented Apr 15, 2021

amogkam commented Apr 15, 2021

krfricke commented Apr 16, 2021

krfricke commented Apr 16, 2021 •

edited

Loading

ijrsvt commented Apr 16, 2021

rkooo567 commented Apr 16, 2021

krfricke commented Apr 19, 2021

amogkam commented Apr 20, 2021

krfricke commented Apr 20, 2021

amogkam commented Apr 20, 2021

[release/microbenchmarks] Please take a look at the results (and possibly sign off) #15247

[release/microbenchmarks] Please take a look at the results (and possibly sign off) #15247

Comments

krfricke commented Apr 12, 2021

krfricke commented Apr 12, 2021

wuisawesome commented Apr 12, 2021

krfricke commented Apr 12, 2021

rkooo567 commented Apr 12, 2021

rkooo567 commented Apr 12, 2021

rkooo567 commented Apr 12, 2021

rkooo567 commented Apr 12, 2021

krfricke commented Apr 12, 2021

krfricke commented Apr 12, 2021 • edited Loading

ericl commented Apr 12, 2021 via email

rkooo567 commented Apr 12, 2021

wuisawesome commented Apr 12, 2021

wuisawesome commented Apr 14, 2021

krfricke commented Apr 14, 2021

AmeerHajAli commented Apr 14, 2021 • edited Loading

krfricke commented Apr 14, 2021

rkooo567 commented Apr 14, 2021

richardliaw commented Apr 14, 2021

wuisawesome commented Apr 15, 2021 • edited Loading

amogkam commented Apr 15, 2021

amogkam commented Apr 15, 2021

amogkam commented Apr 15, 2021

rkooo567 commented Apr 15, 2021

amogkam commented Apr 15, 2021

krfricke commented Apr 16, 2021

krfricke commented Apr 16, 2021 • edited Loading

ijrsvt commented Apr 16, 2021

rkooo567 commented Apr 16, 2021

krfricke commented Apr 19, 2021

amogkam commented Apr 20, 2021

krfricke commented Apr 20, 2021

amogkam commented Apr 20, 2021

krfricke commented Apr 12, 2021 •

edited

Loading

AmeerHajAli commented Apr 14, 2021 •

edited

Loading

wuisawesome commented Apr 15, 2021 •

edited

Loading

krfricke commented Apr 16, 2021 •

edited

Loading