Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](MultiCast) Avoid copying while holding a lock #37462

Merged
merged 1 commit into from
Jul 15, 2024
Merged

[opt](MultiCast) Avoid copying while holding a lock #37462

merged 1 commit into from
Jul 15, 2024

Conversation

Mryange
Copy link
Contributor

@Mryange Mryange commented Jul 8, 2024

Proposed changes

Previously, copying was done while holding a lock;
Now, get block while holding the lock and then copy

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Mryange
Copy link
Contributor Author

Mryange commented Jul 8, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39491 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 02e38a24e3c25975c08c42b1b61c06f73b04affc, data reload: false

------ Round 1 ----------------------------------
q1	17621	4289	4217	4217
q2	2016	186	190	186
q3	10452	1146	1079	1079
q4	10188	786	770	770
q5	7488	2655	2659	2655
q6	220	139	136	136
q7	952	600	604	600
q8	9229	2076	2045	2045
q9	8790	6483	6566	6483
q10	8972	3688	3722	3688
q11	457	228	232	228
q12	460	222	225	222
q13	17760	2956	2974	2956
q14	260	223	230	223
q15	519	488	474	474
q16	495	371	371	371
q17	952	653	705	653
q18	7905	7399	7326	7326
q19	4784	1547	1361	1361
q20	649	323	316	316
q21	4899	3169	3233	3169
q22	389	333	334	333
Total cold run time: 115457 ms
Total hot run time: 39491 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4336	4205	4187	4187
q2	379	275	268	268
q3	3002	2760	2831	2760
q4	1922	1691	1744	1691
q5	5520	5656	5517	5517
q6	220	136	131	131
q7	2210	1849	1806	1806
q8	3280	3408	3410	3408
q9	8665	8600	8759	8600
q10	4080	3878	3775	3775
q11	602	513	507	507
q12	774	620	619	619
q13	16025	3152	3166	3152
q14	298	289	282	282
q15	542	488	508	488
q16	493	432	422	422
q17	1804	1479	1508	1479
q18	8048	7967	7680	7680
q19	1753	1750	1701	1701
q20	2071	1910	1912	1910
q21	10937	4894	4897	4894
q22	632	560	577	560
Total cold run time: 77593 ms
Total hot run time: 55837 ms

@Mryange
Copy link
Contributor Author

Mryange commented Jul 8, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40198 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dc24cdfb2a6794c4cd94e703392a9e252a99d8cb, data reload: false

------ Round 1 ----------------------------------
q1	17612	5280	4325	4325
q2	2030	191	188	188
q3	10473	1164	1096	1096
q4	10182	799	783	783
q5	7498	2705	2648	2648
q6	223	139	140	139
q7	963	609	617	609
q8	9236	2095	2100	2095
q9	8821	6498	6452	6452
q10	8979	3730	3741	3730
q11	460	242	244	242
q12	459	241	229	229
q13	18726	2996	2996	2996
q14	271	227	238	227
q15	527	492	496	492
q16	530	396	385	385
q17	977	715	676	676
q18	8230	7590	7478	7478
q19	6033	1537	1554	1537
q20	697	325	335	325
q21	4952	3205	3999	3205
q22	404	341	341	341
Total cold run time: 118283 ms
Total hot run time: 40198 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4419	4313	4266	4266
q2	378	280	278	278
q3	3155	2846	2928	2846
q4	1976	1775	1730	1730
q5	5584	5538	5458	5458
q6	229	134	138	134
q7	2233	1911	1889	1889
q8	3314	3481	3427	3427
q9	8716	8924	8838	8838
q10	4138	3657	3892	3657
q11	610	510	494	494
q12	835	648	648	648
q13	15877	3162	3250	3162
q14	311	282	289	282
q15	533	492	504	492
q16	493	456	424	424
q17	1822	1558	1550	1550
q18	8360	8086	7798	7798
q19	1812	1729	1604	1604
q20	2238	1869	1854	1854
q21	9812	4737	4870	4737
q22	633	571	570	570
Total cold run time: 77478 ms
Total hot run time: 56138 ms

@Mryange
Copy link
Contributor Author

Mryange commented Jul 8, 2024

run performance

@doris-robot
Copy link

TPC-H: Total hot run time: 39591 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dc24cdfb2a6794c4cd94e703392a9e252a99d8cb, data reload: false

------ Round 1 ----------------------------------
q1	17619	4296	4253	4253
q2	2014	197	192	192
q3	10449	1174	1153	1153
q4	10192	722	833	722
q5	7478	2622	2612	2612
q6	222	135	141	135
q7	965	605	604	604
q8	9245	2059	2083	2059
q9	8925	6475	6488	6475
q10	8965	3647	3679	3647
q11	468	235	235	235
q12	475	226	238	226
q13	17902	2966	2991	2966
q14	260	228	242	228
q15	526	492	486	486
q16	500	375	384	375
q17	959	656	767	656
q18	8031	7459	7371	7371
q19	4619	1521	1388	1388
q20	652	321	330	321
q21	4901	3149	3915	3149
q22	383	340	338	338
Total cold run time: 115750 ms
Total hot run time: 39591 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4414	4202	4252	4202
q2	381	271	278	271
q3	3001	2771	2895	2771
q4	1984	1700	1716	1700
q5	5596	5581	5528	5528
q6	229	145	131	131
q7	2180	1868	1874	1868
q8	3309	3437	3383	3383
q9	8709	8731	8839	8731
q10	4131	3870	3804	3804
q11	584	510	490	490
q12	805	630	653	630
q13	17008	3193	3166	3166
q14	319	288	283	283
q15	543	496	484	484
q16	491	441	433	433
q17	1808	1529	1525	1525
q18	8006	7895	7798	7798
q19	1805	1624	1574	1574
q20	2201	1873	1914	1873
q21	5248	4949	4993	4949
q22	616	540	550	540
Total cold run time: 73368 ms
Total hot run time: 56134 ms

@Mryange
Copy link
Contributor Author

Mryange commented Jul 8, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40561 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cd4eef91a46d07b16c03041c2cc763640956a748, data reload: false

------ Round 1 ----------------------------------
q1	17973	4446	4444	4444
q2	2635	199	202	199
q3	10958	1176	1151	1151
q4	10247	811	751	751
q5	7579	2684	2691	2684
q6	228	142	151	142
q7	972	620	626	620
q8	9295	2102	2103	2102
q9	9068	6498	6489	6489
q10	9016	3722	3682	3682
q11	441	240	238	238
q12	418	227	233	227
q13	17757	2985	2988	2985
q14	271	223	222	222
q15	527	472	484	472
q16	489	370	378	370
q17	975	613	697	613
q18	8025	7388	7501	7388
q19	1950	1499	1305	1305
q20	677	323	323	323
q21	5046	3813	4001	3813
q22	399	348	341	341
Total cold run time: 114946 ms
Total hot run time: 40561 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4328	4221	4234	4221
q2	378	263	266	263
q3	2984	2790	2774	2774
q4	1865	1606	1622	1606
q5	5254	5347	5294	5294
q6	221	129	132	129
q7	2107	1758	1683	1683
q8	3198	3347	3294	3294
q9	8397	8361	8395	8361
q10	3859	3715	3724	3715
q11	593	486	515	486
q12	775	627	583	583
q13	16392	2970	3007	2970
q14	293	264	261	261
q15	531	468	486	468
q16	475	418	418	418
q17	1788	1466	1466	1466
q18	7570	7537	7419	7419
q19	4329	1619	1527	1527
q20	1998	1778	1808	1778
q21	4851	4753	4910	4753
q22	602	534	554	534
Total cold run time: 72788 ms
Total hot run time: 54003 ms

@Mryange
Copy link
Contributor Author

Mryange commented Jul 8, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39372 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1637feea169bff9cf5889e561213cede0f558771, data reload: false

------ Round 1 ----------------------------------
q1	17635	4340	4284	4284
q2	2020	192	192	192
q3	10440	1125	1156	1125
q4	10205	731	756	731
q5	7494	2635	2584	2584
q6	218	136	137	136
q7	954	607	600	600
q8	9230	2049	2075	2049
q9	8837	6508	6489	6489
q10	9025	3717	3718	3717
q11	485	244	236	236
q12	491	237	230	230
q13	18742	2987	2954	2954
q14	280	214	228	214
q15	521	485	494	485
q16	519	375	372	372
q17	961	581	592	581
q18	8014	7429	7321	7321
q19	6570	1356	1484	1356
q20	696	324	325	324
q21	4930	3064	3276	3064
q22	403	333	328	328
Total cold run time: 118670 ms
Total hot run time: 39372 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4406	4266	4323	4266
q2	385	280	265	265
q3	2930	2954	2904	2904
q4	1980	1665	1738	1665
q5	5613	5510	5459	5459
q6	219	129	127	127
q7	2176	1895	1862	1862
q8	3246	3416	3423	3416
q9	8751	8826	8734	8734
q10	4097	3780	3705	3705
q11	616	508	491	491
q12	803	660	632	632
q13	16346	3139	3147	3139
q14	320	285	293	285
q15	515	494	486	486
q16	485	436	443	436
q17	1830	1542	1516	1516
q18	8174	7975	7812	7812
q19	1846	1704	1566	1566
q20	2127	1871	1849	1849
q21	5091	4894	4607	4607
q22	615	567	540	540
Total cold run time: 72571 ms
Total hot run time: 55762 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173064 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1637feea169bff9cf5889e561213cede0f558771, data reload: false

query1	924	364	375	364
query2	6918	2406	2413	2406
query3	6636	207	216	207
query4	25429	17484	17450	17450
query5	3714	465	470	465
query6	265	174	173	173
query7	4582	292	282	282
query8	305	305	295	295
query9	8476	2404	2391	2391
query10	574	296	283	283
query11	10705	9981	10066	9981
query12	119	93	86	86
query13	1648	376	380	376
query14	10412	7692	6898	6898
query15	250	190	179	179
query16	7637	320	310	310
query17	1772	553	536	536
query18	1635	278	303	278
query19	202	159	154	154
query20	91	83	82	82
query21	220	132	132	132
query22	4304	4101	4003	4003
query23	34313	34167	33678	33678
query24	10626	2861	2851	2851
query25	620	423	386	386
query26	704	159	153	153
query27	2297	274	274	274
query28	6309	2130	2112	2112
query29	913	663	713	663
query30	249	153	158	153
query31	956	785	751	751
query32	95	52	56	52
query33	685	281	299	281
query34	894	484	503	484
query35	680	557	610	557
query36	1154	977	960	960
query37	151	77	84	77
query38	2984	2854	2773	2773
query39	868	803	798	798
query40	203	122	114	114
query41	55	50	52	50
query42	122	98	97	97
query43	571	531	531	531
query44	1049	731	713	713
query45	192	158	157	157
query46	1067	702	710	702
query47	1827	1807	1764	1764
query48	367	285	284	284
query49	841	402	413	402
query50	792	383	388	383
query51	6839	6801	6761	6761
query52	116	88	94	88
query53	358	286	279	279
query54	864	436	433	433
query55	77	73	72	72
query56	287	257	306	257
query57	1105	1045	1060	1045
query58	255	237	263	237
query59	3204	2952	3215	2952
query60	292	267	283	267
query61	94	94	96	94
query62	785	630	655	630
query63	322	281	284	281
query64	9078	2179	1610	1610
query65	3221	3149	3111	3111
query66	743	327	324	324
query67	15770	15022	14858	14858
query68	5784	526	530	526
query69	707	450	340	340
query70	1140	1160	1151	1151
query71	479	281	269	269
query72	8588	5242	5328	5242
query73	770	317	321	317
query74	5962	5496	5499	5496
query75	4118	2629	2670	2629
query76	3571	963	1013	963
query77	672	305	290	290
query78	9645	9109	8994	8994
query79	3089	513	504	504
query80	2817	468	465	465
query81	592	217	221	217
query82	1407	135	128	128
query83	303	166	172	166
query84	273	90	88	88
query85	1436	315	356	315
query86	473	314	279	279
query87	3363	3083	3059	3059
query88	4108	2337	2329	2329
query89	470	375	382	375
query90	1877	182	187	182
query91	134	101	101	101
query92	60	46	48	46
query93	2704	505	495	495
query94	1341	208	204	204
query95	395	310	311	310
query96	605	261	269	261
query97	3173	3034	3032	3032
query98	217	202	198	198
query99	1798	1263	1306	1263
Total cold run time: 283883 ms
Total hot run time: 173064 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.37 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1637feea169bff9cf5889e561213cede0f558771, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.68	0.07	0.08
query5	0.50	0.46	0.48
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.55	0.50	0.51
query10	0.54	0.54	0.55
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.58
query14	0.78	0.78	0.79
query15	0.86	0.82	0.83
query16	0.37	0.37	0.37
query17	0.96	1.03	1.03
query18	0.23	0.22	0.23
query19	1.91	1.69	1.72
query20	0.01	0.01	0.01
query21	15.40	0.74	0.64
query22	4.30	7.36	1.64
query23	18.26	1.45	1.28
query24	2.11	0.24	0.21
query25	0.16	0.08	0.08
query26	0.30	0.21	0.21
query27	0.46	0.23	0.23
query28	13.28	1.01	0.99
query29	12.60	3.27	3.30
query30	0.25	0.06	0.05
query31	2.89	0.39	0.41
query32	3.25	0.48	0.47
query33	2.87	2.90	2.95
query34	16.98	4.33	4.46
query35	4.38	4.41	4.42
query36	0.65	0.49	0.49
query37	0.18	0.15	0.16
query38	0.16	0.14	0.15
query39	0.04	0.04	0.03
query40	0.16	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.73 s
Total hot run time: 30.37 s

@Mryange
Copy link
Contributor Author

Mryange commented Jul 9, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40251 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 607342e02c019e0d66872c2b21756ecfe842cebf, data reload: false

------ Round 1 ----------------------------------
q1	17601	4351	4300	4300
q2	2015	192	197	192
q3	10973	1233	1117	1117
q4	10782	840	881	840
q5	7566	2716	2655	2655
q6	225	146	148	146
q7	987	625	634	625
q8	9260	2114	2150	2114
q9	8990	6623	6643	6623
q10	9071	3712	3672	3672
q11	482	244	249	244
q12	421	235	240	235
q13	17821	3019	2975	2975
q14	276	215	227	215
q15	524	493	504	493
q16	526	373	374	373
q17	970	675	716	675
q18	8191	7460	7475	7460
q19	3473	1374	1436	1374
q20	657	333	353	333
q21	4944	3955	3248	3248
q22	390	342	343	342
Total cold run time: 116145 ms
Total hot run time: 40251 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4374	4252	4252	4252
q2	369	273	266	266
q3	3014	2886	2859	2859
q4	1999	1779	1769	1769
q5	5561	5485	5481	5481
q6	227	134	141	134
q7	2226	1940	1869	1869
q8	3319	3427	3439	3427
q9	8853	8768	8652	8652
q10	4026	3886	3879	3879
q11	612	516	505	505
q12	813	628	639	628
q13	16056	3221	3142	3142
q14	316	291	279	279
q15	563	482	499	482
q16	478	431	426	426
q17	1851	1553	1538	1538
q18	8121	7911	7864	7864
q19	1849	1669	1703	1669
q20	2083	1862	1896	1862
q21	5137	4855	4950	4855
q22	658	604	542	542
Total cold run time: 72505 ms
Total hot run time: 56380 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172148 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 607342e02c019e0d66872c2b21756ecfe842cebf, data reload: false

query1	914	376	370	370
query2	6277	2442	2304	2304
query3	6644	213	220	213
query4	28570	17607	17504	17504
query5	3822	479	486	479
query6	257	173	165	165
query7	4590	287	289	287
query8	314	298	301	298
query9	8561	2444	2413	2413
query10	585	307	293	293
query11	11371	10144	10220	10144
query12	122	85	87	85
query13	1669	395	388	388
query14	10174	7839	7731	7731
query15	254	187	188	187
query16	7795	317	337	317
query17	1780	586	547	547
query18	2001	287	296	287
query19	207	158	158	158
query20	94	82	88	82
query21	214	137	141	137
query22	4243	3969	4051	3969
query23	34043	33644	33748	33644
query24	12018	2876	2855	2855
query25	655	434	441	434
query26	1733	161	155	155
query27	2915	277	289	277
query28	7901	2144	2113	2113
query29	1068	667	675	667
query30	262	156	154	154
query31	990	750	794	750
query32	99	57	57	57
query33	772	323	333	323
query34	939	512	503	503
query35	757	620	631	620
query36	1157	981	994	981
query37	286	95	87	87
query38	2937	2808	2746	2746
query39	843	833	803	803
query40	252	124	124	124
query41	55	54	54	54
query42	117	102	104	102
query43	582	547	553	547
query44	1221	735	832	735
query45	187	161	159	159
query46	1080	716	711	711
query47	1859	1781	1784	1781
query48	368	294	304	294
query49	957	417	426	417
query50	791	399	406	399
query51	6884	6784	6741	6741
query52	111	93	99	93
query53	364	288	298	288
query54	996	445	445	445
query55	72	73	73	73
query56	278	264	265	264
query57	1216	1081	1027	1027
query58	253	243	255	243
query59	3519	3255	3159	3159
query60	307	303	283	283
query61	100	95	91	91
query62	802	662	627	627
query63	323	291	289	289
query64	10238	2180	1658	1658
query65	3162	3115	3087	3087
query66	1257	326	334	326
query67	15705	14969	14878	14878
query68	8831	549	543	543
query69	713	506	399	399
query70	1261	1174	1175	1174
query71	568	297	290	290
query72	8523	5574	2775	2775
query73	1780	319	317	317
query74	5921	5489	5473	5473
query75	5676	2673	2712	2673
query76	5292	856	906	856
query77	843	306	325	306
query78	11162	9125	8786	8786
query79	12270	527	516	516
query80	828	490	488	488
query81	571	222	214	214
query82	262	130	128	128
query83	349	212	165	165
query84	280	84	85	84
query85	866	328	304	304
query86	371	319	320	319
query87	3251	3071	3039	3039
query88	5158	2362	2355	2355
query89	515	391	425	391
query90	2146	193	195	193
query91	135	104	101	101
query92	61	49	50	49
query93	5904	511	500	500
query94	1379	213	217	213
query95	402	315	316	315
query96	614	269	260	260
query97	3221	3038	3069	3038
query98	211	199	199	199
query99	1487	1263	1327	1263
Total cold run time: 313740 ms
Total hot run time: 172148 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.55 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 607342e02c019e0d66872c2b21756ecfe842cebf, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.22	0.04	0.04
query4	1.68	0.07	0.08
query5	0.50	0.48	0.50
query6	1.14	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.56	0.49	0.50
query10	0.55	0.52	0.52
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.60	0.58
query14	0.80	0.79	0.77
query15	0.86	0.82	0.81
query16	0.37	0.38	0.37
query17	1.05	1.02	0.95
query18	0.22	0.21	0.21
query19	1.86	1.76	1.84
query20	0.02	0.01	0.01
query21	15.39	0.76	0.65
query22	4.27	7.16	2.00
query23	18.26	1.32	1.21
query24	2.17	0.22	0.21
query25	0.16	0.09	0.08
query26	0.30	0.21	0.21
query27	0.45	0.24	0.23
query28	13.26	1.01	0.99
query29	12.65	3.29	3.29
query30	0.24	0.06	0.05
query31	2.85	0.39	0.39
query32	3.28	0.47	0.47
query33	2.91	2.88	2.95
query34	16.89	4.33	4.42
query35	4.42	4.42	4.42
query36	0.67	0.48	0.47
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.03	0.03
query40	0.14	0.12	0.11
query41	0.09	0.05	0.04
query42	0.06	0.04	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.78 s
Total hot run time: 30.55 s

@Mryange
Copy link
Contributor Author

Mryange commented Jul 9, 2024

run p0

@Mryange
Copy link
Contributor Author

Mryange commented Jul 9, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@Mryange
Copy link
Contributor Author

Mryange commented Jul 10, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40394 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cec03f335dac54affad7780054489dafe6d17783, data reload: false

------ Round 1 ----------------------------------
q1	18052	4523	4392	4392
q2	2331	188	188	188
q3	10454	1192	1190	1190
q4	10196	855	771	771
q5	7496	2664	2648	2648
q6	220	142	142	142
q7	951	621	606	606
q8	9238	2052	2076	2052
q9	9060	6517	6509	6509
q10	8783	3783	3753	3753
q11	465	247	242	242
q12	401	237	240	237
q13	17816	3020	2968	2968
q14	275	233	220	220
q15	522	491	494	491
q16	507	378	386	378
q17	971	769	737	737
q18	8130	7493	7379	7379
q19	2530	1603	1508	1508
q20	671	316	321	316
q21	5030	3327	3919	3327
q22	413	347	340	340
Total cold run time: 114512 ms
Total hot run time: 40394 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4341	4274	4255	4255
q2	376	266	266	266
q3	3003	2727	2759	2727
q4	1894	1609	1638	1609
q5	5269	5268	5267	5267
q6	225	133	135	133
q7	2136	1727	1723	1723
q8	3223	3335	3301	3301
q9	8396	8346	8397	8346
q10	3864	3681	3703	3681
q11	588	492	503	492
q12	791	609	583	583
q13	16369	2996	3022	2996
q14	320	258	253	253
q15	517	484	484	484
q16	493	434	430	430
q17	1767	1501	1497	1497
q18	7805	7640	7396	7396
q19	1674	1522	1605	1522
q20	2036	1814	1787	1787
q21	4885	4766	4784	4766
q22	636	582	554	554
Total cold run time: 70608 ms
Total hot run time: 54068 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173064 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cec03f335dac54affad7780054489dafe6d17783, data reload: false

query1	924	371	361	361
query2	6477	2450	2331	2331
query3	6650	217	217	217
query4	28424	17659	17389	17389
query5	4191	498	496	496
query6	281	189	168	168
query7	4594	292	284	284
query8	317	283	287	283
query9	8452	2415	2373	2373
query10	430	285	262	262
query11	12080	9913	10030	9913
query12	132	90	82	82
query13	1643	377	375	375
query14	10367	6900	7035	6900
query15	250	189	189	189
query16	7231	327	305	305
query17	1364	537	547	537
query18	1899	272	277	272
query19	187	150	148	148
query20	88	84	79	79
query21	207	127	128	127
query22	4358	4103	4120	4103
query23	33796	33010	33131	33010
query24	12234	2849	2827	2827
query25	652	369	416	369
query26	1825	151	150	150
query27	2970	270	274	270
query28	7568	2070	2047	2047
query29	1175	639	602	602
query30	283	148	146	146
query31	991	737	729	729
query32	101	53	53	53
query33	776	319	300	300
query34	935	498	495	495
query35	669	558	569	558
query36	1129	939	924	924
query37	203	77	78	77
query38	2878	2744	2721	2721
query39	860	823	795	795
query40	281	120	122	120
query41	53	50	52	50
query42	118	99	99	99
query43	576	568	546	546
query44	1246	746	776	746
query45	189	160	162	160
query46	1068	713	728	713
query47	1882	1738	1782	1738
query48	373	295	309	295
query49	1204	425	414	414
query50	773	395	396	395
query51	6918	6850	6733	6733
query52	107	90	97	90
query53	355	291	288	288
query54	984	460	442	442
query55	76	78	72	72
query56	289	256	270	256
query57	1138	1057	1052	1052
query58	270	237	253	237
query59	3465	3200	3187	3187
query60	301	275	313	275
query61	98	100	97	97
query62	831	649	648	648
query63	327	298	286	286
query64	10584	2194	1632	1632
query65	3217	3108	3093	3093
query66	1424	374	346	346
query67	15442	14942	15046	14942
query68	4610	544	548	544
query69	468	327	319	319
query70	1156	1134	1183	1134
query71	409	276	276	276
query72	7117	5941	5597	5597
query73	750	332	329	329
query74	5897	5467	5530	5467
query75	3401	2727	2685	2685
query76	2297	947	965	947
query77	449	317	293	293
query78	9653	8894	8896	8894
query79	3008	530	525	525
query80	2074	491	469	469
query81	576	223	226	223
query82	727	133	143	133
query83	278	168	169	168
query84	276	88	90	88
query85	2125	323	297	297
query86	490	321	322	321
query87	3300	3121	3122	3121
query88	4543	2466	2470	2466
query89	491	383	397	383
query90	1844	188	188	188
query91	128	104	104	104
query92	59	49	49	49
query93	4888	519	515	515
query94	1264	208	211	208
query95	408	323	315	315
query96	615	273	269	269
query97	3167	3041	3056	3041
query98	221	199	197	197
query99	1573	1287	1280	1280
Total cold run time: 291442 ms
Total hot run time: 173064 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cec03f335dac54affad7780054489dafe6d17783, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.04
query4	1.68	0.07	0.07
query5	0.50	0.47	0.48
query6	1.15	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.56	0.55	0.55
query11	0.15	0.11	0.12
query12	0.15	0.13	0.12
query13	0.60	0.59	0.59
query14	0.76	0.78	0.79
query15	0.86	0.82	0.81
query16	0.35	0.38	0.37
query17	1.01	0.99	0.97
query18	0.23	0.21	0.22
query19	1.75	1.70	1.82
query20	0.01	0.00	0.01
query21	15.43	0.80	0.67
query22	4.06	6.89	2.48
query23	18.32	1.35	1.26
query24	2.10	0.23	0.22
query25	0.17	0.09	0.09
query26	0.28	0.21	0.20
query27	0.44	0.23	0.22
query28	13.31	1.03	1.00
query29	12.89	3.28	3.27
query30	0.26	0.06	0.05
query31	2.88	0.39	0.39
query32	3.27	0.48	0.47
query33	2.92	2.91	2.86
query34	17.04	4.39	4.47
query35	4.47	4.43	4.48
query36	0.65	0.46	0.50
query37	0.18	0.16	0.16
query38	0.16	0.15	0.14
query39	0.04	0.03	0.03
query40	0.14	0.12	0.11
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.05	0.05	0.04
Total cold run time: 109.92 s
Total hot run time: 31.14 s

HappenLee
HappenLee reviewed Jul 12, 2024
View reviewed changes
be/src/pipeline/exec/multi_cast_data_streamer.cpp

void MultiCastDataStreamer::_wait_copy_block(vectorized::Block* block, int& un_finish_copy) {
std::unique_lock l(_mutex);
_cv.wait(l, [&]() { return un_finish_copy == 0; });
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

danger will notify different queue last

HappenLee
HappenLee approved these changes Jul 12, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 12, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

HappenLee
HappenLee approved these changes Jul 15, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HappenLee HappenLee merged commit ad3e4d5 into apache:master Jul 15, 2024
27 of 33 checks passed
wsjz pushed a commit to wsjz/incubator-doris that referenced this pull request Jul 15, 2024
@Mryange @wsjz
[opt](MultiCast) Avoid copying while holding a lock (apache#37462)
9729feb 
Previously, copying was done while holding a lock;
Now, get block while holding the lock and then copy
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@Mryange @seawinde
[opt](MultiCast) Avoid copying while holding a lock (apache#37462)
9c315a2 
Previously, copying was done while holding a lock;
Now, get block while holding the lock and then copy
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@Mryange @dataroaring
[opt](MultiCast) Avoid copying while holding a lock (#37462)
010c5ed 
Previously, copying was done while holding a lock;
Now, get block while holding the lock and then copy
@Mryange Mryange mentioned this pull request Jul 25, 2024
Merged
HappenLee pushed a commit that referenced this pull request Jul 29, 2024
@Mryange
[opt](MultiCast) Avoid copying while holding a lock (#38348)
535ae2c 
pick #37462
The difference is quite large, so it can't be directly picked.
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@Mryange @Baymine
[opt](MultiCast) Avoid copying while holding a lock (apache#38348)
dfe3093 
pick apache#37462
The difference is quite large, so it can't be directly picked.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
@shuke987 shuke987 added dev/2.1.x usercase Important user case type label labels Aug 22, 2024
Mryange added a commit to Mryange/doris that referenced this pull request Aug 23, 2024
@Mryange
[opt](MultiCast) Avoid copying while holding a lock (apache#37462)
5457e9b 
Previously, copying was done while holding a lock;
Now, get block while holding the lock and then copy
@Mryange Mryange mentioned this pull request Aug 23, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 23, 2024
@Mryange
[opt](MultiCast) Avoid copying while holding a lock (#37462) (#39816)
e03b887 
Previously, copying was done while holding a lock; Now, get block while
holding the lock and then copy
#37462
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HappenLee HappenLee HappenLee approved these changes

@Gabriel39 Gabriel39 Gabriel39 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.14-merged dev/2.1.6-merged dev/3.0.1-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@Mryange @doris-robot @HappenLee @Gabriel39 @yiguolei @shuke987 @wm1581066 @dataroaring