Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhance](mtmv)Improve the performance of obtaining partition/table versions #39301

Merged
merged 3 commits into from
Aug 14, 2024
Merged

[enhance](mtmv)Improve the performance of obtaining partition/table versions #39301

merged 3 commits into from
Aug 14, 2024

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Aug 13, 2024

Batch retrieve version information of all tables and partitions used by MTMV and store it in MTMVRefreshContext

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added the doing label Aug 13, 2024
@zddr
Copy link
Contributor Author

zddr commented Aug 13, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39664 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e0315646217db47de61c3c9b2351dcb697b575b4, data reload: false

------ Round 1 ----------------------------------
q1	17644	4361	4260	4260
q2	2017	180	170	170
q3	11786	1029	1068	1029
q4	10527	717	677	677
q5	7780	2783	2777	2777
q6	225	138	142	138
q7	964	590	599	590
q8	9535	2021	2088	2021
q9	9598	6523	6525	6523
q10	6977	2167	2187	2167
q11	467	247	252	247
q12	397	226	225	225
q13	18910	2998	2962	2962
q14	276	242	234	234
q15	532	478	498	478
q16	497	407	381	381
q17	964	749	691	691
q18	8078	7448	7345	7345
q19	3223	1010	1004	1004
q20	693	344	328	328
q21	5760	4413	4527	4413
q22	1103	1004	1009	1004
Total cold run time: 117953 ms
Total hot run time: 39664 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4419	4282	4257	4257
q2	370	284	260	260
q3	2870	2730	2762	2730
q4	1959	1725	1697	1697
q5	5693	5593	5498	5498
q6	222	142	133	133
q7	2107	1775	1801	1775
q8	3264	3436	3411	3411
q9	8731	8782	8763	8763
q10	3515	3283	3200	3200
q11	607	507	517	507
q12	812	656	618	618
q13	16070	3140	3143	3140
q14	313	272	296	272
q15	535	490	498	490
q16	522	447	425	425
q17	1806	1532	1512	1512
q18	8072	7838	7878	7838
q19	1820	1590	1721	1590
q20	2159	1924	1915	1915
q21	5613	5258	5149	5149
q22	1139	1045	1022	1022
Total cold run time: 72618 ms
Total hot run time: 56202 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189092 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e0315646217db47de61c3c9b2351dcb697b575b4, data reload: false

query1	1260	901	860	860
query2	6441	1951	1846	1846
query3	10595	3938	4042	3938
query4	59603	25081	23180	23180
query5	5426	500	491	491
query6	469	155	175	155
query7	6299	289	286	286
query8	301	208	197	197
query9	8870	2435	2409	2409
query10	512	258	253	253
query11	17971	14958	15241	14958
query12	159	101	104	101
query13	1528	402	381	381
query14	12146	6864	7037	6864
query15	221	172	177	172
query16	7613	501	463	463
query17	1144	579	572	572
query18	2134	304	304	304
query19	298	155	150	150
query20	124	120	109	109
query21	210	104	102	102
query22	4357	4417	4333	4333
query23	34008	33324	32975	32975
query24	5731	2888	2867	2867
query25	541	400	417	400
query26	681	157	158	157
query27	1782	268	275	268
query28	3703	2047	2027	2027
query29	703	425	418	418
query30	228	157	149	149
query31	914	786	751	751
query32	76	54	53	53
query33	478	294	279	279
query34	845	456	467	456
query35	815	706	744	706
query36	1037	917	917	917
query37	131	81	82	81
query38	3905	3804	3894	3804
query39	1436	1399	1373	1373
query40	200	118	117	117
query41	47	47	46	46
query42	120	104	101	101
query43	503	468	464	464
query44	1073	720	727	720
query45	193	162	165	162
query46	1089	729	710	710
query47	1898	1790	1804	1790
query48	364	299	295	295
query49	766	430	441	430
query50	791	401	401	401
query51	6879	6671	6699	6671
query52	103	87	90	87
query53	250	180	185	180
query54	558	447	543	447
query55	75	71	73	71
query56	270	242	250	242
query57	1143	1049	1042	1042
query58	203	216	232	216
query59	2842	2821	2701	2701
query60	278	250	263	250
query61	93	97	96	96
query62	740	623	643	623
query63	209	176	173	173
query64	3175	1702	2760	1702
query65	3181	3123	3169	3123
query66	651	315	323	315
query67	15567	14922	14915	14915
query68	8275	580	546	546
query69	702	366	292	292
query70	1178	1107	1031	1031
query71	524	275	267	267
query72	7128	2216	2054	2054
query73	796	321	322	321
query74	9235	8785	8745	8745
query75	4188	2694	2680	2680
query76	4379	1003	942	942
query77	748	303	295	295
query78	9813	9100	8972	8972
query79	7321	527	523	523
query80	2256	483	485	483
query81	593	227	227	227
query82	612	136	135	135
query83	274	142	143	142
query84	272	76	73	73
query85	1136	271	309	271
query86	356	293	270	270
query87	4399	4164	4196	4164
query88	4205	2420	2382	2382
query89	534	288	288	288
query90	2341	190	190	190
query91	121	94	97	94
query92	59	49	48	48
query93	6128	528	526	526
query94	1085	280	293	280
query95	357	254	261	254
query96	620	273	270	270
query97	3205	2996	3032	2996
query98	214	202	196	196
query99	1566	1251	1240	1240
Total cold run time: 332576 ms
Total hot run time: 189092 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e0315646217db47de61c3c9b2351dcb697b575b4, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.03	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.51	0.50	0.50
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.56	0.48	0.48
query10	0.54	0.55	0.53
query11	0.15	0.11	0.12
query12	0.14	0.12	0.12
query13	0.59	0.60	0.57
query14	0.75	0.78	0.76
query15	0.85	0.82	0.82
query16	0.35	0.35	0.37
query17	1.02	1.00	1.03
query18	0.23	0.21	0.22
query19	1.91	1.80	1.75
query20	0.01	0.01	0.01
query21	15.39	0.77	0.66
query22	4.20	7.17	1.87
query23	18.31	1.37	1.30
query24	2.09	0.21	0.21
query25	0.14	0.09	0.08
query26	0.30	0.21	0.21
query27	0.46	0.22	0.21
query28	13.40	1.02	1.00
query29	12.60	3.31	3.28
query30	0.23	0.05	0.05
query31	2.89	0.39	0.38
query32	3.29	0.48	0.47
query33	2.94	2.98	2.99
query34	17.10	4.36	4.39
query35	4.43	4.37	4.38
query36	0.66	0.46	0.51
query37	0.18	0.15	0.16
query38	0.15	0.15	0.15
query39	0.05	0.03	0.04
query40	0.16	0.12	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.02 s
Total hot run time: 30.65 s

@zddr
Copy link
Contributor Author

zddr commented Aug 14, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39919 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c06b40fb001c9db8b701da13d8618f5b6175520d, data reload: false

------ Round 1 ----------------------------------
q1	17628	4393	4320	4320
q2	2012	177	181	177
q3	11677	961	1059	961
q4	10511	717	732	717
q5	7773	2793	2831	2793
q6	226	143	140	140
q7	960	601	602	601
q8	9545	2080	2065	2065
q9	9749	6597	6583	6583
q10	7031	2181	2229	2181
q11	476	240	242	240
q12	394	225	231	225
q13	17766	2964	3003	2964
q14	287	243	233	233
q15	533	499	481	481
q16	493	397	385	385
q17	973	677	617	617
q18	7908	7422	7462	7422
q19	6777	976	923	923
q20	663	345	341	341
q21	5507	4555	4707	4555
q22	1168	1033	995	995
Total cold run time: 120057 ms
Total hot run time: 39919 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4548	4288	4297	4288
q2	367	279	272	272
q3	2996	2835	2854	2835
q4	2003	1807	1715	1715
q5	5588	5557	5648	5557
q6	239	145	142	142
q7	2189	1725	1774	1725
q8	3300	3439	3433	3433
q9	8710	8821	8772	8772
q10	3405	3253	3310	3253
q11	592	502	507	502
q12	852	642	620	620
q13	15938	3210	3141	3141
q14	312	297	289	289
q15	562	487	506	487
q16	488	448	454	448
q17	1859	1539	1484	1484
q18	8162	7821	7842	7821
q19	1755	1578	1729	1578
q20	2130	1869	1863	1863
q21	6780	5253	5396	5253
q22	1157	1080	1073	1073
Total cold run time: 73932 ms
Total hot run time: 56551 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191135 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c06b40fb001c9db8b701da13d8618f5b6175520d, data reload: false

query1	1261	900	871	871
query2	6348	1977	1883	1883
query3	10614	3967	3640	3640
query4	58220	24359	23033	23033
query5	5793	514	494	494
query6	510	153	161	153
query7	6482	294	289	289
query8	302	217	200	200
query9	9137	2437	2411	2411
query10	509	259	275	259
query11	17961	15053	15066	15053
query12	160	101	105	101
query13	1606	392	383	383
query14	11802	7922	7863	7863
query15	227	193	181	181
query16	7552	551	520	520
query17	1189	617	587	587
query18	1753	321	301	301
query19	288	158	165	158
query20	120	114	118	114
query21	220	103	107	103
query22	4409	4384	4450	4384
query23	34214	33899	33587	33587
query24	5577	2928	2870	2870
query25	532	404	401	401
query26	688	164	161	161
query27	1787	272	270	270
query28	3749	2016	2007	2007
query29	679	419	424	419
query30	237	156	150	150
query31	935	759	744	744
query32	80	53	55	53
query33	466	289	296	289
query34	851	467	483	467
query35	833	703	730	703
query36	1063	940	927	927
query37	142	85	84	84
query38	3935	3886	3892	3886
query39	1431	1385	1374	1374
query40	201	115	122	115
query41	47	46	47	46
query42	115	96	96	96
query43	505	459	488	459
query44	1064	736	740	736
query45	207	166	168	166
query46	1096	777	742	742
query47	1867	1785	1773	1773
query48	360	295	290	290
query49	766	435	438	435
query50	813	409	403	403
query51	6850	6833	6634	6634
query52	103	87	89	87
query53	260	189	188	188
query54	573	449	550	449
query55	79	74	73	73
query56	268	244	252	244
query57	1117	1051	1058	1051
query58	216	231	268	231
query59	2941	2768	2685	2685
query60	287	270	262	262
query61	97	105	94	94
query62	749	656	652	652
query63	209	185	178	178
query64	3174	1779	1704	1704
query65	3219	3138	3179	3138
query66	694	333	326	326
query67	15346	15019	15007	15007
query68	5850	543	560	543
query69	498	281	279	279
query70	1206	1125	1114	1114
query71	477	278	267	267
query72	7215	2246	2008	2008
query73	790	336	328	328
query74	9341	8759	8791	8759
query75	3595	2695	2699	2695
query76	3283	1016	956	956
query77	687	333	306	306
query78	9789	9026	8933	8933
query79	7843	532	536	532
query80	2291	518	501	501
query81	557	225	232	225
query82	992	139	137	137
query83	294	149	152	149
query84	265	74	80	74
query85	1490	303	265	265
query86	407	306	301	301
query87	4410	4201	4253	4201
query88	5522	2422	2413	2413
query89	525	289	286	286
query90	2026	198	198	198
query91	120	97	95	95
query92	65	49	49	49
query93	6328	537	541	537
query94	808	285	272	272
query95	346	261	263	261
query96	615	275	280	275
query97	3205	3084	3085	3084
query98	223	204	193	193
query99	1606	1262	1283	1262
Total cold run time: 329554 ms
Total hot run time: 191135 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c06b40fb001c9db8b701da13d8618f5b6175520d, data reload: false

query1	0.05	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.68	0.07	0.07
query5	0.50	0.47	0.48
query6	1.14	0.74	0.72
query7	0.03	0.01	0.01
query8	0.04	0.05	0.04
query9	0.54	0.49	0.48
query10	0.54	0.55	0.54
query11	0.16	0.11	0.12
query12	0.14	0.13	0.12
query13	0.60	0.59	0.59
query14	0.78	0.76	0.79
query15	0.86	0.81	0.82
query16	0.37	0.36	0.37
query17	1.03	1.05	0.99
query18	0.23	0.22	0.23
query19	1.75	1.69	1.67
query20	0.01	0.01	0.01
query21	15.38	0.75	0.66
query22	3.69	7.75	1.99
query23	18.34	1.33	1.28
query24	2.08	0.21	0.22
query25	0.14	0.08	0.08
query26	0.30	0.22	0.21
query27	0.46	0.22	0.23
query28	13.38	1.01	1.00
query29	12.64	3.31	3.33
query30	0.24	0.05	0.04
query31	2.90	0.40	0.39
query32	3.30	0.48	0.48
query33	2.91	2.96	2.96
query34	17.13	4.34	4.40
query35	4.42	4.46	4.46
query36	0.65	0.47	0.48
query37	0.19	0.16	0.15
query38	0.16	0.14	0.15
query39	0.05	0.03	0.04
query40	0.16	0.13	0.12
query41	0.09	0.04	0.04
query42	0.05	0.05	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.45 s
Total hot run time: 30.75 s

@zddr
Copy link
Contributor Author

zddr commented Aug 14, 2024

run cloud_p1

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 14, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit 109dba6 into apache:master Aug 14, 2024
29 of 30 checks passed
zddr added a commit to zddr/incubator-doris that referenced this pull request Aug 16, 2024
@zddr
[enhance](mtmv)Improve the performance of obtaining partition/table v…
b175d19 
…ersions (apache#39301)

Batch retrieve version information of all tables and partitions used by
MTMV and store it in MTMVRefreshContext
@zddr zddr mentioned this pull request Aug 16, 2024
Merged
dataroaring pushed a commit that referenced this pull request Aug 17, 2024
@zddr @dataroaring
[enhance](mtmv)Improve the performance of obtaining partition/table v…
cbec39f 
…ersions (#39301)

Batch retrieve version information of all tables and partitions used by
MTMV and store it in MTMVRefreshContext
yiguolei pushed a commit that referenced this pull request Aug 21, 2024
@zddr
[enhance](mtmv)Improve the performance of obtaining partition/table v… (
77270a0 
#39478)

…ersions (#39301)
pick: #39301
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged doing reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@zddr @doris-robot @CalvinKirs @morrySnow @yiguolei @dataroaring