Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhancement](nereids) Speedup partition pruner #38191

Merged
merged 3 commits into from
Jul 23, 2024
Merged

[enhancement](nereids) Speedup partition pruner #38191

merged 3 commits into from
Jul 23, 2024

Conversation

924060929
Copy link
Contributor

Proposed changes

  1. fast return when partition predicate is true/false/null
  2. fast compute table's hash code
  3. fast merge two ranges when equals

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@924060929
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40234 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2ad4f7822858d34e05cbc5a44b6dc1f166d59ff3, data reload: false

------ Round 1 ----------------------------------
q1	17929	4480	4411	4411
q2	3027	195	208	195
q3	11182	1157	1142	1142
q4	10602	881	853	853
q5	7813	2758	2703	2703
q6	225	143	141	141
q7	992	624	623	623
q8	9242	2055	2074	2055
q9	8784	6543	6515	6515
q10	8654	3760	3753	3753
q11	449	247	239	239
q12	391	229	226	226
q13	18732	3006	3019	3006
q14	276	233	233	233
q15	536	482	487	482
q16	495	395	389	389
q17	975	624	757	624
q18	8142	7595	7513	7513
q19	4433	1358	1352	1352
q20	703	318	345	318
q21	4943	3182	3456	3182
q22	347	279	282	279
Total cold run time: 118872 ms
Total hot run time: 40234 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4357	4248	4411	4248
q2	384	271	270	270
q3	3002	2795	2737	2737
q4	1891	1588	1610	1588
q5	5317	5316	5324	5316
q6	218	130	131	130
q7	2168	1782	1741	1741
q8	3208	3340	3328	3328
q9	8433	8414	8371	8371
q10	3843	3710	3651	3651
q11	607	494	480	480
q12	768	567	588	567
q13	17543	3002	3008	3002
q14	294	271	280	271
q15	514	488	483	483
q16	457	424	407	407
q17	1752	1491	1453	1453
q18	7599	7576	7461	7461
q19	1660	1536	1591	1536
q20	2047	1831	1792	1792
q21	4899	4778	4622	4622
q22	616	507	510	507
Total cold run time: 71577 ms
Total hot run time: 53961 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174120 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2ad4f7822858d34e05cbc5a44b6dc1f166d59ff3, data reload: false

query1	912	371	397	371
query2	6438	1889	1753	1753
query3	6645	208	217	208
query4	28530	17578	17225	17225
query5	4223	479	472	472
query6	260	176	164	164
query7	4583	290	287	287
query8	237	212	194	194
query9	8499	2469	2437	2437
query10	435	299	268	268
query11	10622	10027	10018	10018
query12	138	91	83	83
query13	1637	398	370	370
query14	10374	7655	7619	7619
query15	215	169	164	164
query16	7707	480	439	439
query17	1574	571	537	537
query18	1837	281	269	269
query19	196	153	142	142
query20	90	78	80	78
query21	204	126	122	122
query22	4541	4177	4104	4104
query23	33612	33215	33365	33215
query24	11907	2903	2872	2872
query25	663	359	370	359
query26	1797	145	148	145
query27	2903	265	272	265
query28	7353	2040	2026	2026
query29	1102	640	613	613
query30	296	145	149	145
query31	959	762	738	738
query32	92	51	55	51
query33	772	317	327	317
query34	923	468	475	468
query35	846	755	736	736
query36	1097	954	940	940
query37	198	79	79	79
query38	2874	2759	2754	2754
query39	874	813	814	813
query40	312	121	118	118
query41	46	49	44	44
query42	120	106	102	102
query43	493	447	463	447
query44	1242	727	725	725
query45	191	159	162	159
query46	1083	718	702	702
query47	1892	1763	1771	1763
query48	366	291	293	291
query49	1173	409	422	409
query50	787	401	392	392
query51	6702	6664	6605	6605
query52	107	90	87	87
query53	360	293	292	292
query54	983	453	443	443
query55	76	73	74	73
query56	282	257	273	257
query57	1156	1065	1072	1065
query58	263	245	246	245
query59	3004	2697	2573	2573
query60	343	277	278	277
query61	96	94	125	94
query62	818	647	661	647
query63	320	288	286	286
query64	10476	2209	6362	2209
query65	3166	3127	3110	3110
query66	1373	345	354	345
query67	15583	15112	15077	15077
query68	4632	539	554	539
query69	548	423	368	368
query70	1174	1123	1165	1123
query71	410	290	283	283
query72	7229	5844	6035	5844
query73	745	327	325	325
query74	6109	5723	5676	5676
query75	3489	2694	2717	2694
query76	2842	940	903	903
query77	487	311	315	311
query78	9677	9139	9338	9139
query79	2174	529	526	526
query80	2093	474	483	474
query81	603	222	218	218
query82	828	132	135	132
query83	297	168	162	162
query84	262	87	88	87
query85	2313	314	306	306
query86	447	311	324	311
query87	3264	3097	3073	3073
query88	4297	2382	2356	2356
query89	483	380	385	380
query90	1886	189	195	189
query91	130	121	103	103
query92	66	49	48	48
query93	2407	512	512	512
query94	1308	281	282	281
query95	407	315	314	314
query96	599	269	272	269
query97	3252	3033	3082	3033
query98	209	208	201	201
query99	1537	1245	1243	1243
Total cold run time: 287182 ms
Total hot run time: 174120 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2ad4f7822858d34e05cbc5a44b6dc1f166d59ff3, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.04
query4	1.69	0.08	0.07
query5	0.57	0.48	0.47
query6	1.14	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.54	0.50	0.49
query10	0.54	0.54	0.55
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.58
query14	0.76	0.79	0.79
query15	0.87	0.83	0.82
query16	0.37	0.37	0.38
query17	1.00	1.00	0.94
query18	0.22	0.21	0.22
query19	1.86	1.69	1.68
query20	0.01	0.01	0.01
query21	15.40	0.81	0.67
query22	3.71	7.48	2.04
query23	18.27	1.32	1.31
query24	2.01	0.25	0.23
query25	0.15	0.08	0.09
query26	0.30	0.22	0.21
query27	0.45	0.23	0.23
query28	13.29	1.00	1.01
query29	12.65	3.34	3.28
query30	0.26	0.06	0.05
query31	2.88	0.38	0.38
query32	3.28	0.48	0.46
query33	2.88	2.89	2.96
query34	16.96	4.43	4.34
query35	4.41	4.37	4.39
query36	0.66	0.46	0.47
query37	0.19	0.15	0.16
query38	0.16	0.14	0.15
query39	0.04	0.03	0.04
query40	0.16	0.12	0.12
query41	0.08	0.04	0.04
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.17 s
Total hot run time: 30.65 s

@924060929
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39681 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 725b6137bb7ebe158f74f97b88637951f92327f1, data reload: false

------ Round 1 ----------------------------------
q1	17613	4351	4280	4280
q2	2008	194	189	189
q3	10446	1228	1085	1085
q4	10183	836	708	708
q5	7531	2688	2647	2647
q6	220	138	136	136
q7	951	592	589	589
q8	9216	2089	2082	2082
q9	8903	6568	6577	6568
q10	8872	3767	3774	3767
q11	454	237	239	237
q12	457	222	218	218
q13	17989	2943	2952	2943
q14	277	234	232	232
q15	530	488	486	486
q16	492	379	378	378
q17	963	667	626	626
q18	8041	7466	7423	7423
q19	6299	1371	1327	1327
q20	700	334	325	325
q21	4974	3148	3906	3148
q22	345	290	287	287
Total cold run time: 117464 ms
Total hot run time: 39681 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4424	4258	4240	4240
q2	382	276	268	268
q3	3000	2885	2899	2885
q4	2034	1730	1759	1730
q5	5756	5554	5514	5514
q6	239	132	136	132
q7	2220	1895	1858	1858
q8	3251	3448	3406	3406
q9	8800	8854	8831	8831
q10	4117	3699	3844	3699
q11	593	506	514	506
q12	791	632	649	632
q13	15937	3197	3132	3132
q14	316	274	277	274
q15	529	486	483	483
q16	480	464	422	422
q17	1856	1550	1511	1511
q18	8145	8100	7781	7781
q19	1726	1535	1655	1535
q20	2293	1876	1871	1871
q21	5086	4901	4656	4656
q22	598	531	508	508
Total cold run time: 72573 ms
Total hot run time: 55874 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174270 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 725b6137bb7ebe158f74f97b88637951f92327f1, data reload: false

query1	895	368	366	366
query2	6435	1970	1972	1970
query3	6640	209	217	209
query4	27600	17564	17362	17362
query5	3668	479	489	479
query6	275	168	156	156
query7	4582	298	287	287
query8	250	198	196	196
query9	8584	2460	2423	2423
query10	451	289	275	275
query11	12483	9999	9946	9946
query12	119	84	87	84
query13	1642	369	373	369
query14	10246	9089	7758	7758
query15	255	166	166	166
query16	7753	476	480	476
query17	1577	576	534	534
query18	1408	280	286	280
query19	198	152	155	152
query20	97	91	80	80
query21	208	127	130	127
query22	4534	4057	4096	4057
query23	34235	33794	33560	33560
query24	10611	2994	2913	2913
query25	590	394	374	374
query26	698	164	164	164
query27	2291	290	290	290
query28	6176	2050	2035	2035
query29	877	637	645	637
query30	262	158	152	152
query31	949	736	773	736
query32	100	55	53	53
query33	676	352	322	322
query34	894	504	512	504
query35	850	770	766	766
query36	1137	965	993	965
query37	149	84	86	84
query38	2977	2829	2892	2829
query39	906	843	813	813
query40	228	121	120	120
query41	49	51	47	47
query42	118	101	99	99
query43	505	457	478	457
query44	1074	729	717	717
query45	192	163	156	156
query46	1074	711	713	711
query47	1876	1768	1787	1768
query48	361	293	297	293
query49	826	415	419	415
query50	777	400	415	400
query51	6740	6659	6635	6635
query52	107	97	97	97
query53	359	298	299	298
query54	879	447	457	447
query55	73	74	77	74
query56	285	272	278	272
query57	1137	1071	1029	1029
query58	256	260	247	247
query59	2882	2672	2696	2672
query60	313	275	273	273
query61	98	97	97	97
query62	776	643	644	643
query63	325	295	339	295
query64	9141	2250	1688	1688
query65	3197	3077	3114	3077
query66	706	324	331	324
query67	15403	14958	14940	14940
query68	4497	544	539	539
query69	555	417	372	372
query70	1182	1110	1174	1110
query71	366	276	283	276
query72	7349	5572	5509	5509
query73	752	330	330	330
query74	6122	5674	5661	5661
query75	3328	2747	2700	2700
query76	2291	905	924	905
query77	479	314	307	307
query78	10206	9941	9017	9017
query79	1999	534	517	517
query80	1199	469	466	466
query81	585	223	220	220
query82	633	140	144	140
query83	237	167	166	166
query84	241	88	84	84
query85	1731	300	302	300
query86	486	322	311	311
query87	3319	3068	3113	3068
query88	4248	2443	2475	2443
query89	471	378	383	378
query90	1775	203	194	194
query91	130	102	101	101
query92	62	51	52	51
query93	1993	497	495	495
query94	1243	289	290	289
query95	413	324	321	321
query96	603	281	274	274
query97	3257	3081	3040	3040
query98	227	193	208	193
query99	1554	1270	1253	1253
Total cold run time: 277568 ms
Total hot run time: 174270 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.84 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 725b6137bb7ebe158f74f97b88637951f92327f1, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.06
query4	1.68	0.08	0.08
query5	0.49	0.50	0.49
query6	1.14	0.74	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.54	0.48	0.50
query10	0.55	0.55	0.54
query11	0.16	0.12	0.12
query12	0.15	0.12	0.13
query13	0.60	0.59	0.58
query14	0.77	0.78	0.76
query15	0.84	0.81	0.82
query16	0.37	0.37	0.38
query17	1.02	1.02	1.00
query18	0.23	0.23	0.22
query19	1.87	1.75	1.69
query20	0.01	0.01	0.01
query21	15.39	0.79	0.64
query22	3.96	6.84	2.02
query23	18.28	1.31	1.25
query24	2.33	0.24	0.22
query25	0.15	0.09	0.09
query26	0.37	0.22	0.21
query27	0.53	0.24	0.23
query28	13.63	1.02	1.00
query29	12.63	3.39	3.37
query30	0.25	0.06	0.06
query31	2.90	0.39	0.39
query32	3.25	0.48	0.47
query33	2.90	2.90	2.86
query34	17.05	4.37	4.38
query35	4.54	4.47	4.48
query36	0.67	0.48	0.46
query37	0.18	0.15	0.14
query38	0.15	0.15	0.14
query39	0.04	0.03	0.04
query40	0.15	0.12	0.12
query41	0.10	0.04	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.39 s
Total hot run time: 30.84 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 23, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@924060929 924060929 merged commit 2e0559b into apache:master Jul 23, 2024
28 of 31 checks passed
dataroaring pushed a commit that referenced this pull request Jul 24, 2024
@924060929 @dataroaring
[enhancement](nereids) Speedup partition pruner (#38191)
c86448e 
1. fast return when partition predicate is true/false/null
2. fast compute table's hash code
3. fast merge two ranges when equals
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Jul 26, 2024
@924060929
[enhancement](nereids) Speedup partition pruner (apache#38191)
614566c 
1. fast return when partition predicate is true/false/null
2. fast compute table's hash code
3. fast merge two ranges when equals

(cherry picked from commit 2e0559b)
@924060929 924060929 mentioned this pull request Jul 26, 2024
Merged
924060929 added a commit that referenced this pull request Jul 26, 2024
@924060929
[enhancement](nereids) Speedup partition pruner (#38191) (#38405)
ee65195 
1. fast return when partition predicate is true/false/null
2. fast compute table's hash code
3. fast merge two ranges when equals
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@englefly englefly englefly left review comments

@starocean999 starocean999 starocean999 approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@924060929 @doris-robot @englefly @starocean999 @morrySnow @dataroaring