[Fix](Nereids) fix fe fold constant evaluate like function #37616

LiBinfeng-01 · 2024-07-10T10:02:46Z

Problem:
When evaluating like function using fe, it can not evaluating nullliteral correctly
Example:
Null like "%string%" can not folded to null on fe
Reason:
Fe fold constant does not deal with like function
Solved:
Add fe fold constant of like function

doris-robot · 2024-07-10T10:02:51Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

LiBinfeng-01 · 2024-07-10T10:02:53Z

run buildall

morrySnow

add ut and test case

morrySnow · 2024-07-11T15:29:30Z

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJoin.java

why remove this?

it was a mistake, fixed

morrySnow · 2024-07-11T15:29:56Z

regression-test/data/nereids_hint_tpcds_p0/shape/query2.out

@@ -0,0 +1,39 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !ds_shape_2 --


should not in this pr?

morrySnow · 2024-07-11T15:31:31Z

...core/src/main/java/org/apache/doris/nereids/rules/expression/rules/FoldConstantRuleOnFE.java

@@ -405,6 +405,11 @@ public Expression visitOr(Or or, ExpressionRewriteContext context) {

    @Override
    public Expression visitLike(Like like, ExpressionRewriteContext context) {


maybe, it is better to just remove visitLike

xzj7019 · 2024-07-11T23:38:33Z

...core/src/main/java/org/apache/doris/nereids/rules/expression/rules/FoldConstantRuleOnFE.java

@@ -405,6 +405,11 @@ public Expression visitOr(Or or, ExpressionRewriteContext context) {

    @Override
    public Expression visitLike(Like like, ExpressionRewriteContext context) {


on 2.0, i think it's more important to investigate why c1 like '%xxx%' can't infer out c1 is not null by foldOnBe, than support like folding on fe.

yes, but i think we can fix this problem first by adding this feature to fe folding constant

LiBinfeng-01 · 2024-07-12T03:57:50Z

run buildall

LiBinfeng-01 · 2024-07-12T05:16:11Z

run buildall

doris-robot · 2024-07-12T06:01:00Z

TPC-H: Total hot run time: 39787 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dda6f828a03111d9747352f608ac0ec7437bafc8, data reload: false

------ Round 1 ----------------------------------
q1	17619	4318	4389	4318
q2	2008	192	187	187
q3	10494	1208	1088	1088
q4	10185	747	743	743
q5	7548	2672	2619	2619
q6	223	137	138	137
q7	948	607	606	606
q8	9219	2062	2029	2029
q9	8627	6560	6501	6501
q10	8652	3785	3793	3785
q11	480	239	236	236
q12	477	226	236	226
q13	17789	3005	2976	2976
q14	278	233	240	233
q15	539	491	492	491
q16	480	379	381	379
q17	941	631	667	631
q18	8127	7519	7329	7329
q19	9054	1485	1414	1414
q20	679	318	318	318
q21	5309	3235	3215	3215
q22	382	326	330	326
Total cold run time: 120058 ms
Total hot run time: 39787 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4394	4229	4202	4202
q2	372	262	261	261
q3	3063	2890	2910	2890
q4	1954	1718	1761	1718
q5	5543	5543	5411	5411
q6	226	144	136	136
q7	2222	1904	1848	1848
q8	3207	3354	3377	3354
q9	8769	8719	8877	8719
q10	4049	3905	3715	3715
q11	602	481	499	481
q12	808	637	645	637
q13	17234	3214	3175	3175
q14	326	280	289	280
q15	531	481	491	481
q16	481	441	460	441
q17	1796	1559	1498	1498
q18	8127	7982	7944	7944
q19	1729	1789	1508	1508
q20	2138	1902	1855	1855
q21	9612	4768	4639	4639
q22	657	577	541	541
Total cold run time: 77840 ms
Total hot run time: 55734 ms

doris-robot · 2024-07-12T06:12:08Z

TPC-DS: Total hot run time: 174432 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dda6f828a03111d9747352f608ac0ec7437bafc8, data reload: false

query1	919	366	362	362
query2	6430	2329	2294	2294
query3	6632	208	223	208
query4	28512	17420	17448	17420
query5	3657	485	501	485
query6	306	185	164	164
query7	4572	287	294	287
query8	293	285	273	273
query9	8493	2452	2451	2451
query10	438	292	269	269
query11	10583	9975	9999	9975
query12	112	83	87	83
query13	1657	368	371	368
query14	10418	7585	7746	7585
query15	239	185	183	183
query16	7328	342	319	319
query17	1334	558	522	522
query18	1245	275	272	272
query19	197	152	145	145
query20	87	82	81	81
query21	200	123	129	123
query22	4425	4005	4004	4004
query23	33986	33523	33671	33523
query24	11442	2926	2945	2926
query25	599	423	406	406
query26	702	154	150	150
query27	2280	284	280	280
query28	6594	2170	2177	2170
query29	895	643	632	632
query30	247	161	150	150
query31	991	779	750	750
query32	97	54	54	54
query33	744	294	346	294
query34	987	502	499	499
query35	729	593	608	593
query36	1129	984	991	984
query37	149	86	88	86
query38	2980	2911	2840	2840
query39	928	787	812	787
query40	202	115	117	115
query41	52	50	51	50
query42	112	99	96	96
query43	572	537	540	537
query44	1222	718	736	718
query45	194	171	164	164
query46	1091	733	724	724
query47	1866	1796	1799	1796
query48	361	294	291	291
query49	838	405	409	405
query50	776	392	395	392
query51	6871	6823	6676	6676
query52	103	91	90	90
query53	360	287	282	282
query54	895	458	454	454
query55	76	75	74	74
query56	283	259	263	259
query57	1138	1057	1047	1047
query58	233	244	276	244
query59	3166	3045	2948	2948
query60	294	270	289	270
query61	95	94	94	94
query62	829	652	631	631
query63	322	283	285	283
query64	9103	2202	1618	1618
query65	3181	3150	3104	3104
query66	747	334	327	327
query67	15360	15079	14974	14974
query68	4510	558	562	558
query69	534	421	361	361
query70	1216	1128	1213	1128
query71	413	280	275	275
query72	7417	5841	5721	5721
query73	748	329	331	329
query74	5942	5524	5496	5496
query75	3381	2759	2663	2663
query76	2128	911	966	911
query77	450	300	294	294
query78	10597	9072	8828	8828
query79	2822	533	531	531
query80	2021	468	471	468
query81	610	213	222	213
query82	822	133	143	133
query83	304	169	168	168
query84	279	86	85	85
query85	1762	314	304	304
query86	476	334	298	298
query87	3326	3085	3086	3085
query88	4134	2449	2470	2449
query89	475	382	376	376
query90	1749	184	192	184
query91	128	100	101	100
query92	60	51	48	48
query93	4464	525	512	512
query94	1103	210	212	210
query95	404	327	327	327
query96	616	271	276	271
query97	3237	2985	3057	2985
query98	290	207	203	203
query99	1548	1269	1258	1258
Total cold run time: 281864 ms
Total hot run time: 174432 ms

doris-robot · 2024-07-12T06:17:41Z

ClickBench: Total hot run time: 30.22 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dda6f828a03111d9747352f608ac0ec7437bafc8, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.22	0.05	0.05
query4	1.69	0.07	0.09
query5	0.50	0.48	0.49
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.49	0.49
query10	0.53	0.53	0.55
query11	0.15	0.12	0.11
query12	0.14	0.12	0.13
query13	0.61	0.59	0.58
query14	0.76	0.78	0.83
query15	0.86	0.82	0.81
query16	0.37	0.37	0.36
query17	0.93	1.00	1.02
query18	0.23	0.21	0.21
query19	1.83	1.75	1.69
query20	0.01	0.01	0.02
query21	15.41	0.74	0.64
query22	4.24	8.53	1.41
query23	18.28	1.44	1.36
query24	2.18	0.22	0.22
query25	0.17	0.08	0.09
query26	0.30	0.22	0.21
query27	0.45	0.23	0.23
query28	13.28	1.02	1.00
query29	12.59	3.27	3.29
query30	0.26	0.06	0.06
query31	2.88	0.40	0.40
query32	3.27	0.48	0.48
query33	2.89	2.97	2.88
query34	17.30	4.41	4.37
query35	4.39	4.42	4.46
query36	0.65	0.47	0.50
query37	0.19	0.16	0.16
query38	0.15	0.15	0.15
query39	0.05	0.03	0.04
query40	0.15	0.12	0.12
query41	0.09	0.04	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.95 s
Total hot run time: 30.22 s

LiBinfeng-01 · 2024-07-12T06:48:51Z

run buildall

doris-robot · 2024-07-12T07:09:33Z

TPC-H: Total hot run time: 39953 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dda6f828a03111d9747352f608ac0ec7437bafc8, data reload: false

------ Round 1 ----------------------------------
q1	18784	4637	4370	4370
q2	2991	200	188	188
q3	11337	1166	1106	1106
q4	10835	898	774	774
q5	7615	2695	2636	2636
q6	227	139	137	137
q7	983	601	587	587
q8	9233	2049	2071	2049
q9	8664	6568	6545	6545
q10	8752	3766	3786	3766
q11	503	247	238	238
q12	394	233	235	233
q13	17756	2992	2963	2963
q14	268	232	239	232
q15	553	469	478	469
q16	476	370	375	370
q17	954	699	647	647
q18	8033	7434	7381	7381
q19	5765	1495	1399	1399
q20	672	329	333	329
q21	4885	3240	3211	3211
q22	385	323	337	323
Total cold run time: 120065 ms
Total hot run time: 39953 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4344	4256	4267	4256
q2	386	278	270	270
q3	2963	2780	2692	2692
q4	1885	1628	1659	1628
q5	5288	5306	5302	5302
q6	223	131	129	129
q7	2108	1709	1732	1709
q8	3206	3345	3284	3284
q9	8401	8387	8388	8387
q10	3898	3767	3730	3730
q11	587	491	499	491
q12	773	654	625	625
q13	17479	2978	2980	2978
q14	305	279	270	270
q15	512	483	473	473
q16	483	411	420	411
q17	1794	1491	1476	1476
q18	7680	7478	7412	7412
q19	2662	1552	1577	1552
q20	2010	1770	1785	1770
q21	4894	4608	4642	4608
q22	638	542	531	531
Total cold run time: 72519 ms
Total hot run time: 53984 ms

doris-robot · 2024-07-12T07:21:04Z

TPC-DS: Total hot run time: 174231 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dda6f828a03111d9747352f608ac0ec7437bafc8, data reload: false

query1	916	369	358	358
query2	6451	2380	2333	2333
query3	6660	210	222	210
query4	27556	17695	17217	17217
query5	4176	494	514	494
query6	299	165	161	161
query7	4596	289	282	282
query8	298	278	282	278
query9	8589	2461	2456	2456
query10	440	301	265	265
query11	10820	10061	10227	10061
query12	125	83	87	83
query13	1633	370	361	361
query14	9423	7651	7453	7453
query15	232	184	183	183
query16	7660	318	322	318
query17	1440	553	536	536
query18	1919	277	274	274
query19	198	148	146	146
query20	89	81	82	81
query21	213	127	128	127
query22	4296	4106	3958	3958
query23	33774	33256	33057	33057
query24	11470	2861	2861	2861
query25	622	366	371	366
query26	1539	148	147	147
query27	2876	274	271	271
query28	7473	2107	2088	2088
query29	988	617	625	617
query30	280	145	148	145
query31	963	732	748	732
query32	90	52	54	52
query33	771	286	301	286
query34	942	489	498	489
query35	668	577	604	577
query36	1122	954	924	924
query37	150	80	87	80
query38	2884	2803	2752	2752
query39	869	793	790	790
query40	279	125	122	122
query41	56	53	54	53
query42	114	102	108	102
query43	583	559	560	559
query44	1181	740	715	715
query45	192	167	171	167
query46	1097	745	745	745
query47	1850	1813	1785	1785
query48	376	313	309	309
query49	1184	415	419	415
query50	777	398	399	398
query51	6995	6879	6816	6816
query52	103	93	92	92
query53	360	293	291	291
query54	1049	462	456	456
query55	78	79	75	75
query56	305	286	287	286
query57	1169	1082	1061	1061
query58	265	255	261	255
query59	3666	3308	3238	3238
query60	327	298	307	298
query61	118	114	115	114
query62	843	674	668	668
query63	321	288	292	288
query64	10643	2278	1762	1762
query65	3178	3091	3100	3091
query66	1410	350	350	350
query67	15450	15023	14917	14917
query68	5612	562	548	548
query69	702	456	356	356
query70	1194	1161	1167	1161
query71	484	283	286	283
query72	8791	5790	5433	5433
query73	805	335	324	324
query74	6083	5543	5646	5543
query75	4327	2670	2672	2670
query76	3694	971	952	952
query77	723	331	311	311
query78	9752	9030	9090	9030
query79	8385	523	530	523
query80	2304	473	476	473
query81	598	224	217	217
query82	1409	141	136	136
query83	312	168	194	168
query84	273	89	86	86
query85	1600	324	297	297
query86	474	323	326	323
query87	3303	3136	3148	3136
query88	5320	2539	2458	2458
query89	544	376	369	369
query90	1923	196	195	195
query91	132	105	103	103
query92	62	48	49	48
query93	6373	517	515	515
query94	1328	211	254	211
query95	407	323	330	323
query96	616	276	275	275
query97	3204	3058	3027	3027
query98	227	205	188	188
query99	1595	1252	1292	1252
Total cold run time: 302035 ms
Total hot run time: 174231 ms

doris-robot · 2024-07-12T07:26:35Z

ClickBench: Total hot run time: 31.01 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dda6f828a03111d9747352f608ac0ec7437bafc8, data reload: false

query1	0.04	0.04	0.04
query2	0.09	0.05	0.04
query3	0.23	0.06	0.05
query4	1.68	0.08	0.08
query5	0.50	0.48	0.47
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.54	0.54	0.53
query11	0.15	0.11	0.12
query12	0.14	0.12	0.12
query13	0.59	0.58	0.58
query14	0.76	0.79	0.79
query15	0.85	0.80	0.81
query16	0.37	0.37	0.36
query17	1.06	1.03	0.98
query18	0.22	0.22	0.22
query19	1.88	1.70	1.68
query20	0.02	0.01	0.01
query21	15.40	0.78	0.68
query22	4.18	7.01	2.25
query23	18.26	1.44	1.30
query24	2.07	0.25	0.22
query25	0.15	0.09	0.08
query26	0.31	0.22	0.21
query27	0.46	0.23	0.22
query28	13.30	1.03	1.00
query29	12.64	3.33	3.30
query30	0.25	0.06	0.05
query31	2.87	0.39	0.39
query32	3.27	0.47	0.48
query33	2.89	2.94	2.89
query34	16.97	4.38	4.34
query35	4.48	4.46	4.42
query36	0.64	0.48	0.49
query37	0.18	0.16	0.15
query38	0.16	0.15	0.15
query39	0.04	0.03	0.03
query40	0.15	0.12	0.12
query41	0.10	0.05	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.74 s
Total hot run time: 31.01 s

LiBinfeng-01 · 2024-07-15T01:59:00Z

run external

github-actions · 2024-07-15T07:14:35Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-15T07:14:37Z

PR approved by anyone and no changes requested.

) Problem: When evaluating like function using fe, it can not evaluating nullliteral correctly Example: Null like "%string%" can not folded to null on fe Reason: Fe fold constant does not deal with like function Solved: Add fe fold constant of like function

…37616) (#37863) cherry-pick from master

…37864) cherry-pick: #37616

) Problem: When evaluating like function using fe, it can not evaluating nullliteral correctly Example: Null like "%string%" can not folded to null on fe Reason: Fe fold constant does not deal with like function Solved: Add fe fold constant of like function

Problem: When evaluating like function using fe, it can not evaluating nullliteral correctly Example: Null like "%string%" can not folded to null on fe Reason: Fe fold constant does not deal with like function Solved: Add fe fold constant of like function

…pache#37616) (apache#37863) cherry-pick from master

morrySnow added dev/2.0.x dev/2.1.x labels Jul 10, 2024

morrySnow reviewed Jul 11, 2024

View reviewed changes

morrySnow added the usercase Important user case type label label Jul 11, 2024

xzj7019 reviewed Jul 11, 2024

View reviewed changes

LiBinfeng-01 force-pushed the fix_fold_constant branch from 3f6aa58 to cb39156 Compare July 12, 2024 03:56

[fix](Nereids) fix fe fold constant failed when using like function

dda6f82

LiBinfeng-01 force-pushed the fix_fold_constant branch from cb39156 to dda6f82 Compare July 12, 2024 05:15

morrySnow approved these changes Jul 15, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 15, 2024

github-actions bot added the reviewed label Jul 15, 2024

englefly approved these changes Jul 15, 2024

View reviewed changes

morrySnow merged commit 420c849 into apache:master Jul 15, 2024
29 of 32 checks passed

LiBinfeng-01 mentioned this pull request Jul 16, 2024

[fix](Nereids) fix fe fold constant failed when using like function #37863

Merged

LiBinfeng-01 mentioned this pull request Jul 16, 2024

[fix](Nereids) fix fe fold constant failed when using like function #37864

Merged

morrySnow added dev/2.0.14-merged and removed dev/2.0.x labels Jul 16, 2024

morrySnow pushed a commit that referenced this pull request Jul 16, 2024

[fix](Nereids) fix fe fold constant failed when using like function (#…

7d09754

…37616) (#37863) cherry-pick from master

yiguolei pushed a commit that referenced this pull request Jul 16, 2024

[fix](Nereids) fix fe fold constant failed when using like function (#…

cb2f95e

…37864) cherry-pick: #37616

yiguolei added dev/2.1.5-merged and removed dev/2.1.x labels Jul 16, 2024

dataroaring added the dev/3.0.1-merged label Aug 5, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024

[fix](Nereids) fix fe fold constant failed when using like function (a…

6017e72

…pache#37616) (apache#37863) cherry-pick from master

gavinchou mentioned this pull request Aug 19, 2024

Release Note 3.0.1 #39570

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fix](Nereids) fix fe fold constant evaluate like function #37616

[Fix](Nereids) fix fe fold constant evaluate like function #37616

LiBinfeng-01 commented Jul 10, 2024 •

edited

Loading

doris-robot commented Jul 10, 2024

LiBinfeng-01 commented Jul 10, 2024

morrySnow left a comment

morrySnow Jul 11, 2024

LiBinfeng-01 Jul 12, 2024

morrySnow Jul 11, 2024

morrySnow Jul 11, 2024

LiBinfeng-01 Jul 12, 2024

xzj7019 Jul 11, 2024

LiBinfeng-01 Jul 12, 2024

xzj7019 Jul 12, 2024

LiBinfeng-01 commented Jul 12, 2024

LiBinfeng-01 commented Jul 12, 2024

doris-robot commented Jul 12, 2024

doris-robot commented Jul 12, 2024

doris-robot commented Jul 12, 2024

LiBinfeng-01 commented Jul 12, 2024

doris-robot commented Jul 12, 2024

doris-robot commented Jul 12, 2024

doris-robot commented Jul 12, 2024

LiBinfeng-01 commented Jul 15, 2024

github-actions bot commented Jul 15, 2024

github-actions bot commented Jul 15, 2024

		@@ -0,0 +1,39 @@
		-- This file is automatically generated. You should know what you did if you want to edit this
		-- !ds_shape_2 --

		@@ -405,6 +405,11 @@ public Expression visitOr(Or or, ExpressionRewriteContext context) {

		@Override
		public Expression visitLike(Like like, ExpressionRewriteContext context) {

[Fix](Nereids) fix fe fold constant evaluate like function #37616

[Fix](Nereids) fix fe fold constant evaluate like function #37616

Conversation

LiBinfeng-01 commented Jul 10, 2024 • edited Loading

doris-robot commented Jul 10, 2024

LiBinfeng-01 commented Jul 10, 2024

morrySnow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LiBinfeng-01 commented Jul 12, 2024

LiBinfeng-01 commented Jul 12, 2024

doris-robot commented Jul 12, 2024

doris-robot commented Jul 12, 2024

doris-robot commented Jul 12, 2024

LiBinfeng-01 commented Jul 12, 2024

doris-robot commented Jul 12, 2024

doris-robot commented Jul 12, 2024

doris-robot commented Jul 12, 2024

LiBinfeng-01 commented Jul 15, 2024

github-actions bot commented Jul 15, 2024

github-actions bot commented Jul 15, 2024

LiBinfeng-01 commented Jul 10, 2024 •

edited

Loading