Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature](mtmv)Support single table mv rewrite #34185

Merged
merged 5 commits into from
Apr 27, 2024

Conversation

seawinde
Copy link
Contributor

@seawinde seawinde commented Apr 26, 2024

Proposed changes

Support Single table query rewrite with out group by
this is useful for complex filter or expresission

the mv def and query is as following
which can be query rewritten

mv def:

          select *
            from lineitem where l_comment like '%xx%'

query:

            select l_linenumber, l_receiptdate
            from lineitem where l_comment like '%xx%'

Co-authored-by: zfr9527 qhu15zhang3294197@163.com

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@seawinde
Copy link
Contributor Author

run buildall

@seawinde
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-DS: Total hot run time: 186921 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8f0134b020ea9e6b1628d3fb2d781a9b4253c8a9, data reload: false

query1	918	365	344	344
query2	6292	2490	2420	2420
query3	6639	202	202	202
query4	23932	21710	22055	21710
query5	3787	419	405	405
query6	254	176	166	166
query7	4621	297	285	285
query8	236	180	198	180
query9	8523	2360	2345	2345
query10	409	247	277	247
query11	15298	14785	14790	14785
query12	120	85	87	85
query13	1629	363	379	363
query14	10414	7535	8186	7535
query15	252	165	167	165
query16	8191	250	262	250
query17	1868	547	540	540
query18	2083	272	266	266
query19	326	143	148	143
query20	92	84	82	82
query21	196	122	125	122
query22	5106	4913	4872	4872
query23	33872	33087	33241	33087
query24	10646	2977	2907	2907
query25	608	357	358	357
query26	1138	147	153	147
query27	2339	308	310	308
query28	6807	2006	1976	1976
query29	849	601	587	587
query30	241	151	157	151
query31	945	710	731	710
query32	89	51	50	50
query33	736	240	263	240
query34	1037	474	469	469
query35	793	656	653	653
query36	1075	885	903	885
query37	125	64	63	63
query38	3158	3015	3001	3001
query39	1562	1539	1549	1539
query40	201	124	122	122
query41	41	38	37	37
query42	100	93	94	93
query43	564	534	519	519
query44	1162	706	717	706
query45	270	271	267	267
query46	1081	748	693	693
query47	1939	1851	1827	1827
query48	371	294	288	288
query49	831	379	390	379
query50	796	366	368	366
query51	6759	6624	6585	6585
query52	94	97	86	86
query53	346	277	298	277
query54	305	229	233	229
query55	79	69	74	69
query56	232	222	222	222
query57	1186	1153	1138	1138
query58	224	198	199	198
query59	3393	3254	3089	3089
query60	258	226	229	226
query61	89	87	86	86
query62	640	439	437	437
query63	295	279	279	279
query64	8390	7137	7182	7137
query65	3133	3048	3049	3048
query66	798	331	322	322
query67	15528	15111	14995	14995
query68	5307	517	517	517
query69	479	302	294	294
query70	1196	1119	1151	1119
query71	408	265	266	265
query72	7200	2659	2417	2417
query73	707	322	315	315
query74	6447	6180	6180	6180
query75	3430	2659	2674	2659
query76	3238	958	1035	958
query77	426	261	259	259
query78	10896	10349	10268	10268
query79	7900	510	507	507
query80	2180	425	457	425
query81	538	230	225	225
query82	1180	91	90	90
query83	262	161	162	161
query84	260	81	82	81
query85	2114	264	262	262
query86	528	297	302	297
query87	3263	3085	3140	3085
query88	5062	2301	2293	2293
query89	481	382	372	372
query90	2011	177	175	175
query91	127	96	95	95
query92	65	45	45	45
query93	5961	497	495	495
query94	1177	173	179	173
query95	389	284	298	284
query96	595	267	257	257
query97	3123	2903	2943	2903
query98	232	210	219	210
query99	1220	873	887	873
Total cold run time: 294713 ms
Total hot run time: 186921 ms

@seawinde
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40402 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2811ee63a1c3f6a43fd76f4d013142590602fa41, data reload: false

------ Round 1 ----------------------------------
q1	18021	4778	4388	4388
q2	2978	187	189	187
q3	10844	1196	1245	1196
q4	10590	785	702	702
q5	8044	2769	2695	2695
q6	220	132	133	132
q7	1053	620	609	609
q8	9399	2158	2129	2129
q9	9245	6883	6731	6731
q10	9056	3749	3762	3749
q11	458	246	230	230
q12	506	215	211	211
q13	17760	2940	2916	2916
q14	265	220	224	220
q15	510	474	470	470
q16	506	373	378	373
q17	964	632	737	632
q18	8152	7476	7394	7394
q19	5017	1547	1543	1543
q20	649	302	305	302
q21	5028	3335	4020	3335
q22	330	258	275	258
Total cold run time: 119595 ms
Total hot run time: 40402 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4375	4191	4240	4191
q2	366	274	254	254
q3	3065	2713	2751	2713
q4	1880	1601	1607	1601
q5	5302	5327	5326	5326
q6	214	123	121	121
q7	2240	1846	1837	1837
q8	3215	3352	3390	3352
q9	8619	8544	8578	8544
q10	3872	3689	3691	3689
q11	574	475	485	475
q12	738	612	606	606
q13	16414	3004	2957	2957
q14	298	285	266	266
q15	525	461	476	461
q16	471	411	424	411
q17	1773	1517	1481	1481
q18	7691	7607	7414	7414
q19	1647	1549	1525	1525
q20	1972	1775	1771	1771
q21	4906	4745	4704	4704
q22	562	475	489	475
Total cold run time: 70719 ms
Total hot run time: 54174 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186036 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2811ee63a1c3f6a43fd76f4d013142590602fa41, data reload: false

query1	905	356	357	356
query2	6482	2351	2281	2281
query3	6651	206	205	205
query4	25717	21178	21284	21178
query5	4124	412	430	412
query6	276	199	191	191
query7	4582	297	289	289
query8	251	188	200	188
query9	8562	2318	2320	2318
query10	427	240	262	240
query11	14656	14234	14213	14213
query12	132	86	87	86
query13	1635	365	356	356
query14	10399	7330	8204	7330
query15	249	166	177	166
query16	8185	259	259	259
query17	1854	576	557	557
query18	2098	282	280	280
query19	284	159	157	157
query20	93	90	87	87
query21	201	132	126	126
query22	5088	4943	4866	4866
query23	34001	33294	33025	33025
query24	10889	3020	2949	2949
query25	604	369	367	367
query26	1341	146	147	146
query27	2944	314	314	314
query28	7654	2013	2011	2011
query29	864	600	591	591
query30	272	157	167	157
query31	963	742	735	735
query32	92	53	52	52
query33	739	245	235	235
query34	1057	473	476	473
query35	811	659	698	659
query36	1064	880	908	880
query37	140	66	68	66
query38	3147	3015	3010	3010
query39	1586	1545	1551	1545
query40	198	123	126	123
query41	44	39	39	39
query42	104	100	96	96
query43	596	552	513	513
query44	1351	754	749	749
query45	282	280	270	270
query46	1090	732	735	732
query47	1976	1865	1883	1865
query48	363	297	305	297
query49	1079	395	393	393
query50	783	377	380	377
query51	6640	6627	6679	6627
query52	107	85	96	85
query53	356	279	274	274
query54	291	231	225	225
query55	74	70	70	70
query56	246	227	224	224
query57	1189	1118	1132	1118
query58	226	199	199	199
query59	3380	3174	3012	3012
query60	250	234	227	227
query61	90	87	86	86
query62	654	444	437	437
query63	304	275	277	275
query64	8609	7193	7197	7193
query65	3077	3038	3066	3038
query66	913	335	327	327
query67	15658	15365	15198	15198
query68	5214	529	551	529
query69	486	308	301	301
query70	1140	1148	1113	1113
query71	412	264	266	264
query72	7741	2620	2463	2463
query73	700	315	322	315
query74	6482	6104	6077	6077
query75	3383	2708	2639	2639
query76	3385	1009	982	982
query77	416	268	261	261
query78	11180	10379	10248	10248
query79	4102	519	524	519
query80	1891	419	429	419
query81	541	222	229	222
query82	820	99	90	90
query83	323	174	168	168
query84	265	84	93	84
query85	2003	267	262	262
query86	513	280	309	280
query87	3288	3109	3086	3086
query88	4693	2330	2329	2329
query89	499	375	373	373
query90	2027	180	181	180
query91	122	98	97	97
query92	61	49	47	47
query93	5127	504	507	504
query94	1268	182	180	180
query95	390	297	295	295
query96	610	266	263	263
query97	3126	2928	2947	2928
query98	227	224	217	217
query99	1255	876	878	876
Total cold run time: 294639 ms
Total hot run time: 186036 ms

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 27, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit 0f95160 into apache:master Apr 27, 2024
26 of 29 checks passed
seawinde added a commit to seawinde/doris that referenced this pull request Apr 27, 2024
Support Single table  query rewrite with out group by
this is useful for complex filter or expresission

the mv def and query is as following
which can be query rewritten

mv def:
```
          select *
            from lineitem where l_comment like '%xx%'
```

query:
```
            select l_linenumber, l_receiptdate
            from lineitem where l_comment like '%xx%'
```


Co-authored-by: zfr9527 <qhu15zhang3294197@163.com>
seawinde added a commit to seawinde/doris that referenced this pull request Apr 27, 2024
Support Single table  query rewrite with out group by
this is useful for complex filter or expresission

the mv def and query is as following
which can be query rewritten

mv def:
```
          select *
            from lineitem where l_comment like '%xx%'
```

query:
```
            select l_linenumber, l_receiptdate
            from lineitem where l_comment like '%xx%'
```


Co-authored-by: zfr9527 <qhu15zhang3294197@163.com>
yiguolei pushed a commit that referenced this pull request Apr 27, 2024
Support Single table  query rewrite with out group by
this is useful for complex filter or expresission

the mv def and query is as following
which can be query rewritten

mv def:
```
          select *
            from lineitem where l_comment like '%xx%'
```

query:
```
            select l_linenumber, l_receiptdate
            from lineitem where l_comment like '%xx%'
```

Co-authored-by: zfr9527 <qhu15zhang3294197@163.com>
seawinde added a commit to seawinde/doris that referenced this pull request May 22, 2024
Support Single table  query rewrite with out group by
this is useful for complex filter or expresission

the mv def and query is as following
which can be query rewritten

mv def:
```
          select *
            from lineitem where l_comment like '%xx%'
```

query:
```
            select l_linenumber, l_receiptdate
            from lineitem where l_comment like '%xx%'
```


Co-authored-by: zfr9527 <qhu15zhang3294197@163.com>
yiguolei pushed a commit that referenced this pull request May 23, 2024
Support Single table  query rewrite with out group by
this is useful for complex filter or expresission

the mv def and query is as following
which can be query rewritten

mv def:
```
          select *
            from lineitem where l_comment like '%xx%'
```

query:
```
            select l_linenumber, l_receiptdate
            from lineitem where l_comment like '%xx%'
```

Co-authored-by: zfr9527 <qhu15zhang3294197@163.com>
morrySnow pushed a commit that referenced this pull request Aug 20, 2024
… group by (#39041)


this is brought by #34185
if query external table without group by, rewrite by materialized view
will fail
such as mv def is 

            select  o_custkey, o_orderdate 
            from ${hive_catalog_name}.${hive_database}.${hive_table};

query is query external table directly as following, this would fail
when try to rewrte by materialized view

            select o_custkey 
            from ${hive_catalog_name}.${hive_database}.${hive_table};

this pr fix the problem.
seawinde added a commit to seawinde/doris that referenced this pull request Aug 21, 2024
… group by (apache#39041)

this is brought by apache#34185
if query external table without group by, rewrite by materialized view
will fail
such as mv def is

            select  o_custkey, o_orderdate
            from ${hive_catalog_name}.${hive_database}.${hive_table};

query is query external table directly as following, this would fail
when try to rewrte by materialized view

            select o_custkey
            from ${hive_catalog_name}.${hive_database}.${hive_table};

this pr fix the problem.
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
… group by (#39041)


this is brought by #34185
if query external table without group by, rewrite by materialized view
will fail
such as mv def is 

            select  o_custkey, o_orderdate 
            from ${hive_catalog_name}.${hive_database}.${hive_table};

query is query external table directly as following, this would fail
when try to rewrte by materialized view

            select o_custkey 
            from ${hive_catalog_name}.${hive_database}.${hive_table};

this pr fix the problem.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.4-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants