Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug](delete) fix delete random distributed tbl #37985

Merged

Conversation

DarvenDuan
Copy link
Contributor

Proposed changes

Bug description:
In PR #33630, Doris supports auto aggregation for random distributed table, but it not only effects query statements, so if we delete from a random distributed table, will get an error because of unexpectedly rewriting.

CREATE TABLE `test_tbl` (
  `k` INT NULL,
  `v` BIGINT SUM NULL
) ENGINE=OLAP
AGGREGATE KEY(`k`)
DISTRIBUTED BY RANDOM BUCKETS AUTO;

mysql > delete from test_tbl where k=1;
ERROR 1105 (HY000): errCode = 2, detailMessage = Where clause only supports compound predicate, binary predicate, is_null predicate or in predicate.

fix:
Check whether it is a query statement before rewriting.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@DarvenDuan
Copy link
Contributor Author

run buildall

Copy link
Contributor

@morrySnow morrySnow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add test case please

@doris-robot
Copy link

TPC-H: Total hot run time: 39822 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 59e20f735c3dc4163c1b2bdef8edb80c79e417e5, data reload: false

------ Round 1 ----------------------------------
q1	17630	4518	4445	4445
q2	2014	195	192	192
q3	10453	1183	1090	1090
q4	10176	808	737	737
q5	7520	2697	2621	2621
q6	224	136	141	136
q7	960	602	609	602
q8	9216	2086	2046	2046
q9	8780	6573	6560	6560
q10	8762	3774	3760	3760
q11	461	241	242	241
q12	480	220	222	220
q13	17957	2967	2956	2956
q14	285	232	229	229
q15	531	468	477	468
q16	500	391	387	387
q17	981	646	657	646
q18	8123	7563	7355	7355
q19	6422	1437	1396	1396
q20	653	318	323	318
q21	5101	3133	3193	3133
q22	348	284	290	284
Total cold run time: 117577 ms
Total hot run time: 39822 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4372	4298	4265	4265
q2	367	268	269	268
q3	3026	2852	2894	2852
q4	2024	1715	1711	1711
q5	5621	5555	5460	5460
q6	227	147	142	142
q7	2185	1902	1812	1812
q8	3295	3407	3434	3407
q9	8790	8850	8848	8848
q10	4088	3951	3798	3798
q11	557	511	522	511
q12	852	640	643	640
q13	16966	3199	3211	3199
q14	329	283	271	271
q15	548	480	494	480
q16	501	437	447	437
q17	1817	1582	1524	1524
q18	8043	8100	7828	7828
q19	1831	1508	1524	1508
q20	2917	1883	1885	1883
q21	8497	4888	4849	4849
q22	571	490	528	490
Total cold run time: 77424 ms
Total hot run time: 56183 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174371 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 59e20f735c3dc4163c1b2bdef8edb80c79e417e5, data reload: false

query1	918	373	378	373
query2	6461	1897	1781	1781
query3	6658	209	216	209
query4	28363	17619	17410	17410
query5	3658	525	487	487
query6	271	193	163	163
query7	4579	302	289	289
query8	239	196	188	188
query9	8522	2413	2399	2399
query10	429	313	279	279
query11	11367	10104	9989	9989
query12	122	96	85	85
query13	1651	381	367	367
query14	10207	7991	7871	7871
query15	239	170	180	170
query16	7557	324	320	320
query17	1423	572	567	567
query18	1486	290	287	287
query19	204	156	158	156
query20	94	85	85	85
query21	203	137	129	129
query22	4349	4042	4000	4000
query23	34122	33782	33740	33740
query24	10988	2882	2937	2882
query25	616	444	436	436
query26	734	160	166	160
query27	2248	363	290	290
query28	5724	2075	2069	2069
query29	881	667	650	650
query30	254	155	155	155
query31	961	764	771	764
query32	94	56	55	55
query33	660	306	298	298
query34	932	506	502	502
query35	715	582	614	582
query36	1159	999	967	967
query37	154	86	87	86
query38	2963	2840	2871	2840
query39	896	816	816	816
query40	207	124	122	122
query41	49	44	45	44
query42	126	101	105	101
query43	492	479	483	479
query44	1091	735	729	729
query45	194	163	159	159
query46	1094	777	748	748
query47	1908	1794	1792	1792
query48	389	294	293	293
query49	839	410	425	410
query50	771	398	399	398
query51	6914	6736	6742	6736
query52	106	97	93	93
query53	366	292	290	290
query54	944	455	447	447
query55	77	75	78	75
query56	293	282	274	274
query57	1122	1069	1053	1053
query58	272	256	267	256
query59	2942	2786	2777	2777
query60	307	303	297	297
query61	99	96	98	96
query62	810	640	640	640
query63	312	291	291	291
query64	9180	2210	1704	1704
query65	3162	3090	3097	3090
query66	763	338	352	338
query67	15488	15032	14883	14883
query68	4535	559	540	540
query69	681	416	356	356
query70	1176	1140	1137	1137
query71	445	284	285	284
query72	7958	5902	5629	5629
query73	782	334	332	332
query74	6217	5723	5655	5655
query75	3697	2698	2668	2668
query76	2989	914	872	872
query77	678	324	317	317
query78	9807	8968	9167	8968
query79	2199	516	532	516
query80	1316	471	492	471
query81	590	274	218	218
query82	737	138	136	136
query83	323	169	168	168
query84	252	86	84	84
query85	1997	317	304	304
query86	470	313	282	282
query87	3316	3068	3120	3068
query88	4253	2360	2382	2360
query89	484	392	402	392
query90	1927	201	203	201
query91	140	104	103	103
query92	60	51	51	51
query93	2499	500	512	500
query94	1232	214	212	212
query95	412	315	326	315
query96	606	270	272	270
query97	3211	3009	3087	3009
query98	274	201	203	201
query99	1518	1241	1269	1241
Total cold run time: 280199 ms
Total hot run time: 174371 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.69 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 59e20f735c3dc4163c1b2bdef8edb80c79e417e5, data reload: false

query1	0.04	0.03	0.04
query2	0.09	0.04	0.04
query3	0.22	0.05	0.06
query4	1.66	0.07	0.06
query5	0.48	0.48	0.47
query6	1.14	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.55	0.49	0.50
query10	0.54	0.55	0.58
query11	0.16	0.11	0.12
query12	0.15	0.12	0.13
query13	0.59	0.58	0.58
query14	0.75	0.80	0.78
query15	0.86	0.81	0.81
query16	0.34	0.38	0.37
query17	1.01	1.01	0.97
query18	0.22	0.22	0.22
query19	1.93	1.77	1.71
query20	0.02	0.01	0.00
query21	15.39	0.75	0.66
query22	3.88	7.93	1.93
query23	18.66	1.29	1.26
query24	2.11	0.22	0.22
query25	0.15	0.09	0.09
query26	0.29	0.21	0.22
query27	0.46	0.23	0.23
query28	13.25	1.02	1.01
query29	12.58	3.37	3.37
query30	0.25	0.06	0.06
query31	2.86	0.39	0.38
query32	3.28	0.47	0.46
query33	2.89	2.87	2.90
query34	16.93	4.36	4.34
query35	4.44	4.42	4.48
query36	0.64	0.47	0.47
query37	0.19	0.16	0.15
query38	0.14	0.14	0.15
query39	0.04	0.04	0.04
query40	0.15	0.12	0.12
query41	0.09	0.04	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.03
Total cold run time: 109.59 s
Total hot run time: 30.69 s

@DarvenDuan
Copy link
Contributor Author

add test case please

done

@doris-robot
Copy link

TPC-H: Total hot run time: 39806 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ccaeac58ff78ab6e706896af0c17a5268e24c4a0, data reload: false

------ Round 1 ----------------------------------
q1	17631	4483	4305	4305
q2	2011	192	180	180
q3	10458	1202	1011	1011
q4	10181	728	786	728
q5	7539	2697	2667	2667
q6	220	136	136	136
q7	958	602	588	588
q8	9211	2059	2107	2059
q9	8731	6596	6592	6592
q10	8900	3763	3756	3756
q11	448	239	233	233
q12	505	222	221	221
q13	17856	2972	2971	2971
q14	270	233	236	233
q15	525	473	483	473
q16	498	375	375	375
q17	955	645	678	645
q18	8118	7506	7474	7474
q19	7112	1422	1461	1422
q20	672	311	329	311
q21	4924	3159	3150	3150
q22	329	276	288	276
Total cold run time: 118052 ms
Total hot run time: 39806 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4460	4290	4240	4240
q2	372	277	263	263
q3	2997	2918	2910	2910
q4	1986	1694	1669	1669
q5	5605	5508	5456	5456
q6	226	140	133	133
q7	2150	1885	1842	1842
q8	3279	3404	3410	3404
q9	8714	8730	8800	8730
q10	4113	3904	3801	3801
q11	597	489	507	489
q12	799	625	639	625
q13	16003	3180	3191	3180
q14	316	295	284	284
q15	546	489	467	467
q16	487	433	436	433
q17	1825	1532	1478	1478
q18	8079	7843	7658	7658
q19	4842	1657	1563	1563
q20	2138	1863	1852	1852
q21	5061	4885	4731	4731
q22	597	502	519	502
Total cold run time: 75192 ms
Total hot run time: 55710 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172758 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ccaeac58ff78ab6e706896af0c17a5268e24c4a0, data reload: false

query1	914	366	374	366
query2	6459	1829	1804	1804
query3	6651	217	226	217
query4	28089	17568	17354	17354
query5	3619	495	485	485
query6	246	172	161	161
query7	4576	286	300	286
query8	239	191	183	183
query9	8454	2388	2379	2379
query10	451	300	293	293
query11	12801	10169	10135	10135
query12	117	83	87	83
query13	1651	374	369	369
query14	10435	6928	7885	6928
query15	238	164	163	163
query16	7311	339	323	323
query17	1761	559	544	544
query18	1752	284	321	284
query19	195	153	147	147
query20	91	77	84	77
query21	207	137	125	125
query22	4395	4197	4091	4091
query23	34315	33707	33586	33586
query24	10484	2902	2881	2881
query25	590	406	389	389
query26	697	151	151	151
query27	2265	279	276	276
query28	6109	2056	2050	2050
query29	914	659	652	652
query30	244	150	155	150
query31	985	761	770	761
query32	94	53	56	53
query33	640	296	294	294
query34	910	506	498	498
query35	701	593	580	580
query36	1127	978	987	978
query37	153	88	93	88
query38	2983	2932	2798	2798
query39	951	852	844	844
query40	197	119	119	119
query41	41	39	42	39
query42	115	100	98	98
query43	499	475	464	464
query44	1087	723	741	723
query45	192	159	159	159
query46	1076	728	723	723
query47	1900	1768	1792	1768
query48	368	302	293	293
query49	828	438	412	412
query50	792	391	391	391
query51	7036	6881	6786	6786
query52	106	98	96	96
query53	365	297	296	296
query54	854	448	449	448
query55	75	76	74	74
query56	296	262	265	262
query57	1115	1042	1044	1042
query58	244	253	251	251
query59	3011	2615	2617	2615
query60	305	271	284	271
query61	95	98	95	95
query62	793	660	646	646
query63	314	284	288	284
query64	9144	2229	1731	1731
query65	3153	3128	3134	3128
query66	705	323	335	323
query67	15440	15067	14901	14901
query68	4553	558	539	539
query69	683	399	345	345
query70	1134	1164	1094	1094
query71	420	288	278	278
query72	7831	5117	5424	5117
query73	763	326	327	326
query74	6157	5739	5672	5672
query75	3405	2714	2748	2714
query76	2649	995	888	888
query77	651	310	310	310
query78	9692	8957	9224	8957
query79	2190	536	529	529
query80	2447	474	481	474
query81	596	221	234	221
query82	673	138	142	138
query83	295	168	175	168
query84	267	89	88	88
query85	1717	402	301	301
query86	473	319	319	319
query87	3300	3148	3156	3148
query88	3755	2341	2349	2341
query89	472	385	389	385
query90	1745	198	187	187
query91	128	103	104	103
query92	61	49	49	49
query93	2183	501	508	501
query94	1087	218	215	215
query95	406	321	320	320
query96	598	277	263	263
query97	3211	3039	3035	3035
query98	215	195	199	195
query99	1609	1274	1274	1274
Total cold run time: 280326 ms
Total hot run time: 172758 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.43 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ccaeac58ff78ab6e706896af0c17a5268e24c4a0, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.68	0.08	0.08
query5	0.48	0.47	0.49
query6	1.16	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.50	0.48
query10	0.55	0.53	0.56
query11	0.15	0.12	0.11
query12	0.15	0.13	0.12
query13	0.59	0.59	0.59
query14	0.77	0.77	0.78
query15	0.85	0.80	0.81
query16	0.38	0.38	0.37
query17	1.07	1.05	0.97
query18	0.23	0.21	0.21
query19	1.90	1.77	1.68
query20	0.02	0.01	0.01
query21	15.40	0.75	0.65
query22	4.24	6.61	2.71
query23	18.24	1.39	1.22
query24	2.10	0.23	0.23
query25	0.15	0.08	0.08
query26	0.30	0.21	0.20
query27	0.46	0.23	0.24
query28	13.28	1.02	1.00
query29	12.67	3.30	3.32
query30	0.25	0.06	0.06
query31	2.87	0.40	0.39
query32	3.26	0.48	0.47
query33	2.91	2.96	2.93
query34	17.27	4.38	4.39
query35	4.47	4.44	4.48
query36	0.66	0.47	0.49
query37	0.20	0.16	0.16
query38	0.16	0.14	0.15
query39	0.04	0.04	0.03
query40	0.14	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.2 s
Total hot run time: 31.43 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 18, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit ee0b4c6 into apache:master Jul 24, 2024
27 of 29 checks passed
dataroaring pushed a commit that referenced this pull request Jul 24, 2024
## Proposed changes

Bug description:
In PR #33630, Doris supports auto
aggregation for random distributed table, but it not only effects query
statements, so if we delete from a random distributed table, will get an
error because of unexpectedly rewriting.
```
CREATE TABLE `test_tbl` (
  `k` INT NULL,
  `v` BIGINT SUM NULL
) ENGINE=OLAP
AGGREGATE KEY(`k`)
DISTRIBUTED BY RANDOM BUCKETS AUTO;

mysql > delete from test_tbl where k=1;
ERROR 1105 (HY000): errCode = 2, detailMessage = Where clause only supports compound predicate, binary predicate, is_null predicate or in predicate.
```

fix:
Check whether it is a query statement before rewriting.
yiguolei pushed a commit that referenced this pull request Aug 23, 2024
## Proposed changes

cherry-pick #37985

<!--Describe your changes.-->
starocean999 pushed a commit that referenced this pull request Aug 27, 2024
## Proposed changes

After pr #37985, explain will not be rewritten for querying random
distributed table, so remove explain action in regression test case.

<!--Describe your changes.-->
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
3 tasks
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
## Proposed changes

After pr #37985, explain will not be rewritten for querying random
distributed table, so remove explain action in regression test case.

<!--Describe your changes.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants