[Fix](function) fix coredump because short of check on randoms arguments #39255

zclllyybb · 2024-08-12T13:38:28Z

Proposed changes

Issue Number: close #xxx

before:
crash or

mysql [test]>select random(1,array_size(split_by_string(fcst_emp,','))) from test_random;
+---------------------------------------------------------+
| random(1, array_size(split_by_string(`fcst_emp`, ','))) |
+---------------------------------------------------------+
|                                     7471044383762196303 |
+---------------------------------------------------------+
1 row in set (0.05 sec)

now:
both for nereids and legacy planner:

mysql [test]>select random(1,array_size(split_by_string(fcst_emp,','))) from test_random;
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.8)[INVALID_ARGUMENT]The param of rand function must be literal

doc pr: apache/doris-website#992

doris-robot · 2024-08-12T13:38:33Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

zclllyybb · 2024-08-12T13:38:42Z

run buildall

github-actions · 2024-08-12T13:45:57Z

clang-tidy review says "All clean, LGTM! 👍"

zhiqiang-hhhh

The test only guarantee exception is thrown from FE. Modification on BE is not covered, and this is important, since we need to upgrade cluster.

When the pr is picked to 2.1, should add more tests to cover non-nereids planner.

zclllyybb · 2024-08-12T14:07:59Z

The test only guarantee exception is thrown from FE. Modification on BE is not covered, and this is important, since we need to upgrade cluster.

When the pr is picked to 2.1, should add more tests to cover non-nereids planner.

实际上，最后面加的那个testcase，就是接收的从BE报的错：

mysql [test]>set enable_nereids_planner=false;
Query OK, 0 rows affected (0.02 sec)

mysql [test]>select random(1,array_size(split_by_string(fcst_emp,','))) from test_random;
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.8)[INVALID_ARGUMENT]The param of rand function must be literal

剩下的交给升降级测试，回归里只能测到这些了（BE的报错本来就是兜底的，且老优化器的问题主要修复方式是不走老优化器）

zclllyybb · 2024-08-12T14:09:45Z

run buildall

github-actions · 2024-08-12T14:18:13Z

clang-tidy review says "All clean, LGTM! 👍"

zhiqiang-hhhh · 2024-08-12T14:43:54Z

The test only guarantee exception is thrown from FE. Modification on BE is not covered, and this is important, since we need to upgrade cluster.
When the pr is picked to 2.1, should add more tests to cover non-nereids planner.

实际上，最后面加的那个testcase，就是接收的从BE报的错：
mysql [test]>set enable_nereids_planner=false;
Query OK, 0 rows affected (0.02 sec)

mysql [test]>select random(1,array_size(split_by_string(fcst_emp,','))) from test_random;
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.8)[INVALID_ARGUMENT]The param of rand function must be literal
剩下的交给升降级测试，回归里只能测到这些了（BE的报错本来就是兜底的，且老优化器的问题主要修复方式是不走老优化器）

master 上还有老优化器么我记得不是删了么已经

doris-robot · 2024-08-12T14:45:42Z

TPC-H: Total hot run time: 39449 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ca2b78798ff35b40b0662b6650c1b17bbda94abd, data reload: false

------ Round 1 ----------------------------------
q1	17648	4380	4307	4307
q2	2017	208	201	201
q3	11483	1075	1062	1062
q4	10511	737	712	712
q5	7524	2601	2559	2559
q6	223	137	137	137
q7	988	590	603	590
q8	9406	1918	1926	1918
q9	10203	6529	6520	6520
q10	7009	2157	2155	2155
q11	443	240	251	240
q12	392	223	222	222
q13	18743	2998	2989	2989
q14	277	240	242	240
q15	520	479	502	479
q16	504	390	381	381
q17	993	641	664	641
q18	8017	7390	7265	7265
q19	4954	1008	1025	1008
q20	667	341	332	332
q21	5374	4485	4548	4485
q22	1079	1013	1006	1006
Total cold run time: 118975 ms
Total hot run time: 39449 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4422	4293	4268	4268
q2	367	271	269	269
q3	2848	2724	2824	2724
q4	2059	1690	1699	1690
q5	5659	5538	5498	5498
q6	227	130	129	129
q7	2151	1782	1736	1736
q8	3234	3387	3476	3387
q9	8763	8765	8812	8765
q10	3560	3305	3122	3122
q11	602	495	528	495
q12	823	665	610	610
q13	17204	3181	3179	3179
q14	313	285	278	278
q15	543	505	485	485
q16	508	422	441	422
q17	1838	1512	1517	1512
q18	7980	8020	7780	7780
q19	2829	1584	1608	1584
q20	2142	1880	1859	1859
q21	5457	5269	5139	5139
q22	1113	1079	1018	1018
Total cold run time: 74642 ms
Total hot run time: 55949 ms

doris-robot · 2024-08-12T14:58:16Z

TPC-DS: Total hot run time: 202994 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ca2b78798ff35b40b0662b6650c1b17bbda94abd, data reload: false

query1	1718	1467	1442	1442
query2	6429	1925	1926	1925
query3	13981	9342	308	308
query4	29917	23074	23102	23074
query5	5566	483	504	483
query6	313	170	177	170
query7	4584	303	309	303
query8	268	209	218	209
query9	8524	2521	2497	2497
query10	548	477	431	431
query11	16460	15016	14834	14834
query12	149	98	92	92
query13	1553	384	364	364
query14	10790	7669	7828	7669
query15	265	240	249	240
query16	7484	512	544	512
query17	1555	568	547	547
query18	1983	296	293	293
query19	215	145	142	142
query20	126	108	108	108
query21	198	100	107	100
query22	4697	4391	4344	4344
query23	34488	33562	33524	33524
query24	6197	2592	2534	2534
query25	478	395	365	365
query26	698	156	156	156
query27	1785	280	287	280
query28	4015	2082	2063	2063
query29	642	411	422	411
query30	216	144	151	144
query31	937	746	745	745
query32	94	54	53	53
query33	489	279	271	271
query34	869	453	468	453
query35	945	881	753	753
query36	1052	943	941	941
query37	136	79	79	79
query38	4211	4187	4102	4102
query39	1408	1367	1380	1367
query40	199	113	113	113
query41	46	41	44	41
query42	120	96	98	96
query43	511	481	460	460
query44	1077	733	722	722
query45	236	208	209	208
query46	1067	756	744	744
query47	1847	1752	1745	1745
query48	361	292	291	291
query49	747	412	427	412
query50	801	412	413	412
query51	6794	6695	6712	6695
query52	101	86	91	86
query53	260	186	183	183
query54	554	477	443	443
query55	74	75	75	75
query56	273	245	239	239
query57	1149	1088	1059	1059
query58	220	222	232	222
query59	2950	2803	2805	2803
query60	289	254	267	254
query61	97	91	94	91
query62	737	641	643	641
query63	211	184	182	182
query64	2709	1794	1730	1730
query65	3196	3121	3130	3121
query66	643	324	342	324
query67	15225	14723	14834	14723
query68	4617	540	546	540
query69	465	429	404	404
query70	1164	1165	1136	1136
query71	372	277	269	269
query72	18465	16796	17210	16796
query73	769	327	326	326
query74	9233	8784	8689	8689
query75	3381	2697	2611	2611
query76	2254	1048	953	953
query77	459	305	309	305
query78	9616	8952	10924	8952
query79	3207	518	530	518
query80	1013	507	539	507
query81	578	219	218	218
query82	746	139	133	133
query83	236	152	152	152
query84	232	78	79	78
query85	1404	270	273	270
query86	485	293	292	292
query87	4591	4451	4580	4451
query88	3847	2510	2465	2465
query89	405	285	278	278
query90	1930	198	201	198
query91	120	95	95	95
query92	64	49	51	49
query93	2705	540	540	540
query94	788	295	304	295
query95	346	270	263	263
query96	618	279	276	276
query97	3161	3078	3029	3029
query98	227	196	194	194
query99	1731	1269	1269	1269
Total cold run time: 297706 ms
Total hot run time: 202994 ms

doris-robot · 2024-08-12T15:03:48Z

ClickBench: Total hot run time: 31.06 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ca2b78798ff35b40b0662b6650c1b17bbda94abd, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.08
query5	0.50	0.49	0.49
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.56	0.50	0.50
query10	0.54	0.55	0.55
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.61	0.59	0.60
query14	0.77	0.78	0.78
query15	0.90	0.81	0.83
query16	0.37	0.35	0.37
query17	0.96	0.95	0.96
query18	0.23	0.23	0.22
query19	1.90	1.72	1.70
query20	0.03	0.01	0.01
query21	15.61	0.76	0.66
query22	4.39	7.16	2.35
query23	18.26	1.31	1.21
query24	2.14	0.22	0.22
query25	0.15	0.08	0.08
query26	0.30	0.22	0.21
query27	0.46	0.23	0.22
query28	13.28	1.02	1.00
query29	12.60	3.32	3.32
query30	0.24	0.05	0.05
query31	2.89	0.41	0.39
query32	3.27	0.47	0.47
query33	2.94	2.93	2.94
query34	17.19	4.36	4.36
query35	4.41	4.52	4.40
query36	0.65	0.46	0.47
query37	0.19	0.16	0.16
query38	0.15	0.14	0.15
query39	0.05	0.04	0.04
query40	0.15	0.12	0.12
query41	0.09	0.05	0.04
query42	0.06	0.05	0.04
query43	0.05	0.03	0.04
Total cold run time: 110.42 s
Total hot run time: 31.06 s

zclllyybb · 2024-08-13T02:14:22Z

The test only guarantee exception is thrown from FE. Modification on BE is not covered, and this is important, since we need to upgrade cluster.
When the pr is picked to 2.1, should add more tests to cover non-nereids planner.

实际上，最后面加的那个testcase，就是接收的从BE报的错：
mysql [test]>set enable_nereids_planner=false;
Query OK, 0 rows affected (0.02 sec)

mysql [test]>select random(1,array_size(split_by_string(fcst_emp,','))) from test_random;
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.8)[INVALID_ARGUMENT]The param of rand function must be literal
剩下的交给升降级测试，回归里只能测到这些了（BE的报错本来就是兜底的，且老优化器的问题主要修复方式是不走老优化器）
master 上还有老优化器么我记得不是删了么已经

还有，不清楚哪个版本删，而且我记得目前还不能直接删来着，有些语句新优化器都没支持上。。

master pr: apache/doris#39255

zhiqiang-hhhh

LGTM

github-actions · 2024-08-13T03:07:47Z

PR approved by anyone and no changes requested.

github-actions · 2024-08-13T07:00:36Z

PR approved by at least one committer and no changes requested.

…nts (apache#39255) ## Proposed changes Issue Number: close #xxx before: crash or ```sql mysql [test]>select random(1,array_size(split_by_string(fcst_emp,','))) from test_random; +---------------------------------------------------------+ | random(1, array_size(split_by_string(`fcst_emp`, ','))) | +---------------------------------------------------------+ | 7471044383762196303 | +---------------------------------------------------------+ 1 row in set (0.05 sec) ``` now: both for nereids and legacy planner: ```sql mysql [test]>select random(1,array_size(split_by_string(fcst_emp,','))) from test_random; ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.8)[INVALID_ARGUMENT]The param of rand function must be literal ``` doc pr: apache/doris-website#992

…nts (#39255) ## Proposed changes Issue Number: close #xxx before: crash or ```sql mysql [test]>select random(1,array_size(split_by_string(fcst_emp,','))) from test_random; +---------------------------------------------------------+ | random(1, array_size(split_by_string(`fcst_emp`, ','))) | +---------------------------------------------------------+ | 7471044383762196303 | +---------------------------------------------------------+ 1 row in set (0.05 sec) ``` now: both for nereids and legacy planner: ```sql mysql [test]>select random(1,array_size(split_by_string(fcst_emp,','))) from test_random; ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.8)[INVALID_ARGUMENT]The param of rand function must be literal ``` doc pr: apache/doris-website#992

…nts (apache#39255) ## Proposed changes Issue Number: close #xxx before: crash or ```sql mysql [test]>select random(1,array_size(split_by_string(fcst_emp,','))) from test_random; +---------------------------------------------------------+ | random(1, array_size(split_by_string(`fcst_emp`, ','))) | +---------------------------------------------------------+ | 7471044383762196303 | +---------------------------------------------------------+ 1 row in set (0.05 sec) ``` now: both for nereids and legacy planner: ```sql mysql [test]>select random(1,array_size(split_by_string(fcst_emp,','))) from test_random; ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.8)[INVALID_ARGUMENT]The param of rand function must be literal ``` doc pr: apache/doris-website#992

## Proposed changes Issue Number: close #xxx #38385 optimize parsing datetime #38978 make stream load failure message more clear and disable some error's stacktrace by default #39255 fix random function coredump #39324 fix function corr inconsistency with doc #39449 check auto partitoin nullity when creating partition #39695 make DynamicPartitionScheduler immediately know interval's change #39754 Add some partition expr check on creating table

…e#992) master pr: apache/doris#39255

github-actions bot added the doing label Aug 12, 2024

zclllyybb mentioned this pull request Aug 12, 2024

[fix](function) Add parameter restrictions for function random apache/doris-website#992

Merged

zhiqiang-hhhh suggested changes Aug 12, 2024

View reviewed changes

1

ca2b787

zclllyybb force-pushed the random branch from c969fb2 to ca2b787 Compare August 12, 2024 14:09

HappenLee pushed a commit to apache/doris-website that referenced this pull request Aug 13, 2024

[fix](function) Add parameter restrictions for function random (#992)

a925249

master pr: apache/doris#39255

zhiqiang-hhhh approved these changes Aug 13, 2024

View reviewed changes

github-actions bot added the reviewed label Aug 13, 2024

yiguolei approved these changes Aug 13, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 13, 2024

yiguolei merged commit 1f87ecf into apache:master Aug 13, 2024
29 of 30 checks passed

BiteTheDDDDt added dev/2.1.x dev/3.0.x labels Aug 13, 2024

dataroaring added dev/3.0.2-merged and removed dev/3.0.x labels Aug 17, 2024

zclllyybb deleted the random branch August 23, 2024 10:00

zclllyybb mentioned this pull request Aug 23, 2024

[branch-2.1] pick some prs #39860

Merged

yiguolei added dev/2.1.6-merged and removed dev/2.1.x labels Aug 24, 2024

Gabriel39 pushed a commit to Gabriel39/doris-website that referenced this pull request Sep 23, 2024

[fix](function) Add parameter restrictions for function random (apach…

8db5073

…e#992) master pr: apache/doris#39255

wyxxxcat mentioned this pull request Sep 29, 2024

[Bug] The second parameter of the Radom function being a variable can cause the BE node to crash #41454

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fix](function) fix coredump because short of check on randoms arguments #39255

[Fix](function) fix coredump because short of check on randoms arguments #39255

zclllyybb commented Aug 12, 2024 •

edited

Loading

doris-robot commented Aug 12, 2024

zclllyybb commented Aug 12, 2024

github-actions bot commented Aug 12, 2024

zhiqiang-hhhh left a comment

zclllyybb commented Aug 12, 2024

zclllyybb commented Aug 12, 2024

github-actions bot commented Aug 12, 2024

zhiqiang-hhhh commented Aug 12, 2024

doris-robot commented Aug 12, 2024

doris-robot commented Aug 12, 2024

doris-robot commented Aug 12, 2024

zclllyybb commented Aug 13, 2024

zhiqiang-hhhh left a comment

github-actions bot commented Aug 13, 2024

github-actions bot commented Aug 13, 2024

[Fix](function) fix coredump because short of check on randoms arguments #39255

[Fix](function) fix coredump because short of check on randoms arguments #39255

Conversation

zclllyybb commented Aug 12, 2024 • edited Loading

Proposed changes

doris-robot commented Aug 12, 2024

zclllyybb commented Aug 12, 2024

github-actions bot commented Aug 12, 2024

zhiqiang-hhhh left a comment

Choose a reason for hiding this comment

zclllyybb commented Aug 12, 2024

zclllyybb commented Aug 12, 2024

github-actions bot commented Aug 12, 2024

zhiqiang-hhhh commented Aug 12, 2024

doris-robot commented Aug 12, 2024

doris-robot commented Aug 12, 2024

doris-robot commented Aug 12, 2024

zclllyybb commented Aug 13, 2024

zhiqiang-hhhh left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 13, 2024

github-actions bot commented Aug 13, 2024

zclllyybb commented Aug 12, 2024 •

edited

Loading