Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FIX](type)fix matchExactType for complex type #28233

Merged
merged 8 commits into from
Dec 21, 2023

Conversation

amorynan
Copy link
Contributor

Proposed changes

Issue Number: close #xxx
fe matchExactType function should call type.matchTypes for its own logic, do not switch case to do special logic otherwise we may meet core in be like this.

F20231208 18:54:39.359673 680131 block.h:535] Check failed: _data_types[i]->is_nullable()  target type: Struct(l_info:Nullable(Array(Nullable(String)))) src type: Struct(col:Nullable(Array(Nullable(UInt8))))
*** Check failure stack trace: ***
   @     0x5584e952b926  google::LogMessage::SendToLog()
   @     0x5584e9527ef0  google::LogMessage::Flush()
   @     0x5584e952c169  google::LogMessageFatal::~LogMessageFatal()
   @     0x5584cf17201e  doris::vectorized::MutableBlock::merge_impl<>()
   @     0x5584ceac4b1d  doris::vectorized::MutableBlock::merge<>()
   @     0x5584d4dd7de3  doris::vectorized::VUnionNode::get_next_const()
   @     0x5584d4dd9a45  doris::vectorized::VUnionNode::get_next()
   @     0x5584bce469bd  std::__invoke_impl<>()
   @     0x5584bce466d0  std::__invoke<>()
   @     0x5584bce465c7  _ZNSt5_BindIFMN5doris8ExecNodeEFNS0_6StatusEPNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS1_St12_PlaceholderILi1EESC_ILi2EESC_ILi3EEEE6__callIS2_JOS4_OS7_OS8_EJLm0ELm1ELm2ELm3EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
   @     0x5584bce46358  std::_Bind<>::operator()<>()
   @     0x5584bce46208  std::__invoke_impl<>()
   @     0x5584bce46178  _ZSt10__invoke_rIN5doris6StatusERSt5_BindIFMNS0_8ExecNodeEFS1_PNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS3_St12_PlaceholderILi1EESD_ILi2EESD_ILi3EEEEJS5_S8_S9_EENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESL_E4typeEOSM_DpOSN_
   @     0x5584bce45c18  std::_Function_handler<>::_M_invoke()
   @     0x5584bce6412f  std::function<>::operator()()
   @     0x5584bce56382  doris::ExecNode::get_next_after_projects()
   @     0x5584bce26218  doris::PlanFragmentExecutor::get_vectorized_internal()
   @     0x5584bce2431b  doris::PlanFragmentExecutor::open_vectorized_internal()
   @     0x5584bce22a96  doris::PlanFragmentExecutor::open()
   @     0x5584bce27c9d  doris::PlanFragmentExecutor::execute()
   @     0x5584bcbdb3f8  doris::FragmentMgr::_exec_actual()
   @     0x5584bcbf982f  doris::FragmentMgr::exec_plan_fragment()::$_0::operator()()
   @     0x5584bcbf9715  std::__invoke_impl<>()
   @     0x5584bcbf96b5  _ZSt10__invoke_rIvRZN5doris11FragmentMgr18exec_plan_fragmentERKNS0_23TExecPlanFragmentParamsERKSt8functionIFvPNS0_12RuntimeStateEPNS0_6StatusEEEE3$_0JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESH_E4typeEOSI_DpOSJ_
   @     0x5584bcbf942d  std::_Function_handler<>::_M_invoke()
   @     0x5584b9dfd883  std::function<>::operator()()
   @     0x5584bd6e3929  doris::FunctionRunnable::run()
   @     0x5584bd6cf8ce  doris::ThreadPool::dispatch_thread()

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@amorynan
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit d5482820c354acb090b7512a47190aeb231b1818, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4697	4528	4431	4431
q2	361	159	153	153
q3	1467	1247	1243	1243
q4	1106	911	900	900
q5	3103	3118	3104	3104
q6	253	127	126	126
q7	993	496	476	476
q8	2189	2207	2164	2164
q9	6629	6671	6637	6637
q10	3209	3467	3261	3261
q11	329	209	202	202
q12	352	211	212	211
q13	4602	3774	3792	3774
q14	241	210	215	210
q15	562	523	521	521
q16	454	394	403	394
q17	1000	532	529	529
q18	7586	8347	7158	7158
q19	1503	1284	1439	1284
q20	552	290	645	290
q21	3075	2655	2659	2655
q22	344	283	290	283
Total cold run time: 44607 ms
Total hot run time: 40006 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4399	4394	4358	4358
q2	267	169	179	169
q3	3559	3603	3545	3545
q4	2388	2382	2382	2382
q5	5754	5759	5747	5747
q6	239	120	125	120
q7	2393	1854	1866	1854
q8	3529	3529	3530	3529
q9	9091	9068	9013	9013
q10	3879	3985	3986	3985
q11	506	421	403	403
q12	767	604	604	604
q13	4281	3589	3587	3587
q14	287	256	253	253
q15	569	528	513	513
q16	515	464	479	464
q17	1920	1886	1847	1847
q18	8737	8464	8433	8433
q19	1746	1745	1727	1727
q20	2249	1959	1957	1957
q21	6548	6205	6179	6179
q22	497	412	441	412
Total cold run time: 64120 ms
Total hot run time: 61081 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.59 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17222749800 Bytes

@BePPPower
Copy link
Contributor

Please add some test cases

@amorynan
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.73 seconds
stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17225036941 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit e3caf16623b3aedd31cec7ba1cc9ec445111dc6b, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4665	4437	4466	4437
q2	369	153	159	153
q3	1452	1230	1229	1229
q4	1105	910	870	870
q5	3105	3156	3133	3133
q6	253	127	128	127
q7	1007	481	479	479
q8	2181	2232	2188	2188
q9	6625	6667	6667	6667
q10	3218	3258	3270	3258
q11	334	208	210	208
q12	351	202	212	202
q13	4550	4479	3831	3831
q14	238	213	215	213
q15	569	519	522	519
q16	442	391	392	391
q17	1004	551	548	548
q18	7598	7155	7090	7090
q19	1529	1265	1458	1265
q20	1608	531	340	340
q21	3046	2665	2734	2665
q22	351	279	293	279
Total cold run time: 45600 ms
Total hot run time: 40092 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4387	4409	4426	4409
q2	266	164	181	164
q3	3560	3542	3557	3542
q4	2375	2392	2367	2367
q5	5741	5745	5763	5745
q6	242	121	121	121
q7	2379	1854	1873	1854
q8	3506	3506	3520	3506
q9	9027	9056	9045	9045
q10	3881	3986	4000	3986
q11	497	390	392	390
q12	766	602	608	602
q13	4308	3559	3544	3544
q14	293	244	255	244
q15	576	523	525	523
q16	494	484	452	452
q17	1873	1837	1847	1837
q18	8780	8323	8460	8323
q19	1739	1760	1736	1736
q20	2260	1952	1948	1948
q21	6559	6163	6181	6163
q22	510	423	425	423
Total cold run time: 64019 ms
Total hot run time: 60924 ms

@BePPPower
Copy link
Contributor

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.71 seconds
stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17217007248 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 054be24c91c6a8f1ea78e0327f1296a4c552b9eb, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4659	4441	4450	4441
q2	360	163	159	159
q3	1441	1248	1238	1238
q4	1116	934	958	934
q5	3093	3151	3136	3136
q6	249	128	128	128
q7	994	480	471	471
q8	2206	2215	2164	2164
q9	6654	6683	6670	6670
q10	3196	3239	3250	3239
q11	317	208	209	208
q12	354	209	202	202
q13	4540	3785	3942	3785
q14	238	212	209	209
q15	566	525	516	516
q16	440	385	386	385
q17	1007	611	498	498
q18	8188	7239	6901	6901
q19	1527	1341	1445	1341
q20	505	309	329	309
q21	3062	2652	2675	2652
q22	349	279	287	279
Total cold run time: 45061 ms
Total hot run time: 39865 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4418	4428	4439	4428
q2	267	167	176	167
q3	3552	3537	3537	3537
q4	2391	2458	2458	2458
q5	5768	5759	5747	5747
q6	240	120	119	119
q7	2389	1854	1894	1854
q8	3520	3510	3541	3510
q9	9065	9012	8979	8979
q10	3912	4001	4003	4001
q11	487	395	381	381
q12	763	591	622	591
q13	4284	3543	3546	3543
q14	285	265	243	243
q15	570	524	520	520
q16	501	455	463	455
q17	1868	1901	1869	1869
q18	8746	8430	8674	8430
q19	1717	1737	1755	1737
q20	2288	1940	1949	1940
q21	6529	6184	6213	6184
q22	509	442	432	432
Total cold run time: 64069 ms
Total hot run time: 61125 ms

@amorynan amorynan changed the title [FIX](type)fix type match type [FIX](type)fix matchExactType for complex type Dec 13, 2023
@amorynan
Copy link
Contributor Author

run buildall

@amorynan amorynan closed this Dec 14, 2023
@amorynan amorynan reopened this Dec 14, 2023
@amorynan
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 331968ed3a39ce0d832a82c3984f3189d99d8ea7, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4719	4441	4488	4441
q2	380	138	159	138
q3	1446	1268	1330	1268
q4	1117	923	874	874
q5	3131	3154	3147	3147
q6	251	129	129	129
q7	978	507	485	485
q8	2172	2209	2174	2174
q9	6678	6677	6618	6618
q10	3218	3261	3258	3258
q11	315	210	210	210
q12	353	206	209	206
q13	4559	3799	3774	3774
q14	239	218	211	211
q15	564	524	512	512
q16	440	387	381	381
q17	1007	577	577	577
q18	7166	6931	6954	6931
q19	1517	1383	1395	1383
q20	547	298	313	298
q21	3068	2582	2637	2582
q22	343	276	283	276
Total cold run time: 44208 ms
Total hot run time: 39873 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4428	4407	4336	4336
q2	269	165	177	165
q3	3558	3523	3515	3515
q4	2376	2363	2361	2361
q5	5719	5701	5737	5701
q6	239	121	121	121
q7	2371	1884	1856	1856
q8	3510	3495	3521	3495
q9	9056	8973	9016	8973
q10	3913	3982	3980	3980
q11	507	383	386	383
q12	761	592	597	592
q13	4279	3566	3573	3566
q14	289	259	269	259
q15	577	518	522	518
q16	501	451	466	451
q17	1860	1835	1842	1835
q18	8617	8165	8176	8165
q19	1719	1718	1704	1704
q20	2265	1940	1938	1938
q21	6500	6151	6133	6133
q22	504	422	416	416
Total cold run time: 63818 ms
Total hot run time: 60463 ms

@amorynan
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 79886fd57accaa49c0248680ec2642b394ff3c49, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4744	4483	4498	4483
q2	372	143	158	143
q3	1455	1251	1238	1238
q4	1118	916	877	877
q5	3192	3180	3140	3140
q6	248	131	133	131
q7	1016	484	477	477
q8	2237	2223	2198	2198
q9	6695	6644	6670	6644
q10	3223	3263	3259	3259
q11	323	201	204	201
q12	354	210	210	210
q13	4562	3778	3805	3778
q14	247	211	223	211
q15	569	530	517	517
q16	446	385	389	385
q17	1005	576	543	543
q18	7195	6959	6904	6904
q19	1506	1347	1448	1347
q20	521	340	318	318
q21	3057	2623	2659	2623
q22	350	278	291	278
Total cold run time: 44435 ms
Total hot run time: 39905 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4431	4424	4408	4408
q2	270	161	177	161
q3	3546	3533	3522	3522
q4	2377	2375	2367	2367
q5	5726	5732	5733	5732
q6	240	121	123	121
q7	2376	1856	1893	1856
q8	3517	3515	3519	3515
q9	9036	9038	9015	9015
q10	3922	4000	3977	3977
q11	504	382	381	381
q12	764	603	604	603
q13	4299	3587	3564	3564
q14	284	257	259	257
q15	579	530	530	530
q16	502	508	471	471
q17	1903	1857	1856	1856
q18	8610	8314	8383	8314
q19	1761	1729	1715	1715
q20	2258	1942	1939	1939
q21	6514	6159	6194	6159
q22	497	419	440	419
Total cold run time: 63916 ms
Total hot run time: 60882 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.35 seconds
stream load tsv: 586 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17219519541 Bytes

@amorynan
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.1 seconds
stream load tsv: 585 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.2 seconds inserted 10000000 Rows, about 354K ops/s
storage size: 17224253563 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 15dee6900b749606b212f25d5e741f2eb27f7c68, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4719	4500	4499	4499
q2	366	152	158	152
q3	1469	1252	1198	1198
q4	1121	927	917	917
q5	3175	3204	3185	3185
q6	246	129	130	129
q7	1023	490	484	484
q8	2195	2225	2212	2212
q9	6677	6681	6649	6649
q10	3238	3271	3288	3271
q11	333	208	201	201
q12	355	208	208	208
q13	4570	3773	3792	3773
q14	250	216	220	216
q15	571	526	526	526
q16	442	389	389	389
q17	1025	639	559	559
q18	7110	6872	6785	6785
q19	1550	1437	1427	1427
q20	519	321	280	280
q21	3101	2629	2675	2629
q22	354	279	286	279
Total cold run time: 44409 ms
Total hot run time: 39968 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4417	4427	4437	4427
q2	266	165	171	165
q3	3512	3506	3502	3502
q4	2392	2376	2371	2371
q5	5748	5752	5758	5752
q6	235	119	120	119
q7	2369	1873	1839	1839
q8	3516	3525	3534	3525
q9	9109	9018	8979	8979
q10	3932	4023	4030	4023
q11	511	384	393	384
q12	763	601	622	601
q13	4306	3583	3527	3527
q14	283	268	262	262
q15	573	524	531	524
q16	531	456	475	456
q17	1897	1898	1871	1871
q18	8583	8201	8197	8197
q19	1724	1720	1741	1720
q20	2255	1951	1941	1941
q21	6481	6146	6156	6146
q22	506	420	429	420
Total cold run time: 63909 ms
Total hot run time: 60751 ms

@amorynan
Copy link
Contributor Author

run pipelinex_p0

1 similar comment
@amorynan
Copy link
Contributor Author

run pipelinex_p0

morrySnow
morrySnow previously approved these changes Dec 19, 2023
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 19, 2023
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@amorynan
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 20, 2023
@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit d8c9bce0d585470e6299593d49a009226b62cd60, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4654	4395	4404	4395
q2	360	152	157	152
q3	1458	1257	1229	1229
q4	1114	920	924	920
q5	3158	3159	3177	3159
q6	244	127	125	125
q7	995	491	496	491
q8	2176	2253	2184	2184
q9	6699	6715	6736	6715
q10	3223	3262	3296	3262
q11	310	184	182	182
q12	349	212	205	205
q13	4578	3855	3779	3779
q14	246	210	218	210
q15	576	525	523	523
q16	452	380	386	380
q17	1002	605	565	565
q18	7264	7018	6911	6911
q19	1515	1345	1375	1345
q20	546	322	297	297
q21	3097	2650	2670	2650
q22	347	276	286	276
Total cold run time: 44363 ms
Total hot run time: 39955 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4356	4358	4325	4325
q2	266	164	179	164
q3	3540	3514	3524	3514
q4	2387	2381	2384	2381
q5	5756	5748	5784	5748
q6	241	120	122	120
q7	2391	1869	1873	1869
q8	3531	3531	3543	3531
q9	9033	9039	9041	9039
q10	3914	3994	4029	3994
q11	492	359	363	359
q12	769	597	594	594
q13	4299	3545	3529	3529
q14	289	253	263	253
q15	575	517	527	517
q16	509	452	466	452
q17	1880	1839	1840	1839
q18	8710	8221	8297	8221
q19	1745	1754	1757	1754
q20	2273	1935	1938	1935
q21	6529	6217	6187	6187
q22	505	420	428	420
Total cold run time: 63990 ms
Total hot run time: 60745 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.02 seconds
stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17183563406 Bytes

Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 21, 2023
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@amorynan
Copy link
Contributor Author

run external

@morrySnow morrySnow merged commit e984806 into apache:master Dec 21, 2023
27 of 29 checks passed
CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Dec 27, 2023
* [Performance](point query)Opimize partition prune for point query (apache#28150)

* [Performance](point query)Opimize partition prune for point query

* [fix](stacktrace) ignore stacktrace for error code INVALID_ARGUMENT INVERTED_INDEX_NOT_IMPLEMENTED (apache#27898) (apache#28598)

* ignore stacktrace for error INVALID_ARGUMENT INVERTED_INDEX_NOT_IMPLEMENTED

* AndBlockColumnPredicate::evaluate

* [bugfix](topn) fix coredump in copy_column_data_to_block when nullable mismatch (apache#28597)

* [bugfix](topn) fix coredump in copy_column_data_to_block when nullable mismatch

return RuntimeError if copy_column_data_to_block nullable mismatch to avoid coredump in input_col_ptr->filter_by_selector(sel_rowid_idx, select_size, raw_res_ptr) .

The problem is reported by a doris user but I can not reproduce it, so there is no testcase added currently.

* clang format

* [feature](Nereids): eliminate semi join (apache#28588)

Eliminate Semi/Anti Join which is FALSE or TRUE.

* [feature](Nereids) support datev1 and datetimev1 (apache#28581)

* [improvement](http) add show_table_data http api (apache#28380)

In some cases, users need to get the data size of single replica of a table, and evaluate certain actions based on this, such as estimating the precise backup size.

Signed-off-by: nextdreamblue <zxw520blue1@163.com>

* [Enhance](regression)enhance jdbc case to adapt to use case concurrency (apache#28565)

enhance jdbc case to adapt to use case concurrency

* [opt](Nereids)when both Nereids and old parsers report errors, prompt error messages for the Nereids (apache#28580)

* [Improvement](regression) change compound predicate regression case name to make it more clear (apache#28612)

* [regression](memtable) add case for memtable flush error handle (apache#28285)

Co-authored-by: ziyang zhang <zhangziyang@stu.cdut.edu.cn>

* [regression-test](memtable)  test memtable flush is high priority for vtable writerV1 (apache#28502)

* enhance performance for broken tablet checking under multi-core scenario with a coarse-grained read lock (apache#28552)

* [fix](load) fix memtracking orphan too large (apache#28600)

* [fix](memtable-limiter) do not block write if load mem usage is low (apache#28602)


Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>

* [refactor](renamefile) rename some files according to the class names (apache#28606)

* [feature](mtmv)create mtmv support refresh_partition_num (apache#28566)

- create/alter mtmv support refresh_partition_num
- mtmv task according to refresh_partition_num executes refresh tasks in batches
- `tasks` tvf add column `CompletedPartitions` and `progress`
- fix mtmv can not `show temp partition` and `drop temp partition`
- fix task can not get error msg when insert overwrite error
- fix when the partition field is capitalized, the verification of creating a mtmv does not pass

* [fix](Nereids) stats estimation of lessThan apache#28444

* [fix](regression)Change analyze_timeout to global.  (apache#28587)

Fix hive statistics regression case. analyze_timeout is a global session variable.

* [chore] Add bvar for meta operations of BE (apache#28374)

* [Bug](cooldown) Fix problem that followers may never completely cooldown (apache#28561)

* [Improve](tvf)jni-avro support split file (apache#27933)

* [feature](mtmv)after creating a materialized view, if other operations fail, roll back  (apache#28621)

after create mtmv, if create job failed,need drop mtmv

* [fix](meta) update killed query state (#) (apache#25917)

* [refactor](profile&names) using dst_id in pipelinex profile to be same as non pipeline; rename some function names (apache#28626)

Co-authored-by: yiguolei <yiguolei@gmail.com>

* [feature][executor]support workload schedule policy (apache#28443)

* [feature](expr) Support kill query by query_id (apache#28530)

Issue Number: open apache#28517

* [chore](user) Add user property `parallel_fragment_exec_instance_num` (apache#28447)

* [feature](inverted index) add ignore_above property to prevent long s… (apache#28585)

When string is too long, clucene will throw an error. 
And the string is too long to analyze. So we ignore the string in index process when the string is longer than 256 bytes by default.
We add an poperty `ignore_above` for user to customize.

* [Enhance](broker) add inputstream expire scheduled checker to avoid memory leak for broker scan (apache#28589)

This pr introduces 2 broker conf:

1. enable_input_stream_expire_check: which indicates whether enable inputStream expire check.
2. input_stream_expire_seconds: which indicates the timeout seconds for inputStream since last update.

* [fix](hive) add support for `quoteChar` and `seperatorChar` for hive (apache#28613)

add support for quoteChar and seperatorChar .

* [Fix](transactional-hive) Fix hive transactional table return empty result. (apache#28518)

* [Fix](memtable) fix `shrink_memtable_by_agg` without duplicated keys (apache#28660)

remove duplicated logic:
```
vectorized::Block in_block = _input_mutable_block.to_block();
_put_into_output(in_block);
```
`_input_mutable_block.to_block()` will move `_input_mutable_block`, and lead to `flush` with empty block

* [enhance](partition_id) check partition id before store meta (apache#28055)

* [opt](task-assignment) use consistent hash as default task assigner and cache the consistent hash ring (apache#28522)

1. Use consistent hash algo as the default assigner for file query scan node
    A consistent assignment can better utilize the page cache of BE node.

2. Cache the consistent hash ring
    Init a consistent hash ring is time-consuming because there a thousands of virtual node need to be added.
    So cache it for better performance

* [Fix](Job)Fixed job scheduling missing certain time window schedules (apache#28659)

Since scheduling itself consumes a certain amount of time, the start time of the time window should not be the current time, but the end time of the last schedule.

* [fix](test) fix ccr test cases (apache#28664)

* [fix](regression) fix test_set_replica_status due to force_olap_table_replication_num=3 (apache#28573)

* [fix](regression) restore reserve num replicas (apache#28541)

* [fix](regression) fix test_alter_colocate_table due to force_olap_table_replication_num=3 (apache#28575)

* [improvement](transaction) reduce publish txn log (apache#28277)

* [fix](stream_load)fix bug for stream (apache#27752)

1. forbid thed  stream_load without content-length or chunked Transfer Encoding
2. forbid thed  stream_load both with content-length and chunked Transfer Encoding

Co-authored-by: xingying01 <xingying01@corp.netease.com>

* [Revert](partial update) Revert "Fix missing rowsets during doing alignment when flushing memtable due to compaction (apache#28062)" (apache#28674)

This reverts commit 485d7db.

* [feat](Nereids) support outer join and aggregate bitmap rewrite by mv (apache#28596)

- Support left outer join rewrite by materialized view
- Support bitmap_union roll up to imp count(distinct)
- Support partition materialized view rewrite

* [refactor](runtimefilter) do not use QueryContext in runtime filter (apache#28559)

* [fix](planner)should bind expr using no slot to correct tuple (apache#28656)

* [fix](planner) ctas update datev1 to datev2 should use equals (apache#28641)

* [fix](Nereids) should only do bind relation in view analyzer (apache#28637)

* [code](pipelineX) refine some pipelineX code  (apache#28570)

* [fix](partial update) report error directly if missing rowsets during doing alignment when flushing memtable due to compaction  (apache#28677)

* [opt](inverted index) Add null document interface to optimize empty string indexing (apache#28661)

* [feature](invert index) match_regexp feature added (apache#28257)

* [Feature](datatype) update be ut codes and fix bugs for IPv4/v6 (apache#28670)

* [test](Nereids): remove shape plan project and distribute in eager test (apache#28701)

* [fix](statistics)Fix drop stats fail silently bug. (apache#28635)

Drop stats use IN predicate to filter the column stats to delete. The default length of IN predicate is 1024, drop table stats with more than 1024 columns may fail.
This pr is to split the delete sql based on the IN predicate length.

* [fix](mtmv) fix insert overwrite getExecTimeout error (apache#28700)

should use InsertTimeout but QueryTimeout

* [Enhancement](auto-partition) change the behaviour when insert overwrite an auto partition table apache#28683

If we specific target partition(s) when inserting overwrite an auto partition table,
before:
could create new partition
now:
behalf just like non-auto partition table

* [FIX](regresstest) fix case with double value apache#28668

double value in case has 27 figures after the point, which will make not standable output

* [test](Nereids): add test for scalar agg (apache#28712)

* [fix](meta_scanner) fix meta_scanner process ColumnNullable (apache#28711)

* [fix](nereids) Fix data wrong using mv rewrite and ignore case when getting mv related partition table (apache#28699)

1. Fix data wrong using mv rewrite
2. Ignore case when getting mv related partition table
3. Enable infer expression column name without alias when create mv

* [test](regression-test) order by decs should only make effect on its nearest column apache#28728

* [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline (apache#28488)

* [fix](Nereids) delete partition failed (apache#28717)

1. parser's partitionSpec changed unexpectly by PR apache#26492
2. delete without using should support un-equals expression

* [fix](tvf)Fixed the avro-scanner projection pushdown failing to query on multiple BEs (apache#28709)

* [fix](block) fix nullptr in MutableBlock::allocated_bytes (apache#28738)

* [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE (apache#28723)

when replay addTaskResult log,will create one ConnectContext,and set Env.getCurrentEnv,then store this ctx in ConnectContext.threadLocalInfo,threadLocalInfo is static,so this ctx can not be recycling,Env of replay thread also can not be recycling

* [Fix](json reader) fix json reader crash due to `fmt::format_to` (apache#28737)

```
4# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75
5# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
6# 0x00005622F33D22B1 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
7# 0x00005622F33D2404 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
8# fmt::v7::detail::error_handler::on_error(char const*) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
9# char const* fmt::v7::detail::parse_replacement_field<char, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char> >&>(char const*, char const*, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char> >&) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
10# void fmt::v7::detail::vformat_to<char>(fmt::v7::detail::buffer<char>&, fmt::v7::basic_string_view<char>, fmt::v7::basic_format_args<fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<fmt::v7::type_identity<char>::type>, fmt::v7::type_identity<char>::type> >, fmt::v7::detail::locale_ref) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
11# doris::vectorized::NewJsonReader::_append_error_msg(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:924
12# doris::vectorized::NewJsonReader::_set_column_value
```

* [bug](coredump) Fix coredump in aggregation node's destruction(apache#28684)

fix coredump in aggregation node's destruction

* [pipelineX](improvement) Support global runtime filter (apache#28692)

* [FIX](explode)fix explode array decimal (apache#28744)

* fix explode with array<decimal> has specific precision at old planner

* [refactor](broadcastbuffer) using a queue to remove ref and unref codes (apache#28698)

Co-authored-by: yiguolei <yiguolei@gmail.com>Add a new class broadcastbufferholderqueue to manage holders
Using shared ptr to manage holders, not use ref and unref, it is too difficult to maintain.

* [enhance](blacklist) seperate blacklist conf from heartbeat (apache#28638)

There is a circuit breaker lasting for 2 minutes in grpc, then if a be is down and up again, send fragments to the be fails lasting for 2 minutes.

* [fix](stmt):fix CreateTableStmt toSql placed comment in wrong place (apache#27504)

Issue Number: close apache#27474
Co-authored-by: tongyang.han <tongyang.han@jiduauto.com>

* [Improve](compile) add `__AVX2__` macro for JsonbParser (apache#28754)

* [Improve](compile) add `__AVX2__` macro for JsonbParser

* throw exception instead of CHECK

* [Feature](materialized-view) support match logicalAggregate(logicalProject(logicalFilter(logicalOlapScan())) without agg (apache#28747)

support match logicalAggregate(logicalProject(logicalFilter(logicalOlapScan())) without agg

* [Fix] (schema change) forbid adding time type column (apache#28751)

* [fix](group_commit) fix group commit cancel stuck (apache#28749)

* [fix](scanner) fix concurrency bugs when scanner is stopped or finished (apache#28650)

`ScannerContext` will schedule scanners even after stopped, and confused with `_is_finished` and `_should_stop`.
 Only Fix the concurrency bugs when scanner is stopped or finished reported in apache#28384

* [fix](regression) fix stream load properties case fail (apache#28680)

* [fix](fe ut) fix PropertyConverterTest (apache#28722)

* [improve](rpc) Log channel state before shutdown backend service client (apache#28667)

* [fix](publish version) fix publish fail but return ok (apache#28425)

* [test](Nereids): remove shape plan project and distribute in eager test (apache#28724)

* [refactor](nereids) make NormalizeAggregate rule more clear and readable (apache#28607)

* [FIX](type) fix matchExactType for complex type (apache#28233)

fe matchExactType function should call type.matchTypes for its own logic, do not switch case to do special logic otherwise we may meet core in be like this.
 ```
F20231208 18:54:39.359673 680131 block.h:535] Check failed: _data_types[i]->is_nullable()  target type: Struct(l_info:Nullable(Array(Nullable(String)))) src type: Struct(col:Nullable(Array(Nullable(UInt8))))
*** Check failure stack trace: ***
    @     0x5584e952b926  google::LogMessage::SendToLog()
    @     0x5584e9527ef0  google::LogMessage::Flush()
    @     0x5584e952c169  google::LogMessageFatal::~LogMessageFatal()
    @     0x5584cf17201e  doris::vectorized::MutableBlock::merge_impl<>()
    @     0x5584ceac4b1d  doris::vectorized::MutableBlock::merge<>()
    @     0x5584d4dd7de3  doris::vectorized::VUnionNode::get_next_const()
    @     0x5584d4dd9a45  doris::vectorized::VUnionNode::get_next()
    @     0x5584bce469bd  std::__invoke_impl<>()
    @     0x5584bce466d0  std::__invoke<>()
    @     0x5584bce465c7  _ZNSt5_BindIFMN5doris8ExecNodeEFNS0_6StatusEPNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS1_St12_PlaceholderILi1EESC_ILi2EESC_ILi3EEEE6__callIS2_JOS4_OS7_OS8_EJLm0ELm1ELm2ELm3EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
    @     0x5584bce46358  std::_Bind<>::operator()<>()
    @     0x5584bce46208  std::__invoke_impl<>()
    @     0x5584bce46178  _ZSt10__invoke_rIN5doris6StatusERSt5_BindIFMNS0_8ExecNodeEFS1_PNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS3_St12_PlaceholderILi1EESD_ILi2EESD_ILi3EEEEJS5_S8_S9_EENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESL_E4typeEOSM_DpOSN_
    @     0x5584bce45c18  std::_Function_handler<>::_M_invoke()
    @     0x5584bce6412f  std::function<>::operator()()
    @     0x5584bce56382  doris::ExecNode::get_next_after_projects()
    @     0x5584bce26218  doris::PlanFragmentExecutor::get_vectorized_internal()
    @     0x5584bce2431b  doris::PlanFragmentExecutor::open_vectorized_internal()
    @     0x5584bce22a96  doris::PlanFragmentExecutor::open()
    @     0x5584bce27c9d  doris::PlanFragmentExecutor::execute()
    @     0x5584bcbdb3f8  doris::FragmentMgr::_exec_actual()
    @     0x5584bcbf982f  doris::FragmentMgr::exec_plan_fragment()::$_0::operator()()
    @     0x5584bcbf9715  std::__invoke_impl<>()
    @     0x5584bcbf96b5  _ZSt10__invoke_rIvRZN5doris11FragmentMgr18exec_plan_fragmentERKNS0_23TExecPlanFragmentParamsERKSt8functionIFvPNS0_12RuntimeStateEPNS0_6StatusEEEE3$_0JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESH_E4typeEOSI_DpOSJ_
    @     0x5584bcbf942d  std::_Function_handler<>::_M_invoke()
    @     0x5584b9dfd883  std::function<>::operator()()
    @     0x5584bd6e3929  doris::FunctionRunnable::run()
    @     0x5584bd6cf8ce  doris::ThreadPool::dispatch_thread()
```

* [test](Nereids): add assert debug log for TopnToMaxTest (apache#28755)

* [improve](move-memtable) limit task num in load stream flush token (apache#28748)

* [feature](Nereids) elimite inner join by foreign key (apache#28486)

* [enhancement](backup/restore) support alter s3 repo info about ak/sk/token (apache#27027)

in some cases:

s3.session_token/AWS_TOKEN will be expired after hours, and may be upload snapshot job will failed if data is big large;
as same reason, repo will be expired too when RepositoryMgr exec repo ping
so it need support alter s3 repo properties about ak/sk/token and update upload snapshot job properties to continue backup.

Signed-off-by: nextdreamblue <zxw520blue1@163.com>

* [improve](transaction) extend abort transaction time (apache#28662)

* [regression](p2) fix test cases result (apache#28768)

regression-test/data/external_table_p2/hive/test_hive_hudi.out
regression-test/data/external_table_p2/hive/test_hive_to_array.out
regression-test/suites/external_table_p2/tvf/test_local_tvf_compression.groovy
regression-test/suites/external_table_p2/tvf/test_path_partition_keys.groovy
regression-test/data/external_table_p2/hive/test_hive_text_complex_type.out

* (fix)[meta][export] fix replay export NPE issue (apache#28752)

The ConnectionContext does not exist in replay thread

* [Bug fix][metrics] correct fe collector type for jvm_gc (apache#28784)

Co-authored-by: 胥剑旭 <xujianxu@xujianxudeMacBook-Pro.local>

* [Chore](decimal) set check_overflow_for_decimal to true when alter table(apache#28777)

set check_overflow_for_decimal to true when alter table

* [pipelineX](fix) Fix TPCH Q2 (apache#28783)

* (topN)runtime_predicate is only triggered when the column name is obtained (apache#28419)

Issue Number: close apache#27485

* [fix](planner)fix bug of bound conjunct to wrong tuple (apache#28811)

this fix bug introduced by apache#28656

* [bug](pipelineX) Fix pipelineX bug on multiple BE (apache#28792)

* [opt](inverted index) ignore_above only affects untokenized strings (apache#28819)

* [improve](load) reduce lock scope in MemTableWriter active consumption (apache#28790)

* [opt](Nereids) improve Shape check cases (apache#28124)

* tmplate update
* log tpcds stats when check shape

* [fix](group commit)Fix the issue of duplicate addition of wal path when encouter exception (apache#28691)

* [fix](compaction) compaction should catch exception when vertical block reader read next block (apache#28625)

* [fix](Nereids) join order is not right after sql parsing (apache#28721)

for sql
```
t1, t2 join t3
```

we should generate plan like:
```
t1 join (t2 join t3)
```

but we generate:
```
(t1 join t2) join t3
```
to follow legancy planner.

* [fix](metrics) fix bvar memtable_input_block_allocated_size (apache#28725)

* [enhancement](audit-log) add status code and error msg to audit log for proxy stmt (apache#27607)

* [improvement](external catalog)Change log level to debug for getRowCountFromExternalSource. (apache#28801)

* [fix](partial update) only report error when in strict mode partial update when finding missing rowsets during flushing memtable (apache#28764)

related pr: apache#28062, apache#28674, apache#28677
fix apache#28677

* [fix](function) Fix from_second functions overflow and wrong result (apache#28685)

* [Enhancement](load) Limit the number of incorrect data drops and add documents (apache#27727)

In the load process, if there are problems with the original data, we will store the error data in an error_log file on the disk for subsequent debugging. However, if there are many error data, it will occupy a lot of disk space. Now we want to limit the number of error data that is saved to the disk.

Be familiar with the usage of doris' import function and internal implementation process
Add a new be configuration item load_error_log_limit_bytes = default value 200MB
Use the newly added threshold to limit the amount of data that RuntimeState::append_error_msg_to_file writes to disk
Write regression cases for testing and verification

Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com>

* [test](partial update) add complex type regression cases for partial update (apache#28758)

NOTE: There's some issue for MAP type to work with row store, so in this PR we don't have cases for MAP type
Will add the support for MAP type in future.

* [fix](test)fix test_create_table test case for nereids (apache#28693)

* [improve](config) set mutable and masterOnly in FE config stream_load_default_memtable_on_sink_node (apache#28835)

* [fix](group-commit) check if wal need recovery is abnormal (apache#28769)

* [Fix](Variant) fix variant predicate rewrite OrToIn with wrong plan (apache#28695)

using the name without paths info will lead to wrong In plan, e.g.
```
where cast(v:a as text) = 'hello' or cast(v:b as text) = 'world'
```
will be rewrite to:
```
where cast(v as text) in ('hello', 'world')
``
This is wrong, because they are different slots

* [refactor](buffer) remove download buffer since it is no longer useful (apache#28832)

remove download buffer since it is no longer useful

* [Feature](Variant) Implement variant new sub column access method (apache#28484)

* [Feature](Variant) Implement variant new sub column access method

The query SELECT v["a"]["b"] from simple_var WHERE cast(v["a"]["b"] as int) = 1 encompasses three primary testing scenarios:

```
1. A basic test involving the variant data type.
2. A scenario dealing with GitHub event data in the context of a variant.
3. A case related to the TPC-H benchmark using a variant.
```

* [fix](memory) Add thread asynchronous purge jemalloc dirty pages (apache#28655)

jemallctl purge all arena dirty pages may take several seconds, which will block memory GC and cause OOM.
So purge asynchronously in a thread.

* [fix](regression) fix regression error of test_compress_type (apache#28826)

* [improvement](executor) Add tvf and regression test for Workload Scheduler (apache#28733)

1 Add select workload schedule policy tvf
2 Add reg test

* [chore](error msg) print type info when colocate with ddl failed due to type mismatch (apache#28773)

* [opt](query cancel) optimization for query cancel apache#28778

* [Fix](multi-catalog) skip hms events if hms table is not supported. (apache#28644)

Co-authored-by: wangxiangyu <wangxiangyu@360shuke.com>

* [Enhancement](job) No need to query some backends which are not alive. (apache#28608)

No need to execute some jobs if backend is not alive

* [fix](paimon)fix type convert for paimon (apache#28774)

fix type convert for paimon

* [fix](stream load)add test case and doc for arrow type of stream load (apache#28098)

add test case and doc for arrow type of stream load

* [feature](mtmv)mtmv partition refresh case (apache#28787)

* (enhance)(InternalQuery) Support to collect profile for intenal query (apache#28762)

* [optimize](count) optimize pk exact query without reading data (apache#28494)

* [opt](sessionVar)show changed sessoin var first apache#28840

“show variables” command list changed vars before not changed vars,

* check  stats and log memo for ds46 (apache#28396)

* [bug](sharedscan) Fix shared scan bug (apache#28841)

* [fix](hash join) fix stack overflow caused by evaluate case expr on huge build block (apache#28851)

* [refactor](executor)remove scan group apache#28847

* [pipelineX](refactor) rename functions (apache#28846)

* [Feature](inverted index) add lowercase option for inverted index analyzer (apache#28704)

* [feature](Nereids) Pull up join from union all (apache#28682)

* [fix](ci) tpch pipeline should not re-load data (apache#28874)

* [fix](ci) tpch pipeline should not re-load data

* 2

---------

Co-authored-by: stephen <hello-stephen@qq.com>

* [exec](compress) use FragmentTransmissionCompressionCodec control the exchange compress behavior (apache#28818)

* [improve](move-memtable) tweak load stream flush token num and max tasks (apache#28884)

* [improve](load) remove extra layer of heavy work pool in tablet_writer_add_block (apache#28550)

* [improve](load) limit delta writer flush task parallelism (apache#28883)

* [fix](multi-catalog)filter impala generated path (apache#28786)

file impala generated dir _imapala_insert_staging

* [enhancement](udf) add prepare function for java-udf (apache#28750)

* [refactor](test)Refactor workload group/schedule policy test apache#28888

[refactor](test)Refactor workload group/schedule policy test
apache#28888

* [improve](move-memtable) avoid using heavy work pool during append data (apache#28745)

* [enhancement](broker-load) fix-move-memtable-session-var-for-s3 (apache#28894)

* [exec](load) change default parallel num from 1 to 8 in no pipeline exec engine (apache#28864)

* [fix](segcompaction) disable segcompaction by default (apache#28906)

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

* [fix](pipelineX) fix cannot runtime obtain profile on pipelineX apache#28795

* [fix](mtmv) fix failed to specify the number of buckets when bucket auto (apache#28854)

Issue Number: close #xxx

- fix failed to specify the number of buckets when bucket auto
- delete unused SessionVariable
- if mtmv used external table ,check `isMaterializedViewRewriteEnableContainForeignTable`

* [bugfix](scannercore) scanner will core in deconstructor during collect profile (apache#28727)

* [FIX](map)fix map with rowstore table (apache#28877)

* [feature](mtmv)MTMV pause and resume (apache#28887)

- PAUSE MATERIALIZED VIEW JOB ON mv1
- RESUME MATERIALIZED VIEW JOB ON mv1
- fix when drop db,not drop job
- add lock for one materialized view can only run one task at a time

* [feature](mtmv)add more test case1 (apache#28910)

* [fix](block) add block columns size dcheck (apache#28539)

* [fix](chore) update dcheck to avoid core during stress test (apache#28895)

* [fix](doc) Add the usage example of bos to the documentation of s3 tvf (apache#28899)

* [fix](mtmv)fix can not create mtmv all use default value (apache#28922)

* [fix](parquet) the end offset of column chunk may be wrong in parquet metadata (apache#28891)

* [fix](paimon)fix `like` predicate (apache#28803)

fix like predict

* [fix](merge-on-write) migration may cause duplicate keys for mow table (apache#28923)

* [fix](mtmv) Related partition exclude null generate column when increment build materialized view (apache#28855)

Infer partition column by materialized view partition column, exclude null generate column in join when increment build materialized view

* [nereids] fix join fd computing bug (apache#28849)

* [Fix](statistics) Fix partition name NPE and sample for all table during auto analyze (apache#28916)

Fix partition name NPE and sample for all table during auto analyze.
Sample for all tables because getData may have latency, which may cause full analyze a huge table and use too much resource. Sample for all tables to avoid this. Will improve the strategy later.

* [fix](nereids) Fix query mv rewrite fail when mv cache build quickly (apache#28876)

* [optimize](zonemap) skip zonemap if predicate does not support_zonemap (apache#28595)

* [optimize](zonemap) skip zonemap if predicate does not support_zonemap apache#27608 (apache#28506)

* Revert "[bugfix](scannercore) scanner will core in deconstructor during collect profile (apache#28727)" (apache#28931)

This reverts commit 4066de3.

* [fix] (nereids) Catch exception when mv fail and fix the npe (apache#28932)

* [fix](regression-test) test_partial_update_native_insert_stmt_complex is flaky (apache#28927)

* [improvement](group commit) make get column function more reliable when replaying wal (apache#28900)

* [opt](nereids) convert or to inpredicate and optimize inpredicate partition prune (apache#28316)

* [enhancement](bulk-load) cancel loading tasks directly without retrying when timeout exceeded (apache#28666)

* [chore](test) Add testing util sync point (apache#28924)

* [chore](prompt) Optimize tablet and replica prompt by pointing out what the numbers mean (apache#28925)

* [fix](join) incorrect result of left semi/anti join with empty build side (apache#28898)

* [docs] (DebugPoints) Update docs about Debug Points (apache#28347)


---------

Co-authored-by: qinhao <qinhao@newland.com.cn>

* [feature](api) add profile text api (apache#28697)

* [opt](invert index) Empty strings are not written to the index in the case of TOKENIZED (apache#28822)

* [feature](pipelineX) control exchange sink by memory usage (apache#28814)

* [fix](pipelineX) fix use global rf when there no shared_scans (apache#28869)

* [refactor](pipelineX)do not override dependency() function in pipelineX (apache#28848)

* [fix](log) regularise some BE error type and fix a load task check apache#28729

* [chore](test) correct create table statement (apache#28863)

* [fix](nereids)subquery unnest need handle subquery in Not expr correnctly (apache#28713)

* [feature](nereids)support decimalv2 (apache#28726)

* [fix](nereids)group by expr may be bound twice in bind agg slot (apache#28771)

* [doc](insert) Add group commit docs (apache#25949)

* [performance](variant) support topn 2phase read for variant column (apache#28318)

 [performance](variant) support topn 2phase read for variant column

* return residual expr of join (apache#28760)

* [fix](load) fix nullptr when getting memtable flush running count (apache#28942)

* [fix](load) fix nullptr when getting memtable flush running count

* style

* (enhance)(regression) Support `force_olap_table_replication_num=3` run test_insert_random_distribution_table case (apache#28903)

* [Test](Job)Add test case (apache#28481)

* [fix](doc) typo fix in auto-partition page (apache#28512)

* [feature](load) enable memtable on sink node by default (apache#28963)

* [Chore](Job)print log before task execute (apache#28962)

* [fix](hdfs) Fix HdfsFileSystem::exists_impl crash (apache#28952)

Calling hdfsGetLastExceptionRootCause without initializing ThreadLocalState
will crash. This PR modifies the condition for determining the existence of
a hdfs file, because hdfsExists will set errno to ENOENT when the file does
not exist, we can use this condition to check whether a file existence rather
than check the existence of the root cause.

* [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block (apache#27943)

* [fix](doc) typo fix in dynamic-partition page (apache#28511)

* [fix](hash join) fix column ref DCHECK failure of hash join node block mem reuse (apache#28991)

Introduced by apache#28851, after evaluating build side expr, some columns in resulting block may be referenced more than once in the same block.

e.g. coalesce(col_a, 'string') if col_a is nullable but actually contains no null values, in this case funcition coalesce will insert a new nullable column which references the original col_a.

* [opt](assert_num_rows) support filter in AssertNumRows operator and fix some explain (apache#28935)

* NEED

* Update pipeline x

* fix pipelinex compile

* [fix](parquet_reader) misused bool pointer (apache#28986)

Signed-off-by: pengyu <pengyu@selectdb.com>

* [doc](multi-catalog)add krb and some ertificates FAQs (apache#28858)

add some security docs

* [fix](planner)should save original select list item before analyze (apache#28187)

* [fix](planner)should save original select list item before analyze

* fix test case

* fix failed case

* [bug](storage) Fix gc rowset bug (apache#28979)

* [fix](mtmv)add log for resolve pending task (apache#28999)

* add lock for resolve pending task

* add lock for resolve pending task

* [improvement](nereids) Get partition related table disable nullable field and complete agg matched pattern mv rules. (apache#28973)

* [improvement] (nereids) Get partition related table disable nullable field and modify regression test, complete agg mv rules.

* make filed not null to create partition mv

* [chore](config) modify `tablet_schema_cache_recycle_interval` from 24h to 1h (apache#28980)

To prevent from too many tablet schema cache in memory and lead to performance issue when hold lock
to erase item

* [opt](scanner) optimize the number of threads of scanners (apache#28640)

1. Remove `doris_max_remote_scanner_thread_pool_thread_num`, use `doris_scanner_thread_pool_thread_num` only.
2. Set the default value `doris_scanner_thread_pool_thread_num` as `std::max(48, CpuInfo::num_cores() * 4)`

* [chore](config) modify `variant_ratio_of_defaults_as_sparse_column` from 0.95 to 1 (apache#28984)

since sparse column is not stable at present

* [feature](nereids)support partition property in nereids (apache#28982)

* [feature](Nereids) support table valued function http_stream (apache#29004)

* [feature](Load)(step2)support nereids load job schedule (apache#26356)

We will  Integrate new load job manager into  new job scheduling framework so that the insert into task can be scheduled after the broker load sql  is converted to insert into TVF(table value function) sql.

issue: apache#24221

Now support:
1. load data by tvf insert into sql, but just for simple load(columns need to be defined in the table)
2. show load stmt
- job id, label name, job state, time info
- simple progress
3. cancel load from db
4. support that enable new load through Config.enable_nereids_load
5. can replay job after restarting doris

TODO:
- support partition insert job
- support show statistics from BE
- support multiple task and collect task statistic
- support transactional task
- need add ut case

* [enhance](tablet) Reduce log in tablet meta (apache#28719)

* [fix](case) Add sync to test case (apache#29034)

* [opt](compound) Optimize by deleting the compound expr after obtaining the final result (apache#28934)

* [hotfix](jdbc catalog) fix load table and column names npe (apache#28865)

This fix is for the npe situation that occurs when FE of non-Master nodes initializes Jdbc Catalog metadata.

* [opt](Nereids) support cast bewteen numeric and  boolean in FE (apache#29006)

* [improvement](jdbc catalog) Optimize connection pool caching logic (apache#28859)

In the old caching logic, we only used jdbcurl, user, and password as cache keys. This may cause the old link to be still used when replacing the jar package, so we should concatenate all the parameters required for the connection pool as the key.

* [fix](Nereids) runtime filter push down failed (apache#28997)

project child not always NamedExpression

failed msg
```
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = class org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral cannot be cast to class org.apache.doris.nereids.trees.expressions.NamedExpression (org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral and org.apache.doris.nereids.trees.expressions.NamedExpression are in unnamed module of loader 'app')
    at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:623) ~[classes/:?]
    at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:478) ~[classes/:?]
    at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:457) ~[classes/:?]
    at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:245) ~[classes/:?]
    at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:166) ~[classes/:?]
    at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:193) ~[classes/:?]
    at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:246) ~[classes/:?]
    at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[classes/:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.lang.ClassCastException: class org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral cannot be cast to class org.apache.doris.nereids.trees.expressions.NamedExpression (org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral and org.apache.doris.nereids.trees.expressions.NamedExpression are in unnamed module of loader 'app')
    at org.apache.doris.nereids.trees.plans.physical.PhysicalSetOperation.pushDownRuntimeFilter(PhysicalSetOperation.java:178) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.physical.PhysicalHashJoin.pushDownRuntimeFilter(PhysicalHashJoin.java:229) ~[classes/:?]
    at org.apache.doris.nereids.processor.post.RuntimeFilterGenerator.pushDownRuntimeFilterCommon(RuntimeFilterGenerator.java:386) ~[classes/:?]
```

* [fix](Nereids) generating function should not folding to NullLiteral (apache#29003)

should not fold table generating function to null when do constant folding.
we should remove Generate node and replaced it by project later.

* [improve](load) add profile for WaitFlushLimitTime (apache#29013)

* [opt](Nereids) let inverted index work with top opt (apache#29000)

* [feature](Nereids) support values inline table in query (apache#28972)

* [enhencement](config) change default memtable size & loadStreamPerNode & default load parallelism (apache#28977)

We change memtable size from 200MB to 100MB to achieve smoother flush
performance. We change loadStreamPerNode from 20 to 60 to avoid stream
rpc to be the bottleneck when enable memtable_on_sink_node. We change
default s3&broker load parallelsim to make the most of CPUs on moderm
multi-core systems.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

* [improve](move-memtable) increase load_stream_flush_token_max_tasks (apache#29011)

* [feature](scan) Implement parallel scanning by dividing the tablets based on the row range (apache#28967)

* [feature](scan) parallel scann on dup/mow mode

* fix bugs

* [refactor](create tablet) default create tablet round robin  (apache#28911)

* [Bug] optimize the collection import Lists dependency packge apache#28523 (apache#28579)

* [fix](regression) Fix flaky test test_partial_update_2pc_schema_change (apache#29037)

* [Bug](security) BE download_files function exists log print sensitive msg apache#28592 (apache#28594)

* Revert "[feature](load) enable memtable on sink node by default (apache#28963)" (apache#29090)

This reverts commit 17917a0.

* [improvement](statistics)Remove retry load when load stats cache fail (apache#28904)

Remove retry load when load stats cache fail. This case usually happens when BE is down or BE OOM, retry doesn't work in these cases and may increase BE work load.

* [fix](pipeline) sort_merge should throw exception in has_next_block if got failed status (apache#29076)

Test in regression-test/suites/datatype_p0/decimalv3/test_decimalv3_overflow.groovy::249 sometimes failed when there are multiple BEs and FE process report status slowly for some reason.

explain select k1, k2, k1 * k2 from test_decimal128_overflow2 order by 1,2,3
--------------

+----------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                            |
+----------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                                                                            |
|   OUTPUT EXPRS:                                                                                                            |
|     k1[#5]                                                                                                                 |
|     k2[#6]                                                                                                                 |
|     (k1 * k2)[#7]                                                                                                          |
|   PARTITION: UNPARTITIONED                                                                                                 |
|                                                                                                                            |
|   HAS_COLO_PLAN_NODE: false                                                                                                |
|                                                                                                                            |
|   VRESULT SINK                                                                                                             |
|      MYSQL_PROTOCAL                                                                                                        |
|                                                                                                                            |
|   111:VMERGING-EXCHANGE                                                                                                    |
|      offset: 0                                                                                                             |
|                                                                                                                            |
| PLAN FRAGMENT 1                                                                                                            |
|                                                                                                                            |
|   PARTITION: HASH_PARTITIONED: k1[#0], k2[#1]                                                                              |
|                                                                                                                            |
|   HAS_COLO_PLAN_NODE: false                                                                                                |
|                                                                                                                            |
|   STREAM DATA SINK                                                                                                         |
|     EXCHANGE ID: 111                                                                                                       |
|     UNPARTITIONED                                                                                                          |
|                                                                                                                            |
|   108:VSORT                                                                                                                |
|   |  order by: k1[#5] ASC, k2[#6] ASC, (k1 * k2)[#7] ASC                                                                   |
|   |  offset: 0                                                                                                             |
|   |                                                                                                                        |
|   102:VOlapScanNode                                                                                                        |
|      TABLE: regression_test_datatype_p0_decimalv3.test_decimal128_overflow2(test_decimal128_overflow2), PREAGGREGATION: ON |
|      partitions=1/1 (test_decimal128_overflow2), tablets=8/8, tabletList=22841,22843,22845 ...                             |
|      cardinality=6, avgRowSize=0.0, numNodes=1                                                                             |
|      pushAggOp=NONE                                                                                                        |
|      projections: k1[#0], k2[#1], (k1[#0] * k2[#1])                                                                        |
|      project output tuple id: 1                                                                                            |
+----------------------------------------------------------------------------------------------------------------------------+
36 rows in set (0.03 sec)
Why failed:

Multiple BEs
Fragments 0 and 1 are MUST on different BEs
Pipeline task of VOlapScanNode which executes k1*k2 failed sets query status to cancelled
Pipeline task of VSort call try close, send Cancelled status to VMergeExchange
sort_curso did not throw exception when it meets error

* [feature](Nereids): support infer join when comapring mv (apache#28988)

* [fix](nereids) "not is null" stats estimation fix (apache#28860)

* fix not is null stats

* [opt](nereids)expr normalize after filter pushdown (apache#28743)

normalize expression after filter push down

* [Enhancement](Wal)Support dynamic wal space limit (apache#27726)

* [fix](doc) spell errors fixes query profile docs (apache#28437)

* [opt](Nerieds) add infer props to expression (apache#28953)

* [fix](read) remove logic of estimating count of rows to read in segment iterator to avoid wrong result of unique key. (apache#29109)

* [fix](paimon) read batch by doris' batch size (apache#29039)

* [fix](mtmv)add log for resolve pending task (apache#29078)

* [Refact](inverted index) refactor inverted index writer init (apache#29072)

* [fix](paimon)Remove the static attribute of the source for paimon (apache#29032)

* [fix](planner) Fix delete statement on MOR unique table (apache#28968)

add conditions for mor table removed in apache#26776

* [fix][compile]unused variable (apache#28992)

* [Fix](WorkFlow)Auto replay not work

* [Fix](WorkFlow)Auto replay not work

* [Fix](WorkFlow)Auto replay not work

---------

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Signed-off-by: pengyu <pengyu@selectdb.com>
Co-authored-by: lihangyu <15605149486@163.com>
Co-authored-by: Kang <kxiao.tiger@gmail.com>
Co-authored-by: jakevin <jakevingoo@gmail.com>
Co-authored-by: morrySnow <101034200+morrySnow@users.noreply.github.com>
Co-authored-by: xueweizhang <zxw520blue1@163.com>
Co-authored-by: zhangguoqiang <18372634969@163.com>
Co-authored-by: zhangdong <493738387@qq.com>
Co-authored-by: airborne12 <airborne08@gmail.com>
Co-authored-by: Ma1oneZhang <73012001+Ma1oneZhang@users.noreply.github.com>
Co-authored-by: ziyang zhang <zhangziyang@stu.cdut.edu.cn>
Co-authored-by: Siyang Tang <82279870+TangSiyang2001@users.noreply.github.com>
Co-authored-by: Kaijie Chen <ckj@apache.org>
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
Co-authored-by: yiguolei <676222867@qq.com>
Co-authored-by: minghong <englefly@gmail.com>
Co-authored-by: Jibing-Li <64681310+Jibing-Li@users.noreply.github.com>
Co-authored-by: Gavin Chou <gavineaglechou@gmail.com>
Co-authored-by: plat1ko <platonekosama@gmail.com>
Co-authored-by: wudongliang <46414265+DongLiang-0@users.noreply.github.com>
Co-authored-by: Nitin-Kashyap <66766227+Nitin-Kashyap@users.noreply.github.com>
Co-authored-by: yiguolei <yiguolei@gmail.com>
Co-authored-by: wangbo <wangbo@apache.org>
Co-authored-by: nanfeng <42513321+nanfeng1999@users.noreply.github.com>
Co-authored-by: Xinyi Zou <zouxinyi02@gmail.com>
Co-authored-by: qiye <jianliang5669@gmail.com>
Co-authored-by: DuRipeng <453243496@qq.com>
Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
Co-authored-by: Qi Chen <kaka11.chen@gmail.com>
Co-authored-by: Mingyu Chen <morningman@163.com>
Co-authored-by: Xin Liao <liaoxinbit@126.com>
Co-authored-by: yujun <yu.jun.reach@gmail.com>
Co-authored-by: walter <w41ter.l@gmail.com>
Co-authored-by: xy <whuxingying@163.com>
Co-authored-by: xingying01 <xingying01@corp.netease.com>
Co-authored-by: seawinde <149132972+seawinde@users.noreply.github.com>
Co-authored-by: Mryange <59914473+Mryange@users.noreply.github.com>
Co-authored-by: starocean999 <40539150+starocean999@users.noreply.github.com>
Co-authored-by: bobhan1 <bh2444151092@outlook.com>
Co-authored-by: zzzxl <33418555+zzzxl1993@users.noreply.github.com>
Co-authored-by: yangshijie <sjyang2022@zju.edu.cn>
Co-authored-by: zclllyybb <zhaochangle@selectdb.com>
Co-authored-by: amory <wangqiannan@selectdb.com>
Co-authored-by: zhiqiang <seuhezhiqiang@163.com>
Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com>
Co-authored-by: Gabriel <gabrielleebuaa@gmail.com>
Co-authored-by: htyoung <hty551020@hotmail.com>
Co-authored-by: Pxl <pxl290@qq.com>
Co-authored-by: Luwei <814383175@qq.com>
Co-authored-by: meiyi <myimeiyi@gmail.com>
Co-authored-by: Ashin Gau <AshinGau@users.noreply.github.com>
Co-authored-by: HHoflittlefish777 <77738092+HHoflittlefish777@users.noreply.github.com>
Co-authored-by: 谢健 <jianxie0@gmail.com>
Co-authored-by: XuJianxu <xjxlzlpzl@163.com>
Co-authored-by: 胥剑旭 <xujianxu@xujianxudeMacBook-Pro.local>
Co-authored-by: Yoko <53637394+TomaYoko@users.noreply.github.com>
Co-authored-by: huanghaibin <284824253@qq.com>
Co-authored-by: lw112 <131352377+felixwluo@users.noreply.github.com>
Co-authored-by: zhannngchen <48427519+zhannngchen@users.noreply.github.com>
Co-authored-by: Xiangyu Wang <dut.xiangyu@gmail.com>
Co-authored-by: wangxiangyu <wangxiangyu@360shuke.com>
Co-authored-by: AlexYue <yj976240184@gmail.com>
Co-authored-by: TengJianPing <18241664+jacktengg@users.noreply.github.com>
Co-authored-by: xzj7019 <131111794+xzj7019@users.noreply.github.com>
Co-authored-by: Dongyang Li <hello_stephen@qq.com>
Co-authored-by: stephen <hello-stephen@qq.com>
Co-authored-by: HappenLee <happenlee@hotmail.com>
Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com>
Co-authored-by: zhangstar333 <87313068+zhangstar333@users.noreply.github.com>
Co-authored-by: zhengyu <freeman.zhang1992@gmail.com>
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
Co-authored-by: HowardQin <hao.qin@esgyn.cn>
Co-authored-by: qinhao <qinhao@newland.com.cn>
Co-authored-by: deardeng <565620795@qq.com>
Co-authored-by: caiconghui <55968745+caiconghui@users.noreply.github.com>
Co-authored-by: py023 <137761033+py023@users.noreply.github.com>
Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-authored-by: Guangming Lu <71873108+LuGuangming@users.noreply.github.com>
Co-authored-by: abmdocrt <Yukang.Lian2022@gmail.com>
hello-stephen pushed a commit to hello-stephen/doris that referenced this pull request Dec 28, 2023
fe matchExactType function should call type.matchTypes for its own logic, do not switch case to do special logic otherwise we may meet core in be like this.
 ```
F20231208 18:54:39.359673 680131 block.h:535] Check failed: _data_types[i]->is_nullable()  target type: Struct(l_info:Nullable(Array(Nullable(String)))) src type: Struct(col:Nullable(Array(Nullable(UInt8))))
*** Check failure stack trace: ***
    @     0x5584e952b926  google::LogMessage::SendToLog()
    @     0x5584e9527ef0  google::LogMessage::Flush()
    @     0x5584e952c169  google::LogMessageFatal::~LogMessageFatal()
    @     0x5584cf17201e  doris::vectorized::MutableBlock::merge_impl<>()
    @     0x5584ceac4b1d  doris::vectorized::MutableBlock::merge<>()
    @     0x5584d4dd7de3  doris::vectorized::VUnionNode::get_next_const()
    @     0x5584d4dd9a45  doris::vectorized::VUnionNode::get_next()
    @     0x5584bce469bd  std::__invoke_impl<>()
    @     0x5584bce466d0  std::__invoke<>()
    @     0x5584bce465c7  _ZNSt5_BindIFMN5doris8ExecNodeEFNS0_6StatusEPNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS1_St12_PlaceholderILi1EESC_ILi2EESC_ILi3EEEE6__callIS2_JOS4_OS7_OS8_EJLm0ELm1ELm2ELm3EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
    @     0x5584bce46358  std::_Bind<>::operator()<>()
    @     0x5584bce46208  std::__invoke_impl<>()
    @     0x5584bce46178  _ZSt10__invoke_rIN5doris6StatusERSt5_BindIFMNS0_8ExecNodeEFS1_PNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS3_St12_PlaceholderILi1EESD_ILi2EESD_ILi3EEEEJS5_S8_S9_EENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESL_E4typeEOSM_DpOSN_
    @     0x5584bce45c18  std::_Function_handler<>::_M_invoke()
    @     0x5584bce6412f  std::function<>::operator()()
    @     0x5584bce56382  doris::ExecNode::get_next_after_projects()
    @     0x5584bce26218  doris::PlanFragmentExecutor::get_vectorized_internal()
    @     0x5584bce2431b  doris::PlanFragmentExecutor::open_vectorized_internal()
    @     0x5584bce22a96  doris::PlanFragmentExecutor::open()
    @     0x5584bce27c9d  doris::PlanFragmentExecutor::execute()
    @     0x5584bcbdb3f8  doris::FragmentMgr::_exec_actual()
    @     0x5584bcbf982f  doris::FragmentMgr::exec_plan_fragment()::$_0::operator()()
    @     0x5584bcbf9715  std::__invoke_impl<>()
    @     0x5584bcbf96b5  _ZSt10__invoke_rIvRZN5doris11FragmentMgr18exec_plan_fragmentERKNS0_23TExecPlanFragmentParamsERKSt8functionIFvPNS0_12RuntimeStateEPNS0_6StatusEEEE3$_0JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESH_E4typeEOSI_DpOSJ_
    @     0x5584bcbf942d  std::_Function_handler<>::_M_invoke()
    @     0x5584b9dfd883  std::function<>::operator()()
    @     0x5584bd6e3929  doris::FunctionRunnable::run()
    @     0x5584bd6cf8ce  doris::ThreadPool::dispatch_thread()
```
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
fe matchExactType function should call type.matchTypes for its own logic, do not switch case to do special logic otherwise we may meet core in be like this.
 ```
F20231208 18:54:39.359673 680131 block.h:535] Check failed: _data_types[i]->is_nullable()  target type: Struct(l_info:Nullable(Array(Nullable(String)))) src type: Struct(col:Nullable(Array(Nullable(UInt8))))
*** Check failure stack trace: ***
    @     0x5584e952b926  google::LogMessage::SendToLog()
    @     0x5584e9527ef0  google::LogMessage::Flush()
    @     0x5584e952c169  google::LogMessageFatal::~LogMessageFatal()
    @     0x5584cf17201e  doris::vectorized::MutableBlock::merge_impl<>()
    @     0x5584ceac4b1d  doris::vectorized::MutableBlock::merge<>()
    @     0x5584d4dd7de3  doris::vectorized::VUnionNode::get_next_const()
    @     0x5584d4dd9a45  doris::vectorized::VUnionNode::get_next()
    @     0x5584bce469bd  std::__invoke_impl<>()
    @     0x5584bce466d0  std::__invoke<>()
    @     0x5584bce465c7  _ZNSt5_BindIFMN5doris8ExecNodeEFNS0_6StatusEPNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS1_St12_PlaceholderILi1EESC_ILi2EESC_ILi3EEEE6__callIS2_JOS4_OS7_OS8_EJLm0ELm1ELm2ELm3EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
    @     0x5584bce46358  std::_Bind<>::operator()<>()
    @     0x5584bce46208  std::__invoke_impl<>()
    @     0x5584bce46178  _ZSt10__invoke_rIN5doris6StatusERSt5_BindIFMNS0_8ExecNodeEFS1_PNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS3_St12_PlaceholderILi1EESD_ILi2EESD_ILi3EEEEJS5_S8_S9_EENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESL_E4typeEOSM_DpOSN_
    @     0x5584bce45c18  std::_Function_handler<>::_M_invoke()
    @     0x5584bce6412f  std::function<>::operator()()
    @     0x5584bce56382  doris::ExecNode::get_next_after_projects()
    @     0x5584bce26218  doris::PlanFragmentExecutor::get_vectorized_internal()
    @     0x5584bce2431b  doris::PlanFragmentExecutor::open_vectorized_internal()
    @     0x5584bce22a96  doris::PlanFragmentExecutor::open()
    @     0x5584bce27c9d  doris::PlanFragmentExecutor::execute()
    @     0x5584bcbdb3f8  doris::FragmentMgr::_exec_actual()
    @     0x5584bcbf982f  doris::FragmentMgr::exec_plan_fragment()::$_0::operator()()
    @     0x5584bcbf9715  std::__invoke_impl<>()
    @     0x5584bcbf96b5  _ZSt10__invoke_rIvRZN5doris11FragmentMgr18exec_plan_fragmentERKNS0_23TExecPlanFragmentParamsERKSt8functionIFvPNS0_12RuntimeStateEPNS0_6StatusEEEE3$_0JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESH_E4typeEOSI_DpOSJ_
    @     0x5584bcbf942d  std::_Function_handler<>::_M_invoke()
    @     0x5584b9dfd883  std::function<>::operator()()
    @     0x5584bd6e3929  doris::FunctionRunnable::run()
    @     0x5584bd6cf8ce  doris::ThreadPool::dispatch_thread()
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.4-merged p0_c reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants