Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](group-commit) check if wal need recovery is abnormal #28769

Merged
merged 4 commits into from
Dec 22, 2023

Conversation

mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Dec 21, 2023

Proposed changes

  1. The original logic does not consider plan fragment execute status, so some wal may be deleted directly, but in fact, it need replay.
  2. If a stream load is cancelled in client side, should remove it from load ids, otherwise, the internal group commit can not commit because it is waiting for the load is finished.
  3. regression test check server parepare statement id; add session varaibels in jdbc url to check if it works
  4. reuse fe plan fragment does not work

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Dec 21, 2023

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@@ -335,7 +334,7 @@ Status WalManager::add_recover_wal(const std::string& db_id, const std::string&
return Status::OK();
}

size_t WalManager::get_wal_table_size(const std::string& table_id) {
size_t WalManager::get_wal_table_size(int64_t table_id) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: method 'get_wal_table_size' can be made static [readability-convert-member-functions-to-static]

be/src/olap/wal_manager.h:56:

-     size_t get_wal_table_size(int64_t table_id);
+     static size_t get_wal_table_size(int64_t table_id);

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.48% (8538/23406)
Line Coverage: 28.60% (69419/242737)
Region Coverage: 27.62% (35926/130054)
Branch Coverage: 24.37% (18363/75362)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c26d13899f79f555be27a26faff9cb69bd35d3b5_c26d13899f79f555be27a26faff9cb69bd35d3b5/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit c26d13899f79f555be27a26faff9cb69bd35d3b5, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4677	4425	4437	4425
q2	357	160	166	160
q3	1467	1289	1249	1249
q4	1108	935	886	886
q5	3165	3140	3205	3140
q6	247	127	131	127
q7	1018	479	492	479
q8	2197	2202	2189	2189
q9	6746	6655	6700	6655
q10	3233	3268	3307	3268
q11	313	182	186	182
q12	355	212	216	212
q13	4570	3787	3847	3787
q14	253	213	218	213
q15	568	530	526	526
q16	442	390	387	387
q17	993	601	573	573
q18	7111	7042	7040	7040
q19	1541	1461	1415	1415
q20	524	337	318	318
q21	3078	2690	2729	2690
q22	352	276	280	276
Total cold run time: 44315 ms
Total hot run time: 40197 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4382	4375	4420	4375
q2	274	169	181	169
q3	3542	3541	3528	3528
q4	2392	2388	2380	2380
q5	5765	5732	5763	5732
q6	244	123	122	122
q7	2381	1866	1884	1866
q8	3531	3523	3518	3518
q9	9070	9043	9013	9013
q10	3934	4013	4038	4013
q11	479	376	375	375
q12	771	607	592	592
q13	4289	3572	3574	3572
q14	287	253	267	253
q15	561	521	518	518
q16	509	467	479	467
q17	1867	1860	1852	1852
q18	8677	8099	8277	8099
q19	1753	1762	1773	1762
q20	2266	1957	1962	1957
q21	6552	6200	6185	6185
q22	506	440	404	404
Total cold run time: 64032 ms
Total hot run time: 60752 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.43 seconds
stream load tsv: 588 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.5 seconds inserted 10000000 Rows, about 338K ops/s
storage size: 17183928369 Bytes

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Dec 21, 2023

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.47% (8536/23406)
Line Coverage: 28.59% (69406/242749)
Region Coverage: 27.61% (35916/130060)
Branch Coverage: 24.36% (18357/75368)
Coverage Report: http://coverage.selectdb-in.cc/coverage/02acd0948784e697f327cb049a4aa265c8231923_02acd0948784e697f327cb049a4aa265c8231923/report/index.html

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.02 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17188562169 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 02acd0948784e697f327cb049a4aa265c8231923, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4658	4376	4396	4376
q2	378	148	158	148
q3	1452	1236	1186	1186
q4	1159	871	849	849
q5	3147	3125	3146	3125
q6	243	127	129	127
q7	991	482	477	477
q8	2166	2218	2176	2176
q9	6705	6678	6704	6678
q10	3222	3272	3252	3252
q11	303	195	185	185
q12	357	213	206	206
q13	4578	3816	3779	3779
q14	239	210	216	210
q15	574	523	524	523
q16	438	385	386	385
q17	994	567	530	530
q18	7104	6980	6897	6897
q19	1508	1339	1431	1339
q20	511	297	294	294
q21	3051	2638	2672	2638
q22	347	286	283	283
Total cold run time: 44125 ms
Total hot run time: 39663 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4331	4341	4319	4319
q2	268	168	171	168
q3	3540	3518	3502	3502
q4	2374	2359	2368	2359
q5	5741	5733	5724	5724
q6	242	121	119	119
q7	2387	1845	1879	1845
q8	3502	3511	3510	3510
q9	8985	9025	9012	9012
q10	3918	3985	4026	3985
q11	493	375	364	364
q12	759	592	592	592
q13	4301	3563	3583	3563
q14	286	255	263	255
q15	565	517	521	517
q16	502	469	457	457
q17	1879	1857	1832	1832
q18	8617	8214	8306	8214
q19	1720	1767	1758	1758
q20	2244	1933	1947	1933
q21	6505	6215	6197	6197
q22	505	428	435	428
Total cold run time: 63664 ms
Total hot run time: 60653 ms

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Dec 21, 2023

run buildall

dataroaring
dataroaring previously approved these changes Dec 21, 2023
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 21, 2023
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.47% (8536/23405)
Line Coverage: 28.59% (69418/242780)
Region Coverage: 27.61% (35915/130078)
Branch Coverage: 24.35% (18357/75384)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e0bc3c45361fd3850ba712f6f1ed0e27057d3903_e0bc3c45361fd3850ba712f6f1ed0e27057d3903/report/index.html

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.09 seconds
stream load tsv: 570 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17183781064 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit e0bc3c45361fd3850ba712f6f1ed0e27057d3903, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4671	4388	4399	4388
q2	359	165	147	147
q3	1470	1266	1239	1239
q4	1093	872	865	865
q5	3131	3167	3147	3147
q6	250	127	127	127
q7	999	491	477	477
q8	2176	2218	2181	2181
q9	6694	6652	6651	6651
q10	3224	3255	3257	3255
q11	299	184	179	179
q12	352	208	208	208
q13	4546	3799	3763	3763
q14	236	213	216	213
q15	562	530	512	512
q16	435	380	381	380
q17	1000	561	581	561
q18	7714	7255	7004	7004
q19	1501	1280	1412	1280
q20	544	309	279	279
q21	3040	2663	2668	2663
q22	351	280	284	280
Total cold run time: 44647 ms
Total hot run time: 39799 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4336	4346	4342	4342
q2	267	166	173	166
q3	3539	3523	3538	3523
q4	2386	2375	2367	2367
q5	5737	5756	5751	5751
q6	242	118	118	118
q7	2378	1875	1882	1875
q8	3523	3517	3510	3510
q9	9007	8973	8971	8971
q10	3883	3976	3979	3976
q11	481	377	364	364
q12	765	597	627	597
q13	4300	3551	3576	3551
q14	296	258	254	254
q15	575	521	527	521
q16	497	451	449	449
q17	1871	1852	1846	1846
q18	8667	8312	8251	8251
q19	1725	1748	1732	1732
q20	2257	1967	1944	1944
q21	6512	6182	6114	6114
q22	501	423	408	408
Total cold run time: 63745 ms
Total hot run time: 60630 ms

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 21, 2023
@mymeiyi
Copy link
Contributor Author

mymeiyi commented Dec 21, 2023

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.47% (8536/23405)
Line Coverage: 28.59% (69415/242784)
Region Coverage: 27.61% (35919/130092)
Branch Coverage: 24.35% (18358/75386)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b0722aaed251206873cbd04299e190051c736810_b0722aaed251206873cbd04299e190051c736810/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit b0722aaed251206873cbd04299e190051c736810, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4685	4400	4398	4398
q2	371	144	159	144
q3	1457	1218	1210	1210
q4	1112	906	947	906
q5	3149	3153	3141	3141
q6	248	128	128	128
q7	1014	494	486	486
q8	2178	2226	2167	2167
q9	6710	6680	6651	6651
q10	3201	3279	3292	3279
q11	314	188	186	186
q12	351	212	203	203
q13	4560	3785	3818	3785
q14	240	219	206	206
q15	565	523	531	523
q16	444	383	384	383
q17	1012	600	551	551
q18	7271	7149	7032	7032
q19	1511	1306	1467	1306
q20	518	281	303	281
q21	3072	2651	2699	2651
q22	355	283	296	283
Total cold run time: 44338 ms
Total hot run time: 39900 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4337	4378	4312	4312
q2	270	170	172	170
q3	3569	3540	3536	3536
q4	2398	2382	2378	2378
q5	5725	5725	5741	5725
q6	241	123	120	120
q7	2386	1928	1865	1865
q8	3524	3531	3526	3526
q9	9012	9011	9008	9008
q10	3898	3986	3981	3981
q11	477	372	380	372
q12	769	615	593	593
q13	4256	3557	3587	3557
q14	278	242	264	242
q15	569	514	527	514
q16	509	464	459	459
q17	1907	1866	1861	1861
q18	8779	8299	8240	8240
q19	1737	1737	1750	1737
q20	2268	1937	1927	1927
q21	6544	6253	6192	6192
q22	496	454	420	420
Total cold run time: 63949 ms
Total hot run time: 60735 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.2 seconds
stream load tsv: 574 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17183887872 Bytes

@dataroaring dataroaring changed the title [fix](group-commit) check if wal need recovery is not correct [fix](group-commit) check if wal need recovery is abnormal Dec 22, 2023
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 22, 2023
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 0af6bd6 into apache:master Dec 22, 2023
27 of 30 checks passed
Copy link
Contributor

@hust-hhb hust-hhb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Dec 27, 2023
* [Performance](point query)Opimize partition prune for point query (apache#28150)

* [Performance](point query)Opimize partition prune for point query

* [fix](stacktrace) ignore stacktrace for error code INVALID_ARGUMENT INVERTED_INDEX_NOT_IMPLEMENTED (apache#27898) (apache#28598)

* ignore stacktrace for error INVALID_ARGUMENT INVERTED_INDEX_NOT_IMPLEMENTED

* AndBlockColumnPredicate::evaluate

* [bugfix](topn) fix coredump in copy_column_data_to_block when nullable mismatch (apache#28597)

* [bugfix](topn) fix coredump in copy_column_data_to_block when nullable mismatch

return RuntimeError if copy_column_data_to_block nullable mismatch to avoid coredump in input_col_ptr->filter_by_selector(sel_rowid_idx, select_size, raw_res_ptr) .

The problem is reported by a doris user but I can not reproduce it, so there is no testcase added currently.

* clang format

* [feature](Nereids): eliminate semi join (apache#28588)

Eliminate Semi/Anti Join which is FALSE or TRUE.

* [feature](Nereids) support datev1 and datetimev1 (apache#28581)

* [improvement](http) add show_table_data http api (apache#28380)

In some cases, users need to get the data size of single replica of a table, and evaluate certain actions based on this, such as estimating the precise backup size.

Signed-off-by: nextdreamblue <zxw520blue1@163.com>

* [Enhance](regression)enhance jdbc case to adapt to use case concurrency (apache#28565)

enhance jdbc case to adapt to use case concurrency

* [opt](Nereids)when both Nereids and old parsers report errors, prompt error messages for the Nereids (apache#28580)

* [Improvement](regression) change compound predicate regression case name to make it more clear (apache#28612)

* [regression](memtable) add case for memtable flush error handle (apache#28285)

Co-authored-by: ziyang zhang <zhangziyang@stu.cdut.edu.cn>

* [regression-test](memtable)  test memtable flush is high priority for vtable writerV1 (apache#28502)

* enhance performance for broken tablet checking under multi-core scenario with a coarse-grained read lock (apache#28552)

* [fix](load) fix memtracking orphan too large (apache#28600)

* [fix](memtable-limiter) do not block write if load mem usage is low (apache#28602)


Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>

* [refactor](renamefile) rename some files according to the class names (apache#28606)

* [feature](mtmv)create mtmv support refresh_partition_num (apache#28566)

- create/alter mtmv support refresh_partition_num
- mtmv task according to refresh_partition_num executes refresh tasks in batches
- `tasks` tvf add column `CompletedPartitions` and `progress`
- fix mtmv can not `show temp partition` and `drop temp partition`
- fix task can not get error msg when insert overwrite error
- fix when the partition field is capitalized, the verification of creating a mtmv does not pass

* [fix](Nereids) stats estimation of lessThan apache#28444

* [fix](regression)Change analyze_timeout to global.  (apache#28587)

Fix hive statistics regression case. analyze_timeout is a global session variable.

* [chore] Add bvar for meta operations of BE (apache#28374)

* [Bug](cooldown) Fix problem that followers may never completely cooldown (apache#28561)

* [Improve](tvf)jni-avro support split file (apache#27933)

* [feature](mtmv)after creating a materialized view, if other operations fail, roll back  (apache#28621)

after create mtmv, if create job failed,need drop mtmv

* [fix](meta) update killed query state (#) (apache#25917)

* [refactor](profile&names) using dst_id in pipelinex profile to be same as non pipeline; rename some function names (apache#28626)

Co-authored-by: yiguolei <yiguolei@gmail.com>

* [feature][executor]support workload schedule policy (apache#28443)

* [feature](expr) Support kill query by query_id (apache#28530)

Issue Number: open apache#28517

* [chore](user) Add user property `parallel_fragment_exec_instance_num` (apache#28447)

* [feature](inverted index) add ignore_above property to prevent long s… (apache#28585)

When string is too long, clucene will throw an error. 
And the string is too long to analyze. So we ignore the string in index process when the string is longer than 256 bytes by default.
We add an poperty `ignore_above` for user to customize.

* [Enhance](broker) add inputstream expire scheduled checker to avoid memory leak for broker scan (apache#28589)

This pr introduces 2 broker conf:

1. enable_input_stream_expire_check: which indicates whether enable inputStream expire check.
2. input_stream_expire_seconds: which indicates the timeout seconds for inputStream since last update.

* [fix](hive) add support for `quoteChar` and `seperatorChar` for hive (apache#28613)

add support for quoteChar and seperatorChar .

* [Fix](transactional-hive) Fix hive transactional table return empty result. (apache#28518)

* [Fix](memtable) fix `shrink_memtable_by_agg` without duplicated keys (apache#28660)

remove duplicated logic:
```
vectorized::Block in_block = _input_mutable_block.to_block();
_put_into_output(in_block);
```
`_input_mutable_block.to_block()` will move `_input_mutable_block`, and lead to `flush` with empty block

* [enhance](partition_id) check partition id before store meta (apache#28055)

* [opt](task-assignment) use consistent hash as default task assigner and cache the consistent hash ring (apache#28522)

1. Use consistent hash algo as the default assigner for file query scan node
    A consistent assignment can better utilize the page cache of BE node.

2. Cache the consistent hash ring
    Init a consistent hash ring is time-consuming because there a thousands of virtual node need to be added.
    So cache it for better performance

* [Fix](Job)Fixed job scheduling missing certain time window schedules (apache#28659)

Since scheduling itself consumes a certain amount of time, the start time of the time window should not be the current time, but the end time of the last schedule.

* [fix](test) fix ccr test cases (apache#28664)

* [fix](regression) fix test_set_replica_status due to force_olap_table_replication_num=3 (apache#28573)

* [fix](regression) restore reserve num replicas (apache#28541)

* [fix](regression) fix test_alter_colocate_table due to force_olap_table_replication_num=3 (apache#28575)

* [improvement](transaction) reduce publish txn log (apache#28277)

* [fix](stream_load)fix bug for stream (apache#27752)

1. forbid thed  stream_load without content-length or chunked Transfer Encoding
2. forbid thed  stream_load both with content-length and chunked Transfer Encoding

Co-authored-by: xingying01 <xingying01@corp.netease.com>

* [Revert](partial update) Revert "Fix missing rowsets during doing alignment when flushing memtable due to compaction (apache#28062)" (apache#28674)

This reverts commit 485d7db.

* [feat](Nereids) support outer join and aggregate bitmap rewrite by mv (apache#28596)

- Support left outer join rewrite by materialized view
- Support bitmap_union roll up to imp count(distinct)
- Support partition materialized view rewrite

* [refactor](runtimefilter) do not use QueryContext in runtime filter (apache#28559)

* [fix](planner)should bind expr using no slot to correct tuple (apache#28656)

* [fix](planner) ctas update datev1 to datev2 should use equals (apache#28641)

* [fix](Nereids) should only do bind relation in view analyzer (apache#28637)

* [code](pipelineX) refine some pipelineX code  (apache#28570)

* [fix](partial update) report error directly if missing rowsets during doing alignment when flushing memtable due to compaction  (apache#28677)

* [opt](inverted index) Add null document interface to optimize empty string indexing (apache#28661)

* [feature](invert index) match_regexp feature added (apache#28257)

* [Feature](datatype) update be ut codes and fix bugs for IPv4/v6 (apache#28670)

* [test](Nereids): remove shape plan project and distribute in eager test (apache#28701)

* [fix](statistics)Fix drop stats fail silently bug. (apache#28635)

Drop stats use IN predicate to filter the column stats to delete. The default length of IN predicate is 1024, drop table stats with more than 1024 columns may fail.
This pr is to split the delete sql based on the IN predicate length.

* [fix](mtmv) fix insert overwrite getExecTimeout error (apache#28700)

should use InsertTimeout but QueryTimeout

* [Enhancement](auto-partition) change the behaviour when insert overwrite an auto partition table apache#28683

If we specific target partition(s) when inserting overwrite an auto partition table,
before:
could create new partition
now:
behalf just like non-auto partition table

* [FIX](regresstest) fix case with double value apache#28668

double value in case has 27 figures after the point, which will make not standable output

* [test](Nereids): add test for scalar agg (apache#28712)

* [fix](meta_scanner) fix meta_scanner process ColumnNullable (apache#28711)

* [fix](nereids) Fix data wrong using mv rewrite and ignore case when getting mv related partition table (apache#28699)

1. Fix data wrong using mv rewrite
2. Ignore case when getting mv related partition table
3. Enable infer expression column name without alias when create mv

* [test](regression-test) order by decs should only make effect on its nearest column apache#28728

* [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline (apache#28488)

* [fix](Nereids) delete partition failed (apache#28717)

1. parser's partitionSpec changed unexpectly by PR apache#26492
2. delete without using should support un-equals expression

* [fix](tvf)Fixed the avro-scanner projection pushdown failing to query on multiple BEs (apache#28709)

* [fix](block) fix nullptr in MutableBlock::allocated_bytes (apache#28738)

* [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE (apache#28723)

when replay addTaskResult log,will create one ConnectContext,and set Env.getCurrentEnv,then store this ctx in ConnectContext.threadLocalInfo,threadLocalInfo is static,so this ctx can not be recycling,Env of replay thread also can not be recycling

* [Fix](json reader) fix json reader crash due to `fmt::format_to` (apache#28737)

```
4# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75
5# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
6# 0x00005622F33D22B1 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
7# 0x00005622F33D2404 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
8# fmt::v7::detail::error_handler::on_error(char const*) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
9# char const* fmt::v7::detail::parse_replacement_field<char, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char> >&>(char const*, char const*, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char> >&) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
10# void fmt::v7::detail::vformat_to<char>(fmt::v7::detail::buffer<char>&, fmt::v7::basic_string_view<char>, fmt::v7::basic_format_args<fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<fmt::v7::type_identity<char>::type>, fmt::v7::type_identity<char>::type> >, fmt::v7::detail::locale_ref) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
11# doris::vectorized::NewJsonReader::_append_error_msg(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:924
12# doris::vectorized::NewJsonReader::_set_column_value
```

* [bug](coredump) Fix coredump in aggregation node's destruction(apache#28684)

fix coredump in aggregation node's destruction

* [pipelineX](improvement) Support global runtime filter (apache#28692)

* [FIX](explode)fix explode array decimal (apache#28744)

* fix explode with array<decimal> has specific precision at old planner

* [refactor](broadcastbuffer) using a queue to remove ref and unref codes (apache#28698)

Co-authored-by: yiguolei <yiguolei@gmail.com>Add a new class broadcastbufferholderqueue to manage holders
Using shared ptr to manage holders, not use ref and unref, it is too difficult to maintain.

* [enhance](blacklist) seperate blacklist conf from heartbeat (apache#28638)

There is a circuit breaker lasting for 2 minutes in grpc, then if a be is down and up again, send fragments to the be fails lasting for 2 minutes.

* [fix](stmt):fix CreateTableStmt toSql placed comment in wrong place (apache#27504)

Issue Number: close apache#27474
Co-authored-by: tongyang.han <tongyang.han@jiduauto.com>

* [Improve](compile) add `__AVX2__` macro for JsonbParser (apache#28754)

* [Improve](compile) add `__AVX2__` macro for JsonbParser

* throw exception instead of CHECK

* [Feature](materialized-view) support match logicalAggregate(logicalProject(logicalFilter(logicalOlapScan())) without agg (apache#28747)

support match logicalAggregate(logicalProject(logicalFilter(logicalOlapScan())) without agg

* [Fix] (schema change) forbid adding time type column (apache#28751)

* [fix](group_commit) fix group commit cancel stuck (apache#28749)

* [fix](scanner) fix concurrency bugs when scanner is stopped or finished (apache#28650)

`ScannerContext` will schedule scanners even after stopped, and confused with `_is_finished` and `_should_stop`.
 Only Fix the concurrency bugs when scanner is stopped or finished reported in apache#28384

* [fix](regression) fix stream load properties case fail (apache#28680)

* [fix](fe ut) fix PropertyConverterTest (apache#28722)

* [improve](rpc) Log channel state before shutdown backend service client (apache#28667)

* [fix](publish version) fix publish fail but return ok (apache#28425)

* [test](Nereids): remove shape plan project and distribute in eager test (apache#28724)

* [refactor](nereids) make NormalizeAggregate rule more clear and readable (apache#28607)

* [FIX](type) fix matchExactType for complex type (apache#28233)

fe matchExactType function should call type.matchTypes for its own logic, do not switch case to do special logic otherwise we may meet core in be like this.
 ```
F20231208 18:54:39.359673 680131 block.h:535] Check failed: _data_types[i]->is_nullable()  target type: Struct(l_info:Nullable(Array(Nullable(String)))) src type: Struct(col:Nullable(Array(Nullable(UInt8))))
*** Check failure stack trace: ***
    @     0x5584e952b926  google::LogMessage::SendToLog()
    @     0x5584e9527ef0  google::LogMessage::Flush()
    @     0x5584e952c169  google::LogMessageFatal::~LogMessageFatal()
    @     0x5584cf17201e  doris::vectorized::MutableBlock::merge_impl<>()
    @     0x5584ceac4b1d  doris::vectorized::MutableBlock::merge<>()
    @     0x5584d4dd7de3  doris::vectorized::VUnionNode::get_next_const()
    @     0x5584d4dd9a45  doris::vectorized::VUnionNode::get_next()
    @     0x5584bce469bd  std::__invoke_impl<>()
    @     0x5584bce466d0  std::__invoke<>()
    @     0x5584bce465c7  _ZNSt5_BindIFMN5doris8ExecNodeEFNS0_6StatusEPNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS1_St12_PlaceholderILi1EESC_ILi2EESC_ILi3EEEE6__callIS2_JOS4_OS7_OS8_EJLm0ELm1ELm2ELm3EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
    @     0x5584bce46358  std::_Bind<>::operator()<>()
    @     0x5584bce46208  std::__invoke_impl<>()
    @     0x5584bce46178  _ZSt10__invoke_rIN5doris6StatusERSt5_BindIFMNS0_8ExecNodeEFS1_PNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS3_St12_PlaceholderILi1EESD_ILi2EESD_ILi3EEEEJS5_S8_S9_EENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESL_E4typeEOSM_DpOSN_
    @     0x5584bce45c18  std::_Function_handler<>::_M_invoke()
    @     0x5584bce6412f  std::function<>::operator()()
    @     0x5584bce56382  doris::ExecNode::get_next_after_projects()
    @     0x5584bce26218  doris::PlanFragmentExecutor::get_vectorized_internal()
    @     0x5584bce2431b  doris::PlanFragmentExecutor::open_vectorized_internal()
    @     0x5584bce22a96  doris::PlanFragmentExecutor::open()
    @     0x5584bce27c9d  doris::PlanFragmentExecutor::execute()
    @     0x5584bcbdb3f8  doris::FragmentMgr::_exec_actual()
    @     0x5584bcbf982f  doris::FragmentMgr::exec_plan_fragment()::$_0::operator()()
    @     0x5584bcbf9715  std::__invoke_impl<>()
    @     0x5584bcbf96b5  _ZSt10__invoke_rIvRZN5doris11FragmentMgr18exec_plan_fragmentERKNS0_23TExecPlanFragmentParamsERKSt8functionIFvPNS0_12RuntimeStateEPNS0_6StatusEEEE3$_0JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESH_E4typeEOSI_DpOSJ_
    @     0x5584bcbf942d  std::_Function_handler<>::_M_invoke()
    @     0x5584b9dfd883  std::function<>::operator()()
    @     0x5584bd6e3929  doris::FunctionRunnable::run()
    @     0x5584bd6cf8ce  doris::ThreadPool::dispatch_thread()
```

* [test](Nereids): add assert debug log for TopnToMaxTest (apache#28755)

* [improve](move-memtable) limit task num in load stream flush token (apache#28748)

* [feature](Nereids) elimite inner join by foreign key (apache#28486)

* [enhancement](backup/restore) support alter s3 repo info about ak/sk/token (apache#27027)

in some cases:

s3.session_token/AWS_TOKEN will be expired after hours, and may be upload snapshot job will failed if data is big large;
as same reason, repo will be expired too when RepositoryMgr exec repo ping
so it need support alter s3 repo properties about ak/sk/token and update upload snapshot job properties to continue backup.

Signed-off-by: nextdreamblue <zxw520blue1@163.com>

* [improve](transaction) extend abort transaction time (apache#28662)

* [regression](p2) fix test cases result (apache#28768)

regression-test/data/external_table_p2/hive/test_hive_hudi.out
regression-test/data/external_table_p2/hive/test_hive_to_array.out
regression-test/suites/external_table_p2/tvf/test_local_tvf_compression.groovy
regression-test/suites/external_table_p2/tvf/test_path_partition_keys.groovy
regression-test/data/external_table_p2/hive/test_hive_text_complex_type.out

* (fix)[meta][export] fix replay export NPE issue (apache#28752)

The ConnectionContext does not exist in replay thread

* [Bug fix][metrics] correct fe collector type for jvm_gc (apache#28784)

Co-authored-by: 胥剑旭 <xujianxu@xujianxudeMacBook-Pro.local>

* [Chore](decimal) set check_overflow_for_decimal to true when alter table(apache#28777)

set check_overflow_for_decimal to true when alter table

* [pipelineX](fix) Fix TPCH Q2 (apache#28783)

* (topN)runtime_predicate is only triggered when the column name is obtained (apache#28419)

Issue Number: close apache#27485

* [fix](planner)fix bug of bound conjunct to wrong tuple (apache#28811)

this fix bug introduced by apache#28656

* [bug](pipelineX) Fix pipelineX bug on multiple BE (apache#28792)

* [opt](inverted index) ignore_above only affects untokenized strings (apache#28819)

* [improve](load) reduce lock scope in MemTableWriter active consumption (apache#28790)

* [opt](Nereids) improve Shape check cases (apache#28124)

* tmplate update
* log tpcds stats when check shape

* [fix](group commit)Fix the issue of duplicate addition of wal path when encouter exception (apache#28691)

* [fix](compaction) compaction should catch exception when vertical block reader read next block (apache#28625)

* [fix](Nereids) join order is not right after sql parsing (apache#28721)

for sql
```
t1, t2 join t3
```

we should generate plan like:
```
t1 join (t2 join t3)
```

but we generate:
```
(t1 join t2) join t3
```
to follow legancy planner.

* [fix](metrics) fix bvar memtable_input_block_allocated_size (apache#28725)

* [enhancement](audit-log) add status code and error msg to audit log for proxy stmt (apache#27607)

* [improvement](external catalog)Change log level to debug for getRowCountFromExternalSource. (apache#28801)

* [fix](partial update) only report error when in strict mode partial update when finding missing rowsets during flushing memtable (apache#28764)

related pr: apache#28062, apache#28674, apache#28677
fix apache#28677

* [fix](function) Fix from_second functions overflow and wrong result (apache#28685)

* [Enhancement](load) Limit the number of incorrect data drops and add documents (apache#27727)

In the load process, if there are problems with the original data, we will store the error data in an error_log file on the disk for subsequent debugging. However, if there are many error data, it will occupy a lot of disk space. Now we want to limit the number of error data that is saved to the disk.

Be familiar with the usage of doris' import function and internal implementation process
Add a new be configuration item load_error_log_limit_bytes = default value 200MB
Use the newly added threshold to limit the amount of data that RuntimeState::append_error_msg_to_file writes to disk
Write regression cases for testing and verification

Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com>

* [test](partial update) add complex type regression cases for partial update (apache#28758)

NOTE: There's some issue for MAP type to work with row store, so in this PR we don't have cases for MAP type
Will add the support for MAP type in future.

* [fix](test)fix test_create_table test case for nereids (apache#28693)

* [improve](config) set mutable and masterOnly in FE config stream_load_default_memtable_on_sink_node (apache#28835)

* [fix](group-commit) check if wal need recovery is abnormal (apache#28769)

* [Fix](Variant) fix variant predicate rewrite OrToIn with wrong plan (apache#28695)

using the name without paths info will lead to wrong In plan, e.g.
```
where cast(v:a as text) = 'hello' or cast(v:b as text) = 'world'
```
will be rewrite to:
```
where cast(v as text) in ('hello', 'world')
``
This is wrong, because they are different slots

* [refactor](buffer) remove download buffer since it is no longer useful (apache#28832)

remove download buffer since it is no longer useful

* [Feature](Variant) Implement variant new sub column access method (apache#28484)

* [Feature](Variant) Implement variant new sub column access method

The query SELECT v["a"]["b"] from simple_var WHERE cast(v["a"]["b"] as int) = 1 encompasses three primary testing scenarios:

```
1. A basic test involving the variant data type.
2. A scenario dealing with GitHub event data in the context of a variant.
3. A case related to the TPC-H benchmark using a variant.
```

* [fix](memory) Add thread asynchronous purge jemalloc dirty pages (apache#28655)

jemallctl purge all arena dirty pages may take several seconds, which will block memory GC and cause OOM.
So purge asynchronously in a thread.

* [fix](regression) fix regression error of test_compress_type (apache#28826)

* [improvement](executor) Add tvf and regression test for Workload Scheduler (apache#28733)

1 Add select workload schedule policy tvf
2 Add reg test

* [chore](error msg) print type info when colocate with ddl failed due to type mismatch (apache#28773)

* [opt](query cancel) optimization for query cancel apache#28778

* [Fix](multi-catalog) skip hms events if hms table is not supported. (apache#28644)

Co-authored-by: wangxiangyu <wangxiangyu@360shuke.com>

* [Enhancement](job) No need to query some backends which are not alive. (apache#28608)

No need to execute some jobs if backend is not alive

* [fix](paimon)fix type convert for paimon (apache#28774)

fix type convert for paimon

* [fix](stream load)add test case and doc for arrow type of stream load (apache#28098)

add test case and doc for arrow type of stream load

* [feature](mtmv)mtmv partition refresh case (apache#28787)

* (enhance)(InternalQuery) Support to collect profile for intenal query (apache#28762)

* [optimize](count) optimize pk exact query without reading data (apache#28494)

* [opt](sessionVar)show changed sessoin var first apache#28840

“show variables” command list changed vars before not changed vars,

* check  stats and log memo for ds46 (apache#28396)

* [bug](sharedscan) Fix shared scan bug (apache#28841)

* [fix](hash join) fix stack overflow caused by evaluate case expr on huge build block (apache#28851)

* [refactor](executor)remove scan group apache#28847

* [pipelineX](refactor) rename functions (apache#28846)

* [Feature](inverted index) add lowercase option for inverted index analyzer (apache#28704)

* [feature](Nereids) Pull up join from union all (apache#28682)

* [fix](ci) tpch pipeline should not re-load data (apache#28874)

* [fix](ci) tpch pipeline should not re-load data

* 2

---------

Co-authored-by: stephen <hello-stephen@qq.com>

* [exec](compress) use FragmentTransmissionCompressionCodec control the exchange compress behavior (apache#28818)

* [improve](move-memtable) tweak load stream flush token num and max tasks (apache#28884)

* [improve](load) remove extra layer of heavy work pool in tablet_writer_add_block (apache#28550)

* [improve](load) limit delta writer flush task parallelism (apache#28883)

* [fix](multi-catalog)filter impala generated path (apache#28786)

file impala generated dir _imapala_insert_staging

* [enhancement](udf) add prepare function for java-udf (apache#28750)

* [refactor](test)Refactor workload group/schedule policy test apache#28888

[refactor](test)Refactor workload group/schedule policy test
apache#28888

* [improve](move-memtable) avoid using heavy work pool during append data (apache#28745)

* [enhancement](broker-load) fix-move-memtable-session-var-for-s3 (apache#28894)

* [exec](load) change default parallel num from 1 to 8 in no pipeline exec engine (apache#28864)

* [fix](segcompaction) disable segcompaction by default (apache#28906)

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

* [fix](pipelineX) fix cannot runtime obtain profile on pipelineX apache#28795

* [fix](mtmv) fix failed to specify the number of buckets when bucket auto (apache#28854)

Issue Number: close #xxx

- fix failed to specify the number of buckets when bucket auto
- delete unused SessionVariable
- if mtmv used external table ,check `isMaterializedViewRewriteEnableContainForeignTable`

* [bugfix](scannercore) scanner will core in deconstructor during collect profile (apache#28727)

* [FIX](map)fix map with rowstore table (apache#28877)

* [feature](mtmv)MTMV pause and resume (apache#28887)

- PAUSE MATERIALIZED VIEW JOB ON mv1
- RESUME MATERIALIZED VIEW JOB ON mv1
- fix when drop db,not drop job
- add lock for one materialized view can only run one task at a time

* [feature](mtmv)add more test case1 (apache#28910)

* [fix](block) add block columns size dcheck (apache#28539)

* [fix](chore) update dcheck to avoid core during stress test (apache#28895)

* [fix](doc) Add the usage example of bos to the documentation of s3 tvf (apache#28899)

* [fix](mtmv)fix can not create mtmv all use default value (apache#28922)

* [fix](parquet) the end offset of column chunk may be wrong in parquet metadata (apache#28891)

* [fix](paimon)fix `like` predicate (apache#28803)

fix like predict

* [fix](merge-on-write) migration may cause duplicate keys for mow table (apache#28923)

* [fix](mtmv) Related partition exclude null generate column when increment build materialized view (apache#28855)

Infer partition column by materialized view partition column, exclude null generate column in join when increment build materialized view

* [nereids] fix join fd computing bug (apache#28849)

* [Fix](statistics) Fix partition name NPE and sample for all table during auto analyze (apache#28916)

Fix partition name NPE and sample for all table during auto analyze.
Sample for all tables because getData may have latency, which may cause full analyze a huge table and use too much resource. Sample for all tables to avoid this. Will improve the strategy later.

* [fix](nereids) Fix query mv rewrite fail when mv cache build quickly (apache#28876)

* [optimize](zonemap) skip zonemap if predicate does not support_zonemap (apache#28595)

* [optimize](zonemap) skip zonemap if predicate does not support_zonemap apache#27608 (apache#28506)

* Revert "[bugfix](scannercore) scanner will core in deconstructor during collect profile (apache#28727)" (apache#28931)

This reverts commit 4066de3.

* [fix] (nereids) Catch exception when mv fail and fix the npe (apache#28932)

* [fix](regression-test) test_partial_update_native_insert_stmt_complex is flaky (apache#28927)

* [improvement](group commit) make get column function more reliable when replaying wal (apache#28900)

* [opt](nereids) convert or to inpredicate and optimize inpredicate partition prune (apache#28316)

* [enhancement](bulk-load) cancel loading tasks directly without retrying when timeout exceeded (apache#28666)

* [chore](test) Add testing util sync point (apache#28924)

* [chore](prompt) Optimize tablet and replica prompt by pointing out what the numbers mean (apache#28925)

* [fix](join) incorrect result of left semi/anti join with empty build side (apache#28898)

* [docs] (DebugPoints) Update docs about Debug Points (apache#28347)


---------

Co-authored-by: qinhao <qinhao@newland.com.cn>

* [feature](api) add profile text api (apache#28697)

* [opt](invert index) Empty strings are not written to the index in the case of TOKENIZED (apache#28822)

* [feature](pipelineX) control exchange sink by memory usage (apache#28814)

* [fix](pipelineX) fix use global rf when there no shared_scans (apache#28869)

* [refactor](pipelineX)do not override dependency() function in pipelineX (apache#28848)

* [fix](log) regularise some BE error type and fix a load task check apache#28729

* [chore](test) correct create table statement (apache#28863)

* [fix](nereids)subquery unnest need handle subquery in Not expr correnctly (apache#28713)

* [feature](nereids)support decimalv2 (apache#28726)

* [fix](nereids)group by expr may be bound twice in bind agg slot (apache#28771)

* [doc](insert) Add group commit docs (apache#25949)

* [performance](variant) support topn 2phase read for variant column (apache#28318)

 [performance](variant) support topn 2phase read for variant column

* return residual expr of join (apache#28760)

* [fix](load) fix nullptr when getting memtable flush running count (apache#28942)

* [fix](load) fix nullptr when getting memtable flush running count

* style

* (enhance)(regression) Support `force_olap_table_replication_num=3` run test_insert_random_distribution_table case (apache#28903)

* [Test](Job)Add test case (apache#28481)

* [fix](doc) typo fix in auto-partition page (apache#28512)

* [feature](load) enable memtable on sink node by default (apache#28963)

* [Chore](Job)print log before task execute (apache#28962)

* [fix](hdfs) Fix HdfsFileSystem::exists_impl crash (apache#28952)

Calling hdfsGetLastExceptionRootCause without initializing ThreadLocalState
will crash. This PR modifies the condition for determining the existence of
a hdfs file, because hdfsExists will set errno to ENOENT when the file does
not exist, we can use this condition to check whether a file existence rather
than check the existence of the root cause.

* [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block (apache#27943)

* [fix](doc) typo fix in dynamic-partition page (apache#28511)

* [fix](hash join) fix column ref DCHECK failure of hash join node block mem reuse (apache#28991)

Introduced by apache#28851, after evaluating build side expr, some columns in resulting block may be referenced more than once in the same block.

e.g. coalesce(col_a, 'string') if col_a is nullable but actually contains no null values, in this case funcition coalesce will insert a new nullable column which references the original col_a.

* [opt](assert_num_rows) support filter in AssertNumRows operator and fix some explain (apache#28935)

* NEED

* Update pipeline x

* fix pipelinex compile

* [fix](parquet_reader) misused bool pointer (apache#28986)

Signed-off-by: pengyu <pengyu@selectdb.com>

* [doc](multi-catalog)add krb and some ertificates FAQs (apache#28858)

add some security docs

* [fix](planner)should save original select list item before analyze (apache#28187)

* [fix](planner)should save original select list item before analyze

* fix test case

* fix failed case

* [bug](storage) Fix gc rowset bug (apache#28979)

* [fix](mtmv)add log for resolve pending task (apache#28999)

* add lock for resolve pending task

* add lock for resolve pending task

* [improvement](nereids) Get partition related table disable nullable field and complete agg matched pattern mv rules. (apache#28973)

* [improvement] (nereids) Get partition related table disable nullable field and modify regression test, complete agg mv rules.

* make filed not null to create partition mv

* [chore](config) modify `tablet_schema_cache_recycle_interval` from 24h to 1h (apache#28980)

To prevent from too many tablet schema cache in memory and lead to performance issue when hold lock
to erase item

* [opt](scanner) optimize the number of threads of scanners (apache#28640)

1. Remove `doris_max_remote_scanner_thread_pool_thread_num`, use `doris_scanner_thread_pool_thread_num` only.
2. Set the default value `doris_scanner_thread_pool_thread_num` as `std::max(48, CpuInfo::num_cores() * 4)`

* [chore](config) modify `variant_ratio_of_defaults_as_sparse_column` from 0.95 to 1 (apache#28984)

since sparse column is not stable at present

* [feature](nereids)support partition property in nereids (apache#28982)

* [feature](Nereids) support table valued function http_stream (apache#29004)

* [feature](Load)(step2)support nereids load job schedule (apache#26356)

We will  Integrate new load job manager into  new job scheduling framework so that the insert into task can be scheduled after the broker load sql  is converted to insert into TVF(table value function) sql.

issue: apache#24221

Now support:
1. load data by tvf insert into sql, but just for simple load(columns need to be defined in the table)
2. show load stmt
- job id, label name, job state, time info
- simple progress
3. cancel load from db
4. support that enable new load through Config.enable_nereids_load
5. can replay job after restarting doris

TODO:
- support partition insert job
- support show statistics from BE
- support multiple task and collect task statistic
- support transactional task
- need add ut case

* [enhance](tablet) Reduce log in tablet meta (apache#28719)

* [fix](case) Add sync to test case (apache#29034)

* [opt](compound) Optimize by deleting the compound expr after obtaining the final result (apache#28934)

* [hotfix](jdbc catalog) fix load table and column names npe (apache#28865)

This fix is for the npe situation that occurs when FE of non-Master nodes initializes Jdbc Catalog metadata.

* [opt](Nereids) support cast bewteen numeric and  boolean in FE (apache#29006)

* [improvement](jdbc catalog) Optimize connection pool caching logic (apache#28859)

In the old caching logic, we only used jdbcurl, user, and password as cache keys. This may cause the old link to be still used when replacing the jar package, so we should concatenate all the parameters required for the connection pool as the key.

* [fix](Nereids) runtime filter push down failed (apache#28997)

project child not always NamedExpression

failed msg
```
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = class org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral cannot be cast to class org.apache.doris.nereids.trees.expressions.NamedExpression (org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral and org.apache.doris.nereids.trees.expressions.NamedExpression are in unnamed module of loader 'app')
    at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:623) ~[classes/:?]
    at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:478) ~[classes/:?]
    at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:457) ~[classes/:?]
    at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:245) ~[classes/:?]
    at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:166) ~[classes/:?]
    at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:193) ~[classes/:?]
    at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:246) ~[classes/:?]
    at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[classes/:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.lang.ClassCastException: class org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral cannot be cast to class org.apache.doris.nereids.trees.expressions.NamedExpression (org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral and org.apache.doris.nereids.trees.expressions.NamedExpression are in unnamed module of loader 'app')
    at org.apache.doris.nereids.trees.plans.physical.PhysicalSetOperation.pushDownRuntimeFilter(PhysicalSetOperation.java:178) ~[classes/:?]
    at org.apache.doris.nereids.trees.plans.physical.PhysicalHashJoin.pushDownRuntimeFilter(PhysicalHashJoin.java:229) ~[classes/:?]
    at org.apache.doris.nereids.processor.post.RuntimeFilterGenerator.pushDownRuntimeFilterCommon(RuntimeFilterGenerator.java:386) ~[classes/:?]
```

* [fix](Nereids) generating function should not folding to NullLiteral (apache#29003)

should not fold table generating function to null when do constant folding.
we should remove Generate node and replaced it by project later.

* [improve](load) add profile for WaitFlushLimitTime (apache#29013)

* [opt](Nereids) let inverted index work with top opt (apache#29000)

* [feature](Nereids) support values inline table in query (apache#28972)

* [enhencement](config) change default memtable size & loadStreamPerNode & default load parallelism (apache#28977)

We change memtable size from 200MB to 100MB to achieve smoother flush
performance. We change loadStreamPerNode from 20 to 60 to avoid stream
rpc to be the bottleneck when enable memtable_on_sink_node. We change
default s3&broker load parallelsim to make the most of CPUs on moderm
multi-core systems.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

* [improve](move-memtable) increase load_stream_flush_token_max_tasks (apache#29011)

* [feature](scan) Implement parallel scanning by dividing the tablets based on the row range (apache#28967)

* [feature](scan) parallel scann on dup/mow mode

* fix bugs

* [refactor](create tablet) default create tablet round robin  (apache#28911)

* [Bug] optimize the collection import Lists dependency packge apache#28523 (apache#28579)

* [fix](regression) Fix flaky test test_partial_update_2pc_schema_change (apache#29037)

* [Bug](security) BE download_files function exists log print sensitive msg apache#28592 (apache#28594)

* Revert "[feature](load) enable memtable on sink node by default (apache#28963)" (apache#29090)

This reverts commit 17917a0.

* [improvement](statistics)Remove retry load when load stats cache fail (apache#28904)

Remove retry load when load stats cache fail. This case usually happens when BE is down or BE OOM, retry doesn't work in these cases and may increase BE work load.

* [fix](pipeline) sort_merge should throw exception in has_next_block if got failed status (apache#29076)

Test in regression-test/suites/datatype_p0/decimalv3/test_decimalv3_overflow.groovy::249 sometimes failed when there are multiple BEs and FE process report status slowly for some reason.

explain select k1, k2, k1 * k2 from test_decimal128_overflow2 order by 1,2,3
--------------

+----------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                            |
+----------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                                                                            |
|   OUTPUT EXPRS:                                                                                                            |
|     k1[#5]                                                                                                                 |
|     k2[#6]                                                                                                                 |
|     (k1 * k2)[#7]                                                                                                          |
|   PARTITION: UNPARTITIONED                                                                                                 |
|                                                                                                                            |
|   HAS_COLO_PLAN_NODE: false                                                                                                |
|                                                                                                                            |
|   VRESULT SINK                                                                                                             |
|      MYSQL_PROTOCAL                                                                                                        |
|                                                                                                                            |
|   111:VMERGING-EXCHANGE                                                                                                    |
|      offset: 0                                                                                                             |
|                                                                                                                            |
| PLAN FRAGMENT 1                                                                                                            |
|                                                                                                                            |
|   PARTITION: HASH_PARTITIONED: k1[#0], k2[#1]                                                                              |
|                                                                                                                            |
|   HAS_COLO_PLAN_NODE: false                                                                                                |
|                                                                                                                            |
|   STREAM DATA SINK                                                                                                         |
|     EXCHANGE ID: 111                                                                                                       |
|     UNPARTITIONED                                                                                                          |
|                                                                                                                            |
|   108:VSORT                                                                                                                |
|   |  order by: k1[#5] ASC, k2[#6] ASC, (k1 * k2)[#7] ASC                                                                   |
|   |  offset: 0                                                                                                             |
|   |                                                                                                                        |
|   102:VOlapScanNode                                                                                                        |
|      TABLE: regression_test_datatype_p0_decimalv3.test_decimal128_overflow2(test_decimal128_overflow2), PREAGGREGATION: ON |
|      partitions=1/1 (test_decimal128_overflow2), tablets=8/8, tabletList=22841,22843,22845 ...                             |
|      cardinality=6, avgRowSize=0.0, numNodes=1                                                                             |
|      pushAggOp=NONE                                                                                                        |
|      projections: k1[#0], k2[#1], (k1[#0] * k2[#1])                                                                        |
|      project output tuple id: 1                                                                                            |
+----------------------------------------------------------------------------------------------------------------------------+
36 rows in set (0.03 sec)
Why failed:

Multiple BEs
Fragments 0 and 1 are MUST on different BEs
Pipeline task of VOlapScanNode which executes k1*k2 failed sets query status to cancelled
Pipeline task of VSort call try close, send Cancelled status to VMergeExchange
sort_curso did not throw exception when it meets error

* [feature](Nereids): support infer join when comapring mv (apache#28988)

* [fix](nereids) "not is null" stats estimation fix (apache#28860)

* fix not is null stats

* [opt](nereids)expr normalize after filter pushdown (apache#28743)

normalize expression after filter push down

* [Enhancement](Wal)Support dynamic wal space limit (apache#27726)

* [fix](doc) spell errors fixes query profile docs (apache#28437)

* [opt](Nerieds) add infer props to expression (apache#28953)

* [fix](read) remove logic of estimating count of rows to read in segment iterator to avoid wrong result of unique key. (apache#29109)

* [fix](paimon) read batch by doris' batch size (apache#29039)

* [fix](mtmv)add log for resolve pending task (apache#29078)

* [Refact](inverted index) refactor inverted index writer init (apache#29072)

* [fix](paimon)Remove the static attribute of the source for paimon (apache#29032)

* [fix](planner) Fix delete statement on MOR unique table (apache#28968)

add conditions for mor table removed in apache#26776

* [fix][compile]unused variable (apache#28992)

* [Fix](WorkFlow)Auto replay not work

* [Fix](WorkFlow)Auto replay not work

* [Fix](WorkFlow)Auto replay not work

---------

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Signed-off-by: pengyu <pengyu@selectdb.com>
Co-authored-by: lihangyu <15605149486@163.com>
Co-authored-by: Kang <kxiao.tiger@gmail.com>
Co-authored-by: jakevin <jakevingoo@gmail.com>
Co-authored-by: morrySnow <101034200+morrySnow@users.noreply.github.com>
Co-authored-by: xueweizhang <zxw520blue1@163.com>
Co-authored-by: zhangguoqiang <18372634969@163.com>
Co-authored-by: zhangdong <493738387@qq.com>
Co-authored-by: airborne12 <airborne08@gmail.com>
Co-authored-by: Ma1oneZhang <73012001+Ma1oneZhang@users.noreply.github.com>
Co-authored-by: ziyang zhang <zhangziyang@stu.cdut.edu.cn>
Co-authored-by: Siyang Tang <82279870+TangSiyang2001@users.noreply.github.com>
Co-authored-by: Kaijie Chen <ckj@apache.org>
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
Co-authored-by: yiguolei <676222867@qq.com>
Co-authored-by: minghong <englefly@gmail.com>
Co-authored-by: Jibing-Li <64681310+Jibing-Li@users.noreply.github.com>
Co-authored-by: Gavin Chou <gavineaglechou@gmail.com>
Co-authored-by: plat1ko <platonekosama@gmail.com>
Co-authored-by: wudongliang <46414265+DongLiang-0@users.noreply.github.com>
Co-authored-by: Nitin-Kashyap <66766227+Nitin-Kashyap@users.noreply.github.com>
Co-authored-by: yiguolei <yiguolei@gmail.com>
Co-authored-by: wangbo <wangbo@apache.org>
Co-authored-by: nanfeng <42513321+nanfeng1999@users.noreply.github.com>
Co-authored-by: Xinyi Zou <zouxinyi02@gmail.com>
Co-authored-by: qiye <jianliang5669@gmail.com>
Co-authored-by: DuRipeng <453243496@qq.com>
Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
Co-authored-by: Qi Chen <kaka11.chen@gmail.com>
Co-authored-by: Mingyu Chen <morningman@163.com>
Co-authored-by: Xin Liao <liaoxinbit@126.com>
Co-authored-by: yujun <yu.jun.reach@gmail.com>
Co-authored-by: walter <w41ter.l@gmail.com>
Co-authored-by: xy <whuxingying@163.com>
Co-authored-by: xingying01 <xingying01@corp.netease.com>
Co-authored-by: seawinde <149132972+seawinde@users.noreply.github.com>
Co-authored-by: Mryange <59914473+Mryange@users.noreply.github.com>
Co-authored-by: starocean999 <40539150+starocean999@users.noreply.github.com>
Co-authored-by: bobhan1 <bh2444151092@outlook.com>
Co-authored-by: zzzxl <33418555+zzzxl1993@users.noreply.github.com>
Co-authored-by: yangshijie <sjyang2022@zju.edu.cn>
Co-authored-by: zclllyybb <zhaochangle@selectdb.com>
Co-authored-by: amory <wangqiannan@selectdb.com>
Co-authored-by: zhiqiang <seuhezhiqiang@163.com>
Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com>
Co-authored-by: Gabriel <gabrielleebuaa@gmail.com>
Co-authored-by: htyoung <hty551020@hotmail.com>
Co-authored-by: Pxl <pxl290@qq.com>
Co-authored-by: Luwei <814383175@qq.com>
Co-authored-by: meiyi <myimeiyi@gmail.com>
Co-authored-by: Ashin Gau <AshinGau@users.noreply.github.com>
Co-authored-by: HHoflittlefish777 <77738092+HHoflittlefish777@users.noreply.github.com>
Co-authored-by: 谢健 <jianxie0@gmail.com>
Co-authored-by: XuJianxu <xjxlzlpzl@163.com>
Co-authored-by: 胥剑旭 <xujianxu@xujianxudeMacBook-Pro.local>
Co-authored-by: Yoko <53637394+TomaYoko@users.noreply.github.com>
Co-authored-by: huanghaibin <284824253@qq.com>
Co-authored-by: lw112 <131352377+felixwluo@users.noreply.github.com>
Co-authored-by: zhannngchen <48427519+zhannngchen@users.noreply.github.com>
Co-authored-by: Xiangyu Wang <dut.xiangyu@gmail.com>
Co-authored-by: wangxiangyu <wangxiangyu@360shuke.com>
Co-authored-by: AlexYue <yj976240184@gmail.com>
Co-authored-by: TengJianPing <18241664+jacktengg@users.noreply.github.com>
Co-authored-by: xzj7019 <131111794+xzj7019@users.noreply.github.com>
Co-authored-by: Dongyang Li <hello_stephen@qq.com>
Co-authored-by: stephen <hello-stephen@qq.com>
Co-authored-by: HappenLee <happenlee@hotmail.com>
Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com>
Co-authored-by: zhangstar333 <87313068+zhangstar333@users.noreply.github.com>
Co-authored-by: zhengyu <freeman.zhang1992@gmail.com>
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
Co-authored-by: HowardQin <hao.qin@esgyn.cn>
Co-authored-by: qinhao <qinhao@newland.com.cn>
Co-authored-by: deardeng <565620795@qq.com>
Co-authored-by: caiconghui <55968745+caiconghui@users.noreply.github.com>
Co-authored-by: py023 <137761033+py023@users.noreply.github.com>
Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-authored-by: Guangming Lu <71873108+LuGuangming@users.noreply.github.com>
Co-authored-by: abmdocrt <Yukang.Lian2022@gmail.com>
hello-stephen pushed a commit to hello-stephen/doris that referenced this pull request Dec 28, 2023
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants