Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](mysql serde) Avoid core dump when converting invalid block to mysql result #28069

Merged
merged 7 commits into from
Dec 8, 2023

Conversation

zhiqiang-hhhh
Copy link
Contributor

BE will core dump if result block is invalid when we doing result serialization.
An existing bug case is described in #28030, so we add check branch to avoid BE core dump due to out of range related problem.

Copy link
Contributor

github-actions bot commented Dec 6, 2023

clang-tidy review says "All clean, LGTM! 👍"

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

yiguolei
yiguolei previously approved these changes Dec 6, 2023
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

github-actions bot commented Dec 6, 2023

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Dec 6, 2023
Copy link
Contributor

github-actions bot commented Dec 6, 2023

PR approved by anyone and no changes requested.

for (int i = 0; i < _output_vexpr_ctxs.size(); ++i) {
RETURN_IF_ERROR(arguments[i].serde->write_column_to_mysql(
*(arguments[i].column), row_buffer, row_idx, arguments[i].is_const));
for (size_t row_idx = 0; row_idx < num_rows; ++row_idx) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don’t check every row

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My fault.
Check loop is divided from for loop and placed in the front.

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.97 seconds
stream load tsv: 574 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17164379282 Bytes

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 6, 2023
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

Copy link
Contributor

github-actions bot commented Dec 6, 2023

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
Copy link
Contributor

github-actions bot commented Dec 6, 2023

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.62 seconds
stream load tsv: 571 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17164804437 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 669f61420139656320a13c458f297cbf69d2eac0, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4746	4493	4501	4493
q2	363	151	166	151
q3	1490	1258	1210	1210
q4	1122	936	913	913
q5	3182	3190	3208	3190
q6	256	128	128	128
q7	994	515	487	487
q8	2200	2243	2217	2217
q9	6717	6686	6694	6686
q10	3219	3268	3266	3266
q11	326	207	207	207
q12	357	220	210	210
q13	4582	3815	3819	3815
q14	244	213	213	213
q15	564	526	525	525
q16	449	391	392	391
q17	1020	582	542	542
q18	7797	7875	7133	7133
q19	1529	1324	1425	1324
q20	546	310	320	310
q21	3110	2680	2706	2680
q22	357	303	303	303
Total cold run time: 45170 ms
Total hot run time: 40394 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4420	4410	4436	4410
q2	269	163	172	163
q3	3548	3540	3553	3540
q4	2389	2395	2376	2376
q5	5736	5747	5739	5739
q6	240	119	124	119
q7	2382	1895	1893	1893
q8	3530	3534	3538	3534
q9	9089	9013	9015	9013
q10	3916	3999	4037	3999
q11	514	398	389	389
q12	779	595	628	595
q13	4319	3534	3591	3534
q14	282	248	260	248
q15	572	520	512	512
q16	507	473	476	473
q17	1878	1837	1874	1837
q18	8700	8454	8133	8133
q19	1753	1745	1734	1734
q20	2262	1944	1947	1944
q21	6547	6195	6195	6195
q22	500	438	421	421
Total cold run time: 64132 ms
Total hot run time: 60801 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 8b9153c7eb08ab7cfad57681e3c7b65f735be90a, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4723	4420	4479	4420
q2	367	148	158	148
q3	1466	1219	1238	1219
q4	1110	966	896	896
q5	3168	3171	3229	3171
q6	250	128	129	128
q7	1034	501	489	489
q8	2224	2214	2210	2210
q9	6698	6710	6682	6682
q10	3200	3295	3249	3249
q11	318	214	214	214
q12	367	213	211	211
q13	4560	3796	3774	3774
q14	246	210	213	210
q15	562	526	529	526
q16	442	386	391	386
q17	1020	629	541	541
q18	7575	7233	6905	6905
q19	1520	1441	1377	1377
q20	523	318	303	303
q21	3077	2667	2657	2657
q22	354	289	299	289
Total cold run time: 44804 ms
Total hot run time: 40005 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4422	4391	4385	4385
q2	268	163	166	163
q3	3535	3532	3509	3509
q4	2383	2365	2362	2362
q5	5736	5751	5766	5751
q6	239	122	122	122
q7	2383	1863	1865	1863
q8	3520	3518	3527	3518
q9	9042	9028	9022	9022
q10	3911	4015	4003	4003
q11	515	378	378	378
q12	764	610	579	579
q13	4294	3608	3554	3554
q14	296	252	262	252
q15	568	522	519	519
q16	501	494	481	481
q17	1868	1841	1872	1841
q18	8687	8421	8235	8235
q19	1742	1745	1763	1745
q20	2265	1966	1961	1961
q21	6512	6160	6174	6160
q22	497	426	421	421
Total cold run time: 63948 ms
Total hot run time: 60824 ms

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

Copy link
Contributor

github-actions bot commented Dec 6, 2023

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.67 seconds
stream load tsv: 574 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17195287338 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 06c78ae90da0f9387d8fd6c0d678c5ab4fed27bd, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4721	4479	4497	4479
q2	369	146	158	146
q3	1466	1294	1288	1288
q4	1124	944	898	898
q5	3199	3206	3195	3195
q6	247	132	129	129
q7	991	494	481	481
q8	2232	2224	2202	2202
q9	6685	6679	6710	6679
q10	3221	3267	3286	3267
q11	317	198	206	198
q12	355	212	214	212
q13	4562	3781	4737	3781
q14	243	216	221	216
q15	563	523	525	523
q16	439	380	389	380
q17	1024	599	579	579
q18	8380	7228	6977	6977
q19	1533	1403	1393	1393
q20	612	326	345	326
q21	3178	2714	2665	2665
q22	368	290	300	290
Total cold run time: 45829 ms
Total hot run time: 40304 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4426	4459	4404	4404
q2	265	163	174	163
q3	3542	3528	3506	3506
q4	2383	2369	2353	2353
q5	5744	5723	5710	5710
q6	243	121	124	121
q7	2377	1848	1884	1848
q8	3533	3528	3530	3528
q9	9022	9033	9019	9019
q10	3908	3993	4006	3993
q11	511	395	398	395
q12	769	585	599	585
q13	4300	3540	3535	3535
q14	276	258	252	252
q15	581	524	528	524
q16	488	440	444	440
q17	1860	1851	1895	1851
q18	8712	8349	8472	8349
q19	1722	1792	1752	1752
q20	2267	1938	1938	1938
q21	6538	6171	6169	6169
q22	507	423	422	422
Total cold run time: 63974 ms
Total hot run time: 60857 ms

BiteTheDDDDt
BiteTheDDDDt previously approved these changes Dec 7, 2023
Copy link
Contributor

github-actions bot commented Dec 7, 2023

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 7, 2023
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 7, 2023
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

Copy link
Contributor

github-actions bot commented Dec 7, 2023

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 9bb3606dca716cc27abdb00192b7d71abb25b359, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4705	4494	4458	4458
q2	369	146	158	146
q3	1450	1264	1271	1264
q4	1132	951	889	889
q5	3174	3176	3193	3176
q6	247	128	131	128
q7	1016	499	482	482
q8	2244	2243	2192	2192
q9	6689	6715	6663	6663
q10	3212	3282	3279	3279
q11	330	210	210	210
q12	350	217	218	217
q13	4572	3827	3788	3788
q14	241	208	231	208
q15	567	527	526	526
q16	446	393	390	390
q17	1019	658	602	602
q18	8123	7539	7734	7539
q19	1527	1368	1397	1368
q20	551	352	309	309
q21	3100	2684	2667	2667
q22	356	292	295	292
Total cold run time: 45420 ms
Total hot run time: 40793 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4422	4399	4386	4386
q2	268	161	175	161
q3	3533	3529	3531	3529
q4	2379	2373	2384	2373
q5	5741	5740	5743	5740
q6	241	122	123	122
q7	2373	1913	1875	1875
q8	3516	3534	3529	3529
q9	9095	9018	8995	8995
q10	3908	4007	4008	4007
q11	505	378	373	373
q12	759	599	607	599
q13	4282	3560	3578	3560
q14	286	265	259	259
q15	573	526	524	524
q16	500	454	462	454
q17	1885	1871	1861	1861
q18	8629	9909	8313	8313
q19	1723	1759	1771	1759
q20	2272	1948	1939	1939
q21	6523	6204	6160	6160
q22	497	440	423	423
Total cold run time: 63910 ms
Total hot run time: 60941 ms

const auto col_index = index_check_const(row_idx, col_const);
const auto row_idx_of_col_arr = index_check_const(row_idx_of_mysql, col_const);

if (column_array.size() <= row_idx_of_col_arr) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not think here should add this check, if here add , string serde should add same check too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we should add check for all serde if we can.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if u add array , do same with map too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check is removed since it is checked per row.
this pr just fixed mysql result writer, and we have other writer types, maybe they are subjected to the same problem

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.82 seconds
stream load tsv: 584 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.1 seconds inserted 10000000 Rows, about 355K ops/s
storage size: 17194601444 Bytes

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

Copy link
Contributor

github-actions bot commented Dec 7, 2023

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.38 seconds
stream load tsv: 587 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 35 seconds loaded 861443392 Bytes, about 23 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17193833700 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 6faf066455b5f69b2e00b427e3dd40f38270e048, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4711	4485	4480	4480
q2	366	150	159	150
q3	1451	1244	1254	1244
q4	1116	912	909	909
q5	3179	3224	3166	3166
q6	257	132	131	131
q7	1008	500	482	482
q8	2207	2212	2190	2190
q9	6671	6673	6646	6646
q10	3220	3269	3272	3269
q11	326	204	210	204
q12	346	210	213	210
q13	4581	3852	3832	3832
q14	246	209	214	209
q15	560	526	527	526
q16	442	383	386	383
q17	1015	594	561	561
q18	7501	7825	7057	7057
q19	1535	1383	1414	1383
q20	2143	334	339	334
q21	3116	2677	2665	2665
q22	353	282	294	282
Total cold run time: 46350 ms
Total hot run time: 40313 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4421	4399	4377	4377
q2	270	163	167	163
q3	3534	3518	3526	3518
q4	2376	2367	2358	2358
q5	5744	5736	5741	5736
q6	240	123	122	122
q7	2381	1893	1891	1891
q8	3527	3531	3522	3522
q9	9035	9004	9011	9004
q10	3915	4011	3979	3979
q11	497	377	372	372
q12	757	595	611	595
q13	4323	3523	3555	3523
q14	292	263	257	257
q15	575	518	516	516
q16	507	437	475	437
q17	1885	1843	1861	1843
q18	8688	8258	8404	8258
q19	1728	1733	1748	1733
q20	2251	1935	1923	1923
q21	6510	6188	6153	6153
q22	532	430	420	420
Total cold run time: 63988 ms
Total hot run time: 60700 ms

Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 7, 2023
Copy link
Contributor

github-actions bot commented Dec 7, 2023

PR approved by at least one committer and no changes requested.

Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Collaborator

@wm1581066 wm1581066 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit 0947bf4 into apache:master Dec 8, 2023
30 of 32 checks passed
@zhiqiang-hhhh zhiqiang-hhhh deleted the opt-ser branch December 8, 2023 14:13
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
…ysql result (apache#28069)

BE will core dump if result block is invalid when we doing result serialization.
An existing bug case is described in apache#28030, so we add check branch to avoid BE core dump due to out of range related problem.
zhiqiang-hhhh added a commit to zhiqiang-hhhh/doris that referenced this pull request Jun 24, 2024
…ysql result (apache#28069)

BE will core dump if result block is invalid when we doing result serialization.
An existing bug case is described in apache#28030, so we add check branch to avoid BE core dump due to out of range related problem.
@xiaokang xiaokang mentioned this pull request Aug 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.14-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants