Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](topN)runtime_predicate is only triggered when the column name is obtained #28315

Closed
wants to merge 3 commits into from
Closed

[fix](topN)runtime_predicate is only triggered when the column name is obtained #28315

wants to merge 3 commits into from

Conversation

TomaYoko
Copy link
Contributor

…s obtained (#27485)

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions[bot]
github-actions bot reviewed Dec 13, 2023
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/exec/vsort_node.cpp Outdated Show resolved Hide resolved
@xiaokang @github-actions
Update be/src/vec/exec/vsort_node.cpp
b0a28d1 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@xiaokang xiaokang changed the title [fix](topN)runtime_predicate is only triggered when the column name i… [fix](topN)runtime_predicate is only triggered when the column name is obtained Dec 13, 2023
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@LemonLiTree
Copy link
Collaborator

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit dfaa5d02b47376515e2cdb6b4595db389c3ccd6e, data reload: false

run tpch-sf100 query with default conf and session variables
q1	51158	31836	31850	31836
q2	378	136	137	136
q3	1554	1244	1205	1205
q4	1111	902	891	891
q5	3185	3193	3169	3169
q6	256	134	143	134
q7	1000	496	487	487
q8	2170	2226	2197	2197
q9	6942	6925	6864	6864
q10	3230	3299	3303	3299
q11	328	202	210	202
q12	346	206	210	206
q13	4558	3853	3798	3798
q14	391	314	331	314
q15	795	1325	1324	1324
q16	437	386	383	383
q17	1017	553	573	553
q18	7298	7198	6852	6852
q19	1535	1316	1421	1316
q20	575	314	292	292
q21	3117	2661	2717	2661
q22	351	282	283	282
Total cold run time: 91732 ms
Total hot run time: 68401 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	30790	31863	31806	31806
q2	279	166	179	166
q3	3684	3646	3641	3641
q4	2421	2403	2397	2397
q5	5806	5782	5788	5782
q6	239	136	129	129
q7	3047	3405	3388	3388
q8	3529	3570	3564	3564
q9	9165	9168	9119	9119
q10	3958	4072	4061	4061
q11	810	913	1359	913
q12	781	611	600	600
q13	4303	3549	3530	3530
q14	510	438	436	436
q15	1099	1357	1309	1309
q16	494	476	482	476
q17	1893	1877	1863	1863
q18	8842	8289	8234	8234
q19	1756	1769	1768	1768
q20	2879	2609	2587	2587
q21	6585	6261	6189	6189
q22	495	420	431	420
Total cold run time: 93365 ms
Total hot run time: 92378 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.7 seconds
stream load tsv: 573 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17219844554 Bytes

@LemonLiTree
Copy link
Collaborator

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.08 seconds
stream load tsv: 582 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17219816517 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit dfaa5d02b47376515e2cdb6b4595db389c3ccd6e, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4701	4456	4480	4456
q2	363	163	157	157
q3	1453	1203	1228	1203
q4	1114	854	859	854
q5	3176	3178	3205	3178
q6	244	129	129	129
q7	984	480	483	480
q8	2217	2225	2198	2198
q9	6706	6646	6654	6646
q10	3193	3282	3292	3282
q11	330	206	211	206
q12	353	211	213	211
q13	4589	3813	3832	3813
q14	242	213	212	212
q15	567	530	532	530
q16	449	386	387	386
q17	1009	579	562	562
q18	7278	7012	7076	7012
q19	1517	1377	1420	1377
q20	533	296	300	296
q21	3082	2686	2689	2686
q22	362	285	288	285
Total cold run time: 44462 ms
Total hot run time: 40159 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4438	4427	4396	4396
q2	267	161	170	161
q3	3551	3533	3543	3533
q4	2426	2390	2389	2389
q5	5722	5740	5760	5740
q6	243	121	122	121
q7	2416	1882	1900	1882
q8	3519	3525	3537	3525
q9	9035	9039	9103	9039
q10	3906	3984	4011	3984
q11	503	383	379	379
q12	772	614	601	601
q13	4309	3533	3572	3533
q14	291	250	252	250
q15	574	523	522	522
q16	526	486	484	484
q17	1884	1865	1846	1846
q18	8706	8169	8262	8169
q19	1750	1728	1742	1728
q20	2253	1940	1942	1940
q21	6583	6193	6168	6168
q22	502	434	443	434
Total cold run time: 64176 ms
Total hot run time: 60824 ms

@LemonLiTree
Copy link
Collaborator

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit dfaa5d02b47376515e2cdb6b4595db389c3ccd6e, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4692	4437	4474	4437
q2	363	171	160	160
q3	1446	1253	1278	1253
q4	1103	924	906	906
q5	3152	3186	3158	3158
q6	249	129	129	129
q7	978	496	481	481
q8	2205	2245	2190	2190
q9	6649	6626	6692	6626
q10	3199	3266	3242	3242
q11	318	206	195	195
q12	361	212	203	203
q13	4530	3813	3797	3797
q14	242	207	219	207
q15	572	519	522	519
q16	438	386	392	386
q17	996	658	574	574
q18	7185	6807	6991	6807
q19	1522	1374	1450	1374
q20	516	281	284	281
q21	3072	2617	2593	2593
q22	350	279	290	279
Total cold run time: 44138 ms
Total hot run time: 39797 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4415	4409	4378	4378
q2	265	162	171	162
q3	3535	3509	3511	3509
q4	2394	2380	2378	2378
q5	5716	5731	5730	5730
q6	241	122	122	122
q7	2384	1883	1871	1871
q8	3518	3518	3525	3518
q9	9006	9017	9058	9017
q10	3918	3966	3996	3966
q11	491	383	375	375
q12	763	596	589	589
q13	4275	3596	3574	3574
q14	286	249	254	249
q15	566	521	532	521
q16	492	480	466	466
q17	1889	1876	1861	1861
q18	8699	8258	8205	8205
q19	1713	1746	1744	1744
q20	2247	1942	1919	1919
q21	6516	6164	6127	6127
q22	491	416	423	416
Total cold run time: 63820 ms
Total hot run time: 60697 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.62 seconds
stream load tsv: 585 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17219894641 Bytes

@TomaYoko TomaYoko closed this Dec 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@TomaYoko @LemonLiTree @doris-robot @xiaokang @wm1581066 @yoko123yoko