Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[chore] Reduce range of nextId when calling advanceNextId() #40160

Merged
merged 4 commits into from
Sep 3, 2024

Conversation

gavinchou
Copy link
Contributor

The previous implementation may result in extremely large nextId.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@gavinchou
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37936 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c7836b80e470abce5ad06cb8da32d49233cb87db, data reload: false

------ Round 1 ----------------------------------
q1	17865	5185	4322	4322
q2	2012	182	177	177
q3	10472	1185	1055	1055
q4	10143	732	729	729
q5	7735	2822	2795	2795
q6	223	138	134	134
q7	970	620	591	591
q8	9334	2062	2052	2052
q9	7185	6547	6512	6512
q10	6998	2213	2224	2213
q11	468	241	246	241
q12	388	223	219	219
q13	17763	3005	3016	3005
q14	279	231	236	231
q15	512	492	486	486
q16	587	507	515	507
q17	981	634	732	634
q18	7280	6933	6829	6829
q19	1388	1052	1079	1052
q20	728	324	330	324
q21	3892	2942	2814	2814
q22	1140	1014	1033	1014
Total cold run time: 108343 ms
Total hot run time: 37936 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4410	4268	4251	4251
q2	385	276	281	276
q3	2901	2709	2685	2685
q4	1933	1687	1617	1617
q5	5367	5382	5392	5382
q6	222	133	128	128
q7	2123	1724	1768	1724
q8	3201	3363	3383	3363
q9	8375	8376	8304	8304
q10	3457	3195	3207	3195
q11	598	502	505	502
q12	779	645	631	631
q13	12589	3043	3036	3036
q14	312	268	280	268
q15	520	477	485	477
q16	606	560	543	543
q17	1769	1491	1487	1487
q18	7806	7668	7464	7464
q19	1644	1585	1585	1585
q20	2047	1827	1801	1801
q21	5567	5209	5236	5209
q22	1136	1021	1035	1021
Total cold run time: 67747 ms
Total hot run time: 54949 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188138 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c7836b80e470abce5ad06cb8da32d49233cb87db, data reload: false

query1	913	366	361	361
query2	6456	2020	1978	1978
query3	6650	226	229	226
query4	33536	23231	23277	23231
query5	4136	493	475	475
query6	250	162	161	161
query7	4588	310	295	295
query8	273	208	222	208
query9	8448	2518	2490	2490
query10	433	276	259	259
query11	17678	15091	15085	15085
query12	151	97	98	97
query13	1622	383	391	383
query14	9919	7287	6856	6856
query15	255	171	174	171
query16	8088	438	417	417
query17	1574	570	564	564
query18	2081	290	300	290
query19	265	149	147	147
query20	115	115	112	112
query21	211	109	102	102
query22	4391	4129	4112	4112
query23	34238	33314	33677	33314
query24	11236	2911	2885	2885
query25	655	401	399	399
query26	1307	167	163	163
query27	2728	291	281	281
query28	7344	2135	2124	2124
query29	870	437	418	418
query30	308	163	158	158
query31	979	760	759	759
query32	101	58	62	58
query33	772	298	297	297
query34	977	490	486	486
query35	894	745	716	716
query36	1073	906	955	906
query37	168	95	95	95
query38	3932	3807	3816	3807
query39	1469	1395	1421	1395
query40	273	121	116	116
query41	51	48	48	48
query42	117	100	100	100
query43	516	473	470	470
query44	1218	763	763	763
query45	196	170	176	170
query46	1095	752	759	752
query47	1867	1792	1794	1792
query48	388	303	304	303
query49	1083	447	437	437
query50	810	417	407	407
query51	7331	7041	7062	7041
query52	103	89	89	89
query53	265	195	192	192
query54	1092	462	473	462
query55	81	76	84	76
query56	292	264	272	264
query57	1234	1089	1087	1087
query58	247	240	246	240
query59	3024	2973	2997	2973
query60	305	287	279	279
query61	125	117	126	117
query62	847	665	670	665
query63	224	188	184	184
query64	5503	720	657	657
query65	3219	3123	3151	3123
query66	1391	336	340	336
query67	15601	15215	15360	15215
query68	4784	584	563	563
query69	472	269	291	269
query70	1181	1099	1085	1085
query71	398	283	289	283
query72	7360	4071	3805	3805
query73	755	334	342	334
query74	9128	8770	8874	8770
query75	3953	2667	2686	2667
query76	3152	979	1014	979
query77	591	301	312	301
query78	9711	9056	9105	9056
query79	1651	545	568	545
query80	925	504	505	504
query81	584	233	232	232
query82	1011	147	142	142
query83	275	154	153	153
query84	242	80	77	77
query85	1285	331	282	282
query86	379	267	250	250
query87	4386	4340	4413	4340
query88	2947	2356	2328	2328
query89	394	296	293	293
query90	1837	196	197	196
query91	127	104	108	104
query92	66	52	56	52
query93	1503	535	544	535
query94	907	297	297	297
query95	350	267	266	266
query96	605	267	271	267
query97	3173	3040	3066	3040
query98	216	215	207	207
query99	1526	1290	1277	1277
Total cold run time: 295456 ms
Total hot run time: 188138 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.43 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c7836b80e470abce5ad06cb8da32d49233cb87db, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.08
query5	0.50	0.48	0.48
query6	1.12	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.48
query10	0.55	0.54	0.55
query11	0.16	0.12	0.11
query12	0.15	0.13	0.13
query13	0.61	0.58	0.57
query14	2.04	2.07	2.07
query15	0.90	0.83	0.83
query16	0.38	0.38	0.37
query17	0.97	1.00	0.98
query18	0.21	0.20	0.20
query19	1.83	1.84	1.79
query20	0.01	0.01	0.02
query21	15.39	0.67	0.66
query22	4.39	7.08	2.34
query23	18.32	1.35	1.28
query24	2.17	0.22	0.22
query25	0.15	0.10	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.08
query28	13.26	1.02	1.01
query29	12.57	3.34	3.31
query30	0.24	0.06	0.05
query31	2.88	0.39	0.40
query32	3.26	0.49	0.48
query33	2.96	3.02	2.96
query34	17.16	4.40	4.41
query35	4.42	4.37	4.46
query36	0.65	0.47	0.47
query37	0.19	0.16	0.16
query38	0.16	0.14	0.14
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.05
Total cold run time: 111.03 s
Total hot run time: 32.43 s

w41ter
w41ter previously approved these changes Aug 30, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 30, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 30, 2024
@gavinchou
Copy link
Contributor Author

run buildall

w41ter
w41ter previously approved these changes Aug 30, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 30, 2024
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 30, 2024
@gavinchou
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38624 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b66748c40ead799378a4b79bc2148704e3df3fd7, data reload: false

------ Round 1 ----------------------------------
q1	14379	4485	4302	4302
q2	1362	183	170	170
q3	6598	1126	1156	1126
q4	4960	706	705	705
q5	3838	2838	2836	2836
q6	231	140	137	137
q7	975	624	604	604
q8	4305	2080	2064	2064
q9	6693	6552	6528	6528
q10	4599	2264	2284	2264
q11	393	250	246	246
q12	394	223	223	223
q13	17081	3077	3053	3053
q14	274	239	239	239
q15	529	495	482	482
q16	584	541	538	538
q17	985	630	737	630
q18	7568	6989	6971	6971
q19	1214	1132	1096	1096
q20	609	341	345	341
q21	3944	3251	3087	3087
q22	1109	982	1016	982
Total cold run time: 82624 ms
Total hot run time: 38624 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4377	4311	4302	4302
q2	382	276	270	270
q3	2882	2684	2695	2684
q4	1928	1708	1647	1647
q5	5444	5414	5440	5414
q6	225	134	132	132
q7	2129	1769	1768	1768
q8	3207	3379	3382	3379
q9	8487	8402	8456	8402
q10	3459	3215	3229	3215
q11	592	513	504	504
q12	806	606	588	588
q13	13576	3069	3071	3069
q14	301	286	277	277
q15	528	477	474	474
q16	580	581	572	572
q17	1790	1500	1503	1500
q18	7821	7349	7416	7349
q19	1685	1651	1527	1527
q20	2077	1819	1808	1808
q21	5471	5306	5133	5133
q22	1091	1025	1026	1025
Total cold run time: 68838 ms
Total hot run time: 55039 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189067 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b66748c40ead799378a4b79bc2148704e3df3fd7, data reload: false

query1	919	380	368	368
query2	6459	1971	1931	1931
query3	6640	207	215	207
query4	34373	23195	23288	23195
query5	4152	522	505	505
query6	261	180	174	174
query7	4585	307	296	296
query8	263	213	214	213
query9	8491	2485	2489	2485
query10	432	281	272	272
query11	17710	14988	15109	14988
query12	151	99	97	97
query13	1628	387	366	366
query14	10192	7252	7372	7252
query15	255	172	175	172
query16	8036	457	512	457
query17	1598	586	568	568
query18	2095	298	296	296
query19	332	154	154	154
query20	119	111	117	111
query21	208	111	105	105
query22	4602	4314	4111	4111
query23	34118	35222	33833	33833
query24	11566	2901	2872	2872
query25	582	403	414	403
query26	1131	163	165	163
query27	2640	286	287	286
query28	7394	2147	2118	2118
query29	671	449	432	432
query30	289	162	159	159
query31	1049	761	794	761
query32	98	58	61	58
query33	768	324	313	313
query34	912	472	492	472
query35	912	761	732	732
query36	1087	941	950	941
query37	155	90	94	90
query38	3982	3873	3887	3873
query39	1451	1401	1389	1389
query40	277	123	120	120
query41	51	50	48	48
query42	117	101	103	101
query43	513	484	458	458
query44	1253	765	758	758
query45	209	173	174	173
query46	1109	747	739	739
query47	1876	1806	1789	1789
query48	394	295	305	295
query49	1156	454	450	450
query50	835	417	422	417
query51	7182	7013	7152	7013
query52	103	91	92	91
query53	278	189	195	189
query54	948	470	475	470
query55	82	84	79	79
query56	290	271	276	271
query57	1176	1094	1065	1065
query58	251	235	238	235
query59	3099	2740	2835	2740
query60	308	284	276	276
query61	127	121	134	121
query62	837	655	647	647
query63	231	189	190	189
query64	5252	703	649	649
query65	3259	3196	3150	3150
query66	1313	342	345	342
query67	15751	15229	15245	15229
query68	4794	560	545	545
query69	430	275	280	275
query70	1192	1085	1146	1085
query71	369	283	282	282
query72	6251	4089	4020	4020
query73	761	329	335	329
query74	9551	9037	8890	8890
query75	3451	2632	2659	2632
query76	2873	1111	977	977
query77	555	334	330	330
query78	10623	9105	9061	9061
query79	1816	552	549	549
query80	985	553	504	504
query81	594	240	239	239
query82	925	147	148	147
query83	222	153	153	153
query84	234	74	80	74
query85	1311	289	277	277
query86	446	295	267	267
query87	4449	4367	4300	4300
query88	3000	2316	2303	2303
query89	426	307	301	301
query90	1768	195	197	195
query91	129	103	99	99
query92	68	57	52	52
query93	1746	551	545	545
query94	745	292	290	290
query95	356	267	266	266
query96	595	268	270	268
query97	3228	3066	3058	3058
query98	223	203	217	203
query99	1480	1257	1229	1229
Total cold run time: 296179 ms
Total hot run time: 189067 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.12 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b66748c40ead799378a4b79bc2148704e3df3fd7, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.09	0.08
query5	0.50	0.51	0.48
query6	1.13	0.73	0.74
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.55	0.49	0.49
query10	0.55	0.55	0.53
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.61	0.59	0.59
query14	2.06	2.06	2.08
query15	0.90	0.83	0.82
query16	0.37	0.36	0.36
query17	0.97	0.97	0.98
query18	0.21	0.21	0.21
query19	1.81	1.70	1.72
query20	0.01	0.00	0.01
query21	15.41	0.69	0.68
query22	4.37	6.91	2.07
query23	18.29	1.43	1.30
query24	2.16	0.22	0.23
query25	0.15	0.09	0.09
query26	0.28	0.19	0.19
query27	0.08	0.09	0.07
query28	13.21	1.03	0.99
query29	12.67	3.28	3.33
query30	0.24	0.06	0.06
query31	2.88	0.41	0.40
query32	3.24	0.51	0.48
query33	2.96	3.03	3.00
query34	17.02	4.36	4.36
query35	4.45	4.44	4.39
query36	0.67	0.48	0.47
query37	0.20	0.16	0.16
query38	0.16	0.16	0.16
query39	0.05	0.04	0.04
query40	0.16	0.13	0.12
query41	0.10	0.05	0.05
query42	0.07	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.95 s
Total hot run time: 32.12 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 30, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@gavinchou
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37849 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e4942f8cc3e207151e0d875072a6add305ffa738, data reload: false

------ Round 1 ----------------------------------
q1	17612	4409	4330	4330
q2	2740	179	182	179
q3	11619	961	1066	961
q4	10506	812	731	731
q5	7774	2798	2814	2798
q6	222	137	137	137
q7	971	619	594	594
q8	9372	2033	2093	2033
q9	7759	6503	6559	6503
q10	7095	2194	2200	2194
q11	442	240	240	240
q12	410	223	225	223
q13	18760	3055	3037	3037
q14	280	236	245	236
q15	508	479	492	479
q16	583	509	496	496
q17	997	672	671	671
q18	7329	6693	6934	6693
q19	1391	1029	1080	1029
q20	680	329	333	329
q21	3902	2942	2977	2942
q22	1091	1014	1023	1014
Total cold run time: 112043 ms
Total hot run time: 37849 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4394	4351	4317	4317
q2	373	268	271	268
q3	2870	2666	2696	2666
q4	1912	1670	1757	1670
q5	5679	5623	5818	5623
q6	234	134	135	134
q7	2184	1851	1815	1815
q8	3252	3436	3499	3436
q9	8871	8879	8833	8833
q10	3645	3421	3343	3343
q11	594	515	514	514
q12	856	663	670	663
q13	13259	3015	3051	3015
q14	313	277	266	266
q15	518	469	482	469
q16	609	581	565	565
q17	1814	1572	1523	1523
q18	7920	7873	7783	7783
q19	1736	1535	1632	1535
q20	2129	1910	1927	1910
q21	5772	5555	5498	5498
q22	1140	1100	1094	1094
Total cold run time: 70074 ms
Total hot run time: 56940 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192750 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e4942f8cc3e207151e0d875072a6add305ffa738, data reload: false

query1	1250	917	860	860
query2	6156	2006	2038	2006
query3	10627	4130	4053	4053
query4	60388	24735	23168	23168
query5	5378	503	501	501
query6	394	157	152	152
query7	5857	299	296	296
query8	282	213	229	213
query9	9015	2513	2503	2503
query10	491	272	268	268
query11	18371	15107	15468	15107
query12	149	103	106	103
query13	1574	394	385	385
query14	11025	7422	7090	7090
query15	248	183	199	183
query16	7590	496	434	434
query17	1156	587	551	551
query18	1929	300	291	291
query19	287	155	156	155
query20	123	124	116	116
query21	210	107	113	107
query22	4656	4571	4171	4171
query23	34013	33744	33441	33441
query24	5987	2871	2908	2871
query25	529	379	374	374
query26	687	157	159	157
query27	1799	281	279	279
query28	3845	2125	2099	2099
query29	679	419	399	399
query30	188	149	149	149
query31	928	759	796	759
query32	79	55	54	54
query33	459	285	289	285
query34	873	483	486	483
query35	835	718	727	718
query36	1080	965	939	939
query37	149	100	87	87
query38	3978	3926	4006	3926
query39	1496	1394	1412	1394
query40	202	116	117	116
query41	48	45	45	45
query42	112	98	94	94
query43	518	485	490	485
query44	1091	772	751	751
query45	205	172	187	172
query46	1118	722	737	722
query47	1893	1835	1864	1835
query48	370	290	294	290
query49	743	411	420	411
query50	807	431	421	421
query51	7221	7022	7046	7022
query52	99	86	87	86
query53	257	177	176	176
query54	572	441	443	441
query55	79	74	76	74
query56	268	245	256	245
query57	1213	1078	1098	1078
query58	229	217	226	217
query59	3161	2863	2888	2863
query60	283	265	268	265
query61	103	102	131	102
query62	748	653	624	624
query63	225	177	188	177
query64	2843	682	645	645
query65	3200	3166	3144	3144
query66	634	330	326	326
query67	15427	15198	15304	15198
query68	2942	586	565	565
query69	400	298	320	298
query70	1150	1045	1153	1045
query71	344	279	276	276
query72	5304	3986	4043	3986
query73	758	331	329	329
query74	9299	8850	8930	8850
query75	3374	2642	2693	2642
query76	1429	954	954	954
query77	499	322	317	317
query78	9721	9413	9064	9064
query79	1046	551	536	536
query80	701	495	511	495
query81	524	237	232	232
query82	249	143	145	143
query83	171	146	149	146
query84	267	74	72	72
query85	678	306	290	290
query86	332	294	308	294
query87	4415	4250	4309	4250
query88	2950	2332	2324	2324
query89	381	290	291	290
query90	1999	192	193	192
query91	127	99	94	94
query92	59	50	51	50
query93	1042	547	537	537
query94	683	298	306	298
query95	321	262	265	262
query96	591	273	267	267
query97	3188	3051	3055	3051
query98	222	207	196	196
query99	1730	1284	1253	1253
Total cold run time: 305325 ms
Total hot run time: 192750 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.3 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e4942f8cc3e207151e0d875072a6add305ffa738, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.04
query4	1.66	0.08	0.07
query5	0.51	0.50	0.49
query6	1.13	0.74	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.55	0.49	0.49
query10	0.55	0.54	0.53
query11	0.16	0.11	0.12
query12	0.16	0.13	0.13
query13	0.62	0.58	0.58
query14	2.04	2.05	2.10
query15	0.84	0.81	0.82
query16	0.38	0.37	0.37
query17	1.01	0.96	0.98
query18	0.22	0.20	0.20
query19	1.81	1.82	1.73
query20	0.01	0.00	0.01
query21	15.38	0.65	0.64
query22	3.68	6.83	2.37
query23	18.26	1.44	1.37
query24	2.09	0.23	0.21
query25	0.14	0.08	0.08
query26	0.26	0.18	0.19
query27	0.08	0.07	0.07
query28	13.24	1.02	0.99
query29	12.68	3.35	3.32
query30	0.24	0.05	0.05
query31	2.87	0.42	0.39
query32	3.23	0.49	0.48
query33	2.99	2.96	3.01
query34	17.03	4.41	4.31
query35	4.42	4.39	4.46
query36	0.66	0.48	0.47
query37	0.18	0.16	0.15
query38	0.16	0.15	0.14
query39	0.05	0.04	0.05
query40	0.15	0.13	0.12
query41	0.09	0.05	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.06 s
Total hot run time: 32.3 s

Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit c363830 into apache:master Sep 3, 2024
31 of 32 checks passed
gavinchou added a commit that referenced this pull request Sep 5, 2024
The previous implementation may result in extremely large nextId.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.2-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants