Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix](cloud) Fix cluster status inconsistent with bes and add config disable auto #40799

Merged
merged 2 commits into from
Sep 14, 2024

Conversation

deardeng
Copy link
Contributor

@deardeng deardeng commented Sep 13, 2024

  1. add switch for auto start in cloud
  2. fix inconsistent with cluster status in be tag when add be node
  3. fix docker compose, [INVALID_ARGUMENT]cloud_instance_id in fe.conf and be.conf are not same, fe: , be: reg_cloud_instance

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@deardeng
Copy link
Contributor Author

run buildall

Copy link
Collaborator

@yujun777 yujun777 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 42685 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6fb7c29a6616ffd21407d184faadec783510c4a6, data reload: false

------ Round 1 ----------------------------------
q1	17607	7293	7216	7216
q2	2041	181	180	180
q3	10756	1221	1404	1221
q4	10179	1006	1088	1006
q5	7715	3157	3143	3143
q6	240	153	148	148
q7	1030	626	614	614
q8	9447	2024	2008	2008
q9	6711	6297	6268	6268
q10	7065	2500	2496	2496
q11	443	252	249	249
q12	402	223	220	220
q13	17756	3034	3023	3023
q14	280	247	257	247
q15	580	540	525	525
q16	555	421	411	411
q17	989	944	953	944
q18	7378	6733	6773	6733
q19	1381	1231	1219	1219
q20	603	336	318	318
q21	3904	3512	3502	3502
q22	1074	1020	994	994
Total cold run time: 108136 ms
Total hot run time: 42685 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7190	7188	7148	7148
q2	339	244	236	236
q3	3098	3087	3013	3013
q4	2080	2141	2035	2035
q5	5714	5605	5676	5605
q6	242	148	157	148
q7	2164	1755	1782	1755
q8	3361	3446	3393	3393
q9	8848	8904	8791	8791
q10	3452	3609	3566	3566
q11	596	477	488	477
q12	798	637	633	633
q13	10540	3209	3212	3209
q14	318	276	319	276
q15	586	541	548	541
q16	532	487	469	469
q17	1812	1737	1746	1737
q18	8601	7986	8019	7986
q19	1754	1754	1722	1722
q20	2132	1888	1868	1868
q21	5820	5675	5647	5647
q22	1113	1066	1059	1059
Total cold run time: 71090 ms
Total hot run time: 61314 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194747 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6fb7c29a6616ffd21407d184faadec783510c4a6, data reload: false

query1	1255	893	909	893
query2	6392	1722	1725	1722
query3	10682	3856	3877	3856
query4	54702	25212	23937	23937
query5	5099	550	526	526
query6	369	162	163	162
query7	5603	312	292	292
query8	276	225	222	222
query9	6132	2568	2581	2568
query10	407	299	288	288
query11	16134	15562	15517	15517
query12	154	110	97	97
query13	1406	396	380	380
query14	10198	6620	7244	6620
query15	211	169	185	169
query16	6908	499	475	475
query17	1189	620	603	603
query18	1868	311	304	304
query19	215	161	157	157
query20	128	113	118	113
query21	219	108	107	107
query22	4662	4406	4647	4406
query23	34781	33914	33823	33823
query24	6116	3157	3090	3090
query25	519	416	434	416
query26	608	159	162	159
query27	1577	289	292	289
query28	2786	2095	2080	2080
query29	689	428	427	427
query30	218	157	158	157
query31	972	768	802	768
query32	73	56	57	56
query33	470	301	303	301
query34	890	494	473	473
query35	876	737	737	737
query36	1019	912	929	912
query37	147	81	82	81
query38	4079	4007	3898	3898
query39	1454	1422	1559	1422
query40	214	118	116	116
query41	49	48	47	47
query42	119	95	99	95
query43	471	438	435	435
query44	1182	779	786	779
query45	202	171	173	171
query46	1095	830	814	814
query47	1894	1769	1764	1764
query48	386	291	290	290
query49	748	478	454	454
query50	920	451	447	447
query51	7062	6936	6989	6936
query52	100	88	87	87
query53	253	182	182	182
query54	561	465	462	462
query55	75	74	76	74
query56	270	262	252	252
query57	1205	1083	1087	1083
query58	219	232	238	232
query59	2738	2611	2388	2388
query60	306	275	280	275
query61	103	102	113	102
query62	790	684	679	679
query63	228	187	183	183
query64	1626	689	682	682
query65	3240	3160	3185	3160
query66	681	290	299	290
query67	15899	15355	15410	15355
query68	1978	870	876	870
query69	466	319	323	319
query70	1156	1154	1134	1134
query71	386	348	345	345
query72	6092	3386	3399	3386
query73	596	585	588	585
query74	9162	9029	8968	8968
query75	3113	3027	2990	2990
query76	1128	852	846	846
query77	541	399	405	399
query78	9467	9322	9168	9168
query79	1034	1010	528	528
query80	731	505	553	505
query81	460	238	233	233
query82	241	136	141	136
query83	169	143	145	143
query84	257	78	74	74
query85	651	287	280	280
query86	324	294	302	294
query87	4381	4234	4275	4234
query88	2878	2353	2353	2353
query89	379	289	294	289
query90	1807	188	191	188
query91	122	101	100	100
query92	58	52	50	50
query93	1221	616	612	612
query94	614	283	253	253
query95	317	255	260	255
query96	601	275	271	271
query97	3228	3120	3102	3102
query98	215	208	202	202
query99	1714	1298	1307	1298
Total cold run time: 289606 ms
Total hot run time: 194747 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.78 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6fb7c29a6616ffd21407d184faadec783510c4a6, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.04	0.05
query4	1.67	0.07	0.07
query5	0.49	0.50	0.50
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.56	0.51	0.51
query10	0.56	0.61	0.58
query11	0.16	0.13	0.12
query12	0.15	0.13	0.12
query13	0.62	0.61	0.60
query14	1.48	1.49	1.49
query15	0.92	0.88	0.88
query16	0.37	0.37	0.37
query17	1.04	1.04	1.01
query18	0.17	0.16	0.16
query19	1.95	1.80	1.82
query20	0.01	0.01	0.02
query21	15.40	0.67	0.68
query22	4.36	8.16	1.02
query23	17.76	1.28	1.28
query24	2.30	0.23	0.22
query25	0.19	0.09	0.08
query26	0.29	0.18	0.18
query27	0.08	0.07	0.08
query28	13.18	1.14	1.10
query29	12.51	3.37	3.32
query30	0.24	0.05	0.05
query31	2.86	0.42	0.41
query32	3.23	0.49	0.49
query33	3.05	3.07	3.04
query34	15.45	4.30	4.36
query35	4.35	4.37	4.34
query36	0.68	0.48	0.51
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.03
query40	0.16	0.14	0.14
query41	0.10	0.05	0.05
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 108.38 s
Total hot run time: 30.78 s

@deardeng
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 43037 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2c035bb3c5038ecb47b5bfd744c104aa4cd74fce, data reload: false

------ Round 1 ----------------------------------
q1	17614	7282	7246	7246
q2	2037	192	185	185
q3	10441	1239	1418	1239
q4	10516	1009	1035	1009
q5	7730	3179	3125	3125
q6	242	155	151	151
q7	1045	640	628	628
q8	9455	1984	2044	1984
q9	6759	6307	6365	6307
q10	7041	2579	2526	2526
q11	435	254	256	254
q12	411	226	230	226
q13	17938	3064	3050	3050
q14	291	255	262	255
q15	580	525	541	525
q16	522	433	434	433
q17	976	957	970	957
q18	7409	6870	7017	6870
q19	1382	1254	1238	1238
q20	608	337	334	334
q21	3917	3551	3524	3524
q22	1079	1023	971	971
Total cold run time: 108428 ms
Total hot run time: 43037 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7226	7266	7267	7266
q2	351	250	248	248
q3	3117	3103	3057	3057
q4	2132	2154	2046	2046
q5	5890	5787	5862	5787
q6	277	159	157	157
q7	2284	1889	1916	1889
q8	3417	3546	3481	3481
q9	9151	9248	8829	8829
q10	3505	3590	3610	3590
q11	589	483	498	483
q12	832	628	618	618
q13	13636	3224	3215	3215
q14	324	285	296	285
q15	592	533	535	533
q16	519	450	459	450
q17	1789	1774	1742	1742
q18	8593	8097	8113	8097
q19	1783	1740	1756	1740
q20	2112	1900	1896	1896
q21	5806	5563	5948	5563
q22	1097	1008	1017	1008
Total cold run time: 75022 ms
Total hot run time: 61980 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 199974 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2c035bb3c5038ecb47b5bfd744c104aa4cd74fce, data reload: false

query1	1238	886	854	854
query2	6258	1811	1789	1789
query3	10676	3909	3800	3800
query4	54238	26053	24078	24078
query5	5001	558	555	555
query6	336	173	172	172
query7	5591	312	301	301
query8	294	227	221	221
query9	5875	2582	2577	2577
query10	396	287	279	279
query11	15884	15674	15551	15551
query12	159	106	100	100
query13	1365	391	376	376
query14	10159	7330	7159	7159
query15	207	178	187	178
query16	7048	513	489	489
query17	1135	629	599	599
query18	1875	316	304	304
query19	213	159	160	159
query20	138	117	113	113
query21	218	109	108	108
query22	4758	4734	4549	4549
query23	35113	33852	33592	33592
query24	6185	3086	3068	3068
query25	518	403	416	403
query26	611	158	158	158
query27	1570	283	287	283
query28	2805	2109	2094	2094
query29	684	433	431	431
query30	226	153	162	153
query31	950	802	787	787
query32	84	59	60	59
query33	461	300	307	300
query34	893	488	469	469
query35	864	758	744	744
query36	1034	912	926	912
query37	141	83	86	83
query38	4038	4010	3926	3926
query39	1463	1415	1396	1396
query40	212	116	118	116
query41	50	49	46	46
query42	119	96	100	96
query43	502	460	435	435
query44	1160	784	763	763
query45	200	171	174	171
query46	1101	848	796	796
query47	1913	1868	1781	1781
query48	374	296	308	296
query49	733	445	435	435
query50	925	431	443	431
query51	7078	6952	6838	6838
query52	98	89	87	87
query53	254	182	184	182
query54	567	463	468	463
query55	75	77	80	77
query56	281	261	261	261
query57	1214	1098	1080	1080
query58	221	237	233	233
query59	2674	2563	2518	2518
query60	293	280	276	276
query61	128	100	101	100
query62	795	680	671	671
query63	219	188	187	187
query64	1448	669	681	669
query65	3228	3170	3169	3169
query66	682	299	294	294
query67	16039	15827	15371	15371
query68	2085	590	591	590
query69	410	278	278	278
query70	1154	1129	1077	1077
query71	352	288	295	288
query72	6195	4059	4016	4016
query73	768	325	336	325
query74	9380	9103	9070	9070
query75	3410	2728	2728	2728
query76	1350	1277	1294	1277
query77	524	319	319	319
query78	10125	9880	9555	9555
query79	1125	875	857	857
query80	1065	830	816	816
query81	535	264	315	264
query82	1246	265	263	263
query83	235	190	190	190
query84	272	106	109	106
query85	737	415	398	398
query86	341	319	292	292
query87	4372	4351	4424	4351
query88	4174	4087	4071	4071
query89	391	367	368	367
query90	1848	321	314	314
query91	123	123	123	123
query92	81	79	76	76
query93	1246	1038	1038	1038
query94	781	366	385	366
query95	501	424	431	424
query96	476	471	471	471
query97	3188	3135	3144	3135
query98	236	230	242	230
query99	1661	1288	1313	1288
Total cold run time: 293672 ms
Total hot run time: 199974 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.19 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2c035bb3c5038ecb47b5bfd744c104aa4cd74fce, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.04	0.04
query3	0.23	0.04	0.04
query4	1.69	0.07	0.07
query5	0.51	0.50	0.51
query6	1.15	0.72	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.57	0.54	0.52
query10	0.56	0.58	0.58
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.62	0.61	0.61
query14	1.48	1.46	1.46
query15	0.91	0.87	0.88
query16	0.36	0.35	0.37
query17	1.02	1.06	1.02
query18	0.17	0.16	0.16
query19	1.87	1.79	1.82
query20	0.01	0.01	0.01
query21	15.42	0.69	0.69
query22	3.81	8.05	1.28
query23	18.03	1.38	1.36
query24	2.28	0.22	0.22
query25	0.18	0.08	0.07
query26	0.29	0.18	0.18
query27	0.08	0.07	0.07
query28	13.18	1.14	1.10
query29	12.55	3.38	3.35
query30	0.24	0.05	0.05
query31	2.90	0.42	0.41
query32	3.22	0.49	0.49
query33	3.07	3.05	3.11
query34	15.42	4.33	4.37
query35	4.34	4.39	4.37
query36	0.68	0.48	0.50
query37	0.19	0.16	0.16
query38	0.16	0.14	0.14
query39	0.04	0.04	0.04
query40	0.16	0.13	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.05
Total cold run time: 108.1 s
Total hot run time: 31.19 s

@deardeng
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 42773 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2c035bb3c5038ecb47b5bfd744c104aa4cd74fce, data reload: false

------ Round 1 ----------------------------------
q1	17579	7361	7486	7361
q2	2051	182	178	178
q3	10775	1166	1365	1166
q4	10303	979	1044	979
q5	7734	3172	3134	3134
q6	241	153	151	151
q7	1037	636	612	612
q8	9456	1995	2045	1995
q9	6821	6273	6274	6273
q10	7070	2519	2529	2519
q11	435	248	247	247
q12	404	226	222	222
q13	17767	3031	3030	3030
q14	294	245	251	245
q15	580	524	536	524
q16	557	438	436	436
q17	977	931	948	931
q18	7383	6815	6674	6674
q19	1384	1238	1236	1236
q20	589	329	326	326
q21	3950	3508	3508	3508
q22	1099	1026	1028	1026
Total cold run time: 108486 ms
Total hot run time: 42773 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7165	7151	7162	7151
q2	360	249	244	244
q3	3060	3076	3024	3024
q4	2108	2120	2079	2079
q5	5750	5633	5715	5633
q6	242	156	155	155
q7	2146	1721	1775	1721
q8	3422	3440	3427	3427
q9	8782	9011	8825	8825
q10	3505	3545	3575	3545
q11	569	490	482	482
q12	837	618	646	618
q13	8607	3230	3206	3206
q14	324	279	288	279
q15	585	542	552	542
q16	515	491	487	487
q17	1799	1760	1706	1706
q18	8487	7930	7974	7930
q19	1775	1758	1768	1758
q20	2102	1899	1862	1862
q21	5891	5618	5621	5618
q22	1138	1033	1012	1012
Total cold run time: 69169 ms
Total hot run time: 61304 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 199666 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2c035bb3c5038ecb47b5bfd744c104aa4cd74fce, data reload: false

query1	1259	881	863	863
query2	6376	1794	1780	1780
query3	10685	3883	3767	3767
query4	56089	25044	24175	24175
query5	5025	548	536	536
query6	330	183	175	175
query7	5573	298	301	298
query8	291	233	219	219
query9	6015	2553	2570	2553
query10	405	292	275	275
query11	15938	15390	15573	15390
query12	153	110	103	103
query13	1439	390	390	390
query14	10589	7296	7074	7074
query15	199	191	179	179
query16	7042	490	478	478
query17	1181	613	602	602
query18	1932	316	314	314
query19	200	165	163	163
query20	125	126	116	116
query21	211	105	107	105
query22	4628	4688	4578	4578
query23	34739	33562	33857	33562
query24	6106	3087	3089	3087
query25	531	409	429	409
query26	614	154	160	154
query27	1574	283	285	283
query28	2799	2070	2065	2065
query29	703	434	430	430
query30	228	156	159	156
query31	978	767	770	767
query32	77	53	54	53
query33	457	306	306	306
query34	896	480	470	470
query35	884	758	716	716
query36	1029	894	911	894
query37	141	82	83	82
query38	4032	3950	3890	3890
query39	1444	1570	1458	1458
query40	202	119	116	116
query41	49	49	47	47
query42	123	98	98	98
query43	495	451	455	451
query44	1202	787	766	766
query45	202	168	172	168
query46	1102	827	818	818
query47	1928	1775	1903	1775
query48	375	289	283	283
query49	730	449	437	437
query50	916	440	429	429
query51	7158	6793	6896	6793
query52	95	86	86	86
query53	248	177	176	176
query54	546	459	459	459
query55	76	75	75	75
query56	286	271	274	271
query57	1205	1078	1060	1060
query58	232	233	259	233
query59	2655	2747	2539	2539
query60	291	280	266	266
query61	107	102	100	100
query62	801	685	672	672
query63	217	188	185	185
query64	1696	700	687	687
query65	3220	3206	3180	3180
query66	681	306	300	300
query67	15844	15708	15522	15522
query68	2032	875	855	855
query69	465	326	320	320
query70	1205	1152	1178	1152
query71	360	344	341	341
query72	5979	3388	3349	3349
query73	594	575	575	575
query74	9331	9062	9277	9062
query75	3062	3019	3017	3017
query76	967	838	832	832
query77	544	407	401	401
query78	9487	9258	9351	9258
query79	915	876	873	873
query80	820	821	807	807
query81	451	271	264	264
query82	264	266	264	264
query83	188	185	184	184
query84	260	105	105	105
query85	659	406	390	390
query86	326	317	320	317
query87	4441	4448	4497	4448
query88	4225	4077	4101	4077
query89	368	375	361	361
query90	1755	309	306	306
query91	121	125	122	122
query92	79	78	77	77
query93	1054	1036	1042	1036
query94	643	378	389	378
query95	453	422	425	422
query96	469	472	471	471
query97	3146	3153	3167	3153
query98	228	231	238	231
query99	1642	1324	1311	1311
Total cold run time: 292132 ms
Total hot run time: 199666 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.72 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2c035bb3c5038ecb47b5bfd744c104aa4cd74fce, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.51	0.50	0.50
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.58	0.52	0.51
query10	0.58	0.56	0.57
query11	0.15	0.12	0.13
query12	0.15	0.12	0.11
query13	0.61	0.61	0.61
query14	1.48	1.47	1.50
query15	0.90	0.89	0.88
query16	0.37	0.36	0.37
query17	1.05	1.03	1.04
query18	0.16	0.16	0.16
query19	1.94	1.85	1.75
query20	0.01	0.00	0.01
query21	15.40	0.65	0.66
query22	4.60	7.79	1.13
query23	17.92	1.25	1.27
query24	2.28	0.22	0.22
query25	0.18	0.09	0.08
query26	0.29	0.18	0.17
query27	0.08	0.08	0.08
query28	13.19	1.15	1.10
query29	12.57	3.32	3.31
query30	0.24	0.05	0.05
query31	2.87	0.42	0.41
query32	3.22	0.49	0.49
query33	3.05	3.01	3.04
query34	15.44	4.33	4.29
query35	4.33	4.36	4.33
query36	0.68	0.49	0.48
query37	0.19	0.17	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.17	0.14	0.14
query41	0.11	0.05	0.05
query42	0.06	0.06	0.05
query43	0.05	0.04	0.05
Total cold run time: 108.87 s
Total hot run time: 30.72 s

@gavinchou gavinchou changed the title [Fix](cloud) Fix cluster status inconsistent with bes and add config … [Fix](cloud) Fix cluster status inconsistent with bes and add config disable auto Sep 14, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 14, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@gavinchou gavinchou merged commit 2ceccac into apache:master Sep 14, 2024
26 of 28 checks passed
gavinchou pushed a commit that referenced this pull request Sep 14, 2024
…disable auto (#40799)

1. add switch for auto start in cloud
2. fix inconsistent with cluster status  in be tag when add be node
3. fix docker compose, `[INVALID_ARGUMENT]cloud_instance_id in fe.conf
and be.conf are not same, fe: , be: reg_cloud_instance`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. cir dev/3.0.2-merged p0_c reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants