Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improvement](statistics)Drop column stats after schema change. #39101

Merged
merged 1 commit into from
Aug 8, 2024

Conversation

Jibing-Li
Copy link
Contributor

Drop column stats after schema change. So the auto analyze could be triggered and will collect stats under new schema.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added the doing label Aug 8, 2024
@Jibing-Li Jibing-Li marked this pull request as ready for review August 8, 2024 08:14
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39378 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e71d9df9067f9e478a3c2b0aea1104c9f87ad89b, data reload: false

------ Round 1 ----------------------------------
q1	18075	4444	4395	4395
q2	2959	186	188	186
q3	11382	1146	1016	1016
q4	10508	776	670	670
q5	7959	2563	2514	2514
q6	224	139	145	139
q7	963	613	601	601
q8	9285	1920	1916	1916
q9	8814	6555	6547	6547
q10	7043	2194	2160	2160
q11	489	247	258	247
q12	392	222	225	222
q13	18405	2962	2982	2962
q14	280	233	240	233
q15	519	485	471	471
q16	511	386	395	386
q17	956	639	668	639
q18	8031	7483	7503	7483
q19	4580	1088	1020	1020
q20	662	333	353	333
q21	5423	4221	4531	4221
q22	1087	1020	1017	1017
Total cold run time: 118547 ms
Total hot run time: 39378 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4419	4302	4213	4213
q2	371	282	267	267
q3	2801	2621	2614	2614
q4	1871	1572	1599	1572
q5	5261	5251	5265	5251
q6	220	131	133	131
q7	2058	1618	1666	1618
q8	3169	3361	3272	3272
q9	8309	8377	8396	8377
q10	3362	3135	3138	3135
q11	594	475	497	475
q12	766	589	583	583
q13	16641	2990	2966	2966
q14	305	273	258	258
q15	524	475	472	472
q16	485	411	406	406
q17	1758	1480	1467	1467
q18	7655	7509	7497	7497
q19	1695	1587	1511	1511
q20	1988	1801	1811	1801
q21	5299	5014	5052	5014
q22	1081	1012	984	984
Total cold run time: 70632 ms
Total hot run time: 53884 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 202155 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e71d9df9067f9e478a3c2b0aea1104c9f87ad89b, data reload: false

query1	947	409	399	399
query2	6471	2071	1965	1965
query3	6661	206	219	206
query4	34346	23099	22994	22994
query5	4188	493	515	493
query6	296	190	179	179
query7	4581	297	296	296
query8	257	198	197	197
query9	8499	2392	2365	2365
query10	926	867	864	864
query11	16330	15073	15038	15038
query12	151	98	99	98
query13	1641	387	411	387
query14	10517	7154	7792	7154
query15	416	352	348	348
query16	7746	421	487	421
query17	1704	548	549	548
query18	2103	396	377	377
query19	262	213	202	202
query20	110	103	105	103
query21	206	117	104	104
query22	4324	4160	3957	3957
query23	33846	33480	33187	33187
query24	11861	2928	2806	2806
query25	624	383	358	358
query26	1727	154	154	154
query27	2867	282	283	282
query28	7570	2014	2019	2014
query29	1044	420	406	406
query30	306	148	146	146
query31	953	729	750	729
query32	94	55	53	53
query33	744	287	289	287
query34	934	475	487	475
query35	967	847	829	829
query36	1063	931	910	910
query37	145	85	80	80
query38	4289	4172	4134	4134
query39	1426	1359	1387	1359
query40	277	118	114	114
query41	46	44	45	44
query42	115	103	97	97
query43	526	496	490	490
query44	1229	735	759	735
query45	411	387	362	362
query46	1108	802	811	802
query47	1833	1802	1763	1763
query48	365	292	292	292
query49	1134	419	437	419
query50	810	408	413	408
query51	6852	6889	6723	6723
query52	99	97	87	87
query53	273	182	181	181
query54	917	451	440	440
query55	76	77	78	77
query56	270	244	260	244
query57	1135	1063	1061	1061
query58	235	226	249	226
query59	3085	2807	2723	2723
query60	283	263	268	263
query61	99	94	96	94
query62	824	646	678	646
query63	213	185	181	181
query64	10591	2428	1899	1899
query65	3263	3123	3170	3123
query66	1368	337	342	337
query67	15270	14881	14916	14881
query68	4616	539	552	539
query69	415	405	417	405
query70	1201	1114	1120	1114
query71	404	272	280	272
query72	18258	16812	16605	16605
query73	755	324	328	324
query74	9054	8726	8900	8726
query75	3417	2695	2669	2669
query76	2783	963	973	963
query77	450	313	309	309
query78	9469	9336	8956	8956
query79	2468	537	513	513
query80	2163	484	481	481
query81	602	230	232	230
query82	680	140	140	140
query83	270	167	151	151
query84	284	89	80	80
query85	1941	310	306	306
query86	483	263	313	263
query87	4863	4548	4466	4466
query88	3845	2490	2585	2490
query89	383	291	288	288
query90	1855	195	196	195
query91	143	119	127	119
query92	63	51	51	51
query93	1959	536	535	535
query94	948	306	288	288
query95	358	268	263	263
query96	607	279	277	277
query97	3259	3085	3059	3059
query98	227	204	200	200
query99	1769	1264	1262	1262
Total cold run time: 315841 ms
Total hot run time: 202155 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.24 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e71d9df9067f9e478a3c2b0aea1104c9f87ad89b, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.08
query5	0.49	0.49	0.50
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.56	0.49	0.49
query10	0.54	0.54	0.55
query11	0.16	0.12	0.12
query12	0.14	0.11	0.12
query13	0.62	0.60	0.58
query14	0.77	0.78	0.79
query15	0.86	0.83	0.82
query16	0.34	0.37	0.36
query17	0.97	1.02	0.99
query18	0.22	0.22	0.22
query19	1.78	1.70	1.77
query20	0.02	0.01	0.01
query21	15.40	0.76	0.65
query22	4.24	7.26	2.52
query23	18.26	1.34	1.25
query24	2.11	0.21	0.22
query25	0.15	0.09	0.07
query26	0.28	0.21	0.21
query27	0.45	0.21	0.22
query28	13.31	1.03	1.01
query29	12.61	3.42	3.31
query30	0.25	0.06	0.06
query31	2.86	0.40	0.38
query32	3.28	0.47	0.47
query33	2.86	2.90	2.90
query34	16.95	4.33	4.37
query35	4.41	4.42	4.41
query36	0.66	0.47	0.47
query37	0.18	0.15	0.16
query38	0.17	0.15	0.14
query39	0.05	0.04	0.04
query40	0.15	0.13	0.12
query41	0.10	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.53 s
Total hot run time: 31.24 s

@Jibing-Li
Copy link
Contributor Author

run cloud_p0

@Jibing-Li
Copy link
Contributor Author

run cloud_p1

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

github-actions bot commented Aug 8, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 8, 2024
Copy link
Contributor

github-actions bot commented Aug 8, 2024

PR approved by anyone and no changes requested.

@Jibing-Li Jibing-Li merged commit 9866542 into apache:master Aug 8, 2024
29 of 31 checks passed
@Jibing-Li Jibing-Li deleted the sc branch August 8, 2024 14:13
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
Drop column stats after schema change. So the auto analyze could be
triggered and will collect stats under new schema.
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
…he#39101)

Drop column stats after schema change. So the auto analyze could be
triggered and will collect stats under new schema.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Aug 15, 2024
…he#39101)

Drop column stats after schema change. So the auto analyze could be
triggered and will collect stats under new schema.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Aug 15, 2024
…he#39101)

Drop column stats after schema change. So the auto analyze could be
triggered and will collect stats under new schema.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Aug 15, 2024
…he#39101)

Drop column stats after schema change. So the auto analyze could be
triggered and will collect stats under new schema.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Aug 15, 2024
…he#39101)

Drop column stats after schema change. So the auto analyze could be
triggered and will collect stats under new schema.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Aug 15, 2024
…he#39101)

Drop column stats after schema change. So the auto analyze could be
triggered and will collect stats under new schema.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Aug 15, 2024
…he#39101)

Drop column stats after schema change. So the auto analyze could be
triggered and will collect stats under new schema.
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
Drop column stats after schema change. So the auto analyze could be
triggered and will collect stats under new schema.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants