Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](restore) Reset next version for remote table when restore #40118 #40166

Merged
merged 1 commit into from
Aug 30, 2024
Merged

[fix](restore) Reset next version for remote table when restore #40118 #40166

merged 1 commit into from
Aug 30, 2024

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented Aug 30, 2024

cherry pick from #40118

@smallx @w41ter
[fix](restore) Reset next version for remote table when restore (apac…
1bd9c40 
…he#40118)

We should reset next version to visible version + 1 for all partitions
of remote table, when restoring table that do not exist locally.

在高并发insert场景，CCR源端表的next version可能比visible
version大比较多，目标集群restore全量快照后，切换到增量binlog后，commit事务时使用的version（来自next
version）就可能比当前visible version大比较多。

此时，对于MoW表，就会出现publish
version不连续，增量binlog一直无法publish的问题。事务状态会一直是`COMMITTED`，并伴随类似ErrMsg `wait
for publishing partition 15027 version 1037597. self version: 1037627.
table 15025`。
@w41ter
Copy link
Contributor Author

w41ter commented Aug 30, 2024

run buildall

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@doris-robot
Copy link

TPC-H: Total hot run time: 49321 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1bd9c40d3835e42ed518f26572576e74d46f7507, data reload: false

------ Round 1 ----------------------------------
q1	17626	4392	4353	4353
q2	2045	195	154	154
q3	10262	1878	1930	1878
q4	10101	1237	1330	1237
q5	8515	3958	3961	3958
q6	229	125	125	125
q7	2033	1588	1612	1588
q8	9506	2743	2714	2714
q9	13982	10229	10031	10031
q10	8637	3518	3514	3514
q11	415	236	248	236
q12	511	305	302	302
q13	18349	3932	4032	3932
q14	362	320	321	320
q15	500	462	458	458
q16	670	565	586	565
q17	1142	943	887	887
q18	7265	6811	6897	6811
q19	1702	1584	1488	1488
q20	521	318	298	298
q21	4431	4147	4084	4084
q22	505	388	397	388
Total cold run time: 119309 ms
Total hot run time: 49321 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4370	4293	4279	4279
q2	320	224	233	224
q3	4179	4166	4144	4144
q4	2745	2730	2754	2730
q5	7194	7074	7144	7074
q6	241	121	120	120
q7	3249	2894	2881	2881
q8	4360	4467	4498	4467
q9	16858	16725	16882	16725
q10	4252	4227	4252	4227
q11	774	691	766	691
q12	1048	868	841	841
q13	6676	3738	3707	3707
q14	463	418	414	414
q15	510	464	459	459
q16	749	688	668	668
q17	3808	3860	3922	3860
q18	8820	8749	8756	8749
q19	1743	1693	1699	1693
q20	2401	2185	2094	2094
q21	8507	8481	8468	8468
q22	1026	929	920	920
Total cold run time: 84293 ms
Total hot run time: 79435 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 204064 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1bd9c40d3835e42ed518f26572576e74d46f7507, data reload: false

query1	927	392	412	392
query2	6532	2212	2071	2071
query3	6925	203	198	198
query4	20299	18002	17955	17955
query5	19722	6508	6534	6508
query6	295	212	228	212
query7	4165	302	312	302
query8	279	259	259	259
query9	3181	2730	2665	2665
query10	414	287	304	287
query11	11263	10855	10623	10623
query12	124	73	74	73
query13	5619	650	661	650
query14	17744	13646	13227	13227
query15	384	217	228	217
query16	6478	299	269	269
query17	1738	1434	873	873
query18	2330	414	414	414
query19	214	147	152	147
query20	81	78	82	78
query21	188	96	93	93
query22	5221	5088	5097	5088
query23	32399	31838	31878	31838
query24	6952	6489	6563	6489
query25	522	437	433	433
query26	533	167	164	164
query27	1831	296	297	296
query28	6187	2308	2265	2265
query29	2870	2851	2764	2764
query30	242	171	166	166
query31	895	763	723	723
query32	67	65	58	58
query33	402	251	248	248
query34	855	476	483	476
query35	1137	907	935	907
query36	1208	1297	1185	1185
query37	91	64	64	64
query38	3068	2984	2932	2932
query39	1380	1322	1326	1322
query40	212	95	99	95
query41	39	37	38	37
query42	84	81	85	81
query43	780	686	565	565
query44	1154	723	721	721
query45	241	230	232	230
query46	1222	952	960	952
query47	1778	1813	1720	1720
query48	987	687	672	672
query49	622	368	368	368
query50	883	662	655	655
query51	4765	4619	4704	4619
query52	88	81	84	81
query53	454	319	323	319
query54	2681	2457	2503	2457
query55	94	85	87	85
query56	240	224	214	214
query57	1150	1108	1093	1093
query58	226	202	213	202
query59	3633	3326	3314	3314
query60	218	194	227	194
query61	95	94	101	94
query62	844	530	449	449
query63	497	343	346	343
query64	2565	1523	1499	1499
query65	3603	3523	3568	3523
query66	805	383	382	382
query67	16094	17366	17543	17366
query68	8356	672	652	652
query69	569	330	324	324
query70	1606	1399	1420	1399
query71	407	304	318	304
query72	6483	3504	3510	3504
query73	734	323	325	323
query74	6301	5902	5870	5870
query75	4545	3720	3676	3676
query76	4631	1177	1195	1177
query77	583	263	262	262
query78	12371	11663	11788	11663
query79	8271	642	647	642
query80	2052	414	408	408
query81	534	232	231	231
query82	1490	105	97	97
query83	179	131	130	130
query84	255	73	71	71
query85	1418	317	316	316
query86	366	303	283	283
query87	3223	3033	3018	3018
query88	5209	2317	2299	2299
query89	349	284	285	284
query90	1746	212	202	202
query91	160	121	133	121
query92	60	55	53	53
query93	4424	587	597	587
query94	896	211	204	204
query95	1108	1074	1053	1053
query96	644	324	326	324
query97	6419	6321	6430	6321
query98	189	181	164	164
query99	2858	841	883	841
Total cold run time: 309252 ms
Total hot run time: 204064 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1bd9c40d3835e42ed518f26572576e74d46f7507, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.05
query4	1.78	0.06	0.06
query5	0.54	0.52	0.51
query6	1.27	0.60	0.60
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.52	0.48	0.48
query10	0.54	0.53	0.53
query11	0.11	0.10	0.08
query12	0.12	0.09	0.08
query13	0.61	0.61	0.60
query14	0.78	0.79	0.79
query15	0.77	0.75	0.76
query16	0.39	0.36	0.36
query17	1.03	1.00	1.01
query18	0.23	0.23	0.26
query19	1.92	1.84	1.80
query20	0.02	0.01	0.01
query21	15.47	0.55	0.57
query22	2.01	2.18	1.45
query23	16.90	0.97	1.00
query24	6.51	0.98	1.20
query25	0.38	0.08	0.06
query26	0.73	0.16	0.15
query27	0.05	0.03	0.03
query28	6.47	0.73	0.73
query29	12.66	2.34	2.31
query30	0.53	0.53	0.54
query31	2.81	0.39	0.38
query32	3.38	0.50	0.50
query33	3.11	3.03	3.10
query34	15.26	4.79	4.77
query35	4.87	4.82	4.83
query36	1.06	1.00	1.02
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.15	0.15
query41	0.07	0.02	0.02
query42	0.02	0.02	0.01
query43	0.03	0.01	0.01
Total cold run time: 103.62 s
Total hot run time: 30.53 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1bd9c40d3835e42ed518f26572576e74d46f7507 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@w41ter w41ter merged commit 03d9296 into apache:branch-2.0 Aug 30, 2024
22 of 24 checks passed
@w41ter w41ter deleted the pick_40118_to_origin_branch-2.0 branch August 30, 2024 06:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@w41ter @doris-robot @smallx