Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](simdjson) fix simdjson reader for read json object array when jsonroot set #38500

Merged
merged 1 commit into from
Jul 30, 2024

Conversation

amorynan
Copy link
Contributor

Proposed changes

backport : #38490
Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@amorynan
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49373 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c103d9b72d93f0f4291661443534fafdd2e8e90d, data reload: false

------ Round 1 ----------------------------------
q1	17667	4340	4315	4315
q2	2076	150	145	145
q3	10282	1883	1951	1883
q4	10448	1221	1301	1221
q5	8412	3821	3914	3821
q6	260	123	123	123
q7	2024	1594	1589	1589
q8	9523	2712	2713	2712
q9	13465	10342	10074	10074
q10	8626	3500	3496	3496
q11	420	240	240	240
q12	478	296	303	296
q13	18628	3920	4021	3920
q14	351	325	326	325
q15	513	459	451	451
q16	682	568	573	568
q17	1139	933	916	916
q18	7340	6864	6998	6864
q19	1846	1684	1593	1593
q20	535	288	302	288
q21	4454	4124	4080	4080
q22	543	453	460	453
Total cold run time: 119712 ms
Total hot run time: 49373 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4386	4333	4347	4333
q2	320	229	229	229
q3	4180	4184	4193	4184
q4	2772	2739	2748	2739
q5	7252	7099	7093	7093
q6	240	125	119	119
q7	3240	2859	2830	2830
q8	4387	4510	4541	4510
q9	17101	17233	17125	17125
q10	4393	4354	4237	4237
q11	774	679	699	679
q12	1021	833	860	833
q13	8107	3778	3805	3778
q14	456	442	429	429
q15	508	463	469	463
q16	757	684	681	681
q17	3914	3813	3897	3813
q18	8853	8819	8792	8792
q19	1726	1617	1675	1617
q20	2349	2104	2097	2097
q21	8407	8398	8455	8398
q22	1065	1003	976	976
Total cold run time: 86208 ms
Total hot run time: 79955 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.80% (8116/21471)
Line Coverage: 29.47% (66563/225887)
Region Coverage: 28.95% (34312/118519)
Branch Coverage: 24.82% (17625/71004)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c103d9b72d93f0f4291661443534fafdd2e8e90d_c103d9b72d93f0f4291661443534fafdd2e8e90d/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 204046 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c103d9b72d93f0f4291661443534fafdd2e8e90d, data reload: false

query1	925	417	376	376
query2	6551	2726	2676	2676
query3	6920	205	200	200
query4	21057	17933	18094	17933
query5	19738	6472	6490	6472
query6	293	223	222	222
query7	4154	296	298	296
query8	435	438	405	405
query9	3135	2714	2649	2649
query10	442	293	317	293
query11	11343	10686	10637	10637
query12	126	76	76	76
query13	5603	682	682	682
query14	17943	13476	13358	13358
query15	364	241	249	241
query16	6440	282	265	265
query17	1710	1438	868	868
query18	2323	406	399	399
query19	215	150	149	149
query20	81	78	77	77
query21	192	95	92	92
query22	5221	5019	5034	5019
query23	32530	31644	31954	31644
query24	6941	6481	6508	6481
query25	511	436	422	422
query26	492	166	159	159
query27	1743	289	291	289
query28	6158	2408	2329	2329
query29	2999	2779	2748	2748
query30	248	171	168	168
query31	920	749	742	742
query32	69	58	62	58
query33	399	259	252	252
query34	836	463	475	463
query35	1111	912	929	912
query36	1381	1159	1193	1159
query37	89	62	60	60
query38	3024	2960	2940	2940
query39	1377	1322	1328	1322
query40	207	96	99	96
query41	47	44	44	44
query42	87	83	84	83
query43	733	715	646	646
query44	1124	707	720	707
query45	247	238	235	235
query46	1247	973	965	965
query47	1829	1666	1679	1666
query48	1028	726	712	712
query49	622	376	366	366
query50	859	620	569	569
query51	4711	4615	4628	4615
query52	88	95	81	81
query53	454	318	324	318
query54	2644	2471	2452	2452
query55	88	81	74	74
query56	236	209	221	209
query57	1157	1012	1078	1012
query58	220	189	210	189
query59	3908	3968	3751	3751
query60	220	204	199	199
query61	99	96	98	96
query62	838	460	478	460
query63	485	332	336	332
query64	2513	1493	1475	1475
query65	3625	3529	3535	3529
query66	771	367	379	367
query67	15531	16392	16802	16392
query68	8624	629	662	629
query69	582	371	358	358
query70	1549	1626	1595	1595
query71	403	313	317	313
query72	6511	3507	3478	3478
query73	736	320	319	319
query74	6300	5889	5867	5867
query75	5398	3777	3712	3712
query76	5394	1121	1162	1121
query77	911	260	250	250
query78	12515	11668	11838	11668
query79	8755	635	632	632
query80	1210	401	398	398
query81	488	245	233	233
query82	1687	106	94	94
query83	185	132	133	132
query84	256	72	73	72
query85	877	324	341	324
query86	337	287	276	276
query87	3265	3069	3047	3047
query88	5042	2314	2300	2300
query89	385	278	285	278
query90	1963	210	210	210
query91	176	141	137	137
query92	60	55	50	50
query93	5273	540	560	540
query94	707	207	216	207
query95	1115	1056	1047	1047
query96	659	321	322	321
query97	6480	6292	6275	6275
query98	176	180	173	173
query99	2858	817	916	817
Total cold run time: 312534 ms
Total hot run time: 204046 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.51 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c103d9b72d93f0f4291661443534fafdd2e8e90d, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.04
query4	1.78	0.08	0.07
query5	0.54	0.53	0.52
query6	1.30	0.66	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.03
query9	0.52	0.47	0.48
query10	0.53	0.53	0.53
query11	0.12	0.09	0.09
query12	0.11	0.09	0.10
query13	0.62	0.61	0.61
query14	0.78	0.78	0.79
query15	0.79	0.76	0.76
query16	0.37	0.37	0.37
query17	1.01	0.98	1.00
query18	0.23	0.26	0.23
query19	1.92	1.82	1.83
query20	0.02	0.01	0.01
query21	15.58	0.54	0.56
query22	2.02	2.73	1.59
query23	17.00	0.94	1.04
query24	7.63	1.01	0.74
query25	0.34	0.11	0.07
query26	0.81	0.16	0.16
query27	0.04	0.04	0.03
query28	5.89	0.78	0.73
query29	12.68	2.32	2.32
query30	0.62	0.53	0.52
query31	2.82	0.40	0.38
query32	3.37	0.51	0.49
query33	3.05	3.08	3.06
query34	15.27	4.79	4.79
query35	4.84	4.82	4.85
query36	1.07	1.01	1.03
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.01
query40	0.15	0.14	0.15
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 104.44 s
Total hot run time: 30.51 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c103d9b72d93f0f4291661443534fafdd2e8e90d with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@xiaokang xiaokang merged commit 973f287 into apache:branch-2.0 Jul 30, 2024
22 of 25 checks passed
@wm1581066 wm1581066 added the usercase Important user case type label label Jul 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/test usercase Important user case type label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants